Augment sarscov2 community builds listing and include datasets in sequence search

[eharkins]

see commit messages for now, just putting this up for review since I came to a good breaking point even though there is more to do.

[eharkins]

@jameshadfield Is this clear enough or should we explicitly indicate somehow whether datasets in sequence search results are from Nextstrain or the community?

(edit: so far you can only get community datasets in search results like this by changing the search results jsonUrl locally, since i havent uploaded this to data.nextstrain.org yet)

[emmahodcroft]

Just an idea, but could we port the same symbols you used for the map view of builds instead of showing the 'folded corner page' symbol? That would also indicate how 'focused' the build in, which might also help people decide which one might be of most help, depending on why/what they're looking for!

[eharkins]

Good idea @emmahodcroft. I think we could do this although the build level isn't immediately accessible in the strain map created for the search results since it's currently just strain -> list of dataset indexes, but we could match up those datasets names with entries from the build list that has the levels in it if we want to go with those icons.

[eharkins]

@jameshadfield this should be functional so I marked ready for review. However, before we merge, I still need to change a few URLs to data.nextstrain.org from staging.nextstrain.org.

Just realized I also need to do this (as we discussed last week):
Go through the sars-cov-2 build catalogue and make sure all entries have URLs that point to an auspice server so that we can include all of them and not filter them as we currently are. I believe you mentioned wanting to include the core nextstrain builds in the "augmenting" process so that we can get metadata for those on the map as well. Just noting to myself here that in doing that I'll need to make sure that doesn't result in duplicate datasets in the strain map, since I imagine the nextstrain builds in the catalogue are (all?) also in the s3 bucket that we have been using for search results.

[emmahodcroft]

Yes, good points @eharkins. I think this is an 'optional extra' rather than a dealbreaker - so if it's a lot of work do to this matching & maintain it - no worries!

[jameshadfield]

Thanks @eharkins -- this is looking really good.

I think there's a number of optional extras which I'd be in favor of making issues for rather than solving in this PR:

• augmenting core datasets (as mentioned above)
• changing the search icons, if possible
• cleaning up the displayed URL in search so that we don't display the query part

The functionality and code looks good - tested it a few different ways and it handles problems well (including if the fetch fails).

Things to add / fix before merge:

• dev docs to explain that the manually curated YAML is being augmented, and that this is happening via GitHub actions (it is, right?)
• brief dev docs to explain what extra information's added and what's needed (backend + frontend) to add more here
• URLs to change (as you noted)
• The augmented YAML currently includes “block chomping indicators” (YAML is crazy when it comes to multi-line strings) which should be removed. For instance:

  - url: >-
      http://auspice.finlaymagui.re/ncov/north-america/canada/ontario?f_division=Ontario
    name: Ontario, Canada
    geo: ontario
    level: division

• ./scripts/collect-pathogen-resources.js --pathogen seasonal-flu fails:

Writing search data to file  ./data/search_seasonal-flu.json
(node:18117) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at validateString (internal/validators.js:124:11)
    at Object.basename (path.js:1156:5)
    at main (/Users/enigma/projects/nextstrain/nextstrain.org/scripts/collect-pathogen-resources.js:96:43)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
...

P.S. I hadn't realised that Nextstrain groups datasets were part of this YAML - it's awesome that they are 😄

[eharkins]

Thanks @jameshadfield! Cleaned up the commit history a bit and took care of these three:

URLs to change (as you noted)

The augmented YAML currently includes “block chomping indicators” (YAML is crazy when it comes to multi-line strings) which should be removed.

gen-resources.js --pathogen seasonal-flu fails

I also ran nextstrain remote upload s3://nextstrain-data data/allSARS-CoV-2Builds.augmented.yaml so that that file exists on s3 for when we merge and we dont have to wait for a GH action to run (and so that it works on the heroku dev app before we merge).

Docs PR to address dev docs concerns: nextstrain/docs.nextstrain.org#44

[jameshadfield]

Closing this as all commits are in #271 - Eli please reopen if i'm missing something!