NWM 1km LDAS kerchunked dataset

[rsignell-usgs]

First try at NWM 1km LDAS recipe

[rsignell-usgs]

@cisaacstern my recipe failed -- had a typo. I fixed it (pushed a change to my fork) but how do I kick off the checks again?

[cisaacstern]

@rsignell-usgs, thanks for this contribution!

Apologies for the confusion. The UI has changed, and is now presented as part of the PR checks:

A green check next to synchronize means your good to go with running a test, which is now done by calling /run {recipe_id} in a comment, where {recipe_id} is the name of your recipe given in meta.yaml. The /run command is currently only activated for members of the pangeo-forge GitHub org. I'll call a test on this recipe now.

[cisaacstern]

/run NWM-2.1-grid1km-LDAS

[pangeo-forge]

🎉 The test run of NWM-2.1-grid1km-LDAS at 7aafd19 succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

[cisaacstern]

@rsignell-usgs, the code snippet is not correct for the same reason as live ocean, because

• Add opener code snippet for kerchunk dataset test runs pangeo-forge-orchestrator#198
• Derive dataset type from recipe object (rather than hardcoding) pangeo-forge-orchestrator#199

are not yet resolved. But https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr should still be openable in normal kerchunk ways?

[rsignell-usgs]

Hmm, the JSON contains only 2 time steps instead of the expected 2920 time steps.
https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr/refrence.json

[cisaacstern]

Hmm, the JSON contains only 2 time steps instead of the expected 2920 time steps.

For the test on an unmerged PR, that is expected. We only run two time steps from /run, so that we can get a gist of the dataset without running the whole thing. The full temporal extent is run on PR merge.

Aside from the that, does the data look as expected? If so, I'll merge!

[rsignell-usgs]

Yes, the data looks as expected except for the two time steps! We really want the entire simulation, not just 2017, so I just changed my file pattern to reflect that.

[rsignell-usgs]

/run NWM-2.1-grid1km-LDAS

[cisaacstern]

pre-commit.ci autofix

[cisaacstern]

/run NWM-2.1-grid1km-LDAS

[rsignell-usgs]

@cisaacstern , I didn't see any updated run here:

import fsspec
fs = fsspec.filesystem('s3', anon=True, skip_instance_cache=True, 
        client_kwargs={'endpoint_url': 'https://ncsa.osn.xsede.org'})
 fs.info('Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr/reference.json')

returns:

{'Key': 'Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr/reference.json',
 'LastModified': datetime.datetime(2022, 12, 2, 16, 22, 22, 505000, tzinfo=tzutc()),
 'ETag': '"f1934363a439ef7f4d8d8c05a6d6ebcc"',
 'Size': 195714,
 'StorageClass': 'STANDARD',
 'type': 'file',
 'size': 195714,
 'name': 'Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr/reference.json'}

[rsignell-usgs]

Also just a note that reference.yaml intake catalog being generated is not quite right.
Instead of:

sources:
  data:
    args:
      chunks: {}
      consolidated: false
      storage_options:
        fo: Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr/reference.json
        remote_options:
          anon: true
        remote_protocol: s3
        skip_instance_cache: true
        target_options: {}
        target_protocol: s3
      urlpath: reference://
    description: ''
    driver: intake_xarray.xzarr.ZarrSource

It should be:

sources:
  data:
    args:
      chunks: {}
      urlpath: "reference://"
      consolidated: false
      storage_options:
        target_options:
          anon: true
          client_kwargs: {'endpoint_url': 'https://ncsa.osn.xsede.org'}
        fo: 's3://Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1393/NWM-2.1-grid1km-LDAS.zarr/reference.json'
        remote_options:
          anon: true
        remote_protocol: "s3"
    driver: intake_xarray.xzarr.ZarrSource
    description: ''

[cisaacstern]

@cisaacstern , I didn't see any updated run here:

Yes, thanks for the ping. Looks like something is failing silently before this job get's submitted. I'll take a look at the logs and get back to you on this.

Also just a note that reference.yaml intake catalog being generated is not quite right.

@rsignell-usgs could you make an issue or PR for this on https://github.com/pangeo-forge/pangeo-forge-recipes? Looks like this is the relevant code.

[sharkinsspatial]

/run NWM-2.1-grid1km-LDAS

[rsignell-usgs]

pre-commit.ci autofix

[rsignell-usgs]

/run NWM-2.1-grid1km-LDAS