Update Zarr Encoding Spec for v3

[tasansal]

Hi, since zarr v3 is around the corner, do we have any plans to update the zarr encoding?

Some neat things that may need modification:

• dimension names are a part of the array metadata spec; ie we don't need to write them to zattrs.
• support irregular chunking in Zarr
• storage transformers (sharding etc)

Cheers!

[jhamman]

Hi @tasansal - Thanks for opening this conversation.

We're of course very excited V3 in Xarray! Blocking any of the new features enabled by V3 is a spec-compliant implementation in Zarr-Python. I'm actually leading a working group on bringing Zarr-Python up to spec: zarr-developers/zarr-python#1480. We hope to have full support for the core of V3 this year.

After that, there will need to be some work on Zarr-Python for things like irregular chunk and sharding support and, as you mention, dimension names in Xarray.

If you are interested in helping move any of these forward, let me know and I can help point you in the right direction.

[tasansal]

Sounds great! I am definitely interested in moving it forward.

I can take a stab at the dimension names.

Found this, will continue there: zarr-developers/zarr-python#1516

Besides that I can also help with Xarray too. At a high level, what are tasks needed to update the Xarray Zarr encoding?

Some immediate/open questions:

1. Does Xarray need backwards compat for v2 zarrs?
2. Xarray has rich metadata. Would be nice to use ZEP0004 like in here ZEP0004 Review - Zarr Conventions zarr-developers/zarr-specs#262.
3. Dimension names can obviously go to the zarr.json
4. Would the sharding spec, uneven chunks etc just be part of the variable encoding? I'm not sure if this is supported when writing Xarray Datasets to HDF5 files (since it supports it)

[jhamman]

I'll chime in on zarr-developers/zarr-python#1516

To your questions -

1. Does Xarray need backwards compat for v2 zarrs?

Yes. There are PBs of V2 Zarr out there today so read/write support for v2 will be something we maintain for quite a while.

Xarray has rich metadata. Would be nice to use ZEP0004 like in here zarr-developers/zarr-specs#262.

+1, today we mostly follow CF-like conventions. You could imagine support here for geo-zarr or other conventions as well.

Dimension names can obviously go to the zarr.json

+1, this will be such a nice improvement.

Would the sharding spec, uneven chunks etc just be part of the variable encoding? I'm not sure if this is supported when writing Xarray Datasets to HDF5 files (since it supports it)

Each backend accepts its own set of encoding. If Zarr-Python support sharding or variable chunk sizes, we'll be able to wire those through.