Losing attributes when exporting data with: write_dataset_to_netcdf

[Nick-Maurer]

When exporting and re-loading data using datacube's write_dataset_to_netcdf function I discovered that some attributes are lost. In particular it appears to be dictionaries. I discovered this problem when exporting some landsat water observation data, where the flag_definitions dictionary doesn't export. The export function should ideally keep the dataset intact for future use.

[robbibt]

@SpacemanPaul @Kirill888 This seems like a bug - writing out datacube data should preserve those important datacube attributes.

However, it also makes me think we need to think about relocating the write_dataset_to_netcdf function to odc.geo in the future, e.g. by adding an additional .odc.write_netcdf method to sit alongside .odc.write_cog (the existing xarray .to_netcdf method doesn't work with datacube data, and fails with serialisation errors).

[Nick-Maurer]

[robbibt]

Copying chat from Slack here @SpacemanPaul @Kirill888:

[Kirill888]

write_dataset_to_netcdf was used by ingestor, it's main purpose was to encode spatial information stored in attributes into NetCDF-CF convention. Datasets created by odc-geo should have all the spatial attributes already populated, so a simple .to_netcdf method of xarray should just work without any information loss, provided attributes were not polluted with non-serializable data, like is the case for mask bands loaded by datacube.

Real issue here is that code that populates and uses flags_definition attribute populates it with a python dictionary causing issues for .to_netcdf() function. Instead it should use some form of serialization to string and back when storing/extracting masking info. JSON would be a reasonable candidate here.

Extraction code is here:

datacube-core/datacube/utils/masking.py

Lines 226 to 240 in 9d3c6c1

	def get_flags_def(variable):
	flags = getattr(variable, FLAGS_ATTR_NAME, None)
	if flags is not None:
	return flags
	data_vars = getattr(variable, 'data_vars', None)
	if data_vars is not None:
	# Maybe we have a DataSet, not a DataArray
	for var in data_vars.values():
	flags = getattr(var, FLAGS_ATTR_NAME, None)
	if flags is not None:
	return flags
	raise ValueError('No masking variable found')

so it's just a matter of adding flags = json.loads(flags) somewhere there.

And on the create side, code here will need to encode flags_definition appropriately:

datacube-core/datacube/model/__init__.py

Lines 347 to 349 in 9d3c6c1

	def dataarray_attrs(self) -> Dict[str, Any]:
	"""This returns attributes filtered for display in a dataarray."""
	return {key: value for key, value in self.items() if key not in self.ATTR_SKIP}

[Kirill888]

but honestly, flags_definition structure is rather odd and tricky to create properly, we need a better mechanism for describing bit-mask information.

[SpacemanPaul]

but honestly, flags_definition structure is rather odd and tricky to create properly, we need a better mechanism for describing bit-mask information.

I think we're starting to get into Enhancement Proposal territory there.