Metadata for color and contrast limits

[tischi]

@joshmoore @will-moore

It would be great to add metadata specifications for colour and contrastLimits (aka displayRange).

Below an example of how I am opening the data with BigDataViewer and how I am setting the colours, thus showing the kind of information I would need:

OMEZarrS3Reader reader = new OMEZarrS3Reader( "https://s3.embassy.ebi.ac.uk", "us-west-2", "idr" );
SpimData image = reader.read( "zarr/v0.1/6001237.zarr" );
List< BdvStackSource< ? > > sources = BdvFunctions.show( image );
sources.get( 0 ).setColor( new ARGBType( ARGBType.rgba( 0,0,255,255 ) ) );
sources.get( 0 ).setDisplayRange( 0, 3000 );
sources.get( 1 ).setColor( new ARGBType( ARGBType.rgba( 0,255,0,255 ) ) );
sources.get( 1 ).setDisplayRange( 0, 3000 );
sources.get( 2 ).setColor( new ARGBType( ARGBType.rgba( 255,0,0,255 ) ) );
sources.get( 2 ).setDisplayRange( 0, 3000 );
sources.get( 3 ).setColor( new ARGBType( ARGBType.rgba( 255,255,255,255 ) ) );
sources.get( 3 ).setDisplayRange( 0, 3000 );

[joshmoore]

Agreed! Currently we are exporting the OMERO metadata in all of the IDR sample images. You can see what the metadata looks like in the spec: https://ngff.openmicroscopy.org/latest/#omero-md and sample code is available in ome-zarr-py for setting contrast limits, etc: https://github.com/ome/ome-zarr-py/blob/master/ome_zarr/reader.py#L293

However, we don't yet consider the "omero" namespace as official. Let's discuss on this issue how best to migrate from "omero" to proper rendering information in the spec. 👍

[tischi]

Hi @d-v-b,
Are you having display settings within the N5 openOrganelle images?

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

[d-v-b]

@tischi I haven't come to a decision for display settings that I feel super comfortable with yet. At the moment I don't put the display settings in n5 metadata -- instead, I have a large json document at the root directory of each dataset that contains consolidated metadata for all the datasets we want to display, and this is where the display settings live. But if there's a need for it I have no problem with putting the display settings in the dataset or group metadata.

My display settings are really simple: e.g. "displaySettings": {"contrastLimits": {"min": 0, "max": 0.006958113984893569}, "color": "white", "invertColormap": false}, and only openorganelle parses this datastructure. I would love to abandon this and use something more standard though.

[joshmoore]

@d-v-b: is there one per channel? Otherwise, looks like there's a good deal of overlap. From https://ngff.openmicroscopy.org/latest/#omero-md:

"channels": [                         # Array matching the c dimension size
    {
        "active": true,
        "coefficient": 1,
        "color": "0000FF",
        "family": "linear",
        "inverted": false,
        "label": "LaminB1",
        "window": {
            "end": 1500,
            "max": 65535,
            "min": 0,
            "start": 0
        }
    }
],
"rdefs": {
    "defaultT": 0,                    # First timepoint to show the user
    "defaultZ": 118,                  # First Z section to show the user
    "model": "color"                  # "color" or "greyscale"
}

[will-moore]

As you can see in the links above, in OMERO we currently use start/end for the current rendering settings, as well as min/max for the range of rendering sliders for the client to display. The min/max is usually the intensity range of the pixels in the image. Maybe this shouldn't live in 'contrast limits' but in other metadata, but it can certainly be useful,
e.g:

"window": {
            "end": 1500,
            "max": 65535,
            "min": 0,
            "start": 0
        }

[d-v-b]

@joshmoore yes there's a lot of overlap :) Switching to the omero format is on my to-do list actually, I just haven't gotten around to it yet. It would also be interesting if neuroglancer could natively read this metadata, which would allow me to delete some code from openorganelle :)

[will-moore]

Also in OMERO, we support LUTs e.g. "lut": "cool.lut", but that relies on the server knowing what LUT that refers to and is probably outside the scope of this issue.

[joshmoore]

Switching to the omero format is on my to-do list actually

👍 but is it worth simplifying the above before you do that?

It would also be interesting if neuroglancer could natively read this metadata

Do we need to ping anyone else for that? Does neuroglancer already have a format it uses?

that relies on the server knowing what LUT that refers to

Yeah, that's likely another spec with an enum and then extensible LUT storage in an array as well.

[d-v-b]

To my knowledge Neuroglancer does not use dataset metadata directly, but the developer (@jbms, who probably needs to be added to this conversation to weigh in) is pretty responsive and I think he would consider this kind of feature in-scope.

Overall I think the current display settings are fine, maybe with the exception of the active field -- what does this convey?

[jbms]

Neuroglancer supports brightness contrast adjustment via the "invlerp" UI control, which is documented here:
https://github.com/google/neuroglancer/blob/master/src/neuroglancer/sliceview/image_layer_rendering.md#invlerp-controls

Currently, as @d-v-b noted, Neuroglancer doesn't have a mechanism for these settings to be determined by the data source, but I think it could in principle be supported. Currently the default settings are just based on the full range of the data type, but they could instead be determined by the datasource with some modifications...

Another feature that could be added to Neuroglancer is a command for automatically setting the range based on the distribution of currently visible values, which might make it mostly unnecessary to even have a default.

[will-moore]

The active field is whether a channel is on/off. Since some images have many channels (e.g. over 50), you only want to turn on a small number at once.

[tischi]

At the moment I don't put the display settings in n5 metadata -- instead, I have a large json document at the root directory of each dataset that contains consolidated metadata for all the datasets we want to display, and this is where the display settings live.

@d-v-b This is very interesting, because I was actually wondering whether that's in some sense the better approach. The idea would be that the display settings could in fact be part of specifications how to visualize collections of images. In other words: there is a the "raw data" and there are certain "views" (could be several) specifying how all the raw data may be nicely visualized, including layout specification (like HTM multi well plate) and display settings.

[d-v-b]

@will-moore in that case active is a bit of a semantic outlier -- all the other display metadata describes how intensities of the image should be displayed, however active is describing something about the UI state of that application, which seems like it should be in a separate/ higher category of metadata that describes a "scene" or a "view".

@tischi consolidating metadata has a very practical advantage: I want to keep the number of HTTP requests low, so putting all the dataset metadata in one place means an application only has to make 1 GET request to get a full index of the datasets. Additionally, I wasn't fully comfortable at a conceptual level with putting display metadata on n5 arrays directly -- display metadata somehow feels very application-specific and not at all intrinsic to the data itself (unlike spatial transform information). I think your idea of multiple "views" of data with different display settings accords with this feeling. But this is just a feeling. :)

[tischi]

@d-v-b I think those are quite valid points. Maybe we could work on a spec proposal along those lines to try to see whether we can convert our feelings into something substantial?

[will-moore]

@d-v-b The way I think of active is certainly a UI state, but this UI state describes how the image should be displayed. This is also true of intensity settings which can be controlled via the UI (e.g. via sliders). Both can be saved in a similar way and used to set the initial UI state (or rendering settings) in a different viewer.
E.g. http://idr.openmicroscopy.org/webclient/img_detail/9836831/ shows the same 3 channels initially active as in https://hms-dbmi.github.io/vizarr/v0.1?source=https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836831.zarr

[d-v-b]

@will-moore good point, that makes sense
@tischi I'd be up for making a spec proposal... how do we do that?

[tischi]

@d-v-b I feel this maybe became quite related to this issue: #31
Because maybe we only want to specify display settings on the level of a "specific view" on the dataset?
Also related, does it makes sense conceptually to think of MutableImageProperties and ImmutableImageProperties? Where the DisplaySettings would be part of the MutableImageProperties? At least in my Java code I currently have such classes.

(Personally, I have the issue right now that I have only time again from 22nd of March to really work on this; if that's not too late maybe could have a zoom in that week with a group of interested people)

[joshmoore]

#23 (comment) I'd be up for making a spec proposal... how do we do that?

The steps from my POV follow, but you can skip any of the former ones based on how much feedback you want before investing more time:

1. Write down one or more proposed specifications (usually as sample JSON blocks initially) and give the pros and cons. That could be done here or image.sc. Usually I update the description of the issue as I make changes and keep a changelog, but that may be overkill.
2. Open a PR against this repository which includes a more formal spec, including MAY/MUST/SHOULD differentiations, more examples, etc.
3. Open a PR against one or more of the implementations (bioformats2raw, ome-zarr-py, zarr.js, bioformats, etc.) to show how the specification would work.
4. Produce sample data that the community can examine. (We're happy to host this if it's of use, e.g. https://s3.embassy.ebi.ac.uk/idr/share/2021-Q1-3D/)