How to organize ephys+opto-projected images

[matham]

We are running slice based e-phys with multi-electrode arrays that are simultaneously stimulated optogenetically. Using a projector through a set of lenses, we stimulate various areas with time varying intensity. I'm not sure how to store the resulting data in NWB or how to think about it.

---

The kind of data we have is a set of shapes on the screen (e.g. polygon/circle/ellispse) and a set of functions that determines the intensity of each shape for every frame projected (at 120Hz). Ideally, we'd have an image series where you can provide a series of masks (ordered in the z-axis) along with a 2D array for each mask, indicating the time-varying RGB value of the mask area.

Maybe I can create an ImageSeries for each mask, and simply use a TimeSeries indicating the RGB values. But then how will other people understand what this is? Or is that not something to worry about?

We also have an alignment, where for every projected frame we have an associated index in the e-phys data when this frame was projected. We have custom hardware digital IO passed between the projection and e-phys systems for this. I saw alignment code somewhere, but can't find it anymore. But, for the alignment indexes and other associated logged digital IO, can I just create a bunch of TimeSeries and use those for this data and rely on the description field for people to understand what the data means? Or do I need to create an extension where everything is laid out "officially"? And how is that different than using the description for documentation?

Side question, I couldn't figure out from the docs, what's the difference between epoch and trial? And again, do I use whichever I like and not worry about what it may mean to other people or tools around NWB as long as people in our lab understand?

---

Similarly, we have lightsheet microscope data from cleared whole-brains. We have/will have an alignment of the brain to the Allen atlas. And then a bunch of regions in the 3D space along with cell counts for each region. We could simply link to the whole brain tiff file from the NWB file. But I don't see structures for the region definitions and cell counts for each. Is there some suggestions on how to approach this?

---

If we made an NDX extension for the projected (and cleared brain) data, how would this work with tools that e.g. know how to read normal ImageSeries? Would they all need to implement code to visualize our data? Or should I only worry about making a schema that our lab can read and not consider outside lab who may access the data when shared? What are the expectations around NWB?

[CodyCBakerPhD]

Wow, that's a lot of questions!

I'll let others handle the broad philosophy points and tackle some of the technical ones

Similarly, we have lightsheet microscope data from cleared whole-brains. We have/will have an alignment of the brain to the Allen atlas. And then a bunch of regions in the 3D space along with cell counts for each region. We could simply link to the whole brain tiff file from the NWB file.

Sounds like standard ophys types are suitable here (Device -> ImagingPlane -> OnePhotonSeries) - yes, we currently refer to lightsheet as a OnePhotonSeries, but we're aware it's not identical to laser scanning approaches, so just ignore the laser-based metadata like the scan line rate

But I don't see structures for the region definitions and cell counts for each. Is there some suggestions on how to approach this?

Because they are not hard-coded areas of the schema, instead they would be free-form columns added to your PlaneSegmentation table, which should be generic enough to allow to associate any aspects you want to segmented regions. Suggested column names for consistency with ecephys side would be x,y,z in Allen coordinates, location as the Allen brain area abbreviation, and cell_type/cell_counts

We are running slice based e-phys with multi-electrode arrays that are simultaneously stimulated optogenetically.

Keep in mind if you're planning to submit to DANDI, include subject metadata (species, subject_id, age or date_of_birth, sex) as they would have been at the time the slice was collected

Using a projector through a set of lenses, we stimulate various areas with time varying intensity. I'm not sure how to store the resulting data in NWB or how to think about it.

This sounds rather like holographic stimulation, is that correct? In that case there is an extension that has been made for this and will be entering official review by the Technical Advisory Board soon: https://github.com/catalystneuro/ndx-patterned-ogen

Check it out and let us know if it fits your needs - the odd thing I see is the bit about 'RGB' values?

We also have an alignment, where for every projected frame we have an associated index in the e-phys data when this frame was projected. We have custom hardware digital IO passed between the projection and e-phys systems for this. I saw alignment code somewhere, but can't find it anymore. But, for the alignment indexes and other associated logged digital IO, can I just create a bunch of TimeSeries and use those for this data and rely on the description field for people to understand what the data means? Or do I need to create an extension where everything is laid out "officially"? And how is that different than using the description for documentation?

All data in the NWB file should be aligned to the session_start_time of the file; this aspect is very important for re-use, so every effort should be undertaken to do so

Whether or not you include the original TTL pulses and whatnot in some capacity is up to you; sometimes it depends on how complicated the rig is such that the act of aligning might be considered a part of replicating the findings of the study

I'd usually add such alignment signals as ElectricalSeries with links to the electrodes table shared with electrodes collecting neural data, pointing to a different Device (common cases include NIDQ or Intan boards, for example)

Side question, I couldn't figure out from the docs, what's the difference between epoch and trial?

In general,

epochs occur over a longer period of time and represent qualitative shifts to the structure of the experiment. For in vivo examples this can include a mouse being moved to a different type of maze, being presented different categories of visual stimuli (this example from Allen Institute Visual Coding dataset has a switch between natural movies and drifting gratings, among others). It's somewhat uncommon to see an epochs table with more than say a dozen rows

trials are shorter and are usually repeated much more frequently, and also typically have a bunch of parameters that change with each trial. For example, this file from the International Brain Lab had quite a lot of values that changed over each individual trial

And again, do I use whichever I like and not worry about what it may mean to other people or tools around NWB as long as people in our lab understand?

Referring to the NIH Data Sharing Policy

in particular, this section

"""
What Scientific Data Need to Be Shared?

Scientific data are defined as the recorded factual material commonly accepted in the scientific community as of sufficient quality to validate and replicate research findings, ...
"""

it is therefore quite important to give consideration to how you represent your data so that others are able to understand it in order to validate and replicate the findings. Especially reviewers of the eventual publication associated with the dataset

[matham]

Thank you for the explanations and quick response to all questions 😊

Because they are not hard-coded areas of the schema, instead they would be free-form columns added to your PlaneSegmentation table, which should be generic enough to allow to associate any aspects you want to segmented regions. Suggested column names for consistency with ecephys side would be x,y,z in Allen coordinates, location as the Allen brain area abbreviation, and cell_type/cell_counts

OnePhotonSeries should indeed work for our lightsheet data. If I understood correctly, PlaneSegmentation stores each ROI in a row, so I can add columns with data for each ROI. This should work for an overall cell count per region. But what if I also wanted to store the x, y, z, intensity etc. for each cell in all ROIs? That wouldn't quite work with columns. I saw the Fluorescence and df/f table that can do this kind of thing, but in time dimension, per ROI(s). I'm assuming a custom schema?

holographic stimulation,
Check it out and let us know if it fits your needs - the odd thing I see is the bit about 'RGB' values?

It is similar. You can see our system diagram here. The reason for RGB is that each pixel of the projector (that projects on the slice) can output any RGB value. But, at any time we use a specific color, e.g. blue, along with filters so the slice will only see the desired wavelengths. And we change out these filters (dichroic) as needed.

ndx-patterned-ogen fits our needs, except for the time dimension. I'm assuming we'd use a OptogeneticStimulus2DPattern with a mask over the full projected area (I'm not quite sure why this mask is needed, other than to define the overall tissue area!?). Next we use a OptogeneticStimulusTarget and add all the ROIs (polygons/circles etc) illuminated by the projector.

Here's where it'd diverge though. I could use PatternedOptogeneticStimulusTable, and for every frame projected, for every ROI, and for every color (e.g if stimulating red/green simultaneously), I can add an interval indicating the start and end time of the frame (it's a 120Hz - 1.44kHz) along with the intensity (watt used by LED). But that loses some of the context. It would be nicer to instead use e.g. a TimeSeries whose timestamps are start frame and data is the RGB intensity for each ROI, along with frame end time.

All data in the NWB file should be aligned to the session_start_time of the file; this aspect is very important for re-use, so every effort should be undertaken to do so

This kinda brings me to something I wondered. Everything in NWB seems to be using timestamps. This makes sense in e.g. videos and behavior. For ephys data where we are typically sampling regularly e.g. at 20kHz it makes me feel slightly uncomfortable to rely on timestamps instead of indices (integer multiples of sampling rate) that can be converted to timestamps on demand. By using timestamps, it's almost an implicit assumption that the data could have been sampled irregularly and you'd need to inspect the timestamps to figure out if it's regular.

Even to synchronize multiple systems with different clocks. If each system samples regularly at a given rate (which is typical), even when the system clocks drift, wouldn't it still be better to compute the clock drift (as a line) and adjust the frequency of the other systems to the common system? Avoiding having to use timestamps, except when really needed when the data is sampled irregularly?

Also, once we lose the assumption of sampling regularity, everything gets more expensive. To find a sample given its time value, if it's regularly sampled, you just need to multiply the rate by time. But now, you'd need to search through the timestamps list. The continuity parameter can be used to mark this. But it would be nice if sampling regularity would be a TimeSeries parameter so we can know at a glance.

[CodyCBakerPhD]

This kinda brings me to something I wondered. Everything in NWB seems to be using timestamps. This makes sense in e.g. videos and behavior. For ephys data where we are typically sampling regularly e.g. at 20kHz it makes me feel slightly uncomfortable to rely on timestamps instead of indices (integer multiples of sampling rate) that can be converted to timestamps on demand. By using timestamps, it's almost an implicit assumption that the data could have been sampled irregularly and you'd need to inspect the timestamps to figure out if it's regular.

TimeSeries and all descendants of it can take either a vector of timestamps or a two scalars starting_time and rate if the sampling frequency is truly regular. That should satisfy all your various concerns

Even to synchronize multiple systems with different clocks. If each system samples regularly at a given rate (which is typical), even when the system clocks drift, wouldn't it still be better to compute the clock drift (as a line) and adjust the frequency of the other systems to the common system?

If by 'adjust the frequency of the other systems to the common system` you mean define the sampling rate of secondary systems as the corrected rate relative to the common clock, then yes that is exactly what we mean by temporal alignment in the NWB file

But what if I also wanted to store the x, y, z, intensity etc. for each cell in all ROIs? That wouldn't quite work with columns.

How so? One column for x, another for y, z, intensity, etc. if they truly vary over ROI but not over time

Or one column for position where every entry is a length 3 list/array (would need to define index=True when defining this column)

A case where it would break down is if the intensity of an ROI itself changes over time, but surely the position would be constant over the experiment?

I saw the Fluorescence and df/f table that can do this kind of thing, but in time dimension, per ROI(s). I'm assuming a custom schema?

Fluorescence and df/f are descendants of TimeSeries, not tables per se. Their data is a 2D array of (number of imaging frames) x (number of ROIs) and values being some function of the weighted summation over the pixel or image masks for each ROI

Here's where it'd diverge though. I could use PatternedOptogeneticStimulusTable, and for every frame projected, for every ROI, and for every color (e.g if stimulating red/green simultaneously), I can add an interval indicating the start and end time of the frame (it's a 120Hz - 1.44kHz) along with the intensity (watt used by LED). But that loses some of the context. It would be nicer to instead use e.g. a TimeSeries whose timestamps are start frame and data is the RGB intensity for each ROI, along with frame end time.

Given how close it is, I'd recommend raising an issue on that particular repo to consult the developers of that extension; my intuition says that the PatternedOptogeneticStimulusTable is generic enough to allow this kind of customization, or that there are other nuances to their hierarchy that can allow other time-dependent notions of intensity (if that's the limiting factor)

[matham]

TimeSeries and all descendants of it can take either a vector of timestamps or a two scalars starting_time and rate if the sampling frequency is truly regular. That should satisfy all your various concerns

Ahh I thought timestamps was necessary.

How so? One column for x, another for y, z, intensity, etc. if they truly vary over ROI but not over time

I may be missing something obvious. But, say you have R ROIs and C columns. Then the table is RxC, right!? Because each ROI is a one row and is not repeated (repeating the same ROI by "adding" it multiple times presumably is not allowed).

Then, say in ROI-1 we counted 5 cells, and in ROI-2 25 cells. Then we'd have 25 * 4 (x, y, z, intensity) additional columns. But for ROI-1, only 20 columns will be used, which feels a little weird. I suppose I can add a column that indicates the number of cells to make it clear. But this is where I know I can literally save it that way, but if I do that I'm not sure other NWB tools will understand how to read it.

On the other hand, if I can repeat the ROI, then I can just repeat the ROI for each cell in the ROI, which feels natural. But I don't see how to do that in the API without potentially confusing tools that would assume each ROI is only repeated once.

Given how close it is, I'd recommend raising an issue on that particular repo to consult the developers of that extension

Will do.

[CodyCBakerPhD]

I may be missing something obvious. But, say you have R ROIs and C columns. Then the table is RxC, right!?

The PlaneSegmentation table would have $R$ rows, one for each ROI (Region Of Interest). By default it has only two columns: the ID associated with the ROI (must be unique over the table; by default it is just the index of the row) and a 'mask' that defines how the region is related to pixels of the imaging space (either an array of identical shape to the imaging or just a list of pixels at the non-zero values; usually the latter because they can tend to be sparse)

You can add however many extra columns you want to the table to associate additional information with each ROI

Because each ROI is a one row and is not repeated (repeating the same ROI by "adding" it multiple times presumably is not allowed).

What is the use case for information you would want to associate multiple times to a single ROI?

There are several approaches that could be taken depending...

(i) You could set the values of each entry in a particular row and column to be a vector of values

(ii) As mentioned above, each ROI must have a unique ID - though there is total ambiguity in overlapping vs. non-overlapping masks, so you could literally define the same mask for multiple ROI IDs and that would be acceptable if there is a good reason to use that representation

Then, say in ROI-1 we counted 5 cells, and in ROI-2 25 cells.

I would add a column called cell_count whose value in each 'macro-ROI' is how you indicate (5, 25, and so on)

I will note that I'm referring to these as 'macro-ROI' in the off chance that you do any individual cell segmentation, which is more usually the norm for this kind of thing in microscopy (using tools such as Caiman, Suite2p, etc). But no one said you couldn't have multiple PlaneSegmengation tables, so if you do both types of segmentation then I'd have two separate tables and name one appropriately to indicate it's a much coarser level of resolution

Then we'd have 25 * 4 (x, y, z, intensity) additional columns. But for ROI-1, only 20 columns will be used, which feels a little weird. I suppose I can add a column that indicates the number of cells to make it clear. But this is where I know I can literally save it that way, but if I do that I'm not sure other NWB tools will understand how to read it.

Then, say in ROI-1 we counted 5 cells, and in ROI-2 25 cells. Then we'd have 25 * 4 (x, y, z, intensity) additional columns.

The table you describe would have 2 rows for IDs 'ROI-1' and 'ROI-2'. Adding columns for x, y, z, and intensity we would have a total of 6 columns

All for a total of 2 * 6 = 12 entries in the table

But for ROI-1, only 20 columns will be used, which feels a little weird.

It seems you may, in that calculation, be alluding to a multi-segmentation approach that I mention above. Specifically, you seem to have ROI's within ROI's? Such that 5 of the cells in ROI-2 are also within ROI-1 (ROI-1 being a strict subset of ROI-2 to produce your count of 20?)

Notice I've been careful not to refer to what NWB PlaneSegmentation encodes as being 'neurons' or even whole 'cells' - that classification is quite strong and the generic notion of an ROI can be any spatial partition of the imaging pixel space (I've seen dendritic branches, axonal tracts, neuropil, and glia all be considered)

So if you did have a second PlaneSegmentation table that was on the 'cellular' level (or I'd just call it a finer grain) and this table only spanned the unique set of individually segmented cells, and would be a 20 x 6 table

Also note that if classification of these cells is for use only with the ndx-patterened-optogenetics type of data, then that extension also harnesses some of it's own data structures to represent this - in particular, certain fields will specify a list of ROIs (a 'macro-region' if you will) that is stimulated even though the individual ROIs are not duplicated over the source PlaneSegmentation table

[matham]

Thanks for your patience! I think I see now what I missed. I assumed the PlaneSegmengation table (and indeed all tables) is like numpy arrays in the sense each entry of the table must be a scalar or special object. I didn't realize each entry can have a differently sized array. I understand now what you meant about adding columns!