Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data read #85

Draft
wants to merge 163 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
163 commits
Select commit Hold shift + click to select a range
1e1f041
Define base classes for reading
oruebel Aug 31, 2024
e58f50f
First draft of reading datasets and attributes
oruebel Aug 31, 2024
250a66b
Split reading of attribute and dataset to separate functions
oruebel Aug 31, 2024
e1bfa2e
Remove debug print
oruebel Aug 31, 2024
4aedefd
Add functions to construct ReadDatasetWrapper and ReadAttributeWrappe…
oruebel Aug 31, 2024
4dae171
Fix formatting
oruebel Aug 31, 2024
140809e
Add test for using the ReadDatasetWrapper
oruebel Aug 31, 2024
4d7b1ea
Start refactor containers for read
oruebel Aug 31, 2024
2db4bfe
Fix format
oruebel Aug 31, 2024
b2959d2
Read ElectricalSeries.data example working
oruebel Sep 1, 2024
37d30d3
Revert change to tests used for debugging
oruebel Sep 1, 2024
6cfe416
Fix codespell issue
oruebel Sep 1, 2024
3d86aff
Move read example to an example file
oruebel Sep 1, 2024
3fc1d7a
Fix bug in Container::create
oruebel Sep 1, 2024
b09a4fa
Add example for data read
oruebel Sep 1, 2024
970f4c8
Fix spelling error
oruebel Sep 1, 2024
26fbae9
Add user docs for data read
oruebel Sep 1, 2024
4299f99
Update read user docs
oruebel Sep 1, 2024
9c9301c
Add toc
oruebel Sep 2, 2024
447337a
Add read software design figure and more details on the design
oruebel Sep 2, 2024
e3d4330
Some adjustment to the edges in the fig
oruebel Sep 2, 2024
77f0861
Some adjustment to the edges in the fig
oruebel Sep 2, 2024
5190731
Add intro for data read page
oruebel Sep 2, 2024
f08e42f
Make Container::create inline
oruebel Sep 3, 2024
1a754b9
Make DataBlock::fromGeneric inline
oruebel Sep 3, 2024
ebfbdf1
Implement function to allow us to get the storage object type for a g…
oruebel Sep 3, 2024
3022861
Rename getObjectType to getStroageObjectType
oruebel Sep 3, 2024
6ba32a5
Fix reading of attributes and add TimeSeries.resolution read
oruebel Sep 3, 2024
1fb5d95
Add example for reading an attribute
oruebel Sep 3, 2024
676b978
Add ReadDataWrapperBase as common base class for ReadDatasetWrapper a…
oruebel Sep 3, 2024
14c8362
Fix docstring
oruebel Sep 3, 2024
d2b1c23
Minor fixed to docs
oruebel Sep 3, 2024
78825f2
Mention ReadDataWrapperBase in the read dox page
oruebel Sep 3, 2024
85db320
Merge branch 'main' into add_read
oruebel Sep 3, 2024
5c2faae
Fix unit tests after merge
oruebel Sep 3, 2024
c481260
Remove NWBFile.identifierText property
oruebel Sep 3, 2024
b418149
Allow setting of value type template parameter for ReadDataWrapperBas…
oruebel Sep 4, 2024
1f3ad24
Save the std::type_index as part of the DataBlock to support runtime …
oruebel Sep 4, 2024
99b5d4e
Fix #88 move io classes to their own module
oruebel Sep 5, 2024
04117e9
Refactor to add IO namespace to match folder structure
oruebel Sep 5, 2024
82a4c08
Move HDF5RecordingData to its own hpp/cpp files
oruebel Sep 5, 2024
6554e29
Moved read I/O classes to new ReadIO.hpp
oruebel Sep 5, 2024
bff7121
Refactor the read data wrappers to use a single ReadDataWrapper class…
oruebel Sep 5, 2024
53365ff
Replace redundant ReadObjectType with StorageObjectType
oruebel Sep 6, 2024
c66b3bc
Add traits for ReadObjectWrapper to prevent instantiation for Group a…
oruebel Sep 6, 2024
0da4800
Fix formatting
oruebel Sep 6, 2024
d161c47
Fix spelling error
oruebel Sep 6, 2024
58e5afb
Inline functions in ReadIO.hpp
oruebel Sep 6, 2024
b7cc149
Some minor fixes
oruebel Sep 6, 2024
b9c6c9b
Merge branch 'main' into add_read
oruebel Sep 7, 2024
0e8ef12
First draft of Container registry
oruebel Sep 8, 2024
69c3c4e
Update ElectrodGroup to provide standard Container constructor and mo…
oruebel Sep 8, 2024
e75c7ad
Fix formatting
oruebel Sep 8, 2024
eeaceae
Merge branch 'add_read' into add_container_read
oruebel Sep 8, 2024
e62164b
Register subtypes
oruebel Sep 8, 2024
e99feee
Move the registry to a new RegisteredType class to also cover non-Gro…
oruebel Sep 8, 2024
3683921
Fix formatting
oruebel Sep 8, 2024
20050c7
Ensure all neurodata_types are registered correctly
oruebel Sep 8, 2024
cf9c017
Add unittest for RegisteredTrype
oruebel Sep 8, 2024
be29fb6
Add developer docs for RegisteredTypes
oruebel Sep 8, 2024
779c2af
Fix name of the unit test file for RegisteredType
oruebel Sep 8, 2024
3179c4b
Add example code for RegisteredType
oruebel Sep 8, 2024
dbdd945
Use both namespace and type name to register and access subtypes
oruebel Sep 8, 2024
4a7197f
Update documentation to clarify the use of namespace::classname to ma…
oruebel Sep 8, 2024
d11870b
Update read test to use the registry to construct the class
oruebel Sep 8, 2024
7a64982
Minor dox fixes
oruebel Sep 8, 2024
b42065b
Fix getTypeName and getNamespace and use them instead of hard-coded v…
oruebel Sep 9, 2024
501bc32
Update docs for getTypeName add getNamespace
oruebel Sep 9, 2024
5ab8464
Merge branch 'main' into add_read
oruebel Sep 9, 2024
803dfcf
Fix ElectricalSeries
oruebel Sep 9, 2024
5d959bd
Merge branch 'add_read' into add_container_read
oruebel Sep 9, 2024
a1dc63d
Attempt to fix reading of string attribute
oruebel Sep 9, 2024
bba4a79
Fix read neurodata_type attribute read test
oruebel Sep 9, 2024
a58a44a
Add test for reading a TimeSeries Container back
oruebel Sep 9, 2024
3ac8ca4
Add RegisteredType::create method to read a type from file
oruebel Sep 9, 2024
5f0bdd6
Fix path ecephys read test
oruebel Sep 9, 2024
edc45cf
Change RegisteredType::create to return shared pointer to support cas…
oruebel Sep 9, 2024
7184f96
Update ecephys read example
oruebel Sep 9, 2024
bda279d
Updated read tutorial and example
oruebel Sep 9, 2024
db8c652
Add missing comment
oruebel Sep 9, 2024
cf06a19
Clarify read docs and rename workflow docs
oruebel Sep 9, 2024
cdffc74
Run test workflows on all PRs
oruebel Sep 19, 2024
b0a28f9
Merge branch 'add_read' into add_container_read
oruebel Sep 19, 2024
89f6d29
Fix static assert for ReadDataWrapper
oruebel Sep 19, 2024
5608c5a
Sync add_read with main branch (#101)
oruebel Sep 19, 2024
8bbecb8
Merge branch 'add_read' into add_container_read
oruebel Sep 19, 2024
c7ced4e
Merge branch 'main' into add_read
oruebel Sep 19, 2024
dd01ad3
Merge branch 'main' into add_read
oruebel Sep 20, 2024
968fcf0
Fix formatting and docs error after merge with main
oruebel Sep 20, 2024
b424eaf
Merge branch 'add_read' into add_container_read
oruebel Sep 20, 2024
8aa5680
Fix formatting after merge
oruebel Sep 20, 2024
5798145
Remove SpikeEventSeries.neurodata_type
oruebel Sep 20, 2024
8832b04
Register SpikeEventSeries as a type
oruebel Sep 20, 2024
f244d8c
Updated RegisterTypes to allow overwrite of the typename to use and o…
oruebel Sep 21, 2024
c78e7cb
Fix code formatting
oruebel Sep 21, 2024
9d7f408
Add missing description for DynamicTableRegion
oruebel Sep 21, 2024
435866c
Fix #109 Add missing axis attribute for channel_conversion and remove…
oruebel Sep 21, 2024
ce3227c
Remove undefined functions from Device
oruebel Sep 21, 2024
cc0cd84
Rename member variable in ElectricalSeries
oruebel Sep 21, 2024
f99ed91
Rename member variable in SpikeEventSeries
oruebel Sep 21, 2024
6a4876f
Added mergePaths functions and made functions in Utils.hpp static inline
oruebel Sep 21, 2024
af28b29
Updated ElectrodeTable to use mergePaths method
oruebel Sep 21, 2024
c222955
Updated DynamicTable to use mergePaths method
oruebel Sep 21, 2024
0d402f3
Update ElectrodeGroup to use mergePaths
oruebel Sep 21, 2024
156910c
Updated ElectricalSeries to use mergePaths
oruebel Sep 21, 2024
962faaa
Updated TimerSeries to use mergePaths function
oruebel Sep 21, 2024
8f13baf
Updated NWBFile to use the mergePaths function
oruebel Sep 21, 2024
bf92ce2
Updated ElectrodeTable static paths to not used trailing / for consis…
oruebel Sep 22, 2024
c2251a8
Fix Doxygen error by extracting also static members in the docs
oruebel Sep 22, 2024
f3e30cd
Fix section level in install.dox
oruebel Sep 22, 2024
cea343b
Update ReadDataWrapper to use m_ member naming convention
oruebel Sep 22, 2024
7c884d4
Updated DataBlock / DataBlockGeneric to avoid shadowing of constructo…
oruebel Sep 22, 2024
8145382
Fix reference error in read.dox
oruebel Sep 22, 2024
078c4f9
Add ReadDataWrapper isType, getPath, getIO methods and update docstrings
oruebel Sep 22, 2024
98377cb
Create macro DEFINE_FIELD to simplify creating access methods for fie…
oruebel Sep 22, 2024
281375c
Replace previous field access methods in TimeSeries with DEFINE_FIELD…
oruebel Sep 22, 2024
9eb00cf
Replace TimeSeries.dataLazy and resolutionLazy with DEFINE_FIELD defi…
oruebel Sep 22, 2024
90803a8
Update the Doxygen built to expand the DEFINE_FIELD macro to document…
oruebel Sep 22, 2024
4dfb541
Fix spellcheck
oruebel Sep 22, 2024
9b9c8c3
Fix section levels in read.dox
oruebel Sep 22, 2024
3c4b2d7
Add docs for using the DEFINE_FIELD macro
oruebel Sep 22, 2024
aad6928
Added attributeExists method on IO and ReadDataWrapper::exists method…
oruebel Sep 22, 2024
5a411c4
Add TimeSeries.descriptionLazy field to test that reading string attr…
oruebel Sep 22, 2024
daca8f2
Added function getGroupObjects to the I/O to allow gettings all objec…
oruebel Sep 22, 2024
1f81e6d
Add BaseIO::findTypes method to search for neurodata_types in a file
oruebel Sep 22, 2024
3affcca
Add example for searching for typed objects
oruebel Sep 23, 2024
30e1226
Update ElectrodeTable to use m_ member names
oruebel Sep 23, 2024
fa6a981
Added field definitions for TimeSeries
oruebel Sep 23, 2024
a0e02b2
Add some design docs about reading registered types
oruebel Sep 24, 2024
fbed92c
Update read types design figure
oruebel Sep 24, 2024
d4c2e64
Minor update to docs of RegisteredType
oruebel Sep 24, 2024
bfaabb8
Update docs/pages/userdocs/read.dox
oruebel Sep 24, 2024
a439212
Update src/io/ReadIO.hpp
oruebel Sep 24, 2024
5f28fb0
Update src/io/ReadIO.hpp
oruebel Sep 24, 2024
a9725b2
Rename ReadDataWrapper.is_dataset
oruebel Sep 24, 2024
e70949c
Update src/io/hdf5/HDF5IO.hpp
oruebel Sep 24, 2024
d1da5d1
Remove duplicate code in NWBFile
oruebel Sep 24, 2024
12a86a5
Remove outdated ToDo item
oruebel Sep 24, 2024
423f3d7
Apply suggestions on docs from code review
oruebel Oct 2, 2024
ae66f33
Merge branch 'main' into add_read
oruebel Oct 22, 2024
69df3e0
Fix build error after merge
oruebel Oct 22, 2024
415926c
Merge branch 'add_read' into add_container_read
oruebel Oct 22, 2024
a93cadd
Fix build error due to error during merge with base branch
oruebel Oct 22, 2024
2554667
Apply suggestions from code review
oruebel Oct 22, 2024
6c57527
Fix read.dox figure based on suggestion
oruebel Oct 23, 2024
a7cf936
Fix docs based on code review
oruebel Oct 23, 2024
3d89d78
update getAttribute method to detect object type
stephprince Dec 12, 2024
c22d658
update read example
stephprince Dec 12, 2024
a91cc91
add file mode options and readonly mode to io
stephprince Dec 12, 2024
b20388c
update example to use readonly
stephprince Dec 12, 2024
20afbae
add getter functions to Container classes with attributes
stephprince Dec 18, 2024
20813c2
fix formatting
stephprince Dec 18, 2024
f211978
update lint workflow
stephprince Dec 18, 2024
0b7d181
add NWBFile fields
stephprince Dec 19, 2024
a5cda51
update filenames
stephprince Dec 19, 2024
6a7bb45
remove duplicate file mode checks
stephprince Dec 19, 2024
bfa740a
Merge pull request #91 from NeurodataWithoutBorders/add_container_read
stephprince Dec 19, 2024
ddb8f50
Merge branch 'main' into add_read
oruebel Dec 22, 2024
54d55da
Fix spelling and complete merge
oruebel Dec 22, 2024
8492bf7
Fix missing merge in file
oruebel Dec 22, 2024
9ed9bd3
Install Boost multi-array in windows action
oruebel Dec 22, 2024
f6b306b
Use std::vector instead of variable-length arrays to avoid Windows bu…
oruebel Dec 22, 2024
7b88122
Add note to docs to clarify non-virtual DEFINE_FIELD
oruebel Dec 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/codespell.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
- main

pull_request:
branches:
- main
#branches:
oruebel marked this conversation as resolved.
Show resolved Hide resolved
#- main
workflow_dispatch:

jobs:
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/doxygen-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ name: Doxygen build test

on:
pull_request:
branches:
- main
#branches:
#- main

jobs:
test:
Expand All @@ -17,7 +17,7 @@ jobs:
uses: actions/checkout@v4
with:
submodules: "true"

- name: Install dependencies - macos
run: brew install hdf5 boost catch2 doxygen graphviz

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
- main

pull_request:
branches:
- main
#branches:
#- main
workflow_dispatch:

jobs:
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
- main

pull_request:
branches:
- main
#branches:
#- main
workflow_dispatch:

jobs:
Expand All @@ -22,7 +22,7 @@ jobs:
fail-fast: false
matrix:
os: [macos-latest, ubuntu-latest]

runs-on: ${{ matrix.os }}

steps:
Expand All @@ -39,7 +39,7 @@ jobs:
git checkout "v3.5.3"
cmake -Bbuild -H. -DBUILD_TESTING=OFF
sudo cmake --build build/ --target install

- name: Install dependencies - macos
if: matrix.os == 'macos-latest'
run: brew install hdf5 boost catch2
Expand Down Expand Up @@ -67,7 +67,7 @@ jobs:

sanitize:
needs: test

runs-on: ubuntu-latest

steps:
Expand Down
5 changes: 3 additions & 2 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ include(cmake/variables.cmake)

add_library(
aqnwb_aqnwb
src/BaseIO.cpp
src/io/BaseIO.cpp
src/Channel.cpp
src/hdf5/HDF5IO.cpp
src/io/hdf5/HDF5IO.cpp
src/io/hdf5/HDF5RecordingData.cpp
src/nwb/NWBFile.cpp
src/nwb/RecordingContainers.cpp
src/nwb/base/TimeSeries.cpp
Expand Down
4 changes: 4 additions & 0 deletions docs/Doxyfile.in
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ EXTRACT_ALL = YES
RECURSIVE = YES
OUTPUT_DIRECTORY = "@DOXYGEN_OUTPUT_DIRECTORY@"

# Also show private members in the docs,
EXTRACT_PRIVATE = YES
# HIDE_UNDOC_MEMBERS = YES

# Enable Markdown support
MARKDOWN_SUPPORT = YES

Expand Down
1 change: 1 addition & 0 deletions docs/pages/1_userdocs.dox
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@
* - \subpage user_install_page
* - \subpage workflow
* - \subpage hdf5io
* - \subpage read_page
*/
2 changes: 1 addition & 1 deletion docs/pages/devdocs/documentation.dox
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
*
* To display a code snippet from our example in the documentation we can use the
* ``\snippet <file> <label>`` Doxygen command in our ``*.dox`` documentation file.
* For example to just show the line where we create the \ref AQNWB::HDF5::HDF5IO "HDF5IO
* For example to just show the line where we create the \ref AQNWB::IO::HDF5::HDF5IO "HDF5IO"
* object we use:
*
* \code{.sh}
Expand Down
10 changes: 5 additions & 5 deletions docs/pages/userdocs/hdf5io.dox
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@
*
* \section hdf5io_swmr Single-Writer Multiple-Reader (SWMR) Mode
*
* The \ref AQNWB::HDF5::HDF5IO "HDF5IO" I/O backend uses by default SWMR mode while recording data.
* The \ref AQNWB::IO::HDF5::HDF5IO "HDF5IO" I/O backend uses by default SWMR mode while recording data.
* Using SWMR, one process can write to the HDF5 file and multiple other processes can read
* from the file concurrently while ensuring that the readers see a consistent view of the data.
*
Expand All @@ -106,7 +106,7 @@
* node [shape=box, style=filled];
*
* // Define the writer node with light blue fill color
* writer [label="AqNWB HDF5IO\n(SWMR Write)", fillcolor=lightblue, URL="\ref AQNWB::HDF5::HDF5IO"];
* writer [label="AqNWB HDF5IO\n(SWMR Write)", fillcolor=lightblue, URL="\ref AQNWB::IO::HDF5::HDF5IO"];
* // Define the HDF5 file node with a distinct shape
* hdf5_file [label="\n.nwb\n(HDF5 File)\n ", shape=folder, fillcolor=lightgray, URL="\ref AQNWB::NWB::NWBFile"];
* // Define reader nodes with light green fill color
Expand Down Expand Up @@ -148,11 +148,11 @@
* \note
* While SWMR mode ensures data integrity, some data loss may still occur if the application crashes.
* Only data that has been completely written and flushed to disk will be readable. To manually
* flush data to disk use \ref AQNWB::HDF5::HDF5IO::flush "HDF5IO::flush".
* flush data to disk use \ref AQNWB::IO::HDF5::HDF5IO::flush "HDF5IO::flush".
*
* \subsection hdf5io_swmr_workflow Writing an NWB file with SWMR mode
*
* SWMR mode is enabled when calling \ref AQNWB::HDF5::HDF5IO::startRecording "HDF5IO::startRecording".
* SWMR mode is enabled when calling \ref AQNWB::IO::HDF5::HDF5IO::startRecording "HDF5IO::startRecording".
* Once SWMR mode is enabled, no new data objects (Datasets, Groups, Attributes etc.)
* can be created, but we can only add and set values to existing data objects. Since other
* processes may read from the HDF5 file, it is not possible to intermittently disable
Expand All @@ -168,7 +168,7 @@
* This workflow is applicable to a wide range of data acquisition use-cases. However,
* for use cases that require creation of new Groups and Datasets during acquisition,
* you can disable the use of SWMR mode by setting `disableSWMRMode=true` when
* constructing the \ref AQNWB::HDF5::HDF5IO object.
* constructing the \ref AQNWB::IO::HDF5::HDF5IO object.
*
* \warning
* While disabling SWMR mode allows Groups and Datasets to be created during and after
Expand Down
Loading
Loading