Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use HDF5 for data storage #40

Open
4 of 5 tasks
tmolteno opened this issue Mar 6, 2019 · 1 comment
Open
4 of 5 tasks

Use HDF5 for data storage #40

tmolteno opened this issue Mar 6, 2019 · 1 comment
Assignees
@tmolteno
Labels
enhancement

Comments

@tmolteno
Copy link
Owner

tmolteno commented Mar 6, 2019 •
edited
Loading

The use of python pickle to store Observation and Visibility objects is not a good long term option. This is not robust to python2 vs python3, so persistence will move to HDF5 files.

This will add a dependency to the 'tart' python package (it will depend on h5py). Some routines will need to be written to convert existing files.

  • Write new persistence methods. with tests.
  • Utilities to convert from pickle to .hdf (written in python2).
  • Update testbenches
  • Web app save HDF RAW files
  • Web app save HDF Vis files
@tmolteno tmolteno self-assigned this Mar 6, 2019
tmolteno pushed a commit that referenced this issue Mar 8, 2019
Added HDF5 input and output for observation objects. First part of Is…
ebc41ca 
…sue #40 addressed
tmolteno pushed a commit that referenced this issue Mar 11, 2019
Updated to store HDF5 files. Part of resolving Issue #40
d2a37dd
tmolteno pushed a commit that referenced this issue Mar 12, 2019
Updated observation load to choose HDF files if the extension is righ…
9236fea 
…t. Refers to Issue #40
tmolteno pushed a commit that referenced this issue Mar 13, 2019
Update testbench to add visibility exporters for HDF format, and to r…
d505a3d 
…ename Visibility_Load/Save to list_load and list_save, since they do not actually save individual visibility objects. Related to Issue #40
tmolteno pushed a commit that referenced this issue Mar 14, 2019
Update testbench, still to get loading visibility lists going. Refer …
3c5facc 
…Issue #40
@tmolteno
Copy link
Owner Author

All that is required now is to change the file extension to .hdf in tart_web_api/service.py. Testbenches are done and passing for exporting and importing lists of visibilities into .hdf5 files.

tmolteno pushed a commit that referenced this issue Jul 2, 2019
Visibilities now stored as HDF5 files. Closes Issue #40
13044c7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tmolteno tmolteno

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tmolteno