New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reformat data in Parquet and benchmark #67

Closed

lewfish opened this issue Jun 15, 2022 · 1 comment

Assignees

Contributor

lewfish commented Jun 15, 2022

Re-encode the data using Parquet for point-based datasets (ie. the ones representing stream reaches). Try different ways of chunking it. This corresponds to Task 1-3: Encode NWM Streamflow Output as Parquet #28. Code in https://github.com/azavea/noaa-hydro-data/blob/master/src/notebooks/save_nwm_sample.ipynb might be a useful starting point.
Time the query using the Parquet-encoded data.

This is a piece of #45

lewfish assigned rajadain and vlulla

lewfish mentioned this issue

Experiments for ESIP Presentation #45

Closed

Contributor Author

lewfish commented Jul 6, 2022 •

edited

Loading

This should roughly follow the workflow for Zarr in rechunk_zarr_subset.ipynb and huc8_streamflow_query.ipynb introduced in #71

lewfish closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment