Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP for zarr alignment store #409

Merged
merged 28 commits into from
Nov 28, 2024
Merged

WIP for zarr alignment store #409

merged 28 commits into from
Nov 28, 2024

Conversation

jeromekelleher
Copy link
Owner

No description provided.

@jeromekelleher
Copy link
Owner Author

Note to self, to read in Viridian metadata need to do

df_in = pd.read_csv("tmp/run_metadata.v04.tsv", sep="\t", dtype={"Artic_primer_version": str})
df_in

TODO

  • Systematic ingest pipeline for metadata
  • Sample reordering
  • Metadata validation
  • A complete dataset

The rest can wait till after merge.

@jeromekelleher jeromekelleher mentioned this pull request Nov 27, 2024
Closed
@jeromekelleher
WIP for zarr alignment store
ac30782
@jeromekelleher
Update the metadata to follow original Viridian format
34ac484
@jeromekelleher
Add basics of new Dataset class
9c6a244
@jeromekelleher
Port alignment store tests to dataset
584f687
@jeromekelleher
Implement cached alignment storage
0085330
@jeromekelleher
Port metadata code to Zarr
cf4bb8c
@jeromekelleher
Add basic date handling and dedicated date field
19fb953 
Partial updates. Broken

Various stuff broken

Revert to older metadata to keep test suite working

Fixup more tests

Fixup more tests

Skip some flagging tests that are tedious to update
@jeromekelleher
Patch up the CLI
0b4d6dd
@jeromekelleher
Patch up some old tests
e7d6d9d
@jeromekelleher
Fixup more tests
a1b8f3d
@jeromekelleher
Patch up some tests
e349b33
@jeromekelleher
Add CLI and tests for validate
bc31321
@jeromekelleher
Tighten up code for validation.
cf05431 
And validate it runs - only a few minutes and 2.5G of RAM ror 440k
samples at 2022-08-01!
@jeromekelleher
Update import alignments command
37ff498
@jeromekelleher
Add metadata import
bf72a33
@jeromekelleher
fixup end-to-end tests
805e683
@jeromekelleher
Fixup unit tests
faf2bfb
@jeromekelleher
Add basic info-dataset command
5a282e4 
Fixup

Fixup

Tweaks
@jeromekelleher
Add raw viridian data fixture to tests
8affedc
@jeromekelleher
Add some zarr information
6f536a4
@jeromekelleher
Tests
ec5bc2b
@jeromekelleher
Delete old modules
127ee59
@jeromekelleher
Add more tweaks for importing Viridian data
74420b2
@jeromekelleher
Implement basic copy function
9e313f5 
fixup suite

Update
@jeromekelleher
Improve metadata validation
4338d34
@jeromekelleher
Add reorder dataset command
d149c08
@jeromekelleher
Fix memory leak
16cbc3f
@jeromekelleher
Fix flushing bug
0ec7c25
@jeromekelleher jeromekelleher marked this pull request as ready for review November 28, 2024 16:06
@jeromekelleher
Copy link
Owner Author

Going to merge this now as we have the basic prerequisites for working with a Zarr store.

@jeromekelleher jeromekelleher merged commit 0ad0923 into main Nov 28, 2024
3 checks passed
@jeromekelleher jeromekelleher deleted the switch-align-to-zarr branch November 28, 2024 16:15
@jeromekelleher jeromekelleher mentioned this pull request Nov 29, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jeromekelleher