Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Export copies of scraped data #148

Closed
AlexAxthelm opened this issue Feb 18, 2024 · 2 comments · Fixed by #150 or #158
Closed

Export copies of scraped data #148

AlexAxthelm opened this issue Feb 18, 2024 · 2 comments · Fixed by #150 or #158

Comments

@AlexAxthelm
Copy link
Collaborator

AlexAxthelm commented Feb 18, 2024 •
edited
Loading

Include a copy of scraped data with other data files, so that data prep can be reconstructed later if neccesary.

  • Currencies currencies.rds
  • index_regions index_regions.rds
@AlexAxthelm
Copy link
Collaborator Author

relates to #147, which needs to be updated to account for this.

@AlexAxthelm
Copy link
Collaborator Author

Note it appears that index_regions is not currently exported anywhere, which is a problem for reproducibility.

AlexAxthelm added a commit that referenced this issue Feb 18, 2024
@AlexAxthelm
feat(app): #148 Export data from scraped files
2d2c72f 
Closes: #148
@AlexAxthelm AlexAxthelm mentioned this issue Feb 18, 2024
Merged
@cjyetman cjyetman mentioned this issue Feb 22, 2024
Merged
@cjyetman cjyetman reopened this Feb 22, 2024
cjyetman added a commit that referenced this issue Feb 22, 2024
@cjyetman
save index_regions to outputs (#158)
e27b2d4 
supersedes #150 
closes #148
AlexAxthelm added a commit that referenced this issue Feb 23, 2024
@AlexAxthelm @cjyetman
feat(app): #148 Export data from scraped files (#150)
5b2617e 
Changes in this PR export data from scraped sources to file, so that all
sources for a dataset are captured, and the dataset can be recreated
later.

By default, exports to data prep outputs path, but can be specified with
`preflight_data_path`. Writing to the outputs path by default, since
this matches current behavior, and we do not have a unified inputs path
that we are currently using.

Closes: #148

Co-authored-by: CJ Yetman <[email protected]>
@cjyetman cjyetman mentioned this issue Apr 29, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@cjyetman @AlexAxthelm