You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Maintaining both sources and origins is cumbersome. Migrating all sources to origins would allow us to deprecate a significant amount of code in both ETL and Grapher, potentially speeding up the process. We've already migrated the most critical datasets to origins and are currently migrating the rest as needed. However, a one-time full migration might be more efficient.
Solution
Full migration to origins in snapshots
This would involve migrating all sources in snapshots to origins. We’d need to devise an automatic source-to-origin transformation and modify all steps to use Table instead of DataFrame to ensure proper origin propagation.
Adding origins to YAML files
For all garden and grapher datasets, we could extract sources, automatically migrate them to origins, and add them to the metadata YAML files (potentially as *.meta.override.yml).
TODO
How many (unarchived) datasets still use sources?
The text was updated successfully, but these errors were encountered:
Problem
Maintaining both sources and origins is cumbersome. Migrating all sources to origins would allow us to deprecate a significant amount of code in both ETL and Grapher, potentially speeding up the process. We've already migrated the most critical datasets to origins and are currently migrating the rest as needed. However, a one-time full migration might be more efficient.
Solution
Full migration to origins in snapshots
This would involve migrating all sources in snapshots to origins. We’d need to devise an automatic source-to-origin transformation and modify all steps to use
Table
instead ofDataFrame
to ensure proper origin propagation.Adding origins to YAML files
For all garden and grapher datasets, we could extract sources, automatically migrate them to origins, and add them to the metadata YAML files (potentially as
*.meta.override.yml
).TODO
The text was updated successfully, but these errors were encountered: