Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port 2-3 logically complicated validations to dbt #3997

Open
2 of 7 tasks
jdangerx opened this issue Dec 31, 2024 · 1 comment
Open
2 of 7 tasks

Port 2-3 logically complicated validations to dbt #3997

jdangerx opened this issue Dec 31, 2024 · 1 comment
Assignees
@zschira
Labels
dagster Issues related to our use of the Dagster orchestrator data-validation Issues related to checking whether data meets our quality expectations. dbt Issues related to the data build tool aka dbt duckdb Issues referring to duckdb, the embedded OLAP database parquet Issues related to the Apache Parquet file format which we use for long tables. testing Writing tests, creating test data, automating testing, etc.

Comments

@jdangerx
Copy link
Member

jdangerx commented Dec 31, 2024 •
edited by zschira
Loading

Success criteria

  • Convert basic validation like bounds checks
  • Convert high memory validations (vcerare)
  • Convert eia923 vs_historical test
  • Fully setup dbt project in basic pudl-dev environment install

Not in scope

  • connecting dbt source test to dagster

Next steps

  • setup multiple profiles for testing nightly builds/etl full/etl fast
  • get dbt set up with dbt-utils, dbt-expectations
  • port those dang tests
@jdangerx jdangerx converted this from a draft issue Dec 31, 2024
@jdangerx jdangerx assigned jdangerx and zschira and unassigned jdangerx Dec 31, 2024
@jdangerx jdangerx changed the title Port 2-3 logically complicated validations to port to GX Port 2-3 logically complicated validations to dbt Jan 8, 2025
@jdangerx jdangerx moved this from Backlog to In progress in Catalyst Megaproject Jan 8, 2025
@jdangerx
Copy link
Member Author

jdangerx commented Jan 8, 2025

@zschira do you mind fleshing out what X/Y/Z validation tests are in this issue?

@zaneselvans zaneselvans added testing Writing tests, creating test data, automating testing, etc. data-validation Issues related to checking whether data meets our quality expectations. dagster Issues related to our use of the Dagster orchestrator dbt Issues related to the data build tool aka dbt parquet Issues related to the Apache Parquet file format which we use for long tables. duckdb Issues referring to duckdb, the embedded OLAP database labels Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zschira zschira

Labels
dagster Issues related to our use of the Dagster orchestrator data-validation Issues related to checking whether data meets our quality expectations. dbt Issues related to the data build tool aka dbt duckdb Issues referring to duckdb, the embedded OLAP database parquet Issues related to the Apache Parquet file format which we use for long tables. testing Writing tests, creating test data, automating testing, etc.
Projects
Catalyst Megaproject
Status: In progress
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zaneselvans @jdangerx @zschira