Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow DBT MODELS to be specified from file location #182

Open
1yuv opened this issue Nov 11, 2024 · 2 comments · May be fixed by #188
Open

Allow DBT MODELS to be specified from file location #182

1yuv opened this issue Nov 11, 2024 · 2 comments · May be fixed by #188
Assignees
Labels
Type: Feature Add something new

Comments

@1yuv
Copy link
Member

1yuv commented Nov 11, 2024

Is your feature request related to a problem? Please describe.
cht-sync currently requires CHT_PIPELINE_BRANCH_URL to be set and requires all dbt models to be committed on github.

Describe the solution you'd like
Current practice makes is difficult to develop or test any new models as it requires the new models need to be pushed on github without testing. Enable alternative environment variable DBT_PIPELINE_PATH that can specify relative or absolute path to pipeline models.

Describe alternatives you've considered
None

@1yuv 1yuv added the Type: Feature Add something new label Nov 11, 2024
@witash witash self-assigned this Nov 13, 2024
@witash
Copy link
Contributor

witash commented Nov 13, 2024

you can use the e2e test framework for local testing, which is what we did for developing the base models. e2e tests are designed for a local couchdb with fake data, but can use a remote couchdb also.

edit models at tests/dbt/package
copy tests/.e2e-env to tests/.local-test-env and change couchdb url
run with

docker compose --env-file ./tests/.local-test-env -f docker-compose.yml -f docker-compose.postgres.yml -f tests/dbt/docker-compose.yml up -d --build

but this is not documented and easy to use.

we can add something to the dbt dockerfile so it copies files into the container...or keep it this way (where it uses DBT_TARBALL_URL env variable, which points to another container that's built by tests/dbt/docker-compose.yml) but just clean up the commands and add it to the docs

@mrjones-plip
Copy link
Contributor

😍

clean up the commands and add it to the docs

witash added a commit that referenced this issue Nov 21, 2024
witash added a commit that referenced this issue Nov 21, 2024
@witash witash linked a pull request Nov 21, 2024 that will close this issue
@andrablaj andrablaj assigned witash and unassigned witash Dec 3, 2024
@andrablaj andrablaj moved this from Todo to In Progress in Product Team Activities Dec 3, 2024
@andrablaj andrablaj moved this to Todo in CHT Stewardship Dec 19, 2024
@andrablaj andrablaj moved this from Todo to In Progress in CHT Stewardship Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Feature Add something new
Projects
Status: In Progress
Status: In Progress
Development

Successfully merging a pull request may close this issue.

3 participants