Patch references to UC schemas to capture dependencies automatically #1989
Conversation
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
|
Test Details: go/deco-tests/12239986056 |
shreyas-goenka
changed the title
${resources.schemas...} syntax
| diags := diag.Diagnostics{} | ||
| for k, p := range rb.Config().Resources.Pipelines { | ||
| // Skip if the pipeline uses hive metastore. | ||
| if p.Catalog == "" { |
There was a problem hiding this comment.
The API allows creating a pipeline without a schema/target specified as long as you are not using UC. It's not a forward-looking use case anyway, so I did not look deeper into this.
shreyas-goenka
requested review from
pietern,
denik,
andrewnester and
lennartkats-db
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
| Severity: diag.Warning, | ||
| Summary: `Use ${resources.schemas.s1.name} syntax to refer to the UC schema instead of directly using its name "schema1"`, | ||
| Detail: `Using ${resources.schemas.s1.name} will allow DABs to capture the deploy time dependency this DLT pipeline | ||
| has on the schema "schema1" and deploy changes to the schema before deploying the pipeline.`, |
There was a problem hiding this comment.
It would be painful to update this test if the message changes, but it's not nice to assert on the message.
I have added a "golden files" test utilities here #2025 , it's a good fit for this cases.
Full output is stored in the file and can be updated by running tests with TESTS_OUTPUT=OVERWRITE
There was a problem hiding this comment.
Note, that PR only targets integration tests for now, so not ready for your use case yet.
There was a problem hiding this comment.
I think it's fine in this case since its unit tests that are colocated with the mutator that implements the functionality. It'll be a simple search and replace when updating.
Are you proposing to extend the golden file approach to mutators or unit tests? Possibly by serializing the diagnostics and persisting them in a file?
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
|
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
shreyas-goenka
changed the title
${resources.schemas...} syntax
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
| "nilschema": {}, | ||
| "emptyschema": { | ||
| CreateSchema: &catalog.CreateSchema{}, | ||
| }, |
There was a problem hiding this comment.
The nilschema should be nil as in actual nil and empty as in what's now the nilschema.
Changes
FIxes #1977.
Tests
Unit tests and manually.