Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make clean_stale_db_objects configurable #142

Merged
merged 1 commit into from
Sep 13, 2024
Merged

Conversation

hancush
Copy link
Collaborator

@hancush hancush commented Sep 11, 2024

Overview

See title! This PR also updates local development instructions in README.

The primary concept here is params: https://airflow.apache.org/docs/apache-airflow/2.6.3/core-concepts/params.html#params

As of 2.0, Airflow has a new concept called task flows, that are less boilerplate-y and more declarative than traditional DAGs. They also allow you to pass the return value of one task directly to another. Here's more, specifically how to declare dependencies between tasks and traditional operators, which we need to here: https://airflow.apache.org/docs/apache-airflow/2.6.3/tutorial/taskflow.html#adding-dependencies-between-decorated-and-traditional-tasks

Checklist

  • PR has a descriptive enough title to be useful in changelogs

Testing Instructions

  • Run the dashboard as described in the README
  • Trigger a clean_stale_db_objects job with configuration. Use different values for window and max, or toggle on report, and confirm the DAG behaves as you've configured it.

Closes #140, closes #141

Also update local development instructions in README
Copy link
Collaborator

@antidipyramid antidipyramid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! Can't wait for more DAGs to be configurable in the same way.

@hancush hancush merged commit 080351d into main Sep 13, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make DAGs configurable on run Set reasonable max for pupa clean DAG
2 participants