Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Archive DB daily #598

Merged
merged 4 commits into from
Nov 6, 2024
Merged

Archive DB daily #598

merged 4 commits into from
Nov 6, 2024

Commits on Nov 6, 2024

  1. db: Return None from get_last_modified() on no-data

    @spbnick
    spbnick committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    62de004 View commit details
    Browse the repository at this point in the history

  2. db: Add get_first_modified()

    @spbnick
    spbnick committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    422cf83 View commit details
    Browse the repository at this point in the history

  3. tests: Make sure to empty all deployed databases

    @spbnick
    spbnick committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    2628fb3 View commit details
    Browse the repository at this point in the history

  4. Copy pieces of operational DB to archive DB daily 

    Add a Cloud Scheduler job posting messages to a Pub/Sub topic every day
at 12:00, which triggers a Cloud Function, which copies one week of
(missing) data from the operational DB (PostgreSQL) to archive DB
(BigQuery). This replaces the previously-used writing through a mux with
rate limiting.
    @spbnick
    spbnick committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    a756f46 View commit details
    Browse the repository at this point in the history