feat: deduplication of metrics #20

desusai7 · 2023-10-27T14:42:02Z

Update database for storing batches

Create Model classes for snapshot table
Create a new table for storing snapshots of batches.
Provide APIs for fetching and storing in the database

take a snapshot of metrics at regular intervals as batches and assign random UUID

Take a snapshot of the metrics table at regular intervals.
Create a batch JSON from the same.
Assign a random UUID to the batch JSON
Use reservoir API to store the same.

syncing batches with server

Whenever a batch is pushed, try syncing it with the server.
At regular intervals fetch batches from the jobs table (if any)
Send the batches to the server
If successful remove the batches.

codecov · 2023-10-27T14:55:59Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6d61209) 0.00% compared to head (8d00b21) 0.00%.

❗ Current head 8d00b21 differs from pull request most recent head 670b13f. Consider uploading reports for the commit 670b13f to get more accurate results

Additional details and impacted files

@@     Coverage Diff      @@
##   master   #20   +/-   ##
============================
============================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

* feat: added batch generator to generate batches periodically * chore: updated sample app to be debuggable and added new files to the project * refactor: made changes to make BatchOperator and BatchGenerator unit-testable * test: fixed failing unit tests in DatabaseTests, MetricOperatorTests after enhancements in MetricOperator * test: added unit tests for batch generator * chore: fixed merge issues and changes as required * fix: updated snapshot generator timer to a default of 30 seconds and deleting metrics and errors only on successful batch generation * fix: minor changes * chore: extended Plugin protocol to create MetricsPlugin which contains execute * chore: changed crashreporter and snapshotgenerators from plugins to managers * test: fixed failing unit tests after changing CrashReporter, SnapshotGenerator to managers from plugins * feat: syncing batches with server (#24) * feat: added functions to get and clear batches in database * feat: added database observer to be notified about the changes to the db * chore: minor improvements in the ServiceManager * feat: added batch uploader to sync batches to metrics service * feat: added support for retrying with backoff when requests to metrics service get failed * chore: renamed batch to snapshot * chore: uploading snapshots on saving new snapshot * chore: added error handling to database observer * feat: added atomic property wrapper for synchronized across across threads * refactor: converted snapshot uploader to a manager and initialized it in snapshot generator * chore: minor changes --------- Co-authored-by: Desu Sai Venkat <[email protected]>