Data Pruning for Anomaly Detection

[aayush-se]

• Create Celery task to delete data that is over 28 days old and update matrix profiles for remaining points
• Update DbDynamicAlert to include data_purge_flag and queued_at columns to reflect Celery task status
• Write unit tests for new accessor methods and cleanup task

[corps]

Before we turn this on in production with ENABLE_CELERY_WORKERs, lets get #1192 in and add move this beat task to a conditional for anomaly detection.

[ram-senth]

We also need some metrics published from the Celerey task so that we can keep track of:

• number of clean ups done in a time window
• time duration taken for each cleanup so we can monitor p90 or p99
• count of times when a cleanup results in the alert having less than 7 days of data
• number of times cleanup task encountered error

These are in addition to the regular queue related metrics like queue size, time spent waiting etc. We should check with Jenn if we need to do anything specific for those to work.

[aayush-se]

We also need some metrics published from the Celerey task so that we can keep track of:

• number of clean ups done in a time window
• time duration taken for each cleanup so we can monitor p90 or p99
• count of times when a cleanup results in the alert having less than 7 days of data
• number of times cleanup task encountered error

These are in addition to the regular queue related metrics like queue size, time spent waiting etc. We should check with Jenn if we need to do anything specific for those to work.

I did include Sentry tracing for the cleanup task so I think everything aside from the third point (less than 7 days of data) should be viewable. I'm unsure about the other queue related metrics though since Sentry doesn't support direct metrics anymore (https://docs.sentry.io/product/explore/metrics/).

[ram-senth]

We can convert this to string enum and use it in both the business logic as well as in the data layer.

[ram-senth]

We also need some metrics published from the Celerey task so that we can keep track of:

• number of clean ups done in a time window
• time duration taken for each cleanup so we can monitor p90 or p99
• count of times when a cleanup results in the alert having less than 7 days of data
• number of times cleanup task encountered error

These are in addition to the regular queue related metrics like queue size, time spent waiting etc. We should check with Jenn if we need to do anything specific for those to work.

I did include Sentry tracing for the cleanup task so I think everything aside from the third point (less than 7 days of data) should be viewable. I'm unsure about the other queue related metrics though since Sentry doesn't support direct metrics anymore (https://docs.sentry.io/product/explore/metrics/).

Wondering if we should add the alert id as a tag so that we can look at metrics at an alert level. Let us ask Zach and Jenn about the celery metrics in the next standup. Rest is good.