[COST-5851] one expiration task per provider_type

[maskarb]

Jira Ticket

COST-5851

Description

This change will:

• utilize cascade_delete to remove the expired OCPUsageReportPeriod instead of the Django delete method
• run one data expiration task per schema per provider_type, rather than 1 task per provider
• order the deletions so OCP is triggered last (because we run multiple workers, it's still possible that OCP will delete concurrently to the Cloud providers, but tis best effort)

Testing

1. load test data:

$ make create-test-customer; make load-test-customer-data test_source=AWS

2. check for report data when done ingesting:
   http://localhost:8000/api/cost-management/v1/reports/aws/costs/?filter[time_scope_value]=-30
   (see data for 2 months)
3. set RETAIN_NUM_MONTHS=1 and restart koku-worker
4. open a python shell:

$ REDIS_HOST=localhost make shell

(set the REDIS_HOST so that the shell can reach Redis)

5. use the Orchestrator to trigger a deletion:

>>> from masu.processor.orchestrator import Orchestrator
>>> orchestrator = Orchestrator()
>>> orchestrator.remove_expired_report_data(simulate=False)
[
    {"schema": "org1234567", "provider_type": "GCP-local", "async_id": "6f896ddc-7590-4d8e-8279-a4bd428c6651"},
    {"schema": "org1234567", "provider_type": "IBM-local", "async_id": "390a5303-91a2-4c96-94f4-0b7a8b65d99f"},
    {"schema": "org1234567", "provider_type": "AWS-local", "async_id": "efdd8a49-a13e-4592-846d-6f46ee197d1e"},
    {"schema": "org1234567", "provider_type": "Azure-local", "async_id": "0122ac56-020e-4fab-b624-77245c54a18b"},
    {"schema": "org1234567", "provider_type": "OCI-local", "async_id": "e0b624e6-7b3b-4383-8c93-f9342b084ddc"},
    {"schema": "org1234567", "provider_type": "OCP", "async_id": "b56d2cf2-6d2e-4988-a4ac-a0971c6b3c50"},
]

See that each provider-type only appears once in the result list AND OCP is called last.

6. clear redis cache:

make delete-redis-cache

7. Revisit the report endpoint and see data only exists for the current month:
   http://localhost:8000/api/cost-management/v1/reports/aws/costs/?filter[time_scope_value]=-30

Release Notes

• proposed release note

* [COST-5851](https://issues.redhat.com/browse/COST-5851) ordered data expiration per provider type

[codecov]

Codecov Report

Attention: Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.

Project coverage is 94.1%. Comparing base (7a51af4) to head (74bb1bf).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #5432     +/-   ##
=======================================
- Coverage   94.1%   94.1%   -0.0%     
=======================================
  Files        371     371             
  Lines      31631   31635      +4     
  Branches    3386    3389      +3     
=======================================
+ Hits       29770   29773      +3     
  Misses      1211    1211             
- Partials     650     651      +1     

[maskarb]

/retest

[maskarb]

/retest

[djnakabaale]

looks good to me, one small nitpick: