Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[COST-5773] Handle multiple resource for OCP on GCP managed flow #5427

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

myersCody
Copy link
Contributor

@myersCody myersCody commented Dec 16, 2024

Jira Ticket

COST-5573

Description

This change will:

  1. Add multiple clusters, add a wait, and trigger ocp ingest before cloud ingest to prevent ocp on cloud parquet file error.
  2. Restructure the managed flow to use a dataclass to pass in build the sql parameters in the db accessors.
  3. Restructure the GCP flow to handle row deduplication
  • Create two temporary tables
  • Table One: Grab all the possible rows from the GCP table and assign a unique identifier to them
  • Table Two: Read from table one to insert all of the OCP resource & tag matched rows
  • When we insert into the final managed table make we make sure the uuid only shows up once preventing duplication

Testing

  • Set the unleash flag to True for ocp on cloud managed processing
  1. using this branch Load test customer data for:
  • AWS
  • Azure
  • GCP
  1. The test data populated for OCP on GCP will end with a verification successful return. Whereas if you were to try on master you will get a failed message.

Release Notes

  • proposed release note
* [COST-####](https://issues.redhat.com/browse/COST-####) Fix some things

Copy link

codecov bot commented Jan 10, 2025

Codecov Report

Attention: Patch coverage is 90.24390% with 16 lines in your changes missing coverage. Please review.

Project coverage is 94.1%. Comparing base (aabdbae) to head (e2430a6).

Additional details and impacted files
@@           Coverage Diff           @@
##            main   #5427     +/-   ##
=======================================
- Coverage   94.2%   94.1%   -0.1%     
=======================================
  Files        370     371      +1     
  Lines      31539   31635     +96     
  Branches    3378    3384      +6     
=======================================
+ Hits       29700   29773     +73     
- Misses      1196    1216     +20     
- Partials     643     646      +3     

@myersCody myersCody added the smoke-tests pr_check will build the image and run minimal required smokes label Jan 14, 2025
@myersCody myersCody marked this pull request as ready for review January 14, 2025 18:57
@myersCody myersCody requested review from a team as code owners January 14, 2025 18:57
@myersCody myersCody added aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests and removed aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests labels Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
smoke-tests pr_check will build the image and run minimal required smokes smokes-required
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant