Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate AWS report from BigQuery #33

Open
natanlao opened this issue Dec 8, 2020 · 2 comments
Open

Generate AWS report from BigQuery #33

natanlao opened this issue Dec 8, 2020 · 2 comments
Assignees
Labels
code [subject] Production code enh [type] New feature or request

Comments

@natanlao
Copy link
Contributor

natanlao commented Dec 8, 2020

From #26.

AWS billing data should be loaded into BigQuery and queried in a fashion similar to how GCP billing data is queried for the GCP report.

@natanlao
Copy link
Contributor Author

natanlao commented Dec 8, 2020

The current CSV delivery format may not be suitable for this due to lack of a unique ID per row (see #30). New approach is to add a new delivery format (parquet compression for Athena import), import the new files to BigQuery, then use a SQL view to mimic the GCP billing report table.

@github-actions github-actions bot added the orange label Dec 8, 2020
@natanlao natanlao mentioned this issue Dec 8, 2020
@melainalegaspi melainalegaspi added code [subject] Production code enh [type] New feature or request labels Dec 9, 2020
@natanlao
Copy link
Contributor Author

I was not able to complete this ticket. There is currently a CUR being generated in Parquet format in AWS; I was able to connect that bucket to a BigQuery Data Transfer job, but it complains about the format of the report. The Athena export format may not be appropriate.

Note that documentation on this was mistakenly committed before the functionality was: 7d1c099#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code [subject] Production code enh [type] New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants