[Utilization][S3 Replicator][fortesting db]Add replicator logic to insert util data into clickhouse #6217
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
facebook-github-bot
added
the
CLA Signed
yangw-dev
changed the title
yangw-dev
changed the title
huydhn
approved these changes
Jan 28, 2025
There was a problem hiding this comment.
LGTM! You probably want to manually test the insert queries with the provide schemas like https://github.com/pytorch/test-infra/wiki/How-to-add-a-new-custom-table-on-ClickHouse#testing
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Jan 29, 2025
upload_utilization_script to generate db-ready-insert records to s3 - generate two files: metadata and timeseries in ossci-utilization buckets - convert log record to db format ones - add unit test job for tools/stats/ Related Prs: setup composite action for data pipeline: #145310 add permission for composite action to access S3 bucket: https://github.com/pytorch-labs/pytorch-gha-infra/pull/595 add insert logic in s3 replicator: pytorch/test-infra#6217 Pull Request resolved: #145327 Approved by: https://github.com/huydhn Co-authored-by: Huy Do <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
currently i add the debug folder for testing, and the data will be inserted into tables in db fortesting on clickhouse
others:
add missing field tags
Related Prs:
setup composite action for data pipeline: pytorch/pytorch#145310
add permission for composite action to access S3 bucket: https://github.com/pytorch-labs/pytorch-gha-infra/pull/595
set up data pipeline script: pytorch/pytorch#145327