Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[compile_trace] Add compile time Kineto trace #148

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

xuzhao9
Copy link
Contributor

@xuzhao9 xuzhao9 commented Jan 30, 2025

Generates Kineto trace focusing on compile time so that we can peek into compile time analysis.

Fixes #117

Test plan:

$ python run.py --op softmax --num-inputs 1 --input-id 0 --metrics compile_trace --only triton_softmax
image

Internal link only: https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftritonbench%2Fcompile_time.json&bucket=tc_bench_ci

With autotuning:

image

@xuzhao9 xuzhao9 temporarily deployed to docker-s3-upload January 30, 2025 13:56 — with GitHub Actions Inactive
@xuzhao9 xuzhao9 temporarily deployed to docker-s3-upload January 30, 2025 13:56 — with GitHub Actions Inactive
@xuzhao9 xuzhao9 requested review from adamomainz and FindHao January 30, 2025 13:58
@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@xuzhao9
Copy link
Contributor Author

xuzhao9 commented Jan 30, 2025

cc @fywkevin @Jokeren This is a Python side view of the Triton compilation process, note that Kineto has no observation on the C++ code.

@xuzhao9
Copy link
Contributor Author

xuzhao9 commented Jan 30, 2025

For autotuning, the Triton's autotuner has some GPU idleness gaps:

image

@Jokeren
Copy link

Jokeren commented Jan 30, 2025

This is a Python side view of the Triton compilation process, note that Kineto has no observation on the C++ code.

I think it's fine to have such a coarse-grained view from profilers.
MLIR can get you a very detailed view of time spent on each compiler passe, but I think it may only provide insights for compiler engineers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Need break-down of compilation time for Triton
3 participants