Wall-time/all tasks profiler

[d-netto]

One limitation of sampling CPU/thread profiles, as is currently done in Julia, is that they primarily capture samples from CPU-intensive tasks.

If many tasks are performing IO or contending for concurrency primitives like semaphores, these tasks won’t appear in the profile, as they aren't scheduled on OS threads sampled by the profiler.

A wall-time profiler, like the one implemented in this PR, samples tasks regardless of OS thread scheduling. This enables profiling of IO-heavy tasks and detecting areas of heavy contention in the system.

Co-developed with @nickrobinson251.

[d-netto]

An additional note: this kind of wall-time/all tasks profiler is also implemented in Go (and denoted as goroutine profiler there), so there is some precedent for this in other languages as well: https://github.com/felixge/fgprof.

[d-netto]

@nickrobinson251 I can't assign you as reviewer... Feel free to assign yourself or post review comments otherwise.

[IanButterworth]

I think this is related to #55103. Could the metrics here be useful in that too?

[d-netto]

I think this is related to #55103. Could the metrics here be useful in that too?

For diagnosing excessive scheduling time? I can't immediately see how this PR would be useful for that.

[d-netto]

For diagnosing excessive scheduling time

#55103 seems like a much more direct approach for doing so, at least.

[d-netto]

Wrote this short demo on how to use the profiler: https://github.com/d-netto/Wall-time-Profiler-Demo.

Feedback is welcome.

Perhaps we should add this to some documentation page in Julia itself.

[vtjnash]

Can you add this to the Profile docs and the NEWS file?

[d-netto]

Can you add this to the Profile docs and the NEWS file?

Sounds good, will do. Thanks for the reviews.

[JZL]

This looks really interesting, I'm excited to use it!

To confirm, there's no way to use this with SIGUSR1/SIGINFO signalling, right? I've been trying to make sense of the code but am unsure -- am I correct that it's this line's [1] hardcoded 0/false which forces it to always be the old version?

Are there plans to make this modifiable using e.g. env variables like for heap snapshots or function references like peek_report.

[1]

julia/src/signals-unix.c

Line 818 in fcf7ec0

if (jl_profile_start_timer(0) < 0)

[nickrobinson251]

there's no way to use this with SIGUSR1/SIGINFO signalling, right?

Your understanding is correct.

Are there plans to make this modifiable using e.g. env variables like for heap snapshots or function references like peek_report.

i think that could be a good addition! Perhaps open a feature request issue for it? We might even want both e.g. I suppose an ENV variable could control whether to take a cpu profile or a wall-time profile, but if you want both it might be best to use peek_report, so we might want to add an easy, official way to do that, and document it. Right now i think you'd have to use the non-public function Profile.start_timer(true). Worth an issue to discuss, i think :)

Also, you might also be interested in #56043 (which depending on your use-case might be even better than a wall-time profile, since it'll print the backtrace for all tasks not just a sample)