Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve import stats performance by emitting start and end together #2417

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

op
Copy link

@op op commented Oct 28, 2024

Decrease lock contention on imports by sending start and end events all at once.

@op op requested a review from danielnorberg October 28, 2024 13:07
@danielnorberg
Copy link
Contributor

With the current internal telemetry collection implementation emitting start and stop events at the same time would cause the resulting spans to become zero length, although maybe that's something that can be fixed.

@dflemstr
Copy link
Contributor

Maybe out of scope for this particular PR but I wonder how we can attack the main bottleneck of this? If the main issue is contending for the queue, this PR reduces the overhead by a constant factor which is great but I guess the contention is still O(n) in terms of something (nr of threads)?

@danielnorberg
Copy link
Contributor

I would be surprised if contention is the problem given that python is mostly single threaded. It might be good to benchmark and measure to get a sense of where the bottleneck is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants