fix: MetaGrpcClient deadlock when drop #16727
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
drmingdrmer
added
C-bug
drmingdrmer
force-pushed
the
159-fix-meta-client-dead-lock
branch
from
939bdaa
to
dfc361d
Compare
Xuanwo
reviewed
Oct 29, 2024
drmingdrmer
force-pushed
the
159-fix-meta-client-dead-lock
branch
2 times, most recently
from
5fb20f8
to
21ffe80
Compare
Move the reference to the dedicated runtime `rt` from `MetaGrpcClient` to `ClientHandle`. `rt` is a reference to the dedicated runtime for running `MetaGrpcClient`. If all ClientHandle are dropped, the runtime will be destroyed. This `rt` previously is stored in `MetaGrpcClient`, which leads to a deadlock: - When all `ClientHandle` are dropped, the two workers `worker_loop()` and `auto_sync_interval()` will quit. - These two futures both held a reference to `MetaGrpcClient`. - The last of these(say, `F`) two will drop `MetaGrpcClient.rt` and `Runtime::_dropper` will block waiting for the runtime to shut down. - But `F` is still held, deadlock occurs. Other changes: - `Runtime::try_spawn` and several other spawn methods now accept a name argument for display in async backtrace. - Add async-backtrace to `MetaGrpcClient` methods
drmingdrmer
force-pushed
the
159-fix-meta-client-dead-lock
branch
from
21ffe80
to
cab7489
Compare
zhang2014
approved these changes
Oct 30, 2024
Xuanwo
approved these changes
Oct 30, 2024
drmingdrmer
commented
Oct 30, 2024
There was a problem hiding this comment.
Reviewed 12 of 12 files at r1.
Reviewable status: 8 of 12 files reviewed, all discussions resolved (waiting on @everpcpc)
drmingdrmer
commented
Oct 30, 2024
There was a problem hiding this comment.
Reviewed 1 of 1 files at r2, 3 of 3 files at r3, all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @everpcpc and @Xuanwo)
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
fix: MetaGrpcClient deadlock when drop
Move the reference to the dedicated runtime
rtfromMetaGrpcClientto
ClientHandle.rtis a reference to the dedicated runtime for runningMetaGrpcClient.If all ClientHandle are dropped, the runtime will be destroyed.
This
rtpreviously is stored inMetaGrpcClient, which leads to a deadlock:ClientHandleare dropped, the two workersworker_loop()andauto_sync_interval()will quit.
MetaGrpcClient.F) two will dropMetaGrpcClient.rtandRuntime::_dropperwill block waiting for the runtime to shut down.
Fis still held, deadlock occurs.Other changes:
Runtime::try_spawnand several other spawn methods now accept a nameargument for display in async backtrace.
Add async-backtrace to
MetaGrpcClientmethodsTests
Type of change
Related Issues
This change is