Implement blocking eventfd #3939
Conversation
|
@rustbot author |
rustbot
added
the
S-waiting-on-author
|
Hmm...why is rustfmt complaining? Let me try rebase it instead. |
|
☔ The latest upstream changes (presumably #3951) made this pull request unmergeable. Please resolve the merge conflicts. |
This comment was marked as resolved.
This comment was marked as resolved.
src/shims/unix/linux/eventfd.rs
Outdated
| let mut waiter = Vec::new(); | ||
| let mut blocked_write_tid = eventfd.blocked_write_tid.borrow_mut(); | ||
| while let Some(tid) = blocked_write_tid.pop() { | ||
| waiter.push(tid); | ||
| } | ||
| drop(blocked_write_tid); | ||
|
|
||
| waiter.sort(); | ||
| waiter.dedup(); | ||
| for thread_id in waiter { | ||
| ecx.unblock_thread(thread_id, BlockReason::Eventfd)?; | ||
| } |
There was a problem hiding this comment.
I couldn't think of a testcase that can actually produce thread id duplication in blocked_write/read_id, but I will just keep the dedup here first.
There was a problem hiding this comment.
we'll get a panic if that happens, so maybe just don't sort and dedup and wait until we get a test case. I don't think it can happen though, as a thread would need to get blocked again without having been removed and unblocked.
There was a problem hiding this comment.
Also: the unblocking order is user-visible, and tho I assume that eventfd does not specify which thread gets woken first, we'd probably want a random (deterministic, but depending on the seed) order of unblocking to happen
cc @RalfJung I think we need a new system for unblocking multiple threads at the same time. So far we only ever unblocked one thread, and "unblock N threads, then randomly run them with the normal rules for picking the next thread" seems like a recurring thing.
Also I wonder if the normal thread unblocking is subtly wrong, too. Or just wrong in this PR:
When a thread gets unblocked and immediately performs some operation that is visible from other threads, then that means when the current thread continues after unblocking the other thread, it may behave as if the unblocked thread already had a few CPU cycles to do something. While that may be expected behaviour depending on preemption, the behaviour here does not depend on preemption. So, to avoid such a footgun, thread unblocking should not immediately execute the unblocking operation, but just unblock the thread and make the thread execute the unblocking operation when it gets scheduled next.
|
@rustbot ready |
rustbot
added
S-waiting-on-review
src/shims/unix/linux/eventfd.rs
Outdated
| let mut waiter = Vec::new(); | ||
| let mut blocked_write_tid = eventfd.blocked_write_tid.borrow_mut(); | ||
| while let Some(tid) = blocked_write_tid.pop() { | ||
| waiter.push(tid); | ||
| } | ||
| drop(blocked_write_tid); | ||
|
|
||
| waiter.sort(); | ||
| waiter.dedup(); | ||
| for thread_id in waiter { | ||
| ecx.unblock_thread(thread_id, BlockReason::Eventfd)?; | ||
| } |
There was a problem hiding this comment.
we'll get a panic if that happens, so maybe just don't sort and dedup and wait until we get a test case. I don't think it can happen though, as a thread would need to get blocked again without having been removed and unblocked.
src/shims/unix/linux/eventfd.rs
Outdated
| let mut waiter = Vec::new(); | ||
| let mut blocked_write_tid = eventfd.blocked_write_tid.borrow_mut(); | ||
| while let Some(tid) = blocked_write_tid.pop() { | ||
| waiter.push(tid); | ||
| } | ||
| drop(blocked_write_tid); | ||
|
|
||
| waiter.sort(); | ||
| waiter.dedup(); | ||
| for thread_id in waiter { | ||
| ecx.unblock_thread(thread_id, BlockReason::Eventfd)?; | ||
| } |
There was a problem hiding this comment.
Also: the unblocking order is user-visible, and tho I assume that eventfd does not specify which thread gets woken first, we'd probably want a random (deterministic, but depending on the seed) order of unblocking to happen
cc @RalfJung I think we need a new system for unblocking multiple threads at the same time. So far we only ever unblocked one thread, and "unblock N threads, then randomly run them with the normal rules for picking the next thread" seems like a recurring thing.
Also I wonder if the normal thread unblocking is subtly wrong, too. Or just wrong in this PR:
When a thread gets unblocked and immediately performs some operation that is visible from other threads, then that means when the current thread continues after unblocking the other thread, it may behave as if the unblocked thread already had a few CPU cycles to do something. While that may be expected behaviour depending on preemption, the behaviour here does not depend on preemption. So, to avoid such a footgun, thread unblocking should not immediately execute the unblocking operation, but just unblock the thread and make the thread execute the unblocking operation when it gets scheduled next.
|
@RalfJung we raced on the review, do you have any thoughts on #3939 (comment) |
|
I think for now making wakeup deterministic with a queue (via VecDeque) is fine. We can leave an issue to randomize this.
Regarding the unblocking callback, it is required for correctness right now to run immediately. As long as this does not perform atomic ops (that can be observed from other threads), I think that is fine? What exact issue are you concerned about?
|
well, in this case, that we perform the reads and writes of the unblocked thread immediately, but then continue on the current thread. So unblocking a thread affects the current thread's next operations (there can now be more space to write to, even tho no thread switch happened). This makes no-preemption tests harder to write, but is not a fundamental issue I guess, just weird and not behavior a real system would have |
Oh that's surprising. I always assumed the current thread should finish first before executing other newly unblocked threads. But I think I observed what is mentioned here before, and I was pretty confused by the execution sequence. |
| // 1. Thread 1 blocks. | ||
| // 2. Thread 2 blocks. | ||
| // 3. Thread 3 unblocks both thread 1 and thread 2. | ||
| // 4. Either thread 1 or thread 2 writes u64::MAX. |
There was a problem hiding this comment.
Due to the preemption rate being zero, this is deterministic no matter the seed, right?
Why is it thread 3 that gets to write first and not thread 2? How does that ordering of the unblocks happen?
There was a problem hiding this comment.
Yes it is deterministic, it consistently blocked on thread2 when running with many-seeds . The unblock sequence is related to the unblocking order below:
let waiting_threads = std::mem::take(&mut *eventfd.blocked_write_tid.borrow_mut());
for thread_id in waiting_threads {
ecx.unblock_thread(thread_id, BlockReason::Eventfd)?;
}In the test, thread1 gets blocked before thread2, so during the unblock, thread1 gets unblocked before thread2. Hence thread1 gets to return from eventfd_write first, while thread2 hits the blocking condition again.
|
Please squash the changes |
|
Thanks! I left FIXME for randomizing unblock sequence, and slightly changed the comments for expected execution in |
This PR
fail-depBlockReason::Eventfdcc #3665