test_deadlock & test_threads sometimes deadlock on Windows refleaks buildbot

[encukou]

The “AMD64 Windows11 Refleaks 3.12” sometimes deadlocks with:

3:58:18 load avg: 0.07 running (1): test.test_concurrent_futures.test_deadlock (2 hour 34 min)
3:58:48 load avg: 0.05 running (1): test.test_concurrent_futures.test_deadlock (2 hour 35 min)
3:59:18 load avg: 0.03 running (1): test.test_concurrent_futures.test_deadlock (2 hour 35 min)
3:59:48 load avg: 0.02 running (1): test.test_concurrent_futures.test_deadlock (2 hour 36 min)
command timed out: 14400 seconds elapsed running [b'Tools\\buildbot\\test.bat', b'-p', b'x64', b'-j2', b'-R', b'3:3', b'-u-cpu', b'-j2', b'--timeout', b'12000'], attempting to kill
4:00:18 load avg: 0.03 running (1): test.test_concurrent_futures.test_deadlock (2 hour 36 min)
program finished with exit code 1
elapsedTime=14430.167703

On some builds, a different test fails:

3:58:59 load avg: 0.03 running (1): test.test_multiprocessing_spawn.test_threads (1 hour 43 min)
3:59:29 load avg: 0.06 running (1): test.test_multiprocessing_spawn.test_threads (1 hour 44 min)
command timed out: 14400 seconds elapsed running [b'Tools\\buildbot\\test.bat', b'-p', b'x64', b'-j2', b'-R', b'3:3', b'-u-cpu', b'-j2', b'--timeout', b'12000'], attempting to kill
3:59:59 load avg: 0.04 running (1): test.test_multiprocessing_spawn.test_threads (1 hour 44 min)
program finished with exit code 1
elapsedTime=14430.167536

I am able to reproduce locally, by running these two (or this and test_concurrent_futures.test_shutdown) in parallel. The deadlock occurs in test_crash_big_data.

Linked PRs

• [3.12] gh-114440: Close writer pipe in multiprocessing.Queue, not concurrent.futures #114489

[encukou]

In the 3.12 backports for gh-109047, gh-109370, and gh-107219, a backporting/rebasing error snuck in:

• test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219 adds self.call_queue._writer.close() to _ExecutorManagerThread in concurrent.futures.
• Unexpected traceback output in test_concurrent_futures and crash #109370 changes this to be only called on Windows.
• test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition) #109047 moves the call to multiprocessing.Queue's _terminate_broken.

The backports to 3.12 were applied in a different order (and with different bases), leaving the Windows-specific close call in concurrent.futures. Moving it to multiprocessing.Queue fixes test_concurrent_futures.test_deadlock for me.

[encukou]

Thanks Victor & Serhiy!

I'll close the issue but continue monitoring buildbots.

[encukou]

Victor pointed out that this might need a backport to 3.11.
( I haven't seen it on the 3.11 buildbots, and didn't try to reproduce there.)

[encukou]

I cannot reproduce on 3.11. (I am using a simplified reproducer. But I also haven't seen this fail on a 3.11 buildbot yet.)
If it comes up in the future, #109047 (356de02) probably needs to be backported before this.