Skip to content

Fix race in shared-stream overflow test that could hang the suite#68652

Closed
potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:fix-flaky-shared-stream-overflow-test
Closed

Fix race in shared-stream overflow test that could hang the suite#68652
potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:fix-flaky-shared-stream-overflow-test

Conversation

@potiuk

@potiuk potiuk commented Jun 17, 2026

Copy link
Copy Markdown
Member

The scheduled canary flaked on
test_ack_mode_queue_full_during_fanout_does_not_break_iteration with a
TimeoutError (e.g. the LowestDeps ARM run).

Root cause is a race in the test, not the production code. The test
pre-fills trigger 1's subscriber queue with a sentinel so the background poll
task hits QueueFull and overflow-fails it at fan-out. But it then started
draining s_full immediately, so depending on event-loop scheduling the drain
could consume the sentinel first — leaving the queue no longer full when the
burst was fanned out. The put_nowait then succeeded, the overflow under test
never happened, and the iteration blocked forever (surfacing as the
TimeoutError). It passed most of the time only because the poll task usually
won the race.

Fix: synchronise on the overflow — wait until trigger 1 is recorded in
_failed_subscribers before draining s_full. The test is now deterministic
(passes 6/6 locally in ~1s instead of hanging), and the production overflow
path is unchanged.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_ack_mode_queue_full_during_fanout_does_not_break_iteration pre-fills a
subscriber queue with a sentinel and expects the background poll task to hit
QueueFull and overflow-fail that subscriber at fan-out. The test then started
draining the subscriber immediately, so depending on event-loop scheduling the
drain could consume the sentinel first — leaving the queue no longer full when
the burst was fanned out, so no overflow occurred and the iteration blocked
forever (surfacing as a TimeoutError under load, e.g. the LowestDeps canary).

Synchronise on the overflow (wait until the subscriber is recorded in
_failed_subscribers) before draining, making the test deterministic.
@potiuk

potiuk commented Jun 17, 2026

Copy link
Copy Markdown
Member Author

Almost identical #68651 merged before.

@potiuk potiuk closed this Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant