Skip to content

test(rdns): fix flaky thread leak across analytics rDNS tests - #211

Merged
mousebrains merged 1 commit into
mainfrom
fix-rdns-test-thread-leak
Jun 21, 2026
Merged

test(rdns): fix flaky thread leak across analytics rDNS tests#211
mousebrains merged 1 commit into
mainfrom
fix-rdns-test-thread-leak

Conversation

@mousebrains

Copy link
Copy Markdown
Owner

Why

main went red right after #210 merged, on test_analytics_rdns.py::test_repeated_calls_dedup_targets_to_bound_budget — its calls list unexpectedly contained 9.9.9.x IPs from the previous test (test_saturated_pool_does_not_poison_unpulled_queue). This is a test-isolation thread leak, not a product bug, and unrelated to the LI-COR change that surfaced it.

Root cause

warm_rdns resolves PTRs on daemon rdns-* workers and abandons any that black-hole past its wall-clock budget (intentional — a hung gethostbyaddr must not pin the process). test_saturated_pool leaves 2 such workers hung on its gate; its finally: gate.set() revives them, and they drain the remaining queue items. By then the next test has monkeypatched _rdns_lookup, so the leaked workers append to its calls list.

It only manifests when xdist schedules the two tests back-to-back on one worker — timing-dependent, which is why the identical tree was green on #210's PR run but red on main (the merge added test files, shifting -n auto distribution). Production is unaffected: each warm_rdns call owns its queue, and _rdns_lookup is never swapped at runtime.

Fix

The rdns_state fixture now joins any lingering rdns-* thread (with a timeout) on teardown, before the next test runs. The test's monkeypatch is still active during teardown (it's set up before rdns_state, so torn down after), so the joined workers call their own test's lookup and exit cleanly — nothing leaks forward. Test-only change.

Verification

  • The file passes 5× sequential + 3× under -n auto (CI mode).
  • Full light suite: 1840 passed; ruff clean.
  • Tests-only change, so the slow deploy suite is unaffected.

(Also re-ran the red main CI run to unblock main immediately; this PR is the durable fix so it stops recurring.)

— Claude · Opus-4.8(1M) · x-high

🤖 Generated with Claude Code

…y leak)

main went red on test_analytics_rdns.py::test_repeated_calls_dedup_targets_
to_bound_budget — its `calls` list contained 9.9.9.x IPs from the PRIOR
test (test_saturated_pool_does_not_poison_unpulled_queue). Root cause: a
test-isolation thread leak, not a product bug.

warm_rdns runs lookups on daemon `rdns-*` workers and abandons any
black-holed past its wall-clock budget (intentional — survives a hung
gethostbyaddr). test_saturated_pool leaves 2 such workers hung on its
gate; the `finally: gate.set()` revives them, and they drain the
remaining queue items — but by then the next test has monkeypatched
_rdns_lookup, so the leaked workers append to ITS calls list. It only
surfaces when xdist schedules the two tests back-to-back on one worker
(timing-dependent — green on the PR run, red on main, same tree; #210's
added test files shifted the distribution). Production is unaffected:
each warm_rdns call owns its queue and _rdns_lookup isn't swapped at
runtime.

Fix: the rdns_state fixture now joins any lingering `rdns-*` thread
(timeout) on teardown, before the next test. The test's monkeypatch is
still active during teardown (set up before rdns_state, torn down after),
so the joined workers call their own test's lookup and exit cleanly.

Verified: the file passes 5x sequential + 3x under -n auto; full light
suite 1840 passed; ruff clean. (Tests-only change — slow deploy suite
unaffected.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018i8zX4iJDjh6nwLCYRVaRf
@mousebrains
mousebrains merged commit 22bfac3 into main Jun 21, 2026
9 checks passed
@mousebrains
mousebrains deleted the fix-rdns-test-thread-leak branch June 21, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant