Intermittent recovered-signature cleanup timeout under TSAN
Summary
feature_llmq_singlenode.py intermittently fails under the TSAN
functional-test job while waiting for an expired recovered signature and its
conflict markers to be cleaned up.
The failure occurs at the final cleanup assertion:
feature_llmq_singlenode.py:178
self.wait_for_sigs(False, False, False, 15)
AssertionError: ... not true after 60.0 seconds
The test advances mock time to 6.5 days after the recovered signature was
created, verifies that it remains present, advances another day, then waits for
NetSigning::WorkThreadSigning() / CSigningManager::Cleanup() to remove it.
Under loaded TSAN runs, the state can remain visible for the full timeout on
every retry.
Evidence
Scope / root-cause direction
This predates #7415 and is not caused by that PR's pending sig-share
admission/verification caps. The failing assertion concerns expiration of an
already recovered signature in CRecoveredSigsDb, after signature recovery
and quorum rotation are complete.
The likely race is around mocked-time advancement and the asynchronous
five-second cleanup loop (CleanupThrottler<NodeClock> in
NetSigning::WorkThreadSigning()), or around which of hasrecsig / conflict
state remains cached after cleanup. The current failure output does not identify
which predicate component remains true, so the test should log each node's
three observed values before timeout.
Reproduction
-
Build with ThreadSanitizer using the linux64_tsan CI configuration.
-
Run the extended functional suite, or repeatedly run:
test/functional/test_runner.py feature_llmq_singlenode.py --jobs=1 --timeout-factor=4
-
Add concurrent functional-test load to increase the reproduction rate.
-
Observe a timeout after Fast forward 1 day, recovered sig should not be valid anymore at line 178.
Suggested investigation
- Log
hasrecsig and both isconflicting results per node on each cleanup wait.
- Log recovered-signature cleanup cutoff/write times and whether the
database/cache entries are erased.
- Make the test synchronize with completion of the cleanup pass after advancing
mock time instead of relying only on wall-clock polling.
Intermittent recovered-signature cleanup timeout under TSAN
Summary
feature_llmq_singlenode.pyintermittently fails under the TSANfunctional-test job while waiting for an expired recovered signature and its
conflict markers to be cleaned up.
The failure occurs at the final cleanup assertion:
The test advances mock time to 6.5 days after the recovered signature was
created, verifies that it remains present, advances another day, then waits for
NetSigning::WorkThreadSigning()/CSigningManager::Cleanup()to remove it.Under loaded TSAN runs, the state can remain visible for the full timeout on
every retry.
Evidence
all three attempts failed at the cleanup assertion.
on 2026-06-09: the same test exhausted all three retries at the same line-178
cleanup assertion.
on 2026-05-07: the test failed its first attempt and passed its retry,
demonstrating intermittent behavior before fix: bound pending sig share queue #7415.
Scope / root-cause direction
This predates #7415 and is not caused by that PR's pending sig-share
admission/verification caps. The failing assertion concerns expiration of an
already recovered signature in
CRecoveredSigsDb, after signature recoveryand quorum rotation are complete.
The likely race is around mocked-time advancement and the asynchronous
five-second cleanup loop (
CleanupThrottler<NodeClock>inNetSigning::WorkThreadSigning()), or around which ofhasrecsig/ conflictstate remains cached after cleanup. The current failure output does not identify
which predicate component remains true, so the test should log each node's
three observed values before timeout.
Reproduction
Build with ThreadSanitizer using the
linux64_tsanCI configuration.Run the extended functional suite, or repeatedly run:
Add concurrent functional-test load to increase the reproduction rate.
Observe a timeout after
Fast forward 1 day, recovered sig should not be valid anymoreat line 178.Suggested investigation
hasrecsigand bothisconflictingresults per node on each cleanup wait.database/cache entries are erased.
mock time instead of relying only on wall-clock polling.