Skip to content

test: feature_llmq_singlenode.py recovered-signature cleanup timeout under TSAN #7435

Description

@thepastaclaw

Intermittent recovered-signature cleanup timeout under TSAN

Summary

feature_llmq_singlenode.py intermittently fails under the TSAN
functional-test job while waiting for an expired recovered signature and its
conflict markers to be cleaned up.

The failure occurs at the final cleanup assertion:

feature_llmq_singlenode.py:178
self.wait_for_sigs(False, False, False, 15)
AssertionError: ... not true after 60.0 seconds

The test advances mock time to 6.5 days after the recovered signature was
created, verifies that it remains present, advances another day, then waits for
NetSigning::WorkThreadSigning() / CSigningManager::Cleanup() to remove it.
Under loaded TSAN runs, the state can remain visible for the full timeout on
every retry.

Evidence

Scope / root-cause direction

This predates #7415 and is not caused by that PR's pending sig-share
admission/verification caps. The failing assertion concerns expiration of an
already recovered signature in CRecoveredSigsDb, after signature recovery
and quorum rotation are complete.

The likely race is around mocked-time advancement and the asynchronous
five-second cleanup loop (CleanupThrottler<NodeClock> in
NetSigning::WorkThreadSigning()), or around which of hasrecsig / conflict
state remains cached after cleanup. The current failure output does not identify
which predicate component remains true, so the test should log each node's
three observed values before timeout.

Reproduction

  1. Build with ThreadSanitizer using the linux64_tsan CI configuration.

  2. Run the extended functional suite, or repeatedly run:

    test/functional/test_runner.py feature_llmq_singlenode.py --jobs=1 --timeout-factor=4
  3. Add concurrent functional-test load to increase the reproduction rate.

  4. Observe a timeout after Fast forward 1 day, recovered sig should not be valid anymore at line 178.

Suggested investigation

  • Log hasrecsig and both isconflicting results per node on each cleanup wait.
  • Log recovered-signature cleanup cutoff/write times and whether the
    database/cache entries are erased.
  • Make the test synchronize with completion of the cleanup pass after advancing
    mock time instead of relying only on wall-clock polling.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions