Skip to content

Baseline fetch (fetchPreviousRun) can hang forever — 10s timeout doesn't survive a half-open socket during body read #163

Description

@veksen

This was generated by AI during triage.

Summary

fetchPreviousRun (the CI baseline fetch, src/reporters/site-api.ts / driven from src/main.ts) can hang indefinitely. Its 10s AbortSignal.timeout(10000) does not survive a half-open socket during body read, so a transient Site API stall becomes a full CI-job hang that only ends at the job's 20-minute timeout.

Evidence

Observed in GitHub Actions run 28626422344 (repo veksen-enterprises/d2armory, PR #907, test job 84893821211, attempt 1). Tests passed in ~70s; the job then hung on the final query-doctor/analyzer@main step and was killed by the job's timeout-minutes: 20 ("The operation was canceled" at 23:08:21, exactly 20 min after job start 22:48:09).

Log trace:

22:50:24.09  Site API ingestion successful: {"id":"019f2506-..."}   # POST /ci/runs = HTTP 200, ~1s
22:50:24.09  Fetching previous run from https://api.querydoctor.com/ci/runs/latest?repo=veksen-enterprises%2Fd2armory&branch=main&excludeId=...
22:50:25.85  [main] API connection broken during CI run: Error: RPC session was shut down by disposing the main stub
             ...total silence until the 23:08:21 kill (~18 min)

Key point: none of fetchPreviousRun's own failure/retry log lines appeared — no Failed to fetch previous run, no Retrying baseline fetch (attempt 2/3). So the 10s timeout never took effect. Execution was sitting in await response.json() on the multi-MB /ci/runs/latest response after headers had already arrived, and the abort did not interrupt the stalled body read once the server (capnweb / Cloudflare-Workers "main stub") was disposed mid-stream and left a half-open TCP socket (no FIN/RST).

The ingest POST succeeded first — this is not the dup-hash ingest 500.

Fix

Enforce a hard wall-clock deadline covering the entire fetch + parse, not just the initial request:

  • Promise.race([fetchAndParse(), rejectAfter(deadlineMs)]), or a total-deadline AbortSignal that also aborts body consumption, so await response.json() on a half-open socket is guaranteed to reject.
  • On deadline, fall through to the existing { kind: "error" }comparisonUnavailable path ("comparison temporarily unavailable — re-run") instead of hanging.

Result: a stalled baseline fetch degrades in ~10s instead of hanging 18+ minutes.

Severity

Medium. Not currently firing (the run self-resolved on re-run with no client change, consistent with a Site API restart), but it recurs on any api.querydoctor.com blip during the baseline fetch, and each recurrence is a 20-minute hung job + red PR check.

Related

  • Root-cause counterpart (server side): Query-Doctor/Site — RPC "main stub" disposal leaving a half-open response socket.
  • Optional CI guardrail: veksen-enterprises/d2armory — step-level timeout on the analyzer step.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions