[aw-failures] Copilot-engine agent step failing immediately (exit 22, zero tokens) — systemic, 2026-06-08

[github-actions]

Summary

• Incident: Copilot-engine agentic workflows fail at the agent step the instant it starts — exit code 22, 0 tokens, 0 turns, no agent artifact uploaded.
• Scope: 6 confirmed scheduled Copilot workflows, ~06:42–08:12 UTC on 2026-06-08.
• Status: ❌ Ongoing at time of analysis (latest failure 08:12 UTC). Claude-engine workflows in the same window are not affected at the agent layer.
• Coverage gap: Each failure already has an individual [aw] * failed issue, but there was no consolidated root-cause rollup for this cluster. This issue is that rollup.
• Recommended P0: Restore Copilot-engine egress/auth — the engine's HTTP requests to the model endpoint are terminated before any response headers (transaction-end-before-headers), looping until the step aborts.

Failure Cluster — Copilot engine, exit 22 (P0)

All members share an identical signature: activation succeeds → agent fails (exit 22) → conclusion succeeds; detection and safe_outputs skipped; zero token usage, meaning the engine never completed a single model call.

Workflow	Engine	Run	Tracking issue	turns / tokens
Auto-Triage Issues	copilot	§27124575784	#37777	– / 0
PR Triage Agent	copilot	§27124164930	#37776	2 / 0
Daily Hippo Learn	copilot	§27124152679	#37775	– / 0
PR Sous Chef	copilot	§27121980478	#37773	– / 0
GitHub Remote MCP Authentication Test	copilot	§27121210064	#37768	– / 0
Daily CLI Tools Exploratory Tester	copilot	§27120569486	#37767	– / 0

Evidence

1. Agent step exits 22 with no work done. audit of all six runs reports error_count=1, the single error being ##[error]Process completed with exit code 22 at the agent job; token_usage and turns are null/zero. Exit 22 is the conventional "HTTP error returned" code for the engine's outbound request.
2. Network layer terminates the request before headers. The firewall (squid) access log for run 27124575784 shows a repeating entry every ~2s: NONE_NONE:HIER_NONE error:transaction-end-before-headers — the connection to the model endpoint ends before any response headers arrive, then retries, until the step gives up.
3. No agent artifact is produced. The failing runs upload no sandbox/agent logs (only baseline comparison runs have them), confirming the engine died before initializing its session.
4. Engine-scoped, not fleet-wide. Claude-engine workflows in the same window ran normally at the agent layer — e.g. Safe Output Health Monitor (§27118532206) completed 35 turns / 9.4M tokens with agent success (its run-level failure is downstream and unrelated).

Per-run audit detail

27124575784 Auto-Triage Issues   agent=failure  exit 22  turns=-   tokens=0
27124164930 PR Triage Agent       agent=failure  exit 22  turns=2   tokens=0
27124152679 Daily Hippo Learn     agent=failure  exit 22  turns=-   tokens=0
27121980478 PR Sous Chef          agent=failure  exit 22  turns=-   tokens=0
27121210064 GitHub Remote MCP Auth agent=failure exit 22  turns=-   tokens=0
27120569486 Daily CLI Tools       agent=failure  exit 22  turns=-   tokens=0

squid (run 27124575784): repeated NONE_NONE:HIER_NONE error:transaction-end-before-headers "-" at ~2s cadence.

Correlation With Existing Issues

• Symptom-level coverage exists; root-cause coverage did not. [aw] Daily CLI Tools Exploratory Tester failed #37767, [aw] GitHub Remote MCP Authentication Test failed #37768, [aw] PR Sous Chef failed #37773, [aw] Daily Hippo Learn failed #37775, [aw] PR Triage Agent failed #37776, [aw] Auto-Triage Issues failed #37777 are auto-filed per-run failure notices. They are linked below as sub-issues of this rollup.
• Distinct from the wave-1 smoke cascade [aw] Failure cascade detected #37721 (02:11–03:11 UTC). That cascade is multi-engine smoke-test failures where the agent did execute (Claude 4 turns, Codex 1 turn, Gemini 156k tokens/exit 123) — a different signature and root cause. Not closing it here.
• Auto-managed trackers left untouched: [aw] No-Op Runs #37689 (No-Op Runs), [aw] Detection Runs #37758 (Detection Runs), [agentic-token-audit] Daily Token Usage Audit — 2026-06-05 #37109 (token-audit). These are not failures.
• No issues closed this run: the incident is unresolved (failures continuing at 08:12 UTC) and every individual issue is fresh (today), so none are stale or fixed.

Fix Roadmap

• P0 — Restore Copilot-engine egress/auth. Verify the firewall allowlist still permits the Copilot model + token-broker endpoints, confirm the Copilot token/credential is valid and not expired, and check provider status. The transaction-end-before-headers loop points to a blocked or failing connection to the model endpoint rather than an application bug.
• P1 — Surface the real HTTP status on agent-step failure. Exit 22 alone is uninformative and the failing-step logs are root-owned (mode 600, unreadable to the investigator). Have the engine wrapper echo the upstream status code / response snippet into the visible step log before exiting.
• P2 — Auto-rollup non-smoke cascades. The wave-1 smoke cascade auto-generated rollup [aw] Failure cascade detected #37721, but this wave-2 scheduled-workflow cascade produced none. Extend cascade detection to scheduled production workflows so same-signature clusters self-aggregate.

Success Criteria / Verification

• Next scheduled Copilot runs report agent job success with token_usage > 0.
• squid logs no longer show repeated transaction-end-before-headers for the model endpoint.
• No new [aw] * failed issues with the exit-22 / zero-token signature.
• Once verified, batch-close the six linked sub-issues and this rollup.

Investigation limitations

• The deterministic pre-fetch payload (prefetch.json) reported failed_run_ids: [] / failures: [] — stale; live gh queries were used instead. Its existing_tracking_issues listed only 2 of 30+ open agentic-workflows issues.
• Failing-step logs (job-*-step-*-failed.log) are root-owned with mode 600 and could not be read, so the exact upstream HTTP status was inferred from exit code 22 + squid evidence, not read directly.
• gh GraphQL operations returned 403; REST (gh api) was used. Firewall blocked api.github.com pagination beyond page 1 (boundary respected).

References:

• §27124575784 — Auto-Triage Issues (representative)
• §27121210064 — GitHub Remote MCP Authentication Test
• §27118532206 — Safe Output Health Monitor (Claude, agent succeeded — comparator)

Generated by 🔍 [aw] Failure Investigator (6h) · 270.2 AIC · ⌖ 13.2 AIC · ⊞ 5.3K · ◷

• expires on Jun 15, 2026, 12:49 AM UTC-08:00

---

✅ Resolved — Copilot-engine egress restored; agent steps executing normally

Close this rollup. The systemic exit-22 / zero-token signature is gone — Copilot model egress (api.githubcopilot.com) is restored and scheduled Copilot agent steps now complete with normal token usage. The single P0 hypothesis in this issue is confirmed and the incident has cleared.

Recovery evidence (6h window, 08:10–14:10 UTC, 2026-06-08)

Workflow	Incident run (failed)	Recovered run (schedule)	agent	tokens
Auto-Triage Issues	§27124575784 (08:12)	§27142389599 (13:51)	✅ success	160,315
PR Triage Agent	§27124164930 (08:04)	§27141913977 (13:43)	✅ success	247,251

Auto-Triage Issues also succeeded at 12:22 and 12:40 UTC. Same schedule trigger as the failed runs — a true recovery, not a different code path.

Root cause confirmed — audit-diff (failed §27124575784 → recovered §27142389599)

• api.githubcopilot.com:443: 0 → 10 allowed requests. In the failed run the model endpoint received zero successful connections (matching the transaction-end-before-headers loop); in the recovered run it is reached normally.
• Sentry telemetry endpoint (o205451.ingest.us.sentry.io) also flowing (9 allowed) in the recovered run.

This directly validates the P0 hypothesis: outbound egress to the Copilot model endpoint was the failure, and it has been restored.

Success criteria — all met

• ✅ Scheduled Copilot runs report agent success with token_usage > 0 (2 confirmed above).
• ✅ No transaction-end-before-headers for the model endpoint in recovered runs (endpoint now allowed).
• ✅ Zero new exit-22 / zero-token agentic failures in the last 6h.

Sub-issues

Batch-closing the six linked per-run notices: #37767, #37768, #37773, #37775, #37776, #37777 — single incident, systemic cause resolved. The four daily-scheduled workflows among them (Daily CLI Tools, GitHub Remote MCP Auth, PR Sous Chef, Daily Hippo Learn) re-run on their next schedule; the auto-filer will re-open tracking only if one genuinely fails again.

P1/P2 hardening items (surface real HTTP status on agent-step failure; auto-rollup non-smoke cascades) remain valid as future work but are not blockers for closing this incident tracker.

References:

• §27142389599
• §27141913977
• §27124575784

Generated by 🔍 [aw] Failure Investigator (6h) · 155.5 AIC · ⌖ 9.46 AIC · ⊞ 5.3K · ◷