DeepReport Intelligence Briefing — 2026-05-17

[github-actions[bot]]

Executive Summary

The gh-aw ecosystem is producing reports at high cadence — 36 discussions in the last 7 days, 26 of them audit reports — but the surface area of unresolved findings continues to outpace remediation. Today's signal is dominated by silent partial failures that show up as success in the workflow UI: a 24h window contained 11 silent 422s on submit_pull_request_review, a case-sensitive isNotFoundError bug in production, a compiler permission omission for Vale steps, and a bundle-push path that skips temporary-ID substitution. These are the kind of failures that don't trip alarms but quietly degrade output quality.

Infrastructure trended the wrong way on 2026-05-16: 30% failure rate (19/63 audited runs), Smoke CI at 64%, LintMonster at 75%. The headline cause was the Safe Outputs MCP HTTP Server startup failure (4+ recurrences) cascading across unrelated workflows. Cost was $17.59 over 5.4h. Copilot agent merge rate fell 10pp to 77.6%.

The largest quick win remains untapped from prior briefings: 63 workflows still inline the same 4-line gh-proxy + cli-proxy + toolsets: [default] block — extracting it to shared/github-proxy-default.md would clear ~252 lines and create the only single-edit lever for this pattern.

---

🚨 Top Findings

1. Silent partial failures dominate this week

Four separate silent-failure patterns surfaced, all marking workflow runs success while losing data or producing broken output:

Pattern	Count	Where	Source
submit_pull_request_review 422 (empty body/zero comments)	10 in 24h	Test Quality Sentinel	#32768
submit_pull_request_review 422 (paths not in diff)	1 in 24h	Matt Pocock Skills Reviewer	#32768
isNotFoundError case-sensitive at outcome_eval_comment.go:34	1 site (latent)	pkg/cli	#32753
Compiler omits pull-requests: read from activation job	1 site (latent)	gh aw compiler	#32846
Bundle-push path skips #aw_<id> substitution	regression in v0.72.1	safe-outputs	#32834

Why this matters: silent failures don't show up in failure-rate dashboards. The Safe Output Health Report flagged this specifically — Successful: 2 / Failed: 0 printed after a logged ##[error]. We need pre-checks and accurate counter accounting.

2. Workflow infrastructure had a rough day on 2026-05-16

• 30% failure rate across 63 audited runs (#32712)
• Smoke CI: 64% failure (7/11), LintMonster: 75% (3/4), PR Sous Chef: 38% (3/8)
• 9 OPEN [aw] failure issues as of today: Daily Cache Strategy Analyzer ([aw] Daily Cache Strategy Analyzer failed #32876), Design Decision Gate ([aw] Design Decision Gate 🏗️ failed #32870), Daily Issues Report Generator ([aw] Daily Issues Report Generator failed #32830), Draft PR Cleanup ([aw] Draft PR Cleanup failed #32797), Dev ([aw] Dev failed #32790), Daily Hippo Learn ([aw] Daily Hippo Learn failed #32776), Detection Runs ([aw] Detection Runs #32316), No-Op Runs ([aw] No-Op Runs #32279), agentic workflows out of sync ([aw] agentic workflows out of sync #32270)
• Top cause: Safe Outputs MCP HTTP Server startup failure (4+ recurrences in 24h)
• Cost outlier: Daily Safe Output Tool Optimizer at $5.99 / 28.6M tokens (~34% of daily spend, failed)

3. Schema vs. parser disagreements

The Schema Consistency Audit (#32772) caught two new high-signal drifts:

• tools.serena: schema declares x-removed: true, parser still treats it as a built-in MCP tool with explicit handling at pkg/parser/mcp.go:186, 214, 425. IDE schema integration shows REMOVED; runtime silently accepts.
• engine.bare for Codex/Gemini: docs feature matrix says ✅, schema description says supported, compiler at agent_validation.go:229 warns and ignores. Three sources disagree.

4. Helper underadoption (Sergo's persistent theme)

Four of five prior Sergo findings are fixed in main, validating that the agent's recommendations land (#32753). Two persist:

• isPermissionError centralized at pkg/cli/audit.go:250 — 3 sites already drifted (logs_download.go:312/802, logs_github_api.go:286-291 with different superset)
• isNotFoundError — pkg/parser/remote_fetch.go:610 is unexported and takes string; 4 pkg/cli sites reimplement inline; outcome_eval_comment.go:34 is silently buggy due to case-sensitive matching
• HTTP 30s timeout literal at 4 sites — Run 8 finding #aw_sg8a2, open 13+ days, no DefaultHTTPClientTimeout in pkg/constants

5. Skill-extraction debt unchanged from prior briefings

The Workflow Skill Extractor (#32054) re-confirmed three high-impact patterns still un-DRY'd:

Pattern	Workflows	Lines
cli-proxy + github: gh-proxy + toolsets: [default]	63	~252
strict: true + features: copilot-requests: true	29	~87
agentic-workflows + cli-proxy + timeout: 300	19	~76

Combined: ~415 lines across 111 distinct workflows. The existing shared/gh.md covers mode only, so the largest pattern can't be DRYed today.

6. Documentation drift for non-Copilot users

The Claude Code User Documentation Review (#32807) flagged the same items for the third day running:

• README has zero engine signal — no mention of Claude/Anthropic/Codex/Gemini
• gh aw init silently installs Copilot-only .github/agents/agentic-workflows.agent.md
• engines.md:28 still recommends against the user's chosen engine: "If you are unsure, start with Copilot."
• Quick Start ordering puts the Copilot setup callout (lines 76–80) before the Anthropic key callout (lines 82–86)

Trend: stable — no regressions, no measurable fixes either.

---

📊 Trends Worth Watching

Encouraging signals

• Copilot agent merge rate: 80.2% across 998 PRs in a 2.5-week window. Inline sub-agent cluster at 100% (n=22).
• PR velocity: avg merge time 1.1h, issue resolution 16.8h — both excellent.
• Secrets posture: 100% workflows with explicit permissions: blocks, 100% with redaction steps, 860 token-cascade fallbacks, zero raw github.event.* interpolations.
• Cache strategy: 12/12 cache-memory runs scored a cache hit today; only watch-item is Chaos PR Bundle Fuzzer (already tracked).
• Hippo memory: 531 memories, avg strength 0.92, top memory verified at score 0.994 (stale lock files).

Headwinds

• Daily success rate down to 65.17% (vs ~78% in early April).
• Code churn stability: 3/15 — weakest quality-score component. 1,222 source files modified across 455 commits in 7 days.
• Test-to-source ratio: 2.10× and slowly declining (-2.5% over 7d).
• CGO/CJS Build Regression: failing on every push since Feb 2026 (90+ days, [CGO] Workflow failure on main - Run #2565 #29669).
• Deployment Monitor: 100 runs/day at 8% success rate = ~92 wasted invocations/day (zombie pattern).
• Daily Observability Report: hit 80M effective token limit ([aw] Daily Observability Report for AWF Firewall and MCP Gateway failed #32717).
• Smoke OTEL: Sentry OTLP ingest returning 401 (Smoke OTEL - 26000186618 #32879) — GH_AW_OTEL_SENTRY_AUTHORIZATION secret invalid/expired.

Observability data gaps

• 47.4% firewall artifact coverage on firewall-enabled runs (10/19 missing access.log)
• 0/17 MCP-enabled runs exposed gateway.jsonl (all relied on rpc-messages.jsonl fallback)
• 38/38 firewall-enabled runs in 7-day window missing firewall-audit-logs artifact

---

🎯 7 Quick Wins (Issues Created)

All 7 high-impact, low-effort tasks have been filed today as issues for agent execution:

#	Quick Win	Effort	Source
1	Fix cache-memory upload-artifact path (/tmp/gh-aw/cache-memory/.)	<1h	#32712
2	Fix silent case-sensitive isNotFoundError bug at outcome_eval_comment.go:34	<2h	#32753
3	Create shared/github-proxy-default.md (63 workflows, ~252 lines)	2–3h	#32054
4	Add DefaultHTTPClientTimeout constant in pkg/constants (13+ days unfixed)	<1h	#32753
5	Add Claude/Anthropic engine mention to README.md (zero engine signal today)	<1h	#32807
6	Resolve tools.serena schema/parser disagreement (schema says REMOVED, parser accepts)	1–2h	#32772
7	Prevent silent 422 in safe-output PR review submission (11 silent failures in 24h)	2–3h	#32768

Total: <13 hours of agent time, all single-owner tasks with clear acceptance criteria.

---

📈 Headline Metrics (24h window ending 2026-05-17)

Metric	Value
Total workflow runs	178 (116 ✅ / 62 ❌)
Success rate	65.17%
GitHub REST API calls	54,436 (peak 102.3K on 2026-05-16)
Safe-output writes	95
Avg duration	266s (p95: 869s)
Top API burner	Smoke CI (29.1% / 15,863 calls)
Total cost (audit window)	$17.59
Total tokens (audit window)	41.0M raw / 241.5M effective
Engine mix (audit window)	47 copilot · 12 claude · 2 codex · 2 unknown
Codebase	1,500,870 LOC across 4,529 files
Quality score (Daily Code Metrics)	81/100
Sentrux quality signal	5226/10000 (first baseline)
Complex functions (Sentrux)	877
Test-to-source ratio	2.10× (-2.5% in 7d)
Hippo memory store	531 (478 episodic, 53 semantic, 3 verified)

🔥 Top Failing Workflows (2026-05-16 audit window)

Workflow	Runs	Failed	Rate
Smoke CI	11	7	64%
LintMonster	4	3	75%
PR Sous Chef	8	3	38%
Design Decision Gate 🏗️	4	1	25%

📦 Active [aw] Failure Issues (9 OPEN)

• Daily Cache Strategy Analyzer (#32876)
• Design Decision Gate 🏗️ (#32870)
• Daily Issues Report Generator (#32830)
• Draft PR Cleanup (#32797)
• Dev (#32790)
• Daily Hippo Learn (#32776)
• Detection Runs (#32316)
• No-Op Runs (#32279)
• agentic workflows out of sync (#32270)

🧱 Chronic Failures Worth Escalating

• CGO/CJS Build Regression — failing on every push to main since Feb 2026 (90+ days). #29669
• Deployment Monitor — 100 runs/day at 8% success rate; ~92 wasted invocations/day. Recommend deprecate or add trigger circuit-breaker.
• Daily Fact about gh-aw — ~50% success; recurring parse failures despite PR Comment out top-level on.labels in compiled workflows to prevent push-time workflow parse failures #31411 merge.
• Codex Smoke Test — attempting OPENAI_API_KEY access outside sandbox scope. #32446
• Sentry OTLP 401 — GH_AW_OTEL_SENTRY_AUTHORIZATION invalid/expired; all gh-aw spans missing from Sentry. #32879

🧠 Copilot Agent Cluster Insights (998 PRs over 2.5 weeks)

Cluster	Tasks	Merge Rate	Avg Effort
Workflow features	308	74.7%	3.75 commits / 20.9 files
Safe outputs & MCP tools	223	86.5%	4.09 / 32.3
Command triggers & HTTP	204	81.4%	3.25 / 25.1
CI/config & job runners	122	72.1%	4.37 / 66.8
Experiments & schemas	85	87.1%	3.86 / 25.5
Targeted validation / comments	23	82.6%	6.83 / 36.0
Inline sub-agents	22	100.0%	3.64 / 27.3
Failing Actions triage	11	72.7%	2.91 / 67.8

Key insight: tighter, smaller-scoped clusters merge more reliably. Inline sub-agents and Experiments & schemas (smaller diffs, tighter abstractions) outperform the bigger, churn-heavy CI/config cluster. Prompt specificity > iteration count.

---

🔄 Continuity Notes

Last DeepReport was 2026-04-03 (six weeks ago). Several patterns flagged then are still active:

• Daily Issues Report Generator — failing again today (#32830)
• Validation debt — 9/54 validators >300 lines (similar to April figure)
• Stale lock files — Hippo memory now lists this as its highest-confidence pattern (score 0.994)
• Duplicate Code Detector — no longer surfacing; status unverified

New this cycle:

• Safe Outputs MCP HTTP Server startup failure (new infra hotspot)
• 80M token limit exhaustion (Daily Observability Report [aw] Daily Observability Report for AWF Firewall and MCP Gateway failed #32717)
• Sergo helper-underadoption theme — 4/5 prior findings now fixed; thread converging
• Sentrux baseline established (5226/10000) — future runs will give 30-day trend data

---

References:

• §26000390483 (this DeepReport run)
• #32712 — Audit-Workflows Daily Audit 2026-05-16
• #32768 — Safe Output Health Report 2026-05-17

Generated by 🔬 DeepReport - Intelligence Gathering Agent · ● 24.9M · ◷

• expires on May 24, 2026, 7:40 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by DeepReport - Intelligence Gathering Agent.

A newer discussion is available at Discussion #33082.