[audit-workflows] 🔍 Agentic Workflow Audit — 2026-07-16 (🟢 healthy-on-baseline, 91.7% prod)

[github-actions[bot]]

🔍 Agentic Workflow Audit — 2026-07-16

Verdict: 🟢 HEALTHY-on-baseline. Non-smoke production success is 91.7% (66/72). The headline raw fleet rate (60.6%, 66/109) is a Smoke-CI-canary artifact — 37 of 43 failures are smoke/test workflows. Every one of the 43 failures is a 0-turn pre-agent driver failure. There were zero agent-logic failures, missing-tools, missing-data, MCP failures, or noops fleet-wide.

⚠️ Coverage caveat: partial ~5h window (07-16 16:26–21:28Z), 109/322 downloaded run dirs had a complete run_summary (MCP logs bridge hit its 120s cap). Also note a 10-day gap since the last recorded audit (07-06).

📊 Key Metrics

Metric	Value
Runs analyzed	109 (66 ✅ / 43 ❌)
Non-smoke production	66/72 = 91.7% ✅
Smoke/test failures	37 (canary + intentional)
Agent-logic failures	0
Missing tools / data	0 / 0
MCP failures / errors / noops	0 / 0 / 0
Tokens (in/out)	1.18M / 0.50M = 1.68M
Est. AI cost (window)	~$3,062

🚦 Engine Health

Engine	Runs	Success	Notes
pi	20	100%	clean
codex	1	100%	clean
claude	16	75%	4 fails, all 0-turn cross-engine driver signature
copilot	72	45.8% raw	skewed by Smoke CI canary (30 fails); non-smoke healthy

📈 Trend Charts

Fleet success rate held in the 85–95% band across late June through early July. Today's 60.6% dip is not a real regression — it reflects a burst of Smoke CI validation runs against many open agentic-workflow PRs in a short window, all hitting the same copilot 0-turn driver abort. Underlying production reliability (91.7%) is on baseline.

Token reporting remains sparse — only 11 of the last ~33 days have usable totals because metrics.TokenUsage is empty fleet-wide; values are recovered from token_usage_summary (1.68M captured today). The moving average is directional only. Fixing the token-reporting gap is needed for reliable cost tracking.

🔴 Failure Taxonomy (all 0-turn / pre-agent)

Full breakdown of 43 failures

Smoke / test workflows — 37 (not production regressions):

• Smoke CI [copilot] — 30 — chronic copilot CLI 0-turn abort. Scope escalation: now 25 PR-branch fails + 5 push/main (was push/main only). 9 fails on a single branch (copilot/add-packages-with-manifest-files). This validation harness reddens nearly every open agentic-workflow PR.
• Smoke Copilot MAI [copilot] — 4 — Max-AI-credits test, intentional.
• Smoke Claude on Copilot [claude] — 2 — 0-turn, workflow_dispatch on copilot/fix-claude-configuration.
• Smoke Copilot Sub Agents [copilot] — 1 — 0-turn.

Non-smoke production — 6 (all copilot-sdk/cross-engine 0-turn, did real work then job-failed):

• Daily Security Observability Report [copilot] — aic 168 → 0-turn job fail (§29514943122)
• PR Triage Agent [copilot] — aic 138 (§29524501227)
• Discussion Task Miner – Code Quality Improvement Agent [copilot] — aic 201
• Daily Project Performance Summary Generator (MCP Scripts) [copilot] — aic 147
• Design Decision Gate 🏗️ [claude] ×2 — aic 74 + aic 0; cross-engine driver signature on claude (§29515578151)

🎯 Recommendations

1. [HIGH — escalated] Root-cause the copilot CLI 0-turn abort in Smoke CI. Now failing on PR branches, not just push/main — it reddens most agentic-workflow PR validations and creates chronic noise that masks real regressions. Highest-ROI single fix. (linked: smoke-ci-copilot-cli-100pct-fail-on-push)
2. [HIGH] Fix the copilot-sdk longrun 0-turn driver abort. 6 production workflows did real work (aic 73–200) then failed at Execute-CLI with 0 turns; the same signature now appears cross-engine on claude (Design Decision Gate). Needs an SDK-level fix, not per-workflow patching. (linked: copilot-sdk-driver-failures, rc=25)
3. [MEDIUM] Close the token-reporting gap. metrics.TokenUsage empty fleet-wide; only token_usage_summary yields data. Cost/token trends are unreliable until fixed. (linked: token-usage-reporting-gap)

✅ What's working

pi (20/20) and codex (1/1) fully clean. Zero missing-tools, missing-data, MCP failures, agent-logic errors, or noops across all 109 runs. Production reliability at 91.7% is squarely on baseline. No novel failure modes detected — everything maps to known chronic 0-turn driver clusters.

References: §29514943122 · §29515578151 · §29524501227

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · 190.1 AIC · ⌖ 40.3 AIC · ⊞ 7.4K · ◷

• expires on Jul 17, 2026, 1:47 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #46310.