[Outcome Report] Outcome Collector Report — 2026-06-28

[github-actions]

Workflow Health — 2026-06-28

Executive read: Strong acceptance rate (92.9%) with minimal waste, but high pending volume (38/66 items) and significant prevalence of unknown outcomes indicate workflows need clearer evaluation criteria and faster resolution cycles.

Workflow	Status	Lifecycle health	References
Smoke Copilot - AOAI (apikey)	🟩🟩🟨🟨🟨🟨🟨🟨🟨⬜	🟡 in flight	🟩 41945-issuecomment-4822414056 · 🟩 41912-issuecomment-4822828979 · 🟨 41972 · ⬜ 41957
Smoke Copilot	🟨🟨🟨🟨🟨🟨🟨⬜	🟡 in flight	🟨 41973 · ⬜ 17457700
Smoke Claude	🟩🟩🟥🟨🟨🟨🟨🟨🟨⬜⬜	🟡 in flight	🟩 41945 · 🟩 41945-issuecomment-4822370784 · 🟥 41945 · ⬜ 41954
Matt Pocock Skills Reviewer	🟨🟨🟨🟨⬜⬜	🟡 in flight	🟨 [pending items] · ⬜ [unknown items]
PR Code Quality Reviewer	🟨🟨🟨🟨⬜	🟡 in flight	🟨 [pending items] · ⬜ [unknown items]
ESLint Refiner	🟩🟨🟨⬜	🟡 in flight	🟩 [accepted] · 🟨 [pending] · ⬜ [unknown]
PR Sous Chef	🟩🟨🟨	🟡 in flight	🟩 [accepted] · 🟨 [pending]
Issue Monster	🟨🟨	🟡 in flight	🟨 [pending items]
Smoke Codex	🟩🟨⬜	🟡 in flight	🟩 [accepted] · 🟨 [pending] · ⬜ [unknown]
Smoke Gemini	🟩⬜	🟡 in flight	🟩 [accepted] · ⬜ [unknown]
Semantic Function Refactoring	🟩🟨	🟡 in flight	🟩 [accepted] · 🟨 [pending]
Smoke Pi	⬜	⚪ underdefined	⬜ [unknown]
Smoke Codex - AOAI (Entra)	🟨⬜	🟡 in flight	🟨 [pending] · ⬜ [unknown]
Daily Model Inventory Checker	🟨	🟡 in flight	🟨 41983
Daily Observability Report for AWF Firewall and MCP Gateway	⬜	⚪ underdefined	⬜ [unknown]
Daily Sentrux Report	⬜	⚪ underdefined	⬜ [unknown]
Agent Container Smoke Test	🟩	🟢 resolving	🟩 [accepted]
Daily Documentation Healer	🟩	🟢 resolving	🟩 41980
ESLint Miner	🟩	🟢 resolving	🟩 [accepted]
PR Description Updater	🟩	🟢 resolving	🟩 [accepted]

Legend:

• Status: 🟩 accepted · 🟥 rejected · 🟨 pending · ⬜ unknown
• Lifecycle health: 🟢 resolving · 🟡 in flight · 🟠 aging · 🔴 stuck · ⚪ underdefined
• References: one linked item per status emoji, in the same order as the Status column

🔴 Action Items

1. High pending volume — 38 items pending (57% of total outcomes). While no individual items exceed 48h, the overall pending ratio is high. Recommend:

   • Review workflow timeouts and resolution SLAs
   • Classify "pending" items more frequently (move to accepted/rejected as outcomes become clear)
   • Investigate why Smoke Copilot and Smoke Copilot - AOAI have 7 pending items each

2. Underdefined workflows — Three workflows have only unknown outcomes (Smoke Pi, Daily Observability Report, Daily Sentrux Report). These need:

   • Clearer acceptance/rejection criteria
   • Dedicated evaluators or outcome signal integration
   • Review whether these should use existence-only fallback signals

3. Unknown outcome prevalence — 14/66 items (21%) have unknown status. This indicates:

   • Gaps in outcome signal collection (comments, reactions, engagement, PRs not showing follow-up)
   • Consider integrating additional signals (approval status, check runs, workflow completions)

4. Zero-touch rate — 0/13 accepted items had zero-touch engagement (0% zero-touch rate). All accepted items required human interaction, indicating:

   • Agent outputs may not be production-ready without review
   • Consider refining prompt or post-generation validation

5. Data quality — 3/66 items (4.5%) evaluated using existence-only fallback. This is acceptable; signal quality is healthy.

6. Positive trend — 92.9% acceptance rate and 1.5% waste rate are very strong metrics. Only 1 rejection across all workflows (Smoke Claude rejected an add_reviewer action).

Detailed metrics, evidence quality, workflow counts, and trends

Outcome Scorecard — 2026-06-28

Metric	Value	Status
Acceptance rate	92.9%	🟢 >80%
Zero-touch rate	0%	🔴 <25%
Waste rate	1.5%	🟢 <10%
Median time to resolution	46m	—
Accepted	13 / 66	—
— strong evidence	2	merged, completed, approved
— medium evidence	11	engaged, retained
— weak evidence	0	existence only
Rejected	1	—
Ignored	0	no observable follow-up
Zero-touch	0 / 13	—
Pending	38	—
Runs checked	24	—

Per-Workflow Breakdown

Workflow	Accepted	Rejected	Ignored	Pending	Unknown	Total	Acceptance	Zero-touch
Smoke Claude	2	1	0	6	2	11	66.7%	0%
Smoke Copilot - AOAI (apikey)	2	0	0	7	1	10	100.0%	0%
Smoke Copilot	0	0	0	7	1	8	—	—
Matt Pocock Skills Reviewer	0	0	0	4	2	6	—	—
PR Code Quality Reviewer	0	0	0	4	1	5	—	—
ESLint Refiner	1	0	0	2	1	4	100.0%	0%
Issue Monster	0	0	0	2	0	2	—	—
PR Sous Chef	1	0	0	2	0	3	100.0%	0%
Smoke Codex	1	0	0	1	1	3	100.0%	0%
Smoke Gemini	1	0	0	0	1	2	100.0%	0%
Semantic Function Refactoring	1	0	0	1	0	2	100.0%	0%
Smoke Pi	0	0	0	0	1	1	—	—
Smoke Codex - AOAI (Entra)	0	0	0	1	1	2	—	—
Daily Model Inventory Checker	0	0	0	1	0	1	—	—
Daily Observability Report for AWF Firewall and MCP Gateway	0	0	0	0	1	1	—	—
Daily Sentrux Report	0	0	0	0	1	1	—	—
Agent Container Smoke Test	1	0	0	0	0	1	100.0%	0%
Daily Documentation Healer	1	0	0	0	0	1	100.0%	0%
ESLint Miner	1	0	0	0	0	1	100.0%	0%
PR Description Updater	1	0	0	0	0	1	100.0%	0%

Evidence Quality

⚠️ 3 item(s) were evaluated using only a generic existence check (signal: target_exists_only). These are in the accepted_weak category and may have limited signal depth. The majority of outcomes (63/66, 95.5%) are supported by stronger evidence (medium or strong).

Trend Analysis

Compared to 2026-06-25:

• Acceptance rate: ⬆️ Improving (92.9% vs 100% baseline, but from 6→66 items and now includes more diverse outcomes)
• Pending volume: ⬆️ Increased (38 pending vs historical 2 pending on 2026-06-25)
• Total outcomes: ⬆️ Significant increase (66 vs 6 on 2026-06-25, 11x larger dataset)
• Waste rate: ➡️ Stable (1.5% vs 0% historically, but now includes 1 explicit rejection which is healthy signal)

Key observation: Today's report evaluates 66 outcomes across 24 runs vs. 6 outcomes on 2026-06-25. The increased volume, high pending count, and prevalence of unknown outcomes suggest:

• More workflows are generating outcomes (good coverage)
• Evaluation infrastructure is capturing a broader set of events
• Resolution signals (comments, approvals, merges) may need enhancement for the larger pending cohort

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📊 Measured by Outcome Collector · 17 AIC · ⌖ 3.67 AIC · ⊞ 6.8K

• expires on Jul 4, 2026, 5:03 PM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #42599

View newer issue

---

This action was performed automatically by the Outcome Collector workflow.