[cli-tools-test] audit: misleading failure diagnosis — 'failed before agent activation' when agent ran for 2 minutes

[github-actions]

Problem Description

The audit tool reports a misleading failure diagnosis for run 28006289881. The key_findings description states:

Workflow Failed: Workflow 'Smoke Claude on Copilot' failed before agent activation — no error logs were available to analyze

However, the jobs data from the same audit response shows the activation job completed successfully and the agent job actually ran for 2 minutes before failing:

activation:    completed, success, duration: 37.0s
agent:         completed, failure, duration: 2.0m   ← agent ran!
conclusion:    completed, success
detection:     skipped
safe_outputs:  skipped

Command/Tool

• Tool: audit
• Command: agenticworkflows audit --run_id 28006289881 --max_tokens 3000

Steps to Reproduce

1. Call agenticworkflows audit --run_id 28006289881 --max_tokens 3000
2. Observe key_findings[0].description = "failed before agent activation — no error logs were available to analyze"
3. Compare against jobs[2] which shows agent job ran with duration: 2.0m and conclusion: failure

Expected Behavior

The failure diagnosis should correctly identify that the agent job ran and failed (not that it failed before activation). The message should say something like: "Agent job ran for 2.0m before failing. No agent telemetry was exported."

Actual Behavior

False diagnosis: key_findings reports "failed before agent activation — no error logs were available to analyze" when the agent clearly ran for 2 minutes.

Root Cause Hypothesis

The code likely checks for the presence of agent_usage.json or populated token_usage.jsonl to decide whether agent activation occurred. For run 28006289881:

• token_usage.jsonl exists but is 0 bytes
• agent_usage.json is absent from the logs directory

This causes the heuristic to incorrectly conclude the agent never started, when in fact it ran for 2 minutes and failed to export telemetry before exit.

Environment

• Repository: github/gh-aw
• Test Run ID: 28006289881 (Smoke Claude on Copilot)
• gh-aw version: 1.0.63
• Date: 2026-06-23

Impact

• Severity: Medium
• Frequency: Reproducible for any agent-job failure without telemetry export
• Workaround: Manually cross-check jobs list vs key_findings in audit output

Logs/Diagnostics

{
  "key_findings": [{
    "category": "error",
    "severity": "critical",
    "title": "Workflow Failed",
    "description": "Workflow failed before agent activation — no error logs were available to analyze"
  }],
  "jobs": [
    {"name": "activation", "conclusion": "success", "duration": "37.0s"},
    {"name": "agent",      "conclusion": "failure", "duration": "2.0m"},
    {"name": "conclusion", "conclusion": "success"}
  ]
}

Files present in /tmp/gh-aw/aw-mcp/logs/run-28006289881/:

• aw_info.json (2284 bytes)
• token_usage.jsonl (0 bytes — empty)
• run_summary.json (TokenUsage: 0, Turns: 0, ErrorCount: 0)
• agent_usage.json — absent

References: §28006976304

Generated by 🧪 Daily CLI Tools Exploratory Tester · 96.6 AIC · ⌖ 14.1 AIC · ⊞ 9.5K · ◷

• expires on Jun 29, 2026, 10:42 PM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-06-30T06:42:49.629Z.

Closed by Workflow