otel: gh-aw.agent.conclusion reports run.status=success / STATUS_CODE_OK when the Copilot CLI step exits non-zero (error_count=0)

[matiloti]

OTel conclusion span reports run.status=success / error_count=0 / STATUS_CODE_OK when the agent job's "Execute GitHub Copilot CLI" step exits non-zero

Summary

The gh-aw.agent.conclusion OTLP span emitted by the setup action's post step reports a healthy run even when the GitHub Actions run conclusion is failure because the agent job's "Execute GitHub Copilot CLI" step exited non-zero.

Observed on a failing run, the conclusion span carried:

• gh-aw.run.status = "success"
• gh-aw.error_count = "0"
• otel.status_code = "STATUS_CODE_OK"
• gh-aw.warning_count = "143"

…while the same run's GitHub Actions conclusion was failure, with the agent job failing at the "Execute GitHub Copilot CLI" step.

The net effect: real agent-run failures are invisible in OTel / Application Insights. A DevOps engineer filtering on otel.status_code = ERROR or gh-aw.run.status IN (failure, timed_out, cancelled) sees ~0 failures, while the Actions UI shows dozens. Failure-based alerting and dashboards are unusable for this class of failure.

This is a residual gap left by the earlier fixes #33037 and #35182 (see Related): those handle the "output errors present" and "conclusion env-var propagation" cases, but neither catches a Copilot-CLI step that exits non-zero while agent_output.json records zero errors (error_count=0).

Environment

• gh-aw pinned version: 0.82.9 (github/gh-aw-actions/setup@v0.82.9, digest ca8678ca22a7aab577514482576720da641e5661)
• Latest upstream at time of report: 0.82.15
• Engine: copilot, model claude-opus-4.8
• Host: GitHub Enterprise Server (soderbergpartners.ghe.com)
• Repo: S-P-Global-Innovation/AdviserNotes
• Workflow: "Jerry - PR Triage"

The 0.82.10 → 0.82.15 changelog contains OTLP-adjacent items (id-token permission fix for OTLP OIDC in 0.82.10; OTEL_SERVICE_NAME respect in 0.82.14) but no change to how the conclusion span computes run.status / status_code from the Copilot CLI exit status. Bumping the pin does not fix this.

Evidence

• is:failure on the "Jerry - PR Triage" workflow: 84 failed runs over ~30 days.
• Failures visible in OTel/telemetry for the same window: ~0.
• Representative failed run IDs (all GH conclusion=failure, agent job failed at "Execute GitHub Copilot CLI"):
  • 162095662 — conclusion span: run.status=success, error_count=0, warning_count=143, otel.status_code=STATUS_CODE_OK
  • 161841472
  • 159573952
  • 158705292

84-vs-~0: the failure count that operators can see in the Actions UI vs. what any standard OTel error query returns. The signal that dashboards and alerts depend on is systematically wrong for CLI-exit failures.

Root-cause hypothesis

The conclusion span's gh-aw.run.status / otel.status_code are derived (in actions/setup/js/send_otlp_span.cjs, the status-computation block around the historical lines 1670–1745) from:

1. GH_AW_AGENT_CONCLUSION / workflow_run.conclusion (a job cannot observe its own needs.<job>.result, so this is empty in the agent job's own post step — the root cause first noted in [grafana-otel-advisor] OTel improvement: gh-aw.run.status silently reports 'success' on real agent failures #32958 / [otel-advisor] populate gh-aw.agent.conclusion span attribute on conclusion spans #35182), then
2. a fallback to agent_output.json output errors — but only when outputErrors.length > 0 (the fix(otlp): derive gh-aw.run.status and status.code from output errors when conclusion env var is absent #33037 fix).

The exit status of the "Execute GitHub Copilot CLI" step is not an input to either path. When that step exits non-zero but writes no errors into agent_output.json (our case: error_count=0), neither signal fires, so the span defaults to success / STATUS_CODE_OK. The span is effectively stamped independent of the Copilot CLI step's actual exit code.

Suggested fix (agentic plan)

1. Capture the Copilot CLI step exit code. In the workflow generator (likely pkg/workflow/…, the engine/agent step emitter that renders "Execute GitHub Copilot CLI"), persist the step's exit code / outcome into a file or env var readable by the setup post step (e.g. write $GITHUB_ENV or an artifact analogous to how conclusion is threaded in pkg/workflow/notify_comment.go).
2. Consume it in the span builder. In actions/setup/js/send_otlp_span.cjs, extend the runStatus / statusCode computation so that a non-zero agent-step exit forces runStatus="failure", statusCode=2 (STATUS_CODE_ERROR), and a statusMessage referencing the CLI exit code — even when outputErrors.length === 0. This mirrors the fix(otlp): derive gh-aw.run.status and status.code from output errors when conclusion env var is absent #33037 fallback but keyed on step exit rather than output errors.
3. Propagate through the post step. Confirm the setup post step (actions/setup/post.js) reads the new signal in the same clean-environment path that [otel-advisor] populate gh-aw.agent.conclusion span attribute on conclusion spans #35182 tried to fix for GH_AW_AGENT_CONCLUSION; the fix must survive the post-step's fresh environment.
4. Add a regression test. Extend send_otlp_span.test/.cjs with a case: GH_AW_AGENT_CONCLUSION absent + error_count=0 + agent-step exit non-zero → expect run.status=failure, status_code=STATUS_CODE_ERROR.
5. Backfill validation. Re-emit or spot-check against the four run IDs above (or equivalents) and confirm they surface as failures under the standard OTel error query.

Related (from issue/PR search)

• [grafana-otel-advisor] OTel improvement: gh-aw.run.status silently reports 'success' on real agent failures #32958 [closed] — [grafana-otel-advisor] gh-aw.run.status silently reports 'success' on real agent failures. Same core symptom; the originating report.
• fix(otlp): derive gh-aw.run.status and status.code from output errors when conclusion env var is absent #33037 [closed, PR] — fix(otlp): derive run.status/status.code from output errors when conclusion env var is absent. Partial fix — only fires when outputErrors.length > 0; does not cover error_count=0 CLI-exit failures.
• [otel-advisor] populate gh-aw.agent.conclusion span attribute on conclusion spans #35182 [closed] — populate gh-aw.agent.conclusion on conclusion spans / propagate conclusion to setup post step. Adjacent root cause (env var empty in post step).
• [otel-advisor] OTel improvement: distinguish partial-success runs in gh-aw.run.status #33518 [closed] — distinguish partial-success runs (errors present but conclusion=success).
• [cli-tools-test] audit: misleading failure diagnosis — 'failed before agent activation' when agent ran for 2 minutes #40982 [closed] — audit: misleading "failed before agent activation" when the agent actually ran.
• audit: distinguish pre-activation failures from agent-run failures without telemetry #41022 [closed, PR] — audit: distinguish pre-activation failures from agent-run failures without telemetry.
• Conclusion job mis-flags an allowed out_of_credits rate_limit_event as Effective-Token-Budget exhaustion (false failure on successful runs) #40835 [closed] — conclusion job mis-flags allowed out_of_credits as budget exhaustion (false failure — the inverse direction).
• [aw-failures] Copilot-engine agent step failing immediately (exit 22, zero tokens) — systemic, 2026-06-08 #37780 [closed] — Copilot-engine agent step failing immediately (exit 22, zero tokens) — related Copilot-CLI-exit failure mode.

No open PR or issue matching run.status telemetry, otel status_code, or agent.conclusion exit was found; the closest matches above are all closed.