[otel-advisor] populate gh-aw.agent.conclusion span attribute on conclusion spans

[github-actions]

📡 OTel Instrumentation Improvement: propagate agent conclusion to setup-action post step

Analysis Date: 2026-05-27
Priority: High
Effort: Medium (2–4h)

Problem

The gh-aw.agent.conclusion span attribute defined at actions/setup/js/send_otlp_span.cjs:1926-1928 is never populated in production conclusion spans. The Sentry query has:gh-aw.agent.conclusion over the last 14 days returns zero results — confirming the attribute is dead code at runtime even though it appears in unit tests and span-builder logic.

Root cause: GH_AW_AGENT_CONCLUSION: ${{ needs.<agent>.result }} is only emitted at the step level for three specific user-facing steps (see pkg/workflow/notify_comment.go:84,216,422 — the Process no-op messages, Handle agent failure, and detection-related steps). The setup action's post step — which is where sendJobConclusionSpan actually runs (actions/setup/post.js:24) — executes after those steps complete with a clean environment, so process.env.GH_AW_AGENT_CONCLUSION is unset when the conclusion span is built.

A DevOps engineer cannot today answer the question "how many failures last week were infrastructure failures (agent crash/timeout/cancellation) versus user-mistake failures (safe-output validation errors)?" even though that distinction is fully knowable inside the workflow — needs.agent.result is visible to every downstream job's YAML.

Why This Matters (DevOps Perspective)

Without this attribute, every alert/dashboard that filters on gh-aw.run.status:failure lumps together two operationally distinct failure modes:

• Infrastructure failures (gh-aw.agent.conclusion:failure|timed_out|cancelled): the agent crashed, timed out, or was cancelled — typically requires platform/engine team attention (OOM, model outage, token budget exhaustion, runner abort).
• Validation failures (no gh-aw.agent.conclusion, gh-aw.error.messages set via the fallback at send_otlp_span.cjs:1854-1858): the agent completed normally but produced output that failed safe-output validation — typically a workflow-author content issue.

These two categories require completely different on-call rotations, escalation paths, and SLO accounting. Today's Sentry dashboards over-page the platform team because the noise from validation failures is indistinguishable from real outages. With the attribute populated, alerts can route on gh-aw.agent.conclusion:failure OR gh-aw.agent.conclusion:timed_out (page platform) versus the residual gh-aw.run.status:failure -has:gh-aw.agent.conclusion (notify workflow authors only). This is a direct MTTR win.

Current Behavior

In actions/setup/js/send_otlp_span.cjs the conclusion-span builder reads the env var (line 1806) and conditionally adds it (line 1926-1928), but the env var is never set when the post step runs:

// actions/setup/js/send_otlp_span.cjs:1806
const agentConclusion = process.env.GH_AW_AGENT_CONCLUSION || "";
// ...
// actions/setup/js/send_otlp_span.cjs:1926-1928
if (agentConclusion) {
  attributes.push(buildAttr("gh-aw.agent.conclusion", agentConclusion));
}

The compiler emits the env var only at STEP scope, not job/post scope:

// pkg/workflow/notify_comment.go:84 (and lines 216, 422)
noopEnvVars = append(noopEnvVars,
    fmt.Sprintf("          GH_AW_AGENT_CONCLUSION: ${{ needs.%s.result }}\n", mainJobName))

In the compiled actions/setup/action.yml there is no agent-conclusion input that would let the main step persist the value to GITHUB_ENV for the post step to read.

Proposed Change

Add an agent-conclusion input to the setup action and have the main step write it to GITHUB_ENV, so the post step (where the conclusion span runs) can read it as a normal env var. The compiler then passes needs.<agent-job>.result to every downstream job's Setup Scripts step.

# actions/setup/action.yml — new input
inputs:
  agent-conclusion:
    description: >-
      Agent job result (success / failure / timed_out / cancelled / skipped).
      Pass ${{ needs.<agent-job>.result }} from downstream jobs so the
      gh-aw.<job>.conclusion span carries gh-aw.agent.conclusion.
    required: false
    default: ''

// actions/setup/js/action_setup_otlp.cjs — write to GITHUB_ENV so post step sees it
const inputAgentConclusion = getActionInput("AGENT_CONCLUSION");
if (inputAgentConclusion && githubEnv) {
  writeEnvLine(githubEnv, "GH_AW_AGENT_CONCLUSION", inputAgentConclusion,
    "GH_AW_AGENT_CONCLUSION", "GITHUB_ENV");
}

// pkg/workflow/* — wire the input into every downstream Setup Scripts call
// (conclusion, safe_outputs, detection, push_repo_memory jobs):
//   with:
//     agent-conclusion: ${{ needs.<agent-job>.result }}

Expected Outcome

After this change:

• In Sentry / Grafana / Honeycomb / Datadog: gh-aw.agent.conclusion:failure becomes a queryable filter, finally distinguishing agent-runtime failures from safe-output validation failures. Dashboards can split-by gh-aw.agent.conclusion to expose timeout vs. crash vs. cancellation rates per workflow/engine.
• In the JSONL mirror (/tmp/gh-aw/otel.jsonl): every gh-aw.<job>.conclusion span downstream of the agent will carry gh-aw.agent.conclusion, so offline post-mortem analysis can categorize failures without re-deriving the lineage.
• For on-call engineers: alert rules can route based on failure category, reducing pages to the platform team for what are really workflow-author content errors.

Implementation Steps

• Add agent-conclusion input to actions/setup/action.yml
• In actions/setup/js/action_setup_otlp.cjs (in run()), read the input via getActionInput("AGENT_CONCLUSION") and append GH_AW_AGENT_CONCLUSION=<value> to GITHUB_ENV when non-empty
• Update actions/setup/js/action_setup_otlp.test.cjs to cover the new env-write path
• In pkg/workflow/* (the compiler), pass agent-conclusion: ${{ needs.<agent-job>.result }} to the uses: ./actions/setup step in every downstream job that already sets GH_AW_AGENT_CONCLUSION at step level (conclusion, safe_outputs, detection, push_repo_memory)
• Update compiler golden tests to assert the new with: line
• Run make recompile to regenerate .github/workflows/*.lock.yml
• Run make test-unit and make fmt
• Verify in a smoke run that a sample gh-aw.<job>.conclusion span in /tmp/gh-aw/otel.jsonl now carries gh-aw.agent.conclusion

Evidence from Live OTel Data (Sentry/Grafana)

Sentry (org=github, project=gh-aw, last 14d):

• has:gh-aw.agent.conclusion → 0 results
• gh-aw.agent.conclusion:success → 0 results
• For comparison, has:gh-aw.run.status → many results; has:gh-aw.event_name → many results. Other gh-aw.* span attributes index normally, confirming the gap is attribute-population, not Sentry indexing.

Grafana Tempo trace 4a27bb1bc4ff7c0a8427f404d1282642 (failed copilot run, 2026-05-27T05:05Z, workflow Copilot CLI Deep Research Agent, run_id 26491933777). Inspecting every gh-aw.*.conclusion span in the trace via tempo_get-trace:

• gh-aw.agent.conclusion (span 192bafaa95cc63a7) — attributes contain gh-aw.run.status: failure, gh-aw.error.messages: "Line 1: create_discussion 'body' is too short (minimum 64 characters)", but no gh-aw.agent.conclusion attribute.
• gh-aw.detection.conclusion (a937f0875968c193) — same gap.
• gh-aw.safe_outputs.conclusion (eb277f99d7dfdb36) — carries gh-aw.detection.conclusion: success (proving the sibling pattern works because that env var IS surfaced via needs.detection.outputs) but no gh-aw.agent.conclusion.
• gh-aw.conclusion.conclusion (fe0b627b6fc84705) — same gap.

This confirms the issue is not engine-specific or workflow-specific — it affects every conclusion span across every workflow.

Related Files

• actions/setup/js/send_otlp_span.cjs (consumer of the env var, lines 1806, 1926-1928)
• actions/setup/js/action_setup_otlp.cjs (where the new input would be written to GITHUB_ENV)
• actions/setup/js/action_conclusion_otlp.cjs (post-step invoker — no direct change needed)
• actions/setup/action.yml (new agent-conclusion input)
• actions/setup/post.js (no change — it already runs action_conclusion_otlp.cjs)
• pkg/workflow/notify_comment.go (existing step-level env-var emission, lines 84, 216, 422)
• pkg/workflow/* (downstream-job uses: ./actions/setup call sites that need the new with: line)

---

Generated by the Daily OTel Instrumentation Advisor workflow

Generated by 📊 Daily OTel Instrumentation Advisor · opus47 34M · ◷

• expires on Jun 3, 2026, 10:45 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #35457

View newer issue

---

This action was performed automatically by the Daily OTel Instrumentation Advisor workflow.