[otel-advisor] OTel improvement: distinguish partial-success runs in gh-aw.run.status

[github-actions]

📡 OTel Instrumentation Improvement: distinguish partial-success runs in gh-aw.run.status

Analysis Date: 2026-05-20
Priority: Medium
Effort: Small (< 2h)

Problem

When the agent process concludes with GH_AW_AGENT_CONCLUSION=success but agent_output.json contains non-empty errors (e.g. a safe-output failed to push, a tool call returned an error message, a sub-action reported a non-fatal failure), the conclusion span at actions/setup/js/send_otlp_span.cjs:1719–1745 is emitted with:

• gh-aw.run.status = "success"
• status.code = 1 (STATUS_CODE_OK)
• gh-aw.error.count > 0
• exception events for each error

The error data is captured (good), but the summary signal says "healthy". As a result, a DevOps engineer running the standard OTel error query in Sentry / Grafana Tempo / Datadog (status.code = ERROR or gh-aw.run.status IN (failure, timeout, cancelled)) cannot answer: "Show me all runs that finished but had operational errors." They have to add a custom gh-aw.error.count > 0 clause, which is not part of any standard out-of-the-box error dashboard.

This is a deliberate design choice — see actions/setup/js/send_otlp_span.test.cjs:4571 ("does not override explicit GH_AW_AGENT_CONCLUSION=success even when outputErrors exist"), which asserts the current behavior. The proposal here is non-breaking: keep status.code = 1, but extend the gh-aw.run.status taxonomy with a new "partial_success" value so partial-success runs are distinguishable from clean-success runs without rewriting alerting rules.

Why This Matters (DevOps Perspective)

The gh-aw.run.status attribute is the single most useful low-cardinality slice for dashboards — it's already used by gh-aw.run.status=timeout queries to separate timeouts from generic failures (see send_otlp_span.cjs:1723–1726). Adding partial_success extends that same pattern to cover the "finished but had errors" case.

Concrete operational scenarios this unblocks:

• Sentry: an alert rule "any gh-aw run with non-empty errors in the last 1h" today either misses partial successes (if it filters on issue status) or fires too broadly (if it filters on gh-aw.error.count).
• Grafana / Tempo: the canonical TraceQL filter { resource.service.name = "gh-aw" && span.gh-aw.run.status != "success" } would surface partial successes alongside failures — currently they slip through.
• MTTR: when an on-call engineer sees a successful run that produced no artifacts (because all safe-outputs errored), they currently have to inspect the JSONL artifact to discover the failure. A partial_success status surfaces it in the first dashboard glance.

This is the classic "misleading data" failure mode — the data exists but is not surfaced in the operational summary signal.

Current Behavior

// actions/setup/js/send_otlp_span.cjs lines 1694–1745
const isAgentTimedOut = agentConclusion === "timed_out";
const isAgentFailure = agentConclusion === "failure" || isAgentTimedOut;
const isAgentCancelled = agentConclusion === "cancelled";
const isAgentNonOK = isAgentFailure || isAgentCancelled;
let statusCode = isAgentNonOK ? 2 : 1;
let statusMessage;
// ...

let runStatus = "success";
const rawRunStatus = agentConclusion || workflowRunConclusion;
if (rawRunStatus === "cancelled") {
  runStatus = "cancelled";
} else if (rawRunStatus === "timed_out") {
  runStatus = "timeout";
} else if (rawRunStatus === "failure") {
  runStatus = "failure";
}

// Only fires when rawRunStatus is empty — not when it is explicitly "success":
if (!rawRunStatus && outputErrors.length > 0) {
  runStatus = "failure";
  statusCode = 2;
  statusMessage = (errorMessages.length > 0 ? `errors detected: ${errorMessages[0]}` : "errors detected").slice(0, 256);
}

if (isAgentFailure && errorMessages.length > 0) {
  statusMessage = `agent ${agentConclusion}: ${errorMessages[0]}`.slice(0, 256);
}
// ...
attributes.push(buildAttr("gh-aw.run.status", runStatus));
attributes.push(buildAttr("gh-aw.error_count", outputErrors.length));

The test at send_otlp_span.test.cjs:4571 pins this behavior:

it("does not override explicit GH_AW_AGENT_CONCLUSION=success even when outputErrors exist", async () => {
  process.env.GH_AW_AGENT_CONCLUSION = "success";
  // ... agent_output.json with { errors: [{ message: "non-fatal warning error" }] }
  expect(attrs["gh-aw.run.status"]).toBe("success");
  expect(span.status.code).toBe(1);
});

Proposed Change

Add partial_success as a new gh-aw.run.status value. Keep status.code = 1 (non-breaking).

// actions/setup/js/send_otlp_span.cjs (replace the block at lines 1719–1738)
let runStatus = "success";
const rawRunStatus = agentConclusion || workflowRunConclusion;
if (rawRunStatus === "cancelled") {
  runStatus = "cancelled";
} else if (rawRunStatus === "timed_out") {
  runStatus = "timeout";
} else if (rawRunStatus === "failure") {
  runStatus = "failure";
} else if (rawRunStatus === "success" && outputErrors.length > 0) {
  // Agent process concluded successfully, but reported non-fatal errors via
  // agent_output.json (e.g. a safe-output failed, a tool call returned an error).
  // Keep status.code = OK so the team's explicit "success when conclusion=success"
  // contract is preserved, but mark the run.status distinctly so operators can
  // query for partial successes without parsing per-span error attributes.
  runStatus = "partial_success";
}

if (!rawRunStatus && outputErrors.length > 0) {
  runStatus = "failure";
  statusCode = 2;
  statusMessage = (errorMessages.length > 0 ? `errors detected: ${errorMessages[0]}` : "errors detected").slice(0, 256);
}

Expected Outcome

After this change:

• In Grafana / Honeycomb / Datadog: a single low-cardinality filter gh-aw.run.status IN (failure, timeout, cancelled, partial_success) surfaces every run that needs attention.
• In Sentry: alert rules can be tuned per status — page on failure/timeout, ticket on partial_success, ignore cancelled.
• In the JSONL mirror: jq 'select(.attributes[] | select(.key == "gh-aw.run.status") | .value.stringValue == "partial_success")' becomes the canonical way to find these runs.
• For on-call engineers: a run with gh-aw.run.status=partial_success is immediately recognizable as "agent ran fine but produced errors" without having to inspect gh-aw.error.messages or the exception events.

Behavior preserved (non-breaking):

• span.status.code = 1 (OK) when GH_AW_AGENT_CONCLUSION=success, matching the existing contract.
• All existing gh-aw.run.status values (success, failure, timeout, cancelled) keep their current semantics.
• Exception events and gh-aw.error.* attributes are unchanged.

Evidence from Live Sentry Data

Live Sentry MCP tooling was not available in this run (the sentry MCP server's tool registry was empty: /home/runner/work/_temp/gh-aw/mcp-cli/tools/sentry.json contained [], and sentry find_organizations returned Error [-32602]: unknown tool). Analysis was grounded in the local OTLP JSONL mirror plus static code inspection.

Live span sampled (/tmp/gh-aw/otel.jsonl, this run):

• traceId: 5356be426197a28ba11634bcba38fb76
• name: gh-aw.agent.setup
• parentSpanId: 61e5c778ff4e3db2 (trace continuity intact — setup is parented under the activation span)
• status.code: 1 (OK)
• Resource attributes present: service.name, service.version=2.1.142, github.repository, github.run_id, github.run_attempt, github.event_name, github.workflow_ref, deployment.environment=production, runner.os/arch/name/environment. All of the candidate "missing" resource attributes (service.version, github.event_name, github.run_id, deployment.environment) are already populated — the resource-attribute layer is in excellent shape.

The live span is for an in-flight run that has not reached the conclusion phase, so this specific span cannot itself exhibit the gap. The gap was confirmed instead by:

1. Tracing the code path in send_otlp_span.cjs:1719–1745.
2. The pinning test at send_otlp_span.test.cjs:4571 ("does not override explicit GH_AW_AGENT_CONCLUSION=success even when outputErrors exist"), whose comment explicitly labels the input as a non-fatal warning error.
3. The asymmetry between the !rawRunStatus && outputErrors.length > 0 branch (which sets runStatus="failure", statusCode=2) and the rawRunStatus==="success" && outputErrors.length > 0 case (which does nothing).

Implementation Steps

• Modify actions/setup/js/send_otlp_span.cjs to add the partial_success branch shown in the Proposed Change section above (lines ~1719–1738).
• Update actions/setup/js/send_otlp_span.test.cjs:4571 so the does not override explicit GH_AW_AGENT_CONCLUSION=success even when outputErrors exist test now asserts gh-aw.run.status === "partial_success" and status.code === 1.
• Add a focused test alongside it asserting that agent_conclusion=success with no errors still yields gh-aw.run.status === "success".
• Run cd actions/setup/js && npx vitest run send_otlp_span.test.cjs and make test-unit to confirm tests pass.
• Run make fmt.
• Mention the new value in docs/src/content/docs/guides/custom-otlp-attributes.md if gh-aw.run.status values are documented there.
• Open a PR referencing this issue.

Related Files

• actions/setup/js/send_otlp_span.cjs (lines 1719–1745)
• actions/setup/js/send_otlp_span.test.cjs (lines 3977–4003 and 4571–4592)
• actions/setup/js/action_conclusion_otlp.cjs (caller — no changes needed)
• docs/src/content/docs/guides/custom-otlp-attributes.md (if gh-aw.run.status taxonomy is documented)

---

Generated by the Daily OTel Instrumentation Advisor workflow

Generated by 📊 Daily OTel Instrumentation Advisor · ● 12.3M · ◷

• expires on May 27, 2026, 11:12 AM UTC

[pelikhan]

@mnkiefer fix this AW so it gets live data from otel providers to do analyzis

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #33576

View newer issue

---

This action was performed automatically by the Daily OTel Instrumentation Advisor workflow.