[reliability] Daily Reliability Review - 2026-07-15

[github-actions]

Executive Summary

Overall health for the last 24h is degraded but not broken. Whole-workflow agent spans complete with span.status:ok (no runtime-timeout status observed), and no truncation (gen_ai.response.finish_reasons:length) was found. However, there is a systemic, recurring pattern of errored model calls against copilot/claude-sonnet-4.6 (invoke_agent gen_ai spans): the query hit its 50-span cap spread across 22+ distinct traces, each with 2–4 errored calls. The failing calls are long-running (27–72s) with duplicate ~57.9s durations that look like retries/timeouts, while sibling calls in the same trace finish in 6–10s.

Two observability caveats bound confidence: the errors and logs datasets are both empty for the window (only spans is populated), and the failing copilot spans carry no workflow identity, so failures cannot be grouped by workflow. Treat the copilot errors as confirmed failures (span.status:error is reliable) with an inconclusive root cause (timeout vs. API error) and a confirmed attribution/instrumentation gap.

Telemetry source: Sentry org github, project gh-aw, spans dataset, statsPeriod:24h (MCP build exposes list_events only — no search_events/get_trace_details).

Top Reliability Findings

Priority	Workflow	Problem	Evidence	Next Action
P1	Unattributed (see gap)	Recurring errored copilot/claude-sonnet-4.6 model calls, likely timeouts/retries	50-span cap across 22+ traces, 2–4 errors each; trace 933db0b3... had 4 errors (72.8s / 57.9s / 57.9s / 27.2s) beside ok calls at 6–10s	Add per-request timeout + capped retry with backoff on copilot model calls; alert on span.status:error span.op:gen_ai rate
P2	All (correlation gap)	Copilot model error spans have no gh-aw.workflow.name — 0 of N error spans carry it	span.status:error gen_ai.request.model:copilot/claude-sonnet-4.6 has:gh-aw.workflow.name → 0 results; native gh-aw spans do carry it (e.g. "PR Sous Chef", "Daily Reliability Review")	Propagate gh-aw.* resource attrs onto Copilot-CLI-emitted spans so model failures are attributable
P3	n/a	gen_ai.response.finish_reasons not queryable in Sentry despite always being emitted	has:gen_ai.response.finish_reasons → 0 results, but send_otlp_span.cjs:2146 always sets it (fallback timeout/unknown)	Verify array-attribute indexing/export path; without it, timeout-vs-length cannot be distinguished from telemetry
P4	n/a	Small class of fast gateway.request error spans (likely gateway rejections)	Error spans 10–240ms on traces 3c4d535a..., e5aca975..., b33c7350..., 60c9bc8c...	Capture status.message/HTTP code on gateway spans to classify (auth vs rate-limit vs routing)
P5	n/a	errors + logs datasets empty for 24h — triage limited to span status	errors→No results; logs→No results	Confirm error/log export is wired; empty datasets are themselves an observability finding

Representative Traces

View representative traces

• P1 copilot errors — trace 933db0b3d014d25cd4bd4d89cff0f358: 4 errored invoke_agent gen_ai spans (72861ms, 57879ms, 57879ms, 27194ms) vs 3 ok (6.6–10.2s), all copilot/claude-sonnet-4.6. https://github.sentry.io/explore/traces/trace/933db0b3d014d25cd4bd4d89cff0f358
• Additional recurring copilot-error traces: 082d623ef3658eb76bb362af4b212679 (incl. one 196992ms span), 17749014b54366740766f20d3cc11536, 86b14e75ae72137307d65145364c77a9 (4 errors), b06ac13a9ff90f6978ac13c65bc87cda (4 errors).
• P4 gateway.request errors — trace 3c4d535a2373454d96486343630e88d3 (10.1ms error), e5aca975feee0062cc82c02a037eca5e (230ms error). https://github.sentry.io/explore/traces/trace/3c4d535a2373454d96486343630e88d3
• Latency (not a failure) — longest gen_ai agent spans run ~27–40 min with span.status:ok; e.g. trace 5e14c38ecff38f303c107efbb5cda59a (2376437ms). These are long workflow runs, not errors.

Recommendations

1. Propagate gh-aw.workflow.name (and gh-aw.run.id/run.status) onto Copilot-CLI model spans. Smallest useful fix; without it the P1 failures cannot be grouped by workflow. Emit side already sets these on native spans (send_otlp_span.cjs:1297,2068,2076).
2. Add a per-request timeout + bounded retry/backoff on copilot/claude-sonnet-4.6 calls. The duplicate ~57.9s durations and 27–72s errored calls point to unbounded retries/hangs.
3. Fix gen_ai.response.finish_reasons searchability (array-attribute indexing/export). It is always emitted but returns 0 hits in Sentry, blocking timeout-vs-truncation triage.
4. Verify errors/logs export. Both datasets are empty for 24h; confirm this is expected (spans-only pipeline) vs. a broken exporter.

Notes

View notes

• Attribute presence (verified via has: filters): span.status ✓ present/queryable; release ✓ present (Sentry maps resource service.version); gh-aw.run.status ✓ present on native spans; gh-aw.workflow.name ✓ present on native spans; gh_aw.workflow_name (underscore) ✗ — this is a query-side naming mismatch, the real emitted key is dotted gh-aw.workflow.name (send_otlp_span.cjs:1297); service.version ✗ as a span field (it is a resource attr → surfaces as release); gen_ai.response.finish_reasons ✗ not queryable despite emit-side always setting it.
• Confirmed vs. inconclusive: copilot model failures are confirmed (span.status:error is reliable). Root cause (timeout vs API error) is inconclusive — status.message/finish_reasons are not queryable in this backend. No finish_reasons:length truncation was observed, so no runaway-token/truncation finding is claimed.
• Tooling limits: this Sentry MCP build lacks search_events and get_trace_details; trace continuity was validated via list_events filtered by trace:<id>.
• Emit-side cross-check (actions/setup/js/send_otlp_span.cjs): workflow identity → gh-aw.workflow.name; runtime outcome → OTLP status.code/status.message + gh-aw.run.status; finish reasons → emitted with fallback timeout/unknown (line 2146); release → resource service.version (backend-mapped to release).

References:

• §29457747547
• Sentry trace 933db0b3...: https://github.sentry.io/explore/traces/trace/933db0b3d014d25cd4bd4d89cff0f358
• Sentry trace 082d623e...: https://github.sentry.io/explore/traces/trace/082d623ef3658eb76bb362af4b212679

Generated by 🚨 Daily Reliability Review · 139.9 AIC · ⌖ 35.4 AIC · ⊞ 5.4K · ◷

• expires on Jul 17, 2026, 3:20 PM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #46098

View newer issue

---

This action was performed automatically by the Daily Reliability Review workflow.