[OTLP Validation] Validation report — run 26556845789 (2026-05-28)

[github-actions]

A. Executive Summary

Overall status: ⚠️ WARN

The single setup span in the local JSONL mirror is correctly formed, all required attributes are present, trace ID propagation is consistent, and no export errors have been recorded. However, the validation is necessarily incomplete because the workflow is still executing at analysis time — conclusion and agent spans have not yet been emitted. Backend visibility (Sentry / Grafana Tempo) could not be confirmed as no MCP query tools are available in this environment.

Main risks:

• Conclusion and agent spans may not reach backends if the workflow fails before conclusion step runs (cannot be ruled out until run ends).
• Backend ingestion success is unconfirmed — export errors file is absent (expected when 0 errors), but no round-trip confirmation was obtained.

Most likely root cause of WARN: Incomplete dataset due to in-flight execution, not a pipeline defect.

---

B. Trace Completeness

Metric	Value
Validation window	2026-05-28T05:36:54Z – present (workflow in progress)
JSONL mirror span count	1 (gh-aw.agent.setup)
Backend observed span count	Unknown — no MCP query tools available
Duplicate spans (same traceId+spanId)	0
Trace IDs in mirror	1 (93eab79f66b7cbe3e7c05d00b7c8f7c0)
GITHUB_AW_OTEL_TRACE_ID env match	✅ YES — mirror traceId equals env var

Missing spans (expected at this stage):

• gh-aw.agent.conclusion — not yet emitted (workflow still running)
• Agent turn spans (if any) — not yet emitted

Confidence: HIGH for what is observable; MEDIUM for overall pipeline health (conclusion path unverified).

---

C. Span Hierarchy Validation

Only one span is present at analysis time. Full hierarchy validation requires at least setup + conclusion spans.

Check	Result	Notes
Single global traceId across all spans	✅ PASS	Only 1 span; 1 unique traceId
Setup span has parentSpanId	✅ PASS	e7ae4af359899ce4 (global parent)
Setup span naming gh-aw.<job>.setup	✅ PASS	gh-aw.agent.setup
kind = 1 (INTERNAL) for setup span	✅ PASS
start < end timestamps	✅ PASS	Duration: 3.67 s
Timestamps within validation window	✅ PASS	2026-05-28T05:36:54Z / 05:36:57Z
Conclusion spans parent under setup	⏳ PENDING	Not yet emitted
Agent spans parent under conclusion	⏳ PENDING	Not yet emitted

---

D. Attribute Contract Validation

Setup span required attributes (spec §10.1)

Attribute	Value	Status
gh-aw.job.name	agent	✅
gh-aw.workflow.name	OTLP Data Quality Validator	✅
gh-aw.run.id	26556845789	✅
gh-aw.run.attempt	1	✅
gh-aw.run.actor	mnkiefer	✅
gh-aw.repository	github/gh-aw	✅
gh-aw.staged	false	✅

All 7 required setup attributes: PRESENT ✅

Setup span conditional attributes (present when available)

Attribute	Value	Status
gen_ai.system	github_models	✅
gh-aw.engine.id	copilot	✅
gh-aw.event_name	schedule	✅
gh-aw.episode.id	26556845789-1:...	✅
gh-aw.episode.kind	run	✅
gh-aw.hop.id	present	✅
gh-aw.workflow_call.id	present	✅
github.actions.run_url	present (duplicated in span + resource)	✅

Conclusion span required attributes (spec §10.2)

⏳ PENDING — Conclusion span not yet emitted.

Required attributes to verify when available: gh-aw.run.status, gh-aw.error_count, gh-aw.warning_count, gh-aw.action_minutes, gh-aw.output.item_count, gh-aw.otlp.export_errors.

Agent span GenAI attributes (spec §10.3)

⏳ PENDING — Agent spans not yet emitted (copilot engine).

Required attributes to verify: gen_ai.system, gen_ai.request.model, gen_ai.operation.name ("chat"), gen_ai.usage.input_tokens, gen_ai.usage.output_tokens.

---

E. Export and Fan-out Health

Metric	Value
Configured OTLP endpoints	2
Endpoint 1	Sentry (o205451.ingest.us.sentry.io) — header: x-sentry-auth ✅ (correct Sentry header rewrite applied)
Endpoint 2	Grafana Tempo (otlp-gateway-prod-eu-west-2.grafana.net) — header: Authorization: Basic ...
Export error count	0 (no error files present)
Export error details	None
JSONL mirror write status	✅ Written (/tmp/gh-aw/otel.jsonl, 1 line)
Multi-endpoint fan-out	✅ Both endpoints configured via GH_AW_OTLP_ENDPOINTS; independence not tested (no failure observed)
Backend round-trip confirmation	⚠️ UNCONFIRMED — no MCP query tools available

Sentry header rewrite note: The GH_AW_OTLP_ENDPOINTS env shows the Sentry entry correctly uses x-sentry-auth=sentry sentry_key=... (not Authorization), confirming the header rewrite required by spec §8 is applied.

---

F. Root-cause Hypothesis

Most likely explanation: No defect. Validation is in-progress.

The single span present is the setup span for the currently-executing agent job. Conclusion and agent spans are emitted at job end — they are pending by design, not lost.

Hypothesis	Evidence	Rank
In-flight execution (no defect)	Workflow is running; only setup span expected at this stage	HIGH
Export silently dropped by backend	No error files; both endpoints configured correctly	LOW
Sentry header rewrite missing	x-sentry-auth correctly set in GH_AW_OTLP_ENDPOINTS	RULED OUT
Network allowlist blocking vendor endpoints	No firewall errors in /tmp/gh-aw/sandbox/firewall/logs/ for OTLP destinations	LOW
OTLP endpoint misconfiguration	Sentry URL contains /integration/otlp suffix (correct for Sentry OTLP integration); Grafana URL is standard	LOW

---

G. Recommended Fixes (Prioritized)

No active defects identified. The following are observability improvements:

1. Add post-run backend reconciliation query — After conclusion spans are emitted, query Sentry/Grafana to confirm span count matches JSONL mirror. This closes the "backend visibility unconfirmed" gap.
2. Verify agent span GenAI attributes — Once agent spans are emitted, validate gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens per spec §10.3.
3. Monitor conclusion span gh-aw.otlp.export_errors — This attribute (spec §10.2) will record any export failures that occurred during the run. Review when available.
4. Consider adding service.version v-prefix consistency — Resource service.version is "1.0.52" while gh-aw.awf.version is "v0.25.56". Aligning prefix style would reduce confusion, though neither violates spec.

---

H. Validation Queries

# All spans with key identity fields
jq -c '.resourceSpans[].scopeSpans[].spans[] | {name, traceId, spanId, parentSpanId, kind, status}' \
  /tmp/gh-aw/otel.jsonl

# Unique trace IDs (expect 1)
jq -r '.resourceSpans[].scopeSpans[].spans[].traceId' /tmp/gh-aw/otel.jsonl | sort -u | wc -l

# Duplicate span check (expect 0)
jq -r '.resourceSpans[].scopeSpans[].spans[] | .traceId + "+" + .spanId' /tmp/gh-aw/otel.jsonl \
  | sort | uniq -d | wc -l

# Setup span required attributes
jq -c '.resourceSpans[].scopeSpans[].spans[] | select(.name | endswith(".setup")) |
  {name, attrs: ([.attributes[]? | {(.key): .value}] | add)}' /tmp/gh-aw/otel.jsonl

# Resource attributes
jq -c '.resourceSpans[].resource.attributes[] | {(.key): .value}' /tmp/gh-aw/otel.jsonl | sort -u

# Timestamp validity
jq -r '.resourceSpans[].scopeSpans[].spans[] |
  "\(.name): start=\(.startTimeUnixNano) end=\(.endTimeUnixNano) valid=\(.startTimeUnixNano < .endTimeUnixNano)"' \
  /tmp/gh-aw/otel.jsonl

# Instrumentation scope
jq -c '.resourceSpans[].scopeSpans[].scope' /tmp/gh-aw/otel.jsonl

# Export errors
cat /tmp/gh-aw/agent/otlp-export-errors.jsonl 2>/dev/null || echo "No export errors"
cat /tmp/gh-aw/agent/otlp-export-errors.count 2>/dev/null || echo "0 export errors"

---

Validation run: github/gh-aw · run 26556845789 · trace 93eab79f66b7cbe3e7c05d00b7c8f7c0
Spec reference: specs/otel-observability-spec.md v0.3.0 (§10.1, §10.2, §10.3, §11.1, §12)

Generated by 🧭 OTLP Data Quality Validator · sonnet46 1.3M · ◷

• expires on Jun 4, 2026, 5:42 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #35641

View newer issue

---

This action was performed automatically by the OTLP Data Quality Validator workflow.