[OTLP Validation] OTLP Data Quality Validation Report — run 26801041670

[github-actions]

Validation window: 2026-06-02T05:46:23Z – 2026-06-02T05:48:36Z (UTC)
Workflow: OTLP Data Quality Validator (otlp-data-quality-validator)
Run ID: 26801041670 · Attempt 1 · Triggered by schedule
OTLP endpoints: Sentry (o205451.ingest.us.sentry.io) + Grafana Tempo (otlp-gateway-prod-eu-west-2.grafana.net)

---

A. Executive Summary

Overall status: ⚠️ WARN

The single emitted span (gh-aw.agent.setup) is fully spec-compliant. The missing conclusion and agent spans are expected — this validator ran during the workflow session; those spans are emitted at job conclusion, which had not yet occurred at validation time. No export errors were detected. Backend query visibility cannot be confirmed without live backend access.

Main risks:

1. The Grafana Tempo endpoint (.../otlp) lacks an explicit /v1/traces suffix — this may or may not be appended automatically by send_otlp_span.cjs; warrants a code-level check (see §F).
2. Backend ingestion cannot be independently verified from within the runner — reconciliation relies solely on the absence of export errors.

---

B. Trace Completeness

Metric	Value
JSONL mirror lines	1
Spans emitted	1 (gh-aw.agent.setup)
Unique traceId values	1 ✅
Duplicate traceId+spanId pairs	0 ✅
Conclusion span present	❌ Not yet emitted (workflow in-progress — expected)
Agent span present	❌ Not yet emitted (workflow in-progress — expected)
Export errors	0 ✅

Confidence: High for setup span integrity; not applicable for conclusion/agent spans (they are post-run artifacts).

The traceId in the JSONL mirror (63f74bdf6b2a0b90131741ebfcd27456) matches GITHUB_AW_OTEL_TRACE_ID in the environment ✅ (spec §12.2).

---

C. Span Hierarchy Validation

Only the setup span was available for analysis. Expected hierarchy per spec §9.3:

Check	Result
Setup span parentSpanId present and non-empty	✅ 310b1ae2383d7efe
Setup span parents under global root parent	✅ (parentSpanId is set to the pre-generated global root)
Conclusion span parents under setup span	⚪ Not yet emitted
Agent span parents under conclusion span	⚪ Not yet emitted
Span name matches gh-aw.<job>.<op> (§9.2)	✅ gh-aw.agent.setup
GITHUB_AW_OTEL_PARENT_SPAN_ID = setup spanId	✅ 10b171217914698b (will parent the conclusion span)

---

D. Attribute Contract Validation

Setup span required attributes (§10.1)

Attribute	Present	Value
gh-aw.job.name	✅	agent
gh-aw.workflow.name	✅	OTLP Data Quality Validator
gh-aw.run.id	✅	26801041670
gh-aw.run.attempt	✅	1
gh-aw.run.actor	✅	mnkiefer
gh-aw.repository	✅	github/gh-aw
gh-aw.staged	✅	false

All 7 required setup span attributes present. ✅

Conclusion span required attributes (§10.2)

⚪ Not yet emitted — cannot validate.

Agent span GenAI attributes (§10.3)

⚪ Not yet emitted — cannot validate.

Note: The setup span carries gen_ai.system: github_models as a conditional attribute (§10.1), which is correct for the copilot engine ✅.

Resource attributes (§11.1)

Attribute	Present	Value
service.name	✅	gh-aw.otlp-data-quality-validator
service.version	✅	1.0.55
github.repository	✅	github/gh-aw
github.run_id	✅	26801041670
github.run_attempt	✅	1
github.actions.run_url	✅	https://github.com/github/gh-aw/actions/runs/26801041670

All 6 required resource attributes present. ✅

Conditional resource attributes also present: github.event_name, github.ref, github.ref_name, github.sha, github.job, github.workflow_ref, github.actor_id, runner.os, runner.arch, runner.name, runner.environment, gh-aw.awf.version, deployment.environment.

Instrumentation scope (§11.3)

Field	Value	Status
scope.name	gh-aw	✅
scope.version	1.0.55	✅ (matches service.version)

Timestamp integrity

Check	Result
startTimeUnixNano < endTimeUnixNano	✅
Duration	2,076 ms (≈ 2.1 s, plausible for setup)
Start offset from validation time	~147 s (within run window ✅)
Far-future timestamps	None ✅

---

E. Export and Fan-Out Health

Two OTLP endpoints configured (GH_AW_OTLP_ENDPOINTS):

Endpoint	Auth header	Status
https://o205451.ingest.us.sentry.io/api/4511347087179777/integration/otlp	x-sentry-auth (Sentry-specific rewrite ✅)	No export errors logged
https://otlp-gateway-prod-eu-west-2.grafana.net/otlp	Authorization: Basic ...	No export errors logged

• /tmp/gh-aw/agent/otlp-export-errors.jsonl — absent (no errors) ✅
• /tmp/gh-aw/agent/otlp-export-errors.count — absent (0) ✅
• JSONL mirror write: /tmp/gh-aw/otel.jsonl (1 line) ✅
• Fan-out independence: endpoints are written as a JSON array in GH_AW_OTLP_ENDPOINTS; fan-out is handled concurrently in send_otlp_span.cjs per spec §6.1.

Backend query visibility: No backend MCP tools available. Cannot independently confirm ingestion. No evidence of loss — absence of errors is the strongest available signal.

---

F. Root-Cause Hypothesis

No active failure detected. The following items are precautionary observations for future runs:

Hypothesis 1 (Low confidence): Grafana Tempo endpoint path missing /v1/traces

• Evidence: GH_AW_OTLP_ENDPOINTS contains ...grafana.net/otlp without an explicit /v1/traces suffix. OTLP/HTTP exporters that do not auto-append the path would POST to the wrong URL (HTTP 404).
• Alternative: send_otlp_span.cjs may already append /v1/traces to the base endpoint URL, making this a non-issue.
• Action: Inspect actions/setup/js/send_otlp_span.cjs for URL construction logic.

Hypothesis 2 (Not triggered): Sentry Authorization → x-sentry-auth rewrite

• Evidence: Header is already correctly set to x-sentry-auth in the environment — the rewrite was applied at compile/config time. No action needed.

Hypothesis 3 (Not applicable): Retry exhaustion or network block

• No export errors recorded; no evidence of retry exhaustion or firewall block for either endpoint.

---

G. Recommended Fixes (Prioritized)

1. [P1] Confirm Grafana Tempo URL path handling — verify that send_otlp_span.cjs appends /v1/traces to the base OTLP endpoint URL, or that .../otlp is the correct full path accepted by otlp-gateway-prod-eu-west-2.grafana.net.

2. [P2] Add post-run backend reconciliation — after the conclusion span is emitted, query both Sentry and Grafana Tempo backends by traceId (63f74bdf6b2a0b90131741ebfcd27456) to confirm all 3 spans (setup, conclusion, agent) are queryable. This would convert the current "no errors" proxy into a real end-to-end signal.

3. [P3] Validate conclusion and agent span attributes in a follow-up run — the current validation window only captured the setup span. A post-run re-execution of this validator would be able to check §10.2 conclusion attributes and §10.3 GenAI agent attributes.

4. [P4] Store trace ID in aw_info.json context — aw_info.context.otel_trace_id was null at validation time. Spec §12.2 specifies it as a resolution source; populating it improves cross-workflow trace context propagation.

---

H. Validation Queries and Commands

# Extract all spans from JSONL mirror
jq -c '.resourceSpans[].scopeSpans[].spans[] | {name, traceId, spanId, parentSpanId, kind, status}' /tmp/gh-aw/otel.jsonl

# Verify unique trace IDs (expect 1)
jq -r '.resourceSpans[].scopeSpans[].spans[].traceId' /tmp/gh-aw/otel.jsonl | sort -u | wc -l

# Check for duplicate spanId
jq -r '.resourceSpans[].scopeSpans[].spans[] | "\(.traceId):\(.spanId)"' /tmp/gh-aw/otel.jsonl | sort | uniq -d

# Validate required setup span attributes (§10.1)
jq -c '.resourceSpans[].scopeSpans[].spans[] | select(.name | endswith(".setup")) | {name, attrs: [.attributes[]? | {(.key): .value}] | add}' /tmp/gh-aw/otel.jsonl

# Check all resource attributes (§11.1)
jq -c '.resourceSpans[].resource.attributes[] | {(.key): .value}' /tmp/gh-aw/otel.jsonl | sort -u

# Confirm JSONL trace ID matches env var
echo "JSONL: $(jq -r '.resourceSpans[].scopeSpans[].spans[].traceId' /tmp/gh-aw/otel.jsonl | sort -u)"
echo "ENV:   $GITHUB_AW_OTEL_TRACE_ID"

# Check export errors
cat /tmp/gh-aw/agent/otlp-export-errors.jsonl 2>/dev/null || echo "No errors"
cat /tmp/gh-aw/agent/otlp-export-errors.count 2>/dev/null || echo "0"

Spec references: §9.2 (span naming), §9.3 (hierarchy), §9.4 (kinds), §10.1 (setup attributes), §11.1 (resource attributes), §11.3 (scope), §12.2 (trace ID propagation)

Generated by 🧭 OTLP Data Quality Validator · sonnet46 1.4M · ◷

• expires on Jun 9, 2026, 5:53 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #36580

View newer issue

---

This action was performed automatically by the OTLP Data Quality Validator workflow.