[OTLP Validation] OTLP validation run 26620247189 — WARN: setup span PASS, backend visibility unconfirmed

[github-actions]

Validation Run Context

• Validation time window: 2026-05-29 05:39:28 UTC – 2026-05-29 05:40:41 UTC (agent still running at report time)
• Workflow run ID: 26620247189 (attempt 1)
• Service name: gh-aw.otlp-data-quality-validator
• OTLP endpoint: Sentry (https://o205451.ingest.us.sentry.io/api/4511347087179777/integration/otlp)
• Expected trace ID: 2384236d85e963e7f7bf1fc55dd1c6bd
• gh-aw CLI version: 1.0.52 / AWF version v0.25.56

Note: This report was produced while the agent job was still executing. The gh-aw.agent.conclusion and gh-aw.agent.agent spans are not yet emitted. Absence of those spans is expected (not data loss).

---

A. Executive Summary

Overall status: ⚠️ WARN

The single emitted span (gh-aw.agent.setup) is well-formed: all required §10.1 setup attributes are present, all §11.1 resource attributes are present, trace ID propagation is consistent, and no export errors were recorded. The data pipeline appears healthy.

Main risks / observations:

1. Backend visibility cannot be confirmed — no MCP backend tool is available in this environment to query Sentry for the emitted span.
2. The OTLP endpoint is Sentry; the Sentry-specific Authorization → x-sentry-auth header rewrite (spec §8) must be applied by the exporter or it will be rejected with HTTP 400/401.
3. Only 1 span has been emitted so far (setup only). Conclusion and agent spans are pending emission.

---

B. Trace Completeness

Metric	Value
Spans in JSONL mirror	1
Unique traceId values	1 (2384236d85e963e7f7bf1fc55dd1c6bd)
Unique traceId+spanId pairs	1
Duplicate spans	0
Export errors	0
Backend query available	❌ No MCP backend tool present

Expected vs. emitted:

Expected span	Status
gh-aw.agent.setup	✅ Emitted
gh-aw.agent.conclusion	⏳ Pending (workflow still running)
gh-aw.agent.agent	⏳ Pending (workflow still running)

Trace ID consistency: GITHUB_AW_OTEL_TRACE_ID=2384236d85e963e7f7bf1fc55dd1c6bd matches span traceId exactly. ✅

Confidence: High for emitted span; N/A for backend visibility (no query tool available).

---

C. Span Hierarchy Validation

Only one setup span exists in the mirror at report time. Full hierarchy validation is deferred until conclusion and agent spans are available.

Check	Result	Notes
Setup span parentSpanId present	✅ Pass	53b651df57ef6847 — the global root parent
Setup spanId == GITHUB_AW_OTEL_PARENT_SPAN_ID	✅ Pass	Both are d1e5be6ae26866c4 — correct for conclusion/agent parenting
All setup spans share global parent	✅ Pass (1 span only)	Can only verify with multiple jobs
Conclusion spans parent under setup	⏳ Pending	Not yet emitted
Agent spans parent under conclusion	⏳ Pending	Not yet emitted
Span naming pattern gh-aw.<job>.<op>	✅ Pass	gh-aw.agent.setup matches

Spec §9.3 hierarchy structure is consistent with what has been emitted.

---

D. Attribute Contract Validation

Setup Span (gh-aw.agent.setup) — spec §10.1

Required attributes:

Attribute	Present	Value
gh-aw.job.name	✅	agent
gh-aw.workflow.name	✅	OTLP Data Quality Validator
gh-aw.run.id	✅	26620247189
gh-aw.run.attempt	✅	1
gh-aw.run.actor	✅	mnkiefer
gh-aw.repository	✅	github/gh-aw
gh-aw.staged	✅	false (boolean)

All 7 required setup span attributes are present. ✅

Conditional attributes present: gen_ai.system, gh-aw.engine.id, gh-aw.event_name, gh-aw.episode.id, gh-aw.episode.kind, gh-aw.hop.id, gh-aw.workflow_call.id, gh-aw.cli.version, github.actions.run_url — all appropriately populated.

Conclusion Span — spec §10.2

⏳ Not yet emitted.

Agent Span — spec §10.3 (GenAI semantic conventions)

⏳ Not yet emitted.

Resource Attributes — spec §11.1

Attribute	Present	Value
service.name	✅	gh-aw.otlp-data-quality-validator
service.version	✅	1.0.52
github.repository	✅	github/gh-aw
github.run_id	✅	26620247189
github.run_attempt	✅	1
github.actions.run_url	✅	https://github.com/github/gh-aw/actions/runs/26620247189

All 6 required resource attributes present. ✅

Conditional resource attributes also present: github.event_name, github.ref, github.ref_name, github.sha, github.job, github.workflow_ref, github.actor_id, runner.os, runner.arch, runner.name, runner.environment, gh-aw.awf.version, deployment.environment.

Instrumentation Scope

Check	Result
scope.name == "gh-aw"	✅ Pass
scope.version == service.version	✅ Pass (1.0.52 == 1.0.52)

---

E. Export and Fan-out Health

Check	Result
JSONL mirror written (/tmp/gh-aw/otel.jsonl)	✅ 1 line
Export error count	✅ 0 (/tmp/gh-aw/agent/otlp-export-errors.count = 0)
Export error details	✅ None (/tmp/gh-aw/agent/otlp-export-errors.jsonl absent)
Backend visibility confirmed	❌ No MCP backend tool available
Multi-endpoint fan-out	N/A (single Sentry endpoint configured)

The local pipeline (JSONL mirror write) is healthy. No export errors means the OTLP HTTP POST to Sentry either succeeded or errors would appear in the errors file. Absence of errors is the best signal available.

---

F. Root-cause Hypothesis

No failures detected in the emitted data. However, two latent risks are worth noting:

Risk 1 — Sentry header rewrite (medium probability):
The OTLP endpoint is o205451.ingest.us.sentry.io. Sentry's OTLP ingestion requires the x-sentry-auth header instead of the standard Authorization header. If OTEL_EXPORTER_OTLP_HEADERS contains Authorization: DSN <key> without rewrite, Sentry will return HTTP 401 and reject the span. Supporting evidence: the Sentry endpoint URL is confirmed; header contents are redacted (in AWF_ONE_SHOT_TOKENS) and cannot be inspected. This is the spec §8 security/privacy boundary working as intended.

Risk 2 — Backend query gap (low probability):
No MCP backend tool is available to confirm the span appeared in Sentry. This is not a data loss indicator — it is a validation environment gap.

Alternative explanations:

• If the span is not visible in Sentry, it could be a Sentry ingestion delay (typically < 60 s) or a rate limit on the free Sentry tier.
• The /integration/otlp path in the Sentry URL is the correct OTLP-compatible ingest path; a misconfigured /api/0/envelope/ URL would cause 400 errors, which would appear in export errors.

---

G. Recommended Fixes (Prioritized)

1. Verify Sentry header rewrite is applied — Confirm that OTEL_EXPORTER_OTLP_HEADERS uses x-sentry-auth (not Authorization) per Sentry's OTLP requirements and spec §8 guidance. If the header is wrong, all spans are silently dropped by Sentry despite no export error being recorded.

2. Add backend validation step — After the conclusion span is emitted, query Sentry via API (or add a Sentry MCP tool) to confirm traceId=2384236d85e963e7f7bf1fc55dd1c6bd is query-visible. This closes the "emitted vs. exported vs. query-visible" gap.

3. Validate conclusion span attributes after run completes — Re-run this validator or inspect /tmp/gh-aw/otel.jsonl after conclusion to confirm all §10.2 required attributes (gh-aw.run.status, gh-aw.error_count, gh-aw.warning_count, gh-aw.action_minutes, gh-aw.output.item_count, gh-aw.otlp.export_errors) are present.

4. Add GenAI agent span validation — Once gh-aw.agent.agent is emitted, confirm gen_ai.operation.name=chat, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens per spec §10.3.

---

H. Validation Queries Used

# 1. Check mirror exists and line count
wc -l /tmp/gh-aw/otel.jsonl

# 2. Span summary
jq -c '.resourceSpans[].scopeSpans[].spans[] | {name, traceId, spanId, parentSpanId, kind}' /tmp/gh-aw/otel.jsonl

# 3. Unique trace IDs (expect 1)
jq -r '.resourceSpans[].scopeSpans[].spans[].traceId' /tmp/gh-aw/otel.jsonl | sort -u | wc -l

# 4. Export errors
cat /tmp/gh-aw/agent/otlp-export-errors.count 2>/dev/null || echo "0"

# 5. Trace ID env var vs span
echo "Env: $GITHUB_AW_OTEL_TRACE_ID"
jq -r '.resourceSpans[].scopeSpans[].spans[].traceId' /tmp/gh-aw/otel.jsonl

# 6. Full attribute inspection
cat /tmp/gh-aw/otel.jsonl | python3 -c "
import sys, json
for line in sys.stdin:
    obj = json.loads(line)
    for rs in obj.get('resourceSpans', []):
        res = {a['key']: list(a['value'].values())[0] for a in rs['resource']['attributes']}
        for ss in rs['scopeSpans']:
            for s in ss['spans']:
                attrs = {a['key']: list(a['value'].values())[0] for a in s.get('attributes', [])}
                print(json.dumps({'name': s['name'], 'resource': res, 'attrs': attrs}, indent=2))
"

Generated by 🧭 OTLP Data Quality Validator · sonnet46 1.5M · ◷

• expires on Jun 5, 2026, 5:46 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #36188

View newer issue

---

This action was performed automatically by the OTLP Data Quality Validator workflow.