feat(recipe): DCGM to hw.* OTTL transform + ADR for metrics-side detector wiring (PR-A for #260)

[trilamsr]

Summary

PR-A for issue #260 — the recipe-side half. Two commits:

1. docs(adr): metrics-to-logs converter for pattern input (Option A) — records the architectural decision and the upstream-blocker analysis under a new docs/adrs/ directory.
2. docs(recipe): extend prometheus-scrape with DCGM to hw.* OTTL transform — adds transform/dcgm_to_hw_semconv (a second OTTL pass after transform/gpu_vendor) and documents every per-pattern projection in docs/integrations/prometheus-scrape.md.

Architectural decision (issue #260 §Proposed plan step 2)

Option A — extend patterndetectorprocessor with processor.WithMetrics alongside the existing WithLogs (deferred to PR-B; this PR ships only the wire-format contract PR-B will consume).

Per [[adopt-over-build]] Option B (OTTL metrics-to-logs converter) was the preferred starting point. It's blocked upstream: at OTel contrib v0.130 (the release pinned in builder-config.yaml) the transformprocessor signal contexts are sealed per-signal — metric_statements can only reference resource | scope | metric | datapoint paths, never log (README @ v0.130.0 § Config). Surveying every contrib connector at v0.130 also turns up zero metrics→logs primitives (countconnector, signaltometricsconnector, spanmetrics, exceptions, servicegraph all output metrics; routing/failover/roundrobin/otlpjson are signal-preserving; datadog and grafanacloud are vendor-specific). The full survey + reasoning is in ADR-0001.

Option C (routing connector + sibling) is strictly heavier than A with the same end state — routing preserves signal type, so it cannot bridge metrics→logs either.

Trade-off accepted: doubled processor surface inside patterndetectorprocessor (one ConsumeLogs, one ConsumeMetrics). Mitigation: the metrics path is a thin projection on top of the same module/pkg/patterns/ library; verdict emission stays log-based via a connector. Upstream-contribution slot opened — a metricthresholdconnector (inverse of signaltometricsconnector) would let us collapse Option A back to Option B without operator-visible churn. Tracked under RFC-0013 §5.

Per-pattern OTTL projection (this PR)

Pattern	Raw DCGM input	OTel output	Trigger-statement shape (PR-B consumes this)
#1 NVLink	DCGM_FI_PROF_NVLINK_L{N}_{TX,RX}_BYTES	hw.gpu.nvlink.io (Counter, By) + hw.gpu.nvlink.link={N} + network.io.direction={transmit|receive}	rate(hw.gpu.nvlink.io[5m]) < 0.5 × per-GPU median over the per-hw.id link set, for 10m
#3 HBM ECC	DCGM_FI_DEV_ECC_{SBE,DBE}_{VOL,AGG}_TOTAL	hw.errors (Counter, {error}) + error.{type,subtype,persistence}	increase(hw.errors{error.type="uncorrected", error.persistence="volatile"}[5m]) > 0
#4 thermal	DCGM_FI_DEV_{THERMAL,POWER,SYNC_BOOST,BOARD_LIMIT,LOW_UTIL}_VIOLATION	hw.gpu.throttle.duration (Counter, s) + hw.gpu.throttle.reason	increase(hw.gpu.throttle.duration{reason="thermal"}[5m]) > 30s, for 10m
#5 PCIe	DCGM_FI_PROF_PCIE_{TX,RX}_BYTES	hw.gpu.io (Counter, By) + network.io.direction + hw.gpu.pci.bdf resource attr	rate(hw.gpu.io[5m]) < 0.3 × host median, for 15m

Resource attribution: UUID or gpu_uuid → hw.id; gpu or GPU → hw.gpu.index; hw.type="gpu" stamped on every DCGM series. Dual-label fallback handles both legacy and modern dcgm-exporter releases.

What's NOT in this PR

The third commit originally planned by issue #260 — feat(recipe): metrics→logs OTTL alert emission — is dropped because OTTL cannot emit log records from metric streams at v0.130 (root cause cited above, full analysis in ADR-0001). The fallback is Option A, which requires modifying patterndetectorprocessor source — out of scope for a recipe-only PR. The alert-emission half lands in PR-B alongside the NVLink detector code.

Test plan

• ./_build/tracecore validate --config=docs/integrations/examples/prometheus-scrape.yaml exits 0.
• bash scripts/doc-check.sh clean (test-name parity, markdown link integrity, banned-phrase lint, integration-recipe rubric all pass).
• bash scripts/validator-recipe.sh validates 6 recipes (skips 2 requires-linux / requires-k8s-cluster — CI ubuntu runner exercises those paths).
• make check (golangci-lint, go vet, mod-verify) clean.
• Adversarial reviewer check that the OTTL set(metric.name, ...) identity-conflict caveat (recipe markdown § "Identity-conflict caveat") matches the operator's downstream backend's deduplication semantics for the four projected metric families.
• CI runner exercises the requires-linux / requires-k8s recipes that this PR's host skipped.

Followup

• PR-B (issue Recipe extension: emit hw.gpu.nvlink.* + wire metrics path for pattern-1 detector #260, blocked by this PR's merge) — extend patterndetectorprocessor with processor.WithMetrics; land module/pkg/patterns/nvlink_degradation.go; wire verdict emission through a sibling logs pipeline via a connector. Decision rationale baked into ADR-0001 so PR-B doesn't relitigate.
• Upstream contribution (v0.3 cycle, RFC-0013 §5) — propose a metricthresholdconnector (or signaltologsconnector) to OTel-contrib. If/when it lands, the recipe can collapse back to Option B without operator-visible churn — the hw.gpu.* wire-format contract this PR ships is the load-bearing customer surface, not the upstream component selection.
• dcgm-exporter throttle-reason mapping verification — the pattern Wave 1: governance bootstrap (CODEOWNERS, DCO, signing) #4 mapping table assumes modern dcgm-exporter DCGM_FI_DEV_*_VIOLATION series names. If your dcgm-exporter release exposes the throttle bitmask as DCGM_FI_DEV_CLOCKS_EVENT_REASONS instead (per the semconv proposal Open Question ci(deps): bump the gh-actions group with 5 updates #1), extend the OTTL block to expand the bitmask into discrete hw.gpu.throttle.duration datapoints — out of this recipe's scope; tracked as a follow-up under the same milestone.

docs(recipe): The `prometheus-scrape` recipe gains a second OTTL transform pass that projects raw `dcgm-exporter` `DCGM_FI_*` series into the customer-stable `hw.gpu.*` / `hw.errors` namespace (semconv proposal). Four pattern families covered — NVLink, HBM ECC, thermal throttle, PCIe — using upstream OTTL functions only (`set`, `IsMatch`, `ExtractPatterns`, `Int`). The wire-format contract this transform produces is the input PR-B's NVLink detector will consume; the in-tree component change for the detector itself lands in a follow-up PR per ADR-0001.