slo-rules.yaml: 'pipeline' label doesn't exist on otelcol_* metrics; 'unless on (instance)' join silently no-ops

[trilamsr]

Summary

Two label-alignment defects in
install/kubernetes/tracecore/dashboards/slo-rules.yaml:

1. pipeline label does not exist on the source metrics

The drop-rate recording rule (tracecore:pipeline_drop_rate:5m)
groups by pipeline:

sum by (pipeline) (
  rate(otelcol_processor_refused_log_records_total[5m])
  + rate(otelcol_exporter_send_failed_log_records_total[5m])
)
/
clamp_min(sum by (pipeline) (
  rate(otelcol_receiver_accepted_log_records_total[5m])
), 0.001)

Upstream processorhelper/obsreport.go only stamps a processor
attribute. Upstream receiverhelper/obsreport.go stamps receiver

• transport. There is no pipeline label. sum by (pipeline)
  collapses to a single empty pipeline="" group. The
  {{ $labels.pipeline }} field in the alert annotations renders empty.

Math: the ratio is still per-pod-correct (since all metrics carry
the same instance), but the runbook's per-pipeline localization
guidance doesn't apply to the alert payload.

2. unless on (instance) join in TracecoreSelftelemetryDown doesn't match

up{job="tracecore"} == 0 unless on (instance) kube_pod_status_phase{phase="Failed"} == 1

up{} labels: instance, job (from scrape config).
kube_pod_status_phase{} labels: namespace, pod, phase, uid.

The two metrics have no shared instance label unless an external
relabel step injects one. The unless on (instance) ... is effectively
a no-op — the alert becomes functionally identical to
TracecorePodDown. The intent (separate "listener wedge" from
"pod crashed") is not actually implemented.

Fix

For (1): either replace sum by (pipeline) with sum by (component_id, instance)
(or sum by (processor) / sum by (exporter) per-rule), or have
operators wire pipeline naming via target_label: pipeline in
metric_relabel_configs. Document the choice.

For (2): join via namespace + pod (after relabeling up{} to
carry pod from the kube-prometheus-stack scrape config) — or
drop the unless clause and document why "self-tel down" implies
"investigate pod first" in the runbook.

References

• Upstream label sources:
  • processor@v0.110.0/processorhelper/obsreport.go line 58 (ProcessorKey only)
  • receiver@v0.110.0/receiverhelper/obsreport.go line 64-65 (ReceiverKey, TransportKey)
• slo-rules.yaml lines 117-126 (drop-rate rule), 181 (selftel unless-clause)
• Found during adversarial review of PR docs(slo): declare v0.4 self-observability SLO targets + runbook #288.

[trilamsr]

Fixed in 11592d0 on chore/v1-rc1-knowledge-gaps.

(1) pipeline label drift: the drop-rate recording rule grouped by (pipeline), but upstream OTel obsreport stamps processor/receiver/exporter/transport — never pipeline. Split the recording rule into per-instance numerator + denominator and aggregate to a per-instance drop ratio. The alert payload now carries the actual instance label and the description points operators at the per-component upstream series for localization.

(2) unless on (instance) no-op: up{} carries instance+job while kube_pod_status_phase{} carries namespace+pod+phase+uid — no shared instance label, so the unless clause never matched. Removed the duplicative TracecoreSelftelemetryDown rule; TracecoreSelftelemetryEmpty (which uses a real join via job+instance) already covers the listener-wedge case the original rule tried to capture.

Verification: promtool check rules dashboards/slo-rules.yaml → SUCCESS: 13 rules.