Skip to content

patterndetectorprocessor: emit verdict counter metric for dashboard panels #261

Description

@trilamsr

Summary

The module/processor/patterndetectorprocessor/ README explicitly defers
self-telemetry to v0.3 ("No self-telemetry yet. Self-telemetry is on the
v0.3 roadmap"). The patterns Grafana dashboard
(install/kubernetes/tracecore/dashboards/patterns.json, shipped in
PR feat/grafana-dashboard-patterns) currently drives all verdict-rate
panels off Loki LogQL against the emitted verdict log records,
because there is no otelcol_processor_patterndetector_verdicts_emitted_total
metric to query.

Request

Emit at least the following self-tel from the processor:

Metric Type Labels
otelcol_processor_patterndetector_verdicts_emitted_total Counter pattern_id, confidence
otelcol_processor_patterndetector_verdict_evidence_count Histogram pattern_id
otelcol_processor_patterndetector_consume_logs_duration_seconds Histogram (none — single hot path)

Naming follows the in-tree-namespace convention pinned in
docs/migration/v0.1-to-v0.2.md ("In-tree receiver / exporter namespace
alignment" §): otelcol_processor_<name>_* where <name> is the
processor's OCB component type (patterndetector).

Why this matters for the dashboard

Three of the six panels in the patterns dashboard would be simpler,
lower-latency, and not Loki-dependent if these metrics existed:

  1. Verdict rate by pattern_id — currently sum by (pattern_id) (rate({...} | json | __error__=""[5m])) against Loki.
  2. Confidence distribution — currently a Loki log-line aggregation.
  3. Verdict count per pattern over time — currently Loki.

Once the metrics ship, those panels can swap to native PromQL and the
dashboard will work on Prometheus-only stacks.

Acceptance

  • New metric instruments wired via the per-processor MeterProvider pattern
    (see docs/migration/v0.1-to-v0.2.md §"In-tree receiver / exporter
    namespace alignment").
  • internal/integration/ocb_scrape_test.go extended to assert the names
    on a real :8888/metrics scrape.
  • Dashboard PR follow-up swaps the LogQL panels to PromQL.

Out of scope

  • Detector code changes (the metric is emitted by the processor wrapper,
    not the underlying module/pkg/patterns/ library).
  • Receiver-side metrics — those already exist via the standard
    otelcol_receiver_* family.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions