From 6b1a42258e769d623dfb011fc00ed145139e65b0 Mon Sep 17 00:00:00 2001 From: Tri Lam Date: Sun, 31 May 2026 22:19:49 -0700 Subject: [PATCH 1/2] docs(multi-cluster): v0 read-only federation recipe (roadmap A18) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the operator pattern for running tracecore across N source clusters with a central aggregation collector. Read-only roll-up contract: source clusters detect verdicts locally + stamp cluster.id via OTTL transform; aggregation cluster forwards via OTLP/HTTP without re-running detection. Cross-cluster verdict dedup is deferred to v1+ (roadmap C4). Every config primitive is upstream OTel collector core/contrib: otlpreceiver, otlphttpexporter, transformprocessor (RFC-0013 Β§1, Β§2 adoption matrix). No tracecore-specific federation code. Ships: - docs/multi-cluster.md (topology diagram, role-by-role config, verdict-routing FAQ, Helmfile example, failure modes) - docs/integrations/examples/multi-cluster-source.yaml (validates) - docs/integrations/examples/multi-cluster-aggregation.yaml (validates) - docs/README.md index row Verified: bash scripts/doc-check.sh, make check, ./_build/tracecore validate on both example configs. Signed-off-by: Tri Lam --- docs/README.md | 1 + .../examples/multi-cluster-aggregation.yaml | 98 ++++++++ .../examples/multi-cluster-source.yaml | 121 ++++++++++ docs/multi-cluster.md | 215 ++++++++++++++++++ 4 files changed, 435 insertions(+) create mode 100644 docs/integrations/examples/multi-cluster-aggregation.yaml create mode 100644 docs/integrations/examples/multi-cluster-source.yaml create mode 100644 docs/multi-cluster.md diff --git a/docs/README.md b/docs/README.md index aa6e9e40..4ba68f23 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,6 +23,7 @@ Legend: πŸ‘€ operator Β· πŸ› οΈ contributor Β· πŸ›οΈ maintainer Β· 🌐 exter | [nps.md](nps.md) | πŸ›οΈ | NPS-style operator-feedback methodology. | | [reproducibility.md](reproducibility.md) | 🌐 πŸ‘€ | Third-party verification recipe for published releases: rebuild + diffoscope + cosign + SLSA + SBOM. | | [maintainership.md](maintainership.md) | πŸ›οΈ πŸ› οΈ | Governance: who has commit access, how RFCs are sponsored, how security issues are handled. | +| [multi-cluster.md](multi-cluster.md) | πŸ‘€ | Multi-cluster federation v0 (read-only roll-up): N source clusters stamp `cluster.id` via OTTL transform, forward OTLP/HTTP to a central aggregation collector that fans out to backends. | ## Subdirectories diff --git a/docs/integrations/examples/multi-cluster-aggregation.yaml b/docs/integrations/examples/multi-cluster-aggregation.yaml new file mode 100644 index 00000000..a43d1f4f --- /dev/null +++ b/docs/integrations/examples/multi-cluster-aggregation.yaml @@ -0,0 +1,98 @@ +# Multi-cluster federation v0 β€” AGGREGATION-cluster role config (one +# central collector). Receives OTLP/HTTP from N source-cluster +# tracecore collectors and fans out to backends (Loki for verdict logs, +# Tempo/Datadog/ClickHouse/etc. for metrics + traces). Every record on +# the wire already carries the `cluster.id` resource attribute stamped +# by the source-cluster collector (see multi-cluster-source.yaml). +# +# Adoption matrix (RFC-0013 Β§1, Β§2): every component is upstream OTel +# collector core or contrib. The aggregation tier deliberately does NOT +# re-run patterndetectorprocessor β€” verdicts arrive as logs that the +# source clusters already emitted, and re-detecting on top would +# require write-path dedup (v1+ roadmap C4, intentionally out of scope +# for federation v0). +# +# receiver/otlp β€” collector core. The aggregation listener; +# accepts OTLP/HTTP from every source cluster. +# exporter/otlphttp β€” collector core. Backend egress; this example +# targets Loki (verdict logs); add additional +# exporters per integrations/{tempo,datadog, +# clickhouse-direct}.md for metrics + traces. +# processor/batch β€” collector core. Standard. +# +# v0 READ-ONLY contract: +# - Aggregation receives verdicts (already-emitted logs) from every +# source cluster. +# - Aggregation does NOT re-emit deduplicated verdicts across +# clusters; per-cluster verdicts roll up side-by-side, keyed by +# `cluster.id` at query time on the backend. +# - patternddetector is intentionally absent from the pipelines below. +# Re-running detection at the aggregation tier would either +# duplicate verdicts (same input β†’ same output) or require write- +# path dedup, both deferred to v1+. +# +# Endpoint (Loki distributor): see docs/integrations/loki.md for the +# full breakdown. The `otlphttp` exporter appends `/v1/logs` per the +# OTLP spec, so the endpoint resolves to `…/otlp/v1/logs`. + +receivers: + # OTLP listener for source-cluster traffic. mTLS posture is the + # recommended deploy shape (configure with `tls:` and the CA bundle + # the gateway uses to issue source-cluster client certs); the + # listener below accepts unauthenticated HTTP for local validation, + # rendering is the operator's responsibility at deploy time. + otlp: + protocols: + http: + endpoint: 0.0.0.0:4318 + # max_request_body_size guards against pathological source- + # cluster bursts. Default is 20 MiB; raise only if source + # batchprocessor flush sizes legitimately exceed it. + max_request_body_size: 20971520 + +processors: + batch: {} + +exporters: + # Loki for verdict logs. The recipe in docs/integrations/loki.md + # documents the X-Scope-OrgID tenant header model and the labels-vs- + # structured-metadata mapping that `cluster.id` lands in (resource + # attribute, so it is a candidate for the Loki label index if added + # to `default_resource_attributes_as_index_labels` on the Loki side). + otlphttp/loki: + endpoint: http://loki-distributor.observability.svc.cluster.local:3100/otlp + compression: gzip + headers: + X-Scope-OrgID: tracecore + sending_queue: + enabled: true + num_consumers: 4 + queue_size: 1000 + retry_on_failure: + enabled: true + initial_interval: 5s + max_interval: 30s + max_elapsed_time: 300s + +service: + pipelines: + # Verdict logs from every source cluster roll up here. The + # `cluster.id` resource attribute is preserved by the otlphttp + # exporter and lands in Loki as either a structured-metadata field + # (default) or a stream label (if the Loki operator adds + # `cluster.id` to `default_resource_attributes_as_index_labels`). + logs/federation: + receivers: [otlp] + processors: [batch] + exporters: [otlphttp/loki] + # Metrics + traces roll-up is optional at v0; uncomment and add a + # backend exporter per integrations/{tempo,datadog,clickhouse- + # direct}.md when the backend is provisioned. + # metrics/federation: + # receivers: [otlp] + # processors: [batch] + # exporters: [otlphttp/metrics_backend] + # traces/federation: + # receivers: [otlp] + # processors: [batch] + # exporters: [otlphttp/traces_backend] diff --git a/docs/integrations/examples/multi-cluster-source.yaml b/docs/integrations/examples/multi-cluster-source.yaml new file mode 100644 index 00000000..255acd08 --- /dev/null +++ b/docs/integrations/examples/multi-cluster-source.yaml @@ -0,0 +1,121 @@ +# Multi-cluster federation v0 β€” SOURCE-cluster role config (one per +# source cluster). The collector runs the standard tracecore signal +# pipeline (receivers, patterndetectorprocessor, batch) and exports +# OTLP/HTTP to an aggregation-cluster collector instead of directly to +# a backend. The aggregation cluster fans out to Loki / Tempo / etc. +# +# Adoption matrix (RFC-0013 Β§1, Β§2): every component below is upstream +# OTel collector core or contrib. No tracecore-specific federation code. +# +# receiver/otlp β€” collector core, accepts traffic from the +# cluster's signal sources (or from a +# cluster-internal feeder collector). Listening +# keeps this file role-symmetric with +# multi-cluster-aggregation.yaml; replace with +# filelog / journald / k8sobjects / prometheus +# per the source recipes if this collector also +# ingests directly. +# processor/transform β€” contrib, stamps the `cluster.id` resource +# attribute that the aggregation tier uses to +# tell verdicts from different clusters apart. +# processor/patterndetector β€” tracecore; detects verdicts AT the source +# cluster so the aggregation tier sees verdict +# logs rather than raw signals (verdict-routing +# model documented in docs/multi-cluster.md). +# processor/batch β€” collector core, standard. +# exporter/otlphttp β€” collector core, OTLP/HTTP egress to the +# aggregation cluster's OTLP receiver. Same +# component the otel-backend / honeycomb / loki +# recipes use; only the endpoint changes. +# +# cluster.id injection: the `transform/cluster_id` processor stamps the +# resource attribute on every log/metric/trace record at this collector +# before it hits the OTLP exporter. The literal value must be replaced +# per cluster (REPLACE_WITH_CLUSTER_ID, e.g. `cluster-a`). Render at +# deploy time via Helm template or envsubst; tracecore does not expand +# environment variables in YAML (doc-check.sh banned-placeholder gate). +# +# Endpoint: the aggregation cluster's OTLP/HTTP receiver. The default +# port for OTLP/HTTP is 4318; the path suffix `/v1/{logs,metrics,traces}` +# is appended by the exporter automatically per the OTLP spec β€” do NOT +# include it in `endpoint`. +# +# Auth: a literal `Authorization: Bearer …` header is the minimum +# upstream OTLP receivers accept. mTLS is the recommended posture for +# cross-cluster ingress; configure with the `tls:` block on both sides +# and a CA / cert bundle issued by the aggregation cluster's gateway. +# The marker below is a placeholder so `tracecore validate` succeeds +# offline; render the real token / cert at deploy time. + +receivers: + otlp: + protocols: + http: + endpoint: 0.0.0.0:4318 + +processors: + # Stamp `cluster.id` on every resource. The aggregation tier uses + # this attribute (and only this attribute) to attribute every verdict + # to its source cluster. Resource context is correct because OTLP + # resource attributes flow downstream attached to every record in the + # batch without re-stamping per datapoint. + transform/cluster_id: + log_statements: + - context: resource + statements: + - set(attributes["cluster.id"], "REPLACE_WITH_CLUSTER_ID") + metric_statements: + - context: resource + statements: + - set(attributes["cluster.id"], "REPLACE_WITH_CLUSTER_ID") + trace_statements: + - context: resource + statements: + - set(attributes["cluster.id"], "REPLACE_WITH_CLUSTER_ID") + + # Detect verdicts AT this source cluster. Emitting verdicts here + # (rather than at the aggregation tier) keeps the federation v0 + # contract READ-ONLY at the aggregation cluster: aggregation never + # re-runs detection on the same raw signal, so there is no per-cluster + # vs cross-cluster verdict-dedup problem at this milestone (v1+ + # roadmap C4 covers write-path dedup). + patterndetector: {} + + batch: {} + +exporters: + # Egress to the aggregation cluster's OTLP receiver. The endpoint + # below is a placeholder; replace with the gateway / service address + # exposed by the aggregation cluster. The `headers:` block carries + # the cross-cluster auth token; mTLS via `tls:` is the recommended + # posture for production deploys. + otlphttp/aggregation: + endpoint: https://REPLACE_WITH_AGGREGATION_HOST:4318 + compression: gzip + timeout: 10s + headers: + Authorization: REPLACE_WITH_BEARER_TOKEN + sending_queue: + enabled: true + num_consumers: 4 + queue_size: 1000 + retry_on_failure: + enabled: true + initial_interval: 5s + max_interval: 30s + max_elapsed_time: 300s + +service: + pipelines: + logs/federation: + receivers: [otlp] + processors: [transform/cluster_id, patterndetector, batch] + exporters: [otlphttp/aggregation] + metrics/federation: + receivers: [otlp] + processors: [transform/cluster_id, batch] + exporters: [otlphttp/aggregation] + traces/federation: + receivers: [otlp] + processors: [transform/cluster_id, batch] + exporters: [otlphttp/aggregation] diff --git a/docs/multi-cluster.md b/docs/multi-cluster.md new file mode 100644 index 00000000..9de40e99 --- /dev/null +++ b/docs/multi-cluster.md @@ -0,0 +1,215 @@ + + + +# Multi-cluster federation v0 (read-only roll-up) + +How to run tracecore across N source clusters with a central +aggregation collector that rolls verdicts up to a single backend. The +contract is **read-only**: source clusters emit verdicts locally, the +aggregation cluster collects and forwards them; nothing dedupes +across clusters at this milestone (v1+ roadmap C4 covers cross-cluster +write-path dedup). + +Every component on both sides is upstream OTel collector core or +contrib (RFC-0013 Β§1, Β§2 adoption matrix). There is no tracecore- +specific federation code. + +## Topology + +``` ++-------------------+ +-------------------+ +-------------------+ +| source cluster A | | source cluster B | | source cluster C | +| | | | | | +| tracecore | | tracecore | | tracecore | +| receivers | | receivers | | receivers | +| transform/ | | transform/ | | transform/ | +| cluster_id=A | | cluster_id=B | | cluster_id=C | +| patterndetector | | patterndetector | | patterndetector | +| otlphttp ────────┐ | otlphttp ────────┐ | otlphttp ────────┐ ++-------------------+| +-------------------+| +-------------------+| + | | | + +------------+--------------+---------------------------+ + v + +---------------------+ + | aggregation cluster | + | | + | tracecore | + | receiver/otlp | + | batch | + | otlphttp ──────────┐ + +---------------------+| + v + +-----------------------+ + | backend (Loki/Tempo/ | + | ClickHouse/Datadog/…) | + +-----------------------+ +``` + +Verdict logs are the load-bearing signal. Metrics and traces roll up +optionally β€” uncomment the metrics/traces pipelines in the aggregation +config and point them at a metrics/traces backend. + +## Role: source cluster + +One tracecore collector per source cluster. The collector runs the +normal tracecore signal pipeline (receivers, patterndetectorprocessor, +batch) and exports OTLP/HTTP to the aggregation cluster instead of +directly to a backend. + +The `cluster.id` injection point is the `transformprocessor` running +**before** patterndetectorprocessor on every signal pipeline. OTTL +stamps the attribute at resource context, so it attaches once per +record batch and travels with every downstream verdict. + +```yaml +# docs/integrations/examples/multi-cluster-source.yaml +processors: + transform/cluster_id: + log_statements: + - context: resource + statements: + - set(attributes["cluster.id"], "REPLACE_WITH_CLUSTER_ID") + metric_statements: + - context: resource + statements: + - set(attributes["cluster.id"], "REPLACE_WITH_CLUSTER_ID") + trace_statements: + - context: resource + statements: + - set(attributes["cluster.id"], "REPLACE_WITH_CLUSTER_ID") +``` + +Replace `REPLACE_WITH_CLUSTER_ID` per cluster (`cluster-a`, +`cluster-b`, `cluster-c`) at deploy time β€” tracecore does not expand +environment variables in YAML; render through Helm template or +envsubst. + +Validate the full file with the in-tree binary: + +```sh +./tracecore validate --config=docs/integrations/examples/multi-cluster-source.yaml +``` + +## Role: aggregation cluster + +One tracecore collector at a central location (its own cluster, a +dedicated namespace in one of the source clusters, or a managed +service). The collector listens on OTLP/HTTP and exports to whichever +backend(s) the operator wants verdicts in. The pipelines have **no +patternddetector**: re-running detection at the aggregation tier on +already-detected verdicts would duplicate output (same input β†’ same +output) without solving any operator problem; cross-cluster verdict +dedup is roadmap C4 (v1+). + +```yaml +# docs/integrations/examples/multi-cluster-aggregation.yaml +receivers: + otlp: + protocols: + http: + endpoint: 0.0.0.0:4318 +exporters: + otlphttp/loki: + endpoint: http://loki-distributor.observability.svc.cluster.local:3100/otlp + headers: + X-Scope-OrgID: tracecore +service: + pipelines: + logs/federation: + receivers: [otlp] + processors: [batch] + exporters: [otlphttp/loki] +``` + +Validate: + +```sh +./tracecore validate --config=docs/integrations/examples/multi-cluster-aggregation.yaml +``` + +## Verdict routing across the boundary + +The aggregation collector forwards verdicts; it does not re-detect +them. This is intentional and is what "read-only roll-up" means at +v0. + +| Question | Answer at v0 | +|---|---| +| Does the aggregation cluster need `patterndetectorprocessor` in its pipelines? | No. Verdicts are already detected at the source cluster and arrive as logs. Re-running detection on the same input would duplicate output. | +| What if the same incident shows up in two source clusters? | Two verdicts roll up, each tagged with its source `cluster.id`. The backend (Loki / Tempo / etc.) sees them as distinct records; the operator queries by `cluster.id` to slice per cluster. | +| Can the aggregation tier emit a single deduplicated verdict per incident? | No, deferred to v1+ (roadmap C4 cross-cluster write-path dedup). Implementing this needs a stateful identity contract for verdicts that the v0 detector does not emit. | +| Where does `cluster.id` land on the backend? | As a resource attribute. On Loki specifically, it lands in structured metadata by default; promote it to a stream label by adding `cluster.id` to `default_resource_attributes_as_index_labels` on the Loki side (see [`loki.md`](integrations/loki.md)). | + +## Helmfile example: 3-cluster topology + +Helmfile renders one `values.yaml` per cluster against the bundled +tracecore Helm chart. Adopt the operator's preferred secret-render +path (External Secrets Operator, sealed-secrets, sops, CSI driver) +for the `REPLACE_WITH_BEARER_TOKEN` literal. + +```yaml +# helmfile.yaml +helmDefaults: + wait: true + timeout: 300 + +releases: + - name: tracecore-source-a + chart: ./install/kubernetes/tracecore + kubeContext: cluster-a + values: + - values/source-cluster-a.yaml + + - name: tracecore-source-b + chart: ./install/kubernetes/tracecore + kubeContext: cluster-b + values: + - values/source-cluster-b.yaml + + - name: tracecore-source-c + chart: ./install/kubernetes/tracecore + kubeContext: cluster-c + values: + - values/source-cluster-c.yaml + + - name: tracecore-aggregation + chart: ./install/kubernetes/tracecore + kubeContext: aggregation + values: + - values/aggregation.yaml +``` + +Per-cluster `values/source-cluster-{a,b,c}.yaml` stubs differ only in +the `cluster.id` literal and (optionally) per-cluster auth; the +collector config is the source-role recipe with that literal rendered. +The `values/aggregation.yaml` stub deploys the aggregation-role +recipe. Both recipes are config-only β€” chart values surface as +top-level config strings, so the chart change for federation v0 is +"pass a different config file"; no chart schema bump is required. + +(Chart schema additions for first-class federation β€” a `federation:` +block that materializes the OTTL transform + bearer header from +typed fields β€” are tracked separately as a v0.4 deliverable.) + +## Failure modes + +| Symptom | First check | +|---|---| +| Aggregation cluster sees no traffic | Source cluster's `otlphttp/aggregation` exporter is dropping requests. Check `otelcol_exporter_send_failed_log_records` and the bearer-token / mTLS posture. | +| Verdicts arrive at aggregation without `cluster.id` | `transform/cluster_id` is not in the source pipeline's processor chain, or runs at the wrong context. Confirm it runs at `context: resource` and is listed before `patterndetector` in `service.pipelines.logs/federation.processors`. | +| Same verdict appears multiple times in the backend per incident | Expected at v0 when the same incident triggers in multiple source clusters. Each source cluster emits independently. Operator queries should `group by cluster.id` to slice; cross-cluster dedup is v1+ (roadmap C4). | +| Aggregation cluster's exporter saturates | Source-cluster traffic exceeds the aggregation backend's ingestion rate. Either tune `batchprocessor` on each source cluster (raise `send_batch_size`, raise `timeout`) or scale the aggregation collector horizontally. | +| `cluster.id` is missing from Loki LogQL queries | Loki's distributor receives `cluster.id` as a resource attribute and stores it as structured metadata by default. Either query as `attributes_cluster_id` (LogQL structured-metadata syntax) or promote to a stream label on the Loki side via `default_resource_attributes_as_index_labels`. | +| Source cluster's tracecore can't reach the aggregation cluster's OTLP endpoint | Cross-cluster network path is not open. Federation v0 assumes a routable network from each source cluster to the aggregation cluster's ingress (LoadBalancer service, ingress controller, mesh gateway, or VPC peering). The recipe does NOT prescribe the network topology. | + +## See also + +- Upstream OTLP/HTTP receiver: + [`receiver/otlpreceiver`](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver). +- Upstream OTLP/HTTP exporter: + [`exporter/otlphttpexporter`](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter). +- Upstream OTTL transform processor: + [`processor/transformprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor). +- Generic backend recipe: [`integrations/otel-backend.md`](integrations/otel-backend.md). +- Loki backend recipe (verdict-logs roll-up target): [`integrations/loki.md`](integrations/loki.md). +- Tempo backend recipe (traces roll-up target): [`integrations/tempo.md`](integrations/tempo.md). From 15f6ea7809c0500ee349f2e9467914dc18bc0599 Mon Sep 17 00:00:00 2001 From: Tri Lam Date: Sun, 31 May 2026 22:30:54 -0700 Subject: [PATCH 2/2] fix(multi-cluster): move to integrations/, fix typos + add failure modes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per fresh-context review of #291: - Move docs/multi-cluster.md to docs/integrations/multi-cluster.md so the scripts/validator-recipe.sh + doc-check integration-index gates pick it up. Validator now sees 9 recipes (was 8). - Add docs/README.md row under backend recipes section. - Fix typo patternddetector β†’ patterndetector in doc + aggregation YAML. - Add 3 failure-mode rows: aggregation cluster down > 300s, source-collector restart loses in-memory queue, naked-HTTP receiver. - Auth-extension example (bearer + mTLS) deferred to v0.4 per #297. Signed-off-by: Tri Lam --- docs/README.md | 2 +- .../examples/multi-cluster-aggregation.yaml | 2 +- docs/{ => integrations}/multi-cluster.md | 13 ++++++++----- 3 files changed, 10 insertions(+), 7 deletions(-) rename docs/{ => integrations}/multi-cluster.md (88%) diff --git a/docs/README.md b/docs/README.md index 4ba68f23..fbfdee9d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,7 +23,6 @@ Legend: πŸ‘€ operator Β· πŸ› οΈ contributor Β· πŸ›οΈ maintainer Β· 🌐 exter | [nps.md](nps.md) | πŸ›οΈ | NPS-style operator-feedback methodology. | | [reproducibility.md](reproducibility.md) | 🌐 πŸ‘€ | Third-party verification recipe for published releases: rebuild + diffoscope + cosign + SLSA + SBOM. | | [maintainership.md](maintainership.md) | πŸ›οΈ πŸ› οΈ | Governance: who has commit access, how RFCs are sponsored, how security issues are handled. | -| [multi-cluster.md](multi-cluster.md) | πŸ‘€ | Multi-cluster federation v0 (read-only roll-up): N source clusters stamp `cluster.id` via OTTL transform, forward OTLP/HTTP to a central aggregation collector that fans out to backends. | ## Subdirectories @@ -53,6 +52,7 @@ Backend (exporter-side) recipes: | [integrations/clickhouse-direct.md](integrations/clickhouse-direct.md) | πŸ‘€ | Self-hosted ClickHouse via the bundled `clickhouseexporter`. | | [integrations/loki.md](integrations/loki.md) | πŸ‘€ | Grafana Loki via OTLP/HTTP native ingestion (`otlphttp` exporter, `X-Scope-OrgID` tenant header); labels-vs-structured-metadata mapping for `pattern.*` verdict attributes. | | [integrations/tempo.md](integrations/tempo.md) | πŸ‘€ | Grafana Tempo (OSS, AGPL-3.0) trace backend via the in-tree `otlphttp` exporter. | +| [integrations/multi-cluster.md](integrations/multi-cluster.md) | πŸ‘€ | Multi-cluster federation v0 (read-only roll-up): N source clusters stamp `cluster.id` via OTTL transform, forward OTLP/HTTP to a central aggregation collector that fans out to backends. | Source (receiver-side) recipes β€” RFC-0013 Β§migration PR-J replacements for the deleted in-tree receivers: diff --git a/docs/integrations/examples/multi-cluster-aggregation.yaml b/docs/integrations/examples/multi-cluster-aggregation.yaml index a43d1f4f..378f58c7 100644 --- a/docs/integrations/examples/multi-cluster-aggregation.yaml +++ b/docs/integrations/examples/multi-cluster-aggregation.yaml @@ -26,7 +26,7 @@ # - Aggregation does NOT re-emit deduplicated verdicts across # clusters; per-cluster verdicts roll up side-by-side, keyed by # `cluster.id` at query time on the backend. -# - patternddetector is intentionally absent from the pipelines below. +# - patterndetector is intentionally absent from the pipelines below. # Re-running detection at the aggregation tier would either # duplicate verdicts (same input β†’ same output) or require write- # path dedup, both deferred to v1+. diff --git a/docs/multi-cluster.md b/docs/integrations/multi-cluster.md similarity index 88% rename from docs/multi-cluster.md rename to docs/integrations/multi-cluster.md index 9de40e99..0e0f3395 100644 --- a/docs/multi-cluster.md +++ b/docs/integrations/multi-cluster.md @@ -96,7 +96,7 @@ One tracecore collector at a central location (its own cluster, a dedicated namespace in one of the source clusters, or a managed service). The collector listens on OTLP/HTTP and exports to whichever backend(s) the operator wants verdicts in. The pipelines have **no -patternddetector**: re-running detection at the aggregation tier on +patterndetector**: re-running detection at the aggregation tier on already-detected verdicts would duplicate output (same input β†’ same output) without solving any operator problem; cross-cluster verdict dedup is roadmap C4 (v1+). @@ -138,7 +138,7 @@ v0. | Does the aggregation cluster need `patterndetectorprocessor` in its pipelines? | No. Verdicts are already detected at the source cluster and arrive as logs. Re-running detection on the same input would duplicate output. | | What if the same incident shows up in two source clusters? | Two verdicts roll up, each tagged with its source `cluster.id`. The backend (Loki / Tempo / etc.) sees them as distinct records; the operator queries by `cluster.id` to slice per cluster. | | Can the aggregation tier emit a single deduplicated verdict per incident? | No, deferred to v1+ (roadmap C4 cross-cluster write-path dedup). Implementing this needs a stateful identity contract for verdicts that the v0 detector does not emit. | -| Where does `cluster.id` land on the backend? | As a resource attribute. On Loki specifically, it lands in structured metadata by default; promote it to a stream label by adding `cluster.id` to `default_resource_attributes_as_index_labels` on the Loki side (see [`loki.md`](integrations/loki.md)). | +| Where does `cluster.id` land on the backend? | As a resource attribute. On Loki specifically, it lands in structured metadata by default; promote it to a stream label by adding `cluster.id` to `default_resource_attributes_as_index_labels` on the Loki side (see [`loki.md`](loki.md)). | ## Helmfile example: 3-cluster topology @@ -201,6 +201,9 @@ typed fields β€” are tracked separately as a v0.4 deliverable.) | Aggregation cluster's exporter saturates | Source-cluster traffic exceeds the aggregation backend's ingestion rate. Either tune `batchprocessor` on each source cluster (raise `send_batch_size`, raise `timeout`) or scale the aggregation collector horizontally. | | `cluster.id` is missing from Loki LogQL queries | Loki's distributor receives `cluster.id` as a resource attribute and stores it as structured metadata by default. Either query as `attributes_cluster_id` (LogQL structured-metadata syntax) or promote to a stream label on the Loki side via `default_resource_attributes_as_index_labels`. | | Source cluster's tracecore can't reach the aggregation cluster's OTLP endpoint | Cross-cluster network path is not open. Federation v0 assumes a routable network from each source cluster to the aggregation cluster's ingress (LoadBalancer service, ingress controller, mesh gateway, or VPC peering). The recipe does NOT prescribe the network topology. | +| Aggregation cluster down > 300s β†’ source-cluster verdicts dropped | `otlphttp/aggregation` exporter's default `retry_on_failure.max_elapsed_time` is 300s. Beyond that, queued records drop. For longer outage tolerance, wire `file_storage` extension + `sending_queue.storage` on each source cluster's exporter to persist to disk. v0 recipe ships in-memory queue only β€” see [`filelog-container.md`](filelog-container.md) for the `file_storage` extension shape. | +| Source collector restart drops in-flight verdicts | In-memory `sending_queue` is non-persistent. Same fix as the row above (`file_storage` extension + `sending_queue.storage`). Tracked for v0.4 chart enablement. | +| Aggregation OTLP receiver accepts naked HTTP | The example ships an authentication-free receiver for v0 simplicity. Production deployments MUST wire an auth extension (`bearertokenauth`, `basicauth`, or mTLS via `tls.client_ca_file` + cert-manager) before exposing the endpoint outside the cluster's trust boundary. Tracked at #297. | ## See also @@ -210,6 +213,6 @@ typed fields β€” are tracked separately as a v0.4 deliverable.) [`exporter/otlphttpexporter`](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter). - Upstream OTTL transform processor: [`processor/transformprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor). -- Generic backend recipe: [`integrations/otel-backend.md`](integrations/otel-backend.md). -- Loki backend recipe (verdict-logs roll-up target): [`integrations/loki.md`](integrations/loki.md). -- Tempo backend recipe (traces roll-up target): [`integrations/tempo.md`](integrations/tempo.md). +- Generic backend recipe: [`otel-backend.md`](otel-backend.md). +- Loki backend recipe (verdict-logs roll-up target): [`loki.md`](loki.md). +- Tempo backend recipe (traces roll-up target): [`tempo.md`](tempo.md).