Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/ATTRIBUTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,6 @@ hardware signal.
| `tracecore.alert.pcie_rate_collapse.bytes_per_second` | double | tracecore-ext | alpha | Current rate sample on a per-GPU `hw.gpu.io` collapse | OTTL metrics→logs recipe (blocked on [ADR-0001](adrs/0001-metrics-to-logs-pattern-input.md) PR-B) | `patterndetectorprocessor` `projectPCIeIORecord` (gates PCIe-AER pattern) |
| `tracecore.alert.pcie_rate_collapse.baseline_bytes_per_second` | double | tracecore-ext | alpha | Baseline rate this sample collapsed from | OTTL metrics→logs recipe | `projectPCIeIORecord` |
| `tracecore.alert.pcie_rate_collapse.direction` | string | tracecore-ext | alpha | `transmit` or `receive` — falls back to upstream `network.io.direction` if absent | OTTL metrics→logs recipe | `projectPCIeIORecord` |
| `tracecore.container.lines_per_s` | double | tracecore-ext | development | Per-rank container-stdout line rate over 15s window | OTTL on filelog records (recipe TBD) | Future: nccl_hang corroboration |

---

Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ them unless you explicitly override.
endpoint via the `healthcheck` extension, and zpages via the
`zpages` extension. The default chart values enable all three on
`localhost`. Standard `otelcol_*` metric names apply per
[RFC-0013 §3](rfcs/0013-distro-first-pivot.md#3-customer-stable-contracts);
[RFC-0013 §3](rfcs/0013-distro-first-pivot.md#3-customer-stable-telemetry-contracts);
alerts must not assume the legacy `tracecore_*` prefix.

## Add a real receiver
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/filelog-container.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ pod, namespace, and workload identity; the `file_storage` extension
checkpoints read offsets across restarts so log lines are not
re-shipped on rollouts. Replaces the in-tree `containerstdout`
receiver scheduled for deletion at v0.2.0 per
[RFC-0013 §migration PR-K](../rfcs/0013-distro-first-pivot.md#migration--rollout)
[RFC-0013 §migration PR-K](../rfcs/0013-distro-first-pivot.md#migration-rollout)
and §7 (Deletion list).

## Config
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/journald-kernel.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ journald and `/dev/kmsg` and normalizing the records through an OTTL
[RFC-0013 §3](../rfcs/0013-distro-first-pivot.md#3-customer-stable-telemetry-contracts)
so existing operator alerts survive the swap. Replaces the in-tree
`kernelevents` receiver scheduled for deletion at v0.2.0 per
[RFC-0013 §migration PR-K](../rfcs/0013-distro-first-pivot.md#migration--rollout)
[RFC-0013 §migration PR-K](../rfcs/0013-distro-first-pivot.md#migration-rollout)
and §7 (Deletion list).

## Config
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/k8sobjects-events.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Tracecore watches the Kubernetes Events API via the upstream
`node_pressure`, `image_pull_failure`) via an OTTL `transform`
processor. Replaces the in-tree `k8sevents` receiver scheduled for
deletion at v0.2.0 per
[RFC-0013 §migration PR-K](../rfcs/0013-distro-first-pivot.md#migration--rollout)
[RFC-0013 §migration PR-K](../rfcs/0013-distro-first-pivot.md#migration-rollout)
and §7 (Deletion list).

> **Validation note.** The upstream `k8sobjectsreceiver` calls
Expand Down
4 changes: 2 additions & 2 deletions docs/migration/v0.1-to-v0.2.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Migration: v0.1.x → v0.2.0

This guide tells operators how to move from a `v0.1.x` deployment to `v0.2.0`. Every operator-visible break gets a row; everything not listed below is unchanged. Sections below mirror [RFC-0013 §migration](../rfcs/0013-distro-first-pivot.md#migration--rollout) (PR-A through PR-L).
This guide tells operators how to move from a `v0.1.x` deployment to `v0.2.0`. Every operator-visible break gets a row; everything not listed below is unchanged. Sections below mirror [RFC-0013 §migration](../rfcs/0013-distro-first-pivot.md#migration-rollout) (PR-A through PR-L).

## TL;DR

Expand Down Expand Up @@ -79,7 +79,7 @@ To verify what's actually registered in the binary you're running:
./_build/tracecore components
```

The chart does not yet ship a per-receiver `recipe:` switch — that mechanism arrives in [RFC-0013 PR-J](../rfcs/0013-distro-first-pivot.md#migration--rollout) along with the upstream-recipe templates. Until PR-J lands, the migration path for the in-tree receivers other than `clockreceiver`/`stdoutexporter` is: pin v0.1.x → wait for PR-J → cut over to the upstream-recipe values shape in one minor.
The chart does not yet ship a per-receiver `recipe:` switch — that mechanism arrives in [RFC-0013 PR-J](../rfcs/0013-distro-first-pivot.md#migration-rollout) along with the upstream-recipe templates. Until PR-J lands, the migration path for the in-tree receivers other than `clockreceiver`/`stdoutexporter` is: pin v0.1.x → wait for PR-J → cut over to the upstream-recipe values shape in one minor.

## Self-telemetry metric vocabulary

Expand Down
2 changes: 1 addition & 1 deletion docs/migration/v0.2-to-v0.3.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ helm upgrade tracecore install/kubernetes/tracecore \

- [#222: PR-M deferral memo](https://github.com/TraceCoreAI/tracecore/issues/222) — current PR-M status + re-evaluation triggers (OTel Profiles → Beta, parca-agent OTLP export)
- [RFC-0013 §Adoption matrix](../rfcs/0013-distro-first-pivot.md#2-adoption-matrix) — why pyspy is on the eventual deletion path in favour of parca-agent (note: timing in the RFC predates the #222 deferral)
- [RFC-0013 §Migration / rollout](../rfcs/0013-distro-first-pivot.md#migration--rollout) — original PR-M and PR-N sequencing (supersede with #222 for current timeline)
- [RFC-0013 §Migration / rollout](../rfcs/0013-distro-first-pivot.md#migration-rollout) — original PR-M and PR-N sequencing (supersede with #222 for current timeline)
- [RFC-0009 §Safety properties](../rfcs/0009-pyspy-receiver-scope.md#proposal) — design record of the cooperative receiver's zero-capability posture (still in force at v0.3.0)
- [`components/receivers/pyspy/README.md`](../../components/receivers/pyspy/README.md) — cooperative receiver's user-facing docs (the receiver ships in v0.3.0)
- [`components/receivers/pyspy/RUNBOOK.md`](../../components/receivers/pyspy/RUNBOOK.md) — per-kind operator triage for the cooperative receiver
Expand Down
2 changes: 1 addition & 1 deletion docs/notes/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ load-bearing lessons. Newest-first.

The release + integration workflow set changes across the v0.1.0 →
v0.3.0 migration window per
[RFC-0013 §Migration](../rfcs/0013-distro-first-pivot.md#migration--rollout)
[RFC-0013 §Migration](../rfcs/0013-distro-first-pivot.md#migration-rollout)
and §7 (Deletion list). Concrete schedule:

- **v0.1.0** - `.github/workflows/release.yml` is rewritten on top
Expand Down
2 changes: 1 addition & 1 deletion docs/reproducibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Verify a published `tracecore` release end-to-end from source. The
release pipeline lives in
[`.github/workflows/release.yml`](../.github/workflows/release.yml)
(`package` job per [RFC-0013 §Migration PR-C](rfcs/0013-distro-first-pivot.md#migration--rollout)):
(`package` job per [RFC-0013 §Migration PR-C](rfcs/0013-distro-first-pivot.md#migration-rollout)):
inline shell builds each `linux/{amd64,arm64}` OCB binary via
`make build`, archives it with `tar --sort=name
--owner=0 --group=0 --numeric-owner --mtime=@$SOURCE_DATE_EPOCH | gzip -n`
Expand Down
8 changes: 4 additions & 4 deletions module/receiver/ncclfrreceiver/RUNBOOK.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# nccl_fr receiver — runbook

Per-Kind incident-response notes. Each section corresponds to one value of `kind` in `tracecore_receiver_errors_total{component="nccl_fr",kind=…}`.
Per-Kind incident-response notes. Each section corresponds to one value of `kind` in `otelcol_receiver_ncclfr_errors_total{component_id=~"ncclfr/.*",kind=…}` (v0.2.0 self-telemetry name; see [migration self-tel vocabulary](../../../docs/migration/v0.1-to-v0.2.md#self-telemetry-metric-vocabulary)).

Cross-links: [`README.md`](README.md) for config + emitted records (including the threat-model boundary around the pickle parser), [`docs/FAILURE-MODES.md`](../../../docs/FAILURE-MODES.md) for runtime-level modes.

Expand All @@ -20,7 +20,7 @@ The receiver tails `dump_dir` for PyTorch NCCL FlightRecorder pickle files. Erro
2. Check kubelet logs around the failure window for volume events.
3. If size-bound: `ls -laS <dump_dir>` to identify the oversize file.

**Remediation.** Restore directory access OR raise `max_file_bytes` for legitimate large dumps. Sustained `enumerate` errors past `degraded_failure_ratio` flip the receiver degraded; alert on `tracecore_receiver_degraded_seconds_total` rising.
**Remediation.** Restore directory access OR raise `max_file_bytes` for legitimate large dumps. Sustained `enumerate` errors past `degraded_failure_ratio` flip the receiver degraded; alert on `otelcol_receiver_ncclfr_degraded_seconds_total` rising.

## kind=read

Expand Down Expand Up @@ -62,7 +62,7 @@ The receiver tails `dump_dir` for PyTorch NCCL FlightRecorder pickle files. Erro

**Likely causes.** Exporter backpressure, OTLP endpoint down, schema mismatch downstream.

**Investigation.** Inspect the exporter's `tracecore_exporter_calls_total{result="fail"}` counter and its own runbook.
**Investigation.** Inspect the exporter's `otelcol_exporter_<name>_calls_total{result="error"}` counter (e.g. `otelcol_exporter_otlphttp_calls_total`) and its own runbook.

**Remediation.** Resolve downstream; the receiver does not retry but will succeed on the next file.

Expand All @@ -72,6 +72,6 @@ The receiver tails `dump_dir` for PyTorch NCCL FlightRecorder pickle files. Erro

**Likely causes.** Bug in `module/pkg/nccl/fr_parser`, malformed input edge case, OTel SDK panic.

**Investigation.** Pod logs include the panic trace at ERROR via the lifecycle's recover. The receiver flips degraded on every panic — `tracecore_receiver_degraded_seconds_total{receiver="nccl_fr"}` will be climbing.
**Investigation.** Pod logs include the panic trace at ERROR via the lifecycle's recover. The receiver flips degraded on every panic — `otelcol_receiver_ncclfr_degraded_seconds_total{component_id=~"ncclfr/.*"}` will be climbing.

**Remediation.** File a tracecore bug with the panic trace and (if possible) the file that triggered it. The receiver stays alive; subsequent files are still processed, but the degraded gauge stays elevated until the binary restarts.