diff --git a/MILESTONES.md b/MILESTONES.md index c75c8075..01f562ed 100644 --- a/MILESTONES.md +++ b/MILESTONES.md @@ -26,8 +26,6 @@ Optional fields where they apply: **Reference** (RFC link), **Landed** (one-line A milestone ships when every rubric is provably satisfied. -The Foundation entries (M1, M2, M4, M9) predate this convention — they ship as prose summaries without inlined rubrics; backfilling them is a deferred cleanup, not a blocker for new milestones. - **M7 is absent by design** — OTel `gen_ai.training.*` semconv work lives in `open-telemetry/semantic-conventions`, not this repo (recurring cadence in NORTHSTARS.md O4). ## Keeping this document current @@ -136,6 +134,7 @@ Critical path to v0.1.0; the only lane in which a single milestone (M21) gates e - **Status:** ☑ delivered (PR #29) - **Depends on:** M1 - **Landed:** Chart at `install/kubernetes/tracecore/`; CI workflow `.github/workflows/chart.yml`; bundled `conftest` policy under `policies/conftest/`. +- **Carry-forward:** Hero-KPI 10-run aggregation — `chart.yml` measures install-to-Ready on a single CI run (≤300s gate); cross-run median across 10 runs is a measurement-harness extension, not a chart deficiency. **Functional rubrics:** - ☑ Chart at `install/kubernetes/tracecore/` installs via `helm install tracecore ./install/kubernetes/tracecore` against a kind/minikube cluster in CI; install returns exit 0 and `helm status` reports `STATUS: deployed`. @@ -149,7 +148,7 @@ Critical path to v0.1.0; the only lane in which a single milestone (M21) gates e - ☑ Rendered pod spec passes the Kubernetes `restricted` Pod Security Standard except for explicit `SYS_PTRACE` and the host-path mounts required by receivers; deviation list is enumerated in the chart README with a one-line justification per item. (per https://kubernetes.io/docs/concepts/security/pod-security-standards/) - ☑ DaemonSet template sets `securityContext.runAsNonRoot: true`, a non-zero `runAsUser`, `seccompProfile.type: RuntimeDefault`, `allowPrivilegeEscalation: false`; CI asserts each field via `yq`/grep gate. (per NORTHSTARS O2) - ☑ `Chart.yaml` declares `apiVersion: v2`, a SemVer `version`, and an `appVersion` matching the tracecore binary tag; CI gate fails on drift. (per PRINCIPLES §15) -- ☑ `helm install` plus DaemonSet `Ready` on a single-node kind cluster completes in ≤5 min median across 10 CI runs. (per NORTHSTARS O2 hero-KPI) +- ⧗ `helm install` plus DaemonSet `Ready` on a single-node kind cluster completes in ≤5 min median across 10 CI runs. *(Single-run ≤300s gate in `chart.yml`; 10-run median aggregation is the carry-forward.)* (per NORTHSTARS O2 hero-KPI) ### M20. Reference-cluster install benchmark (staged) @@ -201,23 +200,23 @@ M20a/b/c are gates against the same artifact (`bench/install/run.sh`) at progres ### M4b. Failure-injection harness -- **Status:** ☑ delivered (PR #30) +- **Status:** ☑ partial (PR #30 — xid / cpu-steal / pod-evict shipped; nccl-hang CLI integration is a stub returning `ErrPending` per `tools/failure-inject/ncclhang/`) - **Depends on:** none -- **Landed:** `tools/failure-inject/` (`xid`/`nccl-hang`/`pod-evict`/`cpu-steal`); `internal/pipeline/chaos_test.go`; `.github/workflows/chaos.yml`. -- **Carry-forward:** Matrix grows as patterns land — when M17/M18/M19 ship, add a `pattern:` matrix entry per pattern (per NORTHSTARS O1). +- **Landed:** `tools/failure-inject/` (`xid`/`pod-evict`/`cpu-steal`); `internal/pipeline/chaos_test.go`; `.github/workflows/chaos.yml`. +- **Carry-forward:** (1) `nccl-hang` CLI subcommand — wrap `pkg/nccl/fr_parser/synthesize.go` (capability already present from M11); (2) cross-arch SHA-256 equality check — `chaos.yml` builds on amd64 + arm64 but does not cross-compare the seeded outputs; (3) matrix grows as patterns land — when M17 / M18 / M19 ship, add a `pattern:` matrix entry per pattern (per NORTHSTARS O1). **Functional rubrics:** - ☑ `failure-inject xid` emits a line matching the kernelevents parser regex `NVRM: Xid \(PCI:[0-9a-fA-F:.]+\): (\d+)` for codes 13, 31, 43, 48, 63, 64, 74, 79, 94, 95; round-trip through `components/receivers/kernelevents/parser.go` populates `kernelevents.xid` and `gpu.id`. (per RFC-0007 Inventions table row 3 — NVRM/Xid regex) - ☑ `xid` output is valid for both kmsg and journald sinks: harness writes either a `/dev/kmsg`-shaped record or `journalctl --output=json` JSON; both ingested by the kernelevents receiver without `Degraded()` flipping. (per RFC-0007 §Design overview — kmsg + journald sources) -- ☑ `failure-inject nccl-hang` produces bytes decodable by `pkg/nccl/fr_parser/`; `synthesize → parse → re-synthesize` is byte-identical. -- ☑ `nccl-hang` pickle stream contains only safe opcodes (dict, list, tuple, int, str, bytes, None, refs); no `REDUCE`, `BUILD`, `GLOBAL`, or `INST` opcodes appear. (per PRINCIPLES §9) +- ⧗ `failure-inject nccl-hang` produces bytes decodable by `pkg/nccl/fr_parser/`; `synthesize → parse → re-synthesize` is byte-identical. *(CLI shim is a stub returning `ErrPending` per `tools/failure-inject/ncclhang/ncclhang.go`; the underlying round-trip exists at `pkg/nccl/fr_parser/synthesize.go` from M11.)* +- ⧗ `nccl-hang` pickle stream contains only safe opcodes (dict, list, tuple, int, str, bytes, None, refs); no `REDUCE`, `BUILD`, `GLOBAL`, or `INST` opcodes appear. *(Underlying `pkg/nccl/fr_parser` enforces this default-deny set; the CLI wrapper that exposes it ships with carry-forward item #1.)* (per PRINCIPLES §9) - ☑ `failure-inject pod-evict` creates a real k8s `Event` with `Reason=Evicted` against the in-cluster ServiceAccount or `--kubeconfig`; event is observable via `kubectl get events -o json` within 5 seconds *(unverified — requires live cluster)*. - ☑ `failure-inject cpu-steal` pins a busy-loop to `--core N` for `--duration D`; `mpstat -P N 1` reports `%steal+%user ≥ 95%` for at least `D-1` seconds; process exits 0. (per NORTHSTARS Appendix A pattern #6) - ☑ `internal/pipeline/chaos_test.go` under `-tags=chaos` pairs panic-or-error receiver with panic-or-error exporter; runtime stays alive for ≥100 iterations without leaking goroutines. (per PRINCIPLES §1) - ☑ `.github/workflows/chaos.yml` runs nightly with a matrix entry per landed pattern (M17 / M18 / M19); each entry invokes the relevant `failure-inject` subcommand and asserts the corresponding pattern emits ≥1 match. Matrix grows as patterns land — M4b ships without forward-referencing unbuilt patterns. (per NORTHSTARS O1) **Non-functional rubrics:** -- ☑ Determinism: same `argv` + `--seed` produces byte-identical stdout/file across two runs on both `linux/amd64` and `linux/arm64`; SHA-256 equality enforced in CI matrix. (per PRINCIPLES §12) +- ⧗ Determinism: same `argv` + `--seed` produces byte-identical stdout/file across two runs on both `linux/amd64` and `linux/arm64`; SHA-256 equality enforced in CI matrix. *(Single-arch SHA gate active in `chaos.yml` via `tools/failure-inject/testdata/golden.sha256`; cross-arch equality check is carry-forward item #2.)* (per PRINCIPLES §12) - ☑ Isolation: harness writes only to stdout or `--out`; never touches `/dev/kmsg`, journald, or kube-apiserver unless the corresponding subcommand is explicitly invoked. (per PRINCIPLES §1) - ☑ `pod-evict` requires `--allow-cluster-write` or non-default `--kubeconfig`; without either, dry-runs and exits 0. (per PRINCIPLES §9) - ☑ Fixture-stability gate: SHA-256 of `failure-inject xid --code 79 --seed 0` pinned in `tools/failure-inject/testdata/golden.sha256`; CI fails on drift. (per PRINCIPLES §12) @@ -313,6 +312,7 @@ Alpha unified-source logs receiver covering L2 + L9 (kernel + system events). Ta - **Status:** ☑ alpha (PR #32) - **Depends on:** M1 - **Landed:** `components/receivers/k8sevents/` (25 files, ~3.1k LOC); RBAC ClusterRole + `kubectl auth can-i` golden; cluster-singleton Deployment manifest. +- **Carry-forward:** End-to-end overhead measurement at 1k events/min asserting CPU + egress + RSS budgets together (per-component benches exist; integrated run is the open work). **Functional rubrics:** - ☑ Watches `events.k8s.io/v1` via SharedInformer with resync ≥10 min; emits one `plog.LogRecord` per Event with `event.uid`, `event.action`, `event.reason`, `regarding.{kind,namespace,name,uid}`, `reporting.controller`, `note`, `series.count`, `event_time` populated; verified by integration test against a fake `kube-apiserver`. @@ -340,7 +340,7 @@ Alpha unified-source logs receiver covering L2 + L9 (kernel + system events). Ta - ☑ Degraded mode: informer `WatchErrorHandler` increments `IncError("watch")`, sets `Degraded()=true`, backs off (1s/2s/5s, ceiling 30s); receiver stays alive without restart; trips `K8sEventsReceiverDegraded` alert; `FAILURE-MODES.md` row exists referencing the test. **Non-functional rubrics:** -- ☑ Overhead at 1k events/min steady-state: ≤0.02% CPU, ≤0.02 Mbps egress, ≤10 MB RSS. (per NORTHSTARS O2 "k8s events / kube-state" row) +- ⧗ Overhead at 1k events/min steady-state: ≤0.02% CPU, ≤0.02 Mbps egress, ≤10 MB RSS. *(`bench_test.go` provides `BenchmarkEmitOne` + `BenchmarkConvertOne` and `rusage_linux_test.go` measures CPU; an end-to-end run at 1k events/min that asserts all three budgets together is the carry-forward.)* (per NORTHSTARS O2 "k8s events / kube-state" row) - ☑ API-courtesy: `rest.Config` `QPS=5`, `Burst=10` pinned explicitly; single shared informer per process; resync ≥10 min; never issues `LIST` outside informer bootstrap. - ☑ Back-pressure: event-burst storms (≥10k events in 60s) MUST NOT block the informer; bounded internal channel (cap 1024); drops with `IncError("backpressure_drop")` past cap; `goleak` verifies no leak. - ☑ Multi-tenancy: optional `namespaces:` allowlist enforced server-side via FieldSelector when length=1; ≥2 namespaces fall back to in-process filter with documented egress cost. @@ -530,15 +530,15 @@ Lane 6 covers NVIDIA-side device telemetry (DCGM), NCCL collective diagnostics ( - **Depends on:** M1, M4b - **Hardware:** parser + fixtures: none (pure Go). E2E receiver validation against real NCCL dumps: Linux + NVIDIA GPU (flood-gated) — pending flood-gate open. - **Landed:** `pkg/nccl/fr_parser/` (parser + `synthesize.go` + version-tagged fixtures); `components/receivers/nccl_fr/`; registered via `components.yaml`. -- **Carry-forward:** E2E validation against real NCCL dumps (flood-gated); 2.31+ fixture additions land under the NORTHSTARS O6 ecosystem-change SLA. +- **Carry-forward:** (1) E2E validation against real NCCL dumps (flood-gated); (2) 2.31-drift fixture test (NORTHSTARS O6 ecosystem-change SLA covers the schedule); (3) 5s dump-watcher emit timing assertion (`receiver_test.go` exercises the watch path; explicit ≤5s p99 timing gate is the open work); (4) overhead bench gate — `bench/overhead/nccl_fr_bench_test.go` replays fixtures and measures HeapAlloc but is advisory, not a CI-gated assertion against the 0.01 / 0.5 Mbps + 30 MB targets; (5) `make generate-fixtures` byte-identical gate — `synthesize.go` encodes deterministically (sorted dict keys), an explicit pinned-seed regen + diff gate is not yet wired; (6) `failure-inject nccl-hang` CLI shim (M4b carry-forward #1). **Functional rubrics:** - ☑ Parser opcode whitelist enforced in `pkg/nccl/fr_parser/`. **Default-deny:** any opcode not in the whitelist returns `ErrUnknownOpcode` and aborts (protects against future protocol additions). Whitelist (protocol 0–5 safe subset NCCL FlightRecorder actually emits): `PROTO`, `FRAME`, `EMPTY_DICT/EMPTY_LIST/EMPTY_TUPLE/EMPTY_SET`, `MARK`, `DICT/LIST/TUPLE/TUPLE1/TUPLE2/TUPLE3`, `SETITEM(S)/APPEND(S)/ADDITEMS`, `FROZENSET`, `SHORT_BINUNICODE/BINUNICODE/BINUNICODE8`, `SHORT_BINBYTES/BINBYTES/BINBYTES8/BYTEARRAY8`, `BININT/BININT1/BININT2/LONG1/LONG4`, `NEWFALSE/NEWTRUE/NONE`, `MEMOIZE/BINGET/LONG_BINGET/BINPUT/LONG_BINPUT`, `POP/POP_MARK`, `STOP`. (per https://docs.python.org/3/library/pickletools.html — Python pickle protocol 0–5) Explicitly reject (return `ErrUnsafeOpcode`): `REDUCE`/`BUILD`/`INST`/`OBJ`/`NEWOBJ`/`NEWOBJ_EX`/`GLOBAL`/`STACK_GLOBAL`/`EXT1/2/4`/`PERSID`/`BINPERSID`/`NEXT_BUFFER`/`READONLY_BUFFER` (out-of-band buffers imply reducer machinery). - ☑ Decoded ring-buffer record exposes PyTorch FlightRecorder fields `record_id`, `pg_id`/`process_group`, `collective_seq_id`, `p2p_seq_id`, `op_id`, `profiling_name`, `state` (`scheduled|started|completed`), `time_created_ns`, `time_discovered_started_ns`, `time_discovered_completed_ns`, `duration_ms`, `input_sizes`, `input_dtypes`, `output_sizes`, `output_dtypes`, `frames`, `is_p2p`. (per `torch/csrc/distributed/c10d/FlightRecorder.hpp`) - ☑ Version-tagged fixture suite at `pkg/nccl/fr_parser/testdata/`: `nccl-2.29.x-healthy.pkl`, `nccl-2.29.x-hang.pkl`, `nccl-2.30.x-healthy.pkl`, `nccl-2.30.x-hang.pkl`, plus one for NCCL 2.30 `ncclCommGrow`/`ncclCommShrink` and one with `is_p2p=true`. Each has sibling `.golden.json`. -- ☑ Synthetic fixture generator `synthesize.go` round-trips: `Synthesize(spec) → bytes → Parse(bytes) == spec`. Same generator reachable from `cmd/tracecore failure-inject nccl-hang` (M4b). +- ⧗ Synthetic fixture generator `synthesize.go` round-trips: `Synthesize(spec) → bytes → Parse(bytes) == spec`. Same generator reachable from `cmd/tracecore failure-inject nccl-hang` (M4b). *(Package round-trip lands in `pkg/nccl/fr_parser/synthesize.go`; CLI reachability is blocked on M4b carry-forward #1.)* - ☑ Receiver emits one OTel log record per FR entry with `nccl.fr.pg_id`, `nccl.rank` (canonical join key — also surfaced as `nccl.fr.rank` alias for backward compat), `nccl.communicator`, `nccl.fr.collective_seq_id`, `nccl.fr.profiling_name`, `nccl.fr.state`, `nccl.fr.duration_ms`, `nccl.version`, plus resource attrs `hw.id`, `k8s.pod.name`/`k8s.pod.uid`. Schema validated by JSON-schema doc-lint. -- ☑ Dump-directory watcher: pickle written into `t.TempDir()` matching configured glob causes receiver to emit decoded records within 5s. Truncated pickle (last 32 bytes dropped) returns `ErrTruncated` without stalling. +- ⧗ Dump-directory watcher: pickle written into `t.TempDir()` matching configured glob causes receiver to emit decoded records within 5s. Truncated pickle (last 32 bytes dropped) returns `ErrTruncated` without stalling. *(Watcher + truncation handling are present and tested; an explicit ≤5s p99 emit-timing assertion is the carry-forward.)* - ☑ Cross-rank join NOT exposed publicly in M11. `pkg/nccl/fr_parser/cross_rank.go` does not exist at M11 tag; only per-rank parser API is exported. M17 lift is falsifiable. - ☑ **Factory registration:** `cmd/tracecore receivers list` reports `nccl_fr`; registered at `cmd/tracecore/components.go` and listed in `components.yaml`. - ☑ README per STYLE.md: stability badge `alpha`, config table, example YAML ≤20 lines, "Limitations" section names exactly which NCCL FR fields are decoded vs ignored. @@ -547,10 +547,10 @@ Lane 6 covers NVIDIA-side device telemetry (DCGM), NCCL collective diagnostics ( - ☑ Zero RCE surface: parser implemented without `os/exec`, `reflect.Call`/`reflect.MakeFunc`, `plugin`, dynamic symbol resolution. CI grep against the package fails the build on any of these. Parser depends only on `encoding/binary`, `errors`, `fmt`, `io`, stdlib container types — verified by `go list -deps`. (per PRINCIPLES §1, §9) - ☑ Fuzz coverage matches kernelevents precedent: `FuzzParseFRPickle` seeded with every `testdata/*.pkl` plus adversarial seeds (truncated, all-zero, all-`REDUCE`, deeply nested 2^16 lists, 4 GiB declared-length `BINBYTES8`). 30s fuzz in `make ci` (matches Go's official [`-fuzztime 30s` tutorial example](https://go.dev/doc/tutorial/fuzz)); nightly 10-min fuzz. Pass: no panic, no OOM, bounded allocation (≤2× input size), recursion ≤`MaxDepth` (default 256 — between protobuf C# default of 100 per [proto-limits](https://protobuf.dev/programming-guides/proto-limits/) and CPython `sys.getrecursionlimit()` default of 1000). - ☑ Resource bounds enforced: parser refuses inputs declaring container sizes >`MaxItems` (default 1<<20 ≈ 1M; derived from NORTHSTARS O2 RSS budget — no direct upstream precedent) or string/bytes lengths >`MaxBytes` (default 1<<26 = 64 MiB, matches protobuf parsed-message ceiling per [proto-limits](https://protobuf.dev/programming-guides/proto-limits/)) with `ErrLimitExceeded`. Test injects oversize headers; asserts no allocation past limit (`runtime.ReadMemStats` delta <64 KiB — bounds Go runtime small-allocation span). -- ☑ NCCL version drift: parser passes 2.29.x and 2.30.x fixture sets; synthetic "2.31-like" fixture succeeds and records unknown field in `extra`. Ecosystem-change SLA (NORTHSTARS O6, 30 days): adding 2.31 support requires only a new `testdata/` fixture — CI check diffs `parser.go` against M11 tag. -- ☑ Overhead within NORTHSTARS O2: idle ≤0.01 Mbps, dump-active ≤0.5 Mbps, RSS ≤30 MB. Measured by `bench/overhead/nccl_fr_bench_test.go` replaying 1 GB of fixtures. +- ⧗ NCCL version drift: parser passes 2.29.x and 2.30.x fixture sets; synthetic "2.31-like" fixture succeeds and records unknown field in `extra`. Ecosystem-change SLA (NORTHSTARS O6, 30 days): adding 2.31 support requires only a new `testdata/` fixture — CI check diffs `parser.go` against M11 tag. *(2.29.x and 2.30.x fixtures present; 2.31-drift fixture + parser-diff CI gate is the carry-forward.)* +- ⧗ Overhead within NORTHSTARS O2: idle ≤0.01 Mbps, dump-active ≤0.5 Mbps, RSS ≤30 MB. Measured by `bench/overhead/nccl_fr_bench_test.go` replaying 1 GB of fixtures. *(Bench replays fixtures and reports HeapAlloc deltas; the bench is advisory per its preamble — promoting it to a fail-closed CI gate against the three named targets is the carry-forward.)* - ☑ Trust posture: every parser error includes component (`nccl_fr.parser`), file path, byte offset, opcode name. Typed sentinels (`ErrUnsafeOpcode`, `ErrTruncated`, `ErrFieldMissing`, `ErrLimitExceeded`, `ErrVersionUnknown`) matched via `errors.Is`. (per PRINCIPLES §1, §9) -- ☑ Reproducibility: every fixture in `testdata/` regeneratable from `synthesize.go` with pinned seed; `make generate-fixtures` produces byte-identical output across two runs on linux/amd64. (per PRINCIPLES §12) +- ⧗ Reproducibility: every fixture in `testdata/` regeneratable from `synthesize.go` with pinned seed; `make generate-fixtures` produces byte-identical output across two runs on linux/amd64. *(`synthesize.go` encodes deterministically — dict keys sorted; an explicit pinned-seed regen + byte-identical diff gate in `make ci` is the carry-forward.)* (per PRINCIPLES §12) ### M12. NCCL Inspector receiver diff --git a/docs/FOLLOWUPS.md b/docs/FOLLOWUPS.md index 22eb98d7..ab4e4a63 100644 --- a/docs/FOLLOWUPS.md +++ b/docs/FOLLOWUPS.md @@ -670,6 +670,15 @@ M21-trigger or post-M3 follow-up cadence. - [ ] **`make doc-check`** — assert every test name referenced in `docs/**.md` actually exists. *Target:* opportunistic, ~30 minutes. Promoted to "Next up" above. +- [ ] **Backfill Foundation milestone rubrics (M1, M2, M4, M9).** + Those entries predate the per-rubric `☑` convention adopted in + PR #53 and ship as prose-only delivery summaries. Restore their + original functional + non-functional rubric blocks from the + respective RFCs (RFC-0003 for M1, RFC-0006 for M2, no RFC for + M4 lint harness, RFC-0007 for M9) with `☑` prefixes. *Target:* + opportunistic; the missing audit trail is mild because each is + already shipped, but consistency across the doc helps future + readers grep for "shipped under what rubric set". ## Considered and explicitly skipped