test(sdk): single-source verdict fixtures across Go + Python (#368)#398
Merged
Conversation
Shipped-pattern verdict fixtures lived in three places — the Go SDK decode test, the Python SDK decode test, and the envelope schema test — and had to be hand-synced. Drift was inevitable. Replace the three literal fixture sets with one canonical JSON file at docs/schemas/fixtures/shipped-patterns-v1.0.0-rc1.json. All three test sites parametrise off it at test time. Adding a pattern verdict is now a one-file edit. New CI gate `make verdict-fixtures-check` asserts the canonical file exists, every consumer still references it by path, and all three test sites round-trip the fixture set green. Wired into `verify` and `ci-full`. Mutation-verified — deleting the canonical file fails the gate, and dropping a reference from any consumer fails the gate. Net diff: -204 / +176 (over 100 lines of duplicated fixture literals eliminated). Signed-off-by: Tri Lam <tri@maydow.com>
This was referenced Jun 1, 2026
trilamsr
added a commit
that referenced
this pull request
Jun 1, 2026
## Summary Audit issue #421 finding-2 root-fix. PR #398 landed the canonical v1.0-rc1 SDK fixture at `docs/schemas/fixtures/shipped-patterns-v1.0.0-rc1.json` with five rows carrying wrong `pattern.id` values — the fixture was hand-typed by sequential counter (..14, 15, 16, **17, 18, 19, 20, 21**) instead of being copied from the `PatternID*` constants in `module/pkg/patterns/*.go`. The envelope schema only constrains `pattern.id` to `type:string, minLength:1`, so per-pattern envelope tests round-tripped the drift silently. ### Before / after pattern.id | fixture row | before | canonical (`module/pkg/patterns/*.go`) | |---|---|---| | `hbm_ecc` | `"17"` | `PatternIDHBMECC = "3"` | | `thermal_throttle` | `"18"` | `PatternIDThermalThrottle = "4"` | | `pcie_aer` | `"19"` | `PatternIDPCIeAER = "5"` | | `cuda_oom` | `"20"` | `PatternIDCUDAOOM = "10"` | | `ib_link_flap` | `"21"` | `PatternIDIBLinkFlap = "2"` | `pod_evicted` ("14"), `nccl_hang` ("15"), `xid_correlation` ("16"), `silent_data_corruption` ("13") were already correct. ### Scoped-keys delta The `ib_link_flap` row also used unscoped `"node"` while `IBLinkFlapVerdict.Node` serializes as `"k8s.node.name"` (customer-stable namespace per `docs/ATTRIBUTES.md` line 152-154 + `docs/patterns/pattern-2-ib-link-flap.md`). Renamed in the fixture to mirror what the Go struct actually emits. The other unscoped keys on this row (`hca_device`, `port`, `transition_count`) match the Go struct's `json:"..."` tags as-is — those are the **envelope** key names; the operator-facing dashboard scoped names (`tracecore.alert.ib_link_flap.transition_count`, `hw.network.ib.device`, `hw.network.ib.port.num`) are the OTel log-record attributes the processor promotes alongside the JSON payload, a separate surface. ### Drift-prevention test `TestCanonicalShippedFixtures_PatternIDsMatchDetectorConsts` in `module/pkg/patterns/verdict_envelope_schema_test.go` pins each fixture row's `pattern.id` to the `PatternID*` const in the detector package and asserts symmetry (every const has a fixture, every fixture matches a const). Confirmed RED with the buggy fixture (5 failures matching the audit prediction exactly), GREEN after the fix. ```release-notes Correct `pattern.id` values for five rows in the v1.0-rc1 canonical SDK fixture and add a drift-prevention test pinning fixtures to detector-package constants. ``` Refs #421 Refs #398 ## Test plan - [x] `go test ./pkg/patterns/...` PASS (new test + existing envelope suite) - [x] `go test ./sdk/verdict/...` PASS (Go SDK round-trip) - [x] `python3 -m pytest python/tracecore_verdict/test_decode.py` 22/22 PASS - [x] `make vet` clean - [x] `make lint` 0 issues - [x] RED→GREEN cycle confirmed on `TestCanonicalShippedFixtures_PatternIDsMatchDetectorConsts`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Shipped-pattern verdict fixtures lived in three places —
module/sdk/verdict/decode_test.go(Go SDK),python/tracecore_verdict/test_decode.py(Python SDK), andmodule/pkg/patterns/verdict_envelope_schema_test.go(envelope schema test) — each as a hand-rolled literal. Adding a pattern verdict at rc1 or v0.4 meant editing three places in lockstep; drift was inevitable (and we've already been bitten by SDK-schema drift on the$idfield).Replace the three literals with one canonical JSON file at
docs/schemas/fixtures/shipped-patterns-v1.0.0-rc1.json. All three test sites parametrise off it at test time. Adding a pattern verdict is now a one-file edit — drift is impossible by construction.Root cause
PR #356 cut the v1.0-rc1 SDKs (criterion 12) by duplicating the shipped-verdict fixture set across Go SDK + Python SDK + envelope schema test. There was no canonical artifact for "the v1.0-rc1 shipped-verdict set"; each test author rewrote the list. This PR creates that artifact and converts each test site to a consumer.
Why not code-gen?
The issue title says "code-gen" but the simpler primitive is runtime-load: both languages parse JSON natively, so a generator step adds zero safety beyond
os.ReadFile/Path.read_text, and runtime-load eliminates the "generated file out of sync with source" failure mode. The existinggenerate-fixturesMake target codes-gen.pklbinary fixtures (where parse-at-test-time wouldn't work); JSON does not need that machinery.CI gate (
make verdict-fixtures-check)Wired into
make verify(pre-push) andmake ci-full(everything CI runs). It asserts:docs/schemas/fixtures/shipped-patterns-v1.0.0-rc1.json.Mutation-verified locally: deleting the canonical file fails the gate; dropping the reference from any consumer fails the gate.
The envelope test gains a new subtest
TestVerdictEnvelopeV1RC1_CanonicalShippedFixturesValidatethat validates every canonical fixture against the published envelope schema — closing the loop so an envelope-non-conforming fixture would fail before either SDK suite runs.Diff shape
Net: -204 deletions / +332 insertions (the canonical JSON file is +146; the rest is the new envelope-validation subtest + Make target + README rows; over 100 lines of duplicated fixture literals deleted).
Test plan
make verdict-fixtures-checkgreenmake verdict-fixtures-checkred when canonical file deleted (mutation test)make verdict-fixtures-checkred when any consumer drops the reference (mutation test)cd module && GOWORK=off go test ./sdk/verdict/ ./pkg/patterns/ -count=1greencd python/tracecore_verdict && python3 -m pytest .green (22/22)golangci-lint run ./module/sdk/verdict/... ./module/pkg/patterns/...clean on touched filesCloses #368