Skip to content

feat(sdk): publish Python + Go verdict-consumption SDKs#356

Merged
trilamsr merged 13 commits into
mainfrom
feat/sdk-verdict-v1-rc1
Jun 1, 2026
Merged

feat(sdk): publish Python + Go verdict-consumption SDKs#356
trilamsr merged 13 commits into
mainfrom
feat/sdk-verdict-v1-rc1

Conversation

@trilamsr

@trilamsr trilamsr commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes v1-rc1 cut criterion 12 — publishes the Python + Go client SDKs operators bind against to decode tracecore-emitted verdicts.

  • Go SDK at module/sdk/verdict/Decode(rawJSON []byte) (*Verdict, error) validates against the embedded v1.0-rc1 envelope schema and returns a typed Verdict with per-pattern extensions on Verdict.Extras.
  • Python SDK at python/tracecore_verdict/decode(raw_json: bytes | str) -> Verdict validates via jsonschema and returns a typed dataclass with per-pattern extensions on Verdict.extras. Pip-installable as tracecore-verdict.
  • Single source of truth: both SDKs ship a byte-identical copy of docs/schemas/verdict-1.0.0-rc1.json (Go via //go:embed, Python via package-data). A drift-detection test in each SDK fails the build if the copy strays from canonical.
  • CI gate: new sdk-python job wired into the verify aggregator; Go SDK already runs via verify-test through go.work resolution.
  • Docs: docs/sdk/README.md indexes both SDKs + version policy; cross-linked from docs/README.md; criterion 12 flipped from ☐ to ☑.

Install

# Python
pip install tracecore-verdict

# Go
go get github.com/tracecoreai/tracecore/module/sdk/verdict

Discipline

  • TDD: RED commits (test(sdk/verdict): RED ..., test(tracecore_verdict): RED ...) precede GREEN. Tests assert: happy-path decode, schema-invalid rejection (6 envelope guards), malformed JSON rejection, per-pattern extras preserved, all 8 shipped patterns round-trip, missing-layers populated.
  • Schema pin: per-SDK TestEmbeddedSchemaMatchesCanonical / test_embedded_schema_matches_canonical byte-compares the embedded schema against docs/schemas/verdict-1.0.0-rc1.json. Drift fails the build with a copy-paste fix hint.
  • Versioning: both SDKs target schema v1.0-rc1 only. A future envelope ships a new sibling package (module/sdk/verdict/v2, tracecore-verdict-v2) — never a silent breaking change.
### Added
- Python verdict-consumption SDK (`tracecore-verdict`, `pip install tracecore-verdict`) — typed dataclass + `decode()` for v1.0-rc1 envelopes. Closes v1-rc1 cut criterion 12 (Python half).
- Go verdict-consumption SDK (`module/sdk/verdict`, `go get github.com/tracecoreai/tracecore/module/sdk/verdict`) — typed `Verdict` + `Decode()` for v1.0-rc1 envelopes. Closes v1-rc1 cut criterion 12 (Go half).
- SDK landing page at `docs/sdk/README.md` with version policy + per-pattern field inventory.

### CI
- New `sdk-python` workflow job (pytest) wired into the `verify` aggregator.

Test plan

  • go test -race ./module/sdk/verdict/... clean
  • cd python/tracecore_verdict && pytest — 21/21 pass
  • cd python/tracecore_verdict && make test — pass
  • go vet ./... clean
  • make tidy-check / make license-check / make actionlint / make doc-check clean
  • make zizmor clean
  • CI green on PR

Tri Lam added 9 commits June 1, 2026 02:01
Adds verdict_envelope_schema_test.go pinning cut-criterion-2 contract
before the artifact lands: schema loadable from docs/schemas/verdict-
1.0.0-rc1.json, every shipped Verdict round-trips through it, ten
envelope-drift falsifiers reject, and $id matches the published CDN
URL contract for external SDK pinning.

Fails with file-not-found until the schema artifact is published.

Signed-off-by: Tri Lam <tri@maydow.com>
Publishes docs/schemas/verdict-1.0.0-rc1.json — the canonical Verdict
envelope every pattern detector emits. Per-pattern detectors layer
pattern-specific fields on top (gpu_id, xid_code, kind, …); per-pattern
testdata schemas pin those extensions. Envelope sets additionalProperties
true so per-pattern fields validate without re-listing here.

Closes v1-rc1 cut criterion 2:
- Schema at docs/schemas/verdict-1.0.0-rc1.json, draft 2020-12, $id
  https://schema.tracecore.io/verdict/1.0.0-rc1.json (matches the CDN
  contract third-party SDKs key off).
- Envelope-conformance + ten drift-falsifier rows already landed RED
  in the prior commit; this commit makes them GREEN.
- docs/schemas/README.md documents versioning + evolution policy and
  cross-links the forthcoming docs/DEPRECATION.md (criterion 4).
- docs/ATTRIBUTES.md cross-links the envelope from the
  pattern.verdict_json column.
- docs/v1-rc1-cut-criteria.md flips criterion 2 from planned to
  shipped with a Landed pointer.

Signed-off-by: Tri Lam <tri@maydow.com>
Signed-off-by: Tri Lam <tri@maydow.com>
Drives criterion 12 (verdict-consumption SDKs) for v1-rc1. Tests

fail with 'no non-test Go files' until verdict.go + decode.go land.

Signed-off-by: Tri Lam <tri@maydow.com>
Publishes module/sdk/verdict — the typed Go client for the v1.0-rc1

Verdict envelope. Decode validates against the embedded copy of

docs/schemas/verdict-1.0.0-rc1.json; a sync test pins byte-equality

with the canonical source. Per-pattern extensions ride through

Verdict.Extras (envelope additionalProperties:true).

Drives v1-rc1 cut criterion 12 (Go half).

Signed-off-by: Tri Lam <tri@maydow.com>
Mirrors the Go SDK's TDD discipline: every envelope guard +

additionalProperties+ shipped-pattern fixture is encoded here as a

pytest row. Fails with ModuleNotFoundError until __init__.py +

verdict.py + decode.py land. Drives criterion 12 (Python half).

Signed-off-by: Tri Lam <tri@maydow.com>
Publishes python/tracecore_verdict — the pip-installable Python

client for the v1.0-rc1 Verdict envelope. decode() validates

against the package-data copy of docs/schemas/verdict-1.0.0-rc1.json;

a sync test pins byte-equality with the canonical source.

Per-pattern extensions ride through Verdict.extras.

Drives v1-rc1 cut criterion 12 (Python half).

Signed-off-by: Tri Lam <tri@maydow.com>
Adds sdk-python CI job (pytest python/tracecore_verdict) wired into

the verify aggregator. Go SDK is already covered by verify-test

via go.work resolution. Publishes docs/sdk/README.md as the SDK

index + cross-links it from docs/README.md. Flips cut criterion 12

to shipped.

Signed-off-by: Tri Lam <tri@maydow.com>
@trilamsr

trilamsr commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Addressed reviewer findings:

🔴 Fixed (real bug): added hw_gpu_sdc + gen_ai_training_eval_accuracy evidence kinds to all three schema copies (canonical + Go SDK + Python SDK). SDC verdicts (pattern #13) now validate.

🟡 Fixed: drift tests now JSON-normalize (Go: json.Unmarshal + require.Equal; Python: json.loads + dict compare) — whitespace-only canonical reformatting won't false-positive.

🔵 Fixed: 9th SDC fixture added to both Go + Python round-trip suites. docs/sdk/README.md updated to '9 shipped pattern verdicts'.

Acknowledged: future patterns still require manual fixture sync across Go + Python — tracked as code-gen opportunity, not blocking rc1.

Tri Lam and others added 2 commits June 1, 2026 03:08
# Conflicts:
#	docs/schemas/README.md
#	docs/schemas/verdict-1.0.0-rc1.json
#	docs/v1-rc1-cut-criteria.md
#	module/pkg/patterns/verdict_envelope_schema_test.go
@trilamsr trilamsr merged commit 57ce53b into main Jun 1, 2026
12 checks passed
@trilamsr trilamsr deleted the feat/sdk-verdict-v1-rc1 branch June 1, 2026 20:21
trilamsr added a commit that referenced this pull request Jun 1, 2026
…398)

## Summary

Shipped-pattern verdict fixtures lived in three places —
`module/sdk/verdict/decode_test.go` (Go SDK),
`python/tracecore_verdict/test_decode.py` (Python SDK), and
`module/pkg/patterns/verdict_envelope_schema_test.go` (envelope schema
test) — each as a hand-rolled literal. Adding a pattern verdict at rc1
or v0.4 meant editing three places in lockstep; drift was inevitable
(and we've already been bitten by SDK-schema drift on the `$id` field).

Replace the three literals with **one canonical JSON file** at
`docs/schemas/fixtures/shipped-patterns-v1.0.0-rc1.json`. All three test
sites parametrise off it at test time. Adding a pattern verdict is now a
one-file edit — drift is impossible by construction.

## Root cause

PR #356 cut the v1.0-rc1 SDKs (criterion 12) by duplicating the
shipped-verdict fixture set across Go SDK + Python SDK + envelope schema
test. There was no canonical artifact for "the v1.0-rc1 shipped-verdict
set"; each test author rewrote the list. This PR creates that artifact
and converts each test site to a consumer.

## Why not code-gen?

The issue title says "code-gen" but the simpler primitive is
runtime-load: both languages parse JSON natively, so a generator step
adds zero safety beyond `os.ReadFile`/`Path.read_text`, and runtime-load
eliminates the "generated file out of sync with source" failure mode.
The existing `generate-fixtures` Make target codes-gen `.pkl` binary
fixtures (where parse-at-test-time wouldn't work); JSON does not need
that machinery.

## CI gate (`make verdict-fixtures-check`)

Wired into `make verify` (pre-push) and `make ci-full` (everything CI
runs). It asserts:

1. The canonical file exists at
`docs/schemas/fixtures/shipped-patterns-v1.0.0-rc1.json`.
2. Each of the three consumer test files still references the canonical
path by string — a reintroduced hand-rolled literal in any site trips
this guard.
3. Both SDK suites + the envelope-schema canonical-fixtures subtest
round-trip the fixture set green.

**Mutation-verified locally:** deleting the canonical file fails the
gate; dropping the reference from any consumer fails the gate.

The envelope test gains a new subtest
`TestVerdictEnvelopeV1RC1_CanonicalShippedFixturesValidate` that
validates every canonical fixture against the published envelope schema
— closing the loop so an envelope-non-conforming fixture would fail
before either SDK suite runs.

## Diff shape

Net: -204 deletions / +332 insertions (the canonical JSON file is +146;
the rest is the new envelope-validation subtest + Make target + README
rows; over 100 lines of duplicated fixture literals deleted).

## Test plan

- [x] `make verdict-fixtures-check` green
- [x] `make verdict-fixtures-check` red when canonical file deleted
(mutation test)
- [x] `make verdict-fixtures-check` red when any consumer drops the
reference (mutation test)
- [x] `cd module && GOWORK=off go test ./sdk/verdict/ ./pkg/patterns/
-count=1` green
- [x] `cd python/tracecore_verdict && python3 -m pytest .` green (22/22)
- [x] `golangci-lint run ./module/sdk/verdict/...
./module/pkg/patterns/...` clean on touched files
- [x] Pre-push hook (vet, mod-verify, attribute-namespace-check,
no-autoupdate-check) clean

```release-notes
sdk: shipped-pattern verdict fixtures consolidated to a single canonical file at `docs/schemas/fixtures/shipped-patterns-v1.0.0-rc1.json`; the Go SDK, Python SDK, and envelope schema tests now consume it directly. Adding a new pattern verdict is a one-file edit. `make verdict-fixtures-check` is the CI drift gate (issue #368).
```

Closes #368

Signed-off-by: Tri Lam <tri@maydow.com>
Co-authored-by: Tri Lam <tri@maydow.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant