From 9be2295f6c31953516438edc56c9ced302239d87 Mon Sep 17 00:00:00 2001 From: Tri Lam Date: Wed, 20 May 2026 16:45:10 -0700 Subject: [PATCH 1/6] [m3] container image publish to ghcr.io/tracecoreai/tracecore Closes the long-standing chart-default-image gap. The chart's install/kubernetes/tracecore/values.yaml has shipped with image.repository=ghcr.io/tracecoreai/tracecore as the default since M5b, but release.yml only ever published the binary + SBOM + cosign-bundle + provenance as GitHub Release artifacts. Operators following the chart's defaults could not pull. RFC-0008 names this as the target operator-pull path. Architecture: - New Dockerfile pinned to gcr.io/distroless/static-debian12:nonroot by digest (sha256:d093aa3e30...). Non-root UID 65532 matches the chart's runAsUser. CGO_ENABLED=0 binary means scratch was viable too, but distroless gives a working CA bundle for the otlphttp exporter's HTTPS path and tzdata for the binary's RFC3339 stamping with zero shell-attack surface. - The image consumes the pre-built reproducible binary from the build job (COPY release/$BINARY_BASENAME), not a recompile. Image reproducibility reduces to binary reproducibility (already gated) plus the digest-pinned base layer. release.yml `image` job: - needs: build (downloads binary artifact, verifies sha256 matches build.outputs.digest before push). - Builds with docker/build-push-action@v6.19.2, SOURCE_DATE_EPOCH threaded through so the COPY layer's mtime is deterministic. - Always tags :TAG. Floats :latest only on stable releases (no `-` in the SemVer pre-release field) so a pre-release does not silently promote alpha bits to the chart's default-pull surface. - cosign sign --yes "$IMAGE_REPO@$DIGEST": sign the manifest BY DIGEST, not tag. A registry rebuild of a floating tag would otherwise let an attacker replace what cosign verify resolves. - cosign verify smoke check pins the same identity binding the binary blob already uses (--certificate-github-workflow-ref refs/tags/$TAG, --trigger push). - attest-build-provenance with push-to-registry=true attaches the SLSA v1.0 provenance to the manifest in the registry, so a verifier pulls everything from one place via `gh attestation verify oci://`. Permissions: id-token: write, attestations: write, packages: write. No long-lived registry credentials (GHCR auth uses the workflow's GITHUB_TOKEN); no long-lived signing keys (cosign keyless via OIDC). Docs: - docs/reproducibility.md grows two steps (8: cosign verify image manifest by digest; 9: gh attestation verify oci://) with the same identity-binding flags as the binary-side steps. The release-doc-parity.sh gate scopes to the binary-side `gh attestation verify "$BINARY_BASENAME"` line specifically, so adding the image verifier does not break parity. - "What this verifies" / "What this does not verify" / "If a step fails" tables extended for steps 8-9. - install/kubernetes/tracecore/README.md "Pre-release note" replaced with the live-publish contract. Troubleshooting "ImagePullBackOff on first install" entry updated with the Dockerfile-based local-build workaround (was: "M3 release stream has not landed yet"). Followup + changelog: - docs/followups/M3.md "Container-image publish" item closed with the project's HTML-comment-above + struck-italic-line-below convention (mirrors the rows already closed in that shard). - CHANGELOG [Unreleased] ### Added gains an M3 entry. Verification: - `make ci` clean (golangci-lint, govulncheck, vet, mod-verify, RCE gate, register-lint, actionlint, zizmor, all unit/race tests). - `make doc-check` clean: 438 markdown links resolve, em-dash + en-dash diff gate clean, release-doc-parity green, chart-appversion green. - actionlint clean on the new release.yml job. - Cannot exercise the end-to-end push without a real tag push; workflow runs at next vX.Y.Z tag. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/release.yml | 131 ++++++++++++++++++++++++- CHANGELOG.md | 1 + Dockerfile | 33 +++++++ docs/followups/M3.md | 25 +++-- docs/reproducibility.md | 49 +++++++-- install/kubernetes/tracecore/README.md | 22 +++-- 6 files changed, 234 insertions(+), 27 deletions(-) create mode 100644 Dockerfile diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f888a762..22ef808a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -18,6 +18,10 @@ env: GO_VERSION_FILE: go.mod BINARY_BASENAME: tracecore_linux_amd64 CYCLONEDX_GOMOD_VERSION: v1.10.0 + # Image publish destination. Chart's values.yaml `image.repository` + # default points here; drift between the two is the contract this + # job upholds. Lowercase per OCI distribution spec. + IMAGE_REPO: ghcr.io/tracecoreai/tracecore jobs: build: @@ -443,10 +447,135 @@ jobs: if-no-files-found: error retention-days: 7 + image: + name: image (build + push + sign + attest) + runs-on: ubuntu-latest + needs: build + permissions: + contents: read + packages: write # push to ghcr.io/tracecoreai/tracecore + id-token: write # cosign keyless + attest-build-provenance OIDC + attestations: write # attest-build-provenance writes a bundle + outputs: + digest: ${{ steps.push.outputs.digest }} + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + # Pin by commit SHA so a force-push to the tag between the + # build and image jobs cannot produce an image manifest for + # a different tree than the signed binary it contains. + ref: ${{ github.sha }} + persist-credentials: false + + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: binary + path: release + + - name: Verify binary digest matches build job + env: + EXPECTED: ${{ needs.build.outputs.digest }} + run: | + set -euo pipefail + actual=$(sha256sum "release/$BINARY_BASENAME" | awk '{print $1}') + if [ "$actual" != "$EXPECTED" ]; then + echo "::error::artifact digest drift: build=$EXPECTED downloaded=$actual" + exit 1 + fi + + - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0 + + - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Compute image tags + id: meta + env: + TAG: ${{ needs.build.outputs.tag }} + IS_PRERELEASE: ${{ contains(needs.build.outputs.tag, '-') }} + run: | + set -euo pipefail + # Always tag with the release version. Only float `:latest` + # for stable releases (no `-` in the SemVer pre-release + # field); a pre-release that takes `:latest` would silently + # promote alpha bits to the chart's default-pull surface. + tags="$IMAGE_REPO:$TAG" + if [ "$IS_PRERELEASE" != "true" ]; then + tags="$tags"$'\n'"$IMAGE_REPO:latest" + fi + { + echo "tags<> "$GITHUB_OUTPUT" + + - name: Compute SOURCE_DATE_EPOCH for image layer + id: sde + env: + TAG: ${{ needs.build.outputs.tag }} + run: | + set -euo pipefail + # Same epoch as the binary build so the image layer carrying + # the binary has a deterministic mtime. Without this, two + # builds at the same SHA produce different image digests + # purely from `now()` in the COPY layer. + epoch=$(git log -1 --pretty=%ct "$TAG") + echo "epoch=$epoch" >> "$GITHUB_OUTPUT" + + - uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2 + id: push + with: + context: . + file: Dockerfile + platforms: linux/amd64 + push: true + tags: ${{ steps.meta.outputs.tags }} + build-args: | + BINARY_PATH=release/${{ env.BINARY_BASENAME }} + SOURCE_DATE_EPOCH=${{ steps.sde.outputs.epoch }} + provenance: false # we attest below with GitHub's reusable attester + sbom: false # SBOM ships as a release artifact, not a manifest sub-attestation + + - uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 + + - name: Sign image (keyless) + env: + DIGEST: ${{ steps.push.outputs.digest }} + run: | + set -euo pipefail + # Sign the manifest by digest, not tag — a registry rebuild + # of `:latest` would otherwise let an attacker replace what + # `cosign verify` resolves. The digest is the trust root. + cosign sign --yes "${IMAGE_REPO}@${DIGEST}" + + - name: Verify image signature smoke check + env: + DIGEST: ${{ steps.push.outputs.digest }} + IDENTITY_REGEXP: "^https://github.com/${{ github.repository }}/\\.github/workflows/release\\.yml@refs/tags/" + TAG: ${{ needs.build.outputs.tag }} + run: | + set -euo pipefail + # Same identity-binding pattern as the binary blob verify: + # pin to release.yml on a tag ref. Mismatch fails closed. + cosign verify "${IMAGE_REPO}@${DIGEST}" \ + --certificate-identity-regexp "$IDENTITY_REGEXP" \ + --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \ + --certificate-github-workflow-ref "refs/tags/$TAG" \ + --certificate-github-workflow-trigger 'push' + + - uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0 + with: + subject-name: ${{ env.IMAGE_REPO }} + subject-digest: ${{ steps.push.outputs.digest }} + push-to-registry: true + release: name: release runs-on: ubuntu-latest - needs: [build, sbom, sign, provenance] + needs: [build, sbom, sign, provenance, image] permissions: contents: write steps: diff --git a/CHANGELOG.md b/CHANGELOG.md index 89bf1fc8..4da7aa2e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ Pre-alpha. The CLI runs the M1 pipeline runtime end-to-end via factory-based ass ### Added +- **M3: container image publish to `ghcr.io/tracecoreai/tracecore`.** Every release tag now pushes a signed and attested container image alongside the existing binary + SBOM + cosign-bundle + provenance artifacts. The image is built from the same byte-reproducible binary the binary-side jobs sign (no in-image recompile), the manifest is cosigned by digest using the existing keyless OIDC identity binding, and a SLSA v1.0 provenance attestation is pushed to the registry via `attest-build-provenance push-to-registry=true`. Stable releases (no `-` in the SemVer pre-release field) also tag `:latest`; pre-releases do not. The Dockerfile pins `gcr.io/distroless/static-debian12:nonroot` by digest and runs as the distroless `nonroot` user (UID 65532) matching the chart's `runAsUser`. Verification walkthrough at [`docs/reproducibility.md`](docs/reproducibility.md) steps 8 and 9. Closes the long-standing chart-default-image gap (chart's `image.repository: ghcr.io/tracecoreai/tracecore` is now a live pull path, not a future one). - **M13 — pyspy receiver Phase 2 (alpha; wire protocol + Python helper)** — Closes the Phase 1 scaffold with an actively-emitting receiver. New Go units handle the UDS handshake, faulthandler parsing, fnv128a `stack.id` hashing, `plog.LogRecord` emission with RFC-0009's prescribed attribute set, and a single-goroutine trigger driving both full and main cadences through the half-duplex wire. Workload-side helper ships as `pip install tracecore-pyspy` (stdlib-only; seven-step shutdown per RFC §Helper lifecycle) with PyPI trusted-publisher OIDC + PEP 740 attestation publishing and typosquat-reservation stubs. End-to-end integration test spawns a real `python3` subprocess and round-trips a dump request to catch wire-protocol drift. Rank-in-hello-frame and per-window LRU dedup deferred to Phase 3 (see [`docs/FOLLOWUPS.md`](docs/FOLLOWUPS.md)). See [RFC-0009 §Phase deliverables](docs/rfcs/0009-pyspy-receiver-scope.md#phase-deliverables) and [`components/receivers/pyspy/README.md`](components/receivers/pyspy/README.md) for the operator-facing degraded-mode table. - **M13 — pyspy receiver scaffold (alpha; Phase 1)** — `components/receivers/pyspy/` ships the package skeleton, factory wired through `tools/components-gen`, config + Validate (operator-facing fields documented in [RFC-0009 §Config schema](docs/rfcs/0009-pyspy-receiver-scope.md#config-schema)), all 12 `IncError` kinds from RFC-0009 §Degraded modes declared in `kinds.go`, and the `target_not_attached` posture as the first observable Phase 1 behavior (empty `uds_dir` → single warning + `disabled_reason="target_not_attached"` self-metric + idle scan loop on a 30s retry cadence derived from kubelet liveness-probe grace window). CI gates landing alongside: `tools/pyspy-lint/` reads `go tool nm` output for the linked binary and rejects symbols whose names match `(?:^|[._/])Ptrace(?:$|[A-Z_])` / `(?:^|[._/])ptrace(?:$|[_])` / `process_vm_readv` (mutation-verify fixtures at `tools/pyspy-lint/testdata/{clean,violating}` so the gate is falsifiable on stdlib-only Go binaries where it would otherwise be vacuously true); chart-render `yq` step in `.github/workflows/chart.yml` asserts `capabilities.drop: [ALL]` + `capabilities.add: []` when `receivers.pyspy.enabled=true` (the receiver requires no capability addition because the helper walks frames in-target via faulthandler); frame-length-ceiling fuzz harness in `fuzz_test.go` plus a deliberate oversize-input mutation-verify test asserts `readFrame` returns `ErrFrameTooLarge` without allocating the 2³¹-1-byte payload; Linux-only `strace`-asserted integration test in `integration_linux_test.go` plus a falsifier counterpart under build tag `pyspy_strace_falsifier` asserts `kill/ptrace/process_vm_readv` are not invoked while the receiver idles. Phase 1 receiver does NOT yet emit OTLP records — the trigger loop, UDS connect, parser, and `stack.id` hash are Phase 2 deliverables. See [RFC-0009](docs/rfcs/0009-pyspy-receiver-scope.md) and [`components/receivers/pyspy/README.md`](components/receivers/pyspy/README.md). - **`docs/proposals/gen-ai-training-semconv.md`** — Upstream proposal draft for the `gen_ai.training.*` semantic-conventions namespace M13 (and Lane 4/5/6 receivers) emit on. Closes the NORTHSTARS O4 first-draft-PR KPI gap; ready to copy-paste into an upstream PR after SIG attendance. Mirrors `docs/proposals/semconv-hw-gpu-extensions.md` shape: full attribute set with cardinality guidance, prior-art table (PyTorch DDP / Slurm / Ray / MLflow / W&B / Kueue / SageMaker / torchrun), cross-language SDK adoption checklist, OTel `transform` processor translation examples. diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 00000000..f0120303 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,33 @@ +# syntax=docker/dockerfile:1.7 +# +# Container image for the tracecore collector. This Dockerfile consumes +# a pre-built reproducible binary from `release/$BINARY_BASENAME`; it +# does NOT recompile. That is load-bearing: image reproducibility +# reduces to binary reproducibility (already gated in release.yml) plus +# the base layer pinned by digest below. +# +# Local build (CI calls the same): +# make build BIN=release/tracecore_linux_amd64 +# docker buildx build --platform=linux/amd64 \ +# --build-arg BINARY_PATH=release/tracecore_linux_amd64 \ +# -t tracecore:dev . +# +# The chart's pod spec assumes: +# * Non-root UID 65532 (distroless `nonroot`) +# * readOnlyRootFilesystem: true (tracecore writes nothing outside +# mounted volumes) +# * No shell, no package manager (distroless guarantees both) + +# Pinned by digest, not tag, so a registry rebuild of `:nonroot` cannot +# silently change what the chart's default install pulls. Refresh the +# digest when bumping; never use the floating tag. +FROM gcr.io/distroless/static-debian12:nonroot@sha256:d093aa3e30dbadd3efe1310db061a14da60299baff8450a17fe0ccc514a16639 + +ARG BINARY_PATH=release/tracecore_linux_amd64 + +COPY --chown=nonroot:nonroot ${BINARY_PATH} /usr/local/bin/tracecore + +USER nonroot:nonroot + +ENTRYPOINT ["/usr/local/bin/tracecore"] +CMD ["collect", "--config=/etc/tracecore/config.yaml"] diff --git a/docs/followups/M3.md b/docs/followups/M3.md index 3d476277..2dcd50e6 100644 --- a/docs/followups/M3.md +++ b/docs/followups/M3.md @@ -10,16 +10,21 @@ natural boundary (per MEMORY.md `feedback_narrow_pr_scope`). Mostly M21-trigger or post-M3 follow-up cadence. - [ ] **`chart-appversion` CI gate enforces only non-empty, not drift-against-binary-tag.** `Chart.yaml`'s leading comment promises the field is "Held in sync with the upstream binary by the `chart-appversion` CI gate." The current chart.yml workflow asserts `appVersion` is non-empty (line 154) but does not compare against the binary release tag. A future tightening: parse the binary tag from the most recent `gh release view` and assert `Chart.yaml.appVersion == binary_tag` (or matches `install/kubernetes/tracecore/.appversion-source`). *Trigger:* M21 release-tag prep (the gate becomes load-bearing when a real tag ships). -- [ ] **Container-image publish to `ghcr.io/tracecoreai/tracecore`.** - The chart's default `image.repository` - (`install/kubernetes/tracecore/values.yaml`) points at - `ghcr.io/tracecoreai/tracecore`, but `release.yml` today - publishes only the binary + SBOM + cosign bundle + SLSA - provenance as GitHub Release artifacts — no container image push. - Operators following the chart's defaults cannot pull yet. - RFC-0008 names this as the target operator-pull path; this - item closes the gap. *Trigger:* M21 v0.1.0 (release-tag-time - requirement) or first operator request, whichever comes first. + +- *Closed (see comment above): `ghcr.io/tracecoreai/tracecore:` + publishes on every release tag, signed and attested. The chart's + default `image.repository` is now a live pull path, not a future + one.* - [ ] **SLSA Build L3 via the reusable-workflow generator.** Replace the user-defined `release.yml` sign + attest steps with a call diff --git a/docs/reproducibility.md b/docs/reproducibility.md index 8acd4dcb..00a6d169 100644 --- a/docs/reproducibility.md +++ b/docs/reproducibility.md @@ -1,11 +1,13 @@ # Reproducibility -Verify a published `tracecore` release end-to-end from source. Four -artifacts ship with every release tag: the binary, a CycloneDX SBOM, a -keyless cosign signature bundle, and an SLSA v1.0 in-toto provenance -attestation. The commands below reproduce the build at the same git SHA, -diff it against the published binary, and verify each of the three -attestations independently. +Verify a published `tracecore` release end-to-end from source. Each +release tag ships four binary-side artifacts (the binary, a CycloneDX +SBOM, a keyless cosign signature bundle, and an SLSA v1.0 in-toto +provenance attestation) plus a signed and attested container image at +`ghcr.io/tracecoreai/tracecore:`. The commands below reproduce the +build at the same git SHA, diff it against the published binary, verify +each binary-side attestation, and verify the image's own signature and +provenance attestation. This walkthrough targets `linux/amd64`. `linux/arm64` is opt-in and follows the same sequence with `GOARCH=arm64`. See @@ -117,14 +119,43 @@ jq -e '.components | length > 0' "$SBOM" jq -r '.components[].purl' "$SBOM" | sort -u | head ``` +```bash +# 8. Resolve the published image digest and verify its cosign signature. +# Resolving by digest, not tag, is load-bearing: a registry rebuild +# of the floating tag would otherwise let an attacker replace what +# cosign verify sees. Same identity binding as step 5. +IMAGE_REPO=ghcr.io/tracecoreai/tracecore +DIGEST=$(cosign triangulate --type digest "$IMAGE_REPO:$TAG") +cosign verify "$DIGEST" \ + --certificate-identity-regexp "^https://github.com/${REPO}/\.github/workflows/release\.yml@refs/tags/" \ + --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \ + --certificate-github-workflow-ref "refs/tags/$TAG" \ + --certificate-github-workflow-trigger 'push' +``` + +```bash +# 9. Verify the image's SLSA v1.0 provenance attestation. The +# attestation is stored alongside the manifest in the registry +# (push-to-registry=true on attest-build-provenance), so gh +# attestation verify fetches it inline; no separate download. +gh attestation verify "oci://$DIGEST" \ + --repo "$REPO" \ + --signer-workflow "${REPO}/.github/workflows/release.yml" \ + --predicate-type 'https://slsa.dev/provenance/v1' \ + --source-ref "refs/tags/$TAG" \ + --source-digest "$(git -C src rev-parse HEAD)" +``` + ## What this verifies | Step | Guarantee | Source spec | |---|---|---| | 4 | Byte-identical rebuild at the same SHA | [PRINCIPLES.md §12](../PRINCIPLES.md) | -| 5 | Signature traces to a GitHub Actions OIDC identity, no long-lived key | [Sigstore keyless](https://docs.sigstore.dev/cosign/verifying/verify/) | -| 6 | Build provenance matches `predicateType: https://slsa.dev/provenance/v1`, signed by this repo's `release.yml` on a tag-ref | [SLSA v1.0 Build L1](https://slsa.dev/spec/v1.0/levels#build-l1) + [GitHub artifact attestations](https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations) | +| 5 | Binary signature traces to a GitHub Actions OIDC identity, no long-lived key | [Sigstore keyless](https://docs.sigstore.dev/cosign/verifying/verify/) | +| 6 | Binary provenance matches `predicateType: https://slsa.dev/provenance/v1`, signed by this repo's `release.yml` on a tag-ref | [SLSA v1.0 Build L1](https://slsa.dev/spec/v1.0/levels#build-l1) + [GitHub artifact attestations](https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations) | | 7 | SBOM coverage of every direct module | [CycloneDX spec](https://cyclonedx.org/specification/overview/) | +| 8 | Image manifest signature traces to the same OIDC identity as the binary | [Sigstore keyless](https://docs.sigstore.dev/cosign/verifying/verify/) | +| 9 | Image provenance matches `predicateType: https://slsa.dev/provenance/v1`, attestation stored in the OCI registry | [SLSA v1.0 Build L1](https://slsa.dev/spec/v1.0/levels#build-l1) | ## What this does not verify @@ -145,6 +176,8 @@ jq -r '.components[].purl' "$SBOM" | sort -u | head | 5 (cosign) | OIDC issuer rotation or revoked cert | Sigstore [transparency log](https://search.sigstore.dev/) entry for the bundle. | | 6 (`gh attestation verify`) | Workflow file altered after the tag, or signer-workflow / source-ref mismatch | `gh release view $TAG --json assets` against the cached provenance; inspect `.dsseEnvelope.payload | @base64d | fromjson .predicate.buildDefinition.externalParameters.workflow` from the bundle. | | 7 (jq) | SBOM truncation during upload | Re-download with `gh release download --clobber`; re-run from step 3. | +| 8 (cosign image) | Registry rebuild of the floating tag, or a different workflow ran on this tag | Resolve the digest explicitly with `crane digest $IMAGE_REPO:$TAG` and re-verify; check Sigstore transparency log for the image manifest. | +| 9 (`gh attestation verify oci://`) | Attestation not pushed to registry, or signer-workflow / source-ref mismatch | `cosign tree $IMAGE_REPO@$DIGEST` to list attestations; confirm the predicate-type DSSE envelope is present. | Reproducibility breakage is a P0 bug. File against [`SECURITY.md`](../SECURITY.md) if the failure is a signing-identity or diff --git a/install/kubernetes/tracecore/README.md b/install/kubernetes/tracecore/README.md index 23a68284..3d31d31b 100644 --- a/install/kubernetes/tracecore/README.md +++ b/install/kubernetes/tracecore/README.md @@ -69,14 +69,17 @@ path; for unattended updates, wire one of the delivery systems above. On failed upgrade, roll back via `helm rollback tracecore --namespace tracecore-system`; confirm the rollback landed by tailing pod logs (`kubectl logs -n tracecore-system -l app.kubernetes.io/name=tracecore`) and curling `/readyz` (per [RFC-0006 self-telemetry surface](../../../docs/rfcs/0006-self-telemetry-surface.md)) — `helm rollback` exits 0 on revision-revert even when the new revision's pod is still failing, so the post-rollback readiness check is the load-bearing signal. In a delivery system (Flux / Argo CD), gate promotion on the same `/readyz` so a failed rollout never auto-promotes. -Pre-release note: container image publication to -`ghcr.io/tracecoreai/tracecore` is tracked as an open follow-up; until -that workflow lands, see the **ImagePullBackOff on first install** -entry under § Troubleshooting below for the local-build workaround. +Container images publish to `ghcr.io/tracecoreai/tracecore:` on +every release tag. Each image carries a keyless cosign signature and a +SLSA v1.0 provenance attestation stored alongside the manifest in the +registry; verify both before deploying per +[`docs/reproducibility.md`](../../../docs/reproducibility.md) steps 8 +and 9. Stable releases (no `-` in the SemVer pre-release field) also +float `:latest`; pre-releases do not. The full rationale and the contract for what tracecore commits to (immutable digests, lockstep `appVersion`/binary) lives in -[RFC-0008 — auto-update boundary](../../../docs/rfcs/0008-auto-update-boundary.md). +[RFC-0008: auto-update boundary](../../../docs/rfcs/0008-auto-update-boundary.md). ## Uninstall @@ -225,9 +228,12 @@ grace (~45s) serializes the rollout. Override with (`ghcr.io/tracecoreai/tracecore`) is a public registry; air-gapped clusters must mirror the image to an internal registry and set `--set image.repository=/tracecore` (+ optional -`imagePullSecrets`). For pre-release builds, the M3 release stream -has not landed yet — use `kind load docker-image` against a local -build for evaluation clusters. +`imagePullSecrets`). For local evaluation against an unreleased SHA, +build the image with the in-tree `Dockerfile` and load it directly: +`make build && docker buildx build --platform=linux/amd64 +--build-arg BINARY_PATH=tracecore -t tracecore:dev . && kind load +docker-image tracecore:dev`; then install with +`--set image.repository=tracecore --set image.tag=dev`. **`helm upgrade --reuse-values` ignores a chart-level default I want.** `--reuse-values` is intentionally additive: a chart that added a new From 7578feb82b7f8a7c564353da690e99af10f09a39 Mon Sep 17 00:00:00 2001 From: Tri Lam Date: Wed, 20 May 2026 20:12:36 -0700 Subject: [PATCH 2/6] [m3] self-review fixes: cosign-triangulate, SOURCE_DATE_EPOCH, parity gate, force-push comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review of #145 surfaced three correctness bugs and one missing anti-regression gate. All fixable now; fixing now beats deferring to the followup list. F3 — `cosign triangulate --type digest` is the wrong tool. `cosign triangulate` resolves the signature/attestation OCI reference for a subject, not the subject's own digest. Step 8 as written likely fails at M21 tag-push exercise. Replaced with `crane digest` (the canonical tag-to-digest resolver) and added `crane` to the prerequisites list. The "If a step fails" row for step 8 already mentioned `crane digest` as a fallback; it is now the primary. F5 — SOURCE_DATE_EPOCH did not actually reach buildkit. Build-args not declared in the Dockerfile are silently ignored by buildkit, so the COPY layer's mtime was non-deterministic despite the intent. Two fixes layered: 1. `env: SOURCE_DATE_EPOCH:` on the build-push-action step so buildkit's layer-timestamp rewrite picks up the epoch from the build environment (buildkit >= 0.11 honors this). 2. `ARG SOURCE_DATE_EPOCH` declared in the Dockerfile so the determinism contract is visible to readers of the Dockerfile alone and so `docker buildx build --build-arg SOURCE_DATE_EPOCH=...` reproduces the CI image bit-for-bit. F1 — release-doc-parity.sh did not cover the image surface. The existing gate only compared the binary-side `gh attestation verify` flag set. Extended with a parallel block covering the image-side `cosign verify` flag set: the `image` job's smoke check vs `docs/reproducibility.md` step 8. Mutation-verified locally — breaking one flag in the workflow makes the gate exit non-zero. F4 — softened the force-push comment on the image-job checkout. The previous wording claimed the SHA pin alone prevents tree drift between `build` and `image`. In fact the pin guarantees the Dockerfile + workflow tree this job reads matches the commit that ran; the binary-digest guard (already present) catches the case of a binary built from a different tree than the Dockerfile read here. Together they close the force-push window. Rewrote to match the actual guarantee. Verification: - make doc-check clean (438 markdown links, em-dash + comment-noise gates, both release-doc-parity blocks, chart-appversion gate). - actionlint clean on release.yml. - release-doc-parity image block mutation-verified (rename one flag in the workflow -> gate fails). Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/release.yml | 19 ++++++++++++++--- Dockerfile | 9 ++++++++ docs/reproducibility.md | 10 ++++++--- scripts/release-doc-parity.sh | 40 +++++++++++++++++++++++++++++++++++ 4 files changed, 72 insertions(+), 6 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 22ef808a..59f71d11 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -461,9 +461,14 @@ jobs: steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: - # Pin by commit SHA so a force-push to the tag between the - # build and image jobs cannot produce an image manifest for - # a different tree than the signed binary it contains. + # Pin by commit SHA so the Dockerfile + workflow this job + # reads match the commit that ran. The binary downloaded + # below has its own digest guard (see "Verify binary digest + # matches build job"), which catches the case of a binary + # built from a different tree than the Dockerfile read here. + # Together these close the force-push window: a force-push + # to the tag between `build` and `image` cannot smuggle in + # either a different binary or a different Dockerfile. ref: ${{ github.sha }} persist-credentials: false @@ -527,6 +532,14 @@ jobs: - uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2 id: push + env: + # SOURCE_DATE_EPOCH must reach buildkit through the build + # environment (not just as a --build-arg) so buildkit's + # layer-timestamp rewrite kicks in. The Dockerfile also + # declares `ARG SOURCE_DATE_EPOCH` so the value is visible + # to a reader of the Dockerfile alone, and so a local + # `docker buildx build` reproduces the CI image bit-for-bit. + SOURCE_DATE_EPOCH: ${{ steps.sde.outputs.epoch }} with: context: . file: Dockerfile diff --git a/Dockerfile b/Dockerfile index f0120303..2c613c05 100644 --- a/Dockerfile +++ b/Dockerfile @@ -25,6 +25,15 @@ FROM gcr.io/distroless/static-debian12:nonroot@sha256:d093aa3e30dbadd3efe1310db0 ARG BINARY_PATH=release/tracecore_linux_amd64 +# Declared so buildkit's SOURCE_DATE_EPOCH layer-rewrite picks it up; +# buildkit >= 0.11 (shipped with docker/build-push-action@v6) rewrites +# COPY layer timestamps to this epoch when the env var is set on the +# build invocation. Declaring the ARG here makes the determinism +# contract visible to readers of the Dockerfile alone and lets a local +# `docker buildx build --build-arg SOURCE_DATE_EPOCH=...` reproduce +# the CI image bit-for-bit. +ARG SOURCE_DATE_EPOCH + COPY --chown=nonroot:nonroot ${BINARY_PATH} /usr/local/bin/tracecore USER nonroot:nonroot diff --git a/docs/reproducibility.md b/docs/reproducibility.md index 00a6d169..0ce19a69 100644 --- a/docs/reproducibility.md +++ b/docs/reproducibility.md @@ -18,6 +18,7 @@ this verifies. - Go ≥ the version pinned in [`.go-version`](../.go-version). - [`cosign`](https://docs.sigstore.dev/cosign/system_config/installation/) ≥ 2.2.0 (Sigstore keyless verification). +- [`crane`](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md) ≥ 0.19 (resolves a registry tag to its content-addressable digest for step 8). - [`diffoscope`](https://diffoscope.org/) (Linux: `apt-get install diffoscope`). - `gh` CLI ≥ 2.49.0, authenticated to GitHub (`gh attestation verify` plus asset download). - A POSIX shell — Linux or macOS. @@ -123,9 +124,12 @@ jq -r '.components[].purl' "$SBOM" | sort -u | head # 8. Resolve the published image digest and verify its cosign signature. # Resolving by digest, not tag, is load-bearing: a registry rebuild # of the floating tag would otherwise let an attacker replace what -# cosign verify sees. Same identity binding as step 5. +# cosign verify sees. `crane digest` is the canonical tag->digest +# resolver; do NOT use `cosign triangulate`, which resolves the +# signature reference for a subject, not the subject's own digest. +# Same identity binding as step 5. IMAGE_REPO=ghcr.io/tracecoreai/tracecore -DIGEST=$(cosign triangulate --type digest "$IMAGE_REPO:$TAG") +DIGEST="${IMAGE_REPO}@$(crane digest "$IMAGE_REPO:$TAG")" cosign verify "$DIGEST" \ --certificate-identity-regexp "^https://github.com/${REPO}/\.github/workflows/release\.yml@refs/tags/" \ --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \ @@ -176,7 +180,7 @@ gh attestation verify "oci://$DIGEST" \ | 5 (cosign) | OIDC issuer rotation or revoked cert | Sigstore [transparency log](https://search.sigstore.dev/) entry for the bundle. | | 6 (`gh attestation verify`) | Workflow file altered after the tag, or signer-workflow / source-ref mismatch | `gh release view $TAG --json assets` against the cached provenance; inspect `.dsseEnvelope.payload | @base64d | fromjson .predicate.buildDefinition.externalParameters.workflow` from the bundle. | | 7 (jq) | SBOM truncation during upload | Re-download with `gh release download --clobber`; re-run from step 3. | -| 8 (cosign image) | Registry rebuild of the floating tag, or a different workflow ran on this tag | Resolve the digest explicitly with `crane digest $IMAGE_REPO:$TAG` and re-verify; check Sigstore transparency log for the image manifest. | +| 8 (cosign image) | Registry rebuild of the floating tag, or a different workflow ran on this tag | Re-confirm the digest with `crane digest $IMAGE_REPO:$TAG`; cross-check against the Sigstore transparency log entry for the image manifest. | | 9 (`gh attestation verify oci://`) | Attestation not pushed to registry, or signer-workflow / source-ref mismatch | `cosign tree $IMAGE_REPO@$DIGEST` to list attestations; confirm the predicate-type DSSE envelope is present. | Reproducibility breakage is a P0 bug. File against diff --git a/scripts/release-doc-parity.sh b/scripts/release-doc-parity.sh index c93479da..ac30ab0d 100755 --- a/scripts/release-doc-parity.sh +++ b/scripts/release-doc-parity.sh @@ -54,3 +54,43 @@ if [ "$ci_flags" != "$doc_flags" ]; then fi echo "release-doc-parity: gh attestation verify flag set matches across $WORKFLOW and $DOC" + +# --------------------------------------------------------------------- +# Image-side parity: the `image` job's `cosign verify` smoke check vs +# docs/reproducibility.md step 8. Same drift class, same fix shape. +# The CI line resolves the image by `"${IMAGE_REPO}@${DIGEST}"`; the +# doc line resolves by `"$DIGEST"` (where DIGEST already includes the +# `repo@sha256:...` form via `crane digest`). Flag NAMES still match. +# --------------------------------------------------------------------- + +ci_image_flags=$(awk ' + /cosign verify "\$\{IMAGE_REPO\}@\$\{DIGEST\}" \\$/ { capture=1 } + capture { print; if (!/\\$/) capture=0 } +' "$WORKFLOW" | grep -oE '\-\-[a-z][a-z0-9-]+' | sort -u) + +doc_image_flags=$(awk ' + /^cosign verify "\$DIGEST" \\$/ { capture=1 } + capture { print; if (!/\\$/) capture=0 } +' "$DOC" | grep -oE '\-\-[a-z][a-z0-9-]+' | sort -u) + +if [ -z "$ci_image_flags" ]; then + echo "release-doc-parity: failed to extract image-side cosign verify flags from $WORKFLOW" >&2 + exit 1 +fi +if [ -z "$doc_image_flags" ]; then + echo "release-doc-parity: failed to extract image-side cosign verify flags from $DOC step 8" >&2 + exit 1 +fi + +if [ "$ci_image_flags" != "$doc_image_flags" ]; then + echo "release-doc-parity: image-side cosign verify flag set diverges between $WORKFLOW and $DOC" >&2 + echo "$WORKFLOW (image smoke check):" >&2 + printf ' %s\n' $ci_image_flags >&2 + echo "$DOC (step 8):" >&2 + printf ' %s\n' $doc_image_flags >&2 + echo "" >&2 + echo "Fix: either widen the image smoke check or tighten the step-8 walkthrough so adopters exercise the same identity binding CI enforces on the image manifest." >&2 + exit 1 +fi + +echo "release-doc-parity: image cosign verify flag set matches across $WORKFLOW and $DOC" From 7034e1a5134c78852c4eafc07f5579740ef06abe Mon Sep 17 00:00:00 2001 From: Tri Lam Date: Wed, 20 May 2026 20:36:33 -0700 Subject: [PATCH 3/6] [m3] bridge residual risks (R1/R2/R3 + gate-the-gate fixture) Self-review surfaced three residual risks (R1 gh CLI semantic drift, R2 distroless base digest rotation, A++ #1 gate-the-gate). All three are bridged before merge instead of carried forward as followups. Also drops the section-banner the comment-noise gate flagged in the prior commit's parity-script edit. R1 - gh attestation verify flag-shape regression lint. New scripts/gh-attestation-flag-lint.sh parses `gh attestation verify --help` and asserts every long flag we use in release.yml + docs/reproducibility.md is still recognised by the installed gh CLI. Catches the failure mode "gh renamed --signer-workflow between point releases and our published verifier walkthrough is now broken." Skips quietly when gh is not installed (local dev); CI ubuntu-latest always has it. Mutation-verified. Wired into `make doc-check`. R2 - Distroless base digest rotation gate. New scripts/base-digest-check.sh calls `crane digest gcr.io/distroless/static-debian12:nonroot` and compares against the Dockerfile pin. Two modes: --warn (default, exits 0) for periodic cadences and --strict (exits non-zero) wired into the M21 release-prep flow via the new `make base-digest-check` target. Deliberately NOT in doc-check: requires outbound network to gcr.io and the pin legitimately lags between rotations. M3 follow-up shard gains a row describing the cadence. A++ #1 - Gate-the-gate fixture for release-doc-parity.sh. scripts/testdata/release-doc-parity/{intact,drift-binary,drift-image} fixtures exercise both parity blocks. scripts/test-release-doc-parity.sh drives them and asserts the expected exit codes. release-doc-parity.sh now accepts WORKFLOW/DOC env overrides (production paths unchanged). Mutation-verified: breaking the image-side awk anchor in the gate makes the intact fixture fail, driver catches it. R3 - timeout-minutes on release.yml jobs: out of scope (no per-job timeouts exist anywhere in release.yml today). Documented as a M3 follow-up with concrete per-job minute suggestions. Items impossible to accomplish locally are listed in docs/followups/ M3.md under a new "Items impossible to accomplish locally" section: end-to-end image-push smoke, real-attestation oci:// verifier exercise, and two-build image-digest equality. All three only become exercisable at M21 v0.1.0 tag-push time. Banner-comment cleanup: removed the section banner from release-doc-parity.sh that the comment-noise gate flagged in 7578feb. Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: Tri Lam --- Makefile | 7 +- docs/followups/M3.md | 64 ++++++++++++++ scripts/base-digest-check.sh | 83 +++++++++++++++++++ scripts/gh-attestation-flag-lint.sh | 77 +++++++++++++++++ scripts/release-doc-parity.sh | 9 +- scripts/test-release-doc-parity.sh | 43 ++++++++++ scripts/testdata/release-doc-parity/README.md | 36 ++++++++ .../drift-binary/release.yml | 17 ++++ .../drift-binary/reproducibility.md | 18 ++++ .../drift-image/release.yml | 19 +++++ .../drift-image/reproducibility.md | 18 ++++ .../release-doc-parity/intact/release.yml | 18 ++++ .../intact/reproducibility.md | 18 ++++ 13 files changed, 422 insertions(+), 5 deletions(-) create mode 100755 scripts/base-digest-check.sh create mode 100755 scripts/gh-attestation-flag-lint.sh create mode 100755 scripts/test-release-doc-parity.sh create mode 100644 scripts/testdata/release-doc-parity/README.md create mode 100644 scripts/testdata/release-doc-parity/drift-binary/release.yml create mode 100644 scripts/testdata/release-doc-parity/drift-binary/reproducibility.md create mode 100644 scripts/testdata/release-doc-parity/drift-image/release.yml create mode 100644 scripts/testdata/release-doc-parity/drift-image/reproducibility.md create mode 100644 scripts/testdata/release-doc-parity/intact/release.yml create mode 100644 scripts/testdata/release-doc-parity/intact/reproducibility.md diff --git a/Makefile b/Makefile index f00b071c..ef6aa072 100644 --- a/Makefile +++ b/Makefile @@ -224,12 +224,17 @@ test-extras: ## Run all test-suite extras NOT in `make ci`. Independent sub-tar done; \ exit $$status -doc-check: ## Verify test identifiers referenced in rot-prone docs exist in the source tree, AND alert names in component RUNBOOKs match the alerts.yaml, AND release.yml + docs/reproducibility.md share the same `gh attestation verify` flag set, AND Chart.yaml appVersion tracks internal/version.Version. +doc-check: ## Verify test identifiers referenced in rot-prone docs exist in the source tree, AND alert names in component RUNBOOKs match the alerts.yaml, AND release.yml + docs/reproducibility.md share the same `gh attestation verify` flag set, AND each `gh attestation verify` flag we use is still recognised by the installed `gh` CLI, AND Chart.yaml appVersion tracks internal/version.Version. @scripts/doc-check.sh @scripts/alert-check.sh @scripts/release-doc-parity.sh + @scripts/test-release-doc-parity.sh + @scripts/gh-attestation-flag-lint.sh @scripts/chart-appversion-check.sh +base-digest-check: ## Compare the Dockerfile pin for gcr.io/distroless/static-debian12:nonroot against the live registry digest. Strict mode (exits non-zero on drift) is for the M21 release-prep checklist; warn mode is for periodic invocation. + @scripts/base-digest-check.sh --strict + register-lint: ## Verify `func Register*` symbols live only under components/** (or an explicit allowlist). Enforces STRATEGY.md §"Each component owns its own Factory var". @scripts/register-lint.sh diff --git a/docs/followups/M3.md b/docs/followups/M3.md index 2dcd50e6..294a12b0 100644 --- a/docs/followups/M3.md +++ b/docs/followups/M3.md @@ -43,6 +43,70 @@ the verifier walkthrough; install/kubernetes/tracecore/README.md's is otherwise exercised only at tag time, which means a toolchain-drift regression goes unnoticed until the next release. *Trigger:* first published v0.1.x. +- [ ] **Distroless base digest rotation cadence.** `scripts/base-digest-check.sh` + lands with this PR (compares the Dockerfile pin for + `gcr.io/distroless/static-debian12:nonroot` against the live + registry digest). `make base-digest-check` invokes it in + `--strict` mode; M21 release-prep should run that target before + cutting `v0.1.0` and re-pin if drift is reported. Past v0.1.0, + wire the same script into a nightly cron alongside the drift + cron above so a stale base layer surfaces between releases, not + at release-prep time. *Trigger:* M21 release-prep checklist + lands as a documented sequence. +- [ ] **`timeout-minutes` on `release.yml` jobs.** None of the six + jobs in `release.yml` set `timeout-minutes`; GitHub's default + (360m / 6h) lets a wedged push or hung verify burn a full runner + hour. Cross-cutting fix — set sensible per-job timeouts (10m + for `build`/`sbom`/`sign`/`provenance`, 20m for `image`, 5m + for `release`). Left out of this PR because adding only to the + new `image` job would be inconsistent. *Trigger:* opportunistic; + pick up the next time `release.yml` is touched for unrelated + reasons. + +## Items impossible to accomplish locally — only verifiable on a real tag push + +These are not "deferred" in the sense of "we could but chose not to": +the tracecore project today has no infrastructure to exercise them +short of a real `vX.Y.Z` tag push. Listed here so M21 release-prep +includes them in its dry-run pass and so a future contributor does +not file a "missing test" issue assuming the gap is oversight. + +- [ ] **End-to-end image push smoke against `ghcr.io/tracecoreai/tracecore`.** + The `image` job in `release.yml` only fires on a `vX.Y.Z` tag + push, so the push + tag-compute + cosign sign + cosign verify + + attest-build-provenance chain is unexercised locally. The + mitigations in place: (a) `actionlint` on `release.yml`, + (b) `release-doc-parity.sh` image block (mutation-verified), + (c) `gh-attestation-flag-lint.sh` (catches gh CLI flag rename), + (d) the binary-digest guard in the `image` job. None of those + mitigations cover the registry-side semantics. *Trigger:* first + `vX.Y.Z` tag push (M21 v0.1.0 or any pre-release tag). +- [ ] **`gh attestation verify "oci://$DIGEST"` against a real + attestation in shape this pipeline emits.** No public OCI image + carries a GitHub Actions provenance attestation in the exact + shape `attest-build-provenance push-to-registry=true` writes + against `release.yml`, so the verifier walkthrough in + `docs/reproducibility.md` step 9 cannot be smoke-tested + end-to-end before M21. `gh-attestation-flag-lint.sh` partially + covers this by asserting each flag is still recognised by the + CLI; the residual risk is a *semantic* change to an existing + flag (e.g., `--source-ref` accepting a different format) that + no flag-name lint would catch. *Trigger:* same as above — + first tag push. +- [ ] **Two-build digest equality for the image.** The + `SOURCE_DATE_EPOCH` plumbing through buildkit's layer-rewrite + claims image reproducibility, but the claim is only verifiable + by building the image twice at the same SHA and diff'ing the + resulting manifest digests. Doing that locally requires a + working `docker buildx` (which the local dev environment + currently lacks; see "needs prod data" cross-shard for the + broader build-env gap). The CI runner has buildx — a second + build step in `release.yml` would close this gap inline, but + doubling the runner time at every tag push is a tradeoff worth + revisiting after the first real release confirms image stability + under churn. *Trigger:* M21 v0.1.0 ships and image-rebuild + drift becomes the next-most-likely supply-chain regression + class. +- *Closed (see comment above): all six jobs in `release.yml` carry + `timeout-minutes` caps; default 6h ceiling no longer applies.* + +- *Closed (see comment above): `cosign verify-attestation` smoke check + shipped in `image` job; image provenance attestation is now CI-verified + by predicate type + by identity binding, not just pushed and trusted.* + +## Out of scope for M3, tracked here for future milestones + +Each item below is an explicit non-goal of M3 (not a "we forgot"), held +out because the deliverable for M3 was *signed and attested image +publish on every release tag*, not *every supply-chain hardening that +references the published image*. Rowed here so a future audit can find +them without reading commit archaeology. + +- [ ] **Multi-arch image build (`linux/arm64` alongside `linux/amd64`).** + Today `release.yml` builds `linux/amd64` only; the binary half of + the pipeline matches (`GOARCH: amd64`). A multi-arch image would + need a buildx matrix + a manifest list, *and* a matching arm64 + binary build in the `build` job (same `SOURCE_DATE_EPOCH` / + reproducible-build guarantees on both architectures). The chart + defaults to amd64 nodes; demand-driven, not foundational. + *Trigger:* first user request for arm64 (Apple-silicon dev cluster + or AWS Graviton deploy) OR M5 cross-arch certification. +- [ ] **Container vulnerability scan gate (trivy / grype) in `release.yml`.** + A distroless base + a single Go binary minimizes the CVE surface, + but "minimal" is not "zero" — a future Go toolchain CVE or a + glibc/libssl CVE in the base layer would land silently today. + Wire `aquasecurity/trivy-action` or `anchore/scan-action` into + the `image` job at `--severity=CRITICAL,HIGH` failing-closed, with + `.trivyignore` for documented-and-accepted findings. *Trigger:* + first reported CVE against a published image OR M21 supply-chain + audit asks for it. +- [ ] **Image SBOM (syft / cyclonedx) attached as a manifest sub-attestation.** + Today the binary CycloneDX SBOM (`cyclonedx-gomod mod`) ships as + a release artifact but is not attached to the image manifest. A + pull-time verifier (admission controller, signature-aware + registry) can't cross-reference image-digest → SBOM without an + out-of-band lookup. Wire `anchore/sbom-action` or `cyclonedx` + with `--upload` to push an SBOM attestation by digest alongside + the provenance attestation. *Trigger:* same as the vuln-scan + gate; usually requested together. ## Items impossible to accomplish locally — only verifiable on a real tag push