ci(policy-matrix): install ServiceMonitor CRD before helm dry-run (#494)

[trilamsr]

Summary

policy-matrix.yml workflow has been failing on every chart-touching PR
(blocked #476, #481, #493) since #475 landed. The chart's production
preset (values-production.yaml) flips serviceMonitor.enabled=true,
which renders a monitoring.coreos.com/v1 ServiceMonitor resource. Kind
clusters don't ship the prometheus-operator CRDs, so
helm install --dry-run=server exits 1 with:

no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first

Root cause

Kind ships only the core Kubernetes API set. monitoring.coreos.com/v1
is supplied by prometheus-operator, which the policy-matrix kind cluster
never installs. The chart's templates/servicemonitor.yaml is gated by
.Values.serviceMonitor.enabled — default false (chart stays
first-install-compatible on bare clusters), but the production preset
enables it (kube-prometheus-stack convention). The policy-matrix gate
exercises both default and production presets across PSA / Kyverno /
Gatekeeper, so the production rows hit the missing CRD on every run.

Fix

Issue #494 recommended option (a) — install the missing CRD prereq.
This PR adds a single workflow step after kind cluster spin-up but
before the smoke script:

- name: Install prometheus-operator ServiceMonitor CRD (issue #494)
  run: |
    kubectl apply -f \
      "https://github.com/prometheus-operator/prometheus-operator/v0.91.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml"
    kubectl wait --for=condition=established crd/servicemonitors.monitoring.coreos.com --timeout=60s

Design choices:

• Slim CRD (ServiceMonitor only) vs full prometheus-operator bundle.
  The chart's production preset references no other monitoring.coreos.com
  kinds. Slim install (~700 lines of YAML) avoids pulling Prometheus,
  Alertmanager, ThanosRuler, PodMonitor, Probe, PrometheusRule we don't
  exercise.
• Applied to every matrix row (not just production). A future flip of
  the default serviceMonitor.enabled toggle cannot silently re-break
  this gate.
• Pinned to v0.91.0 (latest stable, published 2026-05-05). Matches
  the existing KYVERNO_POLICIES_REF / GATEKEEPER_VERSION pin
  convention in scripts/policy-matrix-smoke.sh. Bumping is a reviewed
  code change — never tracks main.
• kubectl wait --for=condition=established before the helm dry-run
  so the apiserver has registered the CRD when the chart template
  reaches the admission chain (avoids a race where the dry-run hits
  before discovery refreshes).

Gatekeeper CRD timing

Re-audited — install_gatekeeper() in scripts/policy-matrix-smoke.sh
already polls kubectl get crd ...constraints.gatekeeper.sh (line 143-149)
and the constraint byPod[*].enforced field (line 270-276) before the
smoke step exits. The kubectl get constraints -A || true in the
failure-collection step is diagnostic only and already tolerates absent
CRDs. No timing fix needed there.

Why not install-bench / chart.yml

• install-bench.yml uses bench/install/tracecore-values.yaml which
  doesn't enable serviceMonitor — same failure shape doesn't apply.
• chart.yml's install and upgrade jobs install with default values
  (serviceMonitor.enabled=false); the render job's production-preset
  check is helm template only (no cluster), so no API discovery runs.

Test plan

• actionlint .github/workflows/policy-matrix.yml — exit 0
• actionlint across all .github/workflows/ — exit 0
• Pre-push hook suite passed locally (golangci-lint, vet, mod verify,
  attribute-namespace-check, zizmor, doc-check, alert-check,
  chart-appversion-check, rfc-status-check, slo-rules-check,
  deprecation-check, no-autoupdate-check, test-flake-audit)
• policy-matrix workflow runs green on this PR — all 6 matrix rows
  (psa × default, psa × production, kyverno × default, kyverno ×
  production, gatekeeper × default, gatekeeper × production) plus all 3
  mutation rows.

Closes #494.

ci: install prometheus-operator ServiceMonitor CRD in policy-matrix kind cluster so chart-touching PRs no longer fail on the production preset's ServiceMonitor render

[trilamsr]

Code Review — B/A/A+ Grading

VERDICT: A (Ship, no blocking changes)

Criteria Assessment

✓ Issue Closure: closes #494 definitively. ServiceMonitor CRD is now installed before helm dry-run validates the production-preset chart.

✓ Simplicity: 23-line addition; 2 operations (apply CRD, wait for condition). No conditional logic, no loops. Documentation is heavy but load-bearing (explains why slim CRD vs full prometheus-operator bundle).

✓ Placement: inserted after kubectl sanity check, before smoke install. Dependency order is correct.

✓ No Regressions: kubectl apply is idempotent; applying the CRD to all 6 matrix rows is safe. Isolated from chart, smoke script, other workflows.

✓ Race-free Wait: uses kubectl wait --for=condition=established crd/servicemonitors.monitoring.coreos.com --timeout=60s. This is the right primitive (not sleep). 60s timeout is conservative for kind.

✓ Version Pinning: v0.91.0 (released 2026-05-05), stable 4+ weeks. Follows repo convention (KYVERNO_POLICIES_REF / GATEKEEPER_VERSION). Not tracking main/latest.

✓ Scope Justification: applied unconditionally to all matrix rows (not just production), which is defensive — a future default-values flip won't silently re-break the gate. Install-bench exclusion is justified (separate workflow, doesn't call policy-matrix-smoke.sh, production preset is specialist criterion-10 gate).

Minor Note (Non-blocking)

If kubectl apply fails (bad URL, network timeout), the subsequent kubectl wait will timeout at 60s before the step fails. This is slow-fail but safe (the step fails either way). Could tighten error handling with set -e at the top of the run: block, but acceptable for B/A grade.

Simplification Sweep

• Comments: 21 lines of documentation. Necessary context for future maintainers understanding the CRD size trade-off. ✓ Keep.
• Commands: Multi-line apply with backslash continuation. Readable; no change needed.
• Polling pattern: Actually improves on existing gatekeeper polling (L139–149 in policy-matrix-smoke.sh); uses kubectl wait --for=condition=established instead of manual retry loop.

Ready to merge. Do not enable auto-merge (per task requirements).