ci: parallelize tests, isolating the slow deploy tests (~10min → ~6min)

[mousebrains]

Speed up CI: parallelize the suite, isolating + uninstrumenting the heavy deploy tests

CI's test job grew ~5 min (#190) → ~10–11 min as test_kayak_deploy.py's slow tests went 3 → 7 (the 4C deploy PRs #192–#196 added the rollback / serving-gate / quiesce activation tests). Each runs the real kayak-deploy.sh — builds an engine wheel + pip install --require-hashes the 30-package prod lock into a fresh venv — ~135 s/test, all serial.

Investigation (measured)

• One slow test = 142 s cold, 135 s warm → work-bound (venv + 30-wheel install + build + engine import), not download-bound. So no prebuilt-wheel seam, no pip pre-warm.
• The 7 deploy tests are independent; slow-only -n auto ran 2.6× faster (366 s vs ~945 s serial), all passing.

Three things, found the hard way

1. Latent test-hermeticity bug. _run passed {**os.environ, ...} to the deploy subprocess, so under -n a concurrent test's leaked DATASET_DIR/METADATA_DIR reached the staged engine's kayak.config import, which raises ValueError on the mismatch. (The harness comment even hinted: "drop a dev box's leaked METADATA_DIR" — but left it per-test.) _run now starts from a hermetic base — the inherited env minus kayak's config vars (_KAYAK_CONFIG_ENV). A real robustness win regardless of speed.

2. Isolation — separate process AND path-scoped collection. The run is split into two -n auto passes: -m 'not slow' (light) and the deploy file (slow). The slow pass is pytest tests/test_scripts/test_kayak_deploy.py -m slow — path-scoped on purpose: collecting the whole suite imports other test modules whose import-time os.environ side effects leak into the staged-engine subprocess (a plain pytest -m slow fails the activation tests; the path-scoped form passes). A new guard test (test_slow_tests_colocated.py) asserts all slow markers live in that file, so a future slow test elsewhere fails loudly instead of silently skipping CI.

3. Coverage instrumentation breaks the activation tests. Under --cov, pytest-cov's subprocess hooks make the activation tests (which spawn the staged engine repeatedly) fail; without --cov they pass. And --cov=kayak can't instrument those subprocesses anyway (separately-built staged venv), so the deploy tests contribute ~0 coverage (light-alone = full-run 83%). So the light suite carries the coverage gate and the slow pass runs uninstrumented.

Changes (test + CI only — no deployer/production code)

• _run: scrub kayak config env from the deploy subprocess (hermeticity fix).
• New test_slow_tests_colocated.py guard.
• pytest-xdist dev dep; CI runs two -n auto passes — light (gate) + path-scoped slow (no --cov).
• cache: pip on the test job (minor — saves the ~7 s/test download portion).

Validated locally

• Light suite -n auto + gate: 83.10% coverage (same as the full run — confirms the deploy tests add ~0), 1771 passed, 61 s.
• Slow suite -n auto (no cov): 7 passed.
• uv lock --check + both lock drift checks pass; ruff + shellcheck clean.

Expected CI: ~10 min → ~5–6 min (light ~1–1.5 min ∥, slow 7 tests over 4 cores ≈ 2 rounds).

🤖 Generated with Claude Code