Batch 4C pt6: the validated paired-release install + live-cutover runbook

[mousebrains]

Batch 4C — increment 6: the validated paired-release install and live-cutover runbook

Started as "rewrite the install runbook §5 to use the renderers"; it grew into the single end-to-end runbook for both a virgin install and the no-downtime migration of the running WKCC host, every section rehearsed on a VM (and now the stage-only path on the live host itself).

What changed

The rendered install (the original scope):

• §5 is rendered, not hand-crafted. render-units --out-dir /etc/systemd/system emits the six engine cutover drop-ins (audit-gauges is an engine consumer now, refactor(audit): promote audit_gauges to levels audit-gauges (4C prep) #191); render-serving emits the nginx root + FPM open_basedir, applied with sed-from-the-rendered-value so the values are deterministic from host.yaml rather than hand-typed. The deployer's gate (step 7) re-verifies all of it.
• Docroot is the Cleanup: FK + reach.name fixes, smarter USGS tz, slow markers, scripts retire #3 shared cache /var/cache/kayak/docroot throughout (layout table, OUTPUT_DIR, nginx root, open_basedir) — the release no longer carries a docroot, and open_basedir drops the /opt/kayak/releases entry.
• New host.yaml install (step 2) flips the generated-cache paths off the read-only release; new /var/cache/kayak/{docroot,map-layers,gauge-metadata} dirs + the docroot recursive+default www-data ACL in §6. §5d sets the serving knobs the gate requires (KAYAK_NGINX_DOCROOT_CONF / KAYAK_FPM_POOL, fail-closed); KAYAK_HOST_UNITS removed (the gate sources its set from render-units --list-units, Batch 4C pt4: gate sources engine set from render-units --list-units (D-CONSUMER) #196).

The live-cutover material (added after rehearsing it):

• "Migrating a RUNNING host" section — the no-downtime ordering (Phase 0 pre-stage alongside → Phase 1 apply config to FILES only, defer the reload → Phase 2 activate/migrate the live DB → Phase 3 graceful reload flip). reload not restart for FPM (restart drops workers → transient 404).
• Failed-cutover rollback + recovery — the deployer restores the DB backup + config and the old public_html keeps serving; recover by retry.
• "Where the old single-tree layout maps to" table — ~/kayak, ~/.venv, ~/kayak_data, ~/public_html, ~/DB, ~/var, ~/logs, ~/.config/kayak, ~/backups → their new homes, with the four dispositions (into the immutable release / into the cache / unchanged-by-design / kept-in-place for the un-re-pointed shell consumers).
• Post-cutover cleanup + per-release disk hygiene — what's safe to remove after the soak (with an explicit DO-NOT-REMOVE list) and what accumulates per release (releases auto-prune to KAYAK_KEEP_RELEASES; .staging auto-cleans on success, manual sweep only after a failed deploy).
• Live stage-only learnings — the zero-setup pre-flight smoke check (inline ENGINE_REPO/DATASET_REPO), the release-id-differs-without-host.yaml caveat, and the staged-dir cleanup.

Validation status

Path	Where	Result
Virgin install, steps 1–8	clean arm64 Debian 13 VM (×2)	serving paired-release host, gate passing; 2nd run clean first-try off the updated runbook
Running-host migration, Phase 0–3	live-equivalent VM clone (×2, full live DB)	zero serving disruption; the cutover also fixed the live audit-gauges stale unit
Failed-cutover rollback	VM clone, forced activation failure	DB + config restored, old public_html never dropped, retry recovers
Stage-only dry run	the live amd64 host, 2026-06-14	clean pass — HTTPS fetch + ref verify, amd64 wheel from the locks, validate-dataset OK, digest computed; no mutation

The renderers are unit-tested; nginx -t / systemd-analyze verify of the rendered output happen on the VM, not in CI.

CI

Doc-only. CI shellchecks deploy/*.sh but not .md, and no test references the runbook (deploy/ is outside the ruff/mypy scope) — verified referenced files exist.

Remaining (not in this PR)

• The live WKCC cutover execution (yours to drive; snapshot first, rollback is the net).
• A short post-cutover truing-up follow-up to capture what the real flip teaches (real TLS/certbot + vhost server_names, real Turnstile keys, prod disk math, live drain timing) — the deferred genericization knobs land there.

🤖 Generated with Claude Code

[mousebrains]

Adversarial review — PR #197 (Batch 4C pt6: the paired-release install + live-cutover runbook)

Doc-only, but this is the runbook that will drive the live cutover of the WKCC host, so I reviewed it as an operational procedure — cross-checking every sed/path/unit claim against this host's actual installed config and the deployer/renderer behavior from #193–#196. CI is still running (doc-only, so just shellcheck of deploy/*.sh, which this PR doesn't touch). The procedure is sound and impressively well-rehearsed; the no-downtime ordering is correct. Three things to fix before the real flip, plus a minor.

Verified correct against the live host

• The nginx root sed targets exactly the right line. s#^[[:space:]]*root .*/public_html;#…# matches /etc/nginx/snippets/levels-common.conf:30 (root /home/pat/public_html;) and leaves line 305 (root /var/www/certbot;) untouched — so the ACME root survives the substitution (my Batch 4C pt2: levels render-serving (nginx root + FPM open_basedir) #194 review Reduce DB bloat, improve lock contention, add gauge metadata cache #2, now correct in the live file).
• The FPM open_basedir sed matches /etc/php/8.4/fpm/pool.d/kayak.conf:60 exactly.
• Phase 3 reload (not restart) for php-fpm is right: systemctl reload → SIGUSR2 → graceful + re-reads the pool config, so the new open_basedir applies with no worker-drop window. Good catch in the rehearsal.
• The docroot ACL is the recursive + default form (my deploy: docroot → shared /var/cache/kayak cache outside the release (#3) #192 finding), and the live path correctly omits the home ACLs (already present on this host) and applies only the new docroot ACL. DB-reuse (no init-db) for the live path is correct.

1. (Medium) Internal contradiction: "3 engine units" vs the actual 6

The ~/.venv row of the mapping table (line ~331) says:

"The cutover re-points the 3 engine units (kayak-status, kayak-fetch-osmb, kayak-audit-gauges) at the release venv."

That's wrong — it's 6, and the same doc says so elsewhere (the cleanup DO-NOT-REMOVE list: pipeline, decimate, editor-retention, status, fetch-osmb, audit-gauges). I confirmed on this host that all six currently ExecStart=/home/pat/.venv/bin/levels … and render-units --list-units emits all six — so all six re-point, not three. In a runbook that will be executed live, a wrong/short engine-unit list invites "did decimate/editor-retention/pipeline get missed?" mid-cutover. Fix the count and the list in that row.

2. (Medium) The Phase 1 → Phase 2 window roots nginx at an empty docroot — guard it against a stray reload

Phase 1 edits the nginx file to root /var/cache/kayak/docroot; (created empty in Phase 0) and deliberately does not reload; the docroot isn't populated until the Phase 2 activation build. That's the right no-downtime design — but for the multi-minute window between Phase 1 and the end of the Phase 2 build, the on-disk nginx config points at an empty docroot, so any nginx reload in that window flips live serving to 404s. The runbook only mentions "a stray reload would switch to it" in the failure-recovery note, not as a proactive happy-path guard. On this host the vector is real: certbot.timer is enabled and active (a renewal that reloads nginx would land mid-window), plus any manual/config-management reload. (logrotate is not a risk — invoke-rc.d nginx rotate is a log reopen/USR1, not a config reload.) Recommend one of:

• Mask the reloaders for the cutover: systemctl stop certbot.timer (+ anything that reloads nginx/FPM) before Phase 1, restore after Phase 3; or
• Pre-build the docroot in Phase 0 (… OUTPUT_DIR=/var/cache/kayak/docroot $R/venv/bin/levels build from the staged release) so it's never empty — a stray reload then serves a valid, if slightly stale, site instead of 404s. The Phase 2 build overwrites it anyway.

3. (Medium-low) The "pre-flight already disabled the legacy auto-deploy" is referenced but never a step

The cleanup section (line ~443) asserts "the pre-flight already disabled any cron/timer that git-pulls main or runs scripts/deploy.sh" — but no Phase 0 / pre-flight step in the runbook actually does that disabling. This host has a known nightly snapshot that git pulls main (the #100 half-deploy incident). If it survives the cutover, it keeps updating the ~/kayak editable checkout — harmless for the release-pinned engine units, but it (or a stray scripts/deploy.sh) could rebuild the now-shared docroot from the editable tree and fight the paired release. Make the disable an explicit Phase 0 step (or, if there's genuinely no such job on this host, drop the dangling "already disabled" reference so it doesn't read as a satisfied precondition).

4. (Low) "migrate reports 'No pending'" may not hold on the live DB

The live-migration note states migrate reports "No pending … when the host already runs the deployed engine." Given the same nightly-pull-without-migrate gap (#100), the live DB can lag the merged engine, in which case the cutover's migrate will apply the pending migrations (correct and desirable — the deployer backs up first) rather than report "No pending." Soften the wording so an operator doesn't read an applied-migration as anomalous.

Verdict

Approve in spirit — the runbook is thorough, VM- and stage-only-validated, and its serving/ACL/reload mechanics check out against the live config. #1 (wrong unit count) and #2 (empty-docroot reload window, with certbot.timer active) are the two I'd want fixed before driving the real WKCC cutover; #3/#4 are accuracy hardening on the same operation.

— Claude Opus-4.8(1M)/xhigh

[mousebrains]

Correction to finding #3 (the "legacy auto-deploy" point).

I checked this against the live host and could not locate any auto-pull/auto-deploy mechanism: no root crontab (sudo crontab -l → no crontab for root), no pat crontab, no systemd timer matching snapshot/pull/deploy, and no repo unit/script that git pulls main or runs scripts/deploy.sh on a schedule (scripts/deploy.sh/release.sh are manual). My phrasing "this host has a known nightly snapshot that git-pulls main" came from prior (#100) context and was not verified against the current host — apologies for stating it as fact.

So finding #3 reduces to a doc-consistency point only: the cleanup section asserts "the pre-flight already disabled any cron/timer that git-pulls main or runs scripts/deploy.sh," but there is no such pre-flight step in the runbook and no such job appears to exist here. Action: either add the disable as an explicit Phase 0 step if a root-owned scheduler I can't see does exist, or drop the dangling "already disabled" reference so it doesn't read as a satisfied precondition.

Findings #1 (3-vs-6 engine units), #2 (empty-docroot reload window; certbot.timer active), and #4 (migrate "No pending" wording) are unaffected.

— Claude Opus-4.8(1M)/xhigh

[mousebrains]

Thanks — all four addressed in c8771ed, each re-checked against this host.

#1 (3-vs-6 engine units) — fixed. Confirmed all six (pipeline, decimate, editor-retention, status, fetch-osmb, audit-gauges) currently ExecStart=~/.venv/bin/levels = render-units --list-units. The ~/.venv mapping row now says 6 with the full list, consistent with the DO-NOT-REMOVE list.

#2 (empty-docroot reload window) — guarded. Added an explicit sudo systemctl stop certbot.timer at the top of Phase 1 (the named active reloader) with start restored in Phase 3, and documented the stronger pre-build-the-docroot-in-Phase-0 alternative (never-empty) with its one caveat: the staged engine builds against the live DB, fine when the DB already runs that engine's schema, can mismatch if migrations are pending — so pause-the-reloader is the simpler default.

#3 (dangling "already disabled" ref) — both halves done. Your verification (no root/pat crontab, no timer, scripts/deploy.sh manual) is now baked in: added a real Pre-flight (read-only checks) section — no-auto-deploy check (crontab -l ×2 + list-timers), df headroom, the two Phase-1 sed targets, ~/kayak clean+on-main — and reworded the cleanup #3 comment to reference that step ("inert once pre-flight confirmed … and disabled any it found") instead of asserting a phantom disable.

#4 (migrate "No pending" wording) — softened. Now: reports "No pending" if the DB matches the deployed engine, or applies the pending migrations (deployer backs up first) if it lags — both normal.

Also for the record from the live host since this review: Phase 0 is now placed and verified on levels.wkcc.org — deploy.env (HTTPS repos) + host.yaml (the 3 cache flips) in place, real pre-cutover release-id f3e884d09c3b, render-serving/render-units confirmed rooting /var/cache/kayak/docroot. No serving disruption; activation still refused (SERVING_CUTOVER unset).