Batch 4C pt1: HostConfig renderer fields + levels render-units (systemd cutover drop-ins)

docs/PLAN_4c_renderers.md

@@ -0,0 +1,171 @@
+# PLAN — Batch 4C: host-config renderers + the paired-release cutover
+
+> Status: **drafting** (2026-06-14). Branch `b4c-renderers` off `main`.
+> Prereqs done: PR #190 (deployer, merged), #191 (`levels audit-gauges`, merged),
+> #192 (#3 docroot → `/var/cache/kayak/docroot`, merging 2026-06-14 AM).
+> This plan finishes Batch 4 — it replaces the hand-crafted step 5 of
+> `deploy/INSTALL-paired-release.md` (branch `b4c-paired-install`) with renderers
+> driven by `kayak.host.HostConfig`, adds the deployer's serving-path verification,
+> and flips the live host to `SERVING_CUTOVER=yes`.
+
+## Goal
+
+The deployer (`kayak-deploy.sh`) refuses to activate until the host is fully cut
+over to the `/opt/kayak/current` paired-release layout: nginx `root`, FPM
+`open_basedir`, and every `levels`-running consumer unit must point at
+`current`/the shared docroot. Today those artifacts are **hand-crafted** (runbook
+step 5). 4C makes them **rendered from `host.yaml`** so the cutover is
+reproducible and verifiable, then performs the cutover on the live host.
+
+## Consumer classification (verified against `systemd/` on `main`, 2026-06-14)
+
+The cutover only re-points units that execute **engine code from the venv**. The
+`levels audit-gauges` promotion (#191) moved audit-gauges from the
+"host-level/source-script" column into the engine column — the runbook §5c
+(which predates #191) lists only 5; it is now **6**.
+
+**A. Engine consumers — re-point `ExecStart` → `/opt/kayak/current/venv/bin/levels`,
+fix `DATASET_DIR`/`OUTPUT_DIR`/`WorkingDirectory`/`ReadWritePaths`:**
+`kayak-pipeline`, `kayak-decimate`, `kayak-editor-retention`, `kayak-fetch-osmb`,
+`kayak-status`, **`kayak-audit-gauges`** (new, via #191).
+
+**B. Checkout-script consumers — run `/home/<svc>/kayak/{scripts,systemd}/*.sh`,
+none of which invoke `levels`.** They keep running from the repo checkout (which
+still exists on the live host), so the cutover does **not** re-point them. Two of
+them touch the DB and so must still be in the deployer's quiesce set
+(`KAYAK_UNITS`) even though they're not re-pointed (descriptions verified against
+the live scripts — PR #193 review #1):
+- `kayak-healthcheck` (`scripts/health-check.sh`) — reads the DB → **quiesce**.
+- `kayak-config-drift` (`scripts/check-config-drift.sh`) — compares repo↔/etc, no
+  DB content read.
+- `kayak-recap` (`systemd/kayak-recap.sh`) — runs `${KAYAK_HOME}/.venv/bin/python3
+  scripts/recap.py` (the **editable-install venv**, NOT a `levels` subcommand);
+  journald only, no DB/dataset. Stays on the checkout — re-pointing buys nothing
+  (running old code to render a journald email is harmless); no quiesce.
+- `kayak-heartbeat` (`systemd/kayak-heartbeat.sh`) — `mail`/msmtp heartbeat (no
+  curl); `stat()`s the DB file mtime (no content read, no lock) → no quiesce.
+- `kayak-cert-expiry` (`scripts/check-cert-expiry.sh`), `kayak-cert-renewal-test`
+  (`certbot`) — no DB.
+
+> Note: recap uses the venv but isn't a class-A unit — it runs `python3
+> scripts/recap.py`, not `levels`, and touches neither the migrated DB nor the
+> release dataset, so §Goal's "every `levels`-running consumer points at current"
+> doesn't reach it. The D-CONSUMER derivation keys re-pointing off "ExecStart runs
+> `…/levels`", which correctly excludes recap.
+
+**C. Pure host-level — exempt via `KAYAK_HOST_UNITS`:** `kayak-backup-{hourly,
+weekly,offsite}` (shell), `kayak-notify-failure@` (template), `kayak-fail-test`.
+
+> **Open decision D-CONSUMER:** the deployer's activation gate verifies every
+> `KAYAK_UNITS` member's `ExecStart` references `$ROOT/current` *unless* it's in
+> `KAYAK_HOST_UNITS`. The repo-shell-script consumers (B) reference neither `current`
+> nor the venv — they reference `/home/<svc>/kayak`. So they must be listed in
+> `KAYAK_HOST_UNITS` to pass the gate, **even though some read the DB** (so they
+> must ALSO be quiesced). Today `KAYAK_UNITS` (quiesce) and `KAYAK_HOST_UNITS`
+> (gate-exempt) are independent lists, which already supports "quiesce but don't
+> verify". The renderer/runbook must emit a `KAYAK_HOST_UNITS` that includes the
+> class-B units, and `KAYAK_UNITS` that includes the DB-touching ones. Plan: derive
+> both lists from the installed `kayak-*.timer` set + this classification table,
+> rather than hand-maintaining them (closes the "complete consumer enumeration"
+> 4C item).
+
+## `HostConfig` additions (increment 1)
+
+New non-secret fields on `kayak.host.HostConfig` (all with current-WKCC defaults,
+keep-current-then-flip — the live `host.yaml` flips them at cutover):
+
+| field | default (current) | cutover value | used by |
+|---|---|---|---|
+| `service_user` | `pat` | `pat` | unit `User=`, ACL, `KAYAK_APP_USER` |
+| `service_home` | `/home/pat` | `/home/pat` | `KAYAK_HOME`, DB/var/log dirs |
+| `release_root` | `/opt/kayak` | `/opt/kayak` | venv/dataset paths, `$ROOT` |
+| `fpm_pool_php` | `8.4` | `8.4` | FPM pool path |
+| `server_names` | (per-vhost, see below) | — | nginx `server_name` |
+
+`docroot` (existing field) is reused: it stays `/home/pat/public_html` until the
+cutover `host.yaml` sets it to `/var/cache/kayak/docroot` (matching the deployer's
+`KAYAK_DOCROOT`). Derived (not stored): venv = `{release_root}/current/venv`,
+release dataset = `{release_root}/current/dataset`, FPM pool =
+`/etc/php/{fpm_pool_php}/fpm/pool.d/kayak.conf`.
+
+The vhost `server_names` need a small structured type (a list of
+`{server_name, cert_host, enabled}`) for the three sites (`levels.wkcc.org`,
+`levels.mousebrains.com`, `levels-test`). That type lands with the **nginx
+renderer (increment 3)** that consumes it, not here — adding it before its
+consumer risks the wrong shape. Increment 1 is the scalar fields only.
+
+## Renderer mechanism
+
+A `levels` subcommand per the `emit-config` precedent (emits text; the installer/
+runbook redirects it). Proposed:
+- `levels render-units [--out-dir DIR]` → the class-A drop-in files
+  (`<unit>.service.d/cutover.conf`) from `HostConfig`. Default: print a manifest;
+  `--out-dir` writes the files.
+- `levels render-nginx [--site NAME]` / `levels render-fpm` → vhost root +
+  `open_basedir` substitutions (or a sed-spec the runbook applies). Increment 3.
+
+Each renderer is **pure** (HostConfig + templates → text), unit-tested by asserting
+the rendered text matches the runbook §5 spec. Real `systemd-analyze verify` /
+`nginx -t` validation happens on the VM (Pat-driven; see handoff).
+
+## The class-A drop-in spec (`<unit>.service.d/cutover.conf`)
+
+```ini
+[Service]
+ExecStart=
+ExecStart=/opt/kayak/current/venv/bin/levels <cmd> <args…>
+Environment=DATASET_DIR=/opt/kayak/current/dataset
+Environment=OUTPUT_DIR=/var/cache/kayak/docroot
+WorkingDirectory=/opt/kayak/current
+ReadWritePaths=/var/cache/kayak/docroot /home/<svc>/DB
+```
+(The leading empty `ExecStart=` resets the base unit's value — required by systemd
+to replace, not append.) `<cmd> <args…>` comes from the base unit's ExecStart
+tail (e.g. `pipeline`; `status --output …`; `audit-gauges --days 16 --email
+${AUDIT_EMAIL}`). `ReadWritePaths` differs per unit (status writes
+`status_output`; audit-gauges writes the metadata cache — see #191). The renderer
+must carry the per-unit write-path set, not a blanket one.
+
+## Increment sequence (each a reviewable PR; VM-validated where noted)
+
+1. **`HostConfig` renderer fields + tests.** Pure schema (scalar fields); no
+   behavior change (defaults = current). Foundational. *(merged in PR #193's branch)*
+2. **`levels render-units` + tests** asserting the 6 class-A drop-ins match the
+   spec. Also adds the two relocatable-cache fields (`map_layers_dir`,
+   `gauge_metadata_cache`) — surfaced here because `fetch-osmb`/`audit-gauges`
+   default those dirs *relative to the install root*, which is read-only under
+   `/opt/kayak/current`, so the drop-ins must point them at `/var/cache/kayak/*`
+   (keep-current-then-flip, like `docroot`). Wires nothing into the host. *(this PR)*
+3. **`levels render-nginx` / `render-fpm` + tests.** The root/`open_basedir`
+   substitutions.
+4. **Deployer serving-path gate + quiesce-timeout fix** (`deploy/kayak-deploy.sh`):
+   when `SERVING_CUTOVER=yes`, verify nginx root / FPM `open_basedir` / unit
+   `OUTPUT_DIR`+`ReadWritePaths` resolve to `$KAYAK_DOCROOT`; back out maintenance
+   on a drain timeout (the [[deploy_quiesce_timeout_followup]] fix). Branch off
+   #192. Slow-test the gate.
+5. **Derive `KAYAK_UNITS`/`KAYAK_HOST_UNITS` from installed timers** (closes the
+   complete-consumer-enumeration item; resolves D-CONSUMER).
+6. **Runbook §5 rewrite** (`deploy/INSTALL-paired-release.md` on `b4c-paired-install`):
+   replace hand-crafted step 5 with the `render-*` calls; update for the #3 docroot
+   and the audit-gauges promotion. Rebase onto merged #192.
+7. **Cutover the live host** (Pat-driven, with the VM rehearsed first): ship the
+   live `host.yaml`, run the renderers, ACLs, flip `SERVING_CUTOVER=yes`, first
+   real `kayak-deploy` activation. Flip the generic defaults last.
+
+## VM-validation handoff
+
+Increments 2–5 produce text/gate logic unit-tested here, but the rendered units/
+vhosts must pass `systemd-analyze verify` + `nginx -t` on the arm64 test VM (ssh
+`levels-mac`) before the live cutover — that's the same drive→codex→Pat rehearsal
+flow as [[vm_rehearsal_plan]]. The hand-crafted artifacts already validated on the
+VM (2026-06-13) are the renderers' golden output to diff against.
+
+## Reproduce / cross-check
+
+```bash
+# consumer classification:
+for f in systemd/kayak-*.service; do printf '%s\t%s\n' "$(basename "$f")" \
+  "$(grep -h '^ExecStart=' "$f" | head -1)"; done
+# the hand-crafted spec the renderers must reproduce:
+git show b4c-paired-install:deploy/INSTALL-paired-release.md   # § step 5
+```

src/kayak/cli/main.py

@@ -27,6 +27,7 @@
     orphan_check,
     pipeline,
     recover_metadata,
+    render_units,
     seed_maintainer,
     status,
     sync_metadata,
@@ -75,6 +76,7 @@ def main() -> None:
     analyze_logs.addArgs(subparsers)
     status.addArgs(subparsers)
     sync_metadata.addArgs(subparsers)
+    render_units.addArgs(subparsers)
     import_metadata.addArgs(subparsers)
     recover_metadata.addArgs(subparsers)
     generate_sources.addArgs(subparsers)

src/kayak/cli/render_units.py

@@ -0,0 +1,76 @@
+"""``levels render-units`` — emit the Batch 4C cutover systemd drop-ins.
+
+Reads ``host.yaml`` and prints (or writes) one
+``<unit>.service.d/cutover.conf`` per engine consumer — the drop-ins that
+re-point each ``levels``-running unit at the ``/opt/kayak/current`` release
+venv. Mirrors ``emit-config``: the tool emits text; the install runbook /
+deployer applies it. See ``docs/PLAN_4c_renderers.md``.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from kayak.host import load_host_config
+from kayak.host_render import render_cutover_dropins
+
+
+def addArgs(subparsers: argparse._SubParsersAction[argparse.ArgumentParser]) -> None:
+    p = subparsers.add_parser(
+        "render-units",
+        help="Render the paired-release cutover systemd drop-ins from host.yaml (4C)",
+    )
+    p.set_defaults(func=render_units)
+    p.add_argument(
+        "--out-dir",
+        type=Path,
+        help="Write <unit>.service.d/cutover.conf trees under this dir "
+        "(e.g. /etc/systemd/system); default: print a manifest to stdout",
+    )
+    p.add_argument(
+        "--host-config",
+        type=Path,
+        help="host.yaml path (default: $KAYAK_HOST_CONFIG or /etc/kayak/host.yaml)",
+    )
+
+
+def render_units(args: argparse.Namespace) -> int:
+    try:
+        host = load_host_config(args.host_config)
+    except ValueError as e:
+        print(f"render-units: host config invalid: {e}", file=sys.stderr)
+        return 1
+
+    dropins = render_cutover_dropins(host)
+
+    if args.out_dir is None:
+        for d in dropins:
+            print(f"# ==> {d.path}")
+            print(d.text)
+        return 0
+
+    # Every drop-in references {release_root}/current (DATASET_DIR, WorkingDirectory,
+    # the ExecStart venv), which exists only AFTER the first paired-release
+    # activation. Installing them onto a host whose release isn't live yet would
+    # point all six consumers at a non-existent dir and break them, so warn — the
+    # cutover runbook installs these only once `current` resolves (PR #193 review #3).
+    current = Path(host.release_root) / "current"
+    if not current.exists():
+        print(
+            f"render-units: WARNING: {current} does not exist yet — these drop-ins "
+            "reference it and will break the consumers if applied before the first "
+            "paired-release activation (cutover order: stage+activate, THEN install).",
+            file=sys.stderr,
+        )
+
+    # NOTE: this writes cutover.conf for the current 6-unit set but does not sweep a
+    # stale drop-in if that set ever shrinks; revisit if a consumer is retired.
+    out_dir: Path = args.out_dir
+    for d in dropins:
+        dest = out_dir / d.path
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        dest.write_text(d.text, encoding="utf-8")
+        print(f"wrote {dest}")
+    return 0

src/kayak/host.py

@@ -36,10 +36,25 @@
 
 HOST_YAML = "/etc/kayak/host.yaml"
 
-_HOSTNAME_RE = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$")
+# All anchored with \A…\Z, NOT ^…$: Python's `$` also matches just before a
+# trailing newline, so `^…$` would accept e.g. "pat\n" and let a newline smuggle
+# a second directive into a rendered unit / shell command (PR #193 review #2).
+_HOSTNAME_RE = re.compile(r"\A[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+\Z")
 # rclone remote names: word characters and hyphens (no ':' — the colon is
 # syntax, appended by consumers).
-_RCLONE_REMOTE_RE = re.compile(r"^[A-Za-z0-9_-]+$")
+_RCLONE_REMOTE_RE = re.compile(r"\A[A-Za-z0-9_-]+\Z")
+# POSIX-portable service account name (useradd's NAME_REGEX): start with a
+# lower-case letter or underscore, then lower/digit/underscore/hyphen.
+_USERNAME_RE = re.compile(r"\A[a-z_][a-z0-9_-]*\Z")
+# PHP-FPM pool version as it appears in the /etc/php/<v>/fpm path (major.minor).
+# [0-9] not \d — \d also matches Unicode digits, which the path can't contain.
+_PHP_VERSION_RE = re.compile(r"\A[0-9]+\.[0-9]+\Z")
+# Whitespace / control chars in a path field. The renderers interpolate paths
+# into systemd directives by plain f-string (Environment=, ReadWritePaths=,
+# WorkingDirectory=, ExecStart=); a newline injects a directive and a space
+# splits a ReadWritePaths= entry, so reject both even though host.yaml is trusted
+# (defense-in-depth, the same reasoning that hardened service_user — review #2).
+_PATH_BAD_CHAR_RE = re.compile(r"[\s\x00-\x1f\x7f]")
 
 
 class HostConfig(BaseModel):
@@ -58,6 +73,34 @@ class HostConfig(BaseModel):
     docroot: str = "/home/pat/public_html"
     cert_host: str = "levels.wkcc.org"
 
+    # --- paired-release cutover / renderers (4C) ---
+    # The service account + its home, the paired-release root, and the PHP-FPM
+    # pool version the systemd-unit / nginx-vhost / FPM renderers need. Defaults
+    # are the current WKCC values (keep-current-then-flip): a host with no
+    # host.yaml renders the live shape, and the cutover host.yaml flips them.
+    # ``docroot`` (above) is reused — it stays ``public_html`` until the cutover
+    # host.yaml sets it to ``/var/cache/kayak/docroot`` (= the deployer's
+    # ``KAYAK_DOCROOT``). Paths the renderers derive, NOT stored: the release venv
+    # (``{release_root}/current/venv``), the release dataset
+    # (``{release_root}/current/dataset``), and the FPM pool
+    # (``/etc/php/{fpm_pool_php}/fpm/pool.d/kayak.conf``). See
+    # docs/PLAN_4c_renderers.md.
+    service_user: str = "pat"
+    service_home: str = "/home/pat"
+    release_root: str = "/opt/kayak"
+    fpm_pool_php: str = "8.4"
+    # Generated-runtime-data dirs that the engine defaults resolve RELATIVE to the
+    # install root (config.py BASE_DIR): fetch-osmb's map-layer staging and the
+    # gauge-audit metadata cache. In the editable install they land under the repo
+    # checkout; under an immutable /opt/kayak/current release that root is
+    # read-only, so the cutover unit drop-ins must point these at stable writable
+    # cache paths. Keep-current defaults (the live repo-relative locations); the
+    # cutover host.yaml flips them to /var/cache/kayak/* (regenerable cache,
+    # alongside the #3 docroot). map_layers_dir is a dir; gauge_metadata_cache is
+    # the sqlite FILE.
+    map_layers_dir: str = "/home/pat/kayak/var/osmb"
+    gauge_metadata_cache: str = "/home/pat/kayak/Gauge-metadata-cache/gauges.db"
+
     # --- backup policy (S8) ---
     backup_dir: str = "/home/pat/backups"
     offsite_remote: str = "gdrive-crypt"
@@ -79,11 +122,41 @@ def _tz(cls, v: str) -> str:
             raise ValueError(f"must be an IANA timezone name (got {v!r})") from e
         return v
 
-    @field_validator("nginx_log_glob", "status_output", "docroot", "backup_dir")
+    @field_validator(
+        "nginx_log_glob",
+        "status_output",
+        "docroot",
+        "backup_dir",
+        "service_home",
+        "release_root",
+        "map_layers_dir",
+        "gauge_metadata_cache",
+    )
     @classmethod
     def _abs_path(cls, v: str) -> str:
         if not v.startswith("/"):
             raise ValueError(f"must be an absolute path (got {v!r})")
+        if _PATH_BAD_CHAR_RE.search(v):
+            raise ValueError(f"must not contain whitespace or control characters (got {v!r})")
+        return v
+
+    @field_validator("service_user")
+    @classmethod
+    def _username(cls, v: str) -> str:
+        # POSIX-portable service account name: lower/digit/underscore/hyphen,
+        # not starting with a hyphen. Renderers interpolate it into unit User=
+        # and shell ACL commands, so reject anything that isn't a bare name.
+        if not _USERNAME_RE.match(v):
+            raise ValueError(f"must be a bare POSIX username (got {v!r})")
+        return v
+
+    @field_validator("fpm_pool_php")
+    @classmethod
+    def _php_version(cls, v: str) -> str:
+        # PHP-FPM pool version as it appears in /etc/php/<v>/fpm — major.minor
+        # only (Debian packages php8.4, never a patch level in the path).
+        if not _PHP_VERSION_RE.match(v):
+            raise ValueError(f"must be a major.minor PHP version like '8.4' (got {v!r})")
         return v
 
     @field_validator("cert_host")