fix(backend): retry + self-heal the cerastream engine connection

[andrescera]

What

Adds modules/streaming/engine-reconnect.ts (initEngineConnection), which becomes the boot pipelines init in main.ts. It runs the first engine capability probe synchronously (boot ordering unchanged), then — if cerastream is unreachable — arms one self-rescheduling reconnect loop:

• a short exponential backoff (~2s, 4s, 8s, 16s → 30s ceiling) that resolves a normal engine-not-ready-yet race, then
• a bounded periodic (30s) health-recheck so a device self-heals minutes/hours later.

On the unavailable → reachable transition it re-broadcasts capabilities + pipelines + sources to already-connected clients (the same trio the setMockHardware RPC uses), so the "Streaming engine offline" banner clears live, without a page reload, then settles.

Why

Confirmed on real hardware: the engine capability contract was fetched over a short-lived probe exactly once at boot inside guardNonCritical("pipelines", initPipelines). If cerastream.service wasn't up yet (a systemd-ordering race being fixed separately in image-building-pipeline, or just a slow start), the fallback ladder marked the engine unavailable permanently — no retry, no recheck — so the offline banner never cleared even though cerastream came up healthy moments later. This is defense-in-depth: it stands on its own even with correct ordering, covering any transient failure (slow start, brief socket hiccup, a cerastream-only restart).

The @ceralive/cerastream client does expose ConnectOptions.autoReconnect, but it only rescues an already-established connection that later drops and throws immediately on the first connect failure — useless for the fresh per-fetch capability probe — and emits no "became available" event. So recovery lives backend-side. Reachability feeds the existing engine-unavailable/engine-starting capability tier; no parallel state machine is introduced. Backoff mirrors the modules/remote-control/channel.ts convention.

Note: the separate version-skew error (get-capabilities additive method #9 missing on an older engine binary) is an already-tracked follow-up (redeploy a newer cerastream) and is not addressed here — the loop correctly treats it as "unavailable" and self-heals once a matching engine ships.

How to verify

From apps/backend/:

• bun test src/tests/engine-reconnect.test.ts — 8 tests: boot-retry heal, later out-of-band reconnect, backoff-ceiling cadence, and the permanently-unavailable case driven through the real capability ladder (proves no regression to engineUnavailable/engineStarting).
• bun tsc --noEmit — clean.
• biome check — clean.

Full bun test is green except 4 pre-existing environment/permission EACCES failures (/run/ceralive/srtla_ips, /var/run/bcrpt) unrelated to this change — none reference the new module or any file touched here.

Risks

• Low. Boot ordering is preserved (first attempt awaited; the background loop only fires ≥~2s later, after boot's synchronous reconcilePersistedPipeline/sources steps). The loop is bounded (30s ceiling — a masked/disabled cerastream just gets a cheap periodic poll, never a tight loop) and settles once the engine is reachable. All collaborators are injected; stopEngineReconnect()/settleEngineReconnect() are the teardown/test seams. Docs updated in apps/backend/AGENTS.md.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 1e444a35-0edf-4a80-999c-32ed45445789

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/cerastream-connection-retry

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}