diff --git a/docs-site/src/content/docs/ko/reference/configuration.md b/docs-site/src/content/docs/ko/reference/configuration.md index a08b19cff..691c40d06 100644 --- a/docs-site/src/content/docs/ko/reference/configuration.md +++ b/docs-site/src/content/docs/ko/reference/configuration.md @@ -45,6 +45,7 @@ namespaced selected id를 bare id로 바꿉니다. | `stallTimeoutSec?` | `number` | `300` | 업스트림 데이터가 오지 않을 때 bridge가 중단하고 `response.incomplete`를 내보내기까지의 초. 최소 1. | | `connectTimeoutMs?` | `number` | `200000` | DNS/TCP/TLS와 최종 응답 헤더만 기다리는 시도별 deadline. 응답 body 생성 전 종료됩니다. | | `shutdownTimeoutMs?` | `number` | `5000` | 진행 중인 turn을 중단하기 전 graceful drain deadline. | +| `memoryWatchdog?` | object | 경고 전용 | 메모리 압력 관찰 + opt-in 우아한 재시작 (아래 [memoryWatchdog](#memorywatchdog) 참고). | | `websockets?` | `boolean` | `false` | `supports_websockets`를 알려 Codex가 Responses WebSocket 경로를 쓰게 합니다. 생략하거나 `false`이면 HTTP/SSE를 유지합니다. | | `apiKeys?` | `OcxApiKey[]` | `[]` | 비-loopback 바인드에서 관리 API와 data plane 인증에 추가로 허용할 생성형 `ocx_…` 자격 증명. 대시보드가 관리하며 항목 필드는 아래에 설명합니다. | | `codexAutoStart?` | `boolean` | `true` | Codex shim이 Codex 실행 전에 `ocx ensure`를 실행하게 합니다. `false`이면 `ocx ensure`가 아무 작업도 하지 않습니다. | @@ -97,6 +98,65 @@ pool 계정 추가와 quota 갱신은 대시보드의 **Codex Auth** 페이지 | `codexWarmupMaxAgeSeconds?` | `number` | `691200` | 계정을 다시 검증할 최대 기간(8일). | | `codexWarmupModel?` | `string` | `gpt-5.4-mini` | 선택형 warmup에 쓸 네이티브 모델. | +### `memoryWatchdog` + +프로세스 메모리 압력을 전체 시스템 RAM 대비 비율로 관찰하고 임계값을 넘으면 경고합니다. 기본 +동작은 경고 전용입니다. 명시적으로 활성화한 경우에만, 애플리케이션 코드로는 해결할 수 없는 +네이티브 committed 메모리 미반환 문제의 완화책으로 위험 임계값에서 프록시를 우아하게 재시작할 +수 있습니다. + +측정은 완전히 비동기입니다: Windows에서는 숨겨진 자식 PowerShell 프로세스 하나가 이 PID의 +Private Bytes와 시스템 전역 **Committed Bytes / Commit Limit**을 한 번의 spawn으로 수집하므로, +느린 측정이 프록시의 이벤트 루프를 막는 일이 없습니다(타임아웃된 프로브는 종료되고 해당 주기는 +RSS 폴백으로 강등). 첫 측정은 시작 직후 즉시 실행됩니다. 시스템 커밋은 별도의 **관찰 전용** +축입니다: 내부 high-water(기본: 커밋 한도의 0.90)를 넘으면 래치된 경고를 한 번 기록하고, 측정된 +회복이 있을 때만 재무장하며, **절대 재시작을 유발하지 않습니다** — 커밋 압력의 원인이 다른 +프로세스일 수 있어 OpenCodex를 재시작해도 해소되지 않기 때문입니다. 실험적 env 전용 오버라이드 +`OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER`(`[0.50, 0.99]`로 clamp)로 조정 가능하며, 실측으로 축이 +검증되기 전까지 config/UI 설정은 추가하지 않습니다. + +| Field | Type | Default | Meaning | +| --- | --- | --- | --- | +| `enabled?` | `boolean` | `true` | 전체 스위치. 기본 관찰 모드는 경고 전용이며 데이터를 잃지 않습니다. | +| `intervalMs?` | `number` | `60000` | 샘플링 간격. 최소 1000ms. | +| `warnFraction?` | `number` | `0.60` | 압력/전체 RAM이 이 비율을 넘으면 경고. `[0.10, 0.99]`로 clamp. | +| `criticalFraction?` | `number` | `0.75` | 위험 임계값. opt-in 재시작을 준비시킵니다. 항상 `warnFraction`보다 높게 유지. | +| `autoRestart?` | `boolean` | `false` | 위험 임계값에서 drain 후 우아한 재시작 opt-in. | +| `requireSupervisor?` | `boolean` | `true` | supervisor가 감지될 때만 재시작 — 프록시가 그냥 죽은 채 남지 않도록. | +| `restartGraceMs?` | `number` | `30000` | quiet-window drain 예산: 진행 중인 turn이 끝나도록 최대 이 시간까지 기다린 뒤 재시작합니다. drain 중 새 요청은 `503` + `Retry-After`를 받습니다. `[1000, 600000]`으로 clamp. | +| `minRestartIntervalMs?` | `number` | `600000` | 재시작 요청 간 cooldown. 최소 `restartGraceMs` 이상으로 보정됩니다. | +| `maxRestarts?` | `number` | `3` | 약 6시간 롤링 윈도우 안에서 허용되는 메모리 기반 재시작 횟수. 초과 시 경고 전용으로 강등. `0`이면 자동 재시작 비활성화. | + +환경변수 오버라이드(최우선): `OCX_MEMORY_WATCHDOG_ENABLED` / `_DISABLED`, `_INTERVAL_MS`, +`_WARN_FRACTION`, `_CRITICAL_FRACTION`, `_AUTO_RESTART`, `_REQUIRE_SUPERVISOR`, +`_RESTART_GRACE_MS`, `_MIN_RESTART_INTERVAL_MS`, `_MAX_RESTARTS`, `_COMMIT_HIGH_WATER`(실험적, +env 전용 — 모두 `OCX_MEMORY_WATCHDOG` 접두사). 환경변수, `config.json`, 대시보드/관리 API 등 +모든 진입 경로가 동일한 최종 검증과 clamp를 거칩니다. + +`/api/memory` 리포트(및 대시보드 Memory 카드)는 마지막 프로브의 측정값을 추가로 노출합니다: +실제 Private Bytes와 판단에 실제 사용된 압력 값의 구분(`processSource`가 출처를 명시 — +`windows-private`, `proc-status`, 또는 강등된 `rss-fallback`), 시스템 커밋 사용량/한도/비율, +가용 물리 메모리, 응답 캐시 지표(항목 수/직렬화 바이트/최대 항목), 강등 시 새니타이즈된 +`probeError` 코드, 그리고 신선도 확인용 `lastProbeAt` / `lastSuccessfulSystemProbeAt` +타임스탬프. + +:::note[Supervisor와 Windows] +메모리 기반 재시작은 **exit code 75로 종료하는 것**이 전부입니다 — 외부 supervisor에게 재기동을 +요청하는 신호일 뿐입니다. supervisor가 없으면 프록시는 그냥 멈춘 채 남으므로 +`requireSupervisor`가 기본 `true`입니다. + +- supervisor 자동 감지는 **pm2**, **systemd**, 명시적 `OCX_SUPERVISED=1` 플래그만 지원합니다. + **NSSM, Windows 서비스(`sc.exe`), 작업 스케줄러 등 Windows 서비스 관리자는 자동 감지되지 + 않습니다** — 서비스 관리자가 종료 시 재시작하도록 구성되어 있다면 서비스 환경에 + `OCX_SUPERVISED=1`을 설정하세요. +- watchdog의 cooldown과 `maxRestarts` 카운터는 작은 best-effort 이력 파일(타임스탬프만 저장)로 + 재시작 경계를 넘어 시드됩니다. 이 파일을 읽거나 쓸 수 없으면 새 프로세스에서 0부터 다시 + 시작하므로, 재시작 폭주를 제한하긴 하지만 **프로세스 경계를 넘는 영구적 보장은 아닙니다**. + supervisor 자체의 재시작 제한/backoff 정책(예: pm2 `max_restarts` + + `exp_backoff_restart_delay`, systemd `StartLimitIntervalSec` + `StartLimitBurst`)을 바깥층 + 보호로 함께 설정하세요. +::: + ## 원격 접근 opencodex는 기본적으로 `127.0.0.1`(loopback 전용)에 바인드합니다. `hostname`을 `0.0.0.0` 같은 diff --git a/docs-site/src/content/docs/reference/configuration.md b/docs-site/src/content/docs/reference/configuration.md index 505a3e9d6..d113e4ff8 100644 --- a/docs-site/src/content/docs/reference/configuration.md +++ b/docs-site/src/content/docs/reference/configuration.md @@ -46,6 +46,7 @@ differing backup and rewrites known legacy namespaced selected ids to bare ids. | `stallTimeoutSec?` | `number` | `300` | Seconds without upstream data before the bridge aborts and emits `response.incomplete`. Minimum 1. | | `connectTimeoutMs?` | `number` | `200000` | Per-attempt deadline for DNS/TCP/TLS and final response headers only; it ends before response-body generation. | | `shutdownTimeoutMs?` | `number` | `5000` | Graceful drain deadline before active turns are aborted. | +| `memoryWatchdog?` | object | warn-only | Memory-pressure observation and opt-in graceful restart (see [memoryWatchdog](#memorywatchdog) below). | | `websockets?` | `boolean` | `false` | Advertise `supports_websockets` so Codex uses the Responses WebSocket path. Omit or set `false` to keep HTTP/SSE. | | `apiKeys?` | `OcxApiKey[]` | `[]` | Additional generated `ocx_…` credentials accepted by management and data-plane auth on non-loopback binds. Managed by the dashboard; entry fields are listed below. | | `codexAutoStart?` | `boolean` | `true` | Let the Codex shim run `ocx ensure` before launching Codex. `false` makes `ocx ensure` a no-op. | @@ -109,6 +110,67 @@ normally dashboard-managed. | `codexWarmupMaxAgeSeconds?` | `number` | `691200` | Revalidate an account after 8 days. | | `codexWarmupModel?` | `string` | `gpt-5.4-mini` | Native model used for optional warmup. | +### `memoryWatchdog` + +Observes process memory pressure as a fraction of total system RAM and warns when it crosses +thresholds. By default it only warns. Optionally — and only when explicitly enabled — it can +gracefully restart the proxy at the critical threshold, as a mitigation for native +committed-memory retention that cannot be fixed in application code. + +Measurement is fully asynchronous: on Windows a hidden child PowerShell process collects this +PID's Private Bytes plus the system-wide **Committed Bytes / Commit Limit** in one spawn, so the +proxy's event loop is never blocked by a slow probe (a timed-out probe is killed and that cycle +degrades to an RSS fallback). The first probe fires immediately at startup. System commit is a +separate **observe-only** axis: crossing an internal high-water (default 0.90 of the commit +limit) logs one latched warning and re-arms only on a measured recovery — it **never triggers a +restart**, because the commit pressure may come from another process that restarting OpenCodex +would not free. Tunable via the experimental env-only override +`OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER` (clamped to `[0.50, 0.99]`; no config/UI knob until the +axis is validated by field measurement). + +| Field | Type | Default | Meaning | +| --- | --- | --- | --- | +| `enabled?` | `boolean` | `true` | Master switch. The default observation mode is warn-only and cannot lose data. | +| `intervalMs?` | `number` | `60000` | Sampling interval. Floored at 1000 ms. | +| `warnFraction?` | `number` | `0.60` | Warn when pressure / total RAM crosses this fraction. Clamped to `[0.10, 0.99]`. | +| `criticalFraction?` | `number` | `0.75` | Critical threshold; arms the opt-in restart. Kept above `warnFraction`. | +| `autoRestart?` | `boolean` | `false` | Opt in to a graceful drain-then-restart at the critical threshold. | +| `requireSupervisor?` | `boolean` | `true` | Only restart when a process supervisor is detected, so the proxy is never left dead. | +| `restartGraceMs?` | `number` | `30000` | Quiet-window drain budget: in-flight turns get up to this long to finish before the restart proceeds; new requests receive `503` + `Retry-After` while draining. Clamped to `[1000, 600000]`. | +| `minRestartIntervalMs?` | `number` | `600000` | Cooldown between restart requests. Raised to at least `restartGraceMs`. | +| `maxRestarts?` | `number` | `3` | Memory-driven restarts allowed within a rolling ~6 h window before the watchdog degrades to warn-only. `0` disables auto-restart. | + +Environment overrides (highest precedence): `OCX_MEMORY_WATCHDOG_ENABLED` / `_DISABLED`, +`_INTERVAL_MS`, `_WARN_FRACTION`, `_CRITICAL_FRACTION`, `_AUTO_RESTART`, `_REQUIRE_SUPERVISOR`, +`_RESTART_GRACE_MS`, `_MIN_RESTART_INTERVAL_MS`, `_MAX_RESTARTS`, `_COMMIT_HIGH_WATER` +(experimental, env-only — all prefixed `OCX_MEMORY_WATCHDOG`). All entry paths — environment, +`config.json`, and the dashboard / management API — go through the same final validation and +clamping. + +The `/api/memory` report (and the dashboard's Memory card) additionally exposes the last probe's +measurements: actual Private Bytes vs. the pressure value the decisions use (`processSource` +names the origin — `windows-private`, `proc-status`, or the degraded `rss-fallback`), system +commit used/limit/fraction, available physical memory, response-cache metrics (entry count / +serialized bytes / largest entry), a sanitized `probeError` code when degraded, plus +`lastProbeAt` / `lastSuccessfulSystemProbeAt` timestamps for staleness. + +:::note[Supervisors and Windows] +A memory-driven restart works by **exiting with code 75** — a request to an external supervisor +to respawn the process, nothing more. Without a supervisor the proxy would simply stay stopped, +which is why `requireSupervisor` defaults to `true`. + +- Supervisor auto-detection covers **pm2**, **systemd**, and the explicit `OCX_SUPERVISED=1` + flag. **NSSM, Windows services (`sc.exe`), Task Scheduler and similar Windows service + managers are not auto-detected** — set `OCX_SUPERVISED=1` in the service environment when the + service manager is configured to restart the process on exit. +- The watchdog's cooldown and `maxRestarts` counters are seeded across restarts from a small + best-effort history file (timestamps only). When that file cannot be read or written, they + restart from zero in the new process — so they limit restart churn but are **not a permanent + cross-process guarantee**. Configure your supervisor's own restart limit / backoff policy + (e.g. pm2 `max_restarts` + `exp_backoff_restart_delay`, systemd `StartLimitIntervalSec` + + `StartLimitBurst`) as the outer layer of protection. +::: + ## Remote access By default opencodex binds to `127.0.0.1` (loopback only). When `hostname` is set to a non-loopback diff --git a/gui/src/components/MemoryCard.tsx b/gui/src/components/MemoryCard.tsx new file mode 100644 index 000000000..e76386dee --- /dev/null +++ b/gui/src/components/MemoryCard.tsx @@ -0,0 +1,397 @@ +import { useCallback, useEffect, useRef, useState } from "react"; +import { useI18n } from "../i18n"; + +/* + * MemoryCard — dashboard (beta) panel for the memory watchdog. Three layered, opt-in options: + * 1. Monitor — read-only live memory pressure vs total RAM (zero risk). + * 2. Recommend — advisory thresholds from observed baseline (never applied automatically). + * 3. Adjust — toggle/tune thresholds + opt-in auto-restart (management-auth gated; auto-restart + * is further gated by supervisor detection so we never exit into "just dead"). + * Self-contained: it owns its own fetch/poll/PUT so it does not couple to the large Dashboard. + */ + +type WatchdogLevel = "ok" | "warn" | "critical"; + +interface WatchdogDecision { + level: WatchdogLevel; + action: string; + fraction: number; + pressureMb: number; + totalMb: number; + source: string; + growthMbPerHour: number | null; + reason: string; +} + +interface ResolvedConfig { + enabled: boolean; + intervalMs: number; + warnFraction: number; + criticalFraction: number; + autoRestart: boolean; + requireSupervisor: boolean; + minRestartIntervalMs: number; + maxRestarts: number; + /** Optional: servers older than the quiet-window feature do not report it. */ + restartGraceMs?: number; + growthWindowMs: number; +} + +interface Recommendation { + warnFraction: number; + criticalFraction: number; + autoRestart: boolean; + rationale: string; +} + +/** Per-probe measurement block (§7); optional so dashboards tolerate older servers. */ +interface MemoryBlock { + processPrivateBytes: number | null; + processPressureBytes: number; + processSource: string; + rssBytes: number; + heapUsedBytes: number; + externalBytes: number; + physicalMemoryBytes: number; + availablePhysicalBytes: number | null; + systemCommittedBytes: number | null; + systemCommitLimitBytes: number | null; + systemCommitFraction: number | null; + systemCommitAvailable: boolean; + probeError?: string; +} + +interface ResponseStateBlock { + count: number; + totalBytes: number; + largestBytes: number; + oldestAgeMs: number; +} + +interface MemoryReport { + enabled: boolean; + decision: WatchdogDecision | null; + samplesCount?: number; + growthMbPerHour?: number | null; + resolvedConfig?: ResolvedConfig; + supervisor?: { supervised: boolean; hint: string }; + recommendation?: Recommendation; + restartCount?: number; + memory?: MemoryBlock | null; + responseState?: ResponseStateBlock | null; + lastProbeAt?: number | null; + lastSuccessfulSystemProbeAt?: number | null; +} + +interface SettingsPatch { + enabled?: boolean; + warnFraction?: number; + criticalFraction?: number; + autoRestart?: boolean; + requireSupervisor?: boolean; + restartGraceMs?: number; +} + +/** Mirror of the server's RESTART_GRACE_MIN_MS/MAX_MS bounds, in seconds (UI unit). */ +const GRACE_MIN_S = 1; +const GRACE_MAX_S = 600; + +const LEVEL_COLOR: Record = { + ok: "var(--green)", + warn: "var(--amber, #d89614)", + critical: "var(--red)", +}; + +const MIB = 1024 * 1024; + +function pct(fraction: number): number { + return Math.round(fraction * 1000) / 10; +} + +function gb1(bytes: number): string { + return (bytes / (1024 * MIB)).toFixed(1); +} + +/** Stateless fetch so the polling effect never calls setState synchronously in its body. */ +async function loadReport(apiBase: string): Promise { + try { + const res = await fetch(`${apiBase}/api/memory`); + if (!res.ok) return null; + return (await res.json()) as MemoryReport; + } catch { + return null; // old server / offline — keep the last report + } +} + +export default function MemoryCard({ apiBase }: { apiBase: string }) { + const { t, locale } = useI18n(); + const [report, setReport] = useState(null); + // Wall-clock of the last successful report fetch — probe age must not call Date.now() in render. + const [reportAt, setReportAt] = useState(0); + const [loaded, setLoaded] = useState(false); + const [saving, setSaving] = useState(false); + const [saveError, setSaveError] = useState(null); + // Draft threshold percents while dragging; committed on "Apply thresholds". + const [warnDraft, setWarnDraft] = useState(null); + const [critDraft, setCritDraft] = useState(null); + // Draft restart grace (seconds); committed on "Apply grace". + const [graceDraft, setGraceDraft] = useState(null); + const editingRef = useRef(false); + + useEffect(() => { + let cancelled = false; + const refresh = async () => { + const data = await loadReport(apiBase); + if (cancelled) return; + if (data) { + setReport(data); + setReportAt(Date.now()); + } + setLoaded(true); + }; + void refresh(); + const interval = setInterval(() => { + // Don't stomp the sliders the user is dragging. + if (!editingRef.current) void refresh(); + }, 5000); + return () => { cancelled = true; clearInterval(interval); }; + }, [apiBase]); + + const save = useCallback(async (patch: SettingsPatch) => { + setSaving(true); + setSaveError(null); + try { + const res = await fetch(`${apiBase}/api/memory/settings`, { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify(patch), + }); + if (!res.ok) { + const body = (await res.json().catch(() => null)) as { error?: string } | null; + setSaveError(body?.error ?? t("memory.httpError", { status: res.status })); + } + } catch (err) { + setSaveError(err instanceof Error ? err.message : String(err)); + } finally { + setSaving(false); + editingRef.current = false; + const data = await loadReport(apiBase); + if (data) setReport(data); + } + }, [apiBase, t]); + + if (!loaded) return null; + if (!report || report.enabled === false) { + return ( +
+
+ {t("memory.title")} + beta +
+

{t("memory.disabledHint")}

+ + {saveError &&
{saveError}
} +
+ ); + } + + const cfg = report.resolvedConfig; + const decision = report.decision; + const rec = report.recommendation; + const sup = report.supervisor; + const mem = report.memory ?? null; + const rs = report.responseState ?? null; + const probeAgeS = report.lastProbeAt == null || reportAt === 0 + ? null + : Math.max(0, Math.round((reportAt - report.lastProbeAt) / 1000)); + const level = decision?.level ?? "ok"; + const fraction = decision?.fraction ?? 0; + const warnPct = warnDraft ?? (cfg ? Math.round(cfg.warnFraction * 100) : 60); + const critPct = critDraft ?? (cfg ? Math.round(cfg.criticalFraction * 100) : 75); + const autoRestartBlocked = !!cfg && cfg.requireSupervisor && !(sup?.supervised); + const nf = (n: number) => n.toLocaleString(locale); + + // Restart grace: the server reports ms; the UI edits whole seconds. The relation guard mirrors + // the runtime's cooldown >= grace normalization so the dashboard cannot save a combination the + // server would have to correct. + const graceSeconds = Math.round((cfg?.restartGraceMs ?? 30_000) / 1000); + const cooldownSeconds = Math.round((cfg?.minRestartIntervalMs ?? 600_000) / 1000); + const graceValue = graceDraft ?? graceSeconds; + const graceInRange = Number.isFinite(graceValue) && graceValue >= GRACE_MIN_S && graceValue <= GRACE_MAX_S; + const graceUnderCooldown = graceValue <= cooldownSeconds; + const graceValid = graceInRange && graceUnderCooldown; + const graceChanged = graceDraft !== null && graceDraft !== graceSeconds; + + const applyThresholds = () => { + // Server clamps + keeps critical > warn; send fractions. + void save({ warnFraction: warnPct / 100, criticalFraction: Math.max(critPct, warnPct + 1) / 100 }); + }; + const applyGrace = () => { + if (!graceValid || !graceChanged) return; + void save({ restartGraceMs: Math.round(graceValue) * 1000 }); + }; + const applyRecommended = () => { + if (!rec) return; + setWarnDraft(Math.round(rec.warnFraction * 100)); + setCritDraft(Math.round(rec.criticalFraction * 100)); + void save({ + warnFraction: rec.warnFraction, + criticalFraction: rec.criticalFraction, + autoRestart: rec.autoRestart && !autoRestartBlocked ? true : false, + }); + }; + + return ( +
+
+ {t("memory.title")} + beta + {level.toUpperCase()} +
+

{t("memory.intro")}

+ + {/* 1. Monitor */} +
+
+ {t("memory.pressure")} + + {decision ? t("memory.pressureValue", { used: nf(decision.pressureMb), total: nf(decision.totalMb), pct: pct(fraction) }) : "—"} + +
+
+
+ {cfg && } + {cfg && } +
+
+ {t("memory.sourceLabel")} {decision?.source ?? "—"} + {t("memory.growthLabel")} {report.growthMbPerHour == null ? t("memory.notAvailable") : t("memory.growthValue", { mb: nf(report.growthMbPerHour) })} + {t("memory.restartsLabel")} {nf(report.restartCount ?? 0)} + {t("memory.supervisorLabel")} {sup?.supervised ? t("memory.supervisorDetected", { hint: sup.hint }) : t("memory.supervisorNotDetected")} +
+ {mem && ( +
+ {/* Actual Private Bytes vs the pressure value in the gauge above — deliberately distinct. */} + {t("memory.privateLabel")} {mem.processPrivateBytes === null ? t("memory.notAvailable") : t("memory.mbValue", { mb: nf(Math.round(mem.processPrivateBytes / MIB)) })} + {t("memory.commitLabel")} {mem.systemCommitAvailable && mem.systemCommittedBytes !== null && mem.systemCommitLimitBytes !== null + ? t("memory.commitValue", { used: gb1(mem.systemCommittedBytes), limit: gb1(mem.systemCommitLimitBytes), pct: mem.systemCommitFraction === null ? "?" : Math.round(mem.systemCommitFraction * 100) }) + : t("memory.notAvailable")} + {rs && ( + {t("memory.responseCacheLabel")} {t("memory.responseCacheValue", { count: nf(rs.count), mb: nf(Math.round(rs.totalBytes / MIB)) })} + )} + {t("memory.probeLabel")} {probeAgeS === null ? t("memory.notAvailable") : t("memory.probeAgo", { s: probeAgeS })}{mem.probeError && ( + {t("memory.probeDegraded", { code: mem.probeError })} + )} +
+ )} +
+ + {/* 2. Recommend */} + {rec && ( +
+
+ {t("memory.recommended")} + {t("memory.recSummary", { + warn: Math.round(rec.warnFraction * 100), + crit: Math.round(rec.criticalFraction * 100), + auto: rec.autoRestart ? t("memory.yes") : t("memory.no"), + })} + +
+
{rec.rationale}
+
+ )} + + {/* 3. Adjust */} +
+ + +
+ { editingRef.current = true; setWarnDraft(v); }} /> + { editingRef.current = true; setCritDraft(v); }} /> +
+ +
+
+ + + {autoRestartBlocked && ( +
{t("memory.autoRestartBlocked")}
+ )} +
{t("memory.autoRestartNote", { seconds: graceSeconds })}
+ + {!graceInRange && ( +
{t("memory.graceRangeError", { min: GRACE_MIN_S, max: GRACE_MAX_S })}
+ )} + {graceInRange && !graceUnderCooldown && ( +
{t("memory.graceCooldownError", { seconds: cooldownSeconds })}
+ )} + + +
+ {saving && {t("common.saving")}} + {!saving && saveError && {saveError}} +
+
+
+ ); +} + +const betaBadge: React.CSSProperties = { + background: "transparent", + color: "var(--muted)", + border: "1px solid var(--border, var(--muted))", + padding: "1px 7px", + borderRadius: "var(--radius-pill)", + fontSize: 11, + textTransform: "uppercase", + letterSpacing: 0.4, +}; + +const rowStyle: React.CSSProperties = { display: "flex", alignItems: "center", gap: 8, cursor: "pointer" }; + +function Marker({ at, color }: { at: number; color: string }) { + return
; +} + +function Slider({ label, value, min, max, onChange }: { label: string; value: number; min: number; max: number; onChange: (v: number) => void }) { + return ( + + ); +} diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 287b83e97..1593f9864 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -1,5 +1,47 @@ // German — generated from en.ts. Must match TKey set (compile-checked). export const de = { + // memory watchdog (beta) + "memory.title": "Speicher-Watchdog", + "memory.disabledHint": "Watchdog deaktiviert. Aktivieren Sie ihn, um den Speicherdruck zu beobachten und Warnungen zu erhalten.", + "memory.enable": "Aktivieren", + "memory.intro": "Prozessspeicher im Verhältnis zum Gesamt-RAM beobachten. Standardmäßig nur Warnung; Auto-Neustart ist optional und wird nur ausgelöst, wenn ein Supervisor erkannt wird.", + "memory.pressure": "Speicherdruck", + "memory.pressureValue": "{used} / {total} MB ({pct}%)", + "memory.sourceLabel": "Quelle:", + "memory.growthLabel": "Wachstum:", + "memory.growthValue": "{mb} MB/h", + "memory.notAvailable": "n/v", + "memory.restartsLabel": "Neustarts:", + "memory.privateLabel": "Privat:", + "memory.commitLabel": "System-Commit:", + "memory.commitValue": "{used} / {limit} GB ({pct}%)", + "memory.responseCacheLabel": "Antwort-Cache:", + "memory.responseCacheValue": "{count} Einträge · {mb} MB", + "memory.probeLabel": "Messung:", + "memory.probeAgo": "vor {s}s", + "memory.probeDegraded": "eingeschränkt ({code})", + "memory.mbValue": "{mb} MB", + "memory.supervisorLabel": "Supervisor:", + "memory.supervisorDetected": "erkannt ({hint})", + "memory.supervisorNotDetected": "nicht erkannt", + "memory.recommended": "Empfohlen", + "memory.recSummary": "Warnung {warn}% · kritisch {crit}% · Auto-Neustart {auto}", + "memory.applyRecommended": "Empfehlung übernehmen", + "memory.enabledLabel": "Aktiviert", + "memory.warnLabel": "Warnung %", + "memory.criticalLabel": "Kritisch %", + "memory.applyThresholds": "Schwellen übernehmen", + "memory.autoRestartLabel": "Auto-Neustart bei kritisch (optional)", + "memory.autoRestartNote": "Beim Start eines automatischen Neustarts werden neue Anfragen vorübergehend angehalten; laufende Antworten erhalten bis zu {seconds} Sekunden Zeit zum Abschluss. Eine Streaming-Antwort, die nicht rechtzeitig fertig wird, kann unterbrochen werden und muss ggf. vom Client wiederholt werden. Bereits abgeschlossene Antworten sind nicht betroffen; Neustarts sind ratenbegrenzt.", + "memory.graceLabel": "Neustart-Karenzzeit (Sekunden)", + "memory.applyGrace": "Karenzzeit übernehmen", + "memory.graceRangeError": "Wert zwischen {min} und {max} Sekunden eingeben.", + "memory.graceCooldownError": "Darf das minimale Neustart-Intervall ({seconds}s) nicht überschreiten.", + "memory.autoRestartBlocked": "Kein Supervisor erkannt — Auto-Neustart bleibt aus, damit der Proxy nie einfach beendet wird. Setzen Sie OCX_SUPERVISED=1 (oder betreiben Sie ihn unter pm2/systemd) oder deaktivieren Sie „Supervisor erforderlich“.", + "memory.requireSupervisorLabel": "Supervisor vor Auto-Neustart erforderlich", + "memory.yes": "ja", + "memory.no": "nein", + "memory.httpError": "HTTP {status}", "nav.dashboard": "Übersicht", "nav.providers": "Anbieter", "nav.models": "Modelle", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 3430c5ba0..41b0fc480 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -1,6 +1,48 @@ // English — source of truth. Its keys define the TKey type; ko/zh/ja must match (compile-checked). // Strings with {cmd} render a chip via ; {var} are plain interpolations. export const en = { + // memory watchdog (beta) + "memory.title": "Memory watchdog", + "memory.disabledHint": "Watchdog disabled. Enable it to observe memory pressure and receive warnings.", + "memory.enable": "Enable", + "memory.intro": "Observe process memory vs total RAM. Warn-only by default; auto-restart is opt-in and only fires when a supervisor is detected.", + "memory.pressure": "Memory pressure", + "memory.pressureValue": "{used} / {total} MB ({pct}%)", + "memory.sourceLabel": "source:", + "memory.growthLabel": "growth:", + "memory.growthValue": "{mb} MB/h", + "memory.notAvailable": "n/a", + "memory.restartsLabel": "restarts:", + "memory.privateLabel": "private:", + "memory.commitLabel": "system commit:", + "memory.commitValue": "{used} / {limit} GB ({pct}%)", + "memory.responseCacheLabel": "response cache:", + "memory.responseCacheValue": "{count} items · {mb} MB", + "memory.probeLabel": "probe:", + "memory.probeAgo": "{s}s ago", + "memory.probeDegraded": "degraded ({code})", + "memory.mbValue": "{mb} MB", + "memory.supervisorLabel": "supervisor:", + "memory.supervisorDetected": "detected ({hint})", + "memory.supervisorNotDetected": "not detected", + "memory.recommended": "Recommended", + "memory.recSummary": "warn {warn}% · critical {crit}% · auto-restart {auto}", + "memory.applyRecommended": "Apply recommended", + "memory.enabledLabel": "Enabled", + "memory.warnLabel": "Warn %", + "memory.criticalLabel": "Critical %", + "memory.applyThresholds": "Apply thresholds", + "memory.autoRestartLabel": "Auto-restart on critical (opt-in)", + "memory.autoRestartNote": "When automatic restart begins, new requests are paused and in-progress responses are given up to {seconds} seconds to finish. A streaming response that does not finish in time may be interrupted and may need to be retried by the client. Completed responses are unaffected, and restarts are rate-limited.", + "memory.graceLabel": "Restart grace period (seconds)", + "memory.applyGrace": "Apply grace", + "memory.graceRangeError": "Enter {min}–{max} seconds.", + "memory.graceCooldownError": "Cannot exceed the minimum restart interval ({seconds}s).", + "memory.autoRestartBlocked": "No supervisor detected — auto-restart stays off so the proxy is never left dead. Set OCX_SUPERVISED=1 (or run under pm2/systemd), or turn off \"Require supervisor\".", + "memory.requireSupervisorLabel": "Require supervisor before auto-restart", + "memory.yes": "yes", + "memory.no": "no", + "memory.httpError": "HTTP {status}", // sidebar / nav / common "nav.dashboard": "Dashboard", "nav.providers": "Providers", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index 0babac0bf..e5c4b9dc8 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -1,6 +1,48 @@ import type { TKey } from "./en"; export const ja: Record = { + // memory watchdog (beta) + "memory.title": "メモリウォッチドッグ", + "memory.disabledHint": "ウォッチドッグは無効です。有効にするとメモリ負荷を監視し警告を受け取れます。", + "memory.enable": "有効化", + "memory.intro": "総RAMに対するプロセスメモリを監視します。デフォルトは警告のみ。自動再起動はオプトインで、スーパーバイザーが検出された場合にのみ実行されます。", + "memory.pressure": "メモリ負荷", + "memory.pressureValue": "{used} / {total} MB ({pct}%)", + "memory.sourceLabel": "ソース:", + "memory.growthLabel": "増加:", + "memory.growthValue": "{mb} MB/h", + "memory.notAvailable": "なし", + "memory.restartsLabel": "再起動:", + "memory.privateLabel": "プライベート:", + "memory.commitLabel": "システムコミット:", + "memory.commitValue": "{used} / {limit} GB（{pct}%）", + "memory.responseCacheLabel": "応答キャッシュ:", + "memory.responseCacheValue": "{count} 件 · {mb} MB", + "memory.probeLabel": "計測:", + "memory.probeAgo": "{s}秒前", + "memory.probeDegraded": "制限あり（{code}）", + "memory.mbValue": "{mb} MB", + "memory.supervisorLabel": "スーパーバイザー:", + "memory.supervisorDetected": "検出 ({hint})", + "memory.supervisorNotDetected": "未検出", + "memory.recommended": "推奨", + "memory.recSummary": "警告 {warn}% · 危険 {crit}% · 自動再起動 {auto}", + "memory.applyRecommended": "推奨値を適用", + "memory.enabledLabel": "有効", + "memory.warnLabel": "警告 %", + "memory.criticalLabel": "危険 %", + "memory.applyThresholds": "しきい値を適用", + "memory.autoRestartLabel": "危険時に自動再起動（オプトイン）", + "memory.autoRestartNote": "自動再起動が始まると新規リクエストの受付を一時停止し、進行中の応答が完了するまで最大 {seconds} 秒待機します。この時間内に完了しなかったストリーミング応答は中断されることがあり、クライアント側での再試行が必要になる場合があります。すでに完了した応答には影響せず、再起動の頻度は制限されています。", + "memory.graceLabel": "再起動の猶予時間（秒）", + "memory.applyGrace": "猶予時間を適用", + "memory.graceRangeError": "{min}〜{max}秒の値を入力してください。", + "memory.graceCooldownError": "最小再起動間隔（{seconds}秒）を超えることはできません。", + "memory.autoRestartBlocked": "スーパーバイザーが検出されません — プロキシがそのまま停止しないよう自動再起動はオフのままです。OCX_SUPERVISED=1 を設定する（または pm2/systemd 下で実行する）か、「スーパーバイザーを必須にする」をオフにしてください。", + "memory.requireSupervisorLabel": "自動再起動の前にスーパーバイザーを必須にする", + "memory.yes": "はい", + "memory.no": "いいえ", + "memory.httpError": "HTTP {status}", // sidebar / nav / common "nav.dashboard": "ダッシュボード", "nav.providers": "プロバイダー", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index acbaf9750..0fc9bc0bf 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -1,6 +1,48 @@ import type { TKey } from "./en"; export const ko: Record = { + // memory watchdog (beta) + "memory.title": "메모리 워치독", + "memory.disabledHint": "워치독이 비활성화되었습니다. 메모리 압력을 관찰하고 경고를 받으려면 활성화하세요.", + "memory.enable": "활성화", + "memory.intro": "전체 RAM 대비 프로세스 메모리를 관찰합니다. 기본적으로 경고만 하며, 자동 재시작은 선택 사항이고 supervisor가 감지된 경우에만 실행됩니다.", + "memory.pressure": "메모리 압력", + "memory.pressureValue": "{used} / {total} MB ({pct}%)", + "memory.sourceLabel": "소스:", + "memory.growthLabel": "증가:", + "memory.growthValue": "{mb} MB/h", + "memory.notAvailable": "해당 없음", + "memory.restartsLabel": "재시작:", + "memory.privateLabel": "프라이빗:", + "memory.commitLabel": "시스템 커밋:", + "memory.commitValue": "{used} / {limit} GB ({pct}%)", + "memory.responseCacheLabel": "응답 캐시:", + "memory.responseCacheValue": "{count}개 · {mb} MB", + "memory.probeLabel": "측정:", + "memory.probeAgo": "{s}초 전", + "memory.probeDegraded": "제한됨 ({code})", + "memory.mbValue": "{mb} MB", + "memory.supervisorLabel": "supervisor:", + "memory.supervisorDetected": "감지됨 ({hint})", + "memory.supervisorNotDetected": "감지되지 않음", + "memory.recommended": "권장", + "memory.recSummary": "경고 {warn}% · 위험 {crit}% · 자동 재시작 {auto}", + "memory.applyRecommended": "권장값 적용", + "memory.enabledLabel": "활성화됨", + "memory.warnLabel": "경고 %", + "memory.criticalLabel": "위험 %", + "memory.applyThresholds": "임계값 적용", + "memory.autoRestartLabel": "위험 시 자동 재시작 (선택)", + "memory.autoRestartNote": "자동 재시작이 시작되면 새 요청 수신을 잠시 중단하고, 진행 중인 응답이 완료되도록 최대 {seconds}초까지 기다립니다. 이 시간 안에 끝나지 않은 스트리밍 응답은 중단될 수 있으며 클라이언트에서 재시도가 필요할 수 있습니다. 이미 완료된 응답에는 영향이 없으며, 재시작 빈도는 제한됩니다.", + "memory.graceLabel": "재시작 유예 시간 (초)", + "memory.applyGrace": "유예 시간 적용", + "memory.graceRangeError": "{min}–{max}초 사이의 값을 입력하세요.", + "memory.graceCooldownError": "최소 재시작 간격({seconds}초)을 초과할 수 없습니다.", + "memory.autoRestartBlocked": "supervisor가 감지되지 않았습니다 — 프록시가 그냥 종료되지 않도록 자동 재시작이 꺼진 상태로 유지됩니다. OCX_SUPERVISED=1을 설정하거나(또는 pm2/systemd에서 실행) \"supervisor 필요\"를 끄세요.", + "memory.requireSupervisorLabel": "자동 재시작 전에 supervisor 필요", + "memory.yes": "예", + "memory.no": "아니오", + "memory.httpError": "HTTP {status}", // sidebar / nav / common "nav.dashboard": "대시보드", "nav.providers": "프로바이더", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index a48724520..4ddd3e458 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -1,6 +1,48 @@ import type { TKey } from "./en"; export const ru: Record = { + // memory watchdog (beta) + "memory.title": "Сторож памяти", + "memory.disabledHint": "Сторож отключён. Включите его, чтобы отслеживать давление памяти и получать предупреждения.", + "memory.enable": "Включить", + "memory.intro": "Отслеживание памяти процесса относительно общего ОЗУ. По умолчанию только предупреждение; автоперезапуск включается по желанию и срабатывает только при обнаружении супервизора.", + "memory.pressure": "Давление памяти", + "memory.pressureValue": "{used} / {total} МБ ({pct}%)", + "memory.sourceLabel": "источник:", + "memory.growthLabel": "рост:", + "memory.growthValue": "{mb} МБ/ч", + "memory.notAvailable": "н/д", + "memory.restartsLabel": "перезапуски:", + "memory.privateLabel": "приватная:", + "memory.commitLabel": "системный коммит:", + "memory.commitValue": "{used} / {limit} ГБ ({pct}%)", + "memory.responseCacheLabel": "кэш ответов:", + "memory.responseCacheValue": "{count} записей · {mb} МБ", + "memory.probeLabel": "замер:", + "memory.probeAgo": "{s} с назад", + "memory.probeDegraded": "ограничен ({code})", + "memory.mbValue": "{mb} МБ", + "memory.supervisorLabel": "супервизор:", + "memory.supervisorDetected": "обнаружен ({hint})", + "memory.supervisorNotDetected": "не обнаружен", + "memory.recommended": "Рекомендуется", + "memory.recSummary": "предупр. {warn}% · критич. {crit}% · автоперезапуск {auto}", + "memory.applyRecommended": "Применить рекомендуемое", + "memory.enabledLabel": "Включено", + "memory.warnLabel": "Предупр. %", + "memory.criticalLabel": "Критич. %", + "memory.applyThresholds": "Применить пороги", + "memory.autoRestartLabel": "Автоперезапуск при критическом (по желанию)", + "memory.autoRestartNote": "При запуске автоматического перезапуска приём новых запросов временно приостанавливается, а выполняющимся ответам даётся до {seconds} секунд на завершение. Потоковый ответ, не завершившийся за это время, может быть прерван, и клиенту может потребоваться повторить запрос. Уже завершённые ответы не затрагиваются; частота перезапусков ограничена.", + "memory.graceLabel": "Время ожидания перед перезапуском (секунды)", + "memory.applyGrace": "Применить", + "memory.graceRangeError": "Укажите значение от {min} до {max} секунд.", + "memory.graceCooldownError": "Не может превышать минимальный интервал между перезапусками ({seconds} с).", + "memory.autoRestartBlocked": "Супервизор не обнаружен — автоперезапуск остаётся выключенным, чтобы прокси не завершился без восстановления. Установите OCX_SUPERVISED=1 (или запустите под pm2/systemd) либо отключите «Требовать супервизор».", + "memory.requireSupervisorLabel": "Требовать супервизор перед автоперезапуском", + "memory.yes": "да", + "memory.no": "нет", + "memory.httpError": "HTTP {status}", // sidebar / nav / common "nav.dashboard": "Дашборд", "nav.providers": "Провайдеры", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index d3ebea742..92fde29f2 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -1,6 +1,48 @@ import type { TKey } from "./en"; export const zh: Record = { + // memory watchdog (beta) + "memory.title": "内存看门狗", + "memory.disabledHint": "看门狗已禁用。启用后可观察内存压力并接收警告。", + "memory.enable": "启用", + "memory.intro": "观察进程内存与总内存的比例。默认仅警告；自动重启为可选项，且仅在检测到 supervisor 时触发。", + "memory.pressure": "内存压力", + "memory.pressureValue": "{used} / {total} MB ({pct}%)", + "memory.sourceLabel": "来源：", + "memory.growthLabel": "增长：", + "memory.growthValue": "{mb} MB/h", + "memory.notAvailable": "不可用", + "memory.restartsLabel": "重启：", + "memory.privateLabel": "专用内存：", + "memory.commitLabel": "系统提交：", + "memory.commitValue": "{used} / {limit} GB（{pct}%）", + "memory.responseCacheLabel": "响应缓存：", + "memory.responseCacheValue": "{count} 项 · {mb} MB", + "memory.probeLabel": "探测：", + "memory.probeAgo": "{s} 秒前", + "memory.probeDegraded": "受限（{code}）", + "memory.mbValue": "{mb} MB", + "memory.supervisorLabel": "supervisor：", + "memory.supervisorDetected": "已检测 ({hint})", + "memory.supervisorNotDetected": "未检测", + "memory.recommended": "推荐", + "memory.recSummary": "警告 {warn}% · 严重 {crit}% · 自动重启 {auto}", + "memory.applyRecommended": "应用推荐值", + "memory.enabledLabel": "已启用", + "memory.warnLabel": "警告 %", + "memory.criticalLabel": "严重 %", + "memory.applyThresholds": "应用阈值", + "memory.autoRestartLabel": "严重时自动重启（可选）", + "memory.autoRestartNote": "自动重启开始时会暂停接收新请求，并为进行中的响应留出最多 {seconds} 秒完成时间。未能在此时间内完成的流式响应可能被中断，客户端可能需要重试。已完成的响应不受影响，重启有频率限制。", + "memory.graceLabel": "重启宽限时间（秒）", + "memory.applyGrace": "应用宽限时间", + "memory.graceRangeError": "请输入 {min}–{max} 秒之间的值。", + "memory.graceCooldownError": "不能超过最小重启间隔（{seconds} 秒）。", + "memory.autoRestartBlocked": "未检测到 supervisor — 自动重启保持关闭，以免代理直接退出。请设置 OCX_SUPERVISED=1（或在 pm2/systemd 下运行），或关闭\"需要 supervisor\"。", + "memory.requireSupervisorLabel": "自动重启前需要 supervisor", + "memory.yes": "是", + "memory.no": "否", + "memory.httpError": "HTTP {status}", // sidebar / nav / common "nav.dashboard": "仪表盘", "nav.providers": "提供方", diff --git a/gui/src/pages/Dashboard.tsx b/gui/src/pages/Dashboard.tsx index 2d2bc5d14..459413ca8 100644 --- a/gui/src/pages/Dashboard.tsx +++ b/gui/src/pages/Dashboard.tsx @@ -5,6 +5,7 @@ import { Trans } from "../i18n/provider"; import { useI18n, type TKey } from "../i18n/shared"; import { formatTokens } from "../format-tokens"; import { EmptyState, Select } from "../ui"; +import MemoryCard from "../components/MemoryCard"; interface HealthData { status: string; version: string; uptime: number } interface ProviderInfo { name: string; adapter: string; baseUrl: string; defaultModel?: string; hasApiKey: boolean } @@ -617,6 +618,8 @@ export default function Dashboard({ apiBase }: { apiBase: string }) {
+ + {projectConfigWarnings.length > 0 && (
diff --git a/scripts/memory-stress-harness.ts b/scripts/memory-stress-harness.ts new file mode 100644 index 000000000..b6b365627 --- /dev/null +++ b/scripts/memory-stress-harness.ts @@ -0,0 +1,436 @@ +/** + * Isolated memory-stress harness for the responses continuation store. + * + * WHY: OpenCodex on Bun exhibits the same "committed memory >> working set + system stutter" + * signature that a sibling Bun CLI (Claude Code #36132) traced to the mimalloc allocator leaking + * committed memory ~0.6-1 GB/h, independent of the JS heap. Before touching the request path we + * need machine-local evidence that separates two hypotheses: + * (A) the JS-side continuation store (this repo) grows RAM, vs + * (B) the Bun native allocator retains memory the JS heap has already released. + * + * WHAT IT DOES (safe, isolated, accelerated): + * - Scenario "chain": one long previous_response_id chain -> proves prefix reference sharing + * (heapUsed grows ~O(N) while serialized totalBytes grows ~O(N^2)). + * - Scenario "sessions": many rotating independent chains with large payloads -> shows how much + * RAM the current COUNT-only cap (MAX_STORED_RESPONSES) still permits. + * - Return-to-OS check: clear the store + force GC, then resample RSS. If RSS does not fall, + * that is the native-retention (hypothesis B) signal. + * + * SAFETY: + * - Runs the workload in a CHILD process (parent orchestrates the A/B). + * - Child self-aborts if its own RSS exceeds --rss-cap-mb (default 1200 MB). + * - Parent enforces a hard wall-clock timeout and SIGKILLs the child if it overruns. + * - Uses an isolated OPENCODEX_HOME temp dir so it never touches the real ~/.opencodex. + * + * USAGE: + * bun run scripts/memory-stress-harness.ts # parent: runs the A/B and prints a table + * bun run scripts/memory-stress-harness.ts --child # single child run (used internally) + * + * TUNING (env, all optional): + * OCX_STRESS_CHAIN_TURNS, OCX_STRESS_CHAIN_PAYLOAD_BYTES, + * OCX_STRESS_SESSIONS, OCX_STRESS_SESSION_PAYLOAD_BYTES, + * OCX_STRESS_SAMPLE_EVERY, OCX_STRESS_RSS_CAP_MB, OCX_STRESS_PARENT_TIMEOUT_MS + */ + +import { spawnSync } from "node:child_process"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { buildResponseJSON } from "../src/bridge"; +import { captureMemorySnapshot } from "../src/lib/memory-usage"; +import { + clearResponseStateMemoryForTests, + expandPreviousResponseInput, + rememberResponseState, + responseStateMetrics, +} from "../src/responses/state"; + +const MODEL = "gpt-5.5"; +const MiB = 1024 * 1024; + +function envInt(name: string, fallback: number): number { + const raw = Number(process.env[name]); + return Number.isFinite(raw) && raw > 0 ? Math.floor(raw) : fallback; +} + +const CONFIG = { + chainTurns: envInt("OCX_STRESS_CHAIN_TURNS", 400), + chainPayloadBytes: envInt("OCX_STRESS_CHAIN_PAYLOAD_BYTES", 16 * 1024), + sessions: envInt("OCX_STRESS_SESSIONS", 1500), + sessionPayloadBytes: envInt("OCX_STRESS_SESSION_PAYLOAD_BYTES", 128 * 1024), + sampleEvery: envInt("OCX_STRESS_SAMPLE_EVERY", 50), + rssCapMb: envInt("OCX_STRESS_RSS_CAP_MB", 1200), + parentTimeoutMs: envInt("OCX_STRESS_PARENT_TIMEOUT_MS", 120_000), +}; + +// --------------------------------------------------------------------------- +// Shared helpers +// --------------------------------------------------------------------------- + +interface MemSample { + label: string; + rssMb: number; + heapUsedMb: number; + externalMb: number; + storeCount: number; + storeTotalMb: number; + storeLargestMb: number; +} + +function sample(label: string): MemSample { + const m = process.memoryUsage(); + const s = responseStateMetrics(); + return { + label, + rssMb: round(m.rss / MiB), + heapUsedMb: round(m.heapUsed / MiB), + externalMb: round(m.external / MiB), + storeCount: s.count, + storeTotalMb: round(s.totalBytes / MiB), + storeLargestMb: round(s.largestBytes / MiB), + }; +} + +function round(n: number): number { + return Math.round(n * 100) / 100; +} + +/** Best-effort force GC across runtimes: Bun.gc(true), then node --expose-gc global gc(). */ +function forceGc(): void { + const bun = (globalThis as { Bun?: { gc?: (sync: boolean) => void } }).Bun; + if (bun?.gc) bun.gc(true); + const g = (globalThis as { gc?: () => void }).gc; + if (g) g(); +} + +function overRssCap(): boolean { + return process.memoryUsage().rss / MiB > CONFIG.rssCapMb; +} + +function makePayload(bytes: number, tag: string): string { + // A recognizable prefix + filler; length dominates the serialized size. + const filler = "x".repeat(Math.max(0, bytes - tag.length)); + return `${tag}${filler}`; +} + +function buildTurnResponse(turnPayload: string): { id?: unknown; output?: unknown; status?: unknown } { + return buildResponseJSON( + [{ type: "text_delta", text: turnPayload }, { type: "done" }], + MODEL, + ) as { id?: unknown; output?: unknown; status?: unknown }; +} + +// --------------------------------------------------------------------------- +// Child role: run the workload, emit JSONL samples on stderr, SUMMARY on stdout +// --------------------------------------------------------------------------- + +interface ChildSummary { + label: string; + aborted: boolean; + abortReason?: string; + chain: { + turns: number; + heapUsedMb: number; + storeTotalMb: number; + approxUniqueContentMb: number; + // heapUsed / uniqueContent ~ 1 confirms sharing; storeTotal / heapUsed >> 1 confirms over-count. + heapToUniqueRatio: number; + storeToHeapRatio: number; + }; + sessions: { + created: number; + peakStoreCount: number; + peakStoreTotalMb: number; + peakRssMb: number; + }; + returnToOs: { + rssBeforeClearMb: number; + rssAfterClearGcMb: number; + reclaimedMb: number; + }; + /** + * Committed-memory probe points (§7): the field signature is PRIVATE/COMMITTED >> RSS, which the + * RSS-only columns above cannot see. Probed asynchronously at three points only (each Windows + * probe spawns a child, ~4-6s on a slow box) — start, before clear, after clear+GC. HONEST + * LIMIT: these columns verify the pattern is DETECTABLE on this machine; a passing run does not + * prove the leak cannot recur. + */ + committed: { + available: boolean; // false on non-Windows (v1) or probe failure + source: string; + startPrivateMb: number | null; + beforeClearPrivateMb: number | null; + afterClearPrivateMb: number | null; + afterClearRssMb: number; + /** Private/RSS after clear+GC — >> 1 is the native-retention (hypothesis B) detectability signal. */ + afterClearPrivateToRssRatio: number | null; + systemCommittedMb: number | null; + systemCommitLimitMb: number | null; + systemCommitFraction: number | null; + }; +} + +function emitSample(s: MemSample): void { + process.stderr.write(JSON.stringify({ kind: "sample", ...s }) + "\n"); +} + +interface CommittedProbe { + privateMb: number | null; + systemCommittedMb: number | null; + systemCommitLimitMb: number | null; + available: boolean; + source: string; +} + +async function probeCommitted(label: string): Promise { + const snap = await captureMemorySnapshot(); + const probe: CommittedProbe = { + privateMb: snap.processPrivateBytes === null ? null : round(snap.processPrivateBytes / MiB), + systemCommittedMb: snap.systemCommittedBytes === null ? null : round(snap.systemCommittedBytes / MiB), + systemCommitLimitMb: snap.systemCommitLimitBytes === null ? null : round(snap.systemCommitLimitBytes / MiB), + available: snap.systemCommitAvailable, + source: snap.processSource, + }; + process.stderr.write(JSON.stringify({ kind: "probe", label, ...probe }) + "\n"); + return probe; +} + +async function runChild(): Promise { + const label = process.env["OCX_STRESS_LABEL"] ?? "child"; + const summary: ChildSummary = { + label, + aborted: false, + chain: { + turns: 0, heapUsedMb: 0, storeTotalMb: 0, approxUniqueContentMb: 0, + heapToUniqueRatio: 0, storeToHeapRatio: 0, + }, + sessions: { created: 0, peakStoreCount: 0, peakStoreTotalMb: 0, peakRssMb: 0 }, + returnToOs: { rssBeforeClearMb: 0, rssAfterClearGcMb: 0, reclaimedMb: 0 }, + committed: { + available: false, source: "rss-fallback", + startPrivateMb: null, beforeClearPrivateMb: null, afterClearPrivateMb: null, + afterClearRssMb: 0, afterClearPrivateToRssRatio: null, + systemCommittedMb: null, systemCommitLimitMb: null, systemCommitFraction: null, + }, + }; + + const startProbe = await probeCommitted("start"); + + // --- Scenario A: one long chain (prefix reference sharing / C-type proof) --- + clearResponseStateMemoryForTests(); + forceGc(); + let prevId: string | undefined; + let turn = 0; + for (turn = 1; turn <= CONFIG.chainTurns; turn++) { + const payload = makePayload(CONFIG.chainPayloadBytes, `chain-t${turn}-`); + const body = prevId + ? { model: MODEL, previous_response_id: prevId, input: [{ role: "user", content: payload }], store: false } + : { model: MODEL, input: [{ role: "user", content: payload }], store: false }; + const expanded = expandPreviousResponseInput(body); + const resp = buildTurnResponse(payload); + rememberResponseState(expanded, resp, undefined, { force: true }); + prevId = resp.id as string; + if (turn % CONFIG.sampleEvery === 0) { + emitSample(sample(`chain@${turn}`)); + if (overRssCap()) { + summary.aborted = true; + summary.abortReason = `RSS cap ${CONFIG.rssCapMb}MB exceeded during chain at turn ${turn}`; + break; + } + } + } + forceGc(); + { + const s = sample(`chain-final@${turn}`); + emitSample(s); + // Unique content is created once per turn (input payload + output text of ~equal size), then + // shared by reference across every later entry: ~ 2 * turns * payload. + const approxUniqueContentMb = round((2 * (turn) * CONFIG.chainPayloadBytes) / MiB); + summary.chain = { + turns: turn, + heapUsedMb: s.heapUsedMb, + storeTotalMb: s.storeTotalMb, + approxUniqueContentMb, + heapToUniqueRatio: approxUniqueContentMb > 0 ? round(s.heapUsedMb / approxUniqueContentMb) : 0, + storeToHeapRatio: s.heapUsedMb > 0 ? round(s.storeTotalMb / s.heapUsedMb) : 0, + }; + } + + if (!summary.aborted) { + // --- Scenario B: many rotating independent sessions with large payloads --- + clearResponseStateMemoryForTests(); + forceGc(); + for (let i = 1; i <= CONFIG.sessions; i++) { + const payload = makePayload(CONFIG.sessionPayloadBytes, `sess-${i}-`); + const body = { model: MODEL, input: [{ role: "user", content: payload }], store: false }; + const resp = buildTurnResponse(payload); + rememberResponseState(body, resp, undefined, { force: true }); + summary.sessions.created = i; + if (i % CONFIG.sampleEvery === 0) { + const s = sample(`sessions@${i}`); + emitSample(s); + summary.sessions.peakStoreCount = Math.max(summary.sessions.peakStoreCount, s.storeCount); + summary.sessions.peakStoreTotalMb = Math.max(summary.sessions.peakStoreTotalMb, s.storeTotalMb); + summary.sessions.peakRssMb = Math.max(summary.sessions.peakRssMb, s.rssMb); + if (overRssCap()) { + summary.aborted = true; + summary.abortReason = `RSS cap ${CONFIG.rssCapMb}MB exceeded during sessions at ${i}`; + break; + } + } + } + } + + // --- Return-to-OS check: clear + GC, then see if RSS (and Private) actually fall --- + const beforeClearProbe = await probeCommitted("before-clear"); + const rssBeforeClear = round(process.memoryUsage().rss / MiB); + clearResponseStateMemoryForTests(); + forceGc(); + // Give the allocator a beat to purge. + await new Promise(resolve => setTimeout(resolve, 250)); + forceGc(); + const rssAfterClearGc = round(process.memoryUsage().rss / MiB); + summary.returnToOs = { + rssBeforeClearMb: rssBeforeClear, + rssAfterClearGcMb: rssAfterClearGc, + reclaimedMb: round(rssBeforeClear - rssAfterClearGc), + }; + + // The detectability signal for the field incident: committed/private staying high after the JS + // heap released everything, i.e. Private >> RSS post-clear. + const afterClearProbe = await probeCommitted("after-clear-gc"); + summary.committed = { + available: afterClearProbe.available || beforeClearProbe.available || startProbe.available, + source: afterClearProbe.source, + startPrivateMb: startProbe.privateMb, + beforeClearPrivateMb: beforeClearProbe.privateMb, + afterClearPrivateMb: afterClearProbe.privateMb, + afterClearRssMb: rssAfterClearGc, + afterClearPrivateToRssRatio: afterClearProbe.privateMb !== null && rssAfterClearGc > 0 + ? round(afterClearProbe.privateMb / rssAfterClearGc) + : null, + systemCommittedMb: afterClearProbe.systemCommittedMb, + systemCommitLimitMb: afterClearProbe.systemCommitLimitMb, + systemCommitFraction: afterClearProbe.systemCommittedMb !== null && afterClearProbe.systemCommitLimitMb !== null && afterClearProbe.systemCommitLimitMb > 0 + ? round(afterClearProbe.systemCommittedMb / afterClearProbe.systemCommitLimitMb) + : null, + }; + + process.stdout.write("SUMMARY " + JSON.stringify(summary) + "\n"); +} + +// --------------------------------------------------------------------------- +// Parent role: orchestrate the A/B, enforce timeout, print a comparison table +// --------------------------------------------------------------------------- + +interface Overlay { label: string; env: Record; } + +function runParent(): void { + const home = mkdtempSync(join(tmpdir(), "ocx-stress-")); + // A/B overlays. The mimalloc knobs are best-effort: if Bun's bundled mimalloc ignores them the + // two rows will simply match, which is itself informative (rules the lever out). + const overlays: Overlay[] = [ + { label: "bun-default", env: {} }, + { label: "mimalloc-purge0", env: { MIMALLOC_PURGE_DELAY: "0", MIMALLOC_PAGE_RESET: "1" } }, + ]; + + console.log("== OpenCodex memory-stress harness =="); + console.log(`bun: ${process.versions?.bun ?? "?"} node-compat: ${process.version} platform: ${process.platform}`); + console.log("config:", JSON.stringify(CONFIG)); + console.log(`isolated OPENCODEX_HOME: ${home}`); + console.log(""); + + const summaries: ChildSummary[] = []; + for (const overlay of overlays) { + process.stdout.write(`-- running scenario [${overlay.label}] ...\n`); + const res = spawnSync( + process.execPath, + [__filename, "--child"], + { + env: { + ...process.env, + ...overlay.env, + OPENCODEX_HOME: home, + OCX_STRESS_LABEL: overlay.label, + }, + timeout: CONFIG.parentTimeoutMs, + killSignal: "SIGKILL", + encoding: "utf8", + maxBuffer: 64 * MiB, + }, + ); + if (res.error) { + console.log(` [${overlay.label}] spawn error: ${res.error.message}`); + continue; + } + if (res.signal) { + console.log(` [${overlay.label}] killed by ${res.signal} (likely parent timeout / safety)`); + } + const summary = parseSummary(res.stdout ?? ""); + if (summary) summaries.push(summary); + else console.log(` [${overlay.label}] no SUMMARY produced. stderr tail:\n${tail(res.stderr ?? "", 5)}`); + } + + rmSync(home, { recursive: true, force: true }); + printReport(summaries); +} + +function parseSummary(stdout: string): ChildSummary | null { + for (const line of stdout.split("\n")) { + if (line.startsWith("SUMMARY ")) { + try { return JSON.parse(line.slice("SUMMARY ".length)) as ChildSummary; } catch { return null; } + } + } + return null; +} + +function tail(text: string, lines: number): string { + return text.split("\n").filter(Boolean).slice(-lines).join("\n"); +} + +function printReport(summaries: ChildSummary[]): void { + console.log("\n================= RESULTS =================\n"); + for (const s of summaries) { + console.log(`### scenario: ${s.label}${s.aborted ? " (ABORTED: " + s.abortReason + ")" : ""}`); + console.log(" chain (prefix-sharing / C-type):"); + console.log(` turns=${s.chain.turns} heapUsed=${s.chain.heapUsedMb}MB storeTotal=${s.chain.storeTotalMb}MB approxUniqueContent=${s.chain.approxUniqueContentMb}MB`); + console.log(` heapUsed/uniqueContent=${s.chain.heapToUniqueRatio} (~1 => content shared by reference)`); + console.log(` storeTotal/heapUsed=${s.chain.storeToHeapRatio} (>>1 => serialized metric over-counts shared prefixes)`); + console.log(" sessions (count-only cap headroom):"); + console.log(` created=${s.sessions.created} peakStoreCount=${s.sessions.peakStoreCount} peakStoreTotal=${s.sessions.peakStoreTotalMb}MB peakRss=${s.sessions.peakRssMb}MB`); + console.log(" return-to-OS (native-retention signal):"); + console.log(` rssBeforeClear=${s.returnToOs.rssBeforeClearMb}MB rssAfterClear+GC=${s.returnToOs.rssAfterClearGcMb}MB reclaimed=${s.returnToOs.reclaimedMb}MB`); + console.log(" committed probes (Private>>RSS detectability; NOT a non-regression proof):"); + if (s.committed.available) { + console.log(` private: start=${s.committed.startPrivateMb}MB beforeClear=${s.committed.beforeClearPrivateMb}MB afterClear+GC=${s.committed.afterClearPrivateMb}MB`); + console.log(` afterClear private/RSS=${s.committed.afterClearPrivateToRssRatio} (>>1 => committed retained past the JS release)`); + console.log(` system commit: ${s.committed.systemCommittedMb}MB / ${s.committed.systemCommitLimitMb}MB (fraction=${s.committed.systemCommitFraction})`); + } else { + console.log(` unavailable on this platform/run (source=${s.committed.source}) — RSS-only columns above still apply`); + } + console.log(""); + } + if (summaries.length >= 2) { + const [a, b] = summaries; + console.log("### A/B (allocator lever)"); + console.log(` ${a.label} peakRss=${a.sessions.peakRssMb}MB reclaimed=${a.returnToOs.reclaimedMb}MB`); + console.log(` ${b.label} peakRss=${b.sessions.peakRssMb}MB reclaimed=${b.returnToOs.reclaimedMb}MB`); + console.log(" (if reclaimed is small in BOTH, RSS is retained after JS release => hypothesis B / native allocator)"); + } + console.log("\n========================================="); + console.log("Interpretation guide:"); + console.log(" - heapUsed/uniqueContent ~ 1 => items are shared by reference; a delta-chain refactor buys little."); + console.log(" - storeTotal/heapUsed >> 1 => the JSON-serialized byte metric over-counts (upper bound), as designed."); + console.log(" - sessions peakRss high with peakStoreCount pinned at the count cap => a BYTE budget would help (hygiene)."); + console.log(" - reclaimed ~ 0 after clear+GC => native retention dominates => the fix is a watchdog + preemptive restart."); + console.log(" - afterClear private/RSS >> 1 => the committed>>RSS field signature is DETECTABLE here (the watchdog's"); + console.log(" windows-private source would see it). A clean run does NOT prove the leak cannot recur."); +} + +// --------------------------------------------------------------------------- + +if (process.argv.includes("--child")) { + await runChild(); +} else { + runParent(); +} diff --git a/src/lib/memory-usage.ts b/src/lib/memory-usage.ts new file mode 100644 index 000000000..987c2dafe --- /dev/null +++ b/src/lib/memory-usage.ts @@ -0,0 +1,331 @@ +/** + * Portable process + system memory snapshot for the memory watchdog. + * + * WHY: OpenCodex on Bun can hit the "committed memory >> working set" signature (field report: + * bun.exe private/committed ~79 GB while the working set stayed ~5.8 GB, freezing the box once the + * system commit charge was exhausted). The Bun/mimalloc native allocator retains committed memory + * the JS heap has already released — proven locally by the return-to-OS stress harness and mirrored + * upstream by Claude Code #36132. A watchdog that only reads RSS would MISS that case entirely, so + * we read the platform's private/committed counter where it is cheaply available and fall back to + * RSS elsewhere. The same field incident showed the SYSTEM commit charge (~97% of the commit + * limit) mattered more than any single process's share of physical RAM, so the Windows probe also + * collects the system-wide Committed Bytes / Commit Limit for a separate observe-and-warn axis. + * + * SAFETY: the Windows probe runs a child PowerShell process ASYNCHRONOUSLY (Bun.spawn, array + * arguments, no shell string, hidden window) — it never blocks the event loop; a slow probe only + * delays its own completion. On timeout the child is killed and the caller receives an RSS + * fallback snapshot; a late result after the timeout already resolved is dropped + * (first-resolution-wins). Linux reads /proc synchronously (no spawn); every other platform uses + * the in-process RSS fallback. All process-pressure thresholds are relative to physical RAM so the + * same code behaves identically on a 16 GB laptop and a 128 GB worker. + */ + +import { readFileSync } from "node:fs"; +import { platform as osPlatform, totalmem } from "node:os"; + +/** Where the process-pressure value came from (always present — the value itself never is null). */ +export type ProcessPressureSource = "windows-private" | "proc-status" | "rss-fallback"; + +export interface MemorySnapshot { + /** Resident set size (physically mapped) in bytes. */ + rssBytes: number; + /** V8/JSC heap in use, bytes. */ + heapUsedBytes: number; + /** Off-heap native allocations tracked by the runtime, bytes. */ + externalBytes: number; + /** Total physical RAM, bytes — the denominator for the process-pressure thresholds. */ + physicalMemoryBytes: number; + + /** Actual Windows Private Bytes for this PID; null on every other platform / probe failure. */ + processPrivateBytes: number | null; + /** + * The bytes the watchdog treats as process pressure. Windows: Private Bytes; Linux: VmRSS+VmSwap + * from /proc; otherwise (and on probe failure): RSS. Always present — no null fallback needed. + */ + processPressureBytes: number; + /** Which of the above sources filled processPressureBytes. */ + processSource: ProcessPressureSource; + + /** System-wide commit charge (bytes); Windows-only in v1, null elsewhere / on failure. */ + systemCommittedBytes: number | null; + /** System commit limit (RAM + page file, bytes); null when not measured. */ + systemCommitLimitBytes: number | null; + /** True only when BOTH system commit values above were freshly measured. */ + systemCommitAvailable: boolean; + /** Available physical memory (bytes) — auxiliary diagnostics only. */ + availablePhysicalBytes: number | null; + + /** Caller's clock when the capture BEGAN (the watchdog stamps probe completion separately). */ + capturedAt: number; + /** Sanitized probe failure code (never raw command output — may echo paths/usernames). */ + probeError?: string; +} + +/** + * Async budget for the Windows probe. Generous compared to the old 2s synchronous cap: the async + * probe cannot block the event loop, so the only cost of a longer budget is a later evaluation — + * while a tight budget would spuriously degrade slow machines to the RSS fallback. Measured on a + * Windows 11 box: cold powershell.exe spawn + one CIM query = 4.0–5.9s per run, so 15s gives + * ~2.5x headroom over the worst observation while staying well inside the 60s sampling interval. + */ +const WINDOWS_PROBE_TIMEOUT_MS = 15_000; + +type PlatformOverride = string | null; +let platformOverride: PlatformOverride = null; + +/** Test seam: force the platform gate (e.g. "linux") so probe selection is deterministic. */ +export function setMemoryPlatformForTests(value: PlatformOverride): void { + platformOverride = value; +} + +function effectivePlatform(): string { + return platformOverride ?? osPlatform(); +} + +// --------------------------------------------------------------------------- +// Windows probe (async, guarded child process) +// --------------------------------------------------------------------------- + +export interface WindowsProbeResult { + privateBytes: number | null; + systemCommittedBytes: number | null; + systemCommitLimitBytes: number | null; + availablePhysicalBytes: number | null; + timedOut: boolean; + /** Sanitized failure code ("timeout" | "spawn-failed" | "probe-exit-N" | "parse-failed" | ...). */ + error?: string; +} + +type WindowsProbeRunner = (pid: number, timeoutMs: number, signal?: AbortSignal) => Promise; + +const EMPTY_PROBE: Omit = { + privateBytes: null, + systemCommittedBytes: null, + systemCommitLimitBytes: null, + availablePhysicalBytes: null, +}; + +interface SpawnedProbe { + stdout: ReadableStream; + exited: Promise; + exitCode: number | null; + kill(): void; +} + +/** + * One PowerShell spawn per probe collecting four numbers, one per line: + * this PID's Private Bytes, then the system's CommittedBytes / CommitLimit / AvailableBytes. + * Win32_PerfRawData_PerfOS_Memory mirrors the \Memory\Committed Bytes & Commit Limit performance + * counters and reports ALL THREE system values in bytes (verified on Windows 11) — no unit + * normalization needed. Arguments are an array (no shell string; the PID comes from process.pid), + * the window is hidden, and the profile is skipped, matching the icacls runner's hardening. + */ +async function defaultWindowsProbeRunner(pid: number, timeoutMs: number, signal?: AbortSignal): Promise { + const bun = (globalThis as { Bun?: { spawn?: (cmd: string[], opts: Record) => SpawnedProbe } }).Bun; + if (!bun?.spawn) return { ...EMPTY_PROBE, timedOut: false, error: "spawn-unavailable" }; + if (signal?.aborted) return { ...EMPTY_PROBE, timedOut: false, error: "aborted" }; + + let proc: SpawnedProbe; + try { + proc = bun.spawn( + [ + "powershell.exe", + "-NoProfile", + "-NonInteractive", + "-Command", + // Labeled output: a $null value would render as an empty line that trim() removes, shifting + // positional indexes — labels make the parse order-independent and loss-explicit. + `$p=(Get-Process -Id ${pid}).PrivateMemorySize64; ` + + `$m=Get-CimInstance Win32_PerfRawData_PerfOS_Memory; ` + + `Write-Output "P=$p" "C=$($m.CommittedBytes)" "L=$($m.CommitLimit)" "A=$($m.AvailableBytes)"`, + ], + { stdin: "ignore", stdout: "pipe", stderr: "ignore", windowsHide: true }, + ); + } catch { + return { ...EMPTY_PROBE, timedOut: false, error: "spawn-failed" }; + } + + // First-resolution-wins: the timeout (or an abort from stopMemoryWatchdog) kills the child and + // resolves the fallback; a late read completion after that is dropped (the spec's + // probeId/consumed invariant, expressed as a race). + let settled = false; + return new Promise(resolve => { + const finish = (result: WindowsProbeResult) => { + if (settled) return; + settled = true; + clearTimeout(timer); + signal?.removeEventListener("abort", onAbort); + resolve(result); + }; + const onAbort = () => { + try { proc.kill(); } catch { /* already gone */ } + finish({ ...EMPTY_PROBE, timedOut: false, error: "aborted" }); + }; + signal?.addEventListener("abort", onAbort, { once: true }); + const timer = setTimeout(() => { + try { proc.kill(); } catch { /* already gone */ } + finish({ ...EMPTY_PROBE, timedOut: true, error: "timeout" }); + }, timeoutMs); + (timer as { unref?: () => void }).unref?.(); + + void (async () => { + try { + const text = await new Response(proc.stdout).text(); + await proc.exited; + if (proc.exitCode !== 0) { + finish({ ...EMPTY_PROBE, timedOut: false, error: `probe-exit-${proc.exitCode ?? "unknown"}` }); + return; + } + finish(parseWindowsProbeOutput(text)); + } catch { + finish({ ...EMPTY_PROBE, timedOut: false, error: "probe-failed" }); + } + })(); + }); +} + +/** + * Parse the labeled probe output (`P=` private, `C=` committed, `L=` limit, `A=` available). + * Order-independent and tolerant of missing/blank labels — a value PowerShell rendered as empty + * (`P=`) parses to NaN and degrades to null instead of shifting the other fields. Exported as a + * pure function so the parse path is unit-testable without spawning a child. + */ +export function parseWindowsProbeOutput(text: string): WindowsProbeResult { + const fields = new Map(); + for (const line of text.trim().split(/\r?\n/)) { + const match = /^([PCLA])=(.*)$/.exec(line.trim()); + if (match) fields.set(match[1]!, Number(match[2])); + } + const val = (n: number | undefined): number | null => + n !== undefined && Number.isFinite(n) && n > 0 ? n : null; + const result: WindowsProbeResult = { + privateBytes: val(fields.get("P")), + systemCommittedBytes: val(fields.get("C")), + systemCommitLimitBytes: val(fields.get("L")), + availablePhysicalBytes: val(fields.get("A")), + timedOut: false, + }; + if (result.privateBytes === null && result.systemCommittedBytes === null) result.error = "parse-failed"; + return result; +} + +let windowsProbeRunner: WindowsProbeRunner = defaultWindowsProbeRunner; + +/** Test seam: replace the Windows probe runner. Pass null to restore the default. */ +export function setWindowsProbeRunnerForTests(runner: WindowsProbeRunner | null): void { + windowsProbeRunner = runner ?? defaultWindowsProbeRunner; +} + +// --------------------------------------------------------------------------- +// Linux /proc probe (no spawn) +// --------------------------------------------------------------------------- + +type ProcStatusReader = () => string | null; + +function defaultProcStatusReader(): string | null { + try { + return readFileSync("/proc/self/status", "utf8"); + } catch { + return null; + } +} + +let procStatusReader: ProcStatusReader = defaultProcStatusReader; + +/** Test seam: replace the /proc/self/status reader. Pass null to restore the default. */ +export function setProcStatusReaderForTests(reader: ProcStatusReader | null): void { + procStatusReader = reader ?? defaultProcStatusReader; +} + +/** Parse VmRSS + VmSwap (kB) out of /proc/self/status → private committed bytes, or null. */ +export function parseProcStatusCommitted(text: string): number | null { + const field = (name: string): number | null => { + const match = new RegExp(`^${name}:\\s*(\\d+)\\s*kB`, "m").exec(text); + return match ? Number(match[1]) * 1024 : null; + }; + const rss = field("VmRSS"); + if (rss === null) return null; + const swap = field("VmSwap") ?? 0; + return rss + swap; +} + +// --------------------------------------------------------------------------- +// Snapshot assembly +// --------------------------------------------------------------------------- + +function baseSnapshot(capturedAt: number): MemorySnapshot { + const usage = process.memoryUsage(); + return { + rssBytes: usage.rss, + heapUsedBytes: usage.heapUsed, + externalBytes: usage.external, + physicalMemoryBytes: totalmem(), + processPrivateBytes: null, + processPressureBytes: usage.rss, + processSource: "rss-fallback", + systemCommittedBytes: null, + systemCommitLimitBytes: null, + systemCommitAvailable: false, + availablePhysicalBytes: null, + capturedAt, + }; +} + +/** + * RSS-fallback snapshot for callers that must evaluate SOMETHING even when capture itself failed. + * captureMemorySnapshot is contractually non-throwing, but the probe loop uses this as a safety + * net so an unexpected runtime exception never silences a whole evaluation cycle (audit #2). + * probeError takes sanitized codes only — never raw command output or exception text. + */ +export function rssFallbackSnapshot(nowMs: number = Date.now(), probeError?: string): MemorySnapshot { + const snapshot = baseSnapshot(nowMs); + if (probeError !== undefined) snapshot.probeError = probeError; + return snapshot; +} + +/** + * Capture one memory snapshot. ASYNC because the Windows path awaits a child-process probe; it + * never throws and never blocks the event loop. Probe failures degrade to the RSS fallback with a + * sanitized probeError code — degraded, but always evaluable. + */ +export async function captureMemorySnapshot(nowMs: number = Date.now(), signal?: AbortSignal): Promise { + const plat = effectivePlatform(); + + if (plat === "win32") { + let probe: WindowsProbeResult; + try { + probe = await windowsProbeRunner(process.pid, WINDOWS_PROBE_TIMEOUT_MS, signal); + } catch { + probe = { ...EMPTY_PROBE, timedOut: false, error: "probe-failed" }; + } + const snapshot = baseSnapshot(nowMs); + if (probe.privateBytes !== null) { + snapshot.processPrivateBytes = probe.privateBytes; + snapshot.processPressureBytes = probe.privateBytes; + snapshot.processSource = "windows-private"; + } + snapshot.systemCommittedBytes = probe.systemCommittedBytes; + snapshot.systemCommitLimitBytes = probe.systemCommitLimitBytes; + snapshot.systemCommitAvailable = probe.systemCommittedBytes !== null && probe.systemCommitLimitBytes !== null; + snapshot.availablePhysicalBytes = probe.availablePhysicalBytes; + if (probe.error) snapshot.probeError = probe.error; + return snapshot; + } + + if (plat === "linux") { + const snapshot = baseSnapshot(nowMs); + const text = procStatusReader(); + const committed = text ? parseProcStatusCommitted(text) : null; + if (committed !== null) { + snapshot.processPressureBytes = committed; + snapshot.processSource = "proc-status"; + } else { + snapshot.probeError = "proc-status-unavailable"; + } + // v1: system commit collection is Windows-only. Linux could read /proc/meminfo + // Committed_AS/CommitLimit (no spawn) later; systemCommitAvailable=false covers it null-safely. + return snapshot; + } + + return baseSnapshot(nowMs); +} diff --git a/src/responses/state.ts b/src/responses/state.ts index ff4a84024..d3de71be8 100644 --- a/src/responses/state.ts +++ b/src/responses/state.ts @@ -179,6 +179,50 @@ export function previousResponseProviderState(responseId: string | undefined): O return providers ? structuredClone(providers) : undefined; } +export interface ResponseStateMetrics { + /** Number of retained continuation entries currently in RAM. */ + count: number; + /** Sum of serialized item bytes across all entries. Because expanded previous_response_id chains + * share prefix item references, this is an UPPER bound on true heap (it re-counts shared history), + * never an under-count — safe to use as a memory-pressure signal. */ + totalBytes: number; + /** Serialized item bytes of the single largest entry (a long chain's head, or an image-heavy turn). */ + largestBytes: number; + /** Age of the oldest retained entry, in ms (0 when the store is empty). */ + oldestAgeMs: number; +} + +/** + * Observe-only snapshot of the in-RAM continuation store. Additive and side-effect free — it does + * NOT lazy-load the disk snapshot, prune, or evict — so a diagnostics endpoint or benchmark harness + * can sample it without perturbing request handling. This is the instrumentation seam for deciding + * whether RAM growth originates in this store (JS heap) or in the runtime allocator (native). + */ +export function responseStateMetrics(): ResponseStateMetrics { + const at = now(); + let totalBytes = 0; + let largestBytes = 0; + let oldestCreatedAt = at; + for (const state of states.values()) { + let bytes = 0; + try { + bytes = JSON.stringify(state.items).length; + if (state.providers) bytes += JSON.stringify(state.providers).length; + } catch { + /* unserializable entry (should not happen): count as 0 rather than throw in a metrics path */ + } + totalBytes += bytes; + if (bytes > largestBytes) largestBytes = bytes; + if (state.createdAt < oldestCreatedAt) oldestCreatedAt = state.createdAt; + } + return { + count: states.size, + totalBytes, + largestBytes, + oldestAgeMs: states.size > 0 ? at - oldestCreatedAt : 0, + }; +} + export function rememberResponseState( requestBody: unknown, response: { id?: unknown; output?: unknown; status?: unknown }, diff --git a/src/server/index.ts b/src/server/index.ts index c79896d1c..7b895e724 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -114,6 +114,8 @@ export { jsonResponse, safeConfigDTO, } from "./auth-cors"; +import { startMemoryWatchdog } from "./memory-watchdog"; +export { memoryWatchdogSnapshot } from "./memory-watchdog"; import { disableResponsesRequestTimeout, handleResponses, handleResponsesCompact } from "./responses"; export { disableResponsesRequestTimeout, linkAbortSignal } from "./responses"; import { handleClaudeCountTokens, handleClaudeMessages } from "./claude-messages"; @@ -641,6 +643,11 @@ export function startServer(port?: number) { const actualPort = server.port ?? listenPort; setCorsOrigin(actualPort); + // Memory watchdog: warn (or, opt-in, gracefully restart) before the Bun/mimalloc native allocator + // can exhaust the system commit charge. Warn-only by default; the interval is unref'd so it never + // holds the process open on its own. Never throws — a watchdog failure must not block startup. + try { startMemoryWatchdog(config); } catch { /* observability is best-effort */ } + console.log(`🚀 opencodex proxy running on http://localhost:${actualPort}`); console.log(` POST /v1/responses → provider translation`); console.log(` POST /v1/chat/completions → OpenAI-compatible clients`); diff --git a/src/server/lifecycle.ts b/src/server/lifecycle.ts index 39f97400d..40320c205 100644 --- a/src/server/lifecycle.ts +++ b/src/server/lifecycle.ts @@ -48,6 +48,17 @@ export function trackStreamLifetime( }); } +/** + * Drain in-flight turns (rejecting new ones via the draining flag), then abort whatever outlived + * the deadline, flush the replay-state snapshot, and stop the server. timeoutMs is a DEADLINE, + * not a delay: the loop returns the moment the in-flight set empties. + * + * Concurrent invocations (e.g. a signal-path shutdown racing the memory watchdog's restart) are + * safe without extra locking: both poll the same turn set, flushResponseState is a no-op when + * nothing is pending, server.stop is idempotent, and whichever caller exits the process first + * wins. Callers that exit afterwards do so synchronously in the same microtask as their drain's + * resolution, so the draining=false reset below cannot let a new request slip in before exit. + */ export async function drainAndShutdown( server: ReturnType | undefined, timeoutMs: number, diff --git a/src/server/management-api.ts b/src/server/management-api.ts index 609d3dba1..7d25d0fec 100644 --- a/src/server/management-api.ts +++ b/src/server/management-api.ts @@ -50,6 +50,14 @@ import { } from "../lib/debug-settings"; import type { OcxClaudeCodeConfig, OcxConfig, OcxCustomModel, OcxProviderConfig } from "../types"; import { drainAndShutdown } from "./lifecycle"; +import { + applyWatchdogRuntimeConfig, + memoryWatchdogReport, + RESTART_GRACE_MAX_MS, + RESTART_GRACE_MIN_MS, + startMemoryWatchdog, + stopMemoryWatchdog, +} from "./memory-watchdog"; import { filterRequestLogs, getRequestLogEntries, type RequestLogEntry } from "./request-log"; import { estimateComboCost, estimateRequestCost, normalizeCostTokens, tokensPerSecond } from "../usage/cost"; import type { PersistedUsageAttempt } from "../usage/log"; @@ -249,6 +257,51 @@ export async function handleManagementAPI(req: Request, url: URL, config: OcxCon return jsonResponse({ ok: true, codexAutoStart: codexAutoStartEnabled(config) }); } + if (url.pathname === "/api/memory" && req.method === "GET") { + // Read-only Monitor + Recommend report. Degrades to { enabled: false } on disabled/old servers. + return jsonResponse(memoryWatchdogReport() ?? { enabled: false }); + } + + if (url.pathname === "/api/memory/settings" && req.method === "PUT") { + let raw: unknown; + try { raw = await req.json(); } catch { return jsonResponse({ error: "invalid JSON body" }, 400); } + if (!isPlainRecord(raw)) return jsonResponse({ error: "body must be a JSON object" }, 400); + const b = raw as Record; + const isBool = (v: unknown) => typeof v === "boolean"; + const isPosNum = (v: unknown) => typeof v === "number" && Number.isFinite(v) && v > 0; + const isFrac = (v: unknown) => typeof v === "number" && Number.isFinite(v) && v > 0 && v < 1; + if (b.enabled !== undefined && !isBool(b.enabled)) return jsonResponse({ error: "enabled must be boolean" }, 400); + if (b.autoRestart !== undefined && !isBool(b.autoRestart)) return jsonResponse({ error: "autoRestart must be boolean" }, 400); + if (b.requireSupervisor !== undefined && !isBool(b.requireSupervisor)) return jsonResponse({ error: "requireSupervisor must be boolean" }, 400); + if (b.intervalMs !== undefined && !isPosNum(b.intervalMs)) return jsonResponse({ error: "intervalMs must be a positive number" }, 400); + if (b.restartGraceMs !== undefined && (!isPosNum(b.restartGraceMs) || (b.restartGraceMs as number) < RESTART_GRACE_MIN_MS || (b.restartGraceMs as number) > RESTART_GRACE_MAX_MS)) { + return jsonResponse({ error: `restartGraceMs must be a number between ${RESTART_GRACE_MIN_MS} and ${RESTART_GRACE_MAX_MS} ms` }, 400); + } + if (b.warnFraction !== undefined && !isFrac(b.warnFraction)) return jsonResponse({ error: "warnFraction must be a number in (0,1)" }, 400); + if (b.criticalFraction !== undefined && !isFrac(b.criticalFraction)) return jsonResponse({ error: "criticalFraction must be a number in (0,1)" }, 400); + + const patch: NonNullable = {}; + if (b.enabled !== undefined) patch.enabled = b.enabled as boolean; + if (b.intervalMs !== undefined) patch.intervalMs = b.intervalMs as number; + if (b.restartGraceMs !== undefined) patch.restartGraceMs = b.restartGraceMs as number; + if (b.warnFraction !== undefined) patch.warnFraction = b.warnFraction as number; + if (b.criticalFraction !== undefined) patch.criticalFraction = b.criticalFraction as number; + if (b.autoRestart !== undefined) patch.autoRestart = b.autoRestart as boolean; + if (b.requireSupervisor !== undefined) patch.requireSupervisor = b.requireSupervisor as boolean; + + config.memoryWatchdog = { ...config.memoryWatchdog, ...patch }; + saveConfig(config); + + // Apply live so the change takes effect without a proxy restart. + if (patch.enabled === false) { + stopMemoryWatchdog(); + } else if (applyWatchdogRuntimeConfig(patch) === null) { + // Not currently running (was disabled / first enable) — (re)start from the saved config. + startMemoryWatchdog(config); + } + return jsonResponse({ ok: true, memoryWatchdog: config.memoryWatchdog, report: memoryWatchdogReport() ?? { enabled: false } }); + } + if (url.pathname === "/api/diagnostics/project-config" && req.method === "GET") { const { getCachedProjectConfigDiagnostics } = await import("../codex/project-config-warnings"); const { warnings, grouped } = getCachedProjectConfigDiagnostics(); diff --git a/src/server/memory-restart-history.ts b/src/server/memory-restart-history.ts new file mode 100644 index 000000000..e4be4e608 --- /dev/null +++ b/src/server/memory-restart-history.ts @@ -0,0 +1,97 @@ +/** + * Cross-process memory-restart history (best-effort). + * + * WHY: the watchdog's cooldown (minRestartIntervalMs) and cap (maxRestarts) live in process + * memory, and a memory-driven restart ENDS the process — so without persistence every freshly + * respawned proxy would start with a clean slate and the two guards could never bind across the + * very restarts they exist to rate-limit. This module persists ONLY the restart decision + * timestamps (epoch ms — no request data, no config values, no secrets) so a new process can + * seed its cooldown clock and its restarts-in-window count. + * + * SAFETY MODEL: + * - Best-effort cache, never a source of truth: every read/write failure (missing file, + * corrupt JSON, wrong schema, permission error) is swallowed. Degradation is exactly the + * pre-persistence behavior — per-process guards plus whatever restart throttling the + * supervisor applies. History I/O must never block or fail a restart, or app startup. + * - Write is temp + renameAtomicFile (the vetted Windows EBUSY/EPERM-retry rename). + * atomicWriteFile is deliberately NOT reused: it NTFS-hardens with required:true (icacls), + * which exists for secret-bearing files and can stall for seconds on Windows — the wrong + * trade for a timestamps-only file written while memory is already critical. + * - Bounded: entries older than RESTART_HISTORY_WINDOW_MS (or in the future — a clock + * rollback must not pin the cooldown forever) are dropped on every read, and the file + * never holds more than HISTORY_MAX_ENTRIES entries. + */ + +import { existsSync, mkdirSync, readFileSync, unlinkSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { getConfigDir, renameAtomicFile } from "../config"; + +/** + * Rolling window over which persisted restarts count toward maxRestarts. Six hours: with the + * default 10-minute cooldown a pathological workload can burn maxRestarts (3) in ~30 minutes; + * after that the watchdog degrades to warn-only until the oldest entry ages out, instead of + * churning restarts all day. A leak that only re-crosses critical hours later still restarts. + */ +export const RESTART_HISTORY_WINDOW_MS = 6 * 3_600_000; + +const HISTORY_MAX_ENTRIES = 20; +const HISTORY_VERSION = 1; + +let pathOverride: string | null = null; + +/** Test seam: redirect the history file. Pass null to restore the default location. */ +export function setRestartHistoryPathForTests(path: string | null): void { + pathOverride = path; +} + +function historyPath(): string { + return pathOverride ?? join(getConfigDir(), "memory-watchdog-restarts.json"); +} + +/** Read, validate, and prune the persisted timestamps. Any failure yields []. */ +function readTimestamps(nowMs: number): number[] { + try { + const path = historyPath(); + if (!existsSync(path)) return []; + const raw = JSON.parse(readFileSync(path, "utf-8")) as { version?: unknown; restarts?: unknown }; + if (raw.version !== HISTORY_VERSION || !Array.isArray(raw.restarts)) return []; + return raw.restarts + .filter((t): t is number => typeof t === "number" && Number.isFinite(t)) + .filter(t => t <= nowMs && t > nowMs - RESTART_HISTORY_WINDOW_MS) + .sort((a, b) => a - b); + } catch { + return []; + } +} + +export interface SeededRestartHistory { + /** Newest persisted decision timestamp (seeds the cooldown clock), or 0 when none. */ + lastRestartAt: number; + /** Persisted restarts inside the rolling window (seeds the maxRestarts count). */ + recentCount: number; +} + +/** Seed values for a fresh process. Missing/corrupt history degrades to { 0, 0 }. */ +export function loadMemoryRestartHistory(nowMs: number): SeededRestartHistory { + const ts = readTimestamps(nowMs); + return { lastRestartAt: ts.length > 0 ? ts[ts.length - 1]! : 0, recentCount: ts.length }; +} + +/** Append a restart decision timestamp. Best-effort: every failure is swallowed. */ +export function recordMemoryRestart(atMs: number): void { + try { + const restarts = [...readTimestamps(atMs), atMs].slice(-HISTORY_MAX_ENTRIES); + const path = historyPath(); + mkdirSync(dirname(path), { recursive: true }); + const tmp = `${path}.${process.pid}.tmp`; + try { + writeFileSync(tmp, JSON.stringify({ version: HISTORY_VERSION, restarts }), "utf-8"); + renameAtomicFile(tmp, path); + } catch (err) { + try { unlinkSync(tmp); } catch { /* already gone */ } + throw err; + } + } catch { + /* best-effort: history must never block the restart itself */ + } +} diff --git a/src/server/memory-watchdog.ts b/src/server/memory-watchdog.ts new file mode 100644 index 000000000..97cd91b3e --- /dev/null +++ b/src/server/memory-watchdog.ts @@ -0,0 +1,780 @@ +/** + * Memory watchdog: observe + warn by default, opt-in graceful restart. + * + * WHY: the Bun/mimalloc native allocator can retain committed memory the JS heap already released + * (field: private/committed ~79 GB, working set ~5.8 GB, box frozen at the system commit limit; + * mirrored by Claude Code #36132). No in-app JS cache cap fixes native retention — the only proven + * mitigation is to notice the pressure and, optionally, hand off to the already-tested graceful + * restart path. This module NEVER decides thresholds in absolute GB: every trigger is a fraction of + * total system RAM, so one code path behaves the same on a 16 GB laptop and a 128 GB worker. + * + * DRIVER SHAPE (async collect, sync decide): a self-rescheduling probe loop captures snapshots + * asynchronously (the Windows probe is a child process — never a synchronous event-loop block) and + * each completed probe triggers exactly one evaluation. The first probe fires immediately at + * start; a generation guard drops late probe results after stop/replace. + * + * SAFETY MODEL: + * - Default action is WARN ONLY. It cannot lose data or kill the process. + * - SYSTEM-COMMIT axis is observe-only (v1): crossing the high-water logs one latched warning + * and never arms a restart — the commit pressure may come from another process, and + * restarting OpenCodex would not free it. Auto-restart integration waits for field + * measurement (see docs). + * - Auto-restart is opt-in (config.memoryWatchdog.autoRestart / env). When it fires it reuses + * drainAndShutdown() — which drains in-flight turns and flushes the response-state snapshot — + * then exits with a distinct code so a supervisor can respawn. + * - Quiet-window restart: the drain budget for a memory-driven restart is restartGraceMs (default + * 30s), much longer than a generic shutdown. Because drainAndShutdown rejects new turns and exits + * the instant the in-flight set empties, the restart normally lands on a natural idle gap — so a + * running turn is only cut when it outlives the whole grace window (bounded so the restart always + * fires). A mid-stream turn that is still cut is regenerated on retry (context is preserved via + * the flushed previous_response_id snapshot); the provider generation itself cannot be resumed. + * - Restart-loop guards, honestly scoped: a cooldown (minRestartIntervalMs, normalized to at + * least restartGraceMs) and a max-restart cap limit repeated restart REQUESTS. Both counters + * live in process memory, and a fired restart ends the process — so on their own they cannot + * bind across the process boundary. A small best-effort history file (timestamps only, see + * memory-restart-history.ts) re-seeds them in the respawned process; when that file cannot be + * read or written, cross-boundary protection falls back to the SUPERVISOR's own restart + * limit/backoff policy, which operators should configure regardless. maxRestarts is therefore + * a rolling-window rate limit (RESTART_HISTORY_WINDOW_MS), not a permanent all-time cap. + * - Exit code 75 (EX_TEMPFAIL) is a REQUEST to the supervisor to respawn — nothing more. A + * supervisor that treats it as a normal exit will simply leave the proxy stopped. + * - The decision core (evaluate) is pure and injectable (clock + reader + restart hook) so the + * logic is unit-tested without real timers or a real process exit. + */ + +import { captureMemorySnapshot, rssFallbackSnapshot, type MemorySnapshot } from "../lib/memory-usage"; +import { responseStateMetrics, type ResponseStateMetrics } from "../responses/state"; +import { drainAndShutdown } from "./lifecycle"; +import { loadMemoryRestartHistory, recordMemoryRestart } from "./memory-restart-history"; +import type { OcxConfig } from "../types"; + +/** Exit code used for an intentional memory-driven restart (distinct from crash/normal exit). */ +export const MEMORY_RESTART_EXIT_CODE = 75; // EX_TEMPFAIL: "transient, retry" — a supervisor should respawn. + +/** + * Hard bounds for the quiet-window drain budget. The floor keeps a "grace" meaningful (below 1s + * it is an immediate abort); the ceiling bounds how long a restart may hold the proxy in the + * draining state (new requests 503) when a stuck turn never finishes — 10 minutes, matching the + * default cooldown so the default config stays self-consistent even at the extreme. + */ +export const RESTART_GRACE_MIN_MS = 1_000; +export const RESTART_GRACE_MAX_MS = 600_000; + +const MiB = 1024 * 1024; + +export interface ResolvedWatchdogConfig { + enabled: boolean; + intervalMs: number; + warnFraction: number; + criticalFraction: number; + autoRestart: boolean; + /** When true, auto-restart only fires if a supervisor is detected (so we never exit into "just dead"). */ + requireSupervisor: boolean; + minRestartIntervalMs: number; + maxRestarts: number; + /** Drain budget (ms) for a memory-driven restart: wait up to this long for in-flight turns to + * finish before aborting, so the restart lands on a natural idle gap (quiet-window). */ + restartGraceMs: number; + /** + * System-commit high-water fraction for the OBSERVE-ONLY warning axis (v1). Crossing it logs a + * separate warning; it NEVER arms a restart — the cause may be another process, so restarting + * OpenCodex would not help. Internal default 0.90; env-only override + * (OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER, experimental — for field measurement, not the UI). + */ + systemCommitHighWater: number; + growthWindowMs: number; +} + +const DEFAULTS: ResolvedWatchdogConfig = { + enabled: true, + intervalMs: 60_000, + warnFraction: 0.60, + criticalFraction: 0.75, + autoRestart: false, + requireSupervisor: true, + minRestartIntervalMs: 600_000, // 10 min + maxRestarts: 3, + restartGraceMs: 30_000, // quiet-window: wait up to 30s for in-flight turns before aborting + systemCommitHighWater: 0.90, // conservative: a healthy system sits well below (field baseline ~0.2-0.7) + growthWindowMs: 600_000, // 10 min growth-rate window (diagnostic) +}; + +/** Parse a boolean-ish string; undefined when unset/unrecognized. */ +function parseFlagValue(raw: string | undefined): boolean | undefined { + const v = raw?.trim().toLowerCase(); + if (v === undefined || v === "") return undefined; + if (v === "1" || v === "true" || v === "yes" || v === "on") return true; + if (v === "0" || v === "false" || v === "no" || v === "off") return false; + return undefined; +} + +function envFlag(name: string): boolean | undefined { + return parseFlagValue(process.env[name]); +} + +function envNum(name: string): number | undefined { + const raw = process.env[name]?.trim(); + if (!raw) return undefined; + const value = Number(raw); + return Number.isFinite(value) && value > 0 ? value : undefined; +} + +function clampFraction(value: number | undefined, fallback: number): number { + if (value === undefined || !Number.isFinite(value)) return fallback; + return Math.min(0.99, Math.max(0.10, value)); +} + +/** Positive finite number, else fallback — config-file values bypass envNum's guard. */ +function posNumOr(value: number | undefined, fallback: number): number { + return value !== undefined && Number.isFinite(value) && value > 0 ? value : fallback; +} + +/** Positive finite duration clamped into [min, max]; anything else falls back. */ +function clampMs(value: number | undefined, min: number, max: number, fallback: number): number { + if (value === undefined || !Number.isFinite(value) || value <= 0) return fallback; + return Math.min(max, Math.max(min, value)); +} + +/** + * cooldown >= grace: a shorter cooldown could let the watchdog arm a second restart while the + * first drain is still inside its grace window. defaultRestart also stops the sampling timer, so + * for the default path this is belt-and-braces — but injected restart hooks (tests, embedders) + * rely on it. Normalization (raise the cooldown) mirrors the warn/critical nudge and is the safe + * direction: it never extends the 503 drain window, only spaces restarts further apart. + */ +function normalizeRestartTiming(cfg: ResolvedWatchdogConfig): ResolvedWatchdogConfig { + if (cfg.minRestartIntervalMs < cfg.restartGraceMs) cfg.minRestartIntervalMs = cfg.restartGraceMs; + return cfg; +} + +/** + * Resolve config → runtime knobs. Precedence: env override > config file > default. Fractions are + * clamped to [0.10, 0.99]; a criticalFraction <= warnFraction is nudged above warn so the two + * levels never collapse. Durations are validated the same way on every entry path (env, config + * file, management API): non-finite/non-positive values fall back, restartGraceMs is clamped into + * [RESTART_GRACE_MIN_MS, RESTART_GRACE_MAX_MS], and the cooldown is raised to at least the grace. + */ +export function resolveWatchdogConfig(config: OcxConfig): ResolvedWatchdogConfig { + const c = config.memoryWatchdog ?? {}; + const enabled = envFlag("OCX_MEMORY_WATCHDOG_DISABLED") === true + ? false + : (envFlag("OCX_MEMORY_WATCHDOG_ENABLED") ?? c.enabled ?? DEFAULTS.enabled); + + const warnFraction = clampFraction( + envNum("OCX_MEMORY_WATCHDOG_WARN_FRACTION") ?? c.warnFraction, + DEFAULTS.warnFraction, + ); + let criticalFraction = clampFraction( + envNum("OCX_MEMORY_WATCHDOG_CRITICAL_FRACTION") ?? c.criticalFraction, + DEFAULTS.criticalFraction, + ); + if (criticalFraction <= warnFraction) criticalFraction = Math.min(0.99, warnFraction + 0.10); + + const maxRestartsRaw = envNum("OCX_MEMORY_WATCHDOG_MAX_RESTARTS") ?? c.maxRestarts; + return normalizeRestartTiming({ + enabled, + // Sub-second sampling is pure CPU churn for a signal that moves over minutes — floor at 1s. + intervalMs: Math.max(1_000, posNumOr(envNum("OCX_MEMORY_WATCHDOG_INTERVAL_MS") ?? c.intervalMs, DEFAULTS.intervalMs)), + warnFraction, + criticalFraction, + autoRestart: envFlag("OCX_MEMORY_WATCHDOG_AUTO_RESTART") ?? c.autoRestart ?? DEFAULTS.autoRestart, + requireSupervisor: + envFlag("OCX_MEMORY_WATCHDOG_REQUIRE_SUPERVISOR") ?? c.requireSupervisor ?? DEFAULTS.requireSupervisor, + minRestartIntervalMs: + posNumOr(envNum("OCX_MEMORY_WATCHDOG_MIN_RESTART_INTERVAL_MS") ?? c.minRestartIntervalMs, DEFAULTS.minRestartIntervalMs), + // 0 is meaningful ("never auto-restart"), so only reject negatives/non-finite here. + maxRestarts: maxRestartsRaw !== undefined && Number.isFinite(maxRestartsRaw) && maxRestartsRaw >= 0 + ? Math.floor(maxRestartsRaw) + : DEFAULTS.maxRestarts, + restartGraceMs: clampMs( + envNum("OCX_MEMORY_WATCHDOG_RESTART_GRACE_MS") ?? c.restartGraceMs, + RESTART_GRACE_MIN_MS, RESTART_GRACE_MAX_MS, DEFAULTS.restartGraceMs, + ), + // Env-only (experimental): no config-file/UI knob until the axis is validated by field + // measurement — a persisted setting for an unproven threshold would just be dead config. + systemCommitHighWater: (() => { + const raw = envNum("OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER"); + if (raw === undefined) return DEFAULTS.systemCommitHighWater; + return Math.min(0.99, Math.max(0.50, raw)); + })(), + growthWindowMs: DEFAULTS.growthWindowMs, + }); +} + +// --------------------------------------------------------------------------- +// Supervisor detection +// --------------------------------------------------------------------------- + +/** + * Best-effort detection of an external process supervisor that will respawn us after an intentional + * exit. Pure over an env map so it is unit-testable. Heuristics: explicit OCX_SUPERVISED flag, pm2 + * (pm_id / PM2_HOME), systemd (INVOCATION_ID / NOTIFY_SOCKET). When nothing matches we assume NO + * supervisor — the safe default, because exiting without a respawner is worse than staying up. + */ +export function detectSupervisor( + env: Record = process.env, +): { supervised: boolean; hint: string } { + const explicit = parseFlagValue(env.OCX_SUPERVISED); + if (explicit === true) return { supervised: true, hint: "OCX_SUPERVISED" }; + if (env.pm_id !== undefined || (env.PM2_HOME ?? "") !== "") return { supervised: true, hint: "pm2" }; + if ((env.INVOCATION_ID ?? "") !== "" || (env.NOTIFY_SOCKET ?? "") !== "") return { supervised: true, hint: "systemd" }; + if (explicit === false) return { supervised: false, hint: "OCX_SUPERVISED=off" }; + return { supervised: false, hint: "none" }; +} + +// --------------------------------------------------------------------------- +// Pure decision core +// --------------------------------------------------------------------------- + +export type WatchdogLevel = "ok" | "warn" | "critical"; +export type WatchdogAction = "none" | "warn" | "restart"; + +export interface WatchdogState { + samples: { t: number; bytes: number; fraction: number }[]; + /** Highest level already reported since the last drop back to ok — prevents per-interval log spam. */ + reportedLevel: WatchdogLevel; + /** + * Latch for the observe-only system-commit warning: set on the first high-water crossing, + * re-armed only when a MEASURED fraction drops back below the high-water. A probe that merely + * fails to measure (systemCommitAvailable=false) holds the latch — measurement loss is not + * recovery, and a flapping probe must not re-warn every time it comes back. + */ + commitReported: boolean; + restartCount: number; + lastRestartAt: number; +} + +export interface WatchdogDecision { + level: WatchdogLevel; + action: WatchdogAction; + fraction: number; + pressureMb: number; + totalMb: number; + source: string; + growthMbPerHour: number | null; + reason: string; + /** Actual Windows Private Bytes (MB); null when the platform/probe does not provide it. */ + processPrivateMb: number | null; + /** System commit axis (observe-only): all null when the probe did not measure it. */ + systemCommitFraction: number | null; + systemCommitUsedMb: number | null; + systemCommitLimitMb: number | null; + /** "warn" exactly once per high-water crossing; NEVER "restart" — see §6 of the v1 spec. */ + commitAction: "none" | "warn"; + commitReason: string; +} + +export function createWatchdogState(): WatchdogState { + return { samples: [], reportedLevel: "ok", commitReported: false, restartCount: 0, lastRestartAt: 0 }; +} + +function levelFor(fraction: number, cfg: ResolvedWatchdogConfig): WatchdogLevel { + if (fraction >= cfg.criticalFraction) return "critical"; + if (fraction >= cfg.warnFraction) return "warn"; + return "ok"; +} + +/** bytes/hour over the retained window, or null when there is not enough spread to be meaningful. */ +function growthBytesPerHour(state: WatchdogState): number | null { + if (state.samples.length < 2) return null; + const first = state.samples[0]!; + const last = state.samples[state.samples.length - 1]!; + const dtMs = last.t - first.t; + if (dtMs <= 0) return null; + return ((last.bytes - first.bytes) / dtMs) * 3_600_000; +} + +/** + * Pure evaluation of one snapshot. Mutates `state` (sample ring, report latches, restart + * bookkeeping) and returns the decision. Two independent axes: + * - PROCESS pressure (processPressureBytes / physicalMemoryBytes): unchanged semantics — emits + * action "restart" only when level is critical AND auto-restart is enabled AND the cooldown + + * max-restart guards allow it; otherwise a critical level degrades to a "warn" action. + * - SYSTEM commit (systemCommittedBytes / systemCommitLimitBytes): observe-only. Crossing the + * high-water emits commitAction "warn" exactly once (latched, re-armed on measured recovery); + * it NEVER contributes to the restart decision — the cause may be another process, and + * restarting OpenCodex would not free it. Skipped null-safely when not measured. + */ +export function evaluate( + state: WatchdogState, + snapshot: MemorySnapshot, + cfg: ResolvedWatchdogConfig, + nowMs: number, + supervised = true, +): WatchdogDecision { + const bytes = snapshot.processPressureBytes; + const total = snapshot.physicalMemoryBytes > 0 ? snapshot.physicalMemoryBytes : bytes; + const fraction = total > 0 ? bytes / total : 0; + + state.samples.push({ t: nowMs, bytes, fraction }); + const cutoff = nowMs - cfg.growthWindowMs; + while (state.samples.length > 2 && state.samples[0]!.t < cutoff) state.samples.shift(); + + const level = levelFor(fraction, cfg); + const growth = growthBytesPerHour(state); + + let action: WatchdogAction = "none"; + let reason = ""; + + if (level === "ok") { + state.reportedLevel = "ok"; // recovered — re-arm warnings for the next crossing + } else if (level === "warn") { + if (state.reportedLevel === "ok") { action = "warn"; reason = "crossed warn fraction"; } + state.reportedLevel = state.reportedLevel === "critical" ? "critical" : "warn"; + } else { + // critical + const cooledDown = nowMs - state.lastRestartAt >= cfg.minRestartIntervalMs || state.lastRestartAt === 0; + const underCap = state.restartCount < cfg.maxRestarts; + const supervisorOk = !cfg.requireSupervisor || supervised; + if (cfg.autoRestart && supervisorOk && cooledDown && underCap) { + action = "restart"; + reason = "crossed critical fraction; auto-restart armed"; + state.restartCount += 1; + state.lastRestartAt = nowMs; + } else { + if (state.reportedLevel !== "critical") { action = "warn"; } + reason = !cfg.autoRestart + ? "critical; auto-restart disabled" + : !supervisorOk + ? "critical; auto-restart suppressed: no supervisor" + : underCap + ? "critical; restart deferred by cooldown" + : "critical; max-restart guard reached"; + } + state.reportedLevel = "critical"; + } + + // System-commit axis (observe-only, null-safe). Never touches `action` above. + let commitAction: "none" | "warn" = "none"; + let commitReason = ""; + let commitFraction: number | null = null; + if ( + snapshot.systemCommitAvailable + && snapshot.systemCommittedBytes !== null + && snapshot.systemCommitLimitBytes !== null + && snapshot.systemCommitLimitBytes > 0 + ) { + commitFraction = snapshot.systemCommittedBytes / snapshot.systemCommitLimitBytes; + if (commitFraction >= cfg.systemCommitHighWater) { + if (!state.commitReported) { + commitAction = "warn"; + commitReason = "system commit charge crossed high-water (observe-only; the cause may be another process — no restart)"; + state.commitReported = true; + } + } else { + state.commitReported = false; // measured recovery below high-water → re-arm + } + } + // Not measured (non-Windows / probe failure): skip entirely — the latch holds either way. + + return { + level, + action, + fraction, + pressureMb: Math.round(bytes / MiB), + totalMb: Math.round(total / MiB), + source: snapshot.processSource, + growthMbPerHour: growth === null ? null : Math.round(growth / MiB), + reason, + processPrivateMb: snapshot.processPrivateBytes === null ? null : Math.round(snapshot.processPrivateBytes / MiB), + systemCommitFraction: commitFraction, + systemCommitUsedMb: snapshot.systemCommittedBytes === null ? null : Math.round(snapshot.systemCommittedBytes / MiB), + systemCommitLimitMb: snapshot.systemCommitLimitBytes === null ? null : Math.round(snapshot.systemCommitLimitBytes / MiB), + commitAction, + commitReason, + }; +} + +// --------------------------------------------------------------------------- +// Recommendation (advisory only — never mutates config or state) +// --------------------------------------------------------------------------- + +export interface WatchdogRecommendation { + warnFraction: number; + criticalFraction: number; + autoRestart: boolean; + rationale: string; +} + +/** + * Suggest thresholds from the observed steady-state. Advisory only: it computes numbers a human (or + * the Adjust UI) can choose to apply, and never touches config or running state. Warn is placed a + * margin above the observed peak fraction so it does not false-fire on normal usage; critical sits a + * further margin above warn. autoRestart is only suggested when a supervisor exists AND memory is + * trending up (a flat process does not benefit from restarts). With < 2 samples it echoes the + * current config and says so. + */ +export function recommend( + state: WatchdogState, + cfg: ResolvedWatchdogConfig, + supervised: boolean, +): WatchdogRecommendation { + if (state.samples.length < 2) { + return { + warnFraction: cfg.warnFraction, + criticalFraction: cfg.criticalFraction, + autoRestart: cfg.autoRestart, + rationale: "insufficient samples; keeping current thresholds", + }; + } + const observedMax = Math.max(...state.samples.map(s => s.fraction)); + const clamp = (v: number) => Math.min(0.99, Math.max(0.10, v)); + const warnFraction = clamp(observedMax + 0.10); + const criticalFraction = clamp(Math.max(warnFraction + 0.10, observedMax + 0.20)); + const growth = growthBytesPerHour(state); + const trendingUp = growth !== null && growth > 0; + const autoRestart = supervised && trendingUp; + const growthMbH = growth === null ? "n/a" : `${Math.round(growth / MiB)}MB/h`; + const rationale = + `observed peak ${(observedMax * 100).toFixed(1)}% of RAM over ${state.samples.length} samples; ` + + `growth ${growthMbH}; supervisor ${supervised ? "detected" : "not detected"} ` + + `→ autoRestart ${autoRestart ? "suggested" : "not suggested"}`; + return { warnFraction, criticalFraction, autoRestart, rationale }; +} + +// --------------------------------------------------------------------------- +// Runtime driver (timer + logging + restart hook) +// --------------------------------------------------------------------------- + +export interface WatchdogDeps { + now: () => number; + /** Async snapshot capture (Windows probes a child process). Must not throw or block the loop. */ + capture: (nowMs: number, signal?: AbortSignal) => Promise; + /** Whether an external supervisor will respawn us; gates auto-restart when requireSupervisor is on. */ + supervised: boolean; + /** Perform the graceful restart. Default: drain + flush snapshot, then exit with the restart code. */ + restart: (cfg: ResolvedWatchdogConfig) => void | Promise; + log: (line: string) => void; + /** Persist a restart decision timestamp (cross-process guard seed). Optional so tests stay disk-free. */ + recordRestart?: (atMs: number) => void; +} + +/** One restart owns the exit: concurrent or repeated invocations must not start a second drain. */ +let restartInFlight = false; + +function defaultRestart(cfg: ResolvedWatchdogConfig): void { + if (restartInFlight) return; + restartInFlight = true; + // The decision is made — stop sampling so mid-drain ticks can neither spam logs nor (with a + // mis-tuned cooldown) arm a second restart while this drain is still inside its grace window. + stopMemoryWatchdog(); + void (async () => { + try { + // Quiet-window: drainAndShutdown rejects new turns and returns the moment the in-flight set + // empties, so this waits for a natural idle gap and only aborts turns that outlive the budget. + await drainAndShutdown(undefined, cfg.restartGraceMs); + } finally { + // drainAndShutdown resets its draining flag, but nothing can interleave here: the server is + // already stopped and this exit runs in the same microtask chain as the drain's resolution. + process.exit(MEMORY_RESTART_EXIT_CODE); + } + })(); +} + +const DEFAULT_DEPS: WatchdogDeps = { + now: Date.now, + capture: (nowMs, signal) => captureMemorySnapshot(nowMs, signal), + supervised: detectSupervisor().supervised, + restart: defaultRestart, + log: line => console.warn(line), + recordRestart: recordMemoryRestart, +}; + +function formatLine(d: WatchdogDecision): string { + const pct = (d.fraction * 100).toFixed(1); + const growth = d.growthMbPerHour === null ? "n/a" : `${d.growthMbPerHour}MB/h`; + return `[opencodex] memory watchdog ${d.level.toUpperCase()}: ${d.pressureMb}MB / ${d.totalMb}MB (${pct}% of RAM, source=${d.source}, growth=${growth}) — ${d.reason}`; +} + +function formatCommitLine(d: WatchdogDecision): string { + const pct = d.systemCommitFraction === null ? "?" : (d.systemCommitFraction * 100).toFixed(1); + return `[opencodex] memory watchdog SYSTEM-COMMIT: ${d.systemCommitUsedMb}MB / ${d.systemCommitLimitMb}MB (${pct}% of commit limit) — ${d.commitReason}`; +} + +interface RunningWatchdog { + /** Monotonic start id — a late probe completion from a stopped/replaced watchdog is ignored. */ + generation: number; + /** Pending self-rescheduled probe timer; null while a probe is in flight. */ + timer: ReturnType | null; + /** Aborts the in-flight capture (kills the child probe) on stop. */ + abort: AbortController | null; + state: WatchdogState; + last: WatchdogDecision | null; + cfg: ResolvedWatchdogConfig; + deps: WatchdogDeps; + // Observability cache — computed once per probe completion (§7), NOT on every 5s dashboard poll. + lastSnapshot: MemorySnapshot | null; + responseState: ResponseStateMetrics | null; + lastProbeAt: number | null; + lastSuccessfulSystemProbeAt: number | null; +} + +let running: RunningWatchdog | null = null; +let generationCounter = 0; + +/** + * Apply one completed probe snapshot: evaluate → log/restart actions. Sync and deterministic — + * exported for tests (the async probe loop is just "capture, then this, then reschedule"). + */ +export function tick( + state: WatchdogState, + cfg: ResolvedWatchdogConfig, + deps: WatchdogDeps, + snapshot: MemorySnapshot, +): WatchdogDecision { + const nowMs = deps.now(); + const decision = evaluate(state, snapshot, cfg, nowMs, deps.supervised); + if (decision.action === "warn") { + deps.log(formatLine(decision)); + } else if (decision.action === "restart") { + deps.log(formatLine(decision)); + // Persist first: even if the restart hook fails, the DECISION happened and must count + // toward the cross-process cooldown/cap when the next process seeds from history. + try { deps.recordRestart?.(nowMs); } catch { /* best-effort */ } + // A failing restart hook must be visible, not an unhandled rejection or a thrown tick: + // the watchdog keeps running (warn-only until the cooldown re-arms) either way. + try { + const result: unknown = deps.restart(cfg); + if (result instanceof Promise) { + result.catch((err: unknown) => deps.log(`[opencodex] memory watchdog restart hook failed: ${err instanceof Error ? err.message : String(err)}`)); + } + } catch (err) { + deps.log(`[opencodex] memory watchdog restart hook failed: ${err instanceof Error ? err.message : String(err)}`); + } + } + // Observe-only commit axis: an independent log line, never a restart (see evaluate()). + if (decision.commitAction === "warn") { + deps.log(formatCommitLine(decision)); + } + return decision; +} + +/** + * One probe cycle: capture (async, non-blocking) → evaluate exactly once → cache observability → + * self-reschedule. Self-rescheduling (next setTimeout armed only after this probe fully completes) + * structurally prevents overlapping probes; the generation guard drops results that complete after + * stopMemoryWatchdog()/a restart replaced the singleton, so a late probe can never revive state. + */ +async function probeOnce(generation: number): Promise { + // tick()'s restart hook can synchronously null the global `running` (defaultRestart calls + // stopMemoryWatchdog before draining) — so pin the instance this probe owns in a local and, + // from here on, decide "am I still current?" by IDENTITY against the global (audit #1). + const current = running; + if (!current || current.generation !== generation) return; + current.timer = null; + current.abort = new AbortController(); + + let snapshot: MemorySnapshot; + try { + snapshot = await current.deps.capture(current.deps.now(), current.abort.signal); + } catch { + // Audit #2: captureMemorySnapshot is contractually non-throwing, but an unexpected runtime + // exception must not silence the cycle — evaluate exactly once on the RSS fallback instead. + snapshot = rssFallbackSnapshot(current.deps.now(), "capture-threw"); + } + + // Stopped/replaced mid-capture: drop the late result (global is null or a different instance). + if (running !== current || running.generation !== generation) return; + current.abort = null; + + try { + const decision = tick(current.state, current.cfg, current.deps, snapshot); + // If tick fired an auto-restart, its hook already stopped (or replaced) the watchdog: leave + // the dead/foreign singleton untouched and do not reschedule — the restart owns the exit. + if (running !== current || running.generation !== generation) return; + current.last = decision; + current.lastSnapshot = snapshot; + // §7: COMPLETION time, not capture start — a slow probe must read as fresh, not stale. + const completedAt = current.deps.now(); + current.lastProbeAt = completedAt; + if (snapshot.systemCommitAvailable) current.lastSuccessfulSystemProbeAt = completedAt; + // Response-state metrics serialize every entry — compute once per probe, not per poll. + try { current.responseState = responseStateMetrics(); } catch { current.responseState = null; } + } catch { + /* an evaluation failure must never crash the proxy */ + } + + if (running !== current || running.generation !== generation) return; + scheduleNextProbe(generation, current.cfg.intervalMs); +} + +function scheduleNextProbe(generation: number, delayMs: number): void { + if (!running || running.generation !== generation) return; + const timer = setTimeout(() => { void probeOnce(generation); }, delayMs); + (timer as { unref?: () => void }).unref?.(); + running.timer = timer; +} + +/** + * Start the watchdog. No-op (returns false) when disabled. Idempotent: a second call replaces the + * previous instance (its generation is invalidated, so in-flight probes are dropped). The first + * probe fires IMMEDIATELY — a proxy booting into an already-critical box is evaluated as soon as + * the first capture completes, not one interval later. Timers are unref'd. + */ +export function startMemoryWatchdog(config: OcxConfig, deps: Partial = {}): boolean { + const cfg = resolveWatchdogConfig(config); + if (!cfg.enabled) return false; + stopMemoryWatchdog(); + const d: WatchdogDeps = { ...DEFAULT_DEPS, ...deps }; + const state = createWatchdogState(); + // Seed the restart guards from the best-effort history so cooldown/maxRestarts bind across the + // process boundary a fired restart creates. A missing/corrupt file seeds zeros (fresh slate). + const seeded = loadMemoryRestartHistory(d.now()); + state.lastRestartAt = seeded.lastRestartAt; + state.restartCount = seeded.recentCount; + const generation = ++generationCounter; + running = { + generation, + timer: null, + abort: null, + state, + last: null, + cfg, + deps: d, + lastSnapshot: null, + responseState: null, + lastProbeAt: null, + lastSuccessfulSystemProbeAt: null, + }; + void probeOnce(generation); + return true; +} + +/** + * Live-update knobs on a running watchdog without a restart (Adjust UI / management API). The same + * final validation as resolveWatchdogConfig applies: fractions are clamped and critical is kept + * above warn, durations are clamped/normalized (grace bounds, cooldown >= grace, 1s interval + * floor). An intervalMs change re-arms the timer. Returns the applied config, or null when the + * watchdog is not running. + */ +export function applyWatchdogRuntimeConfig( + partial: Partial, +): ResolvedWatchdogConfig | null { + if (!running) return null; + const next: ResolvedWatchdogConfig = { ...running.cfg, ...partial }; + next.warnFraction = clampFraction(next.warnFraction, DEFAULTS.warnFraction); + next.criticalFraction = clampFraction(next.criticalFraction, DEFAULTS.criticalFraction); + if (next.criticalFraction <= next.warnFraction) { + next.criticalFraction = Math.min(0.99, next.warnFraction + 0.10); + } + next.restartGraceMs = clampMs(next.restartGraceMs, RESTART_GRACE_MIN_MS, RESTART_GRACE_MAX_MS, running.cfg.restartGraceMs); + next.minRestartIntervalMs = posNumOr(next.minRestartIntervalMs, running.cfg.minRestartIntervalMs); + // Audit #4: not PUT-exposed, but a programmatic partial carrying undefined/NaN would otherwise + // survive the spread — an NaN high-water makes `fraction >= NaN` permanently false and silently + // disables the commit warning. Same finite-guard pattern as resolveWatchdogConfig's maxRestarts. + next.systemCommitHighWater = Number.isFinite(next.systemCommitHighWater) + ? Math.min(0.99, Math.max(0.50, next.systemCommitHighWater)) + : running.cfg.systemCommitHighWater; + next.maxRestarts = Number.isFinite(next.maxRestarts) && next.maxRestarts >= 0 + ? Math.floor(next.maxRestarts) + : running.cfg.maxRestarts; + normalizeRestartTiming(next); + next.intervalMs = Math.max(1_000, posNumOr(next.intervalMs, running.cfg.intervalMs)); + const intervalChanged = next.intervalMs !== running.cfg.intervalMs; + running.cfg = next; + // Self-rescheduling loop: a probe in flight (timer === null) picks the new interval up on its + // own completion; only a PENDING timer needs re-arming to honor the new cadence promptly. + if (intervalChanged && running.timer !== null) { + clearTimeout(running.timer); + running.timer = null; + scheduleNextProbe(running.generation, next.intervalMs); + } + return next; +} + +/** + * Stop the watchdog (graceful shutdown / tests). Idempotent. Cancels the pending probe timer, + * aborts an in-flight capture (killing its child probe process), and clears the singleton — a + * probe result that still arrives sees no matching generation and is dropped instead of reviving + * state. + */ +export function stopMemoryWatchdog(): void { + if (!running) return; + if (running.timer !== null) clearTimeout(running.timer); + try { running.abort?.abort(); } catch { /* already settled */ } + running = null; +} + +/** Last decision + guard state, for the management API / diagnostics. Null when never sampled. */ +export function memoryWatchdogSnapshot(): (WatchdogDecision & { restartCount: number }) | null { + if (!running || !running.last) return null; + return { ...running.last, restartCount: running.state.restartCount }; +} + +/** Snapshot-derived observability block (§7) — cached at probe completion, null before the first. */ +export interface WatchdogMemoryReport { + processPrivateBytes: number | null; + processPressureBytes: number; + processSource: string; + rssBytes: number; + heapUsedBytes: number; + externalBytes: number; + physicalMemoryBytes: number; + availablePhysicalBytes: number | null; + systemCommittedBytes: number | null; + systemCommitLimitBytes: number | null; + systemCommitFraction: number | null; + systemCommitAvailable: boolean; + probeError?: string; +} + +export interface WatchdogReport { + enabled: boolean; + decision: WatchdogDecision | null; + samplesCount: number; + growthMbPerHour: number | null; + resolvedConfig: ResolvedWatchdogConfig; + supervisor: { supervised: boolean; hint: string }; + recommendation: WatchdogRecommendation; + restartCount: number; + /** Last completed probe's measurements; null until the first probe completes. */ + memory: WatchdogMemoryReport | null; + /** Response-state cache metrics, computed once per probe completion (not per poll). */ + responseState: ResponseStateMetrics | null; + /** Last probe COMPLETION time (RSS fallback included); null before the first. */ + lastProbeAt: number | null; + /** Last probe that measured the system commit values successfully; null when never. */ + lastSuccessfulSystemProbeAt: number | null; +} + +/** + * Full observability report for the dashboard (Monitor + Recommend). Read-only: it never mutates + * config or state, and the per-probe blocks (memory/responseState) come from the probe-completion + * cache so a 5s dashboard poll never re-serializes the response store. Returns null when the + * watchdog is not running (disabled); the caller reports that as { enabled: false }. + */ +export function memoryWatchdogReport(): WatchdogReport | null { + if (!running) return null; + const supervisor = detectSupervisor(); + const s = running.lastSnapshot; + return { + enabled: true, + decision: running.last, + samplesCount: running.state.samples.length, + growthMbPerHour: running.last?.growthMbPerHour ?? null, + resolvedConfig: running.cfg, + supervisor, + recommendation: recommend(running.state, running.cfg, supervisor.supervised), + restartCount: running.state.restartCount, + memory: s === null ? null : { + processPrivateBytes: s.processPrivateBytes, + processPressureBytes: s.processPressureBytes, + processSource: s.processSource, + rssBytes: s.rssBytes, + heapUsedBytes: s.heapUsedBytes, + externalBytes: s.externalBytes, + physicalMemoryBytes: s.physicalMemoryBytes, + availablePhysicalBytes: s.availablePhysicalBytes, + systemCommittedBytes: s.systemCommittedBytes, + systemCommitLimitBytes: s.systemCommitLimitBytes, + systemCommitFraction: s.systemCommitAvailable && s.systemCommittedBytes !== null && s.systemCommitLimitBytes !== null && s.systemCommitLimitBytes > 0 + ? s.systemCommittedBytes / s.systemCommitLimitBytes + : null, + systemCommitAvailable: s.systemCommitAvailable, + ...(s.probeError !== undefined ? { probeError: s.probeError } : {}), + }, + responseState: running.responseState, + lastProbeAt: running.lastProbeAt, + lastSuccessfulSystemProbeAt: running.lastSuccessfulSystemProbeAt, + }; +} diff --git a/src/types.ts b/src/types.ts index 341cb7394..fce6397f6 100644 --- a/src/types.ts +++ b/src/types.ts @@ -520,6 +520,44 @@ export interface OcxConfig { connectTimeoutMs?: number; /** Graceful shutdown drain timeout (ms). Active turns are aborted after this deadline. Default 5000. */ shutdownTimeoutMs?: number; + /** + * Memory watchdog: observe process memory pressure relative to total system RAM and warn (or, + * opt-in, gracefully restart) before the Bun/mimalloc native allocator can exhaust the system + * commit charge. Thresholds are fractions of total RAM so behavior is identical across machines. + * Default: enabled, warn-only (autoRestart false). + */ + memoryWatchdog?: { + /** Master switch. Default true (warn-only observation is safe). */ + enabled?: boolean; + /** Sampling interval (ms). Default 60000. */ + intervalMs?: number; + /** Warn when pressure (committed-or-RSS) / total RAM crosses this fraction. Default 0.60. */ + warnFraction?: number; + /** Critical threshold fraction; arms an opt-in restart. Default 0.75. */ + criticalFraction?: number; + /** Opt in to a graceful restart at the critical threshold. Default false (warn only). */ + autoRestart?: boolean; + /** When true (default), auto-restart only fires if a process supervisor is detected. + * Detection covers pm2, systemd and an explicit OCX_SUPERVISED=1; NSSM / Windows services / + * sc.exe are NOT auto-detected — set OCX_SUPERVISED=1 there. */ + requireSupervisor?: boolean; + /** Minimum ms between auto-restart requests. Raised to at least restartGraceMs so a second + * restart can never arm while the first drain is still inside its grace window. Seeded across + * process boundaries from a best-effort history file; when that file is unavailable the + * cooldown restarts from zero in the new process and the supervisor's own restart throttle is + * the remaining cross-boundary protection. Default 600000 (10 min). */ + minRestartIntervalMs?: number; + /** Max memory-driven restarts within a rolling ~6h window before the watchdog degrades to + * warn-only. Counted best-effort across process boundaries via a small timestamps file — NOT + * a permanent all-time cap, and not a substitute for the supervisor's restart limit/backoff + * policy. 0 disables auto-restart entirely. Default 3. */ + maxRestarts?: number; + /** Quiet-window drain budget (ms) for a memory-driven restart: wait up to this long for + * in-flight turns to finish before aborting, so the restart lands on a natural idle gap. + * While draining, new requests receive 503 + Retry-After — this value bounds that window. + * Clamped to [1000, 600000]. Default 30000. */ + restartGraceMs?: number; + }; /** Advertise supports_websockets so Codex opens the WS endpoint. Default false; set true to opt in. */ websockets?: boolean; /** Generated API keys for external access to the proxy's /v1/responses endpoint. */ diff --git a/structure/09_memory-watchdog.md b/structure/09_memory-watchdog.md new file mode 100644 index 000000000..668da61a3 --- /dev/null +++ b/structure/09_memory-watchdog.md @@ -0,0 +1,129 @@ +# Memory Watchdog SOT + +## Why this exists + +Field incident: OpenCodex on Bun reached **~79 GB private/committed memory while the working set +stayed ~5.8 GB**, freezing the box once the **system commit charge hit ~97% of the commit limit**. +The Bun/mimalloc native allocator retains committed memory the JS heap has already released +(mirrored upstream by Claude Code #36132); no in-app JS cache cap can fix native retention. The +locally-run stress harness reproduces the detectability signature on a healthy machine +(`afterClear private/RSS = 4.4–4.8` after clearing the JS store and forcing GC). + +The only proven mitigation is to **observe the pressure and, opt-in, hand off to a graceful +restart**. Everything below follows from that. + +## Platform scope (PR reviewers: read this first) + +The watchdog is **cross-platform**; only the measurement *fidelity* differs per platform. This is +NOT a Windows-only patch, but the motivating incident, the highest-fidelity probe, and the v1 +system-commit axis are Windows-specific. + +| Layer | win32 | linux | other | +| --- | --- | --- | --- | +| Process pressure source | Private Bytes (async PowerShell probe) | VmRSS+VmSwap (`/proc/self/status`, no spawn) | RSS (in-process) | +| System commit axis (v1) | measured (CommittedBytes/CommitLimit) | not measured (`systemCommitAvailable=false`; `/proc/meminfo` `Committed_AS` is a documented follow-up) | not measured | +| Decision core (warn/critical, cooldown, max-restarts, history seeding) | identical | identical | identical | +| Quiet-window drain + exit-75 restart, supervisor gating | identical | identical | identical | +| Dashboard / management API / i18n | identical | identical | identical | + +The commit-axis logic itself is platform-neutral and null-safe: on platforms without measurement +it simply skips (no NaN propagation, latch held), so enabling a Linux collector later is additive. + +## Architecture: async collect, sync decide + +```text +self-rescheduling probe loop (unref'd; first probe fires IMMEDIATELY at start) + └─ captureMemorySnapshot() — async, never throws, never blocks the event loop + win32: ONE hidden PowerShell child (array args, -NoProfile -NonInteractive) returns + labeled values P=/C=/L=/A= (private, committed, commit limit, available physical; + Win32_PerfRawData_PerfOS_Memory — all bytes, verified live). 15s budget + (measured cold spawn: 4.0–5.9s); timeout kills the child; first-resolution-wins + drops late results; failures degrade to an RSS fallback with a sanitized probeError. + └─ tick(state, cfg, deps, snapshot) — sync, pure-core evaluate(), EXACTLY once per probe + axis 1 PROCESS: pressure/physicalRAM vs warn(0.60)/critical(0.75) → opt-in restart path + axis 2 SYSTEM COMMIT: observe-only high-water warning (see below), NEVER restarts + └─ report cache (memory block, responseStateMetrics, lastProbeAt/lastSuccessfulSystemProbeAt) + computed once per probe — a 5s dashboard poll never re-serializes the response store +``` + +Key invariants, each defended by a regression test: + +- **The probe never blocks the event loop.** The original implementation used `Bun.spawnSync` + (up to 2s synchronous stall per tick on Windows) — flagged as a deploy blocker by audit and + replaced. A streaming proxy must not pause SSE for measurement. +- **Exactly one evaluation per probe.** Timeout → RSS-fallback evaluation, and a late success for + the same probe is dropped; an unexpected `capture()` exception evaluates once on + `rssFallbackSnapshot(..., "capture-threw")` instead of silencing the cycle. +- **A restart owns the exit.** `tick()`'s restart hook synchronously stops the watchdog; + `probeOnce` pins its instance in a local and re-checks IDENTITY against the global after + capture, after tick, and before rescheduling, so the stopped/replaced singleton is never + touched or rescheduled (pre-fix this was an unhandled TypeError on every real auto-restart). +- **Stop kills the child.** `stopMemoryWatchdog()` aborts the in-flight capture via + AbortController and a generation guard drops any result that still arrives. + +## System-commit axis: why observe-only (v1) + +`systemCommittedBytes / systemCommitLimitBytes >= 0.90` (env-only override +`OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER`, clamped [0.50, 0.99]) logs ONE latched warning; the latch +re-arms only on a MEASURED recovery — measurement loss (probe failure) holds it, so a flapping +probe cannot re-warn each time it returns. + +It never contributes to the restart decision because the commit pressure may come from **another +process**: restarting OpenCodex would not free it, and an automated restart loop against an +external cause is worse than a loud warning. Honest consequence, stated up front: **v1 does not +auto-mitigate the original incident** (79 GB / 128 GB RAM ≈ 62% process pressure < 0.75 critical; +system commit 97% → warning only). Promoting the commit axis into the restart decision, and any +UI-configurable thresholds, are deliberately deferred until field measurement validates the axis. + +## Restart path (quiet-window) and honest loop guards + +- `restartGraceMs` (default 30s, clamped [1s, 10min] on every entry path — env, config file, + management API which 400s out-of-range values): draining rejects new turns with 503+Retry-After + and returns the moment the in-flight set empties, so the restart lands on a natural idle gap and + the value is a DEADLINE, not a delay. +- `minRestartIntervalMs` is normalized to ≥ `restartGraceMs` so a second restart can never arm + inside the first drain window. +- Cooldown/max-restarts counters die with the process a restart ends, so they are re-seeded from + a best-effort timestamps-only history file (rolling ~6h window, atomic-rename write, every + failure swallowed). This is **not** a permanent cross-process guarantee: the supervisor's own + restart limit/backoff (pm2 `max_restarts`, systemd `StartLimitBurst`) remains the outer layer, + and exit code 75 is only a *request* to respawn. NSSM / Windows services are not auto-detected — + set `OCX_SUPERVISED=1` there. + +## Verification evidence (as merged) + +- Unit/integration: `tests/memory-watchdog.test.ts` (66), `tests/memory-restart-history.test.ts` + (11), `tests/memory-api.test.ts` (16) — includes a REAL Windows end-to-end probe test asserting + private bytes and a sane commit fraction from the live labeled command. +- Live measurements on a Windows 11 box: probe cold spawn 4.0–5.9s (hence the 15s async budget); + `Win32_PerfRawData_PerfOS_Memory` values confirmed to be bytes; harness detectability + `afterClear private/RSS = 4.4–4.8`. +- Gates: `typecheck`, `lint:gui`, `build:gui`, `privacy:scan` all exit 0. `probeError` carries + sanitized codes only (never raw command output/paths) — privacy-scan enforced. +- Full-suite caveat: on the development Windows box the suite exits 1 due to pre-existing + environment failures (NTFS ACL hardening `EACCES` clusters in auth tests + branch-dependent + release-helper tests). Evidence that these are unrelated: the failure set at this branch's HEAD + is a strict subset (321 ⊂ 345, zero new files, zero memory-file failures) of the same run at the + pre-change baseline commit under identical conditions. + +## Audit trail + +Three independent audit rounds shaped this design (summarized honestly rather than re-litigated): + +1. Initial audit (deploy blockers): synchronous `spawnSync` probe stall, missing system-commit + measurement, 60s startup blind spot, RSS-only stress assertions, unwired response-state + metrics → all five confirmed against code and fixed by the async rewrite. +2. Design review: chose "cached-free" probe-completion-driven evaluation over defer-until-idle + and blue-green alternatives (complexity/regression cost vs. benefit), kept warn-only defaults. +3. Post-implementation audit: auto-restart race in `probeOnce` (unhandled TypeError), capture-throw + silence, positional PowerShell parsing, live re-clamp gaps → all four fixed with regression + tests (the race test was validated to fail against the pre-fix code). + +## Follow-ups (out of scope here) + +- Field measurement, then possible commit-axis restart integration + UI thresholds (v2). +- Linux system commit via `/proc/meminfo` (`Committed_AS`/`CommitLimit`, no spawn). +- docs-site ja/zh-cn translations of the memoryWatchdog reference section. +- Separate issue: `windows-secret-acl` hardening is non-atomic under icacls timeouts + (inheritance stripped before grant), which is the root cause of the environmental `EACCES` + test failures on slow-ACL machines. diff --git a/tests/memory-api.test.ts b/tests/memory-api.test.ts new file mode 100644 index 000000000..13dd003d5 --- /dev/null +++ b/tests/memory-api.test.ts @@ -0,0 +1,163 @@ +/** + * Management API surface for the memory watchdog (dashboard beta panel): + * GET /api/memory → read-only Monitor + Recommend report (Runs behind requireApiAuth). + * PUT /api/memory/settings → validate + persist (saveConfig) + apply live without a restart. + * The route handlers are exercised directly; auth/CORS gating is asserted elsewhere. saveConfig is + * pointed at an isolated OPENCODEX_HOME so persistence is checked without touching the real config. + */ +import { afterEach, describe, expect, test } from "bun:test"; +import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { getConfigPath } from "../src/config"; +import { handleManagementAPI } from "../src/server/management-api"; +import { stopMemoryWatchdog } from "../src/server/memory-watchdog"; +import type { OcxConfig } from "../src/types"; + +const savedHome = process.env.OPENCODEX_HOME; +let tempHome: string | null = null; + +afterEach(() => { + stopMemoryWatchdog(); // never leak a running watchdog singleton across tests + if (savedHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = savedHome; + if (tempHome) { + try { rmSync(tempHome, { recursive: true, force: true }); } catch { /* best-effort */ } + tempHome = null; + } +}); + +function isolatedHome(): void { + tempHome = mkdtempSync(join(tmpdir(), "ocx-memory-")); + process.env.OPENCODEX_HOME = tempHome; +} + +function makeConfig(overrides: Partial = {}): OcxConfig { + return { port: 10100, providers: {}, defaultProvider: "openai", ...overrides } as OcxConfig; +} + +async function get(config: OcxConfig): Promise { + const req = new Request("http://localhost/api/memory"); + const res = await handleManagementAPI(req, new URL(req.url), config); + expect(res).not.toBeNull(); + return res!; +} + +async function put(config: OcxConfig, body: unknown): Promise { + const req = new Request("http://localhost/api/memory/settings", { + method: "PUT", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }); + const res = await handleManagementAPI(req, new URL(req.url), config); + expect(res).not.toBeNull(); + return res!; +} + +interface Report { + enabled: boolean; + resolvedConfig?: { warnFraction: number; criticalFraction: number; requireSupervisor: boolean; autoRestart: boolean }; + supervisor?: { supervised: boolean; hint: string }; + recommendation?: unknown; +} + +describe("GET /api/memory", () => { + test("degrades to { enabled: false } when the watchdog is not running", async () => { + isolatedHome(); + const config = makeConfig(); + const res = await get(config); + expect(res.status).toBe(200); + expect(await res.json()).toEqual({ enabled: false }); + }); + + test("after enabling, the report exposes the Monitor + Recommend shape", async () => { + isolatedHome(); + const config = makeConfig(); + await put(config, { enabled: true, warnFraction: 0.55, criticalFraction: 0.8 }); + const data = (await (await get(config)).json()) as Report; + expect(data.enabled).toBe(true); + expect(data.resolvedConfig?.warnFraction).toBe(0.55); + expect(data.resolvedConfig?.criticalFraction).toBe(0.8); + expect(data.supervisor).toBeDefined(); + expect(data.recommendation).toBeDefined(); + }); +}); + +describe("PUT /api/memory/settings", () => { + test("valid patch persists to disk and applies live", async () => { + isolatedHome(); + const config = makeConfig(); + const res = await put(config, { enabled: true, warnFraction: 0.5, criticalFraction: 0.85, requireSupervisor: false }); + expect(res.status).toBe(200); + const body = (await res.json()) as { ok: boolean; report: Report }; + expect(body.ok).toBe(true); + // in-memory config mutated + expect(config.memoryWatchdog).toMatchObject({ warnFraction: 0.5, criticalFraction: 0.85, requireSupervisor: false }); + // persisted to the isolated config file + const persisted = JSON.parse(readFileSync(getConfigPath(), "utf8")) as OcxConfig; + expect(persisted.memoryWatchdog).toMatchObject({ warnFraction: 0.5, criticalFraction: 0.85, requireSupervisor: false }); + // applied live: the running report reflects the new thresholds + expect(body.report.resolvedConfig?.warnFraction).toBe(0.5); + expect(body.report.resolvedConfig?.requireSupervisor).toBe(false); + }); + + test("a live threshold change updates the already-running watchdog", async () => { + isolatedHome(); + const config = makeConfig(); + await put(config, { enabled: true, warnFraction: 0.6, criticalFraction: 0.75 }); + const res = await put(config, { warnFraction: 0.4 }); + const body = (await res.json()) as { report: Report }; + expect(res.status).toBe(200); + expect(body.report.resolvedConfig?.warnFraction).toBe(0.4); + }); + + test("enabled:false stops the watchdog and the report degrades", async () => { + isolatedHome(); + const config = makeConfig(); + await put(config, { enabled: true }); + expect(((await (await get(config)).json()) as Report).enabled).toBe(true); + await put(config, { enabled: false }); + expect(await (await get(config)).json()).toEqual({ enabled: false }); + }); + + test.each([ + ["out-of-range warnFraction", { warnFraction: 2 }], + ["non-numeric criticalFraction", { criticalFraction: "high" }], + ["non-boolean autoRestart", { autoRestart: "yes" }], + ["non-boolean requireSupervisor", { requireSupervisor: 1 }], + ["non-positive intervalMs", { intervalMs: 0 }], + ["below-minimum restartGraceMs", { restartGraceMs: 500 }], + ["above-maximum restartGraceMs", { restartGraceMs: 900_000 }], + ["non-numeric restartGraceMs", { restartGraceMs: "30s" }], + ["non-finite restartGraceMs", { restartGraceMs: Number.NaN }], + ] as const)("rejects %s with 400 and does not persist", async (_label, body) => { + isolatedHome(); + const config = makeConfig(); + const res = await put(config, body); + expect(res.status).toBe(400); + expect(config.memoryWatchdog).toBeUndefined(); + expect(existsSync(getConfigPath())).toBe(false); + }); + + test("a valid restartGraceMs persists, applies live, and raises a shorter cooldown", async () => { + isolatedHome(); + // A config-file cooldown SHORTER than the requested grace: the runtime must raise it. + const config = makeConfig({ memoryWatchdog: { minRestartIntervalMs: 60_000 } } as Partial); + await put(config, { enabled: true }); + const res = await put(config, { restartGraceMs: 300_000 }); + expect(res.status).toBe(200); + const body = (await res.json()) as { ok: boolean; report: { resolvedConfig?: { restartGraceMs?: number; minRestartIntervalMs?: number } } }; + expect(body.ok).toBe(true); + expect(body.report.resolvedConfig?.restartGraceMs).toBe(300_000); + expect(body.report.resolvedConfig?.minRestartIntervalMs).toBe(300_000); // cooldown >= grace + const persisted = JSON.parse(readFileSync(getConfigPath(), "utf8")) as OcxConfig; + expect(persisted.memoryWatchdog?.restartGraceMs).toBe(300_000); + }); + + test("rejects a non-object body with 400", async () => { + isolatedHome(); + const config = makeConfig(); + const res = await put(config, ["not", "an", "object"]); + expect(res.status).toBe(400); + }); +}); diff --git a/tests/memory-restart-history.test.ts b/tests/memory-restart-history.test.ts new file mode 100644 index 000000000..de18785b4 --- /dev/null +++ b/tests/memory-restart-history.test.ts @@ -0,0 +1,115 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + loadMemoryRestartHistory, + recordMemoryRestart, + RESTART_HISTORY_WINDOW_MS, + setRestartHistoryPathForTests, +} from "../src/server/memory-restart-history"; + +const NOW = 10 * RESTART_HISTORY_WINDOW_MS; // comfortably past the window so pruning math never goes negative + +let tempRoot: string | null = null; + +function isolatedHistoryPath(): string { + tempRoot = mkdtempSync(join(tmpdir(), "ocx-restart-history-")); + const path = join(tempRoot, "memory-watchdog-restarts.json"); + setRestartHistoryPathForTests(path); + return path; +} + +afterEach(() => { + setRestartHistoryPathForTests(null); + if (tempRoot) { + rmSync(tempRoot, { recursive: true, force: true }); + tempRoot = null; + } +}); + +describe("loadMemoryRestartHistory", () => { + test("missing file seeds a fresh slate", () => { + isolatedHistoryPath(); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: 0, recentCount: 0 }); + }); + + test("corrupt JSON seeds a fresh slate instead of throwing", () => { + const path = isolatedHistoryPath(); + writeFileSync(path, "{ not json", "utf-8"); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: 0, recentCount: 0 }); + }); + + test("wrong schema (version mismatch / non-array) seeds a fresh slate", () => { + const path = isolatedHistoryPath(); + writeFileSync(path, JSON.stringify({ version: 99, restarts: [NOW - 1_000] }), "utf-8"); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: 0, recentCount: 0 }); + writeFileSync(path, JSON.stringify({ version: 1, restarts: "nope" }), "utf-8"); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: 0, recentCount: 0 }); + }); + + test("entries outside the rolling window are ignored; entries inside count", () => { + const path = isolatedHistoryPath(); + const inside = NOW - RESTART_HISTORY_WINDOW_MS + 60_000; + const outside = NOW - RESTART_HISTORY_WINDOW_MS - 60_000; + writeFileSync(path, JSON.stringify({ version: 1, restarts: [outside, inside] }), "utf-8"); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: inside, recentCount: 1 }); + }); + + test("future timestamps (clock rollback) are dropped so the cooldown cannot be pinned", () => { + const path = isolatedHistoryPath(); + writeFileSync(path, JSON.stringify({ version: 1, restarts: [NOW + 3_600_000] }), "utf-8"); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: 0, recentCount: 0 }); + }); + + test("non-numeric entries are filtered, valid ones survive", () => { + const path = isolatedHistoryPath(); + writeFileSync(path, JSON.stringify({ version: 1, restarts: ["x", null, NOW - 1_000, Number.NaN] }), "utf-8"); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: NOW - 1_000, recentCount: 1 }); + }); +}); + +describe("recordMemoryRestart", () => { + test("record + load roundtrip seeds the next process", () => { + isolatedHistoryPath(); + recordMemoryRestart(NOW - 2_000); + recordMemoryRestart(NOW - 1_000); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: NOW - 1_000, recentCount: 2 }); + }); + + test("recording prunes expired entries and caps the file", () => { + const path = isolatedHistoryPath(); + const stale = Array.from({ length: 5 }, (_, i) => NOW - RESTART_HISTORY_WINDOW_MS - (i + 1) * 1_000); + writeFileSync(path, JSON.stringify({ version: 1, restarts: stale }), "utf-8"); + for (let i = 0; i < 30; i++) recordMemoryRestart(NOW - (30 - i) * 1_000); + const persisted = JSON.parse(readFileSync(path, "utf-8")) as { restarts: number[] }; + expect(persisted.restarts.length).toBeLessThanOrEqual(20); + expect(Math.min(...persisted.restarts)).toBeGreaterThan(NOW - RESTART_HISTORY_WINDOW_MS); + }); + + test("a write failure is swallowed (history must never block the restart)", () => { + tempRoot = mkdtempSync(join(tmpdir(), "ocx-restart-history-")); + // Point the history "file" at an existing DIRECTORY: writeFileSync/rename must fail on every OS. + const dirAsFile = join(tempRoot, "history-dir"); + mkdirSync(dirAsFile); + setRestartHistoryPathForTests(dirAsFile); + expect(() => recordMemoryRestart(NOW)).not.toThrow(); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: 0, recentCount: 0 }); + }); + + test("creates the parent directory when missing", () => { + tempRoot = mkdtempSync(join(tmpdir(), "ocx-restart-history-")); + const nested = join(tempRoot, "deep", "nested", "memory-watchdog-restarts.json"); + setRestartHistoryPathForTests(nested); + recordMemoryRestart(NOW); + expect(loadMemoryRestartHistory(NOW)).toEqual({ lastRestartAt: NOW, recentCount: 1 }); + }); + + test("persists only timestamps — no config, request or account data", () => { + const path = isolatedHistoryPath(); + recordMemoryRestart(NOW); + const raw = JSON.parse(readFileSync(path, "utf-8")) as Record; + expect(Object.keys(raw).sort()).toEqual(["restarts", "version"]); + expect((raw.restarts as unknown[]).every(t => typeof t === "number")).toBe(true); + }); +}); diff --git a/tests/memory-watchdog.test.ts b/tests/memory-watchdog.test.ts new file mode 100644 index 000000000..2129ffdfd --- /dev/null +++ b/tests/memory-watchdog.test.ts @@ -0,0 +1,867 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { setRestartHistoryPathForTests } from "../src/server/memory-restart-history"; +import { + captureMemorySnapshot, + parseProcStatusCommitted, + parseWindowsProbeOutput, + setMemoryPlatformForTests, + setProcStatusReaderForTests, + setWindowsProbeRunnerForTests, + type MemorySnapshot, + type WindowsProbeResult, +} from "../src/lib/memory-usage"; +import { + applyWatchdogRuntimeConfig, + createWatchdogState, + detectSupervisor, + evaluate, + memoryWatchdogReport, + recommend, + resolveWatchdogConfig, + startMemoryWatchdog, + stopMemoryWatchdog, + tick, + type ResolvedWatchdogConfig, + type WatchdogDeps, +} from "../src/server/memory-watchdog"; +import type { OcxConfig } from "../src/types"; + +const MiB = 1024 * 1024; + +function snapshot(pressure: number, opts: Partial = {}): MemorySnapshot { + return { + rssBytes: pressure, + heapUsedBytes: 10 * MiB, + externalBytes: 0, + physicalMemoryBytes: 100 * MiB, + processPrivateBytes: pressure, + processPressureBytes: pressure, + processSource: "windows-private", + systemCommittedBytes: null, + systemCommitLimitBytes: null, + systemCommitAvailable: false, + availablePhysicalBytes: null, + capturedAt: 0, + ...opts, + }; +} + +function cfg(over: Partial = {}): ResolvedWatchdogConfig { + return { + enabled: true, + intervalMs: 60_000, + warnFraction: 0.60, + criticalFraction: 0.75, + autoRestart: false, + requireSupervisor: true, + minRestartIntervalMs: 600_000, + maxRestarts: 3, + restartGraceMs: 30_000, + systemCommitHighWater: 0.90, + growthWindowMs: 600_000, + ...over, + }; +} + +describe("memory-usage snapshot", () => { + test("parseProcStatusCommitted sums VmRSS + VmSwap (kB → bytes)", () => { + const text = "Name:\tbun\nVmRSS:\t 1024 kB\nVmSwap:\t 512 kB\n"; + expect(parseProcStatusCommitted(text)).toBe((1024 + 512) * 1024); + }); + + test("parseProcStatusCommitted tolerates a missing VmSwap and rejects missing VmRSS", () => { + expect(parseProcStatusCommitted("VmRSS:\t 2048 kB\n")).toBe(2048 * 1024); + expect(parseProcStatusCommitted("VmData:\t 100 kB\n")).toBeNull(); + }); + + function probe(over: Partial = {}): WindowsProbeResult { + return { + privateBytes: null, + systemCommittedBytes: null, + systemCommitLimitBytes: null, + availablePhysicalBytes: null, + timedOut: false, + ...over, + }; + } + + test("linux snapshot fills pressure from /proc/self/status; system commit stays unavailable (v1)", async () => { + setMemoryPlatformForTests("linux"); + setProcStatusReaderForTests(() => "VmRSS:\t 4096 kB\nVmSwap:\t 0 kB\n"); + try { + const snap = await captureMemorySnapshot(1_000); + expect(snap.processSource).toBe("proc-status"); + expect(snap.processPressureBytes).toBe(4096 * 1024); + expect(snap.processPrivateBytes).toBeNull(); // Private Bytes is a Windows concept + expect(snap.systemCommitAvailable).toBe(false); + expect(snap.capturedAt).toBe(1_000); + } finally { + setMemoryPlatformForTests(null); + setProcStatusReaderForTests(null); + } + }); + + test("windows snapshot carries private bytes AND the system commit values from one probe", async () => { + setMemoryPlatformForTests("win32"); + setWindowsProbeRunnerForTests(async () => probe({ + privateBytes: 79 * 1024 * MiB, + systemCommittedBytes: 120 * 1024 * MiB, + systemCommitLimitBytes: 140 * 1024 * MiB, + availablePhysicalBytes: 40 * 1024 * MiB, + })); + try { + const snap = await captureMemorySnapshot(2_000); + expect(snap.processSource).toBe("windows-private"); + expect(snap.processPrivateBytes).toBe(79 * 1024 * MiB); + expect(snap.processPressureBytes).toBe(79 * 1024 * MiB); + expect(snap.systemCommitAvailable).toBe(true); + expect(snap.systemCommittedBytes).toBe(120 * 1024 * MiB); + expect(snap.systemCommitLimitBytes).toBe(140 * 1024 * MiB); + expect(snap.availablePhysicalBytes).toBe(40 * 1024 * MiB); + expect(snap.probeError).toBeUndefined(); + } finally { + setMemoryPlatformForTests(null); + setWindowsProbeRunnerForTests(null); + } + }); + + test("a timed-out windows probe degrades to the RSS fallback with an honest probeError", async () => { + setMemoryPlatformForTests("win32"); + setWindowsProbeRunnerForTests(async () => probe({ timedOut: true, error: "timeout" })); + try { + const snap = await captureMemorySnapshot(); + expect(snap.processPrivateBytes).toBeNull(); + expect(snap.processSource).toBe("rss-fallback"); + expect(snap.processPressureBytes).toBe(snap.rssBytes); // watchdog still functions, degraded + expect(snap.systemCommitAvailable).toBe(false); + expect(snap.probeError).toBe("timeout"); + } finally { + setMemoryPlatformForTests(null); + setWindowsProbeRunnerForTests(null); + } + }); + + test("a partial probe (private only) keeps pressure but reports commit as unavailable", async () => { + setMemoryPlatformForTests("win32"); + setWindowsProbeRunnerForTests(async () => probe({ privateBytes: 5 * 1024 * MiB })); + try { + const snap = await captureMemorySnapshot(); + expect(snap.processSource).toBe("windows-private"); + expect(snap.systemCommitAvailable).toBe(false); // BOTH commit values are required + expect(snap.systemCommittedBytes).toBeNull(); + } finally { + setMemoryPlatformForTests(null); + setWindowsProbeRunnerForTests(null); + } + }); + + test("a throwing probe runner degrades to the RSS fallback instead of propagating", async () => { + setMemoryPlatformForTests("win32"); + setWindowsProbeRunnerForTests(async () => { throw new Error("boom"); }); + try { + const snap = await captureMemorySnapshot(); + expect(snap.processSource).toBe("rss-fallback"); + expect(snap.probeError).toBe("probe-failed"); + } finally { + setMemoryPlatformForTests(null); + setWindowsProbeRunnerForTests(null); + } + }); + + test("labeled probe output parses order-independently and tolerates missing values", () => { + const full = parseWindowsProbeOutput("P=100\nC=200\nL=300\nA=400\n"); + expect(full).toMatchObject({ privateBytes: 100, systemCommittedBytes: 200, systemCommitLimitBytes: 300, availablePhysicalBytes: 400 }); + expect(full.error).toBeUndefined(); + + // Shuffled line order must not reassign fields (the old positional parser's failure mode). + const shuffled = parseWindowsProbeOutput("A=400\r\nL=300\r\nP=100\r\nC=200"); + expect(shuffled).toMatchObject({ privateBytes: 100, systemCommittedBytes: 200 }); + + // A $null private renders as "P=" — private degrades to null WITHOUT shifting commit values. + const noPrivate = parseWindowsProbeOutput("P=\nC=200\nL=300\nA=400"); + expect(noPrivate.privateBytes).toBeNull(); + expect(noPrivate.systemCommittedBytes).toBe(200); + expect(noPrivate.error).toBeUndefined(); + + // Garbage / empty output is an explicit parse failure, not a zero-filled snapshot. + expect(parseWindowsProbeOutput("").error).toBe("parse-failed"); + expect(parseWindowsProbeOutput("access denied\nblah").error).toBe("parse-failed"); + }); + + // Real end-to-end probe: only meaningful (and only run) on an actual Windows host. Verifies the + // §4 field/unit assumptions live: private bytes for this PID plus system CommittedBytes / + // CommitLimit in bytes, all from a single async PowerShell spawn that never blocks the loop. + const testOnWindows = process.platform === "win32" ? test : test.skip; + testOnWindows("REAL windows probe measures private bytes and a sane commit fraction", async () => { + const snap = await captureMemorySnapshot(); + expect(snap.processSource).toBe("windows-private"); + expect(snap.processPrivateBytes).toBeGreaterThan(0); + expect(snap.systemCommitAvailable).toBe(true); + const fraction = snap.systemCommittedBytes! / snap.systemCommitLimitBytes!; + expect(fraction).toBeGreaterThan(0); + expect(fraction).toBeLessThan(1); + }, 20_000); +}); + +describe("resolveWatchdogConfig", () => { + test("defaults are warn-only and enabled", () => { + const r = resolveWatchdogConfig({} as OcxConfig); + expect(r.enabled).toBe(true); + expect(r.autoRestart).toBe(false); + expect(r.warnFraction).toBe(0.60); + expect(r.criticalFraction).toBe(0.75); + }); + + test("config file values override defaults", () => { + const r = resolveWatchdogConfig({ + memoryWatchdog: { enabled: true, warnFraction: 0.5, criticalFraction: 0.8, autoRestart: true }, + } as OcxConfig); + expect(r.warnFraction).toBe(0.5); + expect(r.criticalFraction).toBe(0.8); + expect(r.autoRestart).toBe(true); + }); + + test("critical <= warn is nudged above warn so the levels never collapse", () => { + const r = resolveWatchdogConfig({ + memoryWatchdog: { warnFraction: 0.7, criticalFraction: 0.6 }, + } as OcxConfig); + expect(r.criticalFraction).toBeGreaterThan(r.warnFraction); + }); + + test("fractions are clamped into [0.10, 0.99]", () => { + const r = resolveWatchdogConfig({ + memoryWatchdog: { warnFraction: 0.01, criticalFraction: 5 }, + } as OcxConfig); + expect(r.warnFraction).toBe(0.10); + expect(r.criticalFraction).toBe(0.99); + }); + + test("restartGraceMs defaults to the quiet-window budget and is config-overridable", () => { + expect(resolveWatchdogConfig({} as OcxConfig).restartGraceMs).toBe(30_000); + const r = resolveWatchdogConfig({ memoryWatchdog: { restartGraceMs: 120_000 } } as OcxConfig); + expect(r.restartGraceMs).toBe(120_000); + }); + + test("restartGraceMs is clamped into [1s, 10min]; junk falls back to the default", () => { + expect(resolveWatchdogConfig({ memoryWatchdog: { restartGraceMs: 5 } } as OcxConfig).restartGraceMs).toBe(1_000); + expect(resolveWatchdogConfig({ memoryWatchdog: { restartGraceMs: 86_400_000 } } as OcxConfig).restartGraceMs).toBe(600_000); + expect(resolveWatchdogConfig({ memoryWatchdog: { restartGraceMs: -1 } } as OcxConfig).restartGraceMs).toBe(30_000); + expect(resolveWatchdogConfig({ memoryWatchdog: { restartGraceMs: Number.NaN } } as OcxConfig).restartGraceMs).toBe(30_000); + expect(resolveWatchdogConfig({ memoryWatchdog: { restartGraceMs: Number.POSITIVE_INFINITY } } as OcxConfig).restartGraceMs).toBe(30_000); + }); + + test("cooldown < grace is raised to the grace so a drain can never overlap the next restart", () => { + const r = resolveWatchdogConfig({ + memoryWatchdog: { restartGraceMs: 120_000, minRestartIntervalMs: 5_000 }, + } as OcxConfig); + expect(r.minRestartIntervalMs).toBe(120_000); + // A cooldown already >= grace is untouched. + const ok = resolveWatchdogConfig({ + memoryWatchdog: { restartGraceMs: 30_000, minRestartIntervalMs: 900_000 }, + } as OcxConfig); + expect(ok.minRestartIntervalMs).toBe(900_000); + }); + + test("intervalMs gets a 1s floor and junk falls back to the default", () => { + expect(resolveWatchdogConfig({ memoryWatchdog: { intervalMs: 5 } } as OcxConfig).intervalMs).toBe(1_000); + expect(resolveWatchdogConfig({ memoryWatchdog: { intervalMs: -60_000 } } as OcxConfig).intervalMs).toBe(60_000); + expect(resolveWatchdogConfig({ memoryWatchdog: { intervalMs: Number.NaN } } as OcxConfig).intervalMs).toBe(60_000); + }); + + test("maxRestarts keeps 0 (meaning: never auto-restart) and rejects negatives/junk", () => { + expect(resolveWatchdogConfig({ memoryWatchdog: { maxRestarts: 0 } } as OcxConfig).maxRestarts).toBe(0); + expect(resolveWatchdogConfig({ memoryWatchdog: { maxRestarts: 2.9 } } as OcxConfig).maxRestarts).toBe(2); + expect(resolveWatchdogConfig({ memoryWatchdog: { maxRestarts: -3 } } as OcxConfig).maxRestarts).toBe(3); + expect(resolveWatchdogConfig({ memoryWatchdog: { maxRestarts: Number.NaN } } as OcxConfig).maxRestarts).toBe(3); + }); + + test("systemCommitHighWater defaults to 0.90 and the env-only override is clamped", () => { + expect(resolveWatchdogConfig({} as OcxConfig).systemCommitHighWater).toBe(0.90); + try { + process.env.OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER = "0.85"; + expect(resolveWatchdogConfig({} as OcxConfig).systemCommitHighWater).toBe(0.85); + process.env.OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER = "0.10"; // absurdly low → clamped + expect(resolveWatchdogConfig({} as OcxConfig).systemCommitHighWater).toBe(0.50); + process.env.OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER = "junk"; // unparseable → default + expect(resolveWatchdogConfig({} as OcxConfig).systemCommitHighWater).toBe(0.90); + } finally { + delete process.env.OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER; + } + }); +}); + +describe("evaluate (pure decision core)", () => { + test("stays ok below the warn fraction", () => { + const s = createWatchdogState(); + const d = evaluate(s, snapshot(50 * MiB), cfg(), 1_000); + expect(d.level).toBe("ok"); + expect(d.action).toBe("none"); + }); + + test("warns once on the first crossing, then latches", () => { + const s = createWatchdogState(); + const first = evaluate(s, snapshot(65 * MiB), cfg(), 1_000); + expect(first.level).toBe("warn"); + expect(first.action).toBe("warn"); + const second = evaluate(s, snapshot(66 * MiB), cfg(), 2_000); + expect(second.level).toBe("warn"); + expect(second.action).toBe("none"); // latched — no repeat log + }); + + test("recovering to ok re-arms the warning", () => { + const s = createWatchdogState(); + evaluate(s, snapshot(65 * MiB), cfg(), 1_000); + evaluate(s, snapshot(50 * MiB), cfg(), 2_000); // back to ok + const again = evaluate(s, snapshot(65 * MiB), cfg(), 3_000); + expect(again.action).toBe("warn"); + }); + + test("critical with auto-restart disabled warns but never restarts", () => { + const s = createWatchdogState(); + const d = evaluate(s, snapshot(80 * MiB), cfg({ autoRestart: false }), 1_000); + expect(d.level).toBe("critical"); + expect(d.action).toBe("warn"); + expect(d.reason).toContain("auto-restart disabled"); + expect(s.restartCount).toBe(0); + }); + + test("critical with auto-restart armed restarts once, then defers by cooldown", () => { + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, minRestartIntervalMs: 600_000 }); + const first = evaluate(s, snapshot(80 * MiB), c, 1_000); + expect(first.action).toBe("restart"); + expect(s.restartCount).toBe(1); + + // Still critical a moment later — cooldown must suppress a second restart. + const second = evaluate(s, snapshot(85 * MiB), c, 2_000); + expect(second.action).not.toBe("restart"); + expect(second.reason).toContain("cooldown"); + expect(s.restartCount).toBe(1); + }); + + test("max-restart guard stops restarting after the cap", () => { + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, minRestartIntervalMs: 0, maxRestarts: 2 }); + // Each call is critical and past the (zero) cooldown. + expect(evaluate(s, snapshot(80 * MiB), c, 10_000).action).toBe("restart"); + expect(evaluate(s, snapshot(80 * MiB), c, 20_000).action).toBe("restart"); + const third = evaluate(s, snapshot(80 * MiB), c, 30_000); + expect(third.action).not.toBe("restart"); + expect(third.reason).toContain("max-restart"); + expect(s.restartCount).toBe(2); + }); + + test("computes growth rate across the sample window", () => { + const s = createWatchdogState(); + evaluate(s, snapshot(50 * MiB), cfg(), 0); + const d = evaluate(s, snapshot(60 * MiB), cfg(), 3_600_000); // +10 MiB over exactly 1h + expect(d.growthMbPerHour).toBe(10); + }); +}); + +describe("tick (driver wiring)", () => { + function deps(over: Partial = {}): WatchdogDeps { + return { + now: () => 1_000, + capture: async () => snapshot(50 * MiB), + supervised: true, + restart: () => {}, + log: () => {}, + ...over, + }; + } + + test("logs on a warn action and invokes restart on a restart action", () => { + const logs: string[] = []; + let restarts = 0; + let restartGrace = 0; + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, minRestartIntervalMs: 0, restartGraceMs: 45_000 }); + + // warn crossing + tick(s, c, deps({ log: l => logs.push(l) }), snapshot(65 * MiB)); + expect(logs.length).toBe(1); + expect(logs[0]).toContain("WARN"); + + // critical crossing → restart, and the running cfg (incl. the quiet-window budget) is threaded through + const d = tick(s, c, deps({ + log: l => logs.push(l), + restart: rc => { restarts += 1; restartGrace = rc.restartGraceMs; }, + now: () => 2_000, + }), snapshot(80 * MiB)); + expect(d.action).toBe("restart"); + expect(restarts).toBe(1); + expect(restartGrace).toBe(45_000); + }); + + test("a synchronously-throwing restart hook is logged, not thrown out of tick", () => { + const logs: string[] = []; + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, minRestartIntervalMs: 0 }); + const d = tick(s, c, deps({ + log: l => logs.push(l), + restart: () => { throw new Error("hook boom"); }, + }), snapshot(80 * MiB)); + expect(d.action).toBe("restart"); + expect(logs.some(l => l.includes("restart hook failed") && l.includes("hook boom"))).toBe(true); + }); + + test("a rejecting async restart hook is logged instead of surfacing as an unhandled rejection", async () => { + const logs: string[] = []; + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, minRestartIntervalMs: 0 }); + tick(s, c, deps({ + log: l => logs.push(l), + restart: () => Promise.reject(new Error("async boom")), + }), snapshot(80 * MiB)); + await Bun.sleep(0); // let the rejection handler run + expect(logs.some(l => l.includes("restart hook failed") && l.includes("async boom"))).toBe(true); + }); + + test("a restart decision is persisted via recordRestart with the decision timestamp", () => { + const recorded: number[] = []; + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, minRestartIntervalMs: 0 }); + tick(s, c, deps({ + now: () => 42_000, + recordRestart: at => recorded.push(at), + }), snapshot(80 * MiB)); + expect(recorded).toEqual([42_000]); + }); + + test("a warn action does not touch the restart history", () => { + const recorded: number[] = []; + const s = createWatchdogState(); + tick(s, cfg(), deps({ recordRestart: at => recorded.push(at) }), snapshot(65 * MiB)); + expect(recorded).toEqual([]); + }); + + test("a system-commit high-water crossing logs its own warning line without restarting", () => { + const logs: string[] = []; + let restarts = 0; + const s = createWatchdogState(); + // Process pressure LOW (50%), commit HIGH (97%): the field incident's exact shape. + const d = tick(s, cfg({ autoRestart: true, minRestartIntervalMs: 0 }), deps({ + log: l => logs.push(l), + restart: () => { restarts += 1; }, + }), snapshot(50 * MiB, { + systemCommittedBytes: 97 * MiB, + systemCommitLimitBytes: 100 * MiB, + systemCommitAvailable: true, + })); + expect(d.action).toBe("none"); // process axis untriggered + expect(d.commitAction).toBe("warn"); + expect(restarts).toBe(0); // commit axis NEVER restarts + expect(logs.length).toBe(1); + expect(logs[0]).toContain("SYSTEM-COMMIT"); + }); +}); + +describe("detectSupervisor", () => { + test("explicit OCX_SUPERVISED=1 wins", () => { + expect(detectSupervisor({ OCX_SUPERVISED: "1" })).toEqual({ supervised: true, hint: "OCX_SUPERVISED" }); + }); + + test("pm2 markers are detected", () => { + expect(detectSupervisor({ pm_id: "0" })).toEqual({ supervised: true, hint: "pm2" }); + expect(detectSupervisor({ PM2_HOME: "/home/app/.pm2" })).toEqual({ supervised: true, hint: "pm2" }); + }); + + test("systemd markers are detected", () => { + expect(detectSupervisor({ INVOCATION_ID: "abc123" })).toEqual({ supervised: true, hint: "systemd" }); + expect(detectSupervisor({ NOTIFY_SOCKET: "/run/systemd/notify" })).toEqual({ supervised: true, hint: "systemd" }); + }); + + test("nothing set → not supervised (safe default)", () => { + expect(detectSupervisor({})).toEqual({ supervised: false, hint: "none" }); + }); + + test("explicit OCX_SUPERVISED=off reports the reason", () => { + expect(detectSupervisor({ OCX_SUPERVISED: "off" })).toEqual({ supervised: false, hint: "OCX_SUPERVISED=off" }); + }); +}); + +describe("recommend (advisory only)", () => { + test("with < 2 samples it echoes current config and says so", () => { + const s = createWatchdogState(); + const r = recommend(s, cfg({ warnFraction: 0.5, criticalFraction: 0.7 }), true); + expect(r.warnFraction).toBe(0.5); + expect(r.criticalFraction).toBe(0.7); + expect(r.rationale).toContain("insufficient samples"); + }); + + test("places warn above the observed peak and keeps critical > warn", () => { + const s = createWatchdogState(); + const c = cfg(); + evaluate(s, snapshot(40 * MiB), c, 0); // fraction 0.40 + evaluate(s, snapshot(50 * MiB), c, 1_000); // fraction 0.50 (observed peak) + const r = recommend(s, c, true); + expect(r.warnFraction).toBeGreaterThan(0.50); + expect(r.criticalFraction).toBeGreaterThan(r.warnFraction); + }); + + test("clamps into [0.10, 0.99] even at very high pressure", () => { + const s = createWatchdogState(); + const c = cfg(); + evaluate(s, snapshot(95 * MiB), c, 0); + evaluate(s, snapshot(98 * MiB), c, 1_000); + const r = recommend(s, c, true); + expect(r.warnFraction).toBeLessThanOrEqual(0.99); + expect(r.criticalFraction).toBeLessThanOrEqual(0.99); + }); + + test("suggests auto-restart only when supervised AND trending up", () => { + const s = createWatchdogState(); + const c = cfg(); + evaluate(s, snapshot(40 * MiB), c, 0); + evaluate(s, snapshot(50 * MiB), c, 3_600_000); // growing + expect(recommend(s, c, true).autoRestart).toBe(true); + expect(recommend(s, c, false).autoRestart).toBe(false); // no supervisor → never suggested + }); + + test("does not suggest auto-restart for a flat process", () => { + const s = createWatchdogState(); + const c = cfg(); + evaluate(s, snapshot(50 * MiB), c, 0); + evaluate(s, snapshot(50 * MiB), c, 3_600_000); // no growth + expect(recommend(s, c, true).autoRestart).toBe(false); + }); +}); + +describe("evaluate — requireSupervisor gate", () => { + test("critical + autoRestart but no supervisor → warns instead of restarting", () => { + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, requireSupervisor: true, minRestartIntervalMs: 0 }); + const d = evaluate(s, snapshot(80 * MiB), c, 1_000, /* supervised */ false); + expect(d.level).toBe("critical"); + expect(d.action).toBe("warn"); + expect(d.reason).toContain("no supervisor"); + expect(s.restartCount).toBe(0); + }); + + test("critical + autoRestart + supervised → restarts", () => { + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, requireSupervisor: true, minRestartIntervalMs: 0 }); + const d = evaluate(s, snapshot(80 * MiB), c, 1_000, /* supervised */ true); + expect(d.action).toBe("restart"); + expect(s.restartCount).toBe(1); + }); + + test("requireSupervisor=false lets auto-restart fire without a supervisor", () => { + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, requireSupervisor: false, minRestartIntervalMs: 0 }); + const d = evaluate(s, snapshot(80 * MiB), c, 1_000, /* supervised */ false); + expect(d.action).toBe("restart"); + }); +}); + +describe("evaluate — system-commit axis (observe-only, v1)", () => { + const withCommit = (pressure: number, fraction: number, available = true) => + snapshot(pressure, { + systemCommittedBytes: Math.round(fraction * 100) * MiB, + systemCommitLimitBytes: 100 * MiB, + systemCommitAvailable: available, + }); + + test("a healthy commit level (0.70) produces no commit warning", () => { + const s = createWatchdogState(); + const d = evaluate(s, withCommit(50 * MiB, 0.70), cfg(), 1_000); + expect(d.commitAction).toBe("none"); + expect(d.systemCommitFraction).toBeCloseTo(0.70); + }); + + test("crossing the high-water warns exactly once, then latches", () => { + const s = createWatchdogState(); + const first = evaluate(s, withCommit(50 * MiB, 0.95), cfg(), 1_000); + expect(first.commitAction).toBe("warn"); + const second = evaluate(s, withCommit(50 * MiB, 0.96), cfg(), 2_000); + expect(second.commitAction).toBe("none"); // latched — no per-interval spam + }); + + test("a MEASURED recovery below the high-water re-arms the warning", () => { + const s = createWatchdogState(); + evaluate(s, withCommit(50 * MiB, 0.95), cfg(), 1_000); // warn + latch + evaluate(s, withCommit(50 * MiB, 0.70), cfg(), 2_000); // measured recovery → re-arm + const again = evaluate(s, withCommit(50 * MiB, 0.95), cfg(), 3_000); + expect(again.commitAction).toBe("warn"); + }); + + test("measurement loss holds the latch: no re-arm, no duplicate warn when the probe flaps", () => { + const s = createWatchdogState(); + evaluate(s, withCommit(50 * MiB, 0.95), cfg(), 1_000); // warn + latch + // Probe drops out (systemCommitAvailable=false) — NOT a recovery. + evaluate(s, snapshot(50 * MiB), cfg(), 2_000); + // Probe comes back, still above high-water: latch must have held (no duplicate warning). + const back = evaluate(s, withCommit(50 * MiB, 0.95), cfg(), 3_000); + expect(back.commitAction).toBe("none"); + }); + + test("the commit axis NEVER restarts — even with auto-restart armed and commit at 0.99", () => { + const s = createWatchdogState(); + const c = cfg({ autoRestart: true, minRestartIntervalMs: 0 }); + const d = evaluate(s, withCommit(50 * MiB, 0.99), c, 1_000, /* supervised */ true); + expect(d.action).toBe("none"); // process axis at 50% — quiet + expect(d.commitAction).toBe("warn"); + expect(s.restartCount).toBe(0); + }); + + test("unavailable commit values are null-safe: no NaN, all commit fields null", () => { + const s = createWatchdogState(); + const d = evaluate(s, snapshot(50 * MiB), cfg(), 1_000); + expect(d.systemCommitFraction).toBeNull(); + expect(d.systemCommitUsedMb).toBeNull(); + expect(d.systemCommitLimitMb).toBeNull(); + expect(d.commitAction).toBe("none"); + }); + + test("the env-tuned high-water is honored", () => { + const s = createWatchdogState(); + const d = evaluate(s, withCommit(50 * MiB, 0.86), cfg({ systemCommitHighWater: 0.85 }), 1_000); + expect(d.commitAction).toBe("warn"); + }); +}); + +describe("applyWatchdogRuntimeConfig (live tuning)", () => { + test("returns null when the watchdog is not running", () => { + stopMemoryWatchdog(); + expect(applyWatchdogRuntimeConfig({ warnFraction: 0.5 })).toBeNull(); + }); + + test("live-updates fractions with clamping and critical > warn", () => { + try { + startMemoryWatchdog({} as OcxConfig, { capture: async () => snapshot(50 * MiB), now: () => 0, log: () => {} }); + const applied = applyWatchdogRuntimeConfig({ warnFraction: 0.01, criticalFraction: 5 }); + expect(applied).not.toBeNull(); + expect(applied!.warnFraction).toBe(0.10); // clamped up + expect(applied!.criticalFraction).toBe(0.99); // clamped down + const collapsed = applyWatchdogRuntimeConfig({ warnFraction: 0.7, criticalFraction: 0.6 }); + expect(collapsed!.criticalFraction).toBeGreaterThan(collapsed!.warnFraction); + } finally { + stopMemoryWatchdog(); + } + }); + + test("updates requireSupervisor / autoRestart in place", () => { + try { + startMemoryWatchdog({} as OcxConfig, { capture: async () => snapshot(50 * MiB), now: () => 0, log: () => {} }); + const applied = applyWatchdogRuntimeConfig({ autoRestart: true, requireSupervisor: false }); + expect(applied!.autoRestart).toBe(true); + expect(applied!.requireSupervisor).toBe(false); + } finally { + stopMemoryWatchdog(); + } + }); + + test("live-tuned restartGraceMs is clamped and the cooldown is raised to match", () => { + try { + startMemoryWatchdog({} as OcxConfig, { capture: async () => snapshot(50 * MiB), now: () => 0, log: () => {} }); + const clamped = applyWatchdogRuntimeConfig({ restartGraceMs: 86_400_000 }); + expect(clamped!.restartGraceMs).toBe(600_000); + // Default cooldown is exactly 10 min — equal to the clamped grace, so it is untouched. + expect(clamped!.minRestartIntervalMs).toBe(600_000); + const raised = applyWatchdogRuntimeConfig({ restartGraceMs: 300_000, minRestartIntervalMs: 60_000 }); + expect(raised!.restartGraceMs).toBe(300_000); + expect(raised!.minRestartIntervalMs).toBe(300_000); // raised from 60s to the grace + const junk = applyWatchdogRuntimeConfig({ restartGraceMs: Number.NaN }); + expect(junk!.restartGraceMs).toBe(300_000); // invalid live input keeps the running value + } finally { + stopMemoryWatchdog(); + } + }); +}); + +describe("startMemoryWatchdog — probe-driven loop (§2, §8)", () => { + test("the first probe fires immediately and its completion evaluates + fills the report cache", async () => { + try { + startMemoryWatchdog({} as OcxConfig, { + capture: async () => snapshot(65 * MiB, { capturedAt: 7_000 }), + now: () => 7_000, + log: () => {}, + }); + await Bun.sleep(0); // let the immediate probe's microtasks complete + const report = memoryWatchdogReport(); + expect(report!.decision).not.toBeNull(); // evaluated without waiting one interval + expect(report!.decision!.level).toBe("warn"); + expect(report!.lastProbeAt).toBe(7_000); + expect(report!.memory).not.toBeNull(); + expect(report!.responseState).not.toBeNull(); // §7 cache computed at probe completion + } finally { + stopMemoryWatchdog(); + } + }); + + test("a successful system probe stamps lastSuccessfulSystemProbeAt; a fallback does not", async () => { + try { + startMemoryWatchdog({} as OcxConfig, { + capture: async () => snapshot(50 * MiB, { capturedAt: 9_000 }), // systemCommitAvailable=false + now: () => 9_000, + log: () => {}, + }); + await Bun.sleep(0); + expect(memoryWatchdogReport()!.lastProbeAt).toBe(9_000); + expect(memoryWatchdogReport()!.lastSuccessfulSystemProbeAt).toBeNull(); + } finally { + stopMemoryWatchdog(); + } + + try { + startMemoryWatchdog({} as OcxConfig, { + capture: async () => snapshot(50 * MiB, { + capturedAt: 11_000, + systemCommittedBytes: 50 * MiB, + systemCommitLimitBytes: 100 * MiB, + systemCommitAvailable: true, + }), + now: () => 11_000, + log: () => {}, + }); + await Bun.sleep(0); + expect(memoryWatchdogReport()!.lastSuccessfulSystemProbeAt).toBe(11_000); + } finally { + stopMemoryWatchdog(); + } + }); + + test("a probe completing after stop is dropped: no evaluation, no state revival", async () => { + let release: ((s: MemorySnapshot) => void) | null = null; + const logs: string[] = []; + startMemoryWatchdog({} as OcxConfig, { + capture: () => new Promise(resolve => { release = resolve; }), + now: () => 1_000, + log: l => logs.push(l), + }); + expect(release).not.toBeNull(); // immediate probe is in flight + stopMemoryWatchdog(); + release!(snapshot(80 * MiB)); // late result arrives AFTER stop + await Bun.sleep(0); + expect(memoryWatchdogReport()).toBeNull(); // nothing revived + expect(logs.length).toBe(0); // and nothing was evaluated/logged + }); + + test("self-rescheduling: no second capture starts while one is still in flight", async () => { + let calls = 0; + startMemoryWatchdog({} as OcxConfig, { + capture: () => { calls += 1; return new Promise(() => { /* never resolves */ }); }, + now: () => 1_000, + log: () => {}, + }); + try { + await Bun.sleep(30); // give any (buggy) parallel scheduling a chance to fire + expect(calls).toBe(1); // next probe is armed only after the previous one completes + } finally { + stopMemoryWatchdog(); + } + }); + + test("stop aborts the in-flight capture's signal so the child probe can be killed", async () => { + let aborted = false; + startMemoryWatchdog({} as OcxConfig, { + capture: (_now, signal) => new Promise(() => { + signal?.addEventListener("abort", () => { aborted = true; }, { once: true }); + }), + now: () => 1_000, + log: () => {}, + }); + stopMemoryWatchdog(); + expect(aborted).toBe(true); + }); + + test("audit #1: an auto-restart that stops the watchdog mid-tick does not crash or reschedule", async () => { + const root = mkdtempSync(join(tmpdir(), "ocx-watchdog-restart-race-")); + let captures = 0; + try { + setRestartHistoryPathForTests(join(root, "absent.json")); // fresh guards regardless of runner env + startMemoryWatchdog( + { memoryWatchdog: { enabled: true, autoRestart: true, requireSupervisor: false } } as OcxConfig, + { + capture: async () => { captures += 1; return snapshot(90 * MiB); }, // 90/100 = critical (>0.75) + now: () => 1_000, + supervised: true, + restart: () => stopMemoryWatchdog(), // exactly what defaultRestart does synchronously + recordRestart: () => {}, // keep the test disk-free + log: () => {}, + }, + ); + await Bun.sleep(0); // let the immediate probe's microtasks complete + // With the pre-fix code this turn died on `running.cfg` (unhandled TypeError at reschedule). + expect(memoryWatchdogReport()).toBeNull(); // the restart's stop was respected — nothing revived + expect(captures).toBe(1); // and no extra capture was scheduled by the dead loop + } finally { + stopMemoryWatchdog(); + setRestartHistoryPathForTests(null); + rmSync(root, { recursive: true, force: true }); + } + }); + + test("audit #2: a throwing capture still evaluates exactly once via the RSS fallback", async () => { + try { + startMemoryWatchdog({} as OcxConfig, { + capture: async () => { throw new Error("unexpected boom"); }, + now: () => 5_000, + log: () => {}, + }); + await Bun.sleep(0); + const report = memoryWatchdogReport(); + expect(report).not.toBeNull(); + expect(report!.decision).not.toBeNull(); // the cycle was evaluated, not silenced + expect(report!.samplesCount).toBeGreaterThan(0); + expect(report!.lastProbeAt).toBe(5_000); + expect(report!.memory!.probeError).toBe("capture-threw"); + expect(report!.memory!.processSource).toBe("rss-fallback"); + } finally { + stopMemoryWatchdog(); + } + }); + + test("audit #4: applyWatchdogRuntimeConfig defends systemCommitHighWater against NaN/undefined", () => { + try { + startMemoryWatchdog({} as OcxConfig, { capture: async () => snapshot(50 * MiB), now: () => 0, log: () => {} }); + const nan = applyWatchdogRuntimeConfig({ systemCommitHighWater: Number.NaN }); + expect(Number.isFinite(nan!.systemCommitHighWater)).toBe(true); // previous value kept, not NaN + expect(nan!.systemCommitHighWater).toBe(0.90); + const undef = applyWatchdogRuntimeConfig({ systemCommitHighWater: undefined as unknown as number }); + expect(undef!.systemCommitHighWater).toBe(0.90); + const clamped = applyWatchdogRuntimeConfig({ systemCommitHighWater: 5 }); + expect(clamped!.systemCommitHighWater).toBe(0.99); // clamped down into [0.50, 0.99] + const junkRestarts = applyWatchdogRuntimeConfig({ maxRestarts: Number.NaN }); + expect(junkRestarts!.maxRestarts).toBe(3); // finite-guarded like resolveWatchdogConfig + } finally { + stopMemoryWatchdog(); + } + }); +}); + +describe("startMemoryWatchdog — cross-process history seeding", () => { + test("seeds cooldown clock and restarts-in-window count from the persisted history", () => { + const root = mkdtempSync(join(tmpdir(), "ocx-watchdog-seed-")); + const path = join(root, "memory-watchdog-restarts.json"); + const NOW = 100 * 3_600_000; + try { + writeFileSync(path, JSON.stringify({ version: 1, restarts: [NOW - 120_000, NOW - 60_000] }), "utf-8"); + setRestartHistoryPathForTests(path); + startMemoryWatchdog({} as OcxConfig, { capture: async () => snapshot(50 * MiB), now: () => NOW, log: () => {} }); + const report = memoryWatchdogReport(); + expect(report!.restartCount).toBe(2); + } finally { + stopMemoryWatchdog(); + setRestartHistoryPathForTests(null); + rmSync(root, { recursive: true, force: true }); + } + }); + + test("a missing history file seeds a fresh slate", () => { + const root = mkdtempSync(join(tmpdir(), "ocx-watchdog-seed-")); + try { + setRestartHistoryPathForTests(join(root, "absent.json")); + startMemoryWatchdog({} as OcxConfig, { capture: async () => snapshot(50 * MiB), now: () => 1_000, log: () => {} }); + expect(memoryWatchdogReport()!.restartCount).toBe(0); + } finally { + stopMemoryWatchdog(); + setRestartHistoryPathForTests(null); + rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/responses-state.test.ts b/tests/responses-state.test.ts index 35c9f4f49..3bb3d9628 100644 --- a/tests/responses-state.test.ts +++ b/tests/responses-state.test.ts @@ -14,6 +14,7 @@ import { previousResponseConversationId, previousResponseProviderState, rememberResponseState, + responseStateMetrics, } from "../src/responses/state"; describe("Responses previous_response_id state", () => { @@ -360,4 +361,45 @@ describe("Responses previous_response_id state", () => { clearPrior: newlyCompactedRequest.contextUsageReset === true, }).carryForwardTokens).toBeUndefined(); }); + + test("responseStateMetrics reports an empty store as zeroed", () => { + expect(responseStateMetrics()).toEqual({ + count: 0, + totalBytes: 0, + largestBytes: 0, + oldestAgeMs: 0, + }); + }); + + test("responseStateMetrics counts entries and tracks the largest by serialized bytes", () => { + const small = buildResponseJSON([{ type: "text_delta", text: "hi" }, { type: "done" }], "gpt-5.5"); + rememberResponseState({ model: "gpt-5.5", input: "small" }, small); + + const bigText = "y".repeat(200 * 1024); + const big = buildResponseJSON([{ type: "text_delta", text: bigText }, { type: "done" }], "gpt-5.5"); + rememberResponseState({ model: "gpt-5.5", input: "big" }, big); + + const metrics = responseStateMetrics(); + expect(metrics.count).toBe(2); + expect(metrics.largestBytes).toBeGreaterThan(200 * 1024); + expect(metrics.totalBytes).toBeGreaterThanOrEqual(metrics.largestBytes); + expect(metrics.oldestAgeMs).toBeGreaterThanOrEqual(0); + }); + + test("responseStateMetrics is side-effect free (does not lazy-load the disk snapshot)", () => { + // Seed a snapshot on disk, then wipe memory. A pure metrics read must NOT resurrect it: it + // reflects live RAM only, so a diagnostics probe never perturbs the store or triggers a load. + const first = buildResponseJSON([{ type: "text_delta", text: "persisted" }, { type: "done" }], "gpt-5.5"); + rememberResponseState({ model: "gpt-5.5", input: "hi" }, first); + flushResponseState(); + clearResponseStateMemoryForTests(); + + expect(responseStateMetrics().count).toBe(0); + + // A real read path still loads it, proving the snapshot was intact and metrics simply abstained. + expect((expandPreviousResponseInput({ + model: "gpt-5.5", previous_response_id: first.id, input: "next", + }) as { input: unknown[] }).input).toHaveLength(3); + expect(responseStateMetrics().count).toBe(1); + }); });