Skip to content

feat: memory observability + opt-in watchdog (v1, observe-first) #388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dev-shinyu wants to merge 9 commits into from
Closed

feat: memory observability + opt-in watchdog (v1, observe-first) #388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e227ff3
feat(memory): observability + opt-in watchdog with dashboard (Monitor…
dev-shinyu Jul 24, 2026
59daab4
feat(memory): quiet-window watchdog restart via configurable restartG…
dev-shinyu Jul 24, 2026
1ff459d
feat(gui): honest restart-behavior note under auto-restart toggle
dev-shinyu Jul 24, 2026
251d191
feat(memory): watchdog runtime safety — grace bounds, cooldown>=grace…
dev-shinyu Jul 24, 2026
efc546a
feat(gui): grace-aware honesty note, restart-grace control, superviso…
dev-shinyu Jul 24, 2026
6fe99cd
feat(memory): async non-blocking measurement + observe-only system-co…
dev-shinyu Jul 24, 2026
6a95a86
feat(gui): memory observability row + docs for the async probe and co…
dev-shinyu Jul 24, 2026
9942240
fix(memory): harden probe loop against auto-restart race + capture-th…
dev-shinyu Jul 24, 2026
a9590de
docs(structure): memory watchdog SOT — design rationale, platform sco…
dev-shinyu Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions docs-site/src/content/docs/ko/reference/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ namespaced selected id를 bare id로 바꿉니다.
| `stallTimeoutSec?` | `number` | `300` | 업스트림 데이터가 오지 않을 때 bridge가 중단하고 `response.incomplete`를 내보내기까지의 초. 최소 1. |
| `connectTimeoutMs?` | `number` | `200000` | DNS/TCP/TLS와 최종 응답 헤더만 기다리는 시도별 deadline. 응답 body 생성 전 종료됩니다. |
| `shutdownTimeoutMs?` | `number` | `5000` | 진행 중인 turn을 중단하기 전 graceful drain deadline. |
| `memoryWatchdog?` | object | 경고 전용 | 메모리 압력 관찰 + opt-in 우아한 재시작 (아래 [memoryWatchdog](#memorywatchdog) 참고). |
| `websockets?` | `boolean` | `false` | `supports_websockets`를 알려 Codex가 Responses WebSocket 경로를 쓰게 합니다. 생략하거나 `false`이면 HTTP/SSE를 유지합니다. |
| `apiKeys?` | `OcxApiKey[]` | `[]` | 비-loopback 바인드에서 관리 API와 data plane 인증에 추가로 허용할 생성형 `ocx_…` 자격 증명. 대시보드가 관리하며 항목 필드는 아래에 설명합니다. |
| `codexAutoStart?` | `boolean` | `true` | Codex shim이 Codex 실행 전에 `ocx ensure`를 실행하게 합니다. `false`이면 `ocx ensure`가 아무 작업도 하지 않습니다. |
Expand Down Expand Up @@ -97,6 +98,65 @@ pool 계정 추가와 quota 갱신은 대시보드의 **Codex Auth** 페이지
| `codexWarmupMaxAgeSeconds?` | `number` | `691200` | 계정을 다시 검증할 최대 기간(8일). |
| `codexWarmupModel?` | `string` | `gpt-5.4-mini` | 선택형 warmup에 쓸 네이티브 모델. |

### `memoryWatchdog`

프로세스 메모리 압력을 전체 시스템 RAM 대비 비율로 관찰하고 임계값을 넘으면 경고합니다. 기본
동작은 경고 전용입니다. 명시적으로 활성화한 경우에만, 애플리케이션 코드로는 해결할 수 없는
네이티브 committed 메모리 미반환 문제의 완화책으로 위험 임계값에서 프록시를 우아하게 재시작할
수 있습니다.

측정은 완전히 비동기입니다: Windows에서는 숨겨진 자식 PowerShell 프로세스 하나가 이 PID의
Private Bytes와 시스템 전역 **Committed Bytes / Commit Limit**을 한 번의 spawn으로 수집하므로,
느린 측정이 프록시의 이벤트 루프를 막는 일이 없습니다(타임아웃된 프로브는 종료되고 해당 주기는
RSS 폴백으로 강등). 첫 측정은 시작 직후 즉시 실행됩니다. 시스템 커밋은 별도의 **관찰 전용**
축입니다: 내부 high-water(기본: 커밋 한도의 0.90)를 넘으면 래치된 경고를 한 번 기록하고, 측정된
회복이 있을 때만 재무장하며, **절대 재시작을 유발하지 않습니다** — 커밋 압력의 원인이 다른
프로세스일 수 있어 OpenCodex를 재시작해도 해소되지 않기 때문입니다. 실험적 env 전용 오버라이드
`OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER`(`[0.50, 0.99]`로 clamp)로 조정 가능하며, 실측으로 축이
검증되기 전까지 config/UI 설정은 추가하지 않습니다.

| Field | Type | Default | Meaning |
| --- | --- | --- | --- |
| `enabled?` | `boolean` | `true` | 전체 스위치. 기본 관찰 모드는 경고 전용이며 데이터를 잃지 않습니다. |
| `intervalMs?` | `number` | `60000` | 샘플링 간격. 최소 1000ms. |
| `warnFraction?` | `number` | `0.60` | 압력/전체 RAM이 이 비율을 넘으면 경고. `[0.10, 0.99]`로 clamp. |
| `criticalFraction?` | `number` | `0.75` | 위험 임계값. opt-in 재시작을 준비시킵니다. 항상 `warnFraction`보다 높게 유지. |
| `autoRestart?` | `boolean` | `false` | 위험 임계값에서 drain 후 우아한 재시작 opt-in. |
| `requireSupervisor?` | `boolean` | `true` | supervisor가 감지될 때만 재시작 — 프록시가 그냥 죽은 채 남지 않도록. |
| `restartGraceMs?` | `number` | `30000` | quiet-window drain 예산: 진행 중인 turn이 끝나도록 최대 이 시간까지 기다린 뒤 재시작합니다. drain 중 새 요청은 `503` + `Retry-After`를 받습니다. `[1000, 600000]`으로 clamp. |
| `minRestartIntervalMs?` | `number` | `600000` | 재시작 요청 간 cooldown. 최소 `restartGraceMs` 이상으로 보정됩니다. |
| `maxRestarts?` | `number` | `3` | 약 6시간 롤링 윈도우 안에서 허용되는 메모리 기반 재시작 횟수. 초과 시 경고 전용으로 강등. `0`이면 자동 재시작 비활성화. |

환경변수 오버라이드(최우선): `OCX_MEMORY_WATCHDOG_ENABLED` / `_DISABLED`, `_INTERVAL_MS`,
`_WARN_FRACTION`, `_CRITICAL_FRACTION`, `_AUTO_RESTART`, `_REQUIRE_SUPERVISOR`,
`_RESTART_GRACE_MS`, `_MIN_RESTART_INTERVAL_MS`, `_MAX_RESTARTS`, `_COMMIT_HIGH_WATER`(실험적,
env 전용 — 모두 `OCX_MEMORY_WATCHDOG` 접두사). 환경변수, `config.json`, 대시보드/관리 API 등
모든 진입 경로가 동일한 최종 검증과 clamp를 거칩니다.

`/api/memory` 리포트(및 대시보드 Memory 카드)는 마지막 프로브의 측정값을 추가로 노출합니다:
실제 Private Bytes와 판단에 실제 사용된 압력 값의 구분(`processSource`가 출처를 명시 —
`windows-private`, `proc-status`, 또는 강등된 `rss-fallback`), 시스템 커밋 사용량/한도/비율,
가용 물리 메모리, 응답 캐시 지표(항목 수/직렬화 바이트/최대 항목), 강등 시 새니타이즈된
`probeError` 코드, 그리고 신선도 확인용 `lastProbeAt` / `lastSuccessfulSystemProbeAt`
타임스탬프.

:::note[Supervisor와 Windows]
메모리 기반 재시작은 **exit code 75로 종료하는 것**이 전부입니다 — 외부 supervisor에게 재기동을
요청하는 신호일 뿐입니다. supervisor가 없으면 프록시는 그냥 멈춘 채 남으므로
`requireSupervisor`가 기본 `true`입니다.

- supervisor 자동 감지는 **pm2**, **systemd**, 명시적 `OCX_SUPERVISED=1` 플래그만 지원합니다.
**NSSM, Windows 서비스(`sc.exe`), 작업 스케줄러 등 Windows 서비스 관리자는 자동 감지되지
않습니다** — 서비스 관리자가 종료 시 재시작하도록 구성되어 있다면 서비스 환경에
`OCX_SUPERVISED=1`을 설정하세요.
- watchdog의 cooldown과 `maxRestarts` 카운터는 작은 best-effort 이력 파일(타임스탬프만 저장)로
재시작 경계를 넘어 시드됩니다. 이 파일을 읽거나 쓸 수 없으면 새 프로세스에서 0부터 다시
시작하므로, 재시작 폭주를 제한하긴 하지만 **프로세스 경계를 넘는 영구적 보장은 아닙니다**.
supervisor 자체의 재시작 제한/backoff 정책(예: pm2 `max_restarts` +
`exp_backoff_restart_delay`, systemd `StartLimitIntervalSec` + `StartLimitBurst`)을 바깥층
보호로 함께 설정하세요.
:::

## 원격 접근

opencodex는 기본적으로 `127.0.0.1`(loopback 전용)에 바인드합니다. `hostname`을 `0.0.0.0` 같은
Expand Down
62 changes: 62 additions & 0 deletions docs-site/src/content/docs/reference/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ differing backup and rewrites known legacy namespaced selected ids to bare ids.
| `stallTimeoutSec?` | `number` | `300` | Seconds without upstream data before the bridge aborts and emits `response.incomplete`. Minimum 1. |
| `connectTimeoutMs?` | `number` | `200000` | Per-attempt deadline for DNS/TCP/TLS and final response headers only; it ends before response-body generation. |
| `shutdownTimeoutMs?` | `number` | `5000` | Graceful drain deadline before active turns are aborted. |
| `memoryWatchdog?` | object | warn-only | Memory-pressure observation and opt-in graceful restart (see [memoryWatchdog](#memorywatchdog) below). |
| `websockets?` | `boolean` | `false` | Advertise `supports_websockets` so Codex uses the Responses WebSocket path. Omit or set `false` to keep HTTP/SSE. |
| `apiKeys?` | `OcxApiKey[]` | `[]` | Additional generated `ocx_…` credentials accepted by management and data-plane auth on non-loopback binds. Managed by the dashboard; entry fields are listed below. |
| `codexAutoStart?` | `boolean` | `true` | Let the Codex shim run `ocx ensure` before launching Codex. `false` makes `ocx ensure` a no-op. |
Expand Down Expand Up @@ -109,6 +110,67 @@ normally dashboard-managed.
| `codexWarmupMaxAgeSeconds?` | `number` | `691200` | Revalidate an account after 8 days. |
| `codexWarmupModel?` | `string` | `gpt-5.4-mini` | Native model used for optional warmup. |

### `memoryWatchdog`

Observes process memory pressure as a fraction of total system RAM and warns when it crosses
thresholds. By default it only warns. Optionally — and only when explicitly enabled — it can
gracefully restart the proxy at the critical threshold, as a mitigation for native
committed-memory retention that cannot be fixed in application code.

Measurement is fully asynchronous: on Windows a hidden child PowerShell process collects this
PID's Private Bytes plus the system-wide **Committed Bytes / Commit Limit** in one spawn, so the
proxy's event loop is never blocked by a slow probe (a timed-out probe is killed and that cycle
degrades to an RSS fallback). The first probe fires immediately at startup. System commit is a
separate **observe-only** axis: crossing an internal high-water (default 0.90 of the commit
limit) logs one latched warning and re-arms only on a measured recovery — it **never triggers a
restart**, because the commit pressure may come from another process that restarting OpenCodex
would not free. Tunable via the experimental env-only override
`OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER` (clamped to `[0.50, 0.99]`; no config/UI knob until the
axis is validated by field measurement).

| Field | Type | Default | Meaning |
| --- | --- | --- | --- |
| `enabled?` | `boolean` | `true` | Master switch. The default observation mode is warn-only and cannot lose data. |
| `intervalMs?` | `number` | `60000` | Sampling interval. Floored at 1000 ms. |
| `warnFraction?` | `number` | `0.60` | Warn when pressure / total RAM crosses this fraction. Clamped to `[0.10, 0.99]`. |
| `criticalFraction?` | `number` | `0.75` | Critical threshold; arms the opt-in restart. Kept above `warnFraction`. |
| `autoRestart?` | `boolean` | `false` | Opt in to a graceful drain-then-restart at the critical threshold. |
| `requireSupervisor?` | `boolean` | `true` | Only restart when a process supervisor is detected, so the proxy is never left dead. |
| `restartGraceMs?` | `number` | `30000` | Quiet-window drain budget: in-flight turns get up to this long to finish before the restart proceeds; new requests receive `503` + `Retry-After` while draining. Clamped to `[1000, 600000]`. |
| `minRestartIntervalMs?` | `number` | `600000` | Cooldown between restart requests. Raised to at least `restartGraceMs`. |
| `maxRestarts?` | `number` | `3` | Memory-driven restarts allowed within a rolling ~6 h window before the watchdog degrades to warn-only. `0` disables auto-restart. |

Environment overrides (highest precedence): `OCX_MEMORY_WATCHDOG_ENABLED` / `_DISABLED`,
`_INTERVAL_MS`, `_WARN_FRACTION`, `_CRITICAL_FRACTION`, `_AUTO_RESTART`, `_REQUIRE_SUPERVISOR`,
`_RESTART_GRACE_MS`, `_MIN_RESTART_INTERVAL_MS`, `_MAX_RESTARTS`, `_COMMIT_HIGH_WATER`
(experimental, env-only — all prefixed `OCX_MEMORY_WATCHDOG`). All entry paths — environment,
`config.json`, and the dashboard / management API — go through the same final validation and
clamping.

The `/api/memory` report (and the dashboard's Memory card) additionally exposes the last probe's
measurements: actual Private Bytes vs. the pressure value the decisions use (`processSource`
names the origin — `windows-private`, `proc-status`, or the degraded `rss-fallback`), system
commit used/limit/fraction, available physical memory, response-cache metrics (entry count /
serialized bytes / largest entry), a sanitized `probeError` code when degraded, plus
`lastProbeAt` / `lastSuccessfulSystemProbeAt` timestamps for staleness.

:::note[Supervisors and Windows]
A memory-driven restart works by **exiting with code 75** — a request to an external supervisor
to respawn the process, nothing more. Without a supervisor the proxy would simply stay stopped,
which is why `requireSupervisor` defaults to `true`.

- Supervisor auto-detection covers **pm2**, **systemd**, and the explicit `OCX_SUPERVISED=1`
flag. **NSSM, Windows services (`sc.exe`), Task Scheduler and similar Windows service
managers are not auto-detected** — set `OCX_SUPERVISED=1` in the service environment when the
service manager is configured to restart the process on exit.
- The watchdog's cooldown and `maxRestarts` counters are seeded across restarts from a small
best-effort history file (timestamps only). When that file cannot be read or written, they
restart from zero in the new process — so they limit restart churn but are **not a permanent
cross-process guarantee**. Configure your supervisor's own restart limit / backoff policy
(e.g. pm2 `max_restarts` + `exp_backoff_restart_delay`, systemd `StartLimitIntervalSec` +
`StartLimitBurst`) as the outer layer of protection.
:::

Comment on lines +113 to +173

@coderabbitai coderabbitai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Check whether a ru configuration.md exists and whether it already mentions memoryWatchdog.
fd -i 'configuration.md' docs-site/src/content/docs
echo "---"
fd -i 'configuration.md' docs-site/src/content/docs/ru --exec grep -n "memoryWatchdog" {} \;

Repository: lidge-jun/opencodex

Length of output: 446

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "## docs-site locale configuration files"
fd -i 'configuration.md' docs-site/src/content/docs

echo
echo "## ru memoryWatchdog occurrences"
rg -n "memoryWatchdog|OCX_MEMORY_WATCHDOG|Memory card|Private Bytes|Commit Limit" docs-site/src/content/docs/ru/reference/configuration.md

echo
echo "## ko memoryWatchdog occurrences"
rg -n "memoryWatchdog|OCX_MEMORY_WATCHDOG|Memory card|Private Bytes|Commit Limit" docs-site/src/content/docs/ko/reference/configuration.md

echo
echo "## ja memoryWatchdog occurrences"
rg -n "memoryWatchdog|OCX_MEMORY_WATCHDOG|Memory card|Private Bytes|Commit Limit" docs-site/src/content/docs/ja/reference/configuration.md

echo
echo "## zh-cn memoryWatchdog occurrences"
rg -n "memoryWatchdog|OCX_MEMORY_WATCHDOG|Memory card|Private Bytes|Commit Limit" docs-site/src/content/docs/zh-cn/reference/configuration.md

echo
echo "## follow-up note around the translation bullet"
cat -n structure/09_memory-watchdog.md | sed -n '120,130p'

Repository: lidge-jun/opencodex

Length of output: 516

Sync the Russian config docs with the new memoryWatchdog section

docs-site/src/content/docs/reference/configuration.md#L113-L173 adds a new memoryWatchdog block, but docs-site/src/content/docs/ru/reference/configuration.md exists and does not include it, so the Russian docs will drift from the English source. Add the same section there and include ru in structure/09_memory-watchdog.md#L122-L130 so the translation gap is tracked.

📍 Affects 2 files
  • docs-site/src/content/docs/reference/configuration.md#L113-L173 (this comment)
  • structure/09_memory-watchdog.md#L122-L130
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs-site/src/content/docs/reference/configuration.md` around lines 113 -
173, Sync the Russian configuration documentation with the English
memoryWatchdog section by adding the equivalent content to
docs-site/src/content/docs/ru/reference/configuration.md, preserving all fields,
defaults, environment overrides, API details, and supervisor notes. Also update
structure/09_memory-watchdog.md in the specified range to include ru as a
tracked translation locale; no change is needed to the English configuration
source.

## Remote access

By default opencodex binds to `127.0.0.1` (loopback only). When `hostname` is set to a non-loopback
Expand Down
Loading
Loading