Skip to content

feat(storage): archived cleanup with quarantine (phase 2 of #42) - #529

Merged
Ingwannu merged 17 commits into
lidge-jun:devfrom
Wibias:feat/storage-phase2-cleanup
Jul 27, 2026
Merged

feat(storage): archived cleanup with quarantine (phase 2 of #42)#529
Ingwannu merged 17 commits into
lidge-jun:devfrom
Wibias:feat/storage-phase2-cleanup

Conversation

@Wibias

@Wibias Wibias commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Phase 2 (Option A) for Add Storage page for Codex session usage and cleanup policy #42: preview-first cleanup of the oldest archived sessions by percentage (slider + 10/25/50% presets), quarantine by default to CODEX_HOME/.trash/<epoch>/, permanent delete only behind an explicit checkbox.
  • Active sessions/ are never touched. Cleanup probes state_*.sqlite writability first and returns 409 codex_busy with zero FS changes when Codex holds the lock.
  • Reconciles matching threads rows (and thread_spawn_edges when present) with archived JSONL paths.
  • Phase 1 hygiene in the same PR: fix stale scanner comment, exclude .trash from scan totals/other, remove dead scanStorage imports, document Storage on the web dashboard guide.

API

  • POST /api/storage/cleanup/preview { percent }
  • POST /api/storage/cleanup { percent, mode: "quarantine" | "permanent" }

Test plan

  • bun test --isolate tests/storage-scanner.test.ts tests/storage-cleanup.test.ts tests/api-storage-cleanup.test.ts tests/api-storage.test.ts
  • bun run typecheck
  • bun x tsc -p gui --noEmit
  • Manual: Storage page → preview → quarantine; confirm .trash + rescan; busy Codex → refused

Summary by CodeRabbit

  • New Features
    • Added Storage diagnostics for CODEX_HOME (read-only).
    • Introduced “Archived cleanup” with percent-based preview of the oldest items, then quarantine (default) or permanent deletion after explicit confirmation; active items remain write-protected.
  • Bug Fixes
    • Quarantined items in .trash/ no longer affect Storage totals or “largest files.”
  • Documentation
    • Updated the Web Dashboard guide to describe Storage and archived-cleanup behavior/restore limitations.
  • Localization
    • Added/updated Storage UI text for archived-cleanup flow across supported languages.

Add preview-first archived-session cleanup (oldest N%) with quarantine under CODEX_HOME/.trash and optional permanent delete. Fold Phase 1 hygiene: immutable comment, skip .trash in scan, drop dead scanStorage imports, and document Storage on the dashboard guide.
@github-actions github-actions Bot added the enhancement New feature or request label Jul 27, 2026
@Wibias

Wibias commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator Author

Tracks Phase 2 of #42 (Option A: quarantine default + permanent checkbox). Phase 1 hygiene included in the same PR.

@coderabbitai

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

Wibias added 2 commits July 27, 2026 04:56
Strip host paths from preview JSON, harden FS/DB error mapping, localize cleanup copy, and cover permanent/busy API paths.
Require a candidate digest, stage-then-commit with full rollback, exact archived path matching, spawn-edge/FK cleanup, and mapped 409/400 errors with GUI dialog surfacing.
coderabbitai[bot]

This comment was marked as resolved.

Wibias added 2 commits July 27, 2026 05:39
Localize cleanup errors by code, keep dialog focus stable, harden path/history probes, cap preview payloads, and cover mid-batch rename rollback.
Preserve unrestored stage directories on rollback/purge failure and return the relative trashDir so recovery info reaches the API.
coderabbitai[bot]

This comment was marked as resolved.

Avoid reading/writing the ref during render so GUI eslint stays green in CI.
coderabbitai[bot]

This comment was marked as resolved.

Re-check referenced history under the write lock, rewrite partial-purge
manifests to surviving entries, and stop claiming fs_failed is a no-op.
@Wibias
Wibias marked this pull request as draft July 27, 2026 04:02
Wibias added 7 commits July 27, 2026 06:03
CloseConfirm clears the permanent checkbox on cancel/escape/backdrop/stale preview, and preset buttons use locale-aware percent labels.
Match Codex delete_threads_strict across per-thread stores, fail closed on
threads read errors before mutating files, and cover permanent multi-DB cleanup.
Freeze thread IDs before satellite deletes, snapshot logs/memories/goals for
rollback, and keep the recovery trash path when satellite restore fails.
Snapshot logs/memories/goals first, write satellite-backup.json, then mutate.
Backup-write failures leave DBs and rollouts untouched; cover single-store homes.
Acquire BEGIN IMMEDIATE on logs/memories/goals before snapshotting, persist backup before committing satellite deletes, and restore snapshotted PK rows with conflict-safe inserts. Add regression for concurrent log insert and memory-job update surviving injected state-commit failure.
… race-safe

Roll back and close all acquired satellite write locks when any BEGIN IMMEDIATE fails. Restore the global consolidation job only when the live row still matches cleanup's post-delete image, and add regressions for busy final-satellite rollback and concurrent enqueue watermark updates.
…ssion

Store logs and memories read-only Database instances in variables and close them in finally blocks so Windows CI does not leak file locks.
@Wibias
Wibias marked this pull request as ready for review July 27, 2026 06:03
chatgpt-codex-connector[bot]

This comment was marked as resolved.

Bind digest per physical file stats, create exclusive stage dirs with private permissions, journal manifest before staging, persist consolidate post-image before memories commit, batch large SQL IN clauses, validate archive paths, and localize GUI transport errors.
coderabbitai[bot]

This comment was marked as resolved.

Wibias added 2 commits July 27, 2026 08:32
Batch satellite snapshot/delete IN queries to stay under SQLite variable limits, return fs_failed when exclusive stage-dir creation fails, and split injected-failure regressions into test.each cases.
Restrict wire-forwarded _test hooks via pickWireCleanupTestHooks and align runWithDigest typing with ExecuteCleanupOptions.
@Wibias

Wibias commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator Author

Security review — PR #529 (storage phase 2 cleanup)

Reviewer: automated security pass (post CodeRabbit/Codex fixes, including pickWireCleanupTestHooks in 3b0c85d2)
Scope: src/storage/cleanup.ts, src/server/management/logs-usage-routes.ts, gui/src/pages/Storage.tsx, tests/api-storage-cleanup.test.ts, tests/storage-cleanup.test.ts
Verdict: No medium, high, or critical findings. Merge-ready from a security standpoint.

@lidge-jun @Ingwannu — CI is green (ubuntu/macos/windows + npm-global + react-doctor). Codex and CodeRabbit review threads are addressed. This PR is merge-ready pending your maintainer sign-off.


Auth boundary

Cleanup routes sit behind the same stack as the rest of /api/*:

  • requireApiAuth(req, config, "management") in src/server/index.ts
  • isAllowedRequestOrigin enforced in handleManagementAPI before handlers run
Bind mode Behavior
Loopback (127.0.0.1) No API token required — any local process with access to the proxy can preview/execute cleanup. Same trust model as POST /api/stop, config changes, etc.
Non-loopback OPENCODEX_API_AUTH_TOKEN / config API keys required

codexHome is never taken from the request body; execution always uses resolveCodexHomeDir(). No new cross-tenant or cross-user boundary is introduced.


Path traversal / filesystem scope

Control Implementation
API inputs Only percent, mode, digest (+ optional env-gated _test). No caller-supplied paths.
Candidate discovery Flat archived_sessions/ only; isSafeArchiveFileName rejects /, \, ...
DB path normalization normalizeArchivedRolloutPath requires archived_sessions/<single-segment>.jsonl; rejects .. and active sessions/.
Resolve before IO absFromRel rejects ../absolute paths and re-checks containment under CODEX_HOME.
Active sessions sessions/ is never listed or deleted.

Symlinks under archived_sessions/ move the symlink node via resolve(codexHome, rel), not arbitrary targets outside the home tree.


Destructive FS/DB operations

Design matches the stated threat model:

  1. Probe all runtime DBs (state, logs, goals, memories) with BEGIN IMMEDIATE before any rename.
  2. Digest-bound execute — SHA-256 over percent + candidate paths/sizes/mtimes; drift → stale_preview (409).
  3. Stage → manifest → DB reconcile with satellite snapshots, ordered commits, and FS rollback on failure.
  4. Referenced history guard (spawn edges, paginated history_mode, fork columns) → referenced_history (409).
  5. Quarantine default; permanent only via explicit mode: "permanent" (GUI checkbox).
  6. Recovery artifacts: .trash/<epoch>/manifest.json (+ satellite-backup.json during reconcile), 0o700 trash dir / 0o600 manifest via writePrivateFile.

Residual (acceptable): the initial DB probe is not held for the entire operation. If Codex grabs an exclusive lock mid-flight, cleanup returns codex_busy / rollback paths — documented in API messages and covered by tests. This is not an auth bypass.


Test hook exposure (pickWireCleanupTestHooks)

const testHooks =
  process.env.OPENCODEX_CLEANUP_TEST_HOOKS === "1" && "_test" in body
    ? pickWireCleanupTestHooks(body._test)
    : undefined;
  • Without OPENCODEX_CLEANUP_TEST_HOOKS=1: _test is ignored entirely.
  • Allowlist: booleans + string arrays only; afterSatelliteMutations (function callback) is not on the wire.
  • Hooks inject failure points for atomicity tests only — they do not run shell commands, change codexHome, or accept arbitrary paths.

Residual risk (low): if someone sets the env var on a production service, an API-authenticated client could force failure modes. That does not exceed what the same client can already do with mode: "permanent". Recommendation (non-blocking): document "test/CI only; never set in service config."


Information disclosure in API responses

Endpoint Disclosure
POST .../preview Relative relPath only; strips codexHome / absPath (tested).
POST .../cleanup errors Mapped error codes + fixed message strings; trashDir is relative (e.g. .trash/173…).
GET /api/storage Still returns full codexHome (phase 1; unchanged by this PR).

Error JSON is asserted not to contain absolute host paths in tests/api-storage-cleanup.test.ts. GUI renders errors as React text (no dangerouslySetInnerHTML); trashDir in i18n is server-controlled relative path.

Manifest on disk may include threadId / rolloutPath for recovery — local session metadata under 0o600 in user-owned .trash; acceptable for same-user recovery.


Race conditions

Scenario Mitigation
Preview → execute TOCTOU Digest includes per-file mtimes; file set changes → stale_preview.
Concurrent Codex writer Upfront probe + busy mapping; staged FS rollback on DB failure.
Partial permanent purge purgeIncomplete manifest; relative trashDir in 500 response (tested).
Satellite DB partial commit Snapshot + restoreSatelliteBackup; keeps stage when restore fails.
Concurrent satellite writes during cleanup BEGIN IMMEDIATE write locks held through snapshot/delete; conflict-safe INSERT … ON CONFLICT DO NOTHING restore.

GUI (Storage.tsx)

  • No dangerouslySetInnerHTML; paths shown as text in <li> / tables.
  • Confirm modal + permanent checkbox before destructive execute; checkbox resets on dismiss.
  • Preview list capped at 8 rows but count/bytes/digest reflect the full preview response (digest still binds full server-side set).

Findings summary

Area Blocker? Notes
Auth / CSRF No Same model as existing management API.
Path traversal No Strong containment on all changed paths.
Destructive ops No Scoped to archived files; active sessions protected.
Test hooks No Env-gated; allowlisted; no privilege escalation.
Info disclosure No Cleanup responses avoid host paths; relative recovery hints only.
Races / recovery No Documented partial-failure behavior with manifest/trash.

Optional hardening (not required for merge):

  1. API test that _test is ignored when OPENCODEX_CLEANUP_TEST_HOOKS is unset.
  2. One-time startup warning if OPENCODEX_CLEANUP_TEST_HOOKS=1 outside NODE_ENV=test.
  3. Ops note that remote/LAN deployments must use API auth (already required for non-loopback).

Conclusion

PR #529 storage cleanup is well-bounded for a local-first management surface. No concrete medium+ vulnerabilities with realistic exploitability beyond the existing "whoever can call the local management API can manage the local proxy" trust model.

Merge recommendation: approve.

@Ingwannu Ingwannu left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent maintainer re-review completed on 3b0c85d2.

The three still-open Codex threads are stale against the current head:

  • createExclusiveStageDir creates the quarantine directory without recursive, suffixes timestamp collisions, and applies private permissions;
  • the staging manifest is written before the first rename, then finalized before DB reconciliation;
  • the memories consolidate post-image is persisted to satellite-backup.json before that transaction commits.

Required Linux/macOS/Windows, npm-global, and React Doctor checks are green. The recorded security review covers the final cleanup routes, filesystem scope, recovery sequence, and wire test-hook gate. Approved for dev.

@Ingwannu
Ingwannu merged commit 84f39c4 into lidge-jun:dev Jul 27, 2026
9 checks passed
lidge-jun added a commit that referenced this pull request Jul 27, 2026
Inventory all 23 open issues and 14 open PRs on the axis of decisions only
the maintainer can make, clustered into six bundles for later PABCD cycles.

Live findings:
- anthropics/claude-code#1124, cited as upstream evidence on #543, is CLOSED
- #462/#401/#241 carry upstream-tracking with no upstream ticket at all
- #92's encrypted_content body has no filed upstream issue either
- #527 targets #526's head branch, so enforce-target fails
- #529 is already merged; the earlier triage matrix still lists it in flight
lidge-jun added a commit that referenced this pull request Jul 27, 2026
Two read-only audit lenses returned 30 contradictions; the verified ones are
applied here and recorded in 006_corrections.md.

Factual corrections:
- claude-code#1124 closed 2025-05-16, not 2025-08-10
- 13 open enhancement issues, not 11
- #529 is merged, so issue #42 phase 2 is done
- #418 has no PR behind it; it was miscounted as in-flight
- #528 does not contain #424's current head, so it needs a rebase
- #527's retarget is manual and independent of #526

Framing corrections:
- stale-needs-info.yml is absent from the default branch, so bundle C had no
  real deadline
- the debug switch #543's reporter asked for already exists
- whether #545 is our defect is answerable from code, not owner judgment
- #491 is an OAuth credential change, so it is a security boundary rather
  than a warm-up; #533 and #447 keep their security classification too

Structure: bundle G added for roadmap honesty, bundle F rescoped to the
security boundary, #498 split out of bundle B, cycle order rebuilt, and the
loop archetype corrected to decision-elicitation.
@Wibias
Wibias deleted the feat/storage-phase2-cleanup branch July 28, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants