perf: stop reading whole session files, and index them incrementally

[Davidb-2107]

What I saw

On a machine with ~2.3 GB under ~/.claude/projects (1400 sessions, largest file 217 MB), the main process sat at a 139 MB median but spiked past 250 MB in 24% of samples, peaking at 498 MB — 921 MB across all processes.

That was measured with a single terminal open, so neither the terminals, the grid view, nor the WebGL renderers were involved. I chased those first and the measurements ruled them out.

Why

Two variations of the same thing: reading a whole file to use a little of it. A session .jsonl held as a JS string costs ~2x its size in RAM, since V8 stores non-latin1 text as UTF-16.

1. Three sites read an entire file just to get its head.

site	reads	keeps
schedule-runner.js scanSchedules	whole file	4000 chars — every 60s, per project folder
main.js (session slug)	whole file	8000 chars
derive-project-path.js	whole file	the first line carrying cwd

The scheduler one dominates: a 61 MB session file allocates ~122 MB once a minute. That is the sawtooth.

2. readSessionFile re-read the file in full on every append, to produce ~9 KB of metadata. The projects watcher fires it on each write, so an active session re-read its entire history every few seconds.

What this changes

Adds jsonl-scan.js with the two supported ways to walk these files — scanLines (chunked, resumable, early-exit) and readHead — and routes every caller through it.

For the indexer: session files are append-only (folder-index-state.js already relies on this; measured here as 0 rewrites and 0 truncations across 1214 files), and every field readSessionFile extracts is either a first occurrence or a running total. So it resumes from the byte offset the previous pass reached, persisted in four new session_cache columns.

Safety: a 4 KB head hash and a size check catch a file that was rewritten in place or truncated, falling back to a full read rather than emitting a corrupt message count. Migration v4 adds the columns without wiping the cache — indexedBytes stays NULL on existing rows, which reads as "cannot resume", so each session is fully re-read once and incrementally after that.

Measurements

Main process, 4 minutes, same workload:

before   rss 136 -> 520 MB    heap peak 360 MB    3 jumps of +354 MB
after    rss 145 -> 170 MB    heap peak  12 MB    0 jumps

A/B against the released build, both running side by side for 11 minutes on the same watched directory:

peak across all processes    921 -> 526 MB
main process peak            498 -> 158 MB
samples above 250 MB          24% -> 0%

Re-indexing after an append: ~0 MB and 22 ms, down from 152 MB and 447 ms.

The ~480 MB floor (Electron + renderer + GPU) is untouched — this only removes the spikes.

Tests

Four tests in test/read-session-file.test.js cover the resume path: that an append reads only the appended bytes, that the incremental result equals a full re-read, that a rewritten file falls back to a full read, and that a single-message file with no trailing newline still indexes.

Verified by running the build against a copy of a real 1400-session database.

Happy to split this into two PRs (the head-reads and the incremental indexer) if you'd prefer to review them separately.

[abasiri]

Reviewed this properly — it's careful work and the diagnosis holds up on my machine too. For calibration: 3,290 session files, 3.1 GB, largest 136.6 MB. Under the current code an append to that file re-reads all of it into a JS string — roughly 273 MB of transient allocation — every few seconds while the session is live. Your sawtooth reproduces.

Heads up: I pushed a merge commit to this branch (5016379) fixing the two things that were blocking it. Both were merge-order artifacts, not defects in your work — #60 and #65 landed after you wrote this. Details below so nothing is a surprise.

What's done right

jsonl-scan.js avoids both classic chunked-reader bugs, which is not the common outcome. It carries the partial line as a Buffer rather than a string, so a multibyte UTF-8 sequence split across a chunk boundary survives; and it copies out of the reused allocUnsafe scratch buffer instead of aliasing it into data. The byte accounting is exact — data always begins at consumed — which is what makes the resume offset trustworthy rather than approximately right.

The resume-safety triad is the part I'd have most expected to find wrong, and it's complete: headHash catches an in-place rewrite, indexedBytes <= stat.size catches truncation, and a trailing line with no newline sets resumable = false so it can't be double-counted on the next pass.

The migration is additive with no cache wipe — NULL indexedBytes reads as "can't resume", so each session re-reads once and goes incremental after. And the tests assert the property that actually matters: incremental == full re-read.

I also specifically audited the riskiest refactor here, narrowing cacheGetAll from SELECT * to an explicit list. buildProjectsFromCache reads only aiTitle, created, firstPrompt, messageCount, modified, projectPath, sessionId, slug, summary — all still selected — and the customTitle consumers operate on freshly-read sessions, not cached rows. Safe.

What I changed in 5016379

schedule-runner.js conflict. #65 made scanSchedules prefer a cache_meta folder→projectPath lookup, falling back to a JSONL head read only for folders missing from the cache. That collided textually with your change to the same read — but the two compose. I kept #65's structure and moved readHead(…, 4096) inside its readProjectPathFromJsonl fallback, so the common path does no file read at all and the fallback no longer loads a 136 MB file to inspect its first line.

test/reconcile-cache.test.js was failing. refreshFolder now fetches the cached row as resume state, and that test's fake db predates the method:

✖ reconcileCacheFromFilesystem indexes new and stale folders …
  getCachedSession → undefined

I added getCachedSession() { return null; } to the fake, modelling a session with nothing indexed yet. Fixed in the fake rather than guarding the call site — the real db provides the method, and a guard would mask genuine wiring errors. 27/27 pass now.

Remaining — your call

Long lines are quadratic. pending = Buffer.concat([pending, chunk]) recopies on every chunk, so a line longer than CHUNK_BYTES costs O(n²) in line length. This isn't hypothetical: my largest session file has 38 lines over 256 KB, longest 2.6 MB, which is ~10 recopies of a growing multi-MB buffer for a single line. Collecting chunks into an array and concatenating once at the newline would fix it.

Two test gaps. The truncation path (indexedBytes > stat.size → full re-read) is implemented but unasserted. More importantly, a multibyte character split across a chunk boundary — the subtlest correctness property in the PR — currently rests on inspection alone. A file with, say, an emoji straddling the 256 KB mark would pin it.

Smaller notes. readHead can split a multibyte character at maxBytes, leaving U+FFFD in the final partial line — harmless, since that line fails JSON.parse inside the existing try/catch, but worth a comment. It's also a quiet semantic change from .slice(0, 4000) (characters) to 4096 bytes. And a file under 4 KB hashes its whole contents, so crossing 4 KB changes headHash and forces one extra full re-read per session — correct, just non-obvious.

Nothing above blocks merging. Happy to take it as-is and file the long-line concat separately if you'd rather land the win now.