Skip to content

Reconcile cache with filesystem on get-projects so sessions stop going missing - #60

Merged
abasiri merged 1 commit into
doctly:mainfrom
ymajoros:fix/reconcile-cache-on-get-projects
Aug 1, 2026
Merged

Reconcile cache with filesystem on get-projects so sessions stop going missing#60
abasiri merged 1 commit into
doctly:mainfrom
ymajoros:fix/reconcile-cache-on-get-projects

Conversation

@ymajoros

@ymajoros ymajoros commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #59.

Problem

Sessions and whole worktrees silently vanish from the sidebar while their transcripts are still on disk, and reopening/restarting doesn't bring them back.

get-projects only scans the filesystem on a cold start:

const needsPopulate = !isCachePopulated() || !isSearchIndexPopulated();
if (needsPopulate) { populateCacheViaWorker(); return []; }
return buildProjectsFromCache(showArchived);   // cache only, never reconciled

Once the cache is non-empty it's never reconciled again, so folders that changed while the app was closed — or that were never indexed by the current build (cache_meta.indexMtimeMs === 0) — never reappear. The live fs.watch only covers changes while the app is running.

Fix

Reconcile on get-projects before building from cache. The dead populateCacheFromFilesystem() (defined + exported, never called) is repurposed into reconcileCacheFromFilesystem(), now stat-gated: it re-indexes only folders that are new or whose newest .jsonl is newer than what was last indexed.

for (const folder of folders) {
  const meta = metaMap.get(folder);
  const folderPath = path.join(PROJECTS_DIR, folder);
  if (!meta || getFolderIndexMtimeMs(folderPath) > (meta.indexMtimeMs || 0)) {
    refreshFolder(folder);
  }
}

When nothing changed this is just a readdir + stat per folder — no transcript parsing, no DB writes — so the steady-state cost on get-projects is negligible; real work happens only for folders that actually need it. refreshFolder already does its own per-file mtime skip, so this composes cleanly.

Why here (vs. only at startup)

Reconciling on get-projects is self-healing for both the "changed while closed" and "old build never indexed it" cases, and naturally re-runs if folders drift later — without a separate watcher or timer. The mtime gate keeps it cheap. Happy to move it to a one-time startup pass instead if you'd prefer.

Testing

  • node --test passes, including a new test/reconcile-cache.test.js that asserts the gate: a never-indexed folder and a stale folder (indexMtimeMs older than disk) get indexed, while an up-to-date folder is skipped.
  • Verified live on Linux / 0.0.30: 27 worktree folders that were stuck empty (indexMtimeMs = 0) re-indexed and reappeared in the sidebar.

Notes

Pure cache/indexing change — no schema change, no change to session_meta (names/stars/archive) or settings.

…g missing

The session cache is only rebuilt from the filesystem on a cold start
(populateCacheViaWorker runs only when the cache is completely empty).
Once the cache has any rows, a project folder that changed while the app
was closed — or that was created before the build which first indexed it —
is never re-scanned, so its sessions (and whole worktrees) silently
disappear from the sidebar. The transcripts are intact on disk; only the
cache is stale.

Make get-projects reconcile first: re-index folders that are new or whose
newest .jsonl is newer than what we last indexed. The check is a cheap,
stat-only gate (getFolderIndexMtimeMs vs cache_meta.indexMtimeMs), so it's
effectively free when nothing changed and only does real work for folders
that actually need it.

The previously dead populateCacheFromFilesystem() (defined and exported but
never called) is repurposed into this gated reconcileCacheFromFilesystem().
@ymajoros
ymajoros force-pushed the fix/reconcile-cache-on-get-projects branch from 274eb72 to edf06c0 Compare June 1, 2026 10:19
@abasiri
abasiri merged commit a791cdd into doctly:main Aug 1, 2026
abasiri added a commit to Davidb-2107/switchboard that referenced this pull request Aug 1, 2026
…e test

Branch was 14 behind. Two things needed fixing, both merge-order artifacts —
doctly#60 and doctly#65 landed after this branch was written.

schedule-runner.js conflicted, but the two changes compose rather than compete.
doctly#65 made scanSchedules prefer a cache_meta folder→projectPath lookup, falling
back to reading a JSONL head only for folders missing from the cache; this
branch made that head read cheap. Resolved by keeping doctly#65's structure and
putting readHead(…, 4096) inside its readProjectPathFromJsonl fallback, so the
common path does no file read at all and the fallback no longer loads a
possibly-hundreds-of-MB file to look at its first line.

test/reconcile-cache.test.js failed because refreshFolder now fetches the
cached row to use as resume state, and that test's fake db predates the method:

    ✖ reconcileCacheFromFilesystem indexes new and stale folders …
      getCachedSession → undefined

Added getCachedSession() { return null; } to the fake, modelling a session with
nothing indexed yet. Fixed in the fake rather than guarding the call site: the
real db provides the method, and a guard would mask genuine wiring errors.

27/27 tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sessions/worktrees silently disappear from sidebar; cache never reconciled after cold start

2 participants