(perf): FTS dirty-flag — skip reindex on tab switch when files unchanged - #74
Merged
Conversation
Each tab switch to Memory or Work Files called deleteSearchType + upsertSearchEntries unconditionally, re-reading every file from disk on every open. With the recent contentless FTS5 (#69) this also rewrites the external-content + trigram index each time — pure churn when nothing changed. Add three pure helpers (computeIndexSignature, shouldReindex, invalidateFtsSignature) stored in module-level Maps keyed by FTS type. Both handlers now compute a filePath:mtimeMs:size signature over the collected file list and skip the wipe+rebuild when it matches the last-indexed signature. The result payload (file tree returned to the UI) is built and returned unconditionally — only the FTS side-effect is gated. save-memory and delete-work-file call invalidateFtsSignature to force a fresh reindex on the next tab open even when mtime precision could mask a sub-second write. 21 unit tests cover: signature stability, order-independence, mtime/size/path-set change detection, type isolation, invalidation, and static wiring assertions for all four handlers.
save-file-for-panel wrote to disk without calling invalidateFtsSignature, leaving a sub-second window where a save followed immediately by a search would return stale FTS body content. Mirrors the explicit invalidation already present in save-memory and delete-work-file (review MAJOR-1). Paths under /.work-files/ invalidate 'work-file'; .md paths invalidate 'memory'; both can fire on the same path (e.g. a .work-files/note.md). computeIndexSignature now uses NUL field separators and newline record separators instead of ':' and '|', eliminating a theoretical collision where a filePath containing the literal ':<digits>:<digits>|' string could produce the same joined string as two separate entries (MINOR-2). Tests: 2 new static-analysis assertions for save-file-for-panel wiring; 1 new NUL-delimiter collision test; 2 updated format assertions for the changed separator characters. 24 tests total, all pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
get-memories and get-work-files are called every time the user opens or switches to the Memory/Work Files sidebar tabs. Each call unconditionally ran a full FTS re-index: deleteSearchType wipes all rows, then upsertSearchEntries re-reads every file from disk. With the contentless FTS5 schema (#69), this also rewrites the external-content + trigram index each time — pure churn when nothing changed.
Solution: signature-based dirty-flag
Three pure module-level helpers introduced in main.js:
Both handlers compute the signature from the already-gathered file list and skip the deleteSearchType + upsertSearchEntries block (including all fs.readFileSync calls) when unchanged.
Correctness invariants
Tests
21 unit tests in test/fts-dirty-flag.test.js: