Fix deduplicate plan timeout: index + batch RFC822 duplicate-group lookup - #512
Draft
jesserobbins wants to merge 4 commits into
Draft
Fix deduplicate plan timeout: index + batch RFC822 duplicate-group lookup#512jesserobbins wants to merge 4 commits into
jesserobbins wants to merge 4 commits into
Conversation
Fixes kenn-io#510. GetDuplicateGroupMessages ran one unindexed query per RFC822 duplicate group; 22,025 groups at ~190ms/lookup burned the CLI's 30-minute plan-request timeout before content-hash comparison even started. Authored-By: Jesse Robbins (@jesserobbins) <https://jesserobbins.com>
Second half of the kenn-io#510 fix. Adds GetDuplicateGroupMessagesBatch, turning N per-group queries into a handful of chunked IN (...) queries. Engine.Scan is switched over in the next commit; GetDuplicateGroupMessages itself is kept as the equivalence test's reference implementation. Authored-By: Jesse Robbins (@jesserobbins) <https://jesserobbins.com>
Completes the kenn-io#510 fix. Scan() no longer queries the store once per RFC822 duplicate group; it fetches all groups' messages in one batched, chunked call and looks them up from a map. Authored-By: Jesse Robbins (@jesserobbins) <https://jesserobbins.com>
- Equivalence test now exercises sourceIDs + multi-chunk together, matching Engine.Scan's actual call shape (always non-empty sourceIDs). - Fixed GetDuplicateGroupMessagesBatch's godoc; moved the "kept as equivalence-test reference" rationale onto GetDuplicateGroupMessages, which is what it actually describes. - Extracted the shared SELECT column list into one const, removing ~13 lines of duplicated SQL between the two query methods. - Engine.Scan nils msgsByGroup after use instead of holding it live through the content-hash comparison pass. Authored-By: Jesse Robbins (@jesserobbins) <https://jesserobbins.com>
roborev: Combined Review (
|
jesserobbins
marked this pull request as draft
July 26, 2026 18:12
Contributor
Author
|
Holding off to do more testing |
Contributor
Author
|
I am running into multiple issues on my super old email archive. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Running
deduplicate --content-hashon my own archive, the CLI died after 30 minutes withcontext deadline exceededand zero output. Traced it to an N+1 query with no supporting index —Scanwas querying the store once per RFC822 duplicate group, and on a real archive that's tens of thousands of round trips before content-hash comparison even starts.Summary
messages.rfc822_message_id— a partial index (mirroring the existing attachment-hash indexes) turns out to be unusable by SQLite's planner for this query shape, verified viaEXPLAIN QUERY PLANbefore picking the plain form.Store.GetDuplicateGroupMessagesBatch, replacing the per-group query with a handful of chunkedIN (...)queries.Engine.Scanto the batched method.Fixes #510.
Test plan
make testpasses (one unrelated pre-existing failure ininternal/config— a macOS/varvs/private/varsymlink artifact in a file this PR never touches).deduplicate --collection local --content-hash --no-backup -v --dry-runagainst a personal archive with 161,113 duplicate groups. Before this fix, the equivalent run never completed — it hit the CLI's 30-minute client timeout with zero output, having made partial progress through roughly 22,025 of those groups. After: completes in ~311s.Out of scope, tracked in #510 but not touched here:
BackfillRFC822IDswriting to the DB unconditionally during/planbefore confirmation, and the server not respecting client cancellation once a request times out client-side.Authored by Jesse Robbins (@jesserobbins)