Incremental doc-values updates

[jimczi]

Incremental doc-values updates

When you update a doc-values field today, Lucene rewrites the whole column for that field, even if you only touched a
few docs. So a tiny change writes a lot, and it gets worse as the segment grows.

The idea is simple: when an update only sets values (no unset), write just the changed docs as a sparse "delta"
generation, and stack the deltas on top of the base column at read time, newest wins. Updating a field becomes
O(changed docs) instead of O(column).

Deltas would pile up, so there's a small lifecycle per field:

• every flush writes a delta with only the changed docs
• too many deltas -> fold them into one sparse generation
• deltas end up covering the whole column -> fold back to a single dense column
• a merge flattens everything back to a normal column

Numeric and binary only, set-only (a value unset falls back to the current dense rewrite). No doc-values format
change, the deltas use the codec's existing sparse encoding.

An old Lucene can't read the overlay, so this rolls the segments format version (VERSION_11_0), the same way past
segments-file changes did (like the SegmentCommitInfo id in 8.6): every commit written by this version uses it, and an
older reader rejects the index up front with IndexFormatTooNewException instead of reading a delta as if it was the
whole column. Reading a newer index with an older Lucene isn't in the contract anyway (Lucene is forward-only), so
nothing is lost. On main the feature is on by default; pass IndexWriterConfig#setMaxDocValuesOverlays(0) for the
classic full-column rewrite (0 = off, N = keep up to N overlays before a fold). A 10.x backport would be opt-in via
that default.

This is a proposal, I'd like your opinion on the approach before polishing it more. Two things I'm not sure about:
where the overlay bookkeeping should live (I put it on SegmentCommitInfo), and the compaction, which right now
re-folds all the deltas every time. A size-tiered policy would be better, left as a TODO.

Numbers

There are three ways to change one field on a doc, so I compared them head to head. The benchmark is in the last
commit, a plain main, not a test, run as a normal Java app (assertions off, real JIT, mmap directory) so the JVM
behaves like production and not the test harness. It'll go before merge, it's just there so you can reproduce. It
indexes 5M docs, each with an 8-byte field and a 512-dim vector, lets the natural merges finish, then applies updates
in random order (not ingestion order, so the overlay actually gets stressed) from 4 threads, refreshing an NRT reader
every second. It reports update throughput, the bytes written during the update phase, write amplification, and a scan
that reads the field back. All three arms end on the same values.

• full reindex: updateDocument, what you do today when the field isn't doc-values-updatable. Rewrites the whole doc.
• dense update: what Lucene does now. Rewrites the whole column.
• sparse update: this PR. Only the changed docs.

The thing that matters is cadence. Write amplification isn't really a property of the feature, it's a property of how
many docs you change between commits. So I ran every arm two ways: a batch that updates as fast as it can (lots of docs
per refresh), and a throttled stream at 5k updates/s (a few thousand per refresh, closer to a steady trickle on a live
index).

Numeric 8-byte field:

regime	arm	updates/s	bytes/update	write amp	scan
batch	full reindex	71,398	13,138	1,642×	66 ms
batch	dense	564,653	80	10×	35 ms
batch	sparse	625,391	8	1.0×	118 ms
throttled 5k/s	full reindex	4,991	14,359	1,795×	60 ms
throttled 5k/s	dense	4,991	7,202	900×	21 ms
throttled 5k/s	sparse	4,991	18	2.2×	103 ms

Amplification is bytes written over the 8 bytes actually changed. In batch, dense amortizes: thousands of updated docs
share one full-column rewrite, so it lands at 10×. Throttle it and that same full-column rewrite now covers only a few
thousand docs, so it jumps to 900×. Sparse doesn't care about cadence, it writes the changed docs and nothing else, so
it stays flat: 2.2× vs 900× at 5k/s, about 400× less written.

32-byte binary field, same two runs:

regime	arm	updates/s	bytes/update	write amp	scan
batch	full reindex	85,281	12,547	392×	139 ms
batch	dense	833,333	632	20×	29 ms
batch	sparse	1,270,648	32	1.0×	327 ms
throttled 5k/s	full reindex	4,991	15,756	492×	177 ms
throttled 5k/s	dense	4,991	27,734	866×	28 ms
throttled 5k/s	sparse	4,991	55	1.7×	172 ms

Same shape, bigger column so dense costs more (27 KB/update throttled against 55 bytes for sparse).

Soft deletes come along for free, because a soft delete is a numeric doc-values update on the soft-deletes field, so it
takes the same path. The arms here are a hard delete (liveDocs, no column), and the soft-delete mark with the feature
off (dense) and on (sparse):

regime	arm	updates/s	bytes/update	scan
batch	hard delete	1,428,571	0	42 ms
batch	dense soft	2,036,660	1	50 ms
batch	sparse soft	2,123,142	1	40 ms
throttled 5k/s	hard delete	4,991	0	66 ms
throttled 5k/s	dense soft	4,991	33	54 ms
throttled 5k/s	sparse soft	4,991	6	74 ms

The soft-delete column only holds the marked docs, so the absolute numbers are small, but the shape is the same: batch
amortizes and both are basically free, throttle it and dense rewrites the growing column every commit (33 bytes/update)
while sparse writes just the newly marked docs (6 bytes/update).

Reads are the tradeoff. The sparse scan merges the overlay at read time, so it's a few times slower than a plain
column (numeric 118 ms vs 35 ms, binary 327 ms vs 29 ms over 5M docs). Random-access lookup (the sort/facet pattern,
forward advanceExact over a 10% sample) is ~18 ms vs ~1 ms for a plain column, tens of ns per lookup, about the same
per-doc as the scan since a doc absent from a sparse layer is skipped rather than scanned. It's all bounded by
setMaxDocValuesOverlays (default 16, more overlays means cheaper writes but more layers to merge on read), and a merge
flattens it back. I also ran the batch with 4 threads querying while the updates landed and everything stayed correct
with the write numbers holding.

The benchmark is only here so you can look at the numbers for this review, it isn't meant to land. Here's the line I
ran:

java -da -Xmx6g --enable-native-access=ALL-UNNAMED \
  -Ddvbench.docs=5000000 -Ddvbench.dims=512 -Ddvbench.flatVectors=true -Ddvbench.updates=1000000 -Ddvbench.threads=4 \
  -cp lucene/core/build/classes/java/main:lucene/core/build/resources/main:lucene/core/build/classes/java/test \
  org.apache.lucene.index.IncrementalDocValuesUpdatesBenchmark

(add -Ddvbench.rate=5000 to throttle, -Ddvbench.type=binary or -Ddvbench.type=softdelete for the other fields,
-Ddvbench.queryThreads=4 to query under load.)

Commits

Split so each piece is easy to look at on its own:

1. config option, the single setMaxDocValuesOverlays knob
2. overlay iterators, the read side: merge the deltas over the base, unit tested alone
3. SegmentCommitInfo, where the overlay generations are stored + the segments-version bump
4. write path, write the delta, fold, fold-to-dense, thread the config through
5. tests
6. the standalone benchmark (temporary, removed before merge)
7. review follow-ups: collapse to one config knob, make the version bump unconditional

[msokolov]

I like the idea, it would make updatable DV more appealing. quick Q: I read the proposal but not all the changes. Does it cover all the DocValues, or just Numeric? What about multi-valued (SortedNumeric I think it's called)?

[jimczi]

Yeah, builds on the existing update support so numeric and binary only. For multi-valued you'd have to pack it into a binary field with a custom format today, there's no native SortedNumeric update path. SortedNumeric could be added (it's ordinal-free, so tractable), but that's really about extending the core update support, which is out of scope here: this PR just builds sparse deltas on top of what's already updatable.

[ChrisHegarty]

I like this a lot. The sparse deltas with the existing codec encoding is a clean approach, and the write amplification numbers speak for themselves.

The thing I keep coming back to is whether deltas need to survive across commits. That's the decision that forces the segments version bump and the compatibility implications (old readers can't open the index while overlays are active).

What if you just folded back to dense at commit time? You'd still get most of the win: if refreshes are every 1s and commits every 30s, that's 30 cheap sparse delta writes between commits and then one dense rewrite when you commit. You've avoided 29 out of 30 full-column rewrites compared to today. Not as optimal as never rewriting until merge, but zero compatibility cost, segments_N stays unchanged, old Lucene can still read it, and the committed state is always a plain dense column.

Curious whether you considered that path and rejected it, or whether it just fell out naturally that deltas persist. It might be worth it as a simpler first step even if the full "persist across commits" mode comes later?

[ChrisHegarty]

One thought for the future: the feature is all-or-nothing across the index today. In practice some fields are read-heavy (sorting, aggregations) and would benefit from a flat dense column, while others are write-heavy and would prefer cheap sparse deltas. A per-field policy could be a nice follow-up. The plumbing already knows which field is being updated, so it should be straightforward to add later. Something like:

IndexWriterConfig.setMaxDocValuesDeltaGenerations("score", 16);   // sparse deltas for this field
IndexWriterConfig.setMaxDocValuesDeltaGenerations("timestamp", 0); // dense rewrite for this field

[ChrisHegarty]

The conditional version bump in SegmentInfos.write() (only bump to VERSION_11_0 when overlays are active, otherwise write VERSION_86) is clever, but makes compatibility dependent on merge timing rather than config. The same index with the same settings might be readable by old Lucene after a merge flattens everything, then unreadable again when new updates land. That feels a bit fragile; whether a rollback to an older version works depends on whether merges have kept up, which isn't something operators typically control or reason about. Might be simpler to just always write VERSION_11_0 when the feature is enabled, so compatibility is deterministic based on config alone.

[jimczi]

Yeah I looked at that. Deltas need to survive commits because the whole point is to commit often without rewriting the full column each time. Fold-at-commit only helps when commits are rare compared to refreshes. In the case I care about a refresh is also a commit, so you'd fold to dense on every refresh, which is just what we do today. Your 29/30 holds when refresh:commit is big, at 1:1 it's 0/1.

I don't think the compat cost is a real blocker either. An old reader rejects the index up front (IndexFormatTooNewException, before any codec runs), and I made the bump deterministic like you said in the other comment (it always bumps when the feature is on now, see 3eb7cf1) so it only depends on the config. Commits with the feature off are byte-for-byte the same as today, so backports are safe.

The real cost of keeping deltas around is on the read side (the overlay merge). But there's a lot we can do there: intoBitSet, reading the dense base directly and only patching the changed docids, or folding the deltas into one small patch at reader open. That's a bigger scope though, so I'd rather land the mechanism first and do those as follow-ups.

[jimczi]

Yeah I like this. It's index-wide for now to keep the first cut simple, but the write path already routes updates per field, so a per-field override should be easy to add on top. Good follow-up.

[jimczi]

Fair point. The conditional bump made readability depend on merge timing, which isn't something operators should have to reason about. Done in 3eb7cf1: it now always bumps when the feature is enabled, so it's deterministic from the config. With the feature off (and no inherited overlay) it stays on the old version, so nothing changes for backports.

[jimczi]

Simplified the version bump to unconditional: every commit just writes the current segments version (VERSION_11_0), like any past format bump. An older Lucene can't read a newer index anyway (Lucene is forward-only), so there was no reason to gate it.

[ChrisHegarty]

Thanks for addressing my feedback, and answering my questions. This makes a lot of sense to me, and I think that the direction is good.

[ChrisHegarty]

Are you planning to backport this to 10.6? Should the version be encoded as VERSION_10_6?

[jimczi]

Same as the note on the version constant: it is just naming, the value is 11 regardless. I would rename to VERSION_10_6 if/when we backport, deferring it until then.

[ChrisHegarty]

Are you planning to backport this to 10.6? Should the version be encoded as VERSION_10_6?

[jimczi]

Just the name, not the value: the fence is the value (11), which is higher than VERSION_86 so an older reader rejects it either way. On main it is VERSION_11_0 since main is 11.0. If/when we backport to 10.6 I would rename it to VERSION_10_6 (value unchanged), a trivial rename on both branches. So I would rather defer the naming to the actual backport.

[mikemccand]

+1 for the overall idea/approach -- it is insanely surprising / unexpected that updating a DV field has the cost / write amplification (whole column for that segment even if just one doc was updated) it does today. This would fix that, at least for BINARY and NUMERIC cases.

It does presumably make lookup slower (it's trading off faster indexing for slower searching), since each lookup will go through N stacks (newest to oldest), and those docs that were never updated would have the most added cost I think? Have you tested lookup performance vs number of updates applied? In practice it's likely negligible -- the other costs of searching (HNSW crawl, decoding/intersecting postings, whatnot) usually dominate vs looking up DV valuews for sorting, say.

[mikemccand]

Also, CC @shaie -- long ago he had thought about a similar idea for updatable postings I think?

[shaie]

Thanks @mikemccand for CC'ing me. I like the idea and have few comments, although I haven't reviewed the changes so if you've implemented them already, it's even better :).

On the per-field override / cost of writing deltas vs full vectors: we can make the decision dynamic per the number of updates. For example, say up to 30% of docs were updated, you store the deltas. More than 30% fold all the updates to a single vector. If a segment is updated over several flushes and accumulates over 30% total updates, they're collapsed too. That way, it's per-field per-segment policy which changes dynamically with the rate of updates. The threshold (30%) is a hyperparameter you can evaluate how it impacts indexing and search throughput.

On the runtime cost, how about if we folded all the updates to a single buffer, just like a segment without deltas? That way, you pay the cost once at segment open time, but otherwise reads behave just as if the segment had no deltas?

And yes @mikemccand this idea resembles a discussion we've had many years ago about posting updates too - you maintain the updates in "layers/overlays" of the postings and resolve the final values at read time. At the time when we've implemented DV updates, the dominating scenario was updating numeric values such as update timestamps etc., which naturally affected the majority of the index. But I can totally see usecases for this capability, such as updating sparse fields (price, ratings_count) which will benefit from this enhancement.

[jimczi]

Thanks both.

@mikemccand right, it's a write-for-read trade: a read merges the doc across the layers with a small min-heap, newest wins. On never-updated docs paying the most, the delta layers are sparse (IndexedDISI), so a doc that isn't in a layer is skipped via advance rather than scanned, so it's ~O(log N) over the layers, not O(N) per doc, and N is bounded by maxDocValuesOverlays (default 16) with a merge collapsing it back to one column. I measured the sequential scan (aggregate over all live docs): ~118 ms vs ~35 ms for a plain column, 5M docs at 16 overlays (numeric), so a few times slower but bounded. I haven't isolated random-access lookup (sorting/faceting) yet, that's the honest gap, and I agree it's likely dominated by the rest of search. I'll add a probe for it.

@shaie on the dynamic per-field-per-segment policy: that's already the shape of it, the fold is per field per segment and it collapses back to a single dense column once the deltas cover the whole column. Right now that triggers at full coverage; your %-threshold (say 30%) is a nice way to cap read cost earlier instead of waiting for 100%, and it lines up with the size-tiered compaction I flagged as a TODO. Today the bound is the overlay count; a coverage-% trigger could complement or replace it.

Your fold-at-open idea is exactly the read-side optimization I want, and it answers Mike's question: merge the deltas into one buffer at reader open so reads behave like a plain column and you pay the merge once. Not in the first cut, to keep the mechanism minimal, but it's the top follow-up, along with intoBitSet and streaming the base while only patching changed docids. Good to know this connects to the old postings-update discussions too, and agreed the sweet spot is sparse fields (price, ratings_count) rather than the update-everything timestamp case DV updates were originally built for.

[jimczi]

Added the probe: forward advanceExact over an ascending 10% sample (the sort/facet pattern). Numeric, 5M docs, ~500k lookups: at 16 overlays it's ~18 ms vs ~1 ms for a plain column (batch), and ~26 ms vs ~7 ms at ~350 live gens (throttled). So tens of ns per lookup, and per-doc about the same cost as the sequential scan, since forward advanceExact skips absent docs in each sparse layer just like iteration. A few times a plain column but bounded by maxDocValuesOverlays and, as you expected, tiny next to the rest of a query.