Skip to content

Fix FixedShingleFilter emitting a leading 0 position increment (GITHUB#14137) - #16384

Open
vismaytiwari wants to merge 1 commit into
apache:mainfrom
vismaytiwari:fix-fixedshingle-first-posinc
Open

Fix FixedShingleFilter emitting a leading 0 position increment (GITHUB#14137)#16384
vismaytiwari wants to merge 1 commit into
apache:mainfrom
vismaytiwari:fix-fixedshingle-first-posinc

Conversation

@vismaytiwari

Copy link
Copy Markdown
Contributor

Description

Fixes #14137.

With a valid analyzer chain — StandardTokenizerWordDelimiterGraphFilter(GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | CATENATE_NUMBERS)FlattenGraphFilterFixedShingleFilter — indexing input such as 555,0 throws IllegalArgumentException: first position increment must be > 0 (got 0). This is the same chain Elasticsearch's search_as_you_type uses.

FixedShingleFilter took a shingle's position increment from the emitting base token. But when earlier base positions produce no shingle (their graph routes are too short to fill the window) their increments were dropped, and a base reached as a graph alternative can itself carry a posInc of 0 — so the stream's first shingle could be emitted with posInc=0, which the indexer rejects.

For 555,0 the flattened graph feeding the filter is "5550"(posInc=1), "555"(posInc=0), "0"(posInc=1). The "5550" base can't form a bigram and emits nothing, so the first emitted shingle "555 0" is based at "555" (posInc=0) and inherited that 0.

The fix accumulates the base-position increments across bases that emit nothing and applies the total to the next emitted shingle (0 for further graph routes stacked at the same base). So "555 0" now carries 1 + 0 = 1. Normal streams are unchanged: a single base still contributes its own increment, and stacked graph routes at the same base still get 0.

Added TestFixedShingleFilter#testGraphInputDoesNotEmitLeadingZeroPositionIncrement, which builds the reported chain over 555,0. It fails on main (leading posInc=0) and passes with this change. Full TestFixedShingleFilter and ./gradlew tidy are green.

…B#14137)

The shingle position increment was taken from the emitting base token, but
base positions whose graph routes were too short to form a shingle emitted
nothing and their increments were lost. A base reached as a graph alternative
can itself carry a 0 increment, so the stream's first shingle could be emitted
with posInc=0, which fails indexing with "first position increment must be > 0".

Accumulate the base-position increments across bases that emit nothing and
apply the total to the next emitted shingle (0 for further routes stacked at
the same base), so the first token always keeps a valid increment.
@github-actions

Copy link
Copy Markdown
Contributor

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

@github-actions github-actions Bot added the Stale label Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exception raised when using FixedShingleFilter with WordDelimiterGraphFilter

1 participant