Add CaseFoldingFilter with flat lookup table - #16371
Conversation
|
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution! |
I think we've gone thru this before with the ASCIIFoldingFilter, and the most efficient way is to just tableize basic ascii (first 128) and do a switch statement for the remainder. You dont need to build explicit table, just return toLowerCase() for value < 128. its already using two-stage tables behind the scenes. Otherwise you blow away too much cache with the big lookup table. It might look good in a microbenchmark but not in an overall indexer run. Plus the data is pretty sparse (most of the codepoints dont have case), using a big lookup table is overkill. |
Replace 128KB flat lookup table with a generated switch statement. ~200 codepoints differ between foldCase and toLowerCase.
|
Thanks for the feedback. |
|
yeah this is looking good to me. sorry for dropping the ball on the previous PR, thanks for getting all this together |
|
How does this differ from the The |
|
Ah it was inspired by the other one. Maybe rename the groovy files a bit to make clear which is doing what. It's not so obvious. ... and please fix the comment on the generated file by the other one! Thanks! |
uschindler
left a comment
There was a problem hiding this comment.
looks ok, but please do the file renames :-)
Fix stale regeneration command in generated file headers. Add SimpleCaseFolding.java and CaseFolding.java to .gitattributes.
| setupIcuDependencies(project) | ||
|
|
||
| tasks.register("generateUnicodeProps", { | ||
| tasks.register("generateCaseFolding", { |
There was a problem hiding this comment.
I think that's wrong. There's also a UnicodeProps script (that I created long ago).
I think there might be a task missing. Can you try it out and check that all files are regenerated when you delete the hash?
There was a problem hiding this comment.
another idea on the names, one of these is doing expansion and the other is doing compression/folding.
we might want to eventually fix the java file in o.a.l.u.automaton to make this clear too. Its named CaseFolding.java but it isn't doing any folding, it is doing expansion
There was a problem hiding this comment.
I've updated the task names and renamed them to CaseExpansion.
There was a problem hiding this comment.
thank you for that cleanup: I think it makes it a lot less confusing!
I would propose we consider renaming the .java and groovy files to be The difference (this is just conceptual and of course its not doing A/a but only cornercases):
|
Rename tasks to regenerateCaseFolding and regenerateSimpleCaseFolding so they integrate with the checksum framework and are picked up by `gradlew regenerate`. The previous generateUnicodeProps task lost its checksum tracking during a build refactoring (b39e9bc). Regenerate CaseFolding.java with current ICU 78.3.
Add a token filter that applies Unicode simple case folding to normalize text for case-insensitive matching, without requiring the ICU analysis plugin.
Performs simple case folding for ~1200 BMP codepoints, which differs from Character.toLowerCase() such as: Greek final sigma (ς→σ), micro sign to Greek mu (µ→μ) and others. For ASCII text, folding is identical to lowercasing. Supplementary codepoints (> U+FFFF) fall back to Character.toLowerCase().
Uses a flat char[65536] lookup table (128 KB) lazily initialized on first use (once per JVM), built from ~1200 {codepoint, folded} pairs (which get discarded after initialization) generated from ICU4J via ./gradlew :lucene:core:generateFoldTable
Inspired by #14389.
Benchmark
AMD EPYC (c5a.2xlarge), JDK 25, 256 tokens x 8 chars
Baseline is Character.toLowerCase()