[Fix] Binary search the entries when all suffixes have the same length in a leaf block. by vsop-479 · Pull Request #11888 · apache/lucene

vsop-479 · 2022-10-28T11:45:14Z

Fix #11722 's bug.

update

… leaf block.

…s(NOT_FOUND, END).

…nextEnt plus 1.

…nd greater term exists.

vsop-479 · 2022-10-28T11:46:53Z

@jpountz I have fixed the code and passed the tests, please have a review.

jpountz · 2022-11-10T10:03:00Z

Thanks @vsop-479. Do you know if the test you added to terms can be improved in such a way that it would have caught this bug?

vsop-479 · 2022-11-12T03:21:12Z

@jpountz I added a test case for this bug, but it only works for the tests with PostingsFormat which's DEFAULT_MIN_BLOCK_SIZE is 25, add build the first block with 25 entries, such as TestLucene90PostingsFormat.
It does not work for the tests with other PostingsFormat, such as TestFixedGapPostingsFormat and TestVarGapFixedIntervalPostingsFormat.
I am trying to find a better way for this case.

… block to TestLucene90PostingsFormat.

vsop-479 · 2022-11-12T12:22:31Z

@jpountz I moved the test case for this bug to TestLucene90PostingsFormat. Please have a review.

jpountz · 2022-11-14T15:25:29Z

Thanks for looking into a test for this.

@mikemccand Would you have time to have a look at this? I knew it was complex, but even more than I had anticipated, so I would appreciate another look to get a sense of whether there could be another bug lurking?

vsop-479 · 2022-12-19T13:28:10Z

@jpountz Is there any progress on this PR?

Update from main.

vsop-479 · 2023-10-12T07:06:32Z

Append some performance data. Note that the results have quite diversity from different rounds.

round1

                        Task     QPS baseline   StdDev   QPS bsearch    StdDev  Pct diff            p-value
     BrowseRandomLabelTaxoFacets     2601.88      (5.0%)     2437.93     (11.1%)   -6.3% ( -21% -   10%) 0.021
     BrowseRandomLabelSSDVFacets     1796.93      (7.1%)     1696.76      (8.0%)   -5.6% ( -19% -   10%) 0.019
                       MedPhrase     4021.51      (6.7%)     3813.81      (7.6%)   -5.2% ( -18% -    9%) 0.022
                      HighPhrase      821.97      (7.3%)      788.46      (6.3%)   -4.1% ( -16% -   10%) 0.059
                        HighTerm     6221.04      (7.3%)     6032.11      (5.6%)   -3.0% ( -14% -   10%) 0.138
                          Fuzzy2      276.46      (6.0%)      268.79      (4.8%)   -2.8% ( -12% -    8%) 0.105
                         Respell      707.62      (5.9%)      692.48      (4.9%)   -2.1% ( -12% -    9%) 0.211
       BrowseDayOfYearSSDVFacets     6517.91      (5.4%)     6392.55      (6.0%)   -1.9% ( -12% -   10%) 0.287
            BrowseDateSSDVFacets     2195.69     (19.2%)     2155.70      (9.7%)   -1.8% ( -25% -   33%) 0.705
           BrowseMonthSSDVFacets     6724.35      (6.7%)     6606.24      (6.1%)   -1.8% ( -13% -   11%) 0.386
                         Prefix3     3023.99      (4.5%)     2974.60      (5.3%)   -1.6% ( -10% -    8%) 0.293
                          Fuzzy1      886.09      (5.1%)      877.56      (6.0%)   -1.0% ( -11% -   10%) 0.587
                        PKLookup      396.87     (15.8%)      393.83     (19.2%)   -0.8% ( -30% -   40%) 0.890
                       OrHighLow     3242.80      (6.0%)     3223.53      (7.2%)   -0.6% ( -13% -   13%) 0.778
           BrowseMonthTaxoFacets     5874.67      (6.1%)     5856.79      (5.1%)   -0.3% ( -10% -   11%) 0.865
                          IntNRQ     2799.54      (4.7%)     2792.69      (5.4%)   -0.2% (  -9% -   10%) 0.878
             LowIntervalsOrdered     1336.37      (8.1%)     1333.20      (5.6%)   -0.2% ( -12% -   14%) 0.914
                    HighSpanNear     2660.49      (6.7%)     2654.45      (6.1%)   -0.2% ( -12% -   13%) 0.911
                         LowTerm     9965.56      (8.1%)     9961.77     (10.6%)   -0.0% ( -17% -   20%) 0.990
                     AndHighHigh     3384.43      (7.0%)     3388.41     (11.3%)    0.1% ( -16% -   19%) 0.968
                HighSloppyPhrase     1984.76      (5.8%)     1988.83      (5.3%)    0.2% ( -10% -   12%) 0.908
             MedIntervalsOrdered     7914.54      (8.2%)     7944.02     (10.2%)    0.4% ( -16% -   20%) 0.899
                      AndHighMed     4097.29      (7.7%)     4121.75      (8.0%)    0.6% ( -14% -   17%) 0.811
                     LowSpanNear     5107.67      (9.3%)     5145.19      (7.6%)    0.7% ( -14% -   19%) 0.785
               HighTermMonthSort     3221.73      (5.2%)     3245.54      (8.5%)    0.7% ( -12% -   15%) 0.739
            HighIntervalsOrdered     1333.81      (7.6%)     1349.72      (5.3%)    1.2% ( -10% -   15%) 0.564
                       LowPhrase     5029.07      (8.3%)     5091.95     (11.1%)    1.3% ( -16% -   22%) 0.687
                        Wildcard     1327.36      (3.8%)     1346.91      (3.4%)    1.5% (  -5% -    9%) 0.197
                      AndHighLow     4382.38      (7.8%)     4447.59      (6.9%)    1.5% ( -12% -   17%) 0.524
                       OrHighMed     3121.72      (7.2%)     3169.60      (6.4%)    1.5% ( -11% -   16%) 0.478
           HighTermDayOfYearSort     3766.72      (6.2%)     3825.04      (7.2%)    1.5% ( -11% -   15%) 0.467
                         MedTerm     8666.16      (7.3%)     8841.37      (8.1%)    2.0% ( -12% -   18%) 0.406
                 LowSloppyPhrase     3303.11      (6.5%)     3374.89      (7.8%)    2.2% ( -11% -   17%) 0.341
                      OrHighHigh     2458.90      (7.7%)     2512.82      (5.2%)    2.2% (  -9% -   16%) 0.289
            BrowseDateTaxoFacets     6229.11      (5.5%)     6366.43      (5.7%)    2.2% (  -8% -   14%) 0.211
       BrowseDayOfYearTaxoFacets     5695.81      (6.8%)     5830.89      (6.6%)    2.4% ( -10% -   16%) 0.265
                     MedSpanNear     3161.49      (6.8%)     3242.45      (5.4%)    2.6% (  -8% -   15%) 0.186
                 MedSloppyPhrase     3363.11      (7.0%)     3456.66      (7.6%)    2.8% ( -11% -   18%) 0.230

round2

                            Task     QPS baseline   StdDev   QPS bsearch    StdDev  Pct diff            p-value
                         LowTerm     9037.60      (7.0%)     8817.16      (7.1%)   -2.4% ( -15% -   12%) 0.276
                         Respell      724.07      (6.8%)      706.54      (6.1%)   -2.4% ( -14% -   11%) 0.236
                      AndHighLow     5165.41      (9.9%)     5045.20      (9.6%)   -2.3% ( -19% -   18%) 0.449
                       OrHighLow     2393.11      (6.0%)     2341.81      (7.8%)   -2.1% ( -15% -   12%) 0.329
             LowIntervalsOrdered     3400.45      (5.4%)     3345.42      (6.6%)   -1.6% ( -12% -   10%) 0.395
                     AndHighHigh     3465.41      (7.8%)     3429.20      (7.3%)   -1.0% ( -14% -   15%) 0.661
                          IntNRQ     3252.71      (5.6%)     3236.81      (5.4%)   -0.5% ( -10% -   11%) 0.778
            HighIntervalsOrdered     4125.58      (4.1%)     4114.03      (6.3%)   -0.3% ( -10% -   10%) 0.867
                          Fuzzy1      951.24      (5.9%)      949.09      (6.8%)   -0.2% ( -12% -   13%) 0.911
                      HighPhrase     2552.87      (6.3%)     2547.78      (8.1%)   -0.2% ( -13% -   15%) 0.931
           BrowseMonthTaxoFacets     5806.39      (5.8%)     5799.97      (4.7%)   -0.1% ( -10% -   11%) 0.947
           HighTermDayOfYearSort     3677.43      (4.5%)     3676.63      (5.9%)   -0.0% ( -10% -   10%) 0.990
                         Prefix3     2314.90      (3.9%)     2315.43      (4.6%)    0.0% (  -8% -    8%) 0.986
       BrowseDayOfYearTaxoFacets     5734.92      (6.2%)     5739.04      (4.8%)    0.1% ( -10% -   11%) 0.967
                       LowPhrase     3718.54      (9.1%)     3727.36      (8.7%)    0.2% ( -16% -   19%) 0.933
                     MedSpanNear     5401.87      (6.7%)     5417.02      (8.4%)    0.3% ( -13% -   16%) 0.907
                         MedTerm     7734.00      (6.5%)     7767.21      (8.6%)    0.4% ( -13% -   16%) 0.858
                        Wildcard     1520.86      (4.0%)     1528.72      (5.3%)    0.5% (  -8% -   10%) 0.729
               HighTermMonthSort     3750.85      (8.9%)     3783.58      (7.7%)    0.9% ( -14% -   19%) 0.740
                        PKLookup      425.26     (15.3%)      429.46     (18.0%)    1.0% ( -28% -   40%) 0.852
                       OrHighMed     2816.18      (5.7%)     2847.69      (6.5%)    1.1% ( -10% -   14%) 0.564
                          Fuzzy2      324.51      (4.0%)      329.52      (5.4%)    1.5% (  -7% -   11%) 0.302
                    HighSpanNear     2014.03      (5.0%)     2045.92      (5.5%)    1.6% (  -8% -   12%) 0.338
                HighSloppyPhrase      339.81      (9.1%)      345.36      (8.1%)    1.6% ( -14% -   20%) 0.549
             MedIntervalsOrdered     5921.86      (6.3%)     6019.48      (8.3%)    1.6% ( -12% -   17%) 0.477
                 MedSloppyPhrase     2733.55      (5.8%)     2780.57      (7.9%)    1.7% ( -11% -   16%) 0.432
                        HighTerm     5973.94      (7.6%)     6084.16      (7.9%)    1.8% ( -12% -   18%) 0.450
                     LowSpanNear     3859.74      (7.9%)     3951.20      (9.1%)    2.4% ( -13% -   21%) 0.379
                 LowSloppyPhrase     6304.22      (8.9%)     6460.82     (10.6%)    2.5% ( -15% -   24%) 0.424
                      OrHighHigh     2347.03      (5.3%)     2410.71      (6.1%)    2.7% (  -8% -   14%) 0.133
                      AndHighMed     3984.56      (7.4%)     4099.88      (8.7%)    2.9% ( -12% -   20%) 0.257
                       MedPhrase     4069.20      (8.3%)     4191.45      (7.3%)    3.0% ( -11% -   20%) 0.224
            BrowseDateTaxoFacets     6149.31      (4.4%)     6386.12      (5.0%)    3.9% (  -5% -   13%) 0.009
       BrowseDayOfYearSSDVFacets     5832.22     (19.1%)     6062.83     (11.8%)    4.0% ( -22% -   43%) 0.431
     BrowseRandomLabelSSDVFacets     1635.27     (12.1%)     1704.50      (8.5%)    4.2% ( -14% -   28%) 0.200
           BrowseMonthSSDVFacets     5978.68     (17.2%)     6374.07     (17.6%)    6.6% ( -24% -   49%) 0.229
     BrowseRandomLabelTaxoFacets     2459.73     (11.9%)     2630.18      (4.2%)    6.9% (  -8% -   26%) 0.014
            BrowseDateSSDVFacets     2153.79      (8.1%)     2383.48     (15.7%)   10.7% ( -12% -   37%) 0.007

round3

                            Task     QPS baseline   StdDev   QPS bsearch    StdDev  Pct diff            p-value
               HighTermMonthSort     3225.93      (9.2%)     3032.53      (8.5%)   -6.0% ( -21% -   12%) 0.032
                     AndHighHigh     3020.25      (6.9%)     2920.75      (7.4%)   -3.3% ( -16% -   11%) 0.145
           HighTermDayOfYearSort     2255.60      (7.2%)     2185.62     (11.7%)   -3.1% ( -20% -   17%) 0.312
                       MedPhrase     3924.63      (7.7%)     3810.24      (8.4%)   -2.9% ( -17% -   14%) 0.254
                        HighTerm     4736.30      (6.0%)     4609.63      (6.2%)   -2.7% ( -13% -   10%) 0.164
                          Fuzzy1     1127.25      (4.0%)     1107.10      (4.7%)   -1.8% (  -9% -    7%) 0.190
                       LowPhrase     4488.86      (7.4%)     4414.48      (6.8%)   -1.7% ( -14% -   13%) 0.462
                       OrHighMed     2481.56      (4.5%)     2447.67      (7.1%)   -1.4% ( -12% -   10%) 0.468
                     LowSpanNear     5202.58      (7.1%)     5157.55      (6.4%)   -0.9% ( -13% -   13%) 0.685
                         MedTerm     6901.80      (8.7%)     6852.07      (6.7%)   -0.7% ( -14% -   16%) 0.769
                         Respell      796.66      (6.3%)      791.83      (6.8%)   -0.6% ( -12% -   13%) 0.771
             LowIntervalsOrdered     3582.00      (6.0%)     3562.96      (6.8%)   -0.5% ( -12% -   13%) 0.794
     BrowseRandomLabelTaxoFacets     2590.20      (4.1%)     2576.78      (3.1%)   -0.5% (  -7% -    7%) 0.655
                          Fuzzy2      460.42      (3.9%)      459.19      (3.8%)   -0.3% (  -7% -    7%) 0.826
            BrowseDateTaxoFacets     6234.00      (3.6%)     6217.91      (5.0%)   -0.3% (  -8% -    8%) 0.852
                     MedSpanNear     2694.43      (7.5%)     2687.55      (6.1%)   -0.3% ( -12% -   14%) 0.906
                       OrHighLow     3366.04      (7.5%)     3362.25      (7.4%)   -0.1% ( -13% -   15%) 0.962
                        Wildcard     1597.92      (5.4%)     1596.16      (4.2%)   -0.1% (  -9% -   10%) 0.943
                 LowSloppyPhrase     5180.86      (7.8%)     5183.11      (6.9%)    0.0% ( -13% -   15%) 0.985
       BrowseDayOfYearTaxoFacets     5683.56      (4.3%)     5689.03      (3.0%)    0.1% (  -6% -    7%) 0.934
                         Prefix3     2936.08      (6.2%)     2941.73      (5.1%)    0.2% ( -10% -   12%) 0.914
                HighSloppyPhrase     2332.11      (4.9%)     2336.61      (7.6%)    0.2% ( -11% -   13%) 0.924
           BrowseMonthTaxoFacets     5845.66      (5.3%)     5871.05      (4.2%)    0.4% (  -8% -   10%) 0.773
                          IntNRQ     3838.04      (4.2%)     3858.84      (7.0%)    0.5% ( -10% -   12%) 0.767
            BrowseDateSSDVFacets     2438.30     (21.7%)     2453.33     (21.7%)    0.6% ( -35% -   56%) 0.928
                    HighSpanNear     1692.49      (5.4%)     1706.61      (4.4%)    0.8% (  -8% -   11%) 0.592
                      HighPhrase     2510.17      (6.8%)     2540.12      (5.1%)    1.2% ( -10% -   14%) 0.530
                      AndHighLow     6390.20     (11.8%)     6475.17      (9.5%)    1.3% ( -17% -   25%) 0.694
             MedIntervalsOrdered     2377.72      (5.0%)     2411.77      (9.4%)    1.4% ( -12% -   16%) 0.548
                         LowTerm     9977.52      (9.0%)    10125.70     (10.3%)    1.5% ( -16% -   22%) 0.627
     BrowseRandomLabelSSDVFacets     1783.92      (9.7%)     1811.90      (6.9%)    1.6% ( -13% -   20%) 0.555
                      OrHighHigh     2221.50      (6.8%)     2257.74      (6.2%)    1.6% ( -10% -   15%) 0.426
            HighIntervalsOrdered     2496.96      (6.5%)     2568.80      (9.2%)    2.9% ( -12% -   19%) 0.254
       BrowseDayOfYearSSDVFacets     6337.43     (20.1%)     6553.25     (16.4%)    3.4% ( -27% -   49%) 0.557
                 MedSloppyPhrase     4135.87      (7.5%)     4282.32      (8.5%)    3.5% ( -11% -   21%) 0.162
                      AndHighMed     4011.41      (6.5%)     4170.15      (8.5%)    4.0% ( -10% -   20%) 0.099
           BrowseMonthSSDVFacets     6275.39     (17.9%)     6791.72     (15.5%)    8.2% ( -21% -   50%) 0.120
                        PKLookup      442.55     (10.7%)      480.49     (10.2%)    8.6% ( -11% -   33%) 0.010

github-actions · 2024-01-08T12:24:55Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

jpountz · 2024-01-08T13:55:51Z

@mikemccand I could use your help to review this change, it's quite deep in the guts of block tree.

github-actions · 2024-02-20T00:17:36Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

vsop-479 · 2024-03-15T09:26:44Z

@jpountz
I want to move subCheckBinarySearch to BasePostingsFormatTestCase to make this change forward compatible, by judging whether IndexWriterConfig is set DefaultPostingsFormat like this:

if (TestUtil.getDefaultPostingsFormat()
        .getName()
        .equals(TestUtil.getPostingsFormat(iwc.getCodec(), "id"))) {
      // test target greater than the last entry of matched block,
    }

But it won't pass if this DefaultPostingsFormat do not use DEFAULT_MIN_BLOCK_SIZE and DEFAULT_MAX_BLOCK_SIZE, such as TestPerFieldPostingsFormat.

I also tried to set DefaultCodec to test target greater than the last entry of matched block case, like this:

iwc.setCodec(TestUtil.getDefaultCodec());

TestSTUniformSplitPostingFormat.checkEncoding won't pass this, because it must use its own FieldsConsumer to set states like blocksEncoded.

Do you have any idea about this?
Can we expose minTermBlockSize and maxTermBlockSize in LuceneXXPostingsFormat to a DefaultPostingsFormat, to let user use them?

mikemccand · 2024-03-26T13:06:11Z

I like this idea! It seems like it'd especially help primary key lookup against fixed length IDs like UUID?

Hmm, the QPS in the luceneutil runs are way too high (1000s of QPS) to be trustworthy? Was this on wikimediumall?

mikemccand

Thank you for tackling this tricky optimization @vsop-479!

I need some more time to stare at the binarySearchTermLeaf but not until later ... so I'm publishing first comments :)

vsop-479 · 2024-03-27T02:10:12Z

Was this on wikimediumall?

No, this was on wikimedium10k.
I will measure the performance again on wikimediumall.

vsop-479 · 2024-03-27T08:41:57Z

@mikemccand Thanks for your review.
I measured performance on wikimediumall:

iter1

TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
           BrowseMonthTaxoFacets       11.02     (26.9%)       10.62     (28.4%)   -3.7% ( -46% -   70%) 0.727
     BrowseRandomLabelSSDVFacets        6.22      (9.1%)        6.04      (6.5%)   -2.9% ( -16% -   13%) 0.335
               HighTermTitleSort      184.11      (4.0%)      180.43      (4.0%)   -2.0% (  -9% -    6%) 0.187
                      TermDTSort      214.19      (4.6%)      210.41      (4.2%)   -1.8% ( -10% -    7%) 0.290
               HighTermMonthSort     4049.75      (3.8%)     4009.22      (6.2%)   -1.0% ( -10% -    9%) 0.606
          OrHighMedDayTaxoFacets        6.41      (6.7%)        6.35      (7.5%)   -0.9% ( -14% -   14%) 0.737
                         Prefix3      562.57      (1.3%)      558.72      (1.7%)   -0.7% (  -3% -    2%) 0.228
        AndHighHighDayTaxoFacets       23.16      (1.5%)       23.02      (1.6%)   -0.6% (  -3% -    2%) 0.288
                      HighPhrase       49.14      (3.9%)       48.88      (3.1%)   -0.5% (  -7% -    6%) 0.698
            MedTermDayTaxoFacets       17.63      (3.6%)       17.55      (2.9%)   -0.4% (  -6% -    6%) 0.718
                    HighSpanNear       15.30      (1.8%)       15.24      (1.5%)   -0.4% (  -3% -    3%) 0.544
            HighTermTitleBDVSort       10.23      (2.4%)       10.19      (3.0%)   -0.4% (  -5% -    5%) 0.717
       BrowseDayOfYearSSDVFacets        6.97      (6.3%)        6.95      (6.0%)   -0.3% ( -11% -   12%) 0.890
                         Respell       73.65      (1.5%)       73.44      (2.4%)   -0.3% (  -4% -    3%) 0.717
           HighTermDayOfYearSort      525.93      (2.5%)      524.54      (3.0%)   -0.3% (  -5% -    5%) 0.799
                     MedSpanNear       75.25      (2.6%)       75.16      (1.5%)   -0.1% (  -4% -    3%) 0.882
                       MedPhrase       62.74      (4.7%)       62.73      (2.6%)   -0.0% (  -6% -    7%) 0.987
                     LowSpanNear       10.39      (2.1%)       10.39      (1.4%)    0.0% (  -3% -    3%) 0.998
                          Fuzzy1       99.01      (1.7%)       99.07      (1.6%)    0.1% (  -3% -    3%) 0.923
                   OrNotHighHigh      576.76      (4.0%)      578.77      (4.5%)    0.3% (  -7% -    9%) 0.827
         AndHighMedDayTaxoFacets       80.00      (1.3%)       80.29      (1.6%)    0.4% (  -2% -    3%) 0.511
                       LowPhrase      149.97      (2.9%)      150.57      (2.1%)    0.4% (  -4% -    5%) 0.675
                       OrHighLow      675.00      (2.6%)      678.06      (3.1%)    0.5% (  -5% -    6%) 0.674
            HighIntervalsOrdered        2.81     (14.0%)        2.83     (11.6%)    0.5% ( -22% -   30%) 0.921
                      AndHighLow     1027.38      (4.2%)     1032.64      (3.9%)    0.5% (  -7% -    8%) 0.738
                        Wildcard      100.84      (2.4%)      101.44      (2.8%)    0.6% (  -4% -    5%) 0.547
                          Fuzzy2       92.33      (1.5%)       92.98      (1.4%)    0.7% (  -2% -    3%) 0.206
             MedIntervalsOrdered       13.07     (10.5%)       13.18      (9.4%)    0.8% ( -17% -   23%) 0.824
           BrowseMonthSSDVFacets        6.91      (8.0%)        6.97      (7.1%)    0.9% ( -13% -   17%) 0.748
                    OrNotHighMed      341.39      (3.4%)      345.26      (3.2%)    1.1% (  -5% -    7%) 0.362
                      AndHighMed      155.80      (3.0%)      157.63      (3.2%)    1.2% (  -4% -    7%) 0.316
       BrowseDayOfYearTaxoFacets        7.70      (4.0%)        7.79      (4.4%)    1.2% (  -6% -    9%) 0.450
                      OrHighHigh       42.26      (3.9%)       42.77      (3.6%)    1.2% (  -6% -    9%) 0.396
     BrowseRandomLabelTaxoFacets        7.17      (5.0%)        7.26      (4.5%)    1.2% (  -7% -   11%) 0.486
                   OrHighNotHigh      464.54      (4.5%)      470.46      (5.2%)    1.3% (  -8% -   11%) 0.490
                         LowTerm      669.77      (3.4%)      678.39      (4.3%)    1.3% (  -6% -    9%) 0.383
                       OrHighMed      118.91      (3.2%)      120.47      (3.1%)    1.3% (  -4% -    7%) 0.270
             LowIntervalsOrdered       63.73      (8.3%)       64.58      (7.7%)    1.3% ( -13% -   18%) 0.657
            BrowseDateTaxoFacets        7.63      (3.8%)        7.74      (3.8%)    1.4% (  -5% -    9%) 0.324
                     AndHighHigh       30.16      (5.2%)       30.61      (2.2%)    1.5% (  -5% -    9%) 0.323
                    OrNotHighLow     1186.21      (4.7%)     1203.99      (3.2%)    1.5% (  -6% -    9%) 0.326
                 LowSloppyPhrase        7.71      (2.4%)        7.82      (1.8%)    1.5% (  -2% -    5%) 0.058
                    OrHighNotLow      723.44      (4.9%)      734.39      (5.1%)    1.5% (  -8% -   12%) 0.425
                 MedSloppyPhrase       49.22      (2.7%)       49.99      (2.0%)    1.6% (  -3% -    6%) 0.081
                        HighTerm      605.77      (5.9%)      617.81      (8.3%)    2.0% ( -11% -   17%) 0.466
                         MedTerm      554.77      (3.9%)      565.82      (4.9%)    2.0% (  -6% -   11%) 0.234
                    OrHighNotMed      594.17      (4.4%)      606.01      (6.0%)    2.0% (  -8% -   12%) 0.314
                HighSloppyPhrase       10.92      (3.0%)       11.18      (4.1%)    2.4% (  -4% -    9%) 0.076
            BrowseDateSSDVFacets        1.59     (16.7%)        1.67     (12.7%)    4.8% ( -21% -   41%) 0.391
                        PKLookup      294.05      (2.2%)      312.97      (2.7%)    6.4% (   1% -   11%) 0.000
                          IntNRQ       28.82     (12.4%)       31.23      (8.1%)    8.4% ( -10% -   32%) 0.034

iter2

TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
           BrowseMonthTaxoFacets       11.45     (27.9%)       11.02     (28.4%)   -3.8% ( -46% -   72%) 0.678
     BrowseRandomLabelSSDVFacets        6.33      (9.5%)        6.20      (8.4%)   -2.1% ( -18% -   17%) 0.468
               HighTermTitleSort      185.35      (4.0%)      182.85      (4.4%)   -1.3% (  -9% -    7%) 0.322
               HighTermMonthSort     4091.54      (4.2%)     4049.63      (5.8%)   -1.0% ( -10% -    9%) 0.531
            HighTermTitleBDVSort       10.47      (7.8%)       10.37      (8.2%)   -1.0% ( -15% -   16%) 0.708
                      HighPhrase       48.93      (3.8%)       48.56      (3.2%)   -0.8% (  -7% -    6%) 0.501
            MedTermDayTaxoFacets       17.68      (3.7%)       17.55      (2.6%)   -0.7% (  -6% -    5%) 0.491
                       MedPhrase       62.41      (4.5%)       62.00      (3.3%)   -0.7% (  -8% -    7%) 0.610
                         Prefix3      561.61      (1.5%)      558.87      (1.8%)   -0.5% (  -3% -    2%) 0.353
       BrowseDayOfYearSSDVFacets        6.92      (5.8%)        6.89      (5.6%)   -0.4% ( -11% -   11%) 0.812
        AndHighHighDayTaxoFacets       23.15      (1.3%)       23.05      (1.6%)   -0.4% (  -3% -    2%) 0.379
            HighIntervalsOrdered        2.88     (13.5%)        2.87     (11.3%)   -0.4% ( -22% -   28%) 0.920
                         Respell       73.56      (1.7%)       73.43      (2.3%)   -0.2% (  -4% -    3%) 0.786
                    HighSpanNear       15.27      (2.1%)       15.24      (1.6%)   -0.1% (  -3% -    3%) 0.815
          OrHighMedDayTaxoFacets        6.38      (6.7%)        6.37      (7.1%)   -0.1% ( -13% -   14%) 0.962
                       LowPhrase      149.81      (2.6%)      149.67      (2.2%)   -0.1% (  -4% -    4%) 0.906
                   OrNotHighHigh      580.27      (3.8%)      579.77      (4.1%)   -0.1% (  -7% -    8%) 0.946
                          Fuzzy1       98.99      (1.7%)       98.93      (1.9%)   -0.1% (  -3% -    3%) 0.909
                      TermDTSort      213.72      (4.9%)      213.68      (4.5%)   -0.0% (  -9% -    9%) 0.989
                        Wildcard      100.49      (2.9%)      100.53      (3.2%)    0.0% (  -5% -    6%) 0.962
                      AndHighLow     1032.43      (4.0%)     1033.17      (3.8%)    0.1% (  -7% -    8%) 0.955
             MedIntervalsOrdered       13.27     (10.0%)       13.29      (8.9%)    0.1% ( -17% -   21%) 0.976
                     LowSpanNear       10.39      (2.1%)       10.41      (1.3%)    0.2% (  -3% -    3%) 0.776
                     MedSpanNear       75.09      (2.6%)       75.24      (1.4%)    0.2% (  -3% -    4%) 0.773
                       OrHighLow      678.64      (2.6%)      680.45      (3.1%)    0.3% (  -5% -    6%) 0.774
           HighTermDayOfYearSort      526.84      (2.5%)      528.27      (3.1%)    0.3% (  -5% -    6%) 0.769
         AndHighMedDayTaxoFacets       80.31      (1.5%)       80.54      (1.5%)    0.3% (  -2% -    3%) 0.554
                      AndHighMed      156.61      (2.8%)      157.25      (3.2%)    0.4% (  -5% -    6%) 0.679
                          Fuzzy2       92.44      (1.8%)       92.87      (1.7%)    0.5% (  -2% -    3%) 0.414
           BrowseMonthSSDVFacets        6.86      (7.1%)        6.90      (6.6%)    0.5% ( -12% -   15%) 0.810
                   OrHighNotHigh      469.04      (4.6%)      471.62      (5.0%)    0.5% (  -8% -   10%) 0.725
                    OrNotHighMed      342.44      (3.5%)      344.34      (3.1%)    0.6% (  -5% -    7%) 0.604
                    OrNotHighLow     1198.89      (4.9%)     1205.92      (3.8%)    0.6% (  -7% -    9%) 0.678
                       OrHighMed      119.89      (3.2%)      120.59      (2.8%)    0.6% (  -5% -    6%) 0.544
                         LowTerm      676.75      (3.8%)      680.77      (3.9%)    0.6% (  -6% -    8%) 0.637
                      OrHighHigh       42.52      (3.5%)       42.81      (3.2%)    0.7% (  -5% -    7%) 0.542
             LowIntervalsOrdered       64.52      (8.0%)       64.98      (7.2%)    0.7% ( -13% -   17%) 0.777
       BrowseDayOfYearTaxoFacets        7.77      (4.2%)        7.83      (4.5%)    0.8% (  -7% -    9%) 0.592
     BrowseRandomLabelTaxoFacets        7.26      (4.8%)        7.32      (4.2%)    0.8% (  -7% -   10%) 0.589
            BrowseDateTaxoFacets        7.70      (3.9%)        7.77      (3.9%)    0.8% (  -6% -    9%) 0.505
                     AndHighHigh       30.28      (4.6%)       30.56      (2.4%)    0.9% (  -5% -    8%) 0.443
                        HighTerm      613.47      (5.8%)      619.30      (7.4%)    0.9% ( -11% -   14%) 0.659
                    OrHighNotLow      732.44      (5.1%)      739.78      (5.2%)    1.0% (  -8% -   11%) 0.548
                 MedSloppyPhrase       49.44      (2.7%)       49.97      (2.0%)    1.1% (  -3% -    5%) 0.167
                 LowSloppyPhrase        7.74      (2.5%)        7.83      (2.0%)    1.1% (  -3% -    5%) 0.118
                         MedTerm      560.71      (3.8%)      567.47      (4.6%)    1.2% (  -6% -    9%) 0.378
                    OrHighNotMed      599.96      (4.3%)      607.78      (5.3%)    1.3% (  -7% -   11%) 0.404
                HighSloppyPhrase       11.05      (4.0%)       11.22      (3.8%)    1.5% (  -6% -    9%) 0.232
            BrowseDateSSDVFacets        1.58     (16.2%)        1.64     (13.1%)    3.6% ( -22% -   39%) 0.448
                          IntNRQ       29.55     (11.8%)       31.30      (7.8%)    5.9% ( -12% -   28%) 0.066
                        PKLookup      294.20      (2.3%)      312.60      (2.5%)    6.3% (   1% -   11%) 0.000

iter3

TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
     BrowseRandomLabelSSDVFacets        6.32      (9.3%)        6.20      (8.2%)   -1.9% ( -17% -   17%) 0.501
               HighTermTitleSort      185.14      (3.9%)      182.87      (4.3%)   -1.2% (  -9% -    7%) 0.344
                      HighPhrase       48.99      (3.8%)       48.56      (3.1%)   -0.9% (  -7% -    6%) 0.409
               HighTermMonthSort     4102.48      (4.2%)     4067.36      (5.9%)   -0.9% ( -10% -    9%) 0.599
                       MedPhrase       62.50      (4.4%)       62.01      (3.2%)   -0.8% (  -8% -    7%) 0.520
            HighTermTitleBDVSort       10.44      (7.7%)       10.36      (8.0%)   -0.8% ( -15% -   16%) 0.759
            MedTermDayTaxoFacets       17.64      (3.7%)       17.53      (2.6%)   -0.6% (  -6% -    5%) 0.543
           BrowseMonthTaxoFacets       11.18     (29.8%)       11.11     (28.6%)   -0.6% ( -45% -   82%) 0.947
                         Prefix3      561.39      (1.4%)      559.08      (1.7%)   -0.4% (  -3% -    2%) 0.411
            HighIntervalsOrdered        2.89     (13.3%)        2.88     (11.1%)   -0.4% ( -21% -   27%) 0.921
        AndHighHighDayTaxoFacets       23.11      (1.4%)       23.04      (1.6%)   -0.3% (  -3% -    2%) 0.492
       BrowseDayOfYearSSDVFacets        6.92      (5.6%)        6.90      (5.5%)   -0.2% ( -10% -   11%) 0.889
                      AndHighLow     1037.33      (4.4%)     1036.39      (3.9%)   -0.1% (  -8% -    8%) 0.945
                       LowPhrase      150.03      (2.6%)      149.91      (2.3%)   -0.1% (  -4% -    4%) 0.920
                         Respell       73.55      (1.7%)       73.51      (2.3%)   -0.1% (  -3% -    3%) 0.932
                    HighSpanNear       15.25      (2.1%)       15.24      (1.6%)   -0.0% (  -3% -    3%) 0.942
                   OrNotHighHigh      581.80      (3.9%)      581.70      (4.2%)   -0.0% (  -7% -    8%) 0.989
             MedIntervalsOrdered       13.30      (9.8%)       13.31      (8.7%)    0.0% ( -16% -   20%) 0.995
           HighTermDayOfYearSort      528.23      (2.7%)      528.68      (3.1%)    0.1% (  -5% -    6%) 0.926
                          Fuzzy1       98.95      (1.7%)       99.06      (1.9%)    0.1% (  -3% -    3%) 0.843
                        Wildcard      100.55      (2.8%)      100.69      (3.2%)    0.1% (  -5% -    6%) 0.889
          OrHighMedDayTaxoFacets        6.38      (6.5%)        6.39      (7.0%)    0.2% ( -12% -   14%) 0.943
                     LowSpanNear       10.39      (2.0%)       10.41      (1.3%)    0.2% (  -2% -    3%) 0.693
                      TermDTSort      213.54      (4.8%)      214.09      (4.5%)    0.3% (  -8% -   10%) 0.861
                     MedSpanNear       75.05      (2.6%)       75.26      (1.3%)    0.3% (  -3% -    4%) 0.670
                       OrHighMed      120.32      (3.5%)      120.68      (2.7%)    0.3% (  -5% -    6%) 0.762
         AndHighMedDayTaxoFacets       80.24      (1.5%)       80.49      (1.5%)    0.3% (  -2% -    3%) 0.505
                       OrHighLow      680.40      (2.8%)      683.00      (3.4%)    0.4% (  -5% -    6%) 0.698
                      OrHighHigh       42.66      (3.7%)       42.86      (3.2%)    0.5% (  -6% -    7%) 0.675
                          Fuzzy2       92.47      (1.7%)       92.96      (1.7%)    0.5% (  -2% -    4%) 0.327
                      AndHighMed      157.18      (3.2%)      158.05      (3.8%)    0.6% (  -6% -    7%) 0.622
             LowIntervalsOrdered       64.61      (7.8%)       65.02      (7.0%)    0.6% ( -13% -   16%) 0.789
                    OrHighNotLow      735.60      (5.3%)      740.40      (5.0%)    0.7% (  -9% -   11%) 0.690
                   OrHighNotHigh      469.07      (4.5%)      472.19      (4.9%)    0.7% (  -8% -   10%) 0.654
           BrowseMonthSSDVFacets        6.86      (7.0%)        6.90      (6.5%)    0.7% ( -11% -   15%) 0.745
                    OrNotHighMed      342.82      (3.4%)      345.24      (3.2%)    0.7% (  -5% -    7%) 0.503
                         LowTerm      677.45      (3.8%)      682.23      (3.9%)    0.7% (  -6% -    8%) 0.562
                        HighTerm      615.74      (5.8%)      621.00      (7.3%)    0.9% ( -11% -   14%) 0.682
     BrowseRandomLabelTaxoFacets        7.27      (4.7%)        7.34      (4.2%)    0.9% (  -7% -   10%) 0.514
                     AndHighHigh       30.34      (4.5%)       30.63      (2.6%)    0.9% (  -5% -    8%) 0.418
                    OrNotHighLow     1199.25      (4.8%)     1211.93      (4.3%)    1.1% (  -7% -   10%) 0.460
                 MedSloppyPhrase       49.50      (2.7%)       50.05      (2.0%)    1.1% (  -3% -    6%) 0.143
                 LowSloppyPhrase        7.76      (2.5%)        7.84      (2.1%)    1.1% (  -3% -    5%) 0.113
                    OrHighNotMed      602.01      (4.5%)      609.14      (5.2%)    1.2% (  -8% -   11%) 0.439
       BrowseDayOfYearTaxoFacets        7.75      (4.2%)        7.84      (4.4%)    1.2% (  -7% -   10%) 0.384
            BrowseDateTaxoFacets        7.69      (3.9%)        7.78      (3.9%)    1.2% (  -6% -    9%) 0.319
                         MedTerm      560.55      (3.7%)      567.87      (4.5%)    1.3% (  -6% -    9%) 0.315
                HighSloppyPhrase       11.04      (4.0%)       11.21      (3.7%)    1.6% (  -5% -    9%) 0.189
            BrowseDateSSDVFacets        1.60     (16.1%)        1.67     (14.3%)    4.2% ( -22% -   41%) 0.386
                          IntNRQ       29.63     (11.5%)       31.30      (7.5%)    5.6% ( -12% -   27%) 0.067
                        PKLookup      293.94      (2.2%)      312.15      (2.6%)    6.2% (   1% -   11%) 0.000

mikemccand

I added a few more comments -- still need to find some more time to stare at the new binary search some more. Thanks @vsop-479!

vsop-479 · 2024-03-28T07:11:50Z

Thanks for your comments @mikemccand . I have fixed them, and also removed the stale change entry about this change.
Please take a look when you get a chance.

mikemccand

OK this looks good to me -- the PKLookup gains look compelling and significant -- exciting!

I left one tiny grammar comment about javadoc/comment.

mikemccand · 2024-04-01T12:58:04Z

Actually I can just re-merge your prior CHANGES.txt entry from here, so no need to push another rev here. Thanks @vsop-479 !

… the same length in a leaf block. (#11888) * Binary search the entries when all suffixes have the same length in a leaf block. * add comment on allEqual. * BackwardsCompatibility: keep the same logic on fillTerm and SeekStatus(NOT_FOUND, END). * Update comments: modify scan to binary search. * Add unit test for binarySearchTermLeaf. * Format code. * Assert the value of termsEnum.term() is correct after seeking. * Add CHANGES entry. * Clarify "leaf block _of the terms dict_" * Set suffixesReader's position. * Advance to the greater term If binary search ended at the less term. * Assert termsEnum's position after seeking. * Tidy. * Advance to the greater term if binary search ended at the less term: nextEnt plus 1. * Advance to the greater term if binary search ended at the less term and greater term exists. * Add test case: target greater than the last entry of the matched block. * Move test case that target greater than the last entry of the matched block to TestLucene90PostingsFormat. * Move test case for target greater than the last entry of the matched block to TestLucene99PostingsFormat * Clarify code. * Replace ternary with verbose if. * Replace seekExact with seekCeil. * Replace division by 2 with logical right shift. * Remove assert ste.termExists. * Clarify code. * Remove stale change entry. * Fix comment. --------- Co-authored-by: Adrien Grand <jpountz@gmail.com>

mikemccand · 2024-04-02T14:56:46Z

Oooh this change gave a nice pop (~5.4%, ~915 -> 964 K lookups/sec) to the primary key lookup nightly benchy: https://home.apache.org/~mikemccand/lucenebench/PKLookup.html

I'll add an annotation, exciting!

vsop-479 · 2024-04-03T03:36:38Z

Glad to know that. Thanks @mikemccand .

vsop-479 and others added 17 commits August 19, 2022 10:50

Merge pull request #1 from apache/main

8c4eff7

update

Binary search the entries when all suffixes have the same length in a…

82e1401

… leaf block.

add comment on allEqual.

3258d33

BackwardsCompatibility: keep the same logic on fillTerm and SeekStatu…

6eac7c1

…s(NOT_FOUND, END).

Update comments: modify scan to binary search.

dfbb3cd

Add unit test for binarySearchTermLeaf.

3026f18

Format code.

fcd1075

Assert the value of termsEnum.term() is correct after seeking.

73efe67

Add CHANGES entry.

f2b49e0

Merge branch 'main' into binary_search_terms

48fc59a

Clarify "leaf block _of the terms dict_"

9987993

Set suffixesReader's position.

d2e21c0

Advance to the greater term If binary search ended at the less term.

bea403a

Assert termsEnum's position after seeking.

65ae171

Tidy.

c8a71e1

Advance to the greater term if binary search ended at the less term: …

df4a3b5

…nextEnt plus 1.

Advance to the greater term if binary search ended at the less term a…

7084596

…nd greater term exists.

Add test case: target greater than the last entry of the matched block.

2eeba25

Move test case that target greater than the last entry of the matched…

054b397

… block to TestLucene90PostingsFormat.

Merge pull request #7 from apache/main

262e836

Update from main.

github-actions Bot added the Stale label Jan 8, 2024

github-actions Bot added the Stale label Feb 20, 2024

github-actions Bot removed the Stale label Mar 16, 2024

vsop-479 mentioned this pull request Mar 20, 2024

Binary search all terms. #13192

Open

Clarify code.

4ef23f2

mikemccand reviewed Mar 26, 2024

View reviewed changes

Replace ternary with verbose if.

e915b93

mikemccand reviewed Mar 27, 2024

View reviewed changes

vsop-479 added 5 commits March 28, 2024 11:31

Replace seekExact with seekCeil.

65f0ed6

Replace division by 2 with logical right shift.

c92c4a8

Remove assert ste.termExists.

ad9cc76

Clarify code.

2e6afb6

Remove stale change entry.

a695c07

mikemccand approved these changes Mar 28, 2024

View reviewed changes

Comment thread lucene/core/src/java/org/apache/lucene/codecs/lucene90/blocktree/SegmentTermsEnumFrame.java Outdated

Fix comment.

f7854a6

mikemccand approved these changes Apr 1, 2024

View reviewed changes

mikemccand merged commit bf07450 into apache:main Apr 1, 2024

mikemccand added this to the 9.11.0 milestone Apr 1, 2024

mikemccand added a commit that referenced this pull request Apr 1, 2024

Fix PR number reference (#11722 -> #11888)

baecaf5

mikemccand added a commit that referenced this pull request Apr 1, 2024

Fix PR number reference (#11722 -> #11888)

b984448

vsop-479 mentioned this pull request Apr 8, 2024

Remove unnecessary assert and fix comment in SegmentTermsEnumFrame #13279

Merged

Uh oh!

Conversation

vsop-479 commented Oct 28, 2022

Uh oh!

vsop-479 commented Oct 28, 2022

Uh oh!

jpountz commented Nov 10, 2022

Uh oh!

vsop-479 commented Nov 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsop-479 commented Nov 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jpountz commented Nov 14, 2022

Uh oh!

vsop-479 commented Dec 19, 2022

Uh oh!

vsop-479 commented Oct 12, 2023

round1

round2

round3

Uh oh!

github-actions Bot commented Jan 8, 2024

Uh oh!

jpountz commented Jan 8, 2024

Uh oh!

github-actions Bot commented Feb 20, 2024

Uh oh!

vsop-479 commented Mar 15, 2024

Uh oh!

mikemccand commented Mar 26, 2024

Uh oh!

mikemccand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vsop-479 commented Mar 27, 2024

Uh oh!

vsop-479 commented Mar 27, 2024

iter1

iter2

iter3

Uh oh!

mikemccand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vsop-479 commented Mar 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikemccand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mikemccand commented Apr 1, 2024

Uh oh!

mikemccand commented Apr 2, 2024

Uh oh!

vsop-479 commented Apr 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vsop-479 commented Nov 12, 2022 •

edited

Loading

vsop-479 commented Nov 12, 2022 •

edited

Loading

vsop-479 commented Mar 28, 2024 •

edited

Loading