[MOD-9685] Introduce SVS Basic Benchmarks #829

meiravgri · 2025-11-06T14:10:55Z

This PR introduces a benchmarking suite for the SVS algorithm, including basic operations on loaded indices built on top of the existing training phase infrastructure.

New Benchmarks Added

BM_AddLabelOneByOne - Measures time to add individual vectors to a loaded SVS index one-by-one
BM_TriggerUpdateTiered - Measures time to move vectors from frontend (flat buffer) to backend (SVS index) in tiered index
BM_RunGC - Tests graph repairing after deletions

Test Name	Number of Vectors	Thread Count
BM_AddLabelOneByOne	1,024	1
BM_TriggerUpdateTiered	1,024, 5,000, 10,240 (update_threshold)	2, 4, 8
BM_RunGC	50, 100, 250, 500 (num_deletions)	1

introduce BM_VecSimSVSTrain class with 2 methods: Train and TrainAsync add GoogleTest to benchmarks so we can use ASSERT_* API tieredIndexMock: possible to initialize with a specific thread count add train bemchmark to CI benchmark dispatcher

rename svs_training_fp32 ->svs_indices_training_fp32 add to bm_files.sh

move svs params init to CreateTieredSVSIndex only 5 iterations

add compressed index bm

remove some prints

remove 100K

move UNIT_AND_ITERATIONS and QUANT_BITS_ARGS to bm_vecsim_basics_Svs

… to bm_training_initialize.h define DATA_TYPE_INDEX_T in bm_svs_training_fp*.cpp remove th 10k and 50k for arm

fix include header for fp16

codecov · 2025-11-06T14:29:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.63%. Comparing base (1a15b59) to head (a4c95f5).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #829   +/-   ##
=======================================
  Coverage   96.63%   96.63%           
=======================================
  Files         126      126           
  Lines        7379     7379           
=======================================
  Hits         7131     7131           
  Misses        248      248

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tests/benchmark/bm_svs_index.h

we hard code the name of the data file based on quantBits

benchmark deletsion according to a when we exceed 0.5 index size this benchmark takes 20K ms (20s) for 500 vectors!!! that's a lot after revert - benchmark only gc to detrmine

runGC instead of delete label to not be depnd on consolidation_threshold that can't be controloed and runs for vrey very long!

fix mac

tests/benchmark/bm_vecsim_svs.h

Co-authored-by: BenGoldberger <[email protected]>

github-actions · 2025-11-12T13:55:28Z

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-829-to-8.2 origin/8.2
cd .worktree/backport-829-to-8.2
git switch --create backport-829-to-8.2
git cherry-pick -x 11cdc8b7a8435dfb2e62f37293f6dabd55b7465b

* initial imp of training bm introduce BM_VecSimSVSTrain class with 2 methods: Train and TrainAsync add GoogleTest to benchmarks so we can use ASSERT_* API tieredIndexMock: possible to initialize with a specific thread count add train bemchmark to CI benchmark dispatcher * make bm_files general for other algs rename svs_training_fp32 ->svs_indices_training_fp32 add to bm_files.sh * replace std::formtat only supported from gcc13 with ostringstream * format * revrt assert * intialize quantBits move svs params init to CreateTieredSVSIndex only 5 iterations * move iterawtion logic to runTrainBMIteration add compressed index bm * assert depdnding on HAVE_SVS_LVQ * sepearate non compression and compression bm * TO REVERT !!! test abort * fix if else * revrt timeoutgurard vhanges * dont pause after training to see how it affrects performance remove some prints * fix #ifdef HAVE_SVS_LVQ to #if HAVE_SVS_LVQ * use pause timers its faster * do 3 iter instread of 5 and test if results are stable * use 5 again * fix download all all script * fp16 bm remove 100K * remove 100K from fp32 * increase timeout * try bigger machine * try a bigger machine * try 2 iter move UNIT_AND_ITERATIONS and QUANT_BITS_ARGS to bm_vecsim_basics_Svs * unify bm_training_initialize_fp32.h and bm_training_initialize_fp16.h to bm_training_initialize.h define DATA_TYPE_INDEX_T in bm_svs_training_fp*.cpp remove th 10k and 50k for arm * reevet timeout to 10 fix include header for fp16 * move CreateTieredSVSParams and verifyNumThreads to svs params * revert increease machine size * change assert to log * fix * fix2 * format * introduce bm_svs * add tiered add NewIndex from existing svs to tiered factory imp AddLabel add AddLabelBatches(not implmneted) take bm_utils from meiravg_svs_training_bm * introduce setUpdateTriggerThreshold in BUILD_TESTS move initialize index to a function introfuce bm function: addlabel: insert one by one AddLabelBatches: add in batches with one thread AddLabelAsync: add in batches with multiple threads * fix comment add to yml * remove lock * format * fix num threads in addlabelinplace fix assertupdateTriggerThreshold in AddLabelAsync * use train svs instead * format * small fixes * rename BM_VecSimSVSTrain->BM_VecSimSVS bm_vecsim_svs_train.h->bm_vecsim_svs * remove unrelated * align with new name * revert unnecessary changes in bm_vecsim_index add LVQ BM if HAVE_SVS_LVQ * fix include * fix quantbits * extract general * fix missing main on LVQ cpp * replace vectors file * run only BENCHMARK_MAIN * try dummy for mac * fix DATA_TYPE_INDEX_T definition LVQ * quantBits is now static and needs to be intizlied by the CPP file we hard code the name of the data file based on quantBits * TO REVERT: benchmark deletsion according to a when we exceed 0.5 index size this benchmark takes 20K ms (20s) for 500 vectors!!! that's a lot after revert - benchmark only gc to detrmine * REVERT svs.h change consolidation_threshold runGC instead of delete label to not be depnd on consolidation_threshold that can't be controloed and runs for vrey very long! * revert unrelated changes * fix LVQ8 cpp for non LVQ * foirmat * cleanups fix mac * remove new line in cmake * Update tests/benchmark/bm_vecsim_svs.h Co-authored-by: BenGoldberger <[email protected]> --------- Co-authored-by: BenGoldberger <[email protected]> (cherry picked from commit 11cdc8b)

* [MOD-9685] Introduce SVS Basic Benchmarks (#829) * initial imp of training bm introduce BM_VecSimSVSTrain class with 2 methods: Train and TrainAsync add GoogleTest to benchmarks so we can use ASSERT_* API tieredIndexMock: possible to initialize with a specific thread count add train bemchmark to CI benchmark dispatcher * make bm_files general for other algs rename svs_training_fp32 ->svs_indices_training_fp32 add to bm_files.sh * replace std::formtat only supported from gcc13 with ostringstream * format * revrt assert * intialize quantBits move svs params init to CreateTieredSVSIndex only 5 iterations * move iterawtion logic to runTrainBMIteration add compressed index bm * assert depdnding on HAVE_SVS_LVQ * sepearate non compression and compression bm * TO REVERT !!! test abort * fix if else * revrt timeoutgurard vhanges * dont pause after training to see how it affrects performance remove some prints * fix #ifdef HAVE_SVS_LVQ to #if HAVE_SVS_LVQ * use pause timers its faster * do 3 iter instread of 5 and test if results are stable * use 5 again * fix download all all script * fp16 bm remove 100K * remove 100K from fp32 * increase timeout * try bigger machine * try a bigger machine * try 2 iter move UNIT_AND_ITERATIONS and QUANT_BITS_ARGS to bm_vecsim_basics_Svs * unify bm_training_initialize_fp32.h and bm_training_initialize_fp16.h to bm_training_initialize.h define DATA_TYPE_INDEX_T in bm_svs_training_fp*.cpp remove th 10k and 50k for arm * reevet timeout to 10 fix include header for fp16 * move CreateTieredSVSParams and verifyNumThreads to svs params * revert increease machine size * change assert to log * fix * fix2 * format * introduce bm_svs * add tiered add NewIndex from existing svs to tiered factory imp AddLabel add AddLabelBatches(not implmneted) take bm_utils from meiravg_svs_training_bm * introduce setUpdateTriggerThreshold in BUILD_TESTS move initialize index to a function introfuce bm function: addlabel: insert one by one AddLabelBatches: add in batches with one thread AddLabelAsync: add in batches with multiple threads * fix comment add to yml * remove lock * format * fix num threads in addlabelinplace fix assertupdateTriggerThreshold in AddLabelAsync * use train svs instead * format * small fixes * rename BM_VecSimSVSTrain->BM_VecSimSVS bm_vecsim_svs_train.h->bm_vecsim_svs * remove unrelated * align with new name * revert unnecessary changes in bm_vecsim_index add LVQ BM if HAVE_SVS_LVQ * fix include * fix quantbits * extract general * fix missing main on LVQ cpp * replace vectors file * run only BENCHMARK_MAIN * try dummy for mac * fix DATA_TYPE_INDEX_T definition LVQ * quantBits is now static and needs to be intizlied by the CPP file we hard code the name of the data file based on quantBits * TO REVERT: benchmark deletsion according to a when we exceed 0.5 index size this benchmark takes 20K ms (20s) for 500 vectors!!! that's a lot after revert - benchmark only gc to detrmine * REVERT svs.h change consolidation_threshold runGC instead of delete label to not be depnd on consolidation_threshold that can't be controloed and runs for vrey very long! * revert unrelated changes * fix LVQ8 cpp for non LVQ * foirmat * cleanups fix mac * remove new line in cmake * Update tests/benchmark/bm_vecsim_svs.h Co-authored-by: BenGoldberger <[email protected]> --------- Co-authored-by: BenGoldberger <[email protected]> (cherry picked from commit 11cdc8b) * fix factory --------- Co-authored-by: BenGoldberger <[email protected]>

meiravgri added 28 commits November 4, 2025 05:00

initial imp of training bm

36afa1a

introduce BM_VecSimSVSTrain class with 2 methods: Train and TrainAsync add GoogleTest to benchmarks so we can use ASSERT_* API tieredIndexMock: possible to initialize with a specific thread count add train bemchmark to CI benchmark dispatcher

make bm_files general for other algs

3eee014

rename svs_training_fp32 ->svs_indices_training_fp32 add to bm_files.sh

replace std::formtat only supported from gcc13 with ostringstream

6636ae8

format

dda6487

revrt assert

d6d9149

intialize quantBits

8491ae9

move svs params init to CreateTieredSVSIndex only 5 iterations

move iterawtion logic to runTrainBMIteration

3d4f9ba

add compressed index bm

assert depdnding on HAVE_SVS_LVQ

10e6184

sepearate non compression and compression bm

a2dad0c

TO REVERT !!! test abort

af8e7d9

fix if else

b6cd5c6

revrt timeoutgurard vhanges

dfba28f

dont pause after training to see how it affrects performance

b90a085

remove some prints

fix #ifdef HAVE_SVS_LVQ to #if HAVE_SVS_LVQ

7c4efb4

use pause timers its faster

39f66dc

do 3 iter instread of 5 and test if results are stable

f43df17

Merge remote-tracking branch 'origin/main' into meiravg_svs_training_bm

f4f2b57

use 5 again

74f7e67

fix download all all script

11b8c80

fp16 bm

e5c82f9

remove 100K

remove 100K from fp32

f558d63

Merge branch 'main' into meiravg_svs_training_bm

c97b329

increase timeout

c3f72db

try bigger machine

59ac591

try a bigger machine

4d6d992

try 2 iter

acc8036

move UNIT_AND_ITERATIONS and QUANT_BITS_ARGS to bm_vecsim_basics_Svs

unify bm_training_initialize_fp32.h and bm_training_initialize_fp16.h…

1f4e574

… to bm_training_initialize.h define DATA_TYPE_INDEX_T in bm_svs_training_fp*.cpp remove th 10k and 50k for arm

reevet timeout to 10

219bd4a

fix include header for fp16

github-advanced-security bot found potential problems Nov 6, 2025

View reviewed changes

tests/benchmark/bm_svs_index.h Fixed Show fixed Hide fixed

meiravgri added 5 commits November 10, 2025 08:16

run only BENCHMARK_MAIN

18b7ccf

try dummy for mac

a0faa10

fix DATA_TYPE_INDEX_T definition LVQ

da0f834

quantBits is now static and needs to be intizlied by the CPP file

3e062e8

we hard code the name of the data file based on quantBits

TO REVERT:

5a8394f

benchmark deletsion according to a when we exceed 0.5 index size this benchmark takes 20K ms (20s) for 500 vectors!!! that's a lot after revert - benchmark only gc to detrmine

Base automatically changed from meiravg_svs_training_bm to main November 10, 2025 13:59

meiravgri added 6 commits November 10, 2025 15:09

REVERT svs.h change consolidation_threshold

e742636

runGC instead of delete label to not be depnd on consolidation_threshold that can't be controloed and runs for vrey very long!

Merge branch 'main' into meiravg-svs_basic_bm

5ed3e89

revert unrelated changes

67912d0

fix LVQ8 cpp for non LVQ

f48f149

foirmat

0219063

cleanups

f70da31

fix mac

meiravgri added bm-svs-train-fp32 bm-svs-train-fp16 labels Nov 10, 2025

meiravgri changed the title ~~introduce bm_svs~~ [MOD-9685] Introduce SVS Basic Benchmarks Nov 10, 2025

meiravgri requested a review from BenGoldberger November 10, 2025 16:49

meiravgri added the backport 8.2 label Nov 10, 2025

remove new line in cmake

719671c

BenGoldberger reviewed Nov 11, 2025

View reviewed changes

tests/benchmark/bm_vecsim_svs.h Outdated Show resolved Hide resolved

Update tests/benchmark/bm_vecsim_svs.h

a4c95f5

Co-authored-by: BenGoldberger <[email protected]>

meiravgri enabled auto-merge November 11, 2025 13:32

BenGoldberger approved these changes Nov 12, 2025

View reviewed changes

meiravgri added this pull request to the merge queue Nov 12, 2025

Merged via the queue into main with commit 11cdc8b Nov 12, 2025
37 of 38 checks passed

meiravgri deleted the meiravg-svs_basic_bm branch November 12, 2025 13:55

meiravgri mentioned this pull request Nov 13, 2025

[8.2] [MOD-9685] Introduce SVS Basic Benchmarks (#829) #834

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[MOD-9685] Introduce SVS Basic Benchmarks #829

[MOD-9685] Introduce SVS Basic Benchmarks #829

Uh oh!

meiravgri commented Nov 6, 2025 •

edited

Loading

Uh oh!

codecov bot commented Nov 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[MOD-9685] Introduce SVS Basic Benchmarks #829

[MOD-9685] Introduce SVS Basic Benchmarks #829

Uh oh!

Conversation

meiravgri commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Benchmarks Added

Uh oh!

codecov bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

meiravgri commented Nov 6, 2025 •

edited

Loading

codecov bot commented Nov 6, 2025 •

edited

Loading