Add support for running sql benchmarks with command line arguments by Omega359 · Pull Request #23772 · apache/datafusion

Omega359 · 2026-07-21T19:04:31Z

Which issue does this PR close?

Part of [EPIC] Add SQL based benchmarking #21706 and CLI (non criterion) interface for sql runner #21937

Rationale for this change

SQL benchmark suites expose configuration through environment variables. Contributors must consult the source or documentation to discover suite-specific settings, and Cargo does not forward custom arguments to benchmark targets.

This change lets benchmark_runner expose those settings as validated command-line arguments. It also adds suite-specific help and a dry-run mode for inspecting resolved configuration without executing a benchmark.

What changes are included in this PR?

This PR adds TOML .suite metadata files that define each SQL benchmark suite’s:

Description and help examples.
Suite-specific options, defaults, environment variables, and accepted values.
Query filename pattern.
Data-path replacements.

benchmark_runner uses this metadata to:

Generate suite-specific command-line arguments and help output.
Resolve suite options using command-line, environment-variable, and default-value precedence.
Support --path for suites that declare a DATA_DIR replacement.
Support --result-mode for result persistence and validation.
Validate suite names, metadata, option conflicts, query identifiers, and incompatible arguments.
Provide --dry-run JSON output containing the resolved options, paths, value sources, filters, and execution mode without loading benchmark
definitions or executing SQL.

The existing basic runner, Criterion mode, and suite-listing functionality remain in place.

Are these changes tested?

Yes. New tests cover:

Suite metadata parsing and validation.
Dynamic suite options and help output.
Option precedence between command-line arguments, environment variables, and defaults.
Path and query filename resolution.
Dry-run output and validation.
Result-mode resolution.
Invalid option combinations and malformed metadata.

Are there any user-facing changes?

Yes. Contributors can configure suite-specific settings through command-line arguments and inspect them through suite help:

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --help

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
  tpch --query 15 --format csv --path /path/to/tpch

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- \
  clickbench --partitioning partitioned --dry-run

Environment variables remain supported for compatibility with direct cargo bench invocations and existing benchmark definitions.

The SQL benchmark README documents the command-line options, resolution precedence, dry-run behaviour, and .suite metadata format.

I have a script that cover a large number of combinations to exercise the benchmark_runner as much as possible.

…p and --dry-run

codecov-commenter · 2026-07-21T19:28:39Z

Codecov Report

❌ Patch coverage is 96.00232% with 69 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.96%. Comparing base (db0c31b) to head (f3d8593).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
benchmarks/src/bin/benchmark_runner.rs	95.28%	29 Missing and 17 partials ⚠️
benchmarks/src/sql_benchmark_suite.rs	96.74%	9 Missing and 10 partials ⚠️
benchmarks/src/sql_benchmark_runner.rs	97.60%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23772      +/-   ##
==========================================
+ Coverage   80.89%   80.96%   +0.06%     
==========================================
  Files        1102     1103       +1     
  Lines      376111   377722    +1611     
  Branches   376111   377722    +1611     
==========================================
+ Hits       304251   305813    +1562     
- Misses      53753    53776      +23     
- Partials    18107    18133      +26

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Omega359 · 2026-07-21T19:41:18Z

@alamb - this should be the last main PR for the cli. The remaining work is in the bench.sh script to use the benchmark_runner and updated documentation.

alamb · 2026-07-27T06:53:51Z

This is on my list to review shortly -- I am traveling this week but hope to get it reviewed shortly

alamb

Thank you @Omega359 -- I played around with this for a while this afternoon and it works really nicely 👌

I am sorry it took so long, but it is a large PR (as you probably know 😆 )

This will make running benchmarks really nice

alamb · 2026-08-05T17:57:44Z

+description = "Run all ClickBench queries against the single-file dataset."
+
+[[examples]]
+command = "cargo run --release --bin benchmark_runner -- clickbench --query 7"


I tried it like

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench --query 7

(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench --query 7 Finished `release` profile [optimized] target(s) in 0.24s Running `target/release/benchmark_runner clickbench --query 7` clickbench/Q07 iteration 0: 146.3 ms, 18 rows clickbench/Q07 iteration 1: 28.0 ms, 18 rows clickbench/Q07 iteration 2: 22.9 ms, 18 rows

It was also sweet to be able to do

cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench --query 7 --iterations=100

And have it do

(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench --query 7 --iterations=100 Finished `release` profile [optimized] target(s) in 0.21s Running `target/release/benchmark_runner clickbench --query 7 --iterations=100` clickbench/Q07 iteration 0: 37.4 ms, 18 rows clickbench/Q07 iteration 1: 28.1 ms, 18 rows ... clickbench/Q07 iteration 96: 29.0 ms, 18 rows clickbench/Q07 iteration 97: 26.2 ms, 18 rows clickbench/Q07 iteration 98: 29.2 ms, 18 rows clickbench/Q07 iteration 99: 26.4 ms, 18 rows

alamb · 2026-08-05T18:01:12Z

+DATA_DIR = "../../data"
+
+[[options]]
+name = "partitioning"


this is pretty neat -- so I can do

You can easily see the diiffernc

(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench_extended --iterations=1 Finished `release` profile [optimized] target(s) in 0.40s Running `target/release/benchmark_runner clickbench_extended --iterations=1` clickbench_extended/Q00/single iteration 0: 616.4 ms, 1 rows clickbench_extended/Q01/single iteration 0: 120.4 ms, 1 rows clickbench_extended/Q02/single iteration 0: 263.7 ms, 10 rows clickbench_extended/Q03/single iteration 0: 243.9 ms, 10 rows clickbench_extended/Q04/single iteration 0: 1247.9 ms, 2 rows clickbench_extended/Q05/single iteration 0: 9762.0 ms, 2 rows clickbench_extended/Q06/single iteration 0: 11.7 ms, 1 rows clickbench_extended/Q07/single iteration 0: 520.7 ms, 10 rows clickbench_extended/Q08/single iteration 0: 294.6 ms, 10 rows clickbench_extended/Q09/single iteration 0: 1907.5 ms, 1 rows clickbench_extended/Q10/single iteration 0: 576.0 ms, 1 rows clickbench_extended/Q11/single iteration 0: 1140.2 ms, 1 rows clickbench_extended/Q12/single iteration 0: 129.8 ms, 1 rows

And

(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench_extended --iterations=1 --partitioning partitioned Finished `release` profile [optimized] target(s) in 0.40s Running `target/release/benchmark_runner clickbench_extended --iterations=1 --partitioning partitioned` clickbench_extended/Q00/partitioned iteration 0: 651.8 ms, 1 rows clickbench_extended/Q01/partitioned iteration 0: 102.8 ms, 1 rows clickbench_extended/Q02/partitioned iteration 0: 237.8 ms, 10 rows clickbench_extended/Q03/partitioned iteration 0: 226.6 ms, 10 rows clickbench_extended/Q04/partitioned iteration 0: 1289.1 ms, 2 rows clickbench_extended/Q05/partitioned iteration 0: 10181.2 ms, 2 rows clickbench_extended/Q06/partitioned iteration 0: 4.7 ms, 1 rows clickbench_extended/Q07/partitioned iteration 0: 716.6 ms, 10 rows clickbench_extended/Q08/partitioned iteration 0: 281.5 ms, 10 rows clickbench_extended/Q09/partitioned iteration 0: 1800.9 ms, 1 rows clickbench_extended/Q10/partitioned iteration 0: 457.3 ms, 1 rows clickbench_extended/Q11/partitioned iteration 0: 892.2 ms, 1 rows clickbench_extended/Q12/partitioned iteration 0: 117.3 ms, 1 rows

alamb · 2026-08-05T18:41:08Z

+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --query 15 --format csv
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- clickbench --partitioning partitioned --dry-run
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --query 1 --result-mode persist
+cargo run -p datafusion-benchmarks --release --bin benchmark_runner -- tpch --query 1 --result-mode validate


it is really cool

(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/runner$ cargo run -p datafusion-benchmarks --profile=profiling --bin benchmark_runner -- h2o --size=medium --subgroup=groupby
Finished profiling profile [optimized + debuginfo] target(s) in 0.16s
Running target/profiling/benchmark_runner h2o --size=medium --subgroup=groupby
Loading medium groupby csv h2o data
h2o/Q01/groupby iteration 0: 895.4 ms, 100 rows
h2o/Q01/groupby iteration 1: 886.0 ms, 100 rows
h2o/Q01/groupby iteration 2: 884.8 ms, 100 rows
Loading medium groupby csv h2o data

alamb · 2026-08-05T18:44:44Z

Since I think it is unlikely that anyone else is going to review this PR and it has already been outstanding for so long and it is a testing tool (not part of the library) I will merge it now

Omega359 · 2026-08-05T19:44:14Z

Thanks @alamb, much appreciated.

Add support for running benchmarks with command line arguments, --hel…

1c7a4be

…p and --dry-run

Omega359 changed the title ~~Add support for running benchmarks with command line arguments~~ Add support for running sq benchmarks with command line arguments Jul 21, 2026

Omega359 changed the title ~~Add support for running sq benchmarks with command line arguments~~ Add support for running sql benchmarks with command line arguments Jul 21, 2026

Omega359 marked this pull request as ready for review July 21, 2026 19:40

Merge branch 'main' into benchmark_runner_v2

6a5bcd5

Merge branch 'main' into benchmark_runner_v2

f3d8593

gabotechs mentioned this pull request Aug 4, 2026

Add dfbench statistics command #23975

Open

alamb approved these changes Aug 5, 2026

View reviewed changes

alamb added this pull request to the merge queue Aug 5, 2026

alamb mentioned this pull request Aug 5, 2026

CLI (non criterion) interface for sql runner #21937

Open

Merged via the queue into apache:main with commit 373fab7 Aug 5, 2026
38 checks passed

Navigation Menu

Uh oh!

Add support for running sql benchmarks with command line arguments - #23772

Add support for running sql benchmarks with command line arguments#23772
alamb merged 3 commits into
apache:mainfrom
Omega359:benchmark_runner_v2

Omega359 commented Jul 21, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 21, 2026 •

edited

Loading

Uh oh!

Omega359 commented Jul 21, 2026

Uh oh!

alamb commented Jul 27, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb Aug 5, 2026

Uh oh!

alamb Aug 5, 2026

Uh oh!

alamb Aug 5, 2026

Uh oh!

alamb commented Aug 5, 2026

Uh oh!

Uh oh!

Omega359 commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Omega359 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Omega359 commented Jul 21, 2026

Uh oh!

alamb commented Jul 27, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Aug 5, 2026

Uh oh!

Uh oh!

Omega359 commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Omega359 commented Jul 21, 2026 •

edited

Loading

codecov-commenter commented Jul 21, 2026 •

edited

Loading