Add dfbench statistics command

[gabotechs]

Which issue does this PR close?

• Part of Epic: Statistics improvements #8227

Rationale for this change

Provide a repeatable way to measure how closely planning cardinality statistics match runtime output for benchmark query suites.

The goal is not to have an automated benchmark suite, and not something to enforce in the CI. Instead, this is meant to be a development tool for developers to locally iterate over stats estimation improvements.

What changes are included in this PR?

dfbench statistics accepts any Parquet data directory and SQL file or directory, so it can be used with TPC-DS, TPC-H, or another compatible query suite. It:

• prints an indented, per-operator estimate-versus-runtime report as each query completes;
• reports q-error per operator and finite q-error p50/p75/p95/p99 across the run;
• stores results by branch, comparing with the previous run by default or a named branch using --compare.

About q-error calculation: https://vldb.org/pvldb/vol9/p204-leis.pdf

Example: verify a TPC-DS Q21 improvement

1. Run TPC-DS Q21 for storing the baseline

cargo run --bin dfbench statistics \
  --path benchmarks/data/tpcds_sf1 \
  --query_path datafusion/core/tests/tpc-ds \
  --query 21

2. Perform this small change in https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/sorts/sort.rs#L1427-L1427:

-        Ok(Arc::new(stats.with_fetch(self.fetch, 0, 1)?))
+        Ok(Arc::new(stats.with_fetch(
+            self.fetch,
+            0,
+            self.topk_emitter_count(),
+        )?))

3. Run again the command from step 1.

=== 21 ===
SortPreservingMergeExec: rows=Inexact(100) vs 100, q-error: previous=1.00x, current=1.00x, change=0.0%
  SortExec(TopK): rows=Inexact(1600) vs 1417, q-error: previous=14.40x, current=1.13x, change=✅ 92.2%
    ....

Compare against a baseline from main

# Run on main first; the result is stored under main
cargo run --bin dfbench statistics \
  --path benchmarks/data/tpcds_sf1 \
  --query_path datafusion/core/tests/tpc-ds \
  --query 21

# Then, on a feature branch
cargo run --bin dfbench statistics \
  --path benchmarks/data/tpcds_sf1 \
  --query_path datafusion/core/tests/tpc-ds \
  --query 21 \
  --compare main

TPC-H works with its partitioned data layout as well:

cargo run --bin dfbench statistics \
  --path benchmarks/data/tpch_sf1 \
  --query_path benchmarks/queries \
  --query q1

Are these changes tested?

No unit tests were added: this is a reporting CLI over planner statistics and runtime metrics, exercised by the benchmark-suite smoke runs above.

If reviewers think this type of code needs to be covered by unit tests let me know.

Are there any user-facing changes?

No, this is just for extending the dfbench command with a new dfbench statistics subcommand

[codecov-commenter]

Codecov Report

❌ Patch coverage is 18.84368% with 379 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.83%. Comparing base (68d5874) to head (6366a30).
⚠️ Report is 67 commits behind head on main.

Files with missing lines	Patch %	Lines
benchmarks/src/statistics.rs	18.88%	367 Missing and 11 partials ⚠️
benchmarks/src/bin/dfbench.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23975      +/-   ##
==========================================
+ Coverage   80.75%   81.83%   +1.08%     
==========================================
  Files        1096     1103       +7     
  Lines      373282   397640   +24358     
  Branches   373282   397640   +24358     
==========================================
+ Hits       301440   325416   +23976     
- Misses      53869    54627     +758     
+ Partials    17973    17597     -376     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

@gabotechs
Thanks for adding this new statistics command.
I found a few issues that could make benchmark reports incomplete, stale, or nondeterministic.
I also left a testing suggestion to help protect the command's repeatability guarantees.

[kosiew]

It would be helpful to add focused unit tests for SQL-aware multi-statement parsing, partial-run persistence and exit status, and duplicate table-name rejection. These cases define the repeatability contract for the new standalone CLI and would help prevent regressions.

[gabotechs]

Added some tests here 6366a30.

I want to be careful in not introducing too many tests, as those also require maintenance, and this is just a benchmarking tool that is not in a production path.

Let me know if you think we should be covering some more.

[alamb]

I think @Omega359 is in the process of porting dfbench code -- maybe we should use the new format

[gabotechs]

@alamb I imagine you are referring to #23772. Unless I misinterpreted the intentions there, I see that's focusing on performance benchmarks, while this PR focuses on planning-time statistics vs execution-time metrics statistical divergence.

I've tried several things already for shipping a tool that can qualify improvements to DataFusion stats system:

• Modeling stats estimation as integration tests, attempted in Add statistics integration tests #20292 without success because of non-determinism.
• Enhancing existing benchmarks so that they can additionally output the planning stats VS execution metrics as another output. However, this required shoehorning some logic into the existing benchmarks, and the benefit is not very big, as performance-based benchmarks have a different set of requirements that are not needed for planning stats VS execution metrics benchmarks, like executing the same queries a certain amount of times, measuring timing, etc...

As there's really not any precedence about what I'm trying to do here, and it does not quite fit neither in the existing benchmark infrastructure or the current integration tests, I preferred to ship something as isolated as possible, so that:

• It does not get in the way of normal benchmarks with additional (potentially unrelated) code
• We can easily nuke it if we don't find it useful without touching any other pre-existing files.

Also @Omega359, if you have ideas about how to integrate this with your work, they are more than welcome, but my impression is that integrating this with the existing or future (#23772) is going to require some shoehorning that is likely to get in the way of other efforts rather than being helpful.

[alamb]

@alamb I imagine you are referring to #23772. Unless I misinterpreted the intentions there, I see that's focusing on performance benchmarks, while this PR focuses on planning-time statistics vs execution-time metrics statistical divergence.

I've tried several things already for shipping a tool that can qualify improvements to DataFusion stats system:

As there's really not any precedence about what I'm trying to do here, and it does not quite fit neither in the existing benchmark infrastructure or the current integration tests, I preferred to ship something as isolated as possible, so that:

• It does not get in the way of normal benchmarks with additional (potentially unrelated) code
• We can easily nuke it if we don't find it useful without touching any other pre-existing files.

Fair enough.

Another thing we could do potentially is to add some sort of mode to the benchmark runner ("stats verification mode" perhaps?) that runs the query and then verifies that the actual metrics match the statistics 🤔

[Omega359]

Also @Omega359, if you have ideas about how to integrate this with your work, they are more than welcome, but my impression is that integrating this with the existing or future (#23772) is going to require some shoehorning that is likely to get in the way of other efforts rather than being helpful.

Most of the benchmarks in dfbench will be redirected to the SQL benchmark suite, and their corresponding Rust-based implementations will be removed in the somewhat near future. However, dfbench itself will remain because a few benchmarks are not well suited for SQL benchmarking. From a quick look at the code for this PR this looks like it'll be one of those. I don't think it's a concern at this point.

[gabotechs]

Another thing we could do potentially is to add some sort of mode to the benchmark runner ("stats verification mode" perhaps?)

What I found challenging is how to integrate this with existing benchmark runner without it getting in the way of the classical "performance benchmarks" path.

For example, typical performance benchmarks have the option to specify the number of iterations, which does not make much sense with planing stats vs execution metrics benchmarks, and planing stats vs execution metrics benchmarks need to render the output in a very certain way for them to be useful (plans displayed with q-error per operator), which does not make sense for classical benchmarks.

What I found is that integrating it in existing benchmarks would require some "if normal_path; do this; else if stats_vs_metrics_path; do this other different thing" conditional logic that could end up getting in the way of other people contributing to the classical benchmarks.

Most of the code added in this PR is kind of very specific to this planning stats vs execution metrics benchmarks, but there are some common bits that could potentially be reused with other types of benchmarks: fn query_files(), fn register_parquet_files() and fn collect_parquet_files(), making a total of 80 LOC that could potentially be reused from the ~650 LOC these new benchmarks have.

Even if 80 LOC out of 650 is something, it's still not a lot to be reused with existing benchmarks, the other 570 LOC are very specific to this new type of benchmark.