Add CodSpeed CPU, memory, and macro benchmark integration

[adriangb]

Summary

This PR integrates CodSpeed for continuous performance benchmarking of DataFusion with three modes:

Changes

• Workspace dependency update: Replaced criterion = "0.8" with codspeed-criterion-compat = "4.3.0" using Cargo dependency renaming (package = "codspeed-criterion-compat"). This is a drop-in replacement that preserves all existing use criterion::*; imports across the codebase -- no benchmark source code changes required.

• CI workflow (.github/workflows/codspeed.yml) with three jobs:

  • Criterion CPU Simulation: Runs all 107 criterion micro-benchmarks with instruction counting (<1% variance)
  • Criterion Memory Profiling: Tracks heap allocations across all criterion benchmarks
  • Macro Benchmarks: Runs TPC-H SF1 (22 queries) and ClickBench partitioned (43 queries) via codspeed exec for per-query regression tracking

• README badge: Added CodSpeed badge linking to the project dashboard.

• Macro benchmark script (benchmarks/codspeed_macro.sh): Runs each TPC-H and ClickBench query individually through codspeed exec so each gets tracked as a separate benchmark.

How it works

The codspeed-criterion-compat crate acts as a passthrough when running cargo bench normally, so existing benchmark workflows are unaffected. When benchmarks are run through cargo codspeed in CI, CodSpeed instruments the execution to produce consistent, hardware-agnostic measurements.

Affected crates

All 13 crates with criterion benchmarks are covered automatically via the workspace dependency:

• datafusion (core)
• datafusion-common
• datafusion-datasource
• datafusion-datasource-parquet
• datafusion-functions
• datafusion-functions-aggregate
• datafusion-functions-aggregate-common
• datafusion-functions-nested
• datafusion-functions-window
• datafusion-optimizer
• datafusion-physical-expr
• datafusion-physical-plan
• datafusion-spark

Design decisions

• Simulation mode for macro benchmarks (not walltime): Works on standard ubuntu-latest runners with <1% variance
• codspeed exec per query: Gives per-query regression tracking rather than one aggregate measurement
• Iterations set to 1: Under simulation mode, benchmarks are deterministic (instruction-counting)
• ClickBench partitioned: ~14GB across 100 parquet files, downloaded with 10 parallel workers

Test plan

• cargo bench --no-run verifies all criterion benchmarks compile with codspeed-criterion-compat
• CI: Criterion CPU simulation job builds and runs successfully
• CI: Criterion memory profiling job builds and runs successfully
• CI: Macro benchmark job generates data and runs 65 benchmarks (22 TPC-H + 43 ClickBench)

🤖 Generated with Claude Code