Add FixedSizeBinary support for MultiGroupBy by maxburke · Pull Request #23646 · apache/datafusion

maxburke · 2026-07-16T17:11:03Z

Closes #23645

Rationale for this change

Multi-Group-By has cases for regular Binary/LargeBinary types, but not FixedSizeBinary

Are these changes tested?

Yes.

Are there any user-facing changes?

No

codecov-commenter · 2026-07-16T17:39:44Z

Codecov Report

❌ Patch coverage is 98.92761% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.67%. Comparing base (4957f5d) to head (5eb8979).

Files with missing lines	Patch %	Lines
...s/group_values/multi_group_by/fixed_size_binary.rs	98.71%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23646      +/-   ##
==========================================
+ Coverage   80.66%   80.67%   +0.01%     
==========================================
  Files        1087     1088       +1     
  Lines      367409   367782     +373     
  Branches   367409   367782     +373     
==========================================
+ Hits       296367   296726     +359     
- Misses      53390    53400      +10     
- Partials    17652    17656       +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb

Thanks @maxburke -- what is the use case for this query? Is it that you are grouping on multiple columns and if that also includes a fixed size binary the performance drops off?

alamb · 2026-07-16T18:42:47Z

Related discussion

feat(physical-plan): generic Rows-backed GroupColumn keeps mixed schemas on the column-wise path #23523

alamb · 2026-07-16T18:46:00Z

There is also this PR

feat(physical-plan): generic Rows-backed GroupColumn keeps mixed schemas on the column-wise path #23523

However I don't think that will handle FixedSizeBinary

Rich-T-kid

This PR looks very reasonable. I noticed that this is very similar to the ByteGroupValueBuilder implementation.
I think it would be worth it to try and see if we can modify ByteGroupValueBuilder to support similar optimizations such as batched writes. besides that vectorized/equal_to and the other trait methods are 1-1.
If ByteGroupValueBuilder cant be expanded on to include fixed size binary without extensive branching that hurts readability thats also fine.

Rich-T-kid · 2026-07-16T19:14:05Z

+
+impl GroupColumn for FixedSizeBinaryGroupValueBuilder {
+    fn equal_to(&self, lhs_row: usize, array: &ArrayRef, rhs_row: usize) -> bool {
+        debug_assert!(matches!(array.data_type(), DataType::FixedSizeBinary(_)));


the array input is guaranteed to be the correct type. you can remove the assertion here

Rich-T-kid · 2026-07-16T19:17:05Z

+        rhs_rows: &[usize],
+        equal_to_results: &mut BooleanBufferBuilder,
+    ) {
+        debug_assert!(matches!(array.data_type(), DataType::FixedSizeBinary(_)));


similar point to above

alamb · 2026-07-16T19:35:09Z

This PR looks very reasonable. I noticed that this is very similar to the ByteGroupValueBuilder implementation. I think it would be worth it to try and see if we can modify ByteGroupValueBuilder to support similar optimizations such as batched writes. besides that vectorized/equal_to and the other trait methods are 1-1. If ByteGroupValueBuilder cant be expanded on to include fixed size binary without extensive branching that hurts readability thats also fine.

This sounds like a good thing to explore in a follow on / parallel PR

alamb · 2026-07-16T19:35:48Z

Since I think the point of doing this PR is to improve performance it would be nice to have some benchmark results

I created a PR with some benchmarks here

bench: add FixedSizeBinary coverage to multi_group_by benchmark #23650

Hopefully we can merge that and then use it to validate that this PR improves things

maxburke · 2026-07-16T20:47:25Z

Thanks @maxburke -- what is the use case for this query? Is it that you are grouping on multiple columns and if that also includes a fixed size binary the performance drops off?

More so that if we're grouping by a FixedSizeBinary column we run out of memory in the GroupValuesRows interning and crash :)

…he#23650) ## Which issue does this PR close? - Related to apache#23646 - Related to apache#23645. ## Rationale for this change The point of a specialized FixedSizeBinary group values is performance but we have no performance benchmark for it. ## What changes are included in this PR? Adds a `fixed_size_binary` experiment to `datafusion/physical-plan/benches/multi_group_by.rs` Run with: ```bash cargo bench -p datafusion-physical-plan --bench multi_group_by --features test_utils -- fixed_size_binary ``` ## Are these changes tested? This is benchmark-only. The benchmark compiles on `main` and runs end-to-end on top of apache#23646. No product code changes. ## Are there any user-facing changes? No. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

alamb · 2026-07-17T18:44:11Z

Can you please merge this PR up from main (so I can run the benchmarks using the bot?) I can't do it myself b/c I don't have write permissions int he urbanlogiq repo

maxburke · 2026-07-17T18:47:57Z

Can you please merge this PR up from main (so I can run the benchmarks using the bot?) I can't do it myself b/c I don't have write permissions int he urbanlogiq repo

done!

alamb · 2026-08-03T20:49:56Z

run benchmark multi_group_by

adriangbot · 2026-08-03T20:52:31Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5171560183-1360-vphx4 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing multi-group-by-fsb (5eb8979) to 4957f5d (merge-base) diff

Run configuration

run benchmark multi_group_bu

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-03T20:52:53Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5171561354-1361-qb869 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing multi-group-by-fsb (5eb8979) to 4957f5d (merge-base) diff

Run configuration

run benchmark multi_group_by

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-03T20:53:04Z

Benchmark for this request failed.

Run configuration

run benchmark multi_group_by

Last 20 lines of output:

Click to expand

  Downloaded hmac v0.13.0
  Downloaded sysinfo v0.39.6
  Downloaded serde_json v1.0.150
  Downloaded icu_collections v2.2.0
  Downloaded substrait v0.63.0
  Downloaded parquet v59.1.0
  Downloaded zstd-sys v2.0.16+zstd.1.5.7
  Downloaded time v0.3.47
  Downloaded unicode-width v0.2.2
  Downloaded typify-impl v0.5.0
  Downloaded aws-lc-sys v0.40.0
  Downloaded tokio v1.52.3
  Downloaded ring v0.17.14
  Downloaded liblzma-sys v0.4.6
  Downloaded linux-raw-sys v0.12.1
  Downloaded sha2-const-stable v0.1.0
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`

File an issue against this benchmark runner

adriangbot · 2026-08-03T21:09:30Z

Benchmark for this request failed.

Run configuration

run benchmark multi_group_bu

Last 20 lines of output:

Click to expand

# Micro-Benchmarks (specific operators and features)
cancellation:           How long cancelling a query takes
nlj:                    Benchmark for simple nested loop joins, testing various join scenarios
hj:                     Benchmark for simple hash joins, testing various join scenarios
smj:                    Benchmark for simple sort merge joins, testing various join scenarios
dict:                   Benchmark for dictionary-encoded group-by scenarios
compile_profile:        Compile and execute TPC-H across selected Cargo profiles, reporting timing and binary size


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Supported Configuration (Environment Variables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DATA_DIR            directory to store datasets
CARGO_COMMAND       command that runs the benchmark binary
DATAFUSION_DIR      directory to use (default /workspace/datafusion-base)
RESULTS_NAME        folder where the benchmark files are stored
PREFER_HASH_JOIN    Prefer hash join algorithm (default true)
SIMULATE_LATENCY    Simulate object store latency to mimic S3 (default false)
DATAFUSION_*        Set the given datafusion configuration

File an issue against this benchmark runner

alamb · 2026-08-03T21:22:22Z

run benchmark multi_group_by

adriangbot · 2026-08-03T21:25:42Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5171878692-1362-wpk62 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing multi-group-by-fsb (5eb8979) to 4957f5d (merge-base) diff

Run configuration

run benchmark multi_group_by

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-03T21:25:53Z

Benchmark for this request failed.

Run configuration

run benchmark multi_group_by

Last 20 lines of output:

Click to expand

  Downloaded rustls v0.23.39
  Downloaded tokio-stream v0.1.18
  Downloaded zerocopy v0.8.48
  Downloaded regex-syntax v0.8.11
  Downloaded csv v1.4.0
  Downloaded regex-automata v0.4.14
  Downloaded time v0.3.47
  Downloaded rustix v1.1.4
  Downloaded tokio v1.52.3
  Downloaded liblzma-sys v0.4.6
  Downloaded zstd-sys v2.0.16+zstd.1.5.7
  Downloaded toml_datetime v1.1.1+spec-1.1.0
  Downloaded regress v0.10.5
  Downloaded linux-raw-sys v0.12.1
  Downloaded aws-lc-sys v0.40.0
  Downloaded sha2-const-stable v0.1.0
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`

File an issue against this benchmark runner

github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jul 16, 2026

alamb reviewed Jul 16, 2026

View reviewed changes

alamb mentioned this pull request Jul 16, 2026

bench: add FixedSizeBinary coverage to multi_group_by benchmark #23650

Merged

Rich-T-kid reviewed Jul 16, 2026

View reviewed changes

maxburke force-pushed the multi-group-by-fsb branch from e843517 to f8ebf73 Compare July 16, 2026 20:49

Implement GroupColumn for fixed size binary types

5eb8979

maxburke force-pushed the multi-group-by-fsb branch from 3970070 to 5eb8979 Compare July 17, 2026 18:54

This comment was marked as outdated.

Sign in to view

Uh oh!

Conversation

maxburke commented Jul 16, 2026

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

maxburke Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

maxburke Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

maxburke commented Jul 16, 2026

Uh oh!

alamb commented Jul 17, 2026

Uh oh!

maxburke commented Jul 17, 2026

Uh oh!

This comment was marked as outdated.

alamb commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

alamb commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Jul 16, 2026 •

edited

Loading