Enable dynamic filters for range-partitioned joins - #23854
Open
peterxcli wants to merge 10 commits into
Open
Conversation
…/hash-join-dynamic-filter-with-range-partition
saadtajwar
reviewed
Jul 24, 2026
Member
Author
|
cc @gene-bordegaray @jayshrivastava @stuhood @adriangb — kindly pinging on this one. Please take a look if you have time, thanks! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23854 +/- ##
==========================================
+ Coverage 80.85% 80.89% +0.04%
==========================================
Files 1099 1103 +4
Lines 374304 376330 +2026
Branches 374304 376330 +2026
==========================================
+ Hits 302642 304444 +1802
- Misses 53607 53768 +161
- Partials 18055 18118 +63 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
run benchmark tpch tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (1a141f3) to 1763417 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (1a141f3) to 1763417 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Member
Author
@comphead We’d need to enable range partitioning on the join inputs for the effect to show up. |
getChan
reviewed
Jul 26, 2026
getChan
left a comment
Contributor
There was a problem hiding this comment.
make sense to me!
If it's not too much trouble, could you share some performance comparison benchmark results?
Member
Author
this still need some additional plumbing to really work, but let me try to do that and run some benchmark |
Contributor
|
I believe there is much discussion going on around the future of dynamic filters here #23814 . I am unable to give PRs full attention this week but think syncing with @jayshrivastava will be very useful |
…mic-filter-with-range-partition
peterxcli
force-pushed
the
feat/hash-join-dynamic-filter-with-range-partition
branch
from
July 31, 2026 04:29
9056b30 to
bf4e6a1
Compare
|
Hi @peterxcli, thanks for the request (#23854 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: 2010YOUY01, Dandandan, Fokko, Jefffrey, Omega359, Rachelint, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, coderfender, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mkleen, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas. File an issue against this benchmark runner |
Contributor
|
run benchmark tpch tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Member
Author
I looked at #23814, IIUC, it is orthogonal to this patch, it only discussed how to define the api so other nodes can receive the probe side fragment plan and dynamic filter of hash join, because what we're doing here is still building |
Member
Author
Currently the tpch also I'm looking into #23093. |
Contributor
|
@peterxcli It doesn't have to be TPC-H—we could also add some focused benchmarks under datafusion/physical-plan/benches. |
jayshrivastava
left a comment
Contributor
There was a problem hiding this comment.
These code changes look good! Sorry for the delay in reviewing.
I think we could use 3 more tests. I left comments about 2 tests below - pretty small changes.
It would be nice to have an e2e test showing that this works. There's a test here:
which you can probably copy for range partitioning.| { | ||
| Arc::clone(&real_branches[0].1) | ||
| Arc::clone(&partition_filters[real_partition_ids[0]]) | ||
| } else if let Some(range_partitioning) = &self.probe_range_partitioning { |
Contributor
There was a problem hiding this comment.
Nice this is all much easier to read.
jayshrivastava
approved these changes
Aug 4, 2026
jayshrivastava
left a comment
Contributor
There was a problem hiding this comment.
Nice. Left a nit, but this LGTM!
- You might want to bump this in discord or at the community meeting tomorrow so a maintainer can take a look.
- I think a sqllogictest should be possible and would be nice, but it's not blocking because the existing test is good.
- In a separate issue, it would be nice to add benchmarks for range partitioning. I don't think there's any. Can you file one under #22395?
| // Not portable to sqllogictest: this test pins `PartitionMode::Partitioned` | ||
| // by hand-wiring matching Range repartitioning on both join sides, which the | ||
| // SQL planner does not currently produce. | ||
| #[tokio::test] |
Contributor
There was a problem hiding this comment.
I don't think you need this comment. It's totally fine to construct the plan manually. The Hash partitioned test in this file does the same.
For the record, I think there should be a way to get range partitioning working in sqllogictest? See here:
|
|
||
| // Rough sketch of the Range-partitioned MRE we're trying to recreate. The | ||
| // test hand-wires identical Range repartitioning because SQL planning does | ||
| // not currently derive the split points: |
Contributor
There was a problem hiding this comment.
I don't think we need "because SQL planning does not currently derive the split points:" in this comment
Great test though!
|
|
||
| assert!(join.allow_join_dynamic_filter_pushdown(session_config.options())); | ||
|
|
||
| let hash_join = join |
Contributor
There was a problem hiding this comment.
nit: change the test name to not be so specific to range partitioning
| Some(9), | ||
| None, | ||
| None, | ||
| ])), |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Partitioned hash joins build one dynamic filter per build partition. Existing routing uses
hash(key) % N, which cannot reproduce a Range partitioning layout.Compatible Range co-partitioned joins instead need to route probe rows using their existing ordering and split points.
What changes are included in this PR?
CASEexpression that routes probe rows to the corresponding partition filter using the Range ordering and split points.Are these changes tested?
unit test.
Are there any user-facing changes?
no