feat: add ASOF join physical operator - #23828
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23828 +/- ##
==========================================
+ Coverage 80.66% 80.68% +0.01%
==========================================
Files 1095 1096 +1
Lines 372294 373822 +1528
Branches 372294 373822 +1528
==========================================
+ Hits 300324 301630 +1306
- Misses 54055 54185 +130
- Partials 17915 18007 +92 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2010YOUY01
left a comment
There was a problem hiding this comment.
Thank you! This is a really good start. I have done a quick first pass and left some suggestions.
| vec![ChildStats::At(partition), ChildStats::Skip] | ||
| } | ||
|
|
||
| fn statistics_from_inputs( |
There was a problem hiding this comment.
Just an idea to make this PR smaller, could we use the default implementation here? We could implement it later in a follow-up PR.
There was a problem hiding this comment.
I kept this small override because ASOF has two exact facts the default would discard: the output row count equals the left row count, and unmodified left columns retain their statistics. Right-side column statistics remain unknown. I added a comment to make that scope explicit.
Which issue does this PR close?
Rationale for this change
This is the first layer of the ASOF JOIN stack. It establishes a
broadcast-based physical execution contract independently so later logical-plan,
SQL, DataFrame, and serialization changes can be reviewed as smaller follow-up
PRs.
The initial implementation deliberately favors the simpler broadcast design:
the right input must fit in memory and each left partition scans the shared
right-side batches. A repartitioned implementation can be evaluated separately
without changing the ASOF semantics introduced here.
What changes are included in this PR?
AsOfJoinExecfor left-preserving, Snowflake-style ASOF semantics.left partitions.
preserve the left-side output partitioning.
batches are zero-copy slices, and expose build, match, and output metrics.
batch boundaries, unmatched rows, invalid contracts, shared-buffer memory
accounting, and multi-partition broadcast execution.
Are these changes tested?
Yes:
cargo fmt --allcargo clippy --all-targets --all-features -- -D warningscargo test -p datafusion-physical-plan joins::asof_join --all-featuresAre there any user-facing changes?
This adds a new physical operator API. SQL and DataFrame APIs are intentionally
left to dependent PRs in the ASOF JOIN stack.