Skip to content

feat: add ASOF join physical operator - #23828

Open
Xuanwo wants to merge 6 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-physical
Open

feat: add ASOF join physical operator#23828
Xuanwo wants to merge 6 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-physical

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jul 23, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

This is the first layer of the ASOF JOIN stack. It establishes a
broadcast-based physical execution contract independently so later logical-plan,
SQL, DataFrame, and serialization changes can be reviewed as smaller follow-up
PRs.

The initial implementation deliberately favors the simpler broadcast design:
the right input must fit in memory and each left partition scans the shared
right-side batches. A repartitioned implementation can be evaluated separately
without changing the ASOF semantics introduced here.

What changes are included in this PR?

  • Add AsOfJoinExec for left-preserving, Snowflake-style ASOF semantics.
  • Coalesce and collect the ordered right input once, then share it across all
    left partitions.
  • Keep the left input partitioned so each partition can scan independently and
    preserve the left-side output partitioning.
  • Preserve merge state across input and output batch boundaries.
  • Reserve each retained Arrow buffer exactly once, including when right-side
    batches are zero-copy slices, and expose build, match, and output metrics.
  • Define output properties and statistics for the broadcast execution model.
  • Add physical operator tests covering match directions, equality groups,
    batch boundaries, unmatched rows, invalid contracts, shared-buffer memory
    accounting, and multi-partition broadcast execution.

Are these changes tested?

Yes:

  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test -p datafusion-physical-plan joins::asof_join --all-features
  • Extended workspace tests from the contributor guide

Are there any user-facing changes?

This adds a new physical operator API. SQL and DataFrame APIs are intentionally
left to dependent PRs in the ASOF JOIN stack.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jul 23, 2026
@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.14286% with 230 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.68%. Comparing base (e8a65f2) to head (34552d1).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/joins/asof_join.rs 82.14% 166 Missing and 64 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23828      +/-   ##
==========================================
+ Coverage   80.66%   80.68%   +0.01%     
==========================================
  Files        1095     1096       +1     
  Lines      372294   373822    +1528     
  Branches   372294   373822    +1528     
==========================================
+ Hits       300324   301630    +1306     
- Misses      54055    54185     +130     
- Partials    17915    18007      +92     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@2010YOUY01 2010YOUY01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! This is a really good start. I have done a quick first pass and left some suggestions.

Comment thread datafusion/physical-plan/src/joins/asof_join.rs
Comment thread datafusion/physical-plan/src/joins/asof_join.rs
Comment thread datafusion/physical-plan/src/joins/asof_join.rs
Comment thread datafusion/physical-plan/src/joins/asof_join.rs
Comment thread datafusion/physical-plan/src/joins/asof_join.rs
vec![ChildStats::At(partition), ChildStats::Skip]
}

fn statistics_from_inputs(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an idea to make this PR smaller, could we use the default implementation here? We could implement it later in a follow-up PR.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept this small override because ASOF has two exact facts the default would discard: the output row count equals the left row count, and unmodified left columns retain their statistics. Right-side column statistics remain unknown. I added a comment to make that scope explicit.

Comment thread datafusion/physical-plan/src/joins/asof_join.rs
Comment thread datafusion/physical-plan/src/joins/asof_join.rs
Comment thread datafusion/physical-plan/src/joins/asof_join.rs
Comment thread datafusion/physical-plan/src/joins/asof_join.rs
@Xuanwo
Xuanwo marked this pull request as ready for review August 3, 2026 04:13
@Xuanwo
Xuanwo requested a review from 2010YOUY01 August 3, 2026 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants