implement lead and lag built-in window function by jimexist · Pull Request #429 · apache/datafusion

jimexist · 2021-05-26T14:18:09Z

Which issue does this PR close?

implement lead and lag built-in window function.

based on #520 so review that first

Closes #553

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

datafusion/src/physical_plan/windows.rs

codecov-commenter · 2021-05-26T21:14:08Z

Codecov Report

Merging #429 (f763205) into master (321fda4) will decrease coverage by 0.01%.
The diff coverage is 65.47%.

@@            Coverage Diff             @@
##           master     #429      +/-   ##
==========================================
- Coverage   75.16%   75.15%   -0.02%     
==========================================
  Files         150      152       +2     
  Lines       25144    25357     +213     
==========================================
+ Hits        18899    19056     +157     
- Misses       6245     6301      +56

Impacted Files	Coverage Δ
ballista/rust/client/src/columnar_batch.rs	`0.00% <ø> (ø)`
ballista/rust/core/src/serde/scheduler/mod.rs	`14.28% <ø> (ø)`
datafusion/src/physical_plan/expressions/mod.rs	`71.42% <ø> (ø)`
...tafusion/src/physical_plan/expressions/lead_lag.rs	`38.29% <38.29%> (ø)`
...afusion/src/physical_plan/expressions/nth_value.rs	`70.76% <70.76%> (ø)`
datafusion/src/physical_plan/windows.rs	`78.32% <72.11%> (+4.54%)`	⬆️
datafusion/src/physical_plan/mod.rs	`79.09% <100.00%> (+0.38%)`	⬆️
datafusion/tests/sql.rs	`99.89% <100.00%> (ø)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 321fda4...f763205. Read the comment docs.

alamb · 2021-06-01T22:43:20Z

I plan to review this PR tomorrow

jimexist · 2021-06-02T16:46:09Z

Actually let's park this pull request for a while - I plan to implement sort and partition first and then window frame, after which the window shift approach might not be relevant.

jimexist · 2021-06-13T15:02:59Z

Actually let's park this pull request for a while - I plan to implement sort and partition first and then window frame, after which the window shift approach might not be relevant.

now that #520 is implemented, this PR is ready

jimexist · 2021-06-15T01:32:43Z

putting this back to draft as this relies on apache/arrow-rs#388 which is not yet in arrow 4.3

alamb · 2021-06-15T10:27:40Z

putting this back to draft as this relies on apache/arrow-rs#388 which is not yet in arrow 4.3

Oh no!

Can we possibly use the API that is in Arrow 4.3 (and then we can upgrade datafusion to use the new api when the next version of Arrow comes out)?

jimexist · 2021-06-15T11:28:42Z

putting this back to draft as this relies on apache/arrow-rs#388 which is not yet in arrow 4.3

Oh no!

Can we possibly use the API that is in Arrow 4.3 (and then we can upgrade datafusion to use the new api when the next version of Arrow comes out)?

I don't mind parking this one here for a while since there would be many other window frame stuff to be done before revisiting this and by that time newer version would be released

alamb · 2021-06-15T12:29:31Z

I don't mind parking this one here for a while since there would be many other window frame stuff to be done before revisiting this and by that time newer version would be released

Ok, thank you. The plan is to do a 4.4 release in ~ 2 weeks

jimexist · 2021-06-29T13:21:02Z

@alamb and @Dandandan this pull request is ready now

alamb

Looks like a great start @jimexist -- I do have a question if this will generate the correct answer with multiple partitions.

Also, I suggest an end-to-end integration test using your great harness, but I suspect you plan to do so in a subsequent PR :) 👍

alamb · 2021-06-29T18:57:01Z

datafusion/src/physical_plan/expressions/lead_lag.rs

+impl PartitionEvaluator for WindowShiftEvaluator {
+    fn evaluate_partition(&self, _partition: Range<usize>) -> Result<ArrayRef> {
+        let value = &self.values[0];
+        shift(value.as_ref(), self.shift_offset).map_err(DataFusionError::ArrowError)


do you need to restrict the window to the partition bounds? If the input array had 10 rows in 2 partitions, wouldn't this code produce 2 output partitions of 10 rows each (rather than 2 output partitions of 5 rows each)?

@alamb good catch, this is fixed and add with integration tests.

alamb · 2021-07-01T21:31:53Z

Thanks @jimexist -- I ran out of time today but will check this out tomorrow

alamb

Thanks @jimexist !

… release (apache#429)

jimexist mentioned this pull request May 26, 2021

Support window functions with empty OVER clause #298

Closed

Dandandan reviewed May 26, 2021

View reviewed changes

datafusion/src/physical_plan/windows.rs Outdated Show resolved Hide resolved

jimexist force-pushed the add-lag-and-lead branch from b78b355 to 26ca0fe Compare May 26, 2021 20:51

jimexist mentioned this pull request May 28, 2021

add first_value, last_value, and nth_value built-in window functions #403

Merged

jimexist force-pushed the add-lag-and-lead branch 3 times, most recently from 4161e98 to fddf52a Compare May 31, 2021 12:02

jimexist force-pushed the add-lag-and-lead branch 2 times, most recently from 9d153a7 to 737c2dd Compare June 2, 2021 07:44

jimexist force-pushed the add-lag-and-lead branch from 737c2dd to 225c7ec Compare June 4, 2021 15:27

alamb added the datafusion label Jun 4, 2021

jimexist force-pushed the add-lag-and-lead branch 8 times, most recently from a4523e6 to f676db8 Compare June 13, 2021 15:00

jimexist marked this pull request as ready for review June 13, 2021 15:03

jimexist force-pushed the add-lag-and-lead branch from f676db8 to 7db8d17 Compare June 15, 2021 00:13

jimexist marked this pull request as draft June 15, 2021 01:32

jimexist force-pushed the add-lag-and-lead branch 5 times, most recently from e2d40bc to 9f78341 Compare June 23, 2021 00:34

jimexist force-pushed the add-lag-and-lead branch from 9f78341 to 3a88c0d Compare June 26, 2021 02:51

github-actions bot added the ballista label Jun 26, 2021

jimexist force-pushed the add-lag-and-lead branch 4 times, most recently from ca475b4 to 1fae443 Compare June 28, 2021 23:54

jimexist marked this pull request as ready for review June 28, 2021 23:59

alamb reviewed Jun 29, 2021

View reviewed changes

add lead and lag

29fdc24

jimexist force-pushed the add-lag-and-lead branch from 1fae443 to 29fdc24 Compare July 1, 2021 00:21

jimexist added 2 commits July 1, 2021 08:21

add integration tests

4fce406

add partitioned window functions

20114ad

alamb approved these changes Jul 2, 2021

View reviewed changes

alamb merged commit c0de9bb into apache:master Jul 2, 2021

jimexist deleted the add-lag-and-lead branch July 3, 2021 03:15

houqp added the enhancement New feature or request label Jul 29, 2021

unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025

chore: Create initial release process scripts for official ASF source…

c87fabe

… release (apache#429)

Conversation

jimexist commented May 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Uh oh!

codecov-commenter commented May 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb commented Jun 1, 2021

Uh oh!

jimexist commented Jun 2, 2021

Uh oh!

jimexist commented Jun 13, 2021

Uh oh!

jimexist commented Jun 15, 2021

Uh oh!

alamb commented Jun 15, 2021

Uh oh!

jimexist commented Jun 15, 2021

Uh oh!

alamb commented Jun 15, 2021

Uh oh!

jimexist commented Jun 29, 2021

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Jun 29, 2021

Choose a reason for hiding this comment

Uh oh!

jimexist Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 1, 2021

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jimexist commented May 26, 2021 •

edited

Loading

codecov-commenter commented May 26, 2021 •

edited

Loading