Support modulus op by gangliao · Pull Request #510 · apache/datafusion

gangliao · 2021-06-05T14:53:58Z

Which issue does this PR close?

apache/arrow-rs#98
#99
apache/arrow-rs#317

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

no

gangliao · 2021-06-05T15:22:26Z

@alamb Is that possible to let datafusion use the latest code from arrow-rs?

alamb · 2021-06-05T15:43:25Z

Thank you for your contribution @gangliao !

@alamb Is that possible to let datafusion use the latest code from arrow-rs?

As of now, we are only using "released" versions of arrow.rs.

Your change in apache/arrow-rs#317 is currently slated to be released in 4.3.0 (eta ~ 10 or so days) at which time it will be available for datafusion to use without any additional action

Until 4.3.0 is released, for your branch you can point datafusion to a sha from the arrow github to make sure the code works (e.g. how it was done prior to #395)

Basically we should wait until 4.3.0 is released and then this PR can be merged into datafusion

gangliao · 2021-06-05T15:55:39Z

Got it! Thank you for the clarification.

Co-authored-by: Sathis Kumar <sathis.kumar@udemy.com>

) * refactor sort exec stream and combine batches * refactor async function

* 110 support group by positions * try resolve positions via array, not map * Add comment for i64 and simplify the pattern match * combine match and if condition, add more test cases * replace '0 as i64' with 0_i64

* Fix aggregate fn with invalid column * Fix error message * Fix error message * fix clippy * fix message and test * avoid unwrap in test_aggregation_with_bad_arguments * Update datafusion/tests/sql.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix test_aggregation_with_bad_arguments Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* Use pytest * Formatting * Update GHA conf * Remove TODO note * Format * Test requirements file * Update workflow file * Merge requirements file * Update workflow file

…planning (apache#501) * closing up type checks * add fmt

…pache#532) * use logical planner in ballista building * simplify statement * fix unit test * fix per comment

…he#541) * ShuffleReaderExec now supports multiple locations per partition * Remove TODO * avoid clone

* refactor hash aggregates * remove stale comments

* turn on clippy rule for needless borrow * do a format round * use warn not deny

…e#544)

* Cleanup RepartitionExec code * cleanup metric handling * Add elapsed_nanos

* support table alias in join clause * Update datafusion/src/sql/planner.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-06-14T15:40:02Z

Arrow 4.3.0 has been released so if you rebase this PR it will likely be ready to go

…nto mod

codecov-commenter · 2021-06-16T15:38:15Z

Codecov Report

Merging #510 (f63c8fe) into master (a9d04ca) will increase coverage by 0.03%.
The diff coverage is 72.05%.

❗ Current head f63c8fe differs from pull request most recent head 73902b6. Consider uploading reports for the commit 73902b6 to get more accurate results

@@            Coverage Diff             @@
##           master     #510      +/-   ##
==========================================
+ Coverage   76.07%   76.11%   +0.03%     
==========================================
  Files         155      156       +1     
  Lines       26544    27061     +517     
==========================================
+ Hits        20194    20597     +403     
- Misses       6350     6464     +114

Impacted Files	Coverage Δ
ballista/rust/client/src/context.rs	`0.00% <0.00%> (ø)`
...ta/rust/core/src/execution_plans/shuffle_reader.rs	`0.00% <0.00%> (ø)`
...sta/rust/core/src/serde/logical_plan/from_proto.rs	`35.04% <0.00%> (-0.88%)`	⬇️
...lista/rust/core/src/serde/logical_plan/to_proto.rs	`61.41% <0.00%> (-1.00%)`	⬇️
...ista/rust/core/src/serde/physical_plan/to_proto.rs	`49.38% <0.00%> (-0.93%)`	⬇️
ballista/rust/core/src/utils.rs	`25.53% <0.00%> (-2.06%)`	⬇️
ballista/rust/executor/src/flight_service.rs	`0.00% <ø> (ø)`
ballista/rust/scheduler/src/planner.rs	`66.91% <ø> (ø)`
ballista/rust/scheduler/src/state/mod.rs	`70.49% <0.00%> (ø)`
datafusion-cli/src/lib.rs	`0.00% <0.00%> (ø)`
... and 44 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9d04ca...73902b6. Read the comment docs.

alamb · 2021-06-16T16:38:52Z

I am not sure what has happened to this PR but the diff looks not right:

Where I would expect to only see the changes from 97fd9ff

gangliao · 2021-06-16T19:04:38Z

Yeah. Looks strange. I will resubmit a new PR tonight.

gangliao · 2021-06-17T03:02:32Z

#577

Support modulus op

97fd9ff

msathis and others added 11 commits June 5, 2021 23:34

Optimize cast function during planning stage (apache#513)

b84789a

Co-authored-by: Sathis Kumar <sathis.kumar@udemy.com>

Fix display of execution time (apache#514)

ee2b9ef

closing up type checks (apache#506)

767eeb0

refactor sort exec stream and combine batches (apache#515)

5773a03

closing up type checks (apache#518)

63accf8

Refactor window aggregation, simplify batch processing logic (apache#516

2f73e79

) * refactor sort exec stream and combine batches * refactor async function

110 support group by positions (apache#519)

e39f311

* 110 support group by positions * try resolve positions via array, not map * Add comment for i64 and simplify the pattern match * combine match and if condition, add more test cases * replace '0 as i64' with 0_i64

Define the unittests using pytest (apache#493)

42f908e

* Use pytest * Formatting * Update GHA conf * Remove TODO note * Format * Test requirements file * Update workflow file * Merge requirements file * Update workflow file

Add partition by constructs in window functions and modify logical …

d5bca0e

…planning (apache#501) * closing up type checks * add fmt

Return errors properly from RepartitionExec (apache#521)

8f84564

alamb added the datafusion label Jun 10, 2021

jimexist and others added 12 commits June 10, 2021 10:55

add boundary check (apache#530)

77775b7

remove redundant into_iter() calls (apache#527)

5c88450

use nightly nightly-2021-05-10 (apache#536)

3ef7f34

Make BallistaContext::collect streaming (apache#535)

63e3045

reuse datafusion physical planner in ballista building from protobuf (a…

ad70a1e

…pache#532) * use logical planner in ballista building * simplify statement * fix unit test * fix per comment

ShuffleReaderExec now supports multiple locations per partition (apac…

8f4078d

…he#541) * ShuffleReaderExec now supports multiple locations per partition * Remove TODO * avoid clone

Refactor hash aggregates's planner building code (apache#539)

519698a

* refactor hash aggregates * remove stale comments

turn on clippy rule for needless borrow (apache#545)

738f13b

* turn on clippy rule for needless borrow * do a format round * use warn not deny

apache#420: Support for not_eq predicate in pruning predicates (apach…

2568323

…e#544)

Cleanup Repartition Exec code (apache#538)

d382854

* Cleanup RepartitionExec code * cleanup metric handling * Add elapsed_nanos

support table alias in join clause (apache#547)

91af820

* support table alias in join clause * Update datafusion/src/sql/planner.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

reuse code for now function expr creation (apache#548)

fe810e9

Support modulus op

9c23de0

Merge branch 'mod' of https://github.com/DSLAM-UMD/arrow-datafusion i…

73902b6

…nto mod

gangliao closed this Jun 17, 2021

alamb mentioned this pull request Jun 6, 2022

Update sqlparser-rs to 0.18.0 #2705

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support modulus op#510

Support modulus op#510
gangliao wants to merge 26 commits intoapache:masterfrom
flock-lab:mod

gangliao commented Jun 5, 2021 •

edited

Loading

Uh oh!

gangliao commented Jun 5, 2021

Uh oh!

alamb commented Jun 5, 2021

Uh oh!

gangliao commented Jun 5, 2021

Uh oh!

alamb commented Jun 14, 2021

Uh oh!

codecov-commenter commented Jun 16, 2021

Uh oh!

alamb commented Jun 16, 2021

Uh oh!

gangliao commented Jun 16, 2021

Uh oh!

gangliao commented Jun 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Conversation

gangliao commented Jun 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

gangliao commented Jun 5, 2021

Uh oh!

alamb commented Jun 5, 2021

Uh oh!

gangliao commented Jun 5, 2021

Uh oh!

alamb commented Jun 14, 2021

Uh oh!

codecov-commenter commented Jun 16, 2021

Codecov Report

Uh oh!

alamb commented Jun 16, 2021

Uh oh!

gangliao commented Jun 16, 2021

Uh oh!

gangliao commented Jun 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

gangliao commented Jun 5, 2021 •

edited

Loading