Support Duration in min/max agg functions by svranesevic · Pull Request #15310 · apache/datafusion

svranesevic · 2025-03-19T13:34:31Z

Which issue does this PR close?

/

Rationale for this change

Support min/max aggregation of expressions with duration type.

What changes are included in this PR?

Extend min/max agg functions to support Duration.

Are these changes tested?

Yes, extended aggregate.slt with test cases.

Are there any user-facing changes?

Yes, min/max aggregation functions can be used with expressions of duration type.

alamb

Thank you very much @svranesevic -- this looks great to me

alamb · 2025-03-19T19:27:02Z

datafusion/sqllogictest/test_files/aggregate.slt

+  (arrow_cast(11, 'Duration(Second)'),arrow_cast(22, 'Duration(Millisecond)'), arrow_cast(33, 'Duration(Microsecond)'), arrow_cast(44, 'Duration(Nanosecond)'));
+
+query ????
+SELECT min(column1), min(column2), min(column3), min(column4) FROM d;


I verified these queries fail without this change:

> SELECT min(column1), min(column2), min(column3), min(column4) FROM d; Internal error: Min/Max accumulator not implemented for type Duration(Second). This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker >

alamb · 2025-03-19T19:45:21Z

datafusion/sqllogictest/test_files/aggregate.slt

+----
+0 days 0 hours 0 mins 11 secs 0 days 0 hours 0 mins 0.022 secs 0 days 0 hours 0 mins 0.000033 secs 0 days 0 hours 0 mins 0.000000044 secs
+
+# GROUP BY follows a different code path


I also added some tests when there is GROUP BY in the query as that takes a different code path

I actually double checked and there is a way to make this kind of query faster, which I filed as

Improve performance of min/max aggregates for Durations via GroupsAccumulator #15317

svranesevic · 2025-03-20T06:47:31Z

Thanks @alamb!
Feel free to merge whenever, it seems that I can not do so even after approval.

alamb · 2025-03-20T09:48:26Z

Will do -- thanks @svranesevic

I normally try and wait for abour 24 hours before merging non trivial approved PRs to give a chance for other contributors in different timezones to review

In this case, however, I think the PR is small enough and non contrversial that we can merge now

alamb · 2025-03-20T09:48:37Z

Thanks again @svranesevic

svranesevic · 2025-03-20T12:51:33Z

I normally try and wait for abour 24 hours before merging non trivial approved PRs to give a chance for other contributors in different timezones to review

Oh, I was not aware of that 👍 Pinged just to avoid dead-lock situation where we wait for the other one to merge 🙂

* Support Duration in min/max agg functions * Attempt to fix build * Attempt to fix build - Fix chrono version * Revert "Attempt to fix build - Fix chrono version" This reverts commit fd76fe6. * Revert "Attempt to fix build" This reverts commit 9114b86. --------- Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

* Support Duration in min/max agg functions * Attempt to fix build * Attempt to fix build - Fix chrono version * Revert "Attempt to fix build - Fix chrono version" This reverts commit fd76fe6. * Revert "Attempt to fix build" This reverts commit 9114b86. --------- Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

* Get working build * Add pool_size method to MemoryPool (#218) (#230) * Add pool_size method to MemoryPool * Fix * Fmt Co-authored-by: Daniël Heres <danielheres@gmail.com> * Respect `IGNORE NULLS` flag in `ARRAY_AGG` (#260) (apache#15544) v48 * Hook for doing distributed `CollectLeft` joins (#269) * Ignore writer shutdown error (#271) * ignore writer shutdown error * cargo check * Fix bug in `swap_hash_join` (#278) * Try and fix swap_hash_join * Only swap projections when join does not have projections * just backport upstream fix * remove println * Support Duration in min/max agg functions (#283) (apache#15310) v47 * Support Duration in min/max agg functions * Attempt to fix build * Attempt to fix build - Fix chrono version * Revert "Attempt to fix build - Fix chrono version" This reverts commit fd76fe6. * Revert "Attempt to fix build" This reverts commit 9114b86. --------- Co-authored-by: svranesevic <svranesevic@users.noreply.github.com> * Fix panics in array_union (#287) (apache#15149) v48 * Drop rust-toolchain * Fix panics in array_union * Fix the chrono * Backport `GroupsAccumulator` for Duration min/max agg (#288) (apache#15322) v47 * Fix array_sort for empty record batch (#290) (apache#15149) v48 * fix: rewrite fetch, skip of the Limit node in correct order (apache#14496) v46 * fix: rewrite fetch, skip of the Limit node in correct order * style: fix clippy * Support aliases in ConstEvaluator (apache#14734) (#281) v46 * Support aliases in ConstEvaluator (apache#14734) Not sure why they are not supported. It seems that if we're not careful, some transformations can introduce aliases nested inside other expressions. * Format Cargo.toml * Preserve the name of grouping sets in SimplifyExpressions (#282) (apache#14888) v46 Whenever we use `recompute_schema` or `with_exprs_and_inputs`, this ensures that we obtain the same schema. * Support Duration in min/max agg functions (#284) (apache#15310) v47 Co-authored-by: svranesevic <svranesevic@users.noreply.github.com> * fix case_column_or_null with nullable when conditions (apache#13886) v45 * fix case_column_or_null with nullable when conditions * improve sqllogictests for case_column_or_null --------- Co-authored-by: zhangli20 <zhangli20@kuaishou.com> * Fix casewhen (apache#14156) v45 * Cherry-pick topk limit pushdown fix (apache#14192) v45 * fix: FULL OUTER JOIN and LIMIT produces wrong results (apache#14338) v45 * fix: FULL OUTER JOIN and LIMIT produces wrong results * Fix minor slt testing * fix test (cherry picked from commit ecc5694) * Cherry-pick global limit fix (apache#14245) v45 * fix: Limits are not applied correctly (apache#14418) v46 * fix: Limits are not applied correctly * Add easy fix * Add fix * Add slt testing * Address comments * Disable grouping set in CSE * Fix spm + limit (apache#14569) v46 * prost 0.13 / fix parquet dep * Delete unreliable checks * Segfault in ByteGroupValueBuilder (#294) (apache#15968) v50 * test to demonstrate segfault in ByteGroupValueBuilder * check for offset overflow * clippy (cherry picked from commit 5bdaeaf) * Update arrow dependency to include rowid (#295) * Update arrow version * Feat: Add fetch to CoalescePartitionsExec (apache#14499) (#298) v46 * add fetch info to CoalescePartitionsExec * use Statistics with_fetch API on CoalescePartitionsExec * check limit_reached only if fetch is assigned Co-authored-by: mertak-synnada <mertak67+synaada@gmail.com> * Fix `CoalescePartitionsExec` proto serialization (apache#15824) (#299) v48 * add fetch to CoalescePartitionsExecNode * gen proto code * Add test * fix * fix build * Fix test build * remove comments Co-authored-by: 张林伟 <lewiszlw520@gmail.com> * Add JoinContext with JoinLeftData to TaskContext in HashJoinExec (#300) * Add JoinContext with JoinLeftData to TaskContext in HashJoinExec * Expose random state as const * re-export ahash::RandomState * JoinContext default impl * Add debug log when setting join left data * Update arrow version for not preserving dict_id (#303) * Use partial aggregation schema for spilling to avoid column mismatch in GroupedHashAggregateStream (apache#13995) (#302) v45 * Refactor spill handling in GroupedHashAggregateStream to use partial aggregate schema * Implement aggregate functions with spill handling in tests * Add tests for aggregate functions with and without spill handling * Move test related imports into mod test * Rename spill pool test functions for clarity and consistency * Refactor aggregate function imports to use fully qualified paths * Remove outdated comments regarding input batch schema for spilling in GroupedHashAggregateStream * Update aggregate test to use AVG instead of MAX * assert spill count * Refactor partial aggregate schema creation to use create_schema function * Refactor partial aggregation schema creation and remove redundant function * Remove unused import of Schema from arrow::datatypes in row_hash.rs * move spill pool testing for aggregate functions to physical-plan/src/aggregates * Use Arc::clone for schema references in aggregate functions (cherry picked from commit 81b50c4) Co-authored-by: kosiew <kosiew@gmail.com> * Update tag * Push limits past windows (#337) (apache#17347) v50 * Restore old method for DQE * feat(optimizer): Enable filter pushdown on window functions (apache#14026) v45 * Avoid Aliased Window Expr Enter Unreachable Code (apache#14109) v45 (cherry picked from commit fda500a) * Use `Expr::qualified_name()` and `Column::new()` to extract partition keys from window and aggregate operators (#355) (apache#17757) v51 * Update PR template to be relevant to our fork * Make limit pushdown work for SortPreservingMergeExec (apache#17893) (#361) * re-publicise functions DQE relies on * Handle columns in with_new_exprs with a Join (apache#15055) (#384) apache#15055 * handle columns in with_new_exprs with Join * test doesn't return result * take join from result * clippy * make test fallible * accept any pair of expression for new_on in with_new_exprs for Join * use with_capacity Co-authored-by: delamarch3 <68732277+delamarch3@users.noreply.github.com> --------- Co-authored-by: Georgi Krastev <georgi.krastev@coralogix.com> Co-authored-by: Daniël Heres <danielheres@gmail.com> Co-authored-by: Dan Harris <1327726+thinkharderdev@users.noreply.github.com> Co-authored-by: Faiaz Sanaulla <105630300+fsdvh@users.noreply.github.com> Co-authored-by: Sava Vranešević <20240220+svranesevic@users.noreply.github.com> Co-authored-by: svranesevic <svranesevic@users.noreply.github.com> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: Zhang Li <richselian@gmail.com> Co-authored-by: zhangli20 <zhangli20@kuaishou.com> Co-authored-by: Aleksey Kirilishin <54231417+avkirilishin@users.noreply.github.com> Co-authored-by: xudong.w <wxd963996380@gmail.com> Co-authored-by: Qi Zhu <821684824@qq.com> Co-authored-by: Martins Purins <martins.purins@coralogix.com> Co-authored-by: mertak-synnada <mertak67+synaada@gmail.com> Co-authored-by: 张林伟 <lewiszlw520@gmail.com> Co-authored-by: kosiew <kosiew@gmail.com> Co-authored-by: nuno-faria <nunofpfaria@gmail.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> Co-authored-by: Mason Hall <mason.hall@coralogix.com> Co-authored-by: delamarch3 <68732277+delamarch3@users.noreply.github.com>

* Support Duration in min/max agg functions * Attempt to fix build * Attempt to fix build - Fix chrono version * Revert "Attempt to fix build - Fix chrono version" This reverts commit fd76fe6. * Revert "Attempt to fix build" This reverts commit 9114b86. --------- Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

* Support Duration in min/max agg functions * Attempt to fix build * Attempt to fix build - Fix chrono version * Revert "Attempt to fix build - Fix chrono version" This reverts commit fd76fe6. * Revert "Attempt to fix build" This reverts commit 9114b86. --------- Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

Support min/max agg functions on values of duration type

42ec510

github-actions bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Mar 19, 2025

alamb added 2 commits March 19, 2025 15:32

Add tests for GROUP BY as well

6155986

Merge remote-tracking branch 'apache/main' into duration_min_max_agg

de4e066

alamb mentioned this pull request Mar 19, 2025

Improve performance of min/max aggregates for Durations via GroupsAccumulator #15317

Closed

alamb approved these changes Mar 19, 2025

View reviewed changes

alamb merged commit f5821b9 into apache:main Mar 20, 2025
27 checks passed

svranesevic deleted the duration_min_max_agg branch March 20, 2025 12:51

This was referenced Mar 27, 2025

Support Custom Function Registration with Catalog and Schema #15363

Open

Support AVG for durations #15458

Closed

jkosh44 mentioned this pull request Jun 6, 2025

[substrait] [sqllogictest] Unsupported cast type: Duration #16285

Closed

avantgardnerio pushed a commit to coralogix/arrow-datafusion that referenced this pull request Oct 1, 2025

Support Duration in min/max agg functions (#284) (apache#15310) v47

416af46

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

avantgardnerio pushed a commit to coralogix/arrow-datafusion that referenced this pull request Oct 15, 2025

Support Duration in min/max agg functions (#284) (apache#15310) v47

4748db3

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

avantgardnerio pushed a commit to coralogix/arrow-datafusion that referenced this pull request Dec 15, 2025

Support Duration in min/max agg functions (#284) (apache#15310) v47

8c0ba48

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

avantgardnerio pushed a commit to coralogix/arrow-datafusion that referenced this pull request Dec 15, 2025

Support Duration in min/max agg functions (#284) (apache#15310) v47

a8ef612

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

dispanser pushed a commit to coralogix/arrow-datafusion that referenced this pull request Jan 7, 2026

Support Duration in min/max agg functions (#284) (apache#15310) v47

4e1d15b

Co-authored-by: svranesevic <svranesevic@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Duration in min/max agg functions#15310

Support Duration in min/max agg functions#15310
alamb merged 3 commits intoapache:mainfrom
svranesevic:duration_min_max_agg

svranesevic commented Mar 19, 2025

Uh oh!

alamb left a comment

Uh oh!

alamb Mar 19, 2025

Uh oh!

alamb Mar 19, 2025

Uh oh!

svranesevic commented Mar 20, 2025

Uh oh!

alamb commented Mar 20, 2025

Uh oh!

Uh oh!

alamb commented Mar 20, 2025

Uh oh!

svranesevic commented Mar 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

svranesevic commented Mar 19, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

svranesevic commented Mar 20, 2025

Uh oh!

alamb commented Mar 20, 2025

Uh oh!

Uh oh!

alamb commented Mar 20, 2025

Uh oh!

svranesevic commented Mar 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants