Skip to content

chore: update datafusion family#6675

Merged
MichaelScofield merged 8 commits intomainfrom
chore/update-datafusion-family
Aug 15, 2025
Merged

chore: update datafusion family#6675
MichaelScofield merged 8 commits intomainfrom
chore/update-datafusion-family

Conversation

@MichaelScofield
Copy link
Copy Markdown
Collaborator

@MichaelScofield MichaelScofield commented Aug 7, 2025

pass clippy and fmt

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

most notably updates are:

arrow 54 -> 56
datafusion months ago to day 8.12
tonic 0.12 -> 0.13
sqlparser 0.54 -> 0.55

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

@github-actions github-actions Bot added size/XXL docs-not-required This change does not impact docs. labels Aug 7, 2025
@MichaelScofield MichaelScofield force-pushed the chore/update-datafusion-family branch from be39d93 to 3fb6a98 Compare August 7, 2025 06:25
@github-actions github-actions Bot added the ci label Aug 7, 2025
@MichaelScofield MichaelScofield force-pushed the chore/update-datafusion-family branch 19 times, most recently from 09c95e4 to b781652 Compare August 14, 2025 07:29
@MichaelScofield MichaelScofield marked this pull request as ready for review August 14, 2025 07:34
Comment thread .github/workflows/dependency-check.yml Outdated
Comment thread tests/cases/standalone/tql-explain-analyze/analyze.result
Comment thread src/common/datasource/src/file_format/orc.rs Outdated
Comment thread src/common/function/src/aggrs/aggr_wrapper.rs
Comment thread src/common/function/src/aggrs/aggr_wrapper.rs
Comment thread src/flow/src/batching_mode/task.rs
Comment thread src/mito2/src/sst/parquet/metadata.rs
Comment thread src/query/src/range_select/plan.rs Outdated
Comment thread src/query/src/window_sort.rs
Comment thread src/table/src/table/scan.rs
Comment thread tests/cases/distributed/explain/step_aggr_advance.result
Comment thread tests/cases/standalone/common/types/string/unicode.result
Comment thread tests/cases/distributed/explain/step_aggr_advance.result
MichaelScofield and others added 3 commits August 15, 2025 12:39
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
@MichaelScofield MichaelScofield force-pushed the chore/update-datafusion-family branch from 12d70e3 to 3f535cf Compare August 15, 2025 05:25
Signed-off-by: luofucong <luofc@foxmail.com>
@MichaelScofield MichaelScofield force-pushed the chore/update-datafusion-family branch from 3f535cf to 7c4a552 Compare August 15, 2025 05:27
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Comment thread .github/workflows/develop.yml Outdated
Comment thread src/query/src/dist_plan/analyzer.rs Outdated
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
@MichaelScofield MichaelScofield force-pushed the chore/update-datafusion-family branch from 1cd0b77 to d28a7e6 Compare August 15, 2025 09:32
Comment thread src/query/src/plan.rs
Copy link
Copy Markdown
Contributor

@discord9 discord9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I think we need a way to keep track of those sqlness result change and resolve them later

@MichaelScofield MichaelScofield added this pull request to the merge queue Aug 15, 2025
Merged via the queue into main with commit f9d2a89 Aug 15, 2025
80 checks passed
@MichaelScofield MichaelScofield deleted the chore/update-datafusion-family branch August 15, 2025 13:11
@killme2008 killme2008 requested a review from Copilot August 15, 2025 13:50
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the DataFusion family of dependencies, including Arrow (54→56), DataFusion (months ago→day 8.12), Tonic (0.12→0.13), and SQLParser (0.54→0.55). The changes primarily involve updating test expectations to align with new query execution plans and output formats introduced by the updated dependencies.

  • Updates dependency versions for Arrow, DataFusion, Tonic, and SQLParser
  • Adjusts test files to account for new query plan structures and output formats
  • Fixes compatibility issues with updated DataFusion query optimizer and execution engine

Reviewed Changes

Copilot reviewed 196 out of 198 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/cases/standalone/tql-explain-analyze/analyze.sql Adds SQLNESS REPLACE directive for Hash patterns
tests/cases/standalone/tql-explain-analyze/analyze.result Updates query execution plans to include CooperativeExec nodes and additional execution operators
tests/cases/standalone/optimizer/*.result Updates query plans to reflect new DataFusion execution structure with CooperativeExec wrapping
tests/cases/standalone/common/types/string/unicode.sql Adds CAST workaround for Utf8View compatibility issue
tests/cases/standalone/common/types/timestamp/timestamp.result Updates error message format to include quotes around error details
tests/cases/standalone/common/tql/range.sql Quotes 'service' column name as reserved keyword
tests/cases/standalone/optimizer/filter_push_down.sql Removes TODO comments and aligns results with PostgreSQL behavior
tests/cases/distributed/*.result Similar execution plan updates for distributed test cases

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

Comment thread tests/cases/standalone/common/types/string/unicode.sql
Comment thread tests/cases/standalone/common/tql/range.sql
@sunng87
Copy link
Copy Markdown
Member

sunng87 commented Aug 18, 2025

Can we confirm if there is any user space breaking change?

@MichaelScofield
Copy link
Copy Markdown
Collaborator Author

@sunng87 I can think of two minor ones:

  1. substr function needs to be "cast"ed: SELECT substr(s, 1, 1) FROM t => SELECT cast(substr(s, 1, 1) as string) FROM t
  2. "service" is now a reserved word in sqlparser, must be quoted in SQL

codephage2020 pushed a commit to codephage2020/greptimedb that referenced this pull request Aug 25, 2025
* chore: update datafusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* use official otel-arrow-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* rebase

Signed-off-by: luofucong <luofc@foxmail.com>

* use the official orc-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* remove the empty lines

Signed-off-by: luofucong <luofc@foxmail.com>

* try following PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: codephage2020 <tingwangyan2020@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci docs-not-required This change does not impact docs. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants