build(deps): upgrade sqlparser to 0.47.0 by tisonkun · Pull Request #10392 · apache/datafusion

tisonkun · 2024-05-06T14:35:02Z

Upgrade sqlparser to ~~0.46.0~~ 0.47.0. This should handle a few of breaking changes:

Insert and Delete are now encapsulated in a struct.
JsonAccess refactor - Improve parsing of JSON accesses on Postgres and Snowflake datafusion-sqlparser-rs#1215
SQLFunction refactor - Consolidate representation of function calls, remove AggregateExpressionWithFilter, ArraySubquery, ListAgg and ArrayAgg datafusion-sqlparser-rs#1247
Window Function refactor - BigQuery: Support window clause using named window datafusion-sqlparser-rs#1237

This is working in progress.

Blocked at apache/datafusion-sqlparser-rs#1283.

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2024-05-07T15:17:39Z

@jmhain Thanks for your patch. I've applied it to this branch. Let us continue fixing the breaking changes here when we have time :D

tisonkun · 2024-05-08T10:04:54Z

Pushed a commit to handle other breaking changes. Now remains the JsonAccess and Agg func refactor. I'm still understanding how they are logically changed ...

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2024-05-14T02:21:36Z

datafusion/sql/src/expr/mod.rs

-                operator: JsonOperator::Colon,
-                right,
-            } => {
+            SQLExpr::JsonAccess { value, path } => {


@alamb @jmhain, do you have ideas on how we should modify this branch now?

Previously, we handled xxx::yyy in this branch. But now, JsonOperator::Colon no longer exists.

It seems to me that we should handle path.len() == 1 or path.len() == 2 and build a GetFieldAccess::ListRange as the original logic?

I pushed f29af9d to try it out.

The rest agg parts look much wider that I may hardly handle it by myself in a short time ..

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2024-05-14T02:41:33Z

Seems that we should merge AggregateExpressionWithFilter and ArrayAgg handle logic into sql_function_to_expr.

alamb · 2024-05-14T20:00:03Z

I am going to give this PR a look to see if I can help unstick it

datafusion/sql/src/expr/function.rs

datafusion/sql/src/expr/mod.rs

datafusion/sqllogictest/test_files/array.slt

alamb · 2024-05-14T20:45:37Z

There is still a failure which I haven't had a chance to look into:

cargo test --test sqllogictests -- array
...
Running "array_query.slt"
Running "array.slt"
External error: query failed: DataFusion error: SQL error: ParserError("Expected variant object key name, found: 2")
[SQL] select make_array(1, 2, 3)[1:2], make_array(1.0, 2.0, 3.0)[2:3], make_array('h', 'e', 'l', 'l', 'o')[2:4];
at test_files/array.slt:859

Error: Execution("1 failures")
error: test failed, to rerun pass `-p datafusion-sqllogictest --test sqllogictests`

Caused by:
  process didn't exit successfully: `/Users/andrewlamb/Software/datafusion/target/debug/deps/sqllogictests-b3a0a4f302be541c array` (exit status: 1)

datafusion/sql/src/expr/mod.rs

alamb · 2024-05-14T21:09:37Z

datafusion/sqllogictest/test_files/timestamps.slt

 SELECT '2023-03-12 02:00:00' AT TIME ZONE 'EDT';
+
+
+# Test current_time without parentheses


While updating the code I realized I almost broke this, so I added a new test (also propose adding it to main here: #10509)

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2024-06-01T07:43:50Z

Since we're on the sqlparser-rs main branch now, I'd consider this patch as ready to merge.

But of course it's better to have a new release to sqlparser-rs and we move to that version. Hopefully it contains what it has now without new breaking changes :P

tisonkun · 2024-06-01T08:20:24Z

Window specific test failed. I don't have a windows env to debug ...

test cli_quick_test::case_4_set_batch_size ... FAILED

failures:

---- cli_quick_test::case_4_set_batch_size stdout ----
thread 'cli_quick_test::case_4_set_batch_size' panicked at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\ops\function.rs:250:5:
Unexpected stdout, failed var == "[{\"name\":\"datafusion.execution.batch_size\",\"value\":\"1\"}]\n"
├── var: ""
└── var as str: 

command=`"D:\\a\\datafusion\\datafusion\\datafusion-cli\\target\\debug\\datafusion-cli.exe" "--command" "show datafusion.execution.batch_size" "--format" "json" "-q" "-b" "1"`
code=-1073741571
stdout=""
stderr=```

thread \'main\' has overflowed its stack

alamb · 2024-06-01T09:45:57Z

Since we're on the sqlparser-rs main branch now, I'd consider this patch as ready to merge.

But of course it's better to have a new release to sqlparser-rs and we move to that version. Hopefully it contains what it has now without new breaking changes :P

Thank you so much @tisonkun -- I agree

I will go make a sqlparser release now without any new changes

alamb

I just released version 0.47.0 of sqlparser to crates.io: apache/datafusion-sqlparser-rs#1251 / https://crates.io/crates/sqlparser/0.47.0

alamb · 2024-06-01T10:31:41Z

datafusion/expr/src/logical_plan/ddl.rs

-        }
-    }
+    /// RETURN or AS function body
+    pub function_body: Option<Expr>,


FYI @milenkovicm -- sqlparser changed how the body was represented so now this changes as well to a generic Expr

Thanks for letting me know, @alamb

I just wonder if this change will diverge sqlparser from Postgres?

I guess function like this

CREATE FUNCTION strlen(name TEXT) RETURNS int LANGUAGE plrust AS $$ Ok(Some(name.unwrap().len() as i32)) $$;

Would not be possible without wrapping body with string

It seems like we may need to add $$ string parsing support in sqlparser. I'll try and file a ticket later todday

IIRC we should previously support $$ function body. If it fails to parse now, it's a regression.

It used to work, apache/datafusion-sqlparser-rs#1185

I added a test and made this syntax work in this pr in 8446851

alamb · 2024-06-01T10:35:25Z

Thanks @tisonkun -- I updated this PR to use the newly released 0.47.0 version of sqlparser-rs and updated the description. I suppose now we just need to debug the windows test (I have an idea) and we'll be good to go

alamb · 2024-06-01T10:42:50Z

.github/workflows/rust.yml

        run: |
          export PATH=$PATH:$HOME/d/protoc/bin
          cargo test --lib --tests --bins --features avro,json,backtrace
+          # 8MB stack size otherwise the cli_integration test fails  


I added this to workaround the issue @tisonkun reports #10392 (comment)

If it passes CI I will file a ticket to track

Seems failure retains :<

👍 reverting the change and will try something else

I disabled the test and filed #10793 to track

This reverts commit b5743d5.

alamb · 2024-06-03T13:35:12Z

My plan for this PR is to disable the failing test on windows and file a ticket to investigate

alamb

Ok, I think this one is now ready !

I'll plan to leave it open for another day in case anyone wants to comment. Otherwise I think we are ready to merge.

Thank you for the epic work @tisonkun and @jmhain -- great team effort 🦾

tisonkun · 2024-06-04T22:30:22Z

@alamb Thanks for your review and follow-up! @jmhain Thanks for your help :D

alamb · 2024-06-06T12:33:15Z

I did a final merge up from main to make sure we don't have any logical conflicts and then I plan to merge this PR

* build(deps): upgrade sqlparser to 0.46.0 Signed-off-by: tison <wander4096@gmail.com> * function and cast fixups * catchup refactors Signed-off-by: tison <wander4096@gmail.com> * try migrate json expr Signed-off-by: tison <wander4096@gmail.com> * Update for changes in sqlparser * Update dependencies * handle zero argument form * fmt * fixup more Signed-off-by: tison <wander4096@gmail.com> * fixup more Signed-off-by: tison <wander4096@gmail.com> * try use jmhain's branch Signed-off-by: tison <wander4096@gmail.com> * fix compile FunctionArgumentClause exhausted Signed-off-by: tison <wander4096@gmail.com> * fix compile set multi vars Signed-off-by: tison <wander4096@gmail.com> * fix compile new string values Signed-off-by: tison <wander4096@gmail.com> * fix compile set multi vars Signed-off-by: tison <wander4096@gmail.com> * fix compile Subscript Signed-off-by: tison <wander4096@gmail.com> * cargo fmt Signed-off-by: tison <wander4096@gmail.com> * revert workaround on values Signed-off-by: tison <wander4096@gmail.com> * Rework field access * update lock * fix doc * try catchup new sqlparser version Signed-off-by: tison <wander4096@gmail.com> * fixup timezone expr Signed-off-by: tison <wander4096@gmail.com> * fixup params Signed-off-by: tison <wander4096@gmail.com> * lock Signed-off-by: tison <wander4096@gmail.com> * Update to sqlparser 0.47.0 * Update rust stack size on windows * Revert "Update rust stack size on windows" This reverts commit b5743d5. * Add test + support for `$$` function definition * Disable failing windows CI test * fmt * simplify test * fmt --------- Signed-off-by: tison <wander4096@gmail.com> Co-authored-by: Joey Hain <hain.joseph@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

build(deps): upgrade sqlparser to 0.46.0

7393c64

Signed-off-by: tison <wander4096@gmail.com>

github-actions bot added the sql SQL Planner label May 6, 2024

tisonkun mentioned this pull request May 6, 2024

Consolidate representation of function calls, remove AggregateExpressionWithFilter, ArraySubquery, ListAgg and ArrayAgg apache/datafusion-sqlparser-rs#1247

Merged

alamb mentioned this pull request May 6, 2024

chore(deps): update sqlparser requirement from 0.45.0 to 0.46.0 #10388

Closed

function and cast fixups

d64ac06

alamb mentioned this pull request May 7, 2024

Create fixed size list table with syntax <type name>[<length>] #10303

Closed

tisonkun force-pushed the upgrade-sqlparser branch from 2ed2daa to d9fd47f Compare May 8, 2024 10:04

catchup refactors

c112222

Signed-off-by: tison <wander4096@gmail.com>

tisonkun force-pushed the upgrade-sqlparser branch from d9fd47f to c112222 Compare May 8, 2024 10:10

duongcongtoai mentioned this pull request May 13, 2024

Support explicit type and name during table creation #10273

Merged

tisonkun commented May 14, 2024

View reviewed changes

try migrate json expr

f29af9d

Signed-off-by: tison <wander4096@gmail.com>

alamb added 2 commits May 14, 2024 16:34

Update for changes in sqlparser

1ac63fa

Merge remote-tracking branch 'apache/main' into upgrade-sqlparser

f973c0d

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label May 14, 2024

alamb reviewed May 14, 2024

View reviewed changes

datafusion/sql/src/expr/function.rs Show resolved Hide resolved

alamb reviewed May 14, 2024

View reviewed changes

datafusion/sql/src/expr/mod.rs Show resolved Hide resolved

alamb reviewed May 14, 2024

View reviewed changes

datafusion/sqllogictest/test_files/array.slt Outdated Show resolved Hide resolved

Update dependencies

d6da6fc

alamb added 2 commits May 14, 2024 17:04

handle zero argument form

f553243

fmt

f8eed01

alamb mentioned this pull request May 14, 2024

Minor: add a test for current_time (no args) #10509

Merged

alamb reviewed May 14, 2024

View reviewed changes

datafusion/sql/src/expr/mod.rs Show resolved Hide resolved

alamb reviewed May 14, 2024

View reviewed changes

Merge branch 'main' into upgrade-sqlparser

29fac1e

lock

e1d0c6b

Signed-off-by: tison <wander4096@gmail.com>

alamb reviewed Jun 1, 2024

View reviewed changes

alamb added 2 commits June 1, 2024 06:30

Update to sqlparser 0.47.0

685025a

Merge remote-tracking branch 'apache/main' into upgrade-sqlparser

ba24a7c

alamb changed the title ~~build(deps): upgrade sqlparser to 0.46.0~~ build(deps): upgrade sqlparser to 0.47.0 Jun 1, 2024

alamb reviewed Jun 1, 2024

View reviewed changes

Update rust stack size on windows

b5743d5

alamb reviewed Jun 1, 2024

View reviewed changes

Revert "Update rust stack size on windows"

d655ad6

This reverts commit b5743d5.

alamb mentioned this pull request Jun 3, 2024

DataFusion weekly project plan (Andrew Lamb) - June 3, 2024 #10779

Closed

8 tasks

alamb added 2 commits June 4, 2024 13:50

Merge remote-tracking branch 'apache/main' into upgrade-sqlparser

b8d84d5

Add test + support for $$ function definition

8446851

alamb mentioned this pull request Jun 4, 2024

cli_quick_test failing on windows (stack overflow) after sqlparser 0.47.0 upgrade #10793

Closed

alamb added 4 commits June 4, 2024 14:16

Disable failing windows CI test

72f61b0

fmt

089b571

simplify test

c3e475b

fmt

50a5a0f

alamb approved these changes Jun 4, 2024

View reviewed changes

Merge remote-tracking branch 'apache/main' into upgrade-sqlparser

2290a8e

alamb merged commit f054586 into apache:main Jun 6, 2024

tisonkun deleted the upgrade-sqlparser branch June 9, 2024 01:54

alamb mentioned this pull request Jun 11, 2024

DataFusion weekly project plan (Andrew Lamb) - June 10, 2024 #10869

Closed

7 tasks

		SELECT '2023-03-12 02:00:00' AT TIME ZONE 'EDT';


		# Test current_time without parentheses

Comments

Conversation

tisonkun commented May 6, 2024 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tisonkun commented May 7, 2024

Uh oh!

tisonkun commented May 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tisonkun commented May 14, 2024

Uh oh!

alamb commented May 14, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented May 14, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tisonkun commented Jun 1, 2024

Uh oh!

tisonkun commented Jun 1, 2024

Uh oh!

alamb commented Jun 1, 2024

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jun 1, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jun 3, 2024

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

tisonkun commented Jun 4, 2024

Uh oh!

alamb commented Jun 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tisonkun commented May 6, 2024 •

edited by alamb

Loading