Update to arrow/parquet 53.0.0, tonic, prost, object_store, pyo3#12032
Merged
alamb merged 50 commits intoapache:mainfrom Sep 5, 2024
Merged
Update to arrow/parquet 53.0.0, tonic, prost, object_store, pyo3#12032alamb merged 50 commits intoapache:mainfrom
arrow/parquet 53.0.0, tonic, prost, object_store, pyo3#12032alamb merged 50 commits intoapache:mainfrom
Conversation
Co-authored-by: tison <wander4096@gmail.com>
… alamb/update_prost
alamb
commented
Aug 16, 2024
| Some(1), | ||
| Some(10), | ||
| None, | ||
| Some(0), |
Contributor
Author
There was a problem hiding this comment.
Due to apache/arrow-rs#6216 (where null counts is now Option)
arrow/parquet `53.0.0arrow/parquet 53.0.0, tonic, prost, etc
alamb
commented
Aug 16, 2024
alamb
commented
Sep 3, 2024
| arrow = { version = "53.0.0" } | ||
| async-trait = "0.1.73" | ||
| aws-config = "1.5.5" | ||
| # begin pin aws-sdk crates otherwise CI MSRV check fails |
Contributor
Author
There was a problem hiding this comment.
Here is an example failure https://github.com/apache/datafusion/actions/runs/10690110122/job/29633769353?pr=12032
The same thing actually happens on main if you do cargo update in the datafusion-cli directory, just no one had hit that yet
Contributor
There was a problem hiding this comment.
That is an unfortunate choice by aws.
This was referenced Sep 4, 2024
Contributor
Author
|
This PR is now ready for review 🎉 (finally, 53.0.0 is released) |
comphead
reviewed
Sep 4, 2024
|
|
||
| let mut builder = StringViewBuilder::with_capacity(string_view_array.len()) | ||
| .with_block_size(1024 * 1024 * 2); | ||
| let mut builder = StringViewBuilder::with_capacity(string_view_array.len()); |
Contributor
There was a problem hiding this comment.
does that mean the block size is not used in the builder for this case?
Contributor
There was a problem hiding this comment.
yes, we don't need it after this pr is merged: apache/arrow-rs#6136
I ran a local benchmark and show no perf diff
comphead
approved these changes
Sep 4, 2024
Contributor
Author
|
Thanks for the review @comphead and all the help @XiangpengHao -- let's get this merged! |
This was referenced Sep 5, 2024
This was referenced Sep 9, 2024
5 tasks
wiedld
pushed a commit
to influxdata/arrow-datafusion
that referenced
this pull request
Oct 16, 2024
…`, `pyo3` (apache#12032) * Update prost, prost-derive, pbjson * udpate more * Update datafusion/substrait/Cargo.toml Co-authored-by: tison <wander4096@gmail.com> * Update vendored code * revert upgrade in datafusion-examples until arrow-flight is updated * Pin to pre-release arrow-rs * update pyo3 * Update to use new arrow apis * update for new api * Update tonic in examples * update prost * update datafusion-cli/cargo * update test output * update * updates * updates * update math * update more * fix scalar tests * Port statistics to use new API * factor into a function * update generated files * Update test * add new test * update tests * tapelo format * Update other tests * Update datafusion pin * Update for API change * Update to arrow 53.0.0 sha * Update cli deps * update cargo.lock * Update expected output * Remove patch * update datafusion-cli cargo * Pin some aws sdks whose update caused CI failures --------- Co-authored-by: tison <wander4096@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft as this is waiting on the actual arrow 53.0.0 release: apache/arrow-rs#6016Which issue does this PR close?
N/A
Rationale for this change
We need to update the DataFusion ecosystem
What changes are included in this PR?
arrow/parquet`53.0.0prost,prost-derive,pbjson,tonicecosystem #11372object_storeto 0.11.0 (TBD): Release object store0.11.0(breaking API) around Aug 15 2024 arrow-rs-object-store#59Are these changes tested?
Yes by CI
Are there any user-facing changes?