Conversation
Co-authored-by: Yongting You <2010youy01@gmail.com>
Co-authored-by: Yongting You <2010youy01@gmail.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech>
Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
|
I also added a performance chart from https://alamb.github.io/datafusion-benchmarking/ (I am rechecking these numbers, etc) but it is looking good so far. A preview: |
|
😅 -- ok I think I filled out the major content parts of this post. It needs:
|
|
Thanks for your help with this @alamb, sorry I haven't done more. I've been deep trying to figure out a performance issue I've been having and haven't had the spare time to put into this |
I Totally understand -- we are all just trying to scrap together enough time to make progress. It is a struggle for sure! No worries. All your contributions are appreciated |
|
I took another pass -- I think this post is looking quite good The major thing that is missing now is actual performance numbers. I will work on that next, likely tomorrow or Friday |
|
Oh my, the ai loves commas, just make sure to leave out any m-dashes :) |
alamb
left a comment
There was a problem hiding this comment.
We are getting ready for the DataFusion 49 release and we have several shoutouts in the release notes.
I tagged the relevant people on this PR -- please help / suggest additional comments / content / contributors as you are able
|
|
||
| Here are some noteworthy optimizations added since DataFusion 48: | ||
|
|
||
| **Equivalence system upgrade:** The lower levels of the equivalence system, which is used to implement the |
| Information Passing for joins] (Issue | ||
| [#15037](https://github.com/apache/datafusion/issues/15037) PR | ||
| [#15770](https://github.com/apache/datafusion/pull/15770) by | ||
| [adriangb](https://github.com/adriangb)). |
|
|
||
| The last few months, between `46.0.0` and `49.0.0`, have seen our community grow: | ||
|
|
||
| 1. New PMC members and committers: [berkay], [xudong963] and [timsaucer] joined the PMC. |
There was a problem hiding this comment.
|
|
||
| (Issue [#6518](https://github.com/apache/datafusion/issues/6518), | ||
| [PR #14837](https://github.com/apache/datafusion/pull/14837) from | ||
| [goldmedal](https://github.com/goldmedal) 🏆) |
|
|
||
| [resulting PR]: https://github.com/apache/datafusion/pull/16398 | ||
| [blog post]: https://datafusion.apache.org/blog/2025/06/30/cancellation | ||
| [contribution upstream to tokio]: https://github.com/tokio-rs/tokio/pull/7405 |
| 2. Pushdown of filters and expressions (see "Dynamic Filters and TopK pushdown" section above) | ||
|
|
||
| [a long-requested feature]: https://github.com/apache/datafusion/issues/12644 | ||
| [timsaucer]: https://github.com/timsaucer |
| ); | ||
| ``` | ||
|
|
||
| ([Issue #15216](https://github.com/apache/datafusion/issues/15216), |
|
|
||
| (Issue [#11732](https://github.com/apache/datafusion/issues/11732), | ||
| PR [#13511](https://github.com/apache/datafusion/pull/13511), | ||
| by [Garamda](https://github.com/Garamda)) |
| [`REGEXP_INSTR` function]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#regexp-instr | ||
| ([Issue #13009](https://github.com/apache/datafusion/issues/13009), | ||
| [PR #15928](https://github.com/apache/datafusion/pull/15928) | ||
| by [nirnayroy](https://github.com/nirnayroy)) |
|
|
||
| (Issue [#16130](https://github.com/apache/datafusion/issues/16130), | ||
| PR [#16268](https://github.com/apache/datafusion/pull/16268) | ||
| by [ding-young](https://github.com/ding-young)) |
|
I plan to publish this tomorrow (Monday July 28 2025) |
|
🚀 |
Posting as a draft as per @alamb's request @ apache/datafusion#16235 (comment)
49.0.0(July 2025) datafusion#16235