[SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.72 by LuciferYang · Pull Request #34881 · apache/spark

LuciferYang · 2021-12-13T11:38:37Z

What changes were proposed in this pull request?

This PR upgrades Netty from 4.1.68 to 4.1.71.

All the changes from 4.1.68 to 4.1.71 are as follows:

4.1.69: https://netty.io/news/2021/10/11/4-1-69-Final.html
4.1.70: https://netty.io/news/2021/10/11/4-1-70-Final.html
4.1.71: https://netty.io/news/2021/12/09/4-1-71-Final.html
4.1.72: https://netty.io/news/2021/12/13/4-1-72-Final.html

And after Netty#11732, netty-all replaces a fat jar with separate modules, so this pr manually excludes some Netty sub-modules that Spark does not rely on.

After this pr, Spark will rely on the following Netty sub modules:

netty-all/4.1.72.Final//netty-all-4.1.72.Final.jar
netty-buffer/4.1.72.Final//netty-buffer-4.1.72.Final.jar
netty-codec/4.1.72.Final//netty-codec-4.1.72.Final.jar
netty-common/4.1.72.Final//netty-common-4.1.72.Final.jar
netty-handler/4.1.72.Final//netty-handler-4.1.72.Final.jar
netty-resolver/4.1.72.Final//netty-resolver-4.1.72.Final.jar
netty-tcnative-classes/2.0.46.Final//netty-tcnative-classes-2.0.46.Final.jar
netty-transport-classes-epoll/4.1.72.Final//netty-transport-classes-epoll-4.1.72.Final.jar
netty-transport-classes-kqueue/4.1.72.Final//netty-transport-classes-kqueue-4.1.72.Final.jar
netty-transport-native-epoll/4.1.72.Final/linux-aarch_64/netty-transport-native-epoll-4.1.72.Final-linux-aarch_64.jar
netty-transport-native-epoll/4.1.72.Final/linux-x86_64/netty-transport-native-epoll-4.1.72.Final-linux-x86_64.jar
netty-transport-native-kqueue/4.1.72.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.72.Final-osx-aarch_64.jar
netty-transport-native-kqueue/4.1.72.Final/osx-x86_64/netty-transport-native-kqueue-4.1.72.Final-osx-x86_64.jar
netty-transport-native-unix-common/4.1.72.Final//netty-transport-native-unix-common-4.1.72.Final.jar
netty-transport/4.1.72.Final//netty-transport-4.1.72.Final.jar

and the following Netty sub modules are not dependent on Spark :

netty-codec-dns/4.1.72.Final//netty-codec-dns-4.1.72.Final.jar
netty-codec-haproxy/4.1.72.Final//netty-codec-haproxy-4.1.72.Final.jar
netty-codec-http/4.1.72.Final//netty-codec-http-4.1.72.Final.jar
netty-codec-http2/4.1.72.Final//netty-codec-http2-4.1.72.Final.jar
netty-codec-memcache/4.1.72.Final//netty-codec-memcache-4.1.72.Final.jar
netty-codec-mqtt/4.1.72.Final//netty-codec-mqtt-4.1.72.Final.jar
netty-codec-redis/4.1.72.Final//netty-codec-redis-4.1.72.Final.jar
netty-codec-smtp/4.1.72.Final//netty-codec-smtp-4.1.72.Final.jar
netty-codec-socks/4.1.72.Final//netty-codec-socks-4.1.72.Final.jar
netty-codec-stomp/4.1.72.Final//netty-codec-stomp-4.1.72.Final.jar
netty-codec-xml/4.1.72.Final//netty-codec-xml-4.1.72.Final.jar
netty-handler-proxy/4.1.72.Final//netty-handler-proxy-4.1.72.Final.jar
netty-resolver-dns-classes-macos/4.1.72.Final//netty-resolver-dns-classes-macos-4.1.72.Final.jar
netty-resolver-dns-native-macos/4.1.72.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.72.Final-osx-aarch_64.jar
netty-resolver-dns-native-macos/4.1.72.Final/osx-x86_64/netty-resolver-dns-native-macos-4.1.72.Final-osx-x86_64.jar
netty-resolver-dns/4.1.72.Final//netty-resolver-dns-4.1.72.Final.jar
netty-transport-rxtx/4.1.72.Final//netty-transport-rxtx-4.1.72.Final.jar
netty-transport-sctp/4.1.72.Final//netty-transport-sctp-4.1.72.Final.jar
netty-transport-udt/4.1.72.Final//netty-transport-udt-4.1.72.Final.jar

Why are the changes needed?

Avoid some potential bugs by upgrading Netty to 4.1.72

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass the Jenkins or GitHub Action

SparkQA · 2021-12-13T12:16:33Z

Test build #146135 has finished for PR 34881 at commit d8f6160.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-13T13:02:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50609/

SparkQA · 2021-12-13T13:09:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50612/

SparkQA · 2021-12-13T13:49:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50609/

SparkQA · 2021-12-13T14:09:44Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50612/

SparkQA · 2021-12-13T15:18:23Z

Test build #146139 has finished for PR 34881 at commit 3549a4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

attilapiros · 2021-12-14T14:22:38Z

May I ask you to run a maven build as well? Just to be in the safe side... I am not 100% sure but as you changed the exclusion in SparkBuild.scala but not in a pom.xml we might find something here.

srowen · 2021-12-15T00:11:50Z

 minlog/1.3.0//minlog-1.3.0.jar
-netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
+netty-all/4.1.71.Final//netty-all-4.1.71.Final.jar
+netty-buffer/4.1.71.Final//netty-buffer-4.1.71.Final.jar


Whoa, this doesn't look right. netty-all is (probably?) a packaging of all these subcomponents.
Something changes in how it's declared. We might need to depend just on pieces of netty 4.1.71?

And agree, I think you need Maven exclusions to match SBT

Whoa, this doesn't look right. netty-all is (probably?) a packaging of all these subcomponents.
Something changes in how it's declared. We might need to depend just on pieces of netty 4.1.71?

The change occurred after version 4.1.69, Netty#11732 netty-all should not re-package jars.

@srowen Do you mean that we need to introduce only the sub modules we really use? I think it's OK

But we originally introduced all Netty modules before 4.1.68 through use netty-all-4.1.68.Final.jar. If we introduce subsets instead, will it cause some user compatibility problems?

Let me investigate if there is a better way to upgrade first

Or we can give up the upgrade of this version. I found that there are many users in the Netty community are giving feedback on this problem and hope to provide netty-all-xx.Final.jar again for upgrade.

@srowen I read the discussion of Netty#11732 netty-all should not re-package jars and the recent issues&prs in Netty, although many people have dissenting opinion to the current usage way, it seems that netty-all uber jar has not been restored until 4.1.72, and no one has plans to do this. Do we need to wait for a new version that provides uber jar before upgrading?

We can take our time here and should not be in hurry because of the log4j fix went in to netty. As I see in netty log4j-core was used only in test:

$ rg -A2 log4j-core common/pom.xml 78: <artifactId>log4j-core</artifactId> 79- <scope>test</scope> 80- </dependency> pom.xml 706: <artifactId>log4j-core</artifactId> 707- <version>${log4j2.version}</version> 708- <scope>test</scope>

Thanks @attilapiros

LuciferYang · 2021-12-15T03:36:43Z

May I ask you to run a maven build as well? Just to be in the safe side... I am not 100% sure but as you changed the exclusion in SparkBuild.scala but not in a pom.xml we might find something here.

And agree, I think you need Maven exclusions to match SBT

@attilapiros @srowen

In SPARK-35134 (#32230), there is a netty version conflict issue during sbt test process, the phenomenon that triggers the bug is that even if we use netty-4.1.51-all.jar, there will be copying deps/jars/netty-handler-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars and copying deps/jars/netty-transport-native-epoll-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars behavior during sbt testing, this makes classpath conflicts possible

So I add 2 new ExclusionRule in SPARK-35134 to fix this sbt test bug.

ExclusionRule("io.netty", "netty-handler"),
ExclusionRule("io.netty", "netty-transport-native-epoll")

We didn't find this problem during Maven testing, so I didn't make corresponding change for Maven In SPARK-35134.

Then after Netty-4.1.69 (Netty#11732 netty-all should not re-package jars) , netty-all replaces a fat jar with separate modules, keeping ExclusionRule("io.netty", "netty-handler") and ExclusionRule("io.netty", "netty-transport-native-epoll") will result missing netty-transport-native-epoll-4.1.71.Final.jar and netty-handler-4.1.71.Final.jar problem during sbt test because netty-all-4.1.71.jar no longer contains them.

Therefore, after upgrading the Netty to 4.1.71, the pom.xml has nothing to change

SparkQA · 2021-12-15T06:05:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50688/

SparkQA · 2021-12-15T06:53:23Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50688/

SparkQA · 2021-12-15T08:16:21Z

Test build #146215 has finished for PR 34881 at commit 3836e44.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2021-12-15T15:40:12Z

 metrics-jvm/4.2.2//metrics-jvm-4.2.2.jar
 minlog/1.3.0//minlog-1.3.0.jar
-netty-all/4.1.68.Final//netty-all-4.1.68.Final.jar
+netty-all/4.1.72.Final//netty-all-4.1.72.Final.jar


I see, so netty-all no longer has the contents of all the JARs below, so it's not redundant? that is OK if so. We could even exclude probably lots of these (SMTP? SOCKS?). Not necessary but might be nice if they're largeish and unused.

BTW the description says this updates log4j, but, log4j 2 is not in the build before or after this

OK, let me try to clean up unnecessary Netty submodules

BTW the description says this updates log4j, but, log4j 2 is not in the build before or after this

I have deleted the relevant description. As @attilapiros said, the upgrade of log4j2 in Netty does not affect Spark

@srowen @attilapiros e34a2c9 I clean up 19 Netty sub-modules that seem not to be used by spark, waiting for CI to test. I will also test these locally with Maven.

The cleaned sub modules include:

netty-codec-dns/4.1.72.Final//netty-codec-dns-4.1.72.Final.jar netty-codec-haproxy/4.1.72.Final//netty-codec-haproxy-4.1.72.Final.jar netty-codec-http/4.1.72.Final//netty-codec-http-4.1.72.Final.jar netty-codec-http2/4.1.72.Final//netty-codec-http2-4.1.72.Final.jar netty-codec-memcache/4.1.72.Final//netty-codec-memcache-4.1.72.Final.jar netty-codec-mqtt/4.1.72.Final//netty-codec-mqtt-4.1.72.Final.jar netty-codec-redis/4.1.72.Final//netty-codec-redis-4.1.72.Final.jar netty-codec-smtp/4.1.72.Final//netty-codec-smtp-4.1.72.Final.jar netty-codec-socks/4.1.72.Final//netty-codec-socks-4.1.72.Final.jar netty-codec-stomp/4.1.72.Final//netty-codec-stomp-4.1.72.Final.jar netty-codec-xml/4.1.72.Final//netty-codec-xml-4.1.72.Final.jar netty-handler-proxy/4.1.72.Final//netty-handler-proxy-4.1.72.Final.jar netty-resolver-dns-classes-macos/4.1.72.Final//netty-resolver-dns-classes-macos-4.1.72.Final.jar netty-resolver-dns-native-macos/4.1.72.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.72.Final-osx-aarch_64.jar netty-resolver-dns-native-macos/4.1.72.Final/osx-x86_64/netty-resolver-dns-native-macos-4.1.72.Final-osx-x86_64.jar netty-resolver-dns/4.1.72.Final//netty-resolver-dns-4.1.72.Final.jar netty-transport-rxtx/4.1.72.Final//netty-transport-rxtx-4.1.72.Final.jar netty-transport-sctp/4.1.72.Final//netty-transport-sctp-4.1.72.Final.jar netty-transport-udt/4.1.72.Final//netty-transport-udt-4.1.72.Final.jar

Sub modules that may be used include:

netty-all/4.1.72.Final//netty-all-4.1.72.Final.jar netty-buffer/4.1.72.Final//netty-buffer-4.1.72.Final.jar netty-codec/4.1.72.Final//netty-codec-4.1.72.Final.jar netty-common/4.1.72.Final//netty-common-4.1.72.Final.jar netty-handler/4.1.72.Final//netty-handler-4.1.72.Final.jar netty-resolver/4.1.72.Final//netty-resolver-4.1.72.Final.jar netty-tcnative-classes/2.0.46.Final//netty-tcnative-classes-2.0.46.Final.jar netty-transport-classes-epoll/4.1.72.Final//netty-transport-classes-epoll-4.1.72.Final.jar netty-transport-classes-kqueue/4.1.72.Final//netty-transport-classes-kqueue-4.1.72.Final.jar netty-transport-native-epoll/4.1.72.Final/linux-aarch_64/netty-transport-native-epoll-4.1.72.Final-linux-aarch_64.jar netty-transport-native-epoll/4.1.72.Final/linux-x86_64/netty-transport-native-epoll-4.1.72.Final-linux-x86_64.jar netty-transport-native-kqueue/4.1.72.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.72.Final-osx-aarch_64.jar netty-transport-native-kqueue/4.1.72.Final/osx-x86_64/netty-transport-native-kqueue-4.1.72.Final-osx-x86_64.jar netty-transport-native-unix-common/4.1.72.Final//netty-transport-native-unix-common-4.1.72.Final.jar netty-transport/4.1.72.Final//netty-transport-4.1.72.Final.jar

SparkQA · 2021-12-16T05:22:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50733/

SparkQA · 2021-12-16T06:19:30Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50733/

SparkQA · 2021-12-16T07:13:47Z

Test build #146259 has finished for PR 34881 at commit e34a2c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

attilapiros

LGTM

SparkQA · 2021-12-21T06:34:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50894/

SparkQA · 2021-12-21T07:33:13Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50894/

SparkQA · 2021-12-21T08:24:23Z

Test build #146419 has finished for PR 34881 at commit 8994d5e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class RocksDB implements KVStore
public static class TypeAliases
class RocksDBIterator<T> implements KVStoreIterator<T>
class RocksDBTypeInfo
class Index
trait SwitchToDiskStoreListener
(defaultdict(<class 'list'>,
class ExecutorRollPlugin extends SparkPlugin
class ExecutorRollDriverPlugin extends DriverPlugin with Logging
case class RegrCount(left: Expression, right: Expression)
public class LogDivertAppender extends AbstractWriterAppender<WriterManager>

srowen · 2021-12-23T05:25:19Z

Merged to master

### What changes were proposed in this pull request? Bump Netty version from 4.1.118.Final to 4.2.10.Final, which follows the official community migration guide: [Netty-4.2-Migration-Guide](https://github.com/netty/netty/wiki/Netty-4.2-Migration-Guide). ### Why are the changes needed? The Netty 4.2.10.Final version has been released, which netty version is 4.1.118.Final at present. Backport: - apache/spark#34881 - apache/spark#52552 - apache/spark#53499 ### Does this PR resolve a correctness bug? No. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI. Closes #3596 from SteNicholas/CELEBORN-2258. Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: SteNicholas <programgeek@163.com>

upgrade netty to 4.1.71

d8f6160

LuciferYang changed the title ~~[SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71~~ [WIP][SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71 Dec 13, 2021

github-actions Bot added the BUILD label Dec 13, 2021

LuciferYang changed the title ~~[WIP][SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71~~ [SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71 Dec 13, 2021

LuciferYang marked this pull request as draft December 13, 2021 11:50

LuciferYang marked this pull request as ready for review December 13, 2021 11:53

LuciferYang marked this pull request as draft December 13, 2021 12:18

LuciferYang changed the title ~~[SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71~~ [WIP][SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71 Dec 13, 2021

change SparkBuild.scala

3549a4e

LuciferYang marked this pull request as ready for review December 14, 2021 02:38

LuciferYang changed the title ~~[WIP][SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71~~ [SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71 Dec 14, 2021

srowen reviewed Dec 15, 2021

View reviewed changes

LuciferYang changed the title ~~[SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.71~~ [SPARK-37628][BUILD] Upgrade Netty from 4.1.68 to 4.1.72 Dec 15, 2021

upgrade to 4.1.72

3836e44

srowen reviewed Dec 15, 2021

View reviewed changes

remove some netty submodules

e34a2c9

srowen approved these changes Dec 16, 2021

View reviewed changes

attilapiros approved these changes Dec 16, 2021

View reviewed changes

Merge branch 'upmaster' into upgrade-netty-4171

8994d5e

srowen closed this in 6511dbb Dec 23, 2021

LuciferYang deleted the upgrade-netty-4171 branch October 22, 2023 07:43

This was referenced Feb 4, 2026

[CELEBORN-2239] Support Spark 4.1 apache/celeborn#3571

Closed

[CELEBORN-2258] Bump Netty version from 4.1.118.Final to 4.2.10.Final apache/celeborn#3596

Closed

Conversation

LuciferYang commented Dec 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

SparkQA commented Dec 13, 2021

Uh oh!

attilapiros commented Dec 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen Dec 15, 2021

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 15, 2021

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 15, 2021

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 15, 2021

Choose a reason for hiding this comment

Uh oh!

attilapiros Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 15, 2021

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

SparkQA commented Dec 15, 2021

Uh oh!

srowen Dec 15, 2021

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 16, 2021

Uh oh!

SparkQA commented Dec 16, 2021

Uh oh!

SparkQA commented Dec 16, 2021

Uh oh!

attilapiros left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

SparkQA commented Dec 21, 2021

LuciferYang commented Dec 13, 2021 •

edited

Loading

attilapiros commented Dec 14, 2021 •

edited

Loading

LuciferYang Dec 15, 2021 •

edited

Loading

attilapiros Dec 15, 2021 •

edited

Loading

LuciferYang commented Dec 15, 2021 •

edited

Loading

LuciferYang Dec 15, 2021 •

edited

Loading

LuciferYang Dec 16, 2021 •

edited

Loading