[CELEBORN-2218] Bump lz4-java version from 1.8.0 to 1.10.4 to resolve CVE‐2025‐12183 and CVE-2025-66566

[SteNicholas]

What changes were proposed in this pull request?

• Bump lz4-java version from 1.8.0 to 1.10.4 to resolve CVE‐2025‐12183 and CVE-2025-66566.
• Lz4Decompressor follows the suggestion to move from fastDecompressor to safeDecompressor to mitigate the performance.

Backport:

• [SPARK-54597][BUILD] Upgrade lz4-java to 1.10.0 spark#53327
• [SPARK-54611][BUILD] Upgrade lz4-java to 1.10.1 spark#53347
• [SPARK-55189][BUILD] Upgrade lz4-java to 1.10.3 spark#53971
• [SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor to mitigate perf regression spark#53454
• [SPARK-55803][BUILD] Bump lz4-java 1.10.4 to bring back performance spark#54585

Why are the changes needed?

• CVE‐2025‐12183: Various lz4-java compression and decompression implementations do not guard against out-of-bounds memory access. Untrusted input may lead to denial of service and information disclosure. Vulnerable Maven coordinates: org.lz4:lz4-java up to and including 1.8.0.

• CVE-2025-66566: Insufficient clearing of the output buffer in Java-based decompressor implementations in lz4-java 1.10.0 and earlier allows remote attackers to read previous buffer contents via crafted compressed input. In applications where the output buffer is reused without being cleared, this may lead to disclosure of sensitive data. JNI-based implementations are not affected.

Therefore, lz4-java version should upgrade to 1.10.4.

Does this PR resolve a correctness bug?

No.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI.

[yawkat]

I recommend you stick with fastestInstance. It is secure as long as you are on 1.8.1+. It will be much slower than in previous versions, but that can be mitigated by moving to fastestInstance.safeDecompressor like you did in this PR, which is much faster.

[SteNicholas]

@yawkat, thanks for review. I have sticked with fastestInstance for the workaround With the 1.8.1 patch applied, these workarounds are not necessary. It is still recommended to move from fastDecompressor to safeDecompressor to mitigate the performance impact of the fix, however. PTAL.

[yawkat]

Also fyi there was another cve (CVE-2025-66566) that needs a newer version

[Marcono1234]

CVE-2025-66566 affects versions less than or equal to 1.10.0. You should upgrade to 1.10.1.

[SteNicholas]

Ping @pan3793, @yawkat, @Marcono1234.

[pan3793]

lz4 is famous for its ultra-fast speed, the upgrade is not free, my test shows it has perf impact - apache/spark#53453

I understand that security takes precedence over performance, so I'm fine with this change.

for the suggestion of 'moving to fastestInstance.safeDecompressor', I think we can NOT do that blindly - Celeborn Spark/Flink clients use the lz4-java libs provided by the engine libs, since we support a wide range of Spark/Flink versions, it's possible that the engine still ships old lz4-java jar, we may need to dynamiclly check and bind the fastDecompressor or safeDecompressor based on runtime version of lz4-java

[yawkat]

@pan3793 safeDecompressor should work just fine on old versions, and even on those old versions, it should be slightly faster than fastDecompressor. In fact, using safeDecompressor gets rid of most (but not all) of the security impact of the CVEs on old versions.

[pan3793]

@SteNicholas Code change looks fine, but let's wait for a while to collect feedback from other reviewers, about the performance drop.

[SteNicholas]

@yawkat. could you please take a look at performance drop of safeDecompressor which refers to the benchmark report of LZ4TPCDSDataBenchmark-jdk17-results.txt?

[yawkat]

@SteNicholas I don't have time for deep benchmarking, but I just threw an AI agent at it, and it figured out that some build flag changes that distros do can improve performance by 10%. Could you test with the 1.10.4 I just released?

[pan3793]

@yawkat thanks! it indeed solves the performance regression. our benchmark shows 1.10.4 is much faster than 1.10.3, and even faster than 1.8.0! and I got a similar result in Spark apache/spark#54585

[RexXiong]

@yawkat thanks! it indeed solves the performance regression. our benchmark shows 1.10.4 is much faster than 1.10.3, and even faster than 1.8.0! and I got a similar result in Spark apache/spark#54585

Sounds Great, I think we can keep LZ4 as default.

[SteNicholas]

Thanks for all. Merged to main(v0.7.0) and branch-0.6(v0.6.3).

[yawkat]

Given that zstd is from the same authors as lz4 but newer, it may still be a good idea to move to zstd as the default long-term.