Commit dc46b14
[SPARK-53064][CORE] Rewrite MDC LogKey in Java
### What changes were proposed in this pull request?
This PR proposes to rewrite a few classes used by the Structured Logging API from Scala to Java.
### Why are the changes needed?
Previously (before 3.5), modules under `common` were pure Java, and were easy to embed into other services, for example, the YARN External Shuffle Service will be run as a plugin of the YARN Resource Manager daemon process. With recent years' changes, some pure Java modules also require `scala-library` to be present at runtime, i.e., SPARK-52942 reports that YARN ESS causes YARN RM to fail to start due to missing `scala-library` in the classpath.
Instead of bundling `scala-library` into YARN ESS jar, #51650 (comment) suggests making it scala-free again.
This also makes Java's invocation of Structured Logging API much cleaner, now, it can be called without ugly `$.MODULE$`.
```patch
- MDC.of(LogKeys.HOST_PORT$.MODULE$, address);
+ MDC.of(LogKeys.HOST_PORT, address);
```
### Does this PR introduce _any_ user-facing change?
No, they are internal APIs, for those plugin developers who want to provide custom `LogKey`s, there still possible to make it compatible with both Spark 4.0 and the new API proposed by this PR, see https://github.com/pan3793/SPARK-53064
```java
import org.apache.spark.internal.LogKey;
// CUSTOM_LOG_KEY is compatible with both Spark 4.0 and SPARK-53064
public class JavaCustomLogKeys {
// Custom `LogKey` must be `implements LogKey`
public static class CUSTOM_LOG_KEY implements LogKey {
Override
public String name() {
return "custom_log_key";
}
}
// Singleton
public static final CUSTOM_LOG_KEY CUSTOM_LOG_KEY = new CUSTOM_LOG_KEY();
}
```
### How was this patch tested?
Pass GHA, and verified YARN ESS works without `scala-library`.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #51775 from pan3793/SPARK-53064.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: yangjie01 <[email protected]>1 parent d72e028 commit dc46b14
File tree
546 files changed
+1892
-1914
lines changed- common
- network-common/src/main/java/org/apache/spark/network
- client
- crypto
- protocol
- server
- util
- network-shuffle/src/main/java/org/apache/spark/network
- sasl
- shuffle
- checksum
- network-yarn/src/main/java/org/apache/spark/network/yarn
- utils/src
- main
- java/org/apache/spark
- internal
- network/util
- scala/org/apache/spark
- internal
- util
- test
- java/org/apache/spark/util
- scala/org/apache/spark/util
- connector
- docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc
- kafka-0-10-sql/src
- main/scala/org/apache/spark/sql/kafka010
- consumer
- producer
- test/scala/org/apache/spark/sql/kafka010
- kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010
- kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010
- kinesis-asl/src
- main/scala/org/apache/spark/streaming/kinesis
- test/scala/org/apache/spark/streaming/kinesis
- profiler/src/main/scala/org/apache/spark/profiler
- spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia
- core/src/main
- java/org/apache/spark
- io
- memory
- shuffle/sort
- io
- unsafe/map
- util/collection/unsafe/sort
- scala/org/apache/spark
- api
- python
- r
- broadcast
- deploy
- client
- history
- master
- ui
- rest
- security
- worker
- ui
- executor
- internal
- io
- plugin
- mapred
- memory
- metrics
- sink
- network/netty
- rdd
- resource
- rpc/netty
- scheduler
- cluster
- dynalloc
- security
- serializer
- shuffle
- status
- storage
- memory
- ui
- scope
- util
- collection
- logging
- graphx/src/main/scala/org/apache/spark/graphx
- lib
- hadoop-cloud/src/main/scala/org/apache/spark/internal/io/cloud
- mllib/src
- main/scala/org/apache/spark
- mllib
- clustering
- evaluation
- feature
- fpm
- linalg/distributed
- optimization
- recommendation
- regression
- stat/test
- tree/model
- util
- ml
- classification
- clustering
- feature
- optim
- recommendation
- regression
- r
- tree/impl
- tuning
- util
- test/scala/org/apache/spark
- mllib
- linalg
- tree
- ml
- feature
- recommendation
- resource-managers
- kubernetes
- core/src/main/scala/org/apache/spark
- deploy/k8s
- features
- submit
- scheduler/cluster/k8s
- shuffle
- integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest
- yarn/src/main/scala/org/apache/spark
- deploy/yarn
- scheduler/cluster
- sql
- api/src/main/scala/org/apache/spark/sql
- catalyst/parser
- types
- catalyst/src/main
- java/org/apache/spark/sql
- catalyst/expressions
- util
- scala/org/apache/spark/sql/catalyst
- analysis
- resolver
- catalog
- csv
- expressions
- codegen
- optimizer
- parser
- rules
- util
- xml
- connect
- common/src/main/scala/org/apache/spark/sql/connect
- client
- server/src
- main/scala/org/apache/spark/sql/connect
- execution
- planner
- service
- ui
- utils
- test/scala/org/apache/spark/sql/connect
- core/src
- main/scala/org/apache/spark/sql
- api
- python
- r
- artifact
- avro
- catalyst/analysis
- classic
- columnar
- execution
- adaptive
- aggregate
- command
- datasources
- csv
- jdbc
- json
- orc
- parquet
- v2
- jdbc
- python
- state/metadata
- exchange
- python
- streaming
- r
- streaming
- checkpointing
- continuous
- operators/stateful
- join
- transformwithstate/timers
- runtime
- sinks
- sources
- state
- ui
- internal
- jdbc
- test/scala/org/apache/spark/sql
- hive-thriftserver/src/main
- java/org/apache/hive/service
- auth
- cli
- operation
- session
- thrift
- server
- scala/org/apache/spark/sql/hive/thriftserver
- ui
- hive/src/main/scala/org/apache/spark/sql/hive
- client
- execution
- orc
- security
- pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph
- streaming/src
- main/scala/org/apache/spark/streaming
- dstream
- rdd
- receiver
- scheduler
- rate
- util
- test/scala/org/apache/spark/streaming
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
546 files changed
+1892
-1914
lines changedLines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
364 | 364 | | |
365 | 365 | | |
366 | 366 | | |
367 | | - | |
368 | | - | |
| 367 | + | |
| 368 | + | |
369 | 369 | | |
370 | 370 | | |
371 | 371 | | |
| |||
Lines changed: 9 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
196 | | - | |
197 | | - | |
198 | | - | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
199 | 199 | | |
200 | 200 | | |
201 | 201 | | |
| |||
210 | 210 | | |
211 | 211 | | |
212 | 212 | | |
213 | | - | |
| 213 | + | |
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
| |||
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
317 | | - | |
| 317 | + | |
318 | 318 | | |
319 | 319 | | |
320 | 320 | | |
| |||
340 | 340 | | |
341 | 341 | | |
342 | 342 | | |
343 | | - | |
| 343 | + | |
344 | 344 | | |
345 | 345 | | |
346 | 346 | | |
347 | 347 | | |
348 | 348 | | |
349 | 349 | | |
350 | 350 | | |
351 | | - | |
352 | | - | |
353 | | - | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
354 | 354 | | |
355 | 355 | | |
356 | 356 | | |
| |||
Lines changed: 19 additions & 19 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
148 | | - | |
149 | | - | |
| 148 | + | |
| 149 | + | |
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
| |||
156 | 156 | | |
157 | 157 | | |
158 | 158 | | |
159 | | - | |
160 | | - | |
| 159 | + | |
| 160 | + | |
161 | 161 | | |
162 | 162 | | |
163 | 163 | | |
| |||
168 | 168 | | |
169 | 169 | | |
170 | 170 | | |
171 | | - | |
172 | | - | |
| 171 | + | |
| 172 | + | |
173 | 173 | | |
174 | 174 | | |
175 | 175 | | |
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
183 | | - | |
184 | | - | |
185 | | - | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
186 | 186 | | |
187 | 187 | | |
188 | 188 | | |
| |||
192 | 192 | | |
193 | 193 | | |
194 | 194 | | |
195 | | - | |
196 | | - | |
197 | | - | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
| |||
208 | 208 | | |
209 | 209 | | |
210 | 210 | | |
211 | | - | |
212 | | - | |
213 | | - | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
| |||
222 | 222 | | |
223 | 223 | | |
224 | 224 | | |
225 | | - | |
226 | | - | |
227 | | - | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
228 | 228 | | |
229 | 229 | | |
230 | 230 | | |
| |||
269 | 269 | | |
270 | 270 | | |
271 | 271 | | |
272 | | - | |
| 272 | + | |
273 | 273 | | |
274 | 274 | | |
275 | 275 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
96 | | - | |
| 96 | + | |
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
69 | | - | |
70 | | - | |
| 69 | + | |
| 70 | + | |
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
74 | | - | |
75 | | - | |
| 74 | + | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| |||
Lines changed: 7 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
| 77 | + | |
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| |||
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
99 | | - | |
100 | | - | |
| 99 | + | |
| 100 | + | |
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| |||
111 | 111 | | |
112 | 112 | | |
113 | 113 | | |
114 | | - | |
115 | | - | |
| 114 | + | |
| 115 | + | |
116 | 116 | | |
117 | 117 | | |
118 | 118 | | |
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
156 | | - | |
157 | | - | |
| 156 | + | |
| 157 | + | |
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
| |||
Lines changed: 4 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| |||
168 | 168 | | |
169 | 169 | | |
170 | 170 | | |
171 | | - | |
172 | | - | |
173 | | - | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| |||
Lines changed: 15 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
135 | | - | |
136 | | - | |
| 135 | + | |
| 136 | + | |
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
| |||
143 | 143 | | |
144 | 144 | | |
145 | 145 | | |
146 | | - | |
147 | | - | |
| 146 | + | |
| 147 | + | |
148 | 148 | | |
149 | 149 | | |
150 | 150 | | |
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | | - | |
181 | | - | |
| 180 | + | |
| 181 | + | |
182 | 182 | | |
183 | 183 | | |
184 | 184 | | |
| |||
264 | 264 | | |
265 | 265 | | |
266 | 266 | | |
267 | | - | |
268 | | - | |
| 267 | + | |
| 268 | + | |
269 | 269 | | |
270 | 270 | | |
271 | 271 | | |
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
285 | | - | |
| 285 | + | |
286 | 286 | | |
287 | 287 | | |
288 | 288 | | |
| |||
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
| |||
326 | 326 | | |
327 | 327 | | |
328 | 328 | | |
329 | | - | |
330 | | - | |
| 329 | + | |
| 330 | + | |
331 | 331 | | |
332 | 332 | | |
333 | 333 | | |
| |||
Lines changed: 4 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
64 | | - | |
| 64 | + | |
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | | - | |
| 68 | + | |
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| |||
0 commit comments