[SPARK-54173][K8S] Add support for Deployment API on K8s by ForVic · Pull Request #52867 · apache/spark

ForVic · 2025-11-04T01:54:50Z

What changes were proposed in this pull request?

Adds support for K8s Deployment API to allocate pods.

Why are the changes needed?

Allocating individual pods is not ideal, and we can allocate with higher level APIs. #33508 helps this by adding an interface for arbitrary allocators and adds a statefulset allocator. However, dynamic allocation only works if you have implemented a PodDisruptionBudget associated with the decommission label. Since Deployment uses ReplicaSet, which supports pod-deletion-cost annotation, we can avoid needing to create a separate PDB resource, and allow dynamic allocation (w/ shuffle tracking) by adding a low deletion cost to executors we are scaling down. When we scale the Deployment, it will choose to scale down the pods with the low deletion cost.

Does this PR introduce any user-facing change?

Yes, adds user-facing configs

spark.kubernetes.executor.podDeletionCost

How was this patch tested?

New unit tests + passing existing unit tests + tested in a cluster with shuffle tracking and dynamic allocation enabled

Was this patch authored or co-authored using generative AI tooling?

No

ForVic · 2025-11-05T01:10:28Z

cc @sunchao

also @holdenk @dongjoon-hyun

sunchao · 2025-11-05T18:31:48Z

Thanks @dongjoon-hyun for the review!

sunchao

Thanks @ForVic ! Looks mostly good to me (already reviewed internally). Just one question around the new conf.

sunchao

LGTM

sunchao · 2025-11-11T04:20:49Z

Merged to master, thanks @ForVic for the contribution and @dongjoon-hyun for the review!

### What changes were proposed in this pull request? Adds support for K8s `Deployment` API to allocate pods. ### Why are the changes needed? Allocating individual pods is not ideal, and we can allocate with higher level APIs. apache#33508 helps this by adding an interface for arbitrary allocators and adds a statefulset allocator. However, dynamic allocation only works if you have implemented a PodDisruptionBudget associated with the decommission label. Since Deployment uses ReplicaSet, which supports `pod-deletion-cost` annotation, we can avoid needing to create a separate PDB resource, and allow dynamic allocation (w/ shuffle tracking) by adding a low deletion cost to executors we are scaling down. When we scale the Deployment, it will choose to scale down the pods with the low deletion cost. ### Does this PR introduce _any_ user-facing change? Yes, adds user-facing configs ``` spark.kubernetes.executor.podDeletionCost ``` ### How was this patch tested? New unit tests + passing existing unit tests + tested in a cluster with shuffle tracking and dynamic allocation enabled ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#52867 from ForVic/dev/victors/deployment_allocator. Lead-authored-by: Victor Sunderland <victors@openai.com> Co-authored-by: victors-oai <victors@openai.com> Co-authored-by: Victor Sunderland <64456855+ForVic@users.noreply.github.com> Signed-off-by: Chao Sun <chao@openai.com>

dongjoon-hyun

Just FYI, K8s v1.35 was released a few days ago (2025-12-17) and I added a K8s v1.35 test coverage for Apache Spark 4.2.0. Thank you again for contributing SPARK-54173 to the Apache Spark repository, @ForVic and @sunchao .

#53534

dongjoon-hyun · 2025-12-21T23:18:56Z

      .createWithDefault("direct")

+  val KUBERNETES_EXECUTOR_POD_DELETION_COST =
+    ConfigBuilder("spark.kubernetes.executor.podDeletionCost")


Do we have a future plan to manage this dynamically from Apache Spark side, @ForVic ?

If this is static, can we reuse spark.kubernetes.executor.annotation.controller.kubernetes.io/pod-deletion-cost=XXX instead of spark.kubernetes.executor.podDeletionCost=XXX? Or, pod template?

…onCost` config doc ### What changes were proposed in this pull request? This is a follow-up to fix `spark.kubernetes.executor.podDeletionCost` config doc. - #52867 ### Why are the changes needed? **Apache Spark 4.2.0-preview2** ``` Value to set for the controller.kubernetes.io/pod-deletion-cost annotation when Spark asks a deployment-based allocator to remove executor pods. This helps Kubernetes pick the same pods Spark selected when the deployment scales down. This should only be enabled when both ConfigEntry(key=spark.kubernetes.allocation.pods.allocator, defaultValue=direct, doc=Allocator to use for pods. Possible values are direct (the default), statefulset, deployment, or a full class name of a class implementing AbstractPodsAllocator. Future version may add Job or replicaset. This is a developer API and may change or be removed at anytime., public=true, version=3.3.0) is set to deployment, and ConfigEntry(key=spark.dynamicAllocation.enabled, defaultValue=false, doc=, public=true, version=1.2.0) is enabled. ``` **THIS PR** ``` Value to set for the controller.kubernetes.io/pod-deletion-cost annotation when Spark asks a deployment-based allocator to remove executor pods. This helps Kubernetes pick the same pods Spark selected when the deployment scales down. This should only be enabled when both spark.kubernetes.allocation.pods.allocator is set to deployment, and spark.dynamicAllocation.enabled is enabled. ``` ### Does this PR introduce _any_ user-facing change? No this is a new feature of Spark 4.2.0 which is not released yet officially. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: `Gemini 3 Pro (High)` on `Antigravity` Closes #54271 from dongjoon-hyun/SPARK-54173. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…onCost` config doc ### What changes were proposed in this pull request? This is a follow-up to fix `spark.kubernetes.executor.podDeletionCost` config doc. - apache#52867 ### Why are the changes needed? **Apache Spark 4.2.0-preview2** ``` Value to set for the controller.kubernetes.io/pod-deletion-cost annotation when Spark asks a deployment-based allocator to remove executor pods. This helps Kubernetes pick the same pods Spark selected when the deployment scales down. This should only be enabled when both ConfigEntry(key=spark.kubernetes.allocation.pods.allocator, defaultValue=direct, doc=Allocator to use for pods. Possible values are direct (the default), statefulset, deployment, or a full class name of a class implementing AbstractPodsAllocator. Future version may add Job or replicaset. This is a developer API and may change or be removed at anytime., public=true, version=3.3.0) is set to deployment, and ConfigEntry(key=spark.dynamicAllocation.enabled, defaultValue=false, doc=, public=true, version=1.2.0) is enabled. ``` **THIS PR** ``` Value to set for the controller.kubernetes.io/pod-deletion-cost annotation when Spark asks a deployment-based allocator to remove executor pods. This helps Kubernetes pick the same pods Spark selected when the deployment scales down. This should only be enabled when both spark.kubernetes.allocation.pods.allocator is set to deployment, and spark.dynamicAllocation.enabled is enabled. ``` ### Does this PR introduce _any_ user-facing change? No this is a new feature of Spark 4.2.0 which is not released yet officially. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: `Gemini 3 Pro (High)` on `Antigravity` Closes apache#54271 from dongjoon-hyun/SPARK-54173. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to make `AbstractPodsAllocator` documentation up-to-date: - Use `Javadoc` style instead of `Scaladoc` according to the [Apache Spark Code Style Guide](https://spark.apache.org/contributing.html). - Add `Since` annotation for the class and methods. - Update the class documentation with new `DeploymentPodsAllocator`. - Update the method documentation with `since` tag. ### Why are the changes needed? `AbstractPodsAllocator` is one of `DeveloperApi` of the Apache Spark. https://github.com/apache/spark/blob/4da26e4bf9eab119ff2489c5fdf85efe60f6f469/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/AbstractPodsAllocator.scala#L36-L37 Recently, Apache Spark 4.2.0 - Added a new built-in implementation, `DeploymentPodsAllocator`. - #52867 - Added new API, `setExecutorPodsLifecycleManager`. - #53840 ### Does this PR introduce _any_ user-facing change? No, this changes documentations only. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity` Closes #54526 from dongjoon-hyun/SPARK-55725. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? Add `examples/pi-preview-deployment.yaml`, a SparkPi example that uses `spark.kubernetes.allocation.pods.allocator=deployment`. ### Why are the changes needed? To provide a ready-to-apply example demonstrating the `deployment`-based pod allocator alongside the upcoming `4.2.0-preview5` release. - apache/spark#52867 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually tested. ``` $ kubectl apply -f examples/pi-preview-deployment.yaml sparkapplication.spark.apache.org/pi-preview-deployment created $ kubectl get sparkapp NAME AGE CURRENT STATE pi-preview-deployment 3s DriverRequested $ kubectl get sparkapp NAME AGE CURRENT STATE pi-preview-deployment 11s RunningHealthy $ kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE spark-d-pi-preview-deployment-0-0 3/3 3 3 16s spark-kubernetes-operator 1/1 1 1 3d4h $ kubectl get pod NAME READY STATUS RESTARTS AGE pi-preview-deployment-0-driver 1/1 Running 0 23s spark-d-pi-preview-deployment-0-0-7cc95db6bb-8dzln 1/1 Running 0 20s spark-d-pi-preview-deployment-0-0-7cc95db6bb-hhfqn 1/1 Running 0 20s spark-d-pi-preview-deployment-0-0-7cc95db6bb-pglpb 1/1 Running 0 20s spark-kubernetes-operator-584498648c-pxfpc 1/1 Running 4 (46h ago) 3d4h ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #668 from dongjoon-hyun/SPARK-56787. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Implement a DeploymentPodsAllocator

749e75d

github-actions Bot added KUBERNETES DOCS labels Nov 4, 2025

ForVic changed the title ~~[todo] Add support for Deployment API on K8s~~ [todo][K8S] Add support for Deployment API on K8s Nov 4, 2025

ForVic changed the title ~~[todo][K8S] Add support for Deployment API on K8s~~ [SPARK-54173][K8S] Add support for Deployment API on K8s Nov 4, 2025

victors-oai added 4 commits November 3, 2025 18:38

compile with 2.13 scala

347f77f

more compile issues

218b733

import order

efa302d

import order

4d9725f

ForVic marked this pull request as ready for review November 5, 2025 01:03

formatting

6bae7d9

dongjoon-hyun reviewed Nov 5, 2025

View reviewed changes

Comment thread docs/running-on-kubernetes.md Outdated

dongjoon-hyun reviewed Nov 5, 2025

View reviewed changes

Comment thread resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated

4.2.0

b7ca243

sunchao reviewed Nov 6, 2025

View reviewed changes

Comment thread resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated

victors-oai and others added 3 commits November 10, 2025 09:53

address feedback

f55e244

Merge branch 'master' into dev/victors/deployment_allocator

057d466

unused imports

568f6f8

sunchao approved these changes Nov 10, 2025

View reviewed changes

sunchao closed this in d4bc277 Nov 11, 2025

dongjoon-hyun reviewed Dec 21, 2025

View reviewed changes

dongjoon-hyun mentioned this pull request Feb 11, 2026

[SPARK-54173][K8S][FOLLOWUP] Fix spark.kubernetes.executor.podDeletionCost config doc #54271

Closed

dongjoon-hyun mentioned this pull request Feb 27, 2026

[SPARK-55725][K8S][DOCS] Make AbstractPodsAllocator docs up-to-date #54526

Closed

dongjoon-hyun mentioned this pull request May 7, 2026

[SPARK-56787] Add Deployment-based SparkApp example apache/spark-kubernetes-operator#668

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-54173][K8S] Add support for Deployment API on K8s#52867

[SPARK-54173][K8S] Add support for Deployment API on K8s#52867
ForVic wants to merge 10 commits into
apache:masterfrom
ForVic:dev/victors/deployment_allocator

ForVic commented Nov 4, 2025

Uh oh!

ForVic commented Nov 5, 2025

Uh oh!

Uh oh!

Uh oh!

sunchao commented Nov 5, 2025

Uh oh!

sunchao left a comment

Uh oh!

Uh oh!

sunchao left a comment

Uh oh!

sunchao commented Nov 11, 2025

Uh oh!

dongjoon-hyun left a comment •

edited

Loading

Uh oh!

dongjoon-hyun Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ForVic commented Nov 4, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

ForVic commented Nov 5, 2025

Uh oh!

Uh oh!

Uh oh!

sunchao commented Nov 5, 2025

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

sunchao commented Nov 11, 2025

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun left a comment •

edited

Loading