[Feat] Add multi-node multiprocessing deployment via LeaderWorkerSet by ardecode · Pull Request #1007 · vllm-project/production-stack

ardecode · 2026-07-23T05:45:51Z

Summary

Implements first-class multi-node multiprocessing (MP) deployment support.

Fixes #930.

The only distributed path today is ray-cluster.yaml, which requires the KubeRay operator and hardcodes --distributed-executor-backend ray. Since vLLM v0.19.0 dropped Ray from the default image, this adds an MP alternative modeled on the upstream vLLM LeaderWorkerSet example (vllm-project/vllm#39400).

Changes

New modelSpec.lwsSpec (enabled, size, restartPolicy) + values.schema.json entry
New helm/templates/lws-vllm.yaml rendering a LeaderWorkerSet:
- leader (node-rank 0) serves the OpenAI API; workers run --headless
- --nnodes / --node-rank / --master-addr from $(LWS_GROUP_SIZE) / $(LWS_WORKER_INDEX) / $(LWS_LEADER_ADDRESS)
- only the leader carries router/engine-service discovery labels
Exclude LWS-enabled models from the Deployment and KEDA ScaledObject paths (mirrors raySpec)

Notes

Requires the LeaderWorkerSet controller (kubernetes-sigs/lws) installed in the cluster.
A tutorial and Helm unit tests will added once I get a thumbs up on the approach.
Validated with helm template, helm lint, and structural YAML checks; not yet exercised on a live multi-node GPU cluster.

Checklist

Signed-off commit (DCO)
PR classified ([Feat])
pre-commit checks (please run pre-commit run --all-files; I could not run the helm-schema generator locally)

…rSet Adds a first-class multi-node path for the serving engine that uses vLLM's multiprocessing (MP) distributed backend instead of Ray, addressing vllm-project#930. A new modelSpec.lwsSpec renders a LeaderWorkerSet (leaderworkerset.x-k8s.io/v1): the leader (node-rank 0) serves the OpenAI API and workers run with --headless, wired via the LWS-injected LWS_GROUP_SIZE / LWS_WORKER_INDEX / LWS_LEADER_ADDRESS values (--nnodes / --node-rank / --master-addr). Works with the stock vllm/vllm-openai image; Ray is not required. LWS-enabled models are excluded from the plain Deployment and KEDA paths, and only the leader carries the engine discovery labels so the router targets it. Signed-off-by: ardecode <desaiarijit@gmail.com>

gemini-code-assist

Code Review

This pull request introduces support for deploying models across multiple nodes using Kubernetes LeaderWorkerSet (LWS) and vLLM's multiprocessing distributed backend as an alternative to Ray. It adds the new lws-vllm.yaml template and updates values, schemas, and other templates to support this deployment mode. The review feedback highlights several critical feature parity gaps and integration issues in the new LWS template, including missing LMCache configurations and environment variables, lack of support for emptyDir in pvcStorage, missing support for sharedPvcStorage, and several missing CLI arguments and environment variables (such as envFromSecret, --runner, --convert, and explicit prefix caching flags).

gemini-code-assist · 2026-07-23T05:48:24Z

+{{- if $modelSpec.chatTemplate }}
+- "--chat-template"
+- {{ $modelSpec.chatTemplate | quote }}
+{{- end }}


LMCache integration is completely broken in LWS mode because the --kv-transfer-config argument is not passed to the vllm serve command. Add the configuration argument to ensure LMCache is initialized correctly.

{{- if $modelSpec.chatTemplate }} - "--chat-template" - {{ $modelSpec.chatTemplate | quote }} {{- end }} {{- $kv_role := "kv_both" }} {{- if and $modelSpec.lmcacheConfig $modelSpec.lmcacheConfig.kvRole }} {{- $kv_role = $modelSpec.lmcacheConfig.kvRole }} {{- end }} {{- if $modelSpec.lmcacheConfig }} {{- if $modelSpec.lmcacheConfig.enabled }} {{- if hasKey $modelSpec.lmcacheConfig "enablePD" }} - "--kv-transfer-config" - '{"kv_connector":"LMCacheConnectorV1","kv_role":"{{ $kv_role }}","kv_connector_extra_config":{"discard_partial_chunks": false, "lmcache_rpc_port": {{ $modelSpec.lmcacheConfig.nixlRole | quote }}}}' {{- else if and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "v0") (eq (toString $modelSpec.vllmConfig.v0) "1") }} - "--kv-transfer-config" - '{"kv_connector":"LMCacheConnector","kv_role":"{{ $kv_role }}"}' {{- else }} - "--kv-transfer-config" - '{"kv_connector":"LMCacheConnectorV1","kv_role":"{{ $kv_role }}"}' {{- end }} {{- end }} {{- end }}

gemini-code-assist · 2026-07-23T05:48:24Z

+    {{- with $root.Values.servingEngineSpec.env }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+    {{- with $modelSpec.env }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}


LMCache environment variables are completely missing from the LWS container spec. Without these variables (such as LMCACHE_USE_EXPERIMENTAL, LMCACHE_REMOTE_URL, etc.), LMCache will not function. Please add the LMCache environment variables block matching the one in deployment-vllm-multi.yaml.

gemini-code-assist · 2026-07-23T05:48:24Z

+{{- if hasKey $modelSpec "pvcStorage" }}
+- name: {{ $root.Release.Name }}-storage
+  persistentVolumeClaim:
+    claimName: "{{ $root.Release.Name }}-{{ $modelSpec.name }}-storage-claim"
+{{- end }}


The pvcStorage configuration can be defined as an emptyDir map in values.yaml (similar to how it is handled in deployment-vllm-multi.yaml). Currently, lws-vllm.yaml assumes pvcStorage is always a persistent volume claim, which will cause the LeaderWorkerSet deployment to fail if emptyDir is used.

{{- if hasKey $modelSpec "pvcStorage" }} - name: {{ $root.Release.Name }}-storage {{- if and (kindIs "map" $modelSpec.pvcStorage) (hasKey $modelSpec.pvcStorage "emptyDir") }} emptyDir: {{- toYaml $modelSpec.pvcStorage.emptyDir | nindent 4 }} {{- else }} persistentVolumeClaim: claimName: "{{ $root.Release.Name }}-{{ $modelSpec.name }}-storage-claim" {{- end }} {{- end }}

gemini-code-assist · 2026-07-23T05:48:24Z

+{{- if and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "tensorParallelSize") }}
+- name: shm
+  emptyDir:
+    medium: Memory
+    sizeLimit: {{ default "20Gi" $modelSpec.shmSize }}
+{{- end }}


Add support for sharedPvcStorage in chart.lwsVolumes to ensure feature parity with the standard Deployment template. This allows LWS pods to mount the shared storage volume when enabled.

{{- if $root.Values.sharedPvcStorage.enabled }} - name: {{ $root.Release.Name }}-shared-pvc-storage persistentVolumeClaim: claimName: "{{ $root.Release.Name }}-shared-pvc-storage-claim" {{- end }} {{- if and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "tensorParallelSize") }} - name: shm emptyDir: medium: Memory sizeLimit: {{ default "20Gi" $modelSpec.shmSize }} {{- end }}

gemini-code-assist · 2026-07-23T05:48:24Z

+imagePullSecrets:
+  - name: {{ $modelSpec.imagePullSecret }}
+{{- end }}
+{{- $vols := or (hasKey $modelSpec "pvcStorage") (and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "tensorParallelSize")) $modelSpec.extraVolumes }}


Include sharedPvcStorage.enabled in the $vols check to ensure volumes are rendered when shared storage is enabled.

{{- $vols := or (hasKey $modelSpec "pvcStorage") (and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "tensorParallelSize")) $modelSpec.extraVolumes $root.Values.sharedPvcStorage.enabled }}

gemini-code-assist · 2026-07-23T05:48:24Z

+    - name: HF_HOME
+      {{- if hasKey $modelSpec "pvcStorage" }}
+      value: /data
+      {{- else }}
+      value: /tmp
+      {{- end }}


Configure HF_HOME to point to the shared PVC storage path when sharedPvcStorage is enabled, matching the behavior in deployment-vllm-multi.yaml.

- name: HF_HOME {{- if $root.Values.sharedPvcStorage.enabled }} value: /data/shared-pvc-storage {{- else if hasKey $modelSpec "pvcStorage" }} value: /data {{- else }} value: /tmp {{- end }}

gemini-code-assist · 2026-07-23T05:48:24Z

+  {{- $mountVolumes := or (hasKey $modelSpec "pvcStorage") (and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "tensorParallelSize")) $modelSpec.extraVolumeMounts }}
+  {{- if $mountVolumes }}
+  volumeMounts:
+    {{- if hasKey $modelSpec "pvcStorage" }}
+    - name: {{ $root.Release.Name }}-storage
+      mountPath: /data
+    {{- end }}


Mount the shared PVC storage volume in the container volume mounts when sharedPvcStorage is enabled.

{{- $mountVolumes := or (hasKey $modelSpec "pvcStorage") (and $modelSpec.vllmConfig (hasKey $modelSpec.vllmConfig "tensorParallelSize")) $modelSpec.extraVolumeMounts $root.Values.sharedPvcStorage.enabled }} {{- if $mountVolumes }} volumeMounts: {{- if hasKey $modelSpec "pvcStorage" }} - name: {{ $root.Release.Name }}-storage mountPath: /data {{- end }} {{- if $root.Values.sharedPvcStorage.enabled }} - name: {{ $root.Release.Name }}-shared-pvc-storage mountPath: /data/shared-pvc-storage {{- end }}

gemini-code-assist · 2026-07-23T05:48:25Z

+  {{- if $root.Values.servingEngineSpec.configs }}
+  envFrom:
+    - configMapRef:
+        name: "{{ $root.Release.Name }}-configs"
+  {{- end }}


Add support for envFromSecret in envFrom to allow loading environment variables from secrets in LWS mode.

{{- if or $root.Values.servingEngineSpec.configs $modelSpec.envFromSecret }} envFrom: {{- if $root.Values.servingEngineSpec.configs }} - configMapRef: name: "{{ $root.Release.Name }}-configs" {{- end }} {{- if $modelSpec.envFromSecret }} - secretRef: name: {{ $modelSpec.envFromSecret.name }} {{- end } {{- end }}

gemini-code-assist · 2026-07-23T05:48:25Z

+{{- if hasKey . "maxLoras" }}
+- "--max-loras"
+- {{ .maxLoras | quote }}
+{{- end }}
+{{- range .extraArgs }}
+- {{ . | quote }}
+{{- end }}


Add support for --runner and --convert command arguments in chart.lwsServeCommand to match the configuration options available in the standard Deployment template.

{{- if hasKey . "maxLoras" }} - "--max-loras" - {{ .maxLoras | quote }} {{- end }} {{- if hasKey . "runner" }} - "--runner" - {{ .runner | quote }} {{- end }} {{- if hasKey . "convert" }} - "--convert" - {{ .convert | quote }} {{- end }} {{- range .extraArgs }} - {{ . | quote }} {{- end }}

gemini-code-assist · 2026-07-23T05:48:25Z

+{{- if .enablePrefixCaching }}
+- "--enable-prefix-caching"
+{{- end }}


Explicitly pass --no-enable-prefix-caching when enablePrefixCaching is false to ensure consistent default behavior between Deployment and LWS modes.

{{- if .enablePrefixCaching }} - "--enable-prefix-caching" {{- else }} - "--no-enable-prefix-caching" {{- end }}

…arity for LWS Bring the LeaderWorkerSet template to feature parity with deployment-vllm-multi.yaml per PR review: - wire LMCache --kv-transfer-config and the full LMCACHE_* env block - support sharedPvcStorage (volume, mount, HF_HOME) and pvcStorage emptyDir - add envFromSecret, --runner/--convert, and explicit --no-enable-prefix-caching Signed-off-by: ardecode <desaiarijit@gmail.com>

ardecode · 2026-07-24T05:29:28Z

@ruizhang0101 , ready for review!

gemini-code-assist Bot reviewed Jul 23, 2026

View reviewed changes

ardecode force-pushed the feat/lws-multinode-mp branch from c664751 to 5856883 Compare July 23, 2026 06:18

ardecode and others added 3 commits July 26, 2026 10:35

Merge branch 'main' into feat/lws-multinode-mp

9caf8fe

Merge branch 'main' into feat/lws-multinode-mp

86dd634

Merge branch 'main' into feat/lws-multinode-mp

6ebd0c8

ardecode requested review from ApostaC, Shaoting-Feng, YuhanLiu11 and ruizhang0101 as code owners July 29, 2026 03:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feat] Add multi-node multiprocessing deployment via LeaderWorkerSet - #1007

[Feat] Add multi-node multiprocessing deployment via LeaderWorkerSet#1007
ardecode wants to merge 5 commits into
vllm-project:mainfrom
ardecode:feat/lws-multinode-mp

ardecode commented Jul 23, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

ardecode commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ardecode commented Jul 23, 2026

Summary

Changes

Notes

Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

ardecode commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants