[feature/backend-upgrade] Pin arm64-nvidia-l4t-cuda-13 vLLM backend to 0.24.0 — PR vllm-project/vllm#45179 (UMA GPU memory release) fixes deterministic EngineCore init crashes on GB10 / DGX Spark

[pos-ei-don]

Request

Please pin the nvidia-l4t-arm64-cuda-13-vllm backend gallery to vLLM 0.24.0 (currently 0.23.0). The upgrade includes vllm-project/vllm#45179, "Release cached device memory under pressure on UMA GPUs during weight loading" — critical for GB10 (NVIDIA DGX Spark, Grace Blackwell, unified memory architecture).

Evidence

On a DGX Spark (arm64, 121 GB unified memory), vLLM 0.23.0 causes deterministic cold-load failures in 10-14 seconds:

ERROR Failed to load model modelID="coder"
  error=failed to load model with internal loader: could not load model 
  (no success): Unexpected err=RuntimeError('Engine core initialization 
  failed. See root cause above. Failed core proc(s): {}'), 
  type(err)=<class 'RuntimeError'>

The Failed core proc(s): {} set is empty (no Python traceback), the backend subprocess exits with exitCode=0, and GPU memory remains pinned across process exit. Recovery requires a full host reboot — docker restart, container recreate, backend venv swap between older snapshots, and FlashInfer cached_ops/ flush all fail.

Reproduced 2026-07-06/07 through an 8-hour diagnostic session: four orthogonal changes (LocalAI image v4.6.0 / v4.6.2 / v4.5.6, backend venv v0.23-shelved / various v0.23 snapshots, port isolation via -p 8083:8080 to rule out client polling, FlashInfer cached_ops/ flush) — every combination crashed identically in 10-14 s.

After swapping the backend venv to vLLM 0.24.0 and rebooting the host once (to clear previously pinned kernel state), the same setup produces a successful 16:58-minute cold-load followed by stable inference at ~55 tok/s. No more EngineCore init failed. Inference stays healthy across multiple test rounds.

Setup

• Host: NVIDIA DGX Spark (GB10, ARM64, 121 GB unified memory)
• LocalAI image: localai/localai:v4.5.6-nvidia-l4t-arm64-cuda-13 (v4.6.0 / v4.6.2 also affected on 0.23)
• Model: saricles/Qwen3-Coder-Next-NVFP4-GB10 (79.7B MoE, NVFP4)
• Model YAML: context_size: 262144, gpu_memory_utilization: 0.55, kv_cache_dtype: fp8, enable_prefix_caching: true, enable_chunked_prefill: true

Why now

The arm64-l4t-vllm backend image is documented as stale relative to LocalAI mainline (see #10638). vLLM 0.24 was released 2026-06-29 — the fix has been available for a week. On GB10 hardware in particular, 0.23 is not viable in production — every restart / power cycle risks a stuck kernel state that only host reboot clears.

Related

• vLLM PR: vllm-project/vllm#45179 (merged 2026-06-XX)
• Filed 2026-07-07 in the same batch:
  • #10719 (feature: load-debounce — the retry-storm symptom that surfaces this bug)
  • #10720 (bug: install.sh needs uv + ABI-mismatch diagnostics)

Happy to test a v0.24-pinned backend build if you can push one to the gallery.