[feature] Debounce or queue in-flight model load attempts to prevent retry-storm on transient backend failures

[pos-ei-don]

Problem

When a model load fails (e.g. transient CUDA-init error, backend crash on startup), LocalAI accepts new /v1/chat/completions requests for that same modelID immediately and re-triggers a fresh backend start. If an external client keeps polling that model with a short interval (e.g. every 60s), this cascades:

• request N → BackendLoader starts → crashes in ~10s → HTTP 500 returned
• request N+1 (60s later) → new BackendLoader starts → crashes again → 500 → …

Each attempt leaks GPU state (CUDA context is not always released after EngineCore init failure — I'm filing that separately as a vLLM bug). On a shared GPU with LOCALAI_SINGLE_ACTIVE_BACKEND=true, this can also block legitimate concurrent loads of other models because the failing modelID keeps stealing the active slot.

I hit this several times on 2026-07-06 when a client keeps polling a broken model alias. Memory allocated by leaked contexts climbed to ~58 GB before I had to reboot the host to recover.

Repro

1. Configure a broken model yaml (e.g. parameters.model pointing to a non-existent HF repo, or a config that produces Engine core init failed deterministically).
2. Set LOCALAI_SINGLE_ACTIVE_BACKEND=true.
3. Point a client at that alias with a poll interval shorter than time-to-load. Any 60s poll triggers the cascade.
4. Watch docker logs localai:

BackendLoader starting modelID="X"
Backend process exited unexpectedly id="X" exitCode="0"
Failed to load model modelID="X" error=... Engine core initialization failed ...
HTTP request method="POST" path="/v1/chat/completions" status=500
<60s later>
BackendLoader starting modelID="X"   ← identical cycle repeats

Suggested behavior

Introduce one (or a combination) of:

• Load-debounce: after a load fails for a given modelID, refuse new load-triggers for that modelID for N seconds (default e.g. 30–60s). Return HTTP 503 with a Retry-After header.
• Request-queue: while a modelID's load is in-flight, queue further requests instead of triggering additional BackendLoader instances. The in-flight load's result fans out to all queued callers.
• Exponential backoff on repeated load failures per modelID.

The first (debounce) alone would already close the retry-storm surface.

Version

LocalAI v4.6.0 and v4.6.2 (both nvidia-l4t-arm64-cuda-13 build), vLLM backend 0.23.0. Reproducible in both.

Environment

• Host: NVIDIA DGX Spark (GB10, ARM64, 121 GB unified memory)
• Docker image: localai/localai:v4.6.2-nvidia-l4t-arm64-cuda-13
• Env: LOCALAI_SINGLE_ACTIVE_BACKEND=true, LOCALAI_WATCHDOG_IDLE=false. Reproduces both with and without LOCALAI_LOAD_TO_MEMORY set.

Related

vLLM: CUDA context leak on EngineCore init failure — separate bug report incoming.