feat(model): debounce model loads after a failure to stop retry-storms

[localai-bot]

Problem

A client that keeps polling a model whose load fails (e.g. a backend that crashes deterministically on init) triggers a fresh backend start on every request: request → load → crash in ~10s → 500, then the next poll repeats it. Each attempt can leak GPU/CUDA state, and under LOCALAI_SINGLE_ACTIVE_BACKEND the failing model keeps stealing the active slot from healthy ones. The reporter watched leaked contexts climb to ~58 GB before a host reboot was needed.

The existing loading-coalesce map only dedups concurrent loads, so sequential polls (60s apart) were never covered.

Fix

ModelLoader now tracks load failures per modelID. After a load fails, fresh load triggers for that model are refused until a cooldown elapses, returning a typed ModelLoadCooldownError that the HTTP layer maps to 503 + Retry-After. The cooldown grows exponentially per consecutive failure (base → ×2, capped at 5m) and resets on a successful load.

The coalesced follower-retry of an in-flight burst bypasses the gate, so a genuinely concurrent request burst still gets its one retry — only new, independent triggers are refused, matching the report's "refuse new load-triggers" wording.

Configurable via --model-load-failure-cooldown / LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN (default 10s, 0 disables), plumbed through ApplicationConfig and applied unconditionally at startup (independent of the watchdog/LRU limiter).

Testing

• TDD red→green: 3 new specs — cooldown refuses a fresh load without re-invoking the loader; failure state clears after a successful load; cooldown grows across consecutive failures.
• Made the existing "retry if first attempt fails" spec deterministic so it genuinely exercises the preserved coalesced follower-retry path (previously its instant-failure timing tested two independent sequential loads).
• Full pkg/model suite green under -race; core builds; golangci-lint clean (new-from-merge-base).

Notes / scope

• The 7-line HTTP mapping helper (applyModelLoadCooldown) is left to review + CI: an internal unit test conflicts with the package's single-RunSpecs bootstrap, and the full-API() harness is heavyweight. The substantive logic (the debounce state machine) is unit-tested directly.
• Distributed mode is unaffected (it has its own router-level coalescing).

Closes #10719

🤖 Generated with Claude Code

[pos-ei-don]

Field feedback on how this landed — from a case tonight where the same model failed to load twice with a stale post-reboot driver state (SIGABRT within ~10 s each attempt, classic Pattern-A on this GB10 hardware).

Observed cooldown values I saw returned in the retry-after messages, in order, from the same client:

• 48 s (freshly after the first fail-load)
• 34 s (~14 s later — matches "remaining time on the same cooldown")
• 6 s (~28 s further — cooldown almost drained)
• 2 m 32 s (after the drain hit zero, the next real load-attempt failed, and a new cooldown started ~3× longer)

My interpretation, with the caveat that this is 4 data points from one session and not a controlled repro: only actual load-attempts trigger a new/extended cooldown, while queries during a live cooldown just report the remaining time. That part is a nice property. But the length of each new cooldown appears to grow roughly geometrically per failed attempt (exponential-ish backoff), and a naïve retry client that just polls after the retry after Xs message will keep tripping this — each polling cycle that lands after drain triggers a fresh (longer) cooldown.

A few questions I would find useful to have documented, either in the PR description or the log line:

1. Is the geometric growth intentional? If so, is there an upper cap, or does it grow unbounded until first successful load?
2. When does the cooldown reset back to base — first successful load only, container restart, some idle timeout?
3. Would a slightly clearer message help clients — something like retry after 2m32s; further failures will lengthen this — so a client with a max_retries loop knows to back off further rather than poll at the reported interval?

For context on this side: the underlying pattern that keeps producing the fails is a deterministic 10-14 s SIGABRT during vLLM engine init on Blackwell (GB10 / sm_121a) — persistent even after a host reboot (I did one tonight and got the same class of fail on both retries afterwards). Filed separately at vllm-project/vllm#48031 (default VLLM_ENGINE_READY_TIMEOUT_S too low for this hardware) and #10720 (backend stderr capture is what lets these fails be diagnosed at all). The debounce here is doing the right thing keeping a hostile retry loop from spamming — this is really just a UX/documentation question, not a bug report.

Happy to run a more controlled repro (defined number of forced fails, measure cooldown values, extract the growth factor) once the underlying init issue is resolved and I can trigger fails on demand — likely sometime later this week.