server: recover from checkpoint on truncation failure for hybrid/recurrent models

[European-tech]

Problem

When re-using a cached prompt with TAG_PROMPT_LOGITS, the server tries to truncate the KV cache via llama_memory_seq_rm(slot.id, p0, -1). On hybrid/recurrent models (Qwen3.5, Jamba, Falcon-H1), this call always fails because partial truncation is not supported for recurrent layers. The current fallback is prompt_clear(true), which erases the entire cached state and forces a full prompt re-evaluation from scratch.

For long contexts this is devastating: a 35K-token prompt takes ~21 seconds to re-process instead of the sub-100ms it should take from cache.

Root Cause

llama_memory_seq_rm() returns false for recurrent memory types (Mamba/SSM layers) because they store a compressed state that cannot be partially truncated. The server interprets this as "memory is corrupt" and clears everything — but for hybrid models the memory is perfectly fine, it just doesn't support partial removal.

Fix

Instead of clearing all memory on truncation failure, the server now:

1. Searches backward through existing prompt checkpoints (already created during prompt processing for recurrent models)
2. Finds the nearest checkpoint before the truncation point
3. Restores the full recurrent state from that checkpoint via llama_state_seq_set_data_ext() with LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY
4. Re-evaluates only the delta tokens between the checkpoint and the current position

Falls back to the original prompt_clear() only if no suitable checkpoint exists.

Additionally: checkpoint persistence across slot save/restore

The built-in /slots save/restore API persists KV+recurrent memory via llama_state_seq_{save,load}_file, but does not persist the checkpoint metadata. This PR adds:

• slot_checkpoints_save/load() helpers that write/read a companion .checkpoints file next to the slot state file
• auto_save_slots() / auto_restore_slots() methods called at server shutdown/startup for seamless model hot-swap in router mode (multi-model --models-dir)

Measured Impact

Qwen3.5-27B, 35858 token context, RTX 3090 24GB:

Scenario	Time	Tokens re-evaluated
Before (prompt_clear)	21054 ms	35858 (all)
After (checkpoint recovery)	77 ms	~200 (delta only)
Speedup	273x

Related Issues

• Eval bug: Qwen 3.5 Full prompt re-processing on every conversation turn #20225 — Qwen3 inference issues with KV cache
• Eval bug: forcing full prompt re-processing due to lack of cache data in Qwen3.5 27B #20153 — Hybrid model memory management

Files Changed

• tools/server/server-context.cpp — checkpoint save/load helpers, truncation recovery logic, auto-save/restore
• tools/server/server-context.h — auto_save_slots() / auto_restore_slots() declarations
• tools/server/server.cpp — call auto-restore on startup, auto-save on shutdown

[ggml-gh-bot]

Hi @European-tech, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[llukas]

This MR works for me on M4 Macbook. I checked Qwen-122B and Nemotron 3 Super.

Minor comment: slot auto-save/auto-restore could be an option not unconditional behavior.

[dark-penguin]

This was closed because this mechanism is already implemented, right?