Skip to content

server: fix abort on exact prompt-cache match with hybrid/recurrent models#175

Merged
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
kmehan:fix/recurrent-seq-rm-exact-match-abort
Jun 12, 2026
Merged

server: fix abort on exact prompt-cache match with hybrid/recurrent models#175
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
kmehan:fix/recurrent-seq-rm-exact-match-abort

Conversation

@kmehan

@kmehan kmehan commented Jun 10, 2026

Copy link
Copy Markdown

Problem

When an incoming prompt exactly matches a slot's cached tokens, the server backs off one token (n_past--) to guarantee at least one token is decoded for logits. The subsequent truncation calls seq_rm with p0 = n_past > 0, which recurrent memory cannot satisfy (the state can't be rewound to a mid-sequence position when the rollback exceeds n_rs_seq), so common_context_seq_rm hits GGML_ABORT and the whole server dies:

common/common.cpp:1472: failed to remove sequence 0 with p0=490, p1=-1

Observed in production with Qwen3.6-35B-A3B (hybrid attention + GatedDeltaNet): any client re-sending an identical prompt with cache_prompt enabled (regenerate, retry, n_predict: 0 scoring) crashes the server.

Reproduction

Any recurrent model, e.g. mamba-130m Q8_0 on CPU:

llama-server -m mamba-130m.q8_0.gguf --port 8199 -c 2048 -np 1
# POST /completion {"prompt": "<~500-token text>", "n_predict": 0, "cache_prompt": true}  — twice, identical

Unpatched, the second request aborts the server:

W slot update_slots: ... need to evaluate at least 1 token for each active slot (n_past = 491, task.n_tokens() = 491)
W slot update_slots: ... n_past was set to 490
common/common.cpp:1472: failed to remove sequence 0 with p0=490, p1=-1

Fix

Include the exact-match case (n_past == slot.prompt.n_tokens()) in the existing checkpoint-restore / full-reprocess branch by relaxing its condition to n_past <= slot.prompt.n_tokens() and extending pos_min_thold by 1 when there are no new tokens, so the upcoming one-token back-off is accounted for. Recurrent/hybrid models now restore a context checkpoint (or fall back to full re-processing) instead of aborting; attention models are unaffected.

This is a faithful backport of upstream ggml-org/llama.cpp ggml-org#23280 (ccee426) and ggml-org#24110 (6f3a9f3); this fork's merge-base predates both.

Verification

Patched, same repro sequence (×3): HTTP 200 every time, server stays alive —

W ... restored context checkpoint (pos_min = 486, pos_max = 486, n_tokens = 487, n_past = 487, ...)
  • exact resend: prompt_n=4 (checkpoint restore; falls back to n_past = 0 full reprocess when no checkpoint exists)
  • grown (appended) prompt: prompt_n=21 — prefix cache reuse unchanged
  • attention-model regression check (stories15M): exact resend prompt_n=1 (unchanged one-token back-off), grown prompt prompt_n=8, no checkpoint restores — no behavior change for non-recurrent models

…odels

When an incoming prompt exactly matches the slot's cached tokens, the
server backs off one token (n_past--) to guarantee at least one token is
decoded for logits [TAG_PROMPT_LOGITS]. The subsequent truncation then
calls seq_rm with p0 = n_past > 0, which recurrent memory cannot satisfy
(the state cannot be rewound to a mid-sequence position when the rollback
exceeds n_rs_seq), so common_context_seq_rm hits GGML_ABORT and the whole
server dies:

  common/common.cpp:1472: failed to remove sequence 0 with p0=490, p1=-1

Observed in production with Qwen3.6-35B-A3B (GatedDeltaNet layers): any
client re-sending an identical prompt with cache_prompt enabled
(regenerate / retry) crashed the server. Reproducible with any recurrent
model, e.g. mamba-130m, by sending the same /completion prompt twice.

Fix: include the exact-match case (n_past == slot.prompt.n_tokens()) in
the existing checkpoint-restore/full-reprocess branch by relaxing its
condition to n_past <= slot.prompt.n_tokens() and extending pos_min_thold
by 1 when there are no new tokens, so the upcoming back-off is accounted
for. Recurrent/hybrid models now restore a context checkpoint (or fall
back to full re-processing) instead of aborting; attention models are
unaffected.

This backports upstream ggml-org/llama.cpp PRs ggml-org#23280 and ggml-org#24110
(commits ccee426, 6f3a9f3).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TheTom

TheTom commented Jun 12, 2026

Copy link
Copy Markdown
Owner

Thank you @kmehan, this is a model fix: minimal diff, honest repro, and it backports the exact logic from upstream ggml-org#24110 (merged June 4), so it carries upstream's blessing. Verified the blast radius: full-attention models never reach the inner block (pos_min stays 0 below the threshold), so only SWA/hybrid/recurrent paths change, which is exactly the bug population. Worst case on a checkpoint-less recurrent model is a full reprocess instead of a dead server. Merging.

@TheTom TheTom merged commit f0d5cc9 into TheTom:feature/turboquant-kv-cache Jun 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants