server: fix abort on exact prompt-cache match with hybrid/recurrent models#175
Merged
TheTom merged 1 commit intoJun 12, 2026
Conversation
…odels When an incoming prompt exactly matches the slot's cached tokens, the server backs off one token (n_past--) to guarantee at least one token is decoded for logits [TAG_PROMPT_LOGITS]. The subsequent truncation then calls seq_rm with p0 = n_past > 0, which recurrent memory cannot satisfy (the state cannot be rewound to a mid-sequence position when the rollback exceeds n_rs_seq), so common_context_seq_rm hits GGML_ABORT and the whole server dies: common/common.cpp:1472: failed to remove sequence 0 with p0=490, p1=-1 Observed in production with Qwen3.6-35B-A3B (GatedDeltaNet layers): any client re-sending an identical prompt with cache_prompt enabled (regenerate / retry) crashed the server. Reproducible with any recurrent model, e.g. mamba-130m, by sending the same /completion prompt twice. Fix: include the exact-match case (n_past == slot.prompt.n_tokens()) in the existing checkpoint-restore/full-reprocess branch by relaxing its condition to n_past <= slot.prompt.n_tokens() and extending pos_min_thold by 1 when there are no new tokens, so the upcoming back-off is accounted for. Recurrent/hybrid models now restore a context checkpoint (or fall back to full re-processing) instead of aborting; attention models are unaffected. This backports upstream ggml-org/llama.cpp PRs ggml-org#23280 and ggml-org#24110 (commits ccee426, 6f3a9f3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
|
Thank you @kmehan, this is a model fix: minimal diff, honest repro, and it backports the exact logic from upstream ggml-org#24110 (merged June 4), so it carries upstream's blessing. Verified the blast radius: full-attention models never reach the inner block (pos_min stays 0 below the threshold), so only SWA/hybrid/recurrent paths change, which is exactly the bug population. Worst case on a checkpoint-less recurrent model is a full reprocess instead of a dead server. Merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When an incoming prompt exactly matches a slot's cached tokens, the server backs off one token (
n_past--) to guarantee at least one token is decoded for logits. The subsequent truncation callsseq_rmwithp0 = n_past > 0, which recurrent memory cannot satisfy (the state can't be rewound to a mid-sequence position when the rollback exceedsn_rs_seq), socommon_context_seq_rmhitsGGML_ABORTand the whole server dies:Observed in production with Qwen3.6-35B-A3B (hybrid attention + GatedDeltaNet): any client re-sending an identical prompt with
cache_promptenabled (regenerate, retry,n_predict: 0scoring) crashes the server.Reproduction
Any recurrent model, e.g. mamba-130m Q8_0 on CPU:
llama-server -m mamba-130m.q8_0.gguf --port 8199 -c 2048 -np 1 # POST /completion {"prompt": "<~500-token text>", "n_predict": 0, "cache_prompt": true} — twice, identicalUnpatched, the second request aborts the server:
Fix
Include the exact-match case (
n_past == slot.prompt.n_tokens()) in the existing checkpoint-restore / full-reprocess branch by relaxing its condition ton_past <= slot.prompt.n_tokens()and extendingpos_min_tholdby 1 when there are no new tokens, so the upcoming one-token back-off is accounted for. Recurrent/hybrid models now restore a context checkpoint (or fall back to full re-processing) instead of aborting; attention models are unaffected.This is a faithful backport of upstream ggml-org/llama.cpp ggml-org#23280 (ccee426) and ggml-org#24110 (6f3a9f3); this fork's merge-base predates both.
Verification
Patched, same repro sequence (×3): HTTP 200 every time, server stays alive —
prompt_n=4(checkpoint restore; falls back ton_past = 0full reprocess when no checkpoint exists)prompt_n=21— prefix cache reuse unchangedprompt_n=1(unchanged one-token back-off), grown promptprompt_n=8, no checkpoint restores — no behavior change for non-recurrent models