Eval bug: Server Fails with HTTP 400 (Context Size Exceeded) Instead of Truncating Chat History

[lyesrock]

Name and Version

/home/sysops/llama.cpp/build/bin/llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: Tesla P40, compute capability 6.1, VMM: yes
Device 1: Tesla P40, compute capability 6.1, VMM: yes
version: 7062 (9b17d74)
built with gcc-14 (Ubuntu 14.2.0-19ubuntu2) 14.2.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

2 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Device 0 [Tesla P40] PCIe GEN 3@ 8x
Device 1 [Tesla P40] PCIe GEN 3@16x

Models

hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Problem description & steps to reproduce

When using the llama.cpp server (via the OpenAI-compatible API, /v1/chat/completions), if the total number of tokens in the prompt exceeds the --ctx-size limit, the server stops processing the request and returns an HTTP 400 Bad Request error with the message "the request exceeds the available context size, try increasing it."

I expected the server, especially when using the --context-shift flag, to handle the context limit by automatically truncating/shifting the oldest messages in the conversation history, similar to how other platforms (like Ollama, which uses a client-side approach) manage chat history for long contexts.

Steps to Reproduce

1. Setup the Server: Run llama.cpp using the following command (with a model, e.g., a Qwen model):

   "$LLAMA_SERVER_PATH" \
     --host 0.0.0.0 \
     --port 10000 \
     --threads -1 \
     --ctx-size 131072 \
     --context-shift \
     --alias "Qwen3-Coder-30B-A3B" \
     -m "$MODEL_PATH" \
     --api-key 1234567890 \
     --jinja \
     --temp 0.7 \
     --min-p 0.01 \
     --top-p 0.80 \
     --top-k 20 \
     --repeat-penalty 1.05 \
     --flash-attn on \
     --batch-size 4096 \
     --ubatch-size 2048 \
     --threads 32 \
     --metrics

2. Send Long Chat Requests: Continuously send chat completion requests to http://0.0.0.0:10000/v1/chat/completions using a client (like a custom application running on VSCode/Cline) that sends the full conversation history with each request.

3. Observe Failure: Once the cumulative token count of the conversation history (task.n_tokens) exceeds the configured limit (--ctx-size 131072), the server fails the request.

Observed Log Output (Server Side)

The server log shows the failure when the request size exceeds the context:

slot update_slots: id  1 | task 21 | new prompt, n_ctx_slot = 131072, n_keep = 15000, task.n_tokens = 133046
srv    send_error: task id = 21, error: the request exceeds the available context size, try increasing it
slot      release: id  1 | task 21 | stop processing: n_tokens = 0, truncated = 0
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 10.2.0.153 400

Expected Behavior

The server should either:

1. Automatically truncate the oldest messages (excluding the system prompt and the final user message) to fit the context window (--ctx-size 131072) and process the request successfully.
2. Or, if --context-shift is intended for this use case, it should be enabled and working to manage the context window without returning a 400 error.

First Bad Commit

No response

Relevant log output

srv  log_server_r: request: POST /v1/chat/completions 10.2.0.153 400
srv  log_server_r: request: GET /v1/models 10.2.0.153 200
srv  params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id  3 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.978
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id  3 | task 12 | processing task
slot update_slots: id  3 | task 12 | new prompt, n_ctx_slot = 131072, n_keep = 15000, task.n_tokens = 314
slot update_slots: id  3 | task 12 | need to evaluate at least 1 token for each active slot (n_past = 314, task.n_tokens() = 314)
slot update_slots: id  3 | task 12 | n_past was set to 313
slot update_slots: id  3 | task 12 | n_tokens = 313, memory_seq_rm [313, end)
slot update_slots: id  3 | task 12 | prompt processing progress, n_tokens = 314, batch.n_tokens = 1, progress = 1.000000
slot update_slots: id  3 | task 12 | prompt done, n_tokens = 314, batch.n_tokens = 1
slot print_timing: id  3 | task 12 |
prompt eval time =      21.27 ms /     1 tokens (   21.27 ms per token,    47.01 tokens per second)
       eval time =     134.18 ms /     8 tokens (   16.77 ms per token,    59.62 tokens per second)
      total time =     155.45 ms /     9 tokens
slot      release: id  3 | task 12 | stop processing: n_tokens = 321, truncated = 0
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 10.2.0.153 200
srv  params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  1 | task -1 | sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id  1 | task 21 | processing task
slot update_slots: id  1 | task 21 | new prompt, n_ctx_slot = 131072, n_keep = 15000, task.n_tokens = 133046
srv    send_error: task id = 21, error: the request exceeds the available context size, try increasing it
slot      release: id  1 | task 21 | stop processing: n_tokens = 0, truncated = 0
srv  update_slots: no tokens to decode
srv  update_slots: all slots are idle
srv          stop: cancel task, id_task = 21
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 10.2.0.153 400

[lyesrock]

Hi

I've tested against an older build, and it turns out the behavior I expected was actually working before:

• Working Build (No 400): b6721 (56b479584)
• Failing Build (Current): 7062 (9b17d74)

In build b6721, when I send an oversized chat history via the /v1/chat/completions API (e.g., $133 \text{K}$ tokens vs. --ctx-size 131072), the server handles it correctly: it automatically truncates the old context to make space. This is the behavior we expect from --context-shift.

However, in recent versions, this exact scenario returns an HTTP 400 Bad Request.

🤔 My questions regarding the implementation:

The Right Flag: Is --context-shift still the correct flag for the server to handle chat size automatically, or is there a new/replacement flag for this?

We heavily rely on this server-side context management, so any clarification on the current state and future plans would be greatly appreciated!

Thanks for your work!

[hennyg888]

I can confirm I'm also experiencing the same bug on latest release b7210.

[hosseinalipour]

Same issue, any solutions?

[hennyg888]

I just rolled back to working build b6721 as reported by original issue reporter.

[maglat]

There is no fix in the newest builds yet? I am on the build b7364 and get the context error as well

srv log_server_r: request: POST /v1/chat/completions 172.17.0.3 200 srv log_server_r: request: GET /v1/models 172.17.0.3 200 srv params_from_: Chat format: GPT-OSS slot get_availabl: id 1 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.771 slot launch_slot_: id 1 | task -1 | sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist slot launch_slot_: id 1 | task 1178 | processing task slot update_slots: id 1 | task 1178 | new prompt, n_ctx_slot = 131072, n_keep = 0, task.n_tokens = 773 slot update_slots: id 1 | task 1178 | need to evaluate at least 1 token for each active slot (n_past = 773, task.n_tokens() = 773) slot update_slots: id 1 | task 1178 | n_past was set to 772 slot update_slots: id 1 | task 1178 | n_tokens = 772, memory_seq_rm [772, end) slot update_slots: id 1 | task 1178 | prompt processing progress, n_tokens = 773, batch.n_tokens = 1, progress = 1.000000 slot update_slots: id 1 | task 1178 | prompt done, n_tokens = 773, batch.n_tokens = 1 slot print_timing: id 1 | task 1178 | prompt eval time = 9.35 ms / 1 tokens ( 9.35 ms per token, 106.97 tokens per second) eval time = 2511.94 ms / 268 tokens ( 9.37 ms per token, 106.69 tokens per second) total time = 2521.29 ms / 269 tokens slot release: id 1 | task 1178 | stop processing: n_tokens = 1040, truncated = 0 srv update_slots: all slots are idle srv log_server_r: request: POST /v1/chat/completions 172.17.0.3 200 srv log_server_r: request: GET /v1/models 172.17.0.3 200 srv params_from_: Chat format: GPT-OSS slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1 slot launch_slot_: id 0 | task -1 | sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist slot launch_slot_: id 0 | task 1447 | processing task slot update_slots: id 0 | task 1447 | new prompt, n_ctx_slot = 131072, n_keep = 0, task.n_tokens = 802458 srv send_error: task id = 1447, error: the request exceeds the available context size, try increasing it slot release: id 0 | task 1447 | stop processing: n_tokens = 0, truncated = 0 srv update_slots: no tokens to decode srv update_slots: all slots are idle srv stop: cancel task, id_task = 1447 srv update_slots: all slots are idle srv log_server_r: request: POST /v1/chat/completions 172.17.0.3 400

[Tom-Neverwinter]

i am experiencing the same issue

[Mo-Hashem]

same here o_O

[Iris-Colt-IF]

Same issue... It's annoying...

[AiAiCaptainn]

I was getting similar on the router and now it doesn't even say the context size exceeded, with verbose logging i'm just getting

srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request: POST /v1/completions 127.0.0.1 400
srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request:   127.0.0.1 400
srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request: POST /v1/completions 127.0.0.1 400
srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request:   127.0.0.1 400
srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request: POST /v1/completions 127.0.0.1 400
srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request:   127.0.0.1 400
srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request: POST /v1/completions 127.0.0.1 400
srv  log_server_r: request:
srv  log_server_r: response:
srv  log_server_r: request:   127.0.0.1 400

etc...

funny thing is I checked and I should be well below the context size.. 8192 context size and by my estimate i'm using about 3000 tokens, so the error doesn't even make sense for me

[Iris-Colt-IF]

I was getting similar on the router and now it doesn't even say the context size exceeded, with verbose logging i'm just getting

Same here. I am far from exceeding the 16384 context size but still triggering this infuriating bug with Qwen3-VL-32B-thinking.

[ggerganov]

launch_slot_: id 0 | task 1447 | processing task slot update_slots: id 0 | task 1447 | new prompt, n_ctx_slot = 131072, n_keep = 0, task.n_tokens = 802458 srv send_error: task id = 1447, error: the request exceeds the available context size, try increasing it slot

You are trying to process a 800k prompt with a 128k context. Whatever logic we implement, it is just not going to work for this case.

Besides, context truncation is the job of the client, not the server. The old logic for auto-truncating in llama-server had various side-effects, so it was removed at some point. Modern tools/client perform context compaction when necessary - this is the recommended approach.

Same here. I am far from exceeding the 16384 context size but still triggering this infuriating bug with Qwen3-VL-32B-thinking.

This sounds like a different issue. Provide logs.

[MartinEmrich]

Besides, context truncation is the job of the client, not the server.

I experience this, in the built-in webapp of llama-server. I found not option to truncate in the settings, or another way to continue the conversation.

[donington]

I think I might have found the culprit for this one. Comparing the differences between update_slot() on the current versus b6721, it appears that a bunch of the server stuff was pulled into a separate file.

// current: tools/server/server-context.cpp:1977
            if (slot.state == SLOT_STATE_GENERATING && slot.prompt.n_tokens() + 1 >= slot.n_ctx) {
                if (!params_base.ctx_shift) {
                    /* ... */
                }

// line 2210:
                            if (slot.task->n_tokens() >= slot.n_ctx) {
                                send_error(slot,
                                           string_format("request (%d tokens) exceeds the available context size (%d "
                                                         "tokens), try increasing it",
                                                         slot.task->n_tokens(), slot.n_ctx),
                                           ERROR_TYPE_EXCEED_CONTEXT_SIZE);
                                slot.release();
                                continue;
                            }

vs the working build:

// b6721: tools/server/server.cpp:3281
            if (slot.is_processing() && slot.n_past + 1 >= slot.n_ctx) {
                if (!params_base.ctx_shift) {
                    /* ... */
                }

// line 3434:
                            if (!params_base.ctx_shift) {
                                // if context shift is disabled, we make sure prompt size is smaller than KV size
                                // TODO: there should be a separate parameter that control prompt truncation
                                //       context shift should be applied only during the generation phase
                                if (slot.n_prompt_tokens >= slot.n_ctx) {
                                    slot.release();
                                    send_error(slot, "the request exceeds the available context size. try increasing the context size or enable context shift", ERROR_TYPE_EXCEED_CONTEXT_SIZE);
                                    continue;
                                }
                            }

Noteworthy is that following that last section there was another truncate step if the tokens overflow the context in the old version. I think that's the major difference here.

I managed to get it to function properly again with some local edits to the server-context.cpp in the update_slots function. Basically I migrated the context shift into a function so that it could be run after the entire prompt was processed, managing an edge case where the context shift never fires if the context window overflows on the last step. I also readded the disabled context shift check on the error regarding available context size, since that only makes sense if context shift isn't happening.

I was still getting errors about inconsistent sequence positions error during the sampler stage, which I tracked down to llama-batch. To get around it's sanity checks I had to add a call to update the batch positions in the context shift process. That was a total shot in the dark guess on my part, so if anyone more familiar with the batching can verify if there needs to be a safety check or not that would be great.

Attached is a git diff if anyone wants to try it and see if it works for them as well.

server-context.cpp diff

diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
index 73cb4c75..446bfcb7 100644
--- a/tools/server/server-context.cpp
+++ b/tools/server/server-context.cpp
@@ -1971,16 +1971,23 @@ private:
             queue_tasks.post(std::move(task));
         }
 
-        // apply context-shift if needed
-        // TODO: simplify and improve
-        for (server_slot & slot : slots) {
-            if (slot.state == SLOT_STATE_GENERATING && slot.prompt.n_tokens() + 1 >= slot.n_ctx) {
+        auto try_context_shift = [&](auto const& params_base, auto& slot) {
+            switch (slot.state) {
+                case SLOT_STATE_PROCESSING_PROMPT:
+                case SLOT_STATE_DONE_PROMPT:
+                case SLOT_STATE_GENERATING:
+                    break;
+                default:
+                    return;
+            }
+
+            if (slot.prompt.n_tokens() + 1 >= slot.n_ctx) {
                 if (!params_base.ctx_shift) {
                     // this check is redundant (for good)
                     // we should never get here, because generation should already stopped in process_token()
                     send_error(slot, "context shift is disabled", ERROR_TYPE_SERVER);
                     slot.release();
-                    continue;
+                    return;
                 }
 
                 if (mctx) {
@@ -1992,7 +1999,7 @@ private:
                 if (slot.task->is_parent() || slot.task->is_child()) {
                     send_error(slot, "context shift cannot be used for shared prompt", ERROR_TYPE_SERVER);
                     slot.release();
-                    continue;
+                    return;
                 }
 
                 // Shift context
@@ -2026,10 +2033,21 @@ private:
 
                     slot.prompt.tokens.clear();
                     slot.prompt.tokens.insert(new_tokens);
+
+                    // update the batch positions, or we get an error about sequences being out of order
+                    for (int i = 0; i < batch.n_tokens; ++i) {
+                        batch.pos[i] -= n_discard;
+                    }
                 }
 
                 slot.truncated = true;
             }
+        };
+
+        // apply context-shift if needed
+        // TODO: simplify and improve
+        for (server_slot & slot : slots) {
+            try_context_shift(params_base, slot);
         }
 
         // start populating the batch for this iteration
@@ -2205,14 +2223,16 @@ private:
                                 continue;
                             }
                         } else {
-                            if (slot.task->n_tokens() >= slot.n_ctx) {
-                                send_error(slot,
-                                           string_format("request (%d tokens) exceeds the available context size (%d "
-                                                         "tokens), try increasing it",
-                                                         slot.task->n_tokens(), slot.n_ctx),
-                                           ERROR_TYPE_EXCEED_CONTEXT_SIZE);
-                                slot.release();
-                                continue;
+                            if (!params_base.ctx_shift) {
+                                if (slot.task->n_tokens() >= slot.n_ctx) {
+                                    send_error(slot,
+                                               string_format("request (%d tokens) exceeds the available context size (%d "
+                                                             "tokens), try increasing it",
+                                                             slot.task->n_tokens(), slot.n_ctx),
+                                               ERROR_TYPE_EXCEED_CONTEXT_SIZE);
+                                    slot.release();
+                                    continue;
+                                }
                             }
 
                             if (slot.task->params.cache_prompt) {
@@ -2532,6 +2552,10 @@ private:
 
                     // entire prompt has been processed
                     if (slot.prompt.n_tokens() == slot.task->n_tokens()) {
+                        // if using context shift but the last step overflowed, we need to do another pass
+                        if (params_base.ctx_shift && slot.prompt.n_tokens() + 1 >= slot.n_ctx)
+                            try_context_shift(params_base, slot);
+
                         slot.state = SLOT_STATE_DONE_PROMPT;
 
                         GGML_ASSERT(batch.n_tokens > 0);