Skip to content

llama-cli: fix not copying preserved tokens - #24258

Merged
CISC merged 1 commit into
ggml-org:masterfrom
michaelw9999:fix-cli
Jun 14, 2026
Merged

llama-cli: fix not copying preserved tokens#24258
CISC merged 1 commit into
ggml-org:masterfrom
michaelw9999:fix-cli

Conversation

@michaelw9999

Copy link
Copy Markdown
Contributor

Overview

Llama-server will properly read preserved token IDs and copy chat_params.preserved_tokens into task.params.sampling.preserved_tokens. However, it appears this was left out of llama-cli which does not.
This PR fixes that.

I had just finished a new PR (about to post ) for the new Cohere2 MoE arch and BLS-Mini-Code-1.0 model, and kept fighting chat template issues repeatedly until I saw it only was happening in llama-cli testing, but was not a problem happening when trying via llama-server.
So some special tags, for example, <|END_THINKING|> were getting detokenized, so the model was never showing output when a response would start, leading to the response not showing. This only was in llama-cli.

Previously:

> Hi BLS

[Start thinking]
The user says "Hi BLS". Likely they are greeting. As an AI, we should respond politely. No policy issues. We can ask how can we help.Hello! How can I assist you with today?

was really because of: ... how can we help.<|END_THINKING|><|START_TEXT|>Hello! ...

After this fix:

> Hi BLS

[Start thinking]
The user says "Hi BLS". Likely they are greeting. As an AI, we should respond politely. There's no request for content. We can respond with a
[End thinking]

Hello! I'm Command, a Cohere-built language model. How can I assist you with your query today?

[ Prompt: 23.6 t/s | Generation: 195.3 t/s ]

Additional information`

Tested the fix with a few other models previously working and did not see any change in behavior.
test-chat, test-chat-auto-parser, test-chat-template, test-jinja, test-reasoning-budget, test-sampling all pass.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: Yes, used to find the real root cause of the issue and propose a fix.

@michaelw9999
michaelw9999 requested a review from ngxson as a code owner June 7, 2026 09:24
@michaelw9999 michaelw9999 changed the title Fix CLI not copying preserved tokens llama-cli: fix not copying preserved tokens Jun 7, 2026
@CISC
CISC merged commit 6e14286 into ggml-org:master Jun 14, 2026
24 of 25 checks passed
@CISC

CISC commented Jun 14, 2026

Copy link
Copy Markdown
Member

@michaelw9999 Thanks for the reminder.

kinchahoy pushed a commit to kinchahoy/llama.cpp that referenced this pull request Jun 16, 2026
am17an pushed a commit to am17an/llama.cpp that referenced this pull request Jun 20, 2026
adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants