server : host-memory prompt caching

[ggerganov]

target #16440
rel #16117

Initial version of automatic memory offloading to host memory using an extended logic for minimizing the prompt reprocessing. The host-memory prompt cache acts as "extra slots" with which we can calculate prefix similarity and decide to hot-swap them into the llama_context if it would reduce the processing. The cache is stored in regular RAM.

The RAM size that is used for caching prompts has 2 limits:

• Max size in bytes (controlled with new --cache-ram, -cram CLI arg)
• Max number of cached tokens (by default, equal to --context-size)

The server logs provide detailed prompt cache information each time the cache is updated:

• A small QoL improvement is that update_slots() now also logs the old and new prompt for each task around n_past (up to 10 tokens) so we can have a better understanding what caused the particular choice of the n_past value for the new task.

• Setting LLAMA_SERVER_SLOTS_DEBUG=1 env will make the /slots endpoint output a more detailed output containing the prompt and the generated text of the current or last task. This is useful for debugging purposes.

Note: mtmd workarounds are starting to cause some headaches. For example server_tokens is not copyable which complicates the cache logic and makes the prompt caching feature incompatible with mtmd.

Usage

# use 8192 MiB of host RAM for caching prompts
llama-server ... --cache-ram 8192

# use as much host RAM is available (i.e. no limit)
llama-server ... -cram -1

# disable prompt caching in RAM
llama-server ... -cram 0

Server refactor

• Replace server_slot members with a single server_task
• Remove server_slot.n_predict
• Remove prompt truncation logic (obsolete and not useful anymore)
• slot.task is now const ptr to reflect that the task parameters should not change when it is passed to the slot
• Bump default context checkpoints from 3 to 8

TODOs

• Set memory limit for the host-memory cache from CLI
• Clean-up implementation
• Test with agentic workflows
• Multi-slot tests
• Fix progress report

[ggerganov]

Looking for some feedback of how this new logic performs in different use cases. I've been testing it with the llama.vscode agent and it significantly improves the experience since we can now use a single server slot without trashing the prompt cache.

The current implementation should work with any model (dense, MoE, SWA, SSM, etc.). I think the default settings should be good for most use cases, though we'll probably add some options to adjust cache limits if needed.

Pay attention to these new messages in the logs:

Interested in testing agentic use cases, such as Claude Code and similar, where we have a single large context with various auxilary calls (keyword extraction, summarization, etc.) interleaved. The expectation is that prompt reprocessing should be significantly reduces in such cases.

[dark-penguin]

Fantastic work indeed! Now local orchestrators can actually be usable!

The RAM size that is used for caching prompts has 2 limits:

• Max size in bytes (controlled with new --cache-ram, -cram CLI arg)
• Max number of cached tokens (by default, equal to --context-size)

What argument can I use to make "max number of cached tokens" larger than "by default, equal to --context-size"?

Just to make sure I understand everything correctly:

• Let's say I have 9GB KV cache (96k tokens) that fits in VRAM, I have 64GB RAM, and I'm making 10k-sized prompts (so, a little under 1GB for each prompt).
• By default, I can have 8 prompts cached, because 8GB is the default -cram size
• With -cram -1, I can have 9 prompts cached, because now we are limited by "--context-size 96000"
• If I can find which argument to use to remove that limit, I can have over 60 prompts cached, as long as I have enough free RAM

Is this all correct?