Ollama reloads the model upon each request

[lhofhansl]

Issue

I noticed that Ollama re-loads the model upon every single request.

Version and model info

Aider v0.86.1

I can see at the Ollama server that the model is reloaded at every request.
This is probably due to varying context sizes.

        num_ctx = int(self.token_count(messages) * 1.25) + 8192

This really slows down usage in Ollama.
I have not been able to set the context window via .aider.model.settings.yml
num_ctx probably comes from litellm, it would be nice if there was a way in Aider to just set it, to avoid reloading of the model each time.

[datadriftershow]

why not ask aider ?

You know aider has a self-service help mode.

> /help how to configure the context window of my active mode                                              11:42:48 [23/2616]
> ls ?                                                                                                                       
                                                                                                                             
Loading llama_index.core.storage.kvstore.simple_kvstore from /home/mentat/.aider/caches/help.0.86.2.dev53+gbb21fc03f/docstore
.json.                                                                                                                       
Loading llama_index.core.storage.kvstore.simple_kvstore from /home/mentat/.aider/caches/help.0.86.2.dev53+gbb21fc03f/index_st
ore.json.                                                                                                                    
Warning: it's best to only add files that need changes to the                                                                
chat.                          
https://aider.chat/docs/troubleshooting/edit-errors.html      
Based on the provided documentation for aider, you can        
configure the context window for your models in a few ways,   
primarily for models that aider doesn't know about by         
default.                       

Here's a summary of the methods:                              

1. For Ollama Models (Special Case)                           

Ollama defaults to a very small 2k context window. aider      
helps by automatically setting it to be large enough for your 
request plus 8k tokens for the reply.                         

If you want to set a fixed context window for an Ollama                                                                      
model, you can use a .aider.model.settings.yml file:                                                      
                                                                                                          
                                                                                                          
- name: ollama/qwen2.5-coder:32b-instruct-fp16                
  extra_params:                                               
    num_ctx: 65536                                                                                        
                                                                                                          
                          
2. For Any Unknown Model       
                                                                                                          
For models that aider has never heard of (and shows a warning                                             
like "Unknown context window size and costs, using sane                                                   
defaults"), you can provide the metadata using a                                                          
.aider.model.metadata.json file.                                                                          
                                                                                                          
This file should be placed in one of these locations (in                                                  
order of priority):                                                                                       
                                                                                                          
 • Your home directory.                                                                                   
 • The root of your git repo.                                                                             
 • The current directory where you launch aider.                                                          
 • Or specify a file with --model-metadata-file <filename>.                                               
                                                                                                          
The JSON file should look like this:                                                                      
                                                                                                          
                                                                                                          
{                                                                                                         
    "deepseek/deepseek-chat": {                                                                           
        "max_tokens": 4096,                                                                               
        "max_input_tokens": 32000,                            
        "max_output_tokens": 4096,                                                                        
        "input_cost_per_token": 0.00000014,                   
        "output_cost_per_token": 0.00000028,                  
        "litellm_provider": "deepseek",                       
        "mode": "chat"                                        
    }                                                         
}                                                             
                                                              
                                                              
Important: Use the fully qualified model name with the        
provider/ prefix (e.g., deepseek/deepseek-chat), which should 
match the litellm_provider field.                             

3. For Known Models (Overriding Defaults)                     

You can also override or add settings for any model, even     
ones aider knows about, using a .aider.model.settings.yml     
file.                          

This file is also searched for in the same locations as the   
metadata file. The YAML file should be a list of              
dictionaries, one for each model:                             

                                                              
- name: openai/gpt-4o                                         
  edit_format: diff                                           
  weak_model_name: openai/gpt-4o-mini                         
  use_repo_map: true                                          
  extra_params:                                               
    max_tokens: 8192                                          
                                                              

 • https://aider.chat/docs/config/adv-model-settings.html     
 • https://aider.chat/docs/llms/ollama.html                   

Good luck. I think you will be able to unblock yourself.

[lhofhansl]

That's actually what I tried first...

I have not been able to set the context window via .aider.model.settings.yml

I could have been more specific.

And maybe there's another reason why the model is reloaded each time.
When I use litellm directly, Ollama does not reload the model (unless I change the context size).

Update: Turns out the correct way to specify this now is:

- name: ollama_chat/qwen2.5-coder:32b-instruct-fp16                
  extra_params:                                               
    num_ctx: 65536  

(notice ollama_chat)

I'll close this now,