Auto-load a matching mmproj so vision models just work

[GiuseppeCapaldo93]

Motivation

Multimodal (vision) GGUF models need a companion multimodal projector
(mmproj) passed via -mm. Today the projector has to be located and wired up
by hand; if it is forgotten the model loads without vision and there is no
obvious hint why.

Change

When a model is launched, look for an mmproj whose filename correlates with
the model file and pass it as -mm automatically:

• Searches the dedicated models/mmproj/ tree recursively (so a projector the
  downloader stored under models/mmproj/<repo-slug>/ is rediscovered after a
  reload) and models/ shallowly, recognizing mmproj / clip* / projector
  names.
• Correlates by a normalized key that ignores quantization/precision tokens and
  shard suffixes, so e.g. Nemotron-…-Q4_K_M.gguf matches
  mmproj-Nemotron-…-BF16.gguf.
• Requires exact normalized-key equality, so a close-but-different family version
  (e.g. -V-2.6 vs -V-2) or an unrelated model will not grab a stray
  projector.
• Uses the effective (last) model source, so a repeated -m or a later remote
  -hf/--model-url is honored rather than the first -m.
• Respects an explicit -mm / --mmproj / --mmproj-url / --no-mmproj, and
  only applies to llama-server / llama-cli / llama-mtmd-cli.
• Can be disabled with LLAMA_GUI_AUTO_MMPROJ=0.

Testing

• Added AutoMmprojTests covering key normalization, companion matching,
  false-positive avoidance, and the override/disable paths.
• Full backend test suite passes (486 tests).

[thomas9120]

P2 — Make automatic mmproj matching conservative

The feature direction looks good, but the current substring/length fallback can attach an incompatible same-family projector. For example:

• model MiniCPM-V-2.6-Q4_K_M.gguf normalizes to minicpmv26
• projector mmproj-MiniCPM-V-2-F16.gguf normalizes to minicpmv2

Because minicpmv2 is contained in minicpmv26 and is longer than eight characters, it is accepted for V2.6. Automatically attaching the wrong projector can make the model fail to load or leave vision behavior broken; in this case, declining to auto-match is safer.

A narrow fix would be:

1. Normalize common Qn_n suffixes such as Q4_0 and Q5_1 as complete quantization tokens. The current regex can consume only the Q4 portion and leave a trailing 0.
2. Accept a candidate only when its normalized key equals the model's normalized key; remove the substring/score fallback.
3. Add a positive regression test for Vision-Model-Q4_0 ↔ mmproj-Vision-Model-BF16, and a negative test proving MiniCPM-V-2.6 does not match mmproj-MiniCPM-V-2.

This keeps the existing directory scan, explicit -mm / --no-mmproj handling, tool scope, environment toggle, and launch integration unchanged. Exact normalized equality still supports the advertised quantization/precision-insensitive matches, while broader fuzzy matching can be added later if real filename cases demonstrate a need for it.

[GiuseppeCapaldo93]

Thanks for the detailed review — addressed in the latest push (fdfdcbc).

• Full Qn_n quant tokens: the quant regex now consumes the whole token, including legacy Q4_0 / Q5_1, so no trailing digit is left behind.
• Exact equality only: find_matching_mmproj now requires the model and mmproj filenames to reduce to the same normalized key; the substring/score fallback is removed, so an uncertain case declines to auto-match rather than attaching the wrong projector.
• Regression tests: added positive Vision-Model-Q4_0 ↔ mmproj-Vision-Model-BF16 and negative MiniCPM-V-2.6 vs mmproj-MiniCPM-V-2 (both at the _normalize_model_key level and end-to-end through find_matching_mmproj).

The directory scan, explicit -mm / --no-mmproj handling, tool scope, env toggle, and launch integration are unchanged. Full backend suite passes (489 tests).

[thomas9120]

Almost there,

Thanks — fdfdcbc resolves the previous concerns cleanly. The complete Q4_0/Q5_1 normalization, exact-key matching, and positive/negative regression tests all look good.

I found one remaining P2 edge case in compute_auto_mmproj_args(): it stops at the first local -m/--model argument, while the rest of Llama-GUI treats the last model-source flag as effective. Custom Launch Args intentionally permit duplicates and are placed before the selected UI model, so arguments such as:

-m models/old-Q4_K_M.gguf -m models/selected-Q4_K_M.gguf

can launch the selected model while auto-attaching the old model’s projector. Similarly, a later -hf or --model-url source could receive an unrelated local projector.

The narrow fix would be to reuse the existing helper:

model_flag, model_path = _last_launch_flag_entry(
    flat_launch_args, _MODEL_VALUE_FLAGS
)
if model_flag not in _LOCAL_MODEL_VALUE_FLAGS or not model_path:
    return []

[GiuseppeCapaldo93]

Thanks — all three points are addressed (f550445).

• Effective (last) model source: compute_auto_mmproj_args() now uses the existing precedence helper exactly as suggested —

  model_flag, model_path = _last_launch_flag_entry(flat_launch_args, _MODEL_VALUE_FLAGS)
  if model_flag not in _LOCAL_MODEL_VALUE_FLAGS or not model_path:
      return []

  so a repeated -m uses the last one and an effective remote -hf/--model-url suppresses auto-detection. Tests cover both (two local -m → last wins; trailing -hf → no projector).
• Recursive projector discovery: the dedicated models/mmproj/ tree is now scanned recursively (so a projector stored under models/mmproj/<repo-slug>/<file>.gguf is rediscovered after a reload), while models/ stays a shallow scan for manually placed files. Candidate recognition reuses the downloader's own is_mmproj_filename() (mmproj / clip* / projector). Added a regression test with a projector under models/mmproj/<repo-slug>/, plus one for clip/projector name recognition.
• Doc nit: the PR description now says "exact normalized-key equality" instead of "substantial name overlap".

Full backend suite passes (493 tests).

(For reference, the same review was also posted on #253 — these fixes live here on the mmproj branch.)

[thomas9120]

This one will take a little extra time on my end, I'm putting it on hold for a bit.

[thomas9120]

Thanks — all three points are addressed (f550445).

* **Effective (last) model source**: `compute_auto_mmproj_args()` now uses the existing precedence helper exactly as suggested —
  ```python
  model_flag, model_path = _last_launch_flag_entry(flat_launch_args, _MODEL_VALUE_FLAGS)
  if model_flag not in _LOCAL_MODEL_VALUE_FLAGS or not model_path:
      return []
  ```
  
  
      
        
      
  
        
      
  
      
    
  so a repeated `-m` uses the last one and an effective remote `-hf`/`--model-url` suppresses auto-detection. Tests cover both (two local `-m` → last wins; trailing `-hf` → no projector).

* **Recursive projector discovery**: the dedicated `models/mmproj/` tree is now scanned recursively (so a projector stored under `models/mmproj/<repo-slug>/<file>.gguf` is rediscovered after a reload), while `models/` stays a shallow scan for manually placed files. Candidate recognition reuses the downloader's own `is_mmproj_filename()` (`mmproj` / `clip*` / `projector`). Added a regression test with a projector under `models/mmproj/<repo-slug>/`, plus one for clip/projector name recognition.

* **Doc nit**: the PR description now says "exact normalized-key equality" instead of "substantial name overlap".

Full backend suite passes (493 tests).

(For reference, the same review was also posted on #253 — these fixes live here on the mmproj branch.)

Oops, didn't realize that happened.

[thomas9120]

Sadly this conflicts with changes that were in progress regarding model discoverability in subfolders. I'll have to close for now.