Add Vision Support for Minimax-M3

[timkhronos]

Overview

Implement MiniMax-M3 vision support. The vision tower itself is a Qwen2.5-VL style ViT (now reuses build_vit). The major differences are that M3 uses a 3-axis (T/H/W) RoPE, a gate-less GELU-erf FFN and a two-stage patch-merge projector.

Stacked on #24908, so the full diff carries the MSA base until that merges. The vision-only changes are [here]

Additional information

The preprocessing matches Qwen2.5-VL's. Further, in the graph the summed-temporal-Conv2D patch embed, the 2×2 spatial-merge reorder, separate biased q/k/v attention, and pre-LN should also match.

Expanding a bit on the differences, the most substantial one is the 3-axis RoPE. The 3 bands are laid out as cat([f,f]) with a HF split half pairing and an axis_dim keyed frequency schedule. I don't think this can reuse the existing qwen ggml_rope_multi and the ggml_rope_type_vision, as the existing op can't express it without a q/k weight permute at conversion plus a vision mode that doesn't exist. The graph-level cos/sin matches HF directly, and uses the same approach build_rope_2d already uses for the 2-axis vision rope, generalized to 3 axes. T is the temporal axis, and for still images it's coordinate 0, but the layout should stay so H/W keep the same channels as HF.

Vision MLP is a plain GELU-erf, while qwen2.5vl uses a gated FFN.

The projector itself is a two-stage projector. Uses per patch MLP (mm.1 / mm.2), 2×2 group concat, then merge MLP (mm.merge.fc1 / fc2), both using GELU-erf, while qwen uses a single post-merge MLP.

There is also no post-layer norm and no window attention, only pre_layernorm.

Validation

The metrics below are for the pre build_vit change. Will retest.
Generated vision embeddings vs the HF reference on an identical sample image:

shape : 256 tokens x 6144 embd
overall cosine : 0.999949
per-token cosine: mean=0.999454 min=0.963887 (worst token 95)
relative L2 err : 0.010137
abs err : mean=0.03815 max=15.16844
(The high max-abs is most likely a single high-magnitude channel; cosine and relative-L2 are the
embedding-level metrics.)

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES.

AI assistance disclosure

AI assistance was used during development, but the code is not an unreviewed AI-generated code drop.

Scope of AI assistance:

• Helped write and debug a small local python comparison script, that was used to dump MiniMax-M3-VL vision embeddings from the Hugging Face implementation for parity checks.
• Helped review and organize explanations of the implementation while I was debugging.
• Helped reason through code comments and possible wording.
• The submitted llama.cpp implementation was not accepted blindly from AI output.
• I can explain the implemented code paths and the MiniMax/HF behavior they are matching.

If a stricter or differently formatted disclosure is preferred, please specify the exact wording/fields expected.

[ngxson]

feels like > 90% of the code is AI-generated, do you really understand it?

[ngxson]

A standard ViT backbone (separate biased q/k/v/o, LayerNorm, GELU MLP, full bidirectional attention, no mask, no windowing) that diverges from vanilla CLIP in four ways:

* **Conv3D patch embed, run as summed Conv2D slices.** The HF model uses a Conv3D   patch embedding with `temporal_patch_size` slices, conversion splits the 5D weight into per-slice Conv2D kernels (`V_ENC_EMBD_PATCH` + `.weight.{t}`) and the graph sums the outputs. Exact for still images (video out of scope). No patch-embed bias (asserted).

* **Custom 3-axis (T/H/W) RoPE.** `axis_dim = 26`, `rope_dim = 3·26 = 78`, applied to the first 78 channels of each head with HF `rotate_half` semantics, tail passed through. Cos/sin are host-precomputed and fed as graph inputs (`minimax_cos`/`minimax_sin`). Since `rope_dim (78) < d_head (80)` this is partial rotary,  same pattern as the text tower, 3-axis.

* **2×2 spatial-merge token reduction.** Patches are reordered raster -> block (matching the HF flatten) and merged 2×2, so the projector consumes groups of 4. `spatial_merge_size` is emitted in conversion.

* **No class token, no absolute position table, no post-layernorm.** A `pre_layernorm` only; sinks / abs-pos / class-embedding all absent and asserted null.

@timkhronos did you even read what AI generates here? and ask yourself how wrong it is?

there is no "vanilla CLIP" in mtmd, this model is just qwen-vl with some subtle differents

[ngxson]

I refuse to proceed with this PR until you being honest about AI usage

[timkhronos]

@ngxson I expanded on the AI disclosure in the PR description, to cover precisely what AI was used to assist with.

[timkhronos]

For anyone testing, the latest changes require the mmproj file to be reconverted.

I uploaded one here.
Alternatively you can also convert a fresh one with this pr, without downloading the full model weights by using:

python convert_hf_to_gguf.py MiniMaxAI/MiniMax-M3 --mmproj --remote --outtype bf16 --outfile Minimax-mmproj.gguf