Add arch support for cohere2-MoE

[michaelw9999]

Overview

There's a new early preview of CohereLab's North-Mini-Code-1.0 MoE coding model with weights that I wanted to test and add full implementation for.

So, this PR adds new arch cohere2_moe in addition to the already implemented cohere2 arch.

The models are not finalized, but I have converted from HF SFT into preliminary first-gen models:

• BF16, Q4_K, and NVFP4 quantizations posted at michaelw9999/North-Mini-Code-1.0, which is a 30B-A3B model.
• Q4_K was converted using an imatrix based off wikitrain.
• The NVFP4 version was quantized with my tool separately, including creating an NVFP4 MTP head.

This model is incredibly fast at creating code and I was surprised by what it could do, and look forwarding to testing out more.

• Conversion from HF to BF16 gguf succeeds with the convert_hf_to_gguf script.
• Llama-quantize will successfully convert the BF16.gguf into a quantized model
• Llama-cli will show thinking properly with llama-cli: fix not copying preserved tokens #24258
• Supports optional MTP
• NVFP4 weight and input scales(future use) are attached; also fixed this in cohere2.cpp while we're here
• Tested the model is working across extended multiple turns on BF16, Q4_K, and NVFP4.
• All arch wiring and semantics are linked up properly as far as I can tell in the normal llama.cpp convention

Test-llama-arches passes:

|     cohere2_moe|       NVIDIA GeForce RTX 5090|   MoE|  OK (8.58e-08)|       OK|
|     cohere2_moe|Intel(R) Core(TM) Ultra 9 285K|   MoE|  OK (0.00e+00)|       OK|
|     cohere2_moe|                          Meta|   MoE|  OK (8.58e-08)|     SKIP|

The model is fast:

  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
| model                          |       size |     params | backend    | ngl |   test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| cohere2_moe 30B.A3B NVFP4      |  16.15 GiB |    30.48 B | CUDA       |  -1 |  pp512 |     11120.91 ± 47.90 |
| cohere2_moe 30B.A3B NVFP4      |  16.15 GiB |    30.48 B | CUDA       |  -1 |  tg128 |        284.04 ± 4.20 |

Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
| model                          |       size |     params | backend    | ngl |   test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| cohere2_moe 30B.A3B Q4_K - Medium |  17.44 GiB |    30.48 B | CUDA       |  -1 |     pp512 |      9805.59 ± 94.17 |
| cohere2_moe 30B.A3B Q4_K - Medium |  17.44 GiB |    30.48 B | CUDA       |  -1 |     tg128 |        324.19 ± 2.84 |

Tool calling with build in tools is working through llama-server:

Tell me all about the tools you can use!

Hello! I'd be happy to tell you about the tools I have available. Here are the main functions I can use to help you:

File System Operations
read_file: Read contents of files with optional line ranges and line number prefixes
write_file: Create new files or write content to existing files
edit_file: Make precise edits to files using line-based operations (replace, delete, append)
apply_diff: Apply unified diffs to files for complex changes
file_glob_search: Recursively search for files matching glob patterns
Text Processing
grep_search: Search for regex patterns in files under a path
exec_shell_command: Execute shell commands and capture output
System Information
get_datetime: Get current date and time
Capabilities Summary:
File Management: Create, read, edit, and search files
Content Editing: Precise line-based modifications
Pattern Matching: Regular expression searches
System Operations: Execute commands and get system info
Search: Find files by patterns and search within files
These tools work together to help you with various programming, file management, and text processing tasks. Each tool has specific parameters and options that allow for flexible usage.

What specific task would you like to work on using these tools?

Hey, what's the time today?

get_datetime
Toggle content
The current date and time is Sunday, June 7, 2026 at 4:01:43 AM.

Additional information

The models are currently experimental and in development, the creators are seeking community feedback to shape its future, which is the reason I have created this PR!

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure:
  Yes, used to help code fast and identify how to create the arch, then debug issues, especially with chat parser and tokenizer incompatabilities. De-slop/adjust/review/quant/convert/full testing done manually.

[pwilkin]

@CISC FWIW I feel this implementation looks much cleaner: csabakecskemeti@7b08a0e

[CISC]

@CISC FWIW I feel this implementation looks much cleaner: csabakecskemeti@7b08a0e

Ah, wasn't aware there already was a MoE. The conversion doesn't look completely compatible though.

A few more tweaks here should bring both in line I think.

[michaelw9999]

@CISC FWIW I feel this implementation looks much cleaner: csabakecskemeti@7b08a0e

Well shucks, I didn't see any other version had been done before. I'll still try to get this cleaned up ASAP

[arki05]

Ha, I guess someone beat me to the punch ^^, was working on my own impl of this on arki05/llama.cpp-fork.

Instead of opening a rival pr; i assumed i should just diff our impls and suggest fixes/learn from yours. during that i noticed some discrepancies between our implementations; lots of tiny different choices , but i think these following few differences are worth noting on.

I'm happy to help fixing / collaborating on this pr; looking forward to using this model ^^

[michaelw9999]

Ha, I guess someone beat me to the punch ^^, was working on my own impl of this on arki05/llama.cpp-fork.

Instead of opening a rival pr; i assumed i should just diff our impls and suggest fixes/learn from yours. during that i noticed some discrepancies between our implementations; lots of tiny different choices , but i think these following few differences are worth noting on.

I'm happy to help fixing / collaborating on this pr; looking forward to using this model ^^

Appreciate all the fixes, thank you! I've put them all in place. Would be great to get another test confirmation. HF repo Q4K/NVFP4 new versions with new arch name just finished up.

[coder543]

The model has officially been released now: https://huggingface.co/CohereLabs/North-Mini-Code-1.0

[sdroege]

Just wanted to add here that this works great with the Vulkan backend. prompt processing is slower than with other models of the ~same size but still fast enough.

[michaelw9999]

Pushed one more change. Cohere needed build_attn_inp_kv_iswa() for MTP and was crashing with build_attn_inp_kv() since of course it is sliding window attention type. I doubt anyone has any MTP version yet but worth fixing ahead of time. I never pushed the MTP model up because my GPU couldn't keep up trying to train it more, only was getting about 7-10% acceptance and it was slowing it down significantly.

[imnecho]

I am getting some errors here while trying to use the llama-b9626 with the https://huggingface.co/DevQuasar/CohereLabs.command-a-plus-05-2026-bf16-GGUF

0.00.619.476 I llama_model_loader: - type f32: 65 tensors
0.00.619.476 I llama_model_loader: - type q4_K: 272 tensors
0.00.619.477 I llama_model_loader: - type q6_K: 49 tensors
0.00.619.478 I print_info: file format = GGUF V3 (latest)
0.00.619.479 I print_info: file type = Q4_K - Medium
0.00.619.483 I print_info: file size = 123.17 GiB (4.85 BPW)
0.00.623.681 E llama_model_load: error loading model: unknown model architecture: 'cohere2-moe'
0.00.623.796 E llama_model_load_from_file_impl: failed to load model
0.00.623.800 E common_init_from_params: failed to load model 'd:\AI_Models\CohereLabs.command-a-plus-05-2026-bf16.Q4_K_M-00001-of-00009.gguf'
0.00.623.814 E srv load_model: failed to load model, 'd:\AI_Models\CohereLabs.command-a-plus-05-2026-bf16.Q4_K_M-00001-of-00009.gguf'
0.00.623.818 I srv operator(): operator(): cleaning up before exit...
0.00.624.883 E srv llama_server: exiting due to model loading error

[CISC]

I am getting some errors here while trying to use the llama-b9626 with the https://huggingface.co/DevQuasar/CohereLabs.command-a-plus-05-2026-bf16-GGUF

It helps to read the model card:

100% vibe coded support. You need to use this branch: https://github.com/csabakecskemeti/llama.cpp/tree/cohere2-moe-support

[michaelw9999]

This is the problem:
0.00.623.681 E llama_model_load: error loading model: unknown model architecture: 'cohere2-moe'
We changed it at some point towards the end of the PR to cohere2moe. I see this actually was meant for a different branch so that version would not be compatible with the current official upstream llama.cpp, unfortunately.

[imnecho]

This is the problem: 0.00.623.681 E llama_model_load: error loading model: unknown model architecture: 'cohere2-moe' We changed it at some point towards the end of the PR to cohere2moe. I see this actually was meant for a different branch so that version would not be compatible with the current official upstream llama.cpp, unfortunately.

But details of b9626 literaly says:
"- Add arch support for cohere2-MoE".

And the model at https://huggingface.co/DevQuasar/CohereLabs.command-a-plus-05-2026-bf16-GGUF is a "cohere2-MoE". So the change to "cohere2moe" was a typo or what? Because it doesn't seem to me logycal to change the naming inside the llama to something that would not recognize the official model itsel.f..

[CISC]

And the model at https://huggingface.co/DevQuasar/CohereLabs.command-a-plus-05-2026-bf16-GGUF is a "cohere2-MoE". So the change to "cohere2moe" was a typo or what? Because it doesn't seem to me logycal to change the naming inside the llama to something that would not recognize the official model itsel.f..

That model was not created using this PR, thus it is not compatible.

[coder543]

That random GGUF is not exactly “the official model”. The official model uses an underscore not a hyphen: https://huggingface.co/CohereLabs/command-a-plus-05-2026-bf16/blob/main/config.json#L75

It’s just an unofficial GGUF that says it requires a vibecoded branch. Someone will hopefully push up better GGUFs soon.

[jacekpoplawski]

I am getting some errors here while trying to use the llama-b9626 with the https://huggingface.co/DevQuasar/CohereLabs.command-a-plus-05-2026-bf16-GGUF

I believe the valid solution to this problem is to comment on this model that it should be regenerated

[CISC]

The official model uses an underscore not a hyphen

The transformers model_type also has nothing to do with llama.cpp model arch.

[coder543]

I actually do understand that, and thought about expanding my comment, but if we’re nitpicking, then I was just pointing out that the original creators did not even use a hyphen for anything, anywhere. The closest they came was an underscore.

[CISC]

I actually do understand that, and thought about expanding my comment, but if we’re nitpicking, then I was just pointing out that the original creators did not even use a hyphen.

Nitpicking is my favorite hobby. :)

[bartowski1182]

@CISC something is actually off with this, we're missing a change to llama-vocab.cpp to recognize the new cohere2moe

|0.00.374.386 E llama_model_load: error loading model: error loading model vocabulary: unknown pre-tokenizer type: 'cohere2moe'

[bartowski1182]

That said, the North-Mini-Code-1.0 feels off even with a fix to map the tokenizer to LLAMA_VOCAB_PRE_TYPE_TINY_AYA.. but maybe it's just model instability, half the time it seems to respond normally, the other half it doesn't even write any code and responds like this:

./build/bin/llama-cli -m ./North-Mini-Code-1.0-Q4_0.gguf -p "Write me a C++ function to sort a list using bubble sort" --special --temp 1.0 --top-p 0.95

Loading model...
<truncated>
> Write me a C++ function to sort a list using bubble sort

We need to respond with a C++ function to sort a list using bubble sort. Provide code and explanation. Use markdown. Follow style guidelines. Provide friendly tone, maybe ask follow-up. Provide code block with comments. Use active voice. Provide explanation after code. Use American English. Provide code in C++. Ensure it's correct. Let's produce. <|channel|>analysis<|message|>We need to write a C++ function to sort a list using bubble sort. Provide code, explanation. Follow style guidelines: friendly, informative tone, maybe intro and follow-up question. Use markdown. Provide code block. Provide explanation after code. Use active voice. Use American English. Use APA style guidelines for punctuation etc. Probably a simple function.

We should ask clarifying question? The request is clear. So we can proceed.

We need to output code block with function. Could be a vector<int> bubbleSort(vector<int>& arr). Provide comments. Provide explanation after code.

We need to ensure we follow guidelines: Use markdown, include code block. Provide explanation after code. Use active voice. Use friendly tone. Possibly ask if they want other languages.

We'll produce final answer.<|END_TEXT|><|END_OF_TURN_TOKEN|>

[ Prompt: 351.6 t/s | Generation: 46.1 t/s ]

> Write a python program to check if a string is a palindrome or not.

We need to respond as per guidelines: friendly, conversational, include intro and follow-up question. Provide Python code to check palindrome. Also include explanation after code. Use active voice. Use markdown. No need for extra. Provide code in Python. Should be under 7 sentences? Actually we can have multiple paragraphs. The response length is fine. Provide explanation after code. Use active voice. Let's craft.<|END_TEXT|><|END_OF_TURN_TOKEN|>

[CISC]

That said, the North-Mini-Code-1.0 feels off even with a fix to map the tokenizer to LLAMA_VOCAB_PRE_TYPE_TINY_AYA.. but maybe it's just model instability, half the time it seems to respond normally, the other half it doesn't even write any code and responds like this:

Interesting, @pwilkin want to have a look? :)

[pwilkin]

@CISC yeah I'll look at it.

[michaelw9999]

I've been trying to reproduce this issue, but I always get correct output and code with -p "Write me a C++ function to sort a list using bubble sort" --special --temp 1.0 --top-p 0.95 .

However: with --special it's actually crashing after the response for me (with, or without PR #24258):

terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to parse input at pos 3912: <|END_OF_TURN_TOKEN|>
fish: Job 1, './llama-cli -m /data/north/BLS-…' terminated by signal SIGABRT (Abort)

Here <|END_OF_TURN_TOKEN|> is coming after <|END_TEXT|> which is not expected for in the pre-existing check in parser for Cohere here in chat-diff-analyzer.cpp:

      // Cohere Command R+ - content wrapped in <|CHATBOT_TOKEN|>...<|END_OF_TURN_TOKEN|>
      [](const common_chat_template & tmpl, autoparser & analysis) -> void {
          if (tmpl.src.find("<|CHATBOT_TOKEN|>") != std::string::npos &&
              tmpl.src.find("<|END_OF_TURN_TOKEN|>") != std::string::npos && analysis.content.start.empty()) {
              analysis.content.mode  = content_mode::ALWAYS_WRAPPED;
              analysis.content.start = "<|CHATBOT_TOKEN|>";
              analysis.content.end   = "<|END_OF_TURN_TOKEN|>";
              analysis.preserved_tokens.push_back("<|CHATBOT_TOKEN|>");
              analysis.preserved_tokens.push_back("<|END_OF_TURN_TOKEN|>");
              analysis.user_start = "<|START_OF_TURN_TOKEN|><|USER_TOKEN|>";
              LOG_DBG(ANSI_ORANGE "[Patch: Cohere Command R+]\n" ANSI_RESET);
          }
      },

@bartowski1182 I think the model is probably OK and the issue may be the template in the gguf, because all the ones I tried were all crashing for me with ``--special` because of the end of turn token.. But yours is not, but just giving bad answers, so what is the chat template?

[bartowski1182]

I was using just the default template, didn't touch anything

Also tried the suggested settings and prompt and still got the odd behaviour, though not every time it was consistent enough to be concerning

[arki05]

Cohere seems to ship 2 templates, one in tokenizer_config.json and one in a chat_template.jinja file, and those seem to differ; and the Jinja one seems to be the up to date one.

I also created a patched version that should (fingers crossed) support the default reasoning settings from llama.cpp.
HF link to my chat_template.jinja

Not sure which you were using @bartowski1182 but main seems to default to using the outdated tokenizer_config.json one.

[pwilkin]

Dedicated parser is on its way.

[bartowski1182]

@pwilkin would that fix the existing one or do I still need to remake with the one in the Jinja file?

[pwilkin]

#24615 <= it's up

Nah, the existing chat template is beyond repair, I think it's some early default or something - I got the correct one by clicking "chat template" on their model main page (think that redirects to chat_template.jinja)

[CISC]

@pwilkin would that fix the existing one or do I still need to remake with the one in the Jinja file?

I would recommend removing the ones in tokenizer_config.json, then chat_template.jinja will be picked up, which is an all-in-one template, no idea why they have these conflicting ones there?

[CISC]

@pwilkin would that fix the existing one or do I still need to remake with the one in the Jinja file?

I would recommend removing the ones in tokenizer_config.json, then chat_template.jinja will be picked up, which is an all-in-one template, no idea why they have these conflicting ones there?

I've submitted PRs on affected models to Cohere Labs.

Edit: Funnily the FP8 version of Command A+ did not have these.

[coder543]

I have published a few Command A Plus GGUFs here: https://huggingface.co/coder543/command-a-plus-05-2026-gguf

I cleared out the tokenizer_config templates prior to conversion so the chat_template.jinja would be used, which should be up to date. (But I still notice a couple of differences against the Cohere2MoE.jinja template, even apart from the “Your name is North Mini Code.” which wouldn’t be correct for Command)

Apart from being text only, the Command A Plus model seems to be working fine in some limited testing I did.

[jacekpoplawski]

command-a-plus:

Details

[michaelw9999]

Ahh sorry @CISC , time zone challenges got in the way!😴 I've been hard at work improving the tool creating the MTP tensors. It is substantially improved but still was not getting the right acceptance rate, however I think I've solved that issue now,, but it is not a particularly super fast thing to test and to train, but this will not be dead. Indeed, it is running all the candidates. Hopefully I can get something really good out of it soon and maybe within a week post up the NVFP4 MTP version, I don't ever give up so it's coming.

Status update on this @CISC : I think I'm getting much closer. Amazing learning experience on learning more about ML even if this never works out (AdamW, backwards, etc) and working on it every day! Annoyingly each full training run is about ~16 hours in Bf16. The idea with the tool is to train any gguf model and get MTP tensors out of it. Last run got:

top1_acc=0.915093
Depth breakdown:
d1 top1=0.685959, ce=1.784
d2 top1=0.989238
d3 top1=0.992477
d4 top1=0.992697

According to AI review, I was still training tokens in the wrong order: d2 was not training on d1, d3 not on d2, etc, recursively that way. So n1 acceptance was only 32.4% which gave no speedup. Latest training run is training d2 on d1's prediction, d3 on d2's, d4 on d3's. Greedy on the full 256k vocab, LR 1e-04, d1 weighted extra high, and added a gate to only accept >95%.

My whole (long) training params for the latest attempt below. It is resumable and can warmstart anytime I change anything.

Details

trainer_command=/data/models/bls-mini-code/mtp-sidecar/venvs/torch-cuda314-transplant/bin/python 
/home/mw/blackwell-mxfp6-sidecar-training/tools/quant-train/north_mtp_torch_bundle.py
 --trainpack /data/models/bls-mini-code/mtp-sidecar/trainpack-fullblock-q8-original-layer48-seed-20260621 
--mtp-hidden-cache /data/models/bls-mini-code/mtp-sidecar/cache-codeparrot-q8-original-k128-s512-l4096-gpu40-t20-o1t2-fixed-20260623T210859Z 
--config-gguf /data/models/bls-mini-code/BLS-Mini-Code-1.0-BF16.gguf 
--row-start 0 --rows 2096128 --batch-rows 16 --optimizer-step-rows 512 
--attention-prefix-rows 4096 --max-panels 0 --lr 1e-4 --weight-decay 0 
--dtype bf16 --input-token-offset 1 
--target-token-offset 2 
--input-token-source tokens 
--key-token-source tokens 
--label-source teacher-top1 
--row-sample-mode random 
--row-sample-seed 2026062801 
--moe-impl batched 
--distill-temperature 1.0 
--loss-vocab-scope full 
--sample-negative-count 512 
--sample-device-cache-size 1 
--recursive-depth 4 
--recursive-token-source model-greedy 
--depth1-loss-weight 12 
--depth2-loss-weight 5 
--depth3-loss-weight 3
--depth4-loss-weight 2 
--hidden-loss-weight 0 
--head-update-period 1 
--log-every 100 
--max-runtime-seconds 3600 
--parity-check-rows 0 
--parity-atol 0.3 
--skip-gradient-bundle 
--torch-checkpoint-in /data/models/bls-mini-code/mtp-sidecar/north-mtp-selfgreedy-aggressive-b16-fit-20260628T2200Z/torch-checkpoints/final 
--torch-checkpoint-out /data/models/bls-mini-code/mtp-sidecar/north-mtp-selfgreedy-aggressive-adaptive-20260628T2215Z/chunk-01-lr-1e-4/torch-checkpoints/final --trained-mtp-pack /data/models/bls-mini-code/mtp-sidecar/north-mtp-selfgreedy-aggressive-adaptive-20260628T2215Z/chunk-01-lr-1e-4/mtp-pack-final --pack-dtype bf16-f32norm 
--pack-name north-minicode-resident-mtp --pack-base-model /data/models/bls-mini-code/BLS-Mini-Code-1.0-BF16.gguf 
launch device=cuda dtype=bf16 rows=2096128 row_start=0 batch_rows=16 prefix=4096 input_offset=1
target_offset=2 input_source=tokens key_source=tokens label_source=teacher-top1 row_sample_mode=random
row_sample_seed=2026062801 moe_impl=batched loss_vocab_scope=full sample_negative_count=512
hidden_loss_weight=0.0 sample_device_cache_size=1 recursive_depth=4 depth_weights=[12.0, 5.0, 3.0, 2.0]
recursive_token_source=model-greedy selected=
['blk.49.nextn.embed_tokens.weight', 'blk.49.nextn.enorm.weight', 
'blk.49.nextn.hnorm.weight', 'blk.49.nextn.eh_proj.weight', 'blk.49.nextn.shared_head_norm.weight', 
'blk.49.nextn.shared_head_head.weight', 'blk.49.attn_norm.weight', 'blk.49.attn_q.weight', 'blk.49.attn_k.weight', 
'blk.49.attn_v.weight', 'blk.49.attn_output.weight', 'blk.49.ffn_gate_inp.weight', 'blk.49.ffn_gate_exps.weight', 
'blk.49.ffn_up_exps.weight', 'blk.49.ffn_down_exps.weight']

@am17an do you have any thoughts on what might be the best tuning values to train MTP like this to get the best/highest acceptance? I presume every model will have its own nuances and of course still learning all about MTP and training. I feel like we're so close! 🤯 🥲 Thanks