llama-quantize: use static quantization level for tensors missing from imatrix data by de-wim · Pull Request #23575 · ggml-org/llama.cpp

de-wim · 2026-05-23T13:33:04Z

Overview

This PR adds support to llama-quantize to quantize models using an imatrix file which does not have importance data for all blocks.

This is currently mainly MTP-enabled GGUFs while using an imatrix file, where MTP layers are not evaluated during imatrix generation, but could also affect future architectures or features.

It is an alternative to #23258 and takes a much more minimalist approach, suggested by @bartowski1182:

imatrix generation is not modified and does not use MTP layers
Missing layers are instead quantized using a simple, fast quantization method:
- Q4_0 for Q6_* and below presets
- MXFP4 for MXFP4
- Q8_0 for Q8_0

Since MTP benefits far more from speed, even at the cost of some accuracy, this might be preferable to the earlier PR.

I am yet to run comparative benchmarks between the two.

Additional information

N/A

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: not this time :)

…l for MTP layers This fixes an error when trying to quantize an MTP enabled GGUF with an imatrix.

ggml-gh-bot · 2026-05-23T13:37:08Z

Hi @de-wim, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

bartowski1182 · 2026-05-23T14:25:34Z

Hmmm this wouldn't catch the other tensors in the MTP layer that aren't labelled nextn, there's several others that won't have imatrix data as well right?

Previous commit logic took presedence over the manual CLI args, which is obviously undesirable. Also removed the logic for floating-point types, since for those ggml_is_quantized() is always false, and no imatrix is involved, ever.

de-wim · 2026-05-23T17:13:03Z

Hmmm this wouldn't catch the other tensors in the MTP layer that aren't labelled nextn, there's several others that won't have imatrix data as well right?

You're absolutely right! Good catch! (I'll stop acting like an LLM now)

There are two possible solutions to this I can see:

Either I make it scan for blocks containing nextn and keep track of those layers, and apply the logic to them.
Or I make a generic handler for missing imatrix data, and apply a simple, static fallback quantization to those layers. This would affect more than just MTP layers (other missing / unused/ unsupported layers would get the same treatment), but it seems more sensible to me?

I'm not an expert on any of this, so please point out if I'm wrong on any of this.

…atrix data quantization

de-wim · 2026-05-25T07:47:48Z

I want with option 2 (applying a simple static quant to tensors that are missing imatrix data), and did some wildly unscientific benchmarking with Qwen3.5-122B-A10B, all using a IQ4_XS with imatrix on my 70W Strix Halo laptop:

config	pp/s	tg/s	draft acceptance rate
mtp disabled	266	22.5	N/A
imatrix applied to MTP layers (#23258)	263	24	0.5985
static quant for MTP layers (Q4_0, this PR)	253	26	0.5950

bartowski1182 · 2026-05-25T15:28:58Z

Hmm I'm surprised by how low the acceptance rate is.. When using the smaller models I was getting >80%

How many tokens are you trying to draft? I think the sweet spot for MTP is 3-5, and the speed up was anywhere a lot more

but maybe that has more to do with the hardware.. can you try with Q8_0? I'm wondering if the Q8_0 will have a higher acceptance rate and still be fast enough, I think IQ4_XS is particularly slow on Strix Halo

edit to clarify: Q8_0 instead of Q4_0 as the MTP layers, not the whole model

de-wim · 2026-05-25T21:26:17Z

How many tokens are you trying to draft? I think the sweet spot for MTP is 3-5, and the speed up was anywhere a lot more

I was using 3 draft tokens, since it was already fairly low acceptance I didn't see a reason to increase that.

I'm currently cooking a Qwen3.5-3.6-27B because it's more manageable file size to iterate on and there's more benchmark data to compare with. Will update tomorrow.

de-wim · 2026-05-26T17:09:57Z

I have since retested with Qwen3.6-27B, comparing the following quants:

Quant	Acceptance rate
Q4_K_M (this PR, Q4_0 MTP layers)	0.44
Q8_0 (this PR, Q8_0 MTP layers)	0.44
bartowski/Qwen_Qwen3.6-27B-GGUF:Q6_K	0.41

This is a sample size of 1, on a llama.cpp web conversation which mixes code and markdown. Not the best benchmark, I know, but I was expecting better acceptance rate across the board.

Maybe my setup is not right? My full llama.cpp command is:

/opt/llama.cpp/vulkan/bin/llama-server --flash-attn on --jinja --port 10015 --no-warmup -ngl 999 --batch-size 2048 --ubatch-size 2048 --parallel 1 --cache-ram -1 --threads -1 --mmap -hf bartowski/Qwen_Qwen3.6-27B-GGUF:Q6_K_L --fit-ctx 72000 --spec-type draft-mtp --spec-draft-n-max 4 --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --kv-unified --temp 1.0 --top-p 0.95 --top-k 20 --min_p 0.0 --presence_penalty 1.5 --repeat_penalty 1.0

But at least it doesn't seem the acceptance rate was degraded by the changes in the PR (thought that could be masked by any misconfiguration to be fair).

…-layers

de-wim · 2026-06-03T07:58:18Z

@bartowski1182 Not sure if you can give this another look? I did some comparisons vs your quants and unsloths, and overall my conclusion was:

No meaningful difference in acceptance between:

Models generated with the settings of this PR and existing models with manual quant level overrides for MTP layers
Different quant levels for the MTP layers (it seems like Q4_0 does just as good of a job predicting as Q8_0)

Merging this PR would make it easier for anyone to generate MTP-enabled quants without cryptic error messages for newcomers, without breaking any of the stuff power users have been doing.

bartowski1182 · 2026-06-03T12:37:18Z

Yes sorry I'll give this another look today/in a few hours, was away for a bit and forgot about this, thanks for the ping

bartowski1182 · 2026-06-05T14:38:46Z

Okay so looking at this again, I like this idea

The one small issue in the grand-scheme of things is it doesn't specifically target MTP layers, it just happens to target them because MTP doesn't have imatrix data associated with it

I think there may be other situations where this would occur accidentally, I've seen some models (most recently the gemma-4-E2B) where some specific tensors end up with no imatrix data

I think the best bet would be to wrap this in a flag of some kind that forces static quantization when imatrix data is missing so we have to opt-in to silencing the failures

The only other alternative would be to figure out how to specifically identify MTP layers, but I don't know if that can be done cleanly.. Maybe with an initial pass over all tensors to find which blocks contain any named "nextn"?

Wondering if @am17an has any thoughts, specifically about identifying MTP layers

de-wim · 2026-06-05T15:27:32Z

@bartowski1182 Yep, that's pretty much the alternative I outlined in #23575 (comment) as well!

However, I chose the current approach because it would allow quantizing models which introduce architectural features we don't yet support in llama.cpp.

I realize that's a double edged sword, but it would avoid people having to requant, reupload and everyone having to redownload the GGUFs for a few added layers.

I'm fine with either approach, I defer to your judgement on what is best in this case.

bartowski1182 · 2026-06-05T15:30:14Z

I think as long as you put it behind an explicit flag then it would be reasonable, best to avoid changing default behaviour without warning/opt-in :)

I'd also maybe consider making everything above Q4_0 quantize to their default? It's a tricky one, with MTP specifically Q4_0 is best for raw speed, for non-mtp the default would be better for accuracy..

de-wim · 2026-06-05T18:14:46Z

I think as long as you put it behind an explicit flag then it would be reasonable, best to avoid changing default behaviour without warning/opt-in :)

I'd also maybe consider making everything above Q4_0 quantize to their default? It's a tricky one, with MTP specifically Q4_0 is best for raw speed, for non-mtp the default would be better for accuracy..

I'm thinking I'll look into the following:

Put it behind an explicit flag, as you suggested. Update the error message when encountering missing imatrix data to inform the user of the existence of the flag to bypass the problem.
Do a scan for layers containing nextn tensors, and keeping a list of MTP layer names for use during quantization
When encountering tensors with missing imatrix data:
- If the layer was identified in the earlier pass as an MTP layer, quantize according to the strategy set out before (Q4_0 or Q8_0)
- If not, use the default quant level for the target quant format.

I need to see how complicated this would make the code. But can you find yourself in the strategy?

bartowski1182 · 2026-06-05T18:24:08Z

That sounds like it would be very useful yeah

for the default quant level fallback, you could borrow tensor_requires_imatrix, if it does require imatrix then just blindly set it to GGML_TYPE_IQ3_S

llama.cpp/src/llama-quant.cpp

Line 767 in e82beaa

    
           static bool tensor_requires_imatrix(const char * tensor_name, const ggml_type dst_type, const llama_ftype ftype) {

I think another useful addition could be a specific flag like --mtp q4_0 which similar to --tensor-type would set the MTP tensors to q4_0 or whatever is specified, but I would consider that a value-add for another update, for now just q8_0 and q4_0 as you specified is more than enough, especially since that alone may end up being quite complex (hopefully not!)

bartowski1182 · 2026-06-05T18:25:29Z

You could also leave the nextn scanning itself for a future PR and just focus on the flag that will set any layers missing imatrix data to q8_0 or q4_0, just so you can quickly get some value added and focus on minimal increments of work?

…tp-layers

…ing from imatrix data behind command line flag

de-wim · 2026-06-06T09:54:48Z

You could also leave the nextn scanning itself for a future PR and just focus on the flag that will set any layers missing imatrix data to q8_0 or q4_0

@bartowski1182 You are the voice of reason :)

I've added a --imatrix-allow-missing-tensors flag which guards the new behaviour. Without it, the behaviour should be identical to before.

Let me know if anything else is required for this PR. And thank you for your time and patience :)

de-wim · 2026-06-15T08:58:03Z

@bartowski1182 Sorry to keep bothering you, but could you have another look?

bartowski1182 · 2026-06-15T15:19:16Z

Oh sorry I guess I missed this update..! I was actually thinking about it the other day and was curious if you were still working on it but never looked, my bad :)

This looks overall fine, the one thing I can see being pushed back on is that you end up down-casting tensors that are missing imatrix data, like if you're making a Q6_K and there's imatrix data missing then suddenly that tensor is now at Q4_0 instead of Q6_K.. in the MTP case this makes perfect sense, we want that raw speed, but if it happens to literally any other tensor we've now sacrificed quality. I think for this change here we should only be upcasting to Q4_0, and then when we in the future detect specifically MTP layers we can downcast to Q4_0

de-wim · 2026-06-15T15:29:13Z

This looks overall fine, the one thing I can see being pushed back on is that you end up down-casting tensors that are missing imatrix data, like if you're making a Q6_K and there's imatrix data missing then suddenly that tensor is now at Q4_0 instead of Q6_K.. in the MTP case this makes perfect sense, we want that raw speed, but if it happens to literally any other tensor we've now sacrificed quality. I think for this change here we should only be upcasting to Q4_0, and then when we in the future detect specifically MTP layers we can downcast to Q4_0

I thought of that as well. I would like to add a specific --mtp flag in a future PR that allows you to set the tensor size for MTP layers and then just use the default tensor size for other blocks missing imatrix info.

But if you feel like it would be better to include that in this PR, to avoid "damaged" GGUFs using this flag, I'm fine with that, too.

bartowski1182 · 2026-06-15T17:15:01Z

Yeah I think for this specifically a wise move would be to either specifically set what a user wants missing tensor data to map to in the CLI, or use the "requires imatrix" helper to determine if the tensor needs imatrix, and if it does and it's missing (and this Arg is provided) then upcast

Otherwise it's possible to accidentally degrade the quality by quite a bit in a situation where a single tensor happens to not have imatrix data and you're making Q6_K

It's super narrow and honestly feels like overkill to get you to do this, but in terms of likelyhood to get it merged I think that'll be better

…-layers

…s when imatrix data is missing - For MTP blocks, we keep using the Q8_0/MXFP4/Q4_0 logic from before - For other blocks missing from the imatrix data, we use the default tensor type for the requested quant, or IQ3_S if the default requires an imatrix.

de-wim · 2026-06-18T15:57:52Z

        // if not manual - use the standard logic for choosing the quantization type based on the selected mixture
        if (!manual) {
            new_type = llama_tensor_get_type_impl(qs, new_type, tensor, params->ftype, tm.category);
+            if (params->mtp_tensor_type < GGML_TYPE_COUNT && tm.is_mtp) {


Not sure if this bool is the right way to go about this.

Alternatively, I could add another value to the tensor_category enum, tensor_category::MTP and add another if clause similar to the above for TOKEN_EMBD and OUTPUT. That would however supercede the manual overrides applied above, so not sure if that's desired?

The current category detection code (see

llama.cpp/src/llama-quant.cpp

Line 115 in d2c6795

static tensor_category tensor_get_category(const std::string & tensor_name) {

) also only matches on tensor name, so it would only identify and mark the nextn tensors specifically, not the entire block like the current code.

What do you think would be the most sensible approach @bartowski1182 ?

I do like the idea of adding a new tensor_category::MTP but it would have to be some kind of super-category.. almost like layer_category, since there's still things like ATTN and FFN within MTP layers

that said, I do like the flagging of is_mtp, if the logic works nicely then I think that's a good inclusion and solve

the only other thing that I think could be "better" would be if we went up a layer from here and started encoding the MTP layers in the GGUF metadata itself, like including a list of integers at conversion time that lists which layers specifically are MTP, but that would be a bigger change and need other feedback

bartowski1182 · 2026-06-19T16:06:42Z

 };

+// extract the block index from a tensor name like "blk.N.foo.weight", or -1 if not a block tensor
+static int tensor_block_index(const std::string & name) {


can this be done in the same way as below with sscanf instead?

https://github.com/ggml-org/llama.cpp/pull/23575/changes/BASE..77ef7153912bc49fbf270cde056e4123a8d57440#diff-bcc71d02e033731949e3b56973150c7385b3e2a1f6d62d34557fce3d4c0dec22R460

…-layers

de-wim · 2026-06-28T18:49:41Z

@bartowski1182 I've refactored the block detection code to use sscanf (through the same helper function) in both cases. Hope this helps.

I've not touched the layer category stuff - it seems like your suggestion to encode which layers are MTP would need at least some kind of agreement of how to embed that metadata in the GGUF format.

That would of course be nice, but since this is my first PR to the project, I'd like to keep impact and controversy as low as possible.

bartowski1182 · 2026-06-30T15:36:58Z

Taking a look :)

llama-quantize: when using an imatrix, use a simple, fixed quant leve…

58a25c5

…l for MTP layers This fixes an error when trying to quantize an MTP enabled GGUF with an imatrix.

de-wim mentioned this pull request May 23, 2026

Fix imatrix generation for MTP models #23258

Closed

de-wim force-pushed the static-quantize-mtp-layers branch from a315ba0 to eac3564 Compare May 23, 2026 14:58

llama-quant: remove MTP quantization handling, add generic missing im…

ef1095a

…atrix data quantization

de-wim marked this pull request as ready for review May 25, 2026 07:47

de-wim requested a review from ggerganov as a code owner May 25, 2026 07:47

Merge remote-tracking branch 'origin/master' into static-quantize-mtp…

248e25c

…-layers

de-wim added 2 commits June 6, 2026 07:00

Merge remote-tracking branch 'upstream/master' into static-quantize-m…

c28cae0

…tp-layers

llama-quantize: hide fallback to larger static quant for tensors miss…

4689069

…ing from imatrix data behind command line flag

github-actions Bot added the examples label Jun 6, 2026

de-wim changed the title ~~Static quantize mtp layers~~ llama-quantize: use static quantization level for tensors missing from imatrix data Jun 6, 2026

de-wim added 3 commits June 17, 2026 16:18

Merge remote-tracking branch 'origin/master' into static-quantize-mtp…

28efeb9

…-layers

llama-quantize: add --mtp flag for setting tensor type for mtp blocks

77ef715

de-wim force-pushed the static-quantize-mtp-layers branch from aa509c2 to 77ef715 Compare June 18, 2026 14:54

de-wim commented Jun 18, 2026

View reviewed changes

bartowski1182 reviewed Jun 19, 2026

View reviewed changes

de-wim added 2 commits June 27, 2026 13:57

Merge remote-tracking branch 'origin/master' into static-quantize-mtp…

02073ee

…-layers

llama-quant: refactor block/layer index detection code

42611bb

Uh oh!

Conversation

de-wim commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

ggml-gh-bot Bot commented May 23, 2026

Uh oh!

bartowski1182 commented May 23, 2026

Uh oh!

de-wim commented May 23, 2026

Uh oh!

de-wim commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartowski1182 commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

de-wim commented May 25, 2026

Uh oh!

de-wim commented May 26, 2026

Uh oh!

de-wim commented Jun 3, 2026

Uh oh!

bartowski1182 commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartowski1182 commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

de-wim commented Jun 5, 2026

Uh oh!

bartowski1182 commented Jun 5, 2026

Uh oh!

de-wim commented Jun 5, 2026

Uh oh!

bartowski1182 commented Jun 5, 2026

Uh oh!

bartowski1182 commented Jun 5, 2026

Uh oh!

de-wim commented Jun 6, 2026

Uh oh!

de-wim commented Jun 15, 2026

Uh oh!

bartowski1182 commented Jun 15, 2026

Uh oh!

de-wim commented Jun 15, 2026

Uh oh!

bartowski1182 commented Jun 15, 2026

Uh oh!

de-wim Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

bartowski1182 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

bartowski1182 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

de-wim commented Jun 28, 2026

Uh oh!

bartowski1182 commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

de-wim commented May 23, 2026 •

edited

Loading

de-wim commented May 25, 2026 •

edited

Loading

bartowski1182 commented May 25, 2026 •

edited

Loading

bartowski1182 commented Jun 3, 2026 •

edited

Loading

bartowski1182 commented Jun 5, 2026 •

edited

Loading