Model: Qwen3 Next

[pwilkin]

EDIT: README FIRST
This is an implementation of a new type of attention gating in GGML.
Therefore, this implementation will be focused on CORRECTNESS ONLY.
Speed tuning and support for more architectures will come in future PRs.
Please do not spam this threads with reports about performance, especially on backend architectures (CUDA, Vulkan).

CURRENT STATE: core is done

===
It's been a real learning experience, not gonna lie, but if someone with hybrid model implementation experience (@gabe-l-hart ?) has some quick tips, I'd be grateful.

Resolves #15940

[gabe-l-hart]

I'll try to get into it in more detail soon, but here are a few general thoughts after quickly skimming the PR:

1. The structure of what you've got smells correct, so it's likely close, but missing something small yet critical
2. A full repro with the error it's raising would definitely help debug
3. My debugging process for this would be:
   1. Make sure tokenization is solid (print statements as necessary to compare tokens before input)
   2. Use llama-eval-callback to dump tensors for a single prefill step
   3. Run an identical single prefill with the reference impl (transformers or otherwise), and inject prints as needed to dump tensors along the way
   4. Visually comb through them (particularly the sum at each point) to see where things start diverging significantly

[bugparty]

It's been a real learning experience, not gonna lie, but if someone with hybrid model implementation experience (@gabe-l-hart ?) has some quick tips, I'd be grateful.

Currently at the stage of "graph builds, but first decode complains about wrong memory model", probably not building the inputs correctly.

Resolves #15940

interesting, maybe we can learn together

[pwilkin]

2. A full repro with the error it's raising would definitely help debug

Running llama-cli -m reference/qwen3_next_500m/Qwen3_Next_500M-8x417M-BF16.gguf -ngl 999 -p "Who are " yields this weird memory error:

#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56      in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x000070552b29eb63 in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
warning: 49     ./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75      in ./nptl/cancellation.c
#3  0x000070552b31afdf in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x000070552bb45c31 in ggml_print_backtrace () at /devel/tools/llama.cpp/ggml/src/ggml.c:196
warning: Source file is more recent than executable.
196             waitpid(child_pid, NULL, 0);
#5  0x000070552bb45de5 in ggml_abort (file=0x70552bbcdac8 "/devel/tools/llama.cpp/ggml/src/ggml-backend.cpp", line=189, fmt=0x70552bbcd8af "GGML_ASSERT(%s) failed") at /devel/tools/llama.cpp/ggml/src/ggml.c:230
230             ggml_print_backtrace();
#6  0x000070552bb6091e in ggml_backend_buffer_get_type (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:189
189         GGML_ASSERT(buffer);
#7  0x000070552bb6080e in ggml_backend_buffer_is_host (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:170
170         return ggml_backend_buft_is_host(ggml_backend_buffer_get_type(buffer));
#8  0x000070552c07a114 in llm_graph_input_rs::set_input (this=0x5f11bdf6aea0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:241
241             GGML_ASSERT(ggml_backend_buffer_is_host(s_copy->buffer));
#9  0x000070552c07b03c in llm_graph_input_mem_hybrid::set_input (this=0x5f11bdf6aee0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:437
437         inp_rs->set_input(ubatch);
#10 0x000070552c07b549 in llm_graph_result::set_inputs (this=0x5f11be01ddf0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:480
480             input->set_input(ubatch);
#11 0x000070552c01ddb3 in llama_context::process_ubatch (this=0x5f11c05b5b50, ubatch=..., gtype=LLM_GRAPH_TYPE_DECODER, mctx=0x5f11be00ff00, ret=@0x7fff74d22ea4: 538976288) at /devel/tools/llama.cpp/src/llama-context.cpp:779
779             res->set_inputs(&ubatch);
#12 0x000070552c01f367 in llama_context::decode (this=0x5f11c05b5b50, batch_inp=...) at /devel/tools/llama.cpp/src/llama-context.cpp:1088
1088            const auto * res = process_ubatch(ubatch, LLM_GRAPH_TYPE_DECODER, mctx.get(), status);
#13 0x000070552c025e49 in llama_decode (ctx=0x5f11c05b5b50, batch=...) at /devel/tools/llama.cpp/src/llama-context.cpp:2726
2726        const int ret = ctx->decode(batch);
#14 0x00005f11a2021559 in common_init_from_params (params=...) at /devel/tools/llama.cpp/common/common.cpp:1066
1066                llama_decode(lctx, llama_batch_get_one(tmp.data(), std::min(tmp.size(), (size_t) params.n_batch)));
#15 0x00005f11a1e4a3c0 in main (argc=7, argv=0x7fff74d25968) at /devel/tools/llama.cpp/tools/main/main.cpp:140
140         common_init_result llama_init = common_init_from_params(params);

I'll try to merge the op into the ggml_delta_net function call as @ngxson suggested.

[CISC]

2. A full repro with the error it's raising would definitely help debug

Running llama-cli -m reference/qwen3_next_500m/Qwen3_Next_500M-8x417M-BF16.gguf -ngl 999 -p "Who are " yields this weird memory error:

...
#6  0x000070552bb6091e in ggml_backend_buffer_get_type (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:189
189         GGML_ASSERT(buffer);
#7  0x000070552bb6080e in ggml_backend_buffer_is_host (buffer=0x0) at /devel/tools/llama.cpp/ggml/src/ggml-backend.cpp:170
170         return ggml_backend_buft_is_host(ggml_backend_buffer_get_type(buffer));
...

The backend buffer is NULL.

[ngxson]

#9  0x000070552c07b03c in llm_graph_input_mem_hybrid::set_input (this=0x5f11bdf6aee0, ubatch=0x5f11be011300) at /devel/tools/llama.cpp/src/llama-graph.cpp:437
437         inp_rs->set_input(ubatch);

The model doesn't seem to have any recurrence layers. This makes the set input fails due to input node not being present in cgraph.

I'll try to merge the op into the ggml_delta_net function call as @ngxson suggested.

Hmm I think I said the reverse: not to merge it but make the op simple

I feel like this op can be implemented using other ggml ops like mul, mul_mat, sum. Which part of the calculation do you think that can't be constructed using existing ops?

This is the more important question: should we try to implement it using existing ops, or add a new op and spend even more time to optimize it cross all backends?

[pwilkin]

Now this is an error I haven't expected to encounter:

GGML_ABORT("not enough space in the context's memory pool");

[pwilkin]

The model doesn't seem to have any recurrence layers. This makes the set input fails due to input node not being present in cgraph.

How do I allocate the memory for the linear layers then? I seem to have misunderstood how build_inp_mem_hybrid() works...

[yarikdevcom]

@pwilkin any chance to buy you a coffee?(Paterson etc.) so community able to donate for your efforts. Thank you!

[pwilkin]

@pwilkin any chance to buy you a coffee?(Paterson etc.) so community able to donate for your efforts. Thank you!

Added a buymeacoffee link to my profile (do consider first funding the Llama.cpp project itself, though!)

[ServeurpersoCom]

@pwilkin any chance to buy you a coffee?(Paterson etc.) so community able to donate for your efforts. Thank you!

Added a buymeacoffee link to my profile (do consider first funding the Llama.cpp project itself, though!)

I send a coffee also.

[ngxson]

GGML_ABORT("not enough space in the context's memory pool");

Probably there are too many nodes on cgraph, try increasing the limit via llama_context::graph_max_nodes()

[ngxson]

these ggml_cont can be removed if Q/gate are separated. ggml_cont is not recommended when dealing with big tensors

[CISC]

Actually none of these need ggml_cont, Q is 3D already, Q/K are RoPEd so can be views and V can also be a 3D view now.

Edit: sorry, not quite true about V, only if QKV is fused, the weird gate fuse threw me off. Nevertheless, K/V are already contiguous at this point.

[ngxson]

the problem is that Q is non-contiguous and ggml_rope(_ext) does not work very well with non-cont tensors, it's still buggy on certain backends

[CISC]

the problem is that Q is non-contiguous and ggml_rope(_ext) does not work very well with non-cont tensors, it's still buggy on certain backends

Are you sure? AFAIK those issues are fixed.

Edit: Also, if there still are issues they will never get fixed if we work around them. :)

[ggerganov]

the problem is that Q is non-contiguous and ggml_rope(_ext) does not work very well with non-cont tensors, it's still buggy on certain backends

I think all of these cases are fixed now.

[ngxson]

This was an impl of 2D rope that relies on ggml_view: https://github.com/ngxson/ggml-easy/blob/f56e5e499b1f21a4aae73010e9d9582840428457/demo/2d-rope.cpp

It works on CPU and Metal, but doesn't work on CUDA/Vulkan. Couldn't tested on other backends, but feel free to make a PR to address this issue.

[ggerganov]

Does it still fail? I think these PRs should have addressed the problem:

• cuda : fix rope with partial rotation and non-cont src #14580
• vulkan: fix rope with partial rotation and non-cont src #14582

[ngxson]

yes that seems to work. sorry @pwilkin you will need to manually revert the change where I split Q/gate. the tensor shape for Q will be:

layer.wq = create_tensor(tn(LLM_TENSOR_ATTN_Q, "weight", i), { n_embd, n_embd_head_k * n_head * 2 }, 0);

[ngxson]

Shape of LLM_TENSOR_ATTN_Q and LLM_TENSOR_SSM_OUT should not contain n_ff

[ngxson]

^ proposed fix for the 3 comments above: 46110e0

[pwilkin]

@ngxson Thanks, scale_bias was one op I was missing in my endeavors :>

I got an LLM to rewrite the internal delta into tensor logic. After a day of manually fixing that crap, I think I understand it enough to rewrite it myself ;)

[ngxson]

Honestly I would prefer taking time to understand the mamba/ssm implementation then writing the code manually. Code written by LLM are mostly attempts for 1-to-1 translation from pytorch --> GGML which looks quite confusing

[pwilkin]

Honestly I would prefer taking time to understand the mamba/ssm implementation then writing the code manually. Code written by LLM are mostly attempts for 1-to-1 translation from pytorch --> GGML which looks quite confusing

Yeah, for me getting a rough outline then going over it manually is the best way to learn :)

I tried the "one-to-one" approach and ended up with a graph that wouldn't fit in 16 GB of RAM for a 500M model...

[pwilkin]

Aight, I cleaned up the main graph calculation, now I have to figure out how to include conv_states_all in my delta_net function in order to not get the memory error.

[ddh0]

@mrfrosty009 You need to rebuild llama.cpp from source! See https://docs.unsloth.ai/models/qwen3-next#llama.cpp-run-qwen3-next-80b-a3b-instruct-tutorial for llama.cpp rebuilding

Or, y'know, they could use the actual build instructions from this repo instead of your third-party guide :)

https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md

[danielhanchen]

@ddh0 Oh apologies I linked it because they're specifically using the Unsloth quant - but yes using the official llama.cpp build instructions is recommend! I edited my comment as well

[mrfrosty009]

@ddh0 @danielhanchen Thanks! Just couple of min ago I noticed that it was linking to old llama folder on another disk, not the one I wanted. After typing right folder, it worked, thanks!

[M3l-Idk]

@pwilkin i just wanted to say that you did a great job in this pr

[juanml82]

Guys, I genuinely appreciate all the valuable time you've put into this, but for whatever reasons, there is some problem with this implementation. I'm comparing using fastllm, in both instances with an instruct Q4 quant (fastllm uses this one, which doesn't use gguf https://huggingface.co/fastllm/Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_M/tree/main ), same temperature, top_k etc (--temp 0.6 --top-k 20 --top-p 0.95 but there is minium p that I can see in fastllm). When prompted "You must reason step by step, but do NOT output the steps — only the final answer.

A dragon has 3 boxes:

• Box A always contains a truth.
• Box B always contains a lie.
• Box C may contain either.

The dragon makes three statements:

1. “Box A and Box B contain the same type of item.”
2. “If Box C contains a truth, then Box A contains a lie.”
3. “Box C contains a lie.”

Based only on these statements, determine with certainty which boxes contain truth or lie. If it cannot be fully determined, output: “Undetermined”.

Final answer only." Both implementation expose what they are thinking (they are not the thinking models), but the llama.cpp implementation reaches the conclusion "Undetermined" in 6580 words while the fastllm implementation accomplishes the task in 2235 words (I don't think fastllm exposes the token count). This does not happen on shorter prompts (e.g., joke explanation), so it appears specifically in longer reasoning chains. Not saying it’s “dumb,” but the verbosity difference suggests something may be off internally.

[pwilkin]

@juanml82 Given that (a) you're doing non-greedy decoding (b) you have not fixed a seed (c) both answers are correct it's much more likely that the difference you're reporting is based on the random seed than on anything in the implementation details.

[M3l-Idk]

there is one issue that im having tho, no matter which unsloth's quantization i use, after prompting "create a flappy bird game in html" model gets stuck while trying to code some path to a non existing png file, it just starts hallucinating random numbers as the image's name.

[M3l-Idk]

and the code is always the same if i remember correctly

[IIIIIllllIIIIIlllll]

there is one issue that im having tho, no matter which unsloth's quantization i use, after prompting "create a flappy bird game in html" model gets stuck while trying to code some path to a non existing png file, it just starts hallucinating random numbers as the image's name.

try https://chat.qwen.ai/, the same issue:

The model entered an infinite loop and eventually stopped outputting.

[jdvpro]

CPU-only benchmark: Qwen3-Next vs Qwen3 MoE performance

Hardware: AMD EPYC 9454P (48c/96t, Zen 4), DDR5-4800 12-channel, 377GB RAM

model	size	params	backend	threads	pp512	tg128
qwen3moe 30B.A3B Q4_0	16.11 GiB	30.53 B	CPU	24	236.31 t/s	63.10 t/s
gpt-oss-120B Q4_0	60.87 GiB	116.83 B	CPU	24	108.56 t/s	35.75 t/s
qwen3next 80B.A3B Q4_0	41.98 GiB	79.67 B	CPU	24	73.74 t/s	11.76 t/s

Qwen3-Next shows ~5x slower tg than Qwen3-30B-A3B despite both having 3B active parameters. Even gpt-oss-120B with 5B active params runs 3x faster on tg.

Thanks for all the effort on this either way!

[pwilkin]

@jdvpro oh, we'll get to optimizing it on CPU as well, don't worry.

[LynxPDA]

CPU-only benchmark: Qwen3-Next vs Qwen3 MoE performance

Hardware: AMD EPYC 9454P (48c/96t, Zen 4), DDR5-4800 12-channel, 377GB RAM
model size params backend threads pp512 tg128
qwen3moe 30B.A3B Q4_0 16.11 GiB 30.53 B CPU 24 236.31 t/s 63.10 t/s
gpt-oss-120B Q4_0 60.87 GiB 116.83 B CPU 24 108.56 t/s 35.75 t/s
qwen3next 80B.A3B Q4_0 41.98 GiB 79.67 B CPU 24 73.74 t/s 11.76 t/s

Qwen3-Next shows ~5x slower tg than Qwen3-30B-A3B despite both having 3B active parameters. Even gpt-oss-120B with 5B active params runs 3x faster on tg.

Thanks for all the effort on this either way!

It seems that pp performance on the CPU should be at least 3 times higher.

On the Ryzen AI Max+ 395, I get these results with build: ab6726e (7227)

CPU (Vulkan -ngl 0)

./llama-bench -m /home/lynx/llama-swap/models/Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf -fa 1 -r 1 -ngl 0
load_backend: loaded RPC backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 8060S (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-vulkan.so
load_backend: loaded CPU backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-cpu-icelake.so

model	size	params	backend	ngl	fa	test	t/s
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	0	1	pp512	134.41 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	0	1	tg128	14.65 ± 0.00

build: ab6726e (7227)

CPU (pure CPU backend)

./llama-bench -m /home/lynx/llama-swap/models/Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf -fa 1
load_backend: loaded RPC backend from /home/lynx/tmp/tmp/libggml-rpc.so
load_backend: loaded CPU backend from /home/lynx/tmp/tmp/libggml-cpu-icelake.so

model	size	params	backend	threads	fa	test	t/s
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	CPU	16	1	pp512	101.56 ± 1.59
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	CPU	16	1	tg128	17.32 ± 0.08

build: ab6726e (7227)

Vulkan

./llama-bench -m /home/lynx/llama-swap/models/Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf -fa 1 -r 1 -ngl 999
load_backend: loaded RPC backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 8060S (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-vulkan.so
load_backend: loaded CPU backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-cpu-icelake.so

model	size	params	backend	ngl	fa	test	t/s
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512	313.90 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128	36.07 ± 0.00

build: ab6726e (7227)

Update: Added more relevant values ​​with pure CPU backend

[LynxPDA]

It's also interesting to note that, thanks to its architecture, Qwen3 Next's performance and tg speeds don't drop as much as Qwen3 30b (Vulkan backend) as the context increases.

And starting with a context of 32768, Qwen3 30b doesn't show any significant speed advantages.

Vulkan

./llama-bench -m /home/lynx/llama-swap/models/Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf,/home/lynx/llama-swap/models/Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf -fa 1 -r 1 -ngl 999 -d 512,2048,4096,8192,16384,32768,65536
load_backend: loaded RPC backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 8060S (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-vulkan.so
load_backend: loaded CPU backend from /home/lynx/llama-swap/llama.cpp/vulkan/libggml-cpu-icelake.so

model	size	params	backend	ngl	fa	test	t/s
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512 @ d512	307.01 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128 @ d512	36.25 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512 @ d2048	301.11 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128 @ d2048	35.71 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512 @ d4096	276.94 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128 @ d4096	35.68 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512 @ d8192	252.42 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128 @ d8192	34.28 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512 @ d16384	187.12 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128 @ d16384	33.31 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512 @ d32768	140.82 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128 @ d32768	30.78 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	pp512 @ d65536	95.80 ± 0.00
qwen3next ?B Q4_K - Medium	45.10 GiB	79.67 B	Vulkan	999	1	tg128 @ d65536	26.52 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	pp512 @ d512	822.34 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	tg128 @ d512	86.51 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	pp512 @ d2048	684.52 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	tg128 @ d2048	80.05 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	pp512 @ d4096	545.75 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	tg128 @ d4096	72.61 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	pp512 @ d8192	384.72 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	tg128 @ d8192	62.98 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	pp512 @ d16384	183.72 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	tg128 @ d16384	50.16 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	pp512 @ d32768	94.64 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	tg128 @ d32768	35.33 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	pp512 @ d65536	46.86 ± 0.00
qwen3moe 30B.A3B Q4_K - Medium	16.45 GiB	30.53 B	Vulkan	999	1	tg128 @ d65536	21.86 ± 0.00

build: ab6726e (7227)

However, if Qwen3 Next's performance and tg speeds can be further increased over time through optimizations, that would be incredible.

Thank you again to everyone for the opportunity to run Qwen3 Next locally!

P.S. Sorry for the spam.

Updated: Added graphs for clarity

[M3l-Idk]

I dont know if this model is supposed to do that, but it seems like the more text it generates the faster it gets over time, like it can even get twice as fast over time than in the beggining.

[theo77186]

I dont know if this model is supposed to do that, but it seems like the more text it generates the faster it gets over time, like it can even get twice as fast over time than in the beggining.

This could happen in the first few runs if the model isn't entirely loaded on GPU as llama.cpp loads the model into RAM. Otherwise it shouldn't happen at all. Text generation at long context get much less slowdown compared to standard attention models, though.