Gemma 4 E2B: GGUF INT4 direct conversion + QNN HTP lowering - #407
Merged
Conversation
rui-ren
approved these changes
Jul 16, 2026
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Member
Author
Update: unit tests added + HTP execution validated on-deviceTests (commit
On-device HTP validation (Snapdragon X, ORT 1.27, onnxruntime-qnn 2.4.0 / QNN SDK 2.48.40):
A few index/control ops ( |
Member
Author
Session update: build-gguf
|
Member
Author
gemma4 static KV cache (commit
|
Member
Author
Full HTP op offload via mobius qnn lowering (commit
|
| CPU-forced op | Lowered to |
|---|---|
fused Attention |
SDPA primitives (already) |
RotaryEmbedding (opset-24) |
rotate-half (Reshape/Slice/Mul/Sub/Add/Concat) — new DecomposeRotaryEmbedding |
TensorScatter (static cache) |
ScatterND (batch=1) — new TensorScatterToScatterND |
Tile (GQA head-repeat) |
Expand |
Range (bias) |
Constant(arange) + Slice |
MatMulNBits |
QDQ (already); nibble-unpack BitwiseAnd/BitShift constant-folds away |
Two new EP capability flags (supports_rotary_embedding, supports_tensor_scatter, both False for qnn) gate the new rules. All replacements are numerically identical (qnn-lowered gemma4 static model matches the fused reference at maxdiff 0.0). Added parity + op-replacement unit tests for both rules and a gemma4 build test asserting the qnn graph has no RotaryEmbedding/TensorScatter/Tile/Range.
Result: a non-quantized (fp16) gemma4 static model loads on QNN HTP with disable_cpu_ep_fallback=1 and composes + runs entirely on the NPU.
Remaining INT4 constraint (not a mobius rule): weight-only DequantizeLinear→MatMul (float activations) runs on HTP only for per-tensor weights; per-channel/blocked int8/int4 weight DQ is CPU-forced unless the activations are also quantized (full int QDQ: DQ(act)+DQ(weight per-channel)→MatMul→Q, verified HTP-OK). So GGUF INT4 weights need Olive OnnxStaticQuantization (activation calibration) to keep their matmuls on HTP. All checks pass; ruff clean; no new mypy errors. Recipe docs updated (olive-recipes PR #432).
justinchuby
commented
Jul 17, 2026
Comment on lines
+143
to
+150
| x5 = tape.op("Unsqueeze", [x, c_ints([2])]) | ||
| # Broadcast the size-1 group axis to ``group`` via Expand. | ||
| x5 = tape.op("Expand", [x5, c_ints([1, 1, group, 1, 1])]) | ||
| shp = tape.op("Shape", [x]) # (4,) == [B, Nkv, S, H] | ||
| batch = tape.op("Slice", [shp, c_ints([0]), c_ints([1])]) | ||
| s_h = tape.op("Slice", [shp, c_ints([2]), c_ints([4])]) | ||
| target = tape.op("Concat", [batch, c_ints([q_heads]), s_h], {"axis": 0}) | ||
| return tape.op("Reshape", [x5, target]) |
Comment on lines
+219
to
+223
| zero_s = tape.op("Constant", [], {"value_int": 0}) | ||
| one_s = tape.op("Constant", [], {"value_int": 1}) | ||
| rows = tape.op("Range", [zero_s, sq, one_s]) # (Sq,) | ||
| cols = tape.op("Range", [zero_s, sk, one_s]) # (Sk,) | ||
| offset = tape.op("Sub", [sk, sq]) # scalar Sk - Sq |
Comment on lines
+98
to
+103
| inputs = node.inputs | ||
| q, k, v = inputs[0], inputs[1], inputs[2] | ||
| attn_mask = inputs[3] if len(inputs) > 3 else None | ||
| past_key = inputs[4] if len(inputs) > 4 else None | ||
| past_value = inputs[5] if len(inputs) > 5 else None | ||
|
|
Comment on lines
+64
to
+69
| # The cache axis being scattered must be a concrete size (static cache). | ||
| if cache.shape is None or len(cache.shape) != 3: | ||
| return result.fail("expected rank-3 cache (B, max, D)") | ||
| if not isinstance(cache.shape[1], int): | ||
| return result.fail("cache axis 1 (max_seq_len) must be a concrete int") | ||
| return result |
Comment on lines
+76
to
+79
| # x must be rank-3 (B, S, N*H) as mobius emits. | ||
| if x.shape is None or len(x.shape) != 3: | ||
| return result.fail("expected rank-3 x (B, S, N*H)") | ||
| return result |
`mobius build-gguf --keep-quantized` silently produced a fully dequantized (float) model for Gemma 4: `Gemma4TextModel` built its attention/MLP projections with the default float `Linear`, never wiring the quantization descriptor from the config. This fixes that and the config/weight-mapping gaps that blocked E2B GGUF conversion entirely. - gemma4.py: add `_linear_class_from_config` and thread a quantized `linear_class` (MatMulNBits factory) through Gemma4TextModel -> Gemma4DecoderLayer -> Gemma4TextAttention + GatedMLP. Q/K/V/O and the gate/up/down MLP now emit INT4 MatMulNBits; per-layer input gates and norms stay float (they are F32/BF16 in GGUF). Tie the (always-float) LM head to the float scaled embedding so tied E2B checkpoints resolve. - _config_mapping.py: collapse Gemma 4's per-layer `feed_forward_length` array into a scalar `intermediate_size` + `use_double_wide_mlp` flag (E2B: 15 layers @ 6144, tail 20 KV-shared @ 12288). - _tensor_mapping.py: map the top-level per-layer-input tensors (per_layer_token_embd / per_layer_model_proj / per_layer_proj_norm). - _builder.py: strip the `.weight` artefact from the `layer_scalar` nn.Parameter during GGUF normalization. Verified: `build-gguf --keep-quantized` on unsloth/gemma-4-E2B-it-GGUF Q4_K_M now emits 205 MatMulNBits (INT4) for both cpu (GQA) and qnn (standard Attention) EPs; mixed Q6_K tensors are requantized to 4/32 by the existing loader path. All gguf + gemma4 unit tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
…tion (QNN) path When Gemma 4 is built for an EP without a GroupQueryAttention builder (e.g. `qnn`, which advertises empty `gqa_dtypes`), KV-shared decoder layers borrow K,V from a source layer's opset-24 `Attention` `present_key`/`present_value` outputs. ORT does not run shape inference on those `present` outputs, so they arrive rank-unknown; the downstream Transpose/Reshape then lose the head dimension and the KV-shared layer's `Attention` infers a zero-width output. That made `o_proj`'s MatMul fail shape inference at model-load time (`Incompatible dimensions for matrix multiplication`), so the E2B QNN model could not be loaded by ORT at all. Pin the known 4D BNSH shape (`[batch, kv_heads, kv_seq, head_dim]`) on the borrowed K,V when the source did not supply one. The KV-shared layer shares the source's KV-head/head-dim configuration, so this is exact and lets inference propagate through to `o_proj`. Verified: a tiny E2B-shaped `gemma4_text` model built for the `qnn` EP (standard Attention + per-layer inputs + 2 KV-shared layers) now loads and runs on both CPU and QNN sessions and produces finite logits; the GQA (`cpu`) path is unaffected. Full build_graph + gemma4 + gguf suites pass (1412 passed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Gemma GGUF files omit the feed-forward activation metadata key, so the GGUF config extractor fell back to `_default_activation`, which returned the generic `silu`. Every Gemma generation uses the approximate-tanh GELU (GeGLU) in its MLP, so building a Gemma model from GGUF silently produced a wrong MLP and near-garbage output (e.g. Gemma 4 E2B echoed / looped the prompt instead of answering). Return `gelu_pytorch_tanh` for any `gemma*` model type. Verified end-to-end: `mobius build-gguf --keep-quantized` on unsloth/gemma-4-E2B-it-GGUF (Q4_K_M) now generates correct text via ORT CPU — "What is the capital of France?" -> "The capital of France is **Paris**.", "Name a primary color." -> "Red". The dequantized GGUF weights were confirmed to match the HF `google/gemma-4-E2B-it` checkpoint (all projection/norm/embedding correlations >= 0.98), so the activation default was the sole remaining correctness gap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
- _default_activation returns gelu_pytorch_tanh for gemma* model types - map_gguf_to_hf_names covers the top-level per-layer-input tensors (per_layer_token_embd / per_layer_model_proj / per_layer_proj_norm) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
The Qualcomm Hexagon HTP (QNN EP) has no kernel for the fused opset-24 `Attention` op nor for `com.microsoft::MatMulNBits`, so a mobius decoder built for `qnn` had both forced onto the CPU EP (verified empirically on a Snapdragon X box: only primitives — MatMul/Softmax/RMSNormalization/ Gelu — run on HTP; fused Attention and MatMulNBits do not). This wires two EP-capability-gated lowerings so the decoder can be claimed by the QNN partitioner. Two new EpCapabilities flags (default True; both False for `qnn`): - `supports_attention`: when False, `DecomposeAttentionPass` rewrites the fused Attention op into SDPA primitives (Reshape/Transpose/MatMul/ Softmax/Add, plus Tile for GQA), faithfully reproducing scale, softcap, additive mask, bottom-right causal masking, GQA head repeat, and the past→present KV concat. Implemented as an onnx_ir InPlacePass (not a pattern rewrite rule) so it can rewire the op's 3 outputs, whose present KV tensors are graph outputs on cache layers but dead on shared-KV layers (mixed arity the pattern rewriter can't express). - `supports_matmul_nbits`: when False, the MatMulNBits contrib op is lowered to a standard blocked-uint4 `DequantizeLinear` + `MatMul` (QDQ) pair. Implemented as a model-local `com.microsoft::MatMulNBits` ir.Function body expanded by the existing InlinePass mechanism (like SkipLayerNormalization / PackedMultiHeadAttention) — supported EPs keep the compact contrib op and its native kernel; the registered-but- uninlined function body is harmless (kernels take precedence). Both transforms are verified numerically identical to the originals via ORT (maxdiff 0.0 for prefill/decode/GQA/mask; ~5e-7 with softcap due to fp rounding). A full gemma-4-E2B GGUF INT4 build for `qnn` now lowers to 0 Attention / 35 Softmax and 0 MatMulNBits / 205 DequantizeLinear, with all 30 present-KV graph outputs preserved; CPU/CUDA/etc. builds are unchanged (still emit GQA + MatMulNBits). functions + rewrite_rules unit suites pass (116). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
- _decompose_attention_test.py: numerical parity vs the fused opset-24 Attention op across prefill / decode-with-past / MHA / GQA / softcap / mask-only (atol 0 except ~1e-5 with softcap); KV-cache graph-output rewiring; rank-4 left untouched; and EP gating (qnn decomposes to Softmax SDPA, cpu keeps fused GQA). - matmul_nbits_test.py: the inlined QDQ function body matches the native MatMulNBits op bit-for-bit (incl. odd n_blocks zero-point slice); EP gating (qnn inlines to DequantizeLinear, cpu keeps the contrib op). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Thread static KV cache support through build_from_gguf and the build-gguf CLI, mirroring the existing 'build' subcommand. When --static-cache is set, the GGUF path constructs a CausalLMTask with pre-allocated fixed-width KV buffers (TensorScatter in-place writes), producing a fully static-shaped graph as required by fixed-shape runtimes such as the QNN HTP backend. - build_from_gguf gains static_cache/max_seq_len kwargs; rejects combining static_cache with an explicit task override. - CLI validates --max-seq-len requires --static-cache and is positive. - Add TestBuildGgufStaticCache: verifies fixed-width key_cache/value_cache I/O (KV axis == max_seq_len, no dynamic past_key_values.* inputs) and the explicit-task rejection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Add StaticCacheState dispatch to Gemma4TextAttention/Gemma4DecoderLayer so CausalLMTask(static_cache=True) works for gemma4 (previously rejected). This produces a fully static-shaped graph (pre-allocated fixed-width KV buffers written in place via TensorScatter), required by fixed-shape runtimes such as the QNN HTP backend. Handles gemma4's architectural specifics: - KV-shared layers (trailing num_kv_shared_layers) own no cache and read the source layer's full static buffer directly via op.Attention (is_causal=0, static-cache float bias); - dual head_dim (sliding uses head_dim, full uses global_head_dim) via a new Gemma4TextModel.static_kv_cache_specs() returning per-cache-layer (num_kv_heads, head_dim); the generic CausalLMTask consumes it so only the cache-owning layers get buffers, each sized correctly; - static-cache attention bias for both sliding and full layer types (create_static_cache_attention_bias), replacing the attention_mask-based create_attention_bias when attention_mask is None. Task changes: - _validate_static_cache_support accepts layers with _supports_static_cache=True (set on Gemma4DecoderLayer) in addition to DecoderLayer/MoEDecoderLayer; - _make_static_cache_inputs accepts optional per-layer cache_specs; - build() queries module.static_kv_cache_specs() in static mode. Verified: static-cache decode logits match the dynamic-cache reference at every position (~1e-6 fp32) across sliding-only, sliding+full dual-head_dim, and KV-shared configs. Adds 3 build-graph tests (cache-owning-layer count, dual head_dim buffer sizes, TensorScatter presence). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Change the mobius QNN lowering so a static-cache decoder runs fully on the Qualcomm Hexagon HTP (no CPU fallback). Empirically, the QNN HTP backend has no kernel for these ops (each forces its node onto the CPU EP): Tile, TensorScatter, Range, opset-24 RotaryEmbedding (plus the already-handled fused Attention and MatMulNBits). Replace them with HTP-supported equivalents: - DecomposeAttention: GQA head-repeat Tile -> Expand (broadcast). - create_static_cache_attention_bias: Range -> Constant(arange) + Slice (max_seq_len is a concrete int; q_offsets = full_range[:S_q]). - New DecomposeRotaryEmbedding rule: opset-24 RotaryEmbedding -> rotate-half (Reshape/Slice/Mul/Sub/Add/Concat), non-interleaved / full-rotation form, gated by new Ep capability supports_rotary_embedding (False for qnn). - New TensorScatterToScatterND rule: static-cache TensorScatter (axis=1) -> ScatterND (batch=1 on-device inference), gated by new Ep capability supports_tensor_scatter (False for qnn). All replacements are numerically identical to the originals (verified: a qnn-lowered gemma4 static model matches the fused reference at maxdiff 0.0). With these changes a non-quantized (fp16) gemma4 static model loads on QNN HTP with session.disable_cpu_ep_fallback=1 and composes/runs entirely on the NPU. Note: INT4 (GGUF) weights additionally require quantized activations (full int QDQ) for HTP — QNN HTP claims per-channel/blocked DequantizeLinear only inside an int-activation QDQ MatMul group, not weight-only DQ->MatMul with float activations. That activation quantization needs calibration (Olive OnnxStaticQuantization) and is out of scope for a pure mobius lowering. Adds parity + op-replacement tests for both new rules and a gemma4 build test asserting the qnn lowering emits no RotaryEmbedding/TensorScatter/Tile/Range. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
…apability The Range→Constant(arange)+Slice replacement in create_static_cache_attention_bias was applied unconditionally, even for CPU/CUDA EPs where Range works fine and is more efficient (1 dynamic node vs a large constant initializer). Add a new `supports_range: bool = True` capability to EpCapabilities (False for qnn). In create_static_cache_attention_bias, use Range for EPs that support it and Constant+Slice only when supports_range=False (QNN HTP), matching the pattern used for supports_rotary_embedding / supports_tensor_scatter. Signed-off-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply Ruff formatting to the QNN lowering additions and remap GPT-NeoX lm_head weights to embed_out so the unittest synthetic parity case uses the exported head weights. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby/gemma4-gguf-qnn
branch
from
July 25, 2026 00:15
11c637b to
d66b21c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (2)
src/mobius/rewrite_rules/_decompose_attention.py:146
- In the GQA path,
repeat_kv()usesExpandwith a hard-coded output shape[1, 1, group, 1, 1]. This is incompatible with inputs whereB != 1and/orNkv != 1(and will fail at runtime even for commonNkv>1cases), so the decomposition will not be shape-correct for typical GQA configs.
x5 = tape.op("Unsqueeze", [x, c_ints([2])])
# Broadcast the size-1 group axis to ``group`` via Expand.
x5 = tape.op("Expand", [x5, c_ints([1, 1, group, 1, 1])])
shp = tape.op("Shape", [x]) # (4,) == [B, Nkv, S, H]
src/mobius/rewrite_rules/_decompose_attention.py:203
_causal_bias()builds the causal mask using the ONNXRangeop. But the PR also declaressupports_range=Falsefor QNN HTP (and the stated goal is an HTP-friendly graph). As-is, decomposed Attention withis_causal=1will still containRangeand will be CPU-forced on HTP (e.g., static-cache models that rely onis_causalrather than a precomputed bias).
def _causal_bias(tape, c_ints, c_floats, q_bnsh, k_full, scores):
"""Bottom-right-aligned ``(1, 1, Sq, Sk)`` causal additive bias.
Query row ``i`` (within the current chunk of ``Sq``) may attend to key
columns ``j <= i + (Sk - Sq)``; later columns get ``_MASK_NEG``. Uses
``Range`` over the dynamic Sq/Sk so it is correct for both prefill
(Sk == Sq) and decode (Sk == Sq + past).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Addressed the Copilot review in c5a1db2:
Validation: lint clean; targeted lowering tests 22 passed; full suite 4718 passed, 334 skipped, 62 xfailed, with only the 11 known NeMo failures (#428). The same suite with NeMo ignored passes completely. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
src/mobius/rewrite_rules/_scatternd.py:70
TensorScatterToScatterNDrewrites by squeezing the batch axis, so it is only correct when runtimebatch==1. However, the current check allows a symbolic batch dimension (it only rejectsintbatch sizes != 1), which means the rule can silently rewrite a model that is later executed with batch>1 and fail at runtime. Consider either (1) ensuring QNN/static-cache graphs pinbatch=1before applying this rule, or (2) tightening the rule to require a statically-known batch dimension of 1 (and leaving TensorScatter untouched otherwise).
if cache.shape is None or len(cache.shape) != 3:
return result.fail("expected rank-3 cache (B, max, D)")
if isinstance(cache.shape[0], int) and cache.shape[0] != 1:
return result.fail("only batch=1 static-cache scatter is rewritten")
if not isinstance(cache.shape[1], int):
return result.fail("cache axis 1 (max_seq_len) must be a concrete int")
Comment on lines
+59
to
+63
| # Large negative additive-bias value used for masked-out positions. Matches the | ||
| # magnitude ORT's Attention kernel uses for float masks; after softmax these | ||
| # positions contribute ~0. Kept finite (not -inf) so ``0 * -inf = NaN`` cannot | ||
| # arise for fully-masked rows. | ||
| _MASK_NEG = -3.0e38 |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
src/mobius/rewrite_rules/_scatternd.py:70
TensorScatterToScatterNDassumes batch=1 (it squeezes axis 0), butcheck()currently allows a symbolic batch dimension (only rejects when batch is an int != 1). That means the rewrite can silently produce a model that will fail at runtime for batch>1 (Squeeze on a non-1 dim). Either (a) make qnn/static-cache builds pin the batch dim to 1, and requirecache.shape[0] == 1here, or (b) extend the rewrite to handle batch>1 without squeezing.
# The cache axis being scattered must be a concrete size (static cache).
if cache.shape is None or len(cache.shape) != 3:
return result.fail("expected rank-3 cache (B, max, D)")
if isinstance(cache.shape[0], int) and cache.shape[0] != 1:
return result.fail("only batch=1 static-cache scatter is rewritten")
if not isinstance(cache.shape[1], int):
return result.fail("cache axis 1 (max_seq_len) must be a concrete int")
src/mobius/rewrite_rules/_decompose_attention.py:63
_MASK_NEGis intended to stay finite afterCastLikeinto the compute dtype, but-3.0e38cast to float16 becomes-inf, which reintroduces the fully-masked-row NaN risk the comment is trying to avoid. Use a value that is representable in float16 (e.g. float16 min) so it remains finite across dtypes.
# Large negative additive-bias value used for masked-out positions. Matches the
# magnitude ORT's Attention kernel uses for float masks; after softmax these
# positions contribute ~0. Kept finite (not -inf) so ``0 * -inf = NaN`` cannot
# arise for fully-masked rows.
_MASK_NEG = -3.0e38
Reconcile native sub-4-bit GGUF modules with static-cache task selection and preserve Gemma 4 QNN static KV handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
src/mobius/rewrite_rules/_scatternd.py:70
- TensorScatterToScatterND rewrites even when the batch dimension is symbolic (only rejects when it's an int != 1). The rewrite then unconditionally Squeezes axis 0, which will raise at runtime if the model is ever executed with batch>1. Either make the rewrite strictly require a statically-known batch==1, or ensure qnn static-cache graphs pin the batch dimension to 1 before this rewrite runs.
if cache.shape is None or len(cache.shape) != 3:
return result.fail("expected rank-3 cache (B, max, D)")
if isinstance(cache.shape[0], int) and cache.shape[0] != 1:
return result.fail("only batch=1 static-cache scatter is rewritten")
if not isinstance(cache.shape[1], int):
return result.fail("cache axis 1 (max_seq_len) must be a concrete int")
Comment on lines
+45
to
+50
| shape=ir.Shape([batch, "sq", dim]), | ||
| type=ir.TensorType(ir.DataType.FLOAT), | ||
| ) | ||
| widx = ir.Value(name="widx", shape=ir.Shape([1]), type=ir.TensorType(ir.DataType.INT64)) | ||
| y = ir.Value(name="y") | ||
| node = ir.Node( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enables converting the unsloth/gemma-4-E2B-it GGUF (Q4_K_M) directly into a working INT4 ONNX model that targets the QNN EP / Hexagon HTP — reusing the GGUF's existing INT4 weights instead of running expensive Olive quantization. Pairs with olive-recipes#432 (adds a GGUF-direct
config_gguf.json).What's here
1. Gemma 4 GGUF INT4 conversion fixes (make
mobius build-gguf --keep-quantizedwork for E2B):feed_forward_lengtharray → scalarintermediate_size+use_double_wide_mlp.layer_scalar.weightartefact.MatMulNBitslinear class through Gemma4 (--keep-quantizedwas silently producing a float model) + tie the float LM head.Attentionpresent outputs are un-inferred →o_projcollapsed → model unloadable).silu→gelu_pytorch_tanh; the wrong default produced garbage output).2. QNN HTP lowering (two EP-capability-gated transforms; both default-on, off for
qnn):supports_attention=False→DecomposeAttentionPassrewrites the fused opset-24Attentionop into SDPA primitives (Reshape/Transpose/MatMul/Softmax/Add, Tile for GQA). Anonnx_irInPlacePass so it can rewire the 3 outputs (present-KV are graph outputs on some layers, dead on shared-KV layers — mixed arity the pattern rewriter can't express).supports_matmul_nbits=False→com.microsoft::MatMulNBitslowered to blocked-uint4DequantizeLinear+MatMul(QDQ), implemented as a model-local ir.Function expanded by the existing InlinePass mechanism (like SkipLayerNormalization / PackedMultiHeadAttention). Supported EPs keep the native contrib op + kernel.Verification
google/gemma-4-E2B-itcheckpoint (corr ≥ 0.98).qnnbuild lowers to 0 Attention / 35 Softmax and 0 MatMulNBits / 205 DequantizeLinear, all 30 present-KV graph outputs preserved; CPU/CUDA builds unchanged (still emit GQA + MatMulNBits).add_provider_for_devices, and HTP requires fully static shapes.)Remaining / follow-up
DecomposeAttentionPassand theMatMulNBitsfunction body (in progress).config_gguf.jsonpath) not yet hardware-validated.🤖 Generated with GitHub Copilot CLI