feat(loader): support Gemma 4 E4B QAT q4_0 GGUF (#211)

[pekkah]

Closes #211.

Problem

Google's official gemma-4-E4B-it-qat-q4_0 GGUF failed to load:

Error: Missing bias tensor: blk.24.attn_k_norm.weight

The Q8_0 E4B loaded fine on the same path, so this was a tensor-layout gap, not a quant-decode issue.

Root cause (verified by tensor diff)

The E4B KV-share tail — 18 layers (shared_kv_layers=18, layers 24–41) — reuses an earlier layer's already-normed K, so those layers carry no attn_k/attn_v/attn_k_norm. The diff confirms the q4_0 file omits exactly those three tensor types × 18 layers and nothing else; the Q8_0 ships dead, never-read copies (which is why it loaded).

The CPU ForwardPass already skipped attn_k/attn_v for KV-share layers (if (!kvShared)) but loaded attn_k_norm unconditionally. The dense CUDA (CudaForwardPass) and hybrid loaders already guarded it — this aligns the CPU loader. RunCommand always constructs the CPU ForwardPass as the base pass (line 362), so the throw fired even on -g -1.

Why it's numerically identical

ApplyQkNormLayer passes k=null for KV-share layers (ForwardPass.cs:1492) and only dereferences _kNorm[layer] when k != null. So the shared layers' K-norm slot was never read — loading the dead Q8_0 tensor vs leaving the slot null produces bit-identical output. Pure loader change.

Verification (RTX 4070 Ti)

• Loads + decodes coherently on CUDA and CPU — both produce "The capital of France is Paris." (cross-backend top-1 agreement, per feedback_cross_backend_parity_test).
• Decode 100.4 t/s (q4_0) vs 70.5 (Q8_0) = 1.42×; prefill on par (3283 vs 3444 t/s); frees ~3 GB VRAM.
• New CPU regression test Gemma4_E4B_Q4_0_CpuForward_LoadsWithAbsentSharedKvNorm — asserts the file genuinely omits the shared-layer K-norm (so it would have thrown pre-fix), constructs the pass, and decodes with variety. All 4 Gemma4 CPU tests green (no regression from Q8_0 no longer loading its dead K-norms).

Notes

• README gets one row for the q4_0 variant. The gemma4-e4b-qat download entry already exists.
• No server config change: the model auto-detects as gemma4, and appsettings.Local.json (the "recommended model" convenience) is gitignored.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request updates the model loader to skip loading attn_k_norm for KV-share layers, which are omitted in Gemma 4 QAT q4_0 GGUF models, and adds a unit test to verify this behavior. The review feedback suggests using the per-layer KV head count instead of the global _numKvHeads when calculating kNormSize to avoid potential size calculation or loading errors on certain model variants.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

For models with per-layer KV head counts (such as some Gemma 4 variants), using the global _numKvHeads to calculate kNormSize when _perChannelQkNorm is enabled could lead to incorrect size calculations or loading failures. It is safer to use the per-layer KV head count from hp.LayerKvHeads if available.

                    int layerKv = hp.LayerKvHeads is { } lkv ? lkv[i] : _numKvHeads;
                    int kNormSize = _perChannelQkNorm ? layerKv * layerHd : layerHd;

[pekkah]

Addressed the review feedback:

pr-test-analyzer (CUDA coverage gap): Added Gemma4_E4B_Q4_0_CudaForward_LoadsAndMatchesCpuArgmax — loads the q4_0 file via CudaForwardPass (the -g -1 path the issue exercised), asserts the shared-layer attn_k_norm is genuinely absent, and matches the CPU reference at first-decode argmax. The 12B q4_0 CUDA tests don't cover this (it ships all k_norms and uses k_eq_v, a different branch), so this is the first automated coverage of the CUDA !kvShared upload guard with an actually-absent tensor.

code-reviewer + silent-failure-hunter (Vulkan GpuForwardPass loads attn_k_norm unconditionally): Confirmed moot — GpuForwardPass rejects gemma4 up front (GpuForwardPass.cs:136: if (hp.LayerHeadDim is not null) throw new NotSupportedException(...)), so it never reaches the line-361 load. E4B on the Vulkan dense path fails loud with a clean "not supported" message (correct — there's no Vulkan gemma4 path). No change needed.

silent-failure-hunter (the unguarded ApplyQkNorm in 6 batched paths): Verified safe and intentionally left strict. Those paths (batched prefill, BatchForwardMulti, MTP/BatchVerify, PrefillPackedMulti, ForwardCore) are all gated off for _layerHeadDim != null, and KvSourceLayer != null ⟹ LayerHeadDim != null (same isGemma4 origin in ModelGraph), so no gemma4 model ever reaches them. I deliberately did not make ApplyQkNorm null-tolerant: for non-gemma4 models _kNorm[i] is always loaded, so a null there indicates a real load bug that should fail loud rather than be silently skipped. The strict-vs-tolerant split (guarded ApplyQkNormLayer for the KV-share-aware sequential path; strict ApplyQkNorm elsewhere) is the correct invariant.

[pekkah]

gemini (ForwardPass.cs:389 — per-layer KV head count for kNormSize): Declined. This is the pre-existing formula (I only rescoped it under !kvShared, unchanged), and it matches the sibling loaders verbatim (CudaHybridForwardPass.cs:838, CudaForwardPass). The _numKvHeads branch is only taken when _perChannelQkNorm is true, which is auto-detected from the q_norm tensor size (ModelGraph.cs:275: ElementCount >= numHeads*headDim). E4B's norm tensors are [256]/[512] = headDim → per-head, so _perChannelQkNorm is false for it and that branch is dead (the test decoding coherently confirms the sizing is correct). No current model combines per-channel QK-norm with per-layer KV head counts (per-channel = OLMoE, which has uniform heads; Gemma 4 has per-layer heads but per-head norm), so the concern isn't reachable. "Fixing" only this loader would also make it inconsistent with the hybrid path — out of scope for #211.