Gemma 4 batched decode: validate on a GEMM-N-batchable 12B (real k_eq_v coverage + Q4_K decode-MMQ) — #275/#276 follow-up

[pekkah]

Context

PR #280 (#275) routed the Gemma 4 batched decode through BatchDecodeMatMul (WS / decode-MMQ / GEMM-N), and PR #281 (#276) added synthetic coverage for the 12B-global attention_k_eq_v batched path. Both were limited by the lack of a GEMM-N-batchable 12B (the only local 12B is gemma-4-12b-it-qat-q4_0.gguf, Q4_0 → single-user fallback; a Q8_0 12B is ~13 GB and doesn't fit the 4070 Ti's 12 GB at a useful context):

• perf(cuda): Gemma 4 batched decode uses cuBLAS GEMM, not the dense WS/MMQ decode routing (#195 follow-up) #275 decode-MMQ is untested for Gemma 4. On E4B Q8_0 the decode-MMQ tile (Q4_K-only) correctly falls back to the WS matvec, so only the WS path was benched (2.70×/2.45×/1.79×/1.10× at N=1/2/4/8). The perf(cuda): batched-decode WS matvecs are LSU/latency-bound at N=8 — 27.7 of 32.5 ms/step, 3.3x over weight floor (#197 follow-up) #201/perf(cuda): evaluate SHARPI_BATCH_DECODE_MMQ default-on for batched serving (#203 item 5) #206 int8 decode-MMQ tile that BatchDecodeMatMul plumbs for big Q4_K shapes at N≥5 is never exercised on a Gemma 4 model.
• test(cuda): Gemma 4 k_eq_v batched-decode path is uncovered — needs a GEMM-N-batchable 12B fixture (#195 follow-up) #276 k_eq_v coverage is synthetic-only. The 12B-global CopyDevice(vAll, kAll) batched branch is validated against the single-token oracle on a tiny synthetic all-global F32 fixture (mutation-verified), not on a real 12B. The realistic 12B pairing of k_eq_v-on-global + real-V-on-SWA in one model is not reproduced (the synthetic fixture is all-global; SWA is orthogonal and covered separately by the E4B oracles).

Goal

When a GEMM-N-batchable (Q4_K / Q5_K / Q6_K / Q8_0) 12B that fits 12 GB becomes available (e.g. a Q4_K_M 12B, or a larger-VRAM box):

1. Add a real-12B Gemma4CudaBatchForwardMultiTests-style oracle that exercises the global k_eq_v layers end-to-end (mirrors the synthetic Gemma4CudaKEqVBatchedDecodeTests), closing the test(cuda): Gemma 4 k_eq_v batched-decode path is uncovered — needs a GEMM-N-batchable 12B fixture (#195 follow-up) #276 synthetic-vs-real gap.
2. Bench the Gemma 4 batched decode on a Q4_K 12B so the decode-MMQ tile (N≥5, rows≥2048, cols%256) actually engages, and confirm it beats WS at higher N (the perf(cuda): Gemma 4 batched decode uses cuBLAS GEMM, not the dense WS/MMQ decode routing (#195 follow-up) #275 routing already dispatches it; this validates the win exists for Gemma 4, not just the dense path).

Secondary (lower priority, from #275)

• The PLE pre-pass + injection matmuls (_gpuInpGate / _gpuPleProj / per_layer_model_proj) deliberately stay on cuBLAS GpuMatMulBatched (their pleWidth shapes weren't a decode-matvec win and the issue scoped them out). If a Gemma 4 decode profile shows PLE GEMM is a non-trivial decode-time cost, evaluate routing them through BatchDecodeMatMul too — bench before shipping (cuBLAS GEMM is compute-bound for small-N decode, same reasoning as perf(cuda): Gemma 4 batched decode uses cuBLAS GEMM, not the dense WS/MMQ decode routing (#195 follow-up) #275; but pleWidth is narrow, so the win may not materialize).

Refs: #275 (PR #280), #276 (PR #281), #195, #201/#206 (decode-MMQ), #190 (umbrella).

[pekkah]

Done — both deliverables shipped to master

The prerequisite (a GEMM-N-batchable 12B that fits 12 GB) is satisfiable: download-model.ps1 -Model gemma4-12b-q4km pulls gemma-4-12b-it-Q4_K_M.gguf (~7.1 GiB), which loads on the 4070 Ti and leaves room for batched-decode KV. The Q4_K trunk weights are SoA-repacked at load (dense path), so the decode-MMQ tile engages for the big shapes at N≥5.

1. Real-12B k_eq_v oracle — PR #284 (merged 7244a2a)

New Gemma4Cuda12BBatchForwardMultiTests closes the #276 synthetic-vs-real gap: it drives the realistic 12B pairing (k_eq_v on the global layers + real attn_v on the SWA layers, in ONE model — sliding_window_pattern=[T,T,T,T,T,F,T,T], the F/global layers are MQA k_eq_v) through BatchForwardMulti and matches the single-user ForwardGemma4 loop. Includes an N=6 step that runs the global k_eq_v layers end-to-end through the decode-MMQ tile, argmax-stable. 5/5 pass on a 4070 Ti.

2. Q4_K 12B decode-MMQ bench — PR #285 (merged 94f0c84)

Gemma4CudaBatchedDecodeBench parameterized (SHARPI_BENCH_MODEL / SHARPI_BENCH_CTX; E4B default unchanged). Decode-MMQ vs WS A/B (ctx 512, 128 steps, 4070 Ti):

N	decode-MMQ (default)	WS only (SHARPI_BATCH_DECODE_MMQ=0)	speedup
2	76.1	75.9	1.00× (N<5, both WS)
4	113.7	113.4	1.00× (N<5, both WS)
5	149.0	117.9	1.26×
6	166.3	126.8	1.31×
8	197.0	138.4	1.42×

The crossover sits exactly on the N≥5 decode-MMQ gate — a real win, not neutral/negative, and no regression below the gate (identical to WS). This is the first time the #201/#206 decode-MMQ tile is exercised on a Gemma 4 model (E4B Q8_0 always fell back to WS).

Secondary (PLE routing) — assessed, not shipping

Routing the PLE matmuls (inp_gate / proj / per_layer_model_proj) through BatchDecodeMatMul is unlikely to pay off, for two concrete reasons beyond "pleWidth is narrow":

1. The PLE matmul weights are F32 (_gpuInpGate/_gpuPleProj upload as F32), so the decode-MMQ tile (Q4_K/Q6_K-only) can never engage — only the marginal WS-vs-cuBLAS-GEMM choice would change.
2. The shapes are tiny (inp_gate rows = pleWidth ≈ 256 < the 2048 decode-MMQ floor; proj is [embDim × 256]) — a small fraction of per-layer decode cost dominated by the trunk GEMMs. And the 12B (this issue's model) has no PLE at all, so this only ever concerns E4B.

Recommendation: leave PLE on cuBLAS GEMM. If a future Gemma 4 (E-series) decode profile flags PLE as a hotspot, revisit as its own issue with a profile-backed bench.

Closing — both primary deliverables shipped and verified; secondary assessed and deferred.