perf(cuda): Q6_K decode MMQ kernel — per-16 int8 scales vs the m16n8k32 fragment (#203 item 1, ~5 ms/step at N=8)

[pekkah]

Child of #203 (umbrella; per-shape data there). After #201/PR #202, the Q6_K tensors are the largest batched-decode chunk still on the bit-exact WS matvecs at N=8:

shape	ms/step	vs floor
ffn_down [4096x12288] (Q6_K half)	4.77	~3.2x
lm-head [151936x4096]	2.76	~2.7x
attn_v [1024x4096]	0.77	~16x

The Q4_K n16 decode MMQ (llm_mmq_q4k_soa_acts_n16) showed the shape of the win (gate/up 11.6 -> 6.9 ms/step); Q6_K needs its own kernel because its int8 scales are per 16 elements while the m16n8k32 mma reduces k=32 into one s32 accumulator — the two scale groups inside a fragment cannot be folded after the dot.

Candidate designs (pick after a probe):

1. k-permuted fragments: permute the k-dimension (same permutation on A and B) so each mma`s k=32 covers... still mixes two scale groups in the accumulator — only works combined with (2).
2. Two mmas per 32 elements with zero-masked halves — 2x mma count, simple, scales fold per-16 cleanly; mma is cheap at decode tile sizes (the n16 kernel is staging-bound, not mma-bound).
3. **Lift llama.cpps mmq q6_K tiling** (their vec_dot_q6_K_q8_1_mma` path) wholesale.
4. Skip the lm-head: it runs once per step and is already 2.7x floor; ffn_down/attn_v are the per-layer wins.

Contract: argmax-stable under the existing SHARPI_BATCH_DECODE_MMQ=1 gate, same per-tensor dispatch fallback as Q4_K. A Q6_K SoA weight repack (like #156`s Q4_K one) may be a prerequisite for coalesced staging — the 210-B AoS block is word-hostile.

Validation pattern: CudaDecodeMmqTests (fp32 CPU reference per-RMS tolerance) + the BatchForwardMulti MMQ argmax oracle.

🤖 Generated with Claude Code

[pekkah]

Status after PR #263 (the #205 BM=32 + GPU-argmax bundle, +6.6% at N=8): the Q6_K WS-sw shapes (ffn_down half + lm-head + attn_v, ~8.3 ms/step at N=8) remain the largest untouched chunk and are this issue's target.

Design-panel notes for whoever picks this up:

• Do NOT try the cheap dequant→fp16→cuBLAS route as a probe — it's a measured regression, already reachable via SHARPI_BATCH_DECODE_GEMM=1 (_batchDecodeComputeBound → GpuMatMulBatchedCore) and explicitly kept only as the very-high-concurrency A/B toggle. MatMulBatchedGemm writes a full fp16 [rows×cols] scratch then cuBLAS reads it back ≈ 4 B/elem weight traffic vs the Q6_K block's ~0.82 B/elem read-once = ~4.9× the HBM traffic on a bandwidth-bound decode step. It loses at N=8.
• Only viable design = bespoke kernel: RepackQ6KSoa (model on RepackQ4_0Soa, split the 210-B AoS block into [ql 128][qh 64][scales 16×int8][d fp16], new _soaQ6kHandles) + llm_mmq_q6k_soa_acts_n16 (twin of the Q4_K n16 tile but two m16n8k32 s8 mmas per 32-K sub-block with 16 lanes zeroed so each mma covers one Q6_K 16-element scale group) + extend the MatMulBatchedDecodeMmq gate to Q6_K (rows≥2048 → lm-head + ffn_down-half qualify; attn_v rows=1024 stays WS-sw). Validate via CudaDecodeMmqTests (SimdKernels.DotQ6K) + the N=6 argmax oracle.
• Honest e2e: +3–5%, as a stack on top of perf(cuda): GPU-side argmax for batched decode + BM=32 MMQ tile (#205, #203) #263. Large-effort, low-confidence per adversarial review — worth it for the throughput target but not a quick win.

[pekkah]

Built + validated, but DEFERRED — the companion-SoA approach is VRAM-prohibitive on 12 GB (net regression at N=8). Not merged.

A full bit-exact Q6_K decode-MMQ kernel was implemented (worktree branch worktree-agent-a6e124184316d61d3, commit 5522a31, not pushed): RepackQ6KSoa + llm_q6k_repack_soa, llm_mmq_q6k_soa_acts_n16/_bm32 (the per-16-scale split via two masked m16n8k32 mmas on the clean a0/a1=k0-15 | a2/a3=k16-31 boundary, symmetric no-min fixup), Q6_K branch in MatMulBatchedDecodeMmq, SHARPI_Q6K_DECODE_MMQ / SHARPI_Q6K_SOA kill-switches.

Correctness: PASS. New CudaDecodeMmqTests.DecodeMmq_Q6K_Soa_TracksCpuReference (vs SimdKernels.DotQ6K, 3%-RMS, both tiles) + all CudaBatchForwardMultiTests green (the N=6 oracle + argmax oracle now route the model's real Q6_K shapes — lm-head + ffn_down-halves — through the kernel). 11/11. Clean build. The kernel is correct and, when it fits, fast.

Blocker: VRAM. The implementation keeps a SoA companion alongside the interleaved Q6_K weight (the AoS copy can't be freed because the single-token / GEMM-N / WS readers still read AoS). That ~doubles the eligible Q6_K weights (~1.6 GB: lm-head ~0.65 GB + the ffn_down Q6_K-halves). On the 4070 Ti 12 GB target, measured (true baseline = SHARPI_Q6K_SOA=0):

N	baseline (no companion)	companion ON	peak VRAM (ON)
6	269.7	305.1 (+13.1%)	11478 MiB (fits)
8	325.4	169.3 (−48%)	11963 / 12282 MiB (WDDM spill)

The kernel is a real +13% win at N=6, but the companion pushes N≥7 into spill → a ~2× regression at the N=8 serving cell. (An earlier "+4.5%" measurement was an artifact of an A/B where both arms had the companion allocated and both spilled.)

The only VRAM-neutral path is to port the single-token / GEMM-N / WS Q6_K readers to the SoA layout so the AoS copy can be FREED (like Q4_K's RepackQ4KSoa does) — then there's no doubling and the +13%-class win should hold at all N. That's a larger, higher-risk effort (touches every Q6_K read path) and is the proper follow-up. Deferring #204 until then; the worktree branch is kept as the working reference kernel.