SnapKV: CUDA hybrid GDN port (#51 follow-up)

[pekkah]

Follow-up to #51 / PR #57.

The CPU SnapKV foundation is in master once #57 merges. The CUDA hybrid GDN path (CudaHybridGdnForwardPass — used by qwen35moe and qwen35 27B-MTP) is the primary 12 GB VRAM target for SnapKV but isn't covered by v1.

Scope

1. Last-W per-layer query capture during prefill. CudaHybridGdnForwardPass.Prefill is a sequential token-by-token loop (Forward(tokens[i], …)), so there's no batched Q buffer to harvest. Add a GPU scratch ring [numAttnLayers × W × qDim] that captures _gpuQ per layer per token for the last W tokens. Reuse the trailing slots across the loop — only the last W need to survive into the eviction pass.

2. Post-prefill scoring. Run per-layer scoring against the GPU K cache (_gpuKCache[layer]). Two options:

   • Pure GPU: dispatch Attention-style kernel that emits softmaxed weights summed across heads/queries into a [promptLen] device float buffer.
   • GPU+CPU split: download captured Q's (small, ~6 MiB total for Qwen3.6-27B-MTP at W=64) and use existing SimdKernels.DotF32 on a host-side K download. Simpler but pays the KV download cost; only worth it if the GPU scoring kernel is non-trivial to fuse.

3. GPU-side compaction. New NVRTC kernel llm_kv_compact that takes a [K] keep-position list and copies survivors into slots [0..K) within the flat ring. Two passes (K then V) or fused.

4. Wire up bookkeeping. CudaHybridGdnForwardPass keeps its own PagedKvCache purely for length tracking — switch it to call .Compact(keep) after the GPU compaction. Decode-side: AttnLayer already uses kvPosition + 1 for the kernel seqLen — change to _kvCache.Length + 1 so it sees the post-compaction count.

5. Compose with feat: Bf16 KV cache for hybrid GDN models (#27) #56 (bf16 KV). The kernel needs a bf16-store variant too (i.e. llm_kv_compact_bf16), or generic byte-stride copy.

Acceptance

• SHARPI_SNAPKV_BUDGET=2048 on a 16 K prompt: GPU KV cache shrinks ~8× (matching the issue's projection), decode stays well-formed.
• No decode regression vs no-eviction (prefill-only cost).
• Existing CudaHybridGdnForwardPass_* tests pass with eviction disabled (default).

Notes

• Captures of Q are post-RoPE / post-Q-norm (matches CPU path semantics).
• The bookkeeping cache (_kvCache) already supports the LogicalLength split from PR feat: SnapKV prefill KV eviction (#51 v1, CPU) #57.

[pekkah]

Shipped in 2f3631a.

What landed:

• 4 NVRTC kernels: llm_snapkv_score / _bf16 (per-(query, head) softmaxed attention against the layer's K ring with atomicAdd pooling into a single per-position score buffer), llm_kv_compact / _bf16 (sorted keep-list gather into a staging tensor, then CopyDeviceRegion writes back over the [0, K*kv_dim) prefix). Composes with the bf16 KV ring (feat: Bf16 KV cache for hybrid GDN models (#27) #56).
• CudaHybridGdnForwardPass: lazy [numAttnLayers × W × qDim] capture ring, inline post-RoPE/post-Q-norm capture inside GpuAttnBlockAt for the trailing W tokens of prefill, post-pass score → host-side keep-set → per-layer compaction. GpuAttnBlock uses kvPosition = _kvCache.Length so decode addresses the compacted slot. BatchForward2's two-tokens-without-Increment path stays correct (kept the kvPosition + 1 kernel arg).
• PagedKvCache.CompactLengthOnly(K): bookkeeping-only counterpart of Compact for backends that store KV payload elsewhere (the GPU hybrid case has no host-side data to copy).
• Default policy: auto-on when the full bf16 KV ring would exceed ~256 MiB at the configured ctx (i.e. ctx ≳ 16K on qwen35-class models). Auto-budget is min(maxSeqLen/4, 4096) floored at 1024. SHARPI_SNAPKV_BUDGET=0 opts out; any positive value overrides. SnapKvConfig.IsBudgetExplicit distinguishes env-unset from env=0. CPU ForwardPass (feat: SnapKV prefill KV eviction (#51 v1, CPU) #57) keeps the explicit-opt-in convention.

Acceptance:

• ✅ SHARPI_SNAPKV_BUDGET=256 on a 384-token prompt: GPU KV slot count drops to 256, LogicalLength stays at 384, decode produces ≥2 distinct argmaxes (CudaHybridGdnSnapKv_LongPrompt_*).
• ✅ No decode regression with eviction disabled — existing CudaHybridGdnForwardPass_* tests untouched (SmallCtxDefault_AutoBudgetStaysOff covers the env-unset path explicitly).
• ✅ Full ForwardPass suite: 251/251 pass.

Remaining #51 work — #59 (dense CUDA + Vulkan ports), #60 (TurboQuant composition), #61 (LongBench accuracy eval).