perf(cuda,engine): batched GDN trunk — fused delta-net scan + batched-query SDPA (#114-B)

[pekkah]

Issue #114 direction (B): batch the per-position GDN-hybrid prefill trunk

#114-A (register-tiled N-input routed-MoE dots, PR #116) made the routed MoE flat
with context. With that merged, the trunk became the dominant prefill cost and
it grew with context: profiled on a 4070 Ti (SHARPI_PREFILL_PROFILE=1), trunk
≈1.9s at startPos 0, climbing ~+0.5ms/token to ~20s at ~34K cumulative tokens.

Cause: #111 GEMM-batched the trunk projections but deliberately left conv1d,
the delta-net recurrence (GdnRecurrenceDecode), KV-append and SDPA per-position
over N via CudaBackend.View. GDN launches dominate ~12:1 over attention, so the
trunk was launch-overhead bound at short context and under-utilised the GPU at long
context. CudaBackend.FullSeqAttention previously threw.

What this does

Collapses those per-position launches into batched kernels, bit-identical to the
per-token TrunkLayerSequential path.

New CUDA kernels (CudaTextKernels.cs):

• llm_gdn_recurrence_scan — one block per v-head loops the N positions internally,
  carrying per-head state in global memory. This is the fused sequential scan, not
  the parallel chunked-scan (which reorders FP reductions and would break parity).
• llm_gdn_conv1d_decode_batched (+ _state_update_batched), _l2_norm_per_head_batched,
  _tile_heads_batched — each GDN sub-stage batched over N.
• llm_kv_append_batched (+bf16) and llm_full_seq_attention (+bf16) — the N prompt
  queries attend over their causal prefix in one launch (shared-scores path,
  startPos+N ≤ 4096; falls back to the per-position loop beyond — wave-based >4096
  batched SDPA is a noted follow-up).

Host wiring (CudaHybridGdnForwardPass): GdnBlockBatched / AttnBlockBatched
use the batched paths behind SHARPI_BATCHED_GDN_SCAN / SHARPI_BATCHED_ATTN
(default on, A/B-toggleable like SHARPI_BATCHED_TRUNK).

Parity (hard constraint)

• 6 new per-kernel bit-exactness tests (CudaGdnBatchedTrunkTests): conv1d +
  state-advance, L2-norm, tile-heads, fused recurrence scan (output and final
  state), batched KV-append + SDPA (fp32 and bf16) — all bit-identical to the
  single-token kernels.
• Existing oracles green: BatchedTrunk_BitwiseMatchesSequentialTrunk_Carnice,
  BatchedPrefill_BitwiseMatchesSequential_Carnice (single + multi-chunk),
  MtpDecoder_GreedyParity_LlamaCpp.
• Full Tests.ForwardPass suite: 0 failures. (A process-exit crash during a
  Vulkan-shader test's teardown is a pre-existing, non-deterministic GPU-driver flake,
  unrelated to these CUDA changes.)

Performance (warm, RTX 4070 Ti)

A/B vs the #111 per-position trunk (SHARPI_BATCHED_GDN_SCAN/_ATTN off vs on):

• Trunk profiling at ~4K ctx: −35–50% trunk time, growth slope ~5× flatter
  (+330ms → +64ms per 512-token chunk); routedMoE unchanged (perf(engine,cpu,cuda): remaining GDN-hybrid prefill headroom after #111/#112 (N-input MoE dots + per-position recurrence/SDPA batching) #114-A preserved).
• End-to-end prefill at ~4K ctx: +17% (Qwen3.6-35B-A3B and -MTP), +31% (Carnice).
• The win scales with context — on a trivially short prompt it's within noise (the
  batched kernels need real context to amortise over).

README

The throughput table's Prefill t/s column was previously measured on ~10–60-token
prompts (launch-overhead noise). Re-benchmarked every on-disk row at a uniform
~1K-token warm-cache context so the column is comparable; decode left as the
near-zero-ctx generation rate (added a methodology note). Llama-4 Scout (not on disk)
and the Qwen3-Coder Vulkan-hybrid row (errored at 1K) keep their prior values, annotated.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request implements batched GDN trunk and batched-query scaled dot-product attention (SDPA) kernels (Issue #114-B) to collapse per-position decode launches into single batched launches, significantly improving prompt prefill performance. It includes a new benchmarking script (bench-allrows-1k.ps1) and updates the README with consistent ~1K-token prefill measurements. Additionally, a comprehensive suite of bit-exactness tests is added to ensure the batched implementations match sequential outputs. Review feedback identified a potential stack buffer overflow in the llm_gdn_conv1d_state_update_batched CUDA kernel where the local array tmp is hardcoded to size 4, recommending a larger size and a safety guard for larger kernel sizes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The local array tmp is hardcoded to size 4 with the assumption that kernel_size - 1 <= 4. If a model with a larger kernel_size is loaded, this will cause a stack buffer overflow/corruption in the CUDA kernel. To prevent undefined behavior or GPU crashes, we should increase the array size to a safe maximum (e.g., 16) and add a guard to safely handle cases where retained exceeds this limit.

    int retained = kernel_size - 1;
    if (retained > 16) return;
    float tmp[16];
    #pragma unroll