Skip to content

perf(cuda,engine): extend batched-prefill trunk to dense GDN-hybrid + GPU-SLRU MoE paths (#114 follow-up) #119

Description

@pekkah

Follow-up to #110/#111/#114-B. The batched-prefill harness (PrefillBatchedCpuMoeTrunkLayerBatchedGdnBlockBatched/AttnBlockBatched) is gated on _cpuMoe:

// CudaHybridGdnForwardPass.Prefill
if (BatchedPrefillEnabled && _cpuMoe && batchedSafe) { PrefillBatchedCpuMoe(...); }
else { for (i..N) Forward(tokens[i], startPos+i); }   // sequential per-token prefill

So two CUDA GDN-hybrid configurations still prefill sequentially per token and get neither the #111 GEMM-batched projections nor the #114-B fused GDN scan / batched-query SDPA:

  1. Dense GDN-hybrid (!hp.IsMoE, e.g. Qwen3.6-27B-MTP) — _cpuMoe = false (constructor line ~809). Prefill is per-token ForwardGpuGdnBlockAt / GpuAttnBlockAt + per-layer CPU/GPU dense FFN.
  2. GPU-SLRU MoE (_cpuMoe = false when SLRU capacity ≥ ~50% of experts, e.g. Qwen3-Coder-30B on CUDA) — same sequential prefill loop.

The trunk batching (#111 + #114-B) is independent of how the FFN/MoE runs — it only needs the per-token GDN/attn launches collapsed — so it should extend to both paths. The routed-MoE amortization (#110/#112/#114-A) is CPU-MoE-specific and would not apply to the SLRU path, but the trunk half would.

Scope

  • Factor the trunk-batching path (TrunkLayerBatched + the host residual/norm/shared-expert plumbing) so it can drive a batched prefill for the dense and SLRU configs, with the FFN/MoE stage swapped per config (dense FFN batched à la CpuDenseFfn2 / on-GPU FFN; SLRU routed experts).
  • Keep the _faulted non-transactional latch and the same bit-exactness contract.

Tests

  • Extend BatchedPrefill_BitwiseMatchesSequential coverage to a dense GDN-hybrid model (27B-MTP) and a forced GPU-SLRU MoE config (SHARPI_CPU_MOE=0 on a model that fits SLRU), asserting bit-identical prefill logits vs the sequential loop.

Payoff

Prefill speedup for the 27B-MTP rows (sequential today) and the Coder-30B CUDA-SLRU row; quantify with bench-prefill.ps1 / bench-allrows-1k.ps1. (Vulkan and CPU-only GDN-hybrid backends would need their own batched kernels — out of scope here; track separately if wanted.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions