Follow-up to #110/#111/#114-B. The batched-prefill harness (PrefillBatchedCpuMoe → TrunkLayerBatched → GdnBlockBatched/AttnBlockBatched) is gated on _cpuMoe:
// CudaHybridGdnForwardPass.Prefill
if (BatchedPrefillEnabled && _cpuMoe && batchedSafe) { PrefillBatchedCpuMoe(...); }
else { for (i..N) Forward(tokens[i], startPos+i); } // sequential per-token prefill
So two CUDA GDN-hybrid configurations still prefill sequentially per token and get neither the #111 GEMM-batched projections nor the #114-B fused GDN scan / batched-query SDPA:
- Dense GDN-hybrid (
!hp.IsMoE, e.g. Qwen3.6-27B-MTP) — _cpuMoe = false (constructor line ~809). Prefill is per-token Forward → GpuGdnBlockAt / GpuAttnBlockAt + per-layer CPU/GPU dense FFN.
- GPU-SLRU MoE (
_cpuMoe = false when SLRU capacity ≥ ~50% of experts, e.g. Qwen3-Coder-30B on CUDA) — same sequential prefill loop.
The trunk batching (#111 + #114-B) is independent of how the FFN/MoE runs — it only needs the per-token GDN/attn launches collapsed — so it should extend to both paths. The routed-MoE amortization (#110/#112/#114-A) is CPU-MoE-specific and would not apply to the SLRU path, but the trunk half would.
Scope
- Factor the trunk-batching path (
TrunkLayerBatched + the host residual/norm/shared-expert plumbing) so it can drive a batched prefill for the dense and SLRU configs, with the FFN/MoE stage swapped per config (dense FFN batched à la CpuDenseFfn2 / on-GPU FFN; SLRU routed experts).
- Keep the
_faulted non-transactional latch and the same bit-exactness contract.
Tests
- Extend
BatchedPrefill_BitwiseMatchesSequential coverage to a dense GDN-hybrid model (27B-MTP) and a forced GPU-SLRU MoE config (SHARPI_CPU_MOE=0 on a model that fits SLRU), asserting bit-identical prefill logits vs the sequential loop.
Payoff
Prefill speedup for the 27B-MTP rows (sequential today) and the Coder-30B CUDA-SLRU row; quantify with bench-prefill.ps1 / bench-allrows-1k.ps1. (Vulkan and CPU-only GDN-hybrid backends would need their own batched kernels — out of scope here; track separately if wanted.)
Follow-up to #110/#111/#114-B. The batched-prefill harness (
PrefillBatchedCpuMoe→TrunkLayerBatched→GdnBlockBatched/AttnBlockBatched) is gated on_cpuMoe:So two CUDA GDN-hybrid configurations still prefill sequentially per token and get neither the #111 GEMM-batched projections nor the #114-B fused GDN scan / batched-query SDPA:
!hp.IsMoE, e.g. Qwen3.6-27B-MTP) —_cpuMoe = false(constructor line ~809). Prefill is per-tokenForward→GpuGdnBlockAt/GpuAttnBlockAt+ per-layer CPU/GPU dense FFN._cpuMoe = falsewhen SLRU capacity ≥ ~50% of experts, e.g. Qwen3-Coder-30B on CUDA) — same sequential prefill loop.The trunk batching (#111 + #114-B) is independent of how the FFN/MoE runs — it only needs the per-token GDN/attn launches collapsed — so it should extend to both paths. The routed-MoE amortization (#110/#112/#114-A) is CPU-MoE-specific and would not apply to the SLRU path, but the trunk half would.
Scope
TrunkLayerBatched+ the host residual/norm/shared-expert plumbing) so it can drive a batched prefill for the dense and SLRU configs, with the FFN/MoE stage swapped per config (dense FFN batched à laCpuDenseFfn2/ on-GPU FFN; SLRU routed experts)._faultednon-transactional latch and the same bit-exactness contract.Tests
BatchedPrefill_BitwiseMatchesSequentialcoverage to a dense GDN-hybrid model (27B-MTP) and a forced GPU-SLRU MoE config (SHARPI_CPU_MOE=0on a model that fits SLRU), asserting bit-identical prefill logits vs the sequential loop.Payoff
Prefill speedup for the 27B-MTP rows (sequential today) and the Coder-30B CUDA-SLRU row; quantify with
bench-prefill.ps1/bench-allrows-1k.ps1. (Vulkan and CPU-only GDN-hybrid backends would need their own batched kernels — out of scope here; track separately if wanted.)