feat: Bf16 KV cache for hybrid GDN models (#27)#56
Merged
Conversation
Default the CUDA hybrid GDN forward pass KV cache to bf16 (was fp32), halving cache VRAM on the 16 attention layers without touching arithmetic precision — bf16→fp32 promotion happens at the kernel read sites, fp32 accumulators throughout. SHARPI_KV_DTYPE=fp32 reverts. On Qwen3.6-27B-MTP at ctx=4096 the freed ~128 MiB admits one more dense FFN layer on GPU (22/64 vs 21/64), lifting --temp 0 decode 4.3 → 6.4 t/s. Output is bit-identical vs fp32 cache. Greedy first-token parity vs the CPU/llama.cpp baseline still holds on qwen35moe with bf16 active. New CUDA NVRTC kernels llm_kv_append_bf16 / llm_attention_bf16 mirror their fp32 counterparts; CudaBackend.Clear is now dtype-aware so it doesn't over-write the smaller bf16 buffers. New parity test CudaHybridGdnForwardPass_Qwen35Mtp_Bf16KvCache_GreedyMatchesFp32 asserts both top-1 match and a < 0.5 absolute logit gap on top-16. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced May 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SHARPI_KV_DTYPE=fp32reverts). Arithmetic stays fp32 — bf16 → fp32 promotion happens at the kernel read sites.llm_kv_append_bf16/llm_attention_bf16mirror their fp32 counterparts;CudaBackend.Clearis now dtype-aware so it can't over-write the smaller bf16 buffers.CudaHybridGdnForwardPass_Qwen35Mtp_Bf16KvCache_GreedyMatchesFp32) asserts top-1 match and a < 0.5 absolute gap on top-16 logits between bf16 and fp32 cache runs.Measured (Qwen3.6-27B-MTP Q4_K_M, --temp 0, ctx=4096, RTX 4070 Ti)
Output is bit-identical between the two paths on this prompt. Greedy first-token parity vs the CPU/llama.cpp baseline still holds on qwen35moe with bf16 active.
Test plan
dotnet test tests/SharpInference.Tests.ForwardPass -c Release— all 239 tests pass, including the newBf16KvCache_GreedyMatchesFp32parity case🤖 Generated with Claude Code