TurboQuant: Apple Accelerate + norm correction for CPU dequant#21817
TurboQuant: Apple Accelerate + norm correction for CPU dequant#21817sachmans wants to merge 5 commits into
Conversation
Replace scalar/AVX2 matrix-vector multiply in TurboQuant rotation with cblas_sgemv from Apple's Accelerate framework. This uses the AMX coprocessor on Apple Silicon for the 128x128 dense rotation matrix multiply during dequantization. Results on Mac Studio M3 Ultra, Qwen3-32B Q4_K_M: TBQ4_0 K + F16 V: 7.40 → 14.00 t/s (1.89x faster, 86% of F16) TBQ4_0 K + TBQ4_0 V: 5.32 → 11.04 t/s (2.07x faster, 67% of F16) Quality unchanged (PPL within noise margin).
After inverse rotation, re-normalize the reconstructed unit vector to unit length before scaling by the original norm. This removes magnitude bias from codebook quantization error. Results on Qwen3-32B Q4_K_M (PPL, lower is better): TBQ4_0 K+V: 2.751 → 2.733 (+1.3% → +0.6% vs F16) TBQ4_0 K+F16 V: 2.733 → 2.724 (+0.6% → +0.3% vs F16) TBQ4_0 K+TBQ3_0 V: 2.767 → 2.730 (+1.9% → +0.5% vs F16) 4.5x compression at +0.5% quality loss (TBQ4_0 K + TBQ3_0 V).
|
Hi @sachmans, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
Summary
Two CPU-side improvements to the TurboQuant KV cache dequant path (builds on PR #21089):
Apple Accelerate
cblas_sgemv: Replace the hand-rolled matrix-vector multiply in TBQ dequant withcblas_sgemvfrom Apple's Accelerate framework, which routes to AMX on M-series silicon. 2× faster dequant on Apple Silicon. Guarded by#ifdef GGML_USE_ACCELERATE, zero impact on other platforms.Norm correction: Re-normalize the unit vector after Householder inverse rotation to correct accumulated floating-point error. Halves the PPL gap vs F16 (from +0.6% to +0.3% on Qwen3-32B wikitext-2).
Benchmark results (Qwen3-32B Q4_K_M, Mac Studio M3 Ultra 96GB)
Max context (Qwen3-32B on 96GB)
Files changed
ggml/src/ggml-turboq.c— Accelerate cblas_sgemv + norm correctionggml/src/CMakeLists.txt— Accelerate framework link (if not already linked)Test plan
#ifdefguarded)Depends on: #21089
🤖 Generated with Claude Code