TurboQuant: Accelerate cblas + norm correction for CPU dequant#23049
TurboQuant: Accelerate cblas + norm correction for CPU dequant#23049sachmans wants to merge 5 commits into
Conversation
|
Hi @sachmans, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
Replace scalar/AVX2 matrix-vector multiply in TurboQuant rotation with cblas_sgemv from Apple's Accelerate framework. This uses the AMX coprocessor on Apple Silicon for the 128x128 dense rotation matrix multiply during dequantization. Results on Mac Studio M3 Ultra, Qwen3-32B Q4_K_M: TBQ4_0 K + F16 V: 7.40 → 14.00 t/s (1.89x faster, 86% of F16) TBQ4_0 K + TBQ4_0 V: 5.32 → 11.04 t/s (2.07x faster, 67% of F16) Quality unchanged (PPL within noise margin).
After inverse rotation, re-normalize the reconstructed unit vector to unit length before scaling by the original norm. This removes magnitude bias from codebook quantization error. Results on Qwen3-32B Q4_K_M (PPL, lower is better): TBQ4_0 K+V: 2.751 → 2.733 (+1.3% → +0.6% vs F16) TBQ4_0 K+F16 V: 2.733 → 2.724 (+0.6% → +0.3% vs F16) TBQ4_0 K+TBQ3_0 V: 2.767 → 2.730 (+1.9% → +0.5% vs F16) 4.5x compression at +0.5% quality loss (TBQ4_0 K + TBQ3_0 V).
6a2e0ac to
14b0868
Compare
|
I've been running TurboQuant locally for a couple of weeks on a Mac Studio M3 Ultra (96GB) and wanted to share what I've seen in practice. The biggest win is running multiple large models simultaneously. With TBQ4_0 K + TBQ3_0 V as my default KV cache, I can keep both Qwen3-32B and Llama 3.1 8B loaded at the same time in router mode — something that wasn't practical with f16 KV because the combined cache pressure would push past available memory under real workloads. Quality-wise, I haven't noticed any degradation in daily use across coding tasks, reasoning, and general chat. Still doing more checks. I did notice it only works well with dense-attention models (Qwen3-32B, Llama 3.1 8B). Not beneficial for models with mostly sliding-window attention (e.g. Gemma 3 4B) so run them without the optimisation. |
|
@ggml-org/maintainers can we get this account banned? violated for the 3rd time: https://github.com/ggml-org/llama.cpp/issues?q=is%3Apr%20state%3Aclosed%20author%3Asachmans |
Summary
Two improvements to the TurboQuant CPU dequant path from PR #21089:
1. Apple Accelerate
cblas_sgemv— Replaces the hand-rolled matrix-vector multiply withcblas_sgemvfrom Apple's Accelerate framework, which routes to AMX on M-series silicon. Guarded by#if defined(__APPLE__), zero impact on other platforms. The existing scalar and AVX2 paths are retained as fallbacks.2. Norm correction — Re-normalizes the reconstructed unit vector after inverse Householder rotation to correct accumulated floating-point error. This is a single
sqrtf+ scale per block and halves the perplexity gap vs F16.Changes
ggml/src/ggml-turboq.c—cblas_sgemvformatvec,matvec_row,matvec_ton Apple; norm correction indequantize_row_tbq3_0anddequantize_row_tbq4_0ggml/src/CMakeLists.txt— Link Accelerate framework when availableBenchmarks (Qwen3-32B Q4_K_M, Mac Studio M3 Ultra 96GB, wikitext-2)
Without norm correction the PPL gap is roughly double (+0.6% → +1.1%).
Disclosure
AI tools (Claude) were used to assist with writing and testing this code. All changes were reviewed and validated by a human.
Depends on: #21089