(AI policy violation) Metal: TurboQuant GPU dequant kernels + host buffer type

[sachmans]

Summary

Adds Metal GPU support for TurboQuant KV cache types (TBQ3_0/TBQ4_0) on Apple Silicon. Two components:

Stage 1: Metal dequant kernels

• GPU compute kernels for TBQ3_0/TBQ4_0 → F16/F32 conversion via GGML_OP_CPY
• Performs inverse Householder QR rotation (baked 128×128 matrix) and Lloyd-Max codebook lookup entirely on GPU
• 4 kernel variants: kernel_cpy_tbq{3,4}_0_{f16,f32}

Stage 2: Metal host buffer type (MTL0_Host)

• New buffer type backed by shared-storage MTLBuffer that reports is_host = true
• Tensors allocated here are accessible by both the Metal backend (GPU dequant via CPY) and the CPU backend (SET_ROWS for F32→TBQ quantize)
• The ggml scheduler routes each op to the appropriate backend automatically
• Follows the same pattern as CUDA's ggml_backend_cuda_host_buffer_type() (pinned host memory)

Architecture

Token generation:
  Attention scores → SET_ROWS (F32→TBQ quantize) → CPU backend claims this op
  KV read → CPY (TBQ→F16 dequant) → Metal backend claims this op
  Both operate on the SAME tensor in MTL0_Host shared memory

This avoids needing a Metal SET_ROWS (forward rotation + quantize) kernel, which would be significantly more complex.

Benchmark results (Gemma 3 4B Q4_K_M, Mac Studio M3 Ultra)

K cache	V cache	pp512 (t/s)	tg64 (t/s)
f16	f16	2210	126.7
q4_0	f16	2153	117.9
tbq4_0	f16	869	38.6
tbq3_0	f16	938	37.2

Memory (KV cache @ 8K context)

Config	KV size	vs f16
f16 K+V	1024 MiB	1.00×
tbq4_0 K+V	260 MiB	3.94×
tbq4_0 K + tbq3_0 V	228 MiB	4.49×

Accuracy verification (Gemma 3 4B, greedy decoding)

All configs (f16, tbq4_0, tbq3_0, tbq4_0+tbq4_0) produce correct factual answers ("The three primary colors are red, yellow, and blue") with coherent elaboration. TBQ matches F16 baseline quality.

Files changed

• ggml/src/ggml-metal/ggml-metal.metal — TBQ dequant kernel implementations
• ggml/src/ggml-metal/ggml-metal-turboq-rot.h — Baked 128×128 Householder rotation matrix
• ggml/src/ggml-metal/ggml-metal.cpp — Host buffer type + device callback
• ggml/src/ggml-metal/ggml-metal-ops.cpp — TBQ CPY dispatch
• ggml/src/ggml-metal/ggml-metal-device.m — TBQ op whitelist
• ggml/src/ggml-metal/CMakeLists.txt — Header inclusion
• src/llama-kv-cache.cpp — Host buft routing for TBQ types

Test plan

• Build clean on macOS (M3 Ultra)
• llama-bench speed comparison across f16/q4_0/tbq4_0/tbq3_0
• KV buffer allocation confirmed on MTL0_Host (not CPU fallback)
• Generation correctness on Gemma 3 4B (greedy, deterministic)
• Generation correctness on Llama 3.1 8B (parity with f16 baseline)
• test-backend-ops for CPY on TBQ types
• Verify no regression for non-TBQ KV types

Depends on: #21089

🤖 Generated with Claude Code

[ggml-gh-bot]

Hi @sachmans, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[ddh0]

🤖 Generated with Claude Code

This violates the AI Usage Policy as stated in CONTRIBUTING.md:

This project does not accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.

[ngxson]

I believe this account should be banned if they continue to violate project's policy.