perf(engine): TierPlanner prices Gemma 4 KV at uniform full-context fp32 — runtime uses SWA rings, KV-share aliasing and per-layer head_dim, so auto -g/ctx underestimates capacity ~4-6×

[pekkah]

Problem

TierPlanner.Plan prices GPU KV uniformly: 2 × gpuLayers × NumKvHeads × HeadDim × sizeof(float) × ctx for every layer (TierPlanner.cs:97-126). The Gemma 4 runtime allocates nothing like that:

• SWA layers get window-sized rings, not full-context buffers: CudaForwardPass.cs:809,833-834 — layerCtx = SwaRingSize(_maxSeqLen, swaWindow) when hp.IsSwaLayer[i]. Gemma 4 is ~5:1 SWA:global with a window far smaller than typical contexts.
• KV-share layers allocate no KV at all — they alias the source layer's K/V (CudaForwardPass.cs:802-803: "KV-share layers (KvSourceLayer[L] >= 0) alias the…").
• Per-layer head_dim (Gemma 4's mixed dims, cf. feat(cuda): Gemma 4 continuous batching — per-layer head_dim / SWA rings / shared-KV / PLE (#190 follow-up) #195) is priced at the single hp.HeadDim.
• KV dtype is priced fp32 even when SHARPI_KV_DTYPE=bf16/q8_0 narrows it 2–4× (also noted in perf(cuda,engine): CPU-MoE mode + expert-aware TierPlanner for CudaHybridForwardPass — auto placement is structurally wrong for pure-attention MoE on 12 GB #215; perf(cuda): decode throughput degrades at long context with q8_0 KV (12 vs 65 tok/s) #213's live config runs q8_0 KV).

Net effect: for Gemma 4 the planner overestimates per-token KV cost by roughly 4–6× (more with narrowed KV). On a 12 GB card with -g -1, that means fewer GPU layers and/or a much smaller RecommendedCtxSize than what actually fits — the loader's ClampGemma4KvShareBoundary (InferenceEngineLoader.cs:216) already patches one Gemma-4 blind spot in the planner, and this is the rest of the family.

This matters most for the hybrid configs (Gemma 4 12B-class on 12 GB, where every reclaimed GB is another GPU layer) and for auto-context on E4B long-context runs.

Proposed work

1. Teach TierPlanner the per-layer KV shape: when hp.IsSwaLayer is populated, price SWA layers at SwaRingSize(ctx, hp.SlidingWindowSize) positions; skip KV-share layers (hp.KvSourceLayer[L] >= 0); use per-layer head_dim where the hyperparams carry it. Reuse the same helpers CudaForwardPass uses so the two can't drift (single source of truth for ring sizing).
2. Accept the resolved KV dtype (already proposed in perf(cuda,engine): CPU-MoE mode + expert-aware TierPlanner for CudaHybridForwardPass — auto placement is structurally wrong for pure-attention MoE on 12 GB #215 item 3) so bf16/q8_0 runs budget correctly.
3. Re-derive RecommendedCtxSize from the corrected per-token cost.

Acceptance

• A planner unit test (GGUF-free, cf. Test coverage: GGUF-free regression tests for k_eq_v detection + TierPlanner packed sizing (PR #174 follow-up) #175's pattern) with a synthetic Gemma-4-shaped ModelHyperparams (SWA pattern + KV-share + per-layer head_dim) asserting planned GpuKvBytes matches the runtime allocator's arithmetic.
• On a 12 GB profile, Gemma 4 12B -g -1 gains GPU layers and/or context vs today's plan; no change for uniform-attention models.
• No OOM regression: validate the plan's headroom on an actual long-context run (the planner must not under-reserve once SWA-aware).

References

• Gemma 4 SWA KV cache: window-sized buffer vs absolute indexing → OOB when ctx > sliding_window #138/SWA KV ring: bf16-append wrap + CudaHybridForwardPass batched path are latent-only / untested #166 (SWA ring allocation), feat(cuda): Gemma 4 continuous batching — per-layer head_dim / SWA rings / shared-KV / PLE (#190 follow-up) #195 (per-layer head_dim), feat(loader): support Gemma 4 E4B QAT q4_0 GGUF (missing attn_k_norm tensor layout) #211 (Gemma 4 E4B QAT), perf(cuda): decode throughput degrades at long context with q8_0 KV (12 vs 65 tok/s) #213 (q8_0-KV live config that had to hand-tune ctx), perf(cuda,engine): CPU-MoE mode + expert-aware TierPlanner for CudaHybridForwardPass — auto placement is structurally wrong for pure-attention MoE on 12 GB #215 (MoE/KV-dtype half of the planner work), Test coverage: GGUF-free regression tests for k_eq_v detection + TierPlanner packed sizing (PR #174 follow-up) #175 (planner test pattern).

[pekkah]

Findings from an empirical probe (RTX 4070 Ti 12 GB, master 56abf87)

Ran gemma-4-12b-it-qat-q4_0.gguf (48L, headDim=512, 262144 vocab, model max ctx 262144) with -g -1 --backend cuda across all three KV dtypes:

--kv-type	auto-ctx	expected if dtype-priced
fp32	1770	1770
bf16	1770	~3540 (2×)
q8_0	1770	~7080 (4×)

It runs full-GPU (CudaForwardPass, all 48 layers uploaded — not hybrid), and SnapKV auto-enables citing "full cache ~2655 MiB" at that ctx. So a Gemma 4 12B user on 12 GB silently gets 1770 tokens of context regardless of --kv-type, vs a 262144 model max.

Root cause is narrower (and different) than the issue title

1. The dense full-GPU path does NOT use TierPlanner.RecommendedCtxSize. RunCommand.cs:536 constructs new CudaForwardPass(model, cuda, hp, ctxSize, …) with the raw requested ctxSize (0 when -c is unset) and throws away placement.RecommendedCtxSize. TierPlanner is run (:450, with kvDtype passed at :451) but only placement.GpuLayers is consumed for the full-GPU case.

2. The auto-ctx actually comes from CudaForwardPass.EstimateMaxContext (CudaForwardPass.cs:569, the ctxSize==0 branch → 1770).

3. EstimateMaxContext is ALREADY SWA-aware — it has a full per-layer / SWA-ring / KV-share branch (:4086-4127: skips aliased layers, caps SWA layers at SwaRingSize, uses per-layer LayerHeadDim). So the "teach the planner about SWA rings" half of this issue is effectively already implemented here.

4. The live bug is dtype-blindness: EstimateMaxContext:4100 hardcodes 2L * hp.NumKvHeads * lhd[i] * sizeof(float) — always fp32, never the resolved --kv-type. That is exactly why bf16/q8_0 don't expand auto-ctx (1770 flat across all three). It also uses hp.NumKvHeads rather than per-layer LayerKvHeads, and skips the pow2 alloc round-up, so it can diverge from the real allocator estimate EstimateKvCacheBytes (:3994-4021), which is dtype + per-layer-KV-heads + round-up aware.

Revised proposed work for the dense path (the 12 GB-user case)

• Make the dense auto-ctx dtype-aware. Cleanest: have EstimateMaxContext (or the dense CudaForwardPass ctor) binary-search ctx against EstimateKvCacheBytes(hp, ctx, resolvedKvDType, gpuLayers) — the single source of truth that already matches the allocator (SWA rings, KV-share skip, per-layer KV heads, pow2 round-up). That subsumes the hand-rolled :4086-4127 math and fixes the NumKvHeads/round-up drift at the same time.
• Alternatively/additionally, thread placement.RecommendedCtxSize (TierPlanner already computes it SWA+dtype-aware via SolveGpuCtxForPerLayerKv → EstimateKvCacheBytes, :109/:186-197) into the dense CudaForwardPass construction at RunCommand.cs:536 when -c is unset, so the two estimators can't disagree.
• The TierPlanner half of this issue still applies to the hybrid path; verify both paths converge on the allocator's arithmetic (the acceptance test below).

Acceptance refinement

• Add the GGUF-free planner/estimator unit test as proposed, and assert EstimateMaxContext (or whatever feeds the dense ctor) responds to KV dtype: bf16 ≈ 2× fp32, q8_0 ≈ 4× fp32 on a synthetic Gemma-shaped ModelHyperparams.
• Manual check the fix closes the observed gap: gemma-4-12b-it-qat-q4_0 -g -1 --kv-type q8_0 should jump well past 1770 (target tens of K), and a long-context run must not OOM.