perf(gemma4-cuda): MMQ + flash-attention prefill + dp4a decode (#141/#142)

README.md

@@ -51,13 +51,19 @@ coherent (`scripts/bench-all.ps1`); top-1 parity vs llama.cpp b8585 verified on
 | Qwen3.6-35B-A3B-MTP (GDN+MoE) | (same) | 22 GB | **CUDA** `-g -1 --no-thinking` (hybrid) | **65.0** | **22.9** | Requires `SHARPI_CPU_MOE=1`: 30 GDN + 10 attn + shared expert on GPU, routed experts CPU mmap. 100% acceptance. Fused GDN scan + batched SDPA (#114-B/#118), bit-identical, grows with ctx |
 | Carnice (Qwen3.6-35B-A3B-MTP finetune) | [mudler](https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF) | 17 GB | **CUDA** `-g -1 --no-thinking` (hybrid) | **43.6** | **25.0** | agentic finetune of 35B-A3B-MTP; 77% acceptance (`bench-carnice.ps1` — the default prompt 1-token-EOSes on this terser tune). APEX mixed-precision (Q3_K + Q8_0 experts); Q8_KS per-32 int dots auto-enable at load (#99/#101/#107), +4.6% decode at ~4× tighter parity vs plain Q8_K (`SHARPI_Q3K_Q8K=0`/`SHARPI_Q8_0_Q8K=0` to disable). Fused GDN scan + wave SDPA (#114-B/#118) bit-identical past 4096 |
 | Gemma 4 E4B-it Q8 | [unsloth](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | 8 GB | CPU | 4.9 | 5.0 | dense 42-layer gemma4: per-layer head_dim (256 SWA / 512 global), dual-RoPE, KV-share tail (18 layers), 5:1 SWA:global, logit softcap 30, PLE-256 injection (~4.2 GB mmap-resident) |
-| Gemma 4 E4B-it Q8 | (same) | 8 GB | **CUDA** `-g -1 -c 2048` | **109.4** | **49.1** | all 42 layers fit at `-c 2048`. KV-share alias + SWA/global split per layer; PLE projections (~215 MB) upload at construction. Batched-trunk prefill (#136, bit-identical; `SHARPI_BATCHED_PREFILL=0` → 53.4 t/s to bisect) collapses the per-position attention/FFN/PLE launches into batched GEMM-N + batched SWA/full attention (≈2.05× prefill). Launch-bound decode also gains (44→49) from the fused PLE projection, dropped Q-prescale, and fused Q/K HeadNorm |
+| Gemma 4 E4B-it Q8 | (same) | 8 GB | **CUDA** `-g -1 -c 2048` | **2853** | **59** | all 42 layers fit at `-c 2048`. KV-share alias + SWA/global split per layer; PLE projections (~215 MB) upload at construction. **Prefill (#141):** int8 **tensor-core MMQ** matmul (`mma.m16n8k32.s8`, each Q8_0 weight read once as int8 — beats the dequant→fp16→cuBLAS GEMM, drops its fp16 HBM temp; `SHARPI_PREFILL_MMQ=0` reverts) + a memory-efficient **flash-attention** prefill (shared K/V tiles + online softmax + half2 fp16x2 QK dot — replaces the scalar O(n²) per-query attention that re-streamed each query's K/V window up to ~512×; `SHARPI_PREFILL_FLASH=0` reverts) + a batched Q8_0 embedding lookup. **~1.8× at ~1K ctx (1564→2853), ~2.05× at 1.8K** — profiling showed *attention*, not the matmul, was the dominant prefill cost at realistic prompt lengths, so the win grows with prompt length. **Decode (#142):** dp4a/Q8_1 int8 matvec (`SHARPI_Q80_DP4A=0` to bisect) + CUDA-graph capture/replay default-on (`SHARPI_CUDA_GRAPH=0` to bisect). All prefill/decode fast paths are argmax-stable vs the fp32 path, not bit-exact. Remaining gap to llama.cpp (~8000 prefill / ~78 decode): full tensor-core flash at d=512 + decode matvec work |
 | Gemma 4 E4B-it Q8 | (same) | 8 GB | **CUDA** `-g 22 -c 2048` (hybrid) | 6.6 | 6.8 | 22 GPU + 20 CPU layers. `-g ≤ 22` required so the CPU shared-KV tail can read its own-KV source layers; CPU dense-FFN dominates decode (bandwidth-bound). `SHARPI_CUDA_PROFILE=1` for per-phase breakdown |
 
 _Numbers re-measured across every on-disk row at ~1K ctx so the prefill column is comparable; per-issue
 before/after figures in the notes are historical. Llama-4 Scout and Qwen3-Coder Vulkan-hybrid keep their
 prior values (not re-runnable on the bench machine)._
 
+**Recommended sampling for Gemma 4 E4B-it:** `--temp 1.0 --top-k 64 --top-p 0.95 --min-p 0`
+(the Gemma 3/4 family defaults). Gemma 4 E4B-it is **not** a reasoning model, so the CLI now
+defaults `enable_thinking=false` for it automatically (no `--no-thinking` needed) — otherwise the chat
+template renders a `<think>` block the model wasn't trained to fill and the output degenerates. Greedy
+(`--temp 0`) is not recommended for it either; use the sampling values above.
+
 `--backend auto` (default) picks CUDA when available, sizing the GPU/CPU split from VRAM via TierPlanner;
 falls through to Vulkan only when CUDA isn't present. For hybrid `qwen35moe` models the CUDA backend keeps
 attention KV, the GDN layers, and the shared expert in VRAM; routed-expert dispatch auto-selects between

src/SharpInference.Cli/RunCommand.cs

@@ -230,9 +230,20 @@ protected override int Execute(CommandContext context, Settings settings, Cancel
             s_endThinkTokenId = endChannelId;
         }
 
+        // Gemma 4 E4B-it brackets a <|channel>thought block in its chat template but is
+        // NOT a reasoning model — rendering enable_thinking=true makes it try to fill a
+        // think section it wasn't trained for and the output degenerates. Default thinking
+        // OFF for Gemma 4 (its recommended config); --no-thinking still forces it off for
+        // every other model. Pass --temp 1.0 --top-k 64 --top-p 0.95 for Gemma 4.
+        bool modelDefaultsThinkingOff = s_arch == "gemma4";
+        s_noThinking = settings.NoThinking || modelDefaultsThinkingOff;
+        if (modelDefaultsThinkingOff && !settings.NoThinking)
+            AnsiConsole.MarkupLine("[dim]Gemma 4 is not a reasoning model — defaulting to --no-thinking " +
+                "(recommended: --temp 1.0 --top-k 64 --top-p 0.95).[/]");
+
         // Greedy on a reasoning model tends to "wait, but actually" itself into infinite
         // loops; --no-thinking sidesteps the issue since the model won't reason at all.
-        if (s_thinkTokenId > 0 && settings.Temperature == 0f && !settings.NoThinking)
+        if (s_thinkTokenId > 0 && settings.Temperature == 0f && !s_noThinking)
         {
             AnsiConsole.MarkupLine("[yellow]Warning:[/] Greedy decoding (--temp 0) on a reasoning model often produces");
             AnsiConsole.MarkupLine("infinite \"wait, but actually\" loops. Consider [yellow]--temp 0.6 --top-p 0.95 --top-k 20[/].");
@@ -668,7 +679,7 @@ private static int RunSpeculativeSinglePrompt(Settings s,
         ForwardPass target, ForwardPass draft,
         GgufTokenizer tok, SamplingParams sp)
     {
-        var prompt = FormatPrompt(s.Prompt!, s.SystemPrompt, enableThinking: !s.NoThinking);
+        var prompt = FormatPrompt(s.Prompt!, s.SystemPrompt, enableThinking: !s_noThinking);
         var tokens = tok.Encode(prompt);
 
         if (!s.NoDisplayPrompt)
@@ -726,7 +737,7 @@ private static int RunSpeculativeInteractive(Settings s,
             if (input is null or "/exit" or "/quit") break;
             if (string.IsNullOrWhiteSpace(input)) continue;
 
-            var prompt = FormatPrompt(input, s.SystemPrompt, enableThinking: !s.NoThinking);
+            var prompt = FormatPrompt(input, s.SystemPrompt, enableThinking: !s_noThinking);
             var tokens = tok.Encode(prompt);
 
             target.Cache.Reset();
@@ -775,7 +786,7 @@ private static int RunSinglePrompt(Settings s,
         GgufTokenizer tok, SamplingParams sp, Random rng,
         IForwardPass? mtpFwd)
     {
-        var prompt = FormatPrompt(s.Prompt!, s.SystemPrompt, enableThinking: !s.NoThinking);
+        var prompt = FormatPrompt(s.Prompt!, s.SystemPrompt, enableThinking: !s_noThinking);
         var tokens = tok.Encode(prompt);
 
         // SHARPI_RAW_PROMPT bypasses the chat template, so we need to add BOS
@@ -808,7 +819,7 @@ private static int RunSinglePrompt(Settings s,
         // MTP head AND sampling is greedy AND the user disabled thinking on the chat
         // template (--no-thinking) AND sp.SpecType permits. SHARPI_DISABLE_MTP=1 is
         // a back-compat off-switch that wins.
-        bool useMtp = ResolveCliMtp(mtpFwd, sp, s.NoThinking, out string? mtpReject);
+        bool useMtp = ResolveCliMtp(mtpFwd, sp, s_noThinking, out string? mtpReject);
         if (mtpReject != null)
         {
             AnsiConsole.MarkupLine($"[red]Error:[/] {Markup.Escape(mtpReject)}");
@@ -956,7 +967,7 @@ private static int RunInteractive(Settings s,
             if (input is null or "/exit" or "/quit") break;
             if (string.IsNullOrWhiteSpace(input)) continue;
 
-            var prompt = FormatPrompt(input, s.SystemPrompt, enableThinking: !s.NoThinking);
+            var prompt = FormatPrompt(input, s.SystemPrompt, enableThinking: !s_noThinking);
             var tokens = tok.Encode(prompt);
 
             resetCache();
@@ -1072,6 +1083,9 @@ private static bool EmitToken(int next, GgufTokenizer tok, Utf8StreamDecoder str
     }
 
     private static string s_arch = "qwen2"; // set during model load
+    // Effective "thinking off" state: --no-thinking OR a model whose recommended config
+    // disables reasoning (Gemma 4 E4B-it is not a reasoning model). Set during model load.
+    private static bool s_noThinking;
     private static int s_thinkTokenId = -1;    // <think> token for any model using the <think>/</think> special-token convention
     private static int s_endThinkTokenId = -1; // </think> token for any model using the <think>/</think> special-token convention
     private static JinjaChatTemplate? s_jinja;  // parsed from GGUF tokenizer.chat_template