Problem
RunCommand.DecodeLoop (src/SharpInference.Cli/RunCommand.cs, ~line 1025) logs the top-5 next-token candidates every decode step when --verbose-prompt is set:
var logitsArr = logits.ToArray(); // ~1 MB alloc + copy (Gemma 4 vocab = 262144)
var top5 = Enumerable.Range(0, logitsArr.Length)
.OrderByDescending(j => logitsArr[j]).Take(5) ... // full-vocab sort, every token
For a 262144-token vocab this is a full O(V·logV) LINQ sort plus a 1 MB allocation per token — ~1.5–2.5 ms/token of CPU overhead. It badly skews decode t/s for anyone benchmarking or generating with --verbose-prompt on.
Surfaced while diagnosing #142 — scripts/bench-textgen.ps1 hard-coded --verbose-prompt, understating decode by 10–25%. The bench was fixed in #154 (flag removed), but the underlying logging path is still O(V·logV)/token for direct CLI users.
Fix
Replace the OrderByDescending().Take(5) with a partial top-5 selection (single O(V) pass keeping the 5 best, like Sampler does for top-k) and drop the logits.ToArray() copy — read the span directly. This makes --verbose-prompt a near-free debug aid instead of a per-token full sort.
Low priority (debug flag only), but cheap and removes a foot-gun for benchmarking.
Problem
RunCommand.DecodeLoop(src/SharpInference.Cli/RunCommand.cs, ~line 1025) logs the top-5 next-token candidates every decode step when--verbose-promptis set:For a 262144-token vocab this is a full O(V·logV) LINQ sort plus a 1 MB allocation per token — ~1.5–2.5 ms/token of CPU overhead. It badly skews decode t/s for anyone benchmarking or generating with
--verbose-prompton.Surfaced while diagnosing #142 —
scripts/bench-textgen.ps1hard-coded--verbose-prompt, understating decode by 10–25%. The bench was fixed in #154 (flag removed), but the underlying logging path is still O(V·logV)/token for direct CLI users.Fix
Replace the
OrderByDescending().Take(5)with a partial top-5 selection (single O(V) pass keeping the 5 best, likeSamplerdoes for top-k) and drop thelogits.ToArray()copy — read the span directly. This makes--verbose-prompta near-free debug aid instead of a per-token full sort.Low priority (debug flag only), but cheap and removes a foot-gun for benchmarking.