Skip to content

cli: --verbose-prompt does a full-vocab LINQ sort per decode token #155

Description

@pekkah

Problem

RunCommand.DecodeLoop (src/SharpInference.Cli/RunCommand.cs, ~line 1025) logs the top-5 next-token candidates every decode step when --verbose-prompt is set:

var logitsArr = logits.ToArray();                       // ~1 MB alloc + copy (Gemma 4 vocab = 262144)
var top5 = Enumerable.Range(0, logitsArr.Length)
    .OrderByDescending(j => logitsArr[j]).Take(5) ...    // full-vocab sort, every token

For a 262144-token vocab this is a full O(V·logV) LINQ sort plus a 1 MB allocation per token — ~1.5–2.5 ms/token of CPU overhead. It badly skews decode t/s for anyone benchmarking or generating with --verbose-prompt on.

Surfaced while diagnosing #142scripts/bench-textgen.ps1 hard-coded --verbose-prompt, understating decode by 10–25%. The bench was fixed in #154 (flag removed), but the underlying logging path is still O(V·logV)/token for direct CLI users.

Fix

Replace the OrderByDescending().Take(5) with a partial top-5 selection (single O(V) pass keeping the 5 best, like Sampler does for top-k) and drop the logits.ToArray() copy — read the span directly. This makes --verbose-prompt a near-free debug aid instead of a per-token full sort.

Low priority (debug flag only), but cheap and removes a foot-gun for benchmarking.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions