Skip to content

fix(vllm): non-streaming tool-call regression after #10351#10638

Merged
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:fix/vllm-non-streaming-tool-parser-regression
Jul 2, 2026
Merged

fix(vllm): non-streaming tool-call regression after #10351#10638
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:fix/vllm-non-streaming-tool-parser-regression

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Summary

Non-streaming chat.completions with tools=[...] silently drops the tool
call (finish_reason: "stop", empty content, no tool_calls[]) when a
tool_parser is configured. Streaming requests are unaffected.

Root cause

#10351 introduced native streaming via parser.extract_tool_calls_streaming
and gated the post-loop extract_tool_calls block on
native_streaming and not native_streaming_error.

native_streaming is a capability flag — it becomes True as soon as
tp_request builds and the parser exposes extract_tool_calls_streaming.
It is not a state flag telling us the per-delta loop actually ran.

For non-streaming requests it is still True, so:

  • the if branch (extract_tool_calls on the full text) was skipped
  • the elif branch (content = "", "the stream already emitted everything")
    fired

The result: the tool call is neither extracted nor streamed — it's lost.

Fix

Add an explicit streaming and … guard on both branches so
extract_tool_calls runs for the non-streaming path (and for streaming
paths that fell back to the buffered mode). No change to streaming
requests where native streaming ran cleanly.

Reproduction

Model: saricles/Qwen3-Coder-Next-NVFP4-GB10, vLLM 0.24.0, LocalAI
v4.5.6, coder.yaml with options: - tool_parser:qwen3_coder.

curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"coder","stream":false,
       "messages":[{"role":"user","content":"7*8 with the calc tool"}],
       "tools":[{"type":"function","function":{"name":"calc",
         "parameters":{"type":"object",
           "properties":{"expression":{"type":"string"}}}}}]}'

Before this patch: finish_reason: "stop", content: "", tool_calls: [].

After this patch: finish_reason: "tool_calls",
tool_calls[0].function.name: "calc",
arguments: '{"expression":"7*8"}'.

Streaming (stream: true) re-verified after the change: delta.tool_calls
arrives token-by-token, finish_reason: "tool_calls", no raw XML in
content (behaviour from #10351 unchanged).

Scope

Two lines of behaviour change plus a comment explaining the capability-vs-state
distinction so the guard isn't accidentally removed again.

…ive_streaming is a capability flag, not a state flag)

mudler#10351 introduced native streaming via `parser.extract_tool_calls_streaming`
and gated the post-loop `extract_tool_calls` block on `native_streaming and
not native_streaming_error`. That works for streaming requests, but for
non-streaming requests the same flag is still True (it only means "the
parser can stream", not "we actually streamed"), so the block was skipped
and the `elif` cleared `content = ""` — the tool call was silently lost.

Symptom: non-streaming chat.completions with `tools=[...]` returns
`finish_reason: "stop"` with `content: ""` and no `tool_calls`. Streaming
requests are unaffected.

Fix: gate both branches on `streaming` too, so the extract_tool_calls
block runs for non-streaming requests (and for streaming requests that
fell back to the buffered path).

Reproduction (vLLM 0.24, Qwen3-Coder-Next-NVFP4, qwen3_coder parser):

    curl -s -X POST http://localhost:8080/v1/chat/completions \
      -H 'Content-Type: application/json' \
      -d '{"model":"coder","stream":false,
           "messages":[{"role":"user","content":"7*8 via calc"}],
           "tools":[{"type":"function","function":{"name":"calc",
             "parameters":{"type":"object",
               "properties":{"expression":{"type":"string"}}}}}]}'

Before: finish_reason: "stop", content: "", tool_calls: []
After:  finish_reason: "tool_calls", tool_calls[0].function.name: "calc"

Streaming path re-verified in the same setup: delta.tool_calls arrives
token-by-token, finish_reason: "tool_calls", no raw XML in content.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
@pos-ei-don pos-ei-don force-pushed the fix/vllm-non-streaming-tool-parser-regression branch from 0a58c23 to f6c3875 Compare July 1, 2026 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants