fix(vllm): non-streaming tool-call regression after #10351#10638
Merged
mudler merged 1 commit intoJul 2, 2026
Merged
Conversation
…ive_streaming is a capability flag, not a state flag) mudler#10351 introduced native streaming via `parser.extract_tool_calls_streaming` and gated the post-loop `extract_tool_calls` block on `native_streaming and not native_streaming_error`. That works for streaming requests, but for non-streaming requests the same flag is still True (it only means "the parser can stream", not "we actually streamed"), so the block was skipped and the `elif` cleared `content = ""` — the tool call was silently lost. Symptom: non-streaming chat.completions with `tools=[...]` returns `finish_reason: "stop"` with `content: ""` and no `tool_calls`. Streaming requests are unaffected. Fix: gate both branches on `streaming` too, so the extract_tool_calls block runs for non-streaming requests (and for streaming requests that fell back to the buffered path). Reproduction (vLLM 0.24, Qwen3-Coder-Next-NVFP4, qwen3_coder parser): curl -s -X POST http://localhost:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"coder","stream":false, "messages":[{"role":"user","content":"7*8 via calc"}], "tools":[{"type":"function","function":{"name":"calc", "parameters":{"type":"object", "properties":{"expression":{"type":"string"}}}}}]}' Before: finish_reason: "stop", content: "", tool_calls: [] After: finish_reason: "tool_calls", tool_calls[0].function.name: "calc" Streaming path re-verified in the same setup: delta.tool_calls arrives token-by-token, finish_reason: "tool_calls", no raw XML in content. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
0a58c23 to
f6c3875
Compare
mudler
approved these changes
Jul 2, 2026
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Non-streaming
chat.completionswithtools=[...]silently drops the toolcall (
finish_reason: "stop", empty content, notool_calls[]) when atool_parseris configured. Streaming requests are unaffected.Root cause
#10351 introduced native streaming via
parser.extract_tool_calls_streamingand gated the post-loop
extract_tool_callsblock onnative_streaming and not native_streaming_error.native_streamingis a capability flag — it becomesTrueas soon astp_requestbuilds and the parser exposesextract_tool_calls_streaming.It is not a state flag telling us the per-delta loop actually ran.
For non-streaming requests it is still
True, so:ifbranch (extract_tool_callson the full text) was skippedelifbranch (content = "", "the stream already emitted everything")fired
The result: the tool call is neither extracted nor streamed — it's lost.
Fix
Add an explicit
streaming and …guard on both branches soextract_tool_callsruns for the non-streaming path (and for streamingpaths that fell back to the buffered mode). No change to streaming
requests where native streaming ran cleanly.
Reproduction
Model:
saricles/Qwen3-Coder-Next-NVFP4-GB10, vLLM 0.24.0, LocalAIv4.5.6,
coder.yamlwithoptions: - tool_parser:qwen3_coder.Before this patch:
finish_reason: "stop",content: "",tool_calls: [].After this patch:
finish_reason: "tool_calls",tool_calls[0].function.name: "calc",arguments: '{"expression":"7*8"}'.Streaming (
stream: true) re-verified after the change:delta.tool_callsarrives token-by-token,
finish_reason: "tool_calls", no raw XML incontent (behaviour from #10351 unchanged).
Scope
Two lines of behaviour change plus a comment explaining the capability-vs-state
distinction so the guard isn't accidentally removed again.