fix(vllm): parse tool_call function arguments before applying the chat template#10256
Conversation
…t template Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
54b671b to
a3ee603
Compare
|
Re-verified this today against the current quay master backend build ( One detail worth knowing for review: the bug only triggers with chat templates that iterate over With this patch applied on top of the same build: the identical request returns 200 and the template renders the parsed arguments correctly. |
…hat template OpenAI wire format carries `function.arguments` as a JSON-encoded string, but chat templates (e.g. Qwen3-Coder) iterate over it as a mapping. The vllm backend already parses arguments before applying the chat template (PR mudler#10256); this mirrors that fix in the sglang backend. Without this fix the second turn of any tool-using session (assistant returns tool_calls, user posts `role:"tool"` result, model is invoked with arguments still as a string) crashes inside transformers' Jinja chat-template rendering with: TypeError: Can only get item pairs from a mapping. File ".../transformers/utils/chat_template_utils.py", in render_jinja_template File ".../jinja2/filters.py", in do_items raise TypeError("Can only get item pairs from a mapping.") Reproduced on `lmsysorg/sglang:v0.5.14` via LocalAI v4.5.4 with `saricles/Qwen3-Coder-Next-NVFP4-GB10` (W4A4 NVFP4 / compressed-tensors) on NVIDIA DGX Spark (GB10, sm_121). After the patch, a tool-call roundtrip (assistant tool_calls -> tool result -> assistant final answer) returns http=200 with the expected follow-up content; no behaviour change on requests that don't carry tool_calls. Signed-off-by: Poseidon <philipp.wacker@ibf-solutions.com>
…hat template (#10558) OpenAI wire format carries `function.arguments` as a JSON-encoded string, but chat templates (e.g. Qwen3-Coder) iterate over it as a mapping. The vllm backend already parses arguments before applying the chat template (PR #10256); this mirrors that fix in the sglang backend. Without this fix the second turn of any tool-using session (assistant returns tool_calls, user posts `role:"tool"` result, model is invoked with arguments still as a string) crashes inside transformers' Jinja chat-template rendering with: TypeError: Can only get item pairs from a mapping. File ".../transformers/utils/chat_template_utils.py", in render_jinja_template File ".../jinja2/filters.py", in do_items raise TypeError("Can only get item pairs from a mapping.") Reproduced on `lmsysorg/sglang:v0.5.14` via LocalAI v4.5.4 with `saricles/Qwen3-Coder-Next-NVFP4-GB10` (W4A4 NVFP4 / compressed-tensors) on NVIDIA DGX Spark (GB10, sm_121). After the patch, a tool-call roundtrip (assistant tool_calls -> tool result -> assistant final answer) returns http=200 with the expected follow-up content; no behaviour change on requests that don't carry tool_calls. Signed-off-by: Poseidon <philipp.wacker@ibf-solutions.com> Co-authored-by: Poseidon <philipp.wacker@ibf-solutions.com>
Problem
Multi-turn tool calling fails with a 500 once the client sends back an assistant message containing
tool_calls:_messages_to_dicts()parses the outertool_callsJSON array but leavesfunction.argumentsas a JSON-encoded string (that's the OpenAI wire format). Chat templates like Qwen3's iterate overargumentsas a mapping, soapply_chat_template()blows up. vLLM's own OpenAI-compatible server parsesargumentsinto a dict before rendering the template.Fix
After decoding the outer array, decode each
function.argumentsstring into a dict. Strings that aren't valid JSON are left untouched.Tested
On an NVIDIA DGX Spark (GB10, ARM64),
cuda13-nvidia-l4t-arm64-vllmbackend, Qwen3-Coder withuse_tokenizer_template: true: without the patch every second-turn tool call 500s; with the patch multi-turn tool calling works (finish_reason: tool_calls, correct arguments round-trip).