Skip to content

fix(vllm): parse tool_call function arguments before applying the chat template#10256

Merged
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:fix/vllm-tool-call-arguments
Jun 11, 2026
Merged

fix(vllm): parse tool_call function arguments before applying the chat template#10256
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:fix/vllm-tool-call-arguments

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Problem

Multi-turn tool calling fails with a 500 once the client sends back an assistant message containing tool_calls:

TypeError: Can only get item pairs from a mapping.

_messages_to_dicts() parses the outer tool_calls JSON array but leaves function.arguments as a JSON-encoded string (that's the OpenAI wire format). Chat templates like Qwen3's iterate over arguments as a mapping, so apply_chat_template() blows up. vLLM's own OpenAI-compatible server parses arguments into a dict before rendering the template.

Fix

After decoding the outer array, decode each function.arguments string into a dict. Strings that aren't valid JSON are left untouched.

Tested

On an NVIDIA DGX Spark (GB10, ARM64), cuda13-nvidia-l4t-arm64-vllm backend, Qwen3-Coder with use_tokenizer_template: true: without the patch every second-turn tool call 500s; with the patch multi-turn tool calling works (finish_reason: tool_calls, correct arguments round-trip).

…t template

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
@pos-ei-don
pos-ei-don force-pushed the fix/vllm-tool-call-arguments branch from 54b671b to a3ee603 Compare June 11, 2026 19:30
@pos-ei-don

Copy link
Copy Markdown
Contributor Author

Re-verified this today against the current quay master backend build (master-nvidia-l4t-cuda-13-arm64-vllm) on the DGX Spark, this time with a small model so it's easy to reproduce: Qwen/Qwen3-0.6B, use_tokenizer_template: true, and a two-turn tool-call conversation (assistant message with tool_calls, then a tool result).

One detail worth knowing for review: the bug only triggers with chat templates that iterate over arguments as a mapping — e.g. the Qwen3-Coder template does {%- for args_name, args_value in tool_call.arguments|items %} and fails with TypeError: Can only get item pairs from a mapping (HTTP 500). The base Qwen3 template just runs arguments | tojson, which happens to work with the string, so the bug stays invisible there. That's probably why this hasn't been reported more widely.

With this patch applied on top of the same build: the identical request returns 200 and the template renders the parsed arguments correctly.

@mudler
mudler merged commit b962f4a into mudler:master Jun 11, 2026
1 check passed
@pos-ei-don
pos-ei-don deleted the fix/vllm-tool-call-arguments branch June 15, 2026 04:27
pos-ei-don pushed a commit to pos-ei-don/LocalAI that referenced this pull request Jun 29, 2026
…hat template

OpenAI wire format carries `function.arguments` as a JSON-encoded string,
but chat templates (e.g. Qwen3-Coder) iterate over it as a mapping. The
vllm backend already parses arguments before applying the chat template
(PR mudler#10256); this mirrors that fix in the sglang backend.

Without this fix the second turn of any tool-using session (assistant
returns tool_calls, user posts `role:"tool"` result, model is invoked
with arguments still as a string) crashes inside transformers' Jinja
chat-template rendering with:

  TypeError: Can only get item pairs from a mapping.
  File ".../transformers/utils/chat_template_utils.py", in render_jinja_template
  File ".../jinja2/filters.py", in do_items
      raise TypeError("Can only get item pairs from a mapping.")

Reproduced on `lmsysorg/sglang:v0.5.14` via LocalAI v4.5.4 with
`saricles/Qwen3-Coder-Next-NVFP4-GB10` (W4A4 NVFP4 / compressed-tensors)
on NVIDIA DGX Spark (GB10, sm_121).

After the patch, a tool-call roundtrip (assistant tool_calls -> tool
result -> assistant final answer) returns http=200 with the expected
follow-up content; no behaviour change on requests that don't carry
tool_calls.

Signed-off-by: Poseidon <philipp.wacker@ibf-solutions.com>
mudler pushed a commit that referenced this pull request Jun 30, 2026
…hat template (#10558)

OpenAI wire format carries `function.arguments` as a JSON-encoded string,
but chat templates (e.g. Qwen3-Coder) iterate over it as a mapping. The
vllm backend already parses arguments before applying the chat template
(PR #10256); this mirrors that fix in the sglang backend.

Without this fix the second turn of any tool-using session (assistant
returns tool_calls, user posts `role:"tool"` result, model is invoked
with arguments still as a string) crashes inside transformers' Jinja
chat-template rendering with:

  TypeError: Can only get item pairs from a mapping.
  File ".../transformers/utils/chat_template_utils.py", in render_jinja_template
  File ".../jinja2/filters.py", in do_items
      raise TypeError("Can only get item pairs from a mapping.")

Reproduced on `lmsysorg/sglang:v0.5.14` via LocalAI v4.5.4 with
`saricles/Qwen3-Coder-Next-NVFP4-GB10` (W4A4 NVFP4 / compressed-tensors)
on NVIDIA DGX Spark (GB10, sm_121).

After the patch, a tool-call roundtrip (assistant tool_calls -> tool
result -> assistant final answer) returns http=200 with the expected
follow-up content; no behaviour change on requests that don't carry
tool_calls.

Signed-off-by: Poseidon <philipp.wacker@ibf-solutions.com>
Co-authored-by: Poseidon <philipp.wacker@ibf-solutions.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants