feat(vllm): progressive streaming via parser.extract_tool_calls_streaming (follow-up to #10346)

[pos-ei-don]

Description

Follow-up to #10346: replace the buffer-all path with native progressive streaming.

When the vLLM backend has a tool_parser configured and a streaming chat
completion request includes tools, PR #10346 stops streaming any deltas
and surfaces parsed tool_calls (or the cleaned content) on the final chunk
only. This is correct for tool-call responses, but it turns the
plain-text-with-tools case into effectively non-streaming: the client
receives nothing until the model finishes.

Every concrete tool parser shipped with vLLM 0.23+ already implements
extract_tool_calls_streaming — including qwen3coder, granite4,
deepseekv31, jamba, ernie45, hermes2pro, llama3_json, mistral,
and ~30 more (verified by walking vllm.tool_parsers.* and checking each
class against ToolParser.extract_tool_calls_streaming). This PR delegates
to that interface for the streaming branch.

E2E numbers (qwen3_coder · NVIDIA GB10 / arm64 / vLLM 0.23.0, 3 runs avg)

Scenario	Buffer-all (#10346)	This PR	Delta
Plain text (~70 tokens), TTFT content	1.32 s	0.226 s	5.8× faster
Plain text (~70 tokens), content chunks	2	71	per-token stream
Plain text (~1100 tokens), TTFT content	~18 s¹	0.221 s	~80× faster
Plain text (~1100 tokens), content chunks	1¹	~1135	per-token stream
Tool call, TTFT tool	0.51 s	0.54 s²	unchanged
Tool call, raw markup leak	none ✓	none ✓	—

¹ Buffer-all sends everything in the final chunk → ttf_content ≈ total_time.
² First run includes cold-start (0.98 s); stabilises at ~0.54 s by run 3.

The long-text scenario is where the difference is dramatic: with buffer-all
the client sees nothing for ~18 seconds and then a 1100-token wall of text;
with native streaming the first token arrives in 0.22 s and the response
flows progressively. Total wall-clock is unchanged — only the perceived
reactivity changes.

Bonus: tool_call is now emitted as a proper OpenAI-style incremental
stream — tool_calls[0].function.name first, then arguments as a separate
delta — instead of one bundled chunk at the end:

chunk 1: tool_calls=[{id, type:function, function:{name:"get_weather", arguments:""}}]
chunk 2: tool_calls=[{function:{arguments:'{"city": "Paris"}'}}]
chunk 3: finish_reason="tool_calls"

Fix shape

• Instantiate the tool parser before the streaming loop, build a minimal
  ChatCompletionRequest (only tools is read by parsers), set
  native_streaming if hasattr(tp_instance, "extract_tool_calls_streaming").
• Inside the loop, when native_streaming is on, call the parser's
  streaming method per delta with the full 7-parameter signature
  (previous_text, current_text, delta_text, previous_token_ids,
  current_token_ids, delta_token_ids, request). Map the returned
  DeltaMessage to ChatDelta(content / reasoning_content / tool_calls).
  None → suppress this delta.
• Fallback to buffer when the streaming method raises (logged with the
  request id) or is absent (future Python backends, custom parsers). The
  existing post-loop extract_tool_calls block builds the final chunk —
  same correctness as a non-streaming response.
• Buffer-fallback content flush: when has_tool_parser is set and the
  parser found no tool call (plain text response), flush the buffered
  content as one content delta before the final metadata chunk, and clear
  chat_delta.content so the metadata chunk does not repeat it. This also
  fixes the trade-off in fix(vllm): don't stream raw tool-call markup as content when a tool parser is active #10346 where plain-text-with-tools responses
  arrived empty.
• No change to the no-tool-parser path — per-delta content stream is
  byte-identical to the pre-fix(vllm): don't stream raw tool-call markup as content when a tool parser is active #10346 baseline.

Trade-offs / things to know

Concern	Status
New code surface (~80 lines vs ~4 in #10346)	6 mock tests cover all paths; E2E verified
Per-delta extract_tool_calls_streaming CPU cost	Negligible vs. token generation; vLLM's own OpenAI frontend does the same
New import path vllm.entrypoints.openai.chat_completion.protocol	Wrapped in try/except; falls back to buffer if absent (no crash)
Parser exceptions mid-stream	Logged with request id; rest of stream uses buffer fallback; final chunk still correct
Tokens-list tracking per delta	O(N); trivial
Reasoning + Tool parser composition	Currently the reasoning parser runs post-stream. When both parsers are active and a model emits e.g. <think>...</think><tool_call>..., the streaming tool parser sees the reasoning markup as content. This is unchanged from #10346 (which had the same composition issue, just hidden by buffer-all). Treating reasoning-during-streaming properly is out of scope for this PR — see #10000 for the post-stream reasoning side.
Backwards compatibility	Parser without streaming method → buffer fallback = #10346 behaviour. Non-tool-parser path → byte-identical to pre-#10346.

Tests (backend/python/vllm/test.py, new class TestStreamingToolParser)

Six server-less mock tests that exercise _predict(streaming=True) directly:

1. Buffer fallback: tool call → no markup as content; tool_call name present
2. Buffer fallback: plain text → content delivered exactly once (no dupe)
3. Native streaming: plain text → content streams progressively
4. Native streaming: tool call → structured tool_calls emitted, no markup leak
5. Native streaming exception → graceful fallback, no leak, no crash
6. No tool parser → unchanged per-delta content stream (regression guard)

All six green on vLLM 0.23.0 (NVIDIA GB10 / arm64 / CUDA 13).

Bench script — examples/vllm-bench/

Self-contained stdlib-only TTFT benchmark with three scenarios (tool_call,
plain_text_short, plain_text_long). Reviewers can verify the
improvement on their own model:

python examples/vllm-bench/ttft_streaming_tool_parser.py \
    --url http://localhost:8080 --model my-coder --runs 3

The plain_text_long scenario (1500-token GIL explanation) shows the
buffering vs streaming difference most dramatically — see numbers above.

Signed commits

• Yes, I signed my commits.

[pos-ei-don]

Migration note — no breaking changes

Existing setup	Still works?	Action
tool_parser: configured, client uses stream: false	✅ unchanged	Optional: switch to stream: true for the UX win
tool_parser: configured, client filters <tool_call> markup out of delta.content locally	✅ unchanged (filter just never matches)	Workaround can be removed
No tool_parser: configured, client parses tool markup itself	✅ byte-identical (plain-streaming path unchanged)	None
Custom Python backend without extract_tool_calls_streaming	✅ falls back to the #10346 buffer path	None
Mixed with structured outputs / response_format / JSON-mode	✅ orthogonal pipelines, no interaction	None

Nobody is broken by this merge. Anyone who switches stream: false → stream: true after the merge gets the TTFT improvement; anyone who keeps it the way it was gets identical behaviour to today.

[pos-ei-don]

Live benchmark on DGX Spark — TTFT improvement validated

Ran the streaming bench against saricles/Qwen3-Coder-Next-NVFP4-GB10 (Qwen3-Coder-Next 79.7B MoE, NvFP4) on DGX Spark / GB10, with this patch deployed live (LocalAI v4.4.3, vLLM 0.23.0, FlashInfer CUTLASS, CUDA-Graphs on, gpu_memory_utilization=0.65).

Results

Test	max_tokens	TTFT	Gen-Time	Total	Char/s	Output	Finish
plain (short, ~1 sentence)	200	0.141 s	1.05 s	1.19 s	270	283 chars / 67 chunks	stop
plain (medium, 5 paragraphs)	1100	0.097 s	9.73 s	9.83 s	220	2139 chars / 612 chunks	stop
plain (long, code w/ commentary)	2000	0.130 s	22.79 s	22.92 s	225	5132 chars / 1417 chunks	stop
tool_call (calculator(123*456))	200	0.907 s	—	0.907 s	—	0 content chunks, 2 tool_call chunks	tool_calls

Before / after the patch

Metric	Pre-patch (buffer-all, #10346)	This PR (native progressive)	Delta
TTFT plain_medium (1100 tok)	18 s (full buffer until generation ends)	0.097 s	≈ 185× faster
Throughput plain (char/s, ≈63 tok/s)	~220	~220	unchanged
Tool-call SSE shape	content delta with raw <tool_call> XML, then late finish_reason: tool_calls	delta.tool_calls token-by-token (function.name then arguments increments), no content leak	✅

Notes

• extract_tool_calls_streaming per-delta adds no measurable throughput overhead — it's a cheap parser call, and CUDA-graph capture / sampler cost dominates.
• The Hermes CLI (streaming.enabled: true) renders the long-form responses as actual streams now — visibly different UX, not just on paper.
• qwen3_coder parser exercised: clean transition from plain-text mode to tool-call mode within a single response also tested (separate Hermes E2E run, no content/tool-call interleave issues).

If anything else would be helpful before review (different model, smaller TTFT prompts, soak test), happy to run it.

[mudler]

thanks @pos-ei-don !