Skip to content

[None][perf] Reduce OpenAI stream postprocess overhead - #14708

Merged
2ez4bz merged 3 commits into
NVIDIA:mainfrom
2ez4bz:dev-serving-opts
Jun 3, 2026
Merged

[None][perf] Reduce OpenAI stream postprocess overhead#14708
2ez4bz merged 3 commits into
NVIDIA:mainfrom
2ez4bz:dev-serving-opts

Conversation

@2ez4bz

@2ez4bz 2ez4bz commented May 28, 2026

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • Bug Fixes

    • Streaming responses now include consistent IDs and timestamps across all message chunks for improved data consistency.
  • Tests

    • Added test coverage for streaming response metadata consistency validation.

Review Change Stack

Description

Streaming responses created fresh metadata for each chunk when callers did not pass it explicitly. High-concurrency workloads can emit hundreds of thousands of chunks, which makes UUID generation, and time lookups part of the CPU hot path.

Reuse stream metadata per request.

Details

Prior to this, the stream postprocessors created each SSE chunk (e.g. ChatCompletionStreamResponse) without passing id or created, thus calling their factories uuid.uuid64 and time.time, respectively, so every streamed chunk got fresh metadata.

Semantically, this was wrong - all chunks for one streamed OpenAI response should have the same id and created timestamp.

As a side-effect, this also benefits performance: at high concurrency on lower-tier CPUs, this change can by itself lead to a ~5% ITL decrease.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@2ez4bz
2ez4bz requested a review from a team as a code owner May 28, 2026 21:00
@2ez4bz
2ez4bz requested a review from JunyiXu-nv May 28, 2026 21:00
@coderabbitai

coderabbitai Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Added stream response metadata tracking to chat and completion streaming processors. Both ChatPostprocArgs and CompletionPostprocArgs now store stream_response_id and stream_created, initialized once per request via a new _ensure_stream_metadata() helper. Stream chunks (initial, per-delta, and final usage) now carry consistent id and created values. Completion streaming now uses CompletionStreamResponse for the final chunk instead of ChatCompletionStreamResponse.

Changes

Stream Response Metadata Tracking

Layer / File(s) Summary
Data contracts and metadata helper
tensorrt_llm/serve/postprocess_handlers.py
Added import time and fields stream_response_id and stream_created to both ChatPostprocArgs and CompletionPostprocArgs. New _ensure_stream_metadata() helper lazily initializes these values from the response id and current Unix timestamp, returning both for reuse across chunks.
Chat streaming metadata integration
tensorrt_llm/serve/postprocess_handlers.py
Updated chat_stream_post_processor() to initialize stream metadata once, then include consistent id and created on the first streamed chunk, all subsequent per-delta chunks, and the final usage chunk.
Completion streaming metadata integration
tensorrt_llm/serve/postprocess_handlers.py
Updated completion_stream_post_processor() to initialize stream metadata once, include id and created on per-delta chunks, and use CompletionStreamResponse (instead of ChatCompletionStreamResponse) for the final usage chunk with metadata.
Stream metadata consistency test
tests/unittest/llmapi/test_llm.py
New test test_chat_stream_post_processor_reuses_stream_metadata() validates that id is fixed and created remains identical across multiple stream chunks, and verifies correct delta fields (role on first payload, incremental content on final payload).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed Title clearly indicates a performance optimization for streaming response post-processing, which aligns with the main objective of reducing CPU overhead.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed PR description clearly explains the problem (repeated metadata generation in hot path) and the solution (reuse metadata per request), with details on semantic correctness and performance benefits.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
tests/unittest/llmapi/test_llm.py (1)

2578-2578: ⚡ Quick win

Add an explicit None return type on the test function.

Please annotate the test signature to keep function typing consistent with repo rules.

Suggested change
-def test_chat_stream_post_processor_reuses_stream_metadata():
+def test_chat_stream_post_processor_reuses_stream_metadata() -> None:

As per coding guidelines: "Static type checking with mypy is opt-in by submodule; always annotate functions with return types (use None if function does not return)".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/llmapi/test_llm.py` at line 2578, The test function
test_chat_stream_post_processor_reuses_stream_metadata lacks an explicit return
type; update its signature to include a return annotation of None (i.e., def
test_chat_stream_post_processor_reuses_stream_metadata() -> None:) to satisfy
the repo's typing rule requiring all functions to declare return types.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/llmapi/test_llm.py`:
- Line 2578: The test function
test_chat_stream_post_processor_reuses_stream_metadata lacks an explicit return
type; update its signature to include a return annotation of None (i.e., def
test_chat_stream_post_processor_reuses_stream_metadata() -> None:) to satisfy
the repo's typing rule requiring all functions to declare return types.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: df80d662-dbba-4264-ada5-341cc18b0933

📥 Commits

Reviewing files that changed from the base of the PR and between f6ba936 and 86eb826.

📒 Files selected for processing (2)
  • tensorrt_llm/serve/postprocess_handlers.py
  • tests/unittest/llmapi/test_llm.py

@2ez4bz
2ez4bz force-pushed the dev-serving-opts branch from 86eb826 to 8d3d2a0 Compare May 28, 2026 21:08
Comment thread tensorrt_llm/serve/postprocess_handlers.py
@2ez4bz

2ez4bz commented May 29, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@2ez4bz
2ez4bz enabled auto-merge (squash) May 29, 2026 05:00
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #50971 [ run ] triggered by Bot. Commit: 04f638f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #50971 [ run ] completed with state SUCCESS. Commit: 04f638f
/LLM/main/L0_MergeRequest_PR pipeline #40426 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Comment thread tensorrt_llm/serve/postprocess_handlers.py
Comment thread tensorrt_llm/serve/postprocess_handlers.py
@2ez4bz

2ez4bz commented May 29, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51125 [ run ] triggered by Bot. Commit: 89a95dc Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51125 [ run ] completed with state SUCCESS. Commit: 89a95dc
/LLM/main/L0_MergeRequest_PR pipeline #40563 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

2ez4bz added 3 commits June 1, 2026 09:43
Streaming responses created fresh metadata for each chunk when callers did
not pass it explicitly. High-concurrency workloads can emit hundreds of
thousands of chunks, which makes UUID generation, and time lookups part
of the CPU hot path.

Reuse stream metadata per request.

Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
@2ez4bz
2ez4bz force-pushed the dev-serving-opts branch from 89a95dc to 8d05f99 Compare June 1, 2026 16:43
@2ez4bz

2ez4bz commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51387 [ run ] triggered by Bot. Commit: 8d05f99 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51387 [ run ] completed with state FAILURE. Commit: 8d05f99
/LLM/main/L0_MergeRequest_PR pipeline #40797 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@2ez4bz

2ez4bz commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51493 [ run ] triggered by Bot. Commit: 8d05f99 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51493 [ run ] completed with state SUCCESS. Commit: 8d05f99
/LLM/main/L0_MergeRequest_PR pipeline #40899 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@2ez4bz

2ez4bz commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51636 [ run ] triggered by Bot. Commit: 8d05f99 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51636 [ run ] completed with state SUCCESS. Commit: 8d05f99
/LLM/main/L0_MergeRequest_PR pipeline #41021 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@2ez4bz

2ez4bz commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51682 [ run ] triggered by Bot. Commit: 8d05f99 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51682 [ run ] completed with state SUCCESS. Commit: 8d05f99
/LLM/main/L0_MergeRequest_PR pipeline #41062 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@2ez4bz

2ez4bz commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51700 [ run ] triggered by Bot. Commit: 8d05f99 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51700 [ run ] completed with state SUCCESS. Commit: 8d05f99
/LLM/main/L0_MergeRequest_PR pipeline #41077 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@2ez4bz

2ez4bz commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51746 [ run ] triggered by Bot. Commit: 8d05f99 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51746 [ run ] completed with state SUCCESS. Commit: 8d05f99
/LLM/main/L0_MergeRequest_PR pipeline #41119 completed with status: 'SUCCESS'

CI Report

Link to invocation

@2ez4bz
2ez4bz merged commit 6ab5005 into NVIDIA:main Jun 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants