[None][feat] Add duration-based execution to benchmark by weikuo0506 · Pull Request #13385 · NVIDIA/TensorRT-LLM

weikuo0506 · 2026-04-23T14:19:57Z

Description

This PR adds a duration-based execution feature to the TensorRT-LLM Python benchmark suite (trtllm-bench). Currently, benchmarks rely on a fixed num_requests, which can lead to excessively long run times for scenarios with high Input Sequence Length (ISL) and Output Sequence Length (OSL).

The new --duration option (in seconds) allows users to limit the benchmark run time. The implementation uses a wall-clock check in the worker loop to stop pulling new requests after the duration has elapsed, while allowing in-flight requests to drain to ensure valid statistics.

This affects both throughput and latency commands as they share the same execution logic.

Fixes #13487

Potential Impacts

Performance: This change only affects the benchmarking tool and has no impact on inference performance.
Functional: It adds a new optional CLI parameter and changes the stop condition of the benchmark loop when used.

Test Coverage

Added a unit test test_bench_async.py to verify LlmManager duration logic using mocks.
Local verification was skipped due to missing dependencies (torch, pytest) in the host environment, but the test is designed to be run in the standard container.

PR Checklist

[✓] PR description clearly explains what and why.
[✓] PR Follows TRT-LLM CODING GUIDELINES.
[✓] Test cases are provided for new code paths.

Signed-off-by: Kuo Wei <weikuo@google.com>

weikuo0506 · 2026-04-23T14:20:08Z

/bot run

coderabbitai · 2026-04-23T14:33:18Z

📝 Walkthrough

Walkthrough

This PR adds an optional duration parameter to benchmark execution settings and CLI options, enabling runtime-limited benchmarks. The parameter threads through the benchmark stack from CLI arguments to the async execution manager, where it enforces a time-based execution cap by monitoring elapsed time and stopping request processing when reached.

Changes

Cohort / File(s)	Summary
Settings & Configuration `tensorrt_llm/bench/benchmark/__init__.py`, `tensorrt_llm/bench/benchmark/low_latency.py`, `tensorrt_llm/bench/benchmark/throughput.py`	Add `duration` field to `GeneralExecSettings` and introduce `--duration` CLI parameter in latency and throughput benchmark commands, forwarding it to async execution.
Core Implementation `tensorrt_llm/bench/benchmark/utils/asynchronous.py`	Implement duration-based execution control: add `duration` parameter to `async_benchmark` and `LlmManager`, track request start time, check elapsed time in worker loop, drain inbox when duration limit reached, and modify shutdown logic to conditionally cancel tasks.
Testing `tests/unittest/llmapi/test_bench_async.py`	Add unit test for `LlmManager` duration enforcement, verifying request processing stops when time limit is exceeded.

Sequence Diagram

sequenceDiagram
    actor CLI
    participant async_benchmark
    participant LlmManager
    participant worker as Worker Loop
    participant inbox as Inbox Queue
    
    CLI->>async_benchmark: Call with duration=N seconds
    async_benchmark->>LlmManager: Create with duration=N
    
    Note over CLI,LlmManager: Execution Phase
    activate worker
    worker->>worker: Initialize start_time=None
    
    loop For each request
        inbox->>worker: Receive request
        worker->>worker: Set start_time on first request
        worker->>worker: Calculate elapsed = now() - start_time
        alt elapsed < duration
            worker->>worker: Process request
        else elapsed >= duration
            worker->>worker: Log duration reached
            worker->>inbox: Drain remaining requests
            worker->>worker: Break loop
        end
    end
    
    deactivate worker
    
    Note over worker: Shutdown Phase
    alt stop event is set
        worker->>worker: Cancel remaining tasks
    else stop event not set
        worker->>worker: Wait for in-flight tasks
    end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: adding duration-based execution capability to the benchmark system.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	PR description provides clear explanation of the feature, changes, impacts, test coverage, and checklist verification.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

🧹 Nitpick comments (1)

tests/unittest/llmapi/test_bench_async.py (1)

17-17: Remove unused import.

The time module is imported but never used in this file.

🧹 Proposed fix

 import asyncio
-import time
 from unittest.mock import MagicMock

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/unittest/llmapi/test_bench_async.py` at line 17, Remove the unused
top-level import "time" from tests/unittest/llmapi/test_bench_async.py; locate
the import statement at the top of the file and delete the line "import time" so
the file no longer imports an unused module.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@tests/unittest/llmapi/test_bench_async.py`:
- Line 17: Remove the unused top-level import "time" from
tests/unittest/llmapi/test_bench_async.py; locate the import statement at the
top of the file and delete the line "import time" so the file no longer imports
an unused module.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0a86d31f-8f27-450a-8a37-83b52e329f83

📥 Commits

Reviewing files that changed from the base of the PR and between b3a3b94 and abb477e.

📒 Files selected for processing (5)

tensorrt_llm/bench/benchmark/__init__.py
tensorrt_llm/bench/benchmark/low_latency.py
tensorrt_llm/bench/benchmark/throughput.py
tensorrt_llm/bench/benchmark/utils/asynchronous.py
tests/unittest/llmapi/test_bench_async.py

Signed-off-by: Kuo Wei <weikuo@google.com>

weikuo0506 · 2026-04-23T14:36:07Z

/bot run

dc3671 · 2026-04-27T03:35:57Z

@FrankD412 Can you help review this?

FrankD412 · 2026-04-27T03:36:52Z

@FrankD412 Can you help review this?

Can do -- will handle it asap 🙂

Signed-off-by: Kuo Wei <weikuo@google.com>

…async_benchmark Signed-off-by: Kuo Wei <weikuo@google.com>

Signed-off-by: Kuo Wei <weikuo@google.com>

weikuo0506 · 2026-05-13T10:00:56Z

Hi @dc3671 , can you help to review? Thanks

karljang · 2026-06-12T01:33:25Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-12T01:39:52Z

PR_Github #53742 [ run ] triggered by Bot. Commit: a18ac06 Link to invocation

tensorrt-cicd · 2026-06-12T07:11:39Z

PR_Github #53742 [ run ] completed with state FAILURE. Commit: a18ac06
/LLM/main/L0_MergeRequest_PR pipeline #42867 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

karljang · 2026-06-16T06:15:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T06:20:56Z

PR_Github #54496 [ run ] triggered by Bot. Commit: a18ac06 Link to invocation

karljang · 2026-06-16T06:25:25Z

/bot kill

karljang · 2026-06-16T06:27:07Z

@weikuo0506,
It appears there are pre-commit failures. Could you please address them before we proceed?

tensorrt-cicd · 2026-06-16T06:31:53Z

PR_Github #54501 [ kill ] triggered by Bot. Commit: a18ac06 Link to invocation

tensorrt-cicd · 2026-06-16T06:31:58Z

PR_Github #54496 [ run ] completed with state ABORTED. Commit: a18ac06

Link to invocation

tensorrt-cicd · 2026-06-16T06:32:08Z

PR_Github #54501 [ kill ] completed with state SUCCESS. Commit: a18ac06
Successfully killed previous jobs for commit a18ac06

Link to invocation

Signed-off-by: Kuo Wei <weikuo@google.com>

BowenFu

Re-verified on bcc4ab038. With --duration unset every new branch is inert (_duration_exceeded() false, drain_in_flight false, the multi-turn truncated early-return unreachable), and the rewritten worker() finally is equivalent to main's cancel-then-wait — the pending = set(self._tasks) snapshot additionally fixes main's race where _task_done_callback mutates that set while asyncio.wait iterates it.

Both points I raised earlier are addressed, and one better than I asked: --duration without a concurrency limit is now a hard click.UsageError raised before model load on both commands rather than a warning followed by a full-dataset run, so the flag can no longer be a silent no-op on trtllm-bench throughput.

Two things I'm explicitly not blocking on: the duration drain's final await asyncio.wait(pending) has no timeout, so in duration mode a single non-terminating request can hang the run; and requests dropped at the deadline are excluded from StatsKeeper.requests, so a truncated run reports throughput/latency over completed requests only, not the submitted dataset — worth a line in the --duration help text.

karljang · 2026-07-30T17:59:54Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-30T18:05:33Z

PR_Github #62825 [ run ] triggered by Bot. Commit: bcc4ab0 Link to invocation

tensorrt-cicd · 2026-07-30T22:02:05Z

PR_Github #62825 [ run ] completed with state FAILURE. Commit: bcc4ab0
/LLM/main/L0_MergeRequest_PR pipeline #50951 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-31T04:23:31Z

@weikuo0506, I regret to inform you that, as @BowenFu mentioned earlier, we’ve encountered conflicts. Could you kindly resolve them?

# Conflicts: # tests/integration/test_lists/test-db/l0_cpu_x86.yml

Requests dropped at the deadline never reach StatsKeeper, so a duration-bounded run reports throughput and latency over the requests that completed rather than the dataset that was submitted. Say so in the help text, along with the concurrency requirement. Signed-off-by: Kuo Wei <weikuo@google.com>

weikuo0506 · 2026-07-31T14:12:26Z

@karljang Conflict resolved and pushed (69ac657); the branch is level with main again. It was another append to the same line in l0_cpu_x86.yml, so both entries are kept.

@BowenFu On your two non-blocking notes: I documented the reporting scope in the --duration help text, since a truncated run reporting over completed requests only is easy to misread. I left the drain without a timeout — a bound there needs its own decision on what value is right for long ISL/OSL requests and what to do when it expires, and picking one arbitrarily would risk cancelling healthy requests and losing exactly the statistics the drain exists to preserve. Note the same exposure predates --duration: a non-terminating request already keeps backend.busy true and spins async_benchmark's loop on main.

CI has now failed three times (6d74e9b, eea4060, bcc4ab0) and the pipeline links are internal, so I still can't see which tests failed or whether they relate to this PR. Could you share the failing cases? Verified locally against 1.3.0rc21 on CPU: 7 passed.

karljang · 2026-07-31T15:25:54Z

The last failure looks not relevant to this PR. I will retrigger the CI, thank you!

karljang · 2026-07-31T15:26:05Z

/bot run --disable-fail-fast

karljang · 2026-07-31T15:27:16Z

/bot run

tensorrt-cicd · 2026-07-31T15:31:55Z

PR_Github #63055 [ run ] triggered by Bot. Commit: 69ac657 Link to invocation

tensorrt-cicd · 2026-07-31T15:32:42Z

PR_Github #63056 [ run ] triggered by Bot. Commit: 69ac657 Link to invocation

tensorrt-cicd · 2026-07-31T15:32:45Z

PR_Github #63055 [ run ] completed with state ABORTED. Commit: 69ac657

Link to invocation

tensorrt-cicd · 2026-07-31T18:10:28Z

PR_Github #63056 [ run ] completed with state FAILURE. Commit: 69ac657
/LLM/main/L0_MergeRequest_PR pipeline #51156 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-31T18:27:59Z

/bot run

tensorrt-cicd · 2026-07-31T18:37:39Z

PR_Github #63097 [ run ] triggered by Bot. Commit: 69ac657 Link to invocation

tensorrt-cicd · 2026-07-31T18:56:46Z

PR_Github #63097 [ run ] completed with state FAILURE. Commit: 69ac657
/LLM/main/L0_MergeRequest_PR pipeline #51188 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-31T19:46:25Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-31T19:52:43Z

PR_Github #63112 [ run ] triggered by Bot. Commit: 69ac657 Link to invocation

tensorrt-cicd · 2026-08-01T00:38:50Z

PR_Github #63112 [ run ] completed with state FAILURE. Commit: 69ac657
/LLM/main/L0_MergeRequest_PR pipeline #51204 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-08-01T06:58:24Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-01T07:04:32Z

PR_Github #63189 [ run ] triggered by Bot. Commit: 69ac657 Link to invocation

tensorrt-cicd · 2026-08-01T09:08:59Z

PR_Github #63189 [ run ] completed with state SUCCESS. Commit: 69ac657
/LLM/main/L0_MergeRequest_PR pipeline #51273 completed with status: 'SUCCESS'

CI Report

Link to invocation

karljang · 2026-08-01T18:58:33Z

@weikuo0506, it’s finally merged! Thank you for your contribution and efforts 👍

Signed-off-by: Kuo Wei <weikuo@google.com> Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com> Co-authored-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

[None][feat] Add duration-based execution to benchmark

abb477e

Signed-off-by: Kuo Wei <weikuo@google.com>

weikuo0506 requested a review from a team as a code owner April 23, 2026 14:19

weikuo0506 requested a review from dc3671 April 23, 2026 14:19

github-actions Bot assigned weikuo0506 Apr 23, 2026

coderabbitai Bot reviewed Apr 23, 2026

View reviewed changes

[None][chore] Improve docstring coverage in asynchronous.py

00c7a53

Signed-off-by: Kuo Wei <weikuo@google.com>

svc-trtllm-gh-bot added the Community want to contribute PRs initiated from Community label Apr 23, 2026

dc3671 requested a review from FrankD412 April 27, 2026 03:35

FrankD412 reviewed Apr 28, 2026

View reviewed changes

Comment thread tensorrt_llm/bench/benchmark/utils/asynchronous.py

weikuo0506 added 3 commits May 4, 2026 08:55

[None][chore] Remove unused import in test_bench_async.py

1d1ca51

Signed-off-by: Kuo Wei <weikuo@google.com>

[None][fix] Allow partial request completion when duration is set in …

16c8c70

…async_benchmark Signed-off-by: Kuo Wei <weikuo@google.com>

[None][chore] Add informative log when duration limit is reached

a18ac06

Signed-off-by: Kuo Wei <weikuo@google.com>

FrankD412 approved these changes May 8, 2026

View reviewed changes

[None][chore] Merge upstream/main into feat/duration-bench

7446e94

Signed-off-by: Kuo Wei <weikuo@google.com>

BowenFu approved these changes Jul 30, 2026

View reviewed changes

karljang enabled auto-merge (squash) July 30, 2026 18:44

weikuo0506 added 2 commits July 31, 2026 22:04

Merge remote-tracking branch 'origin/main' into feat/duration-bench

f8f8b90

# Conflicts: # tests/integration/test_lists/test-db/l0_cpu_x86.yml

auto-merge was automatically disabled July 31, 2026 14:12
Head branch was pushed to by a user without write access

karljang enabled auto-merge (squash) July 31, 2026 15:26

karljang merged commit a9544e0 into NVIDIA:main Aug 1, 2026
8 checks passed

Uh oh!

Conversation

weikuo0506 commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Potential Impacts

Test Coverage

PR Checklist

Uh oh!

weikuo0506 commented Apr 23, 2026

Uh oh!

coderabbitai Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

weikuo0506 commented Apr 23, 2026

Uh oh!

dc3671 commented Apr 27, 2026

Uh oh!

FrankD412 commented Apr 27, 2026

Uh oh!

Uh oh!

weikuo0506 commented May 13, 2026

Uh oh!

karljang commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

karljang commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

karljang commented Jun 16, 2026

Uh oh!

karljang commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

BowenFu left a comment

Choose a reason for hiding this comment

Uh oh!

karljang commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

karljang commented Jul 31, 2026

Uh oh!

weikuo0506 commented Jul 31, 2026

Uh oh!

karljang commented Jul 31, 2026

Uh oh!

karljang commented Jul 31, 2026

Uh oh!

karljang commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

karljang commented Jul 31, 2026

Uh oh!

weikuo0506 commented Apr 23, 2026 •

edited

Loading

coderabbitai Bot commented Apr 23, 2026 •

edited

Loading