[None][feat] BREAKING Support saving last N turns in per-conversation policy by jiaganc · Pull Request #16883 · NVIDIA/TensorRT-LLM

jiaganc · 2026-07-27T03:16:37Z

Dev Engineer Review

Added nested BlockReuseConfig under KvCacheConfig.
Added max_num_turns, with a default value of 1.
Updated cache managers, Mamba handling, manifests, examples, and tests to use the nested configuration.
Multi-turn per_conversation reuse now retains the configured number of completed turns.
This is a breaking API change. Replace kv_cache_config.block_reuse_policy with kv_cache_config.block_reuse_config.block_reuse_policy.
Pre-commit checks and MARS compilation passed.
Disaggregated AGA benchmarks started successfully for supported reuse modes.

QA Engineer Review

Updated test_kv_cache_manager_v2.py.
- Added test_per_conversation_policy_retains_configured_number_of_turns.
- Updated the manager fixture to accept max_num_turns.
- Added the max_num_turns fixture.
Updated configuration usage in:
- test_cache_transceiver_single_process.py
- test_kv_transfer.py
- test_kv_cache_stats_behavior.py
- test_llm_args.py
- test_mamba_cache_manager.py
Tests cover nested configuration defaults, serialization, validation, warning behavior, Mamba snapshot handling, and multi-turn cache eviction.
No integration test-list coverage is provided. Verdict: needs follow-up.

Description

Conversation-scoped KV-cache reuse currently retains only one completed turn, which prevents reuse when a conversation rewinds to an older turn.

This change:

introduces a nested BlockReuseConfig under KvCacheConfig;
moves block_reuse_policy into that config;
adds max_num_turns, a positive integer that defaults to 1;
retains the configured number of completed conversation turns for per_conversation reuse.

The default remains one retained turn, so runtime behavior is unchanged unless max_num_turns is configured. The configuration shape changes from:

kv_cache_config:
  block_reuse_policy: per_conversation

to:

kv_cache_config:
  block_reuse_config:
    block_reuse_policy: per_conversation
    max_num_turns: 5

Test Coverage

Pre-commit checks passed.
MARS compilation completed successfully in Slurm job 277527.
Disaggregated AGA benchmarks started successfully for all_reusable, per_request, per_conversation with one turn, and per_conversation with five turns. Each run processed more than 7,000 main-conversation requests before cancellation.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

jiaganc · 2026-07-31T03:13:09Z

/bot run

tensorrt-cicd · 2026-07-31T03:18:32Z

PR_Github #62933 [ run ] triggered by Bot. Commit: 59b867a Link to invocation

coderabbitai · 2026-07-31T03:22:32Z

Walkthrough

The change introduces nested BlockReuseConfig settings for KV cache reuse, adds bounded conversation-turn retention, updates cache-manager consumers, and migrates examples, tests, telemetry, and public exports.

Changes

Block reuse configuration and conversation retention

Layer / File(s)	Summary
BlockReuseConfig API contract `tensorrt_llm/llmapi/llm_args.py`, `tensorrt_llm/llmapi/__init__.py`, `tensorrt_llm/usage/llm_args_golden_manifest.json`, `tests/unittest/llmapi/test_llm_args.py`	Adds nested block reuse policy and `max_num_turns` configuration. Updates exports, manifest paths, validation, serialization, and warning tests.
Runtime conversation retention `tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py`, `tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py`, `tests/unittest/_torch/executor/*`	Uses nested block reuse settings. Retains bounded conversation drop handles and tests eviction after the configured number of turns.
Configuration migration across integrations `examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py`, `tests/unittest/disaggregated/*`, `tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py`	Replaces string-based `block_reuse_policy` fields with per-instance `BlockReuseConfig` values.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Request
  participant KVCacheManagerV2
  participant ConversationManager
  participant KVCache
  Request->>KVCacheManagerV2: Start conversation request
  KVCacheManagerV2->>ConversationManager: Pass block reuse policy and max_num_turns
  ConversationManager->>KVCache: Store planned drop handle
  ConversationManager->>KVCache: Drop oldest handle after the turn limit
  KVCacheManagerV2-->>Request: Return cache reuse result

Possibly related PRs

NVIDIA/TensorRT-LLM#14047: Modifies KV cache manager v2 block-reuse configuration and implementation integration.
NVIDIA/TensorRT-LLM#16957: Relates to TriAttention block-reuse compatibility and KVCacheManagerV2 configuration.

Suggested reviewers: schetlur-nv, qijune, eopxd

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the breaking feature change: configurable retention of the last N turns for per-conversation KV-cache reuse.
Description check	✅ Passed	The description explains the problem, solution, API change, default behavior, test coverage, and checklist status.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/llmapi/llm_args.py`:
- Around line 3621-3649: Update stale block_reuse_policy references: in
test_mamba_cache_manager.py, pass the policy through KvCacheConfig’s
block_reuse_config using BlockReuseConfig(block_reuse_policy=...) instead of
passing the field directly; in telemetry.md, access it via
kv_cache_config.block_reuse_config.block_reuse_policy.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e7fd2494-2650-4401-a6c8-cc06c53c6e64

📥 Commits

Reviewing files that changed from the base of the PR and between 138eb43 and 59b867a.

📒 Files selected for processing (11)

examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py
tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py
tensorrt_llm/llmapi/__init__.py
tensorrt_llm/llmapi/llm_args.py
tensorrt_llm/usage/llm_args_golden_manifest.json
tests/unittest/_torch/executor/test_kv_cache_manager_v2.py
tests/unittest/disaggregated/test_cache_transceiver_single_process.py
tests/unittest/disaggregated/test_kv_transfer.py
tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py
tests/unittest/llmapi/test_llm_args.py

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

jiaganc · 2026-07-31T03:38:09Z

/bot run

coderabbitai

🧹 Nitpick comments (1)

tests/unittest/_torch/executor/test_mamba_cache_manager.py (1)
1651-1684: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Expose max_num_turns in the test helper.

The generic KVCacheManagerV2 test covers max_num_turns=2, but no MambaHybridCacheManagerV2 test covers multi-turn retention or eviction. Add the parameter, pass it to BlockReuseConfig, and add hybrid coverage, including the five-turn case.

Test coverage verdict: insufficient.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/_torch/executor/test_mamba_cache_manager.py` around lines 1651
- 1684, Update _build_v2_hybrid_with_mamba_layer to accept a max_num_turns
parameter and pass it into BlockReuseConfig. Add MambaHybridCacheManagerV2 tests
covering multi-turn retention and eviction, including the five-turn case,
matching the existing KVCacheManagerV2 max_num_turns=2 coverage.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/_torch/executor/test_mamba_cache_manager.py`:
- Around line 1651-1684: Update _build_v2_hybrid_with_mamba_layer to accept a
max_num_turns parameter and pass it into BlockReuseConfig. Add
MambaHybridCacheManagerV2 tests covering multi-turn retention and eviction,
including the five-turn case, matching the existing KVCacheManagerV2
max_num_turns=2 coverage.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ba22a143-156c-4b0c-84da-c197c2383f09

📥 Commits

Reviewing files that changed from the base of the PR and between 59b867a and 8f8ad1a.

📒 Files selected for processing (2)

docs/source/developer-guide/telemetry.md
tests/unittest/_torch/executor/test_mamba_cache_manager.py

tensorrt-cicd · 2026-07-31T03:43:37Z

PR_Github #62940 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

tensorrt-cicd · 2026-07-31T03:47:06Z

PR_Github #62933 [ run ] completed with state ABORTED. Commit: 59b867a

Link to invocation

tensorrt-cicd · 2026-07-31T04:36:48Z

PR_Github #62940 [ run ] completed with state SUCCESS. Commit: 8f8ad1a
/LLM/main/L0_MergeRequest_PR pipeline #51058 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

jiaganc · 2026-07-31T04:54:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-31T05:00:42Z

PR_Github #62959 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

qiaoxj07 · 2026-08-01T02:10:15Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-01T02:16:42Z

PR_Github #63169 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

tensorrt-cicd · 2026-08-01T02:16:50Z

PR_Github #62959 [ run ] completed with state ABORTED. Commit: 8f8ad1a

Link to invocation

tensorrt-cicd · 2026-08-01T05:41:41Z

PR_Github #63169 [ run ] completed with state FAILURE. Commit: 8f8ad1a
/LLM/main/L0_MergeRequest_PR pipeline #51253 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

jiaganc · 2026-08-02T14:11:11Z

/bot run

tensorrt-cicd · 2026-08-02T14:17:04Z

PR_Github #63282 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

tensorrt-cicd · 2026-08-02T14:17:05Z

PR_Github #63282 [ run ] completed with state DISABLED
Pipeline is freezed and top-1 instance is under maintenance. For urgent request, contact Yiteng Niu

Link to invocation

qiaoxj07 · 2026-08-03T01:27:25Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-03T01:33:43Z

PR_Github #63319 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

tensorrt-cicd · 2026-08-03T05:15:08Z

PR_Github #63319 [ run ] completed with state FAILURE. Commit: 8f8ad1a
/LLM/main/L0_MergeRequest_PR pipeline #51313 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

qiaoxj07 · 2026-08-03T05:18:37Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-03T05:24:01Z

PR_Github #63358 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

tensorrt-cicd · 2026-08-03T06:48:43Z

PR_Github #63358 [ run ] completed with state SUCCESS. Commit: 8f8ad1a
/LLM/main/L0_MergeRequest_PR pipeline #51345 completed with status: 'SUCCESS'

CI Report

Link to invocation

yuanjingx87

Approved on oss compliance perspective.

VALLIS-NERIA · 2026-08-04T01:08:57Z

Please add a "save last N turns" test case of MambaHybridCacheManagerV2.

QiJune · 2026-08-04T01:41:29Z

+class BlockReuseConfig(StrictBaseModel):
+    """Configuration for KV cache block reuse policies."""
+
+    block_reuse_policy: Literal[


Could we rename block_reuse_policy to policy? The current path，block_reuse_config.block_reuse_policy, is redundant.

github-actions Bot assigned jiaganc Jul 27, 2026

jiaganc added the api-breaking Accepted LLM API contract change that is backwards-incompatible label Jul 27, 2026

jiaganc force-pushed the codex/block-reuse-config branch from 5df809d to c53368e Compare July 27, 2026 03:29

Add configurable multi-turn block reuse

59b867a

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

jiaganc force-pushed the codex/block-reuse-config branch 2 times, most recently from 65e1b71 to 59b867a Compare July 31, 2026 03:00

jiaganc changed the title ~~[None][feat] BREAKING Add configurable multi-turn block reuse~~ [None][feat] BREAKING Support saving last N turns in per-conversation policy Jul 31, 2026

jiaganc marked this pull request as ready for review July 31, 2026 03:12

jiaganc requested review from a team as code owners July 31, 2026 03:12

jiaganc requested review from QiJune, SimengLiu-nv, arysef, chuangz0, nvpohanh and pcastonguay July 31, 2026 03:12

coderabbitai Bot reviewed Jul 31, 2026

View reviewed changes

Comment thread tensorrt_llm/llmapi/llm_args.py

[None][fix] Address stale block reuse policy references

8f8ad1a

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

coderabbitai Bot reviewed Jul 31, 2026

View reviewed changes

arysef approved these changes Jul 31, 2026

View reviewed changes

coderabbitai Bot mentioned this pull request Jul 31, 2026

[None][feat] Enable block reuse for flashinfer #17106

Open

1 task

longlee0622 enabled auto-merge (squash) August 3, 2026 22:15

longlee0622 approved these changes Aug 3, 2026

View reviewed changes

pcastonguay approved these changes Aug 3, 2026

View reviewed changes

yuanjingx87 approved these changes Aug 4, 2026

View reviewed changes

VALLIS-NERIA reviewed Aug 4, 2026

View reviewed changes

QiJune reviewed Aug 4, 2026

View reviewed changes

QiJune approved these changes Aug 4, 2026

View reviewed changes

Uh oh!

Conversation

jiaganc commented Jul 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

jiaganc commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

coderabbitai Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jiaganc commented Jul 31, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

jiaganc commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

qiaoxj07 commented Aug 1, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

jiaganc commented Aug 2, 2026

Uh oh!

tensorrt-cicd commented Aug 2, 2026

Uh oh!

tensorrt-cicd commented Aug 2, 2026

Uh oh!

qiaoxj07 commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

qiaoxj07 commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

yuanjingx87 left a comment

Choose a reason for hiding this comment

Uh oh!

VALLIS-NERIA Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

QiJune Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

jiaganc commented Jul 27, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 31, 2026 •

edited

Loading