[https://nvbugs/6503299][fix] Default fabric memory KV pool for Python cache transceiver by chuangz0 · Pull Request #16832 · NVIDIA/TensorRT-LLM

chuangz0 · 2026-07-24T07:24:41Z

Description

When the Python cache transceiver (cache_transceiver_config.transceiver_runtime='PYTHON') is used together with the C++ V1 KV cache manager, the KV cache pool should be allocated with fabric memory so KV blocks can be transferred over MNNVL.

This PR:

Defaults TRTLLM_KVCACHE_POOL_USE_FABRIC_MEMORY=1 in KvCacheCreator.__init__ when:
- the cache transceiver is enabled with transceiver_runtime='PYTHON', and
- the resolved KV cache manager is the C++ V1 route (KVCacheManager and its hybrid subclasses; KVCacheManagerV2 is excluded).
The default is applied before any pool allocation (the C++ env getter caches on first read). An explicit user setting is always respected, and platforms without fabric memory support fall back to standard GPU allocation in C++ with a warning.
Sets kv_cache_bounce_size_mb: 512 in the GPT-OSS disaggregated perf configs (from [None][perf] Validate GPT-OSS transceiver v2 performance #15765) where the ctx/gen KV-head sharding differs (ctx TP1 → gen TP4, no attention DP), which requires the bounce buffer to regroup KV heads during transfer. Configs with matching per-rank KV-head layout (equal TP, or gen with attention DP) are intentionally left unchanged.

Test Coverage

Covered by existing disaggregated serving tests using transceiver_runtime: PYTHON.
The updated perf configs are exercised by the disaggregated perf-sanity/perf pipelines.

PR Checklist

PR title and description are self-explanatory
Coding guidelines followed
Test coverage considered

🤖 Generated with Claude Code

Dev Engineer Review

Added KvCacheCreator._maybe_enable_fabric_memory_for_python_transceiver() in tensorrt_llm/_torch/pyexecutor/_util.py, called from KvCacheCreator.__init__.
When llm_args.cache_transceiver_config is present, backend is non-None, transceiver_runtime == "PYTHON", and the resolved KV cache manager class is the C++ V1 KVCacheManager, the helper defaults TRTLLM_KVCACHE_POOL_USE_FABRIC_MEMORY=1 only if the env var isn’t already set; otherwise it leaves existing env/config unchanged.
Updated GB200 GPT-OSS disaggregated performance/stress YAMLs to set kv_cache_bounce_size_mb: 512 under both:
- worker_config.gen.cache_transceiver_config
- worker_config.ctx.cache_transceiver_config
  for the NIXL variants:
- tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_1k1k_con64_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
- tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
- tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
- tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
- tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
- tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
- tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
- tests/scripts/perf/disaggregated/gb200_stress-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_eplb0_eagle3_ccb-NIXL.yaml

QA Engineer Review

Updated performance/disaggregated YAMLs under tests/scripts/ to add kv_cache_bounce_size_mb: 512 to both worker_config.gen.cache_transceiver_config and worker_config.ctx.cache_transceiver_config in the files listed in the Dev Engineer Review section above.
Updated tests/integration/test_lists/waives.txt by removing three perf/test_perf_sanity.py::test_e2e waiver entries for disagg_upload targeting gb300_glm-5-fp4_8k1k:
- disagg_upload-e2e: con1024/ctx1/dep2/gen1/dep8/eplb256/mtp1
- disagg_upload-gen_only: con1024/ctx1/dep2/gen1/dep8/eplb256/mtp1
- disagg_upload-gen_only: con512/ctx1/dep2/gen1/dep32/eplb0/mtp3
Test functions added/modified/removed: none (only YAML/config and the waiver list were updated).
Verdict: needs follow-up.

coderabbitai · 2026-07-24T07:29:52Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

The change enables Fabric-memory pooling for compatible Python/V1 KV-cache setups, adds a 512 MB KV-cache bounce size to eight GB200 GPT-OSS NIXL benchmark configurations, and removes three related performance-test waivers.

Changes

KV-cache Fabric memory configuration

Layer / File(s)	Summary
Fabric memory defaulting `tensorrt_llm/_torch/pyexecutor/_util.py`	`KvCacheCreator` conditionally sets `TRTLLM_KVCACHE_POOL_USE_FABRIC_MEMORY=1` for Python transceivers using the V1 `KVCacheManager`, without overriding explicit environment settings.
Benchmark cache bounce settings and waiver updates `tests/scripts/perf/disaggregated/gb200_NIXL.yaml`, `tests/integration/test_lists/waives.txt`	Generation and context transceiver configurations set `kv_cache_bounce_size_mb: 512`, and three matching performance-test waiver entries are removed.

Estimated code review effort: 2 (Simple) | ~15 minutes

Suggested reviewers: qijune

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the PR’s main change and follows the required [ticket][type] summary format.
Description check	✅ Passed	The description includes the required Description, Test Coverage, and PR Checklist sections and is specific enough.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

🧹 Nitpick comments (1)

tensorrt_llm/_torch/pyexecutor/_util.py (1)
347-347: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add the required return annotation.

As per coding guidelines, every Python function must be annotated; declare this helper as returning None.
Proposed fix
-def _maybe_enable_fabric_memory_for_python_transceiver(self):
+def _maybe_enable_fabric_memory_for_python_transceiver(self) -> None:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/_torch/pyexecutor/_util.py` at line 347, Update
_maybe_enable_fabric_memory_for_python_transceiver to include an explicit None
return annotation, preserving its existing behavior.
Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tensorrt_llm/_torch/pyexecutor/_util.py`:
- Line 347: Update _maybe_enable_fabric_memory_for_python_transceiver to include
an explicit None return annotation, preserving its existing behavior.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e54489eb-50c5-42cb-9963-81b94b2899ed

📥 Commits

Reviewing files that changed from the base of the PR and between 8514fa3 and 442491c.

📒 Files selected for processing (9)

tensorrt_llm/_torch/pyexecutor/_util.py
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_1k1k_con64_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_stress-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_eplb0_eagle3_ccb-NIXL.yaml

chuangz0 · 2026-07-27T06:05:21Z

/bot run --disable-fast-fail

tensorrt-cicd · 2026-07-27T06:11:26Z

PR_Github #61828 Bot args parsing error: usage: /bot [-h]
{run,kill,skip,submit,reviewers,reuse-pipeline,reuse-review} ...
/bot: error: unrecognized arguments: --disable-fast-fail

Link to invocation

chuangz0 · 2026-07-27T06:28:20Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T06:34:17Z

PR_Github #61833 [ run ] triggered by Bot. Commit: 7014802 Link to invocation

chuangz0 · 2026-07-28T01:53:47Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-28T02:00:04Z

PR_Github #62056 [ run ] triggered by Bot. Commit: 7821757 Link to invocation

ZhanruiSunCh

LGTM for infra part.

YihuiLu512

LGTM

tensorrt-cicd · 2026-07-28T11:38:23Z

PR_Github #62056 [ run ] completed with state FAILURE. Commit: 7821757
/LLM/main/L0_MergeRequest_PR pipeline #50241 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Default TRTLLM_KVCACHE_POOL_USE_FABRIC_MEMORY=1 when the Python cache transceiver (transceiver_runtime='PYTHON') is used with the C++ V1 KV cache manager, so the KV pool is allocated with fabric memory for MNNVL transfers. Explicit user settings are respected, and unsupported platforms fall back to standard allocation in C++. Also set kv_cache_bounce_size_mb=512 in the GPT-OSS disagg perf configs where the ctx/gen KV-head sharding differs (ctx TP1 -> gen TP4), which requires the bounce buffer to regroup KV heads during transfer. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/_torch/pyexecutor/_util.py`:
- Line 426: Update _maybe_enable_fabric_memory_for_python_transceiver to
explicitly declare a -> None return annotation, without changing its
implementation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d9358445-7f04-4bfa-b196-3d7182d88e47

📥 Commits

Reviewing files that changed from the base of the PR and between 7821757 and 524d48b.

📒 Files selected for processing (10)

tensorrt_llm/_torch/pyexecutor/_util.py
tests/integration/test_lists/waives.txt
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_1k1k_con64_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_stress-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_eplb0_eagle3_ccb-NIXL.yaml

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (7)

tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_1k1k_con64_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_stress-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_eplb0_eagle3_ccb-NIXL.yaml

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

chuangz0 · 2026-07-29T01:59:09Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-2"

tensorrt-cicd · 2026-07-29T02:05:13Z

PR_Github #62362 [ run ] triggered by Bot. Commit: 66471a3 Link to invocation

tensorrt-cicd · 2026-07-29T03:52:15Z

PR_Github #62362 [ run ] completed with state SUCCESS. Commit: 66471a3
/LLM/main/L0_MergeRequest_PR pipeline #50529 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chuangz0 · 2026-07-29T06:27:30Z

/bot skip --comment "all tests have passed"

tensorrt-cicd · 2026-07-29T06:32:57Z

PR_Github #62452 [ skip ] triggered by Bot. Commit: 66471a3 Link to invocation

tensorrt-cicd · 2026-07-29T06:45:47Z

PR_Github #62452 [ skip ] completed with state SUCCESS. Commit: 66471a3
Skipping testing for commit 66471a3

Link to invocation

chuangz0 requested review from a team as code owners July 24, 2026 07:24

chuangz0 requested review from Tabrizian, YihuiLu512, bo-nv, dc3671 and nv-ananjappa July 24, 2026 07:24

github-actions Bot assigned chuangz0 Jul 24, 2026

chuangz0 changed the title ~~[None][feat] Default fabric memory KV pool for Python cache transceiver~~ [https://nvbugs/6503299][fix] Default fabric memory KV pool for Python cache transceiver Jul 24, 2026

coderabbitai Bot reviewed Jul 24, 2026

View reviewed changes

Shixiaowei02 approved these changes Jul 27, 2026

View reviewed changes

chuangz0 force-pushed the use_fabric_pool_default_for_python_cache_transceiver branch from 442491c to ab93d7a Compare July 27, 2026 06:04

chuangz0 requested a review from chenfeiz0326 July 27, 2026 06:23

chuangz0 requested review from a team as code owners July 27, 2026 06:27

chuangz0 requested review from LarryXFly, QiJune, StanleySun639, brnguyen2, dpitman-nvda and yiqingy0 July 27, 2026 06:27

chenfeiz0326 approved these changes Jul 27, 2026

View reviewed changes

StanleySun639 approved these changes Jul 27, 2026

View reviewed changes

coderabbitai Bot mentioned this pull request Jul 28, 2026

[https://nvbugs/6255417][fix] Unwaive qwen3next ci test #16924

Merged

1 task

yiqingy0 approved these changes Jul 28, 2026

View reviewed changes

lori-ren approved these changes Jul 28, 2026

View reviewed changes

EmmaQiaoCh reviewed Jul 28, 2026

View reviewed changes

Comment thread tests/integration/test_lists/waives.txt

tongyuantongyu reviewed Jul 28, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/_util.py

ZhanruiSunCh approved these changes Jul 28, 2026

View reviewed changes

YihuiLu512 approved these changes Jul 28, 2026

View reviewed changes

coderabbitai Bot mentioned this pull request Jul 28, 2026

[TRTLLM-13948][feat] Set DeepSeekV3 to use Python KV-cache transceiver V2 by default #16908

Open

1 task

chuangz0 added 2 commits July 29, 2026 09:13

unwaive some case on gb300

524d48b

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

chuangz0 force-pushed the use_fabric_pool_default_for_python_cache_transceiver branch from 7821757 to 524d48b Compare July 29, 2026 01:13

chuangz0 requested a review from a team as a code owner July 29, 2026 01:13

chuangz0 requested a review from jieli-matrix July 29, 2026 01:13

coderabbitai Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/_util.py Outdated

[None][chore] Add return annotation to fabric memory helper

66471a3

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

chuangz0 enabled auto-merge (squash) July 29, 2026 06:26

chuangz0 merged commit 097cbc1 into NVIDIA:main Jul 29, 2026
8 checks passed

nv-xtf mentioned this pull request Jul 30, 2026

[TRTLLM-12499][feat] Pipelined KVCache transfer for disaggregated serving in Python Cache Transceiver #15727

Open

This was referenced Jul 30, 2026

[https://nvbugs/6481034][test] Trace Kimi gen-only KV transfer timeout #16918

Closed

[https://nvbugs/6490049][test] Unwaive GB300 Kimi disagg e2e #16920

Open

chuangz0 mentioned this pull request Jul 31, 2026

[https://nvbugs/6541356][fix] align cache transceiver harness setup #17121

Open

Uh oh!

Conversation

chuangz0 commented Jul 24, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Dev Engineer Review

QA Engineer Review

Uh oh!

coderabbitai Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

chuangz0 commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

chuangz0 commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

chuangz0 commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

ZhanruiSunCh left a comment

Choose a reason for hiding this comment

Uh oh!

YihuiLu512 left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chuangz0 commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

chuangz0 commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

chuangz0 commented Jul 24, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading