[None][feat] KVCacheManagerV2 C++ translation by lowsfer · Pull Request #14047 · NVIDIA/TensorRT-LLM

lowsfer · 2026-05-12T11:44:10Z

Dev Engineer Review

Review the C++ KVCacheManagerV2 translation, including storage tiers, eviction, radix-tree reuse, CUDA virtual memory, copy/event handling, statistics, lifecycle management, and Python bindings for correctness, performance, API consistency, error handling, and regressions.
Review SHA-256 vendoring, architecture-specific dispatch, CMake integration, pre-commit exclusions, backend selection, and public API compatibility.
Validate configuration checks, memory/resource cleanup, cross-backend behavior, and the MPI shutdown-safety change.
CI follow-up is required: recent pipeline runs failed despite successful helper jobs; rerun with fail-fast disabled, including PerfSanity.

QA Engineer Review

Test code was added or updated in:
- kvCacheManagerV2HostMemTest.cpp
- kvCacheManagerV2StatsTest.cpp
- kvCacheManagerV2TypedIndexTest.cpp
- KV-cache event manager, manager behavior, salting, and stats API Python tests.
Coverage includes host-memory behavior, typed-index contracts, statistics and pending accounting, event parity, reuse introspection, salting, native C++ bindings, and request-ID handling.
No tests/integration/test_lists/, test-db/, or qa/ coverage changes are listed. Test functions should be mapped to CI or manual-QA entries.
Verdict: needs follow-up.

hyukn · 2026-05-15T09:51:10Z

/bot run --disable-fail-fast --stage-list "PerfSanity"

lowsfer · 2026-07-29T05:32:11Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T05:37:41Z

PR_Github #62430 [ run ] triggered by Bot. Commit: f0852d3 Link to invocation

tensorrt-cicd · 2026-07-29T05:39:48Z

PR_Github #62382 [ run ] completed with state ABORTED. Commit: d38b964

Link to invocation

tensorrt-cicd · 2026-07-29T09:40:20Z

PR_Github #62430 [ run ] completed with state SUCCESS. Commit: f0852d3
/LLM/main/L0_MergeRequest_PR pipeline #50588 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-29T10:10:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T10:16:22Z

PR_Github #62500 [ run ] triggered by Bot. Commit: 392cfea Link to invocation

tensorrt-cicd · 2026-07-29T13:48:40Z

PR_Github #62500 [ run ] completed with state FAILURE. Commit: 392cfea
/LLM/main/L0_MergeRequest_PR pipeline #50646 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-29T17:15:51Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T17:21:49Z

PR_Github #62563 [ run ] triggered by Bot. Commit: b9b5a75 Link to invocation

tensorrt-cicd · 2026-07-29T23:36:00Z

PR_Github #62563 [ run ] completed with state FAILURE. Commit: b9b5a75
/LLM/main/L0_MergeRequest_PR pipeline #50707 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…dings Migrates tensorrt_llm/runtime/kv_cache_manager_v2 from pure Python to a C++ implementation under cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/ with nanobind bindings compiled into bindings.so, preserving the same public API. The dispatcher __init__.py selects the backend via TLLM_KV_CACHE_MANAGER_V2_BACKEND (default "cpp"); both backends pass the shared test suite. CODEOWNERS assigns the new C++ tree to trt-llm-kv-cache-manager-devs. Includes ports of subsequent main features: commit-min-snapshot + SWA-slot reservation, SHA-256 block-key hashing, CUDA-graph request IDs, event manager and stats API to C++, uint64 ReuseScope salt/lora_id, resume-utilization KV constraints, per-conversation KV cache block reuse (PlannedDropHandle), the reuse-probe _KVCache refactor (compute the ReuseMatch once in the manager and pass it into the cache, NVIDIA#14333), SSM-snapshot iteration stats (NVIDIA#16598), the simplified plan_committed_block_drop (db117e6276), and an MPI teardown fix for the unittest/bindings CI shard. Migration planning docs (TODO.md, MIGRATION_PLAN_CPP.md, CPP_MIGRATION_PLAN_MAIN_15633.md) are kept on a separate branch. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

lowsfer · 2026-07-30T03:17:03Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-30T03:22:28Z

PR_Github #62674 [ run ] triggered by Bot. Commit: 1c6c8f2 Link to invocation

tensorrt-cicd · 2026-07-30T14:44:51Z

PR_Github #62674 [ run ] completed with state FAILURE. Commit: 1c6c8f2
/LLM/main/L0_MergeRequest_PR pipeline #50814 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-30T15:31:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-30T15:39:41Z

PR_Github #62796 [ run ] triggered by Bot. Commit: 1c6c8f2 Link to invocation

lowsfer · 2026-07-30T19:21:11Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-30T19:23:02Z

PR_Github #62796 [ run ] completed with state FAILURE. Commit: 1c6c8f2
/LLM/main/L0_MergeRequest_PR pipeline #50923 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-07-30T19:28:24Z

PR_Github #62838 [ run ] triggered by Bot. Commit: 1c6c8f2 Link to invocation

tensorrt-cicd · 2026-07-30T21:58:24Z

PR_Github #62838 [ run ] completed with state FAILURE. Commit: 1c6c8f2
/LLM/main/L0_MergeRequest_PR pipeline #50961 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-30T22:21:20Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-30T22:28:08Z

PR_Github #62862 [ run ] triggered by Bot. Commit: 1c6c8f2 Link to invocation

tensorrt-cicd · 2026-07-30T23:17:17Z

PR_Github #62862 [ run ] completed with state SUCCESS. Commit: 1c6c8f2
/LLM/main/L0_MergeRequest_PR pipeline #50985 completed with status: 'SUCCESS'

CI Report

Link to invocation

github-actions Bot assigned lowsfer May 12, 2026

lowsfer force-pushed the kvCacheManagerV2-cpp branch 2 times, most recently from 5bd984d to 456cfa8 Compare May 26, 2026 08:44

lowsfer force-pushed the kvCacheManagerV2-cpp branch 3 times, most recently from 4f4dd8d to 0802c6c Compare June 15, 2026 10:25

lowsfer force-pushed the kvCacheManagerV2-cpp branch 6 times, most recently from 4c24d8a to 44c797c Compare June 18, 2026 09:30

lowsfer changed the title ~~Draft: Kv cache manager v2 cpp~~ [None][feat] KVCacheManagerV2 C++ translation Jun 19, 2026

lowsfer force-pushed the kvCacheManagerV2-cpp branch 5 times, most recently from b4592c1 to 5a1fa77 Compare July 10, 2026 10:10

lowsfer added the api-compatible Accepted LLM API contract change that is backwards-compatible label Jul 10, 2026

juney-nvidia requested a review from thorjohnsen July 14, 2026 06:37

lowsfer force-pushed the kvCacheManagerV2-cpp branch 3 times, most recently from fd7d167 to 99bfc50 Compare July 20, 2026 03:03

lowsfer marked this pull request as ready for review July 20, 2026 05:16

lowsfer requested review from a team as code owners July 20, 2026 05:16

StanleySun639 approved these changes Jul 29, 2026

View reviewed changes

karljang mentioned this pull request Jul 31, 2026

[https://nvbugs/6379316][fix] Reject MNNVL on split NVLink topology #16603

Open

Uh oh!

Conversation

lowsfer commented May 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Uh oh!

hyukn commented May 15, 2026

Uh oh!

lowsfer commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

lowsfer commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

lowsfer commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

lowsfer commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

lowsfer commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

lowsfer commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

lowsfer commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

lowsfer commented May 12, 2026 •

edited by coderabbitai Bot

Loading