Skip to content

[https://nvbugs/6104831][test] Add cascade-prune reproducer tests - #13572

Merged
chienchunhung merged 1 commit into
NVIDIA:mainfrom
chienchunhung:dev/nvbug-6104831-cascade-prune-fix
May 5, 2026
Merged

[https://nvbugs/6104831][test] Add cascade-prune reproducer tests#13572
chienchunhung merged 1 commit into
NVIDIA:mainfrom
chienchunhung:dev/nvbug-6104831-cascade-prune-fix

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • Bug Fixes

    • Fixed a critical assertion error in batch manager operations that could cause crashes during memory block removal and reallocation sequences.
  • Tests

    • Added comprehensive unit tests covering batch manager edge cases to prevent assertion failure regression.

Summary

Add four RadixBlockTreeTest regression tests that lock in the fixed behavior for the KV-cache-block trie cascade-prune assertion reported in NVBugs 6104831 (templatedTrie.h:249 cascade prune: parent did not find this node as a child).

Note on the fix

The original templatedTrie.h fix from this PR is already in main — it landed earlier as part of #13578 ("[None][fix] Add TestServePrefixAwareScheduling base on LMBenchmark/synthetic-multi-round-qa"), where the same itr->second->setPrevNode(NodePtr{}) line was bundled together with a prefix-aware scheduling test. PR #13578 was opened on 2026-04-20, ~8 days before this PR, and merged into main ~47 minutes before this PR was opened.

After rebasing onto current main, this PR no longer carries any change to cpp/include/tensorrt_llm/batch_manager/templatedTrie.h; only the four reproducer tests remain. They were not part of #13578, so they're still useful as a focused regression net for the trie invariant.

Test Coverage

Four new RadixBlockTreeTest cases isolate the cascade-prune regression at three layers, all using EXPECT_NO_THROW against the fixed behavior:

  • Regression_NVBugs6104831_DetachOrphanedBlockDoesNotFireAssertion — minimal direct repro (one block, no descendants).
  • Regression_NVBugs6104831_FreeBlockAndAllDescendantsOnOrphanedSubtreeDoesNotFireAssertion — same root cause through freeBlockAndAllDescendants, matching the bug report's stack trace literally.
  • Regression_NVBugs6104831_StoreBlocksRekeyDoesNotTriggerAssertion — production storeBlocks-style re-keying via removeNextBlock + addNextBlock.
  • Regression_NVBugs6104831_StressPrefixOverlappingInsertEvictReuseLoop — stress loop on prefix-overlapping sequences mirroring the disagg insert/evict workload.

The full radixBlockTreeTest suite remains green.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why.
  • PR follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
  • Test cases are provided for new code paths.
  • No new dependencies introduced.
  • No CODEOWNERS changes.
  • No documentation changes needed.
  • No tava architecture changes.
  • The reviewers assigned automatically/manually are appropriate for the PR.
  • Please check this after reviewing the above items as appropriate for this PR.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@chienchunhung
chienchunhung force-pushed the dev/nvbug-6104831-cascade-prune-fix branch from a873694 to d301166 Compare April 28, 2026 20:48
chienchunhung added a commit to chienchunhung/TensorRT-LLM that referenced this pull request Apr 30, 2026
…rmanent wedge

Document the multi-signature disaggregated-serving wedge surfaced by the
rc11 deployment. The report covers the 1P1D reproducer harness, the six
labelled failure signatures (sender-side broken-promise after ready,
trie cascade-prune assertion, decode-side bad optional access, gen-side
checkGenTransferStatus blocking on at_least_num=1, receiver-side queued
cancel broken-promise, and the suspected control-path send stall),
their mapping to chained test/fix PR pairs (NVIDIA#13571/NVIDIA#13572 for sig #2,
NVIDIA#13639/NVIDIA#13640 for sig #1), the in-flight fixes for sig #4 and sig #5,
and the relationship to the unrelated companion fixes NVIDIA#12718 and NVIDIA#13119
which are not in rc11. Includes an investigation timeline that explains
why each signature surfaced only after the previous one was fixed, and
a test-coverage analysis of why the existing unit and integration tests
did not catch any of these bugs.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Made-with: Cursor
@chienchunhung
chienchunhung requested a review from thorjohnsen May 1, 2026 00:32
@chienchunhung
chienchunhung marked this pull request as ready for review May 1, 2026 00:32
@coderabbitai

coderabbitai Bot commented May 1, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

This PR adds a bidirectional edge cleanup in the Node::clearNode method to properly detach child nodes from their parent by resetting back-pointers, and introduces four comprehensive unit tests to validate the fix prevents cascade-prune assertion failures during block reattachment operations.

Changes

Cohort / File(s) Summary
Bidirectional edge cleanup and regression tests
cpp/include/tensorrt_llm/batch_manager/templatedTrie.h, cpp/tests/unit_tests/batch_manager/radixBlockTreeTest.cpp
Added bidirectional edge update in Node::clearNode to reset child node back-pointers before removal, and introduced four new unit tests covering orphaned block detachment, cascade-prune scenarios, block re-keying sequences, and stress testing for overlapping prefix operations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately summarizes the main change: adding reproducer tests for the cascade-prune assertion regression (NVBugs 6104831), matching the file changes in the PR.
Description check ✅ Passed The PR description comprehensively explains the issue, fix, test coverage, and includes a properly formatted title with NVBugs ID and [fix] type.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Comment @coderabbitai help to get the list of available commands and usage tips.

@chienchunhung
chienchunhung requested a review from SimengLiu-nv May 1, 2026 17:31
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #46515 [ run ] triggered by Bot. Commit: d301166 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #46515 [ run ] completed with state SUCCESS. Commit: d301166
/LLM/main/L0_MergeRequest_PR pipeline #36574 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@SimengLiu-nv SimengLiu-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Add unit tests that isolate the KV-cache-block trie invariant violation behind the cascade-prune assertion seen in NVBugs 6104831.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung chienchunhung changed the title [https://nvbugs/6104831][fix] Detach pruned trie children [https://nvbugs/6104831][test] Add cascade-prune reproducer tests May 4, 2026
@chienchunhung
chienchunhung force-pushed the dev/nvbug-6104831-cascade-prune-fix branch from d301166 to d8395a5 Compare May 4, 2026 18:34
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #46675 [ run ] triggered by Bot. Commit: d8395a5 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #46675 [ run ] completed with state SUCCESS. Commit: d8395a5
/LLM/main/L0_MergeRequest_PR pipeline #36716 completed with status: 'SUCCESS'

CI Report

Link to invocation

@chienchunhung
chienchunhung merged commit ad2fc22 into NVIDIA:main May 5, 2026
8 checks passed
yufeiwu-nv pushed a commit to yufeiwu-nv/TensorRT-LLM that referenced this pull request May 19, 2026
…IDIA#13572)

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants