[https://nvbugs/6104831][test] Add cascade-prune reproducer tests - #13572
Conversation
a873694 to
d301166
Compare
…rmanent wedge Document the multi-signature disaggregated-serving wedge surfaced by the rc11 deployment. The report covers the 1P1D reproducer harness, the six labelled failure signatures (sender-side broken-promise after ready, trie cascade-prune assertion, decode-side bad optional access, gen-side checkGenTransferStatus blocking on at_least_num=1, receiver-side queued cancel broken-promise, and the suspected control-path send stall), their mapping to chained test/fix PR pairs (NVIDIA#13571/NVIDIA#13572 for sig #2, NVIDIA#13639/NVIDIA#13640 for sig #1), the in-flight fixes for sig #4 and sig #5, and the relationship to the unrelated companion fixes NVIDIA#12718 and NVIDIA#13119 which are not in rc11. Includes an investigation timeline that explains why each signature surfaced only after the previous one was fixed, and a test-coverage analysis of why the existing unit and integration tests did not catch any of these bugs. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
📝 WalkthroughWalkthroughThis PR adds a bidirectional edge cleanup in the Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
|
/bot run --disable-fail-fast |
|
PR_Github #46515 [ run ] triggered by Bot. Commit: |
|
PR_Github #46515 [ run ] completed with state
|
Add unit tests that isolate the KV-cache-block trie invariant violation behind the cascade-prune assertion seen in NVBugs 6104831. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
d301166 to
d8395a5
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #46675 [ run ] triggered by Bot. Commit: |
|
PR_Github #46675 [ run ] completed with state |
…IDIA#13572) Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Summary by CodeRabbit
Bug Fixes
Tests
Summary
Add four
RadixBlockTreeTestregression tests that lock in the fixed behavior for the KV-cache-block trie cascade-prune assertion reported in NVBugs 6104831 (templatedTrie.h:249 cascade prune: parent did not find this node as a child).Note on the fix
The original
templatedTrie.hfix from this PR is already inmain— it landed earlier as part of #13578 ("[None][fix] Add TestServePrefixAwareScheduling base on LMBenchmark/synthetic-multi-round-qa"), where the sameitr->second->setPrevNode(NodePtr{})line was bundled together with a prefix-aware scheduling test. PR #13578 was opened on 2026-04-20, ~8 days before this PR, and merged intomain~47 minutes before this PR was opened.After rebasing onto current
main, this PR no longer carries any change tocpp/include/tensorrt_llm/batch_manager/templatedTrie.h; only the four reproducer tests remain. They were not part of #13578, so they're still useful as a focused regression net for the trie invariant.Test Coverage
Four new
RadixBlockTreeTestcases isolate the cascade-prune regression at three layers, all usingEXPECT_NO_THROWagainst the fixed behavior:Regression_NVBugs6104831_DetachOrphanedBlockDoesNotFireAssertion— minimal direct repro (one block, no descendants).Regression_NVBugs6104831_FreeBlockAndAllDescendantsOnOrphanedSubtreeDoesNotFireAssertion— same root cause throughfreeBlockAndAllDescendants, matching the bug report's stack trace literally.Regression_NVBugs6104831_StoreBlocksRekeyDoesNotTriggerAssertion— productionstoreBlocks-style re-keying viaremoveNextBlock+addNextBlock.Regression_NVBugs6104831_StressPrefixOverlappingInsertEvictReuseLoop— stress loop on prefix-overlapping sequences mirroring the disagg insert/evict workload.The full
radixBlockTreeTestsuite remains green.PR Checklist
Please review the following before submitting your PR:
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.