Skip to content

[TRTLLM-14388][refactor] BREAKING: Remove 2-model code and tests - #16380

Open
mikeiovine wants to merge 4 commits into
NVIDIA:mainfrom
mikeiovine:remove-2-model
Open

[TRTLLM-14388][refactor] BREAKING: Remove 2-model code and tests#16380
mikeiovine wants to merge 4 commits into
NVIDIA:mainfrom
mikeiovine:remove-2-model

Conversation

@mikeiovine

@mikeiovine mikeiovine commented Jul 14, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Collaborator

Description

Remove 2-model code and related tests. We are past the deprecation period and all feature gaps have been closed.

Some 2-model code is left behind for a few niche things that can't use 1-model (ngram, user-provided).

Test Coverage

Existing tests.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Dev Engineer Review

  • Removes deprecated two-model speculative decoding paths, including draft-engine orchestration, ModelDrafter, MTPDraftModelForCausalLM, Eagle3SpecMetadata, and related CUDA-graph and KV-cache branches.
  • Consolidates Eagle3 and MTP Eagle routing around one-model execution while retaining NGram and user-provided speculative decoding paths.
  • Removes breaking API fields, enum members, constructor parameters, and helper arguments such as eagle3_one_model, mtp_eagle_one_model, is_draft_model, and draft_model_engine.
  • Updates speculative-decoding validation, sampler selection, resource-manager creation, metadata handling, attention metadata APIs, and model constructors for consistent mode dispatch.
  • Updates examples, documentation, manifests, configuration files, test baselines, QA lists, and waiver entries to remove deprecated options.
  • Verify all callers use the revised public signatures and that remaining two-model support is limited to documented niche modes.
  • Verify CI failures reported in the L0 merge-request pipeline before approval. Confirm that the updated test-list entries match valid parametrized test identifiers and contain no duplicates.

QA Engineer Review

  • Modified integration tests include Eagle3 and MTP accuracy tests, disaggregated serving tests, AutoDeploy tests, speculative-decoding examples, performance sanity tests, and end-to-end quickstart tests.
  • Removed speculative-decoding tests include test_draft_token_prepare_for_generation and test_draft_token_tree_sampling.
  • Renamed or simplified tests include test_eagle3, test_eagle3_spec_decoding_stats, test_multi_eagle3, test_get_num_spec_layers_*, test_accepts_mtp_eagle, test_detect_hidden_states_capture_last_layer_for_mtp_eagle, and test_autodeploy_eagle3_acceptance_rate.
  • Updated test implementations are represented in tests/integration/test_lists/test-db/, tests/integration/test_lists/qa/, and tests/integration/test_lists/waives.txt where applicable.
  • Removed two-model MTP coverage from l0_b200.yml.
  • Verdict: needs follow-up until the reported L0 pipeline failures are investigated and CBTS coverage is confirmed.

@mikeiovine mikeiovine added the api-breaking Accepted LLM API contract change that is backwards-incompatible label Jul 14, 2026
@mikeiovine

mikeiovine commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #59235 [ run ] triggered by Bot. Commit: 95afa29 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #59235 [ run ] completed with state ABORTED. Commit: 95afa29

Link to invocation

@mikeiovine
mikeiovine force-pushed the remove-2-model branch 6 times, most recently from 4764a86 to 97e26c8 Compare July 16, 2026 18:03
@mikeiovine

mikeiovine commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #59758 [ run ] triggered by Bot. Commit: 97e26c8 Link to invocation

@mikeiovine mikeiovine changed the title [None][chore] BREAKING: Remove 2-model code and tests [None][refactor] BREAKING: Remove 2-model code and tests Jul 16, 2026
@mikeiovine mikeiovine changed the title [None][refactor] BREAKING: Remove 2-model code and tests [TRTLLM-14388][refactor] BREAKING: Remove 2-model code and tests Jul 16, 2026
@mikeiovine
mikeiovine force-pushed the remove-2-model branch 3 times, most recently from 4807177 to cfe1571 Compare July 16, 2026 19:44
@mikeiovine

mikeiovine commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@mikeiovine
mikeiovine marked this pull request as ready for review July 16, 2026 20:12
@mikeiovine
mikeiovine requested review from a team as code owners July 16, 2026 20:12
@tensorrt-cicd

tensorrt-cicd commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #59996 [ run ] triggered by Bot. Commit: e6bb6c9 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #59996 [ run ] completed with state FAILURE. Commit: e6bb6c9
/LLM/main/L0_MergeRequest_PR pipeline #48389 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@mikeiovine

mikeiovine commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61388 [ run ] triggered by Bot. Commit: f917b0b Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61388 [ run ] completed with state FAILURE. Commit: f917b0b
/LLM/main/L0_MergeRequest_PR pipeline #49615 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@coderabbitai coderabbitai Bot mentioned this pull request Jul 23, 2026
Open
1 task
@mikeiovine

mikeiovine commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61624 [ run ] triggered by Bot. Commit: eedbdef Link to invocation

coderabbitai[bot]
coderabbitai Bot reviewed Jul 24, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tensorrt_llm/_torch/attention_backend/sparse/dsa.py (1)

1531-1541: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Per-step pinned host buffer allocation in a hot path.

host_block_table is freshly allocated (with pin_memory=prefer_pinned()) on every call to prepare_for_indexer_k_cache, which runs every forward step via prepare(). Every sibling staging buffer in this class (host_indexer_k_cache_block_offsets, host_slot_mapping_fp8, etc.) is pre-allocated once in create_buffers_for_indexer/__post_init__ and reused. Repeated pinned-memory allocation/free is a comparatively costly page-locking operation on a per-forward-step path.

♻️ Suggested fix: pre-allocate a persistent staging buffer 
-        # pool_indices already has correct values; set padding to -1.
-        # Stage through a fresh pinned buffer: an async H2D from pageable
-        # memory would block the host behind the busy execution stream.
-        host_block_table = torch.empty((pool_indices.shape[0], max_blocks_used),
-                                       dtype=pool_indices.dtype,
-                                       pin_memory=prefer_pinned())
-        host_block_table.copy_(pool_indices[:, :max_blocks_used])
+        # pool_indices already has correct values; set padding to -1.
+        # Stage through a persistent pinned buffer (allocated once in
+        # create_buffers_for_indexer) to avoid a per-step pinned alloc.
+        host_block_table = self.host_block_table_staging[:pool_indices.shape[0], :max_blocks_used]
+        host_block_table.copy_(pool_indices[:, :max_blocks_used])

And add a self.host_block_table_staging buffer of shape [max_num_sequences, max_blocks_per_seq] alongside the other persistent host buffers in create_buffers_for_indexer.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/_torch/attention_backend/sparse/dsa.py` around lines 1531 -
1541, Pre-allocate a persistent host_block_table_staging buffer in
create_buffers_for_indexer (or the corresponding __post_init__ setup) with shape
[max_num_sequences, max_blocks_per_seq], matching pool_indices’ dtype and
pinned-memory policy. Update prepare_for_indexer_k_cache to reuse this buffer
for copying and padding instead of allocating host_block_table on every call,
while preserving the existing per-step contents and masking behavior.
🤖 Prompt for all review comments with AI agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tensorrt_llm/_torch/attention_backend/sparse/dsa.py`:
- Around line 1531-1541: Pre-allocate a persistent host_block_table_staging
buffer in create_buffers_for_indexer (or the corresponding __post_init__ setup)
with shape [max_num_sequences, max_blocks_per_seq], matching pool_indices’ dtype
and pinned-memory policy. Update prepare_for_indexer_k_cache to reuse this
buffer for copying and padding instead of allocating host_block_table on every
call, while preserving the existing per-step contents and masking behavior.
ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ba4defc3-cdbf-42a2-905c-473d215f6d0f

📥 Commits

Reviewing files that changed from the base of the PR and between f917b0b and eedbdef.

📒 Files selected for processing (83)
  • .pre-commit-config.yaml
  • docs/source/blogs/tech_blog/blog06_Llama4_maverick_eagle_guide.md
  • docs/source/features/torch_compile_and_piecewise_cuda_graph.md
  • docs/source/legacy/advanced/speculative-decoding.md
  • examples/auto_deploy/model_registry/configs/super_v3_mtp.yaml
  • examples/llm-api/llm_speculative_decoding.py
  • examples/llm-api/quickstart_advanced.py
  • examples/models/core/deepseek_v4/README.md
  • examples/models/core/exaone/README.md
  • examples/models/core/mistral_large_3/README.md
  • examples/models/core/nemotron/README_nemotron_super_v3.md
  • legacy-files.txt
  • pyproject.toml
  • ruff-legacy.toml
  • tensorrt_llm/_torch/attention_backend/interface.py
  • tensorrt_llm/_torch/attention_backend/sparse/dsa.py
  • tensorrt_llm/_torch/attention_backend/trtllm.py
  • tensorrt_llm/_torch/auto_deploy/llm_args.py
  • tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py
  • tensorrt_llm/_torch/models/modeling_auto.py
  • tensorrt_llm/_torch/models/modeling_deepseekv3.py
  • tensorrt_llm/_torch/models/modeling_glm.py
  • tensorrt_llm/_torch/models/modeling_nemotron_h.py
  • tensorrt_llm/_torch/models/modeling_qwen3vl.py
  • tensorrt_llm/_torch/models/modeling_speculative.py
  • tensorrt_llm/_torch/pyexecutor/_util.py
  • tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py
  • tensorrt_llm/_torch/pyexecutor/model_engine.py
  • tensorrt_llm/_torch/pyexecutor/model_loader.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
  • tensorrt_llm/_torch/pyexecutor/sampler/sampler.py
  • tensorrt_llm/_torch/speculative/__init__.py
  • tensorrt_llm/_torch/speculative/drafter.py
  • tensorrt_llm/_torch/speculative/drafting_loops.py
  • tensorrt_llm/_torch/speculative/eagle3.py
  • tensorrt_llm/_torch/speculative/interface.py
  • tensorrt_llm/_torch/speculative/model_drafter.py
  • tensorrt_llm/_torch/speculative/mtp.py
  • tensorrt_llm/_torch/speculative/utils.py
  • tensorrt_llm/llmapi/llm_args.py
  • tensorrt_llm/usage/llm_args_golden_manifest.json
  • tests/integration/defs/.test_durations
  • tests/integration/defs/.test_durations_aws_dfw
  • tests/integration/defs/accuracy/test_disaggregated_serving.py
  • tests/integration/defs/accuracy/test_llm_api_autodeploy.py
  • tests/integration/defs/accuracy/test_llm_api_pytorch.py
  • tests/integration/defs/disaggregated/test_ad_disagg.py
  • tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_eagle_triton.yaml
  • tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_eagle_trtllm.yaml
  • tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py
  • tests/integration/defs/examples/serve/test_spec_decoding_metrics.py
  • tests/integration/defs/examples/test_ad_speculative_decoding.py
  • tests/integration/defs/perf/pytorch_model_config.py
  • tests/integration/defs/perf/test_perf_sanity.py
  • tests/integration/defs/test_e2e.py
  • tests/integration/test_lists/qa/llm_function_core.txt
  • tests/integration/test_lists/qa/llm_function_rtx6k.txt
  • tests/integration/test_lists/qa/llm_spark_func.yml
  • tests/integration/test_lists/test-db/l0_b200.yml
  • tests/integration/test_lists/test-db/l0_dgx_b200.yml
  • tests/integration/test_lists/test-db/l0_dgx_h100.yml
  • tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml
  • tests/integration/test_lists/test-db/l0_h100.yml
  • tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml
  • tests/integration/test_lists/waives.txt
  • tests/scripts/perf-sanity/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf/disaggregated/gb200_stress-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_eplb0_eagle3_ccb-NIXL.yaml
  • tests/scripts/perf/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/unittest/_torch/attention/sparse/deepseek_v4/test_deepseek_v4_cache_manager.py
  • tests/unittest/_torch/executor/test_dual_pool_kv_cache.py
  • tests/unittest/_torch/executor/test_kv_cache_estimation.py
  • tests/unittest/_torch/executor/test_model_loader_mx.py
  • tests/unittest/_torch/executor/test_py_executor.py
  • tests/unittest/_torch/executor/test_seq_slot_sizing.py
  • tests/unittest/_torch/helpers.py
  • tests/unittest/_torch/modeling/test_modeling_llama.py
💤 Files with no reviewable changes (34)
  • tests/unittest/_torch/executor/test_dual_pool_kv_cache.py
  • examples/auto_deploy/model_registry/configs/super_v3_mtp.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_eagle_triton.yaml
  • tensorrt_llm/_torch/speculative/drafting_loops.py
  • legacy-files.txt
  • tensorrt_llm/_torch/models/modeling_auto.py
  • pyproject.toml
  • tests/integration/defs/perf/pytorch_model_config.py
  • examples/models/core/nemotron/README_nemotron_super_v3.md
  • tensorrt_llm/usage/llm_args_golden_manifest.json
  • tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_eagle_trtllm.yaml
  • tests/integration/defs/disaggregated/test_ad_disagg.py
  • examples/llm-api/quickstart_advanced.py
  • examples/models/core/mistral_large_3/README.md
  • tests/scripts/perf-sanity/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/unittest/_torch/executor/test_kv_cache_estimation.py
  • tests/scripts/perf/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/integration/test_lists/test-db/l0_b200.yml
  • tests/scripts/perf/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/unittest/_torch/helpers.py
  • tensorrt_llm/_torch/attention_backend/interface.py
  • .pre-commit-config.yaml
  • tests/unittest/_torch/modeling/test_modeling_llama.py
  • tensorrt_llm/_torch/speculative/model_drafter.py
  • tests/scripts/perf/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/integration/defs/perf/test_perf_sanity.py
  • ruff-legacy.toml
  • tests/scripts/perf-sanity/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tensorrt_llm/_torch/models/modeling_nemotron_h.py
  • tests/scripts/perf/disaggregated/gb200_stress-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_eplb0_eagle3_ccb-NIXL.yaml
  • tests/integration/defs/test_e2e.py
  • tests/scripts/perf/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
🚧 Files skipped from review as they are similar to previous changes (30)
  • docs/source/blogs/tech_blog/blog06_Llama4_maverick_eagle_guide.md
  • docs/source/features/torch_compile_and_piecewise_cuda_graph.md
  • tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py
  • tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py
  • tests/unittest/_torch/attention/sparse/deepseek_v4/test_deepseek_v4_cache_manager.py
  • tests/unittest/_torch/executor/test_seq_slot_sizing.py
  • tests/integration/test_lists/qa/llm_spark_func.yml
  • tensorrt_llm/_torch/auto_deploy/llm_args.py
  • tensorrt_llm/_torch/speculative/drafter.py
  • tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml
  • tests/integration/test_lists/test-db/l0_dgx_h100.yml
  • tests/integration/test_lists/qa/llm_function_rtx6k.txt
  • tests/integration/defs/.test_durations_aws_dfw
  • tests/unittest/_torch/executor/test_py_executor.py
  • tensorrt_llm/_torch/pyexecutor/sampler/sampler.py
  • tensorrt_llm/_torch/speculative/mtp.py
  • tests/integration/defs/accuracy/test_llm_api_autodeploy.py
  • tests/integration/defs/examples/test_ad_speculative_decoding.py
  • tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py
  • tensorrt_llm/_torch/models/modeling_deepseekv3.py
  • tests/integration/test_lists/test-db/l0_dgx_b200.yml
  • tensorrt_llm/_torch/speculative/interface.py
  • tests/integration/test_lists/test-db/l0_h100.yml
  • tensorrt_llm/_torch/models/modeling_speculative.py
  • tests/integration/test_lists/qa/llm_function_core.txt
  • tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
  • tensorrt_llm/llmapi/llm_args.py
  • tensorrt_llm/_torch/speculative/utils.py
  • tests/integration/defs/accuracy/test_llm_api_pytorch.py
  • tensorrt_llm/_torch/pyexecutor/model_engine.py

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61624 [ run ] completed with state SUCCESS. Commit: eedbdef
/LLM/main/L0_MergeRequest_PR pipeline #49831 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@coderabbitai coderabbitai Bot mentioned this pull request Jul 31, 2026
Merged
1 task
mikeiovine and others added 3 commits August 3, 2026 11:10
@mikeiovine
[None][chore] Remove 2-model code and tests
985ff38 
Signed-off-by: Mike Iovine <miovine@nvidia.com>
@mikeiovine
[None][fix] Remove stale MTP mode argument
b2244c4 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Mike Iovine <miovine@nvidia.com>
@mikeiovine
Fix lints
aef7074 
Signed-off-by: Mike Iovine <miovine@nvidia.com>
@mikeiovine
mikeiovine requested a review from a team as a code owner August 3, 2026 15:14
@mikeiovine
mikeiovine requested a review from ruodil August 3, 2026 15:14
@mikeiovine

mikeiovine commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

coderabbitai[bot]
coderabbitai Bot reviewed Aug 3, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/source/legacy/advanced/speculative-decoding.md`:
- Line 178: Add the repository-standard NVIDIA copyright header to the top of
the modified speculative-decoding documentation file, with the copyright year
updated through 2026. Preserve the existing content unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 313fda1f-8886-4cae-a8a3-750aef5610fe

📥 Commits

Reviewing files that changed from the base of the PR and between eedbdef and aef7074.

📒 Files selected for processing (23)
  • .pre-commit-config.yaml
  • docs/source/blogs/tech_blog/blog06_Llama4_maverick_eagle_guide.md
  • docs/source/features/torch_compile_and_piecewise_cuda_graph.md
  • docs/source/legacy/advanced/speculative-decoding.md
  • examples/auto_deploy/model_registry/configs/super_v3_mtp.yaml
  • examples/llm-api/llm_speculative_decoding.py
  • examples/llm-api/quickstart_advanced.py
  • examples/models/core/deepseek_v4/README.md
  • examples/models/core/exaone/README.md
  • examples/models/core/mistral_large_3/README.md
  • examples/models/core/nemotron/README_nemotron_super_v3.md
  • legacy-files.txt
  • pyproject.toml
  • ruff-legacy.toml
  • tensorrt_llm/_torch/attention_backend/interface.py
  • tensorrt_llm/_torch/attention_backend/sparse/dsa.py
  • tensorrt_llm/_torch/attention_backend/trtllm.py
  • tensorrt_llm/_torch/auto_deploy/llm_args.py
  • tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py
  • tensorrt_llm/_torch/models/modeling_auto.py
  • tensorrt_llm/_torch/models/modeling_deepseekv3.py
  • tensorrt_llm/_torch/models/modeling_glm.py
  • tensorrt_llm/_torch/models/modeling_nemotron_h.py
💤 Files with no reviewable changes (11)
  • pyproject.toml
  • legacy-files.txt
  • examples/models/core/nemotron/README_nemotron_super_v3.md
  • ruff-legacy.toml
  • tensorrt_llm/_torch/models/modeling_auto.py
  • tensorrt_llm/_torch/attention_backend/interface.py
  • examples/models/core/mistral_large_3/README.md
  • examples/auto_deploy/model_registry/configs/super_v3_mtp.yaml
  • tensorrt_llm/_torch/attention_backend/sparse/dsa.py
  • tensorrt_llm/_torch/models/modeling_nemotron_h.py
  • examples/llm-api/quickstart_advanced.py
🚧 Files skipped from review as they are similar to previous changes (11)
  • tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py
  • docs/source/blogs/tech_blog/blog06_Llama4_maverick_eagle_guide.md
  • examples/llm-api/llm_speculative_decoding.py
  • tensorrt_llm/_torch/models/modeling_deepseekv3.py
  • tensorrt_llm/_torch/auto_deploy/llm_args.py
  • examples/models/core/deepseek_v4/README.md
  • docs/source/features/torch_compile_and_piecewise_cuda_graph.md
  • tensorrt_llm/_torch/attention_backend/trtllm.py
  • examples/models/core/exaone/README.md
  • tensorrt_llm/_torch/models/modeling_glm.py
  • .pre-commit-config.yaml

Comment thread docs/source/legacy/advanced/speculative-decoding.md
@tensorrt-cicd

tensorrt-cicd commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #63477 [ run ] triggered by Bot. Commit: aef7074 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #63477 [ run ] completed with state FAILURE. Commit: aef7074
/LLM/main/L0_MergeRequest_PR pipeline #51447 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@mikeiovine
[None][fix] Drop stale Eagle3SpecMetadata reference in model engine
53ac1e7 
The 2-model removal deleted Eagle3SpecMetadata and every producer of
request_accepted_path but left one use behind, which would raise
NameError on any speculative decoding step. Also tighten the legacy
ruff baseline to match the current tree.

Signed-off-by: Mike Iovine <miovine@nvidia.com>
@coderabbitai coderabbitai Bot mentioned this pull request Aug 3, 2026
Open
@mikeiovine

mikeiovine commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #63546 [ run ] triggered by Bot. Commit: 53ac1e7 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #63546 [ run ] completed with state FAILURE. Commit: 53ac1e7
/LLM/main/L0_MergeRequest_PR pipeline #51511 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments
@cascade812 cascade812 cascade812 approved these changes
@Tabrizian Tabrizian Tabrizian approved these changes
@VALLIS-NERIA VALLIS-NERIA VALLIS-NERIA approved these changes
@QiJune QiJune QiJune approved these changes
@LarryXFly LarryXFly LarryXFly approved these changes
@arysef arysef Awaiting requested review from arysef arysef is a code owner automatically assigned from NVIDIA/trt-llm-usage-telemetry-devs
@nv-xtf nv-xtf Awaiting requested review from nv-xtf nv-xtf is a code owner automatically assigned from NVIDIA/trt-llm-disagg-devs
@crazydemo crazydemo Awaiting requested review from crazydemo crazydemo is a code owner automatically assigned from NVIDIA/trt-llm-qa-function
@kris1025 kris1025 Awaiting requested review from kris1025 kris1025 is a code owner automatically assigned from NVIDIA/trt-llm-torch-attention-devs
@chenfeiz0326 chenfeiz0326 Awaiting requested review from chenfeiz0326 chenfeiz0326 is a code owner automatically assigned from NVIDIA/trt-llm-perf-devs
@dhansen-nvidia dhansen-nvidia Awaiting requested review from dhansen-nvidia dhansen-nvidia is a code owner automatically assigned from NVIDIA/trt-llm-perf-devs
@nvpohanh nvpohanh Awaiting requested review from nvpohanh nvpohanh is a code owner automatically assigned from NVIDIA/trt-llm-kv-cache-manager-devs
@SimengLiu-nv SimengLiu-nv Awaiting requested review from SimengLiu-nv SimengLiu-nv is a code owner automatically assigned from NVIDIA/trt-llm-kv-cache-manager-devs
@schetlur-nv schetlur-nv Awaiting requested review from schetlur-nv schetlur-nv is a code owner automatically assigned from NVIDIA/trt-llm-devs
@tburt-nv tburt-nv Awaiting requested review from tburt-nv tburt-nv is a code owner automatically assigned from NVIDIA/trt-llm-infra-devs
@yiqingy0 yiqingy0 Awaiting requested review from yiqingy0 yiqingy0 is a code owner automatically assigned from NVIDIA/trt-llm-infra-devs
@ruodil ruodil Awaiting requested review from ruodil ruodil is a code owner automatically assigned from NVIDIA/trt-llm-qa-serving

At least 1 approving review is required to merge this pull request.

Assignees

@mikeiovine mikeiovine

Labels

api-breaking Accepted LLM API contract change that is backwards-incompatible

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@mikeiovine @tensorrt-cicd @cascade812 @Tabrizian @VALLIS-NERIA @QiJune @LarryXFly