[https://nvbugs/6312828][fix] Recover disaggregated requests after timeouts by chienchunhung · Pull Request #16402 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-07-15T05:08:06Z

Summary

Fixes NVBug 6312828, where high-concurrency disaggregated Qwen3-32B requests could become terminal at the HTTP layer while backend KV-transfer work remained stranded, preventing service recovery.

The change aligns serving, executor, and C++ transceiver lifecycle state. It does not enable in-flight C++ cancellation or tune admission policy.

Change Breakdown

The PR changes 30 existing files (+5,379/-422). Most of the diff is regression coverage, not production logic:

Scope	Diff	Purpose
Serving/HTTP production	`+785/-138`	Propagate one deadline; define 504/SSE terminal behavior; bound retry, reservation, task, stream, and shutdown cleanup
PyExecutor production	`+428/-59`	Terminalize queued cancellations; continue polling/reaping asynchronous transfers; coordinate rank-wide drain
C++ transceiver production	`+364/-65`	Make default-off cancellation phase-safe; reject late/replayed handshakes; bound retained handshake state
Tests and CI configuration	`+3,802/-160`	Add C++/Python lifecycle matrices, the 10K stress-and-recovery checks, QA accounting, and waiver updates

Production code totals +1,577/-262; tests and CI configuration account for about 68% of total churn. There are no generated, vendored, binary, or copied-fixture changes.

Fix Scope

Apply one request deadline across routing, CTX/GEN dispatch, retries, and response streaming.
Return HTTP 504 before headers; after SSE begins, emit an in-band timeout error followed by [DONE]. Expired requests and upstream 504 responses are not retried.
Make coordinator accounting, reservations, sibling tasks, upstream streams, shutdown, and request cleanup cancellation-safe and bounded. Queued requests reach a terminal state before executor bookkeeping is removed.
Keep an otherwise-idle CTX executor polling while any rank owns asynchronous KV-transfer work, and defer drain-required control operations until every rank releases ownership.
Make default-off C++ cancellation phase-safe: cancellation wins before peer-session publication; after the first peer arrives, the transfer drains naturally. GEN remains observe-only until C++ reports a rank-consistent terminal result.
Reject late or replayed handshakes with ready=false, and bound retained pre-handshake/finalized state.
Preserve the reported workload at 10,000 requests and concurrency 512, with a 180-second request deadline and 5-second/10-second heartbeat settings.

Scope boundaries:

C++ in-flight transfer cancellation remains disabled by default; permanently wedged active transport operations are not claimed resolved.
The Python V2 transceiver protocol is unchanged, and kv_transfer_admission_window_multiplier remains 1.
B200 DeepSeek (NVBug 6472256) and H100 Qwen3.5 (NVBug 6479324) are separate workloads and are not claimed resolved here.

Validation

Current head: d97d6c7cf53f72a68a4e42ed6fde6cff14a149a1

Full pre-merge CI passed on the current head: helper #62586 / pipeline #50728 / report. The fresh rerun reported 411 passed, 0 failed, and 193 skipped, while reusing 77 successful same-head stages.
The exact B200 and H100 Qwen3-32B stress leaves passed focused QA on earlier production revision fca8e33571 and are unwaived. These exact QA-only workloads were not rerun on final head d97d6c7; the current-head evidence is the full CI result above.

Test / workload	Evidence	Result and waiver status
B200 `test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]` 10,000 requests, concurrency 512	helper #60877 / QA #311 at `fca8e33571`	Passed; unwaived
H100 `test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]` 10,000 requests, concurrency 512	Test-only mirror PR #16641: helper #60878 / QA #312 / cluster #1767 at `fca8e33571`	Leaf passed; 10,000 terminal records and GSM8K recovery score 0.674 (>0.42); unwaived. QA was overall unstable because the separate Qwen3.5 leaf failed.
B200 `test_disaggregated_stress_test[input8k-output1k-conc512-deepseek_r1_v2_fp4_stress]` 35,000 requests (NVBug 6472256)	Not verified or claimed resolved here	Remains waived
H100 `test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_5_4b_fp8_stress]` 3,000 requests (NVBug 6479324)	Executed alongside Qwen3-32B in QA #312; failed on the separate hybrid-cache dtype issue targeted by PR #16505	Excluded; remains waived

chienchunhung · 2026-07-15T05:37:38Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T05:43:30Z

PR_Github #59370 [ run ] triggered by Bot. Commit: fb8b982 Link to invocation

…16402) This update ensures that commented-out lines in the Groovy configuration are ignored during stage parsing, preventing disabled stage configurations from being processed. Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>

tensorrt-cicd · 2026-07-15T12:32:17Z

PR_Github #59370 [ run ] completed with state FAILURE. Commit: fb8b982
LLM_FUNCTION_AUTO_V2C #262 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-15T18:42:24Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T19:06:55Z

PR_Github #59512 [ kill ] triggered by Bot. Commit: 0ad94a2 Link to invocation

tensorrt-cicd · 2026-07-15T19:07:08Z

PR_Github #59512 [ kill ] completed with state SUCCESS. Commit: 0ad94a2
Successfully killed previous jobs for commit 0ad94a2

Link to invocation

chienchunhung · 2026-07-15T19:07:39Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T19:14:14Z

PR_Github #59514 [ run ] triggered by Bot. Commit: 0ad94a2 Link to invocation

chienchunhung · 2026-07-15T22:46:01Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T22:51:30Z

PR_Github #59545 [ run ] triggered by Bot. Commit: 6fa5884 Link to invocation

tensorrt-cicd · 2026-07-16T04:03:11Z

PR_Github #59545 [ run ] completed with state FAILURE. Commit: 6fa5884
LLM_FUNCTION_AUTO_V2C #270 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-16T15:19:18Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-16T15:26:01Z

PR_Github #59737 [ run ] triggered by Bot. Commit: 92bfff6 Link to invocation

tensorrt-cicd · 2026-07-16T16:22:35Z

PR_Github #59737 [ run ] completed with state FAILURE. Commit: 92bfff6
LLM_FUNCTION_AUTO_V2C #271 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-17T03:15:52Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-17T03:21:35Z

PR_Github #59856 [ run ] triggered by Bot. Commit: f58de71 Link to invocation

tensorrt-cicd · 2026-07-17T04:35:11Z

PR_Github #59856 [ run ] completed with state FAILURE. Commit: f58de71
LLM_FUNCTION_AUTO_V2C #272 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-17T05:24:41Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-17T05:30:21Z

PR_Github #59886 [ run ] triggered by Bot. Commit: e877ea8 Link to invocation

tensorrt-cicd · 2026-07-17T07:24:59Z

PR_Github #59886 [ run ] completed with state FAILURE. Commit: e877ea8
LLM_FUNCTION_AUTO_V2C #273 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

tensorrt-cicd · 2026-07-26T22:02:14Z

PR_Github #61746 [ run ] completed with state FAILURE. Commit: 0c4771a
/LLM/main/L0_MergeRequest_PR pipeline #49948 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-27T17:25:39Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T17:32:52Z

PR_Github #61959 [ run ] triggered by Bot. Commit: cdd5594 Link to invocation

tensorrt-cicd · 2026-07-27T21:25:27Z

PR_Github #61959 [ run ] completed with state FAILURE. Commit: cdd5594
/LLM/main/L0_MergeRequest_PR pipeline #50148 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-07-28T01:37:17Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-28T01:43:06Z

PR_Github #62054 [ run ] triggered by Bot. Commit: d97d6c7 Link to invocation

tensorrt-cicd · 2026-07-28T05:56:24Z

PR_Github #62054 [ run ] completed with state SUCCESS. Commit: d97d6c7
/LLM/main/L0_MergeRequest_PR pipeline #50237 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-28T16:23:09Z

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-4"

tensorrt-cicd · 2026-07-28T16:29:07Z

PR_Github #62234 [ run ] triggered by Bot. Commit: d97d6c7 Link to invocation

tensorrt-cicd · 2026-07-28T17:39:24Z

PR_Github #62234 [ run ] completed with state SUCCESS. Commit: d97d6c7
/LLM/main/L0_MergeRequest_PR pipeline #50403 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

chienchunhung · 2026-07-28T19:37:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-28T19:44:00Z

PR_Github #62279 [ run ] triggered by Bot. Commit: d97d6c7 Link to invocation

tensorrt-cicd · 2026-07-28T22:54:11Z

PR_Github #62279 [ run ] completed with state FAILURE. Commit: d97d6c7
/LLM/main/L0_MergeRequest_PR pipeline #50447 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-29T00:16:12Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T00:22:59Z

PR_Github #62344 [ run ] triggered by Bot. Commit: d97d6c7 Link to invocation

tensorrt-cicd · 2026-07-29T02:48:48Z

PR_Github #62344 [ run ] completed with state FAILURE. Commit: d97d6c7
/LLM/main/L0_MergeRequest_PR pipeline #50511 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-29T19:31:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T19:39:02Z

PR_Github #62586 [ run ] triggered by Bot. Commit: d97d6c7 Link to invocation

tensorrt-cicd · 2026-07-29T22:07:45Z

PR_Github #62586 [ run ] completed with state SUCCESS. Commit: d97d6c7
/LLM/main/L0_MergeRequest_PR pipeline #50728 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

github-actions Bot assigned chienchunhung Jul 15, 2026

chienchunhung added the api-compatible Accepted LLM API contract change that is backwards-compatible label Jul 15, 2026

chienchunhung mentioned this pull request Jul 15, 2026

[https://nvbugs/6448152][perf] TEST ONLY; DO NOT REVIEW: diagnose local CTX completion #16449

Closed

chienchunhung force-pushed the codex/nvbug6312828-proxy-deadline branch from 92bfff6 to f58de71 Compare July 17, 2026 02:48

NVIDIA deleted a comment from tensorrt-cicd Jul 17, 2026

chienchunhung force-pushed the codex/nvbug6312828-proxy-deadline branch from 0c4771a to cdd5594 Compare July 27, 2026 17:25

chienchunhung added 9 commits July 27, 2026 18:33

[NVBUG 6312828][fix] Terminalize disaggregated request timeouts

5025449

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Bound post-stress recovery concurrency

f9eef2f

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Gate post-stress recovery readiness

624a825

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Reap idle context transfers safely

5121cb9

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Coordinate default-off transfer cleanup

af22afc

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Reject replayed KV handshakes

1e4b9c5

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Correct transfer cleanup assertions

db4df3e

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Repair focused CI regressions

49f0ac1

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Bound pre-handshake cancellation state

d97d6c7

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/nvbug6312828-proxy-deadline branch from cdd5594 to d97d6c7 Compare July 28, 2026 01:36

Uh oh!

Conversation

chienchunhung commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Breakdown

Fix Scope

Validation

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

chienchunhung commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

chienchunhung commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

chienchunhung commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 26, 2026

Uh oh!

chienchunhung commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

chienchunhung commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

chienchunhung commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

chienchunhung commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

chienchunhung commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

chienchunhung commented Jul 15, 2026 •

edited

Loading