[NVBUG-6448152][perf] remove synchronous Python PP transfer rendezvous

[chienchunhung]

Summary

Remove synchronous pipeline-parallel rendezvous from the qualified Python NIXL transceiver path while preserving one authoritative cross-rank decision for every request transition.

This is the production merge vehicle for NVBUG-6448152. It was rebuilt from the validated production subset of the TEST ONLY Python validation draft. Performance harnesses and diagnostic instrumentation remain excluded.

The branch contains nine logical DCO-signed commits covering:

1. Drain-safe native transfer cancellation and lifetime handling.
2. A nonblocking cross-rank agreement engine.
3. Python transceiver integration without synchronous PP rendezvous.
4. Final-runtime endpoint ownership and bounded progress.
5. Peer-ready generation-first activation.
6. Fail-closed ownership and bounded metadata-admission hardening.
7. Permanent focused tests, H100 test-list entries, and deterministic fixture corrections.
8. The two production prerequisites discovered by the generation-first E2E: an independently configurable metadata-only admission window and idempotent readiness preparation.
9. Receiver-owned bounce planning with lazy, bounce-only rank-0 peer registration and fail-open fallback before reservation.

Problem

The existing Python path uses blocking collectives to reconcile rank-local readiness and terminal outcomes. Pipeline stages reach those calls at different times, so ordinary PP arrival skew becomes serialized waiting on the hot path. Publishing a local terminal result is also unsafe until native writes, callbacks, bounce scatters, and cancellation obligations can no longer touch request memory.

Generation-first validation exposed two additional defects in the production boundary:

• Metadata-only CTX admission was coupled to compute activation capacity. On the validated PP4, max_batch_size=1 topology that exposed only four of 256 independently ordered requests for peer-metadata association, so a complete four-rank cohort was rarely formed.
• The executor intentionally calls prepare_context_requests during admission and again while polling schedulability. A READY event could move the same request from waiting to prepared between those calls; the second call then repinned and reinserted it as a new waiter. After epoch 0 completed, that stale waiter could publish epoch 1 for the already-active request and fail the authoritative schedule invariant.

Generation-first startup had also exposed a separate endpoint-lifetime issue. Executor creation constructs an estimation transceiver before the final serving transceiver so KV capacity accounts for transfer registration and bounce memory. The temporary executor could nevertheless publish its endpoint through cached server metadata, leaving the router with a retired endpoint.

Full CI then exposed a receiver-owned bounce-planning issue: first-contact metadata was published GEN-to-CTX, but the receiving rank needed receiver-local CTX rank-0 metadata to derive the exact coalesced destination plan. The missing entry caused rank-0 lookup failure and fail-open per-fragment fallback.

Design

• Negotiate a native transfer protocol with request incarnation, endpoint, rank, slice, and operation identity.
• Seal sessions before observing a terminal outcome and retain KV, AUX, and bounce ownership until exact drain acknowledgement.
• Validate each bounce writer against an exact receiver-derived, rank-bound destination plan before scatter.
• For bounce-enabled first contact, lazily register receiver-local rank-0 peer metadata inside the existing fail-open planning path; registration failure or missing higher-rank metadata falls back before any reservation. Non-bounce behavior is unchanged.
• Use point-to-point messages on a duplicated MPI communicator for ordered global agreement; no per-request collective is added.
• Reduce immutable terminal votes to one coordinator-authored commit.
• Treat generation-first readiness as a separate safety state machine: vote, prepare, irrevocable lease acknowledgement, scheduler release, activation by the exact propagated PP schedule, and acknowledged completion.
• Activate ready requests from the canonical peer-ready cohort and materialize the same activation decision on every PP follower.
• Bound metadata-only pre-active ownership independently from compute activation through CacheTransceiverConfig.max_num_generation_first_pre_active_requests. None preserves the compute-cap default; deployments with independently ordered high-concurrency CTX/GEN traffic can select a larger bounded window. Compute activation remains capped by max_num_active_requests.
• Make readiness preparation idempotent across waiting, prepared, activated, and aborted ownership. A duplicate ID owned by another object or by multiple lifecycle states fails closed.
• Track transceiver and connector transfer legs independently so cancellation retires only the provider that owns work.
• Preserve the estimation transceiver and its memory accounting, but prevent it from publishing serving metadata.
• Publish a per-lifetime endpoint generation only from the final serving transceiver and explicitly publish final-executor disaggregated metadata.
• Coalesce bounded endpoint refreshes without holding the router lock, validate endpoint ownership, and release coordinator reservations on refresh failure or cancellation.
• Replace idle busy-polling with an interruptible backoff capped at 10 ms.
• Fail closed on ambiguous partial sends, protocol violations, stale endpoint ownership, missing peer readiness, unsafe graph teardown, or unsafe transfer teardown.

Readiness is a separate factor from terminal agreement: a readiness vote is withdrawable until PREPARE and becomes active only through the scheduler, whereas a terminal vote is immutable and authorizes resource reclamation.

Initial scope

The asynchronous protocol is explicit opt-in and default off. Its qualified domain is:

• Python cache transceiver
• NIXL backend
• MPI distributed runtime
• TP1, CP1, non-ADP
• PP greater than one, with the PP participant domain equal to the worker world

Unsupported or flag-off configurations retain their existing runtime behavior. A mismatched explicit opt-in fails during startup negotiation before request traffic.

Validation

The instrumented generation-first E2E evidence is valid:

• Exact Python/NIXL CTX and GEN runtimes, generation-first scheduling, PP4 CTX, and a bounded metadata window of 256.
• 256/256 HTTP 200 responses, zero failed requests, clean application shutdown, and b_is_valid=true.
• 454.92 output tok/s, 7,733.63 total tok/s, and 2,097,152 output tokens over 4,609.94 seconds.
• All four CTX PP ranks activated 256 requests with the identical activation digest; READY vote, prepare, activate, and complete each reached 256 on every rank.

This proves functional generation-first PP consensus for the integrated candidate. It does not demonstrate a raw throughput improvement: 454.92 output tok/s is 1.885% below the unmatched historical 463.66 context-first result, which used two rounds and no asynchronous flags or instrumentation. A matched same-head A/B is still required for a throughput claim.

Client completion is independent of the final CTX cleanup tail. Strict terminal consensus reached 251 commits on every rank before harness teardown; four remaining rounds lacked rank 3's nonblocking quiescence vote and one lacked all votes. There was no shutdown failure, but this description does not claim 256/256 terminal commits. A pre-teardown drain/assertion remains required for that stronger proof.

Prior exact-head focused CI evidence for the production boundary includes 9,583 passed, 8,557 skipped, and zero test failures across the allocated targeted stages. The real four-rank MPI agreement test passed. Two H100 shards repeatedly failed before allocation or pytest because the requested Slurm node configuration was unavailable; that is missing coverage, not a product failure.

The exact-head receiver-bounce and executor-lifetime proof is terminal green on the current production head:

• A100X: 65 passed, 3 skipped, zero failed; the current-executor-lifetime regression passed.
• GB200: 69 passed, 1 explicit skip, zero failed; the Python bounce E2E passed in 99.242 seconds.
• The bounce E2E explicitly requires the [kv-bounce] coalesced marker, all captured client responses were HTTP 200, and there was no exact-plan fallback, KeyError, or traceback.

For the current production head:

• All changed Python files compile under Python 3.13.
• All configured changed-file hooks, Ruff/YAPF formatting, YAML/test-list AST validation, pinned-memory policy, DCO, and diff checks pass.
• Iterative code-owner-style self-review fixed the discovered issues; a fresh independent full P0-P3 pass is clean.
• Permanent regressions cover bounded/backlogged metadata admission, a 256-request metadata window without expanded compute activation, cancellation overflow, canonical cross-rank cohorts, duplicate preparation through each lifecycle phase, exact object identity, multiple-owner fail-closed behavior, and continued consensus progress.

Remaining merge gates

Before this draft is marked ready:

• Complete full CI on the current exact head, including the affected executor, transceiver, transfer-quiescence, native-metadata, timing, cancellation, bounce, router, and endpoint-publication coverage.
• Obtain strict pre-teardown terminal-drain evidence if 256/256 terminal commits are required as a merge criterion.
• Merge order with the overlapping serving-lifecycle draft is resolved: this focused production draft merges first, then the lifecycle draft rebases and combines—rather than overwrites—the endpoint-refresh, reservation-release, cancellation-owner, bounce, bounded-polling, and endpoint-fixture assertions.
• Run a matched same-head synchronous-versus-asynchronous experiment before claiming output-token-throughput improvement.

This pull request remains a draft until those gates are complete.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #61988 [ run ] triggered by Bot. Commit: bff176b Link to invocation

[tensorrt-cicd]

PR_Github #61988 [ run ] completed with state SUCCESS. Commit: bff176b
/LLM/main/L0_MergeRequest_PR pipeline #50177 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-PyTorch-2,A10-PyTorch-3,DGX_H100-PyTorch-1,DGX_H100-PyTorch-2,DGX_H100-PyTorch-3,DGX_H100-PyTorch-4,DGX_H100-PyTorch-5,DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #62040 [ run ] triggered by Bot. Commit: e31fca7 Link to invocation

[tensorrt-cicd]

PR_Github #62040 [ run ] completed with state SUCCESS. Commit: e31fca7
/LLM/main/L0_MergeRequest_PR pipeline #50225 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-PyTorch-1,A10-PyTorch-2,A10-PyTorch-3,A100X-PyTorch-Post-Merge-1,DGX_H100-PyTorch-1,DGX_H100-PyTorch-2,DGX_H100-PyTorch-3,DGX_H100-PyTorch-4,DGX_H100-PyTorch-5,DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #62267 [ run ] triggered by Bot. Commit: da03373 Link to invocation

[tensorrt-cicd]

PR_Github #62267 [ run ] completed with state FAILURE. Commit: da03373
/LLM/main/L0_MergeRequest_PR pipeline #50435 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-4,DGX_H100-PyTorch-5,DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #62332 [ run ] triggered by Bot. Commit: da03373 Link to invocation

[tensorrt-cicd]

PR_Github #62332 [ run ] completed with state FAILURE. Commit: da03373
/LLM/main/L0_MergeRequest_PR pipeline #50500 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62653 [ run ] triggered by Bot. Commit: db61448 Link to invocation

[tensorrt-cicd]

PR_Github #62653 [ run ] completed with state FAILURE. Commit: db61448
/LLM/main/L0_MergeRequest_PR pipeline #50792 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-1,A100X-PyTorch-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62810 [ run ] triggered by Bot. Commit: 61be245 Link to invocation

[tensorrt-cicd]

PR_Github #62810 [ run ] completed with state SUCCESS. Commit: 61be245
/LLM/main/L0_MergeRequest_PR pipeline #50935 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

Focused validation at the current production head

The exact focused trigger is terminal and green at head 61be245602937e6ed4c90f33fef0558df55ee149. It ran only the requested GB200 and A100X stages with fail-fast disabled.

• A100X: 65 passed, 3 skipped, 0 failed. test_disaggregated_params_comes_from_current_executor_lifetime passed, closing the previously omitted executor-lifetime coverage.
• GB200: 69 passed, 1 explicit skip, 0 failed. The exact Python bounce E2E passed in 99.242 seconds.
• The bounce E2E requires the [kv-bounce] coalesced marker, so its pass proves receiver-owned coalesced planning executed; silent per-fragment fallback cannot satisfy the test.
• All captured completion/chat client requests returned HTTP 200. No KeyError, traceback, or exact-plan fallback marker appeared, and teardown completed as expected.

This validates the receiver-side rank-0 peer-registration correction together with the productionized generation-first metadata-window and idempotent readiness-lifecycle fixes. The final full CI merge gate and the retained-assertion audit against the serving-lifecycle draft remain pending; no additional focused rerun is needed.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62828 [ run ] triggered by Bot. Commit: 61be245 Link to invocation

[tensorrt-cicd]

PR_Github #62828 [ run ] completed with state FAILURE. Commit: 61be245
/LLM/main/L0_MergeRequest_PR pipeline #50953 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63083 [ run ] triggered by Bot. Commit: c55d778 Link to invocation

[tensorrt-cicd]

PR_Github #63083 [ run ] completed with state FAILURE. Commit: c55d778
/LLM/main/L0_MergeRequest_PR pipeline #51175 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation