[None][feat] coordinate disaggregated request lifecycle

[chienchunhung]

Summary

• add a generation-safe v1 lifecycle protocol for the Python-native disaggregated transceiver, with exact request, artifact, grant, session, endpoint-incarnation, and transfer-operation identities;
• coordinate CTX and GEN with bounded, renewable generation-admission grants and context-artifact obligations for both context-first and generation-first scheduling;
• protect KV and auxiliary allocations with allocator-issued generation leases, explicit transfer-submission state, terminal-result replay and acknowledgement, and fail-closed retirement;
• propagate cancellation, expiry, and handoff completion across serving, executor, scheduler, and transceiver boundaries while preserving qualified legacy behavior.

Motivation: NVBUG 6519709

NVBUG 6519709 shows that large attention-DP deployments can encounter many independent, partial per-rank KV-transfer failures during cross-rack operation. The synchronized request-ID union is working as designed; the lifecycle risk is coordinating terminal state and retaining ownership until physical transfer quiescence after such failures. This PR does not eliminate the underlying NIXL/IB transfer errors, but its Python-transceiver v1 path makes their handling fail-closed through exact-attempt fencing, coordinated CTX/GEN cancellation, allocation leases, and acknowledged terminal outcomes.

Transceiver support and enablement

Lifecycle protocol v1 is opt-in in this PR. A v1 handoff requires both CTX and GEN to advertise the complete v1 contract and use the Python transceiver; mixed Python/C++ v1 handoffs are rejected before destination-address publication.

	Python transceiver	C++ transceiver
Runtime selection	transceiver_runtime: PYTHON	transceiver_runtime: CPP or unset/default
Transfer backends	NIXL, including DEFAULT resolution	Existing DEFAULT/NIXL/UCX/MOONCAKE/MPI paths
Default lifecycle	Qualified legacy v0	Qualified legacy v0
With TRTLLM_DISAGG_LIFECYCLE_PROTOCOL_VERSION=1	Full v1 when both endpoints and the allocator/topology advertise all required capabilities	Not qualified; v1 negotiation fails closed rather than silently weakening the contract
New behavior in this PR	Cross-side grants and artifact obligations, exact attempt/session/incarnation identities, publication and submission fencing, allocation leases, exact writer tracking, quiescence, and terminal replay	Existing v0 transfer behavior plus shared lifecycle result types, structured cancellation/shutdown adapters, allocation-generation lease primitives, and fail-closed ownership hardening; no cross-side lifecycle-v1 protocol

Unsupported v1 allocator or topology combinations also fail closed. In particular, MixedMambaHybridCacheManager is not yet qualified because its independent Mamba state slots do not provide allocation-generation leases.

Why lifecycle v1 is Python-first

The safety properties are end to end: every participant must implement exact identities, publication closure, future-submission fencing, and physical quiescence. Advertising only part of that contract on C++ would be worse than retaining an explicit v0 path because one weak endpoint can invalidate the memory-reuse guarantee for the whole handoff.

The Python-native transceiver already owns the control and wire state needed to implement exact session identities, writer accounting, replay, and endpoint-incarnation fencing. The C++ transceiver spans several transport plugins and topology modes, and each requires a backend-specific, testable definition of submission fencing and quiescence before it can honestly advertise v1. The serving stack is also actively moving toward the Python transceiver as the default direction, making it the practical first complete implementation.

This PR therefore lands one complete, fail-closed Python v1 slice while adding C++ lifecycle types, allocator primitives, and legacy hardening as foundations. Full C++ v1 support is a follow-up implementation and qualification effort, not a version-bit change.

Design overview

The coordinator owns placement and cross-side obligations, while endpoint-local schedulers and allocators remain the authority for capacity and memory reuse. A GEN grant admits one exact attempt; CTX retains the corresponding artifact until the grant is committed, abandoned, or expires. Transfer operations are fenced by endpoint incarnation and allocation generation, and logical request completion is kept separate from physical transport quiescence so cancellation or timeout cannot authorize premature memory reuse.

Lifecycle v1 is negotiated explicitly. Unsupported v1 topology/backend combinations fail closed, while identity-free v0 traffic keeps the existing compatibility path. Terminal delivery uses exact replay plus ACK/confirmation, with explicit endpoint-incarnation or global fences as the escape hatch for a disappeared peer; lightweight control replay does not retain drained KV mappings or transfer agents. The C++ transceiver remains on its qualified v0 contract in this change.

Python lifecycle-v1 request workflows

Context-first

sequenceDiagram
    participant F as Frontend
    participant O as Coordinator
    participant C as CTX
    participant G as GEN

    F->>O: logical request
    O->>C: prefill
    C->>C: compute and lease artifact
    C-->>O: ARTIFACT_READY
    O->>G: admission request
    alt GEN rejects
        G-->>O: GEN_REJECT
        O->>G: try another GEN
    else GEN accepts
        G-->>O: GEN_INTENT_GRANT
        G->>C: ARTIFACT_LEASE_RENEW
        G->>G: allocate destination and acquire lease
        G-->>C: RECEIVER_READY
        C->>G: transfer
        G->>G: validate all receive obligations
        G-->>O: HANDOFF_COMMITTED
        G-->>C: artifact obligation complete
        C->>C: release after source quiescence
    end

Loading

Generation-first

sequenceDiagram
    participant F as Frontend
    participant O as Coordinator
    participant G as GEN
    participant C as CTX

    F->>O: logical request
    O->>G: intent admission request
    G-->>O: GEN_INTENT_GRANT
    O->>C: prefill under grant
    C->>C: compute and lease artifact
    C-->>G: ARTIFACT_READY
    G->>C: ARTIFACT_LEASE_RENEW
    G->>G: allocate destination and acquire lease
    G-->>C: RECEIVER_READY
    C->>G: transfer
    G->>G: validate all receive obligations
    G-->>O: HANDOFF_COMMITTED
    G-->>C: artifact obligation complete

Loading

Relationship to #16396

This change carries forward and evolves the Python-native ownership containment implemented by #16396. That PR establishes strong endpoint-local ownership, exact writer accounting, bounce-slot retention, and fail-closed teardown on the existing protocol; this PR adds the generation-safe protocol, coordinated CTX/GEN obligations, admission control, endpoint incarnation fencing, allocator leases, and acknowledged terminal replay.

It is therefore a runtime/behavioral superset of #16396's supported Python-native scope and is intended to supersede it. It is not a byte-for-byte append-only stack because several containment patches are adapted to the new identities and retirement rules. Neither PR implements lifecycle v1 for the C++ transceiver.

Change size

Classified by path: tests/** and cpp/tests/** are test code; all remaining changed C++ and Python paths are production code.

Area	Files	Additions	Deletions	Changed lines
Production code	55	24,512	1,920	26,432 (57.1%)
Test code	38	19,533	326	19,859 (42.9%)
Total	93	44,045	2,246	46,291

Validation

• changed-file Python bytecode compilation
• changed-file formatting, lint, spelling, and whitespace hooks
• focused lifecycle, cancellation, replay, admission, allocation-lease, and handoff unit coverage added

Full unit/integration CI remains pending on this draft. Focused pytest collection is unavailable in the local environment because transformers is not installed.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62001 [ run ] triggered by Bot. Commit: cbe3861 Link to invocation

[tensorrt-cicd]

PR_Github #62001 [ run ] completed with state FAILURE. Commit: cbe3861
/LLM/main/L0_MergeRequest_PR pipeline #50188 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62095 [ run ] triggered by Bot. Commit: 31f0eb1 Link to invocation

[tensorrt-cicd]

PR_Github #62095 [ run ] completed with state SUCCESS. Commit: 31f0eb1
/LLM/main/L0_MergeRequest_PR pipeline #50278 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #62276 [ run ] triggered by Bot. Commit: 13aa5b7 Link to invocation

[tensorrt-cicd]

PR_Github #62276 [ run ] completed with state FAILURE. Commit: 13aa5b7
/LLM/main/L0_MergeRequest_PR pipeline #50445 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation