[NVBUG-6448152][test] Improve consensus efficiency in Python transceiver (DO NOT REVIEW YET)

[chienchunhung]

Purpose

TEST ONLY — DO NOT REVIEW OR MERGE.

This draft runs a matched Python/NIXL throughput A/B for the terminal pipeline-parallel rendezvous derived from the production implementation in PR #16645.

Exact frozen candidate head: fb3981ffa4d8e3e27f3a3fdc586198f359d4c5c2 (tree 976b1ae763b8111faebe2ace6b92c7a806b3883d). Its parent is the preserved failed experiment head a837ed1f49c7c154746b3c2f8347fedfcb957704. This TEST ONLY draft must never merge.

The first failed paired head and the second invalid split head are preserved on backup branches. Their sanitized evidence is public:

• first paired attempt: transfer-timeout/lifecycle failure before any metric;
• second split attempt: both stages actually ran the blocking terminal mode.

Why the experiment was repaired

The earlier valid async-only run completed 256/256 requests at 454.92 output tok/s, but it had no matched synchronous control and therefore proved functionality rather than throughput uplift.

The first paired attempt used sequential arms in one allocation. Two requests hit the 600-second KV-transfer timeout, followed by backend failure and unbounded cleanup. One allocation also cannot safely contain both this roughly 77-minute workload and a potentially slower control under the four-hour allocation limit.

The second attempt split the arms into independent allocations, but it found strict harness defects before a causal comparison could be accepted: the nested arm environment did not reach outer CTX ranks, the lifecycle barrier counted GPU workers rather than outer pytest controllers, cumulative CTX evidence mixed profiling and measured runtimes, and allocation-specific request IDs made cross-stage digests incomparable. Both raw workloads completed, but both ran terminal=0; their raw throughput values are intentionally rejected.

Matched split design

One exact-head CI invocation requests two stage suffixes. Each stage uses a separate fresh three-node allocation built from the same downstream image:

1. Post-Merge-1: asynchronous terminal agreement (terminal=1, peer-ready=1).
2. Post-Merge-2: legacy blocking terminal agreement (terminal=0, peer-ready=1).

The two arms hold constant the exact source head, rendered image, model, selector, topology, generation_first, Python/NIXL transfer, CTX PP4, GEN DEP8, metadata capacity 256, compute cap 4, 256 requests at concurrency 256, one round, dataset order, and instrumentation. Only terminal agreement changes.

The repaired harness now:

• propagates the singleton arm environment to every outer CTX worker only for the e2e experiment;
• asserts the configured mode on CTX ranks 0–3 before launching clients;
• waits for the exact four outer controllers at teardown/evidence barriers;
• namespaces status, abort, hostname, generated-config, log-offset, and barrier coordination by one launch token so requeues cannot consume stale files;
• scopes cumulative CTX evidence by a byte offset and the first complete measured-runtime mode set after profiling shutdown;
• hashes original prompt-token content rather than runtime request IDs and pins sha256-length-prefixed-prompt-sha256-v1 in evidence validation.

Each arm retains bounded client, KV-transfer, coordination, and test deadlines sized above the observed healthy workload duration. Atomic failure propagation and bounded terminate/kill handling prevent a backend failure from leaving clients or barriers alive.

Acceptance criteria

Each arm must independently provide:

• 256/256 successful requests, zero failures, and a valid throughput record;
• expected terminal and peer-ready modes on CTX ranks 0–3 before client launch and at teardown;
• exactly 256 prompt-content activations with one identical pinned-algorithm digest across CTX ranks;
• clean CTX, GEN, and disaggregated-service shutdown with no timeout, protocol, or cancellation failure.

The asynchronous arm must additionally report 256 terminal commits on every CTX rank. Because the arms are separate stages, stage success alone is insufficient: the two prompt-content activation digests must be compared across the published stage artifacts before accepting the throughput delta. Missing or unequal evidence censors the comparison.

This experiment measures only the terminal-rendezvous effect. Because the arms use separate allocations, a single-run delta below roughly 3–5% is treated as inconclusive allocation noise.

Preflight

• Python 3.13 compilation and shell syntax checks passed.
• 49 focused submission, lifecycle, evidence, and fail-fast tests passed.
• Test-list uniqueness and AST validation passed for 2,057 entries.
• Configured changed-file hooks and DCO passed.
• Replay against the failed artifact selects the final runtime exactly and rejects the falsely labeled asynchronous arm before client launch.
• Three independent reviews found no code blocker or workload/configuration confound. The explicit external cross-artifact digest gate above remains mandatory.

No replacement CI trigger will be posted until the automatic PR gates pass on the exact frozen head.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61137 [ run ] triggered by Bot. Commit: 0453b6d Link to invocation

[tensorrt-cicd]

PR_Github #61137 [ run ] completed with state SUCCESS. Commit: 0453b6d
/LLM/main/L0_MergeRequest_PR pipeline #49389 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61190 [ run ] triggered by Bot. Commit: 9f71926 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61363 [ run ] triggered by Bot. Commit: e026195 Link to invocation

[tensorrt-cicd]

PR_Github/16766-9f71926 #61190 was force-killed by a newer pipeline run.
L0 job information not available (job may not have been triggered yet).

Link to superseding invocation

[tensorrt-cicd]

PR_Github #61363 [ run ] completed with state FAILURE. Commit: e026195
/LLM/main/L0_MergeRequest_PR pipeline #49591 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5, DGX_H100-PyTorch-6"

[tensorrt-cicd]

PR_Github #61381 [ run ] triggered by Bot. Commit: a2928c4 Link to invocation

[tensorrt-cicd]

PR_Github #61381 [ run ] completed with state SUCCESS. Commit: a2928c4
/LLM/main/L0_MergeRequest_PR pipeline #49607 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-2, DGX_H100-PyTorch-3, DGX_H100-PyTorch-4, DGX_H100-PyTorch-5"

[tensorrt-cicd]

PR_Github #61628 [ run ] triggered by Bot. Commit: 1301fb8 Link to invocation

[tensorrt-cicd]

PR_Github #61628 [ run ] completed with state SUCCESS. Commit: 1301fb8
/LLM/main/L0_MergeRequest_PR pipeline #49834 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-1, DGX_H100-PyTorch-4"

[tensorrt-cicd]

PR_Github #61655 [ run ] triggered by Bot. Commit: 95911b7 Link to invocation

[tensorrt-cicd]

PR_Github #61655 [ run ] completed with state SUCCESS. Commit: 95911b7
/LLM/main/L0_MergeRequest_PR pipeline #49861 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #61671 [ run ] triggered by Bot. Commit: 7005411 Link to invocation

[tensorrt-cicd]

PR_Github #61671 [ run ] completed with state FAILURE. Commit: 7005411
/LLM/main/L0_MergeRequest_PR pipeline #49877 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #61674 [ run ] triggered by Bot. Commit: 84ea0ba Link to invocation

[tensorrt-cicd]

PR_Github #61674 [ run ] completed with state FAILURE. Commit: 84ea0ba
/LLM/main/L0_MergeRequest_PR pipeline #49880 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62347 [ run ] triggered by Bot. Commit: 25c79db Link to invocation

[tensorrt-cicd]

PR_Github #62347 [ run ] completed with state FAILURE. Commit: 25c79db
/LLM/main/L0_MergeRequest_PR pipeline #50514 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62609 [ run ] triggered by Bot. Commit: e4456db Link to invocation

[tensorrt-cicd]

PR_Github #62609 [ run ] completed with state SUCCESS. Commit: e4456db
/LLM/main/L0_MergeRequest_PR pipeline #50749 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

Terminal generation-first Python throughput evidence

The exact targeted run for head e4456db1938ec6e77753ec68fd612573bb2a9bb3 is terminal and passed.

Frozen configuration:

• generation_first scheduling;
• Python/NIXL transceiver on both context and generation workers;
• asynchronous peer-ready and terminal consensus enabled;
• metadata capacity 256 with compute activation capacity four;
• 256 requests at concurrency 256, one round;
• context PP4 and generation DEP8 on three nodes/twelve GPUs;
• exact requested GB300 stage and selector.

Results:

• 256/256 successful requests and zero failed requests;
• 256 HTTP 200 completions at the context, generation, and disaggregated frontends;
• official performance record valid: b_is_valid=true;
• output-token throughput: 454.92 tok/s;
• total-token throughput: 7733.63 tok/s;
• request throughput: 0.06 req/s;
• benchmark duration: 4609.94 s;
• all four context PP ranks completed 256 READY rounds and activated 256 requests in the same order;
• all four ranks emitted activation count 256 with identical SHA-256 digest 34b93aa3a3e26cf9a1863b158a0d5c81d722d28a05ba327bf8f8d69488e30c31;
• context, generation, and disaggregated services shut down cleanly with no protocol failure, request timeout, cancellation, or local send failure.

Interpretation and remaining caveats

This proves that the integrated generation-first Python-transceiver candidate completes the workload end to end. The result is 8.74 tok/s (1.89%) below the historical 463.66 tok/s context-first result, so it demonstrates restored generation-first functionality at approximately the historical Python throughput magnitude; it is not evidence of a raw throughput improvement over that unmatched context-first run.

The stronger per-request terminal-agreement criterion was not fully observed before harness teardown. READY completed 256/256 on every context rank, but the shutdown summaries contained 251/256 terminal commits. Four tail rounds lacked rank three's final terminal vote and one lacked all final votes. All client requests had already completed successfully, and consensus/service shutdown was clean, so this does not invalidate the official client-throughput measurement. It does mean this run must not be described as proving 256 terminal commits. A pre-teardown drain/assertion is needed for that stronger evidence.

The validated head contains two behavioral corrections after the production-equivalent boundary that are not yet present in the official Python production draft:

• 25c79db90df2756f2e83546ce69b7eb876c7def7: separate the generation-first metadata-admission window from the compute-activation cap and set the experiment's metadata capacity to 256;
• e4456db1938ec6e77753ec68fd612573bb2a9bb3: make peer-readiness preparation idempotent across the waiting/prepared/activated/aborted lifecycle.

Those changes require productionization and review before this evidence can be attributed to the official production draft alone.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62847 [ run ] triggered by Bot. Commit: 53574c9 Link to invocation

[tensorrt-cicd]

PR_Github #62847 [ run ] completed with state FAILURE. Commit: 53574c9
/LLM/main/L0_MergeRequest_PR pipeline #50970 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Paired terminal-consensus A/B — censored run

The exact paired A/B trigger for head 53574c9f04b2e14525a4301d0b851eeb98d55cf8 is terminal, but it produced no valid throughput comparison.

Observed evidence:

• only the asynchronous arm started (terminal=1, peer_ready=1 on context ranks 0–3);
• the synchronous control arm never started;
• no benchmark summary, b_is_valid record, output-token throughput, 256/256 completion, 256-activation digest, or terminal_commit=256 evidence was produced;
• two generation requests first exceeded the configured 600-second KV-transfer timeout;
• the context executor later failed during response serialization with RuntimeError: bad optional access, and 35 pending requests received the propagated failure;
• asynchronous-consensus shutdown then failed to quiesce; the later allocation time limit was cleanup fallout from the already-failed workload, not the initiating failure.

Interpretation: this run is censored by a real transfer-timeout/lifecycle failure plus a fail-fast gap. It says nothing about the performance effect of asynchronous versus blocking terminal PP agreement.

The unchanged head will not be rerun. The next attempt must first correct the experiment's transfer-timeout contract for a roughly 77-minute, concurrency-256 generation-first arm and retain strict fail-closed evidence requirements, then pass review before one new exact-stage trigger.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

[tensorrt-cicd]

PR_Github #63019 [ run ] triggered by Bot. Commit: a837ed1 Link to invocation

[tensorrt-cicd]

PR_Github #63019 [ run ] completed with state FAILURE. Commit: a837ed1
/LLM/main/L0_MergeRequest_PR pipeline #51126 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Split paired terminal-consensus A/B — invalid run

The exact split paired-A/B trigger for head a837ed1f49c7c154746b3c2f8347fedfcb957704 is terminal, but it produced no valid asynchronous-versus-blocking throughput comparison.

Both raw workloads completed 256/256 requests with zero client failures:

• stage labeled asynchronous: 721.16 output tok/s and 12,259.77 total tok/s;
• blocking-control stage: 697.84 output tok/s and 11,863.32 total tok/s.

Those numbers must not be compared. Exact CTX evidence shows that both stages actually ran terminal=0, peer_ready=1 on ranks 0–3. The asynchronous arm-specific CTX environment was present in the nested experiment definition but was omitted from the outer multi-rank launch environment. Accordingly, the asynchronous stage emitted no terminal-commit evidence.

The run also exposed three evidence-harness defects after client completion:

• lifecycle barriers expected fourteen GPU ranks even though only four outer rank-zero pytest controllers participate (CTX_0, GEN_0, disaggregated server, and benchmark), so both stages timed out after completing their clients;
• cumulative CTX logs contained both memory-profiling and measured-runtime lifecycles without an exact final-runtime boundary;
• activation digests hashed allocation-specific runtime request IDs, so the differing stage digests were not cross-allocation evidence of different request order.

There is therefore no accepted throughput delta, no valid asynchronous arm, and no b_is_valid record from this run. The terminal bot report is a harness failure, not a product-performance conclusion.

The failed head is preserved on codex/backup-pr16766-pre-second-paired-ab-repair-20260731-a837ed1 and will never be rerun unchanged.

Replacement head fb3981ffa4d8e3e27f3a3fdc586198f359d4c5c2 repairs only the TEST ONLY experiment and evidence path: e2e-only CTX arm propagation, four-controller barriers, per-invocation coordination namespaces, byte-exact/final-runtime log scoping, a fail-before-client mode assertion, and prompt-content activation digests with a pinned algorithm. The workload configurations and selectors are unchanged. A new run will be requested only after the replacement head's automatic gates pass.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

[tensorrt-cicd]

PR_Github #63066 [ run ] triggered by Bot. Commit: fb3981f Link to invocation

[tensorrt-cicd]

PR_Github #63066 [ run ] completed with state FAILURE. Commit: fb3981f
/LLM/main/L0_MergeRequest_PR pipeline #51165 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Repaired terminal-consensus throughput A/B — censored

The exact repaired trigger is terminal. Both arms used their intended CTX modes and completed 256/256 client requests with zero HTTP failures:

• async terminal=1 / peer-ready=1: 710.72 output tok/s
• blocking terminal=0 / peer-ready=1: 710.44 output tok/s
• raw difference: +0.039%

That raw difference is not interpretable. Every async CTX rank recorded only 255/256 terminal votes and commits, so the strict evidence check failed. The activation count was 256 in both arms, but the prompt-content digests differed across the two allocations, and neither downloaded result archive contained the required b_is_valid evidence. The async services otherwise shut down cleanly; the later cancellation was coordinated cleanup after the evidence failure.

Therefore this run provides no valid terminal-rendezvous throughput conclusion. It will not be rerun unchanged. The diagnostic branch is preserved and this TEST ONLY draft is being closed unmerged. See the terminal CI report.