Skip to content

[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: benchmark combined Python asynchronous agreement - #16651

Closed
chienchunhung wants to merge 6 commits into
NVIDIA:mainfrom
chienchunhung:codex/python-consensus-gf-both
Closed

[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: benchmark combined Python asynchronous agreement#16651
chienchunhung wants to merge 6 commits into
NVIDIA:mainfrom
chienchunhung:codex/python-consensus-gf-both

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Purpose

TEST ONLY treatment for the generation-first Python-transceiver consensus performance experiment. The branch is exactly one signed YAML-only child of corrected harness base da6e140bda4c831bc813b3b192d7cb98a37025b2. That base is one harness-only child of the official asynchronous Python transceiver draft at product head 0f9acdd4dddbde7ccbf31973ce420bf4da6cb28a: it only passes the allowlisted server_config_extra.schedule_style value into the generated runtime config and adds focused tests. It does not change product behavior.

The previous run is not generation-first evidence: although its YAML requested generation_first, the old harness omitted that key from the generated server config and the runtime used context_first. This corrected head requires both the generated config and runtime log to confirm generation_first.

Experiment

Setting Value
Schedule generation_first
Async terminal agreement on
Async readiness agreement on
Context transceiver Python
Generation transceiver Python
Transport NIXL

Both agreement flags are enabled only on the four CTX PP ranks. The GEN workers use the Python transceiver but receive no CTX-only agreement flags, preserving the qualified TP1/CP1/non-ADP/PP-greater-than-one participant domain.

The run must use the exact GB300 disaggregated perf-sanity selector, verify both agreement protocols and the exact PP activation schedule, complete every submitted request, and publish an official output-token-throughput metric. Any failed request censors the throughput result.

This diagnostic is not intended to merge.

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@chienchunhung
chienchunhung force-pushed the codex/python-consensus-gf-both branch from b71dd1c to b6c413f Compare July 21, 2026 00:30
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60583 [ run ] triggered by Bot. Commit: b6c413f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60583 [ run ] completed with state FAILURE. Commit: b6c413f
/LLM/main/L0_MergeRequest_PR pipeline #48893 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chienchunhung chienchunhung changed the title [NVBUG-6448152][test] benchmark combined Python asynchronous agreement [NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: benchmark combined Python asynchronous agreement Jul 21, 2026
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
chienchunhung force-pushed the codex/python-consensus-gf-both branch from b6c413f to 00d416c Compare July 21, 2026 23:33
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60822 [ run ] triggered by Bot. Commit: 00d416c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60822 [ run ] completed with state FAILURE. Commit: 00d416c
/LLM/main/L0_MergeRequest_PR pipeline #49097 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
…tment

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
chienchunhung force-pushed the codex/python-consensus-gf-both branch from 00d416c to 4e31c4b Compare July 22, 2026 03:38

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60875 [ run ] triggered by Bot. Commit: 4e31c4b Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60875 [ run ] completed with state FAILURE. Commit: 4e31c4b
/LLM/main/L0_MergeRequest_PR pipeline #49147 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants