[https://nvbugs/6487040][test] Wait for gen-log end-of-write sentinel before parsing per-iter step time by chenfeiz0326 · Pull Request #16717 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-07-22T08:52:53Z

Description

The disagg gen_only perf-sanity metric
mean_gen_worker_per_iter_device_step_time was intermittently computed from a
truncated gen_server_{i}.log. That log is the gen srun's &> aggregate
of every TP rank; the fd is owned by the srun, so the file is only complete
once the srun has been reaped. The BENCHMARK srun was reading it while the
gen srun was still flushing across NFS/Lustre, and the previous settle-poll
heuristic (row count unchanged across two polls) could latch onto a mid-flush
prefix — silently averaging a partial run of iterations
(nvbugs 6487036 / 6487040).

This PR waits for a real end-of-write signal before parsing, so the read
always sees the fully-flushed log.

Constraints honored: no per-iteration step time is computed inside the
PyExecutor, the gen worker does not push per-iteration step time to the client,
and the metric parse + OpenSearch upload stay in the BENCHMARK srun (only that
srun owns the trtllm-benchmark.0.0.log parse + upload).

Changes:

jenkins/scripts/perf/disaggregated/slurm_launch_draft.sh — run each gen
srun in the foreground of a backgrounded subshell and touch gen_server_{i}.done immediately after the srun returns. Because the srun
owns the aggregate fd, the sentinel fires strictly after reap, i.e. after
the log is fully flushed. A stale sentinel from a re-used output dir is
removed first.
tests/integration/defs/perf/test_perf_sanity.py — the BENCHMARK
run_cmd branch defers the parse out of the client loop. It writes
benchmark_status in finally (which releases the gen workers so their
srun can exit and the sentinel can fire — this handshake is why the wait is
not circular), then blocks in wait_for_gen_log_sentinels() (bounded by
self.timeout; falls through on timeout so CI never hangs), then parses each
client log exactly once. The parse_gen_worker_device_step_time settle loop
is removed (settle_timeout / poll_interval kwargs dropped) since the
sentinel now guarantees completeness. The all-iter Welford fallback for
unbucketable num_generation_tokens (the PR [None][test] Restrict gen-worker per-iter mean to steady-state iterations #16298 None-metric regression) is
retained.
tests/integration/test_lists/waives.txt — un-waive the 20
disagg_upload-gen_only-* perf-sanity cases whose only failure mode was
this gen-log parse race, so the fix is re-validated end-to-end in CI. Scope is
deliberately limited to the disaggregated gen_only path the root cause
explains: aggregated (aggr_upload-*), disagg_upload-e2e, and
disagg_upload-ctx_only waivers are left untouched (aggregated serving
produces no gen_server_{i}.log and no sentinel, so the truncation race
cannot be what they were waived for). The 5 disagg_upload-gen_only cases
that still regress independent of the parse race (6478615, 6374872, and three
under the 6490049 umbrella) also stay waived.

Test Coverage

The 20 un-waived disagg_upload-gen_only-* perf/test_perf_sanity.py::test_e2e
cases run end-to-end in CI. They exercise the full disagg launch → gen-log
write → sentinel → BENCHMARK single-parse → check_test_failure path that this
change touches, which is the exact path that produced the intermittent failure.

PR Checklist

Please review the following before submitting your PR:

PR title follows the format [JIRA/NVBugs/GitHub ID][type] Summary
Changes are limited to a single logical scope
Any change to a public API / behavior is documented in the PR description
New/updated tests cover the change (the un-waived perf-sanity suite runs in pre-merge CI)
Commits are signed off (DCO)

coderabbitai · 2026-07-22T08:57:43Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

The gen-worker step-time parser now retains usable samples without parseable num_generation_tokens, falls back to an all-iteration mean when needed, and uses the updated aggregation during polling. New isolated CPU-only tests cover these paths. The MTP worker entry method is renamed to _forward_impl, and selected performance waivers are removed.

Changes

Gen-worker device step-time parsing

Layer / File(s)	Summary
Per-file scan fallback `tests/integration/defs/perf/test_perf_sanity.py`	Per-file scans retain generation-token buckets alongside all-iteration counts and means for usable `prev_device_step_time` rows.
Aggregation and parser integration `tests/integration/defs/perf/test_perf_sanity.py`	Aggregation selects the most-sampled generation-token bucket or falls back to the all-iteration mean, while polling consumes the updated scan structure.
Isolated parser regression tests `tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py`	Synthetic-log tests validate filtering, bucket selection, fallback formats, multi-worker averaging, empty results, and scan counts.

MTP worker entrypoint

Layer / File(s)	Summary
Worker method rename `tensorrt_llm/_torch/speculative/mtp_dynamic_tree.py`	`MTPEagleDynamicTreeWorker.forward(...)` is renamed to `_forward_impl(...)` with the same signature and body flow.

Performance waiver cleanup

Layer / File(s)	Summary
Performance waiver removals `tests/integration/test_lists/waives.txt`	The `disagg_upload-gen_only` waiver block is narrowed to three DeepSeek v32 32k4k entries, removing several other SKU-specific entries.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Parser
  participant GenWorkerLogs
  participant Scanner
  participant Aggregator
  Parser->>GenWorkerLogs: read gen_server_*.log files
  Parser->>Scanner: scan usable rows
  Scanner-->>Parser: return per-file scans and total count
  Parser->>Aggregator: select settled or timeout result
  Aggregator-->>Parser: return bucket mean or all-iteration fallback

Suggested reviewers: emmaqiaoch, brnguyen2, larryxfly, zhanruisunch, junyixu-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 93.75% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is specific, concise, and accurately describes the sentinel-based parsing fix.
Description check	✅ Passed	The description follows the template with Description, Test Coverage, and PR Checklist sections filled in.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py`:
- Around line 30-34: Update the isolated parser loader used by test_perf_sanity
to accurately document that it reads and executes a source slice rather than
using importlib. At the exec call, add a narrowly scoped Ruff S102 suppression
with justification that _TPS_PATH is a fixed, repository-local trusted path;
avoid broad lint disables or unrelated changes.
- Around line 214-229: Strengthen
test_scan_counts_all_usable_rows_regardless_of_ngen by assigning distinct
step-time values to the bucketed and fallback-only row groups, while preserving
their separate num_generation_tokens states. Assert _mean_at_mode_ngen() returns
the mean for the selected ngen bucket rather than all_mean, and update expected
aggregate values accordingly; complete the required coverage review for this
test change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2b12569b-ab7d-4b3f-8135-ad7d00ad5c39

📥 Commits

Reviewing files that changed from the base of the PR and between 9095cc1 and 5fb50f1.

📒 Files selected for processing (2)

tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py
tests/integration/defs/perf/test_perf_sanity.py

chenfeiz0326 · 2026-07-22T09:41:19Z

This is a [test]-only change to the perf-sanity harness parser
(parse_gen_worker_device_step_time) that runs inside every gen_only
disaggregated PerfSanity stage. To validate the fix against real CI, I picked 10
gen_only disagg cases spanning B200 / GB200 / GB300 and
deepseek-r1 / deepseek-v32 / gpt-oss-120b / qwen3-235b / glm-5 (including the
exact nvbugs/6487040 diagnostic case, gb200_deepseek-v32-fp4_8k1k_con1_...).

The 10 cases map to 5 post-merge Disagg-PerfSanity stages (these stages shard
1-test-per-shard by runtime duration, so all shards of each stage are listed to
guarantee the picked cases run):

DGX_B200-16_GPUs-2_Nodes-...-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge

disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con256_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL
disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL

GB200-8_GPUs-2_Nodes-...-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge
3. disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con512_ctx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL

GB200-8_GPUs-2_Nodes-...-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge
4. disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL
5. disagg_upload-gen_only-gb200_qwen3-235b-fp4_8k1k_con64_ctx1_tp1_gen1_tep4_eplb0_mtp0_ccb-NIXL

GB200-12_GPUs-3_Nodes-...-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge (bug's own family)
6. disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL ← nvbugs/6487040
7. disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL
8. disagg_upload-gen_only-gb200_deepseek-r1-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL

GB300-12_GPUs-3_Nodes-...-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge
9. disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL
10. disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL

/bot run --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2"

chenfeiz0326 · 2026-07-22T09:51:00Z

/bot run --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2"

tensorrt-cicd · 2026-07-22T09:57:30Z

PR_Github #60958 [ run ] triggered by Bot. Commit: abe8454 Link to invocation

tensorrt-cicd · 2026-07-22T10:26:56Z

PR_Github #60958 [ run ] completed with state FAILURE. Commit: abe8454
/LLM/main/L0_MergeRequest_PR pipeline #49221 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-07-22T14:11:24Z

/bot run --disable-fail-fast --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2"

tensorrt-cicd · 2026-07-22T14:17:56Z

PR_Github #60997 [ run ] triggered by Bot. Commit: 7a43ab8 Link to invocation

tensorrt-cicd · 2026-07-22T15:33:33Z

PR_Github #60997 [ run ] completed with state SUCCESS. Commit: 7a43ab8
/LLM/main/L0_MergeRequest_PR pipeline #49256 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

fredricz-20070104

Approve.

chenfeiz0326 · 2026-07-23T09:41:31Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-6,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-7,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-8"

tensorrt-cicd · 2026-07-23T09:47:00Z

PR_Github #61272 [ run ] triggered by Bot. Commit: a30c829 Link to invocation

tensorrt-cicd · 2026-07-23T11:22:04Z

PR_Github #61272 [ run ] completed with state FAILURE. Commit: a30c829
/LLM/main/L0_MergeRequest_PR pipeline #49507 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-07-23T11:49:49Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-8,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3"

tensorrt-cicd · 2026-07-23T11:55:53Z

PR_Github #61284 [ run ] triggered by Bot. Commit: 91be512 Link to invocation

tensorrt-cicd · 2026-07-23T12:31:56Z

PR_Github #61284 [ run ] completed with state FAILURE. Commit: 91be512
/LLM/main/L0_MergeRequest_PR pipeline #49518 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-07-24T04:38:34Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

chenfeiz0326 · 2026-07-28T09:48:15Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5"

tensorrt-cicd · 2026-07-28T09:54:39Z

PR_Github #62172 [ run ] triggered by Bot. Commit: b865b4f Link to invocation

tensorrt-cicd · 2026-07-28T11:12:46Z

PR_Github #62172 [ run ] completed with state SUCCESS. Commit: b865b4f
/LLM/main/L0_MergeRequest_PR pipeline #50346 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 · 2026-07-29T06:30:09Z

/bot run --disable-fail-fast --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-6,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-2,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE4-GPU16-Post-Merge-1,GB300-56_GPUs-14_Nodes-PyTorch-Disagg-PerfSanity-CTX12-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE8-GPU32-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-1,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge-1,GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX6-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1"

tensorrt-cicd · 2026-07-29T06:36:33Z

PR_Github #62453 [ run ] triggered by Bot. Commit: 1fa1c42 Link to invocation

tensorrt-cicd · 2026-07-29T09:50:15Z

PR_Github #62453 [ run ] completed with state FAILURE. Commit: 1fa1c42
/LLM/main/L0_MergeRequest_PR pipeline #50606 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

BowenFu · 2026-07-29T11:20:01Z

The sentinel handshake itself reads correctly — backgrounding the gen srun in a subshell and touching gen_server_{i}.done after it returns does give you a reap-ordered signal, and writing benchmark_status in finally before blocking is what keeps it from deadlocking. Removing the settle-poll heuristic once you have a real end-of-write signal is the right call.

My concern is the waiver removal, not the fix.

11 of the 41 unwaived cases are aggr_upload-*, i.e. aggregated serving, spanning 8 distinct nvbugs (6153575, 6329155, 6374910, 6418510, 6422339 ×2, 6432948, 6517846 ×3, 6530213). The change is entirely in the disaggregated path — jenkins/scripts/perf/disaggregated/slurm_launch_draft.sh and the gen-log parse. Aggregated runs produce no gen_server_{i}.log and no sentinel, so the truncation race can't be what those 11 were waived for. The 30 disagg_upload-* removals are the ones the root cause actually explains.

The bug records don't support the rest either. Checked just now:

6490049 (13 removals) — P0, Open issue / Dev - Open - To fix, no FixedDate
6517846 (10 removals) — Open issue / Dev - Open - To fix
6530213 (4 removals) — Open issue / Dev - Open - To fix
6487040 (this PR's own root cause) — P0, Open issue / Dev - Open - To fix

All still open, and the first three are umbrella "Perf test failures in L0_PostMerge #2850/#2862/#2865" bugs that collect failures from many models — not evidence that a specific case was fixed by this change.

Given the description says these go live in pre-merge, and blossom-ci is currently failing on this head, unwaiving all 41 in the same PR means a red pre-merge signal can't be attributed. Would suggest splitting: land the sentinel fix plus the 30 disagg_upload unwaives it explains, and handle the 11 aggregated ones separately with their own evidence. Happy to look again once CI is green.

… before parsing per-iter step time The gen_only perf-sanity metric mean_gen_worker_per_iter_device_step_time was intermittently parsed from a truncated gen_server_{i}.log: the benchmark srun read the log while the gen srun was still flushing its &> aggregate across NFS/Lustre, so the mean was computed over a partial prefix of iterations (nvbugs 6487036 / 6487040). The prior settle-poll heuristic (row count unchanged across two polls) accepted the first repeated count and could latch onto a mid-flush prefix. Fix (no pyexecutor change; no gen-worker -> client push): - slurm_launch_draft.sh: run each gen srun in the foreground of a backgrounded subshell and touch gen_server_{i}.done immediately after the srun returns. The srun owns the aggregate fd, so the sentinel fires strictly after reap, i.e. after the log is fully flushed. A stale sentinel from a re-run output dir is removed first. - test_perf_sanity.py: the BENCHMARK run_cmd branch defers the parse out of the client loop. It writes benchmark_status in finally (which releases the gen workers so their srun can exit and the sentinel can fire -- this handshake is why the wait is not circular), then wait_for_gen_log_sentinels() blocks (bounded by self.timeout; falls through on timeout so CI never hangs), then each client's log is parsed exactly once. Parse + OpenSearch upload stay in the BENCHMARK srun. - parse_gen_worker_device_step_time: drops the settle loop and reads the now-complete log once (settle_timeout / poll_interval kwargs removed). The all-iter Welford fallback for unbucketable num_generation_tokens (Mode 1, PR NVIDIA#16298 regression) is retained. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

…xed by the sentinel handshake The per-iter device_step_time parse failure (nvbugs 6487036 / 6487040) was caused by the benchmark srun reading gen_server_*.log before the gen srun had flushed and been reaped, so a truncated iteration segment was averaged. The end-of-write sentinel handshake in the previous commit makes the benchmark block on gen_server_$i.done before parsing, removing that race. Un-waive the 20 gen_only test_perf_sanity cases whose only failure mode was this parse race. The remaining gen_only cases that still fail for unrelated reasons stay waived. Non-gen_only (aggr_upload / e2e / ctx_only) waivers are left untouched. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-07-29T12:15:29Z

Thanks for the careful read, @BowenFu — your concern about the waiver removal is exactly right, and I've narrowed the PR to address it.

What changed on the new head: the un-waive set is now scoped to only disagg_upload-gen_only-* — the precise path the gen-log sentinel race explains. Concretely, the diff vs origin/main is now:

20 disagg_upload-gen_only   (removed)
 0 aggr_upload-*            (untouched)
 0 disagg_upload-e2e        (untouched)
 0 disagg_upload-ctx_only   (untouched)

So the 11 aggr_upload-* cases you flagged (4 ctx_only + 1 super_ad_blackwell + 6 aggregated e2e, across 6153575 / 6329155 / 6374910 / 6418510 / 6422339 / 6432948) — none of which produce a gen_server_{i}.log or a sentinel — are left waived. Aggregated serving is untouched by this change, so those keep their own separate handling with their own evidence, as you suggested.

On the umbrella bugs (6490049 / 6517846 / 6530213): I agree the bug records don't show a per-case FixedDate, so I'm not treating "bug closed" as the un-waive signal. Instead I'm keying off the actual per-case post-merge signal: the disagg_upload-gen_only cases that still fail independent of the parse race stay waived. That's 5 of them — 6478615, 6374872, and three tracked under the 6490049 umbrella — which remain in waives.txt; only the 20 that the sentinel handshake accounts for are removed.

Net: this PR now lands the sentinel fix plus exactly the disagg_upload-gen_only un-waives the root cause accounts for — the split you asked for. The PR description is updated to match. Happy to have you look again once blossom-ci is green.

ZhanruiSunCh

LGTM for infra part.

chenfeiz0326 · 2026-07-29T13:11:42Z

/bot skip --comment "Only fix perf tests, no need to run the whole CI pipeline"

tensorrt-cicd · 2026-07-29T13:17:42Z

PR_Github #62521 [ skip ] triggered by Bot. Commit: 2334bab Link to invocation

tensorrt-cicd · 2026-07-29T13:23:52Z

PR_Github #62521 [ skip ] completed with state SUCCESS. Commit: 2334bab
Skipping testing for commit 2334bab

Link to invocation

chenfeiz0326 requested review from a team as code owners July 22, 2026 08:52

chenfeiz0326 requested review from EmmaQiaoCh, LarryXFly, ZhanruiSunCh, brnguyen2 and schetlur-nv July 22, 2026 08:52

github-actions Bot assigned chenfeiz0326 Jul 22, 2026

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py Outdated

Comment thread tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py Outdated

chenfeiz0326 requested a review from a team as a code owner July 22, 2026 14:05

chenfeiz0326 requested a review from JunyiXu-nv July 22, 2026 14:05

chenfeiz0326 requested a review from fredricz-20070104 July 23, 2026 02:51

fredricz-20070104 approved these changes Jul 23, 2026

View reviewed changes

chenfeiz0326 force-pushed the fix/gen-only-device-step-time-parse-none-6487040 branch from 91be512 to daa4def Compare July 24, 2026 04:13

chenfeiz0326 requested a review from a team as a code owner July 28, 2026 09:48

chenfeiz0326 force-pushed the fix/gen-only-device-step-time-parse-none-6487040 branch 2 times, most recently from 4b089a3 to ca85e9e Compare July 29, 2026 05:37

chenfeiz0326 changed the title ~~[https://nvbugs/6487040][test] Robustify gen_only per-iter device-step-time metric parsing~~ [https://nvbugs/6487040][test] Wait for gen-log end-of-write sentinel before parsing per-iter step time Jul 29, 2026

chenfeiz0326 force-pushed the fix/gen-only-device-step-time-parse-none-6487040 branch from ca85e9e to 1fa1c42 Compare July 29, 2026 06:18

Shixiaowei02 approved these changes Jul 29, 2026

View reviewed changes

chenfeiz0326 and others added 2 commits July 29, 2026 05:06

chenfeiz0326 force-pushed the fix/gen-only-device-step-time-parse-none-6487040 branch from 1fa1c42 to 2334bab Compare July 29, 2026 12:11

StanleySun639 approved these changes Jul 29, 2026

View reviewed changes

BowenFu approved these changes Jul 29, 2026

View reviewed changes

ZhanruiSunCh approved these changes Jul 29, 2026

View reviewed changes

chenfeiz0326 enabled auto-merge (squash) July 29, 2026 13:11

EmmaQiaoCh approved these changes Jul 29, 2026

View reviewed changes

chenfeiz0326 merged commit 99bdffc into NVIDIA:main Jul 29, 2026
11 checks passed

Uh oh!

Conversation

chenfeiz0326 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chenfeiz0326 commented Jul 22, 2026

Uh oh!

chenfeiz0326 commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

chenfeiz0326 commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

fredricz-20070104 left a comment

Choose a reason for hiding this comment

Uh oh!

chenfeiz0326 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chenfeiz0326 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chenfeiz0326 commented Jul 24, 2026

Uh oh!

chenfeiz0326 commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

chenfeiz0326 commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

BowenFu commented Jul 29, 2026

Uh oh!

chenfeiz0326 commented Jul 29, 2026

Uh oh!

ZhanruiSunCh left a comment

Choose a reason for hiding this comment

Uh oh!

chenfeiz0326 commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

chenfeiz0326 commented Jul 22, 2026 •

edited

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading