Skip to content

feat: Support Nemotron-nano-v3 Omni AutoModel Path #2362

Merged
yuki-97 merged 79 commits into
NVIDIA-NeMo:mainfrom
yuekaizhang:nemotron
Jun 17, 2026
Merged

feat: Support Nemotron-nano-v3 Omni AutoModel Path #2362
yuki-97 merged 79 commits into
NVIDIA-NeMo:mainfrom
yuekaizhang:nemotron

Conversation

@yuekaizhang

Copy link
Copy Markdown
Contributor

This PR follows the nano-v3-omni mbridge training branch to add AutoModel backend support for Nemotron-Nano-Omni.

yuekaizhang and others added 28 commits April 29, 2026 02:55
Signed-off-by: Yuekai Zhang <yuekaiz@cw-dfw-cs-001-vscode-02.cm.cluster>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Add .claude/settings.local.json, .codex, and .humanize/ to
.gitignore as these are local tool configuration/cache files
that should not be tracked.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
- Add MMPRTinyDataset class with HF download fallback and local cache support
- Add format_mmpr_tiny_dataset for OpenAI-API message conversion
- Port verl_geo3k reward function from old Megatron-Bridge implementation
- Register mmpr-tiny in DATASET_REGISTRY and vlm_hf_data_processor
- Register verl_geo3k reward in VLMVerifyWorker
- Add mathruler dependency to pyproject.toml for answer grading
- Create debug-friendly YAML config with step_400 checkpoint
- Create launch script with uv sync pre-step
- Create CoT prompt file with \boxed{} instruction

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
- Remove self.preprocessor from MMPRTinyDataset to prevent double
  formatting (vlm_hf_data_processor already handles format dispatch)
- Add pylatexenc to pyproject.toml as transitive dependency of mathruler
  (mathruler does not declare it in its own metadata)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
- Fix prompt file brace escaping: \boxed{} -> \boxed{{}} so
  str.format() has exactly one replacement placeholder (fixes AC-2/AC-5)
- Add explicit split parameter validation to MMPRTinyDataset with
  ValueError for unsupported splits (fixes AC-1 negative test)
- Regenerate uv.lock with mathruler and pylatexenc entries (fixes AC-6)
- Add unit tests for dataset formatting, split validation, prompt
  file format compatibility, and verl_geo3k_reward (14 tests, all pass)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Add 3 processor smoke tests using a stub NemotronNanoVLV2Processor:
- test_processor_produces_valid_datum_spec: verifies DatumSpec fields
- test_prompted_text_contains_boxed_literal: verifies \boxed{} survives
- test_placeholder_conversion_for_nemotron_processor: verifies <image>
  placeholder and question text in vllm_content

Uses a tiny 1x1 PNG fixture for image resolution. All 17 MMPR tests
pass (11 dataset + 6 reward).

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
- Stub now captures the exact text arg passed to __call__
- Assert exact equality: vllm_content == "<image>\n" + prompted_question
- Assert exactly one <image> token in output (no duplicates)
- Negative assertion: raw dataset string "<image>\nQuestion" not in output
- Assert captured __call__ text matches expected tokenizer input
- All tests run with -p no:testmon --override-ini='addopts='

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Clean up partial state (stale images_dir, parquet, temp dir) before
re-downloading when the ready marker is absent. Prevents shutil.move
from nesting images/images when images_dir already exists from an
interrupted prior attempt.

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
MMPR-Tiny has 4,192 rows with multiple images (up to 11). The previous
code truncated to images[0], losing visual context for multi-image
questions.

- _load_mmpr_tiny_from_cache: keep all image paths instead of [imgs[0]]
- format_mmpr_tiny_dataset: split question on <image> and <image_N>
  placeholders, interleave image content items with text segments
- Add tests for multi-image and numbered-placeholder formatting

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: root <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: root <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
…nting

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
…sound_projection to resume training

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
@yuekaizhang yuekaizhang requested review from a team as code owners April 29, 2026 10:25
@yuekaizhang

Copy link
Copy Markdown
Contributor Author

/ok to test 4177252

@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: 4177252 (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

@yuekaizhang

Copy link
Copy Markdown
Contributor Author

@jinglinglingling Hi Linglin, could you please review the PR again? Thanks!

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: 2a49359 (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

Resolve the unresolved merge conflict between the nemotron (1970h) and
main (2000h) branches by setting the nightly compute budget to 2070 hours.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
@yuekaizhang

Copy link
Copy Markdown
Contributor Author

/ok to test 86c1c1f

@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: 86c1c1f (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @yuekaizhang , overall LGTM and left some minor comments.

Comment thread 3rdparty/Automodel-workspace/Automodel
Comment thread nemo_rl/environments/rewards.py Outdated
Comment thread tests/unit/rewards/test_geo3k_reward.py
Rename the reward function and its "verl_geo3k" config registration name to
"geo3k", and note in the docstring that it has the same behavior as verl's
geo3k reward function. Updates the function definition, the VLM environment
dispatch/import, the unit test (file + call sites), the exemplar/recipe YAMLs,
and the docs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: root <zhangyuekai@foxmail.com>
@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: 15cc685 (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

@yuekaizhang

Copy link
Copy Markdown
Contributor Author

/ok to test 15cc685

yuki-97
yuki-97 previously approved these changes Jun 16, 2026
@yuekaizhang

Copy link
Copy Markdown
Contributor Author

/ok to test c4f6247

@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: c4f6247 (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

@yuekaizhang

Copy link
Copy Markdown
Contributor Author

/ok to test 04c62f0

@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: 04c62f0 (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

…y recipes

Move and rename the Nemotron-Omni VLM GRPO configs into
examples/configs/recipes/vlm/ following the VLM recipe naming convention,
and run both as functional nightly tests:
- vlm_grpo-nemotron-omni-30ba3b-clevr-1n8g-automodel-ep8.v1 (CLEVR-CoGenT, 1 node)
- vlm_grpo-nemotron-omni-30ba3b-mmpr-4n8g-automodel-ep8.v1 (MMPR-Tiny, 4 nodes)

Both configs are minimized snapshots that inherit from vlm_grpo_3B.yaml; the
MMPR config carries cluster.num_nodes=4. The previous thin wrapper recipe
(same mmpr name) is replaced by the full config under that name, and the
nightly now points at these recipes directly via matching driver scripts.

Nightly GPU-hours total stays at 2041h, under the 2070h budget.

Update the Nemotron-Omni guide to the new config paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: root <zhangyuekai@foxmail.com>
@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: bc85d5f (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

@yuekaizhang

Copy link
Copy Markdown
Contributor Author

/ok to test bc85d5f

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #2362 — feat: Support Nemotron-nano-v3 Omni AutoModel Path

Nice work — this adds Nemotron-Nano-Omni 30B-A3B AutoModel recipes (CLEVR-CoGenT + MMPR-Tiny), the MMPR-Tiny loader, a geo3k reward, docs, and nightly tests. Reviewed by a team of agents (RL/config, Automodel SME, tests, bug-finder) plus an adversarial pass.

No blocking bugs found. Every bug candidate was verified and refuted: the geo3k brace parser is correct (validated with nested/multi-last/no-close/empty cases), the MMPR download-cache lock is safe, the placeholder <image> token count matches len(images), and AutoConfig.from_pretrained in ModelFlag.matches() runs only at worker init (not a hot path). Convergence curves vs. Megatron-Bridge are present in the comments, and the nightly GPU-hour bump (2000→2070) is justified (actual total ≈2041h).

The inline comments are all suggestions/nits — the most substantive is migrating off the deprecated enable_deepep knob (Automodel upstream did the same for its own Nemotron-Omni recipes in this bump). Existing unit tests all pass locally (geo3k 6/6, mmpr_tiny 10/10).

Generated by Claude Code

Comment thread docs/guides/nemotron-3-nano-omni.md Outdated
Comment thread nemo_rl/data/datasets/response_datasets/mmpr_tiny.py Outdated
Comment thread tests/unit/rewards/test_geo3k_reward.py
yuekaizhang and others added 2 commits June 17, 2026 07:08
- Migrate off deprecated enable_deepep knob to dispatcher=deepep +
  experts=torch_mm in both CLEVR and MMPR recipes (avoids grouped_gemm dep)
- Remove leftover NRL_DEBUG_PICKLE debug env var from MMPR recipe
- Fix docs max_total_sequence_length 10000 -> 8192 to match MMPR recipe
- Update mmpr_tiny.py copyright year to 2026
- Add geo3k tests for last-wins multi-boxed and nested/unbalanced braces
- Declare experts/dispatcher and freeze_config in Automodel TypedDicts

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: root <zhangyuekai@foxmail.com>
# Conflicts:
#	tests/unit/test_recipes_and_test_suites.py
@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: a3ca738 (PR #2362 from nemotron)

❌ Submodules that need attention:

Automodel: ❌ Commits have DIVERGED from a common ancestor
TARGET (main branch): https://github.com/NVIDIA-NeMo/Automodel/commits/6de0c3612156c1d882b6e951af9b0f5d9af29d4a/
CURRENT (PR #2362 from nemotron): https://github.com/NVIDIA-NeMo/Automodel/commits/24b47e856263d313b942f0ed666c63fff83306b4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

@yuekaizhang

Copy link
Copy Markdown
Contributor Author

/ok to test a3ca738

@qiaochuz-nv

Copy link
Copy Markdown
Contributor

Doc cross-check for docs/guides/nemotron-3-nano-omni.md: I verified the stated recipe/config details against the resolved PR recipes, including inheritance from examples/configs/vlm_grpo_3B.yaml. The doc looks technically accurate.

Two non-blocking doc nits from the POR run:

  1. MMPR-Tiny data prep: the doc says MMPR-Tiny is auto-downloaded from HF / no manual prep needed. That is true with network access, but the standard air-gapped/offline test environment sets HF_HUB_OFFLINE=1, so OpenGVLab/MMPR-Tiny (notably images.zip) must be pre-staged in the HF dataset cache. Consider adding a note for offline clusters.

  2. MMPR launch snippet: the snippet uses a placeholder CONTAINER and injects PYTHONPATH=...:/path/to/automodel-omni, which reads like the Omni code may not yet be part of the base container. If that is temporary/intended, no issue; otherwise it may be worth clarifying the expected container/code path.

Neither is a blocker for the PR.

@qiaochuz-nv

Copy link
Copy Markdown
Contributor

POR Test Summary — Nemotron-3-Nano-Omni-30B VLM GRPO (PR #2362)

PR under test: NVIDIA-NeMo/RL #2362 — Nemotron-Nano-v3 Omni AutoModel VLM GRPO support
MR: https://gitlab-master.nvidia.com/dl/DLQASH/nmfw_tests/-/merge_requests/270
Branch: qiaochuz/por_2362_nano_omni_vlmmain
Commit: e786d6b (3 files, 152 insertions — test logic only; local infra reverted)
Status: ✅ Both testcases REAL GREEN


Model & Backend Under Test

Item Value
Model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
Backend AutoModel
Parallelism Expert Parallel = 8 / vLLM Tensor Parallel = 8 (triton MoE backend)
Towers Vision + audio frozen; LM trained
Max sequence length 8192

What the Tests Do

Both drivers are a 1:1 mirror of the PR's own oracle scripts
(tests/test_suites/vlm/vlm_grpo-nemotron-omni-30ba3b-*.v1.sh):

  • Entrypoint: examples/run_vlm_grpo.py --config <recipe>
  • Step shrink: grpo.max_num_steps=10 (CI-sized; full run is longer)
  • Logging: wandb off, tensorboard on, GPU monitor on, checkpointing on
  • Assertion (gated): dump TB logs → metrics.json; only if train/loss reached
    step 10, run check_metrics.py 'max(data["train/reward"]) > 0.5'

Function Tests

1. CLEVR — function tier (testcases/pipelines/)

test_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_automodel_ep8.sh

Field Value
Task CLEVR-CoGenT
Scale 1 node × 8 GPU
Execution Interactive
Timeout 7200 s
Recipe vlm_grpo-nemotron-omni-30ba3b-clevr-1n8g-automodel-ep8.v1.yaml

Result: REAL GREEN — run 5532636, 10 steps, reward 0.83 (>0.5), val acc 0.713

2. MMPR-tiny — e2e tier (testcases/e2e/)

test_vlm_grpo_nemotron_omni_30ba3b_mmpr_4n8g_automodel_ep8.sh

Field Value
Task MMPR-tiny (geo3k / MMPR reward, vlm_environment path)
Scale 4 nodes × 8 GPU = 32 GPU
Execution sbatch via ray.sub (batched-job model)
Recipe vlm_grpo-nemotron-omni-30ba3b-mmpr-4n8g-automodel-ep8.v1.yaml

Result: REAL GREEN (verified ×2) — run 5533255, SLURM COMPLETED 0:0,
runner failures=0, reward 0.799 (>0.5), step 10/10, val acc 0.664


Coverage

Covers both VLM GRPO recipes that PR #2362 introduces, across two scales and two execution models:

  • Single-node interactive (CLEVR) + multi-node batched ray.sub (MMPR) — exercises the batched-job submission path the PR adds.
  • New mmpr-tiny dataset + geo3k / MMPR reward function + vlm_environment path.
  • AutoModel VLM with frozen vision/audio towers, MoE EP=8 / TP=8 triton backend.

Tags: codecoverage, por_v06, current_por, rc_p0


Doc Check — docs/guides/nemotron-3-nano-omni.md

The recipes run green exactly as the PR ships them — no functional/run-breaking doc issue.
5 minor cosmetic/consistency doc nits were noted during the doc-vs-recipe re-check; no PR
comment or patch filed (per decision to leave doc fix out of scope).


Artifacts

Test Run ID Cluster
CLEVR 5532636 EOS
MMPR 5533255 EOS, nodes eos[0548,0558-0559,0563]

Remote logs base: /lustre/fsw/coreai_dlalgo_ci/qiaochuz/nemo_llm_nightly/test_suite/rl/logs/
Container: nemo-rl-nightly-20260629.sqsh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests Documentation Improvements or additions to documentation QA:Verified

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants