Added cosmos3 model by MaciejBalaNV · Pull Request #46146 · huggingface/transformers

MaciejBalaNV · 2026-05-21T19:47:46Z

What does this PR do?

This PR adds a support for Cosmos3 Reasoner model (not released yet). It's a Mixture Of Transformers model, where we have a Generator and a Reasoner tower in a unified checkpoint. The Reasoner tower has Qwen3-VL architecture, so we can directly reuse it. However, we need extra code to handle the checkpoint mapping, since the final checkpoint will be in a unified Reasoner+Generator diffusers format.

Additionally, this PR fixes one issue which currently is present on top of tree - when using latest vllm and latest transformers build from source, even basic vllm serve Qwen/Qwen3-VL-8B-Instruct fails during dummy run. This root-cause of the bug is this commit: ba06e3fbdf355c363ac067ebcda210017e90a852, reverting it also fixes Qwen-VL.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yonigozlan for a vision model review

Signed-off-by: Maciej Bala <mbala@nvidia.com>

MaciejBalaNV · 2026-05-22T21:46:07Z

Converting to draft for now until we decide on the final checkpoint format

zucchini-nlp

I am not sure if we need a whole new model for it since the arch is identical to qwen3-vl from what I see. The only diff seems to be the conversion mapping and keys_to_ignore_at_load, for which I added questions below

Since you are still deciding on ckpt, maybe we could release it by re-using a qwen3-vl model. WDYT?

Signed-off-by: Maciej Bala <mbala@nvidia.com>

MaciejBalaNV · 2026-05-25T14:47:28Z

The code works now with the planned Cosmos3 checkpoint format, reopening for review.

Signed-off-by: Maciej Bala <mbala@nvidia.com>

zucchini-nlp

Nice, let's merge after addressing a few nitty-picky comments :)

vasqu · 2026-05-28T12:49:22Z

Btw, we reverted some bigger PR, need to sync with main for the CI to pass again

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Signed-off-by: MaciejBalaNV <mbala@nvidia.com>

zucchini-nlp · 2026-06-05T09:41:23Z

run-slow: cosmos3_omni

github-actions · 2026-06-05T09:42:44Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/cosmos3_omni"]
quantizations: []

zucchini-nlp · 2026-06-05T12:52:49Z

run-slow: cosmos3_omni

github-actions · 2026-06-05T12:54:11Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/cosmos3_omni"]
quantizations: []

github-actions · 2026-06-05T13:48:11Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, cosmos3_omni, qwen3_5, qwen3_5_moe, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe

zucchini-nlp · 2026-06-05T13:56:55Z

run-slow: cosmos3_omni

github-actions · 2026-06-05T13:58:19Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/cosmos3_omni"]
quantizations: []

github-actions · 2026-06-05T14:05:36Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	90ad29f9	workflow commit (merge commit)
PR	04fbe459	branch commit (from PR)
main	98eea724	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

zucchini-nlp · 2026-06-05T14:08:48Z

Thanks god, slow CI decided to turn green. Merging...

ydshieh · 2026-06-05T14:56:40Z

FAILED tests/models/auto/test_modeling_auto.py::AutoModelTest::test_from_pretrained_dynamic_model_distant - AssertionError: <class 'transformers_modules.hf_hyphen_internal_hyphen_testing.test_dynamic_model.2efddce40dddaccd37bae208c3c7ca66dbedf68a.modeling.NewModel'> is not <class 'transformers_modules.hf_hyphen_internal_hyphen_testing.test_dynamic_model.c8ae8a2e5845fc3ad4e5237e4346e5a9ba1e87d2.modeling.NewModel'>

i don't think it's relevant, ping me if you want me to force merge.

zucchini-nlp · 2026-06-05T15:09:05Z

Yep, the CI is fine, it's just me. The model has sampling on by default and I didn't get at first why the CI keeps complaining after I adjust expectations with runners 😅

* Added cosmos3 model Signed-off-by: Maciej Bala <mbala@nvidia.com> * bugfix for processing_utils.py Signed-off-by: Maciej Bala <mbala@nvidia.com> * Add extra tests * Rename architecture; linter fixes Signed-off-by: Maciej Bala <mbala@nvidia.com> * Rename the architecture again for now Signed-off-by: Maciej Bala <mbala@nvidia.com> * Adapted to new Cosmos3 checkpoint format * Improve the checkpoint loading Signed-off-by: Maciej Bala <mbala@nvidia.com> * Simplified docs Signed-off-by: Maciej Bala <mbala@nvidia.com> * Renamed cosmos3 to cosmos3_reasoner Signed-off-by: Maciej Bala <mbala@nvidia.com> * Removed cosmos3 tests Signed-off-by: Maciej Bala <mbala@nvidia.com> * Revert processing fix Signed-off-by: Maciej Bala <mbala@nvidia.com> * Answered review comments Signed-off-by: Maciej Bala <mbala@nvidia.com> * Rename to Cosmos3ReasonerForConditionalGeneration Signed-off-by: Maciej Bala <mbala@nvidia.com> * Renamed config key Signed-off-by: Maciej Bala <mbala@nvidia.com> * Run make fix-repo Signed-off-by: Maciej Bala <mbala@nvidia.com> * Refactored naming Signed-off-by: Maciej Bala <mbala@nvidia.com> * Some more renaming; fix linter issues; Add Cosmos3 Processor Signed-off-by: MaciejBalaNV <mbala@nvidia.com> * Linter fixes Signed-off-by: MaciejBalaNV <mbala@nvidia.com> * Multiple fixes Signed-off-by: Maciej Bala <mbala@nvidia.com> * fix test and linter Signed-off-by: Maciej Bala <mbala@nvidia.com> * Make fix-repo fixes Signed-off-by: Maciej Bala <mbala@nvidia.com> * Linter fix Signed-off-by: Maciej Bala <mbala@nvidia.com> * Change date Signed-off-by: Maciej Bala <mbala@nvidia.com> * rename it all and load existing AutoModel * docs * fix repo * fix docs * ugly typo * adjust expectation * dont sample in test! * forgot this one * style --------- Signed-off-by: Maciej Bala <mbala@nvidia.com> Signed-off-by: MaciejBalaNV <mbala@nvidia.com> Co-authored-by: raushan <raushan@huggingface.co>

After the transformer PR was merged for Reasoner-only here: huggingface/transformers#46146, I'm updating the cosmos docs to reflect this change. Also made some small updates for vllm, like installing 0.23 version, which is the first one with Cosmos Reasoner-only support. --------- Signed-off-by: Maciej Bala <mbala@nvidia.com>

MaciejBalaNV added 3 commits May 21, 2026 17:06

Added cosmos3 model

c37442d

Signed-off-by: Maciej Bala <mbala@nvidia.com>

bugfix for processing_utils.py

e453738

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Add extra tests

61cd694

MaciejBalaNV mentioned this pull request May 21, 2026

Add Cosmos3 Reasoner model vllm-project/vllm#43356

Merged

4 tasks

zucchini-nlp self-requested a review May 22, 2026 00:45

MaciejBalaNV added 2 commits May 22, 2026 13:26

Rename architecture; linter fixes

4f7087b

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Rename the architecture again for now

2d096f8

Signed-off-by: Maciej Bala <mbala@nvidia.com>

MaciejBalaNV marked this pull request as draft May 22, 2026 21:45

zucchini-nlp reviewed May 25, 2026

View reviewed changes

Comment thread docs/source/en/model_doc/cosmos3.md Outdated

Comment thread src/transformers/conversion_mapping.py

Comment thread src/transformers/processing_utils.py Outdated

MaciejBalaNV added 2 commits May 25, 2026 15:17

Adapted to new Cosmos3 checkpoint format

4bf38dc

Improve the checkpoint loading

859bac7

Signed-off-by: Maciej Bala <mbala@nvidia.com>

MaciejBalaNV marked this pull request as ready for review May 25, 2026 14:47

Simplified docs

c66247f

Signed-off-by: Maciej Bala <mbala@nvidia.com>

SAY-5 mentioned this pull request May 26, 2026

utils: handle flash_attn missing from importlib packages_distributions without crashing #45524

Merged

zucchini-nlp reviewed May 28, 2026

View reviewed changes

MaciejBalaNV added 8 commits May 28, 2026 16:05

Renamed cosmos3 to cosmos3_reasoner

bf4150e

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Removed cosmos3 tests

1b3c0c5

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Revert processing fix

2a843ca

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Answered review comments

ca72165

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Rename to Cosmos3ReasonerForConditionalGeneration

738cce2

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Renamed config key

eec3e05

Signed-off-by: Maciej Bala <mbala@nvidia.com>

Merge branch 'main' into mbala/cosmos3

e819da8

Run make fix-repo

c0423fd

Signed-off-by: Maciej Bala <mbala@nvidia.com>

zucchini-nlp reviewed May 28, 2026

View reviewed changes

Refactored naming

4fe4398

Signed-off-by: Maciej Bala <mbala@nvidia.com>

MaciejBalaNV changed the title ~~Added cosmos3 model and bugfixed Qwen3-VL~~ Added cosmos3 model May 28, 2026

Some more renaming; fix linter issues; Add Cosmos3 Processor

bccc86e

Signed-off-by: MaciejBalaNV <mbala@nvidia.com>

huggingface deleted a comment from github-actions Bot Jun 5, 2026

zucchini-nlp added 2 commits June 5, 2026 14:49

Merge remote-tracking branch 'upstream/main' into mbala/cosmos3

61d9a8f

dont sample in test!

9fc85eb

huggingface deleted a comment from github-actions Bot Jun 5, 2026

forgot this one

04fbe45

huggingface deleted a comment from github-actions Bot Jun 5, 2026

style

c9215d8

zucchini-nlp enabled auto-merge June 5, 2026 14:09

Merge branch 'main' into mbala/cosmos3

208bfa0

zucchini-nlp added this pull request to the merge queue Jun 5, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 5, 2026

zucchini-nlp added this pull request to the merge queue Jun 5, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 5, 2026

zucchini-nlp added this pull request to the merge queue Jun 5, 2026

Merged via the queue into huggingface:main with commit f677e3a Jun 5, 2026
29 checks passed

MaciejBalaNV mentioned this pull request Jun 22, 2026

Add Reasoner-only transformer docs and update vllm docs NVIDIA/cosmos#228

Merged

Uh oh!

Conversation

MaciejBalaNV commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

MaciejBalaNV commented May 22, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaciejBalaNV commented May 25, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasqu commented May 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zucchini-nlp commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

zucchini-nlp commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

zucchini-nlp commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

CI Results

Commit Info

Uh oh!

zucchini-nlp commented Jun 5, 2026

Uh oh!

Uh oh!

ydshieh commented Jun 5, 2026

Uh oh!

zucchini-nlp commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MaciejBalaNV commented May 21, 2026 •

edited

Loading