fix: tie weights outside _init_model#1817
Merged
Merged
Conversation
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Contributor
Author
|
/ok to test a45ba52 |
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Contributor
Author
|
/ok to test 07a94bc |
Contributor
Author
|
@HuiyingLi confirmed this working on phi4mm too. |
svcnvidia-nemo-ci
pushed a commit
that referenced
this pull request
Apr 14, 2026
* tie weights outside _init_model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add typecasting back Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
akoumpa
added a commit
that referenced
this pull request
Apr 14, 2026
fix: tie weights outside _init_model (#1817) * tie weights outside _init_model * update test * undo test * add typecasting back --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
edjson
pushed a commit
to edjson/Automodel
that referenced
this pull request
Apr 17, 2026
* tie weights outside _init_model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add typecasting back Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
edjson
pushed a commit
to edjson/Automodel
that referenced
this pull request
Apr 18, 2026
* tie weights outside _init_model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add typecasting back Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Edison <edisonggacc@gmail.com>
linnanwang
pushed a commit
that referenced
this pull request
Apr 24, 2026
* tie weights outside _init_model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add typecasting back Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
kashif
pushed a commit
to kashif/Automodel
that referenced
this pull request
May 25, 2026
* tie weights outside _init_model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add typecasting back Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
This was referenced Jul 6, 2026
Closed
yuhezhang-ai
added a commit
that referenced
this pull request
Jul 8, 2026
Nemotron-Flash-1B is the tied list-form-tied-keys remote-code model (the #1817 case): enabling Phase 0 on it proves the force-HF load still ties when the config requests it — the mirror of the untied Nano-9B coverage. mistral_7b_hellaswag_fp8 covers the training-time FP8 quantization path; Phase 0 bounds the FP8-prepared trainer against the raw bf16 reference, with thresholds loosened for expected quantization error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
yuhezhang-ai
added a commit
that referenced
this pull request
Jul 9, 2026
…#2942) _tie_weights_nemo re-tied lm_head.weight to the input embedding for any model exposing _nemo_tied_weights_keys, without consulting config.tie_word_embeddings. For untied models this aliased away the trained lm_head loaded by from_pretrained, and corrupted embeddings/head on checkpoint resume (both tensors loaded into one shared storage). Gate the re-tie on the controlling config flag via get_controlling_tie_word_embeddings. Tied models (the #1817 Nemotron-Flash case) keep the existing re-tie behavior; models without a config fall back to re-tying. Fixes #2941 Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Changelog
Before your PR is "Ready for review"
Pre checks:
If you haven't finished some of the above items you can still open "Draft" PR.
Additional Information