fix(DSV3): parity between native DeepseekV3MoE and remote official implementation

[casinca]

What does this PR do?

Please see fix #45440 for more details

Discussed with @vasqu

Also fixed via regen/inheritance:

• exaone-moe
• glm4-moe
• glm4v-moe
• glm4-moe-lite
• glm_moe_dsa
• nemotron_h
• solar_open

 

Warning

Edit: The test passes for me I’m getting back my string (with and without fix).
I realized it makes sense, biases are zero init, registered as a buffer but the model for the test is untrained, so router_logits_for_choice = router_logits + self.gate.e_score_correction_bias is always the same, fix or not.
This test, for other models, would only fail if the model is loaded from a pretrained checkpoint (non zero biases)

 

transformers/tests/models/deepseek_v3/test_modeling_deepseek_v3.py

Lines 398 to 409 in 5b565a5

	@slow
	@require_torch_accelerator
	@pytest.mark.torch_compile_test
	def test_compile_static_cache(self):
	NUM_TOKENS_TO_GENERATE = 40
	# https://github.com/huggingface/transformers/pull/38562#issuecomment-2939209171
	# The reason why the output is gibberish is because the testing model bzantium/tiny-deepseek-v3 is not trained
	# one. Since original DeepSeek-V3 model is too big to debug and test, there was no testing with the original one.
	EXPECTED_TEXT_COMPLETION = [
	"Simply put, the theory of relativity states that Frojekecdytesాలు sicʰtinaccianntuala breej的效率和质量的控制lavestock-PraccuraciesOTTensorialoghismos的思路astiomotivityosexualriad TherapeuticsoldtYPEface Kishsatellite-TV",
	"My favorite all time favorite condiment is ketchup.ieden沟渠係室温 Fryrok般地Segmentation Cycle/physicalwarenkrautempsాలు蹈梗 Mesomac一等asan lethality suspended Causewaydreamswith Fossilsdorfాలు蹈 ChristiansenHOMEbrew",
	]

 

So I reran the DSV3 forward pass to get the new hardcoded string for myself and retest but maybe this needs to be run and tested on your side too?

       EXPECTED_TEXT_COMPLETION = [
"Simply put, the theory of relativity states that aportersh455elike injection tactics-altitude蹲在那儿 >Loregefruitakosdeckingredientsuchtroni李世umontיםplicitlyShadowoldtriad Therapeutics不减-ste 的希望和价值 >kerretteylesheetzimnasium的品质 Talm",
"My favorite all time favorite condiment is ketchup. Lan overhead excite-ment好用>cileriaceaeagnainesogaslipadicSiggleESHalseawarriorsrattieri佐iented >Parrheta-counterousseanatysisoglCTSinkeheilbronnenlaceslide tactauralick",
       ]

 

• Also potential cascading effect with the inherited models too and their own def test_compile_static_cache(self): test?

 

I greedily flagged 3 other models with masked_fill(~score_mask.bool(), 0.0) but I'm not blindly touching these for now,
need to verify their logic first (if using loss free load balancing) + what their remote are doing. If the remote is
wrong, not sure we should change the native implementation or keep it wrong with the remote.

This is likely for a follow-up PR if these need to be fixed too.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: deepseek_v3, exaone_moe, glm4_moe, glm4_moe_lite, glm4v_moe, glm_moe_dsa, nemotron_h, solar_open

[vasqu]

I am inclined to fix this for deepseek v3 itself as it is indeed added within its remote code; however, even if it is a bug, we need to be careful for other models as that may now be their expected behavior. core cc @ArthurZucker @Cyrilvallez

Maybe we should ping the other model maintainers but would like to hear the opinion of at least one other core maintainer on this

[vasqu]

Thanks a lot for the PR, very nice find! Wanna make sure we go right and don't rush it as it affects quite a few models

[casinca]

Concerning the 3 other models in the repo using explicitly masked_fill(~score_mask.bool(), 0.0):

There's Mistral4 (doesn't use loss free load balancing), DSV2 (can't be, since it was introduced in DSV3), but dots1 seems so.

Probably another PR for this one? it inherits from DeepSeekV3 with a comment saying # main diff with deepseekv3 but I don't see any diff with DVS3(left)? Could have been a direct pass like the others I guess.

[ArthurZucker]

Having a look!

[ArthurZucker]

For dots, indeed if there is no diff feel free to inherit!

This is minor:

1. the scores for choice are only used to compute the index, not the weights
2. the router_logits_for_choice can probably be 0 or negative (with bias), I thought it would be super rare? what you are trying to avoid is that "natural" 0 could be part of the topk vs the masking? makes sense

IMO does not warrant a patch, but you tell me if this is super crucial!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[casinca]

For dots, indeed if there is no diff feel free to inherit!

This is minor:

1. the scores for choice are only used to compute the index, not the weights

2. the `router_logits_for_choice` can probably be 0 or negative (with bias), I thought it would be super rare? what you are trying to avoid is that "natural" 0 could be part of the topk vs the masking? makes sense

IMO does not warrant a patch, but you tell me if this is super crucial!

1. yes, weights use unbiased scores.

2. Following your question, I just checked the distribution on biases, well... it's funny because the minimum bias across all layers is +2.02. So specifically for DSV3 it's not even rare but impossible xD, 0 will act the same as -inf, no behavior change. But might not be the case for all other models.

   Not just about tie: natural 0 vs masked 0, it was more about avoiding a valid unmasked expert with a negative score being not selected/outranked by a masked one (with a masked score of 0.0)

Imho, since some models inherit from DSV3 and also DSV3.2 does -inf, I believe it was still a good thing to match 1:1.
Initially it was just to be rigorous for the MiMo-V2 impl and to match their remote modeling file (which was taken from the remote DSV3)

 

• Will open a PR for dots1, thanks.