Register qwen3_5_moe_text for text-only Qwen3.6 export - #445
Open
justinchuby wants to merge 2 commits into
Open
Conversation
Qwen/Qwen3.6-35B-A3B ships model_type=qwen3_5_moe with a vision_config, so
the builder overrides it to qwen3_5_moe_vl and unwraps text_config (whose own
model_type is qwen3_5_moe_text). Unlike every other VL family (gemma3_text,
qwen3_vl_text, qwen3_5_vl_text), the MoE VL had no registered text-only
sibling, so:
- registry.get("qwen3_5_moe_text") raised KeyError, and
- build(..., text_only=True) on the VL checkpoint failed (no
_TEXT_ONLY_MODEL_TYPE entry for qwen3_5_moe_vl).
Qwen35MoECausalLMModel already strips the language_model. prefix, drops
visual./MTP keys, and unpacks fused experts, so it consumes the VL text
weights directly. Register qwen3_5_moe_text -> Qwen35MoECausalLMModel, wire
the text-only override (qwen3_5_moe_vl -> qwen3_5_moe_text, idempotent
self-map), and add default-id/family map entries.
Add an L1 build-graph config (reusing the qwen3_5_moe hybrid MoE tiny config)
so registry-completeness passes and the new key builds. Verified against the
real Qwen3.6 config that both text_only routing and text_config.model_type
now resolve to Qwen35MoECausalLMModel.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: justinchuby <justinchuby@users.noreply.github.com>
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes text-only export routing for the Qwen3.6-35B-A3B vision-language checkpoint by registering the missing qwen3_5_moe_text registry sibling and wiring the text_only=True remap path so the VL wrapper can export its text backbone as a standalone decoder-only model.
Changes:
- Register
qwen3_5_moe_text→Qwen35MoECausalLMModeland add text-only remappingqwen3_5_moe_vl→qwen3_5_moe_textin_TEXT_ONLY_MODEL_TYPE. - Add L1 coverage via a tiny build-graph config for
qwen3_5_moe_textand skip ONNX checker for it (same custom-op limitations as its sibling). - Add metadata for L2/family defaults (
_TEST_MODEL_IDS, family override entries) so dashboards/tests can resolve the new model_type.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/build_graph_test.py | Add qwen3_5_moe_text to the ONNX-checker skip list to avoid known checker limitations for these custom-op graphs. |
| tests/_test_configs.py | Add a representative tiny config entry for qwen3_5_moe_text (hybrid linear/full attention + MoE) for L1/L3 coverage. |
| src/mobius/_registry.py | Register qwen3_5_moe_text, add text-only remap for qwen3_5_moe_vl, and attach default test model id + family metadata for the new type. |
Comment on lines
+801
to
+805
| # Qwen3.5-MoE-VL (Qwen3.6-35B-A3B): export just the hybrid MoE text | ||
| # backbone as a standalone decoder-only LLM. The builder overrides | ||
| # ``qwen3_5_moe`` -> ``qwen3_5_moe_vl`` when a ``vision_config`` is present, | ||
| # so the text-only override keys off the VL type here. | ||
| "qwen3_5_moe_vl": "qwen3_5_moe_text", |
Member
Author
There was a problem hiding this comment.
@copilot Fix the code for all comments in this review comment.
When a review comment includes a suggested change, apply the suggestion exactly.
Do not make changes beyond what is described in the linked review comment.
Contributor
There was a problem hiding this comment.
Updated the build() docstring in _builder.py to include qwen3_5_moe_vl (Qwen/Qwen3.6-35B-A3B) alongside gemma4_unified in the list of supported text_only=True model types. Committed in the latest push.
… supported models The text_only=True parameter docstring listed only gemma4_unified as a supported model type. Since this PR also adds qwen3_5_moe_vl → qwen3_5_moe_text to _TEXT_ONLY_MODEL_TYPE, update the docstring to reflect the new supported type. Addresses review comment: #445 (comment) Signed-off-by: Copilot <copilot@github.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the missing text-only registry sibling for Qwen3.6-35B-A3B (
Qwen/Qwen3.6-35B-A3B, HFmodel_type=qwen3_5_moewith avision_config).Every other Qwen/Gemma VL family registers a
*_textsibling (gemma3_text,qwen3_vl_text,qwen3_5_vl_text), but the MoE VL did not. As a result:registry.get("qwen3_5_moe_text")raisedKeyError(the VLtext_config's ownmodel_typeisqwen3_5_moe_text); andbuild(..., text_only=True)on the VL checkpoint failed — no_TEXT_ONLY_MODEL_TYPEentry forqwen3_5_moe_vl.How
Qwen35MoECausalLMModel.preprocess_weightsalready strips thelanguage_model.prefix, dropsvisual./MTP keys, and unpacks fused MoE experts, so it consumes the VL text weights directly (no new class needed).qwen3_5_moe_text→Qwen35MoECausalLMModel._TEXT_ONLY_MODEL_TYPE:qwen3_5_moe_vl→qwen3_5_moe_text(+ idempotent self-map).Qwen/Qwen3.6-35B-A3B) and family (qwen) map entries.qwen3_5_moehybrid-MoE config (added to_CHECKER_SKIP_MODELSlike its sibling, since it usesLinearAttention/CausalConvWithStatecustom ops).Verification
text_only=Truerouting andtext_config.model_typenow resolve toQwen35MoECausalLMModel(previously aKeyError).Please review — do not merge yet.