Fix SM120 FP8xFP4 MoE undefined symbol when both FP8 and FP4 QMoE enabled - #29852
Merged
Conversation
tianleiwu
enabled auto-merge (squash)
July 27, 2026 22:29
jiafatom
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes an undefined-symbol link failure that occurs when both
--enable_fp8_qmoeand--enable_fp4_qmoeare enabled together (the FP8+FP4 CUDA build config). The build only breaks in that combined config because it is the only one that instantiatesMoeGemmRunner<__nv_fp8_e4m3, __nv_fp4_e2m1, ...>.Root cause
The SM120 compile-time tile-shape gate
are_tile_shapes_supported_sm120()inmoe_gemm_template_dispatch_tma_ws.hallowed several extra tile shapes (128×128×64/256, 128×256×64/128, 256×128×64/128) for an FP8 activation type. However, the launcher generatorgenerate_moe_gemm_tma_ws_sm120_fp4.pyemits only the128×128×128tile for FP8×FP4 — that is the only shape that fits in shared memory with ≥2 pipeline stages. The dispatch cascade therefore referenced launcher template instantiations that were never generated, producing undefined symbols at link time.Fix
Restrict the SM120 compile-time dispatch to
128×128×128when the activation type is__nv_fp8_e4m3, matching both the generator and the runtime heuristic:isValidSM120MOESpecialisationonly supports FP4×FP4 and FP8×FP4, soDataType == __nv_fp8_e4m3always means FP8×FP4.get_candidate_configs_sm120()only ever selectsCtaShape128x128x128B.So the prune causes no functional loss — the removed tiles were never generated nor selected. FP4×FP4 tiles are unchanged.
Validation
Object-level verification on
build/cu130_fp8_fp4(nm -Cuonmoe_gemm_kernels_fp8_fp4.cu.o):<128,128,128>(fp16 + bf16). Both are defined (weak) by the generated launcher objectsmoe_gemm_tma_ws_sm120_fp4_fp8_{fp16,bf16}_m128_n128_k128.generated.cu.o, so the link resolves.Motivation and Context
Unblocks building ONNX Runtime with FP8 and FP4 QMoE enabled simultaneously. Delivered as a standalone change independent of the FP8/FP4 QMoE feature PRs.