Skip to content

Fix SM120 FP8xFP4 MoE undefined symbol when both FP8 and FP4 QMoE enabled - #29852

Merged
tianleiwu merged 1 commit into
mainfrom
tlwu/20260723/fix_sm120_fp8xfp4_moe_undef
Jul 27, 2026
Merged

Fix SM120 FP8xFP4 MoE undefined symbol when both FP8 and FP4 QMoE enabled#29852
tianleiwu merged 1 commit into
mainfrom
tlwu/20260723/fix_sm120_fp8xfp4_moe_undef

Conversation

@tianleiwu

Copy link
Copy Markdown
Contributor

Description

Fixes an undefined-symbol link failure that occurs when both --enable_fp8_qmoe and --enable_fp4_qmoe are enabled together (the FP8+FP4 CUDA build config). The build only breaks in that combined config because it is the only one that instantiates MoeGemmRunner<__nv_fp8_e4m3, __nv_fp4_e2m1, ...>.

Root cause

The SM120 compile-time tile-shape gate are_tile_shapes_supported_sm120() in moe_gemm_template_dispatch_tma_ws.h allowed several extra tile shapes (128×128×64/256, 128×256×64/128, 256×128×64/128) for an FP8 activation type. However, the launcher generator generate_moe_gemm_tma_ws_sm120_fp4.py emits only the 128×128×128 tile for FP8×FP4 — that is the only shape that fits in shared memory with ≥2 pipeline stages. The dispatch cascade therefore referenced launcher template instantiations that were never generated, producing undefined symbols at link time.

Fix

Restrict the SM120 compile-time dispatch to 128×128×128 when the activation type is __nv_fp8_e4m3, matching both the generator and the runtime heuristic:

  • On SM120, isValidSM120MOESpecialisation only supports FP4×FP4 and FP8×FP4, so DataType == __nv_fp8_e4m3 always means FP8×FP4.
  • The runtime config heuristic get_candidate_configs_sm120() only ever selects CtaShape128x128x128B.

So the prune causes no functional loss — the removed tiles were never generated nor selected. FP4×FP4 tiles are unchanged.

Validation

Object-level verification on build/cu130_fp8_fp4 (nm -Cu on moe_gemm_kernels_fp8_fp4.cu.o):

  • Before: 12 undefined SM120 FP8×FP4 launcher references (the extra tiles) → link failure.
  • After: only 2 remaining references, both <128,128,128> (fp16 + bf16). Both are defined (weak) by the generated launcher objects moe_gemm_tma_ws_sm120_fp4_fp8_{fp16,bf16}_m128_n128_k128.generated.cu.o, so the link resolves.

Motivation and Context

Unblocks building ONNX Runtime with FP8 and FP4 QMoE enabled simultaneously. Delivered as a standalone change independent of the FP8/FP4 QMoE feature PRs.

@tianleiwu
tianleiwu enabled auto-merge (squash) July 27, 2026 22:29
@tianleiwu
tianleiwu merged commit f665014 into main Jul 27, 2026
97 of 102 checks passed
@tianleiwu
tianleiwu deleted the tlwu/20260723/fix_sm120_fp8xfp4_moe_undef branch July 27, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants