Sync with Microsoft ONNX Runtime - 29072026 - #1234
Merged
Merged
Conversation
…bled (microsoft#29852) ### Description Fixes an undefined-symbol link failure that occurs when **both** `--enable_fp8_qmoe` and `--enable_fp4_qmoe` are enabled together (the FP8+FP4 CUDA build config). The build only breaks in that combined config because it is the only one that instantiates `MoeGemmRunner<__nv_fp8_e4m3, __nv_fp4_e2m1, ...>`. ### Root cause The SM120 compile-time tile-shape gate `are_tile_shapes_supported_sm120()` in `moe_gemm_template_dispatch_tma_ws.h` allowed several extra tile shapes (128×128×64/256, 128×256×64/128, 256×128×64/128) for an FP8 activation type. However, the launcher generator `generate_moe_gemm_tma_ws_sm120_fp4.py` emits **only** the `128×128×128` tile for FP8×FP4 — that is the only shape that fits in shared memory with ≥2 pipeline stages. The dispatch cascade therefore referenced launcher template instantiations that were never generated, producing undefined symbols at link time. ### Fix Restrict the SM120 compile-time dispatch to `128×128×128` when the activation type is `__nv_fp8_e4m3`, matching both the generator and the runtime heuristic: - On SM120, `isValidSM120MOESpecialisation` only supports FP4×FP4 and FP8×FP4, so `DataType == __nv_fp8_e4m3` always means FP8×FP4. - The runtime config heuristic `get_candidate_configs_sm120()` only ever selects `CtaShape128x128x128B`. So the prune causes **no functional loss** — the removed tiles were never generated nor selected. FP4×FP4 tiles are unchanged. ### Validation Object-level verification on `build/cu130_fp8_fp4` (`nm -Cu` on `moe_gemm_kernels_fp8_fp4.cu.o`): - **Before:** 12 undefined SM120 FP8×FP4 launcher references (the extra tiles) → link failure. - **After:** only 2 remaining references, both `<128,128,128>` (fp16 + bf16). Both are defined (weak) by the generated launcher objects `moe_gemm_tma_ws_sm120_fp4_fp8_{fp16,bf16}_m128_n128_k128.generated.cu.o`, so the link resolves. ### Motivation and Context Unblocks building ONNX Runtime with FP8 and FP4 QMoE enabled simultaneously. Delivered as a standalone change independent of the FP8/FP4 QMoE feature PRs.
…oft#29866) ### Description The NuGet native packages shipped `onnxruntime_experimental_c_api.h` but not `onnxruntime_experimental_c_api.inc`, which the header `#include`s directly — causing a fatal compile error for any C/C++ consumer. **`tools/nuget/generate_nuspec_for_native_nuget.py`** - Add a `onnxruntime_*.inc` glob entry alongside the existing `onnxruntime_*.h` glob so all `.inc` files from `include/onnxruntime/core/session/` are packaged. **`tools/nuget/validate_package.py`** - Add `onnxruntime_experimental_c_api.h` and `onnxruntime_experimental_c_api.inc` to `gpu_related_header_files`, `dmlep_related_header_files`, and `training_related_header_files` so CI validation catches this class of regression. ### Motivation and Context `onnxruntime_experimental_c_api.h` ends with `#include "onnxruntime_experimental_c_api.inc"`. The NuGet packaging script used a `*.h` glob that by construction excludes `.inc` files, so the included file was never shipped. The CMake install rules already handle both files correctly; this aligns NuGet packaging with that behavior. Affected packages: all native NuGet packages (`Microsoft.ML.OnnxRuntime`, `.Gpu.Windows`, `.Gpu.Linux`, etc.). <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes microsoft#29865 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
…rosoft#29880) ### Description `import onnxruntime` segfaults during `dlopen` of `onnxruntime_pybind11_state.so` on Linux. This removes the global initializer in `onnxruntime_pybind_state.cc` that eagerly calls `Env::Default()`, and resolves the platform `Env` on first use at its two call sites instead. ### Motivation and Context The module has a namespace-scope dynamic initializer: ```cpp static Env& platform_env = Env::Default(); ``` Since POSIX telemetry landed (microsoft#27379), `Env::Default()` constructs `PosixEnv`, whose `PosixTelemetry` member initializes the 1DS SDK in its constructor. That path reads `defaultRuntimeConfig`, a namespace-scope `static ILogConfiguration` defined in the 1DS SDK's `RuntimeConfig_Default.hpp`, which lives in a **different translation unit of the same shared library**. Dynamic initialization order across translation units is unspecified, and the pybind TU's initializer runs first. `Variant::merge_map` therefore iterates a still zero-initialized `std::map`: `_M_node_count == 0`, but `_M_header._M_left` is `nullptr` rather than self-pointing, so `begin() != end()` and the loop dereferences null. Textbook static initialization order fiasco. Backtrace (Release build relinked without `--strip-all` to recover symbols): ``` #0 std::map<..., Variant>::lower_bound stl_map.h:1307 #1 std::map<..., Variant>::operator[] stl_map.h:509 #2 Variant::merge_map VariantType.hpp:508 #3 RuntimeConfig_Default::RuntimeConfig_Default RuntimeConfig_Default.hpp:97 #4 LogManagerImpl::LogManagerImpl LogManagerImpl.cpp:183 #5 LogManagerFactory::Create LogManagerFactory.cpp:36 #6 LogManagerFactory::lease #7 LogManagerFactory::Get LogManagerFactory.hpp:71 #8 LogManagerProvider::Get LogManagerProvider.cpp:16 #9 onnxruntime::PosixTelemetry::Initialize() #10 onnxruntime::PosixTelemetry::PosixTelemetry() #11 onnxruntime::(anonymous namespace)::PosixEnv::PosixEnv() #12 onnxruntime::Env::Default() #13 _GLOBAL__sub_I_onnxruntime_pybind_state.cc #14 call_init elf/dl-init.c:74 ... #21 _dl_open elf/dl-open.c:905 ``` The crash is independent of `LD_LIBRARY_PATH` and `CUDA_VISIBLE_DEVICES` — it happens before any ORT runtime code runs. `libonnxruntime.so` is unaffected because it contains no global initializer that reaches `Env::Default()`, which is why C/C++ and onnxruntime-genai consumers do not see it. Both remaining uses of `platform_env` are inside pybind lambdas that run long after load, so calling `Env::Default()` there is safe. This also removes the now-stale `TODO: we may delay-init this variable` and a pre-existing `#pragma warning(push)` that should have been `pop`. Note for follow-up: `Env::Default()` is now unsafe to call from any dynamic initializer. This was the only such call site in the tree, but hardening `PosixTelemetry` to defer SDK initialization out of its constructor would remove the hazard entirely. ### Tests Verified on a Linux CUDA 13 Release build (`onnxruntime_USE_TELEMETRY` enabled): - `dlopen` of `onnxruntime_pybind11_state.so` succeeds (previously SIGSEGV). - `import onnxruntime` reports the version and `['CUDAExecutionProvider', 'CPUExecutionProvider']`. - `onnxruntime.enable_telemetry_events()` / `disable_telemetry_events()` — the two call sites changed here — work. - CPU and CUDA inference sessions produce correct results. - Reproduced and verified with `LD_LIBRARY_PATH` unset and `CUDA_VISIBLE_DEVICES` empty.
### Description Extends the WebGPU EP `Add` operator to accept `tensor(int64)` inputs by converting its registration from a static macro to the same conditional factory-function pattern already used by `Sub` and `Equal` (microsoft#29392) ### Motivation and Context Int64 `Add` nodes (e.g. token-position arithmetic in LLMs, [mobileclip_s0 text model](https://huggingface.co/Xenova/mobileclip_s0/blob/main/onnx/text_model_fp16.onnx)) fail kernel lookup in the WebGPU EP. The shader already handles int64 generically for all binary ops; only the kernel registration was missing the int64 type constraint for `Add`. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
…soft#29903) ### Description Adds `docs/MLAS_ISA_Validation.md`: how to exercise a chosen MLAS dispatch tier with Intel SDE (chip flag to tier mapping), and how to prove the ISA kernel actually executed rather than the fallback passing in its place, using the instruction histogram (`-mix`) with its two pitfalls spelled out (the `$static-counts` sections count unexecuted disassembly, and running with no chip flag exposes every feature rather than a native baseline). Links from the MLAS README's testing section. ### Motivation and Context microsoft#29862 measured that several dispatch tiers (the AVX-VNNI-INT8 int8 GEMMs, the AMX U8S8 path, the AVX-NE-CONVERT cast) pass the suite identically whether or not the ISA kernel ran, and no current CI machine reaches them. This writes down the manual workflow so it is reproducible by anyone; the pipeline-job half of that issue's proposal is unchanged and separate.
…#29801) `GatherBlockQuantized` previously passed its quantized `data` and `zero_point` initializers straight to `dequantizeLinear` via `GetOperand`. When 4-bit data is stored bit-packed in a `uint8` tensor (2 nibbles per byte, e.g. Qwen3.5's `embed_tokens`), the declared `uint8` shape is half the logical size, so `dequantizeLinear` received a `scale` and `zero_point` with mismatched shapes and failed with "The shape of scale and zero point must be the same." Re-register the uint8-packed `data` and `zero_point` as WebNN `uint4` constants with their logical (unpacked) shapes — a zero-copy relabel of the same byte buffer — and skip their default `uint8` registration. Now `data`, `scale` and `zero_point` are dimensionally consistent for the blockwise dequant.
### Description
`ConstSessionImpl::GetOverridableInitializerNames()` was leaking the
allocator-owned `char*` buffer after copying it into `std::string`,
unlike the sibling functions `GetInputNames()` and `GetOutputNames()`
which correctly call `allocator.Free(name)`.
**Fix:** Add the missing `allocator.Free(name)` after `emplace_back`:
```cpp
// Before
for (size_t i = 0; i < num_initializers; ++i) {
char* name;
ThrowOnError(GetApi().SessionGetOverridableInitializerName(this->p_, i, allocator, &name));
initializer_names.emplace_back(name); // name copied, but buffer never freed
}
// After
for (size_t i = 0; i < num_initializers; ++i) {
char* name;
ThrowOnError(GetApi().SessionGetOverridableInitializerName(this->p_, i, allocator, &name));
initializer_names.emplace_back(name);
allocator.Free(name); // matches pattern in GetInputNames() / GetOutputNames()
}
```
### Motivation and Context
Every call to `GetOverridableInitializerNames()` on a model with
overridable initializers leaked one heap allocation per initializer.
Reproducible by calling the function repeatedly and observing
monotonically increasing resident memory.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>
… of experts (microsoft#29907) ### Description <!-- Describe your changes. --> Fix MoE security issue when the attribute k is larger than the number of experts (CPU contrib_ops) ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
…crosoft#29901) ### Description microsoft#29791 and microsoft#29864 gave the CompInt8 compute path a direct fp16 C output: each worker runs the kernel into a small per thread fp32 scratch and converts its strip, so a fp16 `MatMulNBits` never allocates a full `M x N` fp32 copy of the result. The CompFp32 path still did: a 4 bit fp16 model at `accuracy_level` 0, 1, 2 or 3, which includes the schema default of 0, computed into a full fp32 output buffer and then ran a separate conversion pass over it. This closes that hole for the 4 bit CompFp32 path. `SQ4BitGemm_CompFp32` gains the same strip conversion the CompInt8 wrappers use, in both of its shapes: the `M == 1` GEMV loop converts each 128 column chunk after the kernel call, and the `M > 1` loop runs its dequant strip SGEMM into the scratch and converts after the rows of a strip are done, bias included. Since both loops fully finish a strip before moving on (the SGEMM runs with `ZeroMode` over the whole `K`), the conversion slots in without touching the compute structure, and the fp16 values are bitwise identical to converting the fp32 result. The shared helpers move above `SQ4BitGemm_CompFp32` and lose the misleading `CompInt8` prefix (`StripCToFp16`, `GetStripCFp16Scratch`) since they now serve both compute types. `MlasQNBitGemmFp16DirectCOutputSupported` takes the compute type so the operator can ask per path; the CompInt8 answer is unchanged, and CompFp32 answers true for 4 bit wherever its kernels exist. The operator side is the same three lines as before, now also taken when `compute_type_` is `SQNBIT_CompFp32`: skip the fp32 result buffer, set `CFp16`, skip the final conversion pass. ### Motivation and Context Follow up to microsoft#29791/microsoft#29864, which noted the C epilogue is portable. `accuracy_level=0` is what an export gets when the attribute is not set at all, so this is the default path for a 4 bit fp16 model on x64, and it was the last `MatMulNBits` configuration still paying the `M x N` fp32 output buffer plus a full extra conversion pass at compute time. Validation: a new MLAS test (`test_sqnbitgemm_fp16_c_fp32path.cpp`) checks the fp16 output against the fp32 result converted with `MlasConvertFloatToHalfBuffer` for byte equality across BlkLen 16/32/64/128, both loop shapes (`M == 1` and `M > 1`), column remainders against both the 128 and 32 column chunkings, symmetric and asymmetric B, bias on and off, and with and without a thread pool; all pass. An AddressSanitizer build of the full suite is clean. The existing fp16 operator tests (`MatMulNBits.Float16_4b_Accuracy0`) run through the new branches, verified with temporary probes on both branches during development and then removed. Full MLAS suite green.
…rosoft#29709) ### Description This PR builds on [fp16-split/02-mlas-half-api-squashed](microsoft#28786) by connecting its FP16 MLAS APIs and KleidiAI backends to the CPU Execution Provider. ```markdown It: - Registers CPU `MLFloat16` kernels for: - `MatMul` opsets 1–8, 9–12, and 13+ - `Gemm` opsets 7–8, 9–10, 11–12, and 13+ - Adds a dedicated FP16 `MatMul` implementation which: - Preserves ONNX broadcasting and batched execution through `MatMulComputeHelper`. - Handles empty outputs and zero-K inputs. - Uses the existing HGEMM path for eligible small, unpacked matrices. - Otherwise dispatches through `MlasHalfGemmBatch`. - Carries the configured MLAS backend selector through every batch entry. - Pre-packs constant two-dimensional B matrices into the selected backend’s native layout when supported. - Keeps backend-native FP16 MatMul packs owned by the kernel. These layouts depend on the active backend, so they are not placed in the cross-session shared-prepack cache without suitable layout metadata. - Adds an explicit kernel-owned packed-weights marker to `PrePackedWeights`. Session initialization continues to reject broken kernels which report successful prepacking without either providing cacheable buffers or explicitly identifying a valid kernel-owned pack. - Extends FP16 `Gemm` to use `MlasHalfGemmBatch` for supported no-transpose, `alpha == 1` cases: - No effective bias, including a missing C input or `beta == 0`. - `beta == 1` with an MLAS-compatible N-element bias. - Unsupported transpose, scaling, or bias-broadcast combinations retain the existing Eigen fallback. - Adds an MLAS HalfConv fast path to the existing FP16 CPU Conv implementation: - Uses `MlasHalfConvPrepare` to determine runtime and backend eligibility for two-dimensional convolutions without a fused Sum input. - Runs eligible convolutions through `MlasHalfConv`. - Supports NCHW and NHWC layouts and forwards padding, stride, dilation, activation, and workspace information. - Pre-packs eligible constant filters using `MlasHalfConvPackWeightsAndBias`. - Includes a compatible constant bias in the native pack when safe. - Falls back to the existing reordered-weight/im2col plus HalfGemm implementation when HalfConv is unavailable or rejects the configuration. - Keeps combined HalfConv weight-and-bias packs session-local because their contents depend on the bias while the existing shared-prepack key identifies only W. Generic Conv weight packs remain shareable across sessions. - Centralizes HalfGemm backend-selector handling in the public MLAS dispatcher and updates the affected MatMul, Gemm, Conv, and MoE call sites. - Updates the CPU operator documentation to advertise FP16 support for `MatMul` and `Gemm`. Test coverage includes: - FP16 Gemm with no C input and with `beta == 0`. - FP16 MatMul with constant and dynamic B inputs. - MatMul backend-native B prepacking. - Kernel-owned MatMul packs remaining intentionally unshared across sessions. - Session-state validation for broken prepacking implementations. - HalfConv-eligible NCHW and NHWC convolutions. - HalfConv execution with and without bias. - Correct fallback behavior when KleidiAI is explicitly disabled. - Safe sharing of generic Conv prepacked weights while keeping bias-dependent HalfConv packs local. - Updated HalfGemm selector and backend-native packed-B contract coverage. ``` ### Motivation and Context This is pr 3 of 5 as per microsoft#28786 splitting suggestion The preceding PR introduces the MLAS primitives for FP16 HalfGemm, backend-native B packing, and HalfConv, but standard ONNX CPU operators do not yet consume them. This PR connects those primitives to CPU `MatMul`, `Gemm`, and `Conv`, making native FP16 execution and KleidiAI acceleration available to the operator layer while retaining portable fallback behavior for unsupported hardware, shapes, attributes, or backend configurations. The prepacking changes avoid repeatedly packing constant weights while preventing backend-specific or bias-dependent layouts from being incorrectly reused through the shared prepack cache. This PR is limited to CPU kernel registration and execution. The follow-up InsertCastTransformer PR introduces the opt-in policy and shape heuristics that decide when graph optimization should preserve FP16 CPU nodes. Apple ARM64 build enablement is also handled separately. --------- Signed-off-by: Cathal Lawlor <cathal.lawlor@arm.com> Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> Co-authored-by: Cathal Lawlor <cathal.lawlor@arm.com>
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
July 28, 2026 20:35
hdharpure9922
self-requested a review
July 29, 2026 03:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.