Sync with Microsoft ONNX Runtime - 31072026 - #1236
Merged
Merged
Conversation
### Description <!-- Describe your changes. --> Fix TensorScatter CPU security issue ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
…icrosoft#29919) ## Description At decode (one token per sequence) the MoE/QMoE path launches a standalone softmax/top-k kernel and then a second single-block kernel that builds the expert permutation maps. Both are one-block kernels whose cost is launch and drain latency rather than work, so the second launch is nearly pure overhead. This PR merges them. `fusedRoutingBuildExpertMapsSingleTokenKernel` handles the single-token case with one warp: lane `e` owns expert `e`'s logit, warp reductions give the softmax, and `WarpBitonicSortDescending` gives the top-k. For a single token the permutation is just "the k selected experts ordered by expert id", which every lane derives with shuffles, so no CUB block rank and no shared memory are needed. The kernel writes `token_selected_experts`, `token_final_scales` and all three permutation maps. ## Summary of Changes - **Fused routing prologue.** `launchFusedRoutingBuildExpertMaps` replaces the `LaunchSoftmaxTopK` + `fusedBuildExpertMapsSortFirstToken` pair on the supported decode shape. - **Shared softmax/top-k math.** The routing arithmetic must stay bit-identical to `SoftmaxTopKWarpBitonicKernel`, so `SoftmaxScale`, `SafeInvSum`, `TopKNormalizeDenom`, `WarpReduceMax` and `WarpReduceSum` move from `qmoe_kernels.cu` into `topk_warp_sort.cuh`, and both translation units now share one definition. - **Single source of truth for the guard.** `isFusedMoeRoutingSupported()` is host-only and deterministic. The QMoE op uses it to decide whether to skip its own softmax/top-k launch, and `runMoe` re-checks it, so the two can never disagree about who computed the routing. It requires one token, no expert parallelism, and `num_experts <= 32`. The FP4 family is excluded in the op because its decode fast path consumes `expert_indices`/`expert_scales` itself instead of calling `runMoe`. - **Kill switch:** `ORT_DISABLE_MOE_FUSED_ROUTING=1` restores the two-kernel prologue. ## Performance The win scales with how large a launch is relative to the rest of the call, so it is much bigger on a smaller GPU. Isolated QMoE node at the gpt-oss-20b decode shape (hidden 2880, intermediate 2880, 32 experts, top-4, batch 1, fp16, int4), arms interleaved within every pass via the kill switch: **RTX 3060 (sm_86), per-channel int4, 12 passes x 1000 reps** | arm | mean | min | vs fused | |---|---|---|---| | fused routing | **255.6 us** | 250.1 us | — | | `ORT_DISABLE_MOE_FUSED_ROUTING=1` | 277.6 us | 262.7 us | +8.6% | Fused wins **12 of 12** paired passes (median +7.0%). **RTX 3060 (sm_86), block-wise int4 (`block_size=64`), 8 passes x 1000 reps** | arm | mean | vs fused | |---|---|---| | fused routing | **263.7 us** | — | | `ORT_DISABLE_MOE_FUSED_ROUTING=1` | 277.7 us | +5.3% | Fused wins **8 of 8** paired passes (median +5.0%). On an RTX 5060 Ti (sm_120) the same change measured +0.24% end-to-end decode throughput on gpt-oss-20b — inside that harness's noise floor. A tripwire build confirmed the fused path really does engage at decode there, so the sm_120 number is a genuine null result rather than a dead path; the faster GPU simply amortizes the extra launch better. ## Testing On the RTX 3060 (sm_86), with and without `ORT_DISABLE_MOE_FUSED_ROUTING=1`: - `onnxruntime/test/python/transformers/test_qmoe_cuda.py`: 109 passed, 13 skipped - `onnxruntime/test/python/transformers/test_moe_cuda.py`: 48 passed, 15 skipped --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…icrosoft#30514) ### Description Apply `Split-K` on more MatMul parameter combinations on Intel `xe-3lpg` (Panther Lake) GPUs. ### Motivation and Context After updating Dawn to the latest now we can differ the 12Xe Panther Lake iGPUs (`xe-3lpg`) from the 2Xe Wildcat Lake iGPUs (`xe-3lpg-xs`), so we can apply `Split-K` optimization on more MatMul parameter combinations on `xe-3lpg` as it has more EUs. This patch improves 22% of `jina-embeddings-v2-base-code-fp16` and 9% of `efficientnet-lite-f16-demo`.
…29898) ### Description This change integrates the Arm® KleidiAI™ symmetric Q4 SME2 kernels into the existing symmetric-quantized Q4 MatMulNBits path. The preexisting kernels and fallback behavior is preserved and is used wherever SME2 is not available. The accompanying MLAS tests cover symmetric Q4 correctness and verify feature-based kernel selection. ### Motivation and Context The new kernel integrations reduce latency on SME2-equipped hardware. The benchmarks below were run on a Mac Mini with M4 Pro. End-to-end testing using a symmetric Q4 Phi-4 model produced the following result: | Patch | Base | No KleidiAI | Patch vs base | |---:|---:|---:|---:| | 31.416 ms | 46.041 ms | 69.120 ms | **31.8% faster** | The numbers are the median of the mean inference times of 3 benchmark runs, each with 100 timed inferences. Synthetic MatMulNBits shapes representative of common LLM tensors show similar improvement (with `b` denoting quantization block size): | Shape | Patch | Base | No KleidiAI | Patch vs base | |---|---:|---:|---:|---:| | M1 K3072 N3072, b32 | 0.0637 ms | 0.1334 ms | 0.1943 ms | **52.2% faster** | | M1 K3072 N8192, b32 | 0.1693 ms | 0.2990 ms | 0.4503 ms | **43.4% faster** | | M1 K8192 N3072, b32 | 0.1752 ms | 0.3008 ms | 0.4488 ms | **41.7% faster** | | M1 K4096 N4096, b128 | 0.0840 ms | 0.1884 ms | 0.2737 ms | **55.4% faster** | The synthetic shape numbers were obtained as the arithmetic mean of 2 benchmark runs with 200 timed inferences each. The built-in MLAS benchmark across 10 repetitions also shows consistent kernel-level improvement: | Case | Patch | Base | No KleidiAI | |---|---:|---:|---:| | Hidden M1 b32 | 61.84 µs | 110.23 µs | 167.03 µs | | Up M1 b32 | 170.61 µs | 297.93 µs | 447.89 µs | | Common M1 b128 | 81.62 µs | 187.73 µs | 272.33 µs | --------- Signed-off-by: Martin Klacer <martin.klacer@arm.com>
### Description Add PRelu support for the WebGPU EP: kernel implementation plus registrations for opsets 7-8, 9-15 and 16. Opset 6 is not registered, matching the CPU and CUDA EPs. Coverage: the existing ActivationOpTest.PRelu* cases fan out to every registered EP, so they exercise the new kernel on the scalar/no-broadcast and general-broadcast paths in BinaryElementwise, plus the inf/NaN edge cases. Those run at OpTester's default opset (7), so two new float16 cases cover the 9-15 and 16 registrations; the opset-9 case uses a last dim of 4, which is also the only coverage of the vectorized path. Both use float16 because the CPU EP's PRelu is float-only, so they cannot silently pass via CPU fallback. ### Motivation and Context Fixes microsoft#29859 Without it, PRelu falls back to the CPU EP and Memcpy nodes get inserted around it when it sits between WebGPU nodes. That costs a GPU→CPU→GPU round trip per inference, and causes a session-init failure when graph capture is enabled — the scenario named in the issue.
The int64 Clip kernel was registered unconditionally via the static kernel table, unlike every other int64 kernel (Cast, Expand, Equal, Sub, Where, ReduceSum, Reshape, Unsqueeze, Range), which register their int64 variant only when the enable_int64 option is set. As a result an int64 Clip node was claimed by the EP even when int64 support was left disabled. Remove the int64 Clip entries from the static kernel table and register the dedicated ClipInt64 kernels conditionally in RegisterKernels, gated on enable_int64. The int64 Clip WebGPU tests now build the EP with the option enabled. This is an enhancement for landed PR microsoft#29834
### Description The WebNN EP maps ONNX BOOL → WebNN uint8. When WebNN uses WebGPU as its backend, a Gather on a bool tensor (e.g., in [detr-resnet-50](https://huggingface.co/Xenova/detr-resnet-50/blob/main/onnx/model_fp16.onnx)) is lowered to WebGPU Gather on uint8, which previously had no matching kernel. ### Motivation and Context Models such as detr-resnet-50 use Gather on bool tensors. The WebNN EP converts bool to uint8 before dispatching to the WebGPU backend, so the WebGPU Gather kernel must accept uint8 to handle this path end-to-end. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: huningxin <1005673+huningxin@users.noreply.github.com>
…n wasm_inferencing_webgpu_jspi (microsoft#29777) ### Description Extends `cmake/patches/dawn/dawn_parallel_build_fix.patch` with a second hunk and updates the explanatory comment in `onnxruntime_external_deps.cmake`. Two fixes in the patch address the race: **Fix 1 — missing output directory** (existing hunk): adds `cmake -E make_directory` before `cmake -E copy` in `emdawnwebgpu_headers_gen_add` so the destination directory is guaranteed to exist before any copy runs, regardless of parallel job scheduling. **Fix 2 — duplicate custom-command recipe** (new hunk): removes `webgpu_enum_class_bitmasks.h` from `emdawnwebgpu_cpp`'s `HEADERS` list: ```cmake dawn_add_library( emdawnwebgpu_cpp ... HEADERS "${EM_BUILD_GEN_DIR}/include/dawn/webgpu_cpp_print.h" "${EM_BUILD_GEN_DIR}/include/webgpu/webgpu_cpp.h" "${EM_BUILD_GEN_DIR}/include/webgpu/webgpu_cpp_chained_struct.h" - "${EM_BUILD_GEN_DIR}/include/webgpu/webgpu_enum_class_bitmasks.h" DEPENDS emdawnwebgpu_c ) ``` `emdawnwebgpu_cpp` is an `INTERFACE` library, so anything in `HEADERS` goes into `INTERFACE_SOURCES`. When `onnxruntime_providers_webgpu` links to it, CMake propagates the generated file into ORT's directory scope and emits a **second** `cmake -E copy` recipe for the same output path. With `-j32`, both recipes run concurrently → race → copy failure. Removing the entry eliminates the duplicate recipe. The header remains reachable via `${EM_BUILD_GEN_DIR}/include` (already on `emdawnwebgpu_c_include`'s `INTERFACE_INCLUDE_DIRECTORIES`), and build ordering is preserved through the existing `emdawnwebgpu_c → emdawnwebgpu_c_include → emdawnwebgpu_headers_gen` chain. Also removes a stale comment in `cmake/onnxruntime_providers_webgpu.cmake` that described an older, different bug. ### Motivation and Context The `wasm_inferencing_webgpu_jspi` step (CI job `88023093249`) was failing because `webgpu_enum_class_bitmasks.h` was being copied by two parallel make jobs simultaneously. The JSPI build is most susceptible because it runs after a warmed ccache, so all ORT compilation steps finish quickly and many more jobs compete for the Dawn header generation step at the same time. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
…001 (microsoft#29885) ### Description `sqnbitgemm_kernel_avx512vnni.cpp` included the three W2 (2-bit) kernel headers `sqnbitgemm_kernel_avx512_2bit_blklen{32,64,128}.h` — ~2500 lines of fully `MLAS_FORCEINLINE` AVX-512-VNNI intrinsic bodies — on top of the W4/W8 VNNI kernels. The combined TU is large enough that MSVC 19.44 aborts with `fatal error C1001` during Release code generation. This is pure code motion; the dispatch table entry, symbol name, and linkage are unchanged. - **New TU `onnxruntime/core/mlas/lib/sqnbitgemm_kernel_avx512vnni_2bit.cpp`** — holds `sq2bit_avx512::SQ2BitGemmKernel_BlkSum_CompInt8_Avx512Vnni_Dispatch` and the three heavy `blklen*` includes. - **`sqnbitgemm_kernel_avx512vnni.cpp`** — now includes only the declaration-only `sqnbitgemm_kernel_avx512_2bit.h`, which already declared the dispatch symbol, so the `d.SQ2BitGemmKernel_BlkSum_CompInt8` wiring and the `kBlockGroupBlks` `static_assert` are untouched. - **`cmake/onnxruntime_mlas.cmake`** — registers the new source in the Windows x64 list and in `mlas_platform_srcs_avx512vnni`, so it inherits the same `-mfma -mavx512vnni …` flags as its sibling TU. `sqnbitgemm_kernel_avx512.cpp` (non-VNNI) is left alone — it only instantiates the non-VNNI kernel variants and was not reported as failing. If it later hits the same limit, the same split applies. ### Motivation and Context The ICE breaks Windows Release pipeline builds of `onnxruntime_mlas` (surfaced via the WebGPU plugin EP packaging build, but the failing target is generic MLAS). Debug builds are unaffected, and the reported line 541 is EOF, confirming the failure is whole-TU codegen rather than a specific construct. Shrinking the TU sidesteps it without changing generated kernel code. <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes microsoft#29869 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
### Description PR microsoft#27379 introduced a static dependency on shell32.dll. Since shell32.dll depends on user32.dll, this prevents loading onnxruntime.dll in processes that enabled the mitigation PROCESS_MITIGATION_SYSTEM_CALL_DISABLE_POLICY.DisallowWin32kSystemCalls (Win32k lockdown), causing an ERROR_INTERNAL_ERROR (1359). The use of shell32.dll is only for svchost processes, so it's okay to just delay load it. ### Motivation and Context WebNN is migrating ORT graph compilation to a sandboxed process with Win32k lockdown. Dependency on shell32.dll will cause onnxruntime.dll to fail to get loaded in that process.
…ft#29589) ### Description <!-- Describe your changes. --> Use real stream for memory pattern block allocation so that the cleanup of chunks from the arena correctly marks it as unused. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> microsoft#29351
…ft#29910) ## Description This PR addresses follow-up review feedback from recently merged CUDA PRs covering block-scaled FP4/FP8 MatMul and MoE GEMV paths. The update consolidates duplicated CUDA type helpers into a shared header, tightens kernel contracts and comments, and adds edge-case validation tests for FP4 block-scaled MatMul behavior. The goal is to improve maintainability and correctness without changing intended operator semantics. ## Summary of Changes ### CUDA helper consolidation and kernel cleanup | File | Change | |------|--------| | onnxruntime/core/providers/cuda/cu_inc/cuda_type_helper.cuh | Expanded shared CUDA helper utilities (float/bfloat16 conversion helpers, packed vec2 traits, bit_cast helper, FP8 E4M3 raw-byte conversion helpers) and removed anonymous-namespace scoping so helpers are reusable across translation units. | | onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cu | Switched to shared helper usage and removed local duplicate helper patterns; updated call sites to shared conversion and vec2 helper API. | | onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cu | Switched to shared helper usage and updated FP8 GEMV path call sites/comments to use the consolidated helper API. | | onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4_sm120.cu | Switched to shared helper usage and kept SM120 quantization path aligned with consolidated type conversion helpers. | ### FP4/NVFP4 behavior and API cleanups | File | Change | |------|--------| | onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cc | Updated PrePack path and argument handling based on feedback; removed obsolete weight-scale handoff usage in native SM120 launch path. | | onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.h | Clarified launcher contracts/documentation and aligned signatures with current runtime behavior. | ### MoE GEMV and related style/documentation updates | File | Change | |------|--------| | onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv.cu | Applied coding-style consistency updates and readability improvements requested in review. | | onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv.h | Applied coding-style consistency updates requested in review. | | onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv_fp4.cu | Applied coding-style consistency updates requested in review. | | onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv_fp4.h | Applied coding-style consistency updates requested in review. | | onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_kernels.cu | Added clarifying comments around invariants/assumptions raised in review. | | onnxruntime/contrib_ops/cuda/bert/xqa/mha_impl.cuh | Applied targeted const-ordering style alignment for PR-introduced lines. | ### Tests | File | Change | |------|--------| | onnxruntime/test/contrib_ops/matmul_block_scaled_fp4_test.cc | Added edge-case tests covering partial trailing scale block handling, K multiple-of-16 but not 32 path, and zero-K behavior with and without bias. | ## Testing - Build and install: - bash /home/tianlei/git/onnxruntime/.env/cuda130_fp4_bench.sh --build --install - Additional verification for sources excluded by the default build config: - bash /tmp/sm120_check.sh - bash /tmp/test_syntax.sh - GPU functional checks: - python /tmp/verify_fp4_tests.py - Formatting verification: - lintrunner -a --take CLANGFORMAT onnxruntime/core/providers/cuda/cu_inc/cuda_type_helper.cuh onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cu onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cu onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4_sm120.cu ## Motivation and Context In microsoft#29818 and microsoft#29887 and microsoft#29896, reviewer feedback identified duplicate helper logic and opportunities to tighten contracts/comments across the CUDA block-scaled MatMul and MoE GEMV code paths. Consolidating type/packing helpers in one shared CUDA header reduces duplication and future drift, while the added tests lock in edge-case expectations for FP4 block-scaled behavior. ## Checklist - [x] Tests added/updated - [x] No breaking changes - [ ] Documentation updated (not required for this internal CUDA refactor/cleanup)
### Description Adds a new `sliding_window_cache` attribute to `com.microsoft::GroupQueryAttention` so that layers using local (sliding-window) attention can be backed by a **window-sized** KV cache instead of a full-length one. Today `local_window_size` narrows the attention *mask* only — the cache is still allocated for the whole sequence, so a model like gpt-oss pays for `max_length` positions on every layer even though half of its layers can never look back more than 128 tokens. The attribute is opt-in and defaults to `0`, so every existing model keeps the exact current code path. When set to `1`, the operator treats the bound past/present buffer as a fixed-capacity ring of the most recent tokens and does all cache addressing in **cache-relative** coordinates, while RoPE keeps using absolute positions. Because both the causal mask and the local-window mask depend only on the query/key *distance*, translating the cache origin does not change the math. ### Design For a windowed layer with capacity `C`, the invariant maintained is: > the cache holds the `L = min(T, C)` most recent tokens, contiguously, at cache indices `[0, L)`, where `T` is the absolute total sequence length. Appending `S` new tokens onto past length `P`: ``` Lp = min(P, C) # tokens currently resident E = max(0, Lp + S - C) # tokens evicted by this step shift: cache[E .. Lp) -> cache[0 .. Lp-E) write: new tokens -> cache[Lp-E .. Lp-E+S) ``` Two details are worth calling out: * **Everything is derived on device from `seqlens_k`.** `is_first_prompt` cannot be used here: it is computed from `sequence_length == total_sequence_length` (input 6), and onnxruntime-genai binds input 6 to the *allocated buffer width*, not the running total. `seqlens_k` (input 5) is the true running total, so `GetSequenceLengths` emits both absolute lengths (for RoPE) and cache-relative lengths (for cache addressing) plus a per-batch eviction count, all on device. This also keeps the launches CUDA-graph capturable. * **Multi-token steps run against a staging cache.** A naive in-place scheme would require `C >= local_window_size + sequence_length - 1`, coupling the cache size to the prefill chunk size. Instead any step with `S > 1` is seeded into a temporary `C + S` staging cache and the surviving window is copied back afterwards, which reduces the requirement to just `C >= local_window_size`. Decode (`S = 1`) runs in place with a compaction shift of at most one row. ### Summary of Changes **Op schema and docs** | File | Change | |------|--------| | `onnxruntime/core/graph/contrib_ops/bert_defs.cc` | New `sliding_window_cache` INT attribute (default `0`); shape inference keeps `present_*` at the past buffer's own sequence dim instead of growing it to `total_sequence_length`. | | `docs/ContribOperators.md`, `docs/contrib_ops/cuda/gqa.md` | Regenerated op docs plus a new "Sliding Window (Windowed) KV Cache" section covering capacity, cache-relative indexing, RoPE bounds, staging, and constraints. | **Parameters and validation** | File | Change | |------|--------| | `onnxruntime/contrib_ops/cpu/bert/attention_parameters.h` | Adds `is_windowed_kv_cache`, `kv_cache_capacity`, `kv_cache_real_capacity`, `rotary_max_position`. | | `onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h` | `CheckInputs` takes the new attribute; in windowed mode `present_sequence_length = past_sequence_length` (the bound buffer *is* the capacity); validates `C >= local_window_size`, `past_present_share_buffer`, and `local_window_size > 0`; adds `rotary_max_position` so RoPE stays bounded when rotary caches are absent. Rejects `attention_bias` combined with a windowed cache. | | `onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc`, `js/...`, `webgpu/...` | CPU / JS / WebGPU GQA return `NOT_IMPLEMENTED` for `sliding_window_cache=1` rather than silently computing the wrong result. | **CUDA kernel** | File | Change | |------|--------| | `group_query_attention_impl.cu` / `.h` | `GetSequenceLengths` additionally emits `cache_past_seq_lens`, `cache_total_seq_lens` and `evict_counts`; new fused `LaunchCompactKvCache` (in-place eviction shift) and `LaunchCopyKvCacheWindow` (staging seed / copy-back) row-movement kernels — K and V are moved in a single launch, one row-tile per block. | | `group_query_attention_qkv.cuh` | KV append uses the cache-relative offset; RoPE keeps the absolute position. | | `group_query_attention.cc` / `.h`, `attention_data.h` | Allocates the staging cache and compaction scratch, wires the windowed buffers through, and disables the Flash-Attention fast-decode path for windowed caches (its RoPE/cache indexing assumptions do not hold). | **Tests** | File | Change | |------|--------| | `onnxruntime/test/python/transformers/test_gqa.py` | New `TestGQAWindowedKvCache` suite (11 tests) plus `test_windowed_cache_attention_bias_is_rejected`, and a `has_fp8_xqa()` SM>=8.9 guard so the fp8 XQA sliding-window parity tests skip on older GPUs instead of failing. | ### Testing ```bash pytest onnxruntime/test/python/transformers/test_gqa.py -k TestGQAWindowedKvCache pytest onnxruntime/test/python/transformers/test_gqa.py ``` `TestGQAWindowedKvCache` covers: prompt shorter/longer than capacity followed by decode, chunked prefill, eviction boundaries, RoPE, gpt-oss shape with `head_sink`, int8 quantized cache, batch-1 packed head layout, capacity exactly equal to the window size, and the two negative cases (capacity too small, `local_window_size` missing). End-to-end validation on gpt-oss-20b (24 layers, 12 of them sliding with window 128), same weights with and without the attribute: | | GPQA-diamond (198 q, `reasoning=medium`) | KV cache @ `max_length=131072`, int8 KV | |---|--:|--:| | full-length cache | 0.6010 (119/198) | 5138 MiB | | windowed cache | 0.5960 (118/198) | 3090 MiB (**-39.9%**) | Paired McNemar over the 198 questions gives `p = 1.0000` (both correct 102, only-baseline 17, only-windowed 16), i.e. the accuracy difference is not significant. Results are bit-identical to a full-length cache while nothing has been evicted; once the cache origin shifts, flash accumulates the same keys in a different block order, and on an MoE model that rounding difference can flip an expert route — hence the aggregate/paired comparison rather than greedy token parity. Decode cost of the extra row movement is ~1-2% (measured on A100 with CUDA graphs enabled, prompt 512-65536); the copies are launch-bound rather than bandwidth-bound. **Backward compatibility:** `sliding_window_cache` defaults to `0`, which is byte-for-byte the existing code path. No existing model or exported graph changes behavior. ### Motivation and Context Reduce KV cache memory for LLM models that use sliding-window attention (gpt-oss and similar). For gpt-oss, half the layers are sliding-window, so this saves roughly 50% of the KV cache for long contexts such as 128k — measured at ~40% of the total KV allocation on the int8-KV variant above. The corresponding onnxruntime-genai change (model-builder emits `sliding_window` for the CUDA EP, and the KV-cache allocator sizes windowed layers to the window) is a separate PR. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…microsoft#29556) ### Description `GraphTransformerManager::Register` guarded against duplicate registration with a `std::find` over `level_to_transformer_map_[level]` (a container of `std::unique_ptr<GraphTransformer>`) looking for the `transformer` unique_ptr about to be inserted. Since that unique_ptr is not yet in the container and owns a distinct pointer, the comparison can **never** match — the guard is dead code. As a result, a transformer registered with a duplicate **name** at the same level slips through: it is appended a second time to `level_to_transformer_map_[level]` and applied **twice** per optimization pass, and the intended "already registered" error is never returned. ### Fix Replace the dead identity check with a **per-level name** check backed by an **O(1) hash lookup**: each transformer's name is recorded in the existing `transformers_info_` map keyed by `(level, name)`, so a duplicate at the same level is rejected in constant time instead of scanning the transformers already registered at that level (avoiding the quadratic registration cost). This preserves the original per-level intent — the same transformer name at *different* levels remains valid (e.g. `LayerNormFusion` is registered at both Level1 and Level2 by `GenerateTransformers`), which a global-name check would have wrongly rejected. ### Tests Adds `GraphTransformationTests.RegisterDuplicateTransformerNameFailsPerLevel`: - same name, same level → `FAIL` - same name, different level → `OK` Verified locally: the test **fails on the pre-fix code** (`duplicate_status.IsOK()` was `true`, i.e. the duplicate was wrongly accepted) and **passes with the fix**. The full `GraphTransformationTests` suite remains green after the O(1) revision — re-verified on a RelWithDebInfo build (350 passed, 15 CUDA/WebGpu skipped, 0 failed), including the layernorm tests that register the same fusion at Level1 and Level2. ### Motivation and Context Fixes the design-audit **F10** finding — a non-functional safeguard that silently allowed duplicate transformer registration and double-application. --------- Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>
…29558) ### Description Extends error-path / negative test coverage for `InferenceSession` model ingestion (continuation of the audit's "thin error-path coverage" finding). Adds two tests to `inference_session_test.cc`: - **`LoadModelTwiceReturnsError`** — a second `Load()` on a session that already holds a model must be rejected with `MODEL_LOADED` ("This session already contains a loaded model"). - **`LoadInvalidGraphReturnsError`** — a model whose only node consumes an input that is never defined (not a graph input, initializer, or another node's output) is constructed via the ONNX proto API, serialized, and loaded. Graph `Resolve` must reject it gracefully (error at `Load` or `Initialize`) rather than crash or silently accept it. ### Motivation and Context Failure/validation paths are under-tested relative to the numeric happy-path bulk. These cover the double-load guard and the graph-resolve rejection path for a malformed graph. ### Verification Built `onnxruntime_test_all` (Release, CPU) and ran both tests locally — both pass: ``` [ OK ] InferenceSessionTests.LoadModelTwiceReturnsError [ OK ] InferenceSessionTests.LoadInvalidGraphReturnsError [ PASSED ] 2 tests. ``` Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>
### Description Adds verbose trace logging around `GraphTransformerManager::ApplyTransformers`. The manager now logs: - when no transformers are registered for a requested level; - how many transformers will run for a level and the maximum number of steps; - step start/end; - each transformer invocation; - when a `ShouldOnlyApplyOnce()` transformer is skipped on a later step; - early stop when a step does not modify the graph. This complements the existing per-transformer `GraphTransformer::Apply` log (`GraphTransformer <name> modified: ...`) with manager-level pass/iteration context. ### Motivation and Context Graph optimizer changes are a frequent source of performance differences. When a fusion does or does not happen, it is useful to see not just an individual transformer's result, but also the manager-level iteration flow: which level ran, which step ran, which transformer was invoked, and why iteration stopped. This makes future perf investigations safer and easier to localize without changing optimization behavior. ### Testing Added `GraphTransformationTests.GraphTransformerManagerEmitsVerboseTrace`, which uses a no-op transformer and `CapturingSink` to assert the manager-level trace messages are emitted. Built `onnxruntime_test_all` (Release, Windows) and ran: ```powershell .\onnxruntime_test_all.exe --gtest_filter="GraphTransformationTests.GraphTransformerManagerEmitsVerboseTrace" ``` Result: ```text [==========] 1 test from 1 test suite ran. [ PASSED ] 1 test. ``` Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>
### Description - Replace the two function-local `InlinedHashMap` tables used by `PropagateCastOps` with one `constexpr` operator argument descriptor table. - Derive each operator's active input/output count from its initializer via `ArgIndexSet`, so a count can never disagree with the indices actually listed, and listing more indices than the capacity fails to compile on the offending table row. - Bound every argument index by its capacity (`kMaxRelevantInputs` / `kMaxRelevantOutputs`), which also gives the tests a finite index range to cover exhaustively. - Validate sorted unique operator names, non-empty argument sets, in-range indices, and duplicate indices with `consteval` checks, split across separately named predicates so a failure names the rule that broke. - Add an independent exhaustive compile-time truth table for all seven operators, plus runtime tests for the unlisted-operator fallback, the `-1` not-found sentinel, and out-of-range indices. ### Motivation and Context The relevant input/output policy is fixed program metadata, but it was represented by two separately initialized runtime hash maps. Consolidating it removes duplicate operator keys and avoids runtime hash-table initialization and allocation. Unknown operators still use the existing fallback where every input/output is relevant. ### What the compile-time checks do and do not cover The checks verify that the table is internally consistent and that it agrees with the independently maintained truth table in the test. They do not verify the policy against the ONNX operator definitions: an edit applied consistently to both the header and the test truth table will still compile. The policy itself is unchanged by this PR. Specifically caught at compile time: | Mistake | Caught by | | --- | --- | | Count disagreeing with the listed indices | unrepresentable, the count is deduced | | More indices than the capacity | the table row itself | | Index >= capacity, negative, or duplicated | `AreAllArgIndicesValid` | | Empty argument set | `AreAllArgIndicesValid` | | Operator out of order, duplicated, or empty-named | `AreOpTypesSortedAndUnique` | | Table disagreeing with the expected truth table | the test's `static_assert`s | | Capacity grown without widening the truth table | the test's `RelevanceRow` | ### Behavior Unchanged. The previous implementation stored zero-padded `std::array<int, 3>` rows and scanned all three slots, so equivalence depended on every single-index row happening to hold `0`. The count-bounded scan no longer depends on that coincidence. Verified with a harness that reconstructs the original `InlinedHashMap` implementation and `static_assert`s agreement between old and new for all seven operators plus unlisted ones, over indices -3 to 10. That range covers the `-1` that `IndexOfNodeInput` and `IndexOfNodeOutput` return when a `NodeArg` does not belong to the node. Seven mutations of the table were each confirmed to fail compilation. ### Measurement Windows RelWithDebInfo, MSVC 19.51, touched-file rebuild of `propagate_cast_ops.cc`: | Artifact | Before | After | Delta | | --- | ---: | ---: | ---: | | `propagate_cast_ops.obj` | 7,196,523 B | 6,353,108 B | -843,415 B | | `onnxruntime_optimizer.lib` | 255,830,720 B | 254,651,324 B | -1,179,396 B | These numbers were taken on the first revision of this branch and have not been re-measured since the review follow-up commit, which changed the header. They should be refreshed before they are relied on. Build timing is noisy and this PR does not claim a compile-time improvement. The single baseline rebuild was 10.83 s; post-change samples were 13.45 s, 12.37 s, and 12.06 s. One baseline sample is not enough to separate noise from a regression. This PR also makes no runtime performance claim. The lookup is now a linear scan of seven `std::string_view` comparisons rather than a hash probe, and the common case is an operator absent from the table, which costs all seven comparisons. That has not been benchmarked in either direction. ### Testing - `cmake --build build/Windows/RelWithDebInfo --config RelWithDebInfo --target onnxruntime_test_all --parallel 8` - `onnxruntime_test_all.exe --gtest_filter=PropagateCastOpsMetadataTest.*:GraphTransformationTests.PropagateCastOpsTests*` from the test output directory: 2 tests from 2 suites passed. - `lintrunner -a` on all three changed files: no lint issues. The three items above were run on the first revision of this branch. The review follow-up commit has so far been verified with standalone MSVC 19.51 compilation of the header and the test at `/std:c++20 /W4 /permissive-`, the old-versus-new equivalence harness, the seven mutation cases, and `clang-format` 20.1.8, the pinned version, reporting clean. The full `onnxruntime_test_all` run needs to be repeated. --------- Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
July 30, 2026 20:35
hdharpure9922
self-requested a review
July 31, 2026 08:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.