Sync with Microsoft ONNX Runtime - 26/09/2025 - #818
Merged
Conversation
…or (microsoft#26016) Compiling C application dependent on onnxruntime with flag -Werror=strict-prototypes results in an error because the function declarations without parameters are considered to have an implicit int parameter list, which is not allowed in strict prototype mode. To fix this, we need to explicitly specify void in the parameter list of functions that do not take any arguments. ### Description <!-- Describe your changes. --> ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> When compiling a C program that depends on ONNX Runtime with the `-Werror=strict-prototypes` flag enabled, the build fails due to function declarations in `onnxruntime_c_api.h` that are not strict prototypes. For example, in the **onnxruntime-inference-examples** project, the following errors are reported: ``` ~/install/onnxruntime/include/onnxruntime_c_api.h:4929:3: error: function declaration isn't a prototype [-Werror=strict-prototypes] const OrtModelEditorApi*(ORT_API_CALL* GetModelEditorApi)(); ^~~~~ ~/install/onnxruntime/include/onnxruntime_c_api.h:4987:3: error: function declaration isn't a prototype [-Werror=strict-prototypes] const OrtCompileApi*(ORT_API_CALL* GetCompileApi)(); ^~~~~ ~/install/onnxruntime/include/onnxruntime_c_api.h:5253:3: error: function declaration isn't a prototype [-Werror=strict-prototypes] const OrtEpApi*(ORT_API_CALL* GetEpApi)(); ^~~~~ ``` This occurs because function declarations with empty parentheses are not considered proper prototypes in C, and strict prototype checking treats this as an error. Our project uses ONNX Runtime and enforces -Werror=strict-prototypes in our build system. It would be very helpful if these declarations in ONNX Runtime could be updated to use (void) for functions that take no arguments, so that strict prototype builds succeed without errors. Signed-off-by: Lifang Zhang <lifang.zhang@nokia.com>
…evices (microsoft#26028) ### Description Adds the Java bits mirroring microsoft#25878, and renames a few things in microsoft#25131 for uniformity with the other APIs. ### Motivation and Context Java API parity.
### Description <!-- Describe your changes. --> This pull request introduces several enhancements to ONNX Runtime's Python and C++ APIs, focusing on improved device and memory information handling, synchronization stream support, and tensor copy functionality. It adds new Python bindings for device/memory types, exposes more detailed session input/output metadata, and provides a Python-accessible tensor copy API. The changes also refactor and extend the C++ API for better stream and memory info management. Key changes include: ### Device and Memory Information Enhancements * Added Python bindings for `OrtMemoryInfoDeviceType`, `OrtDeviceMemoryType`, and expanded `OrtDevice` to expose the memory type via a new `mem_type` method. The `OrtMemoryInfo` Python class now supports both legacy and new V2 constructors and exposes additional properties such as device memory type and vendor ID. [[1]](diffhunk://#diff-c46fc0e05521f706449c04aed599ac0229012c007a78b584519e71a57601d63eR1801-R1810) [[2]](diffhunk://#diff-c46fc0e05521f706449c04aed599ac0229012c007a78b584519e71a57601d63eR1839) [[3]](diffhunk://#diff-c46fc0e05521f706449c04aed599ac0229012c007a78b584519e71a57601d63eL1941-R2005) * Extended the Python `InferenceSession` object to provide access to input/output `OrtMemoryInfo` and `OrtEpDevice` objects through new properties and methods. [[1]](diffhunk://#diff-c46fc0e05521f706449c04aed599ac0229012c007a78b584519e71a57601d63eR2702-R2729) [[2]](diffhunk://#diff-f0e8ba8cb8cb07b51b3be675bf62cec07e2eae1461341ce5801d33a57c8f57fdR202-R213) [[3]](diffhunk://#diff-f0e8ba8cb8cb07b51b3be675bf62cec07e2eae1461341ce5801d33a57c8f57fdR591-R593) [[4]](diffhunk://#diff-f0e8ba8cb8cb07b51b3be675bf62cec07e2eae1461341ce5801d33a57c8f57fdR607-R609) ### Synchronization Stream and Execution Provider Device Support * Introduced Python bindings for `OrtSyncStream`, including creation via `OrtEpDevice.create_sync_stream()` and retrieval of device-specific `OrtMemoryInfo` via `OrtEpDevice.memory_info()`. [[1]](diffhunk://#diff-c46fc0e05521f706449c04aed599ac0229012c007a78b584519e71a57601d63eR1890-R1938) [[2]](diffhunk://#diff-44e70fbe60cba71c94f1a46ec2b1facaa8e9475232dad6df5ecbea301e76d475R34-R44) * Refactored the C++ API to generalize `SyncStream` handling, allowing for unowned streams and improved type safety. [[1]](diffhunk://#diff-17f64e8b38fcdcd25e90abcabeec4b420956b15fe63868a5d0b270c376bde209L1066-R1084) [[2]](diffhunk://#diff-cc93f5f9d8078d3d3af14c9bb4c0c59e25a99f3ec75d7772ea20111ed7eb6ddeL672-R677) ### Tensor Copy Functionality * Added a new Python-level `copy_tensors` function and corresponding C++ binding, enabling efficient copying of tensor data between `OrtValue` objects, optionally using a synchronization stream. [[1]](diffhunk://#diff-c46fc0e05521f706449c04aed599ac0229012c007a78b584519e71a57601d63eR1588-R1599) [[2]](diffhunk://#diff-f0e8ba8cb8cb07b51b3be675bf62cec07e2eae1461341ce5801d33a57c8f57fdR1155-R1163) [[3]](diffhunk://#diff-44e70fbe60cba71c94f1a46ec2b1facaa8e9475232dad6df5ecbea301e76d475R84) ### Miscellaneous Improvements and Fixes * Changed the return type of the `OrtValue.data_ptr` method in the Python binding from `int64_t` to `uintptr_t` for better cross-platform compatibility. [[1]](diffhunk://#diff-666c9002698d1bbd4215237231e5be98d7b33e5054f018dce952407027bd0473L336-R336) [[2]](diffhunk://#diff-666c9002698d1bbd4215237231e5be98d7b33e5054f018dce952407027bd0473L347-R347) * Minor improvements to error messages and device type handling in the Python API (e.g., for `OrtDevice`). [[1]](diffhunk://#diff-f0e8ba8cb8cb07b51b3be675bf62cec07e2eae1461341ce5801d33a57c8f57fdR1176) [[2]](diffhunk://#diff-f0e8ba8cb8cb07b51b3be675bf62cec07e2eae1461341ce5801d33a57c8f57fdR1219-R1221) * Included necessary C++ includes for plugin stream support. These changes collectively improve the flexibility and introspection capabilities of ONNX Runtime's device, memory, and execution provider interfaces, and make advanced features available to Python users. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Depends on: microsoft#26021
### Description I created a new machine pool with the latest VS version. This PR upgrades all the ADO pipelines to use this VS version, except for the CUDA pipelines which are still using CUDA 12.2. Added code in Windows ARM64 jobs to pin cmake and vcpkg's versions. Added a powershell script to help setup nodejs on Windows ARM64 machines.
### Description as x86_64 Mac machines are retiring, move jobs to arm64. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
### Description Add Inverse Op from com.microsoft domain to QNN EP. ### Motivation and Context Add Inverse Op from com.microsoft domain to QNN EP. --------- Signed-off-by: Mu-Chein Hsu <quic_muchhsu@quicinc.com>
### Description The package size limit for PyPI and Nuget are: - python package size under 300MB - Nuget package size under 250MB To meet the size limit, this PR firstly removes some old GPU arch support in CMAKE_CUDA_ARCHITECTURE. Secondly, it removes the FPA_INTB_GEMM support in Linux Python wheel. #### Python wheel | OS | cmake_cuda_architecture | CUDA kernel removal |Package size | Under 300MB| |---------|--------------------------------------------------------|-|-------------|---| | Linux | 60-real;70-real;75-real;80-real;86-real;90a-real;90a-virtual | |341 MB |No (original)| | Linux | 70-real;75-real;80-real;86-real;90a-real;90a-virtual | | 329 MB |No| | Linux | 75-real;80-real;86-real;90a-real;90a-virtual | |319 MB |No| | Linux | 80-real;86-real;90a-real;90a-virtual | |304 MB |No| | Linux | 60-real;70-real;75-real;80-real;86-real;90a-real;90a-virtual. | FPA_INTB_GEMM|287 MB |Yes| | Windows | 52-real;61-real;75-real;86-real;89-real;90a-virtual | | 272 MB |Yes (original)| #### Nuget | OS | cmake_cuda_architecture | CUDA kernel removal |Package size |Under 250MB| |---------|--------------------------------------------------------|---|--------------|---| | Linux | 60-real;70-real;75-real;80-real;90a-real;90a-virtual | |276 MB |No (original)| | Linux | 75-real;80-real;90a-real;90a-virtual | |253 MB |No| | Linux | 60-real;70-real;75-real;80-real;90a-real;90a-virtual |FPA_INTB_GEMM| 230 MB |Yes| | Windows | 52-real;61-real;75-real;86-real;89-real;90a-virtual || 264 MB |No (original)| | Windows | 61-real;75-real;86-real;89-real;90a-virtual || 254 MB |No| | Windows | 75-real;86-real;89-real;90a-virtual || 242 MB |Yes| ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
### Description This PR adds the dispatchWorkgroupsIndirect capability for the program. It's part of the work to enable graph capture in phi4 microsoft#25868 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
### Description This PR updates multiple NPM dependencies to their latest versions to resolve security concerns - Upgraded `electron` to `38.1.2`. - Upgraded `axios` to `1.12.2`. - Upgraded `@types/node` to `22.18.6`. - Added `undici-types` as a new dependency. - Updated `brace-expansion` to versions `2.0.2` and `1.1.12`.
This PR adds model compilation support in ort perf test * Add `--compile_ep_context` flag in onnxruntime_perf_test. This generates EP context model and prints out compilation time and perf statistics for the compiled model * Prints `Compile time cost` for EP context compilation Sample usage: ```sh $ ./onnxruntime_perf_test -e trt-rtx -I -r 1 "/path/to/model.onnx" --compile_ep_context --compile_model_path "/path/to/model_ctx.onnx" ``` Output: ```sh Compile time cost: 17.8012 s Session creation time cost: 0.966619 s First inference time cost: 8151 ms Total inference time cost: 8.08084 s Total inference requests: 1 Average inference time cost: 8080.84 ms Total inference run time: 8.08085 s Number of inferences per second: 0.123749 Avg CPU usage: 6 % Peak working set size: 7861874688 bytes Avg CPU usage:6 Peak working set size:7861874688 Runs:1 Min Latency: 8.08084 s Max Latency: 8.08084 s P50 Latency: 8.08084 s P90 Latency: 8.08084 s P95 Latency: 8.08084 s P99 Latency: 8.08084 s P999 Latency: 8.08084 s ```
- Added new STFT Op Builder - Added unit tests for running op on HTP - Disabled QNN CPU for STFT Op due to bug in QNN CPU ### Description Added support for STFT with a new op builder ### Motivation and Context To enable audio and signal processing on models within the QNN framework --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
## Description Make the GPU backend visible to users of the WinML API. The V2 EP selection mechanism will now expose the GPU device for QNN. The GPU can be picked when: * The device list contains only GPU * The execution policy is set to PREFER_GPU If multiple devices (e.g. HTP and GPU) are provided to `AppendExecutionProviders_V2`, whichever device was provided last in the list will be used. ## Motivation and Context Required to enable WinML usage with the QNN GPU backend
Convert React Native pipeline to Github Actions. The new place use ARM64 machines instead of x86 machines to run the tests.
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/setup-python/releases">actions/setup-python's releases</a>.</em></p> <blockquote> <h2>v6.0.0</h2> <h2>What's Changed</h2> <h3>Breaking Changes</h3> <ul> <li>Upgrade to node 24 by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1164">actions/setup-python#1164</a></li> </ul> <p>Make sure your runner is on version v2.327.1 or later to ensure compatibility with this release. <a href="https://github.com/actions/runner/releases/tag/v2.327.1">See Release Notes</a></p> <h3>Enhancements:</h3> <ul> <li>Add support for <code>pip-version</code> by <a href="https://github.com/priyagupta108"><code>@priyagupta108</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1129">actions/setup-python#1129</a></li> <li>Enhance reading from .python-version by <a href="https://github.com/krystof-k"><code>@krystof-k</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/787">actions/setup-python#787</a></li> <li>Add version parsing from Pipfile by <a href="https://github.com/aradkdj"><code>@aradkdj</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1067">actions/setup-python#1067</a></li> </ul> <h3>Bug fixes:</h3> <ul> <li>Clarify pythonLocation behaviour for PyPy and GraalPy in environment variables by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1183">actions/setup-python#1183</a></li> <li>Change missing cache directory error to warning by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1182">actions/setup-python#1182</a></li> <li>Add Architecture-Specific PATH Management for Python with --user Flag on Windows by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1122">actions/setup-python#1122</a></li> <li>Include python version in PyPy python-version output by <a href="https://github.com/cdce8p"><code>@cdce8p</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1110">actions/setup-python#1110</a></li> <li>Update docs: clarification on pip authentication with setup-python by <a href="https://github.com/priya-kinthali"><code>@priya-kinthali</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1156">actions/setup-python#1156</a></li> </ul> <h3>Dependency updates:</h3> <ul> <li>Upgrade idna from 2.9 to 3.7 in /<strong>tests</strong>/data by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/setup-python/pull/843">actions/setup-python#843</a></li> <li>Upgrade form-data to fix critical vulnerabilities <a href="https://redirect.github.com/actions/setup-python/issues/182">#182</a> & <a href="https://redirect.github.com/actions/setup-python/issues/183">#183</a> by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1163">actions/setup-python#1163</a></li> <li>Upgrade setuptools to 78.1.1 to fix path traversal vulnerability in PackageIndex.download by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1165">actions/setup-python#1165</a></li> <li>Upgrade actions/checkout from 4 to 5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/setup-python/pull/1181">actions/setup-python#1181</a></li> <li>Upgrade <code>@actions/tool-cache</code> from 2.0.1 to 2.0.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/setup-python/pull/1095">actions/setup-python#1095</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/krystof-k"><code>@krystof-k</code></a> made their first contribution in <a href="https://redirect.github.com/actions/setup-python/pull/787">actions/setup-python#787</a></li> <li><a href="https://github.com/cdce8p"><code>@cdce8p</code></a> made their first contribution in <a href="https://redirect.github.com/actions/setup-python/pull/1110">actions/setup-python#1110</a></li> <li><a href="https://github.com/aradkdj"><code>@aradkdj</code></a> made their first contribution in <a href="https://redirect.github.com/actions/setup-python/pull/1067">actions/setup-python#1067</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/setup-python/compare/v5...v6.0.0">https://github.com/actions/setup-python/compare/v5...v6.0.0</a></p> <h2>v5.6.0</h2> <h2>What's Changed</h2> <ul> <li>Workflow updates related to Ubuntu 20.04 by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1065">actions/setup-python#1065</a></li> <li>Fix for Candidate Not Iterable Error by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1082">actions/setup-python#1082</a></li> <li>Upgrade semver and <code>@types/semver</code> by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1091">actions/setup-python#1091</a></li> <li>Upgrade prettier from 2.8.8 to 3.5.3 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1046">actions/setup-python#1046</a></li> <li>Upgrade ts-jest from 29.1.2 to 29.3.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1081">actions/setup-python#1081</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/setup-python/compare/v5...v5.6.0">https://github.com/actions/setup-python/compare/v5...v5.6.0</a></p> <h2>v5.5.0</h2> <h2>What's Changed</h2> <h3>Enhancements:</h3> <ul> <li>Support free threaded Python versions like '3.13t' by <a href="https://github.com/colesbury"><code>@colesbury</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/973">actions/setup-python#973</a></li> <li>Enhance Workflows: Include ubuntu-arm runners, Add e2e Testing for free threaded and Upgrade <code>@action/cache</code> from 4.0.0 to 4.0.3 by <a href="https://github.com/priya-kinthali"><code>@priya-kinthali</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1056">actions/setup-python#1056</a></li> <li>Add support for .tool-versions file in setup-python by <a href="https://github.com/mahabaleshwars"><code>@mahabaleshwars</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1043">actions/setup-python#1043</a></li> </ul> <h3>Bug fixes:</h3> <ul> <li>Fix architecture for pypy on Linux ARM64 by <a href="https://github.com/mayeut"><code>@mayeut</code></a> in <a href="https://redirect.github.com/actions/setup-python/pull/1011">actions/setup-python#1011</a> This update maps arm64 to aarch64 for Linux ARM64 PyPy installations.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/actions/setup-python/commit/e797f83bcb11b83ae66e0230d6156d7c80228e7c"><code>e797f83</code></a> Upgrade to node 24 (<a href="https://redirect.github.com/actions/setup-python/issues/1164">#1164</a>)</li> <li><a href="https://github.com/actions/setup-python/commit/3d1e2d2ca0a067f27da6fec484fce7f5256def85"><code>3d1e2d2</code></a> Revert "Enhance cache-dependency-path handling to support files outside the w...</li> <li><a href="https://github.com/actions/setup-python/commit/65b071217a8539818fdb8b54561bcbae40380a54"><code>65b0712</code></a> Clarify pythonLocation behavior for PyPy and GraalPy in environment variables...</li> <li><a href="https://github.com/actions/setup-python/commit/5b668cf7652160527499ee14ceaff4be9306cb88"><code>5b668cf</code></a> Bump actions/checkout from 4 to 5 (<a href="https://redirect.github.com/actions/setup-python/issues/1181">#1181</a>)</li> <li><a href="https://github.com/actions/setup-python/commit/f62a0e252fe7114e86949abfa6e1e89f85bb38c2"><code>f62a0e2</code></a> Change missing cache directory error to warning (<a href="https://redirect.github.com/actions/setup-python/issues/1182">#1182</a>)</li> <li><a href="https://github.com/actions/setup-python/commit/9322b3ca74000aeb2c01eb777b646334015ddd72"><code>9322b3c</code></a> Upgrade setuptools to 78.1.1 to fix path traversal vulnerability in PackageIn...</li> <li><a href="https://github.com/actions/setup-python/commit/fbeb884f69f0ac1c0257302f62aa524c2824b649"><code>fbeb884</code></a> Bump form-data to fix critical vulnerabilities <a href="https://redirect.github.com/actions/setup-python/issues/182">#182</a> & <a href="https://redirect.github.com/actions/setup-python/issues/183">#183</a> (<a href="https://redirect.github.com/actions/setup-python/issues/1163">#1163</a>)</li> <li><a href="https://github.com/actions/setup-python/commit/03bb6152f4f691b9d64579a1bd791904a083c452"><code>03bb615</code></a> Bump idna from 2.9 to 3.7 in /<strong>tests</strong>/data (<a href="https://redirect.github.com/actions/setup-python/issues/843">#843</a>)</li> <li><a href="https://github.com/actions/setup-python/commit/36da51d563b70a972897150555bb025096d65565"><code>36da51d</code></a> Add version parsing from Pipfile (<a href="https://redirect.github.com/actions/setup-python/issues/1067">#1067</a>)</li> <li><a href="https://github.com/actions/setup-python/commit/3c6f142cc0036d53007e92fa1e327564a4cfb7aa"><code>3c6f142</code></a> update documentation (<a href="https://redirect.github.com/actions/setup-python/issues/1156">#1156</a>)</li> <li>Additional commits viewable in <a href="https://github.com/actions/setup-python/compare/v5...v6">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### Description Fix CUDA 12.8+ build issues with regards to fp4 type ### Motivation and Context Fix CUDA 12.8+ build issues with regards to fp4 type --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
### Description <!-- Describe your changes. --> Ports the `MlasErfKernel`, `MlasLogisticKernel` and `MlasComputeSoftmax` kernels to the ARM SVE backend. Specifically, the following functions have been ported. - `MlasErfKernel` (lib/erf.cpp) - `MlasLogisticKernel` (lib/logistic.cpp) - `MlasComputeSumExpF32Kernel` (lib/compute.cpp) - `MlasReduceMaximumF32Kernel` (lib/compute.cpp) - `MlasComputeSoftmaxOutputF32Kernel` (lib/compute.cpp) - `MlasComputeSoftmaxThreaded` (lib/compute.cpp) This PR uses the following design structure: adds new wrapper implementations of SVE functions in `lib/mlasi_sve.h` similar to `mlasi.h` and calls these wrapper functions in each kernel's implementation. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> This work is a step toward making ONNX Runtime more performant and architecture-aware on ARM platforms. ### Performance Analysis  - Observed upto 1.4x speedup at the operator level - Performance is tested on AWS Graviton3E This PR is a joint contribution by: - @NishantPrabhuFujitsu - @sanketkaleoss --------- Co-authored-by: Sanket Kale <sanketk.kale@fujitsu.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
### Description 1. Delete python 3.10 from packaging pipeline 2. Increase min supported macOS version from 13.4 to 14.0 3. Remove macOS x86 wheels from the job configs. (Because Apple has stopped producing x86 machines for quite a while) ### Motivation and Context Reduce nightly pipelines complexity
### Description <!-- Describe your changes. --> Improve InlinedVector Remove stale raw_hash_set
## Description `MatMul` in the WebGPU EP contains a const& member initialized with a temporary object (`Activation()`). This introduces a dangling reference issue. ## Motivation and Context While working on another PR, I accidentally discovered this potential problem. Although no errors have been observed in the current code, a dangling reference leads to undefined behavior and could cause potential issues in the future.
### Description <!-- Describe your changes. --> Add support for `MemcpyFromHost` and `MemcpyToHost` ops with plugin EPs. - Add CPU EP fallback kernels for the memcpy ops. These are generic implementations using a data transfer manager. - Update `SessionState::PopulateKernelCreateInfo()` to fall back to CPU memcpy kernels if a node's assigned provider doesn't have them. - Update `MemcpyTransformer` to determine whether providers are CPU-based or compatible with other providers by looking at the device type instead of matching against a hardcoded list of provider types. This accommodates plugin EPs, where the provider type can't be hardcoded. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Allow plugin EPs to work with models where memcpy ops are required (i.e., models where connected nodes are not fully assigned to the plugin EP).
…6134) ### Description As title ### Motivation and Context Fix Windows CUDA 12.8+ builds wrt to fp4 type usage
…lity (microsoft#26132) ### Description In current TRT RTX EP/ TRT EP implementation, when constructing the `IndexedSubGraph`, for some cases, it will include the node's unused output as the SubGraph's output. So, it will return the incorrect `IndexedSubGraph` from its GetCapability to ORT. Add the logic to prevent adding the unused node's output. With this fix, we can avoid generating the incorrect EPContext model where the EPContext node has unused output.
Fix build warnings using cuda 12.8 in Linux like the following:
```
<command-line>: error: "_FORTIFY_SOURCE" redefined [-Werror]
gather_block_quantized.cc:95:40: warning: comparison of integer expressions of different signedness: ‘int64_t’ {aka ‘long int’} and ‘long unsigned int’ [-Wsign-compare]
95 | for (int64_t i = gather_axis_ + 1; i < data_rank; ++i) {
attention_op_test.cc:304:85: warning: comparison of integer expressions of different signedness: ‘int’ and ‘std::vector<float>::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
304 | } else if (batch_size * q_num_heads * q_sequence_length * total_sequence_length == attn_mask.size()) {
/cast_op_test.cc:1487:24: warning: comparison of integer expressions of different signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘int’ [-Wsign-compare]
1487 | for (size_t i = 0; i < num_pairs; ++i) {
| ~~^~~~~~~~~~~
```
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This pull request adds new APIs and updates existing ones to improve memory and device information handling in the ONNX Runtime C# bindings. The most significant changes introduce methods for fetching memory info and device info for session inputs/outputs, and add support for shared allocators and synchronization streams. There are also several updates and renamings for LoraAdapter delegates and related APIs. ### Memory and Device Info APIs * Added `GetMemoryInfosForInputs`, `GetMemoryInfosForOutputs`, and `GetEpDeviceForInputs` methods to `InferenceSession.shared.cs` to fetch memory info and device info for session inputs/outputs. These methods utilize new native delegates for retrieving memory and device information. * Introduced native delegates in `NativeMethods.shared.cs` for `OrtSessionGetMemoryInfoForInputs`, `OrtSessionGetMemoryInfoForOutputs`, and `OrtSessionGetEpDeviceForInputs`, and wired them up in the static constructor. [[1]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73R530-R532) [[2]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73R1312-R1335) ### Shared Allocator and Synchronization Stream Support * Added delegates and static fields for creating, getting, and releasing shared allocators, as well as for creating and managing synchronization streams (`OrtCreateSharedAllocator`, `OrtGetSharedAllocator`, `OrtReleaseSharedAllocator`, `OrtCreateSyncStreamForEpDevice`, `OrtSyncStream_GetHandle`, `OrtReleaseSyncStream`). * Added delegate for copying tensors (`OrtCopyTensors`). ### LoraAdapter API Updates * Renamed LoraAdapter-related delegates to use the `Ort` prefix (`OrtCreateLoraAdapter`, `OrtCreateLoraAdapterFromArray`, `OrtReleaseLoraAdapter`) and updated their usage throughout the codebase. [[1]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73L699-R710) [[2]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73L1561-R1672) [[3]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73L1578-R1695) ### MemoryInfo Enhancements * Added new delegates for creating memory info with more parameters (`OrtCreateMemoryInfoV2`), and for querying device memory type and vendor ID (`OrtMemoryInfoGetDeviceMemType`, `OrtMemoryInfoGetVendorId`). [[1]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73R594-R596) [[2]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73R1804-R1817) [[3]](diffhunk://#diff-f9f2aaafc076365917de8ab96628da427d9dd0fd6a214fb9c266733f90d6fc73R1866-R1877) ### Minor API Documentation Update * Clarified the lifetime of allocators in the documentation, noting they can be explicitly unregistered.### Description <!-- Describe your changes. --> ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
### Description - Regenerates the `input_propagate_to_output.onnx` model used in [this unit test](https://github.com/microsoft/onnxruntime/blob/35dcab5088118117acc6086c9b6dd6dd92c7060f/onnxruntime/test/shared_lib/test_inference.cc#L497-L506) so that it uses an ONNX IR version compatible with ONNX 1.18.0 (i.e., IR version < 12). - Adds script `input_propagate_to_output.py` that can be used to regenerate the `input_propagate_to_output.onnx` model. - Embed missing weight values that are needed to run the existing `test_dangling_input_segment_ids.py` script. ### Motivation and Context The main branch is using ONNX 1.19. However, this unit test also needs to pass in the `rel-1.23.1` branch, which is still using ONNX 1.18.0. So, by downgrading the model's IR version, the unit test can run in both branches. See original PR that added the test models: microsoft#26021
…23 on CUDA (microsoft#26075) ### Description Fixes microsoft#26065.
### Description This PR adds missing kernels for bfloat16 precision across several ops in both the `ai.onnx` and `com.microsoft` domains. 1. `SkipLayerNormalization` [contrib op](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftskiplayernormalization) 2. `Conv` for [opset 22](https://onnx.ai/onnx/operators/onnx__Conv.html) 3. `Pow` for [opset 15](https://onnx.ai/onnx/operators/onnx__Pow.html) 4. `AveragePool` for [opset 22](https://onnx.ai/onnx/operators/onnx__AveragePool.html) This PR also enables weight-only quantization of a bfloat16 `MatMul` op to a bfloat16 `MatMulNBits` [contrib op](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits). ### Motivation and Context This PR enables running ONNX models from the Gemma-3 family that are generated with bfloat16 precision.
…Intel GPUs (microsoft#25908) ### Description This PR improves the performance of `Conv-Transpose` by ~12x on Lunar Lake. For a specific shape: (3, 3, 2560, 1280), the default normalization produced a dispatch group size of (679, 679, 1) resulted in extremely slow performance on LNL (likely due to a driver issue). By manually normalize the dispatch group size to (5, 640, 160), we achieve a significant ~12x performance improvement on LNL. ### Motivation and Context See above. Co-authored-by: Yang, Wenqin <wenqin.yang@intel.com>
### Description MultiheadAttention CUDA BF16 Support ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
…soft#26150) ### Description Enable largest attr and remove sorted check for QNN topk op builder. ### Motivation and Context Signed-off-by: Mu-Chein Hsu <quic_muchhsu@quicinc.com>
### Description Fix logic flow bug where rpc polling interval is set to 9999 when perf performance is NOT burst. The interval should be set to 9999 when the perf performance is burst ### Motivation and Context Co-authored-by: quic_calvnguy <quic_calvnguy@quic_inc.com>
…6147) This PR fixes few unused variables
Fix cuda 13 build errors and warnings. Related: microsoft#25936 I've verified the build in Linux and Windows using the following test settings: ### Build command line You may need change cuda_home and cudnn_home to your installation directories, also update CMAKE_CUDA_ARCHITECTURES according to your GPU. #### Linux Build ``` pip install cmake ninja packaging numpy sh build.sh --config Release --build_dir build/cuda13 --parallel --use_cuda \ --cuda_version 12.8 --cuda_home /nvida/cuda13.0/ \ --cudnn_home /nvida/cudnn9.12_cu13/ \ --build_wheel --skip_tests \ --cmake_generator Ninja \ --enable_cuda_nhwc_ops \ --use_binskim_compliant_compile_flags \ --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=90-real;90-virtual \ --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON \ --cmake_extra_defines onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON ``` #### Windows Build ``` IF "%VCToolsVersion%"=="" call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" build.bat --cmake_generator "Visual Studio 17 2022" --config Release --build_dir build\cuda13 --build_wheel ^ --parallel 4 --nvcc_threads 1 --build_shared_lib ^ --use_cuda --cuda_version "13.0" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0" ^ --cudnn_home "D:\cudnn\9.13.0.50_cuda13" ^ --skip_tests ^ --use_binskim_compliant_compile_flags ^ --enable_cuda_nhwc_ops ^ --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=native" ^ --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON ^ --cmake_extra_defines FETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER ``` The onnxruntime_test_all.exe is passed for RTX 5060 Ti GPU, so the binary can support blackwell GPU (CUDA_ARCHITECTURES=120) properly with CUDA 13.0: ``` [----------] Global test environment tear-down [==========] 1242 tests from 111 test suites ran. (83468 ms total) [ PASSED ] 1242 tests. ```
Use vmlaq_f32 on MLAS_TARGET_ARM (armv7) so builds on Linux/Android arm32 don't attempt to use the armv8 (aarch64) vfmaq_f32 intrinsic. Fixes build failures referencing vfmaq_f32 in arm_neon.h. ### Description On ARMv7 targets (32-bit ARM with NEON) the vfmaq_f32 intrinsic is unavailable and causes build failures like: target specific option mismatch 1728 | vfmaq_f32 (float32x4_t __a, float32x4_t __b, float32x4_t __c). The previous code only fell back to vmlaq_f32 when __ANDROID__ was defined, which fixes Android but not Linux ARM32 builds. ### Motivation and Context The proposal is to change: if defined(__ANDROID__) && defined(MLAS_TARGET_ARM) to: if defined(MLAS_TARGET_ARM) This change ensures 32-bit ARM builds (Linux and Android) use vmlaq_f32 while 64-bit ARM uses vfmaq_f32. This pr is related to issue microsoft#25949 which was opened also by me. --------- Signed-off-by: Wei Du <wei.du@tpv-tech.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
ankitm3k
approved these changes
Sep 26, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Synchronizing intel/onnxruntime ovep-develop branch with latest changes from microsoft/onnxruntime master branch.