Add BF16 kernels in several ops for Gemma-3 - #26102
Merged
kunal-vaishnavi merged 15 commits intoSep 25, 2025
Merged
Conversation
nenad1002
reviewed
Sep 23, 2025
nenad1002
left a comment
Contributor
There was a problem hiding this comment.
You are missing docs which fails some pipelines
tianleiwu
reviewed
Sep 23, 2025
tianleiwu
reviewed
Sep 25, 2025
tianleiwu
approved these changes
Sep 25, 2025
kunal-vaishnavi
added a commit
to microsoft/onnxruntime-genai
that referenced
this pull request
Sep 25, 2025
### Description This PR enables multi-modal models to use bfloat16 precision for other models besides the text decoder. It requires the changes from this [ORT PR](microsoft/onnxruntime#26102). ### Motivation and Context Models from the Gemma-3 family can lose accuracy with float16 precision and require bfloat16 precision for higher quality output.
hariharans29
added a commit
that referenced
this pull request
Sep 26, 2025
### Description Fix warnings in non-CUDA EP Windows builds ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Minor fix from #26102
fs-eire
pushed a commit
that referenced
this pull request
Oct 24, 2025
### Description This PR adds missing kernels for bfloat16 precision across several ops in both the `ai.onnx` and `com.microsoft` domains. 1. `SkipLayerNormalization` [contrib op](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftskiplayernormalization) 2. `Conv` for [opset 22](https://onnx.ai/onnx/operators/onnx__Conv.html) 3. `Pow` for [opset 15](https://onnx.ai/onnx/operators/onnx__Pow.html) 4. `AveragePool` for [opset 22](https://onnx.ai/onnx/operators/onnx__AveragePool.html) This PR also enables weight-only quantization of a bfloat16 `MatMul` op to a bfloat16 `MatMulNBits` [contrib op](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits). ### Motivation and Context This PR enables running ONNX models from the Gemma-3 family that are generated with bfloat16 precision.
fs-eire
pushed a commit
that referenced
this pull request
Oct 24, 2025
### Description Fix warnings in non-CUDA EP Windows builds ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Minor fix from #26102
naomiOvad
pushed a commit
to naomiOvad/onnxruntime
that referenced
this pull request
Nov 2, 2025
### Description This PR adds missing kernels for bfloat16 precision across several ops in both the `ai.onnx` and `com.microsoft` domains. 1. `SkipLayerNormalization` [contrib op](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftskiplayernormalization) 2. `Conv` for [opset 22](https://onnx.ai/onnx/operators/onnx__Conv.html) 3. `Pow` for [opset 15](https://onnx.ai/onnx/operators/onnx__Pow.html) 4. `AveragePool` for [opset 22](https://onnx.ai/onnx/operators/onnx__AveragePool.html) This PR also enables weight-only quantization of a bfloat16 `MatMul` op to a bfloat16 `MatMulNBits` [contrib op](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits). ### Motivation and Context This PR enables running ONNX models from the Gemma-3 family that are generated with bfloat16 precision.
naomiOvad
pushed a commit
to naomiOvad/onnxruntime
that referenced
this pull request
Nov 2, 2025
### Description Fix warnings in non-CUDA EP Windows builds ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Minor fix from microsoft#26102
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds missing kernels for bfloat16 precision across several ops in both the
ai.onnxandcom.microsoftdomains.SkipLayerNormalizationcontrib opConvfor opset 22Powfor opset 15AveragePoolfor opset 22This PR also enables weight-only quantization of a bfloat16
MatMulop to a bfloat16MatMulNBitscontrib op.Motivation and Context
This PR enables running ONNX models from the Gemma-3 family that are generated with bfloat16 precision.