Skip to content

Integrate SME1 SGEMM KleidiAI kernels - #25760

Merged
edgchen1 merged 1 commit into
microsoft:mainfrom
patryk-kaiser-ARM:SME1_sgemm_integration
Sep 12, 2025
Merged

Integrate SME1 SGEMM KleidiAI kernels#25760
edgchen1 merged 1 commit into
microsoft:mainfrom
patryk-kaiser-ARM:SME1_sgemm_integration

Conversation

@patryk-kaiser-ARM

@patryk-kaiser-ARM patryk-kaiser-ARM commented Aug 15, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Key changes
This PR integrates KleidiAI SME1 FP32 kernels into the existing kleidiai_sgemm.cpp implementation.

Adding SME2 flag in onnxruntime/core/common/cpuid_info.h & onnxruntime/core/common/cpuid_info.cc
Previous SME2 kernels integrated were using SME(1) check, this change will correctly distinguish between when SME1 and SME2 kernels are to be used.

Bumping KleidiAI version to 1.10.0

Indicative performance data
Single thread Mac Mini M4 runs on various models using: onnxruntime_perf_test -v -e cpu -I -m times -x 1 -y 1 -r 1
image

Next steps
Additional commits to come will address outstanding to-do issues from previous PR linked below:
KleidiAI SGEMM/IGEMM/Quantized MatMul - Modular MLAS API Changes for KleidiAI #25187

@patryk-kaiser-ARM

patryk-kaiser-ARM commented Aug 15, 2025

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree company="Arm"

@patryk-kaiser-ARM
patryk-kaiser-ARM marked this pull request as draft August 21, 2025 10:06
edgchen1
edgchen1 reviewed Aug 21, 2025
View reviewed changes
Comment thread onnxruntime/contrib_ops/cpu/quantization/dynamic_quantize_matmul.cc
@patryk-kaiser-ARM
patryk-kaiser-ARM marked this pull request as ready for review August 25, 2025 12:10
@hariharans29

hariharans29 commented Aug 25, 2025

Copy link
Copy Markdown
Member

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

@azure-pipelines

azure-pipelines Bot commented Aug 25, 2025

Copy link
Copy Markdown
Azure Pipelines successfully started running 5 pipeline(s).

@patryk-kaiser-ARM
patryk-kaiser-ARM force-pushed the SME1_sgemm_integration branch 2 times, most recently from 443a898 to d165482 Compare September 1, 2025 12:03
@hariharans29

hariharans29 commented Sep 2, 2025

Copy link
Copy Markdown
Member

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

@azure-pipelines

azure-pipelines Bot commented Sep 2, 2025

Copy link
Copy Markdown
Azure Pipelines successfully started running 5 pipeline(s).

edgchen1
edgchen1 reviewed Sep 4, 2025
View reviewed changes
Comment thread onnxruntime/core/common/cpuid_info.cc
Comment thread onnxruntime/core/mlas/lib/kleidiai/sgemm_kleidiai.cpp Outdated
Comment thread onnxruntime/core/mlas/lib/kleidiai/sgemm_kleidiai.cpp Outdated
Comment thread onnxruntime/core/mlas/lib/kleidiai/sgemm_kleidiai.cpp
@patryk-kaiser-ARM
patryk-kaiser-ARM force-pushed the SME1_sgemm_integration branch 2 times, most recently from 40526c4 to 9800e11 Compare September 9, 2025 12:50
@hariharans29

hariharans29 commented Sep 9, 2025

Copy link
Copy Markdown
Member

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

@azure-pipelines

azure-pipelines Bot commented Sep 9, 2025

Copy link
Copy Markdown
Azure Pipelines successfully started running 5 pipeline(s).

@patryk-kaiser-ARM
Integrate SME1 SGEMM KleidiAI kernels
39b7e05 
Signed-off-by: Patryk Kaiser <patryk.kaiser@arm.com>
@hariharans29

hariharans29 commented Sep 10, 2025

Copy link
Copy Markdown
Member

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

@azure-pipelines

azure-pipelines Bot commented Sep 10, 2025

Copy link
Copy Markdown
Azure Pipelines successfully started running 5 pipeline(s).

@hariharans29 hariharans29 reopened this Sep 12, 2025
@hariharans29

hariharans29 commented Sep 12, 2025

Copy link
Copy Markdown
Member

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline

@azure-pipelines

azure-pipelines Bot commented Sep 12, 2025

Copy link
Copy Markdown
Azure Pipelines successfully started running 5 pipeline(s).

@hariharans29

hariharans29 commented Sep 12, 2025

Copy link
Copy Markdown
Member

/azp run Windows ARM64 QNN CI Pipeline

@azure-pipelines

azure-pipelines Bot commented Sep 12, 2025

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@edgchen1
edgchen1 merged commit ec3bf7f into microsoft:main Sep 12, 2025
155 of 163 checks passed
hariharans29 added a commit that referenced this pull request Sep 16, 2025
@hariharans29
Revert "Integrate SME1 SGEMM KleidiAI kernels (#25760)"
3c7d339 
This reverts commit ec3bf7f.
@hariharans29 hariharans29 mentioned this pull request Sep 16, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hariharans29 hariharans29 hariharans29 approved these changes
@edgchen1 edgchen1 edgchen1 approved these changes
+1 more reviewer
@JonathanC-ARM JonathanC-ARM JonathanC-ARM left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

KleidiAI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@patryk-kaiser-ARM @hariharans29 @edgchen1 @JonathanC-ARM @jywu-mysoft