[MLAS][KleidiAI] Add MatMul, Gemm, and Conv FP16 execution paths

[JonathanC-ARM]

Description

This PR builds on fp16-split/02-mlas-half-api-squashed by connecting its FP16 MLAS APIs and KleidiAI backends to the CPU Execution Provider.

It:

- Registers CPU `MLFloat16` kernels for:
  - `MatMul` opsets 1–8, 9–12, and 13+
  - `Gemm` opsets 7–8, 9–10, 11–12, and 13+
- Adds a dedicated FP16 `MatMul` implementation which:
  - Preserves ONNX broadcasting and batched execution through `MatMulComputeHelper`.
  - Handles empty outputs and zero-K inputs.
  - Uses the existing HGEMM path for eligible small, unpacked matrices.
  - Otherwise dispatches through `MlasHalfGemmBatch`.
  - Carries the configured MLAS backend selector through every batch entry.
  - Pre-packs constant two-dimensional B matrices into the selected backend’s native layout when supported.
- Keeps backend-native FP16 MatMul packs owned by the kernel. These layouts depend on the active backend, so they are not placed in the cross-session shared-prepack cache without suitable layout metadata.
- Adds an explicit kernel-owned packed-weights marker to `PrePackedWeights`. Session initialization continues to reject broken kernels which report successful prepacking without either providing cacheable buffers or explicitly identifying a valid kernel-owned pack.
- Extends FP16 `Gemm` to use `MlasHalfGemmBatch` for supported no-transpose, `alpha == 1` cases:
  - No effective bias, including a missing C input or `beta == 0`.
  - `beta == 1` with an MLAS-compatible N-element bias.
  - Unsupported transpose, scaling, or bias-broadcast combinations retain the existing Eigen fallback.
- Adds an MLAS HalfConv fast path to the existing FP16 CPU Conv implementation:
  - Uses `MlasHalfConvPrepare` to determine runtime and backend eligibility for two-dimensional convolutions without a fused Sum input.
  - Runs eligible convolutions through `MlasHalfConv`.
  - Supports NCHW and NHWC layouts and forwards padding, stride, dilation, activation, and workspace information.
  - Pre-packs eligible constant filters using `MlasHalfConvPackWeightsAndBias`.
  - Includes a compatible constant bias in the native pack when safe.
  - Falls back to the existing reordered-weight/im2col plus HalfGemm implementation when HalfConv is unavailable or rejects the configuration.
- Keeps combined HalfConv weight-and-bias packs session-local because their contents depend on the bias while the existing shared-prepack key identifies only W. Generic Conv weight packs remain shareable across sessions.
- Centralizes HalfGemm backend-selector handling in the public MLAS dispatcher and updates the affected MatMul, Gemm, Conv, and MoE call sites.
- Updates the CPU operator documentation to advertise FP16 support for `MatMul` and `Gemm`.

Test coverage includes:

- FP16 Gemm with no C input and with `beta == 0`.
- FP16 MatMul with constant and dynamic B inputs.
- MatMul backend-native B prepacking.
- Kernel-owned MatMul packs remaining intentionally unshared across sessions.
- Session-state validation for broken prepacking implementations.
- HalfConv-eligible NCHW and NHWC convolutions.
- HalfConv execution with and without bias.
- Correct fallback behavior when KleidiAI is explicitly disabled.
- Safe sharing of generic Conv prepacked weights while keeping bias-dependent HalfConv packs local.
- Updated HalfGemm selector and backend-native packed-B contract coverage.

Motivation and Context

This is pr 3 of 5 as per #28786 splitting suggestion
The preceding PR introduces the MLAS primitives for FP16 HalfGemm, backend-native B packing, and HalfConv, but standard ONNX CPU operators do not yet consume them.

This PR connects those primitives to CPU MatMul, Gemm, and Conv, making native FP16 execution and KleidiAI acceleration available to the operator layer while retaining portable fallback behavior for unsupported hardware, shapes, attributes, or backend configurations.

The prepacking changes avoid repeatedly packing constant weights while preventing backend-specific or bias-dependent layouts from being incorrectly reused through the shared prepack cache.

This PR is limited to CPU kernel registration and execution. The follow-up InsertCastTransformer PR introduces the opt-in policy and shape heuristics that decide when graph optimization should preserve FP16 CPU nodes. Apple ARM64 build enablement is also handled separately.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR wires newly introduced MLAS FP16 (HalfGemm + backend-native packed-B + HalfConv) capabilities into the CPU Execution Provider’s MatMul, Gemm, and FP16 Conv kernels, including prepacking behavior and tests, and updates operator docs to advertise FP16 support.

Changes:

• Registers CPU MLFloat16 kernels for MatMul and Gemm across multiple opset ranges, and routes FP16 execution through MLAS (MlasHalfGemmBatch) with backend-selector propagation and optional backend-native B prepacking.
• Extends FP16 CPU Conv to opportunistically use MLAS HalfConv (including optional constant bias baking into a session-local packed buffer), while keeping generic prepacked weights shareable across sessions.
• Strengthens prepacking validation by adding an explicit “kernel-owned packed weights” marker and adding coverage for broken prepack implementations and sharing behavior.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
onnxruntime/test/providers/cpu/nn/conv_fp16_test.cc	Adds FP16 Conv tests covering HalfConv eligibility, bias/no-bias, and KleidiAI disable fallback.
onnxruntime/test/providers/cpu/math/matmul_test.cc	Adds test coverage for kernel-owned FP16 MatMul prepacking not entering shared prepack cache.
onnxruntime/test/providers/cpu/math/gemm_test.cc	Adds FP16 Gemm tests for “missing C” and beta == 0 cases.
onnxruntime/test/mlas/unittest/test_halfgemm.h	Updates unit test for updated HalfGemm override signature.
onnxruntime/test/mlas/unittest/test_halfgemm.cpp	Updates override signature and related KleidiAI HalfGemm tests to match new API.
onnxruntime/test/framework/session_state_test.cc	Adds a broken-prepack kernel and a test validating session-state rejects non-cacheable successful prepack.
onnxruntime/core/util/math_cpu.cc	Adds MatMul<MLFloat16> specialization that uses MlasHalfGemmBatch when available.
onnxruntime/core/providers/cpu/math/matmul.h	Introduces MatMul<MLFloat16> kernel with backend selector + prepack plumbing.
onnxruntime/core/providers/cpu/math/matmul.cc	Registers FP16 MatMul kernels, implements MLAS-backed FP16 MatMul compute + backend-native packed-B prepack.
onnxruntime/core/providers/cpu/math/gemm.cc	Extends FP16 Gemm MLAS path to support no-bias (beta==0) and MLAS-compatible bias shapes.
onnxruntime/core/providers/cpu/fp16/fp16_conv.cc	Adds MLAS HalfConv fast path, including session-local packed weights+bias handling.
onnxruntime/core/providers/cpu/cpu_execution_provider.cc	Registers FP16 MatMul/Gemm kernel class declarations and creation entries for CPU EP.
onnxruntime/core/mlas/lib/mlasi.h	Updates KleidiAI HalfGemm override signature to drop explicit selector config parameter.
onnxruntime/core/mlas/lib/kleidiai/mlasi_kleidiai.h	Updates KleidiAI header declaration for updated override signature.
onnxruntime/core/mlas/lib/kleidiai/halfgemm_kleidiai.cpp	Removes redundant per-call selector check; relies on centralized dispatcher gating.
onnxruntime/core/mlas/lib/halfgemm.cpp	Centralizes backend selector handling before calling platform override.
onnxruntime/core/framework/session_state.cc	Updates prepack validation to allow explicit kernel-owned packed weights marker.
onnxruntime/core/framework/prepacked_weights.h	Adds has_kernel_owned_packed_weights_ marker to PrePackedWeights.
onnxruntime/core/framework/prepacked_weights_container.cc	Ensures the kernel-owned packed marker is preserved in referring copies.
onnxruntime/contrib_ops/cpu/moe/moe_cpu.cc	Updates HalfGemm params initialization style ({}) consistent with new patterns.
docs/OperatorKernels.md	Updates documentation to list FP16 support for CPU MatMul and Gemm.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[hariharans29]

Re-review: PR #29709 — [MLAS][KleidiAI] Add MatMul, Gemm, and Conv FP16 execution paths (head 3ef65e5)

Author: @JonathanC-ARM. Now 4 commits. Head advanced past two force-pushes since the pass-1 review: 7d86dcb → 37a0249d (initial rebase) → 3aeb947760cd → 00de02c → 3ef65e5 (current). CI: 67/83 checks OK at fetch time; hariharans29 closed/reopened once already to retrigger Azure Pipelines.

Verdict: no blockers. All substantive Copilot feedback has been addressed correctly (either accepted or rebutted with valid rationale). The two design-level changes since pass-1 (the has_kernel_owned_packed_weights_ marker and the HalfGemm dispatch fix on non-FP16-accel ARM64) are both correct improvements. Merge when CI is green and a Microsoft-side approver signs off.

---

Copilot review disposition

Copilot #1 (Medium): input_defs[2]->Exists() in fp16_conv.cc — "risks null-pointer crash for Conv without B".

Author correctly rebutted this: ORT does not represent an omitted optional input as a null NodeArg* in InputDefs(). An optional slot that is absent from the NodeProto altogether shortens InputDefs().size() (handled by the preceding input_defs.size() >= 3 guard). An optional slot that is present but empty in the NodeProto is materialized as a valid NodeArg* whose Name() is empty and whose Exists() returns false. Nullptr entries do not appear in InputDefs() by ORT convention — you can grep for input_defs[i]->Exists() throughout the codebase to confirm the pattern (fp16_conv.cc itself, conv.cc, gemm.cc, and many others all do this without null checks). The Copilot suggestion is a defensive change against a case that cannot occur.

Copilot #2 (Medium): try/catch on ORT_THROW_IF_ERROR breaks ORT_NO_EXCEPTIONS builds.

Author accepted, converted to ASSERT_STATUS_NOT_OK_AND_HAS_SUBSTR. Correct fix — the macro handles both build modes uniformly and matches the pre-existing test-file style. Verified in the diff.

---

Substantive changes since pass-1

1. has_kernel_owned_packed_weights_ marker on PrePackedWeights (new)

New boolean field in prepacked_weights.h:

// Set when PrePack successfully produced kernel-owned packed weights that
// intentionally cannot be stored in a shared pre-packed weights container.
bool has_kernel_owned_packed_weights_{false};

Propagated through CreateReferringCopy. Consumed in session_state.cc:

• The pre-existing invariant ORT_ENFORCE(weights_to_be_filled_in.buffers_.size() > 0, ...) is relaxed to ORT_RETURN_IF_NOT(!buffers.empty() || has_kernel_owned_packed_weights_, ...). Also flipped from ORT_ENFORCE (throws) to ORT_RETURN_IF_NOT (returns Status) — better error propagation, and consistent with the Copilot-driven ASSERT_STATUS_NOT_OK_AND_HAS_SUBSTR refactor in the test.
• Container insertion is guarded on !has_kernel_owned_packed_weights_ — kernel-owned packs never enter the shared container.

This is a clean opt-out mechanism. Default is false, so every existing kernel is unaffected. Only MatMul<MLFloat16> sets it in this PR.

The new negative test BrokenKernelWithoutCacheableBuffersFails in session_state_test.cc verifies the invariant holds: a synthetic kernel that reports is_packed = true without providing buffers and without setting has_kernel_owned_packed_weights_ = true causes FinalizeSessionState to fail with the documented substring. Good regression guard.

2. MatMul<MLFloat16> opts out of cross-session sharing by design

matmul.cc:

if (is_packed && prepacked_weights != nullptr) {
    prepacked_weights->has_kernel_owned_packed_weights_ = true;
}

and

Status MatMul<MLFloat16>::UseSharedPrePackedBuffers(...) {
    // Native fp16 packed-B buffers are backend-layout-specific. Decline shared
    // buffers until the shared prepack cache can validate that layout.
    used_shared_buffers = false;
    return Status::OK();
}

Interaction with our earlier design-doc discussion: this is a conservative-but-correct choice for exactly the class of hazard we talked about. Even though the framework's "PrePack-first-then-hash" invariant prevents cross-config buffer adoption today (mismatched configs produce different bytes → different hash keys → distinct slots), this PR simply doesn't rely on that invariant for the FP16 path. Cost: one packed buffer per session when they could otherwise share. Benefit: immune to any future container-lookup optimization. Reasonable trade-off given the layout opacity.

New test MatMulFloat16SharedInitializerWithKernelOwnedPrepack in matmul_test.cc verifies both sessions pack independently and number_of_shared_pre_packed_weights_counter == 0 for the FP16 MatMul path.

3. Conv HalfConv weights+bias combined pack — safe by construction

fp16_conv.cc:

const bool has_valid_constant_halfconv_bias =
    constant_B_ != nullptr &&
    !share_prepacked_weights &&  // <-- disable when sharing is on
    constant_B_->Shape().NumDimensions() == 1 &&
    constant_B_->Shape()[0] == static_cast<int64_t>(output_channels);

The bias is baked into the combined pack only when the session is not participating in cross-session sharing. In shared mode, the HalfConv combined pack is skipped and the generic packed_W_/reordered_W_ path (which is bias-agnostic and safely shareable) is used instead. Comment near share_prepacked_weights confirms:

packed_halfconv_weights_and_bias_buffer_ is session-local only. It encodes a bias choice even when the bias is null, so it must not be shared under the W-only prepack cache key.

Exactly the right analysis — the shared-prepack key is over W bytes, so a W-hash collision could otherwise cause a session with bias b1 to adopt a combined pack from a session with bias b2. Guarding on !share_prepacked_weights at construction time makes this impossible. New tests SharedPrepackedWeights_HalfConvEligible_NoBias and SharedPrepackedWeights_HalfConvEligible_BiasNotShared in conv_fp16_test.cc cover both paths.

UseSharedPrePackedBuffers for the FP16 Conv now has a proper else arm:

} else {
    ORT_ENFORCE(false, "Unexpected number of shared prepacked fp16 conv buffers.");
}

Defensive against future contract drift. Fine.

4. Latent HalfGemm dispatch fix on non-FP16-accel ARM64

halfgemm.h MlasHalfGemmGetDispatch:

#if defined(MLAS_F16VEC_INTRINSICS_SUPPORTED) && defined(MLAS_TARGET_ARM64)
-    return &MlasHalfGemmDispatchNeon;
+    if (MLAS_CPUIDINFO::GetCPUIDInfo().HasFp16VectorAcceleration()) {
+        return &MlasHalfGemmDispatchNeon;
+    }
+    return &MlasHalfGemmDispatchDefault;

Before this fix, any ARM64 build with MLAS_F16VEC_INTRINSICS_SUPPORTED returned the Neon dispatch unconditionally, regardless of whether the actual runtime CPU supported FP16 vector acceleration (FEAT_FP16). On an ARMv8-A core without FEAT_FP16, running MlasHalfGemmDispatchNeon would either produce wrong results (float16 emulation via scalar) or crash (SIGILL on unsupported intrinsics), depending on how MlasHalfGemmDispatchNeon's implementation looks. This is a real correctness fix covered by the new MlasFp16AccelerationSupported() gate. MlasHGemmSupported gained the same early-return check for consistency.

5. MatmulTransposeFusion dtype exclusion for CPU FP16

matmul_transpose_fusion.cc:

if (execution_provider_type == kCpuExecutionProvider) {
    return element_type == FLOAT || element_type == DOUBLE;
}
return element_type == FLOAT || element_type == FLOAT16 || element_type == DOUBLE || ...;

The FP16 MatMul CPU kernel added in this PR handles Transpose+MatMul separately (no com.microsoft.FusedMatMul for CPU FP16). If the fusion ran, it would produce a FusedMatMul node with FP16 that no CPU kernel could execute. New test TransposeMatmulNoFusionForCpuFp16 verifies the fusion is suppressed. Correct scope.

6. HalfGemm backend-selector plumbing cleanup

Simplified MlasHalfGemmBatchOverride signature — dropped the trailing const MLAS_BACKEND_KERNEL_SELECTOR_CONFIG* parameter. The config check (use_kleidiai etc.) is now performed once by TryGetHalfGemmBackendSelectorConfig before the override is called, and the internal check inside ArmKleidiAI::MlasHalfGemmBatch was removed. Same behavior, one authoritative place. All corresponding test-side stubs (TestHalfGemmBatchOverride, MlasHalfGemmTest::CallHalfGemm, and the three HalfGemmKleidiAIPath unit tests) were updated to the new signature.

7. Value-initialization of MLAS data-params structs

Multiple call sites now do MLAS_HALF_GEMM_DATA_PARAMS params{} and MLAS_HGEMM_DATA_PARAMS data[i] fields with zero-init. Small correctness improvement — the Bias and BackendKernelSelectorConfig pointers must be nullptr when unused, and the pre-existing pattern of uninitialized struct + selective field assignment left those pointers as whatever was on the stack.

8. FP16 Gemm no-bias / beta==0 fast path

gemm.cc:

const bool use_mlas_no_bias = beta == onnxruntime::MLFloat16::Zero;
const bool use_mlas_bias = beta == onnxruntime::MLFloat16::One && support_mlas_bias;
if (trans_a == CblasNoTrans && trans_b == CblasNoTrans &&
    alpha == onnxruntime::MLFloat16::One && (use_mlas_no_bias || use_mlas_bias)) {
    ...
    if (use_mlas_bias && c_shape != nullptr) {
        data.Bias = c_data;
    }
    ...
}

Extends the pre-existing MLAS fast path to also cover the case where C is present but beta == 0 (bias is effectively ignored). New tests in gemm_test.cc cover both "missing C" and "beta == 0 with a full-shape C" paths. Clean.

---

CI and merge status

• 67/83 checks OK on the most recent commit 3ef65e5. 16 not-OK.
• The PR was closed and reopened once by @hariharans29 to retrigger Azure Pipelines. Some of the not-OK legs may be recovering from that.
• "At least 1 approving review is required to merge." No approving human review yet; Copilot AI review is present with both comments resolved.

Recommend keeping an eye on the CI legs that haven't come back yet. If any legs fail rather than pending, they'll need triage — but nothing in the code diff itself looks like a plausible cause of new-regression failures at first pass.

---

Nits (not blockers)

1. In MatMul<MLFloat16>::Compute, data[i].BackendKernelSelectorConfig = &mlas_backend_kernel_selector_config_; is set on every entry of the batch even though the pointer is identical for all entries. Same pattern as float, so consistent — not worth refactoring here.
2. The zero-K MatMul refactor EigenMatrixMapRowMajor<T>(...).setZero() → std::fill(..., T{}) is a drive-by simplification and equivalent for float/double/MLFloat16 (all have T{} == 0 bit-representation). Fine.
3. The prepacked_weights.h change is API-adjacent — adding a public bool. Since PrePackedWeights is an ORT-internal type (not exposed through the public C API), backward compat is not an issue. But it's worth flagging that any future PR that serializes PrePackedWeights to disk (which does exist via the external-data path we discussed) would need to think about whether has_kernel_owned_packed_weights_ needs to be part of the serialized manifest. Not this PR's problem.

---

Bottom line

Pass-1 verdict (approve pending Copilot resolution) stands. Both Copilot comments handled well — one accepted with the right fix, one correctly rebutted. The has_kernel_owned_packed_weights_ mechanism is a well-scoped extension point that connects cleanly to our earlier design-doc discussion. The HalfGemm dispatch fix on non-FP16-accel ARM64 is a genuine latent bug fix. Bias-in-Conv-pack correctness is preserved. Test coverage is thorough. Approve when CI is green.

[hariharans29]

Review: PR #29709 — [MLAS][KleidiAI] Add MatMul, Gemm, and Conv FP16 execution paths (head 467007c)

Author: @JonathanC-ARM (with @Laan33 co-author on the initial commit). 9 commits, part 3 of 5 in the fp16-split series. CI on head: 35 / 78 checks OK (still recovering after the last push); pipelines pre-push consistently at 82–83/86. Previously approved by @hariharans29 on 3ef65e5; approval dismissed by a subsequent push and needs to be re-issued.

Verdict: approve when CI settles. This is a substantive, well-scoped PR that wires the previously-landed MLAS FP16 primitives (MlasHalfGemmBatch, backend-native packed-B, MlasHalfConv) into the CPU EP's MatMul, Gemm, and FP16 Conv kernels. Design decisions on prepack sharing are conservative in the right direction, and along the way it fixes a real latent bug where MLAS_F16VEC_INTRINSICS_SUPPORTED (compile-time) was conflated with runtime FP16 vector acceleration. Both Copilot AI comments have been resolved (one accepted with correct fix, one correctly rebutted).

---

What it does

1. Registers CPU MLFloat16 kernels for MatMul (opsets 1–8, 9–12, 13+) and Gemm (opsets 7–8, 9–10, 11–12, 13+) in onnxruntime/core/providers/cpu/cpu_execution_provider.cc.
2. Adds a dedicated MatMul<MLFloat16> implementation in onnxruntime/core/providers/cpu/math/matmul.cc / onnxruntime/core/providers/cpu/math/matmul.h that:
   • Preserves ONNX broadcast + batched execution through MatMulComputeHelper.
   • Handles zero-K / empty-output degenerate cases.
   • Routes to MlasHGemmBatch for small unpacked matrices (M ≤ 2), otherwise MlasHalfGemmBatch.
   • Carries MLAS_BACKEND_KERNEL_SELECTOR_CONFIG* through every batch entry.
   • Optionally pre-packs a constant 2-D B into the active backend's native layout via MlasHalfGemmNativePackBSize / MlasHalfGemmNativePackB.
   • Falls back to math::MatMul<MLFloat16> (Eigen Eigen::half path) when no runtime FP16 acceleration is available and there's no pre-packed B.
3. Extends Gemm_MLFloat16 in onnxruntime/core/providers/cpu/math/gemm.cc to reach MlasHalfGemmBatch for alpha == 1, no-transpose, and effective-zero-bias or MLAS-compatible N-element bias (including beta == 0 with a full-shape C). Retains Eigen fallback for unsupported shapes.
4. Adds an MLAS HalfConv fast path to onnxruntime/core/providers/cpu/fp16/fp16_conv.cc:
   • Detects eligibility via MlasHalfConvPrepare for 2-D convolutions without a fused Sum input.
   • Pre-packs eligible constant filters via MlasHalfConvPackWeightsAndBias, folding a constant 1-D bias into the pack when session-local (see prepack-safety note below).
   • Falls back to the existing reordered-weight + im2col + HalfGemm implementation on rejection.
5. Adds a has_kernel_owned_packed_weights_ marker on PrePackedWeights, consumed by onnxruntime/core/framework/session_state.cc, that lets a kernel legitimately report is_packed=true with no cacheable buffers.
6. Centralizes HalfGemm backend-selector handling in MlasHalfGemmBatch, dropping the trailing selector parameter from MlasHalfGemmBatchOverride and updating all call sites (MoE, MatMul, Gemm, Conv, tests).
7. Excludes CPU FP16 from MatmulTransposeFusion in onnxruntime/core/optimizer/matmul_transpose_fusion.cc (no com.microsoft.FusedMatMul MLFloat16 kernel exists on CPU).
8. Fixes a latent HalfGemm dispatch bug in onnxruntime/core/mlas/lib/halfgemm.h: unconditionally returned MlasHalfGemmDispatchNeon on ARM64 when compile-time intrinsics were enabled, regardless of runtime FEAT_FP16. Adds a MlasFp16AccelerationSupported() gate, plus consistent early-return in MlasHGemmSupported. Same probe added at three CPU EP call sites (MatMul<MLFloat16>::Compute, math::MatMul<MLFloat16>, Gemm_MLFloat16) so the ARM64-without-FEAT_FP16 configuration falls through to Eigen instead of crashing or producing wrong output.
9. Updates operator documentation in docs/OperatorKernels.md to advertise FP16 for CPU MatMul and Gemm.

---

Design-level analysis

has_kernel_owned_packed_weights_ — a clean opt-out from cross-session sharing

struct PrePackedWeights final {
  InlinedVector<IAllocatorUniquePtr<void>> buffers_;
  InlinedVector<size_t> buffer_sizes_;
  // Set when PrePack successfully produced kernel-owned packed weights that
  // intentionally cannot be stored in a shared pre-packed weights container.
  bool has_kernel_owned_packed_weights_{false};
  ...
};

Propagated through CreateReferringCopy. Consumed in onnxruntime/core/framework/session_state.cc:

• The pre-existing invariant ORT_ENFORCE(weights_to_be_filled_in.buffers_.size() > 0, ...) is relaxed to ORT_RETURN_IF_NOT(!buffers.empty() || has_kernel_owned_packed_weights_, ...). Flipping from ORT_ENFORCE (throws) to ORT_RETURN_IF_NOT (returns Status) is a correctness improvement for ORT_NO_EXCEPTIONS builds — a broken kernel now aborts session init cleanly instead of calling abort().
• Container insertion is guarded on !has_kernel_owned_packed_weights_ — kernel-owned packs never enter the shared container.

Default false, so every existing kernel is unaffected. Only MatMul<MLFloat16> opts in this PR. The negative test BrokenKernelWithoutCacheableBuffersFails in onnxruntime/test/framework/session_state_test.cc pins the invariant with ASSERT_STATUS_NOT_OK_AND_HAS_SUBSTR, which handles both ORT_NO_EXCEPTIONS and default builds uniformly.

MatMul<MLFloat16> opts out by design

Status MatMul<MLFloat16>::PrePack(...) {
  ...
  if (is_packed && prepacked_weights != nullptr) {
    prepacked_weights->has_kernel_owned_packed_weights_ = true;
  }
  return Status::OK();
}

Status MatMul<MLFloat16>::UseSharedPrePackedBuffers(...) {
  // Native fp16 packed-B buffers are backend-layout-specific. Decline shared
  // buffers until the shared prepack cache can validate that layout.
  used_shared_buffers = false;
  return Status::OK();
}

Conservative-but-correct choice. Even though the framework's "PrePack-first-then-hash" invariant prevents cross-config buffer adoption today (mismatched configs produce different bytes → different hash keys → distinct slots), this PR simply doesn't rely on that invariant for the FP16 path. Cost: one packed buffer per session where they could otherwise share. Benefit: immune to any future container-lookup optimization. Reasonable tradeoff given the layout opacity across KleidiAI vs. non-KleidiAI backends. New test MatMulFloat16SharedInitializerWithKernelOwnedPrepack in onnxruntime/test/providers/cpu/math/matmul_test.cc verifies both sessions pack independently and number_of_shared_pre_packed_weights_counter == 0.

Conv HalfConv weights+bias combined pack — safe by construction

const bool has_valid_constant_halfconv_bias =
    constant_B_ != nullptr &&
    !share_prepacked_weights &&        // <-- disable when sharing is on
    constant_B_->Shape().NumDimensions() == 1 &&
    constant_B_->Shape()[0] == static_cast<int64_t>(output_channels);

Bias is baked into the combined pack only when the session is not participating in cross-session sharing. In shared mode, the HalfConv combined pack is skipped and the generic packed_W_ / reordered_W_ path (bias-agnostic, safely shareable) is used instead. Comment in the diff makes it explicit:

packed_halfconv_weights_and_bias_buffer_ is session-local only. It encodes a bias choice even when the bias is null, so it must not be shared under the W-only prepack cache key.

Exactly the right analysis. The shared-prepack key is over W bytes, so a W-hash collision would otherwise let a session with bias b1 adopt a combined pack from a session with bias b2. UseSharedPrePackedBuffers in the FP16 Conv now has a proper else arm that ORT_ENFORCE-rejects unexpected buffer counts — defensive against future contract drift. New tests SharedPrepackedWeights_HalfConvEligible_NoBias and SharedPrepackedWeights_HalfConvEligible_BiasNotShared in onnxruntime/test/providers/cpu/nn/conv_fp16_test.cc cover both paths.

Latent HalfGemm dispatch fix on non-FP16-accel ARM64

The pre-existing MlasHalfGemmGetDispatch had:

#if defined(MLAS_F16VEC_INTRINSICS_SUPPORTED) && defined(MLAS_TARGET_ARM64)
    return &MlasHalfGemmDispatchNeon;

which was wrong: an ARM64 build with the compile flag set would return the Neon dispatch even at runtime on an ARMv8-A core without FEAT_FP16, producing wrong results or SIGILL depending on the intrinsic path. Fixed to:

#if defined(MLAS_F16VEC_INTRINSICS_SUPPORTED) && defined(MLAS_TARGET_ARM64)
    if (MLAS_CPUIDINFO::GetCPUIDInfo().HasFp16VectorAcceleration()) {
        return &MlasHalfGemmDispatchNeon;
    }
    return &MlasHalfGemmDispatchDefault;

The same runtime probe (MlasFp16AccelerationSupported() || MlasHalfGemmNativePackBSize(...) != 0 — the OR-branch handles KleidiAI-native-B backends that supply acceleration through packing) is added at three additional CPU EP call sites:

• MatMul<MLFloat16>::Compute in onnxruntime/core/providers/cpu/math/matmul.cc — fallback to math::MatMul<MLFloat16> on the non-packed_b_ path.
• math::MatMul<MLFloat16> in onnxruntime/core/util/math_cpu.cc — replaces the whole-function #ifdef MLAS_F16VEC_INTRINSICS_SUPPORTED (which made the helper a silent no-op on non-intrinsics builds) with a runtime check + Eigen fallback that is always compiled.
• Gemm_MLFloat16 in onnxruntime/core/providers/cpu/math/gemm.cc — analogous replacement of the whole-function #ifdef with a runtime gate that falls through to the Eigen bias-broadcast path when acceleration isn't available.

This is a coherent 4-site fix for the same underlying latent bug class. Real correctness improvement, not just a defensive change.

MatmulTransposeFusion CPU-FP16 exclusion

if (execution_provider_type == kCpuExecutionProvider) {
    return element_type == FLOAT || element_type == DOUBLE;
}
return element_type == FLOAT || element_type == FLOAT16 || element_type == DOUBLE || ...;

com.microsoft.FusedMatMul has no CPU MLFloat16 kernel. Without this guard, the fusion would produce a FusedMatMul node with FP16 T that no CPU kernel could execute. New test TransposeMatmulNoFusionForCpuFp16 in onnxruntime/test/optimizer/graph_transform_test.cc verifies suppression. Correctly scoped.

HalfGemm backend-selector plumbing cleanup

MlasHalfGemmBatchOverride and ArmKleidiAI::MlasHalfGemmBatch drop the trailing const MLAS_BACKEND_KERNEL_SELECTOR_CONFIG* parameter. The use_kleidiai check is now performed once by TryGetHalfGemmBackendSelectorConfig before the override is invoked, and the duplicate check inside ArmKleidiAI::MlasHalfGemmBatch is removed. Same behavior, one authoritative place. All corresponding test-side stubs (TestHalfGemmBatchOverride, MlasHalfGemmTest::CallHalfGemm, and the three HalfGemmKleidiAIPath unit tests in onnxruntime/test/mlas/unittest/test_halfgemm.cpp / onnxruntime/test/mlas/unittest/test_halfgemm.h) are updated to the new signature.

Value-initialization of MLAS data-params structs

Multiple call sites now use MLAS_HALF_GEMM_DATA_PARAMS params{} and value-initialize MLAS_HGEMM_DATA_PARAMS data[i] fields. Small correctness improvement — the Bias and BackendKernelSelectorConfig pointers must be nullptr when unused, and the pre-existing pattern of uninitialized struct + selective field assignment left those pointers indeterminate.

FP16 Gemm no-bias / beta==0 fast path

const bool use_mlas_no_bias = beta == onnxruntime::MLFloat16::Zero;
const bool use_mlas_bias    = beta == onnxruntime::MLFloat16::One && support_mlas_bias;
if (has_accelerated_half_gemm && trans_a == CblasNoTrans && trans_b == CblasNoTrans &&
    alpha == onnxruntime::MLFloat16::One && (use_mlas_no_bias || use_mlas_bias)) {
  ...
  if (use_mlas_bias && c_shape != nullptr) {
    data.Bias = c_data;
  }
  ...
}

Extends the pre-existing MLAS fast path to cover both "no C" and "C present but beta == 0" (bias effectively ignored). Two new sub-tests in onnxruntime/test/providers/cpu/math/gemm_test.cc exercise each.

---

Test coverage

Framework-level (session state). BrokenKernelWithoutCacheableBuffersFails — synthetic kernel that reports is_packed=true without buffers and without has_kernel_owned_packed_weights_=true → FinalizeSessionState fails with the documented substring. Regression guard for the new invariant.

MatMul.

• MatMulFloat16Cpu — M=3, K=2, N=2, integer inputs, no custom tolerance. Exercises the dynamic-B, M>2 dispatch path.
• MatMulFloat16SharedInitializerWithKernelOwnedPrepack — verifies kernel-owned packs across two sessions with shared-initializers enabled; asserts number_of_shared_pre_packed_weights_counter == 0. GTEST_SKIP() when the accelerated path isn't produced.

Gemm.

• GemmNoTrans_f16 gets two new inline sub-blocks: (a) missing C with effective beta == 0, (b) C present but beta == 0 explicitly. Both SetOutputTolerance(0.005f) — comfortable for the small integer-ish outputs used.

Conv.

• Conv2D_KleidiAiImatmulEligibleNoBias, Conv2D_KleidiAiImatmulEligibleBiasAndDisabledFallback, NhwcFusedConv2D_KleidiAiImatmulEligibleBiasAndDisabledFallback — HalfConv-eligible NCHW/NHWC, with/without bias, KleidiAI-enabled and KleidiAI-disabled fallback. Correct MLAS gate: #if !defined(__aarch64__) && !defined(_M_ARM64) skips + MlasFp16AccelerationSupported() skips.
• SharedPrepackedWeights_HalfConvEligible_NoBias and SharedPrepackedWeights_HalfConvEligible_BiasNotShared — verify that generic W prepacks share across sessions while bias-dependent HalfConv packs don't.

MLAS unit tests. Signature updates for the four call sites of the pruned MlasHalfGemmBatchOverride. HalfGemmKleidiAIPath tests continue to exercise the BIsBackendNativePacked invariant, output-processor rejection, and zero-K handling.

Optimizer. TransposeMatmulNoFusionForCpuFp16 — pins the CPU-FP16 fusion suppression.

---

Tolerance changes — all defensible

1. _expanded overrides in onnxruntime/test/testdata/onnx_backend_test_series_overrides.jsonc: three new entries matching their fused siblings (5e-4, 6e-4, 5e-4). Correct strategy — the expanded form does more materialization of fp16 intermediates but the underlying arithmetic and reference are the same order of error. Empirically OK on 959658d's CI. If any of the three flakes post-merge, a one-line bump is the fix.

2. GatherRobertaE2E in onnxruntime/test/optimizer/compute_optimizer_test.cc: loosened 2e-3 → 1e-2 for CPU only, other EPs keep 2e-3. Justified because CPU now executes the Roberta MatMuls natively in fp16 (K≈768). Rough worst-case fp16 relative error is $\sqrt{K}\cdot\varepsilon_{\mathrm{fp16}}\approx 1.35\times10^{-2}$, so 1e-2 sits near the theoretical boundary — tight for adversarial inputs, comfortable for realistic ones. Correctly scoped to kCpuExecutionProvider. Same value used for both atol and rtol, preserving the original both-2e-3 semantic.

3. New conv/gemm/matmul tests use 0.002f rel_error (matching pre-existing FP16 test style) or SetOutputTolerance(0.005f) (small integer outputs). No regression in tolerance strictness.

---

Copilot review disposition

Both Copilot comments handled correctly:

• Copilot Set up CI with Azure Pipelines #1 (Medium): input_defs[2]->Exists() in onnxruntime/core/providers/cpu/fp16/fp16_conv.cc, "risks null-pointer crash for Conv without B". Correctly rebutted by the author. ORT does not represent an omitted optional input as a null NodeArg* in InputDefs(). Absent-from-NodeProto shortens InputDefs().size() (handled by the preceding input_defs.size() >= 3 guard). Present-but-empty is materialized as a valid NodeArg* whose Exists() returns false. input_defs[i]->Exists() is the ORT canonical pattern — the same file, conv.cc, gemm.cc, and many others use it without null checks.
• Copilot Remove vsts test runner in cmake file #2 (Medium): try/catch on ORT_THROW_IF_ERROR breaks ORT_NO_EXCEPTIONS builds. Accepted by the author and converted to ASSERT_STATUS_NOT_OK_AND_HAS_SUBSTR, which handles both build modes uniformly. Correct fix.

---

Nits (not blockers)

1. In MatMul<MLFloat16>::Compute, data[i].BackendKernelSelectorConfig = &mlas_backend_kernel_selector_config_ is set on every entry even though the pointer is identical for all entries. Same pattern as float, so consistent — not worth refactoring here.
2. The three has_accelerated_half_gemm check sites duplicate the same 4-line probe. Reasonable to factor into a helper (e.g. bool MlasHalfGemmDispatchAvailable(N, K, config) in onnxruntime/core/mlas/lib/mlasi.h or a public MLAS header) in a follow-up. Not blocking.
3. The zero-K MatMul refactor EigenMatrixMapRowMajor<T>(...).setZero() → std::fill(..., T{}) is a drive-by simplification and equivalent for float / double / MLFloat16 (all have T{} == 0 bit-representation).
4. onnxruntime/core/framework/prepacked_weights.h is API-adjacent, but PrePackedWeights is ORT-internal (not exposed through the public C API) so backward compat isn't a concern. Worth flagging that any future PR serializing PrePackedWeights to disk (external-data path) would need to think about whether has_kernel_owned_packed_weights_ belongs in the serialized manifest. Not this PR's problem.
5. In onnxruntime/core/providers/cpu/math/gemm.cc, has_accelerated_half_gemm is computed unconditionally even when alpha != 1 or transpose flags mean the MLAS path can't be taken. Minor wasted work (one MlasHalfGemmNativePackBSize probe on every FP16 Gemm) but well below noise.

---

CI / merge state

• 35 / 78 checks OK on 467007c at fetch time, still recovering after the last push. Pre-push commits (3ef65e5, b61832e) hit 82–83 / 86. If any leg lands as a fail rather than pending, the three latest fallback fixes are the likely suspects to check first — they touch dispatch paths that a CPU-FP16-heavy test could probe.
• @hariharans29's approval was dismissed on 959658d (auto-merge disabled by a "user without write access" push signature). Needs re-approval and auto-merge re-enable.
• "At least 1 approving review is required to merge this pull request."

---

Bottom line

Substantial, well-motivated PR that lands the CPU EP side of ORT's FP16 story on ARM64 (and elsewhere as backends materialize). The design decisions on prepack safety are conservative in the right direction, the correctness improvements around runtime FP16 gating fix a genuine latent bug across four sites, and the test coverage exercises every meaningful new path including the negative session-init invariant. Both Copilot comments are handled correctly. Approve when CI is green and re-approval is issued.

[hariharans29]

Pass-3 re-review: PR #29709 — [MLAS][KleidiAI] Add MatMul, Gemm, and Conv FP16 execution paths (head 385bdf1)

Author: @JonathanC-ARM (with @Laan33 co-author on the initial commit). Force-pushed since my pass-2 review. The branch has been rebased to 7 flat commits (all authored 1 hour ago at fetch time):

#	SHA	Title	Since pass
1	b25c723	Add CPU EP fp16 MatMul Gemm and Conv prepack paths (main)	pass-1 base
2	1642972	Address CPU fp16 prepack review feedback	pass-1
3	c8aa059	Address review feedback	pass-2
4	0031eaa	Address CPU FP16 review feedback	pass-2
5	0cd6d71	Align expanded FP16 attention tolerances	new
6	7d97b21	Fix CPU FP16 MatMul fallback and test tolerance	new
7	385bdf1	Fix CPU FP16 Gemm fallback dispatch	new (head)

CI: 1 / 68 checks OK on 385bdf1 — pipelines just spun up on the force-push, too early to draw conclusions. Pre-push (at pass-2 heads 467007c and 3ef65e5) the same code was at 35-83 / 78-86 depending on stage. Force-push also auto-dismissed @hariharans29's previous approval — the PR now shows "At least 1 approving review is required to merge."

Verdict: pass-2 approval stands. The three new commits are targeted follow-ups to CI failures I explicitly flagged as the most likely regression source in pass-2, and they land exactly where I said to look. Re-request approval from @hariharans29 and merge when CI settles.

---

Delta from pass-2 head (467007c) — three targeted fixes

Commit 5 (0cd6d71): "Align expanded FP16 attention tolerances" — proactive follow-up to pass-2 nit. My pass-2 flagged:

If any of the three [_expanded overrides in onnx_backend_test_series_overrides.jsonc] flakes post-merge, a one-line bump is the fix.

This commit aligns the tolerances before waiting for a flake. Sensible.

Commit 6 (7d97b21): "Fix CPU FP16 MatMul fallback and test tolerance" — targeted fix. My pass-2 explicitly called out:

If any leg lands as a fail rather than pending, the three latest fallback fixes are the likely suspects to check first — they touch dispatch paths that a CPU-FP16-heavy test could probe.

This commit fixes the MatMul<MLFloat16>::Compute fallback path — one of the exact three sites where the MlasFp16AccelerationSupported() || MlasHalfGemmNativePackBSize(...) != 0 gate was added in pass-2. Along with a test tolerance update.

Commit 7 (385bdf1): "Fix CPU FP16 Gemm fallback dispatch" — same shape for Gemm_MLFloat16. Second of the three sites. Consistent with commit 6.

The pattern (fix MatMul fallback → fix Gemm fallback) suggests CI on the pass-2 head caught an issue in the runtime-FP16-gating check on some non-FEAT_FP16 leg, and these two commits are the surgical fixes. Given the pass-2 correctness argument for those sites was fundamentally sound (the fix went from unconditional Neon dispatch → runtime probe + Eigen fallback), the follow-up here is likely a corner case in how the probe/fallback interacts with the pre-packed-B path or a specific test's assumptions about the dispatched kernel.

Without the specific diff content for those two commits, I can't rule out that they introduce a design-level change, but the commit titles and small-scope naming (Fix ... fallback, Fix ... dispatch) strongly suggest one-site-per-file surgical fixes rather than architectural revisions.

---

Everything from passes 1 and 2 still stands

Design substance unchanged since pass-2:

• has_kernel_owned_packed_weights_ marker on PrePackedWeights — clean opt-out from cross-session sharing. Consumed in session_state.cc via ORT_RETURN_IF_NOT(!buffers.empty() || has_kernel_owned_packed_weights_, ...). Container insertion guarded on !has_kernel_owned_packed_weights_. Default false so every existing kernel is unaffected.
• MatMul<MLFloat16> opts out by design — PrePack sets the marker; UseSharedPrePackedBuffers returns used_shared_buffers = false. Conservative-but-correct choice given FP16 packed-B layout opacity across KleidiAI vs. non-KleidiAI backends.
• Conv HalfConv weights+bias combined pack — bias baked in only when !share_prepacked_weights, correctly preventing W-hash-collision-based cross-session bias adoption.
• Latent HalfGemm dispatch bug fix in halfgemm.h — added runtime HasFp16VectorAcceleration() probe. Real correctness fix.
• MatmulTransposeFusion CPU-FP16 exclusion — correct scope; no CPU com.microsoft.FusedMatMul for MLFloat16.
• HalfGemm backend-selector plumbing cleanup — pruned trailing MLAS_BACKEND_KERNEL_SELECTOR_CONFIG* from MlasHalfGemmBatchOverride and downstream stubs.
• Value-initialization of MLAS data-params structs — zero-init Bias and BackendKernelSelectorConfig pointers.
• FP16 Gemm no-bias / beta==0 fast path — extends pre-existing fast path to the beta == 0 with full-shape C case.
• Both Copilot AI comments handled correctly — one accepted (ORT_THROW_IF_ERROR → ASSERT_STATUS_NOT_OK_AND_HAS_SUBSTR), one correctly rebutted (input_defs[i]->Exists() is the canonical ORT pattern).

Test coverage across framework (BrokenKernelWithoutCacheableBuffersFails), MatMul (dynamic + shared-initializer), Gemm (no-C + beta==0), Conv (HalfConv-eligible NCHW/NHWC, with/without bias, KleidiAI-enabled/disabled fallback), and the fusion-exclusion (TransposeMatmulNoFusionForCpuFp16) is unchanged from pass-2 and remains thorough.

---

CI / merge state

• Head 385bdf1: 1 / 68 OK — brand new, still spinning up.
• Approval history: @hariharans29 had approved 3ef65e5 earlier; auto-dismissed by the force-push. Currently zero approvals.
• Both Copilot comments remain resolved on the force-pushed history.
• The three "Fix ... fallback / dispatch / tolerance" commits are the most direct evidence that a CI leg on the previous head surfaced a specific failure the author has now targeted. Whether the fix sticks will be clear once ARM64 / non-FP16-accel test legs report back — most likely candidates to watch:
  • The Linux ARM64 CI leg (if it lands on a non-FEAT_FP16 machine): watch MatMulFloat16* and Gemm*_f16 results.
  • The TransposeMatmulNoFusionForCpuFp16 and BrokenKernelWithoutCacheableBuffersFails framework tests: unchanged from pass-2, should stay green.

---

Nits from prior passes still apply (all non-blocking)

1. data[i].BackendKernelSelectorConfig set redundantly per batch entry in MatMul<MLFloat16>::Compute. Matches the pattern for float, not worth refactoring here.
2. The three has_accelerated_half_gemm probe sites (MatMul / math::MatMul / Gemm) duplicate the same 4-line check. Refactoring into a helper bool MlasHalfGemmDispatchAvailable(N, K, config) would tighten it. Follow-up.
3. PrePackedWeights gains a public bool has_kernel_owned_packed_weights_. Internal type; no C-API impact. Future PRs that serialize PrePackedWeights (external-data path) will need to decide whether to include it in the manifest. Not this PR's problem.
4. In gemm.cc, has_accelerated_half_gemm computed unconditionally even when alpha != 1 or transpose flags rule out MLAS. Minor wasted probe per call. Sub-noise.

---

Bottom line

The three new commits since pass-2 are targeted follow-ups to exactly the failure class I flagged as most likely to surface in CI (the runtime FP16 gate at the MatMul, math::MatMul, and Gemm_MLFloat16 fallback sites). Naming and scope suggest surgical fixes rather than architectural changes. The design substance from the pass-2 approval — has_kernel_owned_packed_weights_ marker, prepack-safety analysis, HalfGemm dispatch fix, MatmulTransposeFusion exclusion, backend-selector plumbing cleanup — is untouched.

Action items:

1. @JonathanC-ARM — nothing required beyond letting CI settle on 385bdf1.
2. @hariharans29 — re-approval needed; the force-push auto-dismissed the earlier one. The pass-2 approval rationale (3ef65e5) all carries forward except the three new commits, which are described here.
3. Watch the ARM64 non-FEAT_FP16 CI legs first; that's where the pass-2 fixes were probing new dispatch behavior and where these three follow-up commits are aimed.

Ready to merge when CI reaches green on 385bdf1 and a Microsoft-side approver signs off.