[WebGPU EP] Fix unnecessary CPU fallback and improve fallback diagnostic by mingmingtasd · Pull Request #29682 · microsoft/onnxruntime

mingmingtasd · 2026-07-13T08:40:49Z

Some nodes in MobileNet and the SD-Turbo pipeline were assigned to the CPU EP even though the WebGPU EP can run them, causing extra GPU<->CPU copies and, in the device-free compile-only flow (where CPU fallback is disabled), aborting session initialization outright. This adds the missing WebGPU kernel coverage and improves the diagnostics that explain why a node is not claimed by an EP. With these fixes all four SD-Turbo sessions (text encoder, UNet, VAE, safety checker) partition fully onto the WebGPU EP with zero CPU fallback.

Kernel coverage fixes, by model

MobileNet — AveragePool (pool.cc, webgpu_execution_provider.cc): register kernels for opsets 11-18, 19-21 and 22, replacing the single opset-11 registration. Root cause: MobileNet (opset 21) uses AveragePool at SinceVersion 19. ORT clamps an open-ended SinceVersion(11) registration to [11, 18] (not [11, INT_MAX]), so the opset-19 node was not claimed and fell back to the CPU EP; the CPU-only NchwcTransformer (Level3) then amplified the single pool into 3 CPU nodes.

SD-Turbo text encoder (CLIP) — Clip int32/uint32 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): register int32 and uint32 Clip for opsets 11-11, 12-12 and 13. Root cause: the CLIP text position-id / index subgraph uses integer Clip, which was previously CPU-only.

SD-Turbo text encoder (CLIP) — Clip int64 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): add a dedicated ClipInt64 kernel/shader for opsets 11-11, 12-12 and 13. Root cause: the WebNN-inserted index Clip before ArgMax (Inserted_Clip_951) is int64. The templated Clip supports only 4-byte element types (sizeof(T)==sizeof(float) static_assert), so int64 previously stayed on the CPU EP and, under compile-only (CPU fallback disabled), stopped the text encoder session from compiling. WebGPU has no native 64-bit integer type; the EP already stores int64 as vec2<u32> and operates on the truncated low 32 bits as i32 (shader_variable.cc GetByOffset/SetByOffset). Because Clip is monotonic, ClipInt64 clamps that i32 value (min/max saturated into i32 range) and sign-extends on write, consistent with the EP's existing int64 semantics.

SD-Turbo safety checker — Cast to uint8 (cast.cc, shader_variable.cc): allow the WebGPU Cast to produce uint8 output. Add uint8 to the Cast output (T2) type constraint, a to-uint8 shader expression, and a Uint8x4 packing path in SetByOffset (4 uint8 packed per u32, low element -> low byte -- the same byte layout already used for bool output). Root cause: WebNN has no bool type and represents a bool graph output as uint8, so the safety checker ends in bool->uint8 casts (Inserted_Cast_972, Inserted_Cast_974). uint8 was not a supported WebGPU Cast output type, so these terminal casts fell back to the CPU EP. Non-terminal bool->uint8->int32 cast chains can be collapsed upstream by CastChainElimination (a Level1 rewrite rule), but a terminal cast has no downstream cast to fold into and must be produced directly, hence this uint8 output support.

Correctness fix exposed by the above

AveragePool count_include_pad divisor (pool.cc): when count_include_pad is set, the shader divided by the full kernel size. That is wrong with ceil_mode, which can extend the last window past the padded region (input + end padding) -- those overflow cells must not be counted. Now count is accumulated in the kernel loop and, under count_include_pad, excludes cells beyond the padded region (detected with signed arithmetic, since u32 window indices underflow for begin padding, which is inside the region). This latent WebGPU pool-kernel bug was previously masked because opset-19 AveragePool fell back to CPU; the registration above exposes it (see PoolTest.AveragePool_19_ceil_count_include_pad_1d).

Diagnostics

These do not change placement; they make unexpected fallbacks diagnosable.

execution_provider.cc (IExecutionProvider::GetCapability): log (VERBOSE) when an EP has no kernel for a still-unassigned node, including op type, opset and domain.
kernel_lookup.h (LookUpKernel): capture and log (VERBOSE) the reason TryFindKernel failed (version vs. type-constraint mismatch, or no registration at all). The reason was previously built but discarded by the pointer-only return, leaving only an opaque "kernel not found". This identified Inserted_Clip_951 as int64 (not int32) and the safety checker fallbacks as uint8 casts.
session_state.cc (verbose node-placement dump): include the op set version and the input/output types for each node, so a fallback caused by an unsupported element type is obvious from the placement log.

Tests

clip_test.cc: Clip_int64_WebGpu — int64 Clip on the WebGPU EP.
cast_op_test.cc: BoolToUint8_WebGpu — bool->uint8 Cast on the WebGPU EP, with a non-multiple-of-4 shape to exercise the partial packed word.
The existing PoolTest.AveragePool_19_ceil_count_include_pad_1d now runs on the WebGPU EP and passes with the divisor fix.

azure-pipelines · 2026-07-13T08:41:03Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

…tics Some nodes in MobileNet and the SD-Turbo pipeline were assigned to the CPU EP even though the WebGPU EP can run them, causing extra GPU<->CPU copies and, in the device-free compile-only flow (where CPU fallback is disabled), aborting session initialization outright. This adds the missing WebGPU kernel coverage and improves the diagnostics that explain why a node is not claimed by an EP. With these fixes all four SD-Turbo sessions (text encoder, UNet, VAE, safety checker) partition fully onto the WebGPU EP with zero CPU fallback. Kernel coverage fixes, by model: * MobileNet -- AveragePool (pool.cc, webgpu_execution_provider.cc): register kernels for opsets 11-18, 19-21 and 22, replacing the single opset-11 registration. Root cause: MobileNet (opset 21) uses AveragePool at SinceVersion 19. ORT clamps an open-ended SinceVersion(11) registration to [11, 18] (not [11, INT_MAX]), so the opset-19 node was not claimed and fell back to the CPU EP; the CPU-only NchwcTransformer (Level3) then amplified the single pool into 3 CPU nodes. * SD-Turbo text encoder (CLIP) -- Clip int32/uint32 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): register int32 and uint32 Clip for opsets 11-11, 12-12 and 13. Root cause: the CLIP text position-id / index subgraph uses integer Clip, which was previously CPU-only. * SD-Turbo text encoder (CLIP) -- Clip int64 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): add a dedicated ClipInt64 kernel/shader for opsets 11-11, 12-12 and 13. Root cause: the WebNN-inserted index Clip before ArgMax (Inserted_Clip_951) is int64. The templated Clip supports only 4-byte element types (sizeof(T)==sizeof(float) static_assert), so int64 previously stayed on the CPU EP and, under compile-only (CPU fallback disabled), stopped the text encoder session from compiling. WebGPU has no native 64-bit integer type; the EP already stores int64 as vec2<u32> and operates on the truncated low 32 bits as i32 (shader_variable.cc GetByOffset/SetByOffset). Because Clip is monotonic, ClipInt64 clamps that i32 value (min/max saturated into i32 range) and sign-extends on write, consistent with the EP's existing int64 semantics. * SD-Turbo safety checker -- Cast to uint8 (cast.cc, shader_variable.cc): allow the WebGPU Cast to produce uint8 output. Add uint8 to the Cast output (T2) type constraint, a to-uint8 shader expression, and a Uint8x4 packing path in SetByOffset (4 uint8 packed per u32, low element -> low byte -- the same byte layout already used for bool output). Root cause: WebNN has no bool type and represents a bool graph output as uint8, so the safety checker ends in bool->uint8 casts (Inserted_Cast_972, Inserted_Cast_974). uint8 was not a supported WebGPU Cast output type, so these terminal casts fell back to the CPU EP. Non-terminal bool->uint8->int32 cast chains can be collapsed upstream by CastChainElimination (a Level1 rewrite rule), but a terminal cast has no downstream cast to fold into and must be produced directly, hence this uint8 output support. Correctness fix exposed by the above: * AveragePool count_include_pad divisor (pool.cc): when count_include_pad is set, the shader divided by the full kernel size. That is wrong with ceil_mode, which can extend the last window past the padded region (input + end padding): those overflow cells must not be counted. Now count is accumulated in the kernel loop and, under count_include_pad, excludes cells beyond the padded region (detected with signed arithmetic, since u32 window indices underflow for begin padding, which is inside the region). This latent WebGPU pool-kernel bug was previously masked because opset-19 AveragePool fell back to CPU; the registration above exposes it (see PoolTest.AveragePool_19_ceil_count_include_pad_1d). Diagnostics (these do not change placement; they make unexpected fallbacks diagnosable): * execution_provider.cc (IExecutionProvider::GetCapability): log (VERBOSE) when an EP has no kernel for a still-unassigned node, including op type, opset and domain. This is the root-cause signal for why a node is not claimed. * kernel_lookup.h (LookUpKernel): capture and log (VERBOSE) the reason TryFindKernel failed (version vs. type-constraint mismatch, or no registration at all). The reason was previously built but discarded by the pointer-only return, leaving only an opaque "kernel not found". This identified Inserted_Clip_951 as int64 (not int32) and the safety checker fallbacks as uint8 casts. * session_state.cc (verbose node-placement dump): include the op set version and the input/output types for each node, so a fallback caused by an unsupported element type is obvious from the placement log. Tests: * clip_test.cc: Clip_int64_WebGpu -- int64 Clip on the WebGPU EP. * cast_op_test.cc: BoolToUint8_WebGpu -- bool->uint8 Cast on the WebGPU EP, with a non-multiple-of-4 shape to exercise the partial packed word. * The existing PoolTest.AveragePool_19_ceil_count_include_pad_1d now runs on the WebGPU EP and passes with the divisor fix.

mingmingtasd · 2026-07-15T00:56:09Z

@adrastogi Please help review, thanks!

cc/ @huningxin @ibelem

Copilot

Pull request overview

This PR improves WebGPU Execution Provider (EP) graph partitioning by adding missing kernel coverage (so supported nodes don’t unnecessarily fall back to CPU) and enhancing verbose diagnostics to explain why an EP did not claim a node. It also fixes a correctness issue in the WebGPU AveragePool shader exposed once opset coverage is expanded.

Changes:

Expand WebGPU kernel registrations for AveragePool (opset/version ranges) and Clip (int32/uint32/int64), and add WebGPU Cast output support for uint8.
Fix WebGPU AveragePool count_include_pad handling under ceil_mode by computing the divisor per-window.
Add/extend tests to validate WebGPU execution for bool→uint8 Cast and int64 Clip, plus richer verbose placement/kernel-miss logging.

Show a summary per file

File	Description
onnxruntime/test/providers/cpu/tensor/cast_op_test.cc	Adds a WebGPU EP test for bool→uint8 Cast, including non-multiple-of-4 packing coverage.
onnxruntime/test/providers/cpu/math/clip_test.cc	Adds a WebGPU EP test for int64 Clip behavior.
onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc	Registers additional WebGPU kernels for Clip integer types and AveragePool opset ranges.
onnxruntime/core/providers/webgpu/tensor/cast.cc	Enables uint8 as a Cast output type and emits uint8-specific shader casting expression.
onnxruntime/core/providers/webgpu/shader_variable.cc	Adds Uint8x4 packing support in SetByOffset for uint8 outputs.
onnxruntime/core/providers/webgpu/nn/pool.cc	Adjusts AveragePool divisor calculation for `count_include_pad` with `ceil_mode` overflow exclusion.
onnxruntime/core/providers/webgpu/math/unary_elementwise_ops.cc	Adds a dedicated int64 Clip kernel/program for WebGPU.
onnxruntime/core/framework/session_state.cc	Enhances verbose node placement dump with opset version and input/output types.
onnxruntime/core/framework/kernel_lookup.h	Logs (VERBOSE) kernel lookup miss reasons instead of discarding TryFindKernel diagnostics.
onnxruntime/core/framework/execution_provider.cc	Logs (VERBOSE) when an EP has no kernel for a still-unassigned node.

Review details

Comments suppressed due to low confidence (1)

onnxruntime/core/providers/webgpu/nn/pool.cc:127

AveragePool shader generation can now produce count == 0 when count_include_pad is enabled and ceil_mode creates an output window entirely beyond the padded region (all cells flagged is_outside_padded_region). That leads to a division by zero (value /= f32(count) or later count_shared[0]) and NaNs/inf in output. Clamp the divisor to a minimum of 1 (or guard the divide) so empty windows yield 0 consistently.

    SS(downsampling_ss, kStringInitialSize);
    if (are_small_output_big_kernel_) {
      downsampling_ss << "  sum_or_max_shared[local_idx] = value;\n"
                      << "  count_shared[local_idx] = count;\n";
    } else {
      downsampling_ss << "  value /= " << (is_float16_ ? "f16" : "f32") << "(count);\n";
    }
    downsampling_code = SS_GET(downsampling_ss);

Files reviewed: 10/10 changed files
Comments generated: 2
Review effort level: Low

+    // Diagnostic: surface *why* no kernel matched (version vs. type-constraint mismatch, or no
+    // registration at all). TryFindKernel builds this reason but the pointer-only return discards it,
+    // which otherwise leaves EP GetCapability logging only an opaque "kernel not found".
+    LOGS(logger_, VERBOSE) << "LookUpKernel[" << provider_type_ << "] no kernel for "
+                           << node.OpType() << "(" << node.Name() << "): "
+                           << (last_reason.empty() ? "no candidate kernels registered for this op/domain/provider"
+                                                   : last_reason);


+    } else if (node.GetExecutionProviderType().empty()) {
+      // No kernel was found in this EP's registry for a still-unassigned node. Logging the op type,
+      // domain and resolved opset makes it obvious why a node is not claimed by the EP (and therefore
+      // falls back to another provider such as the CPU EP). This is the root-cause signal for
+      // unexpected CPU fallbacks, e.g. an op registered only for a different opset range or
+      // domain/layout. Skipping nodes already assigned to another EP keeps the output readable.
+      LOGS_DEFAULT(VERBOSE) << "EP [" << Type() << "] has no kernel for node '" << node.Name()
+                            << "': " << node.OpType() << "(" << node.SinceVersion() << ") domain=["
+                            << node.Domain() << "] -> not claimed by this EP.";
    }


mingmingtasd · 2026-07-17T01:57:41Z

A soft ping, @edgchen1 @adrastogi please help review this pr, thanks!

huningxin · 2026-07-21T20:06:04Z

@mingmingtasd, thanks for making this PR!

My main feedback is scope: bundling several independent fixes across 10 files makes this slow to review. Does it make sense by spitting it into:

Integer Clip
Cast to uint8 (do we also need cast from uint8?)
AveragePool fix
Diagnostics

Add a ClipInt64 kernel/shader for opsets 12-12 and 13, so int64 Clip stops falling back to the CPU EP (which aborts session init under the compile-only flow). int64 (like all integer types) is only a valid Clip element type from opset 12 on -- the opset-11 Clip schema constrains T to float types -- so there is no 11-11 registration. The WebNN-inserted index Clip before ArgMax in the SD-Turbo text encoder is int64. The 4-byte templated Clip cannot cover it (sizeof(T)==sizeof(float) static_assert). WebGPU has no native 64-bit integer type; the EP stores int64 as vec2<u32> and operates on the truncated low 32 bits as i32 (shader_variable.cc GetByOffset/SetByOffset). Because Clip is monotonic, ClipInt64 clamps that i32 value (min/max saturated into the i32 range) and sign-extends on write, consistent with the EP's existing int64 semantics. This is a split CL from #29682 to fix part of #29756

…ns (#29835) Register missing opset ranges for existing WebGPU kernels so these nodes stop falling back to the CPU EP (extra GPU<->CPU copies; session-init abort under the compile-only flow). * AveragePool (pool.cc, webgpu_execution_provider.cc): register opsets 11-18, 19-21 and 22, replacing the single opset-11 registration. ORT clamps an open-ended SinceVersion(11) registration to [11, 18] (not [11, INT_MAX]), so MobileNet (opset 21, AveragePool SinceVersion 19) was not claimed and fell back to CPU; the CPU-only NchwcTransformer (Level3) then amplified the single pool into 3 CPU nodes. * Clip int32/uint32 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): register the 4-byte templated Clip for int32/uint32 at opsets 12-12 and 13, via a dedicated WEBGPU_CLIP_KERNEL_FROM_12 macro. Integer types are only valid Clip element types from opset 12 on (the opset-11 Clip schema constrains T to float types), so there is no 11-11 integer registration. Used by shape/index/mask subgraphs (e.g. CLIP text position ids). * AveragePool count_include_pad divisor (pool.cc): a pre-existing WebGPU pool-kernel bug exposed by the opset-19 registration. With count_include_pad the shader divided by the full kernel size, which is wrong under ceil_mode (the last window can extend past the padded region; those overflow cells must not be counted). Now count is accumulated in the kernel loop and, under count_include_pad, excludes cells beyond the padded region (signed arithmetic, since u32 window indices underflow for begin padding, which is inside the region). This is a split CL from #29682 to fix part of #29756

Add uint8 support to the WebGPU Cast in both directions, so uint8 Cast nodes stop falling back to the CPU EP. uint8 was in neither the input (T1) nor output (T2) type constraint, so both casts *to* uint8 (e.g. a terminal bool->uint8 at a graph output) and casts *from* uint8 (back to int32/float/bool) fell back. WebGPU stores uint8 packed as Uint8x4 (4 per u32, lane 0 -> low byte), the same layout already used for bool. * to uint8 (cast.cc, shader_variable.cc): add uint8 to the output (T2) type constraint, a to-uint8 shader expression, and a Uint8x4 packing path in SetByOffset. * from uint8 (cast.cc, shader_variable.cc): add uint8 to the input (T1) type constraint, a Uint8x4 unpacking path in GetByOffset (reads the 4 bytes into a vec4<u32>, inverse of the packing), and include uint8 in is_from_unsigned so uint8->int64 zero-extends. This is a split CL from #29682 to fix part of #29756

mingmingtasd · 2026-07-27T00:37:11Z

@mingmingtasd, thanks for making this PR!

My main feedback is scope: bundling several independent fixes across 10 files makes this slow to review. Does it make sense by spitting it into:

Integer Clip

Cast to uint8 (do we also need cast from uint8?)

AveragePool fix

Diagnostics

The first three PRs are merged. The last Diagnostics one is low priority. Close this PR now. @huningxin

mingmingtasd force-pushed the pr-webgpu-cpu-fallback-fix branch from b16a4c7 to 6c75b70 Compare July 14, 2026 05:43

mingmingtasd force-pushed the pr-webgpu-cpu-fallback-fix branch from 6c75b70 to 47147f8 Compare July 15, 2026 00:38

adrastogi requested review from Copilot and edgchen1 July 15, 2026 02:50

Copilot started reviewing on behalf of adrastogi July 15, 2026 02:51 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

qwu16 mentioned this pull request Jul 17, 2026

Microsoft WebNN demo applications cannot successfully run in ORT-WebGPU backend with CPU fallback disabled #29756

Open

ibelem mentioned this pull request Jul 22, 2026

[Meta] WebNN - Windows ML / ONNX Runtime Issues and Feature Requests #26722

Open

40 tasks

This was referenced Jul 23, 2026

[WebGPU EP] Support int64 Clip via a dedicated ClipInt64 kernel #29834

Merged

[WebGPU EP] Add AveragePool 19/22 and integer Clip kernel registrations #29835

Merged

[WebGPU EP] Support Cast to and from uint8 #29839

Merged

mingmingtasd closed this Jul 27, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WebGPU EP] Fix unnecessary CPU fallback and improve fallback diagnostic - #29682

[WebGPU EP] Fix unnecessary CPU fallback and improve fallback diagnostic#29682
mingmingtasd wants to merge 1 commit into
microsoft:mainfrom
mingmingtasd:pr-webgpu-cpu-fallback-fix

mingmingtasd commented Jul 13, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

mingmingtasd commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mingmingtasd commented Jul 17, 2026

Uh oh!

huningxin commented Jul 21, 2026

Uh oh!

mingmingtasd commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mingmingtasd commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kernel coverage fixes, by model

Correctness fix exposed by the above

Diagnostics

Tests

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

mingmingtasd commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

mingmingtasd commented Jul 17, 2026

Uh oh!

huningxin commented Jul 21, 2026

Uh oh!

mingmingtasd commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mingmingtasd commented Jul 13, 2026 •

edited

Loading