[Performance] Shape operator in a model defined with opset 21 falls back to CPU

[xadupre]

Describe the issue

I tried to run an model defined with Opset 21 on CUDA but many node are moved to CPUExecutionProvider.

The reason is onnx defines an new version of Shape in opset 21 but onnxruntime does not and the following condition fails because kernel_start_version != since_ver and kernel_end_version == INT_MAX.

https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/framework/kernel_registry.cc#L126

As a result, many memcpy to/from host are inserted.

To reproduce

Loading a model with defined with opset 21 on CUDA with at least a Shape node.

Urgency

No response

Platform

Windows

OS Version

11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

main branch

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 12.6

Model File

No response

Is this a quantized model?

No

[titaiwangms]

Looks like we probably just need to register it. Let me try.

[yuslepukhin]

Shape is always on CPU, it is one of the CPU preferred nodes because tensors shape always is in CPU memory.
.OutputMemoryType(OrtMemTypeCPUInput, 0).

[xadupre]

That's true but there is a CUDA kernel declared in onnxruntime. The same goes for Transpose. Enabling the log shows the following and that should not happen.

2025-09-18 10:26:59.413612084 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Shape node name: index_Tensor_n0_rk1
2025-09-18 10:26:59.413716348 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Shape node name: sym_size_int3
2025-09-18 10:26:59.413796426 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Squeeze node name: sym_size_int4
2025-09-18 10:26:59.413947224 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: ShapeBasedReshapeIsSqueezePattern--reshape
2025-09-18 10:26:59.414074909 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Shape node name: sym_size_int5
2025-09-18 10:26:59.414152757 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Shape node name: sym_size_int
2025-09-18 10:26:59.414218333 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Squeeze node name: SqueezeAddPattern--sym_size_int6
2025-09-18 10:26:59.414298410 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: ShapeBasedReshapeIsSqueezePattern--reshape3
2025-09-18 10:26:59.414446513 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Shape node name: index_Tensor_n0_rk16
2025-09-18 10:26:59.414533606 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Reshape node name: index_Tensor_n0_rk15
2025-09-18 10:26:59.414663131 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Reshape node name: index_Tensor_n0_rk14
2025-09-18 10:26:59.414700102 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Reshape node name: index_Tensor_n0_rk17
2025-09-18 10:26:59.414770462 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Squeeze node name: sym_size_int6
2025-09-18 10:26:59.414848733 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: ShapeBasedReshapeIsSqueezePattern--reshape2
2025-09-18 10:26:59.415324828 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: ShapeBasedReshapeIsSqueezePattern--view3
2025-09-18 10:26:59.415424651 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Reshape node name: TransposeEqualReshapePattern--B--transpose_int3
2025-09-18 10:26:59.415458659 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: unsqueeze7
2025-09-18 10:26:59.415660166 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Squeeze node name: ShapeBasedReshapeIsSqueezePattern--reshape5
2025-09-18 10:26:59.415758894 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: ShapeBasedReshapeIsSqueezePattern--view2
2025-09-18 10:26:59.415835343 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Reshape node name: TransposeEqualReshapePattern--B--transpose_int2
2025-09-18 10:26:59.415956736 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: unsqueeze3
2025-09-18 10:26:59.416159107 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Transpose node name: transpose_int5
2025-09-18 10:26:59.416307772 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: unsqueeze5
2025-09-18 10:26:59.416494348 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: unsqueeze4
2025-09-18 10:26:59.416669932 [I:onnxruntime:, cuda_execution_provider.cc:2879 GetCapability] CUDA kernel not found in registries for Op type: Unsqueeze node name: unsqueeze6