Add OrtEp-level conformance checks for plugin EPs - #29924
Open
GopalakrishnanN wants to merge 10 commits into
Open
Add OrtEp-level conformance checks for plugin EPs#29924GopalakrishnanN wants to merge 10 commits into
GopalakrishnanN wants to merge 10 commits into
Conversation
Parameterized GoogleTest suite (EpContract/EpConformanceTest) encoding universal IExecutionProvider invariants every EP must satisfy: stable Type(), valid preferred layout, CPU mem types map to CPU-accessible devices, non-null/repeatable preferred allocators, usable allocations, CPU data-transfer round-trip integrity, and consistent metadata queries. EPs are registered via factories (no static-init construction); unavailable EPs skip rather than fail. Adding a new EP is a single guarded line in GetEpConformanceParams().
PreferredAllocatorsAllocateUsableMemory previously called Alloc()/Free() on every preferred allocator, only gating the host memory read/write behind UsesCpuMemory(). On WebGPU (arm64 Debug CI) the GpuBufferAllocator creates buffers mapped at creation; Free() routes through the buffer manager which throws EnforceBufferUnmapped ('Buffer is still mapped'), failing the test.
Restrict the standalone Alloc/Free exercise to CPU-accessible allocators, whose lifecycle is backend-agnostic, and GTEST_SKIP when an EP exposes no such allocator. Device allocators remain covered by PreferredAllocatorsAreNonNullAndRepeatable. Fixes the webgpu build-and-test (arm64, Debug) leg on PR #28968.
Address PR review: in ORT_USE_EP_API_ADAPTERS builds, DefaultWebGpuExecutionProvider() ORT_ENFORCEs (aborting the entire unit-test run) when the dynamic plugin EP is initialized to a different EP name, instead of cleanly returning nullptr. Mirror the guard used by base_tester.cc and default_providers.cc by listing the built-in WebGPU EP only under 'defined(USE_WEBGPU) && !defined(ORT_USE_EP_API_ADAPTERS)'.
…ext nodes, allocator info Extends the conformance suite with backend-agnostic checks that use only the base IExecutionProvider API (no dependency on the capability-segregation work): - GraphCaptureNodeAssignmentPolicyIsValid: policy is a defined enum value. - GetOrtEpIsNullForBuiltInEp: built-in EPs are not plugin-wrapped. - EpContextNodesEmptyOnFreshEp: no EPContext nodes before any compilation. - PreferredAllocatorInfoIsConsistent: allocator OrtMemoryInfo has a non-empty name and a valid allocator type. Also extends MetadataQueriesAreCallable with IsGraphCaptureEnabled() and ShouldConvertDataLayoutForOp() smoke calls. All 22 instances (11 invariants x 2 CPU configs) pass.
Extract the backend-agnostic EP conformance invariants into a shared header (test/util/include/ep_conformance_invariants.h) so the built-in and plugin suites share one source of truth. Add a plugin EP conformance suite in onnxruntime_provider_test (test/providers/ep_conformance_plugin_test.cc) that runs the same invariants against a dynamically-loaded plugin EP -- an OrtEp reached through the PluginExecutionProvider wrapper -- obtained from dynamic_plugin_ep_infra::MakeEp(). It covers arbitrary plugin EPs (example plugin EP, CUDA/WebGPU-as-plugin, or a runtime-specified one) and skips cleanly when no plugin EP is configured. Asserts the plugin-specific stable non-null GetOrtEp(). Refactor the built-in suite (test/framework/execution_provider_conformance_test.cc) to delegate to the shared invariants (behavior-preserving). Addresses review feedback to also test the plugin OrtEp interface for arbitrary plugin EPs in onnxruntime_provider_test, reusing the invariant definitions.
- Skip (not fail) when an EP factory throws during construction, e.g. CUDA's cudaSetDevice on a GPU-less machine, by wrapping MakeEp() in try/catch and logging the exception text before returning nullptr. - Stop asserting GetDeviceId() == GetDevice().Id(): GetDeviceId() is provider-defined (WebGPU returns a configurable context id while its OrtDevice id is fixed at 0) and is not required to match; it is now only smoke-called. - Drop the non-empty allocator-name assertion: an empty OrtMemoryInfo.name is permitted by the contract, so requiring one over-states it. - Strengthen CreatePreferredAllocators() repeatability: null-check the second call and compare per-position OrtMemoryInfo, not just the element count. - Rename DataTransferCpuRoundTrip* to DataTransferCpuCopy* across both suites and the shared helper (it is a single CPU-to-CPU copy, not a round-trip). - Self-document the plugin-EP factory flag (/*expects_plugin_ep=*/true), clarify in the header that coverage is opt-in per registered EP, and note that skip-capable helpers must be invoked as the last statement of a test body. - Plugin suite: move test-only includes inside the dynamic-plugin guard, drop a redundant self-namespace alias, and simplify a factory lambda. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The allocator-name assertion was removed from the invariant helper, but the test-side comment still claimed a non-empty name is required. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The conformance suite's coverage was a hand-maintained list guarded by #ifdef USE_*, so an EP added to the build was silently left unchecked -- nothing failed and nothing warned. Cross-check that list against GetAvailableExecutionProviderNames(), the registry of EPs compiled into the build, and fail when a compiled EP is neither registered nor explicitly exempted. - Add ep_name (the canonical kXxxExecutionProvider) to EpConformanceParam so the registered list can be compared against the registry. - Drop the USE_* guards around the CUDA, DML and XNNPACK entries. Every Default*ExecutionProvider() is declared unconditionally and returns nullptr when its EP is not compiled in, so an absent EP now reports a skip instead of vanishing from the suite. The WebGPU guard stays: DefaultWebGpuExecutionProvider() ORT_ENFORCEs under ORT_USE_EP_API_ADAPTERS rather than returning nullptr, and the guard already matches the one on kWebGpuExecutionProvider in get_execution_providers.cc, so availability and registration stay in agreement. - Record the EPs that are not covered in two separate lists: kStructurallyExemptEps for web-only, remote-endpoint and external-runtime EPs that have no Default* factory, and kNotYetVettedEps for EPs that do have a factory and are expected to graduate into the registered list. - Add EpConformanceCoverage.ExemptionsAreWellFormed so a typo or an entry left behind after an EP is renamed cannot silently widen an exemption. The set of EPs actually exercised is unchanged, so no CI leg begins running the invariants against an EP that was not already covered. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The EP conformance suites assert contracts on IExecutionProvider, which is the internal interface. Plugin EP authors implement the public OrtEp C API instead, and some of its contracts have no IExecutionProvider equivalent, so they were not covered. Add two invariants that inspect the OrtEp behind a plugin EP: - CheckOrtEpRequiredFunctionsArePresent asserts GetCapability is implemented. It is the only OrtEp function ORT dereferences without a null check (PluginExecutionProvider::GetCapability), so a null one faults during graph partitioning rather than producing a diagnosable error. Every other entry point ORT calls is null-guarded or falls back to the OrtEpFactory implementation, so it is genuinely optional and is not asserted. - CheckOrtEpDeclaresCompileOrKernelRegistry asserts the EP declares a coherent execution mode: it either exposes a kernel registry, or implements both Compile and ReleaseNodeComputeInfos. ORT already requires this, but only on the first actual compilation, so a structural mistake surfaces late and partially. Implementing both is legal and is not flagged. Both read OrtEp members only after checking ort_version_supported, because a plugin built against an older ORT allocated a shorter struct. Both skip for built-in EPs, which have no backing OrtEp. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds OrtEp-level conformance invariants (reached via IExecutionProvider::GetOrtEp()) so the existing EP contract test suites can validate plugin-EP-specific ABI contracts that are not expressible via the internal IExecutionProvider interface.
Changes:
- Added shared
OrtEp-level invariant checks (e.g., required function presence and coherent “kernel registry vs compile” execution mode) to the shared invariants header. - Extended the plugin EP conformance suite to run the shared invariant set against a dynamically loaded plugin EP.
- Extended the built-in EP conformance suite to invoke the new
OrtEpinvariants (skipping for built-in EPs with no backingOrtEp).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| onnxruntime/test/util/include/ep_conformance_invariants.h | Adds shared invariant helpers, including new OrtEp-level checks gated by GetOrtEp() and ABI version. |
| onnxruntime/test/providers/ep_conformance_plugin_test.cc | Adds/updates the plugin EP conformance gtests to invoke the shared invariants against the dynamically loaded plugin EP. |
| onnxruntime/test/framework/execution_provider_conformance_test.cc | Adds/updates the built-in EP conformance suite to invoke the shared invariants and cover the new OrtEp-level checks. |
Comment on lines
+311
to
+313
| if (ort_ep->ort_version_supported >= 24 && ort_ep->GetKernelRegistry != nullptr) { | ||
| return; // Kernel-registry-based EP: Compile is optional for it. | ||
| } |
Member
|
Which one is the PR I should look at: #28968 or this one? |
Member
|
This one too: #29915, I would merge all of them into a single. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The EP conformance suites added in #28968 assert contracts on
IExecutionProvider, which is ONNX Runtime's internal EP interface. External EP authors implement the public plugin EP C API (OrtEp/OrtEpFactory) instead, and several of its contracts have noIExecutionProviderequivalent — so nothing covered them.This PR adds the first
OrtEp-level checks, reached throughIExecutionProvider::GetOrtEp(). They live in the shared invariants header so the built-in and plugin suites stay in lockstep, and they skip for built-in EPs (which have no backingOrtEp).CheckOrtEpRequiredFunctionsArePresent— assertsOrtEp::GetCapabilityis implemented.It is the only
OrtEpfunction ORT dereferences without a null check (ep_plugin_provider_interfaces.cc), so a null one faults during graph partitioning instead of producing a diagnosable error. Every other entry point ORT calls is either null-guarded or falls back to theOrtEpFactoryimplementation (CreateAllocator,CreateSyncStreamForDevice), so those are genuinely optional and are deliberately not asserted.CheckOrtEpDeclaresCompileOrKernelRegistry— asserts the EP declares a coherent execution mode: it either exposes a kernel registry, or implements bothCompileandReleaseNodeComputeInfos.ORT already requires this, but only on the first actual compilation (
PluginExecutionProvider::Compile), so a structural mistake surfaces late and partway through session initialization. Checking it at construction makes the failure immediate and self-describing. Implementing both a kernel registry andCompileis legal and is not flagged.ABI versioning
Both checks read
ort_version_supportedbefore touching any member. A plugin built against an older ORT allocated a shorterOrtEp, so reading a newer field would read past it.GetCapability,CompileandReleaseNodeComputeInfosare\since1.23;GetKernelRegistryis\since1.24. This mirrors what ORT itself does before touching versioned fields (e.g.ep_kernel_registration.cc).Notes on what is not asserted, and why
These were considered and rejected as already-covered or incorrect:
OrtEp::GetName()non-null / non-emptySanityCheckOrtEp()already rejects a nullGetNameand a null return before anOrtEpcan reach anIExecutionProvider. Non-emptiness is covered by the existingTypeIsNonEmptyAndStable, because a plugin EP's provider type isOrtEp::GetName()'s return value.OrtEp::GetName() == OrtEpFactory::GetName()PluginExecutionProvideris constructed withIExecutionProvider(ep->GetName(...)).OrtEp::ort_version_supported >= 22SanityCheckOrtEp()enforces it, so anOrtEpthat failed it never becomes anIExecutionProvider.GetKernelRegistry()must return a non-empty registryCopyEpKernelRegistryexplicitly returns OK onnullptr.CompilexorGetKernelRegistryGetKernelRegistry's docs say only that a NULL registry means "ORT assumes the EP compiles nodes"; nothing forbids implementing both. Encoded as an implication, not an xor.OrtEpFactory-level checks (CreateEpFactories()returning ≥ 1 — currently unenforced;GetVendor()non-empty;GetVersion()valid semver) are real gaps, but they need anOrtEpFactoryhandle rather than anIExecutionProvider, so they need separate plumbing and are out of scope here.Verification
Built and ran locally on Windows (MSVC, Ninja, Release):
No plugin EP is configured in a default local build, so the new tests skip there. To confirm they are not vacuous, each branch was driven with a fabricated
OrtEpthrough a temporary local probe (not part of this PR):OrtEpGetCapability == nullptrCompile == nullptr, no kernel registryReleaseNodeComputeInfos == nullptrCompile == nullptr)ort_version_supported == 23ort_version_supported == 22OrtEp)Every in-tree
OrtEpimplementation was also checked statically and satisfies both invariants: all five setGetCapability, the compile-based ones (example_plugin_ep,example_plugin_ep_virt_gpu) set bothCompileandReleaseNodeComputeInfos, and the kernel-registry ones (CUDA plugin, per-kernel example) setGetKernelRegistrywithCompile = nullptr.Motivation and Context
Third in a series on EP conformance coverage, after #28968 (the suites) and #29915 (registry-driven coverage). Those assert the internal interface; this one starts on the public plugin EP API that external EP authors actually implement.
This is stacked on #29915, so until both merge the diff here also shows their commits. Only the three conformance test files are changed by this PR.