Fix NuGet DLL Loading on Linux and macOS

[tianleiwu]

Summary

This PR addresses persistent native library loading issues in the ONNX Runtime NuGet package, specifically on macOS and Linux, by implementing a robust DllImportResolver. It also includes necessary pipeline and packaging adjustments to ensure required macOS artifacts are correctly located and validated during CI.

Problem

#27263 reports that Unable to load shared library 'onnxruntime.dll' or one of its dependencies. It was caused by #26415 since the commit hard-coded onnxruntime.dll even for Linux and MacOS (The correct filename shall be libonnxruntime.so for Linux, and libonnxruntime.dylib for MacOS).

The Nuget test pipeline has been broken for a while, so we also need fix the pipeline to test our change. It has the following issues:

• MacOS nuget is for arm64, but the vmImage macOS-15 is x64.
• MacOS nuget test need libcustom_op_library.dylib, but it is not copied from artifacts to test environment.
• MacOS artifact contains libonnxruntime.dylib and libonnxruntime.1.24.1.dylib, where libonnxruntime.dylib is symlink. It causes issue since the later is excluded by nuspec.
• MacOS nuget test use models from onnx repo. However, latest onnx has some models with data types like float8 that are not supported by C#, so those model test failed.
• Linux nuget test uses a docker Dockerfile.package_ubuntu_2404_gpu, but docker build failed due to libnvinfer-headers-python-plugin-dev and libnvinfer-win-builder-resource10 version.

Changes

1. Robust C# DLL Resolution

The DllImportResolver has been enhanced to handle various deployment scenarios where standard .NET resolution might fail:

• Platform-Specific Naming: Maps extension-less library names (onnxruntime, ortextensions) to appropriate filenames (onnxruntime.dll, libonnxruntime.so, libonnxruntime.dylib) based on the OS.
• Multi-Stage Probing:
  1. Default Loading: Attempts NativeLibrary.TryLoad with the mapped name.
  2. NuGet runtimes Probing: If the above fails, it probes the runtimes/{rid}/native/ subdirectories relative to the assembly location, covering common RIDs (win-x64, linux-arm64, osx-arm64, etc.).
  3. Base Directory Fallback: As a final attempt, it looks in AppContext.BaseDirectory.
• Case-Sensitivity Handling: Ensures lowercase extensions are used on Windows to prevent lookup failures on case-sensitive filesystems.

2. macOS CI/Packaging Improvements

• Templates (test_macos.yml):
  • Updated to extract artifacts from TGZ files.
  • Ensures libcustom_op_library.dylib is placed in the expected location (testdata/testdata) for end-to-end tests.
  • Initializes the ONNX submodule to provide required test data.
• Node.js:
  • Restored the Node.js macOS test stage in c-api-noopenmp-test-pipelines.yml, configured to run on the ARM64 pool (AcesShared).
  • Updated test_macos.yml template to support custom agent pools (similar to the NuGet template).
• Pipeline Config: Adjusted agent pool selection and demands for macOS jobs to ensure stable execution.
• Binary Robustness: The copy_strip_binary.sh script now ensures libonnxruntime.dylib is a real file rather than a symlink, improving NuGet packaging reliability.

3. Test Refinements

• Inference Tests: Skips a specific set of pretrained-model test cases on macOS that are currently known to be flaky or unsupported in that environment, preventing noise in the CI results.

Verification

Pipelines

• Verified in NuGet_Test_MacOS.
• Verified in NuGet_Test_Linux.
• Verified in Windows test pipelines.

Net Effect

The C# bindings are now significantly more resilient to different deployment environments. The CI process for macOS is also more robust, correctly handling the artifacts required for comprehensive NuGet validation.

[Copilot]

Pull request overview

This PR addresses cross-platform native library resolution for the C# ONNX Runtime NuGet packages by removing hardcoded .dll names and introducing custom resolution logic, alongside CI/packaging adjustments to ensure required macOS artifacts (notably the custom op library and dylib layout) are present during NuGet validation.

Changes:

• Updated C# P/Invoke library names to use extension-less names and added a DllImportResolver to control native library loading behavior.
• Adjusted macOS NuGet test pipeline and macOS packaging steps to ensure required test artifacts (e.g., libcustom_op_library.dylib) are available/located as expected.
• Tweaked Linux packaging Docker image dependencies and related CI pipeline configuration.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tools/ci_build/github/linux/docker/Dockerfile.package_ubuntu_2404_gpu	Adds additional TensorRT-related packages for the Ubuntu 24.04 GPU packaging image.
tools/ci_build/github/linux/copy_strip_binary.sh	Ensures libonnxruntime.dylib is a real file (not a symlink) for NuGet packaging robustness.
tools/ci_build/github/azure-pipelines/templates/mac-cpu-packaging-steps.yml	Copies libcustom_op_library.dylib into packaging/testdata locations for macOS artifacts.
tools/ci_build/github/azure-pipelines/nuget/templates/test_macos.yml	Updates macOS NuGet test job pool selection and unpacks macOS artifacts into testdata/; initializes ONNX submodule for test data.
tools/ci_build/github/azure-pipelines/c-api-noopenmp-test-pipelines.yml	Switches macOS NuGet tests to a specific pool/demands and removes the Node.js macOS stage from this pipeline.
csharp/test/Microsoft.ML.OnnxRuntime.Tests.NetCoreApp/InferenceTest.netcore.cs	Skips a set of pretrained-model testcases on macOS.
csharp/src/Microsoft.ML.OnnxRuntime/NativeMethods.shared.cs	Changes DllImport names to extension-less and adds a DllImportResolver to map/load native libraries.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[skottmckay]

Is part of the problem that the props file for the nuget package does not have logic to copy files for non-windows platforms to the build output directory?

https://github.com/microsoft/onnxruntime/tree/main/csharp/src/Microsoft.ML.OnnxRuntime/targets/netstandard

Typically the build process would copy from the relevant runtimes directory in the package to the build output if that was setup.

[tianleiwu]

Is part of the problem that the props file for the nuget package does not have logic to copy files for non-windows platforms to the build output directory?

I added a section of Problem in PR description to list all issues that addressed by this PR.

[skottmckay]

Is part of the problem that the props file for the nuget package does not have logic to copy files for non-windows platforms to the build output directory?

I added a section of Problem in PR description to list all issues that addressed by this PR.

but would the runtimes probing be required if the props file copied the library from the applicable runtimes directory to the build output directory like we do on Windows?

2. NuGet runtimes Probing: If the above fails, it probes the runtimes/{rid}/native/ subdirectories relative to the assembly location, covering common RIDs (win-x64, linux-arm64, osx-arm64, etc.).

FWIW we're having to manually add this copy step in the FL Local samples for Linux and we wouldn't need to do that if the props file included logic for linux.

https://github.com/microsoft/Foundry-Local/blob/58b81a15d0bbe7a13398f74a20aa490a5545a1a5/samples/cs/GettingStarted/ExcludeExtraLibs.props#L36-L43

[tianleiwu]

but would the runtimes probing be required if the props file copied the library from the applicable runtimes directory to the build output directory like we do on Windows?

2. NuGet runtimes Probing: If the above fails, it probes the runtimes/{rid}/native/ subdirectories relative to the assembly location, covering common RIDs (win-x64, linux-arm64, osx-arm64, etc.).

If we have props for Linux/MacOS, we can simplify runtime probing logic.

We can address props issue in another pull request since it is a different issue.
This PR mainly address hard-coded "onnxruntime.dll" for Linux/MacOS.

[philipag]

@tianleiwu If the library client already called SetDllImportResolver() for the same assembly, this change causes a fatal error and makes Onnx unusable. Therefore if the client already has a custom way of handling DLL loading issues through its own resolver, the onnx attempt at registering a resolver should not cause catastrophic failure. The minimal code fix would be something like this:

#if !NETSTANDARD2_0 && !__ANDROID__ && !__IOS__
            // Register a custom DllImportResolver to handle platform-specific library loading.
            // Replaces default resolution specifically on Windows for case-sensitivity.
try
{
// This will fail if the library client already registered a resolver for the assembly.
            NativeLibrary.SetDllImportResolver(typeof(NativeMethods).Assembly, DllImportResolver);
}
catch
{
// perhaps some logging here or an informative exception thrown?
}
#endif

An better approach would be to add a static bool somewhere to configure onnx to not try to register a custom handler. That way one could prevent race conditions of who tries to register first (which can give hard to debug and unexpected results).

Anyway, as-is I cannot use the latest release because we definitely require our own handler for onnx dll loading (Windows).

[tianleiwu]

@tianleiwu If the library client already called SetDllImportResolver() for the same assembly, this change causes a fatal error and makes Onnx unusable.

@philipag, see #27535