[NV EP] fix EP context options by gedoensmax · Pull Request #24545 · microsoft/onnxruntime

gedoensmax · 2025-04-25T10:57:47Z

Description

While cleaning up the options I missed the part in the provider bridge that translates session options to TRT options.
To better integrate with current IHV work I adopted the principle that QNN and OV use to pipe through session options. Since all this is string based magic it would be great to be access a general point of truth like EpContextModelGenerationOptions in the provider wrappedtypes.

onnxruntime/onnxruntime/core/framework/session_options.h

Line 73 in 6df6206

struct EpContextModelGenerationOptions {

This is a fix on top of #24456 @ankan-ban and @chilo-ms to review.

ankan-ban · 2025-04-25T11:34:01Z

looks good. Thanks, Max, for the fix and additional cleanup!

chilo-ms · 2025-04-25T16:38:47Z

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

azure-pipelines · 2025-04-25T16:39:09Z

Azure Pipelines successfully started running 5 pipeline(s).

chilo-ms · 2025-04-25T16:49:52Z

please address lintrunner failure

chilo-ms · 2025-04-25T18:14:08Z

Since we haven't set up a pipeline for NV EP, i tested it locally and encountered following compile error.

...
D:\ort\onnxruntime\core\providers\nv_tensorrt_rtx\nv_provider_factory_creator.h(16,60): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int [D:\ort\build\Windows\Debug\on
nxruntime_session.vcxproj]
(compiling source file '../../../onnxruntime/core/session/provider_registration.cc')

D:\ort\onnxruntime\core\providers\nv_tensorrt_rtx\nv_provider_factory_creator.h(16,80): error C2143: syntax error: missing ',' before '*' [D:\ort\build\Windows\Debug\onnxruntime_session.vcxproj]
(compiling source file '../../../onnxruntime/core/session/provider_registration.cc')

D:\ort\onnxruntime\core\session\provider_registration.cc(292,61): error C2665: 'onnxruntime::NvProviderFactoryCreator::Create': no overloaded function could convert all the argument types [D:\ort\build\Window
s\Debug\onnxruntime_session.vcxproj]
D:\ort\onnxruntime\core\providers\nv_tensorrt_rtx\nv_provider_factory_creator.h(15,53):
could be 'std::shared_ptronnxruntime::IExecutionProviderFactory onnxruntime::NvProviderFactoryCreator::Create(const onnxruntime::ProviderOptions &,const int)'
D:\ort\onnxruntime\core\session\provider_registration.cc(292,61):
'std::shared_ptronnxruntime::IExecutionProviderFactory onnxruntime::NvProviderFactoryCreator::Create(const onnxruntime::ProviderOptions &,const int)': cannot convert argument 2 from 'onnxruntime::
SessionOptions *' to 'const int'
D:\ort\onnxruntime\core\session\provider_registration.cc(292,86):
There is no context in which this conversion is possible
D:\ort\onnxruntime\core\session\provider_registration.cc(292,61):
while trying to match the argument list '(onnxruntime::ProviderOptions, onnxruntime::SessionOptions *)'
...

chilo-ms · 2025-04-25T20:15:02Z

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

azure-pipelines · 2025-04-25T20:15:24Z

Azure Pipelines successfully started running 5 pipeline(s).

chilo-ms · 2025-04-25T20:38:38Z

/azp run Test Linux CUDA x64 Release, Test Linux TensorRT x64 Release, web_Debug / build_onnxruntime_web, web_Release / build_onnxruntime_web

azure-pipelines · 2025-04-25T20:38:45Z

No pipelines are associated with this pull request.

snnn · 2025-04-26T04:25:58Z

/azp run Windows ARM64 QNN CI Pipeline

azure-pipelines · 2025-04-26T04:26:07Z

Azure Pipelines successfully started running 1 pipeline(s).

gedoensmax · 2025-04-29T07:21:42Z

@anujj noticed that during rebasing I accidentally deleted the profile shape parsing. Just reverted that.

chilo-ms · 2025-04-29T22:21:34Z

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

azure-pipelines · 2025-04-29T22:21:53Z

Azure Pipelines successfully started running 5 pipeline(s).

chilo-ms · 2025-04-30T15:32:56Z

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

azure-pipelines · 2025-04-30T15:33:21Z

Azure Pipelines successfully started running 5 pipeline(s).

jywu-mysoft · 2025-05-01T06:37:39Z

the 2 conflicts are due to #24552
take the changes from there.

chilo-ms · 2025-05-01T15:57:42Z

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

azure-pipelines · 2025-05-01T15:58:03Z

Azure Pipelines successfully started running 5 pipeline(s).

@ankan-ban

### Description While cleaning up the options I missed the part in the provider bridge that translates session options to TRT options. To better integrate with current IHV work I adopted the principle that QNN and OV use to pipe through session options. Since all this is string based magic it would be great to be access a general point of truth like `EpContextModelGenerationOptions` in the provider wrappedtypes. https://github.com/microsoft/onnxruntime/blob/6df620675290d97d7e406faf232b8b521333b6e8/onnxruntime/core/framework/session_options.h#L73 This is a fix on top of #24456 @ankan-ban and @chilo-ms to review.

### Description Cherry pick the following into [rel-1.22.0](https://github.com/microsoft/onnxruntime/tree/rel-1.22.0) - (#24608) - (#24545) --------- Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: Maximilian Müller <44298237+gedoensmax@users.noreply.github.com>

@ankan-ban

### Description While cleaning up the options I missed the part in the provider bridge that translates session options to TRT options. To better integrate with current IHV work I adopted the principle that QNN and OV use to pipe through session options. Since all this is string based magic it would be great to be access a general point of truth like `EpContextModelGenerationOptions` in the provider wrappedtypes. https://github.com/microsoft/onnxruntime/blob/6df620675290d97d7e406faf232b8b521333b6e8/onnxruntime/core/framework/session_options.h#L73 This is a fix on top of microsoft#24456 @ankan-ban and @chilo-ms to review.

snnn · 2025-09-05T20:48:27Z

This PR has been included in the rel-1.22.0 branch. Removing the release:1.22.0 label.

chilo-ms previously approved these changes Apr 25, 2025

View reviewed changes

gedoensmax dismissed chilo-ms’s stale review via 81abc62 April 29, 2025 07:21

chilo-ms reviewed Apr 29, 2025

View reviewed changes

Comment thread onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc Outdated

chilo-ms added the release:1.22.0 label Apr 30, 2025

gedoensmax added 7 commits May 1, 2025 09:13

take EP context options from session

72772ea

remove unused options

aed45c3

fix a missing pybind API changes

ad23f83

lint changes and win compile

1bdd004

add profile shape parsing

eb30442

refactoring logging string

a26ae31

fix provider bridge

ffa63b0

gedoensmax force-pushed the fix_ep_context_nv branch from dfbc6d3 to ffa63b0 Compare May 1, 2025 09:31

chilo-ms approved these changes May 1, 2025

View reviewed changes

chilo-ms merged commit 716a88f into microsoft:main May 1, 2025

vraspar mentioned this pull request May 2, 2025

Cherry-picks into rel-1.22.0 #24624

Merged

snnn removed the release:1.22.0 label Sep 5, 2025

Uh oh!

Conversation

gedoensmax commented Apr 25, 2025

Description

Uh oh!

ankan-ban commented Apr 25, 2025

Uh oh!

chilo-ms commented Apr 25, 2025

Uh oh!

azure-pipelines Bot commented Apr 25, 2025

Uh oh!

chilo-ms commented Apr 25, 2025

Uh oh!

chilo-ms commented Apr 25, 2025

Uh oh!

chilo-ms commented Apr 25, 2025

Uh oh!

azure-pipelines Bot commented Apr 25, 2025

Uh oh!

chilo-ms commented Apr 25, 2025

Uh oh!

azure-pipelines Bot commented Apr 25, 2025

Uh oh!

snnn commented Apr 26, 2025

Uh oh!

azure-pipelines Bot commented Apr 26, 2025

Uh oh!

gedoensmax commented Apr 29, 2025

Uh oh!

chilo-ms commented Apr 29, 2025

Uh oh!

azure-pipelines Bot commented Apr 29, 2025

Uh oh!

Uh oh!

chilo-ms commented Apr 30, 2025

Uh oh!

azure-pipelines Bot commented Apr 30, 2025

Uh oh!

jywu-mysoft commented May 1, 2025

Uh oh!

chilo-ms commented May 1, 2025

Uh oh!

azure-pipelines Bot commented May 1, 2025

Uh oh!

snnn commented Sep 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants