Avoid repetitive creation of fp4/fp8 native-custom-op domains for NvTensorRtRtx EP - #27192
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors the NvTensorRTRTX execution provider's custom ops initialization to improve efficiency and correctness. It introduces a flag-based approach to avoid redundant initialization of FP4/FP8 native custom ops and removes potentially dangerous manual deletion of statically-allocated objects.
Changes:
- Added
native_custom_ops_initializedflag to track whether native custom ops (TRT_FP4DynamicQuantize, TRT_FP8QuantizeLinear, TRT_FP8DequantizeLinear) have been created - Restructured control flow to add already-initialized native ops to the domain list without re-creating them
- Made release functions no-ops since the custom op domains are static objects managed by unique_ptr with static storage duration
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
304db7c to
3e87df1
Compare
I have synced the branch. |
|
@yuslepukhin @tianleiwu Can I get a review for this? |
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
3e87df1 to
88bf125
Compare
tianleiwu
left a comment
There was a problem hiding this comment.
The changes look correct and robust.
Thread Safety: The use of static std::mutex effectively protects the initialization logic.
Resource Management: Removing the manual delete corresponds correctly to the static ownership model.
Logic Correctness: The separation of native_custom_ops_initialized check ensures that native ops are returned correctly even if custom_op_domain is empty, while preventing duplicate initialization.
The PR improves stability and prevents potential memory corruption.
|
Error in React Native CI Pipeline / React Native CI Android (pull_request) doesn't look related. Raw Logs - link
|
88bf125 to
3506183
Compare
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
tianleiwu
left a comment
There was a problem hiding this comment.
custom_op_domain->custom_ops_ could accumulate duplicates: The plugin registration path (the try/catch block that iterates getPluginCreatorList) doesn't check whether custom_op_domain already has ops — it relies on custom_op_domain->domain_ != "" as the guard. If that guard somehow fails (domain_ never set because the try block threw), on a subsequent call the plugins could be re-registered and created_custom_op_list / custom_op_domain->custom_ops_ would accumulate duplicates. This is a pre-existing issue not introduced by this PR, so it can be deferred.
The original TRT EP (tensorrt_execution_provider_custom_ops.cc) has the same double-free bug: It uses static unique_ptr but its ReleaseTensorRTCustomOpDomain still does delete ptr. This PR only fixes it for TRTRTX EP. This should be tracked as a follow-up.
Approve with minor suggestion: The code is correct, addresses all actionable review feedback, and the deferred architectural work is appropriately scoped for a follow-up. Consider filing a tracking issue for the broader ownership redesign (covering both TRT and TRTRTX EPs).
He is on vacation right now. Update the status to unblock the PR.
Apply the same fix from NvTensorRtRtx EP (PR #27192) to the TRT EP: - ReleaseTensorRTCustomOpDomain: make no-op since objects are owned by static unique_ptrs in CreateTensorRTCustomOpDomainList() - ReleaseTensorRTCustomOpDomainList: just clear the vector instead of deleting static-owned objects - Add ownership documentation comments to static members Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
### Description Apply the same double-free fix from NvTensorRtRtx EP ([PR #27192](#27192)) to the TRT EP. `CreateTensorRTCustomOpDomainList()` owns domains/ops via static `unique_ptr`s, but `ReleaseTensorRTCustomOpDomain()` was manually `delete`-ing the same objects through raw pointers — double-free at program exit. - `ReleaseTensorRTCustomOpDomain()` → no-op (static `unique_ptr`s own the lifetime) - `ReleaseTensorRTCustomOpDomainList()` → `clear()` the reference vector only - Added ownership comments to static members matching NvTensorRtRtx EP style ### Motivation and Context PR #27192 review ([thread](#27192 (comment))) identified TRT EP has the identical bug pattern that was fixed in NvTensorRtRtx EP. The TRT EP code was the original source this pattern was borrowed from. @tianleiwu noted a follow-up PR was needed. <!-- START COPILOT CODING AGENT TIPS --> --- 🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security) --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Description
Motivation and Context