Skip to content

[OVEP] Fix for precision accuracy - #603

Merged
sfatimar merged 1 commit into
ovep-developfrom
jatin_fix_precison_acc_issue
Mar 11, 2025
Merged

[OVEP] Fix for precision accuracy#603
sfatimar merged 1 commit into
ovep-developfrom
jatin_fix_precison_acc_issue

Conversation

@jatinwadhwa921

@jatinwadhwa921 jatinwadhwa921 commented Mar 6, 2025 •
edited by MayureshV1
Loading

Copy link
Copy Markdown

This PR handles precision for devices "AUTO:,MULTI: and HETERO:"

HAFP-2913

preetha-intel
preetha-intel reviewed Mar 6, 2025
View reviewed changes
Comment thread onnxruntime/core/providers/openvino/backends/basic_backend.cc Outdated
@jatinwadhwa921
jatinwadhwa921 force-pushed the jatin_fix_precison_acc_issue branch from 9967507 to f5428af Compare March 7, 2025 07:08
sfatimar
sfatimar reviewed Mar 7, 2025
View reviewed changes
Comment thread onnxruntime/core/providers/openvino/openvino_provider_factory.cc Outdated
sfatimar
sfatimar reviewed Mar 7, 2025
View reviewed changes
Comment thread onnxruntime/core/providers/openvino/openvino_provider_factory.cc Outdated
@sfatimar

sfatimar commented Mar 7, 2025

Copy link
Copy Markdown

Please move parsing out of the main file.

@sfatimar

sfatimar commented Mar 7, 2025

Copy link
Copy Markdown

Tested with AUTO, Multi Heter, GPU and CPU and NPU with Accuracy . AUTO:GPU, CPU. Need to check for GPU.0 1and GPU.1

preetha-intel
preetha-intel reviewed Mar 7, 2025
View reviewed changes
Comment thread onnxruntime/core/providers/openvino/openvino_provider_factory.cc Outdated
preetha-intel
preetha-intel reviewed Mar 7, 2025
View reviewed changes
Comment thread onnxruntime/core/providers/openvino/openvino_provider_factory.cc Outdated
<< " HETERO/MULTI/AUTO options and set 'precision' separately. \n";
auto delimit = device_type.find("_");
device_type = device_type.substr(0, delimit);
return device_type.substr(delimit + 1);

@preetha-intel preetha-intel Mar 7, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here it directly returns the precision passed by the user or there is a validity check?

@jatinwadhwa921 jatinwadhwa921 Mar 7, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No validity check, just return whatever is written after "_"

@jatinwadhwa921
jatinwadhwa921 force-pushed the jatin_fix_precison_acc_issue branch from f5428af to 204163d Compare March 7, 2025 12:07
@jatinwadhwa921

jatinwadhwa921 commented Mar 7, 2025

Copy link
Copy Markdown
Author

Please move parsing out of the main file.

fixed

@MayureshV1

MayureshV1 commented Mar 7, 2025

Copy link
Copy Markdown

@sfatimar , @jatinwadhwa921 - Please do not merge till there is a corresponding test written and merged for the Accuracy feature.

@sfatimar

sfatimar commented Mar 10, 2025

Copy link
Copy Markdown

Approved from my side.

sfatimar
sfatimar approved these changes Mar 10, 2025
View reviewed changes

@sfatimar sfatimar left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@sfatimar

sfatimar commented Mar 10, 2025

Copy link
Copy Markdown

Please make sure to rebase it

@jatinwadhwa921
jatinwadhwa921 force-pushed the jatin_fix_precison_acc_issue branch from 204163d to 61b36ef Compare March 10, 2025 12:52
@jatinwadhwa921

jatinwadhwa921 commented Mar 10, 2025

Copy link
Copy Markdown
Author

@sfatimar , @jatinwadhwa921 - Please do not merge till there is a corresponding test written and merged for the Accuracy feature.

I have tested it from my end, it works fine, test case has been added in the repo

@sfatimar
sfatimar merged commit ea13a05 into ovep-develop Mar 11, 2025
@MayureshV1

MayureshV1 commented Mar 11, 2025 •
edited
Loading

Copy link
Copy Markdown

@jatinwadhwa921 . Can you please share the tests with me and @preetha-intel.

These are the 3 scenarios that should be covered in validation for mixed precision models (FP32/ FP16):

  1. Objective accuracy test to ensure FP32 like accuracy is retained once Accuracy mode gets set correctly in OV
  2. Performance test to ensure perf is better than FP32 and closer to FP16 execution on GPU- Accuracy mode actually picks based on model precision not FP32 static
  3. Regression testing- Perf comparison between OV EP generations (if possible) to ensure the behavior has not changed between two OV and OV EP versions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MayureshV1 MayureshV1 MayureshV1 left review comments
@preetha-intel preetha-intel preetha-intel left review comments
@sfatimar sfatimar sfatimar approved these changes
@ankitm3k ankitm3k Awaiting requested review from ankitm3k
@vthaniel vthaniel Awaiting requested review from vthaniel
@saurabhkale17 saurabhkale17 Awaiting requested review from saurabhkale17

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jatinwadhwa921 @sfatimar @MayureshV1 @preetha-intel