Skip to content

Make cache_dir and num_stream options session-local - #809

Merged
preetha-intel merged 3 commits into
intel:ovep-developfrom
mklimenk:private/mklimenk/per_model_cache_dir
Sep 12, 2025
Merged

Make cache_dir and num_stream options session-local#809
preetha-intel merged 3 commits into
intel:ovep-developfrom
mklimenk:private/mklimenk/per_model_cache_dir

Conversation

@mklimenk

@mklimenk mklimenk commented Sep 5, 2025

Copy link
Copy Markdown

This is a follow-up PR to #782, that moved most of the options from being set globally (per-process) to locally (per-model).
It is required to properly enable parallel session creation of multiple models with different set of options for each model.

https://jira.devtools.intel.com/browse/CVS-169874

@mklimenk

mklimenk commented Sep 5, 2025

Copy link
Copy Markdown
Author

@MayureshV1 could you please review it?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors cache directory and number of streams configuration from global (per-process) settings to session-local (per-model) settings in the OpenVINO provider. This change enables parallel creation of multiple models with different configuration options.

  • Removes global SetCache and SetStreams methods from the OVCore class
  • Updates EnableCaching and EnableStreams methods to accept and populate a device configuration map parameter
  • Shifts from setting properties globally on the core to setting them locally via device configuration

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
ov_interface.h Removes global SetCache and SetStreams method declarations from OVCore
ov_interface.cc Removes global SetCache and SetStreams method implementations from OVCore
basic_backend.h Updates EnableCaching and EnableStreams method signatures to accept device config parameter
basic_backend.cc Refactors EnableCaching and EnableStreams to populate device config instead of setting global properties

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@MayureshV1 MayureshV1 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implementation looks good and would make the existing provider options thread safe.

The same two provider options: cache_dir and num_streams are OV properties and can be set using load_config provider option.

@preetha-intel

Copy link
Copy Markdown

@mklimenk As this is refactoring the properties to set cache and num streams, can it all be broght into the same function , "PopulateConfigValue" instead of seperate calls which does populate the same map.

It can also be loaded from OV config. Can that be validated as well?

@preetha-intel preetha-intel left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consolidate seperate function calls into single PopulateConfigValue

@preetha-intel

Copy link
Copy Markdown

Consolidate seperate function calls into single PopulateConfigValue

Can be taken as a follow up as its only cosmetic

@preetha-intel
preetha-intel dismissed their stale review September 12, 2025 06:48

Can be added as a follow up

@preetha-intel
preetha-intel merged commit 02cf7e3 into intel:ovep-develop Sep 12, 2025
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants