Make cache_dir and num_stream options session-local - #809
Conversation
|
@MayureshV1 could you please review it? |
There was a problem hiding this comment.
Pull Request Overview
This PR refactors cache directory and number of streams configuration from global (per-process) settings to session-local (per-model) settings in the OpenVINO provider. This change enables parallel creation of multiple models with different configuration options.
- Removes global
SetCacheandSetStreamsmethods from theOVCoreclass - Updates
EnableCachingandEnableStreamsmethods to accept and populate a device configuration map parameter - Shifts from setting properties globally on the core to setting them locally via device configuration
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| ov_interface.h | Removes global SetCache and SetStreams method declarations from OVCore |
| ov_interface.cc | Removes global SetCache and SetStreams method implementations from OVCore |
| basic_backend.h | Updates EnableCaching and EnableStreams method signatures to accept device config parameter |
| basic_backend.cc | Refactors EnableCaching and EnableStreams to populate device config instead of setting global properties |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
MayureshV1
left a comment
There was a problem hiding this comment.
This implementation looks good and would make the existing provider options thread safe.
The same two provider options: cache_dir and num_streams are OV properties and can be set using load_config provider option.
|
@mklimenk As this is refactoring the properties to set cache and num streams, can it all be broght into the same function , "PopulateConfigValue" instead of seperate calls which does populate the same map. It can also be loaded from OV config. Can that be validated as well? |
preetha-intel
left a comment
There was a problem hiding this comment.
Consolidate seperate function calls into single PopulateConfigValue
Can be taken as a follow up as its only cosmetic |
This is a follow-up PR to #782, that moved most of the options from being set globally (per-process) to locally (per-model).
It is required to properly enable parallel session creation of multiple models with different set of options for each model.
https://jira.devtools.intel.com/browse/CVS-169874