Add per-LLM-node generation_config.json path override (#4233) - #4330
Open
exzile wants to merge 6 commits into
Open
Add per-LLM-node generation_config.json path override (#4233)#4330exzile wants to merge 6 commits into
exzile wants to merge 6 commits into
Conversation
Adds an optional generation_config_path field to LLMCalculatorOptions so several deployments backed by the same model weights can use different generation defaults without duplicating the model directory. Mirrors how graph_path already lets one model directory back several deployments. When unset, generation_config.json from models_path is used as before. An explicit path may be absolute or relative to models_path; a relative path is resolved against the model directory (its parent when models_path points at a file, e.g. a GGUF). An explicit path that does not exist is a load error. Shared resolveGenerationConfigPath helper is used by both the continuous batching and legacy initializers. Adds a unit test covering default, absolute, relative, and missing-path cases. Implements openvinotoolkit#4233 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
exzile
force-pushed
the
feature/generation-config-path
branch
from
June 27, 2026 01:03
79ec0ce to
44a52e6
Compare
# Conflicts: # src/test/llm/llmnode_test.cpp
Contributor
Author
|
@dkalinowski rebased and conflict-free now (additive test collision with main's cache_dir tests — both kept). Adds a per-LLM-node |
mzegla
reviewed
Jul 22, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a per-LLM-node override for generation_config.json via a new generation_config_path option on LLMCalculatorOptions, and wires a shared resolveGenerationConfigPath helper into both legacy and continuous-batching initializers so path resolution is consistent across pipeline types.
Changes:
- Adds
generation_config_pathtoLLMCalculatorOptions(proto + docs). - Introduces
resolveGenerationConfigPath(...)and uses it from both legacy and continuous-batching servable initializers. - Adds a unit test covering default/absolute/relative/missing-path resolution cases.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/llm/llmnode_test.cpp | Adds unit test coverage for generation config path resolution. |
| src/llm/servable_initializer.hpp | Declares the shared resolveGenerationConfigPath helper. |
| src/llm/servable_initializer.cpp | Implements resolveGenerationConfigPath including override and existence checks. |
| src/llm/llm_calculator.proto | Adds generation_config_path field to LLM node options. |
| src/llm/language_model/legacy/servable_initializer.cpp | Uses shared resolver to load base generation config. |
| src/llm/language_model/continuous_batching/servable_initializer.cpp | Uses shared resolver to load base generation config. |
| docs/llm/reference.md | Documents the new generation_config_path node option. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Use FileSystem::resolvePathWithBase (added in openvinotoolkit#4385) for the relative/absolute branch instead of hand-rolling is_relative(), keeping the generic path resolution in one place. The GGUF parent-directory adjustment stays local, since the shared helper is deliberately model-agnostic. - Fix resolveGenerationConfigPath never assigning the resolved override back to outPath, so an explicit generation_config_path was validated but then discarded and the models_path default returned instead. - Report a missing/invalid override via a dedicated LLM_NODE_GENERATION_CONFIG_DOES_NOT_EXIST rather than reusing LLM_NODE_DIRECTORY_DOES_NOT_EXIST, whose "workspace path does not exist" message was misleading for this option. - Harden the unit test: unique temp directory via createTempPath with RAII cleanup (a fixed name could collide and leaked on ASSERT failure), assert the exact error code, and cover the directory-valued override and the GGUF models_path case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mzegla
reviewed
Jul 29, 2026
mzegla
left a comment
Collaborator
There was a problem hiding this comment.
@exzile
Generation config is relevant also for VLM and Omni pipelines (@dkalinowski ).
I believe their respective servable initializers should also include the generation config path resolution logic so we are not creating a gap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #4233.
Adds an optional
generation_config_pathfield toLLMCalculatorOptions, so several deployments backed by the same model weights can use different generation defaults without duplicating the model directory. This mirrors howgraph_pathalready lets one model directory back several deployments with different graphs.Behavior
generation_config.jsonfrommodels_path, exactly as before (no change).models_path(resolved against the model directory, or its parent whenmodels_pathpoints at a file such as a GGUF).A shared
resolveGenerationConfigPathhelper is used by both the continuous-batching and legacy initializers, so behavior is identical across pipeline types (and the VLM CB servable, which reuses the CB initializer).Placement rationale
Per the discussion in #4233 (and the config taxonomy from #4221):
graph_pathalready lets one model directory back several deployments with different node options, so putting the override in the node options keeps the whole per-deployment configuration ingraph.pbtxt.Testing
Added
LLMGenerationConfigPath.ResolveGenerationConfigPathcovering default, absolute, relative, and missing-path cases. Built and ran locally on Windows (MSVC).🤖 Generated with Claude Code