OpenVINO Model Server fails to load LFM2-24B-A2B-int4-ov

[ss4workonly]

Describe the bug

When attempting to load the LFM2-24B-A2B-int4-ov model with OpenVINO Model Server on GPU, the initialization fails with an undeclared parameter error for attention_mask, preventing the model from starting.

To Reproduce

Steps to reproduce the behavior:

1. Pull OVMS Docker image: docker pull openvino/model_server:latest-gpu
2. Download model from https://huggingface.co/OpenVINO/LFM2-24B-A2B-int4-ov
3. Place model in ~/models/OpenVINO/LFM2-24B-A2B-int4-ov/
4. Run OVMS launch command:

docker run \
  --name ovms-lfm2 \
  --user "$(id -u):$(id -g)" \
  -p 8000:8000 \
  --device /dev/dri:/dev/dri \
  --group-add "$(getent group render | cut -d: -f3)" \
  -v "${HOME}/models:/models:rw" \
  openvino/model_server:latest-gpu \
  --source_model OpenVINO/LFM2-24B-A2B-int4-ov \
  --model_repository_path models \
  --task text_generation \
  --tool_parser lfm2 \
  --rest_port 8000 \
  --target_device GPU

5. See error in logs

Expected behavior

The model server should start successfully with the model loaded and ready for inference.

Logs

[2026-07-24 17:21:13.612][1][serving][info][server.cpp:117] OpenVINO Model Server 2026.2.1.1122f03bf
[2026-07-24 17:21:13.612][1][serving][info][server.cpp:118] OpenVINO backend 2026.2.1-21919-ede283a88e3-releases/2026/2
[2026-07-24 17:21:13.612][1][serving][info][server.cpp:121] OpenVINO GenAI backend 2026.2.1.0-3123-7dea0459b2a
[2026-07-24 17:21:13.612][1][serving][info][pythoninterpretermodule.cpp:37] PythonInterpreterModule starting
[2026-07-24 17:21:13.612][1][serving][info][pythoninterpretermodule.cpp:40] Initializing python interpreter
[2026-07-24 17:21:13.670][1][serving][info][pythoninterpretermodule.cpp:59] PythonInterpreterModule started
[2026-07-24 17:21:13.714][1][modelmanager][info][modelmanager.cpp:180] Available devices for Open VINO: CPU, GPU
[2026-07-24 17:21:13.714][1][serving][info][modelmanager.cpp:171] Running inside Docker container
[2026-07-24 17:21:13.714][1][serving][info][modelmanager.cpp:172] cpu quota: 0, cpu affinity: 14, max_open_files: 1024
[2026-07-24 17:21:13.714][1][serving][info][capimodule.cpp:40] C-APIModule starting
[2026-07-24 17:21:13.714][1][serving][info][capimodule.cpp:42] C-APIModule started
[2026-07-24 17:21:13.714][1][serving][info][grpcservermodule.cpp:110] GRPCServerModule starting
[2026-07-24 17:21:13.714][1][serving][info][grpcservermodule.cpp:114] GRPCServerModule started
[2026-07-24 17:21:13.714][1][serving][info][grpcservermodule.cpp:115] Port was not set. GRPC server will not be started.
[2026-07-24 17:21:13.714][1][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2026-07-24 17:21:13.714][1][serving][info][httpservermodule.cpp:39] Will start 14 REST workers
[2026-07-24 17:21:13.715][51][serving][info][drogon_http_server.cpp:157] Binding REST server to address: 0.0.0.0:8000
[2026-07-24 17:21:13.765][1][serving][info][drogon_http_server.cpp:184] REST server listening on port 8000 with 14 unary threads and 14 streaming threads
[2026-07-24 17:21:13.765][1][serving][info][http_server.cpp:249] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2026-07-24 17:21:13.765][1][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2026-07-24 17:21:13.765][1][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:8000
[INFO] LFS resume: attempts=5 interval=10 s
[2026-07-24 17:21:13.801][1][serving][info][libgit2.cpp:1282] Path already exists on local filesystem. Skipping download to path: models/OpenVINO/LFM2-24B-A2B-int4-ov
Model: OpenVINO/LFM2-24B-A2B-int4-ov downloaded to: models/OpenVINO/LFM2-24B-A2B-int4-ov
Graph: graph.pbtxt created in: models/OpenVINO/LFM2-24B-A2B-int4-ov
[2026-07-24 17:21:13.803][1][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-07-24 17:21:13.803][1][serving][info][mediapipegraphdefinition.cpp:101] Graph queue globally disabled via OVMS_GRAPH_QUEUE_OFF=1 for mediapipe: OpenVINO/LFM2-24B-A2B-int4-ov
[2026-07-24 17:21:13.803][1][serving][info][mediapipegraphdefinition.cpp:479] MediapipeGraphDefinition initializing graph nodes
[2026-07-24 17:21:13.803][1][modelmanager][info][servable_initializer.cpp:439] Initializing Language Model Continuous Batching servable
[2026-07-24 17:21:14.050][1][serving][error][servable_initializer.cpp:240] Error during llm node initialization for models_path: //models/OpenVINO/LFM2-24B-A2B-int4-ov/./ exception: Check 'unregistered_parameters.str().empty()' failed at src/core/src/model.cpp:264:
Model references undeclared parameters: opset1::Parameter attention_mask () -> (i64[?,?])


[2026-07-24 17:21:14.050][1][modelmanager][error][servable_initializer.cpp:444] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2026-07-24 17:21:14.050][1][serving][error][llm_node_initializer.cpp:60] Failed to process LLM node graph OpenVINO/LFM2-24B-A2B-int4-ov
[2026-07-24 17:21:14.050][1][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: OpenVINO/LFM2-24B-A2B-int4-ov state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent: 
[2026-07-24 17:21:14.050][1][modelmanager][error][modelmanager.cpp:208] Couldn't start model manager
[2026-07-24 17:21:14.050][1][serving][error][servablemanagermodule.cpp:58] ovms::ModelManager::Start() Error: The LLM Node resource initialization failed
[2026-07-24 17:21:14.050][1][serving][info][grpcservermodule.cpp:201] GRPCServerModule shutting down
[2026-07-24 17:21:14.050][1][serving][info][grpcservermodule.cpp:211] GRPCServerModule shutdown
[2026-07-24 17:21:14.050][1][serving][info][httpservermodule.cpp:59] HTTPServerModule shutting down
[2026-07-24 17:21:14.053][1][serving][info][httpservermodule.cpp:64] Shutdown HTTP server
[2026-07-24 17:21:14.053][1][serving][info][servablemanagermodule.cpp:65] ServableManagerModule shutting down
[2026-07-24 17:21:14.054][1][serving][info][servablemanagermodule.cpp:71] ServableManagerModule shutdown
[2026-07-24 17:21:14.054][1][serving][info][pythoninterpretermodule.cpp:70] PythonInterpreterModule shutting down
[2026-07-24 17:21:14.054][1][serving][info][pythoninterpretermodule.cpp:74] PythonInterpreterModule shutdown
Python version:
3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0]
Python sys.path output:
['', '/ovms/lib/python', '/ovms/python_deps', '/lib/python312.zip', '/lib/python3.12', '/lib/python3.12/lib-dynload']
[2026-07-24 17:21:14.056][1][serving][info][capimodule.cpp:50] C-APIModule shutting down
[2026-07-24 17:21:14.056][1][serving][info][capimodule.cpp:52] C-APIModule shutdown

Configuration

1. OVMS version: latest-gpu (as of 2026-07-24)
2. OVMS command line arguments:

   --source_model OpenVINO/LFM2-24B-A2B-int4-ov
   --model_repository_path models
   --task text_generation
   --tool_parser lfm2
   --rest_port 8000
   --target_device GPU
   

3. GPU, accelerator's versions if applicable:
   • Intel GPU with /dev/dri support
4. Model repository directory structure:

   ~/models/
   └── OpenVINO/
       └── LFM2-24B-A2B-int4-ov/
           ├── README.md
           ├── chat_template.jinja
           ├── config.json
           ├── generation_config.json
           ├── graph.pbtxt
           ├── openvino_config.json
           ├── openvino_detokenizer.bin
           ├── openvino_detokenizer.xml
           ├── openvino_model.bin
           ├── openvino_model.xml
           ├── openvino_tokenizer.bin
           ├── openvino_tokenizer.xml
           ├── tokenizer.json
           └── tokenizer_config.json
   

5. Model or publicly available similar model that reproduces the issue:

   https://huggingface.co/OpenVINO/LFM2-24B-A2B-int4-ov
   

Additional context

The same error occurs when using --target_device CPU.

Another error happens if openvino/model_server:2026.2.1-gpu is used to start the server, for example, for another model LFM2.5-350M-int8-ov:

docker run \
  --name ovms-lfm2 \
  --user "$(id -u):$(id -g)" \
  -p 8000:8000 \
  --device /dev/dri:/dev/dri \
  --group-add "$(getent group render | cut -d: -f3)" \
  -v "${HOME}/models:/models:rw" \
  openvino/model_server:2026.2.1-gpu \
  --source_model OpenVINO/LFM2.5-350M-int8-ov \
  --model_repository_path models \
  --task text_generation \
  --tool_parser lfm2 \
  --rest_port 8000 \
  --target_device GPU

Model loads successfully however if fails to serve an example request.

curl -s http://localhost:8000/v3/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "OpenVINO/LFM2.5-350M-int8-ov",
    "max_tokens": 1000,
    "temperature": 0,
    "stream": false,
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "What are the 3 main tourist attractions in Paris?. Explain in detail." }
    ]
  }'

[2026-07-25 16:02:40.346][1][serving][info][server.cpp:117] OpenVINO Model Server 2026.2.1.1122f03bf
[2026-07-25 16:02:40.346][1][serving][info][server.cpp:118] OpenVINO backend 2026.2.1-21919-ede283a88e3-releases/2026/2
[2026-07-25 16:02:40.346][1][serving][info][server.cpp:121] OpenVINO GenAI backend 2026.2.1.0-3123-7dea0459b2a
[2026-07-25 16:02:40.346][1][serving][info][pythoninterpretermodule.cpp:37] PythonInterpreterModule starting
[2026-07-25 16:02:40.346][1][serving][info][pythoninterpretermodule.cpp:40] Initializing python interpreter
[2026-07-25 16:02:40.403][1][serving][info][pythoninterpretermodule.cpp:59] PythonInterpreterModule started
[2026-07-25 16:02:40.446][1][modelmanager][info][modelmanager.cpp:180] Available devices for Open VINO: CPU, GPU
[2026-07-25 16:02:40.447][1][serving][info][modelmanager.cpp:171] Running inside Docker container
[2026-07-25 16:02:40.447][1][serving][info][modelmanager.cpp:172] cpu quota: 0, cpu affinity: 14, max_open_files: 1024
[2026-07-25 16:02:40.447][1][serving][info][capimodule.cpp:40] C-APIModule starting
[2026-07-25 16:02:40.447][1][serving][info][capimodule.cpp:42] C-APIModule started
[2026-07-25 16:02:40.447][1][serving][info][grpcservermodule.cpp:110] GRPCServerModule starting
[2026-07-25 16:02:40.447][1][serving][info][grpcservermodule.cpp:114] GRPCServerModule started
[2026-07-25 16:02:40.447][1][serving][info][grpcservermodule.cpp:115] Port was not set. GRPC server will not be started.
[2026-07-25 16:02:40.447][1][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2026-07-25 16:02:40.447][1][serving][info][httpservermodule.cpp:39] Will start 14 REST workers
[2026-07-25 16:02:40.447][55][serving][info][drogon_http_server.cpp:157] Binding REST server to address: 0.0.0.0:8000
[2026-07-25 16:02:40.498][1][serving][info][drogon_http_server.cpp:184] REST server listening on port 8000 with 14 unary threads and 14 streaming threads
[2026-07-25 16:02:40.498][1][serving][info][http_server.cpp:249] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2026-07-25 16:02:40.498][1][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2026-07-25 16:02:40.498][1][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:8000
[INFO] LFS resume: attempts=5 interval=10 s
[2026-07-25 16:02:40.534][1][serving][info][libgit2.cpp:1282] Path already exists on local filesystem. Skipping download to path: models/OpenVINO/LFM2.5-350M-int8-ov
Model: OpenVINO/LFM2.5-350M-int8-ov downloaded to: models/OpenVINO/LFM2.5-350M-int8-ov
Graph: graph.pbtxt created in: models/OpenVINO/LFM2.5-350M-int8-ov
[2026-07-25 16:02:40.535][1][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-07-25 16:02:40.535][1][serving][info][mediapipegraphdefinition.cpp:101] Graph queue globally disabled via OVMS_GRAPH_QUEUE_OFF=1 for mediapipe: OpenVINO/LFM2.5-350M-int8-ov
[2026-07-25 16:02:40.535][1][serving][info][mediapipegraphdefinition.cpp:479] MediapipeGraphDefinition initializing graph nodes
[2026-07-25 16:02:40.535][1][modelmanager][info][servable_initializer.cpp:439] Initializing Language Model Continuous Batching servable
[2026-07-25 16:02:43.862][1][modelmanager][info][mediapipegraphdefinition.cpp:261] Mediapipe: OpenVINO/LFM2.5-350M-int8-ov inputs: 
name: input; mapping: ; shape: (); precision: UNDEFINED; layout: ...
[2026-07-25 16:02:43.862][1][modelmanager][info][mediapipegraphdefinition.cpp:262] Mediapipe: OpenVINO/LFM2.5-350M-int8-ov outputs: 
name: output; mapping: ; shape: (); precision: UNDEFINED; layout: ...
[2026-07-25 16:02:43.862][1][modelmanager][info][mediapipegraphdefinition.cpp:263] Mediapipe: OpenVINO/LFM2.5-350M-int8-ov kfs pass through: false
[2026-07-25 16:02:43.862][1][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: OpenVINO/LFM2.5-350M-int8-ov state changed to: AVAILABLE after handling: ValidationPassedEvent: 
[2026-07-25 16:02:43.862][126][llm_executor][info][llm_executor.hpp:128] All requests: 0; Scheduled requests: 0;
[2026-07-25 16:02:43.862][1][serving][info][servablemanagermodule.cpp:55] ServableManagerModule started
[2026-07-25 16:02:43.862][127][modelmanager][info][modelmanager.cpp:1060] Started model manager thread
[2026-07-25 16:02:43.862][128][modelmanager][info][modelmanager.cpp:1079] Started cleaner thread
[2026-07-25 16:02:48.589][126][llm_executor][error][llm_executor.hpp:132] Error occurred in LLM executor: Exception from src/inference/src/cpp/infer_request.cpp:224:
Check 'std::all_of(indices_shape.begin(), indices_shape.begin() + last_idx_pos, is_update_compatible)' failed at src/core/shape_inference/include/scatter_nd_base_shape_inference.hpp:53:
While validating node 'opset4::ScatterNDUpdate ScatterNDUpdate_73454 () -> ()' with friendly_name 'ScatterNDUpdate_73454':
updates_shape[0:indices_rank-1] shape must be indices_shape[:-1]

Based on these blog posts this model should be supported: https://www.intel.com/content/www/us/en/developer/articles/community/lfm2-24b-a2b-with-openvino.html
https://docs.openvino.ai/nightly/about-openvino/release-notes-openvino.html

[dtrawins]

In version 2026.2 model LFM2 wasn't supported with continuous batching pipeline (default).
It can be used however with latest weekly drop https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/weekly/2026.3.0_cb34be47/
LFM2.5 models are also supported.

[ss4workonly]

Thank a lot for the clarification. Going forward, it would be great if the application failed fast with a clear error message in such scenarios.