openvino: OV 2026.2, context-shift, Q5_1 support, gemma4 dense/embedding, and -fa off by wine99 · Pull Request #24503 · ggml-org/llama.cpp

wine99 · 2026-06-12T06:22:29Z

Overview

Updated OpenVINO version to 2026.2, which resolves quantized model accuracy issues on GPU
Enabled context-shift (--context-shift) on CPU and GPU (stateless mode)
Added Q5_1 quantization support
Added support for Gemma 4 dense models and embedding models (e.g. bge-m3)
Enabled -fa off (non-flash-attention path)
Code structure refactoring: centralized env var handling, refactored VIEW operations, improved attention pattern abstraction
New ops: REPEAT, ADD_ID, IM2COL, SSM_CONV, GATED_DELTA_NET, L2_NORM, PAD, NORM/TANH, SOFTPLUS, ARGSORT, CLAMP, SUM_ROWS, DIV, CONCAT, MUL_MAT_ID
Manual GQA attention enabled by default for stateless GPU to improve performance
Added Windows CI for OpenVINO
Updated documentation with Windows build instructions and automated build scripts for both Ubuntu and Windows
Updated validated models table across CPU, GPU, and NPU
Various bug fixes and stability improvements

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES. AI was used in an assistive capacity for identifying bugs and expanding on verbose modifications.

…GGML graph in api compute_model_outputs()

… models

…backend

)

…on handling

…ention

…ntion_pattern_case to easy extand

* added translate_1to1_match_1_input function and updated gelu and tanh translations * Remove unused translation function calls --------- Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>

* OpenVINO backend: refactor VIEW related operation * Enable VIEW handling in following ops * OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO

…matched types

ggerganov

The gpu-openvino-low-perf job has been cancelled regularly: https://github.com/ggml-org/llama.cpp/actions/runs/27530076257/job/81365779158

Could you resolve that first? I think the concurrency group is misconfigured.

ggerganov · 2026-06-15T07:26:01Z

+      - name: Setup Cache
+        uses: actions/cache@v5
+        id: cache-openvino
+        with:
+          path: ./openvino_toolkit
+          key: cache-gha-openvino-toolkit-v${{ env.OPENVINO_VERSION_FULL }}-${{ runner.os }}


How big is this cache?

The cache-gha-openvino-toolkit-v2026.2.0.21903.52ddc073857-Windows: 180 MB

ggerganov · 2026-06-15T07:28:08Z

+  windows-2022-openvino:
+    runs-on: windows-2022
+
+    env:
+      # Sync versions in build-openvino.yml, build-self-hosted.yml, release.yml, build-cache.yml, .devops/openvino.Dockerfile
+      OPENVINO_VERSION_MAJOR: "2026.2"
+      OPENVINO_VERSION_FULL: "2026.2.0.21903.52ddc073857"


How long does this build take? Can it be offloaded to self-hosted runners?

The recent time for windows-2022-openvino: succeeded 5 days ago in 13m 14s

I will check internally and try to move to self-hosted runner. Will follow up with another PR.

OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision

ravi9 · 2026-06-15T16:25:00Z

The gpu-openvino-low-perf job has been cancelled regularly: https://github.com/ggml-org/llama.cpp/actions/runs/27530076257/job/81365779158

Could you resolve that first? I think the concurrency group is misconfigured.

Done. Removed the strict concurrency check for gpu-openvino-low-perf and openvino CI (GPU) runs.

ggerganov · 2026-06-16T06:16:23Z

+      - name: ccache
+        uses: ggml-org/ccache-action@v1.2.21
+        with:
+          key: windows-2022-openvino


For consistency:

Suggested change

key: windows-2022-openvino

key: openvino-windows-2022

ggerganov · 2026-06-16T06:17:49Z

+      - name: ccache
+        uses: ggml-org/ccache-action@v1.2.21
+        with:
+          key: windows-2022-openvino-release


Suggested change

key: windows-2022-openvino-release

key: release-windows-2022-openvino

ggerganov · 2026-06-16T06:19:43Z

+      - name: ccache
+        uses: ggml-org/ccache-action@v1.2.21
+        with:
+          key: windows-2022-openvino-release
+          variant: ccache
+          evict-old-files: 1d


Need to add the ccache-clear action after the build - see how the rest of the jobs in the release workflow are configured.

CISC

Please provide successful docker and release runs on your fork.

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

ravi9 · 2026-06-16T14:29:12Z

Please provide successful docker and release runs on your fork.

@CISC Following are successful runs on our fork with ce52f0a

Release: https://github.com/ravi9/llama.cpp/actions/runs/27623189204
Docker (openvino): https://github.com/ravi9/llama.cpp/actions/runs/27623169984/job/81677654775

CISC

The \r\n GHUI ghost strikes again, please normalize release.yml to \n. :)

ravi9 · 2026-06-16T15:08:04Z

@CISC normalized release.yml :)
Thanks for reviewing!

ravi9 · 2026-06-17T06:08:24Z

@ggerganov Let us know if anything else is needed..
Thanks.

…ing, and -fa off (ggml-org#24503) * Add interface is_model_splitted() to check the c-graph is splited or not * Infer and propagate dynamic-dimension indices for all tensors in the GGML graph in api compute_model_outputs() * Only do this for fallback sub graph * Move dynamic dims compute in graph missmatch * ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split models * ggml-openvino:add comments * ggml-openvino: override VIEW op_case to 0 for split model inputs * openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO backend * Enable additional mul_mat tests and add tensor data saving function (ggml-org#81) * ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimension handling * OpenVINO: add NORM/TANH support and rework SOFT_MAX translation * ggml-openvino: extend VIEW handling * Enable -fa off (ggml-org#118) * Enable --context-shift * Fix llm param compute error for normal softmax not the softmax in attention * OpenVINO backend: fix error for attention size compute in llm param * use tensor->extra in infer_request i/o * OpenVINO backend: refacter the compute_llm_params() func add get_attention_pattern_case to easy extand * OpenVINO backend: clean unused code * 1to1 match op update (ggml-org#146) * added translate_1to1_match_1_input function and updated gelu and tanh translations * Remove unused translation function calls --------- Co-authored-by: Mustafa Cavus <mustafacavus@intel.com> * initial gemma4 support * removed hardcoded names for kv cache slicing * OpenVINO backend: Add new attention pattern for llm parameters compute * flash attn Q shape static conversion * Remove slice in permute translation when n_seq is 1 * return optional in extract_layer_from_name * OpenVINO backend: refactor VIEW related operation (ggml-org#148) * OpenVINO backend: refactor VIEW related operation * Enable VIEW handling in following ops * OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO * OpenVINO backend: Add ops l2_norm & pad * OpenVINO backend does not support CPY with non-contiguous data or mismatched types * add op SSM_CONV GATED_DELTA_NET * OpenVINO backend: fix error for bf16 in OV gpu plugin * reverted static Q input shape for attention layer * OpenVINO backend: remove hardcode name inp_tokens, which ignore some leaf case * Disable remote tensor due to bug in ov gpu * Disable n_token > 1 GATED_DELTA_NET on gpu * OpenVINO backend: fix the view op dynamic handling issue in gemma4 & enable view + get_row * OpenVINO backend: clean code * OpenVINO backend: enable view + norm/rms_norm * OpenVINO backend: concat op * OpenVINO backend: argsort op * OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS * Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops result, VIEW node in OpenVINO no need compare, the whole graph result is correct * OpenVINO backend: enable sum_rows * OpenVINO backend: enable clamp * OpenVINO backend: enable DIV * OpenVINO backend: enable GGML_OP_MUL_MAT_ID * OpenVINO backend: disable MUL_MAT_ID_FUSION case with large mem needed * OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed * OpenVINO backend: fix issue in mul_mat_id * OpenVINO backend: Disable DIV with broadcast on GPU * OpenVINO backend: update DIV * use ov internal op GatedDeltaNet * OpenVINO backend: enable llama erch test qwen3next * OpenVINO backend: enable RMS_NORM + VIEW & remove op_case 2 for rope * OpenVINO backend: fix error * suggested changes, need review * suggested changes, need review * OpenVINO backend: clean unused code & fix build warning * OpenVINO backend: enable minicpm3 for arch test * Disable GDN op (ggml-org#177) * disable gated_delta_net * update stateful_kv_size correctly in mismatch case * OpenVINO backend: enable arch test for qwen3vl * OpenVINO backend: enable cohere2 for arch test * OpenVINO backend: enable t5 for arch test * OpenVINO backend: enable jamba for arch test * OpenVINO backend: remove warning for tmp * OpenVINO backend: enable kimi-linear for arch test * Remove unused * Fix gpt-oss accuracy issue * OpenVINO backend: enable arctic for arch test * OpenVINO backend: enable grok for arch test * Gemma4 initial npu support (ggml-org#179) * Initiall gemma4 npu support * temp. fix for gemma4 accuracy bug on npu * Remove hardcoded names for npu-fold handling * revert static n tokens for cont translation as it is not needed * removed unused variable * ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control decoder cache. Add environment variable GGML_OPENVINO_ENABLE_CACHE (default: YES). When set to NO, the decoder_cache is bypassed and models are rebuilt from the cgraph on every inference call in both dynamic and static compute paths. This is useful for debugging and verifying correctness without caching interference. * Revert "Gemma4 initial npu support (ggml-org#179)" This reverts commit 0d29a9c. * OpenVINO backend: disable debug log print * Update TBB discovery. Delegated to OpenVINOs own config. * OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1 * OpenVINO backend: fallback FLASH_ATTN_EXT in gemma3n to CPU backend * Add raw ov infer profiling metric * Add OV raw infer time metric to static compute path Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com> * Modify precision of static profiling * update to OV 2026.2, add OV windows CI * fix editorconfig-checks * Initiall gemma4 npu support * temp. fix for gemma4 accuracy bug on npu * Remove hardcoded names for npu-fold handling * revert static n tokens for cont translation as it is not needed * removed unused variable * test-llama-archs fix * Fix gemma4 flash_attn fallback * support im2col * fix code style * disable add_rope_sin_cos optimization * stateless boradcast and rope optimizations * Enable manual gqa attn by default for stateless gpu * manual gqa: fixed static batch * gemma4 llama-bench ctx update fix * Update OV win CI * stateful rope fusion temp. fix * OpenVINO backend: Conslolidate supported ops * Exclude unsupported GGML_OP_SUB cases * Exclude unsupported TOPK_MOE cases * OpenVINO Backend: MUL_MAT enhancements * Update OV CI * support f16 mask input for npu * Make GGML_OPENVINO_* env vars usage uniform Standardize all GGML_OPENVINO_* env flags: positive integers >0 to enable. Unset, empty, =0, or non-numeric values to disable. This fixes cases where text values or empty strings enabled features. * OpenVINO backend: Enhance envvar handling * more cleanup * move ggml_openvino_env_flag to appropriate place * OpenVINO backend: add REPEAT translator, Q5_1 weights, and GLU view-input fix * ggml-openvino: fix -Werror=cast-qual in extract_q5_1_data * Update openvino.Dockerfile Use BuildKit cache mounts for faster Docker rebuilds. Use apt instead of dpkg, remove unused .ddeb downloads, add DLLAMA_BUILD_TESTS=OFF. * ggml-openvino: centralize env var access via *getenv_str/getenv_int helpers Replace getenv and legacy flags with _str and _int helpers.Minor cleanup, doc updates. * OpenVINO backend: Enable GGML_OP_ADD_ID * Uptade openvino backend clamg-format * clang-format * Update OPENVINO.md (ggml-org#211) * OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision * Remove strict concurrency for gpu-openvino-low-perf * Update openvino CI keynames; add ccache-clear * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Fix formatting --------- Co-authored-by: Xuejun Zhai <Xuejun.Zhai@intel.com> Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com> Co-authored-by: Mustafa Cavus <mustafacavus@intel.com> Co-authored-by: Xuejun <XuejunZhai@intel.com> Co-authored-by: Wang Yang <yang4.wang@intel.com> Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com> Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com> Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

zhaixuejun1993 and others added 30 commits June 9, 2026 12:20

Add interface is_model_splitted() to check the c-graph is splited or not

4bc1eb1

Infer and propagate dynamic-dimension indices for all tensors in the …

c49ec28

…GGML graph in api compute_model_outputs()

Only do this for fallback sub graph

76eb69e

Move dynamic dims compute in graph missmatch

7e6caef

ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split…

d306b0b

… models

ggml-openvino:add comments

01088c2

ggml-openvino: override VIEW op_case to 0 for split model inputs

126d758

openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO …

32f9cb7

…backend

Enable additional mul_mat tests and add tensor data saving function (#81

f812f78

)

ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimensi…

865f121

…on handling

OpenVINO: add NORM/TANH support and rework SOFT_MAX translation

ca3a176

ggml-openvino: extend VIEW handling

a73a6dc

Enable -fa off (#118)

bfa4c53

Enable --context-shift

9c922b1

Fix llm param compute error for normal softmax not the softmax in att…

59f0e3c

…ention

OpenVINO backend: fix error for attention size compute in llm param

c8e9ce4

use tensor->extra in infer_request i/o

9f355ed

OpenVINO backend: refacter the compute_llm_params() func add get_atte…

dc5ed75

…ntion_pattern_case to easy extand

OpenVINO backend: clean unused code

e2ce59c

1to1 match op update (#146)

130ef39

* added translate_1to1_match_1_input function and updated gelu and tanh translations * Remove unused translation function calls --------- Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>

initial gemma4 support

13ddbf3

removed hardcoded names for kv cache slicing

7597773

OpenVINO backend: Add new attention pattern for llm parameters compute

a1baa1a

flash attn Q shape static conversion

dad8acd

Remove slice in permute translation when n_seq is 1

760e86d

return optional in extract_layer_from_name

5a39967

OpenVINO backend: refactor VIEW related operation (#148)

d289bbd

* OpenVINO backend: refactor VIEW related operation * Enable VIEW handling in following ops * OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO

OpenVINO backend: Add ops l2_norm & pad

e0caf43

OpenVINO backend does not support CPY with non-contiguous data or mis…

8c83092

…matched types

add op SSM_CONV GATED_DELTA_NET

a08546f

ggerganov reviewed Jun 15, 2026

View reviewed changes

zhaixuejun1993 and others added 3 commits June 15, 2026 15:52

OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision

90ae917

Merge pull request #214 from zhaixuejun1993/xuejun/fix-error-op-concat

383d163

OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision

Remove strict concurrency for gpu-openvino-low-perf

00e80a9

ggerganov self-assigned this Jun 16, 2026

ggerganov reviewed Jun 16, 2026

View reviewed changes

Update openvino CI keynames; add ccache-clear

65d4041

ggerganov requested a review from CISC June 16, 2026 06:44

CISC reviewed Jun 16, 2026

View reviewed changes

Comment thread .github/workflows/release.yml

Comment thread .github/workflows/release.yml Outdated

Apply suggestions from code review

ce52f0a

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

CISC approved these changes Jun 16, 2026

View reviewed changes

Fix formatting

3481530

ggerganov merged commit 890f1a2 into ggml-org:master Jun 17, 2026
27 checks passed

	key: windows-2022-openvino-release
	key: release-windows-2022-openvino

Uh oh!

Conversation

wine99 commented Jun 12, 2026

Overview

Requirements

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ravi9 commented Jun 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CISC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ravi9 commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CISC left a comment

Choose a reason for hiding this comment

Uh oh!

ravi9 commented Jun 16, 2026

Uh oh!

ravi9 commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

ravi9 commented Jun 16, 2026 •

edited

Loading