Skip to content

ci: fix vulkan docker images#24595

Merged
0cc4m merged 3 commits into
ggml-org:masterfrom
Kononnable:fix-24474
Jun 17, 2026
Merged

ci: fix vulkan docker images#24595
0cc4m merged 3 commits into
ggml-org:masterfrom
Kononnable:fix-24474

Conversation

@Kononnable

@Kononnable Kononnable commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Overview

Starting with b9438 vulkan docker images produced by CI are broken. We run out of memory during shaders generation. CI doesn't report an error, but build artifact are corrupted.
https://github.com/ggml-org/llama.cpp/actions/runs/27397273833/job/80967212036#step:9:2828

#14 337.5 Cannot allocate memory
#14 337.5 Error executing command for matmul_id_subgroup_q5_0_f32_aligned: Failed to fork process

Fixes #24474

Container build on fork: https://github.com/Kononnable/llama.cpp/actions/runs/27477825868/job/81220277837#step:9:2810

Additional information

Recent additions to mul_nn.comp caused RSS memory counters to hit system limits during shaders generation. Building docker images outside of CI worked fine - usually personal computers have more RAM than CI agents. On CI problems started to happen when RSS was around 21GB.

In string_to_spv we call std::async to be able to generate multiple shaders in parallel. We store the futures in compiles and never delete them. We never release the resources taken by those futures.
Creating a future creates a thread which reserves virtual memory for things like the call stack. They don't use a lot of RAM, but each allocates 8MB of virtual memory. We do that thousands of times - it goes into gigabytes of virtual memory. This becomes a problem when we hit limits like /proc/sys/vm/overcommit_*.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, faster loop during problem debugging

@Kononnable Kononnable requested a review from a team as a code owner June 13, 2026 21:15
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Jun 13, 2026
@jeffbolznv

Copy link
Copy Markdown
Contributor

I don't understand fully this change. Is this just trying to clean up threads from finished compiles in order to recover memory? I'm not really clear on what memory they're holding on to that this is releasing.

I've experimented with some changes to reduce the number of matmul shaders we need to generate, maybe that would help too.

@Kononnable

Copy link
Copy Markdown
Contributor Author

Sorry, I could have provided more context from the beginning. Also, I've messed up my notes about RSS memory(It's about virtual memory).
I've updated the original comment with proper description on why that happens (we don't clean up threads).

@jeffbolznv

Copy link
Copy Markdown
Contributor

OK, that makes more sense. When you say thousands I assume that's across all processes? I wouldn't expect that many within a process.

Please add comments saying what the code is doing and why, then I think it's OK.

add comment describing code change intention
@Kononnable

Copy link
Copy Markdown
Contributor Author

For most files number of string_to_spv calls is small (<200). mul_mm.comp is the exception with 1850 calls.

Comment thread ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp Outdated

@CISC CISC left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@0cc4m Merge?

@CISC CISC linked an issue Jun 16, 2026 that may be closed by this pull request
@0cc4m 0cc4m merged commit d5376cf into ggml-org:master Jun 17, 2026
27 checks passed
papamoose pushed a commit to papamoose/llama.cpp that referenced this pull request Jun 27, 2026
* Update vulkan-shaders-gen.cpp

* Update vulkan-shaders-gen.cpp

add comment describing code change intention

* Update vulkan-shaders-gen.cpp

fix potential UB
adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026
* Update vulkan-shaders-gen.cpp

* Update vulkan-shaders-gen.cpp

add comment describing code change intention

* Update vulkan-shaders-gen.cpp

fix potential UB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

4 participants