ci: fix vulkan docker images#24595
Conversation
|
I don't understand fully this change. Is this just trying to clean up threads from finished compiles in order to recover memory? I'm not really clear on what memory they're holding on to that this is releasing. I've experimented with some changes to reduce the number of matmul shaders we need to generate, maybe that would help too. |
|
Sorry, I could have provided more context from the beginning. Also, I've messed up my notes about RSS memory(It's about virtual memory). |
|
OK, that makes more sense. When you say thousands I assume that's across all processes? I wouldn't expect that many within a process. Please add comments saying what the code is doing and why, then I think it's OK. |
add comment describing code change intention
|
For most files number of |
fix potential UB
* Update vulkan-shaders-gen.cpp * Update vulkan-shaders-gen.cpp add comment describing code change intention * Update vulkan-shaders-gen.cpp fix potential UB
* Update vulkan-shaders-gen.cpp * Update vulkan-shaders-gen.cpp add comment describing code change intention * Update vulkan-shaders-gen.cpp fix potential UB
Overview
Starting with
b9438vulkan docker images produced by CI are broken. We run out of memory during shaders generation. CI doesn't report an error, but build artifact are corrupted.https://github.com/ggml-org/llama.cpp/actions/runs/27397273833/job/80967212036#step:9:2828
Fixes #24474
Container build on fork: https://github.com/Kononnable/llama.cpp/actions/runs/27477825868/job/81220277837#step:9:2810
Additional information
Recent additions tomul_nn.compcaused RSS memory counters to hit system limits during shaders generation. Building docker images outside of CI worked fine - usually personal computers have more RAM than CI agents. On CI problems started to happen when RSS was around 21GB.In
string_to_spvwe callstd::asyncto be able to generate multiple shaders in parallel. We store the futures incompilesand never delete them. We never release the resources taken by those futures.Creating a future creates a thread which reserves virtual memory for things like the call stack. They don't use a lot of RAM, but each allocates 8MB of virtual memory. We do that thousands of times - it goes into gigabytes of virtual memory. This becomes a problem when we hit limits like
/proc/sys/vm/overcommit_*.Requirements