Misc. bug: Model layer memory not split to multiple CUDA cards after release b9439

[amv]

Name and Version

$ build/bin/llama-cli --version
version: 9439 (22cadc1)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

$ build/bin/llama-server -m Qwen3.6-27B-MTP-UD-Q6_K_XL.gguf -ngl all -c 4096 -ts "10,10"

Problem description & steps to reproduce

I have 2 NVIDIA cards detected as CUDA0 and CUDA1.

On b9437 things work, and model layer memory is split to both cards according to --tensor-split.

After b9439 it detects and lists both CUDA cards, but all of the model is loaded on CUDA0.

I think the relevant PR was this: #23897

First Bad Commit

tags/b9439

Relevant log output

Logs

0.00.345.957 I device_info:
0.00.436.742 I   - CUDA0   : NVIDIA GeForce RTX 5070 Ti (15838 MiB, 58499 MiB free)
0.00.528.104 I   - CUDA1   : NVIDIA GeForce RTX 5070 Ti (15841 MiB, 58443 MiB free)
0.00.528.109 I   - CPU     : AMD Ryzen 7 7800X3D 8-Core Processor (62151 MiB, 62151 MiB free)
.....
0.02.473.647 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 23511.49 MiB on device 0: cudaMalloc failed: out of memory

[amv]

I started to ponder why my dual RTX GPU setup would be going to the GGML_BACKEND_DEVICE_TYPE_IGPU case that was touched in the PR, and not the GGML_BACKEND_DEVICE_TYPE_GPU case, but given that this case is the only thing changed in the PR, I can see no other explanation.

One theory I could come up with was that having my CPU as one of the devices would cause going to that switch case, so tried compiling with -DGGML_CPU=OFF. Unfortunately my build failed with this flag, so I could not test my theory.

[amv]

I quickly patched the cuda ggml_backend_dev_type detection to always report GPU instead of IGPU, and it seems to solve the issue for me. So this would indicate that the problem is not really the PR in question, but it surfaces a problem with the GPU/IGPU device type detection:

diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
index 18aaa0983..2c8b58353 100644
--- a/ggml/src/ggml-cuda/ggml-cuda.cu
+++ b/ggml/src/ggml-cuda/ggml-cuda.cu
@@ -4999,9 +4999,7 @@ static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend
     cudaDeviceProp prop;
     CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device));

-    return prop.integrated
-        ? GGML_BACKEND_DEVICE_TYPE_IGPU
-        : GGML_BACKEND_DEVICE_TYPE_GPU;
+    return GGML_BACKEND_DEVICE_TYPE_GPU;
 }

 static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_backend_dev_props * props) {

[amv]

This is very peculiar... This would indicate that the cards would not be returning "integrated", but still the above patch that forces type GPU irregardless of the "integrated" prop fixes the issue for me:

cat << 'EOF' > query.cu
#include <stdio.h>
int main() {
    int count;
    cudaGetDeviceCount(&count);
    printf("Found %d device(s)\n", count);
    for (int i = 0; i < count; i++) {
        cudaDeviceProp p;
        cudaGetDeviceProperties(&p, i);
        printf("Device %d: %s\n", i, p.name);
        printf("  Integrated: %d\n", p.integrated);
    }
}
EOF
nvcc query.cu -o query && ./query
Found 2 device(s)
Device 0: NVIDIA GeForce RTX 5070 Ti
  Integrated: 0
Device 1: NVIDIA GeForce RTX 5070 Ti
  Integrated: 0

[gopinath87607]

same issues

[gopinath87607]

cpu moe flag and ts for each gpu also not working

[amv]

Debugged this further with some temporary logging. I found out that cudaGetDeviceProperties returns "integrated" prop value of 70 inside the llama-server runtime.

After asking about this from several LLMs, I was able to figure out that compilation has been detecting an older version of the cuda libraries, and a newer version of nccl. It has been a miracle that things have worked as well as they have.

I got my problem fixed with the following build command:

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_NCCL=ON -DCUDAToolkit_ROOT=/usr/local/cuda-13.3

[amv]

For someone else experiencing a similar problem, running the following commands allowed me to pinpoint the version mismatch:

❯ ~/llama.cpp$ ldd ./build/bin/llama-server 2>&1 | grep -i cuda
      libggml-cuda.so.0 => /home/amv/llama.cpp/build/bin/libggml-cuda.so.0 (0x00007d9502800000)
      libcudart.so.12 => /usr/lib/x86_64-linux-gnu/libcudart.so.12 (0x00007d9502400000)
      libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007d94f6000000)
  
❯ ~/llama.cpp$ nvcc --version
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2026 NVIDIA Corporation
  Built on Fri_Apr_24_07:22:02_PM_PDT_2026
  Cuda compilation tools, release 13.3, V13.3.33
  Build cuda_13.3.r13.3/compiler.37862127_0

The fix was to help cmake find the same version of the toolkit using -DCUDAToolkit_ROOT

I believe the issue can be closed.