CUDA on Kubernetes

[obriensystems]

CUDA (NVIDIA GPUs) on Kubernetes - options

1. Kubeadm
2. RKE

A) requirements from NVidia

• NVidia container toolkit https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

[obriensystems]

check RTX-3500 - docker desktop does no support CUDA passthrough - use RKE2

docker run --rm --gpus all --name ml-tensorflow-win ml-tensorflow-win

 exporting to image
#5 exporting layers done
#5 writing image sha256:8148465eef5f0c74de65b8a926ea4f7723b21e92ce7cc53ea5f2f919bfa8f829 done
#5 naming to docker.io/library/ml-tensorflow-win done
#5 DONE 0.0s

 1 warning found (use docker --debug to expand):
 - JSONArgsRecommended: JSON arguments recommended for CMD to prevent unintended behavior related to OS signals (line 3)

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/v1woacchs0ac3bpzhxc44yzlw

==========
== CUDA ==
==========

CUDA Version 12.2.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Mon Jul 13 03:53:28 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.45.03              Driver Version: 595.71         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX 3500 Ada Gene...    On  |   00000000:01:00.0 Off |                  Off |
| N/A   53C    P3             25W /   84W |       0MiB /  12282MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/20260712/machine-learning/cuda (main)
$ kubectl get nodes   -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
NAME                    GPU
desktop-control-plane   <none>

There is a disconnect with the docker desktop kubernetes plugin - docker is ok

[obriensystems]

on ubuntu 25 with an RTX-A6000

https://github.com/nvidia/k8s-device-plugin
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Welcome to Ubuntu 25.10 (GNU/Linux 6.17.0-35-generic x86_64)
ubuntu@14900c:~$ nvidia-smi
Mon Jul 13 18:08:05 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05              Driver Version: 595.71.05      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:01:00.0 Off |                  Off |
| 30%   40C    P8             17W /  300W |     175MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            3377      G   /usr/bin/gnome-shell                    132MiB |


sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2
[sudo: authenticate] Password: 
Get:1 file:/var/cuda-repo-ubuntu2404-13-1-local  InRelease [1,572 B]
Get:1 file:/var/cuda-repo-ubuntu2404-13-1-local  InRelease [1,572 B]
Hit:2 http://security.ubuntu.com/ubuntu questing-security InRelease
Ign:3 https://packagecloud.io/ookla/speedtest-cli/ubuntu questing InRelease                                                           
Err:4 https://packagecloud.io/ookla/speedtest-cli/ubuntu questing Release                                                             
  404  Not Found [IP: 2600:1f1c:2e5:6901:6f20:8924:aad9:7c7c 443]


sudo apt-get install -y --no-install-recommends    ca-certificates 
sudo apt-get install -y --no-install-recommends   gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /
#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/experimental/deb/$(ARCH) /

sudo apt-get update
Get:1 file:/var/cuda-repo-ubuntu2404-13-1-local  InRelease [1,572 B]
Get:1 file:/var/cuda-repo-ubuntu2404-13-1-local  InRelease [1,572 B]
Get:2 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  InRelease [1,477 B]
Get:3 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  Packages [26.1 kB]                                                                                                
Ign:4 https://packagecloud.io/ookla/speedtest-cli/ubuntu questing InRelease                                                                                                                      
Hit:5 http://ca.archive.ubuntu.com/ubuntu questing InRelease                                                                                                              
Hit:6 http://ca.archive.ubuntu.com/ubuntu questing-updates InRelease         
Hit:7 http://security.ubuntu.com/ubuntu questing-security InRelease 
Err:8 https://packagecloud.io/ookla/speedtest-cli/ubuntu questing Release
  404  Not Found [IP: 2600:1f1c:2e5:6900:a033:5af:e178:4155 443]

ubuntu@14900c:~$ export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.19.1-1
ubuntu@14900c:~$ sudo apt-get install -y \
      nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Solving dependencies... Done
The following packages were automatically installed and are no longer required:
  libnvidia-cfg1-590 libnvidia-common-590 libnvidia-compute-590 libnvidia-decode-590 libnvidia-encode-590 libnvidia-extra-590 libnvidia-fbc1-590 libnvidia-gl-590 nvidia-compute-utils-590 nvidia-firmware-590-590.48.01
  nvidia-kernel-source-590-open nvidia-utils-590 xserver-xorg-video-nvidia-590
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  libnvidia-container-tools libnvidia-container1 nvidia-container-toolkit nvidia-container-toolkit-base
0 upgraded, 4 newly installed, 0 to remove and 45 not upgraded.
Need to get 8,155 kB of archives.
After this operation, 35.3 MB of additional disk space will be used.
Get:1 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  libnvidia-container1 1.19.1-1 [1,191 kB]
Get:2 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  libnvidia-container-tools 1.19.1-1 [20.8 kB]
Get:3 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  nvidia-container-toolkit-base 1.19.1-1 [5,609 kB]
Get:4 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  nvidia-container-toolkit 1.19.1-1 [1,334 kB]
Fetched 8,155 kB in 1s (11.3 MB/s)              
Selecting previously unselected package libnvidia-container1:amd64.
(Reading database ... 228605 files and directories currently installed.)
Preparing to unpack .../libnvidia-container1_1.19.1-1_amd64.deb ...
Unpacking libnvidia-container1:amd64 (1.19.1-1) ...
Selecting previously unselected package libnvidia-container-tools.
Preparing to unpack .../libnvidia-container-tools_1.19.1-1_amd64.deb ...
Unpacking libnvidia-container-tools (1.19.1-1) ...
Selecting previously unselected package nvidia-container-toolkit-base.
Preparing to unpack .../nvidia-container-toolkit-base_1.19.1-1_amd64.deb ...
Unpacking nvidia-container-toolkit-base (1.19.1-1) ...
Selecting previously unselected package nvidia-container-toolkit.
Preparing to unpack .../nvidia-container-toolkit_1.19.1-1_amd64.deb ...
Unpacking nvidia-container-toolkit (1.19.1-1) ...
Setting up nvidia-container-toolkit-base (1.19.1-1) ...
Created symlink '/etc/systemd/system/multi-user.target.wants/nvidia-cdi-refresh.path' → '/etc/systemd/system/nvidia-cdi-refresh.path'.
Created symlink '/etc/systemd/system/multi-user.target.wants/nvidia-cdi-refresh.service' → '/etc/systemd/system/nvidia-cdi-refresh.service'.
Regenerating NVIDIA CDI specification...
Setting up libnvidia-container1:amd64 (1.19.1-1) ...
Setting up libnvidia-container-tools (1.19.1-1) ...
Setting up nvidia-container-toolkit (1.19.1-1) ...
Processing triggers for libc-bin (2.42-0ubuntu3.1) ...
Scanning processes...                                                                                                                                                                                                                     
Scanning candidates...                                                                                                                                                                                                                    
Scanning processor microcode...                                                                                                                                                                                                           
Scanning linux images...                                                                                                                                                                                                                  

Pending kernel upgrade!
Running kernel version:
  6.17.0-35-generic
Diagnostics:
  The currently running kernel version is not the expected kernel version 6.17.0-40-generic.

Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting.

The processor microcode seems to be up-to-date.

Restarting services...

Service restarts being deferred:
 systemctl restart NetworkManager.service

No containers need to be restarted.

User sessions running outdated binaries:
 ubuntu @ user service: dbus.service[3437,3471,3665,3720], evolution-addressbook-factory.service[3821], evolution-calendar-factory.service[3771], evolution-source-registry.service[3457], gnome-session-manager@ubuntu.service[3361],
  gnome-terminal-server.service[4210], localsearch-3.service[3735], org.freedesktop.IBus.session.GNOME.service[3478], org.gnome.SettingsDaemon.A11ySettings.service[3481], org.gnome.SettingsDaemon.Datetime.service[3492],
  org.gnome.SettingsDaemon.MediaKeys.service[3499], org.gnome.SettingsDaemon.Power.service[3504], org.gnome.SettingsDaemon.PrintNotifications.service[3509,3753], org.gnome.Shell@wayland.service[3377], pipewire-pulse.service[3260],
  update-notifier-release.service[4499], xdg-desktop-portal-gnome.service[3922], xdg-desktop-portal-gtk.service[3978], xdg-desktop-portal.service[3907]

No VM guests are running outdated hypervisor (qemu) binaries on this host.

ubuntu@14900c:~$ kubectl get nodes
NAME     STATUS   ROLES                AGE     VERSION
14900c   Ready    control-plane,etcd   5h54m   v1.35.6+rke2r1

ubuntu@14900c:~$ kubectl get nodes   -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
NAME     GPU
14900c   <none>
ubuntu@14900c:~$ sudo reboot now