Add GitHub Actions workflow for compiling CUDA, ROCm, and macOS prebuilts daily and on demand

[oobabooga]

Right now the app-*-linux-x64-cuda* bundles that install_llama_prebuilt.py downloads are compiled and uploaded by hand. This replaces that with a GitHub Action that builds and publishes them automatically: CUDA (Linux x64 and arm64, Windows x64), per-gfx-target ROCm (Windows and Linux), and macOS (arm64 and x64). The macOS build is @danielhanchen's from #18 and #19, folded into this pipeline instead of running on its own.

The plain CPU and Vulkan bundles (Linux-CPU, Windows-CPU, the Vulkan and CPU-arm64 variants) still come from ggml-org's releases; the action records their checksums but doesn't rebuild them.

How it runs

Daily at 22:13 UTC (the time @danielhanchen's macOS workflow used). It builds the newest ggml-org release that has been public for at least 6 hours. That aging guard (his #19) gives a bad release time to be yanked first; this PR applies it pipeline-wide. The job compiles the 25 bundles (13 CUDA, 10 ROCm, 2 macOS) from that tag and publishes a b<N> release with llama-prebuilt-manifest.json, llama-prebuilt-sha256.json, and the source tarballs. Already-published tags are skipped.

Manually (Actions tab):

• tag: latest or a specific b####.
• only_profile: all, or one CUDA variant for a quick test.
• min_age_hours: override the 6h window.
• operating_systems / gfx_target / rocm_version: ROCm knobs.
• publish: false (default) uploads artifacts only; true publishes.

ccache skips unchanged sources. Publishing is atomic (draft, then flip).

What it produces

• CUDA: 6 Linux x64, 1 Linux arm64, 6 Windows x64 profiles (cuda{12,13}-{older,newer,portable}), Toolkit 12.8 / 13.3. Jimver has no 13.3, so cuda13 pulls the toolkit from NVIDIA's redist CDN (what ggml-org uses for its own Windows CUDA build); cuda12 uses Jimver.
• ROCm: 10 bundles (5 gfx targets, Windows and Linux), full ROCm 7 runtime bundled. Adapted from lemonade-sdk/llamacpp-rocm.
• macOS: @danielhanchen's build (ci: build and publish our own macOS llama.cpp prebuilts daily #18, ci: age upstream release (6h) and pin per-slice macOS deployment targets (arm64 14.0, x64 13.3) #19): ggml-org's macos-cpu recipe with a pinned CMAKE_OSX_DEPLOYMENT_TARGET (arm64 14.0, x64 13.3) so the binaries load on macOS < 26. This PR integrates it and fixes two load-gate bugs a real run surfaced: macOS bash 3.2 has no mapfile, and llama-quantize --help exits 1 by design (so the gate checks output, not exit code).

All bundles build from clean ggml-org source at the tag. CUDA bundles ship no CUDA runtime; they use the user's torch runtime, matched via runtime_line.

Next steps

Once merged, the schedule turns on. A matching unslothai/unsloth installer PR will follow, teaching it the new linux-arm64-cuda, windows-cuda, linux-rocm, windows-rocm, macos-arm64, and macos-x64 install kinds.

@mmathew23 @danielhanchen @shimmyshimmer

[oobabooga]

I'm doing a new test run here https://github.com/oobabooga/llama.cpp/actions/runs/26790528456

[oobabooga]

I'm doing a new test run here https://github.com/oobabooga/llama.cpp/actions/runs/26790528456

Test has been successful and from my side this looks ready to merge.

I wrote the wiring for all the new artifacts these new workflows generate here: unslothai/unsloth#5963

[danielhanchen]

Nice work!!

[danielhanchen]

Ran the prebuilts on real hardware across the whole fleet. Putting the coverage on record.

NVIDIA (CUDA), every one confirmed with real GPU offload plus tool calling:

GPU	CC	Result
T4	sm_75	pass
A100	sm_80	pass
RTX 3090	sm_86	pass
L4 / RTX 4090	sm_89	pass
H100 PCIe	sm_90	pass
B200	sm_100	pass
RTX PRO 6000 / RTX 5090	sm_120	pass
DGX Spark (arm64)	sm_121	pass

That is every compute capability from Turing through Blackwell, consumer and data center, running the same bundle.

macOS: confirmed on real M2 Pro, M3 Max and M5 Max, plus all GitHub macOS runners (14, 15, 26 on both arm64 and Intel). The arm64 Metal bundle loads on macOS 26 from the 14.0 floor as intended, with correct version when built on this branch.

Two things to land:

1. cuda12 line. The latest published build (b9518) is missing the cuda12 x64 line, so every cuda12-runtime host falls back to b9493 (which also reports version: 1). I opened ci: refuse to publish a partial CUDA set (cuda12 coverage guard) oobabooga/llama.cpp#1 against this branch: a publish-only guard that fails the run if the cuda12 and cuda13 x64 coverage classes do not match, so a partial set can never go public. After it lands, a fresh only_profile=all publish restores cuda12 and carries the correct build number (the fetch-depth: 0 fix is already here, the old assets just predate it).

2. macOS Metal 4 (optional, not urgent). The arm64 bundle is built on macos-14, so on M5 the runtime tensor-API shader compile fails and it falls back to standard Metal (ggml_metal_library_init_from_source: error compiling source, then the tensor API is disabled). It still runs on the GPU correctly. The M5 tensor path needs a macos-26 SDK build, and since Metal 4 is macOS 26 only that implies a second floor-26 arm64 slice next to the floor-14 one. Noting it for later.

AMD is unchanged (still sourced from lemonade).

[danielhanchen]

Closing out item 2 (macOS Metal 4) with M5 data.

I built two arm64 bundles on the macos-26 SDK and A/B'd them on an M5 Max (Llama-3.1-8B Q4_K_M, best of 3): one at deploy target 14.0 and one at 26.0, which is what actually turns the Metal 4 tensor API on.

build	deploy target	prompt-processing	gen
floor 14	14.0	2566.2 tok/s	99.1 tok/s
floor 26 (tensor API)	26.0	2568.7 tok/s	99.9 tok/s

Delta is +0.1%, inside run-to-run noise. The tensor API gives no measurable gain for GGUF inference at this llama.cpp version, so a second floor-26 slice is not worth the extra bundle and selection logic.

The useful finding is that the M5 error compiling source is an SDK artifact, not a floor one. Building the existing single slice on macos-26 (keeping deploy target 14.0, so minos stays 14.0) compiles cleanly on M5 and still loads on macOS 14+. One line in the resolve matrix, no second slice. Opened as oobabooga#2.

Verified on M5 Max, M3 Max, M2 Pro and the macos-14 CI runner: minos 14.0, no compile error, throughput identical to the macos-14-SDK build.

[danielhanchen]

Another real-hardware confirmation, plus a note on the build number.

RTX 3090 (sm_86) re-confirmed on a fresh stack, with real GPU offload and tool calling:

host	OS	driver / CUDA	bundle	result
RTX 3090	Ubuntu 26.04 LTS (kernel 7.0)	595.71.05 / 13.2	app-b9518-linux-x64-cuda13-portable	pass

The runtime detected libcudart.so.13 and selected the cuda13-portable line, the model loaded onto the card (CUDA0: NVIDIA GeForce RTX 3090 (24126 MiB, 23645 MiB free), ARCHS includes 860), generation was coherent, and the tool call fired. This extends the sm_86 confirmation onto a current Ubuntu 26.04 / driver 595 / CUDA 13.2 stack.

Note on version: 1: this run shows the published b9518 cuda13-portable binary reports version: 1 (7c158fb), so the build-number bug is present in the latest release itself, not only in b9493. The commit is correct and the binary works; the fetch-depth: 0 fix in this PR only applies to the next publish, so the pending cuda12 republish carries the right build number too. Confirms the republish is worth doing for both reasons.

[oobabooga]

Nice work testing this across so much real hardware, that coverage is really reassuring.

I've reviewed both PRs (build arm64 macOS on the newer SDK with the deployment target pinned for backward compatibility; guard that cuda12 and cuda13 x64 coverage match so a partial set can't publish) and agree with both. They are now merged.

The version: 1 issue is fixed in this PR by the full-history checkouts: CUDA via fetch-depth: 0, and ROCm via 860e2ba (full clone). The next publish will carry the correct build number, as you noted.

Doing a final build for testing now.

[danielhanchen]

Windows coverage, on real hardware.

Native Windows NVIDIA confirmed on an RTX 5070 Ti (Blackwell, sm_120), Windows 11, driver 591.86 / CUDA 13.1, with real GPU offload and tool calling. Both the pinned Blackwell fallback and the current build offload at full speed:

build	runtime	GPU offload	generation
llama-b9360-bin-win-cuda-13.1 (Blackwell pin)	cuda 13.1	yes	~529 tok/s
llama-b9518-bin-win-cuda-13.3	cuda 13.3	yes	~525 tok/s

Windows CUDA is pulled from upstream ggml-org (llama-bin-win-cuda plus the paired cudart zip), the same assets the installer selects, so the unslothai bundles here cover Linux. The Windows CPU fallback (llama-bin-win-cpu) was also confirmed on a GPU-less box, and a GPU-less host correctly auto-selects it.

That rounds out platform coverage: Linux (Turing through Blackwell, consumer and data-center), macOS (M2/M3/M5 plus all CI images), and now native Windows including Blackwell, all confirmed with real GPU offload. The only remaining action is the publish side already noted above: republish at latest to restore the cuda12 line and the correct build number.

[danielhanchen]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44e8a37750

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Enable shared libs with backend loading

When the CUDA jobs configure a current llama.cpp checkout, -DGGML_BACKEND_DL=ON is rejected unless BUILD_SHARED_LIBS is also enabled (ggml/src/CMakeLists.txt has a fatal check for this combination). These CUDA configure steps set backend dl but never pass -DBUILD_SHARED_LIBS=ON, so the Linux CUDA matrix will fail during CMake configure before producing any bundles; the Windows CUDA workflow has the same flag combination and needs the same fix.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Replace Unix mkdir in PowerShell step

In the Windows CUDA 13.3 jobs this step runs under the workflow's default pwsh shell, where mkdir is PowerShell's New-Item alias rather than Unix mkdir, so mkdir -p ... is parsed as a PowerShell parameter and fails before the toolkit is installed. Because the default publish matrix includes the cuda13 Windows profiles, this prevents the full release from being assembled; use New-Item -ItemType Directory -Force -Path ... or a bash/cmd shell here.

Useful? React with 👍 / 👎.

[oobabooga]

Both codex comments are wrong: BUILD_SHARED_LIBS defaults to ON in llama.cpp, so the GGML_BACKEND_DL check never fires, and pwsh's mkdir -p binds -p to -Path and works. Upstream's own windows-setup-cuda action uses it under pwsh. Confirmed by a fully green run: https://github.com/oobabooga/llama.cpp/actions/runs/27211662436

[danielhanchen]

Closing the loop on the bundle validation, consolidated in one place. All on real hardware, with generation tok/s shown as the offload proof.

Linux CUDA (x86_64): the full Turing-through-Blackwell matrix posted above stays confirmed, re-verified with the corrected detector on B200 (562 tok/s) and on a dual RTX 6000 Ada + RTX 3090 box, real offload with both cards enumerated.

Linux CUDA (arm64, Spark / GB10, sm_121): confirmed natively and under WSL2, 145 tok/s, native sm_121 SASS (no PTX-JIT).

macOS (arm64 Metal bundle, 14.0 floor):

chip	macOS	result
M1	14.5	pass (71 tok/s)
M2 Pro	26.5	pass (182 tok/s)
M5 Max	26.5	pass

M3 Max also confirmed. The 14.0 deployment floor loads from macOS 14.5 through 26.x as intended, verified on the M1 at 14.5 (oldest chip, lowest OS) and the M5 at 26.5.

Detection note: the confirm scripts now key on device enumeration plus measured generation tok/s. llama.cpp's newer auto-fit flow ("fitting params to device memory") stopped printing the old "offloaded N/N layers to GPU" / "model buffer size" lines, so a log-string check alone under-reports now; the model is on the GPU regardless and the throughput proves it.

Windows and AMD exercise the installer's upstream / lemonade selection rather than these bundles, so that coverage is recorded on unslothai/unsloth#5963.

[danielhanchen]

RTX 5090 (Blackwell, sm_120) on the cuda12 portable bundle.

The cuda12-portable bundle covers Blackwell. On an RTX 5090 the installer selected app-b9493-linux-x64-cuda12-portable.tar.gz (b9518 ships no cuda12 asset, so the last release that did), whose bundle metadata reads toolkit 12.8 and cuda archs 70;75;80;86;89;90;100;120. It enumerated CUDA0 and ran fully offloaded at 424.9 tok/s on the 1B Q4 test (CUDA 12 runtime via libcudart.so.12, libcuda from the WSL stub).

bundle	toolkit	cuda archs	GPU	result
app-b9493-linux-x64-cuda12-portable	12.8	70;75;80;86;89;90;100;120	RTX 5090 (sm_120)	424.9 tok/s, full offload

This also explains the Windows split recorded on unslothai/unsloth#5963: upstream's Windows cuda-12.4 build is toolkit 12.4 (no sm_120), so Windows Blackwell is routed to the b9360 cuda-13.1 pin, while the Linux cuda12 bundle here is toolkit 12.8 and covers sm_120 directly.

The b9518 cuda12 gap is why this run used b9493; the republish at latest restores the cuda12 line.

[danielhanchen]

Audited the workflow at the current head against the fleet validation findings. The big items are all in and enforced, not just configured: the macOS matrix builds on macos-26 with deploy_target 14.0 and assert_macho_minos.sh hard-gates minos and dyld-launches the binaries, every build child has fetch-depth: 0 with the build-number rationale spelled out, and the "Verify cuda12/cuda13 coverage before publish" step closes the partial-publish hole that produced b9518. Two narrower points from the audit:

1. The publish guard only asserts the CUDA x64 lines. macOS, ROCm, and the linux arm64 cuda13 bundles are enforced indirectly (child if-no-files-found: error plus the atomic needs), but a download-artifact anomaly inside assemble - children green, files absent from dist/ - would still pass the guard, and assemble_metadata.py only prints a WARNING when zero macOS bundles are present. Cheapest hardening: make that warning fatal, or extend the guard step with presence checks for llama-{tag}-bin-macos-{arm64,x64}.tar.gz, the arm64 cuda13 bundle, and the ROCm zips.

2. Friendly reminder that latest is still the partial b9518 (cuda13-only, pre-fix version stamp), so cuda12 hosts continue to walk back to b9493 until the republish lands. Once it does, the new release should show the cuda12 lines restored and a correct version: from the fetch-depth fix; happy to re-run the confirm scripts against it for a final sweep.

The RTX 5090 result recorded on this thread is the concrete payoff of the cuda12 bundle being toolkit 12.8: 424.9 tok/s fully offloaded from app-b9493-linux-x64-cuda12-portable on a card upstream's win-cuda-12.4 build cannot serve.

[danielhanchen]

The two items from the audit note above now have a ready fix in #20, stacked on this branch: the publish guard extended to macOS/ROCm/arm64 presence, and the manifest recording the upstream win-cuda binary digests under their own names (the fork-path Windows hash bug reported on unslothai/unsloth#5963). It is this branch verbatim plus two commits, so it can merge right after this PR does.

[danielhanchen]

Hardware confirmation for the arm64 CUDA profile - tested app-b9571-linux-arm64-cuda13-portable.tar.gz (from this pipeline's run publishing to @oobabooga's fork) on an N1X "RTX Spark" (Windows-on-ARM -> WSL2 Ubuntu 24.04, Blackwell sm_121, driver CUDA 13.1, glibc 2.39):

• Loads and runs out of the box: CUDA0: JMJWOA-Generic-GPU (18725 MiB), CUDA : ARCHS = 900,1000,1200,1210 - sm_121 native, no PTX JIT - and BLACKWELL_NATIVE_FP4 = 1.
• Served Qwen3-0.6B (UD-Q4_K_XL) at 215.9 tok/s, GPU 76% peak / 12.2 GB - on par with a locally compiled CUDA 13.3 build that takes 1.5+ hours (and risks thermal shutdowns) on this laptop class.
• Manifest metadata is right for selection: install_kind=linux-arm64-cuda, supported_sms=[90, 100, 103, 120, 121], toolkit 13.3 - unsloth#5963's manifest path picked it correctly on this host.

Once this publishes to unslothai/llama.cpp releases, the whole DGX Spark / N1X class skips the long source build entirely.