Skip to content

fix(ci): shard single-arch backend matrix under GitHub's 256-job limit#10703

Merged
mudler merged 1 commit into
masterfrom
fix/singlearch-matrix-256-limit
Jul 6, 2026
Merged

fix(ci): shard single-arch backend matrix under GitHub's 256-job limit#10703
mudler merged 1 commit into
masterfrom
fix/singlearch-matrix-256-limit

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

The v4.6.1 backend build (run 28786533892) concluded failure while every job in the run was green — and the backend-jobs-singlearch / backend-merge-jobs-singlearch groups sat forever at "Waiting for pending jobs".

Root cause: GitHub Actions refuses to instantiate a matrix that would generate more than 256 jobs, and it does so silently — the caller job hangs at "Waiting for pending jobs" and the run is marked failure with no failing job to point at. The single-arch build matrix had grown to 268 entries (> 256), so those jobs were never created. That's why they don't appear in the jobs API or check-runs, yet the release build "failed".

generate-matrix          1
backend-jobs-multiarch   111   ✅
backend-jobs-darwin      106   ✅
backend-merge-multiarch  57    ✅
backend-jobs-singlearch  0     ❌ never instantiated (268 > 256)
backend-merge-singlearch 0     ❌

Fix

The single-arch list is the one that grows unbounded as backends are added, so shard it across a fixed number of matrix jobs:

  • SINGLEARCH_SHARDS = 4 in changed-backends.js — the 268 entries split into 4 balanced shards of ~67 (headroom to ~1020 backends before the next bump).
  • Each merge shard needs: only its matching build shard, preserving the "merge waits only on its own build" property (PR ci: split backend-jobs into single-arch and multi-arch matrices #9746) that keeps slow CUDA/ROCm single-arch builds from gating multi-arch manifest assembly.
  • The script throws loudly if any shard ever reaches the 256 limit, naming exactly what to bump — instead of letting GitHub drop the overflow silently again.
  • backend.yml and backend_pr.yml gain the four build + four merge shard jobs. Multi-arch and darwin groups are untouched.

Validation

  • Ran changed-backends.js against the full matrix: 268 → shards [67, 67, 67, 67], total preserved, no aggregate output leaked, multi-arch (111) / darwin (53) unchanged.
  • Unit-tested chunkEqually edge cases (0–7 entries, 268, 1020, 1024): balanced, sums preserved, guard trips exactly at 1024 (256/shard).
  • actionlint passes clean on both workflows; both parse and each merge shard correctly needs: its build shard; fail-fast semantics preserved (false in backend.yml, true in backend_pr.yml).

🤖 Generated with Claude Code

GitHub Actions refuses to instantiate a matrix that would generate more
than 256 jobs. It does so silently: the job hangs forever at "Waiting for
pending jobs" and the whole run is marked `failure` while every other job
stays green. This is exactly what happened on the v4.6.1 tag build
(run 28786533892): the single-arch build matrix had grown to 268 entries,
so `backend-jobs-singlearch` (and its downstream merge) never produced a
single job, and the release build "failed" with no failing job to point at.

The single-arch list is the one that grows unbounded as backends are added,
so shard it across a fixed number of matrix jobs (SINGLEARCH_SHARDS=4,
~67 entries each today, headroom to ~1020 backends). Each merge shard
`needs:` only its matching build shard, preserving the "merge waits only on
its own build" property that keeps slow CUDA/ROCm builds from gating
multi-arch manifest assembly.

changed-backends.js now emits per-shard matrix/has-* outputs and throws
loudly if a shard ever reaches the 256 limit (telling the maintainer to
bump SINGLEARCH_SHARDS and add matching job blocks) instead of letting
GitHub drop the overflow silently. backend.yml and backend_pr.yml define
the four build + four merge shard jobs; multi-arch and darwin groups are
untouched.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
@mudler mudler merged commit 29db4ab into master Jul 6, 2026
66 checks passed
@mudler mudler deleted the fix/singlearch-matrix-256-limit branch July 6, 2026 19:38
mudler added a commit that referenced this pull request Jul 7, 2026
…10725)

The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the
prebuilt image drifted onto whatever aarch64 wheel was latest at build
time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory),
0.23 crashes deterministically during cold model loads with an empty
"Engine core initialization failed" set and pins GPU memory until a host
reboot.

vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device
memory under pressure on UMA GPUs during weight loading"), which the
reporter verified fixes the crash on GB10. Pin the L4T requirements to
0.24.0 to match the already-pinned cublas13 build
(requirements-cublas13-after.txt) and keep the image deterministic.

Editing this file also re-triggers the single-arch L4T image build via
the path filter, republishing the gallery image with 0.24.0 (the
single-arch matrix builds again after #10703).

Closes #10722


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
treilhes pushed a commit to treilhes/LocalAI that referenced this pull request Jul 7, 2026
…udler#10725)

The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the
prebuilt image drifted onto whatever aarch64 wheel was latest at build
time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory),
0.23 crashes deterministically during cold model loads with an empty
"Engine core initialization failed" set and pins GPU memory until a host
reboot.

vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device
memory under pressure on UMA GPUs during weight loading"), which the
reporter verified fixes the crash on GB10. Pin the L4T requirements to
0.24.0 to match the already-pinned cublas13 build
(requirements-cublas13-after.txt) and keep the image deterministic.

Editing this file also re-triggers the single-arch L4T image build via
the path filter, republishing the gallery image with 0.24.0 (the
single-arch matrix builds again after mudler#10703).

Closes mudler#10722

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: treil <p.treilhes@free.fr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants