Sycl --split-mode tensor#24152
Conversation
* SYCL: tensor parallelism (--split-mode tensor) for dual-GPU
Adds the comm_init/comm_free/comm_allreduce_tensor trio that the
meta-backend queries via get_proc_address to enable backend-specific
all-reduce, mirroring the pattern used by ggml-cuda.cu.
For N=2 (the common dual-GPU case) implements a degenerate ring
all-reduce with two size-branched paths:
* Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel
chained via depends_on(memcpy_event). 4 SYCL submissions/call.
* Large (nelem >= 32768): BF16-compressed. Each device compresses
FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's
inbox (HALF the PCIe bytes), then decompresses + adds into the
local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved
-- wins for any tensor where PCIe dominates kernel time.
Threshold and BF16 path pattern mirror the CUDA NCCL allreduce.
Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes
(matches both path layouts: FP32 nelem floats; BF16 outbox+inbox =
2 * nelem uint16_t each). Single alloc+free per device keeps the
SYCL pool's strict-LIFO invariant trivial.
Initial impl handles N=2 FP32 contiguous tensors. Other cases return
false, causing the meta-backend to use its generic butterfly fallback.
Per-call sync is intentionally omitted. SYCL in-order queue semantics
ensure that the meta-backend's next compute on the same per-device
queue waits for our final ADD, and the next allreduce's first op on
the same persistent buffer waits via the same queue. Only comm_free
does an explicit final wait.
OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process
in communicator_impl.hpp:47 (condition devices.size() == 1), which is
incompatible with llama.cpp's single-process multi-GPU model.
Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 +
DPC++ nightly):
Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16:
pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%)
tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%)
Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE):
pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%)
tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%)
Qwen3-4B Q4_K_M:
pp64 = 984.51 t/s, tg16 = 49.29 t/s
Llama-3.3-70B in SYCL TP now comfortably beats production layer mode
on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on
both — the BF16 path is what unlocks the many-medium-allreduces
prefill pattern.
Build/CMake: no changes. No new dependencies. ~210 lines added across
ggml-sycl.h and ggml-sycl.cpp.
* Fix comments
|
Hi @Spruill-1, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
@Spruill-1 B580x2, Ubuntu 26.04, oneAPI 2025.3.3 |
|
@Spruill-1
How to avoid it for my case? Thank you! |
|
For smaller LLM (5.6GB), only add Could you check this issue? |
|
After add |
|
For bigger LLM: Qwen3.5-27B-heretic-v2-Q6_K.gguf (22GB), the performance ratio is about 1. I think it should be more than 1 when the LLM size > 22GB. But I only have B580x2 (12x2GB), so I can't test bigger LLM. The only issue is missing |
arthw
left a comment
There was a problem hiding this comment.
When missing -ctk f16 -ctv f16 in cmd, it will lead to error of create context in the case -sm tensor.
|
As a test with heterogeneous GPU setups (1xB70, 1xB580) For a model that will fit when split in half evenly between the two: --bench -m /models/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -p 512,2048 -n 128,512 --split-mode tensor -ctk f16 -ctv f16
--bench -m /models/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -p 512,2048 -n 128,512 --split-mode tensor --tensor-split 3,1 -ctk f16 -ctv f16
--bench -m /models/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -p 512,2048 -n 128,512 --split-mode layer -ctk f16 -ctv f16
But what about a model that has a size greater than twice the VRAM of the smaller card? --bench -m /models/gemma-4-31B-it-UD-Q6_K_XL.gguf -p 512,2048 -n 128,512 --split-mode tensor -ctk f16 -ctv f16 --bench -m /models/gemma-4-31B-it-UD-Q6_K_XL.gguf -p 512,2048 -n 128,512 --split-mode tensor --tensor-split 3,1 -ctk f16 -ctv f16
--bench -m /models/gemma-4-31B-it-UD-Q6_K_XL.gguf -p 512,2048 -n 128,512 --split-mode layer -ctk f16 -ctv f16
|
There was a problem hiding this comment.
@Spruill-1
This PR shows good performance increasing on multiple GPUs.
Here are the comments：
- Why missing the parameters
-ctk f16 -ctv f16lead to the error in create context? - If the parameters
-ctk f16 -ctv f16are mandatory for this feature, please update it in SYCL.md or help in parameter-sm tensor. - Please fix to avoid the error in create context when missing
-ctk f16 -ctv f16. It's allowed to disable this feature in this case, but avoid the error.
Thank you!
|
I tested with a medium Dense (Qwen3.6-27B-UD-Q8_K_XL) and a medium MoE (Qwen3.6-35B-A3B-Q8_0) model on release System InfoDense Qwen3.6-27B-UD-Q8_K_XL
MoE Qwen3.6-35B-A3B-Q8_0
|
|
@arthw @NeoZhangJianyu can we have Intel folks review these PRs? @ggerganov is it possible to make a label in the repo that can be used for intel b70 gpus? This way all the B70 owners can band together and test things out. It's good for the ecosystem if Intel has a competitive software stack. |
|
@mjsabby For this PR, the only issue to be fixed is: missing the parameters There is the label "SYCL" to be added to every issue and PR of SYCL backend. It's a little complex to add the hardware/GPU mode as label to issue/PR. Thank you! |
|
@Spruill-1 the community thanks you for your contribution and would love this getting checked in. @ggerganov as well |
|
Hey folks sorry for the delay in getting back here... But, @arthw I think that the error is not actually '-ctk/-ctv f16', and instead is '-fa 0'. Tensor mode requires flash attention and hits this context error without. Running the bench with -fa 0,1 will run the fa=0 case first - fail with the context error and just exit. Running here myself with '-fa 1' and without the explicit f16 runs fine, f16 is the default KV format anyway. I'm not sure that a code change is warranted here - the guard for ensuring FA for tensor mode in production use is higher up (src/llama-context.cpp) and if my understanding of the architecture here is correct - is shared with non-SYCL backends. I'll add a note to the sycl.md file that indicates that tensor requires flash attention. |
|
@arthw I pushed an additional commit to correct some multi-gpu handling with memcpy's. This had been on a separate fork on a different machine of mine and I missed it when getting this change together but was included in my original benchmarks here. (I've also done a careful walkthrough my changes to make sure I didn't miss anything else between these two systems) A/B on dual Arc Pro B70 (-sm tensor -fa 1) (pre-warmed JIT cache):
built the PR branch as-is vs with the cross-context copy routed through dev2dev_memcpy. Note that the decrease in t/s here is largely due to my testing machine's slow device->device link bottleneck (limited to PCIe 3.0 x8 per card right now). I'd welcome benching with a newer CPU/chipset combo if anyone else has dual B70s. |
|
@Spruill-1 I think the communication depended on the PCIE/driver and GPU hardware too. Hope more community members can help verify this PR! Thank you! |
|
@Spruill-1 I'm still getting garbage after applying your change. Can you share your full command and what other modifications beyond Kernel 7.1 you have going on? |
|
I have a Two B70 setup on Intel host, moving to AMD host this coming weekend. |
|
@Spruill-1 Thank you! |
|
@arthw I'll fix the CI issue after work today - problem is my use of a 5-parameter d2d command where the upstream has moved to a 7-parameter for safety - should be an easy fix. @mjsabby When I get the change in this evening it should resolve your problem without needing different commands - one of my earlier commands regressed it until the dev2dev memcpy fixed the regression, but that is causing this CI failure due to a silent merge conflict with upstream. Once it is in, I would be very interested in your results with dual B70s and a faster PCIe link (my old threadripper 2970wx is limiting the per-card budget to Gen 3.0 x8) |
|
Sorry for the noise y'all - I had a spare trailing empty space on a comment line and it caused the tooling to reject it. There was also a typo to fix at the same time. |
|
Thanks @Spruill-1 works for me, thanks. |
|
For whatever reason I can't seem to build this. |
How are you building? What is the command you are running and what is your repo state? |
Fresh pull of llama.cpp, I follow SYCL build instructions, initialise environment, I try with both FP16 and FP32, build stops at the CPU section.
|
|
@HumerousGorgon If you just sync to my PR branch and try to build you'll see build issues because my branch is a bit behind the mainstream (something like ~230 commits right now). You'll need to use a merge commit of my branch played on top of main to have it fully reconciled. |
* Sycl tp stage1 (#1) * SYCL: tensor parallelism (--split-mode tensor) for dual-GPU Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend queries via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu. For N=2 (the common dual-GPU case) implements a degenerate ring all-reduce with two size-branched paths: * Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel chained via depends_on(memcpy_event). 4 SYCL submissions/call. * Large (nelem >= 32768): BF16-compressed. Each device compresses FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's inbox (HALF the PCIe bytes), then decompresses + adds into the local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved -- wins for any tensor where PCIe dominates kernel time. Threshold and BF16 path pattern mirror the CUDA NCCL allreduce. Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes (matches both path layouts: FP32 nelem floats; BF16 outbox+inbox = 2 * nelem uint16_t each). Single alloc+free per device keeps the SYCL pool's strict-LIFO invariant trivial. Initial impl handles N=2 FP32 contiguous tensors. Other cases return false, causing the meta-backend to use its generic butterfly fallback. Per-call sync is intentionally omitted. SYCL in-order queue semantics ensure that the meta-backend's next compute on the same per-device queue waits for our final ADD, and the next allreduce's first op on the same persistent buffer waits via the same queue. Only comm_free does an explicit final wait. OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process in communicator_impl.hpp:47 (condition devices.size() == 1), which is incompatible with llama.cpp's single-process multi-GPU model. Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 + DPC++ nightly): Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16: pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%) tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%) Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE): pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%) tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%) Qwen3-4B Q4_K_M: pp64 = 984.51 t/s, tg16 = 49.29 t/s Llama-3.3-70B in SYCL TP now comfortably beats production layer mode on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on both — the BF16 path is what unlocks the many-medium-allreduces prefill pattern. Build/CMake: no changes. No new dependencies. ~210 lines added across ggml-sycl.h and ggml-sycl.cpp. * Fix comments * documentation update to address PR feedback * Bring over my device-to-device memcpy chagnes * move the dev2dev_memcpy calls to the upstream 7-parameter variety * Fix a typo and remove a trailing whitespace
|
@arthw @Spruill-1 looks like tensor mode is broken when VMM=1 for basically non trivial stuff. AI came up with a following fix, master...mjsabby:llama.cpp:sm-tensor-vmm-fix My current workarounds are host staging or disabling VMM if I want tensor mode. |
* Sycl tp stage1 (#1) * SYCL: tensor parallelism (--split-mode tensor) for dual-GPU Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend queries via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu. For N=2 (the common dual-GPU case) implements a degenerate ring all-reduce with two size-branched paths: * Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel chained via depends_on(memcpy_event). 4 SYCL submissions/call. * Large (nelem >= 32768): BF16-compressed. Each device compresses FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's inbox (HALF the PCIe bytes), then decompresses + adds into the local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved -- wins for any tensor where PCIe dominates kernel time. Threshold and BF16 path pattern mirror the CUDA NCCL allreduce. Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes (matches both path layouts: FP32 nelem floats; BF16 outbox+inbox = 2 * nelem uint16_t each). Single alloc+free per device keeps the SYCL pool's strict-LIFO invariant trivial. Initial impl handles N=2 FP32 contiguous tensors. Other cases return false, causing the meta-backend to use its generic butterfly fallback. Per-call sync is intentionally omitted. SYCL in-order queue semantics ensure that the meta-backend's next compute on the same per-device queue waits for our final ADD, and the next allreduce's first op on the same persistent buffer waits via the same queue. Only comm_free does an explicit final wait. OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process in communicator_impl.hpp:47 (condition devices.size() == 1), which is incompatible with llama.cpp's single-process multi-GPU model. Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 + DPC++ nightly): Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16: pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%) tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%) Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE): pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%) tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%) Qwen3-4B Q4_K_M: pp64 = 984.51 t/s, tg16 = 49.29 t/s Llama-3.3-70B in SYCL TP now comfortably beats production layer mode on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on both — the BF16 path is what unlocks the many-medium-allreduces prefill pattern. Build/CMake: no changes. No new dependencies. ~210 lines added across ggml-sycl.h and ggml-sycl.cpp. * Fix comments * documentation update to address PR feedback * Bring over my device-to-device memcpy chagnes * move the dev2dev_memcpy calls to the upstream 7-parameter variety * Fix a typo and remove a trailing whitespace
Overview
Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend wants via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu.
For N=2 (dual-GPUs) implements a degenerate ring allreduce with two size-branched paths to mirror CUDA NCCL allreduce:
Small (N< 32k): FP32 direct memcpy + per-device ADD kernel chained via depends_on(memcpy_event). 4 submissions/call.
Large (N>= 32k): BF16-compressed. Each device compresses FP32 -> BF16 locally, cross-device memcpys to the peer, then decompresses + adds into the local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved. Wins overall for larger dense models.
Storage: A single persistent uint8_t buffer per device, 4 * N bytes (matches both path layouts: FP32 N floats; BF16 outbox+inbox = 2 * N uint16_t each). Single alloc+free per device keeps the SYCL pool's strict-LIFO invariant.
Initial impl handles N=2 FP32 contiguous tensors. Other cases return false, causing the meta-backend to use its generic butterfly fallback. This should be easy to extend to N>2, but I don't have the hardware.
Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 + DPC++ nightly) with a Threadripper 2970WX, 128GB system running Ubuntu Server 26 LTS.
Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16:
pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%)
tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%)
Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE):
pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%)
tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%)
Qwen3-4B Q4_K_M:
pp64 = 984.51 t/s, tg16 = 49.29 t/s
Requirements
Build/CMake: no changes. No new dependencies. ~200 lines added across ggml-sycl.h and ggml-sycl.cpp.