spec: add DSpark speculative decoding - #25173
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
|
Hi @CISC @ggerganov , this adds DSpark speculative decoding on top of the merged DFlash drafter (#22105). It's a small change — a new dspark draft arch and draft-dspark spec type that reuse DFlash's graph, feature extraction, KV-cache injection and verify path unchanged; the only new logic is the semi-autoregressive Markov head in draft(). Greedy stays lossless. I benchmarked it against the merged DFlash using DeepSeek's released Qwen3 DSpark drafts. On Qwen3-8B at bf16 / Q8_0 / Q4_K_M, DSpark beats DFlash on every domain (e.g. GSM8K bf16 4.06× vs 3.12×; full per-domain tables in the PR description). I believe it's ready for review and I'm happy to walk through any part of it. |
|
Can you run SpeedBench to do the full comparison between DFlash and DSpark with the same |
|
@ruixiang63 I've run the SpeedBench test set as you suggested, and updated the results in the PR description. DSpark does outperform DFlash across the board. |
aae2941 to
d8b38f2
Compare
|
could you give some examples how to use? |
Good point — I've updated the PR description with a more detailed, copy-pasteable end-to-end example (download → convert → build → run → curl). Let me know if anything's unclear. |
|
DSV4 support was merged in #24162, ideally this PR should cover that model as well and try to replicate a similar speedup |
Thanks! DeepSeek hasn't open-sourced the DSpark weights for DeepSeek-V4 though — only the Qwen3 and Gemma4 drafts are released. So this PR covers Qwen3 for now, and I'll add Gemma4 as a small follow-up. |
|
I think they're a part of the spec decoding module https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark i.e not distributed separately |
Sorry, and thanks for the heads-up. For this PR I'd like to keep the scope a bit narrower for now - land the Qwen3 DSpark path first and get it solid, then add Gemma4 and DSV4 as follow-ups. Does that sound ok? |
|
Okay, will try to review. From a cursory look it does not look like adding |
I agree that |
I found that your command is incorrect. If you want to put the command on a single line, please remove the . Alternatively, you can copy the command from my description and keep the line breaks as they are. |
|
I would have posted results from speed-bench, but since they were going to take 12+ hours, i decided not to |
|
/bot review |
Automated code reviewReview of PR #25173 — adding the I traced the new graph code in Blocking(point 1) Markov head assumes all drafting sequences use an identical block size, and silently corrupts logits when they don't. In Will slow the review(point 2) (point 3) (point 4) Verify the conversion's (point 5) No tests or acceptance/perf numbers. This adds a new speculative draft type, a new conversion path, and new GGUF tensors, with no test, no sample GGUF, and no acceptance-rate / latency numbers. The skill guidance for new drafting features expects at least a converted sample plus measured acceptance on a supported target (e.g. Nits(point 6) (point 7) The no-op returns in (point 8) (point 9) The clamp warning text ( The overall approach — reusing the DFLASH architecture, gating the Markov head on tensor presence, and routing the confidence output through the existing This review was generated automatically by pi coding agent using |
|
@wjinxu can you address the first point in the auto review? (feel free to tell if it's valid or not - other points are optional) |
…tion The DSpark markov head views the draft batch as a uniform [n_seqs x block] grid, but the per-seq dp.n_max clamp could produce blocks of different sizes, silently corrupting the strided views and the resulting logits. Drop the clamp and always draft the full n_max block for every sequence: dp.n_max is already enforced by the central truncation in common_speculative_draft(), the same way eagle3 handles it. Co-authored-by: Zaire404 <3147879462@qq.com>
|
Same approach as eagle3: draft a full n_max block for every sequence and let the central truncation in |
|
I only did a quick code review, not a deeper logic review, but all seems good. Just running another auto review to see if there is any other blocking issues /bot review |
Automated code reviewReview of PR #25173 (DSpark speculative decoding). I read Correctness (should fix before merge)(point 1) const float * conf = params.conf_min > 0.0f ? llama_get_embeddings_nextn(ctx_dft) : nullptr;
...
if (conf && conf[(size_t) idx * n_embd_dec] < params.conf_min) { break; }But the graph only writes (point 2) Silent fallback in Will slow the review(point 3) (point 4) (point 5) Description vs. code disagree on whether confidence is "used at inference". The description says "The confidence head is converted/loaded but not used at inference in this PR (phase 1)", yet this PR adds (point 6) Shared DFlash Nits(point 7) Description overstates the architecture. It claims "a new draft architecture dspark ( (point 8) Minor wording: the anchor-view comment "anchor (committed last) token of every block: token 0" conflates "committed last" (the accepted target token) with "token 0 of the block input". The anchor is This review was generated automatically by pi coding agent using |
|
I think point 1 and 2 worth having a look, but feel free to tell me if it's valid / not valid otherwise. Other points are optional and can be follow-up if needed |
…e conf head With the draft batch always submitting equal-size n_max blocks, a non-divisible token count can only mean the batch was split across ubatches or a caller broke the layout - fail loudly instead of silently dropping the markov bias. The block_drafts > block_size early return stays: worst-case graph reserve passes legitimately build with n_seq_tokens > block_size. Also make conf_proj required when the markov head is present: the confidence head is part of the DSpark checkpoint format, and a missing head would otherwise leave --spec-draft-conf-min silently reading stale embeddings instead of confidences. Co-authored-by: Zaire404 <3147879462@qq.com>
The confidence head is part of the DSpark checkpoint format (every DSpark model ships conf_proj alongside the markov head), so conf_proj is now a required tensor whenever the markov head is present — a checkpoint lacking it fails at load time instead of silently misbehaving at runtime. The one remaining way to reach it would be pointing --spec-type draft-dspark at a plain DFlash draft and setting --spec-draft-conf-min — a stacked misconfiguration we're treating as user error rather than a supported path.
Partially done. The divisibility check is now a GGML_ASSERT: with the batch always submitting equal-size blocks (see the earlier fix), a non-divisible token count can only mean the batch was split across ubatches or a caller broke the layout, so it should fail loudly — agreed. The block_drafts > block_size early return has to stay, we tried asserting it and the server aborts at startup, before serving a single request. |
| float p_split = 0.1f; // speculative decoding split probability | ||
| float p_min = 0.0f; // minimum speculative decoding probability (greedy) | ||
|
|
||
| float conf_min = 0.0f; // DSpark: min predicted acceptance from the confidence head (0 = disabled) |
There was a problem hiding this comment.
Can we reuse the p_min and avoid introducing the new conf_min?
p_min and conf_min express the same thing - the minimum predicted survival probability for a drafted position - differing only in how the estimate is obtained: token probability for regular drafters, the trained confidence head for DSpark. The DSpark readback never used p_min, so reuse it for the confidence threshold and drop the separate --spec-draft-conf-min flag. Both defaulted to 0 (disabled), so behavior is unchanged. Co-authored-by: Zaire404 <3147879462@qq.com>
This comment was marked as off-topic.
This comment was marked as off-topic.
Comments like these are spam. Pull requests are for discussing the code itself to move it forward. If you don't have a meaningful contribution or constructive criticism relating to the code, take it elsewhere. They don't increase the priority or make things go faster. All they do is waste the time of every subscriber to the thread. The only positive analogue to "is it done yet?" is "what tasks remain before this is ready to merge?", because the answer is useful to contributors. When progress has stalled for a long time, you may politely ask if the work will continue. This is clearly not the case here. This PR is active, and as such the answer is implicit. None of us are entitled to the time and effort of others. Stop it. |
| conf = ggml_cont(ctx0, ggml_permute(ctx0, conf, 0, 2, 1, 3)); | ||
| conf = ggml_reshape_2d(ctx0, conf, 1, n_tok); | ||
|
|
||
| conf = ggml_repeat(ctx0, conf, res->t_embd); |
There was a problem hiding this comment.
Is this repeat a placeholder? If yes, add a TODO that later the correct confidences will be computed.
There was a problem hiding this comment.
Actually, this looks like a workaround to reuse the llama_get_embeddings_nextn for extracting the confidences.
That's fine for now - just add a // note: ... to avoid confusion in the future.
Requested in review: the ggml_repeat only adapts the [1, n_tok] confidences to the n_embd-wide embd_nextn transport so that llama_get_embeddings_nextn can be reused - not a placeholder. Co-authored-by: Zaire404 <3147879462@qq.com>
[no ci]
| } | ||
| conf = ggml_sigmoid(ctx0, conf); | ||
|
|
||
| cat_conf = cat_conf ? ggml_concat(ctx0, cat_conf, conf, 1) : conf; |
There was a problem hiding this comment.
These concats might be possible to avoid and optimize via ggml_set_inplace, similar to:
llama.cpp/src/models/delta-net-base.cpp
Lines 261 to 263 in 6ba5ef2
Can be tried in a follow-up PR
|
I haven't tested the code. Unless @ruixiang63 has additional comments, we can merge I think? |
Yes, looks good to me! Let's merge. @ggerganov |
|
Added DSpark here: https://huggingface.co/ggml-org/Qwen3-8B-GGUF Let me know if someone gives it a try. |
This PR adds DSpark speculative decoding, layered on the merged DFlash drafter. DSpark (DeepSeek + PKU, 2026 — "Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation", the DeepSpec repo) is DFlash plus a small semi-autoregressive Markov head: where DFlash takes an independent argmax at each block position (every position marginalizes over all possible predecessors, so acceptance decays along the block), DSpark adds a low-rank, previous-token-conditioned logit bias and samples the block left-to-right, so each draft conditions on the one actually sampled before it. This lifts accepted length at near-zero extra draft cost.
DSpark reuses the entire DFlash machinery unchanged — the encoder/decoder graph, target-layer feature extraction (
llama_set_embeddings_layer_inp/_nextn), KV-cache injection, and the verify/accept path. The only additions are:dspark(llama_model_dspark : llama_model_dflash) that reuses the DFlash graph and additionally loads the Markov head (markov_w1,markov_w2) and an optional confidence head; it shares the target's token-embeddings / lm_head (same as DFlash);draft-dspark(common_speculative_impl_draft_dspark : common_speculative_impl_draft_dflash) that reusesprocess()(extraction + injection) and overrides onlydraft(): the block is anchor-first (position 0 already predicts the first draft token) and sampled with the Markov biasbias(prev) = markov_w2 · markov_w1[prev], computed on-device (llama_dspark_markov_bias);Qwen3DSparkModelconverter.Greedy decoding is lossless: the Markov bias only changes which tokens are proposed; every draft is still verified against the target, so the output is identical to non-speculative greedy.
The confidence head is converted/loaded but not used at inference in this PR (phase 1); the draft-quality win from the Markov head is self-contained and is what the numbers below measure.
How to run
Complete example from scratch (Qwen3-8B). Drafts for other sizes are on the same org:
deepseek-ai/dspark_qwen3_{4b,8b,14b}_block7.1. Get the models — target + its DSpark draft:
2. Convert to GGUF — the draft ships no tokenizer and reuses the target's, so pass
--target-model-dir:python convert_hf_to_gguf.py Qwen3-8B --outtype bf16 --outfile Qwen3-8B.gguf python convert_hf_to_gguf.py dspark_qwen3_8b --outtype bf16 \ --target-model-dir Qwen3-8B --outfile Qwen3-8B-DSpark.ggufYou may quantize the target (e.g.
llama-quantize Qwen3-8B.gguf Qwen3-8B-Q4_K_M.gguf Q4_K_M); keep the draft bf16 — it's tiny, and acceptance is unaffected by target quant.3. Build with CUDA:
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j4. Run — the only DSpark-specific flags are
-md <draft>and--spec-type draft-dspark(
--spec-draft-n-max= draft tokens per step; the released checkpoints use block size 7):./build/bin/llama-server -m Qwen3-8B.gguf -md Qwen3-8B-DSpark.gguf \ --spec-type draft-dspark --spec-draft-n-max 7 \ --temp 0 --top-k 1 -np 1 -c 4096 -ngl 99 -fa on --jinja5. Send a request (the server logs
draft acceptance = ...per request):llama-cliworks the same way (-m ... -md ... --spec-type draft-dspark). Note:draft-dsparkneeds the target's hidden states (KV-cache injection), so usellama-server/llama-cli— thespeculative-simpleexample does not drive that path.Performance
SpeedBench (llama.cpp's own
tools/server/bench/speed-bench)Qwen3-8B (bf16), matched
--spec-draft-n-max 7,qualitativesplit (11 categories), greedy. Baseline is the same server with no draft model. DSpark reaches 1.88× overall decode speedup vs baseline (DFlash is 1.55×), and beats the merged DFlash on every one of the 11 categories (overall 1.21×).DSpark vs baseline:
DSpark vs the merged DFlash (same
--spec-draft-n-max 7):Hardware: RTX 4090. Target Qwen/Qwen3-8B (bf16), draft deepseek-ai/dspark_qwen3_8b_block7 (bf16). Greedy (
--temp 0 --top-k 1), no-thinking,--spec-draft-n-max 7. Baseline = same llama-server with no draft model. DFlash is the merged drafter (z-lab/Qwen3-8B-DFlash, b16), run at the same matched draft size for an apples-to-apples comparison. Per-domain aggregate over the listed prompt counts.Losslessness
Greedy decoding is lossless by construction (the draft is verified against the target). Output is coherent and matches non-speculative greedy.
Qwen3-4B, target bf16
DSpark vs baseline (DFlash was not benchmarked at 4B — no nested-schema 4B DFlash draft available):
Qwen3-8B, target bf16
Qwen3-8B, target Q8_0
Qwen3-8B, target Q4_K_M
DSpark beats the merged DFlash on every domain (higher accept rate and higher throughput), for a ~1.16× geomean speedup over DFlash. The gains are largest on reasoning (GSM8K +25pp accept, 1.30× over DFlash) and open chat (MT-Bench, 1.29×); on code (HumanEval) the two are close as both already accept ~80%.
Confidence Evaluation
Qwen3-8B Q4_K_M target, SPEED-Bench qualitative, 132 completed samples, OSL 512.
Percentage changes on the unified-KV rows are relative to the same-concurrency unified-KV
conf_min=0.0baseline.Confidence pruning has no benefit at concurrency 1, begins to help at concurrency 8.The intended operating environment is high-concurrency serving with packed/unified KV batching.
Do not enable confidence pruning with non-unified KV at high concurrency. Ragged verification causing CUDA Graph reuse to collapse.
Future work
Requirements