diff --git a/docs/speculative.md b/docs/speculative.md index 8f91256c4a4d..4100b92f8f18 100644 --- a/docs/speculative.md +++ b/docs/speculative.md @@ -52,6 +52,32 @@ Supported EAGLE-3 draft models include: For the full and up-to-date list of supported models, see #18039. +### DFlash (`draft-dflash`) + +DFlash produces an entire block of draft tokens in a single forward pass (block diffusion) and +injects the target model's hidden states into the draft model's attention, instead of drafting one +token at a time. This keeps the draft model small while making drafting GPU-friendly. Unlike EAGLE-3 +(a single-layer autoregressive draft), the DFlash draft uses several transformer layers but emits a +whole block per draft step. + +The draft is a small block-diffusion model trained for a specific target (for example +`z-lab/Qwen3-4B-DFlash` for `Qwen/Qwen3-4B`). Convert it with `--target-model-dir` so it inherits the +target's tokenizer and token embeddings: + +```bash +python convert_hf_to_gguf.py z-lab/Qwen3-4B-DFlash \ + --target-model-dir Qwen/Qwen3-4B --outtype bf16 --outfile Qwen3-4B-DFlash.gguf + +llama-server -m Qwen3-4B.gguf -md Qwen3-4B-DFlash.gguf \ + --spec-type draft-dflash --spec-draft-n-max 15 -fa on --jinja +``` + +`--spec-draft-n-max` is clamped to the draft model's trained block size. + +See: + +- #22105 + ### n-gram Cache (`ngram-cache`) An n-gram is a sequence of n tokens. The n-gram cache implementation maintains statistics about short n-gram sequences. @@ -147,7 +173,7 @@ If a draft model is combined with a draftless decoding the draftless decoding ha ### General Speculative Parameters ``` ---spec-type [none|draft-simple|draft-eagle3|draft-mtp|ngram-cache|ngram-simple|ngram-map-k|ngram-map-k4v|ngram-mod] +--spec-type [none|draft-simple|draft-eagle3|draft-dflash|draft-mtp|ngram-cache|ngram-simple|ngram-map-k|ngram-map-k4v|ngram-mod] comma-separated list of types of speculative decoding to use (default: none) (env: LLAMA_ARG_SPEC_TYPE) @@ -287,6 +313,7 @@ Specifies a comma-separated list of speculative decoding types to use. | `none` | No speculative decoding (default) | | `draft-simple` | Use a simple draft model for speculation | | `draft-eagle3` | Use an EAGLE-3 draft model that reads the target's hidden states | +| `draft-dflash` | Use a DFlash block-diffusion draft model that emits a block per step | | `draft-mtp` | Use Multi Token Prediction (MTP) heads from the main model | | `ngram-cache` | Use n-gram cache lookup | | `ngram-simple` | Use simple n-gram pattern matching |