feat(speculative): support sequence packing in the DeepSeek MLA EAGLE-3 draft#3002
Merged
HuiyingLi merged 5 commits intoJul 11, 2026
Merged
Conversation
…-3 draft The DeepSeek MLA EAGLE-3 draft (DeepseekV3Eagle3DraftModel) shares the EAGLE-3 trainer (Eagle3TrainerModule), which already threads position_ids / seq_lens / doc_remaining for sequence packing, but the draft's forward rejected seq_lens with NotImplementedError. Wire packing through the eager MLA path exactly like the dense Llama draft: when seq_lens is present, build the [B, 1, T, T] block-causal additive mask (build_block_causal_additive_mask) instead of the plain causal + padding mask, so Block-1 attention is document-level block-causal. The TTT diagonal block is position-wise and stays document-safe; cross-document supervision is masked by the trainer's doc_remaining gate (unchanged). The MLA attention is eager-only, so packing reuses the eager mask path (no FA2 varlen kernel). Tests: replace the "packing not supported" xfail with (1) single-doc packed == unpacked equivalence, (2) two-document attention isolation (perturbing doc B leaves doc A bit-identical), and (3) an end-to-end packed trainer step with per-document position_ids / seq_lens / doc_remaining. Signed-off-by: khazic <khazzz1c@gmail.com>
…-3 draft Signed-off-by: khazic <khazzz1c@gmail.com>
Apply /simplify cleanups: drop the redundant packed trainer-integration test (subsumed by the golden packing-vs-padding parity test), trim the forward docstring to a proportionate Args block that still documents the seq_lens packing contract, and shorten the inline packing comment to match the sibling dense draft. Signed-off-by: khazic <khazzz1c@gmail.com>
Address /code-review findings on the DeepSeek MLA draft packing path: - build_block_causal_additive_mask now validates that seq_lens is 2D and each row sums to seq_length, mirroring the FlashAttention varlen path's existing check in _seq_lens_to_cu_seqlens. A row that does not sum to T would otherwise silently misbucket the boundary cumsum (isolating the final token into a phantom document) and train the draft on wrong-context supervision. This guard protects every eager packing caller (the MLA draft, the dense Llama draft, and the target wrapper). - DeepseekV3Eagle3DraftModel.forward raises when seq_lens is passed without per-document position_ids instead of falling back to a global arange (which gives every document after the first a wrong RoPE phase), matching the target-side guard in target.py. Tests: reject non-2D / non-summing seq_lens in the mask helper, and reject packing without position_ids / with a non-summing seq_lens in the MLA draft. Signed-off-by: khazic <khazzz1c@gmail.com>
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The DeepSeek MLA EAGLE-3 draft (
DeepseekV3Eagle3DraftModel) shares the EAGLE-3 trainer (Eagle3TrainerModule), which already threadsposition_ids/seq_lens/doc_remainingfor sequence packing, but the draft'sforwardrejectedseq_lenswithNotImplementedError. This wires packing through the eager MLA path exactly like the sibling dense Llama draft (LlamaEagle3DraftModel).When
seq_lensis present, the draft builds the shared[B, 1, T, T]block-causal additive mask (build_block_causal_additive_mask) instead of the plain causal + padding mask, so Block-1 attention is document-level block-causal. The TTT diagonal block is position-wise and stays document-safe; cross-document supervision is masked by the trainer'sdoc_remaininggate (unchanged). The MLA attention is eager-only, so packing reuses the eager mask path (no FlashAttention varlen kernel). The trainer, recipe, and target wrapper are untouched: enabling packing for a DeepSeek target now works end to end (colocated backend,cp_size=1).Robustness
Packing inputs now fail loud instead of silently corrupting supervision:
build_block_causal_additive_maskvalidates thatseq_lensis 2D and each row sums toseq_length, mirroring the check the FlashAttention varlen path already performs in_seq_lens_to_cu_seqlens. A row that does not sum toTwould otherwise misbucket the boundarycumsum(isolating the final token into a phantom document). This guard covers every eager packing caller: the MLA draft, the dense Llama draft, and the target wrapper.DeepseekV3Eagle3DraftModel.forwardraises whenseq_lensis passed without per-documentposition_ids(the globalarangefallback would give every document after the first a wrong RoPE phase), matching the target-side guard.Correctness verification
CPU unit tests (27 pass across the two touched test files), including:
seq_lensand missingposition_idsraise.GPU smoke (1x A100-80GB, bf16) with a tiny HF DeepSeek-V3 target wrapped in the real
HFEagle3TargetModel, draft config derived from the target config exactly as the recipe does: the packed and unpacked paths both train (finite, decreasing loss), and the target's aux hidden states are document-isolated under packing (perturbing document B leaves document A's target aux states bit-identical), confirming the DeepSeek target honors the block-causal mask.GPU smoke log
Scope
DeepSeek MLA draft only, per the incremental plan. Extending packing to DFlash / DSpark / EAGLE-1/2 and resolving packing with context parallelism are separate follow-ups (each draft builds attention differently).