chore: CI proxy for #3002#3014
Closed
HuiyingLi wants to merge 5 commits into
Closed
Conversation
…-3 draft The DeepSeek MLA EAGLE-3 draft (DeepseekV3Eagle3DraftModel) shares the EAGLE-3 trainer (Eagle3TrainerModule), which already threads position_ids / seq_lens / doc_remaining for sequence packing, but the draft's forward rejected seq_lens with NotImplementedError. Wire packing through the eager MLA path exactly like the dense Llama draft: when seq_lens is present, build the [B, 1, T, T] block-causal additive mask (build_block_causal_additive_mask) instead of the plain causal + padding mask, so Block-1 attention is document-level block-causal. The TTT diagonal block is position-wise and stays document-safe; cross-document supervision is masked by the trainer's doc_remaining gate (unchanged). The MLA attention is eager-only, so packing reuses the eager mask path (no FA2 varlen kernel). Tests: replace the "packing not supported" xfail with (1) single-doc packed == unpacked equivalence, (2) two-document attention isolation (perturbing doc B leaves doc A bit-identical), and (3) an end-to-end packed trainer step with per-document position_ids / seq_lens / doc_remaining. Signed-off-by: khazic <khazzz1c@gmail.com>
…-3 draft Signed-off-by: khazic <khazzz1c@gmail.com>
Apply /simplify cleanups: drop the redundant packed trainer-integration test (subsumed by the golden packing-vs-padding parity test), trim the forward docstring to a proportionate Args block that still documents the seq_lens packing contract, and shorten the inline packing comment to match the sibling dense draft. Signed-off-by: khazic <khazzz1c@gmail.com>
Address /code-review findings on the DeepSeek MLA draft packing path: - build_block_causal_additive_mask now validates that seq_lens is 2D and each row sums to seq_length, mirroring the FlashAttention varlen path's existing check in _seq_lens_to_cu_seqlens. A row that does not sum to T would otherwise silently misbucket the boundary cumsum (isolating the final token into a phantom document) and train the draft on wrong-context supervision. This guard protects every eager packing caller (the MLA draft, the dense Llama draft, and the target wrapper). - DeepseekV3Eagle3DraftModel.forward raises when seq_lens is passed without per-document position_ids instead of falling back to a global arange (which gives every document after the first a wrong RoPE phase), matching the target-side guard in target.py. Tests: reject non-2D / non-summing seq_lens in the mask helper, and reject packing without position_ids / with a non-summing seq_lens in the MLA draft. Signed-off-by: khazic <khazzz1c@gmail.com>
Contributor
Author
|
/ok to test bc706cc |
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
Contributor
Author
|
/ok to test c64cf8f |
Contributor
Author
|
/ok to test c64cf8f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI-only proxy PR. Do not merge / do not review. Points at the exact head commit of #3002 (bc706cc) so internal CI runs under the internal-contributor queue; results post back to #3002 via the shared SHA. Source of truth: #3002. Close once CI completes.