S1-27: OUT_PROD_ID_GRP CPU kernel — the weight half of MUL_MAT_ID's backward#20
Open
dillon-blake wants to merge 1 commit into
Open
Conversation
…backward
d_as[k,j,e] = sum_{(i,t) : ids[i,t] == e} b[k, i % ne_b1, t] * grad[j,i,t]
This is the op that makes MoE LoRA possible at all: build_lora_mm_id computes
mul_mat_id(B, mul_mat_id(A, cur, ids), ids), so the trainable A/B tensors ARE the 3D
expert operand, and without d(as) a LoRA MoE run trains nothing.
Threaded BY EXPERT, deliberately rather than incidentally. Each expert's output slice
is written by exactly one thread, so the scatter-add needs no atomics and no reduction
barrier, and the summation order inside a slice is fixed by the (token, slot) loop
rather than by which thread arrives first. Changing the thread count changes WHICH
thread owns an expert, never the order of the float sum within it. Verified
bit-identical at 1/2/4/8 threads on data chosen to make float addition order-sensitive
(magnitudes mixed 1e6 against 1e-6, where a reordered sum would show).
Experts nothing routes to still get their slice zeroed -- it is a gradient, and the
optimizer reads it whether or not anything routed there this step.
THE TEST WOULD HAVE PASSED ON A KERNEL THAT IGNORED THE GRADIENT ENTIRELY.
MODE_GRAD's default objective is sum(out), which makes the incoming gradient all-ones,
which makes
d_as[k,j,e] = sum_{(i,t): ids=e} b[k,i,t] * 1
independent of j -- constant along the whole output axis. So a kernel that never reads
`grad` and just sums b-columns per expert satisfies it. Measured, not argued:
poisoned kernel (grad ignored) + default sum(out) -> OK, passes
poisoned kernel (grad ignored) + weighted grad_loss -> FAIL, MAA 6.56 vs 1e-4
test_mul_mat_id therefore overrides grad_loss with a weighted sum -- the same hook, for
the same structural reason, that S1-34 had to add for SOFT_MAX.
test_mul_mat_id also takes a grad_param selector now, because the two halves of
MUL_MAT_ID's backward are separate ops in separate tickets: `as`-only cases exercise
this kernel ALONE, so S1-27 has a green test to stand on while OUT_PROD_ID (S1-26) is
still unimplemented and its cases report not-supported. Shapes are deliberately tiny
(4x6x8) -- grad_nmax() is 10000 elements and every pre-existing test_mul_mat_id shape
is far above it, so they would have been silently skipped.
Both forward broadcast modes are covered (ne_b1 == 1 and ne_b1 == n_used); both are
live in the real LoRA MoE graph and they are different index arithmetic in the kernel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the S1-25 wiring PR (#19).
This is the op that makes MoE LoRA possible at all.
build_lora_mm_idcomputesmul_mat_id(B, mul_mat_id(A, cur, ids), ids)— the trainable A/B tensors are the 3D expertoperand — so without
d(as)a LoRA MoE run trains nothing.Threaded by expert, and that is load-bearing
Each expert's output slice is written by exactly one thread, so the scatter-add needs no atomics
and no reduction barrier, and the summation order inside a slice is fixed by the
(token, slot)loop rather than by whichever thread arrives first. Changing the thread count changes which thread
owns an expert, never the order of the float sum within it.
Verified bit-identical at 1 / 2 / 4 / 8 threads, on data chosen so that float addition is
order-sensitive (magnitudes mixed 1e6 against 1e-6 — where a reordered sum would actually show).
Determinism is an ADR-0002 hard requirement on a gradient path, and a sum whose order depends on
thread scheduling breaks reproducibility silently.
Experts that nothing routes to still get their slice zeroed. It is a gradient; the optimizer reads
it whether or not anything routed there this step.
The test would have passed on a kernel that ignored the gradient entirely
MODE_GRAD's default objective is
sum(out), which makes the incoming gradient all-ones, whichmakes
d_as[k,j,e]independent ofj— constant along the whole output axis. A kernel thatnever reads
gradand just sums b-columns per expert satisfies it completely.Measured, not argued — same broken kernel, two objectives:
sum(out)OK— passesgrad_lossFAIL, MAA 6.56 vs a 1e-4 boundSo
test_mul_mat_idoverridesgrad_losswith a weighted sum — the same hook, for the samestructural reason, that S1-34 had to add for
SOFT_MAX.Test structure
test_mul_mat_idtakes agrad_paramselector, because the two halves of MUL_MAT_ID's backwardare separate ops in separate tickets.
as-only cases exercise this kernel alone, so S1-27 has agreen test to stand on while
OUT_PROD_ID(S1-26) is still unimplemented and its cases reportnot-supported rather than aborting.
Shapes are deliberately tiny (4×6×8):
grad_nmax()is 10000 elements and every pre-existingtest_mul_mat_idshape is far above it, so they would have been silently skipped.Both forward broadcast modes are covered (
ne_b1 == 1andne_b1 == n_used) — both are live in thereal LoRA MoE graph, and they are different index arithmetic in the kernel.