Skip to content

S1-27: OUT_PROD_ID_GRP CPU kernel — the weight half of MUL_MAT_ID's backward#20

Open
dillon-blake wants to merge 1 commit into
ticket/S1-25-mul-mat-id-add-id-backwardfrom
ticket/S1-27-out-prod-id-grp
Open

S1-27: OUT_PROD_ID_GRP CPU kernel — the weight half of MUL_MAT_ID's backward#20
dillon-blake wants to merge 1 commit into
ticket/S1-25-mul-mat-id-add-id-backwardfrom
ticket/S1-27-out-prod-id-grp

Conversation

@dillon-blake

Copy link
Copy Markdown
Owner

Stacked on the S1-25 wiring PR (#19).

d_as[k,j,e] = sum over {(i,t) : ids[i,t] == e}  of  b[k, i % ne_b1, t] * grad[j,i,t]

This is the op that makes MoE LoRA possible at all. build_lora_mm_id computes
mul_mat_id(B, mul_mat_id(A, cur, ids), ids) — the trainable A/B tensors are the 3D expert
operand — so without d(as) a LoRA MoE run trains nothing.

Threaded by expert, and that is load-bearing

Each expert's output slice is written by exactly one thread, so the scatter-add needs no atomics
and no reduction barrier
, and the summation order inside a slice is fixed by the (token, slot)
loop rather than by whichever thread arrives first. Changing the thread count changes which thread
owns an expert, never the order of the float sum within it.

Verified bit-identical at 1 / 2 / 4 / 8 threads, on data chosen so that float addition is
order-sensitive (magnitudes mixed 1e6 against 1e-6 — where a reordered sum would actually show).
Determinism is an ADR-0002 hard requirement on a gradient path, and a sum whose order depends on
thread scheduling breaks reproducibility silently.

Experts that nothing routes to still get their slice zeroed. It is a gradient; the optimizer reads
it whether or not anything routed there this step.

The test would have passed on a kernel that ignored the gradient entirely

MODE_GRAD's default objective is sum(out), which makes the incoming gradient all-ones, which
makes d_as[k,j,e] independent of j — constant along the whole output axis. A kernel that
never reads grad and just sums b-columns per expert satisfies it completely.

Measured, not argued — same broken kernel, two objectives:

objective verdict
default sum(out) OK — passes
weighted grad_loss FAIL, MAA 6.56 vs a 1e-4 bound

So test_mul_mat_id overrides grad_loss with a weighted sum — the same hook, for the same
structural reason, that S1-34 had to add for SOFT_MAX.

Test structure

test_mul_mat_id takes a grad_param selector, because the two halves of MUL_MAT_ID's backward
are separate ops in separate tickets. as-only cases exercise this kernel alone, so S1-27 has a
green test to stand on while OUT_PROD_ID (S1-26) is still unimplemented and its cases report
not-supported rather than aborting.

Shapes are deliberately tiny (4×6×8): grad_nmax() is 10000 elements and every pre-existing
test_mul_mat_id shape is far above it, so they would have been silently skipped.

Both forward broadcast modes are covered (ne_b1 == 1 and ne_b1 == n_used) — both are live in the
real LoRA MoE graph, and they are different index arithmetic in the kernel.

…backward

    d_as[k,j,e] = sum_{(i,t) : ids[i,t] == e}  b[k, i % ne_b1, t] * grad[j,i,t]

This is the op that makes MoE LoRA possible at all: build_lora_mm_id computes
mul_mat_id(B, mul_mat_id(A, cur, ids), ids), so the trainable A/B tensors ARE the 3D
expert operand, and without d(as) a LoRA MoE run trains nothing.

Threaded BY EXPERT, deliberately rather than incidentally. Each expert's output slice
is written by exactly one thread, so the scatter-add needs no atomics and no reduction
barrier, and the summation order inside a slice is fixed by the (token, slot) loop
rather than by which thread arrives first. Changing the thread count changes WHICH
thread owns an expert, never the order of the float sum within it. Verified
bit-identical at 1/2/4/8 threads on data chosen to make float addition order-sensitive
(magnitudes mixed 1e6 against 1e-6, where a reordered sum would show).

Experts nothing routes to still get their slice zeroed -- it is a gradient, and the
optimizer reads it whether or not anything routed there this step.

THE TEST WOULD HAVE PASSED ON A KERNEL THAT IGNORED THE GRADIENT ENTIRELY.

MODE_GRAD's default objective is sum(out), which makes the incoming gradient all-ones,
which makes

    d_as[k,j,e] = sum_{(i,t): ids=e} b[k,i,t] * 1

independent of j -- constant along the whole output axis. So a kernel that never reads
`grad` and just sums b-columns per expert satisfies it. Measured, not argued:

    poisoned kernel (grad ignored) + default sum(out)   ->  OK, passes
    poisoned kernel (grad ignored) + weighted grad_loss ->  FAIL, MAA 6.56 vs 1e-4

test_mul_mat_id therefore overrides grad_loss with a weighted sum -- the same hook, for
the same structural reason, that S1-34 had to add for SOFT_MAX.

test_mul_mat_id also takes a grad_param selector now, because the two halves of
MUL_MAT_ID's backward are separate ops in separate tickets: `as`-only cases exercise
this kernel ALONE, so S1-27 has a green test to stand on while OUT_PROD_ID (S1-26) is
still unimplemented and its cases report not-supported. Shapes are deliberately tiny
(4x6x8) -- grad_nmax() is 10000 elements and every pre-existing test_mul_mat_id shape
is far above it, so they would have been silently skipped.

Both forward broadcast modes are covered (ne_b1 == 1 and ne_b1 == n_used); both are
live in the real LoRA MoE graph and they are different index arithmetic in the kernel.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant