Skip to content

perf(mtp): fused-scan ring capture for batched verify (#209 item 4) #290

Description

@pekkah

Follow-up to #209 (work item #4). #209's #1 shipped in #287.

Problem

BatchVerify captures per-token GDN recurrent state into the snapshot ring with gdnSnapRing=true, which forces the per-position recurrence loop (k×8 launches/layer) plus ~96×(k−1) CopyDeviceRegion launches per step (~3 ms at k=4) — instead of the #114-B fused GdnRecurrenceScan path the prefill trunk uses.

Goal

Extend GdnRecurrenceScan to optionally dump each token's post-update (scan, conv) state into the ring slots as it scans, so the verify trunk keeps the fused path and drops the per-position relaunch + the bulk CopyDeviceRegion fan-out.

Notes

  • Ring slot layout is per-layer (scan, conv) after batch token j; the fused scan already produces these states transiently.
  • Must preserve rollback correctness (RestoreBatchSnapshot reads slot lengthAfter − startPos − 1).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions