Skip to content

[Perf] Add portable subgroup reduce_add and reduce_all_add#511

Merged
hughperkins merged 27 commits into
mainfrom
hp/reduce
Apr 20, 2026
Merged

[Perf] Add portable subgroup reduce_add and reduce_all_add#511
hughperkins merged 27 commits into
mainfrom
hp/reduce

Conversation

@hughperkins

@hughperkins hughperkins commented Apr 18, 2026

Copy link
Copy Markdown
Collaborator

Portable subgroup reduce_add / reduce_all_add (replaces SPIR-V-only no-arg reductions)

Builds on hp/shuffle-down's portable subgroup.shuffle_down to add cross-platform
sum reductions, and removes the SPIR-V-only no-arg subgroup reductions for
consistency.

What's in this PR

New API: portable sized sum reductions

  • subgroup.reduce_add(value, log2_size) — sums value across 2**log2_size
    consecutive lanes via a shuffle_down tree. Result valid in lane 0 of each
    group; other lanes hold partial sums.
  • subgroup.reduce_all_add(value, log2_size) — same sum, broadcast to every
    lane via a shuffle butterfly (lane ^ mask).
  • Both are decorated with @qd.func and inlined; log2_size is a
    qd.template() (compile-time constant) so the body unrolls into exactly
    log2_size shuffle+add pairs in the calling kernel's IR.
  • Caller contract: 2**log2_size must not exceed the active subgroup size
    (32 on CUDA / Metal / RDNA, 64 on CDNA).
  • Works on i32 / u32 / f32 / f64 / i64 / u64 across CUDA, AMDGPU, and SPIR-V.
    Removed: SPIR-V-only no-arg reductions
  • subgroup.reduce_add(value) (no log2_size), reduce_mul, reduce_min,
    reduce_max, reduce_and, reduce_or, reduce_xor — all gone.
  • C++ side: dropped the subgroupAdd / subgroupMul / ... ops from
    internal_ops.inc.h, their POLY_OP signatures in type_system.cpp, and
    the reduction_ops set in spirv_codegen.cpp (the dispatch branch now
    handles inclusive_scan_ops only, with group_op fixed to
    InclusiveScan).
  • Inclusive scans (subgroup.inclusive_*) are intentionally untouched — they
    are a different primitive (prefix scan, not reduction) and remain
    SPIR-V-only pending their own portable replacement.
    Tests
  • Removed the old Vulkan-only test_subgroup_reduction_* tests and the
    _test_subgroup_reduce helper.
  • Added parametrised test_subgroup_reduce_add and
    test_subgroup_reduce_all_add covering log2_size ∈ {1, 2, 3, 4, 5} and
    dtypes {i32, f32, f64} on arch=qd.gpu (so all backends).
    Docs
  • Extended docs/source/user_guide/subgroup.md (renamed page title to
    "Subgroup primitives") with: support-matrix rows for both new ops,
    semantics sections covering the log2_size template + size-cap contract
    and the lane-0-only vs all-lanes guarantees, worked sum32 / normalize32
    / partial-subgroup examples, performance notes, and a note explaining that
    the SPIR-V-only no-arg reductions have been removed.
  • Build-time housekeeping: replaced the blanket # type: ignore on
    subgroup.py with a targeted # pyright: reportInvalidTypeForm=false
    (needed for the template() annotation form, mirroring _tile16.py).

Good points

  • Single portable API for the by-far most common reduction (sum). No
    more arch=qd.vulkan carve-outs in callers — reduce_add runs on CUDA,
    AMDGPU, and SPIR-V from the same source.
  • Zero kernel-launch / linkage overhead. @qd.func inlining + static
    range unrolling means the generated IR is byte-equivalent to a
    hand-written shuffle tree — no extra calls, no runtime loop.
  • Better than the old SPIR-V API in two ways: the caller chooses the
    reduction width via log2_size, and reduce_all_add (butterfly) saves a
    follow-up broadcast when every lane needs the result.
  • Strictly fewer C++ surfaces to maintain: 7 ops removed from the
    internal ops table, the SPIR-V reduction_ops set, and the type system.
  • Type-checking is tightened — blanket # type: ignore on subgroup.py
    is gone; only the one template()-related warning is suppressed.
  • Tests are parametrised across log2_size and dtypes, replacing two
    Vulkan-only tests with portable coverage.

Bad points / caveats

  • Breaking change. subgroup.reduce_add(value) (no second arg) and
    reduce_mul / min / max / and / or / xor are gone — any
    caller has to switch to either reduce_add(value, log2_size) or build
    their own sized helper on top of shuffle_down / shuffle. (Codebase
    audit showed no in-tree callers; external users will need to adapt.)
  • No portable replacement for reduce_mul/min/max/and/or/xor yet. This
    PR removes the SPIR-V-only path for consistency but only ships the add
    variants. If you need product/min/max reductions on CUDA or AMDGPU
    today, you have to write the shuffle tree yourself for now.
  • AMDGPU reduce_add performance is currently middling. It rides on
    shuffle_down, which on AMDGPU goes through ds_bpermute (LDS-routed,
    tens of cycles). A native DPP fast path is a future PR.
    reduce_all_add is unaffected — it only uses shuffle, which is native
    on AMDGPU.
  • No size-cap enforcement at trace time. Passing log2_size=6 on a
    32-lane subgroup compiles fine and silently produces wrong results in the
    lanes that read past the end — only documented in the docstring, not
    asserted. Could add a compile-time check later.
  • Tests don't cover i64 / u64. Parametrisation is {i32, f32, f64};
    64-bit integer reductions go through the same shuffle path but are
    untested in this PR.
    [fixed]
  • inclusive_* scans are still SPIR-V-only — they are out of scope here
    but the inconsistency is now more visible (sum reduction is portable,
    sum prefix-scan is not).

Risk

  • Low-risk on the C++ side: deletions only, plus a one-line group_op
    simplification in spirv_codegen.cpp. Full tests/python/test_simt.py
    passes (69 passed, 1 skipped) and pyright is clean (0/0/0) on
    subgroup.py.
  • Higher risk for downstream callers using the removed no-arg API; expect
    to break out-of-tree consumers.

Follow-ups

  • Add portable reduce_mul / reduce_min / reduce_max (and integer
    reduce_and / or / xor) using the same log2_size template
    pattern, when needed.
  • AMD DPP ROW_SHR fast path for shuffle_down (will accelerate
    reduce_add on AMDGPU automatically).
  • Cover i64 / u64 in the parametrised tests.

Issue: #

Brief Summary

copilot:summary

Walkthrough

copilot:walkthrough

Add subgroupShuffleDown support to CUDA and AMDGPU backends.
SPIR-V already handled it. CUDA maps to __shfl_down_sync;
AMDGPU emulates via ds_bpermute (FIXME to upgrade to DPP ROW_SHR).

Includes tests for basic shuffle_down and tree reduction pattern,
passing on both CUDA and Vulkan with i32/f32/f64.
Replaces the SPIR-V-only subgroup.reduce_add(value) (which lowered to
OpGroupNonUniformFAdd/IAdd) with a parameterised pair of portable helpers
on top of the cross-platform shuffle primitives:

  reduce_add(value, log2_size)      -- shuffle_down tree, lane-0 result
  reduce_all_add(value, log2_size)  -- butterfly XOR shuffle, all-lanes result

Both are plain Python helpers that unroll into log2_size shuffle+add ops
in the calling kernel's IR.  The caller picks 2**log2_size as the
reduction width (1, 2, 4, 8, 16, 32 lanes), so the API is wavefront-size
agnostic and works identically on CUDA, AMDGPU, and SPIR-V.

The SPIR-V subgroupAdd internal op, type signature, and codegen branch
are removed.  Other reduce_* ops (mul/min/max/and/or/xor) and the
inclusive scans are left unchanged for now.

Tests: parametrised test_subgroup_reduce_add and
test_subgroup_reduce_all_add over log2_size in {1..5} and dtypes
{i32, f32, f64}, all passing on CUDA.
Switch from plain-Python helpers to the canonical @qd.func form with a
qd.template() log2_size parameter and qd.static(range(...)) loops.

The circular-import blocker (subgroup.py loads during `import quadrants`
when qd.func is not yet bound) is avoided by importing the names
directly from their defining leaf modules:

  - func     from quadrants.lang.kernel_impl
  - template from quadrants.types.annotations
  - u32      from quadrants.types.primitive_types
  - static   referenced as impl.static (impl is partially initialised at
             subgroup.py load time but `impl.static` is resolved at
             call time, after impl has finished loading)

Net result: reduce_add and reduce_all_add are first-class
QuadrantsCallable functions, specialised per log2_size value, with the
body fully unrolled by qd.static at AST-rewrite time.

Tests: all 30 reduce_add/reduce_all_add cases and all 49 subgroup+shuffle
tests still pass on CUDA.
Removes `subgroup.reduce_mul` / `reduce_min` / `reduce_max` / `reduce_and` /
`reduce_or` / `reduce_xor` (and their SPIR-V codegen, op registration, type
signatures, and Vulkan-only tests).  These mirrored `subgroupAdd`, which was
already replaced last commit by the portable `subgroup.reduce_add(value,
log2_size)` built on `subgroupShuffleDown`.  Keeping the other reductions on
the SPIR-V-only path was inconsistent and Vulkan-locked; future replacements
should follow the sized portable pattern (shuffle tree on top of `shuffle_down`
/ `shuffle`) when they are needed.

Inclusive scans (`subgroupInclusive*`) are left untouched - they are a
different primitive (prefix scan, not reduction) and still SPIR-V-only.
Document the portable qd.simt.subgroup.shuffle op: backend support
matrix, semantics, examples (broadcast, identity, neighbour swap,
per-lane gather), and performance notes. Note that shuffle_up,
shuffle_down, and shuffle_xor are not yet implemented across backends.
Link from the SIMT primitives section of the user guide index.
Add shuffle_down to the support matrix (CUDA/AMDGPU/SPIR-V), give it
its own Semantics subsection, add a tree-reduction example, and note
that AMDGPU still goes through ds_bpermute pending a DPP fast path.
The page is the natural home for all qd.simt.subgroup.* docs, not
just shuffle, so give it a name that reflects the broader scope.
Extends the subgroup user guide (inherited from hp/shuffle-down) with
coverage of the new portable sized reductions:

- add rows to the support matrix for `reduce_add(v, log2_size)` and
  `reduce_all_add(v, log2_size)`;
- add semantic sections describing the `log2_size` template contract,
  the lane-0-only vs all-lanes result guarantees, and the fact that
  `reduce_add` builds on `shuffle_down` while `reduce_all_add` builds
  on the native `shuffle` butterfly;
- add worked examples: `sum32` using `reduce_add`, `normalize32` using
  `reduce_all_add`, and a note about partial-subgroup reductions;
- note that the SPIR-V-only no-arg reductions (reduce_mul/min/max/and/
  or/xor and the old reduce_add) have been removed in favour of the
  portable sized API;
- extend the performance-notes and related sections accordingly.

Also renames the page title from "Subgroup shuffles" to "Subgroup
primitives" since it now covers shuffles + reductions.
# Conflicts:
#	docs/source/user_guide/subgroup.md
Extends `test_subgroup_reduce_add` and `test_subgroup_reduce_all_add` from
{i32, f32, f64} to {i32, i64, u64, f32, f64} and updates `_init_field` to
fill 64-bit integer fields with positive integer values.  64-bit ints go
through the same shuffle path (split into two 32-bit shuffles on AMDGPU,
native 64-bit on CUDA) so this just locks the path under the parametrised
exact-equality assertion.

50 reduce-test cases pass on CUDA (5 dtypes x 5 log2_sizes x 2 ops).
@hughperkins
hughperkins marked this pull request as ready for review April 18, 2026 07:31
Comment thread docs/source/user_guide/subgroup.md
Resolve conflict by keeping the shuffle_down content from
hp/shuffle-down and applying the unwrap-paragraphs style from
hp/shuffle-doc to it as well.
# Conflicts:
#	docs/source/user_guide/subgroup.md
The helper does not exist under that name (the actual function is
group_size()), and either way the underlying subgroupSize op is only
wired up in the SPIR-V codegen, so it's not portable across backends.
Remove the bullet rather than document a non-portable helper.
Made-with: Cursor

# Conflicts:
#	docs/source/user_guide/subgroup.md
Metal does not support 64-bit integer types in subgroup shuffle
operations, causing worker crashes in CI on all Mac test jobs.

Made-with: Cursor
MoltenVK does not support 64-bit integer types, causing worker
crashes in test_subgroup_reduce_add on Mac CI.

Made-with: Cursor
Comment thread tests/python/test_simt.py
Comment on lines +14 to +18
arch = qd.lang.impl.current_cfg().arch
if dtype in (qd.i64, qd.u64) and arch == qd.metal:
pytest.skip("64-bit integer types not supported on Metal")
if dtype in (qd.i64, qd.u64) and arch == qd.vulkan and platform.system() == "Darwin":
pytest.skip("MoltenVK does not support 64-bit integer types")

@duburcqa duburcqa Apr 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scalar i64 types and arithmetic (long, ulong, int64_t, uint64_t): Available on every M-series chip from M1 onward. MSL has had them since 2.2, and all Apple-silicon Macs meet that. But Apple GPUs have no native 64-bit integer ALU — the compiler lowers these to sequences of 32-bit ops. Typical costs are roughly 2× for add/sub, ~4× for multiply, and noticeably worse for division. Fine for address math, hashing, bit-packing, and any code where i64 is a small fraction of the work. Painful if i64 is in your hot inner loop.

64-bit buffer addressing: All M-series have it; pointers into buffers are effectively 64-bit. This is what lets you cast buffer arguments to device ulong* and index past 4 GB.

@erizmr erizmr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on the basis that I have reviewed the design and the public facing API, tests, and they look reasonable to me.

Base automatically changed from hp/shuffle-down to main April 20, 2026 19:14
Made-with: Cursor

# Conflicts:
#	docs/source/user_guide/subgroup.md
#	tests/python/test_simt.py
@hughperkins
hughperkins enabled auto-merge (squash) April 20, 2026 19:26
@hughperkins
hughperkins merged commit 12e04f6 into main Apr 20, 2026
47 checks passed
@hughperkins
hughperkins deleted the hp/reduce branch April 20, 2026 20:33
npoulad1 added a commit to ROCm/quadrants that referenced this pull request Jun 8, 2026
* [Misc] Warn user to disable caching when print_ir/QD_DUMP_IR enabled (Genesis-Embodied-AI#425)

Co-authored-by: v01dxyz <v01dxyz@v01d.xyz>

* [Build] Pin torch version to CUDA 12.8 for CUDA tests (Genesis-Embodied-AI#428)

* [Misc] Fixing up taichi-dev urls (Genesis-Embodied-AI#429)

* [Perf] Rename cuda_graph to gpu_graph across the codebase (Genesis-Embodied-AI#430)

* Misc: fix typo integeral -> integral (Genesis-Embodied-AI#434)

Co-authored-by: v01dxyz <v01dxyz@v01d.xyz>

* [Perf] CUDA graph 4: call from multiple locations (Genesis-Embodied-AI#420)

* [Bug] Fix fastcache not restoring graph_do_while_arg (Genesis-Embodied-AI#435)

* [Perf] Cache last-call result in perf_dispatch for single-compatible case (Genesis-Embodied-AI#438)

* Fix gpu_graph fallback on old Nvidia GPU. (Genesis-Embodied-AI#443)

* Fix shared memory offset not reset between CUDA kernels. (Genesis-Embodied-AI#442)

* [Misc] Allow disabling GPU graph via QD_GPU_GRAPH=0 env var (Genesis-Embodied-AI#439)

* [Misc] Add named top-level loops (Genesis-Embodied-AI#440)

* [Misc] Rename gpu_graph to graph (Genesis-Embodied-AI#446)

* [Misc] Add cross-platform shuffle (Genesis-Embodied-AI#447)

* [Bug] Fix graph_do_while on Windows: search for cudadevrt.lib (Genesis-Embodied-AI#456)

* [Bug] Also search default CUDA toolkit install location on Windows (Genesis-Embodied-AI#461)

* [SPIRV] Feature Parity Atomics & Shared Array (Genesis-Embodied-AI#432)

* [Misc] Change clang format to 120 characters (Genesis-Embodied-AI#463)

* [Misc] CUDA graph 5 Add fatbin (Genesis-Embodied-AI#464)

* [Bug] Reuse VkInstance across init/reset cycles (Genesis-Embodied-AI#465)

* [Perf] Tiles 1: _load, _store, _eye_ (Genesis-Embodied-AI#466)

* [Misc] Remove dead InternalFuncStmt type_check override (Genesis-Embodied-AI#471)

* [Perf] Tiles 2: add cholesky and ger (Genesis-Embodied-AI#472)

* [Perf] Tiles 2b: add triangular solve (Genesis-Embodied-AI#474)

* [Misc] Refactor: use _get_col/_set_col in tiles load/store/init (Genesis-Embodied-AI#475)

* [Build] Fix flaky test_clock_accuracy (Genesis-Embodied-AI#436)

* Fix AARCH64 emitting invalid asm in CUDA kernels. (Genesis-Embodied-AI#473)

Co-authored-by: Hugh Perkins <hughperkins@gmail.com>

* [AMDGPU] Enable HIP memory pool and surface pool-exhaustion errors. (Genesis-Embodied-AI#485)

* [AMDGPU] Scope hsaco tmp dir per-user to avoid collisions. (Genesis-Embodied-AI#484)

* [Perf] Tiles 3: Add slice syntax, qd.outer() and initial doc (Genesis-Embodied-AI#477)

* [AMDGPU] Fix gradient computation. (Genesis-Embodied-AI#486)

* Enable all backends that are supported in unit tests. (Genesis-Embodied-AI#488)

* Fix SPIRV ID overflow for large kernels due to autodiff. (Genesis-Embodied-AI#489)

* [Misc] Fix purity checker to allow accessing constants from quadrants modules (Genesis-Embodied-AI#487)

* [Misc] Increase tolerance for clock monotonic test (Genesis-Embodied-AI#492)

* [CI] Serialize api doc workflow (Genesis-Embodied-AI#494)

* [CI] Increase tolerance for clock test (Genesis-Embodied-AI#506)

* [CI] Increase clock test tolerance to 20% (Genesis-Embodied-AI#509)

* [Perf] Add tensor_type parametrization to tile16 tests (Genesis-Embodied-AI#504)

* [Perf] Tiles 4b: Migrate tiles16 tests to enable fastcache (Genesis-Embodied-AI#505)

* [Perf] Tiles 4c: add Tiles16x16 proxy (Genesis-Embodied-AI#507)

* [Perf] Tiles 4d: Consolidate slice error tests using parametrize (Genesis-Embodied-AI#508)

* [Perf] Tiles 4: add SharedArray slice support (Genesis-Embodied-AI#482)

* [Perf] Tiles 5: add Cholesky benchmark demo (Genesis-Embodied-AI#483)

* [Doc] Add user guide page for subgroup shuffle (Genesis-Embodied-AI#512)

* [Perf] Implement cross-platform shuffle_down (Genesis-Embodied-AI#510)

* [Perf] Add portable subgroup reduce_add and reduce_all_add (Genesis-Embodied-AI#511)

* [Perf] Add first warmup config to perf dispatch (Genesis-Embodied-AI#422)

* [AutoDiff] Autodiff 1: Add baseline adstack regression test for unary_collections (Genesis-Embodied-AI#500)

* [AutoDiff] Autodiff 2: Implement derivative for tan (Genesis-Embodied-AI#501)

* [AutoDiff] Autodiff 3: Recompute tanh/exp on the operand in the reverse pass (Genesis-Embodied-AI#502)

* [AutoDiff] Autodiff 4: Mark rsqrt as non-linear for adstack promotion (Genesis-Embodied-AI#503)

* [AutoDiff] Autodiff 5: Fix adjoint-alloca placement for GlobalLoads outside the current range-for (Genesis-Embodied-AI#496)

* [AutoDiff] Autodiff 6: Adstack regression tests (Genesis-Embodied-AI#491)

* [AutoDiff] Autodiff 7: Fix header size in AdStackAllocaStmt to match u64 runtime layout (Genesis-Embodied-AI#534)

* [AutoDiff] Autodiff 8: Surface LLVM adstack push/pop overflow as a Python exception (Genesis-Embodied-AI#535)

* [AutoDiff] Autodiff 9: Guard against LLVM worker-thread stack overflow from large per-task adstack budget (Genesis-Embodied-AI#495)

* [AutoDiff] Autodiff 10: Implement adstack for SPIR-V (Genesis-Embodied-AI#490)

* [AutoDiff] Autodiff 11: Latent adstack-adjacent fixes (AMDGPU hipFree, flush() keeps ctx_buffers_, always-preallocate) (Genesis-Embodied-AI#536)

* [Doc] Add AGENTS.md with instructions for AI agents (Genesis-Embodied-AI#541)

* [Bug] Abort kernel execution on assertion failure instead of segfaulting (Genesis-Embodied-AI#419)

* [Type] ndarray typing 1: Add eval_str=True to inspect.signature() calls (Genesis-Embodied-AI#411)

* [CI] Suppress reportPrivateImportUsage in torch-using files (Genesis-Embodied-AI#552)

* [Misc] QD_DUMP_IR dumps to files with the task_id added to the filename (Genesis-Embodied-AI#441)

* [Type] ndarray typing 2: Fix NDArray single-arg subscript crash (Genesis-Embodied-AI#412)

* [Test] Flush xdist channel before worker exit so test failure reports are visible (Genesis-Embodied-AI#555)

* [CI] Reduce test retries on CI from 3 to 1. (Genesis-Embodied-AI#554)

* [AutoDiff] Autodiff 12: Heap-backed adstack on LLVM backends (CPU/CUDA/AMDGPU) (Genesis-Embodied-AI#537)

* [AutoDiff] Autodiff 13: Heap-backed adstack on SPIR-V backends (Metal, Vulkan) (Genesis-Embodied-AI#493)

* [AutoDiff] Autodiff 14: Resolve bounded-inner-loop adstacks without default_ad_stack_size fallback (Genesis-Embodied-AI#539)

* [SPIRV] Vulkan SPIR-V correctness: atomic-view aliasing, PSB stride, narrow storage caps, u1 cast, per-init layer recheck (Genesis-Embodied-AI#513)

* [Build] Autodiff 15: Replace 2022 MoltenVK pin with LunarG Vulkan SDK fetch and sanitise MoltenVK cap advertisement (Genesis-Embodied-AI#551)

* [Test] Suppress stock pytest-timeout to avoid conflict with pytest_hardtle (Genesis-Embodied-AI#557)

* [Vulkan] Use SDK validation layer for debugPrintf instead of apt package (Genesis-Embodied-AI#562)

* [Test] Fix flaky perf_dispatch tests by increasing work amounts (Genesis-Embodied-AI#559)

* [Test] Add --maxfail CLI option to run_tests.py (default 20) (Genesis-Embodied-AI#558)

* [CI] Vulkan debug printf fix to address flaky tests (Genesis-Embodied-AI#563)

* [Docs] Add a new page to help for first time contributors (Genesis-Embodied-AI#426)

Authored-by: v01dxyz <v01dxyz@v01d.xyz>

* [AutoDiff] Autodiff 16: Resolve reverse-mode adstack depths per-launch via runtime-evaluated SizeExpr (Genesis-Embodied-AI#543)

* Fix: raise error if device memory allocation fails (Genesis-Embodied-AI#451) (Genesis-Embodied-AI#453)

Co-authored-by: v01dxyz <v01dxyz@v01d.xyz>
Co-authored-by: Hugh Perkins <hughperkins@gmail.com>

* [CI] Add CI job to check line wrapping of comments and docs (Genesis-Embodied-AI#564)

* [Misc] Add coverage report to PRs, including kernels (Genesis-Embodied-AI#470)

* [CI] CI wrap check feeds only diffs to agent (Genesis-Embodied-AI#567)

* Skip 'flaky' test on MacOS CI. (Genesis-Embodied-AI#573)

* [Test] Fix missing `import sys` in test_fail_device_memory_allocation (Genesis-Embodied-AI#574)

* [CI] Fix Vulkan debugPrintf flake with session-scoped warmup (Genesis-Embodied-AI#571)

* [AutoDiff] determine_ad_stack_size: replace whole-CFG Bellman-Ford with SCC + DAG DP (Genesis-Embodied-AI#575)

* [Test] Fix macOS OOM skip reason to describe actual root cause (Genesis-Embodied-AI#576)

* [Lang] whole_kernel_cse: 2.5x compile time speedup on large kernels (Genesis-Embodied-AI#577)

* [CI] Add CI check for unnecessarily deleted comments (Genesis-Embodied-AI#570)

* [CI] Migrate coverage report to github Check page (Genesis-Embodied-AI#566)

* [Lang] Skip IR verifier between passes unless debug=true (Genesis-Embodied-AI#579)

* [Lang] Inline AdStack ops on release LLVM codegen: dramatically reduces compile time for adstack-enabled reverse-mode kernels (Genesis-Embodied-AI#584)

* [CUDA] Honor offline_cache=False end-to-end so QD_OFFLINE_CACHE=0 actually gives a cold compile (Genesis-Embodied-AI#580)

* [Type] Tensor 24 (Genesis-Embodied-AI#561)

Co-authored-by: hugh <hugh@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local>

* [Lang] auto_diff host-walk reductions: dramatically faster front-end compile time on adstack-enabled reverse-mode kernels (Genesis-Embodied-AI#587)

* [AutoDiff] Speed up reverse-mode kernel launches on GPU backends (Genesis-Embodied-AI#578)

* [Vulkan] Move adstack-sizer scratch out of Function-scope memory to fix SPIR-V pipeline build failures (Genesis-Embodied-AI#588)

* [AutoDiff] Improve diagnosis of unsupported reverse-mode AD patterns (Genesis-Embodied-AI#590)

* [Bug] Fix: promote Ndarray to AnyArray in build_Name for flattened struct fields (Genesis-Embodied-AI#592)

* [SPIR-V] Shrink reverse-grad kernel MSL by ~50% (Genesis-Embodied-AI#591)

* [CI] Add CI check that PR changes have test coverage (Genesis-Embodied-AI#596)

* [Perf] Enable zero-copy in to_torch() and to_numpy() (Genesis-Embodied-AI#450)

* Add BufferView: safe sub-range ndarray access for kernels (Genesis-Embodied-AI#585)

Co-authored-by: alanray-tech <alanray-tech@users.noreply.github.com>
Co-authored-by: Hugh Perkins <hughperkins@gmail.com>

* [Doc] Add user-facing fastcache documentation (Genesis-Embodied-AI#597)

Co-authored-by: hugh <hugh@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local>

* [Misc] Upgrade to enable v1 dlpack so to_numpy(copy=False) writable (Genesis-Embodied-AI#598)

Co-authored-by: root <root@rtx-209-201.slurm-compute.tenant-slurm.svc.cluster.local>

* [AutoDiff] Cut reverse-mode adstack memory usage 10x on all backends (Genesis-Embodied-AI#599)

* [Misc] Add CI check for feature file factorization (Genesis-Embodied-AI#606)

* [Perf] Skip _recursive_set_args for all-Field frozen dataclass structs (Genesis-Embodied-AI#607)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [AutoDiff] SNode-arm bound-expr capture rejects fold-attack gate indices (Genesis-Embodied-AI#610)

* [Misc] Suppress field fastcache warning for qd.Tensor (Genesis-Embodied-AI#615)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [AutoDiff] Adstack heap: clip reducer count by per-task loop trip count (compile-time and SizeExpr-evaluated) (Genesis-Embodied-AI#611)

* [Misc] Forward copy= through qd.Tensor, add copy=None option (Genesis-Embodied-AI#616)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Doc] Update README (Genesis-Embodied-AI#617)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [CI] Fix coverage report showing def lines as uncovered (Genesis-Embodied-AI#623)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Perf] Generic launcher: persistent context, JIT-pointer reuse, Metal compute encoder, LLVM-GPU async memory ops (Part 1/2) (Genesis-Embodied-AI#619)

* [CI] Encode Python-first testing policy in coverage-check prompt (Genesis-Embodied-AI#622)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [CI] Add PR Line change report (Genesis-Embodied-AI#624)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [CI] Disable quadrants pytest plugin during quadrants internal coverage runs (Genesis-Embodied-AI#629)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [AutoDiff] Adstack load+store eliminations: EliminateRecomputableAdStackPushes pass + leaf extensions (Genesis-Embodied-AI#621)

* [CI] Simplify coverage PR comment to a single linked line (Genesis-Embodied-AI#630)

* [CUDA] Add AGX Thor, SM_110 (Genesis-Embodied-AI#631)

Co-authored-by: Johnny Nunez and Hugh Perkins

* [CI] Lines changed report: collapse PR comment to a single linked totals line (Genesis-Embodied-AI#632)

* [FEATURE] Support external Metal command queue via qd.init (Genesis-Embodied-AI#618)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Perf] Cache adstack-sizer metadata per task across SPIR-V + LLVM-GPU; per-snode / DeviceAllocation invalidation (Part 2/2) (Genesis-Embodied-AI#620)

* [AutoDiff] Disable EliminateRecomputableAdStackPushes pending mutated-SNode chain-leaf fix (Genesis-Embodied-AI#633)

* [AutoDiff] Adstack chain-clone safety: mutated-SNode leaf reject + load_top consumer-aware guard (Genesis-Embodied-AI#634)

* [Docs] Add user-guide page for qd.simt.block.* primitives (Genesis-Embodied-AI#638)

* [Docs] Expand qd.simt.subgroup user-guide page to cover every op (Genesis-Embodied-AI#639)

* [Perf] Streams 1-4 (Genesis-Embodied-AI#410)

* [Docs] Add user-guide page for matrix decompositions and solvers (Genesis-Embodied-AI#643)

* [Bug] Revert "[Perf] Streams 1-4 (Genesis-Embodied-AI#410)" (Genesis-Embodied-AI#650)

* [Docs] Add user-guide page for atomics and bit operations (Genesis-Embodied-AI#640)

* [Docs] Add user-guide page for qd.simt.grid.* primitives (Genesis-Embodied-AI#641)

* [AutoDiff] Adstack max-reducer: parallel multi-axis MaxOverRange dispatch (Genesis-Embodied-AI#635)

* [AMDGPU] Fix amdgpu parallel rand init (Genesis-Embodied-AI#658)

* [Perf] Adstack: skip max-reducer recognizer on CPU + lift host-eval cap (Genesis-Embodied-AI#655)

* [Perf] Re-land Streams 1-4 with bug fixes (Genesis-Embodied-AI#653)

* [AMDGPU] Apply device_memory_GB=0.3 cap to AMDGPU tests (Genesis-Embodied-AI#659)

* [Perf] Per-launch host sync: drop wait_idle on SPIR-V, pin stream and drop stream_synchronize on CUDA/AMDGPU (Genesis-Embodied-AI#654)

* [AMDGPU] Unload hipModule_t in JITModuleAMDGPU destructor (Genesis-Embodied-AI#660)

* [AMDGPU] Trim default mempool on qd.reset() (Genesis-Embodied-AI#669)

* [AMDGPU] Hoist rand-state buffer to process lifetime (Genesis-Embodied-AI#668)

* [Streams] Use events for streams serialization on AMDGPU and CUDA (Genesis-Embodied-AI#667)

* [Perf] Adstack max-reducer: launch cache + zero-copy result map; content-stable registry_id (Genesis-Embodied-AI#671)

* [SPIR-V] dispatch_max_reducers: register each task with the real kernel name (Genesis-Embodied-AI#675)

* [AutoDiff] Debug-mode field/grad/dual: dtype, layout, and access-time invariants (Genesis-Embodied-AI#677)

* [Docs] Add user-guide page for qd.algorithms.* device-wide algorithms (Genesis-Embodied-AI#642)

Co-authored-by: alanray-tech <alan.ray@genesis-ai.company>

* [Docs] Doc for existing atomics: switch support table to per-backend columns (Genesis-Embodied-AI#657)

Co-authored-by: alanray-tech <alan.ray@genesis-ai.company>

* [GPU] Cross gpu atomics (Genesis-Embodied-AI#666)

Co-authored-by: alanray-tech <alan.ray@genesis-ai.company>

* [GPU] Make block operations portable cross-gpu (Genesis-Embodied-AI#664)

* [Perf] CPU LLVM adstack-cache: skip per-launch bump-writes + ndarray_shapes capture on forward-only handles (Genesis-Embodied-AI#685)

* [GPU] Cross-GPU for grid ops (Genesis-Embodied-AI#670)

* [Math] Make bitop operations portable cross-gpu (Genesis-Embodied-AI#662)

* [AMDGPU] Always use wave64, on both RDNA and CDNA (Genesis-Embodied-AI#687)

* [AMDGPU] Use syncscope("agent") for atomix xor to avoid CAS livelock (Genesis-Embodied-AI#672)

* [GPU] New bit ops for QIPC (Genesis-Embodied-AI#679)

* [GPU] Subgroup ops cross-gpu (Genesis-Embodied-AI#665)

* [Graph] Rename CUDA Graph to Graph in docs (Genesis-Embodied-AI#691)

* [SPIR-V] Fix FIFO-queue ordering when sharing command queue. (Genesis-Embodied-AI#694)

* [Atomics] New QIPC ops for atomics (Genesis-Embodied-AI#690)

* Pass dataclass sub-structs into qd.func (Genesis-Embodied-AI#698)

* [AMDGPU] HIP graph runtime support for @qd.kernel(graph=True) (Genesis-Embodied-AI#692)

* [CI] Add per-file timing report to Mac Metal test job (Genesis-Embodied-AI#695)

Co-authored-by: Cursor <cursoragent@cursor.com>

* [CI] Enable kernel disk cache during tests (Genesis-Embodied-AI#696)

* [Math] New QIPC ops for single-threaded linalg (Genesis-Embodied-AI#683)

* [BREAKING][GPU] New QIPC ops for subgroups (Genesis-Embodied-AI#676)

* [GPU] New QIPC ops for block (Genesis-Embodied-AI#684)

* [GPU] New device-level ops for QIPC (Genesis-Embodied-AI#693)

* [algorithms] PrefixSumExecutor: drop unused GRID_SZ local (Genesis-Embodied-AI#701)

* [block] sync(): fix unsupported-arch error message (Genesis-Embodied-AI#700)

* [volatile_load] add qd.volatile_load primitive (closes Genesis-Embodied-AI#648) (Genesis-Embodied-AI#702)

* [AutoDiff] Reject recycled identity_key in AdStackCache::register_adstack_sizing_info (Genesis-Embodied-AI#708)

* [Vulkan] Declare GroupNonUniform SPIR-V caps and enable shaderSubgroupExtendedTypes (Genesis-Embodied-AI#707)

* Fix duplicate HIP graph driver-function declarations after v1.0.0 merge

The amd-integration fork had cherry-picked the HIP graph driver functions
(graph_create / graph_destroy / graph_add_kernel_node / graph_instantiate /
graph_exec_destroy / graph_launch), and upstream v1.0.0 added the same set.
The per-file 3-way merge appended both copies into
amdgpu_driver_functions.inc.h, producing redeclaration errors that broke the
AMDGPU RHI/runtime compile. Drop the upstream duplicate block; the signatures
are identical to the fork's existing declarations.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix AMDGPU launcher coherence and num_instructions visibility after v1.0.0 merge

- kernel_launcher.cpp: the 3-way merge spliced upstream v1.0.0's launch_llvm_kernel
  rewrite (ephemeral arg/context buffers, explicit-stream path, AmdgpuDefaultStream
  PinGuard) onto the AMD fork's kernarg-by-value + persistent-scratch design,
  leaving references to undefined `ephemeral_context_ptr`. Restore the fork's
  coherent launch_llvm_kernel verbatim; it calls the (already merged) enhanced
  launch_offloaded_tasks, which keeps the max-reducer dispatch and stream-parallel
  groups adapted onto the AMD launch path.
- llvm_context.h: both the fork and upstream added `num_instructions`; the merge
  kept upstream's private placement, but the AMDGPU codegen force-inline heuristic
  calls it statically from outside the class. Move it back to the public section.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Restore async result D2H and hoist kernarg vectors in AMDGPU launcher

The v1.0.0 merge resolution regressed two amd-integration baseline
optimizations in launch_llvm_kernel / launch_offloaded_tasks:

  - The per-launch result-buffer copy was a blocking memcpy_device_to_host,
    forcing a host stall on every value-returning launch and serializing the
    GPU pipeline. Restore the async D2H (the caller synchronizes lazily when it
    needs the value); external-array transfers still stream_synchronize once
    before reading back.

  - launch_task constructed the kernarg std::vectors from initializer lists
    ({kernarg_payload} / {kernarg_size}) on every dispatch (heap alloc + free
    per launch). Hoist arg_ptrs/arg_sizes out of the per-task launch and reuse.

Co-authored-by: Cursor <cursoragent@cursor.com>

* amdgpu: default to LDS permlane64 emulation; drop host-x86 barrier asm on retarget

Two AMDGPU JIT-compile crashes surfaced after the v1.0.0 merge pulled in the QIPC subgroup
ops (Genesis-Embodied-AI#676), which made the rigid constraint solver's wave-cooperative reductions route through
`amdgpu_cross_half_shuffle_i32`. Both manifested as a SIGSEGV inside
`llvm::SIInstrInfo::getInstSizeInBytes` during `JITSessionAMDGPU::compile_module_to_hsaco`
(i.e. at first kernel launch), and reproduce on gfx942 / MI300X. Baseline 0.4.6 never emitted
these constructs, which is why it was unaffected.

1. Native `llvm.amdgcn.permlane64` lowering crashes the bundled LLVM 22.1.0 AMDGPU backend.
   Default `amdgpu_permlane64` to the existing LDS-roundtrip software emulation on every target
   (it produces identical results). Add `QD_AMDGPU_USE_NATIVE_PERMLANE64=1` to opt back into the
   native instruction once the backend bug is fixed; the old `QD_AMDGPU_FORCE_PERMLANE64_FALLBACK`
   is now the default and still honored. This is the actual crash fix.

2. The runtime module is compiled by the host x86_64 clang and only retargeted to amdgcn here, so
   `amdgpu_cross_half_shuffle_i32`'s `__asm__ volatile("" : "+v"(byte))` optimization barrier carries
   x86 flag clobbers (`~{dirflag},~{fpsr},~{flags}`) that are meaningless on AMDGPU. The IR verifies
   but the empty-body INLINEASM is invalid on the amdgcn target. Neutralize empty-body barrier asm
   during retarget (forward the tied value, then erase) so no stale host asm reaches codegen. On the
   wave64 targets we ship `ds_bpermute` already addresses the full wave, so the hint is a no-op.

Co-authored-by: Cursor <cursoragent@cursor.com>

* style: apply clang-format (v19.1.7) to AMDGPU fn_attrs and launcher sources

CI pre-commit's clang-format hook reformatted these files (long
declarations/lambda signatures collapsed onto single lines per the repo's
clang-format config). Apply the same formatting so the hook passes.

No functional changes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(amdgpu): use CreateNeg for branchless i32 sgn instead of CreateSub(0, input)

clang-tidy (modernize-use-nullptr, -warnings-as-errors) flagged
`builder->CreateSub(0, input)` in the i32 sgn path: the literal `0` binds to
the `llvm::Value*` LHS parameter as a null pointer, not an integer zero.
Replace with `builder->CreateNeg(input)`, which emits `0 - input` with a proper
zero constant -- identical intended semantics, and clang-tidy clean.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Robert Dazi <14996868+v01dXYZ@users.noreply.github.com>
Co-authored-by: v01dxyz <v01dxyz@v01d.xyz>
Co-authored-by: Hugh Perkins <hughperkins@gmail.com>
Co-authored-by: Alexis DUBURCQ <alexis.duburcq@gmail.com>
Co-authored-by: hugh <hugh@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local>
Co-authored-by: alanray-tech <alan.ray@genesis-ai.company>
Co-authored-by: alanray-tech <alanray-tech@users.noreply.github.com>
Co-authored-by: root <root@rtx-209-201.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Johnny <johnnynuca14@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants