[Perf] Add portable subgroup reduce_add and reduce_all_add#511
Conversation
Add subgroupShuffleDown support to CUDA and AMDGPU backends. SPIR-V already handled it. CUDA maps to __shfl_down_sync; AMDGPU emulates via ds_bpermute (FIXME to upgrade to DPP ROW_SHR). Includes tests for basic shuffle_down and tree reduction pattern, passing on both CUDA and Vulkan with i32/f32/f64.
Replaces the SPIR-V-only subgroup.reduce_add(value) (which lowered to
OpGroupNonUniformFAdd/IAdd) with a parameterised pair of portable helpers
on top of the cross-platform shuffle primitives:
reduce_add(value, log2_size) -- shuffle_down tree, lane-0 result
reduce_all_add(value, log2_size) -- butterfly XOR shuffle, all-lanes result
Both are plain Python helpers that unroll into log2_size shuffle+add ops
in the calling kernel's IR. The caller picks 2**log2_size as the
reduction width (1, 2, 4, 8, 16, 32 lanes), so the API is wavefront-size
agnostic and works identically on CUDA, AMDGPU, and SPIR-V.
The SPIR-V subgroupAdd internal op, type signature, and codegen branch
are removed. Other reduce_* ops (mul/min/max/and/or/xor) and the
inclusive scans are left unchanged for now.
Tests: parametrised test_subgroup_reduce_add and
test_subgroup_reduce_all_add over log2_size in {1..5} and dtypes
{i32, f32, f64}, all passing on CUDA.
Switch from plain-Python helpers to the canonical @qd.func form with a
qd.template() log2_size parameter and qd.static(range(...)) loops.
The circular-import blocker (subgroup.py loads during `import quadrants`
when qd.func is not yet bound) is avoided by importing the names
directly from their defining leaf modules:
- func from quadrants.lang.kernel_impl
- template from quadrants.types.annotations
- u32 from quadrants.types.primitive_types
- static referenced as impl.static (impl is partially initialised at
subgroup.py load time but `impl.static` is resolved at
call time, after impl has finished loading)
Net result: reduce_add and reduce_all_add are first-class
QuadrantsCallable functions, specialised per log2_size value, with the
body fully unrolled by qd.static at AST-rewrite time.
Tests: all 30 reduce_add/reduce_all_add cases and all 49 subgroup+shuffle
tests still pass on CUDA.
…rtInvalidTypeForm suppression
Removes `subgroup.reduce_mul` / `reduce_min` / `reduce_max` / `reduce_and` / `reduce_or` / `reduce_xor` (and their SPIR-V codegen, op registration, type signatures, and Vulkan-only tests). These mirrored `subgroupAdd`, which was already replaced last commit by the portable `subgroup.reduce_add(value, log2_size)` built on `subgroupShuffleDown`. Keeping the other reductions on the SPIR-V-only path was inconsistent and Vulkan-locked; future replacements should follow the sized portable pattern (shuffle tree on top of `shuffle_down` / `shuffle`) when they are needed. Inclusive scans (`subgroupInclusive*`) are left untouched - they are a different primitive (prefix scan, not reduction) and still SPIR-V-only.
Document the portable qd.simt.subgroup.shuffle op: backend support matrix, semantics, examples (broadcast, identity, neighbour swap, per-lane gather), and performance notes. Note that shuffle_up, shuffle_down, and shuffle_xor are not yet implemented across backends. Link from the SIMT primitives section of the user guide index.
Add shuffle_down to the support matrix (CUDA/AMDGPU/SPIR-V), give it its own Semantics subsection, add a tree-reduction example, and note that AMDGPU still goes through ds_bpermute pending a DPP fast path.
The page is the natural home for all qd.simt.subgroup.* docs, not just shuffle, so give it a name that reflects the broader scope.
Extends the subgroup user guide (inherited from hp/shuffle-down) with coverage of the new portable sized reductions: - add rows to the support matrix for `reduce_add(v, log2_size)` and `reduce_all_add(v, log2_size)`; - add semantic sections describing the `log2_size` template contract, the lane-0-only vs all-lanes result guarantees, and the fact that `reduce_add` builds on `shuffle_down` while `reduce_all_add` builds on the native `shuffle` butterfly; - add worked examples: `sum32` using `reduce_add`, `normalize32` using `reduce_all_add`, and a note about partial-subgroup reductions; - note that the SPIR-V-only no-arg reductions (reduce_mul/min/max/and/ or/xor and the old reduce_add) have been removed in favour of the portable sized API; - extend the performance-notes and related sections accordingly. Also renames the page title from "Subgroup shuffles" to "Subgroup primitives" since it now covers shuffles + reductions.
# Conflicts: # docs/source/user_guide/subgroup.md
Extends `test_subgroup_reduce_add` and `test_subgroup_reduce_all_add` from
{i32, f32, f64} to {i32, i64, u64, f32, f64} and updates `_init_field` to
fill 64-bit integer fields with positive integer values. 64-bit ints go
through the same shuffle path (split into two 32-bit shuffles on AMDGPU,
native 64-bit on CUDA) so this just locks the path under the parametrised
exact-equality assertion.
50 reduce-test cases pass on CUDA (5 dtypes x 5 log2_sizes x 2 ops).
Resolve conflict by keeping the shuffle_down content from hp/shuffle-down and applying the unwrap-paragraphs style from hp/shuffle-doc to it as well.
# Conflicts: # docs/source/user_guide/subgroup.md
The helper does not exist under that name (the actual function is group_size()), and either way the underlying subgroupSize op is only wired up in the SPIR-V codegen, so it's not portable across backends. Remove the bullet rather than document a non-portable helper.
Made-with: Cursor # Conflicts: # docs/source/user_guide/subgroup.md
Metal does not support 64-bit integer types in subgroup shuffle operations, causing worker crashes in CI on all Mac test jobs. Made-with: Cursor
MoltenVK does not support 64-bit integer types, causing worker crashes in test_subgroup_reduce_add on Mac CI. Made-with: Cursor
| arch = qd.lang.impl.current_cfg().arch | ||
| if dtype in (qd.i64, qd.u64) and arch == qd.metal: | ||
| pytest.skip("64-bit integer types not supported on Metal") | ||
| if dtype in (qd.i64, qd.u64) and arch == qd.vulkan and platform.system() == "Darwin": | ||
| pytest.skip("MoltenVK does not support 64-bit integer types") |
There was a problem hiding this comment.
Scalar i64 types and arithmetic (long, ulong, int64_t, uint64_t): Available on every M-series chip from M1 onward. MSL has had them since 2.2, and all Apple-silicon Macs meet that. But Apple GPUs have no native 64-bit integer ALU — the compiler lowers these to sequences of 32-bit ops. Typical costs are roughly 2× for add/sub, ~4× for multiply, and noticeably worse for division. Fine for address math, hashing, bit-packing, and any code where i64 is a small fraction of the work. Painful if i64 is in your hot inner loop.
64-bit buffer addressing: All M-series have it; pointers into buffers are effectively 64-bit. This is what lets you cast buffer arguments to device ulong* and index past 4 GB.
erizmr
left a comment
There was a problem hiding this comment.
Approving on the basis that I have reviewed the design and the public facing API, tests, and they look reasonable to me.
Made-with: Cursor # Conflicts: # docs/source/user_guide/subgroup.md # tests/python/test_simt.py
* [Misc] Warn user to disable caching when print_ir/QD_DUMP_IR enabled (Genesis-Embodied-AI#425) Co-authored-by: v01dxyz <v01dxyz@v01d.xyz> * [Build] Pin torch version to CUDA 12.8 for CUDA tests (Genesis-Embodied-AI#428) * [Misc] Fixing up taichi-dev urls (Genesis-Embodied-AI#429) * [Perf] Rename cuda_graph to gpu_graph across the codebase (Genesis-Embodied-AI#430) * Misc: fix typo integeral -> integral (Genesis-Embodied-AI#434) Co-authored-by: v01dxyz <v01dxyz@v01d.xyz> * [Perf] CUDA graph 4: call from multiple locations (Genesis-Embodied-AI#420) * [Bug] Fix fastcache not restoring graph_do_while_arg (Genesis-Embodied-AI#435) * [Perf] Cache last-call result in perf_dispatch for single-compatible case (Genesis-Embodied-AI#438) * Fix gpu_graph fallback on old Nvidia GPU. (Genesis-Embodied-AI#443) * Fix shared memory offset not reset between CUDA kernels. (Genesis-Embodied-AI#442) * [Misc] Allow disabling GPU graph via QD_GPU_GRAPH=0 env var (Genesis-Embodied-AI#439) * [Misc] Add named top-level loops (Genesis-Embodied-AI#440) * [Misc] Rename gpu_graph to graph (Genesis-Embodied-AI#446) * [Misc] Add cross-platform shuffle (Genesis-Embodied-AI#447) * [Bug] Fix graph_do_while on Windows: search for cudadevrt.lib (Genesis-Embodied-AI#456) * [Bug] Also search default CUDA toolkit install location on Windows (Genesis-Embodied-AI#461) * [SPIRV] Feature Parity Atomics & Shared Array (Genesis-Embodied-AI#432) * [Misc] Change clang format to 120 characters (Genesis-Embodied-AI#463) * [Misc] CUDA graph 5 Add fatbin (Genesis-Embodied-AI#464) * [Bug] Reuse VkInstance across init/reset cycles (Genesis-Embodied-AI#465) * [Perf] Tiles 1: _load, _store, _eye_ (Genesis-Embodied-AI#466) * [Misc] Remove dead InternalFuncStmt type_check override (Genesis-Embodied-AI#471) * [Perf] Tiles 2: add cholesky and ger (Genesis-Embodied-AI#472) * [Perf] Tiles 2b: add triangular solve (Genesis-Embodied-AI#474) * [Misc] Refactor: use _get_col/_set_col in tiles load/store/init (Genesis-Embodied-AI#475) * [Build] Fix flaky test_clock_accuracy (Genesis-Embodied-AI#436) * Fix AARCH64 emitting invalid asm in CUDA kernels. (Genesis-Embodied-AI#473) Co-authored-by: Hugh Perkins <hughperkins@gmail.com> * [AMDGPU] Enable HIP memory pool and surface pool-exhaustion errors. (Genesis-Embodied-AI#485) * [AMDGPU] Scope hsaco tmp dir per-user to avoid collisions. (Genesis-Embodied-AI#484) * [Perf] Tiles 3: Add slice syntax, qd.outer() and initial doc (Genesis-Embodied-AI#477) * [AMDGPU] Fix gradient computation. (Genesis-Embodied-AI#486) * Enable all backends that are supported in unit tests. (Genesis-Embodied-AI#488) * Fix SPIRV ID overflow for large kernels due to autodiff. (Genesis-Embodied-AI#489) * [Misc] Fix purity checker to allow accessing constants from quadrants modules (Genesis-Embodied-AI#487) * [Misc] Increase tolerance for clock monotonic test (Genesis-Embodied-AI#492) * [CI] Serialize api doc workflow (Genesis-Embodied-AI#494) * [CI] Increase tolerance for clock test (Genesis-Embodied-AI#506) * [CI] Increase clock test tolerance to 20% (Genesis-Embodied-AI#509) * [Perf] Add tensor_type parametrization to tile16 tests (Genesis-Embodied-AI#504) * [Perf] Tiles 4b: Migrate tiles16 tests to enable fastcache (Genesis-Embodied-AI#505) * [Perf] Tiles 4c: add Tiles16x16 proxy (Genesis-Embodied-AI#507) * [Perf] Tiles 4d: Consolidate slice error tests using parametrize (Genesis-Embodied-AI#508) * [Perf] Tiles 4: add SharedArray slice support (Genesis-Embodied-AI#482) * [Perf] Tiles 5: add Cholesky benchmark demo (Genesis-Embodied-AI#483) * [Doc] Add user guide page for subgroup shuffle (Genesis-Embodied-AI#512) * [Perf] Implement cross-platform shuffle_down (Genesis-Embodied-AI#510) * [Perf] Add portable subgroup reduce_add and reduce_all_add (Genesis-Embodied-AI#511) * [Perf] Add first warmup config to perf dispatch (Genesis-Embodied-AI#422) * [AutoDiff] Autodiff 1: Add baseline adstack regression test for unary_collections (Genesis-Embodied-AI#500) * [AutoDiff] Autodiff 2: Implement derivative for tan (Genesis-Embodied-AI#501) * [AutoDiff] Autodiff 3: Recompute tanh/exp on the operand in the reverse pass (Genesis-Embodied-AI#502) * [AutoDiff] Autodiff 4: Mark rsqrt as non-linear for adstack promotion (Genesis-Embodied-AI#503) * [AutoDiff] Autodiff 5: Fix adjoint-alloca placement for GlobalLoads outside the current range-for (Genesis-Embodied-AI#496) * [AutoDiff] Autodiff 6: Adstack regression tests (Genesis-Embodied-AI#491) * [AutoDiff] Autodiff 7: Fix header size in AdStackAllocaStmt to match u64 runtime layout (Genesis-Embodied-AI#534) * [AutoDiff] Autodiff 8: Surface LLVM adstack push/pop overflow as a Python exception (Genesis-Embodied-AI#535) * [AutoDiff] Autodiff 9: Guard against LLVM worker-thread stack overflow from large per-task adstack budget (Genesis-Embodied-AI#495) * [AutoDiff] Autodiff 10: Implement adstack for SPIR-V (Genesis-Embodied-AI#490) * [AutoDiff] Autodiff 11: Latent adstack-adjacent fixes (AMDGPU hipFree, flush() keeps ctx_buffers_, always-preallocate) (Genesis-Embodied-AI#536) * [Doc] Add AGENTS.md with instructions for AI agents (Genesis-Embodied-AI#541) * [Bug] Abort kernel execution on assertion failure instead of segfaulting (Genesis-Embodied-AI#419) * [Type] ndarray typing 1: Add eval_str=True to inspect.signature() calls (Genesis-Embodied-AI#411) * [CI] Suppress reportPrivateImportUsage in torch-using files (Genesis-Embodied-AI#552) * [Misc] QD_DUMP_IR dumps to files with the task_id added to the filename (Genesis-Embodied-AI#441) * [Type] ndarray typing 2: Fix NDArray single-arg subscript crash (Genesis-Embodied-AI#412) * [Test] Flush xdist channel before worker exit so test failure reports are visible (Genesis-Embodied-AI#555) * [CI] Reduce test retries on CI from 3 to 1. (Genesis-Embodied-AI#554) * [AutoDiff] Autodiff 12: Heap-backed adstack on LLVM backends (CPU/CUDA/AMDGPU) (Genesis-Embodied-AI#537) * [AutoDiff] Autodiff 13: Heap-backed adstack on SPIR-V backends (Metal, Vulkan) (Genesis-Embodied-AI#493) * [AutoDiff] Autodiff 14: Resolve bounded-inner-loop adstacks without default_ad_stack_size fallback (Genesis-Embodied-AI#539) * [SPIRV] Vulkan SPIR-V correctness: atomic-view aliasing, PSB stride, narrow storage caps, u1 cast, per-init layer recheck (Genesis-Embodied-AI#513) * [Build] Autodiff 15: Replace 2022 MoltenVK pin with LunarG Vulkan SDK fetch and sanitise MoltenVK cap advertisement (Genesis-Embodied-AI#551) * [Test] Suppress stock pytest-timeout to avoid conflict with pytest_hardtle (Genesis-Embodied-AI#557) * [Vulkan] Use SDK validation layer for debugPrintf instead of apt package (Genesis-Embodied-AI#562) * [Test] Fix flaky perf_dispatch tests by increasing work amounts (Genesis-Embodied-AI#559) * [Test] Add --maxfail CLI option to run_tests.py (default 20) (Genesis-Embodied-AI#558) * [CI] Vulkan debug printf fix to address flaky tests (Genesis-Embodied-AI#563) * [Docs] Add a new page to help for first time contributors (Genesis-Embodied-AI#426) Authored-by: v01dxyz <v01dxyz@v01d.xyz> * [AutoDiff] Autodiff 16: Resolve reverse-mode adstack depths per-launch via runtime-evaluated SizeExpr (Genesis-Embodied-AI#543) * Fix: raise error if device memory allocation fails (Genesis-Embodied-AI#451) (Genesis-Embodied-AI#453) Co-authored-by: v01dxyz <v01dxyz@v01d.xyz> Co-authored-by: Hugh Perkins <hughperkins@gmail.com> * [CI] Add CI job to check line wrapping of comments and docs (Genesis-Embodied-AI#564) * [Misc] Add coverage report to PRs, including kernels (Genesis-Embodied-AI#470) * [CI] CI wrap check feeds only diffs to agent (Genesis-Embodied-AI#567) * Skip 'flaky' test on MacOS CI. (Genesis-Embodied-AI#573) * [Test] Fix missing `import sys` in test_fail_device_memory_allocation (Genesis-Embodied-AI#574) * [CI] Fix Vulkan debugPrintf flake with session-scoped warmup (Genesis-Embodied-AI#571) * [AutoDiff] determine_ad_stack_size: replace whole-CFG Bellman-Ford with SCC + DAG DP (Genesis-Embodied-AI#575) * [Test] Fix macOS OOM skip reason to describe actual root cause (Genesis-Embodied-AI#576) * [Lang] whole_kernel_cse: 2.5x compile time speedup on large kernels (Genesis-Embodied-AI#577) * [CI] Add CI check for unnecessarily deleted comments (Genesis-Embodied-AI#570) * [CI] Migrate coverage report to github Check page (Genesis-Embodied-AI#566) * [Lang] Skip IR verifier between passes unless debug=true (Genesis-Embodied-AI#579) * [Lang] Inline AdStack ops on release LLVM codegen: dramatically reduces compile time for adstack-enabled reverse-mode kernels (Genesis-Embodied-AI#584) * [CUDA] Honor offline_cache=False end-to-end so QD_OFFLINE_CACHE=0 actually gives a cold compile (Genesis-Embodied-AI#580) * [Type] Tensor 24 (Genesis-Embodied-AI#561) Co-authored-by: hugh <hugh@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local> * [Lang] auto_diff host-walk reductions: dramatically faster front-end compile time on adstack-enabled reverse-mode kernels (Genesis-Embodied-AI#587) * [AutoDiff] Speed up reverse-mode kernel launches on GPU backends (Genesis-Embodied-AI#578) * [Vulkan] Move adstack-sizer scratch out of Function-scope memory to fix SPIR-V pipeline build failures (Genesis-Embodied-AI#588) * [AutoDiff] Improve diagnosis of unsupported reverse-mode AD patterns (Genesis-Embodied-AI#590) * [Bug] Fix: promote Ndarray to AnyArray in build_Name for flattened struct fields (Genesis-Embodied-AI#592) * [SPIR-V] Shrink reverse-grad kernel MSL by ~50% (Genesis-Embodied-AI#591) * [CI] Add CI check that PR changes have test coverage (Genesis-Embodied-AI#596) * [Perf] Enable zero-copy in to_torch() and to_numpy() (Genesis-Embodied-AI#450) * Add BufferView: safe sub-range ndarray access for kernels (Genesis-Embodied-AI#585) Co-authored-by: alanray-tech <alanray-tech@users.noreply.github.com> Co-authored-by: Hugh Perkins <hughperkins@gmail.com> * [Doc] Add user-facing fastcache documentation (Genesis-Embodied-AI#597) Co-authored-by: hugh <hugh@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local> * [Misc] Upgrade to enable v1 dlpack so to_numpy(copy=False) writable (Genesis-Embodied-AI#598) Co-authored-by: root <root@rtx-209-201.slurm-compute.tenant-slurm.svc.cluster.local> * [AutoDiff] Cut reverse-mode adstack memory usage 10x on all backends (Genesis-Embodied-AI#599) * [Misc] Add CI check for feature file factorization (Genesis-Embodied-AI#606) * [Perf] Skip _recursive_set_args for all-Field frozen dataclass structs (Genesis-Embodied-AI#607) Co-authored-by: Cursor <cursoragent@cursor.com> * [AutoDiff] SNode-arm bound-expr capture rejects fold-attack gate indices (Genesis-Embodied-AI#610) * [Misc] Suppress field fastcache warning for qd.Tensor (Genesis-Embodied-AI#615) Co-authored-by: Cursor <cursoragent@cursor.com> * [AutoDiff] Adstack heap: clip reducer count by per-task loop trip count (compile-time and SizeExpr-evaluated) (Genesis-Embodied-AI#611) * [Misc] Forward copy= through qd.Tensor, add copy=None option (Genesis-Embodied-AI#616) Co-authored-by: Cursor <cursoragent@cursor.com> * [Doc] Update README (Genesis-Embodied-AI#617) Co-authored-by: Cursor <cursoragent@cursor.com> * [CI] Fix coverage report showing def lines as uncovered (Genesis-Embodied-AI#623) Co-authored-by: Cursor <cursoragent@cursor.com> * [Perf] Generic launcher: persistent context, JIT-pointer reuse, Metal compute encoder, LLVM-GPU async memory ops (Part 1/2) (Genesis-Embodied-AI#619) * [CI] Encode Python-first testing policy in coverage-check prompt (Genesis-Embodied-AI#622) Co-authored-by: Cursor <cursoragent@cursor.com> * [CI] Add PR Line change report (Genesis-Embodied-AI#624) Co-authored-by: Cursor <cursoragent@cursor.com> * [CI] Disable quadrants pytest plugin during quadrants internal coverage runs (Genesis-Embodied-AI#629) Co-authored-by: Cursor <cursoragent@cursor.com> * [AutoDiff] Adstack load+store eliminations: EliminateRecomputableAdStackPushes pass + leaf extensions (Genesis-Embodied-AI#621) * [CI] Simplify coverage PR comment to a single linked line (Genesis-Embodied-AI#630) * [CUDA] Add AGX Thor, SM_110 (Genesis-Embodied-AI#631) Co-authored-by: Johnny Nunez and Hugh Perkins * [CI] Lines changed report: collapse PR comment to a single linked totals line (Genesis-Embodied-AI#632) * [FEATURE] Support external Metal command queue via qd.init (Genesis-Embodied-AI#618) Co-authored-by: Cursor <cursoragent@cursor.com> * [Perf] Cache adstack-sizer metadata per task across SPIR-V + LLVM-GPU; per-snode / DeviceAllocation invalidation (Part 2/2) (Genesis-Embodied-AI#620) * [AutoDiff] Disable EliminateRecomputableAdStackPushes pending mutated-SNode chain-leaf fix (Genesis-Embodied-AI#633) * [AutoDiff] Adstack chain-clone safety: mutated-SNode leaf reject + load_top consumer-aware guard (Genesis-Embodied-AI#634) * [Docs] Add user-guide page for qd.simt.block.* primitives (Genesis-Embodied-AI#638) * [Docs] Expand qd.simt.subgroup user-guide page to cover every op (Genesis-Embodied-AI#639) * [Perf] Streams 1-4 (Genesis-Embodied-AI#410) * [Docs] Add user-guide page for matrix decompositions and solvers (Genesis-Embodied-AI#643) * [Bug] Revert "[Perf] Streams 1-4 (Genesis-Embodied-AI#410)" (Genesis-Embodied-AI#650) * [Docs] Add user-guide page for atomics and bit operations (Genesis-Embodied-AI#640) * [Docs] Add user-guide page for qd.simt.grid.* primitives (Genesis-Embodied-AI#641) * [AutoDiff] Adstack max-reducer: parallel multi-axis MaxOverRange dispatch (Genesis-Embodied-AI#635) * [AMDGPU] Fix amdgpu parallel rand init (Genesis-Embodied-AI#658) * [Perf] Adstack: skip max-reducer recognizer on CPU + lift host-eval cap (Genesis-Embodied-AI#655) * [Perf] Re-land Streams 1-4 with bug fixes (Genesis-Embodied-AI#653) * [AMDGPU] Apply device_memory_GB=0.3 cap to AMDGPU tests (Genesis-Embodied-AI#659) * [Perf] Per-launch host sync: drop wait_idle on SPIR-V, pin stream and drop stream_synchronize on CUDA/AMDGPU (Genesis-Embodied-AI#654) * [AMDGPU] Unload hipModule_t in JITModuleAMDGPU destructor (Genesis-Embodied-AI#660) * [AMDGPU] Trim default mempool on qd.reset() (Genesis-Embodied-AI#669) * [AMDGPU] Hoist rand-state buffer to process lifetime (Genesis-Embodied-AI#668) * [Streams] Use events for streams serialization on AMDGPU and CUDA (Genesis-Embodied-AI#667) * [Perf] Adstack max-reducer: launch cache + zero-copy result map; content-stable registry_id (Genesis-Embodied-AI#671) * [SPIR-V] dispatch_max_reducers: register each task with the real kernel name (Genesis-Embodied-AI#675) * [AutoDiff] Debug-mode field/grad/dual: dtype, layout, and access-time invariants (Genesis-Embodied-AI#677) * [Docs] Add user-guide page for qd.algorithms.* device-wide algorithms (Genesis-Embodied-AI#642) Co-authored-by: alanray-tech <alan.ray@genesis-ai.company> * [Docs] Doc for existing atomics: switch support table to per-backend columns (Genesis-Embodied-AI#657) Co-authored-by: alanray-tech <alan.ray@genesis-ai.company> * [GPU] Cross gpu atomics (Genesis-Embodied-AI#666) Co-authored-by: alanray-tech <alan.ray@genesis-ai.company> * [GPU] Make block operations portable cross-gpu (Genesis-Embodied-AI#664) * [Perf] CPU LLVM adstack-cache: skip per-launch bump-writes + ndarray_shapes capture on forward-only handles (Genesis-Embodied-AI#685) * [GPU] Cross-GPU for grid ops (Genesis-Embodied-AI#670) * [Math] Make bitop operations portable cross-gpu (Genesis-Embodied-AI#662) * [AMDGPU] Always use wave64, on both RDNA and CDNA (Genesis-Embodied-AI#687) * [AMDGPU] Use syncscope("agent") for atomix xor to avoid CAS livelock (Genesis-Embodied-AI#672) * [GPU] New bit ops for QIPC (Genesis-Embodied-AI#679) * [GPU] Subgroup ops cross-gpu (Genesis-Embodied-AI#665) * [Graph] Rename CUDA Graph to Graph in docs (Genesis-Embodied-AI#691) * [SPIR-V] Fix FIFO-queue ordering when sharing command queue. (Genesis-Embodied-AI#694) * [Atomics] New QIPC ops for atomics (Genesis-Embodied-AI#690) * Pass dataclass sub-structs into qd.func (Genesis-Embodied-AI#698) * [AMDGPU] HIP graph runtime support for @qd.kernel(graph=True) (Genesis-Embodied-AI#692) * [CI] Add per-file timing report to Mac Metal test job (Genesis-Embodied-AI#695) Co-authored-by: Cursor <cursoragent@cursor.com> * [CI] Enable kernel disk cache during tests (Genesis-Embodied-AI#696) * [Math] New QIPC ops for single-threaded linalg (Genesis-Embodied-AI#683) * [BREAKING][GPU] New QIPC ops for subgroups (Genesis-Embodied-AI#676) * [GPU] New QIPC ops for block (Genesis-Embodied-AI#684) * [GPU] New device-level ops for QIPC (Genesis-Embodied-AI#693) * [algorithms] PrefixSumExecutor: drop unused GRID_SZ local (Genesis-Embodied-AI#701) * [block] sync(): fix unsupported-arch error message (Genesis-Embodied-AI#700) * [volatile_load] add qd.volatile_load primitive (closes Genesis-Embodied-AI#648) (Genesis-Embodied-AI#702) * [AutoDiff] Reject recycled identity_key in AdStackCache::register_adstack_sizing_info (Genesis-Embodied-AI#708) * [Vulkan] Declare GroupNonUniform SPIR-V caps and enable shaderSubgroupExtendedTypes (Genesis-Embodied-AI#707) * Fix duplicate HIP graph driver-function declarations after v1.0.0 merge The amd-integration fork had cherry-picked the HIP graph driver functions (graph_create / graph_destroy / graph_add_kernel_node / graph_instantiate / graph_exec_destroy / graph_launch), and upstream v1.0.0 added the same set. The per-file 3-way merge appended both copies into amdgpu_driver_functions.inc.h, producing redeclaration errors that broke the AMDGPU RHI/runtime compile. Drop the upstream duplicate block; the signatures are identical to the fork's existing declarations. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix AMDGPU launcher coherence and num_instructions visibility after v1.0.0 merge - kernel_launcher.cpp: the 3-way merge spliced upstream v1.0.0's launch_llvm_kernel rewrite (ephemeral arg/context buffers, explicit-stream path, AmdgpuDefaultStream PinGuard) onto the AMD fork's kernarg-by-value + persistent-scratch design, leaving references to undefined `ephemeral_context_ptr`. Restore the fork's coherent launch_llvm_kernel verbatim; it calls the (already merged) enhanced launch_offloaded_tasks, which keeps the max-reducer dispatch and stream-parallel groups adapted onto the AMD launch path. - llvm_context.h: both the fork and upstream added `num_instructions`; the merge kept upstream's private placement, but the AMDGPU codegen force-inline heuristic calls it statically from outside the class. Move it back to the public section. Co-authored-by: Cursor <cursoragent@cursor.com> * Restore async result D2H and hoist kernarg vectors in AMDGPU launcher The v1.0.0 merge resolution regressed two amd-integration baseline optimizations in launch_llvm_kernel / launch_offloaded_tasks: - The per-launch result-buffer copy was a blocking memcpy_device_to_host, forcing a host stall on every value-returning launch and serializing the GPU pipeline. Restore the async D2H (the caller synchronizes lazily when it needs the value); external-array transfers still stream_synchronize once before reading back. - launch_task constructed the kernarg std::vectors from initializer lists ({kernarg_payload} / {kernarg_size}) on every dispatch (heap alloc + free per launch). Hoist arg_ptrs/arg_sizes out of the per-task launch and reuse. Co-authored-by: Cursor <cursoragent@cursor.com> * amdgpu: default to LDS permlane64 emulation; drop host-x86 barrier asm on retarget Two AMDGPU JIT-compile crashes surfaced after the v1.0.0 merge pulled in the QIPC subgroup ops (Genesis-Embodied-AI#676), which made the rigid constraint solver's wave-cooperative reductions route through `amdgpu_cross_half_shuffle_i32`. Both manifested as a SIGSEGV inside `llvm::SIInstrInfo::getInstSizeInBytes` during `JITSessionAMDGPU::compile_module_to_hsaco` (i.e. at first kernel launch), and reproduce on gfx942 / MI300X. Baseline 0.4.6 never emitted these constructs, which is why it was unaffected. 1. Native `llvm.amdgcn.permlane64` lowering crashes the bundled LLVM 22.1.0 AMDGPU backend. Default `amdgpu_permlane64` to the existing LDS-roundtrip software emulation on every target (it produces identical results). Add `QD_AMDGPU_USE_NATIVE_PERMLANE64=1` to opt back into the native instruction once the backend bug is fixed; the old `QD_AMDGPU_FORCE_PERMLANE64_FALLBACK` is now the default and still honored. This is the actual crash fix. 2. The runtime module is compiled by the host x86_64 clang and only retargeted to amdgcn here, so `amdgpu_cross_half_shuffle_i32`'s `__asm__ volatile("" : "+v"(byte))` optimization barrier carries x86 flag clobbers (`~{dirflag},~{fpsr},~{flags}`) that are meaningless on AMDGPU. The IR verifies but the empty-body INLINEASM is invalid on the amdgcn target. Neutralize empty-body barrier asm during retarget (forward the tied value, then erase) so no stale host asm reaches codegen. On the wave64 targets we ship `ds_bpermute` already addresses the full wave, so the hint is a no-op. Co-authored-by: Cursor <cursoragent@cursor.com> * style: apply clang-format (v19.1.7) to AMDGPU fn_attrs and launcher sources CI pre-commit's clang-format hook reformatted these files (long declarations/lambda signatures collapsed onto single lines per the repo's clang-format config). Apply the same formatting so the hook passes. No functional changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(amdgpu): use CreateNeg for branchless i32 sgn instead of CreateSub(0, input) clang-tidy (modernize-use-nullptr, -warnings-as-errors) flagged `builder->CreateSub(0, input)` in the i32 sgn path: the literal `0` binds to the `llvm::Value*` LHS parameter as a null pointer, not an integer zero. Replace with `builder->CreateNeg(input)`, which emits `0 - input` with a proper zero constant -- identical intended semantics, and clang-tidy clean. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Robert Dazi <14996868+v01dXYZ@users.noreply.github.com> Co-authored-by: v01dxyz <v01dxyz@v01d.xyz> Co-authored-by: Hugh Perkins <hughperkins@gmail.com> Co-authored-by: Alexis DUBURCQ <alexis.duburcq@gmail.com> Co-authored-by: hugh <hugh@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local> Co-authored-by: alanray-tech <alan.ray@genesis-ai.company> Co-authored-by: alanray-tech <alanray-tech@users.noreply.github.com> Co-authored-by: root <root@rtx-209-201.slurm-compute.tenant-slurm.svc.cluster.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Johnny <johnnynuca14@gmail.com>
Portable subgroup
reduce_add/reduce_all_add(replaces SPIR-V-only no-arg reductions)Builds on
hp/shuffle-down's portablesubgroup.shuffle_downto add cross-platformsum reductions, and removes the SPIR-V-only no-arg subgroup reductions for
consistency.
What's in this PR
New API: portable sized sum reductions
subgroup.reduce_add(value, log2_size)— sumsvalueacross2**log2_sizeconsecutive lanes via a
shuffle_downtree. Result valid in lane 0 of eachgroup; other lanes hold partial sums.
subgroup.reduce_all_add(value, log2_size)— same sum, broadcast to everylane via a
shufflebutterfly (lane ^ mask).@qd.funcand inlined;log2_sizeis aqd.template()(compile-time constant) so the body unrolls into exactlylog2_sizeshuffle+add pairs in the calling kernel's IR.2**log2_sizemust not exceed the active subgroup size(32 on CUDA / Metal / RDNA, 64 on CDNA).
Removed: SPIR-V-only no-arg reductions
subgroup.reduce_add(value)(nolog2_size),reduce_mul,reduce_min,reduce_max,reduce_and,reduce_or,reduce_xor— all gone.subgroupAdd/subgroupMul/ ... ops frominternal_ops.inc.h, theirPOLY_OPsignatures intype_system.cpp, andthe
reduction_opsset inspirv_codegen.cpp(the dispatch branch nowhandles
inclusive_scan_opsonly, withgroup_opfixed toInclusiveScan).subgroup.inclusive_*) are intentionally untouched — theyare a different primitive (prefix scan, not reduction) and remain
SPIR-V-only pending their own portable replacement.
Tests
test_subgroup_reduction_*tests and the_test_subgroup_reducehelper.test_subgroup_reduce_addandtest_subgroup_reduce_all_addcoveringlog2_size ∈ {1, 2, 3, 4, 5}anddtypes
{i32, f32, f64}onarch=qd.gpu(so all backends).Docs
docs/source/user_guide/subgroup.md(renamed page title to"Subgroup primitives") with: support-matrix rows for both new ops,
semantics sections covering the
log2_sizetemplate + size-cap contractand the lane-0-only vs all-lanes guarantees, worked
sum32/normalize32/ partial-subgroup examples, performance notes, and a note explaining that
the SPIR-V-only no-arg reductions have been removed.
# type: ignoreonsubgroup.pywith a targeted# pyright: reportInvalidTypeForm=false(needed for the
template()annotation form, mirroring_tile16.py).Good points
more
arch=qd.vulkancarve-outs in callers —reduce_addruns on CUDA,AMDGPU, and SPIR-V from the same source.
@qd.funcinlining + staticrangeunrolling means the generated IR is byte-equivalent to ahand-written shuffle tree — no extra calls, no runtime loop.
reduction width via
log2_size, andreduce_all_add(butterfly) saves afollow-up broadcast when every lane needs the result.
internal ops table, the SPIR-V
reduction_opsset, and the type system.# type: ignoreonsubgroup.pyis gone; only the one
template()-related warning is suppressed.log2_sizeand dtypes, replacing twoVulkan-only tests with portable coverage.
Bad points / caveats
subgroup.reduce_add(value)(no second arg) andreduce_mul/min/max/and/or/xorare gone — anycaller has to switch to either
reduce_add(value, log2_size)or buildtheir own sized helper on top of
shuffle_down/shuffle. (Codebaseaudit showed no in-tree callers; external users will need to adapt.)
reduce_mul/min/max/and/or/xoryet. ThisPR removes the SPIR-V-only path for consistency but only ships the
addvariants. If you need product/min/max reductions on CUDA or AMDGPU
today, you have to write the shuffle tree yourself for now.
reduce_addperformance is currently middling. It rides onshuffle_down, which on AMDGPU goes throughds_bpermute(LDS-routed,tens of cycles). A native DPP fast path is a future PR.
reduce_all_addis unaffected — it only usesshuffle, which is nativeon AMDGPU.
log2_size=6on a32-lane subgroup compiles fine and silently produces wrong results in the
lanes that read past the end — only documented in the docstring, not
asserted. Could add a compile-time check later.
Tests don't cover i64 / u64. Parametrisation is[fixed]{i32, f32, f64};64-bit integer reductions go through the same shuffle path but are
untested in this PR.
inclusive_*scans are still SPIR-V-only — they are out of scope herebut the inconsistency is now more visible (sum reduction is portable,
sum prefix-scan is not).
Risk
group_opsimplification in
spirv_codegen.cpp. Fulltests/python/test_simt.pypasses (69 passed, 1 skipped) and pyright is clean (0/0/0) on
subgroup.py.to break out-of-tree consumers.
Follow-ups
reduce_mul/reduce_min/reduce_max(and integerreduce_and/or/xor) using the samelog2_sizetemplatepattern, when needed.
ROW_SHRfast path forshuffle_down(will acceleratereduce_addon AMDGPU automatically).Issue: #
Brief Summary
copilot:summary
Walkthrough
copilot:walkthrough