perf: codegen, llvm, host_api#1
Merged
Merged
Conversation
rtmadduri
approved these changes
Apr 17, 2026
rtmadduri
left a comment
Collaborator
There was a problem hiding this comment.
Has been tested and benchmarked! LGTM!
Closed
npoulad1
added a commit
that referenced
this pull request
May 9, 2026
…eline) Closes the remaining ~12pp post-merge throughput gap on the Genesis G1 8192-env benchmark by removing the only AMDGPU pipeline change that the LLVM-22 upgrade actually destabilized. Diagnosis. After committing #1 (RuntimeContext alignment) and #2 (range_for body / SNode child accessor inlining), the benchmark sat at 1.57M env·steps/s vs. a 1.79M pre-merge baseline. A symbol-level diff between the post-fix wheel and the last-known-good `wheels_align_inline_fix` wheel (which was actually built before sccache served the current jit_amdgpu.cpp.o) showed the pre-merge wheel was missing every `AMDGPUFlatToGlobalLoadStorePass` symbol — i.e. it had never run that pass, full stop. The wheel built from current source crashed inside `qd.init(arch=qd.amdgpu)` with EXIT=139 in `compile_module_to_hsaco` running the runtime bitcode. Root cause. The pass's `originatesFromScratch` walks `Argument` values back through `Function::users()` to inspect each direct caller, recursing into the corresponding `CallBase::getArgOperand(ArgNo)`. Each crossing of a caller boundary resets the visited-set (line 165, `CallerVisited`), so cycles through the `runtime_*` / `LLVMRuntime_*` call graph aren't broken. The pre-LLVM-22 runtime bitcode happened to keep the recursion shallow; post-LLVM-22 (with kernarg-by-value RuntimeContext, more internal helper functions, and a different inlining shape) the same pass blows the stack on init. Disabling the pass entirely is not an option — without it Genesis's solver kernels emit wrong addresses on the constraint-force path and the simulation poisons with NaN. Fix. Skip `AMDGPUFlatToGlobalLoadStorePass` only on the runtime bitcode module — detected by the unique presence of `runtime_initialize` — and keep running it on every user-kernel module. The runtime functions are inlined into user kernels at user-kernel JIT time, so all their loads/stores still get the flat→global lowering, just in the right module shape (the user kernel is well-formed; the runtime BC is a graph of mutually-recursive helpers). Validated end-to-end on Genesis G1 8192-env benchmark (CG, 15 iters, 500 steps, MI300X, FP32): pre-merge baseline: 1,790,770 env·steps/s post-merge, no fixes: 1,386,055 env·steps/s (-22.6%) fixes #1+#2 only: 1,569,491 env·steps/s (-12.4%) fixes #1+#2 + this commit: 1,791,235 env·steps/s (+0.03%) Recovers the full 22.6pp regression. The previously-suspected "kernel function attributes (uniform-work-group-size, waves-per-eu, fast-math) flipped under LLVM 22" follow-up is no longer a gap to close — `compile_module_to_hsaco` already reapplies those defaults to AMDGPU_KERNEL functions (lines 75-86), and post this commit the benchmark sits on the baseline regardless. Leaving them as-is. Co-authored-by: Cursor <cursoragent@cursor.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_output.log
Summary
This PR delivers a set of performance-critical changes across the AMDGPU codegen, JIT compilation pipeline, kernel launch path, and RHI driver layer. The optimizations target instruction selection quality, memory access efficiency, kernel launch overhead, and compilation caching to improve end-to-end GPU kernel performance on AMDGPU hardware.
Changes
GetRootStmt, SNodeLookupStmt, GetChStmt: SNode root pointers from hipMalloc'd memory are now cast to addrspace(1) (global), ensuring LLVM emits global_load/global_store instructions instead of flat_load/flat_store. This avoids the FLAT unit's runtime address-space resolution overhead.
GlobalLoadStmt / GlobalStoreStmt: Overridden to cast pointers to addrspace(1) before load/store, with invariant.load metadata applied for read-only SNode fields.
get_runtime() / create_intrinsic_load(): Overridden to load runtime metadata through addrspace(1) with invariant-load hints for read-only caching.
New AMDGPUFlatToGlobalLoadStorePass: Post-optimization pass that converts remaining addrspace(0) (flat) loads/stores to addrspace(1) (global) by tracing pointer origins — accesses derived from allocas or scratch memory (addrspace 5) are left as flat.
Replaced the branch-and-alloca-based sgn codegen for f32/f64 with a branchless select-based implementation. Eliminates private scratch memory usage and control-flow divergence.
Large shared arrays exceeding cuda_dynamic_shared_array_threshold_bytes are now promoted to dynamically-sized LDS allocations (addrspace(3)) via GlobalVariable, matching the CUDA backend's behavior.
Implemented create_bls_buffer() for AMDGPU using LDS global variables (addrspace(3)).
Enabled the bls extension for Arch::amdgpu in the extension registry.
AMDGPU kernels now receive the RuntimeContext struct by value in kernarg instead of through an indirection pointer (hipMalloc + memcpy). This eliminates a device-side memory allocation and a host-to-device copy per kernel launch.
Added kernel_argument_struct_in_kernarg() virtual override and supporting context_val_alloca_ plumbing in the base TaskCodeGenLLVM.
Added hipMallocAsync/hipFreeAsync support with automatic fallback to synchronous hipMalloc/hipFree when the runtime doesn't support memory pools.
Probes hipDeviceGetDefaultMemPool at init and configures a 128 MB release threshold via hipMemPoolSetAttribute.
Kernel launcher and device allocator updated to use async allocation paths when available.
HSACo binaries are now cached in-memory keyed by an MD5 hash of the LLVM module bitcode. Repeated compilations of the same module skip the full ISA compilation pipeline and reuse the cached binary.
unsafe-fp-math=true and no-signed-zeros-fp-math=true applied to all functions (matching CUDA's unconditional FTZ behavior).
Kernel functions annotated with amdgpu-waves-per-eu=1,2, uniform-work-group-size=true, amdgpu-ieee=false, amdgpu-dx10-clamp=false.
__oclc_daz_opt=1 global variable emitted for denormals-as-zero.
FPOpFusion::Fast set unconditionally (not gated behind fast_math).
amdgpu-flat-work-group-size attribute set on kernels with known block dimensions.
Added post-pipeline passes: LoopStrengthReduce, SeparateConstOffsetFromGEP, and EarlyCSE (with MemorySSA) to improve address calculation and reduce register pressure.
Profiler now correctly passes dynamic_shared_mem_bytes to trace calls.
saturating_grid_dim calculation corrected (removed erroneous * 2 multiplier).
amdgpu_auto_waves_per_eu config option added with Python binding.
MAX_THREADS_PER_MULTIPROCESSOR device attribute query for accurate listgen grid sizing.