HSA fault on 8xMI300X all_gather at msg_per_rank >= 768 MiB (bf16, both persistent and partitioned variants)

[ryanswann-amd]

iris HSA fault on 8×MI300X all_gather at msg ≥ 768 MiB (bf16, persistent variant)

iris SHA: 9459a5e95d01ac50bd072b9c6c498a5c8f96af16
ROCm: 7.2 (rocm/pytorch:rocm7.2_ubuntu24.04_py3.12_pytorch_release_2.10.0)
Hardware: 8× MI300X, single node (multiple nodes tested: a04u31, h07u13, e05u43, e06u37, j07u37, g08u07, c09u37)
Cluster: c42

Symptom

iris.ccl.all_gather issued from a minimal 8-rank torchrun program with bf16 input shape (M, 8192) and output shape (8M, 8192) reproducibly aborts with:

Memory access fault by GPU node-6 (Agent handle: 0x3de64ff0) on address 0x7a733b300000. Reason: Write access to a read-only page.
Memory access fault by GPU node-8 (Agent handle: 0x169b3f00) on address 0x7e7477300000. Reason: Write access to a read-only page.
Memory access fault by GPU node-7 (Agent handle: 0x34576640) on address 0x723c3f310000. Reason: Write access to a read-only page.
GPU core dump failed
torch.distributed.elastic.multiprocessing.api: failed (exitcode: -6) local_rank: 4 ... Signal 6 (SIGABRT)

The fault hits ranks 4-7 (second NUMA quadrant) on every reproduction.

Threshold (8 trials, single GPU node, single iris-only repro script)

msg per rank	heap	ag_variant	result	failure step
256 MiB	8 GiB	persistent	PASS	4/4 iters OK
512 MiB	32 GiB	persistent	PASS	4/4 iters OK (ag_in=512MB, ag_out=4GB)
768 MiB	32 GiB	persistent	FAIL	warmup AG kernel
1024 MiB	32 GiB	persistent	FAIL	iris init / warmup AG kernel
1024 MiB	64 GiB	persistent	FAIL	warmup AG kernel (allocs OK)
1024 MiB	96 GiB	persistent	FAIL	warmup AG kernel (allocs OK)
1024 MiB	32 GiB	partitioned	FAIL	warmup AG kernel
1024 MiB	64 GiB	partitioned	FAIL	warmup AG kernel

Threshold lies between 512 MiB and 768 MiB per-rank input, regardless of heap size (32/64/96 GiB) or AG variant (persistent / partitioned). Both variants fail at the same threshold.

Per-rank ag_out at 768 MiB input = 6 GiB; at 1 GiB input = 8 GiB. Heap of 32-96 GiB is more than sufficient for the buffer footprint.

Minimal repro

iris_only_repro.py (no bench scaffolding, no concurrent legs, no streams):

import os, torch, torch.distributed as dist, iris
from iris.ccl import Config

torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="gloo")

ctx = iris.iris(heap_size=32 * (1 << 30))
rank = ctx.get_rank(); world = ctx.get_num_ranks()

# 1 GiB input per rank, output is 8 GiB per rank (full all-gather)
M, N = 65536, 8192
ag_in = ctx.zeros((M, N), dtype=torch.bfloat16); ag_in.fill_(float(rank + 1))
ag_out = ctx.zeros((world * M, N), dtype=torch.bfloat16)

cfg = Config(block_size_m=64, block_size_n=128, comm_sms=32,
             num_stages=1, num_warps=4, waves_per_eu=0,
             all_gather_variant="persistent")

ctx.ccl.all_gather(ag_out, ag_in, async_op=True, config=cfg)  # FAULTS HERE
torch.cuda.synchronize()

Launch:

torchrun --nproc-per-node=8 iris_only_repro.py

Repro file: scripts/iris_only_repro.py in this workspace. Trial logs in logs/repro_*.log.

Root cause hypothesis

"Write access to a read-only page" suggests the AG kernel is attempting to write to a remote rank's input buffer (which is mapped read-only in the symmetric heap of the writer's HSA agent), or to a workspace page that the runtime mapped read-only by mistake. The fault hitting ranks 4-7 (one NUMA quadrant) is consistent with a per-quadrant peer mapping issue at the GCD page-table level, but we have not narrowed it further.

The 512→768 MiB threshold lines up with ag_out crossing 4 GiB → 6 GiB per rank, suggesting the bug is in how the variant computes addresses / partitions when the symmetric output exceeds 4 GiB.

Impact

Blocks K-727 (V5b N=4 BC(GET) gap closure) and likely any iris workload that needs AG ≥ 768 MiB on 8× MI300X. K-727 has worked around by capping AG-included cells at the 256 MiB / 512 MiB regime; the no-AG cell (bc_get_ar_rs_a2a) ran cleanly at 1 GiB.

Workspace evidence

• Logs: /home/ryaswann/mc2/K-727/logs/repro_*.log (8 configs)
• JSONs: /home/ryaswann/mc2/K-727/output/repro_*.json
• Workspace: /home/ryaswann/mc2-workspaces/K-727/