GPE in prove_execution only when opt-level="z" AND codegen-units=1 (x86_64 Linux)

[ch4r10t33r]

Filing this mostly for visibility. We've been chasing a deterministic General Protection Exception on x86_64-unknown-linux-gnu that only reproduces when a downstream consumer builds leanMultisig under opt-level = "z" and codegen-units = 1 simultaneously. Either knob on its own is fine. The crash lands inside rec_aggregation::xmss_aggregate → lean_prover::prove_execution → prove_generic_logup → Poseidon16Precompile::bus.

Our downstream (blockblaz/zeam) already shipped a workaround in #759 by switching to opt-level = "s", so we're unblocked. But the interesting part sits in leanMultisig, so I wanted to write it up here.

What actually faults

Debug symbols point at Poseidon16Precompile::bus, but the real body at the faulting PC is a monomorphization of lean_vm::tables::utils::eval_virtual_bus_column that got inlined into lean_prover's CGU:

pub(crate) fn eval_virtual_bus_column<AB: AirBuilder, EF: ExtensionField<PF<EF>>>(
    extra_data: &ExtraDataForBuses<EF>, flag: AB::IF, data: &[AB::IF],
) -> AB::EF {
    let (logup_alphas_eq_poly, bus_beta) = extra_data.transmute_bus_data::<AB::EF>();
    assert!(data.len() < logup_alphas_eq_poly.len());
    (logup_alphas_eq_poly.iter().zip(data).map(|(c, d)| *c * *d).sum::<AB::EF>()
     + *logup_alphas_eq_poly.last().unwrap() * AB::F::from_usize(LOGUP_PRECOMPILE_DOMAINSEP))
     * *bus_beta + flag
}

Twelve lines of safe Rust, but at the crash site objdump shows a single basic block keeping all 16 YMM registers live through vpblendd/vpbroadcastd/vpmuludq/vinserti128/vpaddq, and dying on a vpinsrd $0x1, 0x14(%rsp), %xmm5, %xmm5 — a stack-relative reload of a spilled vector lane. The whole iterator chain has been inlined into one monstrous SIMD fold, and at -Oz the frame layout for that fold looks wrong. Bumping CGU to ≥ 2 keeps the function outlined at the crate boundary, which is why that mitigation works.

(For what it's worth: the unsafe transmute in ExtraDataForBuses::transmute_bus_data isn't the culprit here — at this monomorphization AB::EF == EF so it's an identity transmute. nm confirms the affected symbol only exists in lean_vm's CGU. Still a pattern I'd love to see go away for readability reasons, but it's sound.)

What we tried

Bisected on an AMD Zen 4 VM with stable rustc 1.95.0. Everything below runs against leanMultisig rev 2eb4b9d983171139af36749f127dd9890c9109e6:

• Per-crate opt-level = "s" overrides across mt-*, rec_aggregation, backend, lean_prover, lean_vm, utils, sub_protocols — none of those combos fixed it. A lean_compiler-only override did, but it turned out to be a CGU-partitioning side-effect that's unstable under small cache changes.
• RUSTFLAGS="-Cllvm-args=-enable-machine-outliner=never" — no effect.
• #[inline(never)] on eval_virtual_bus_column itself — no effect.
• codegen-units = 16 with opt-level = "z" — clean.
• codegen-units = 1 with opt-level = "s" — clean.

So the condition is genuinely the conjunction of { "z", 1 }, not either alone.

Reproducer

I don't have a standalone leanMultisig-only repro yet — it needs a realistic witness flowing into prove_execution. Via zeam it's:

git clone https://github.com/blockblaz/zeam.git
cd zeam && git checkout 17f1083   # pre-#759, opt-level="z" still present
./zig build run -Dprover=risc0 -- prove -z risc0
# crashes in ~25s with "General protection exception" on Linux x86_64

Flipping either opt-level to "s" or codegen-units to anything ≥ 2 in rust/Cargo.toml's [profile.risc0-release] clears it. Also reproduces on GitHub's 2-core ubuntu-latest runners.

If it would help, I'm happy to try to carve a standalone #[test] inside leanMultisig that builds under { "z", 1 } and triggers the same codegen path — something around prove_generic_logup with a trivial witness. Let me know if you'd want that shape or something different.

Probably worth doing regardless of a fix

A short note somewhere in the README / a CODEGEN.md saying "consumers using opt-level = "z" need codegen-units >= 2, or use opt-level = "s" — the { "z", 1 } combo miscompiles on x86_64 rustc ≥ 1.95" would save anyone else hitting this a lot of bisecting. Happy to PR that once you confirm the preferred wording/location.

A proper rustc/LLVM upstream issue is the long-term fix, but needs a minimized reproducer first.

Environment

• rustc 1.95.0 (59807616e 2026-04-14) stable
• AMD EPYC-Genoa (Zen 4), also on GitHub ubuntu-latest
• Ubuntu 24.04, Linux 6.8
• -Ctarget-cpu=x86-64-v3 (so AVX2, not AVX-512)
• leanMultisig 2eb4b9d, zeam pre-fix ref 17f1083, downstream fix blockblaz/zeam#759

[TomWambsgans]

Thanks for reporting, will investigate

[TomWambsgans]

I was able to reproduce the issue. The following is Claude diagnostic, claiming that the issue is in Zig linker. We can totally do Fix B (small modif in leanMultisig), but there is always the risk the same bug appears in the future. What do you think?

Summary

Faulting PC is 0x6f0f3a1. Instruction bytes c5 fc 28 05 af 8b 57 fb
decode to:

vmovaps -0x4a87451(%rip), %ymm0   # target = 0x2487f58

That's a 256-bit aligned AVX2 load. vmovaps requires its memory
operand to be 32-byte aligned or the CPU raises #GP. Target address
0x2487f58 has low nibble 0x18: only 8-byte aligned. Deterministic fault
on any x86_64 CPU. Nothing to do with AVX-512.

The load target is a 32-byte constant living in .rodata.cst of the final
binary. In the Rust object files (lean_vm-*.rcgu.o) it is correctly
placed:

.rodata.cst4     sh_entsize=4    sh_addralign=4
.rodata.cst8     sh_entsize=8    sh_addralign=8
.rodata.cst16    sh_entsize=16   sh_addralign=16
.rodata.cst32    sh_entsize=32   sh_addralign=32   <- the constant

In the final zeam binary, all four collapse into a single output section:

.rodata.cst      sh_entsize=4    sh_addralign=32

Section base is 32-aligned, but entries pack on 4-byte tiles because
sh_entsize=4. Individual 32-byte constants land at arbitrary 4-byte
offsets within the section. This specific constant landed at +0x8758,
which is only 8-aligned. vmovaps assumes the global is aligned to
align_of(global); the Rust object declared it 32-aligned; Zig's linker
broke that invariant at final-link time.

Standard linkers (GNU ld, gold, LLD) don't do this — they either keep
.rodata.cstN as separate output sections or merge with
sh_entsize = max(inputs). Zig's ELF linker merges with
sh_entsize = min(inputs) and loses per-entry alignment for every entry
larger than the minimum.

Offending source pattern

Poseidon16Precompile::<true>::bus:

fn bus(&self) -> Bus {
    Bus {
        direction: BusDirection::Pull,
        selector: POSEIDON_16_COL_FLAG,
        data: vec![
            BusData::Constant(POSEIDON16_PRECOMPILE_DATA),
            BusData::Column(POSEIDON_16_COL_INDEX_INPUT_LEFT),
            BusData::Column(POSEIDON_16_COL_INDEX_INPUT_RIGHT),
            BusData::Column(POSEIDON_16_COL_INDEX_INPUT_RES),
        ],
    }
}

BusData is 16 bytes; the 4-element array is 64 bytes. At
{opt="z", cgu=1} LLVM lowers the vec![...] initializer to:

call   box_new_uninit(64)
vmovaps -0x4a87451(%rip), %ymm0   <- faults here
vmovups %ymm0, (%rax)
vmovaps -0x4a80135(%rip), %ymm0
vmovups %ymm0, 0x20(%rax)

At {s, 1} or {z, 16} LLVM picks a different lowering that doesn't
require the source to be 32-byte aligned, so the linker's broken alignment
goes unnoticed. Same bug, different visibility.

This is not specific to bus(). Anywhere in any Rust dependency where
LLVM emits vmovaps from .rodata.cst32, the same fault is latent. We
hit Poseidon16Precompile::bus first because of call-graph ordering in
xmss_aggregate.

Fix A: force LLD at the final link (zeam side)

if (optimize != .Debug and target.result.os.tag == .linux) {
    cli_exe.use_lld = true;
}

Cons

• -flld -ODebug segfaults Zig 0.15.2 itself (separate Zig bug). Must be
  gated on optimize mode, so Debug builds still hit the original crash.
• Loses Zig self-hosted linker features (incremental relink,
  zig build --watch, in-place patching).

Fix B: stop leanMultisig emitting the pattern

Rewrite the four bus() methods in
lean_vm::tables::{poseidon_16, poseidon_24, execution, extension_op}
from data: vec![a, b, c, d] to:

let mut data = Vec::with_capacity(4);
data.push(a);
data.push(b);
data.push(c);
data.push(d);
Bus { ..., data }

Wire via [patch."https://github.com/leanEthereum/leanMultisig.git"] in
rust/Cargo.toml pointing at a fork.

Disassembly of the new bus():

call   Vec::with_capacity
call   Vec::push
call   Vec::push
call   Vec::push
call   Vec::push

No 64-byte rodata literal. No vmovaps. No alignment assumption.

Fix C: Fix Zig's ELF linker

The defect is in Zig's ELF linker's handling of SHF_MERGE sections with
mixed sh_entsize. It must either keep them as separate output sections
or merge with sh_entsize = max(inputs) and pad to preserve per-entry
alignment. Anything else is a workaround.

[ch4r10t33r]

@TomWambsgans yes, Fix B sounds good to me.

[TomWambsgans]

Should be fixed now. I have also applied the same to the devnet4 branch. Please let me know if the problem persists