ci: harden Phoenix CI against OOM, stale NFS handles, and transient SLURM errors#1304

Merged

sbryngelson merged 15 commits into

MFlowCode:masterfrom

sbryngelson:frontier-runner

Mar 13, 2026

Commits on Mar 12, 2026

ci: harden Phoenix CPU job memory and add sbatch retry on transient errors

sbryngelson
and
claude
committed
ci: tolerate stale NFS file handles on rm -rf build

sbryngelson
and
claude
committed
ci: fix retry-sbatch stdout/stderr and job-ID extraction

sbryngelson
and
claude
committed
ci: use printf over echo for sbatch script piping

sbryngelson
and
claude
committed
ci: add explanatory comment to rm -rf build suppression in build.sh and bench.sh

sbryngelson
and
claude
committed
ci: always rebuild on Phoenix even if rm -rf build leaves partial directory

sbryngelson
and
claude
committed
ci: use rename trick to handle stale NFS file handles on build cleanup

sbryngelson
and
claude
committed
ci: log last sbatch error output on retry exhaustion

sbryngelson
and
claude
committed

Commits on Mar 13, 2026