Skip to content

[aw-failures] P0: Smoke CI sandbox crash — EACCES writing chroot hosts file blocks all PR agent runs (last 6h) #46170

Description

@github-actions

Fix the rootless chroot hosts-file write in the awf sandbox — it hard-fails every Smoke CI agent run (P0, blocks all PR CI)

The agentic-firewall sandbox crashes during config generation before the agent ever starts. All Smoke CI agent jobs in the last 6h died identically at writeConfigs with EACCES writing the chroot hosts file. Token usage is 0 on every run — no agent work happens. This is the dominant failure and has no existing tracking coverage.

Failure clusters (last 6h)

Sev Cluster Workflow(s) Signature Representative run Status
P0 Sandbox chroot hosts EACCES Smoke CI (majority of 31 failed runs) Fatal error: EACCES ... open '/tmp/awf-chroot-*/hosts' at writeConfigs (awf-bundle.js:805) §29561818387 NEW — untracked
P1 CLI hang-on-exit PR Code Quality Reviewer agent completes, CLI never exits §29550557102 Tracked → #44298
P2 Misc / cancelled PR Sous Chef, Daily Skill Optimizer, Tidy (cancelled) insufficient/mixed signal §29560675416 Not actionable yet

Evidence

Confirmed identical, deterministic signature across the 5 fully-logged Smoke CI failures (§29561818387, §29561792838, §29561773855, §29561757674, §29561745591).

Error trace + root-cause chain
[INFO] Network-isolation mode: enforcing egress via Docker network topology (no host iptables, no sudo).
[INFO] Generating configuration files...
[WARN] EACCES writing chroot hosts file (process uid=1001 gid=1001):
  target: /tmp/awf-<id>/chroot-<x>/hosts
  chrootHostsDir: /tmp/awf-<id>/chroot-<x> [uid=1001 gid=1001 mode=600]
  hostsRootDir:   /tmp/awf-<id>         [uid=1001 gid=1001 mode=700]
  Falling back to writing hosts file directly in hostsRootDir.
[ERROR] Fatal error: Error: EACCES: permission denied, open '/tmp/awf-chroot-<y>/hosts'
    at Object.writeFileSync (node:fs:2444:35)
    at Object.Mq [as writeConfigs] (/home/runner/.local/lib/awf/awf-bundle.js:805:3010)

Two smoking guns:

  1. Permission mismatch under rootless remap — the chroot dirs are created mode=600/700 owned by uid=1001, yet the writing process (also reporting uid=1001) still gets EACCES. Under the rootless container uid-remap the effective writer differs from the dir owner, so a 0700 dir is not traversable/writable.
  2. Divergent fallback path — the fallback does not write into the reported hostsRootDir (/tmp/awf-<id>). It targets a different, never-created directory /tmp/awf-chroot-<y>/, guaranteeing a fatal EACCES instead of a recovery.

audit-diff (§29561818387 vs §29550546503): token_usage=0 on both (agent never ran), has_anomalies=false, new_domain_count=0. Deterministic pre-agent crash, not firewall/env drift.

Correlated branches under test include copilot/explore-rootless-cli-installation — i.e. rootless CLI/sandbox work is the likely regression source.

Existing-issue correlation

Fix roadmap

Sub-issues created

References:


Resolved — P0 Smoke CI sandbox crash no longer reproduces (Failure Investigator 6h, 2026-07-20)

The dominant P0 cluster (rootless chroot hosts-file EACCES at writeConfigs) is absent from the last 6h window. All 20 recent failed runs now proceed past sandbox config generation — containers start, agents execute, token usage is non-zero. Remaining failures are unrelated downstream issues (model-policy 403 on Smoke Claude, auto-model AI-credits pricing 400 on Smoke Copilot, CLI hang-on-exit on Tidy → tracked in #44298), each newly tracked/closed separately. Closing this P0 as fixed together with its fix sub-issue #46172. Reopen if the EACCES sandbox-generation crash returns.

Generated by 🔍 [aw] Failure Investigator (6h) · 136.1 AIC · ⌖ 36.6 AIC · ⊞ 5.2K ·

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions