github · lpcox · Jul 22, 2026 · Jul 21, 2026 · Jul 21, 2026 · Jul 22, 2026
diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md
@@ -205,11 +205,11 @@ Establish these facts before matching a failure mode:
 | B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn`. **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Benign permission errors (`Operation not permitted`, `EPERM`, `EACCES`) from the artifact repair container are now logged at `debug` with an "expected on restricted runners" note. Only genuinely unexpected failures still emit `[WARN]`. Post-#6328: a `[WARN] Rootless artifact permission repair failed` message indicates a genuine failure, not a restricted-runner non-issue. | `ls -la <firewall-logs-dir>` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, github/gh-aw-firewall#5542, github/gh-aw-firewall#6328 |
 | B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf-<ts>-chroot-home/<path>` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. **additional hardening** (github/gh-aw-firewall#5766): changes `chown && chmod` to `chown 2>/dev/null; chmod` so `chmod` always runs as a fallback even when `chown` fails within the rootless UID namespace. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717, github/gh-aw-firewall#5766 |
 | B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 |
-| B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 |
+| B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. **Further fixed in AWF (PR github/gh-aw-firewall#6460, merged 2026-07-21):** `/etc/pki/ca-trust` and `/etc/pki/tls` are now included in the chroot mount policy, making RHEL/Amazon Linux CA paths directly accessible inside `/host/etc/pki` without requiring the staging fallback. Workaround (older AWF): copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783, github/gh-aw-firewall#6460 |
 | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. **Additional fix (PRs github/gh-aw-firewall#6342 / github/gh-aw-firewall#6356, merged 2026-07-18):** Even with a tag-only image ref, `docker run` was missing `--entrypoint sh`, causing AWF's `entrypoint.sh` to execute in place of the repair command. `entrypoint.sh` waits ~30 s for an iptables-init container that never starts in this context, then fails. The fix adds `--entrypoint sh` and passes the `chown/chmod` command via `-c`; stdout is now captured alongside stderr so timeout messages surfaced on stdout are not silently swallowed. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025, github/gh-aw-firewall#6342, github/gh-aw-firewall#6356 |
 | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: <n>`; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 |
 | B12 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` or `ENOTFOUND <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns <cluster-dns-ip>` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 |
-| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. Upgrade AWF to version including both PRs. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438 |
+| B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `squid.conf` as `acl allowed_domains dstdomain .<topology-host>` after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473 |
 | B14 | After upgrading to AWF including the centralized mount-policy refactor (PR github/gh-aw-firewall#6339), the Copilot CLI agent exits immediately (exit code 1, ~0.5 s, zero stdout/stderr); no network calls are made; the issue appears on Docker and gVisor runtimes but not sbx (which moves files aside rather than overlay-masking them) | PR github/gh-aw-firewall#6339 added `~/.copilot/config.json` to the credential **deny list** with reason *"Copilot CLI persisted auth token"*. This is incorrect: `config.json` contains only experiment flags, first-launch timestamps, and plugin metadata — no token. AWF masks denied files with a **read-only** `/dev/null` bind-mount overlay. The Copilot CLI reads **and atomically rewrites** `config.json` via temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently. | **Fixed in AWF (PR github/gh-aw-firewall#6374, merged 2026-07-18):** `~/.copilot/config.json` is removed from the credential deny list; the file is bind-mounted read-write as part of the `.copilot` tool subdir (pre-#6339 behaviour). Upgrade to the AWF version including github/gh-aw-firewall#6374. Workaround (older AWF): pass `--keep-containers` and inspect the agent container for `EROFS` or rename errors; downgrade AWF past github/gh-aw-firewall#6339 if unpatched. | `awf ... copilot agent --version` — if it exits 1 with no output in ~0.5 s, confirm AWF version includes github/gh-aw-firewall#6374; inspect agent container logs for `EROFS`/`rename` errors on `~/.copilot/config.json` | github/gh-aw-firewall#6339, github/gh-aw-firewall#6374 |
 
 ## Category C — GHES / GHEC / `ghe.com`