diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 41f3c4358..4f33e6b19 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -148,7 +148,7 @@ Establish these facts before matching a failure mode: | A12 | `mkdirat ... : read-only file system` during agent chroot startup on ARC/DinD | `chroot.binariesSourcePath` set to the same root as `--docker-host-path-prefix` (e.g. both `/tmp/gh-aw`); Docker mounts `/tmp/gh-aw/usr:/host/usr:ro` first, then the attempt to mkdir `/host/usr/local/bin` as a nested overlay mount point fails because the parent is read-only | **Fixed in firewall v0.27.10**: upgrade AWF; the overlay is now mounted at `/host/tmp/awf-runner-bin:ro` (writable `/host/tmp` parent) instead of `/host/usr/local/bin:ro` | Check `awf --version`; inspect agent container logs for `mkdirat`; verify `chroot.binariesSourcePath` equals `docker-host-path-prefix` root | #5481, #5482 | | A13 | `chroot: failed to run command '/bin/sh': No such file or directory` or `[entrypoint][ERROR] capsh not found on host system` on a **glibc/Debian daemon** (not musl/Alpine) | ARC/DinD split-fs: system-mount source dirs (`/tmp/gh-aw/{usr,bin,lib,...}`) are empty because nothing populates them. The entrypoint "musl/Alpine" warning is **misleading** — it fires because no dynamic loader is found, not because the daemon is musl. | **Fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. AWF emits a `sysroot-stage` init container that copies the signed `build-tools` image filesystem (`bash`, `capsh`, `gcc`, dev libs, coreutils) into a named `sysroot` volume mounted at `/host:ro` before the agent starts. Use `runner.sysrootImage` to pin a specific image. | Check `awf --version` ≥ v0.27.15; verify `runner.topology: "arc-dind"` is set; inspect compose output for `sysroot-stage` service and `sysroot` volume | #5541, github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 | | A14 | `unknown shorthand flag: 'd' in -d` / `Command failed with exit code 125: docker compose up -d --pull never` | ARC/DinD sidecar image lacks `docker-compose-plugin`; AWF uses `docker compose` (v2 plugin syntax) to orchestrate containers but the DinD sidecar only has legacy standalone Docker or no Compose support | Add `docker-compose-plugin` to the DinD sidecar Dockerfile: `RUN apt-get update && apt-get install -y docker-compose-plugin` (Debian/Ubuntu) or `RUN apk add docker-cli-compose` (Alpine) | `docker compose version` inside the DinD sidecar — v2 output confirms plugin is present; inspect sidecar Dockerfile for `docker-compose-plugin` | github/gh-aw-firewall#5729 | -| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Open — fix in progress** (PR github/gh-aw-firewall#5817): workaround — run `chmod -R a+rX` inside the squid container before `docker compose down` while it still owns the log volume; or configure squid to write logs as runner UID | `ls -la ` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816 | +| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Fixed in PR github/gh-aw-firewall#5963**: `fixArtifactPermissionsForRootless()` now calls `applyHostPathPrefixToVolumes()` so the repair container bind mount is correctly translated for the DinD daemon. Upgrade AWF to the version that includes github/gh-aw-firewall#5963. Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`. | `ls -la ` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963 | ## Category B — Self-hosted runners @@ -217,4 +217,3 @@ Flag these explicitly instead of implying there is a complete fix: - D4 / #4849 — enterprise header injection extension point - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) -- A15 / github/gh-aw-firewall#5816 — rootless artifact permission repair on ARC/DinD: `fixArtifactPermissionsForRootless()` does not apply `dockerHostPathPrefix` to repair step bind mount; fix tracked in PR github/gh-aw-firewall#5817 diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 85e20e55d..7926f6835 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -31,7 +31,7 @@ Establish these facts before matching a failure mode: | A12 | `mkdirat ... : read-only file system` during agent chroot startup on ARC/DinD | `chroot.binariesSourcePath` set to the same root as `--docker-host-path-prefix` (e.g. both `/tmp/gh-aw`); Docker mounts `/tmp/gh-aw/usr:/host/usr:ro` first, then the attempt to mkdir `/host/usr/local/bin` as a nested overlay mount point fails because the parent is read-only | **Fixed in firewall v0.27.10**: upgrade AWF; the overlay is now mounted at `/host/tmp/awf-runner-bin:ro` (writable `/host/tmp` parent) instead of `/host/usr/local/bin:ro` | Check `awf --version`; inspect agent container logs for `mkdirat`; verify `chroot.binariesSourcePath` equals `docker-host-path-prefix` root | #5481, #5482 | | A13 | `chroot: failed to run command '/bin/sh': No such file or directory` or `[entrypoint][ERROR] capsh not found on host system` on a **glibc/Debian daemon** (not musl/Alpine) | ARC/DinD split-fs: system-mount source dirs (`/tmp/gh-aw/{usr,bin,lib,...}`) are empty because nothing populates them. The entrypoint "musl/Alpine" warning is **misleading** — it fires because no dynamic loader is found, not because the daemon is musl. | **Fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. AWF emits a `sysroot-stage` init container that copies the signed `build-tools` image filesystem (`bash`, `capsh`, `gcc`, dev libs, coreutils) into a named `sysroot` volume mounted at `/host:ro` before the agent starts. Use `runner.sysrootImage` to pin a specific image. | Check `awf --version` ≥ v0.27.15; verify `runner.topology: "arc-dind"` is set; inspect compose output for `sysroot-stage` service and `sysroot` volume | #5541, github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 | | A14 | `unknown shorthand flag: 'd' in -d` / `Command failed with exit code 125: docker compose up -d --pull never` | ARC/DinD sidecar image lacks `docker-compose-plugin`; AWF uses `docker compose` (v2 plugin syntax) to orchestrate containers but the DinD sidecar only has legacy standalone Docker or no Compose support | Add `docker-compose-plugin` to the DinD sidecar Dockerfile: `RUN apt-get update && apt-get install -y docker-compose-plugin` (Debian/Ubuntu) or `RUN apk add docker-cli-compose` (Alpine) | `docker compose version` inside the DinD sidecar — v2 output confirms plugin is present; inspect sidecar Dockerfile for `docker-compose-plugin` | github/gh-aw-firewall#5729 | -| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Open — fix in progress** (PR github/gh-aw-firewall#5817): workaround — run `chmod -R a+rX` inside the squid container before `docker compose down` while it still owns the log volume; or configure squid to write logs as runner UID | `ls -la ` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816 | +| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Fixed in PR github/gh-aw-firewall#5963**: `fixArtifactPermissionsForRootless()` now calls `applyHostPathPrefixToVolumes()` so the repair container bind mount is correctly translated for the DinD daemon. Upgrade AWF to the version that includes github/gh-aw-firewall#5963. Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`. | `ls -la ` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963 | ## Category B — Self-hosted runners @@ -100,4 +100,3 @@ Flag these explicitly instead of implying there is a complete fix: - D4 / #4849 — enterprise header injection extension point - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) -- A15 / github/gh-aw-firewall#5816 — rootless artifact permission repair on ARC/DinD: `fixArtifactPermissionsForRootless()` does not apply `dockerHostPathPrefix` to repair step bind mount; fix tracked in PR github/gh-aw-firewall#5817 diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index 66879d1d9..699f230ce 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -42,9 +42,14 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('github/gh-aw-firewall#5753'); expect(content).toContain('| A14 | `unknown shorthand flag: \'d\' in -d` / `Command failed with exit code 125: docker compose up -d --pull never` |'); expect(content).toContain('| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` |'); + expect(content).toContain('**Fixed in PR github/gh-aw-firewall#5963**'); + expect(content).toContain('`fixArtifactPermissionsForRootless()`'); + expect(content).toContain('`applyHostPathPrefixToVolumes()`'); + expect(content).toContain('Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`.'); + expect(content).toContain('github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963'); expect(content).toContain('| `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` on ARC/DinD | A14 |'); expect(content).toContain('| `Rootless artifact permission repair failed for .../sandbox/firewall/logs` on ARC/DinD | A15 |'); - expect(content).toContain('- A15 / github/gh-aw-firewall#5816 — rootless artifact permission repair on ARC/DinD: `fixArtifactPermissionsForRootless()` does not apply `dockerHostPathPrefix` to repair step bind mount; fix tracked in PR github/gh-aw-firewall#5817'); + expect(content).not.toMatch(/^- A15 \/ /m); } expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)');