Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/agents/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ Establish these facts before matching a failure mode:
| A12 | `mkdirat ... : read-only file system` during agent chroot startup on ARC/DinD | `chroot.binariesSourcePath` set to the same root as `--docker-host-path-prefix` (e.g. both `/tmp/gh-aw`); Docker mounts `/tmp/gh-aw/usr:/host/usr:ro` first, then the attempt to mkdir `/host/usr/local/bin` as a nested overlay mount point fails because the parent is read-only | **Fixed in firewall v0.27.10**: upgrade AWF; the overlay is now mounted at `/host/tmp/awf-runner-bin:ro` (writable `/host/tmp` parent) instead of `/host/usr/local/bin:ro` | Check `awf --version`; inspect agent container logs for `mkdirat`; verify `chroot.binariesSourcePath` equals `docker-host-path-prefix` root | #5481, #5482 |
| A13 | `chroot: failed to run command '/bin/sh': No such file or directory` or `[entrypoint][ERROR] capsh not found on host system` on a **glibc/Debian daemon** (not musl/Alpine) | ARC/DinD split-fs: system-mount source dirs (`/tmp/gh-aw/{usr,bin,lib,...}`) are empty because nothing populates them. The entrypoint "musl/Alpine" warning is **misleading** — it fires because no dynamic loader is found, not because the daemon is musl. | **Fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. AWF emits a `sysroot-stage` init container that copies the signed `build-tools` image filesystem (`bash`, `capsh`, `gcc`, dev libs, coreutils) into a named `sysroot` volume mounted at `/host:ro` before the agent starts. Use `runner.sysrootImage` to pin a specific image. | Check `awf --version` ≥ v0.27.15; verify `runner.topology: "arc-dind"` is set; inspect compose output for `sysroot-stage` service and `sysroot` volume | #5541, github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 |
| A14 | `unknown shorthand flag: 'd' in -d` / `Command failed with exit code 125: docker compose up -d --pull never` | ARC/DinD sidecar image lacks `docker-compose-plugin`; AWF uses `docker compose` (v2 plugin syntax) to orchestrate containers but the DinD sidecar only has legacy standalone Docker or no Compose support | Add `docker-compose-plugin` to the DinD sidecar Dockerfile: `RUN apt-get update && apt-get install -y docker-compose-plugin` (Debian/Ubuntu) or `RUN apk add docker-cli-compose` (Alpine) | `docker compose version` inside the DinD sidecar — v2 output confirms plugin is present; inspect sidecar Dockerfile for `docker-compose-plugin` | github/gh-aw-firewall#5729 |
| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Open — fix in progress** (PR github/gh-aw-firewall#5817): workaround — run `chmod -R a+rX` inside the squid container before `docker compose down` while it still owns the log volume; or configure squid to write logs as runner UID | `ls -la <proxy-logs-dir>` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816 |
| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Fixed in PR github/gh-aw-firewall#5963**: `fixArtifactPermissionsForRootless()` now calls `applyHostPathPrefixToVolumes()` so the repair container bind mount is correctly translated for the DinD daemon. Upgrade AWF to the version that includes github/gh-aw-firewall#5963. Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`. | `ls -la <proxy-logs-dir>` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963 |

## Category B — Self-hosted runners

Expand Down Expand Up @@ -217,4 +217,3 @@ Flag these explicitly instead of implying there is a complete fix:
- D4 / #4849 — enterprise header injection extension point
- C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes
- C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911)
- A15 / github/gh-aw-firewall#5816 — rootless artifact permission repair on ARC/DinD: `fixArtifactPermissionsForRootless()` does not apply `dockerHostPathPrefix` to repair step bind mount; fix tracked in PR github/gh-aw-firewall#5817
3 changes: 1 addition & 2 deletions .github/workflows/shared/self-hosted-failure-modes.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Establish these facts before matching a failure mode:
| A12 | `mkdirat ... : read-only file system` during agent chroot startup on ARC/DinD | `chroot.binariesSourcePath` set to the same root as `--docker-host-path-prefix` (e.g. both `/tmp/gh-aw`); Docker mounts `/tmp/gh-aw/usr:/host/usr:ro` first, then the attempt to mkdir `/host/usr/local/bin` as a nested overlay mount point fails because the parent is read-only | **Fixed in firewall v0.27.10**: upgrade AWF; the overlay is now mounted at `/host/tmp/awf-runner-bin:ro` (writable `/host/tmp` parent) instead of `/host/usr/local/bin:ro` | Check `awf --version`; inspect agent container logs for `mkdirat`; verify `chroot.binariesSourcePath` equals `docker-host-path-prefix` root | #5481, #5482 |
| A13 | `chroot: failed to run command '/bin/sh': No such file or directory` or `[entrypoint][ERROR] capsh not found on host system` on a **glibc/Debian daemon** (not musl/Alpine) | ARC/DinD split-fs: system-mount source dirs (`/tmp/gh-aw/{usr,bin,lib,...}`) are empty because nothing populates them. The entrypoint "musl/Alpine" warning is **misleading** — it fires because no dynamic loader is found, not because the daemon is musl. | **Fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. AWF emits a `sysroot-stage` init container that copies the signed `build-tools` image filesystem (`bash`, `capsh`, `gcc`, dev libs, coreutils) into a named `sysroot` volume mounted at `/host:ro` before the agent starts. Use `runner.sysrootImage` to pin a specific image. | Check `awf --version` ≥ v0.27.15; verify `runner.topology: "arc-dind"` is set; inspect compose output for `sysroot-stage` service and `sysroot` volume | #5541, github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 |
| A14 | `unknown shorthand flag: 'd' in -d` / `Command failed with exit code 125: docker compose up -d --pull never` | ARC/DinD sidecar image lacks `docker-compose-plugin`; AWF uses `docker compose` (v2 plugin syntax) to orchestrate containers but the DinD sidecar only has legacy standalone Docker or no Compose support | Add `docker-compose-plugin` to the DinD sidecar Dockerfile: `RUN apt-get update && apt-get install -y docker-compose-plugin` (Debian/Ubuntu) or `RUN apk add docker-cli-compose` (Alpine) | `docker compose version` inside the DinD sidecar — v2 output confirms plugin is present; inspect sidecar Dockerfile for `docker-compose-plugin` | github/gh-aw-firewall#5729 |
| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Open — fix in progress** (PR github/gh-aw-firewall#5817): workaround — run `chmod -R a+rX` inside the squid container before `docker compose down` while it still owns the log volume; or configure squid to write logs as runner UID | `ls -la <proxy-logs-dir>` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816 |
| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Fixed in PR github/gh-aw-firewall#5963**: `fixArtifactPermissionsForRootless()` now calls `applyHostPathPrefixToVolumes()` so the repair container bind mount is correctly translated for the DinD daemon. Upgrade AWF to the version that includes github/gh-aw-firewall#5963. Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`. | `ls -la <proxy-logs-dir>` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963 |

## Category B — Self-hosted runners

Expand Down Expand Up @@ -100,4 +100,3 @@ Flag these explicitly instead of implying there is a complete fix:
- D4 / #4849 — enterprise header injection extension point
- C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes
- C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911)
- A15 / github/gh-aw-firewall#5816 — rootless artifact permission repair on ARC/DinD: `fixArtifactPermissionsForRootless()` does not apply `dockerHostPathPrefix` to repair step bind mount; fix tracked in PR github/gh-aw-firewall#5817
7 changes: 6 additions & 1 deletion scripts/ci/self-hosted-runner-doctor-workflow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,14 @@ describe('self-hosted runner doctor workflow config', () => {
expect(content).toContain('github/gh-aw-firewall#5753');
expect(content).toContain('| A14 | `unknown shorthand flag: \'d\' in -d` / `Command failed with exit code 125: docker compose up -d --pull never` |');
expect(content).toContain('| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` |');
expect(content).toContain('**Fixed in PR github/gh-aw-firewall#5963**');
expect(content).toContain('`fixArtifactPermissionsForRootless()`');
expect(content).toContain('`applyHostPathPrefixToVolumes()`');
expect(content).toContain('Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`.');
expect(content).toContain('github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963');
expect(content).toContain('| `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` on ARC/DinD | A14 |');
expect(content).toContain('| `Rootless artifact permission repair failed for .../sandbox/firewall/logs` on ARC/DinD | A15 |');
expect(content).toContain('- A15 / github/gh-aw-firewall#5816 — rootless artifact permission repair on ARC/DinD: `fixArtifactPermissionsForRootless()` does not apply `dockerHostPathPrefix` to repair step bind mount; fix tracked in PR github/gh-aw-firewall#5817');
expect(content).not.toMatch(/^- A15 \/ /m);
}

expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)');
Expand Down
Loading