fix: skip /etc bind mounts when sysroot-stage is active (arc-dind) by lpcox · Pull Request #5732 · github/gh-aw-firewall

lpcox · 2026-06-30T20:30:42Z

Problem

On ARC/DinD runners with split filesystem, the agent container fails to start with OCI runtime mount errors. The Docker daemon cannot see the runner's filesystem paths, so bind mounts sourced from the runner fail.

Three categories of broken mounts were identified:

/etc file mounts (e.g., /etc/ld.so.cache:/host/etc/ld.so.cache:ro) — daemon can't find these on its Alpine-based root
Chroot hosts file (/tmp/awf-*/chroot-*/hosts:/host/etc/hosts:ro) — written to runner's /tmp which daemon can't see
All workDir-based and home-based mounts (initSignalDir, agentLogsPath, sessionStatePath, chroot-home, ~/.cache, etc.) — sourced from runner's unshared /tmp/awf-* or /home/runner/

Fix

When isSysrootEnabled(config) is true (i.e., runner.topology: arc-dind), the sysroot named volume at /host already provides a complete glibc filesystem from the build-tools image. These fixes skip or filter mounts the daemon cannot resolve:

buildEtcMounts() returns [] — sysroot volume has /etc
generateHostsFileMount() is skipped — sysroot volume has /etc/hosts; domains resolve via DNS at runtime
Compose generator filters bind mounts whose source starts with config.workDir or whose source starts with effectiveHome targeting /host/home/... — the sysroot volume provides writable home directories

Mounts that are kept:

/tmp:/tmp:rw (daemon's own /tmp)
Workspace (${GITHUB_WORKSPACE}:...) — ARC shares workspace with daemon
Kernel VFS (/sys, /dev)
/dev/null credential-hiding overlays
Custom --mount flags (typically under shared /tmp/gh-aw/)
sysroot:/host:rw named volume

Testing

All 3387 tests pass
Added tests for sysroot gating in etc-mounts.test.ts and compose-generator.test.ts
TypeScript compiles cleanly

Context

Iterative fix discovered during ARC/DinD canary testing. Each run revealed the next mount failure after the previous was fixed:

Run 28472890469: /etc/ld.so.cache mount error
Run 28476477775: /tmp/awf-*/chroot-*/hosts mount error
Next expected: initSignalDir, agentLogsPath, home mounts (addressed proactively)

When runnerTopology is 'arc-dind', the sysroot-stage container already provides a complete /etc from the build-tools image. Bind-mounting the host's /etc files on top fails on split-fs because the Docker daemon cannot resolve runner filesystem paths (e.g., /etc/ld.so.cache doesn't exist on the Alpine-based DinD daemon). Return an empty array from buildEtcMounts() when isSysrootEnabled() is true, since the sysroot volume already has all needed /etc content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-30T20:32:46Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	98.64%	98.68%	📈 +0.04%
Statements	98.56%	98.60%	📈 +0.04%
Functions	99.55%	99.55%	➡️ +0.00%
Branches	94.52%	94.53%	📈 +0.01%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/services/agent-volumes/etc-mounts.ts`	98.2% → 98.3% (+0.09%)	98.2% → 98.3% (+0.09%)
`src/workdir-setup.ts`	92.7% → 94.5% (+1.82%)	92.7% → 94.5% (+1.82%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This pull request fixes ARC/DinD (split filesystem) agent startup failures by preventing AWF from attempting host /etc bind mounts when the sysroot-stage mechanism is active, since the sysroot volume already contains a complete /etc and split-fs daemons can’t reliably bind the runner’s /etc paths.

Changes:

Skip /etc bind mounts entirely when runnerTopology enables sysroot-stage (arc-dind).
Add unit tests covering arc-dind, standard, and undefined topology behavior for buildEtcMounts().

Show a summary per file

File	Description
`src/services/agent-volumes/etc-mounts.ts`	Returns `[]` from `buildEtcMounts()` when sysroot-stage is enabled to avoid broken `/etc` overlays on ARC/DinD split-fs.
`src/services/agent-volumes/etc-mounts.test.ts`	Adds test coverage validating that only `arc-dind` produces an empty mount list, while other topologies keep existing `/etc` mounts.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 2/2 changed files
Comments generated: 1
Review effort level: Low

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-30T20:42:31Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-06-30T20:43:36Z

✅ Smoke Gemini completed. All facets verified. 💎

github-actions · 2026-06-30T20:43:37Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-06-30T20:43:40Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

github-actions · 2026-06-30T20:43:42Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-06-30T20:43:45Z

✅ Contribution Check completed successfully!

Contribution guidelines review complete for PR #5732: the PR includes tests for the code change, has a clear description with related context, keeps files in the appropriate src/ service/test locations, and does not appear to need documentation updates.

github-actions · 2026-06-30T20:43:46Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-06-30T20:43:49Z

✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

github-actions · 2026-06-30T20:43:53Z

✅ Smoke Claude passed

github-actions · 2026-06-30T20:43:57Z

✅ Build Test Suite completed successfully!

github-actions · 2026-06-30T20:43:58Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-06-30T20:44:00Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-06-30T20:44:09Z

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

github-actions · 2026-06-30T20:44:20Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-06-30T20:44:26Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-06-30T20:47:01Z

Smoke Test: Claude Engine Validation

Check	Result
API status	✅ PASS
GH check	✅ PASS
File status	✅ PASS

Overall result: PASS

Generated by Smoke Claude for #5732 · 52.3 AIC · ⊞ 6.1K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:47:29Z

🔬 Smoke Test Results

Test	Result
GitHub MCP	✅ Connected (listed PRs successfully)
HTTP Connectivity	❌ Pre-step data unavailable (template not resolved)
File Write/Read	❌ Pre-step data unavailable (template not resolved)

PR: fix: skip /etc bind mounts when sysroot-stage is active (arc-dind)
Author: @lpcox

Overall: FAIL — MCP connectivity confirmed, but pre-computed step outputs (SMOKE_HTTP_CODE, SMOKE_FILE_PATH, SMOKE_FILE_CONTENT) were not resolved before agent execution.

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:47:38Z

@lpcox
MCP PRs:

fix: skip /etc bind mounts when sysroot-stage is active (arc-dind): ✅
fix: correctly recover runner tool on PATH (after sudo w/ secure_path). remove incorrect reading from GITHUB_PATH: ✅
GitHub.com connectivity: ✅
File I/O test: ✅
BYOK inference test: ✅
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
✅ PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:48:04Z

@lpcox
GitHub MCP Testing: ✅
GitHub.com Connectivity: ✅
File Write/Read: ✅
BYOK Inference: ✅
Running direct BYOK mode (OIDC→Entra) via api-proxy→Foundry (o4-mini-aw)
PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:48:11Z

Gemini Smoke Test Results: PASS. Titles: 1. fix: escape $d as $$d in sysroot-stage compose command, 2. fix: pass host-gateway IP to iptables-init container for NAT bypass. Connectivity: OK. File: OK.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:48:21Z

✅ Smoke Copilot BYOK Test Results

MCP Test: ✅ Last 2 merged PRs retrieved
GitHub.com: ✅ HTTP 200
File I/O: ✅ Write/read test passed
BYOK Inference: ✅ Direct BYOK mode active

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com. PASS ✓

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:48:32Z

🔍 Chroot Smoke Test Results

Runtime	Host Version	Chroot Version	Match?
Python	`3.12.13`	`3.12.3`	❌ NO
Node.js	`v24.18.0`	`v22.23.1`	❌ NO
Go	`go1.22.12`	`go1.22.12`	✅ YES

Overall: ❌ NOT all tests passed — Python and Node.js versions differ between host and chroot environments. Go matches correctly.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:48:41Z

🔥 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Status	Detail
Module Loading	✅	`otel.js` loads cleanly; exports `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled`
Test Suite	✅	59 passed, 0 failed (2 suites: `otel.test.js`, `otel-fanout.test.js`)
Env Var Forwarding	✅	`api-proxy-env-config.ts` forwards `GH_AW_OTLP_ENDPOINTS`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `GITHUB_AW_OTEL_TRACE_ID`, `GITHUB_AW_OTEL_PARENT_SPAN_ID`, `OTEL_SERVICE_NAME`
Token Tracker Integration	✅	`onUsage` callback present in `token-tracker-http.js` (lines 283, 324)
OTEL Diagnostics	✅	No container running (unit-test context); fallback path writes to `/var/log/api-proxy/otel.jsonl` when no OTLP endpoint is set — expected behavior

All scenarios pass. OTEL tracing integration is healthy.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:49:55Z

🔥 Smoke Test Results — Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅ 200
File write/read	✅

Overall: PASS — @lpcox

🔑 PAT report filed by Smoke Copilot PAT
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:50:03Z

Merged PRs:

fix: escape $d as $$d in sysroot-stage compose command
docs: sync schemas and specs with source changes

Checks:

PR list review: ❌ (required safeinputs-gh wrapper missing; gh auth unavailable)
Playwright GitHub title: ✅
File write/read: ✅
Discussion comment: ✅
Build (npm ci && npm run build): ✅

Overall: FAIL

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:50:19Z

Smoke Test: Services Connectivity

Check	Result
Redis PING	❌ Timeout — `host.docker.internal` unreachable
PostgreSQL pg_isready	❌ No response on port 5432
PostgreSQL SELECT 1	❌ Timeout — connection failed

Overall: FAIL

host.docker.internal is not resolvable/reachable from this runner (not running inside a Docker container).

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-06-30T20:50:43Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for #5732 · 52.5 AIC · ⊞ 7.8K · ◷
_{Add label ready-for-aw to run again}

Copilot AI review requested due to automatic review settings June 30, 2026 20:30

Copilot started reviewing on behalf of lpcox June 30, 2026 20:31 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread src/services/agent-volumes/etc-mounts.test.ts Outdated

test: rename sysroot gating suite

b8af93a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

lpcox added the ready-for-aw label Jun 30, 2026

lpcox temporarily deployed to aoai-model June 30, 2026 20:44 — with GitHub Actions Inactive

github-actions Bot added the smoke-claude label Jun 30, 2026

github-actions Bot mentioned this pull request Jun 30, 2026

[aw] No-Op Runs #5368

Closed

lpcox temporarily deployed to aoai-model June 30, 2026 20:47 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok-aoai-apikey label Jun 30, 2026

github-actions Bot added the smoke-copilot-byok-aoai-entra label Jun 30, 2026

lpcox temporarily deployed to aoai-model June 30, 2026 20:48 — with GitHub Actions Inactive

github-actions Bot added the smoke-gemini label Jun 30, 2026

github-actions Bot added the smoke-copilot-byok label Jun 30, 2026

github-actions Bot added the smoke-copilot-pat label Jun 30, 2026

github-actions Bot added the build-test label Jun 30, 2026

lpcox merged commit e277902 into main Jun 30, 2026
89 checks passed

lpcox deleted the fix/skip-etc-mounts-with-sysroot branch June 30, 2026 21:03

lpcox mentioned this pull request Jun 30, 2026

fix: filter split-fs-invisible mounts when sysroot-stage is active (arc-dind) #5734

Merged

github-actions Bot mentioned this pull request Jul 1, 2026

fix: sysroot filter should not drop workspace custom mounts #5739

Merged

This was referenced Jul 1, 2026

gh-aw compiler: ARC/DinD topology requires daemon-visible path redirections #5753

Closed

gh-aw compiler: ARC/DinD topology requires daemon-visible path redirections github/gh-aw#42807

Closed

This was referenced Jul 1, 2026

[WIP] Fix ARC/DinD topology path redirections in gh-aw compiler github/gh-aw#42808

Closed

Compiler: enforce ARC/DinD daemon-visible paths and AWF v0.27.20 minimum github/gh-aw#42815

Merged

Uh oh!

Conversation

lpcox commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Testing

Context

Uh oh!

github-actions Bot commented Jun 30, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Jun 30, 2026

🔬 Smoke Test Results

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

🔍 Chroot Smoke Test Results

Uh oh!

github-actions Bot commented Jun 30, 2026

lpcox commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading