fix: add EACCES diagnostic + fallback for chroot hosts file write#6285
Conversation
The writeFileSync in generateHostsFileMount fails with EACCES on Ubuntu 24.04 GitHub Actions runners when writing inside a mkdtempSync-created subdirectory of the AWF workDir. This has been causing 100% Smoke CI failures in gh-aw since ~July 10 (AWF v0.27.29+). The root cause is environmental (the same code path worked previously and hasn't changed since May 23). The squid.conf write to the same workDir succeeds, confirming the workDir itself is writable — only the file creation inside the mkdtemp'd chroot-XXXXXX subdirectory fails. Changes: - Wrap writeFileSync in a try/catch with EACCES-specific handling - On EACCES, emit a diagnostic warning with process UID/GID and stat info for both the mkdtemp'd directory and the parent hostsRootDir — this will reveal the exact environmental cause on the next CI run - Fall back to writing the hosts file directly in hostsRootDir (as chroot-hosts) — config.workDir is already unique per run so the mkdtemp subdirectory is not required for isolation - Non-EACCES errors are re-thrown unchanged - Add 3 tests covering the fallback path, error re-throw, and diagnostic warning emission Addresses: github/gh-aw#45720 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b946b0b6-9ca4-47ae-ac6d-015374ba00a3
|
| Metric | Base | PR | Delta |
|---|---|---|---|
| Lines | 98.92% | 98.95% | 📈 +0.03% |
| Statements | 98.86% | 98.90% | 📈 +0.04% |
| Functions | 99.35% | 99.35% | ➡️ +0.00% |
| Branches | 95.11% | 95.06% | 📉 -0.05% |
📁 Per-file Coverage Changes (1 files)
| File | Lines (Before → After) | Statements (Before → After) |
|---|---|---|
src/log-directory-setup.ts |
96.2% → 100.0% (+3.78%) | 96.3% → 100.0% (+3.71%) |
Coverage comparison generated by scripts/ci/compare-coverage.ts
There was a problem hiding this comment.
Pull request overview
Adds diagnostics and a fallback for EACCES failures when creating the chroot hosts file.
Changes:
- Logs UID/GID and directory metadata on EACCES.
- Falls back to a hosts file in the root directory.
- Adds fallback and error-handling tests.
Show a summary per file
| File | Description |
|---|---|
src/services/agent-volumes/hosts-file.ts |
Adds EACCES diagnostics and fallback handling. |
src/services/agent-volumes/hosts-file-branches.test.ts |
Tests fallback, diagnostics, and non-EACCES behavior. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Medium
| // Fallback: write hosts file directly in hostsRootDir (config.workDir is | ||
| // already unique per run, so a subdirectory is not required for isolation). | ||
| const fallbackPath = path.join(hostsRootDir, 'chroot-hosts'); | ||
| fs.writeFileSync(fallbackPath, hostsContent, { mode: 0o644 }); | ||
| return `${fallbackPath}:/host/etc/hosts:ro`; |
|
✅ Copilot review passed with no inline comments. @lpcox Add the |
|
@copilot address review feedback |
Fixed in the latest commit. The fallback is now gated behind |
|
@copilot also address the security findings |
The CodeQL "Insecure temporary file" finding was already resolved in |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed... |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
✅ Smoke Claude passed |
|
✅ Smoke Gemini completed. All facets verified. 💎 Smoke test completed with FAIL status. Comment added to PR #6285. |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed... |
|
🔌 Smoke Services — All services reachable! ✅ |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
🚀 Security Guard has started processing this pull request |
|
✅ Contribution Check completed successfully! PR #6285 follows the applicable CONTRIBUTING.md guidelines: it includes focused tests for the new fallback and diagnostics, uses the correct src/ organization and logger, has a clear description with a related issue reference, and does not require documentation updates. |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed... |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed... |
|
📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤 |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
✅ Smoke Claude passed |
Gemini Smoke Test Results
Overall Status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
🔬 Smoke Test Results
Overall: PARTIAL — MCP confirmed; pre-computed outputs were unexpanded template variables. cc @lpcox Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
Smoke test: direct BYOK mode (
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com Overall: PARTIAL (inference path confirmed ✅; pre-fetched data variables unresolved) Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Claude Engine Validation
Overall Result: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Copilot PAT Auth
Overall: PARTIAL — template vars ( Auth mode: PAT (COPILOT_GITHUB_TOKEN) | Author: @lpcox Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: API Proxy OpenTelemetry Tracing
All scenarios pass. OTEL tracing integration is functional. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: GitHub Actions Services Connectivity
Result: FAIL The AWF sandbox blocks non-HTTP/HTTPS traffic (ports 5432/6379 are in the iptables blocked list). Service containers are not reachable from within the AWF agent container. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🔥 Smoke Test: Docker Sbx — PARTIAL
Overall: INCONCLUSIVE — pre-computed step outputs were not passed to the agent (unexpanded @lpcox — please verify the workflow passes
|
|
Smoke test results:\n- Merged PRs: ✅ updated workflows; ✅ updated workflows\n- PR list query: ✅\n- Page title check: ❌ (network error reaching github.com)\n- File write/read: ✅\n- Discussion query: ✅ (#6253)\n- Build: ❌ (npm ci 403 from registry)\nOverall status: FAIL Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"
- "registry.npmjs.org"See Network Configuration for more information.
|
Chroot Version Comparison
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
* fix: chmod mkdtempSync dirs and add file-permissions-checker CI The EACCES fallback in hosts-file.ts (PR #6285) still crashed because the fallback path also used mkdtempSync without overriding the restrictive umask (0177) that some Ubuntu 24.04 runners apply. Both the primary and fallback mkdtempSync now get an explicit chmodSync(dir, 0o700). Additionally, adds a comprehensive file-permissions-checker CI workflow that catches this class of bug via: - Static analysis: detects mkdtempSync without chmod, mkdirSync mode without chmod, unguarded cleanup file ops, execa chmod without error handling - Runtime tests: validates all critical directory creation patterns under a simulated restrictive umask (0177) Closes github/gh-aw#46062 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8bef4805-c013-4d25-908f-125577735990 * fix: chmod test dir before writing file in cross-uid test The cross-uid-cleanup-resilience test creates a directory then writes a file into it. Under umask 0177 (as set by the CI step), mkdirSync creates the dir with mode 0600 (no execute bit), so the writeFileSync fails before we even get to restrict the dir to 0555. Fix: chmod the dir to 0755 immediately after creation, then restrict it after writing the test file. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8bef4805-c013-4d25-908f-125577735990 * fix: address review feedback on file-permissions-checker - Wire matrix.os to runs-on so ubuntu-24.04 actually runs on that image - Handle multiline mkdtempSync assignments in static analysis (look back up to 3 lines for split const/let declarations) - Negative test now fails on Linux if EACCES can't be reproduced (validates the threat model is testable on the CI environment) - Integration test imports actual built generateHostsFileMount function instead of reimplementing the pattern inline - Rename cross-uid test to chmod-repair-retry-pattern (honest about what it tests: the local chmod+retry logic, not Docker-based repair) - Add explicit mode to all writeFileSync calls (fixes CodeQL js/insecure-temporary-file alert) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8bef4805-c013-4d25-908f-125577735990 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Problem
Every Smoke CI run in
github/gh-awwhere the agent step executes fails with:This has been happening 100% of the time since ~July 10 (AWF v0.27.29+). The error occurs in
generateHostsFileMount()athosts-file.ts:72—fs.writeFileSync()fails immediately afterfs.mkdtempSync()succeeds on the line above.Tracked in: github/gh-aw#45720
Root Cause Analysis
squid.confwrite to the sameworkDirsucceeds (Phase 5), confirming the workDir itself IS writablemkdtempSync-createdchroot-XXXXXXsubdirectory failsubuntu24/20260714.240) blocks file creation inside mkdtemp'd directoriesmkdirfailures) — none address thisopen/writeFileSyncfailureFix
EACCES-specific catch around
writeFileSyncwith diagnostic logging:statof the mkdtemp'd directory (ownership, permissions)statof the parent hostsRootDirFallback: Write the hosts file directly in
hostsRootDir(aschroot-hosts) instead of crashing. WhenuseDockerHostStagingis false (the case in all failing runs),config.workDiris already a unique per-run directory, so the mkdtemp subdirectory is not required for isolation.Non-EACCES errors are re-thrown unchanged.
Tests
Added 3 new tests to
hosts-file-branches.test.ts: