[ARC-DinD] Add versioned `build-tools` sysroot image for build-test workflows

[lpcox]

Summary

On ARC/DinD deployments, no code in the workflow or agent container can run as root. Anything requiring root permissions must be done before the action runs — i.e., at image build time.

This means system-level build essentials (gcc, make, linker, dev libraries) cannot be installed by workflow steps or inside the agent. They must be baked into a pre-built image during the release pipeline, where root is available.

This issue proposes a versioned build-tools sysroot image that provides system-level build infrastructure. Language SDKs (Go, Node, Java, .NET, Rust, Python) remain on-demand via setup-* actions, installed to a shared tool cache volume.

Critically, this must be easy for gh-aw to configure. A single runner.topology: arc-dind selector (from #5591) should be all that's needed to activate the entire ARC/DinD mode — including sysroot image usage, tool cache redirection, and split-filesystem handling — rather than requiring callers to coordinate 6+ scattered config keys.

Related

• [ARC-DinD] Chroot /host base userland not staged on split-fs runners — security-preserving fix + empty-/host test harness #5541 — Parent issue: chroot /host base userland not staged on split-fs runners
• Add runner.topology config selector for ARC/DinD mode #5591 — runner.topology config selector for ARC/DinD mode
• Upstream: [ARC-DinD] ARC/DinD support in v0.75.4 still requires workflow-level workarounds gh-aw#34896 — ARC/DinD layer-by-layer progression

Design Principle: Privilege Hierarchy

When	Root?	What can happen
Image build (release pipeline)	✅	apt-get install, system library installation, compiler setup
ARC pod startup (K8s)	✅	DinD daemon starts, shared volumes created
Workflow steps (runner container)	❌	setup-* actions (user-writable tool cache), checkout, config generation
Agent container (AWF)	❌	Build, test, agent execution — reads sysroot + tool cache RO

Nothing in the workflow or agent ever needs root. The build-tools image front-loads all privileged work into the release process.

Configuration: runner.topology Integration

The problem today

Getting AWF to behave correctly on ARC/DinD requires the caller to independently set ~6 scattered keys:

Key	Section	ARC/DinD role
network.isolation	network	no NET_ADMIN/iptables (k8s)
container.dockerHost	container	point at the sibling DinD daemon
container.dockerHostPathPrefix	container	translate bind-mount paths across the split filesystem
container.runnerToolCachePath	container	relocate tool cache off /opt onto shared volume
container.sysrootImage	container	NEW: specify build-tools sysroot image
chroot.binariesSourcePath / chroot.identity	chroot	split-FS chroot overrides
dind.preStageDirs / dind.workDir	dind	stage dirs into daemon-visible filesystem

The solution: one selector activates everything

{
  "runner": {
    "topology": "arc-dind"   // "standard" (default) | "arc-dind"
  }
}

When runner.topology === "arc-dind", AWF applies overridable defaults:

1. network.isolation → true (ARC k8s lacks NET_ADMIN)
2. dind.preStageDirs → true
3. container.sysrootImage → ghcr.io/github/gh-aw-firewall/build-tools:{version} (NEW)
4. Tool cache warning if RUNNER_TOOL_CACHE is unset or under /opt
5. Split-filesystem path-prefix probes activate automatically

An explicit value in any key always wins over the topology default.

What gh-aw emits

The gh-aw compiler just emits:

{ "runner": { "topology": "arc-dind" } }

AWF resolves all the split-FS, sysroot, tool cache, and network isolation details internally. This is the single stable contract between gh-aw and the firewall for ARC deployments.

Architecture

build-tools image provides:          setup-* actions provide (on demand):
─────────────────────────────        ──────────────────────────────────
bash, capsh, coreutils               Go 1.22 (setup-go)
gcc, g++, make, ld                   Node 20 (setup-node)
libssl-dev, libc6-dev, libicu-dev    Java 21 (setup-java)
pkg-config, cmake, autoconf          .NET 8.0 (setup-dotnet)
git, curl, wget                      Rust (setup-rust)
system linker, dynamic loader        Python 3.12 (setup-python)
                                     specific versions per workflow

build-tools = static system base (same for all workflows, requires root → baked at image build time).
Tool cache = dynamic, per-workflow (setup-* actions choose versions, no root needed → installed at workflow runtime).

End-to-End Walkthrough

Phase 1: Image Build Time (root available)

The build-tools image is built once in the release pipeline with docker build (runs as root):

# containers/build-tools/Dockerfile
FROM ubuntu:22.04
RUN apt-get install -y build-essential gcc g++ make \
    autoconf automake libtool pkg-config cmake \
    libssl-dev libc6-dev libicu-dev libsqlite3-dev \
    binutils bison flex m4 libcap2-bin \
    git curl wget jq bash coreutils ca-certificates ...

This produces an image layer containing:

/usr/bin/gcc, /usr/bin/g++, /usr/bin/make, /usr/bin/ld
/usr/lib/x86_64-linux-gnu/libssl.so, libcrypto.so, libc.so, ...
/usr/include/openssl/*, linux/*, ...
/usr/sbin/capsh
/bin/bash, /bin/sh, /usr/bin/find, /usr/bin/git, ...

Published as ghcr.io/github/gh-aw-firewall/build-tools:0.28.0, signed with cosign.

---

Phase 2: Workflow Runtime (no root)

Step 1: Tool cache redirection (early in workflow)

- run: echo "RUNNER_TOOL_CACHE=/tmp/gh-aw/tool-cache" >> "$GITHUB_ENV"

Now setup-* actions will install to /tmp/gh-aw/tool-cache/ — a path on the shared emptyDir between runner and DinD containers.

Step 2: setup-* actions run on the runner (no root needed)

- uses: actions/setup-go@v6       # → /tmp/gh-aw/tool-cache/go/1.22.x/x64/
- uses: actions/setup-node@v6     # → /tmp/gh-aw/tool-cache/node/20.x/x64/
- uses: actions/setup-java@v5     # → /tmp/gh-aw/tool-cache/Java_Temurin-jdk/21.x/x64/
- uses: actions/setup-dotnet@v5   # → /tmp/gh-aw/tool-cache/dotnet/8.0.x/

These download pre-built tarballs and extract them — no root required. The tool cache directory tree after setup:

/tmp/gh-aw/tool-cache/
├── go/1.22.8/x64/bin/go
├── node/20.18.0/x64/bin/node
├── Java_Temurin-jdk/21.0.4/x64/bin/java
└── dotnet/8.0.400/x64/dotnet

Step 3: Capture env vars

- run: echo "GOROOT=$(go env GOROOT)" >> "$GITHUB_ENV"

Step 4: AWF starts the compose stack (from the runner)

AWF receives the config with runner.topology: arc-dind and generates docker-compose.yml:

services:
  sysroot-stage:
    image: ghcr.io/.../build-tools:0.28.0
    volumes:
      - sysroot:/sysroot
    command: ["cp -a /usr /lib /lib64 /bin /sbin /sysroot/"]
    # Copies image FS → named volume, then exits

  agent:
    image: ghcr.io/.../agent:0.28.0
    depends_on:
      sysroot-stage: { condition: service_completed_successfully }
    volumes:
      - sysroot:/host:ro                                      # ← system base (gcc, libs)
      - /tmp/gh-aw/tool-cache:/host/tmp/gh-aw/tool-cache:ro  # ← language SDKs
      - /workspace:/host/workspace:rw                         # ← code
      - /tmp/gh-aw:/host/tmp/gh-aw:rw                        # ← config, prompts

---

Phase 3: Inside the Agent Container (no root)

Step 5: entrypoint.sh runs

• Detects /host/bin/sh exists (from sysroot) ✅
• Finds capsh at /host/usr/sbin/capsh ✅
• chroot /host succeeds

Step 6: Agent wrapper sets up PATH

The bash wrapper command dynamically discovers SDK bin directories:

# Find all bin/ directories in the tool cache
GH_AW_TOOL_CACHE="$RUNNER_TOOL_CACHE"
export PATH="$(find "$GH_AW_TOOL_CACHE" -maxdepth 5 -type d -name bin | tr '\n' ':'):$PATH"

# Add GOROOT/bin
[ -n "$GOROOT" ] && export PATH="$GOROOT/bin:$PATH"

After this, PATH contains:

/tmp/gh-aw/tool-cache/go/1.22.8/x64/bin:                    ← go
/tmp/gh-aw/tool-cache/node/20.18.0/x64/bin:                  ← node, npm
/tmp/gh-aw/tool-cache/Java_Temurin-jdk/21.0.4/x64/bin:      ← java, javac
/usr/bin:                                                     ← gcc, make, git (from sysroot)
/bin:                                                         ← bash, ls, etc. (from sysroot)

Step 7: Agent (Copilot CLI) executes

The agent can now run build commands:

go build ./...     # go from tool cache, gcc/ld/libc from sysroot
npm install        # node from tool cache, node-gyp uses gcc from sysroot
javac Main.java   # javac from tool cache
make              # make from sysroot

---

What Comes From Where (Summary Table)

Source	Mounted as	Contains	How populated
build-tools image	sysroot:/host:ro	gcc, make, ld, libssl-dev, libc-dev, bash, capsh, git	apt-get install at image build time (root)
Tool cache	/tmp/gh-aw/tool-cache:/host/.../tool-cache:ro	Go, Node, Java, .NET, Python SDKs	setup-* actions at workflow runtime (no root)
Workspace	/workspace:/host/workspace:rw	Source code, node_modules/, build artifacts	actions/checkout + agent writes
AWF config/prompts	/tmp/gh-aw:/host/tmp/gh-aw:rw	Prompts, MCP config, AWF metadata	Workflow steps write to shared emptyDir
Copilot CLI	/host/tmp/awf-runner-bin/copilot:ro	The agent engine binary	install_copilot_cli.sh + existing staging mechanism

---

Comparison: Non-ARC vs ARC

Aspect	Non-ARC (GitHub-hosted runner)	ARC (DinD)
System libs (gcc, libssl)	Pre-installed on host VM	build-tools sysroot image
Language SDKs	setup-* → host tool cache → visible via /host/opt	setup-* → shared tool cache → mounted into agent
Workspace	Host FS → AWF mounts to /host	Shared emptyDir → mounted into agent
Copilot CLI	Installed on host → visible via /host mounts	Staged via awf-runner-bin mechanism
AWF config	Various keys set by caller	runner.topology: arc-dind → AWF handles the rest
Workflow YAML	Same	Same (+ tool cache redirection step)

---

Proposed Implementation

1. New Dockerfile: containers/build-tools/Dockerfile

FROM ubuntu:22.04

ENV DEBIAN_FRONTEND=noninteractive

# System-level build essentials only — no language SDKs.
# Language SDKs are installed on-demand via setup-* actions to a shared tool cache.
# Everything here requires root to install → done at image build time.
#
# Source of truth for parity: actions/runner-images toolset-2204.json .apt section
RUN apt-get update && apt-get install -y --no-install-recommends \
    # Compilers and linker
    build-essential g++ gcc make \
    # Build system tools
    autoconf automake libtool pkg-config cmake \
    # Development libraries (needed for native module compilation: cgo, node-gyp, etc.)
    libssl-dev libicu-dev libc6-dev libsqlite3-dev libyaml-dev \
    libcurl4-openssl-dev libgsl-dev \
    # Binary tools
    binutils bison flex m4 \
    # CLI utilities
    git curl wget jq zip unzip tar bzip2 \
    # System essentials for chroot
    ca-certificates bash coreutils findutils file \
    # Capability management (for AWF privilege drop)
    libcap2-bin \
    && rm -rf /var/lib/apt/lists/*

2. Release workflow addition

Add a build-build-tools job to .github/workflows/release.yml, parallel to existing image builds. Same pattern as existing images: build, push, sign with cosign, generate SBOM, attest.

Tags: ghcr.io/${{ github.repository }}/build-tools:${version} and :latest.

3. runner.topology: arc-dind config integration

When runner.topology === "arc-dind", AWF automatically:

Behavior	Default applied	Overridable?
Network isolation (no iptables)	true	Yes
DinD pre-stage dirs	true	Yes
Sysroot image	build-tools:{current-version}	Yes (container.sysrootImage)
Sysroot-stage init container in compose	Emitted	—
buildSystemMounts() replaced with sysroot volume	Yes	—
Tool cache warning if under /opt	Emitted	—
Split-FS path-prefix probes	Activated	—

The gh-aw compiler emits just:

{ "runner": { "topology": "arc-dind" } }

4. Schema changes

• Add runner.topology enum (standard | arc-dind) to docs/awf-config.schema.json
• Add container.sysrootImage string to schema
• Regenerate src/awf-config-schema.json via scripts/generate-schema.mjs
• Add runner?: { topology? } to AwfFileConfig in src/config-file.ts
• In mapAwfFileConfigToCliOptions, apply overridable arc-dind defaults

5. Tool cache integration

The existing workflow logic already handles tool cache mounting:

GH_AW_TOOL_CACHE="${RUNNER_TOOL_CACHE:?RUNNER_TOOL_CACHE must be set}"
if [ -d "$GH_AW_TOOL_CACHE" ]; then
  if [[ "$GH_AW_TOOL_CACHE" != /opt/* ]]; then
    GH_AW_TOOL_CACHE_MOUNT="$GH_AW_TOOL_CACHE:$GH_AW_TOOL_CACHE:ro"
  fi
fi

On ARC with RUNNER_TOOL_CACHE=/tmp/gh-aw/tool-cache, this activates automatically. If topology is arc-dind and tool cache is under /opt, AWF emits an actionable warning explaining the misconfiguration.

6. Parity maintenance: drift-detection CI

A scheduled job comparing the build-tools Dockerfile against upstream actions/runner-images toolset-2204.json apt section.

7. Versioning

All images share the same version tag from the release:

Image	Purpose
agent:0.28.0	Security infra, entrypoint, capsh, iptables
build-tools:0.28.0	System base: gcc, make, dev libraries (no language SDKs)
agent-act:0.28.0	Agent on catthehacker base (existing)
squid:0.28.0	Proxy (existing)
api-proxy:0.28.0	API credential proxy (existing)
cli-proxy:0.28.0	CLI proxy (existing)

Security Considerations

• The sysroot image is signed with cosign and has an SBOM attestation, same as all other AWF images.
• The sysroot volume is mounted read-only into the agent — the agent (post-privilege-drop) cannot modify it.
• Tool cache from setup-* actions is also mounted read-only at agent exec time.
• Provenance is AWF-controlled (built from our Dockerfile, from our release workflow) — not sourced from runner-writable or daemon-writable paths for pre-capsh execution.
• This avoids the trust-boundary concerns raised in [ARC-DinD] Chroot /host base userland not staged on split-fs runners — security-preserving fix + empty-/host test harness #5541: the base userland executed before capsh privilege-drop originates from the AWF-signed image, never from runner/daemon-writable paths.

Acceptance Criteria

• containers/build-tools/Dockerfile exists with system-level build essentials (no language SDKs)
• Release workflow builds, signs, and attests ghcr.io/.../build-tools:{version}
• Multi-arch support (linux/amd64, linux/arm64)
• runner.topology enum (standard | arc-dind) added to AWF config schema
• container.sysrootImage field added to schema
• runner.topology: arc-dind applies overridable defaults (network isolation, sysroot image, pre-stage dirs)
• Compose generator emits sysroot-stage init service + volume when topology is arc-dind
• Tool cache warning emitted when topology is arc-dind and cache is under /opt
• Tool cache on shared volume (RUNNER_TOOL_CACHE=/tmp/gh-aw/tool-cache) works with setup-* actions
• Agent can execute gcc --version (from sysroot) and go version (from tool cache) inside chroot
• Drift-detection CI comparing against upstream toolset-2204.json apt packages
• Documentation in docs/arc-dind.md updated
• Add runner.topology config selector for ARC/DinD mode #5591 closed as subsumed by this issue