network-isolation: topology-attach ordering deadlock starves the cli-proxy health gate (fixable in AWF)

[lpcox]

Summary

In --network-isolation (sandbox.agent.sudo: false) mode, AWF's own awf-cli-proxy sidecar can never become healthy when a --topology-attach peer (the external DIFC proxy / MCP gateway) is its probe target. The container that the cli-proxy must reach is only joined to the internal awf-net network after container startup has already gated on that sidecar's health — a structural ordering deadlock. The firewall fails to start and the agent is never invoked.

This is fully addressable within gh-aw-firewall; no gh-aw change is required. It is a standard-runner network-isolation bug (the ARC/chroot track is tracked separately in #5541; the broader sudo:false rollout regressions, including the rootless-EACCES sibling, in #5542).

Symptom (e.g. github/gh-aw Typist run 28168827390, job 83427672448)

[tcp-tunnel] Upstream error (...): getaddrinfo EAI_AGAIN awmg-cli-proxy
[cli-proxy] ERROR: DIFC proxy liveness probe failed for localhost:18443 ...
[cli-proxy] Failing fast to avoid repeated in-agent retries
AWF firewall failed to start: awf-cli-proxy could not connect to the external
DIFC proxy (or exited before establishing a connection). ... The agent was never invoked.

Config in play: topologyAttach: ["awmg-mcpg","awmg-cli-proxy"], --difc-proxy-host awmg-cli-proxy:18443. The hostname awmg-cli-proxy never resolves on awf-net.

---

Root cause — startup ordering deadlock

Current sequence in src/cli-workflow.ts:

1. Step 2 — startContainers() (src/container-lifecycle.ts:44-66) runs a single docker compose up -d. The dependency chain is agent → cli-proxy (service_healthy) → squid-proxy (service_healthy) (src/services/cli-proxy-service.ts:82-93, src/compose-generator.ts:194-211), and squid-proxy is what creates awf-net. So up -d blocks until the cli-proxy is healthy.
2. The cli-proxy entrypoint (containers/cli-proxy/entrypoint.sh:59-97) probes the external DIFC proxy through a TCP tunnel to AWF_DIFC_PROXY_HOST (awmg-cli-proxy), retrying 10× then exit 1 (fail-fast).
3. The external awmg-cli-proxy is only attached to awf-net in Step 2.5 — connectTopologyContainers() (src/cli-workflow.ts:122-133, src/topology.ts:91-120), which runs only after startContainers() returns successfully.

startContainers()  ── waits for ──▶  awf-cli-proxy healthy
        ▲                                   │ requires
        │ runs only after                   ▼
connectTopologyContainers() ◀── never ── awf-cli-proxy probe reaches awmg-cli-proxy
       (Step 2.5)                           │ requires
                                            ▼
                              docker network connect awf-net awmg-cli-proxy
                                       (== Step 2.5)

awf-net is internal: true (src/compose-generator.ts:234-240); the gh-aw-launched peer sits on the default bridge until AWF explicitly docker network connects it. Until then the name is unresolvable from inside awf-net (EAI_AGAIN). The attach is structurally sequenced behind the health gate it would satisfy → deadlock → fail-fast → agent never invoked.

This is deterministic, not flaky: any --network-isolation + --topology-attach run whose cli-proxy probes a topology peer at startup hits it every time. (The awmg-mcpg gateway's early ECONNRESET health failures are a secondary symptom of the same "peers not yet joined to awf-net" condition.)

---

Fix A — Attach topology peers before the health-gated bring-up (deterministic)

Split the single docker compose up -d so the network and the peer attach happen before the sidecars that depend on them:

1. docker compose up -d --no-deps squid-proxy → creates awf-net (and starts squid). --no-deps avoids pulling in the gated services.
2. connectTopologyContainers('awf-net', topologyAttach) → joins awmg-cli-proxy/awmg-mcpg now that the network exists.
3. docker compose up -d → the cli-proxy probe now resolves the peer; the agent starts normally.

Implementation: move the Step 2.5 attach (src/cli-workflow.ts:122-133) to run between network creation and the gated bring-up, and have startContainers() accept an optional "attach hook" (or split into createNetworkAndBaseline() + startRemaining()). Keep the existing single-up path unchanged when topologyAttach is empty, so non-topology runs are unaffected.

Pros: removes the deadlock by construction; no reliance on timing/retries. Cons: touches the startup sequencing and the retry/diagnostic logic in container-lifecycle.ts (the api-proxy/squid one-shot retry must be preserved across the split).

Considerations for Fix A

• The peer containers must exist when AWF attaches. They do today (gh-aw's "Start CLI Proxy"/"Start MCP Gateway" steps run before awf). To be defensive, connectTopologyContainers() could poll docker inspect <name> briefly and surface a clear error if a named peer is absent.
• docker compose up -d --no-deps squid-proxy must reliably create the external awf-net. Verify under both compose-managed (external: false) and external: true network modes (src/compose-generator.ts:215-256).
• Preserve the partial-startup cleanup signal (onContainersStarted?.()) so a failure in any phase still tears down cleanly.

---

Fix B — Make the cli-proxy probe survive a not-yet-attached peer (hardening)

In containers/cli-proxy/entrypoint.sh:59-97, the probe currently classifies only ECONNREFUSED (not-yet-ready) and timeout (unreachable). A DNS failure (EAI_AGAIN/ENOTFOUND / gh api exit for name resolution) falls into the generic bucket and the loop still fail-fasts after 10 short attempts.

Change: treat EAI_AGAIN/ENOTFOUND/getaddrinfo as "peer not yet attached — keep retrying" with a longer, DNS-aware backoff window (e.g. raise AWF_CLI_PROXY_LIVENESS_ATTEMPTS / cap, or add a dedicated DNS-resolution wait before the liveness probe). This makes the sidecar tolerate slow or momentarily-unattached peers rather than dying.

Pros: cheap, localized, defends against residual races even with Fix A. Cons: insufficient on its own — with today's ordering the attach never runs until startContainers() returns, so a longer retry window alone cannot break the deadlock. Fix B is a complement to Fix A, not a substitute.

Considerations for Fix B

• Keep fail-fast semantics for genuine terminal errors (auth failures, the DIFC proxy listening but rejecting) so we don't mask real misconfiguration behind long retries.
• Make the timeout budget configurable and bounded so a truly-missing peer still fails in reasonable time with a clear message.

---

Other approaches considered

• Run the attach concurrently with up -d. Kick off connectTopologyContainers() in the background as soon as awf-net exists (poll for the network), while the blocking up -d waits on cli-proxy health. Works, but concurrency against compose's own startup is harder to reason about and test than the explicit phasing in Fix A.
• Declare peers in compose via external_links / extra network membership so the attach is part of up rather than a post-step. Cleaner in principle, but the peer names are runtime inputs (--topology-attach) and the peers are externally managed, so compose-native wiring is brittle.
• Inject the peer into the cli-proxy's /etc/hosts (or use the peer IP directly) to sidestep DNS. Avoids the attach-timing dependency for name resolution, but the peer still must share awf-net for L3 reachability, so it doesn't remove the need to attach — only the DNS step.
• Relax the cli-proxy health gate (start the agent before the DIFC proxy is confirmed reachable). Rejected: it defeats the fail-fast design that prevents the agent from burning tokens against a dead proxy.

---

Recommendation

Ship Fix A (removes the deadlock deterministically) + Fix B (hardens against residual races), with a regression test that exercises --network-isolation + --topology-attach + a cli-proxy DIFC peer end-to-end and asserts the agent is invoked (no EAI_AGAIN deadlock).

Acceptance criteria

• A --network-isolation + --topology-attach run whose cli-proxy probes a topology peer starts the firewall and invokes the agent on a standard hosted runner.
• The fix is contained to gh-aw-firewall; no gh-aw change required.
• Non-topology runs (empty topologyAttach) keep the existing single-up startup path and the api-proxy/squid one-shot retry behavior.
• Regression coverage prevents the ordering deadlock from silently returning.

References

• Failing run: github/gh-aw Typist, run 28168827390, job 83427672448
• Sibling/rollout issue: network-isolation (sudo:false) rollout regressions on standard runners: topology-attach ordering deadlock + rootless log EACCES #5542 · ARC/chroot track: [ARC-DinD] Chroot /host base userland not staged on split-fs runners — security-preserving fix + empty-/host test harness #5541
• AWF code: src/cli-workflow.ts:95-133, src/topology.ts:91-120, src/container-lifecycle.ts:44-145, src/services/cli-proxy-service.ts:82-93, containers/cli-proxy/entrypoint.sh:59-97, src/compose-generator.ts:194-256