filesystem.allowWrite breaks agent startup: /tmp/awf-init mountpoint uncreatable when /tmp is narrowed to read-only

[lpcox]

Summary

Any filesystem.allowWrite policy that does not include /tmp prevents the agent container from starting. AWF's own control-plane mount — the iptables init-signal directory bound at /tmp/awf-init — is nested under the user-narrowable /tmp bind and is not treated as internal, so narrowing /tmp to ro makes its mountpoint uncreatable.

This is latent today (nothing in our own CI exercises a live compose start with a policy excluding /tmp), but it was just hit for real: github/gh-aw#55172 bumps DefaultFirewallVersion to v0.28.6, which opens the compiler's allowWrite gate for the Docker runtime, and all six of gh-aw's agentic workflows now fail identically.

Symptom

Error response from daemon: failed to create task for container: ... runc create failed:
unable to start container process: error during container init:
error mounting "/tmp/awf-1787524040566/init-signal" to rootfs at "/tmp/awf-init":
create mountpoint for /tmp/awf-init mount: make mountpoint "/tmp/awf-init":
mkdirat /var/lib/docker/overlay2/<id>/merged/tmp/awf-init: read-only file system

awf-agent never starts → docker compose up -d --pull never exits 1.

Example failing job: https://github.com/github/gh-aw/actions/runs/32670511125/job/97270719242 (5 more with byte-identical errors).

Reproduction

With the policy the gh-aw compiler emits by default (allowWrite: ["/tmp/gh-aw/agent"]), applyFilesystemWritePolicy produces:

/tmp:/tmp:rw                      →  /tmp:/tmp:ro
<workspace>:<workspace>:rw        →  :ro
<initSignalDir>:/tmp/awf-init:rw  →  :ro
(added) /tmp/gh-aw/agent:/tmp/gh-aw/agent:rw

runc applies mounts in order of destination depth, so the read-only /tmp bind is established first; the nested /tmp/awf-init mountpoint then cannot be created (EROFS).

Why the current exemption doesn't help

src/services/agent-volumes/volume-builder.ts:66:

const alwaysWritableMounts = new Set(agentVolumes.filter((spec) =>
  spec.startsWith(`${agentLogsPath}:`) ||
  spec.startsWith(`${sessionStatePath}:`) ||
  spec === '/dev/null:/host/dev/null:rw'
));

${initSignalDir}:/tmp/awf-init:rw (src/services/agent-volumes/workspace-mounts.ts:30, and again for the init container at src/services/agent-service.ts:301) is absent. But adding it is not sufficient — the failure is mkdir of the mountpoint inside a read-only parent, not the mode of the mount itself. Keeping the child rw doesn't help if /tmp is ro and the directory doesn't already exist on the host.

Today with /tmp:/tmp:rw, runc implicitly creates host /tmp/awf-init as a side effect. Narrowing removes that.

Fix options (each has a tradeoff — needs a maintainer call)

A. Move the destination out of the narrowable tree — e.g. /run/awf-init. /run is not bind-mounted, so it lives in the container's writable overlay and is structurally immune to narrowing. Cleanest outcome.

⚠️ The destination is a container-image contract: containers/agent/entrypoint.sh:169 waits on /tmp/awf-init/ready, and containers/agent/setup-iptables.sh:500 writes /tmp/awf-init/iptables-audit.txt. Those are baked into the published agent image, which consumers pin by digest (gh-aw pins agent=sha256:…). A newer CLI with an older pinned image would hang on the ready-file. Needs a coordinated release, and ideally the entrypoint accepting either path for one release cycle.

B. Pre-create the host mountpoint when a policy is active, so runc never needs to mkdir. Preserves the image contract exactly.

⚠️ Requires writing a fixed /tmp/awf-init on the host (small symlink-preplacement surface on shared hosts), and doesn't obviously translate under --docker-host-path-prefix where the daemon's /tmp differs from the runner's. Note DinD is already rejected under allowWrite, which limits but does not eliminate that concern (ARC split-FS prefixing is a separate flag).

C. Force /tmp always-writable whenever internal mounts are nested under it. Trivial, but silently defeats narrowing of /tmp — a security regression for anyone deliberately restricting it. Not recommended.

My preference is A with a transitional dual-path wait in the entrypoint, falling back to B if a coordinated image release isn't desirable.

Suggested scope

• Choose a fix and make AWF's control-plane mounts structurally exempt from user filesystem policy
• Treat init-signal as internal in alwaysWritableMounts regardless of the chosen option
• Audit for any other AWF-internal mount nested under a user-narrowable bind
• Add a live compose-start integration test with a policy that excludes /tmp — this is the coverage gap that let the bug through; unit tests of the policy function alone cannot catch it
• Confirm the equivalent path is safe under Cloud Hypervisor (its exports are planned separately, so it may not be affected)

Related

• Align compiler with gh-aw-firewall v0.28.6 (Cloud Hypervisor filesystem.allowWrite support) gh-aw#55172 — blocked on this; full analysis in this comment
• Align compiler with gh-aw-firewall v0.28.6 (Cloud Hypervisor filesystem.allowWrite support) gh-aw#55169 — compiler alignment tracking issue

_{Found while reviewing CI on the compiler-alignment PR. Reproduced locally against released v0.28.6 policy code.}