fix: preserve agent startup with restricted tmp - #7679
Conversation
There was a problem hiding this comment.
Pull request overview
Moves AWF’s init-signal mount outside user-restricted /tmp to prevent agent startup failures.
Changes:
- Relocates the signal directory to
/run/awf-init. - Adds transitional legacy-path handling and write-policy exemption.
- Adds unit and integration regression coverage.
Show a summary per file
| File | Description |
|---|---|
tests/integration/filesystem-allowwrite.test.ts |
Adds live startup regression test. |
tests/fixtures/awf-runner.ts |
Supports config files in test runs. |
src/services/agent-volumes/workspace-mounts.ts |
Moves the agent signal mount. |
src/services/agent-volumes/workspace-mounts.test.ts |
Updates mount expectation. |
src/services/agent-volumes/volume-builder.ts |
Keeps the signal mount writable. |
src/services/agent-volumes-basic.test.ts |
Tests restricted /tmp volumes. |
src/services/agent-service.ts |
Updates init service signaling. |
src/services/agent-service-build.test.ts |
Tests generated init configuration. |
src/services/agent-environment/core-environment.ts |
Exposes the signal path. |
src/constants.ts |
Defines current and legacy paths. |
containers/agent/setup-iptables.sh |
Writes audits to the selected path. |
containers/agent/entrypoint.sh |
Waits on current or legacy signals. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 12/12 changed files
- Comments generated: 2
- Review effort level: Balanced
| const { agentService, environment, networkConfig, initSignalDir, dockerHostPathPrefix, hostGatewayIp } = params; | ||
| const setupCommand = [ | ||
| 'mkdir -p "$$AWF_INIT_SIGNAL_DIR"', | ||
| `if [ ! -e ${LEGACY_INIT_SIGNAL_DIR} ]; then ln -s "$$AWF_INIT_SIGNAL_DIR" ${LEGACY_INIT_SIGNAL_DIR} 2>/dev/null || true; fi`, |
| } | ||
| }); | ||
|
|
||
| test('starts the legacy Docker agent when /tmp is narrowed read-only', async () => { |
The new tests/integration/filesystem-allowwrite.test.ts was never executed by CI. Every `npm run test:integration` invocation in test-integration-suite.yml is filtered by an explicit --testPathPatterns allowlist, and `filesystem-allowwrite` matched none of the five groups, so the file was collected by the integration jest config but always filtered out. That made it dead coverage for exactly the gap it was added to close: the runc mountpoint-creation-order failure that unit tests of volume rewriting cannot catch. Add it to the Container & Ops group, which already builds the local squid and agent images that the test needs (it runs with buildLocal and legacySecurity). Verified with `jest --listTests`: the file is excluded under the previous pattern and selected under the new one. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Review: plan is complete, with one gap I fixed and one claim to correctI verified the fix end-to-end against the original failure. Summary: the core fix is correct and genuinely covered, one checklist item was dead in CI (fixed in e8c4435), and one checklist item is overstated. ✅ The fix worksRan the exact policy from the failing gh-aw run ( Destination is no longer nested under the narrowable ✅ Regression coverage is real, not decorativeI mutation-tested the ✅ Local validation
🔧 Gap I fixed: the integration test never ran
Proven with
So the one item that closes the coverage gap this bug exposed — live container startup, which unit tests of volume rewriting structurally cannot catch — was dead code. I added it to the Container & Ops job, which already builds the local
|
The filesystem.allowWrite integration test added by this PR failed live in CI with a second instance of the bug it set out to fix: error mounting "/dev/null" to rootfs at "/host/home/runner/.npmrc": make mountpoint "/host/home/runner/.npmrc": openat .npmrc: read-only file system buildCredentialHidingOverlays emitted a /dev/null overlay for every file in the central mount policy, whether or not it existed on the host. A bind mount needs its mountpoint to already exist; runc creates a missing one via openat(O_CREAT) on the parent directory. That silently worked while the $HOME bind was rw, but fails with EROFS as soon as filesystem.allowWrite narrows $HOME to read-only, taking the agent container down before start. Filter the overlays to credential files that actually exist on the host. Verified against a real Docker daemon that masking an existing file inside a read-only bind still succeeds and still yields an empty file, so every credential that could leak is masked exactly as before. A file that does not exist cannot leak anything, so skipping it loses no protection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (6 files)
✨ New Files (2 files)
Coverage comparison generated by |
Replaces the host-existence filter from d4d0614, which was too broad: it also dropped the un-prefixed $HOME overlays that live on the container's own writable rootfs, so credential files stopped appearing as empty files and the credential-hiding integration tests failed with ENOENT. Decide per overlay instead, once the full volume list is known. An overlay is dropped only when the innermost bind covering its mountpoint is read-only *and* the masked path does not exist behind that bind. That is exactly the case runc cannot serve: a bind mount needs its mountpoint to already exist, and creating one under a read-only parent fails with EROFS, killing the agent container before it starts. Nothing is left unmasked. A read-only bind is the only way the agent could reach those paths, and there is nothing behind them to read; the motivating example is $HOME/.docker/config.json, which resolves into the synthesized chroot home because .docker is not a whitelisted home subdirectory. Paths that do exist are still masked, since mounting over an existing path succeeds even inside a read-only bind (verified against a real daemon). The pass is a no-op without filesystem.allowWrite, where every covering bind is read-write; the generated no-policy volume list is byte-identical. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
Third and final instance of the same mountpoint-creation bug class, this time for a real bind rather than a credential overlay. On GitHub-hosted runners the workspace lives under $HOME (/home/runner/work/<repo>/<repo>), so its /host-prefixed bind resolves inside the synthesized chroot home. Docker used to create those parent directories itself, but once filesystem.allowWrite narrows the chroot home bind to read-only runc cannot, and container init dies with: error mounting ".../work/gh-aw-firewall" to rootfs at "/host/home/runner/work/gh-aw-firewall/gh-aw-firewall": mkdirat .../merged/host/home/runner/work: read-only file system prepareChrootHomeMounts already solves exactly this for the whitelisted home tool paths and the runner tool cache, so reuse it for the workspace when the workspace is nested under the effective home. Paths outside $HOME are untouched, keeping current behavior for local runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
Fourth instance of the same class, and the one that kept the agent from running its command once the container finally started: /usr/local/bin/entrypoint.sh: line 1295: /host/tmp/awf-cmd-1.sh: Read-only file system The entrypoint writes the user's command to a script inside the chroot to avoid nested-shell quoting problems. That script lived in /tmp, which filesystem.allowWrite narrows to read-only, so the write failed and the run aborted with exit 1. /run inside the chroot is the container's own writable rootfs rather than a host bind, so it stays writable under any host write policy. This is the same reasoning that moved the init signal to /run/awf-init. Cleanup still removes the script, so there is no residue. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
|
Follow-up to my earlier review. The integration test this PR added was dead in CI, so I wired it up — and once it actually ran, it failed, exposing three more instances of the very bug class this PR set out to fix. All are now fixed and the full integration suite is green: run 32682897449, Root cause, stated onceA bind mount needs its mountpoint to already exist. What I fixed on top of your commits
1. Credential overlays ( Nothing is left unmasked: a read-only bind is the only way the agent could reach those paths and there is nothing behind them to read. The motivating case is I first shipped this as a plain host-existence filter ( 2. Workspace mountpoint ( 3. Command script ( Evidence the no-policy path is untouchedThe prune pass is a no-op without a policy, because every covering bind is read-write then. I generated the full agent volume list at Validation
Still open — filed as #7681, deliberately not bundled here
Non-fatal, so it does not block this PR, but it is a security downgrade triggered by opting into a security feature. The fix is the same One correction to carry into the descriptionThe backward-compatibility claim is still the wrong direction. New image + old CLI works. New CLI + old pinned image does not: the agent mounts only |
|
✅ Smoke Claude passed
|
|
❌ Smoke Gemini reports failed. Facets need polishing...
|
|
🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "example.com"See Network Configuration for more information.
|
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓
|
|
🔌 Smoke Services — All services reachable! ✅
|
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓
|
|
Smoke Test: Copilot BYOK (Direct) Mode ✅ PASS
All tests passed. Running in direct BYOK mode.
|
Smoke Test: Copilot Engine
Overall: PASS cc
|
|
EGRESS_RESULT allow=pass deny=pass ✅ Allowed domain (api.github.com): HTTP 200 Overall status: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "example.com"See Network Configuration for more information.
|
Smoke Test: Claude Engine Validation
Overall result: PASS
|
Smoke Test: API Proxy OTEL Tracing — Results
Overall: all 5 scenarios passed.
|
|
Smoke Test: Services Connectivity
Overall: FAIL —
|
Chroot Version Comparison Results
Overall: FAILED — Node.js version differs between host and chroot environment (
|
Smoke Test
Warning Firewall blocked 3 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "msfeed2.pkgs.visualstudio.com"
- "msfeed25.pkgs.visualstudio.com"
- "registry.npmjs.org"See Network Configuration for more information.
|
|
Docker Sbx Smoke Test
Overall: PASS cc
|
|
PR titles: Tests: MCP: ❌ GH.com: ✅ File I/O: ✅ BYOK: ✅
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — PASS Notes:
|
|
|
Fixes #7678
filesystem.allowWritebroke agent startup. The root cause turned out to be aclass of bug, not a single path.
A bind mount needs its mountpoint to already exist. runc creates a missing one
with
mkdiratagainst the destination, which resolves into whichever bindalready covers that path. That silently works while every covering bind is
read-write, and fails with
EROFSthe momentfilesystem.allowWritenarrowsone to read-only.
/tmp/awf-initwas the reported symptom; four more instanceswere found and fixed, three of them only after the PR's own integration test was
wired into CI (it was being filtered out of every
--testPathPatternsgroup andhad never actually run).
What changed
Init signal directory — moved from
/tmp/awf-initto/run/awf-init, andthe mount is treated as AWF-internal/always-writable. Compose env references are
escaped so Docker Compose does not host-interpolate them.
Nested mountpoints, derived from the final topology.
planNestedMountpoints()reports every mountpoint that would land inside a read-only cover;
ensureNestedMountpoints()creates the directories and then fails closed if anyis still missing, rather than letting container init die with an opaque read-only
filesystem error. This is computed from the final, policy-applied volume list, so
internal mounts added later are covered without another targeted fix. It fixes
$HOME/.copilot/logsandsession-state— which sit under whichever of the emptychroot home or the real
~/.copilotends up covering them — and the workspacemountpoint on GitHub-hosted runners, where the workspace is nested under
$HOME.Credential overlays.
/dev/nullmasks whose target does not exist behind aread-only cover are unmountable; they are now dropped, which loses no protection
because the read-only bind is the only way the agent could reach the path and
there is nothing there to read. Sources are resolved back to runner-local paths
before probing: on split-filesystem runners (
--docker-host-path-prefix) custommounts carry daemon-side sources, so probing them with runner-local
fscoulddrop a real credential mask. Unknown sources keep the overlay (fail closed).
Helper staging moved off
/tmp. The one-shot-tokenLD_PRELOADlibrary, thegh CLI proxy wrapper, the Claude key helper and the CA bundles staged under
/tmp/awf-lib, a bind of the host's/tmp. A write policy narrows it toread-only and every copy failed — silently, leaving token protection and gh proxy
mediation disabled while the run looked healthy. They now stage under
/run/awf-libon the container's own writable rootfs, and both security-criticalhelpers fail closed when enabled. The chroot command script moved to
/runforthe same reason.
File mountpoints, not just directories. Preparation assumed every mountpoint
was a directory and silently dropped any bind whose source did not resolve to a
local directory — so a regular-file bind under a read-only cover was skipped
rather than created or refused. It is reachable with
--docker-host-path-prefix:stageHostFilepublishes a staged binary to/tmp/awf-runner-bin/<name>while apolicy narrows
/tmpto read-only. The local-source resolver cannot attribute thestaged path because it lives under the prefix, so staging is now recorded at the
point of truth and the kind is derived from that first, then from a stat of the
resolved source. File mountpoints are created with an exclusive open, and a
required mountpoint that cannot be classified now refuses to launch instead of
being discarded. Kind matters: against a real daemon a missing mountpoint fails
with
read-only file system, and a directory standing in for a file fails withnot a directory.A shared path prefix is not a daemon-only one. The same resolver drew one
more line in the wrong place: it treated any source under
--docker-host-path-prefixas daemon-side. But a/tmp-rooted prefix is theARC/DinD shared-volume shape, where both sides see the same bytes at the same
path — AWF already depends on that, staging files there with ordinary local
fscalls, and prefix translation deliberately leaves an already-
/tmpsourceunrewritten. With
--docker-host-path-prefix /tmpthe run's own workDir sitsinside the prefix, so
/tmp/awf-<ts>/{init-signal,agent-logs,agent-session-state}all became unresolvable: under a policy those binds are nested in a narrowed
read-only cover, so their kind could not be classified and the run failed closed
before launch, while the chroot home lost its covering source and skipped
preparation entirely — reinstating the EROFS failure this PR exists to fix. The
distinction is now named once as
isSharedDockerHostPathPrefixand shared by thethree places that had each grown a private copy of the
/tmptest (the resolver,daemon staging, and
/etcidentity-file preservation); keeping separate copies iswhat allowed them to disagree. Daemon-only prefixes such as
/hoststill failclosed, unchanged. Earlier split-filesystem tests missed this because the prefix
and the workDir were siblings rather than nested.
Agent image compatibility, honestly. The original approach created a symlink
at the legacy path inside the iptables-init container. That container has its own
mount namespace and rootfs, so the symlink was never visible to the agent — a new
CLI with an older pinned
--image-tagwould have hung for 30s and failed. Thesymlink is removed and replaced by binds of the same host source, because a bind
crosses the namespace boundary that a symlink cannot.
Both containers need the legacy view, for different reasons, so both get one:
/tmp/awf-init, read-onlyreadyandoutput.log./tmp/awf-init, read-writesetup-iptables.shruns underset -eand hardcodes its audit dump to/tmp/awf-init/iptables-audit.txt.That second row was the remaining failure. Without it the audit redirect fails,
set -eaborts the script before the CLI's chainedtouch .../ready, and theagent waits 30s for a signal that is never written. Because the audit dump is the
last step, the iptables rules had in fact been applied — the run failed opaquely
on a missing signal alone. The test executes the old script's audit step under
set -ein a rootfs staged from the generated init volumes, and a negativecontrol proves the test detects the regression.
/run/awf-init, read-write/tmp/awf-initbound in both containersValidation
default
TMPDIRandTMPDIR=/tmp(macOS resolves/tmpto/private/tmp, whichhides nesting bugs that only appear on Linux CI).
credential tests; removing mountpoint creation fails 2 topology tests; removing
the legacy init bind fails 2 old-script contract tests; dropping the staging
metadata fails 2 split-filesystem tests; disabling the unclassifiable fail-closed
branch fails 1; reverting the shared-prefix exemption fails exactly the 3 new
nested-prefix tests; dropping prefix normalisation fails 3 fail-closed tests.
daemon sees runner path
Xat<prefix>/X, anddind-probeonly offers itscandidates (
/host,/runner,/tmp/gh-aw) once a split is confirmed. Thesingle exception is the literal
/tmp: only a user can supply it, AWF's ownworkDir lives under it, and its binds are never translated, so the run only
works if
/tmpreally is shared.isSharedDockerHostPathPrefixis thereforeexactly
/tmp;isTmpRootedDockerHostPathPrefixkeeps the older meaning andstill gates where AWF stages, so
/tmp/gh-awrunners keep binary and/etcstaging. A normalised
/is treated as no prefix, matchingtranslateBindMountHostPath, instead of marking every absolute sourceunresolvable.
unprivileged caller cannot give a directory away, and macOS denies chown to
non-root outright. Only that case is tolerated -- any other error, and any
error while privileged, still propagates.
its prefix from the suite's own tmpdir, which is
/tmp/...(shared) on LinuxCI but
/private/var/...on macOS, so it asserted the opposite of its name onthe platform that runs it.
TMPDIR=/tmpcannot reproduce that, becauserealpathSyncmaps it straight back. It now uses a genuinely daemon-onlyprefix, and new string-level resolver tests hardcode both shapes so this class
of Linux-only divergence cannot hide behind a green local run.
assert zero unsatisfiable mountpoints.
by an AWF-owned bind, so nothing in the agent's home tree is created. The
requirements that do exist without a policy are pre-existing system binds
(a GitHub-hosted runner nests the tool cache inside a read-only
/opt) whosemountpoint resolves to the mount's own source and therefore already exists.
path inside a read-only bind succeeds and still reads as empty; over a missing
path it fails with
read-only file system; and a directory where a filemountpoint is required fails with
not a directory./host/runis on the containeroverlay and is writable and executable, so it is a valid home for an
LD_PRELOADlibrary.tsc --noEmit(src and tests),npm run build,bash -n,shellcheckclean(one pre-existing SC2145 on
main, untouched).filesystem.allowWritetests (agent startup under a narrowed read-only/tmp,the one-shot-token library present at
/run/awf-lib, and no host mount residueafter teardown).
resolveComposeFilesystemAllowWrite()returnsundefinedfor non-compose runtimes, so no cover is ever narrowed and the newpreparation pass is a no-op.