Skip to content

fix: preserve agent startup with restricted tmp - #7679

Merged
lpcox merged 18 commits into
mainfrom
copilot/fix-filesystem-allowwrite-issue
Aug 24, 2026
Merged

fix: preserve agent startup with restricted tmp#7679
lpcox merged 18 commits into
mainfrom
copilot/fix-filesystem-allowwrite-issue

Conversation

Copilot AI commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Fixes #7678

filesystem.allowWrite broke agent startup. The root cause turned out to be a
class of bug, not a single path.

A bind mount needs its mountpoint to already exist. runc creates a missing one
with mkdirat against the destination, which resolves into whichever bind
already covers that path. That silently works while every covering bind is
read-write, and fails with EROFS the moment filesystem.allowWrite narrows
one to read-only. /tmp/awf-init was the reported symptom; four more instances
were found and fixed, three of them only after the PR's own integration test was
wired into CI (it was being filtered out of every --testPathPatterns group and
had never actually run).

What changed

Init signal directory — moved from /tmp/awf-init to /run/awf-init, and
the mount is treated as AWF-internal/always-writable. Compose env references are
escaped so Docker Compose does not host-interpolate them.

Nested mountpoints, derived from the final topology. planNestedMountpoints()
reports every mountpoint that would land inside a read-only cover;
ensureNestedMountpoints() creates the directories and then fails closed if any
is still missing, rather than letting container init die with an opaque read-only
filesystem error. This is computed from the final, policy-applied volume list, so
internal mounts added later are covered without another targeted fix. It fixes
$HOME/.copilot/logs and session-state — which sit under whichever of the empty
chroot home or the real ~/.copilot ends up covering them — and the workspace
mountpoint on GitHub-hosted runners, where the workspace is nested under $HOME.

Credential overlays. /dev/null masks whose target does not exist behind a
read-only cover are unmountable; they are now dropped, which loses no protection
because the read-only bind is the only way the agent could reach the path and
there is nothing there to read. Sources are resolved back to runner-local paths
before probing: on split-filesystem runners (--docker-host-path-prefix) custom
mounts carry daemon-side sources, so probing them with runner-local fs could
drop a real credential mask. Unknown sources keep the overlay (fail closed).

Helper staging moved off /tmp. The one-shot-token LD_PRELOAD library, the
gh CLI proxy wrapper, the Claude key helper and the CA bundles staged under
/tmp/awf-lib, a bind of the host's /tmp. A write policy narrows it to
read-only and every copy failed — silently, leaving token protection and gh proxy
mediation disabled while the run looked healthy. They now stage under
/run/awf-lib on the container's own writable rootfs, and both security-critical
helpers fail closed when enabled. The chroot command script moved to /run for
the same reason.

File mountpoints, not just directories. Preparation assumed every mountpoint
was a directory and silently dropped any bind whose source did not resolve to a
local directory — so a regular-file bind under a read-only cover was skipped
rather than created or refused. It is reachable with --docker-host-path-prefix:
stageHostFile publishes a staged binary to /tmp/awf-runner-bin/<name> while a
policy narrows /tmp to read-only. The local-source resolver cannot attribute the
staged path because it lives under the prefix, so staging is now recorded at the
point of truth and the kind is derived from that first, then from a stat of the
resolved source. File mountpoints are created with an exclusive open, and a
required mountpoint that cannot be classified now refuses to launch instead of
being discarded. Kind matters: against a real daemon a missing mountpoint fails
with read-only file system, and a directory standing in for a file fails with
not a directory.

A shared path prefix is not a daemon-only one. The same resolver drew one
more line in the wrong place: it treated any source under
--docker-host-path-prefix as daemon-side. But a /tmp-rooted prefix is the
ARC/DinD shared-volume shape, where both sides see the same bytes at the same
path — AWF already depends on that, staging files there with ordinary local fs
calls, and prefix translation deliberately leaves an already-/tmp source
unrewritten. With --docker-host-path-prefix /tmp the run's own workDir sits
inside the prefix, so /tmp/awf-<ts>/{init-signal,agent-logs,agent-session-state}
all became unresolvable: under a policy those binds are nested in a narrowed
read-only cover, so their kind could not be classified and the run failed closed
before launch, while the chroot home lost its covering source and skipped
preparation entirely — reinstating the EROFS failure this PR exists to fix. The
distinction is now named once as isSharedDockerHostPathPrefix and shared by the
three places that had each grown a private copy of the /tmp test (the resolver,
daemon staging, and /etc identity-file preservation); keeping separate copies is
what allowed them to disagree. Daemon-only prefixes such as /host still fail
closed, unchanged. Earlier split-filesystem tests missed this because the prefix
and the workDir were siblings rather than nested.

Agent image compatibility, honestly. The original approach created a symlink
at the legacy path inside the iptables-init container. That container has its own
mount namespace and rootfs, so the symlink was never visible to the agent — a new
CLI with an older pinned --image-tag would have hung for 30s and failed. The
symlink is removed and replaced by binds of the same host source, because a bind
crosses the namespace boundary that a symlink cannot.

Both containers need the legacy view, for different reasons, so both get one:

Container Legacy mount Why
agent /tmp/awf-init, read-only An old entrypoint only polls ready and output.log.
iptables-init /tmp/awf-init, read-write An old setup-iptables.sh runs under set -e and hardcodes its audit dump to /tmp/awf-init/iptables-audit.txt.

That second row was the remaining failure. Without it the audit redirect fails,
set -e aborts the script before the CLI's chained touch .../ready, and the
agent waits 30s for a signal that is never written. Because the audit dump is the
last step, the iptables rules had in fact been applied — the run failed opaquely
on a missing signal alone. The test executes the old script's audit step under
set -e in a rootfs staged from the generated init volumes, and a negative
control proves the test detects the regression.

CLI Agent image Mechanism
new new /run/awf-init, read-write
new old /tmp/awf-init bound in both containers
old new entrypoint polls both paths

Validation

  • 323 unit suites / 5159 tests pass (+4 suites, +81 tests), under both the
    default TMPDIR and TMPDIR=/tmp (macOS resolves /tmp to /private/tmp, which
    hides nesting bugs that only appear on Linux CI).
  • Every new guard was mutation-tested: removing the local-source resolver fails 3
    credential tests; removing mountpoint creation fails 2 topology tests; removing
    the legacy init bind fails 2 old-script contract tests; dropping the staging
    metadata fails 2 split-filesystem tests; disabling the unclassifiable fail-closed
    branch fails 1; reverting the shared-prefix exemption fails exactly the 3 new
    nested-prefix tests; dropping prefix normalisation fails 3 fail-closed tests.
  • A Docker host path prefix is daemon-only by construction -- it means the
    daemon sees runner path X at <prefix>/X, and dind-probe only offers its
    candidates (/host, /runner, /tmp/gh-aw) once a split is confirmed. The
    single exception is the literal /tmp: only a user can supply it, AWF's own
    workDir lives under it, and its binds are never translated, so the run only
    works if /tmp really is shared. isSharedDockerHostPathPrefix is therefore
    exactly /tmp; isTmpRootedDockerHostPathPrefix keeps the older meaning and
    still gates where AWF stages, so /tmp/gh-aw runners keep binary and /etc
    staging. A normalised / is treated as no prefix, matching
    translateBindMountHostPath, instead of marking every absolute source
    unresolvable.
  • Preparing a mountpoint no longer dies on a chown that cannot succeed: an
    unprivileged caller cannot give a directory away, and macOS denies chown to
    non-root outright. Only that case is tolerated -- any other error, and any
    error while privileged, still propagates.
  • One pre-existing test had encoded the bug rather than the behaviour: it built
    its prefix from the suite's own tmpdir, which is /tmp/... (shared) on Linux
    CI but /private/var/... on macOS, so it asserted the opposite of its name on
    the platform that runs it. TMPDIR=/tmp cannot reproduce that, because
    realpathSync maps it straight back. It now uses a genuinely daemon-only
    prefix, and new string-level resolver tests hardcode both shapes so this class
    of Linux-only divergence cannot hide behind a green local run.
  • Tests model runc mount ordering against the real generated volume list and
    assert zero unsatisfiable mountpoints.
  • No-policy behaviour verified unchanged: no mountpoint requirement is covered
    by an AWF-owned bind, so nothing in the agent's home tree is created. The
    requirements that do exist without a policy are pre-existing system binds
    (a GitHub-hosted runner nests the tool cache inside a read-only /opt) whose
    mountpoint resolves to the mount's own source and therefore already exists.
  • Empirically confirmed against a real Docker daemon: a bind over an existing
    path inside a read-only bind succeeds and still reads as empty; over a missing
    path it fails with read-only file system; and a directory where a file
    mountpoint is required fails with not a directory. /host/run is on the container
    overlay and is writable and executable, so it is a valid home for an
    LD_PRELOAD library.
  • tsc --noEmit (src and tests), npm run build, bash -n, shellcheck clean
    (one pre-existing SC2145 on main, untouched).
  • Live integration suite green: all 5 jobs pass, including the 3 real-Docker
    filesystem.allowWrite tests (agent startup under a narrowed read-only /tmp,
    the one-shot-token library present at /run/awf-lib, and no host mount residue
    after teardown).
  • Cloud Hypervisor is unaffected: resolveComposeFilesystemAllowWrite() returns
    undefined for non-compose runtimes, so no cover is ever narrowed and the new
    preparation pass is a no-op.

@lpcox lpcox changed the title [WIP] Fix filesystem.allowWrite breaking agent startup Fix filesystem.allowWrite breaking agent startup Aug 24, 2026
@lpcox
lpcox marked this pull request as ready for review August 24, 2026 01:01
Copilot AI balanced review requested due to automatic review settings August 24, 2026 01:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Moves AWF’s init-signal mount outside user-restricted /tmp to prevent agent startup failures.

Changes:

  • Relocates the signal directory to /run/awf-init.
  • Adds transitional legacy-path handling and write-policy exemption.
  • Adds unit and integration regression coverage.
Show a summary per file
File Description
tests/integration/filesystem-allowwrite.test.ts Adds live startup regression test.
tests/fixtures/awf-runner.ts Supports config files in test runs.
src/services/agent-volumes/workspace-mounts.ts Moves the agent signal mount.
src/services/agent-volumes/workspace-mounts.test.ts Updates mount expectation.
src/services/agent-volumes/volume-builder.ts Keeps the signal mount writable.
src/services/agent-volumes-basic.test.ts Tests restricted /tmp volumes.
src/services/agent-service.ts Updates init service signaling.
src/services/agent-service-build.test.ts Tests generated init configuration.
src/services/agent-environment/core-environment.ts Exposes the signal path.
src/constants.ts Defines current and legacy paths.
containers/agent/setup-iptables.sh Writes audits to the selected path.
containers/agent/entrypoint.sh Waits on current or legacy signals.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 12/12 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread src/services/agent-service.ts Outdated
const { agentService, environment, networkConfig, initSignalDir, dockerHostPathPrefix, hostGatewayIp } = params;
const setupCommand = [
'mkdir -p "$$AWF_INIT_SIGNAL_DIR"',
`if [ ! -e ${LEGACY_INIT_SIGNAL_DIR} ]; then ln -s "$$AWF_INIT_SIGNAL_DIR" ${LEGACY_INIT_SIGNAL_DIR} 2>/dev/null || true; fi`,
}
});

test('starts the legacy Docker agent when /tmp is narrowed read-only', async () => {
The new tests/integration/filesystem-allowwrite.test.ts was never executed
by CI. Every `npm run test:integration` invocation in
test-integration-suite.yml is filtered by an explicit --testPathPatterns
allowlist, and `filesystem-allowwrite` matched none of the five groups, so
the file was collected by the integration jest config but always filtered
out.

That made it dead coverage for exactly the gap it was added to close: the
runc mountpoint-creation-order failure that unit tests of volume rewriting
cannot catch.

Add it to the Container & Ops group, which already builds the local squid
and agent images that the test needs (it runs with buildLocal and
legacySecurity).

Verified with `jest --listTests`: the file is excluded under the previous
pattern and selected under the new one.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@lpcox

lpcox commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Review: plan is complete, with one gap I fixed and one claim to correct

I verified the fix end-to-end against the original failure. Summary: the core fix is correct and genuinely covered, one checklist item was dead in CI (fixed in e8c4435), and one checklist item is overstated.

✅ The fix works

Ran the exact policy from the failing gh-aw run (allowWrite: ["/tmp/gh-aw/agent"]) through applyFilesystemWritePolicy on this branch:

/tmp:/tmp:ro
/home/runner/work/x/x:...:ro
/tmp/awf-.../init-signal:/run/awf-init:rw   ← was :/tmp/awf-init:ro
/tmp/gh-aw/agent:/tmp/gh-aw/agent:rw

Destination is no longer nested under the narrowable /tmp bind, and it stays rw. That removes the EROFS mountpoint-creation failure.

✅ Regression coverage is real, not decorative

I mutation-tested the alwaysWritableMounts addition — deleting spec.startsWith(\${initSignalDir}:`)failsagent-volumes-basic.test.ts:66. And the assertion sits inside a test with an **active** filesystemAllowWriteand/tmp:/tmp:ro`, so it guards the real condition rather than the happy path. Both halves of the fix are load-bearing.

✅ Local validation

  • Full unit suite: 319 suites / 5069 tests pass
  • npm run build: clean
  • bash -n on both modified scripts: clean
  • shellcheck -S error: only SC2145 at entrypoint.sh:536, which I confirmed is pre-existing on main (2 occurrences) and outside this diff

🔧 Gap I fixed: the integration test never ran

tests/integration/filesystem-allowwrite.test.ts was collected by the integration jest config but filtered out by CI. Every npm run test:integration in test-integration-suite.yml is constrained by an explicit --testPathPatterns allowlist, and filesystem-allowwrite matched none of the five groups.

Proven with jest --listTests:

pattern selects the file?
previous Container & Ops group no
updated group (e8c4435) yes

So the one item that closes the coverage gap this bug exposed — live container startup, which unit tests of volume rewriting structurally cannot catch — was dead code. I added it to the Container & Ops job, which already builds the local squid and agent images the test needs (buildLocal: true, legacySecurity: true). Note these jobs are label-gated on ready-for-aw for PRs, so the test still needs that label applied on this PR to actually prove the fix.

⚠️ Claim to correct: "Preserve compatibility for existing agent image /tmp/awf-init contract"

What's implemented is the opposite skew direction from the one flagged in #7678.

  • New image + old CLI — handled. The new entrypoint.sh waits on either ${AWF_INIT_SIGNAL_DIR:-/run/awf-init}/ready or the legacy path, so an old CLI mounting /tmp/awf-init still works. ✅
  • New CLI + old pinned agent imagenot handled. The agent now mounts only /run/awf-init (and agent-volumes-basic.test.ts explicitly asserts not.toContain(...:/tmp/awf-init:rw)). An old image's entrypoint.sh hardcodes /tmp/awf-init/ready, which never appears → 30s timeout → hard failure.

The ln -s in the init container doesn't bridge this: it lives in the init container's own mount namespace (the two share only the network namespace), so it helps that container's own old setup-iptables.sh audit write, but is invisible to the agent container.

This matters because pinning is exactly how gh-aw consumes AWF (imageTag: "0.28.6,agent=sha256:…"), and --image-tag defaults to latest, so the mismatch only appears for explicit pins.

I don't think this should be "fixed" by adding a legacy mount — mounting at /tmp/awf-init would reintroduce the very EROFS bug being fixed. The two goals are mutually exclusive. Better to drop the compatibility claim and state that the CLI and agent image must be upgraded together, then make sure the gh-aw bump moves agent=sha256:… and the CLI version in lockstep.

Suggestion

Please apply ready-for-aw before merging — otherwise the integration test still won't have executed even after e8c4435, and the live startup path would ship unproven for a second time.

Reviewed on branch copilot/fix-filesystem-allowwrite-issue; all checks run locally.

The filesystem.allowWrite integration test added by this PR failed live in
CI with a second instance of the bug it set out to fix:

  error mounting "/dev/null" to rootfs at "/host/home/runner/.npmrc":
  make mountpoint "/host/home/runner/.npmrc": openat .npmrc:
  read-only file system

buildCredentialHidingOverlays emitted a /dev/null overlay for every file in
the central mount policy, whether or not it existed on the host. A bind
mount needs its mountpoint to already exist; runc creates a missing one via
openat(O_CREAT) on the parent directory. That silently worked while the
$HOME bind was rw, but fails with EROFS as soon as filesystem.allowWrite
narrows $HOME to read-only, taking the agent container down before start.

Filter the overlays to credential files that actually exist on the host.
Verified against a real Docker daemon that masking an existing file inside
a read-only bind still succeeds and still yields an empty file, so every
credential that could leak is masked exactly as before. A file that does
not exist cannot leak anything, so skipping it loses no protection.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 93.83% 93.87% 📈 +0.04%
Statements 92.71% 92.74% 📈 +0.03%
Functions 93.12% 93.16% 📈 +0.04%
Branches 86.05% 86.12% 📈 +0.07%
📁 Per-file Coverage Changes (6 files)
File Lines (Before → After) Statements (Before → After)
src/services/agent-volumes/docker-host-staging.ts 95.7% → 93.8% (-1.99%) 95.9% → 93.8% (-2.16%)
src/services/host-path-prefix.ts 100.0% → 100.0% (+0.00%) 100.0% → 97.6% (-2.44%)
src/services/agent-volumes/credential-hiding.ts 100.0% → 100.0% (+0.00%) 100.0% → 97.6% (-2.44%)
src/services/agent-volumes/workspace-mounts.ts 96.5% → 96.6% (+0.06%) 96.6% → 96.7% (+0.05%)
src/fs-utils.ts 98.3% → 98.5% (+0.15%) 98.3% → 98.5% (+0.15%)
src/log-directory-setup.ts 96.2% → 100.0% (+3.78%) 96.3% → 100.0% (+3.71%)
✨ New Files (2 files)
  • src/services/agent-volumes/mount-topology.ts: 100.0% lines
  • src/services/agent-volumes/nested-mountpoints.ts: 92.8% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

lpcox and others added 3 commits August 23, 2026 18:51
Replaces the host-existence filter from d4d0614, which was too broad: it
also dropped the un-prefixed $HOME overlays that live on the container's own
writable rootfs, so credential files stopped appearing as empty files and
the credential-hiding integration tests failed with ENOENT.

Decide per overlay instead, once the full volume list is known. An overlay
is dropped only when the innermost bind covering its mountpoint is read-only
*and* the masked path does not exist behind that bind. That is exactly the
case runc cannot serve: a bind mount needs its mountpoint to already exist,
and creating one under a read-only parent fails with EROFS, killing the
agent container before it starts.

Nothing is left unmasked. A read-only bind is the only way the agent could
reach those paths, and there is nothing behind them to read; the motivating
example is $HOME/.docker/config.json, which resolves into the synthesized
chroot home because .docker is not a whitelisted home subdirectory. Paths
that do exist are still masked, since mounting over an existing path
succeeds even inside a read-only bind (verified against a real daemon).

The pass is a no-op without filesystem.allowWrite, where every covering bind
is read-write; the generated no-policy volume list is byte-identical.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
Third and final instance of the same mountpoint-creation bug class, this
time for a real bind rather than a credential overlay.

On GitHub-hosted runners the workspace lives under $HOME
(/home/runner/work/<repo>/<repo>), so its /host-prefixed bind resolves
inside the synthesized chroot home. Docker used to create those parent
directories itself, but once filesystem.allowWrite narrows the chroot home
bind to read-only runc cannot, and container init dies with:

  error mounting ".../work/gh-aw-firewall" to rootfs at
  "/host/home/runner/work/gh-aw-firewall/gh-aw-firewall":
  mkdirat .../merged/host/home/runner/work: read-only file system

prepareChrootHomeMounts already solves exactly this for the whitelisted
home tool paths and the runner tool cache, so reuse it for the workspace
when the workspace is nested under the effective home. Paths outside $HOME
are untouched, keeping current behavior for local runs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
Fourth instance of the same class, and the one that kept the agent from
running its command once the container finally started:

  /usr/local/bin/entrypoint.sh: line 1295:
  /host/tmp/awf-cmd-1.sh: Read-only file system

The entrypoint writes the user's command to a script inside the chroot to
avoid nested-shell quoting problems. That script lived in /tmp, which
filesystem.allowWrite narrows to read-only, so the write failed and the run
aborted with exit 1.

/run inside the chroot is the container's own writable rootfs rather than a
host bind, so it stays writable under any host write policy. This is the
same reasoning that moved the init signal to /run/awf-init. Cleanup still
removes the script, so there is no residue.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 248b577f-2e6f-4e03-90bc-96c75c0d395e
@lpcox

lpcox commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to my earlier review. The integration test this PR added was dead in CI, so I wired it up — and once it actually ran, it failed, exposing three more instances of the very bug class this PR set out to fix. All are now fixed and the full integration suite is green: run 32682897449, PASS tests/integration/filesystem-allowwrite.test.ts (25.3 s).

Root cause, stated once

A bind mount needs its mountpoint to already exist. runc creates a missing one with openat/mkdirat on the parent. That silently works while everything is read-write, and fails with EROFS the moment filesystem.allowWrite narrows a bind to read-only. Same for any AWF-internal write into a narrowed path. The original /tmp/awf-init bug was one symptom of this; there were four.

What I fixed on top of your commits

# Commit Symptom
0 e8c44356 Test was collected but filtered out by every --testPathPatterns allowlist, so it never ran
1 7b3593da error mounting "/dev/null" to rootfs at "/host/home/runner/.npmrc" — credential overlays
2 82a45a53 mkdirat .../host/home/runner/work: read-only file system — the workspace bind
3 0cb81ea0 entrypoint.sh: /host/tmp/awf-cmd-1.sh: Read-only file system — the command script

1. Credential overlays (credential-hiding.ts, volume-builder.ts). buildCredentialHidingOverlays emitted /dev/null overlays unconditionally. New pruneUnmountableCredentialOverlays pass drops an overlay only when the innermost bind covering its mountpoint is read-only and the masked path does not exist behind that bind.

Nothing is left unmasked: a read-only bind is the only way the agent could reach those paths and there is nothing behind them to read. The motivating case is $HOME/.docker/config.json, which resolves into the synthesized chroot home because .docker is not a whitelisted home subdirectory — the agent cannot see the host's real .docker at all. Paths that do exist are still masked, because mounting over an existing path succeeds even inside a read-only bind. I verified that against a real daemon rather than assuming it:

overlay on EXISTING file, parent bind ro  -> OK, file reads as 0 bytes (still masked)
overlay on MISSING  file, parent bind ro  -> read-only file system

I first shipped this as a plain host-existence filter (d4d06141). That was too broad — it also dropped the un-prefixed $HOME overlays that live on the container's writable rootfs, so credential files stopped existing as empty files and credential-hiding.test.ts failed with ENOENT. 7b3593da replaces it with the per-overlay rule. Worth noting the sysroot-only dropUnbackedHostHomeOverlays guard uses a different predicate (no writable backing) and still throws for the genuinely dangerous case, so it is not disarmed.

2. Workspace mountpoint (chroot-home-setup.ts). On GitHub-hosted runners the workspace is under $HOME, so its /host-prefixed bind lands inside the chroot home. prepareChrootHomeMounts already solves exactly this for home tool paths and the tool cache; it just was not applied to the workspace.

3. Command script (entrypoint.sh). Moved /tmp/awf-cmd-$$.sh to /run/awf-cmd-$$.sh, same reasoning as your /run/awf-init move. Cleanup still removes it.

Evidence the no-policy path is untouched

The prune pass is a no-op without a policy, because every covering bind is read-write then. I generated the full agent volume list at e8c44356 and at 0cb81ea0 and diffed:

no-policy volume list: IDENTICAL (103 mounts)
with policy:           33 overlays kept, 30 unmountable /host overlays dropped

Validation

  • Full unit suite 319 suites / 5078 tests pass (+9 new: 7 for the prune rule, 2 for workspace prep)
  • Mutation-tested every new guard — removing the existence check, the rootfs case, or the workspace prep each fails a test
  • npm run build / tsc --noEmit clean, bash -n clean
  • shellcheck SC2145 at entrypoint.sh:536 is pre-existing on main, not from this PR
  • Integration suite green on all five jobs

Still open — filed as #7681, deliberately not bundled here

filesystem.allowWrite silently disables one-shot token protection and the gh CLI proxy wrapper, because both stage into /tmp/awf-lib, which the policy narrows. Observed in run 32681985728:

[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib
[entrypoint][WARN] Token protection will be disabled (tokens may be readable multiple times)

Non-fatal, so it does not block this PR, but it is a security downgrade triggered by opting into a security feature. The fix is the same /run relocation across ~49 references in security-sensitive shell code, and it needs confirmation that /run is not noexec before an LD_PRELOAD .so is served from it — too risky to bolt onto a now-green PR. #7681 also covers /tmp/awf-runner-bin, which has the same hazard on ARC/DinD.

One correction to carry into the description

The backward-compatibility claim is still the wrong direction. New image + old CLI works. New CLI + old pinned image does not: the agent mounts only /run/awf-init, and agent-volumes-basic.test.ts asserts /tmp/awf-init:rw is absent, so an old entrypoint waits on /tmp/awf-init/ready until it times out. The ln -s in the init container does not bridge this — init and agent share only the network namespace, not the mount namespace. Adding a legacy mount would reintroduce the EROFS bug, so the goals are mutually exclusive. Best to drop the claim and document that the CLI and the agent image must move in lockstep, which matters because gh-aw pins --image-tag.

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

Generated by Smoke Claude for #7679

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@lpcox
lpcox deployed to aoai-model August 24, 2026 13:45 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode ✅ PASS

  • ✅ GitHub MCP connectivity: Verified 2 recent merged PRs
  • ✅ GitHub.com connectivity: HTTP 200
  • ✅ File write/read: Confirmed at /tmp/gh-aw/agent/smoke-test-copilot-byok.txt
  • ✅ BYOK inference: Active (COPILOT_PROVIDER_API_KEY → api-proxy → api.githubcopilot.com)

All tests passed. Running in direct BYOK mode.

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Engine

Overall: PASS

cc @lpcox

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox Smoke test: Copilot Network Isolation Egress

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (api.github.com): HTTP 200
✅ Blocked domain (example.com): CONNECT tunnel failed 403 (denied)

Overall status: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Status
API ✅ PASS
gh CLI ✅ PASS
File ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #7679 · haiku45 · 55.7 AIC · ⊞ 4.5K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: API Proxy OTEL Tracing — Results

  • Scenario 1 (Module Loading): otel.js loads cleanly, exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled (plus internal helpers).
  • Scenario 2 (Test Suite): otel.test.js, otel-fanout.test.js, otel-workload-identity.test.js — 3 suites, 68/68 tests passed.
  • Scenario 3 (Env Var Forwarding): env-passthrough.ts forwards COPILOT_OTEL_FILE_EXPORTER_PATH, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID into the agent container; api-proxy-env-config.ts (buildOtelEnv) forwards OTLP endpoint/headers, service name, and parent trace context into the api-proxy container.
  • Scenario 4 (Token Tracker Integration): onUsage callback confirmed present in token-tracker-http.js as the OTEL hook point (invoked after normalized usage extraction).
  • Scenario 5 (OTEL Diagnostics): Spans were exported this run — /tmp/gh-aw/otel.jsonl contains a valid OTLP resourceSpans payload with GitHub context attributes (run id, repo, ref, sha) and a gh-aw.agent.setup span with trace/span/parent IDs.

Overall: all 5 scenarios passed.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Services Connectivity

  • Redis: ❌ Temporary failure in name resolution
  • PostgreSQL pg_isready: ❌ no response (DNS resolution failure)
  • PostgreSQL SELECT 1: ❌ could not translate host name

Overall: FAILhost.docker.internal did not resolve; sandbox cannot reach host service containers.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅ YES
Node.js v24.19.0 v22.23.2 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: FAILED — Node.js version differs between host and chroot environment (v24.19.0 vs v22.23.2). smoke-chroot label not applied since not all tests passed.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test

  • chore(workflows): recompile Firewall Issue Dispatcher lock to clear stale-hash mismatch
  • fix: make Cloud Hypervisor writable overlays privately propagated
  • Merged PR review ✅
  • safeinputs-gh / discussion-query ❌
  • Playwright ✅
  • File write/read ✅
  • Build ✅
  • Overall: FAIL

Warning

Firewall blocked 3 domains

The following domains were blocked by the firewall during workflow execution:

  • msfeed2.pkgs.visualstudio.com
  • msfeed25.pkgs.visualstudio.com
  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "msfeed2.pkgs.visualstudio.com"
    - "msfeed25.pkgs.visualstudio.com"
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Docker Sbx Smoke Test

Overall: PASS

cc @lpcox

📰 BREAKING: Report filed by Smoke Docker Sbx
Add label ready-for-aw to run again

@lpcox
lpcox deployed to aoai-model August 24, 2026 13:50 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

PR titles:
chore(workflows): recompile Firewall Issue Dispatcher lock to clear stale-hash mismatch
fix: make Cloud Hypervisor writable overlays privately propagated

Tests: MCP: ❌ GH.com: ✅ File I/O: ✅ BYOK: ✅
Running direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Entra
Overall: FAIL

@Copilot @lpcox

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
Add label ready-for-aw to run again

@lpcox
lpcox deployed to aoai-model August 24, 2026 13:50 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A (ran, "Hello, World!") ✅ PASS
.NET json-parse N/A (ran, Success: True) ✅ PASS
Go color ok ✅ PASS
Go env ok ✅ PASS
Go uuid ok ✅ PASS
Java gson passed ✅ PASS
Java caffeine passed ✅ PASS
Node.js clsx All tests passed ✅ PASS
Node.js execa All tests passed ✅ PASS
Node.js p-limit All tests passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — PASS

Notes:

  • Java: mvn failed initially with Could not create local repository at /home/runner/.m2/repository (permission denied on the shared .m2 dir). Worked around by setting -Dmaven.repo.local=/tmp/gh-aw/agent/m2repo; both gson and caffeine then compiled and tested successfully through the Squid proxy.
  • All other ecosystems built/installed/tested without issue via the firewall's proxy configuration.

Generated by Build Test Suite for #7679 · auto · 58.8 AIC · ⊞ 12K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox
Upgrade gh-aw workflows to latest pre-release ✅
fix: propagate config fields to all layers ✅
GitHub.com connectivity, File I/O, BYOK inference ✅
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
Overall: PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

filesystem.allowWrite breaks agent startup: /tmp/awf-init mountpoint uncreatable when /tmp is narrowed to read-only

4 participants