feat: support filesystem allowWrite with Cloud Hypervisor

.github/workflows/test-cloud-hypervisor.yml

@@ -19,6 +19,7 @@ on:
       - 'src/cloud-hypervisor-runtime-backend.ts'
       - 'src/cloud-hypervisor-runtime-backend.test.ts'
       - 'src/microvm/**'
+      - 'src/filesystem-policy.ts'
       - 'src/types/runtime-options.ts'
       - 'scripts/ci/cloud-hypervisor-*.sh'
       - 'docs/cloud-hypervisor-foundation.md'

docs/awf-config-spec.md

@@ -99,9 +99,15 @@ existing writable host mount. An empty list makes all non-internal host bind
 mounts read-only.
 
 AWF-owned agent log and session-state mounts and required virtual devices remain
-writable so the sandbox can operate. `filesystem.allowWrite` currently supports
-Docker and gVisor compose runtimes; AWF rejects it with unsupported microVM
-runtimes.
+writable so the sandbox can operate. `filesystem.allowWrite` is supported by the
+Docker and gVisor compose runtimes and by the Cloud Hypervisor microVM runtime,
+where it is enforced by the host mount tree that backs each virtio-fs export
+(see [docs/cloud-hypervisor-foundation.md](./cloud-hypervisor-foundation.md#runtime-integration)).
+Cloud Hypervisor has no always-writable internal mounts, so every export it
+publishes is subject to the policy, including `/tmp/gh-aw` and the guest home
+directory at `/workspace/.awf-home`; paths not covered by `allowWrite` become
+read-only. AWF rejects `filesystem.allowWrite` with the sbx
+runtime and with Docker-in-Docker agent execution.
 
 ### 4.2 Cloud Hypervisor microVM preview
 

docs/awf-config.schema.json

@@ -83,7 +83,7 @@
             "type": "string",
             "pattern": "^/(?!\\.\\.(?:/|$))(?!.*(?:/)\\.\\.(?:/|$))[^*?\\[\\]{}]*$"
           },
-          "description": "Guest-visible absolute paths that may be writable. When present, existing writable host bind mounts are made read-only outside these paths. An empty array makes all non-internal host bind mounts read-only. Paths must exist and cannot use glob patterns. This option is currently supported by Docker and gVisor compose runtimes."
+          "description": "Guest-visible absolute paths that may be writable. When present, existing writable host bind mounts are made read-only outside these paths. An empty array makes all non-internal host bind mounts read-only. Paths must exist and cannot use glob patterns. This option is supported by the Docker and gVisor compose runtimes and by the Cloud Hypervisor microVM runtime; it is rejected with the sbx runtime and with Docker-in-Docker agent execution."
         }
       }
     },

docs/cloud-hypervisor-foundation.md

@@ -164,7 +164,7 @@ Temporary microVM workspace data lives under:
 With `--keep-containers`, AWF preserves this directory, the network namespace,
 and runtime diagnostics for investigation.
 
-### Write-policy planning (inert)
+### Write-policy planning
 
 [`src/cloud-hypervisor/filesystem-write-policy.ts`](../src/cloud-hypervisor/filesystem-write-policy.ts)
 plans how a `filesystem.allowWrite` allowlist would narrow validated exports. It
@@ -190,13 +190,11 @@ escape the export source.
 
 The planner only removes write access: it never widens a read-only export and
 never introduces a host path that an existing read-write export does not
-already cover. It is pure policy planning and is not yet wired into runtime
-execution — `filesystem.allowWrite` is still rejected for the Cloud Hypervisor
-runtime by [`src/filesystem-policy.ts`](../src/filesystem-policy.ts). The host
-side of that boundary — how a `hostRootMode: 'ro'` root with writable overlays is
-actually staged and enforced — is described in
-[Host mount-tree enforcement](#host-mount-tree-enforcement) below. The two layers
-are independent: neither is wired into runtime execution yet.
+already cover. It is pure policy planning; the host side of that boundary — how
+a `hostRootMode: 'ro'` root with writable overlays is actually staged and
+enforced — is described in
+[Host mount-tree enforcement](#host-mount-tree-enforcement) below, and
+[Runtime integration](#runtime-integration) describes how the two are joined.
 
 ## Host mount-tree enforcement
 
@@ -206,7 +204,7 @@ guest-side read-only mount is not a security boundary. The only trustworthy
 boundary is the host VFS.
 
 This is the host-side counterpart to
-[Write-policy planning](#write-policy-planning-inert): the planner decides which
+[Write-policy planning](#write-policy-planning): the planner decides which
 paths stay writable, and this layer stages a host mount tree that enforces it.
 
 `VirtiofsdManager.start()` therefore accepts an optional, strongly typed
@@ -313,6 +311,52 @@ require containment inside the export, but this residual setup-time TOCTOU windo
 cannot be closed without fd-based mount APIs that the current tooling does not
 expose.
 
+## Runtime integration
+
+[`src/cloud-hypervisor/filesystem-write-enforcement.ts`](../src/cloud-hypervisor/filesystem-write-enforcement.ts)
+is the only place where the planner and the host mount tree meet. The Cloud
+Hypervisor runtime backend resolves and validates its exports, then plans the
+policy in a dedicated `filesystem-write-policy` startup stage *before* the boot
+loop, so an invalid allowlist aborts the run before virtiofsd or the guest is
+ever launched, and before any retry can re-attempt it. The resulting
+`VirtiofsdMountEnforcement` is threaded through `createManager()` into
+`CloudHypervisorManager`, which forwards it to `VirtiofsdManager.start()`.
+
+No `internalTags` are passed to the planner. Cloud Hypervisor has no analogue of
+the Docker runtime's always-writable agent-log and session-state binds: every
+export it publishes is host-visible workspace or runner state, and marking one
+internal — `tmp-gh-aw` in particular — would defeat the narrowing that a policy
+such as `allowWrite: ["/tmp/gh-aw/agent"]` exists to express.
+
+The translation is total; there is no fallback path:
+
+| Planner disposition | Guest mount mode | Host staged root | Mount plan passed to virtiofsd |
+| --- | --- | --- | --- |
+| policy absent (`undefined`) | unchanged | unchanged | none — `start()` receives no enforcement argument at all, so behaviour is byte-identical to a run without a policy |
+| unrestricted / fully writable (`hostRootMode: 'rw'`) | `rw` | unchanged | none |
+| fully read-only (`hostRootMode: 'ro'`, no overlays) | `ro` | `ro` | plan with zero overlays |
+| selectively writable (`hostRootMode: 'ro'`, overlays) | `rw` | `ro` | plan with one overlay per allowed path |
+
+A read-only export with zero overlays still gets a plan rather than falling back
+to the legacy single `mount --bind` plus `remount,ro`. The staged tree is the
+only variant that recursively remounts carried-in submounts read-only and
+verifies the result against `/proc/self/mountinfo`, so a policy-narrowed export
+is always served by the stronger path.
+
+Because the host tree is the boundary, a selectively writable export is never
+mounted read-only guest-side. The guest mode is derived from the plan, so
+`validateCloudHypervisorExports()` accepts a read-only `workspace` export only
+when a mount plan for the `workspace` tag actually exists; a read-only workspace
+that nothing enforces is still rejected. Unknown plan tags remain fail-closed
+via `assertPlansMatchExports()`, and the planner's own validation is not
+duplicated here.
+
+One consequence is worth stating plainly: the guest `HOME` is
+`/workspace/.awf-home`, inside the workspace export. A policy that narrows
+`/workspace` — including an empty `allowWrite: []` — makes the agent's home
+directory read-only. That is the policy working as specified, not an oversight;
+add the home path to `allowWrite` if the workload needs it.
+
 ## Limitations
 
 The preview rejects configurations that weaken or conflict with its boundary,
@@ -349,6 +393,10 @@ The live job runs only when explicitly enabled by workflow dispatch or the
 - direct-egress, arbitrary-TCP, DNS, and metadata denial;
 - API proxy reachability and secret non-disclosure;
 - workspace persistence;
+- `filesystem.allowWrite` enforcement — an allowed directory and file write
+  persisting to the host, sibling/parent/create/truncate/rename/delete denial
+  outside the allowlist, an empty allowlist narrowing the whole workspace, and
+  a fail-closed abort on an allowlist entry that matches no export path;
 - exit-code, timeout, and cancellation behavior;
 - device assumptions;
 - partial-start and normal cleanup;

scripts/ci/cloud-hypervisor-ci-scripts.test.ts

@@ -97,6 +97,9 @@ describe('cloud-hypervisor-live-smoke.sh', () => {
       'api-proxy-reflect',
       'workspace-live-share',
       'runtime-cache-readonly',
+      'allow-write',
+      'allow-write-none',
+      'allow-write-invalid',
       'exit-code',
       'timeout-124',
       'partial-start-cleanup',
@@ -162,6 +165,72 @@ describe('cloud-hypervisor-live-smoke.sh', () => {
     expect(source).toMatch(/COMMON=\(\n(?:.*\n)*?\s*--network-isolation\n/);
   });
 
+  it('proves filesystem.allowWrite enforcement end to end, including fail-closed', () => {
+    const source = fs.readFileSync(smokePath, 'utf-8');
+    // Selective policy: host-visible persistence of an allowed write, plus
+    // sibling/parent/create/truncate/rename/delete denial outside the list.
+    expect(source).toContain('"allowWrite": ["/workspace/allowed", "/workspace/allowed-file.txt"]');
+    expect(source).toContain('test "$(cat "$allow_workspace/allowed/created.txt")" = guest-allowed');
+    expect(source).toContain('test "$(cat "$allow_workspace/blocked/file.txt")" = host');
+    expect(source).toContain('test ! -e "$allow_workspace/created-at-root.txt"');
+    expect(source).toContain('test ! -e "$allow_workspace/renamed.txt"');
+    // A selective export stays read-write guest-side; only the zero-overlay
+    // narrowing publishes the guest mount itself read-only.
+    expect(source).toContain('grep -q " /workspace/allowed virtiofs " /proc/mounts');
+    expect(source).toContain('grep -q " /workspace virtiofs ro," /proc/mounts');
+    // Unmatched allowlist entries abort the run instead of widening it.
+    expect(source).toContain('run_case allow-write-invalid 1');
+    expect(source).toContain("grep -q 'filesystem.allowWrite'");
+  });
+
+  it('proves every allowWrite denial probe actually executed under BusyBox ash', () => {
+    const source = fs.readFileSync(smokePath, 'utf-8');
+    // Regression: the guest shell is BusyBox ash, where a redirection failure
+    // on a POSIX *special* builtin is fatal and exits the shell. An earlier
+    // `! : > /workspace/input.txt` truncate probe therefore terminated the
+    // guest command before the rename and delete probes ran, while the host
+    // post-checks still passed vacuously -- a write never attempted also never
+    // Comment lines are excluded: the rationale above quotes the old probe.
+    const executableLines = source
+      .split('\n')
+      .filter((line) => !line.trimStart().startsWith('#'));
+    expect(executableLines.some((line) => /!\s*:\s*>/.test(line))).toBe(false);
+    expect(source).toContain('! ( printf "" > /workspace/input.txt )');
+
+    // Every denial probe is subshell-contained (so even a fatal shell error is
+    // isolated) and followed by a sentinel that the suite then requires.
+    for (const probe of [
+      '! ( printf blocked > /workspace/blocked/file.txt )',
+      '! ( printf blocked > /workspace/created-at-root.txt )',
+      '! ( mkdir /workspace/blocked-dir )',
+      '! ( mv /workspace/rename-me.txt /workspace/renamed.txt )',
+      '! ( rm /workspace/blocked/file.txt )',
+    ]) {
+      expect(source).toContain(probe);
+    }
+
+    expect(source).toContain('assert_sentinels() {');
+    expect(source).toContain('missing sentinel $sentinel (probe never executed)');
+    for (const sentinel of [
+      'AWF-ALLOWWRITE-ALLOWED-OK',
+      'AWF-ALLOWWRITE-SIBLING-DENIED',
+      'AWF-ALLOWWRITE-CREATE-DENIED',
+      'AWF-ALLOWWRITE-MKDIR-DENIED',
+      'AWF-ALLOWWRITE-TRUNCATE-DENIED',
+      'AWF-ALLOWWRITE-RENAME-DENIED',
+      'AWF-ALLOWWRITE-DELETE-DENIED',
+      'AWF-ALLOWWRITE-NONE-READ-OK',
+      'AWF-ALLOWWRITE-NONE-WRITE-DENIED',
+      'AWF-ALLOWWRITE-NONE-CREATE-DENIED',
+    ]) {
+      // Emitted by the guest command, and separately required afterwards.
+      expect(source).toContain(`echo ${sentinel}`);
+      expect(source.split(sentinel).length - 1).toBeGreaterThanOrEqual(2);
+    }
+    expect(source).toContain('assert_sentinels allow-write \\');
+    expect(source).toContain('assert_sentinels allow-write-none \\');
+  });
+
   (shellcheckAvailable() ? it : it.skip)('has no shellcheck errors', () => {
     expect(() =>
       execFileSync('shellcheck', ['--severity=error', smokePath]),

scripts/ci/cloud-hypervisor-live-smoke.sh

@@ -181,6 +181,26 @@ run_case() {
   assert_no_residue
 }
 
+# Requires that a completed case printed every named marker to its guest
+# stdout. `run_case` only checks the final exit status, which cannot
+# distinguish "every probe ran and behaved" from "the guest shell died partway
+# through and the remaining probes never executed" -- the latter leaves host
+# post-checks passing vacuously, because a write that was never attempted
+# also never changed anything.
+assert_sentinels() {
+  local name=$1
+  shift
+  local log="$RUN_ROOT/$name/stdout.log"
+  local sentinel
+  for sentinel in "$@"; do
+    if ! grep -qF "$sentinel" "$log"; then
+      echo "case $name: missing sentinel $sentinel (probe never executed)" >&2
+      tail -50 "$log" >&2
+      return 1
+    fi
+  done
+}
+
 assert_no_residue
 
 # Best-effort, bounded packet capture across the first live case only (never
@@ -280,6 +300,79 @@ run_case runtime-cache-readonly 0 \
   'grep -q cache-readable "$RUNNER_TEMP/gh-aw/awf-virtiofs-ro-probe" && ! printf changed > "$RUNNER_TEMP/gh-aw/awf-virtiofs-ro-probe"'
 test "$(cat "$RUNNER_TEMP/gh-aw/awf-virtiofs-ro-probe")" = cache-readable
 
+# filesystem.allowWrite enforcement (docs/cloud-hypervisor-foundation.md
+# "Runtime integration"). The host mount tree -- not the guest mount mode --
+# is the boundary, so a selective policy publishes /workspace to the guest
+# read-write while virtiofsd only ever sees a read-only staged root with
+# narrow read-write submounts bind-mounted back in.
+allow_workspace="$RUN_ROOT/allow-write/workspace"
+mkdir -p "$allow_workspace/allowed/nested" "$allow_workspace/blocked"
+printf 'host\n' >"$allow_workspace/allowed/removable.txt"
+printf 'host\n' >"$allow_workspace/allowed-file.txt"
+printf 'host\n' >"$allow_workspace/blocked/file.txt"
+printf 'host\n' >"$allow_workspace/rename-me.txt"
+cat >"$RUN_ROOT/allow-write.json" <<'JSON'
+{
+  "filesystem": {
+    "allowWrite": ["/workspace/allowed", "/workspace/allowed-file.txt"]
+  }
+}
+JSON
+# Each denial is wrapped in a subshell and followed by a sentinel echo. The
+# guest shell is BusyBox ash: a redirection failure on a POSIX *special*
+# builtin (`:`, `.`, `eval`, ...) is fatal and exits the shell outright, so an
+# earlier `! : > /workspace/input.txt` truncate probe silently skipped the
+# rename and delete probes below it while the host post-checks still passed --
+# nothing had been attempted, so nothing had changed. `printf ''` is a regular
+# builtin (non-fatal), the subshell contains a fatal error even if one occurs,
+# and requiring every sentinel below proves each probe actually ran.
+run_case allow-write 0 \
+  'printf guest-allowed > /workspace/allowed/created.txt && mkdir -p /workspace/allowed/nested/deeper && printf deep > /workspace/allowed/nested/deeper/file.txt && rm /workspace/allowed/removable.txt && printf guest-file > /workspace/allowed-file.txt && grep -q " /workspace/allowed virtiofs " /proc/mounts && grep -q host-input /workspace/input.txt && echo AWF-ALLOWWRITE-ALLOWED-OK && ! ( printf blocked > /workspace/blocked/file.txt ) && echo AWF-ALLOWWRITE-SIBLING-DENIED && ! ( printf blocked > /workspace/created-at-root.txt ) && echo AWF-ALLOWWRITE-CREATE-DENIED && ! ( mkdir /workspace/blocked-dir ) && echo AWF-ALLOWWRITE-MKDIR-DENIED && ! ( printf "" > /workspace/input.txt ) && echo AWF-ALLOWWRITE-TRUNCATE-DENIED && ! ( mv /workspace/rename-me.txt /workspace/renamed.txt ) && echo AWF-ALLOWWRITE-RENAME-DENIED && ! ( rm /workspace/blocked/file.txt ) && echo AWF-ALLOWWRITE-DELETE-DENIED' \
+  --config "$RUN_ROOT/allow-write.json"
+assert_sentinels allow-write \
+  AWF-ALLOWWRITE-ALLOWED-OK \
+  AWF-ALLOWWRITE-SIBLING-DENIED \
+  AWF-ALLOWWRITE-CREATE-DENIED \
+  AWF-ALLOWWRITE-MKDIR-DENIED \
+  AWF-ALLOWWRITE-TRUNCATE-DENIED \
+  AWF-ALLOWWRITE-RENAME-DENIED \
+  AWF-ALLOWWRITE-DELETE-DENIED
+test "$(cat "$allow_workspace/allowed/created.txt")" = guest-allowed
+test "$(cat "$allow_workspace/allowed/nested/deeper/file.txt")" = deep
+test "$(cat "$allow_workspace/allowed-file.txt")" = guest-file
+test ! -e "$allow_workspace/allowed/removable.txt"
+test "$(cat "$allow_workspace/blocked/file.txt")" = host
+test "$(cat "$allow_workspace/input.txt")" = host-input
+test -f "$allow_workspace/rename-me.txt"
+test ! -e "$allow_workspace/renamed.txt"
+test ! -e "$allow_workspace/created-at-root.txt"
+test ! -e "$allow_workspace/blocked-dir"
+
+# An empty allowlist narrows every writable export with zero overlays: the
+# guest mount itself is published read-only on top of the read-only staged
+# host root.
+cat >"$RUN_ROOT/allow-write-none.json" <<'JSON'
+{ "filesystem": { "allowWrite": [] } }
+JSON
+run_case allow-write-none 0 \
+  'grep -q host-input /workspace/input.txt && grep -q " /workspace virtiofs ro," /proc/mounts && echo AWF-ALLOWWRITE-NONE-READ-OK && ! ( printf blocked > /workspace/input.txt ) && echo AWF-ALLOWWRITE-NONE-WRITE-DENIED && ! ( printf blocked > /workspace/created.txt ) && echo AWF-ALLOWWRITE-NONE-CREATE-DENIED' \
+  --config "$RUN_ROOT/allow-write-none.json"
+assert_sentinels allow-write-none \
+  AWF-ALLOWWRITE-NONE-READ-OK \
+  AWF-ALLOWWRITE-NONE-WRITE-DENIED \
+  AWF-ALLOWWRITE-NONE-CREATE-DENIED
+test "$(cat "$RUN_ROOT/allow-write-none/workspace/input.txt")" = host-input
+test ! -e "$RUN_ROOT/allow-write-none/workspace/created.txt"
+
+# Fail-closed: an allowlist entry that matches no export path must abort the
+# run before any guest boots, and must leave no staged mount residue.
+cat >"$RUN_ROOT/allow-write-invalid.json" <<'JSON'
+{ "filesystem": { "allowWrite": ["/workspace/does-not-exist"] } }
+JSON
+run_case allow-write-invalid 1 'true' \
+  --config "$RUN_ROOT/allow-write-invalid.json"
+grep -q 'filesystem.allowWrite' "$RUN_ROOT/allow-write-invalid/stderr.log"
+
 run_case exit-code 37 'exit 37'
 run_case timeout-124 124 'sleep 90' --agent-timeout 1
 

src/awf-config-schema.json

@@ -83,7 +83,7 @@
             "type": "string",
             "pattern": "^/(?!\\.\\.(?:/|$))(?!.*(?:/)\\.\\.(?:/|$))[^*?\\[\\]{}]*$"
           },
-          "description": "Guest-visible absolute paths that may be writable. When present, existing writable host bind mounts are made read-only outside these paths. An empty array makes all non-internal host bind mounts read-only. Paths must exist and cannot use glob patterns. This option is currently supported by Docker and gVisor compose runtimes."
+          "description": "Guest-visible absolute paths that may be writable. When present, existing writable host bind mounts are made read-only outside these paths. An empty array makes all non-internal host bind mounts read-only. Paths must exist and cannot use glob patterns. This option is supported by the Docker and gVisor compose runtimes and by the Cloud Hypervisor microVM runtime; it is rejected with the sbx runtime and with Docker-in-Docker agent execution."
         }
       }
     },