github · lpcox · Aug 23, 2026 · Aug 23, 2026 · Aug 23, 2026 · Aug 23, 2026
diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md
@@ -95,6 +95,7 @@ Prefer the narrowest match. Examples:
 - Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports)
 - Repeated `EACCES` retries reading `${RUNNER_TEMP}/gh-aw/mcp-config/mcp-servers.json` (or similar `${RUNNER_TEMP}/gh-aw` paths) when AWF was invoked as native root (no `sudo`) → B24 (root fallback to sandbox uid 1000 leaves root-owned config unreadable; fixed in github/gh-aw-firewall#7565 with `repairRunnerTempGhAwOwnership()`)
 - Job exits 0 with no agent output/writes on a native-root runner (no `sudo`), even after `${RUNNER_TEMP}/gh-aw` ownership is fixed, or `Host workspace is not writable by the sandbox identity (<uid>:<gid>): <path>` → B25 (checkout/container-workdir left root-owned while agent runs as fallback sandbox uid; fixed in github/gh-aw-firewall#7599 with `repairContainerWorkDirOwnership()`)
+- `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode → B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure Blob storage; fixed in github/gh-aw-firewall#7635 with a scoped Squid ACL keyed to cli-proxy's fixed IP)
 - `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; AWF's chroot-side gap is fixed in github/gh-aw-firewall#7245; only the upstream installer/harness mismatch in github/gh-aw-firewall#7130 remains open)
 - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/<path>` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files)
 - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs)
@@ -165,6 +166,8 @@ B24 / github/gh-aw-firewall#7564, github/gh-aw-firewall#7565 — On native-root
 
 B25 / github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 — On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout. PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run. **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R <uid>:<gid> <workdir>` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. Probe: run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (<uid>:<gid>): <path>`.
 
+B26 / github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 — In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net` in ~350ms. Two independent changes created the failure: `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI running inside cli-proxy must follow the `Location` header itself; meanwhile cli-proxy is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066), so it had no route to `productionresultssa*.blob.core.windows.net`. **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is routed through Squid while cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is not added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. Probe: reproduce with `gh run download <id>` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP.
+
 D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor.
 
 D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime.
@@ -278,6 +281,7 @@ Establish these facts before matching a failure mode:
 | B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Two gaps contributed to the symptom. The still-open upstream gap is that gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns that hardcoded absolute path. Before github/gh-aw-firewall#7245, AWF also mounted host `/usr`/`/usr/local` read-only without creating the missing hardcoded entry inside the chroot, so the harness failed unless the host symlink already existed. | **Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):** `containers/agent/entrypoint.sh` adds `resolve_chroot_binary_path()`, `ensure_usr_local_bin_shims()`, and `prepare_usr_local_bin_overlay()`, invoked after `copy_dind_runner_binary`. When `AWF_ENSURE_USR_LOCAL_BIN=copilot` is set (auto-set for Copilot runs in `tool-specific-environment.ts`), AWF resolves the real `copilot` binary from `$GITHUB_PATH`, `AWF_HOST_PATH`, staged bin dirs, or system dirs, and creates `/usr/local/bin/copilot` inside the chroot via a read-only symlink-farm overlay without modifying host `/usr/local/bin`. Upgrade AWF to include github/gh-aw-firewall#7245. **Older AWF only:** before invoking `awf`, use the host workaround `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot`; it is unnecessary on patched AWF. The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130. PR github/gh-aw-firewall#7151 documents the older behavior and workaround. | Confirm gh-aw took the cache-hit path (`GITHUB_PATH` already set). On AWF including github/gh-aw-firewall#7245, check entrypoint logs for `ensure_usr_local_bin_shims` / `prepare_usr_local_bin_overlay`; on older AWF, absence of host `/usr/local/bin/copilot` reproduces the ENOENT | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245 |
 | B24 | Repeated `EACCES` retries from the harness reading `${RUNNER_TEMP}/gh-aw/mcp-config/mcp-servers.json` (or other files under `${RUNNER_TEMP}/gh-aw`) with no root-cause diagnostic explaining the native-root fallback ownership mismatch; occurs when AWF is invoked as native root (no `sudo`, so no `SUDO_UID`) | AWF falls back to a default sandbox uid/gid (`1000:1000`) when it cannot recover the original host identity from `SUDO_UID`. Files under `${RUNNER_TEMP}/gh-aw` created by the root-run harness stay root-owned, so the sandbox identity cannot read them once they are mounted read-only into the agent container | **Fixed in AWF (PR github/gh-aw-firewall#7565, merged 2026-08-20):** `isNativeRootWithoutSudo()` (`src/host-identity.ts`) detects root execution without `SUDO_UID` and logs an explicit warning identifying the fallback sandbox identity. `repairRunnerTempGhAwOwnership()` (`src/config-writer.ts`) recursively `chown`s `${RUNNER_TEMP}/gh-aw` to the resolved sandbox uid/gid (via `chown -h -P -R`, never following symlinks) before Docker Compose generation and container launch. Upgrade AWF to include github/gh-aw-firewall#7565. | Run AWF as root directly (no `sudo`) and check for the warning `Host process is running as root with no SUDO_UID; AWF will use sandbox identity ...`; compare `stat -c '%U:%G' "$RUNNER_TEMP/gh-aw/mcp-config/mcp-servers.json"` before/after a run — it should show the sandbox uid:gid after the fix instead of `root` | github/gh-aw-firewall#7564, github/gh-aw-firewall#7565 |
 | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R <uid>:<gid> <workdir>` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (<uid>:<gid>): <path>` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 |
+| B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download <id>` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 |
 
 ## Category C — GHES / GHEC / `ghe.com`
 
@@ -365,6 +369,7 @@ Establish these facts before matching a failure mode:
 | `EACCES` retries reading `${RUNNER_TEMP}/gh-aw/mcp-config/mcp-servers.json` (or similar `${RUNNER_TEMP}/gh-aw` paths) when AWF was invoked as native root (no `sudo`) | B24 (root fallback to sandbox uid 1000 leaves root-owned config unreadable; fixed in github/gh-aw-firewall#7565 with `repairRunnerTempGhAwOwnership()`) |
 | Job exits 0 with no agent output/writes on a native-root runner (no `sudo`), even after `${RUNNER_TEMP}/gh-aw` ownership is fixed | B25 (checkout/container-workdir left root-owned while agent runs as fallback sandbox uid; fixed in github/gh-aw-firewall#7599) |
 | `Host workspace is not writable by the sandbox identity (<uid>:<gid>): <path>` | B25 |
+| `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure Blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) |
 | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) |
 | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 |
 | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 |