fix: wipe /tmp before the Coder agent starts, not after

[ppat]

What this fixes

A regression in the released template. /tmp is wiped from script-agent-startup.sh, and that wipe deletes the Coder agent's own CLI.

Coder's generated bootstrap (/workspace-init.sh) downloads the agent binary into a per-boot mktemp -d -t coder.XXXXXX directory under /tmp, cds into it, appends that directory to the PATH of every session and script the agent runs (cli/agent.go does an os.Setenv("PATH", …) on startup), and only then runs the startup script. The wipe therefore removes the binary the agent installed seconds earlier.

Symptom: all eight coder_agent metadata panels report /bin/bash: line 1: coder: command not found. The agent process itself keeps running from an unlinked binary, so nothing else fails visibly — on a workspace recreated from the released template, /proc/1/cwd reads /tmp/coder.XXXXXX (deleted).

What the agent owns under /tmp

Established from the deployed agent version's source, not inferred from the observed file names:

Path	Origin
coder.XXXXXX/ (holds the CLI, is the agent's cwd, is appended to PATH)	provisionersdk/scripts/bootstrap_linux.sh, BINARY_DIR
coder-agent.sock	agent/agentsocket/socket_unix.go — a hardcoded absolute /tmp/…, not TMPDIR-relative
coder-agent.log, coder-agent-<ts>.log, coder-agent-init.log, coder-startup-script.log	agent LogDir, defaults to os.TempDir()
coder-script-data/ (bin/ is prepended to PATH, <uuid>/ per script)	agent ScriptDataDir, defaults to os.TempDir()
coder-screen/	agent/reconnectingpty/screen.go
boundary-audit.sock	agent/boundarylogproxy/proxy.go

The set is version-dependent, and boundary-audit.sock already breaks the coder* prefix that an exclusion glob would key on.

The fix

The wipe moves from the agent startup script to the workspace container's entrypoint.

• New script-container-entrypoint.sh becomes the container command. It wipes /tmp, then execs Coder's /workspace-init.sh.
• script-agent-startup.sh no longer wipes anything.

The container entrypoint is the only hook that runs on a container-only restart within a live Pod — the exact case the wipe exists for, since the /tmp volume is per-Pod and init containers run once per Pod. It runs before the agent exists, so there is nothing to exclude and no coupling to Coder's internals. exec keeps the agent as PID 1, which the orphan reaping and the pgrep -f "coder agent" liveness probe both depend on.

Alternatives rejected

• Exclusion list in the startup script. Trades this breakage for a later one: the path set above is a Coder implementation detail, so an allowlist stops matching on some upgrade and fails exactly as invisibly as the original bug.
• Drop the wipe. Scratch space then accumulates across container restarts on a fixed-size 20Gi volume, which is the growth the volume change existed to bound.
• Relocate the agent's state out of /tmp (BINARY_DIR, CODER_AGENT_LOG_DIR, CODER_AGENT_SCRIPT_DATA_DIR). Cannot be done completely — coder-agent.sock and coder-screen/ are not fully configurable — and it would break the dotfiles .bashrc block, which locates the CLI by globbing /tmp/coder*.

The check that would have caught this

script-agent-startup.sh now asserts, after the agent is up:

• The agent CLI resolves and runs. command -v coder goes through the same PATH a coder stat metadata script uses, then the resolved binary is executed. Against the currently released template this fails immediately.
• The wipe completed. The entrypoint records its outcome, and a missing or non-ok outcome fails here.

startup_script_behavior is blocking and the script sets errexit, so either failure surfaces as a failed startup script in the workspace UI while the agent keeps running and the workspace stays reachable to debug.

Failure posture

The wipe in the entrypoint is best-effort rather than fatal. That entrypoint is the only path to a running agent, so aborting there converts stale scratch space into a CrashLoopBackOff with no way to shell in and look. Visibility is preserved by the recorded outcome that the startup script asserts on.

Verification

• Sandboxed runs of both scripts against a stand-in directory: full wipe including dotfiles and nested directories; the partial-failure path records the failure and still starts the agent; the startup script exits non-zero for a failed wipe, a missing outcome, and an absent CLI, and exits zero when the CLI resolves and runs.
• Live on a disposable workspace built from this branch's dry-run template, including a container-only restart within the live Pod: see the verification comment.
• The new check run unmodified against the test workspace, which is still on the released version: it exits 1. See the same comment.

[ppat]

Live verification

Run against a disposable workspace created from homelab-workspace-test, the template this PR's dry-run published — template version 2930cdb, this branch's commit. Both scripts on the running pod match the branch byte for byte (md5sum of /container-entrypoint.sh and /agent-startup.sh).

Cold Pod start

/proc/1/cwd -> /tmp/coder.t7v1Id        # present, not "(deleted)"
/tmp/.tmp-wipe-status -> ok
command -v coder -> /tmp/coder.t7v1Id/coder   (Coder v2.35.4)
ps: PID 1 = ./coder agent               # the exec handoff kept the agent as PID 1

/tmp/coder-startup-script.log:

/tmp wiped on container start
Coder agent CLI: /tmp/coder.t7v1Id/coder
Done

Metadata scripts, resolved through the same PATH the agent gives them:

coder stat cpu            -> 0.003/2 cores (0%)
coder stat mem --prefix Gi -> 0.04/4 GiB (1%)
coder stat disk --path $HOME -> 92.1/125 GiB (74%)

Container-only restart within the live Pod — the case the wipe exists for. A 64 MiB scratch file was written to /tmp, then the agent worker was killed; the kubelet restarted the container in place (RESTARTS 0 -> 1, same Pod, same IP, same /tmp PVC).

/tmp PVC             -> unchanged (per-Pod volume survived, as designed)
scratch file (64 MiB) -> gone           # the wipe ran on the restart
/tmp usage           -> 120M -> 56M
binary dir           -> coder.t7v1Id -> coder.OsqZ4M, present, not "(deleted)"
/tmp/.tmp-wipe-status -> ok
coder stat cpu       -> 0.003/2 cores (0%)
coder stat mem       -> 0.039/4 GiB (1%)

The workspace was deleted afterwards.

The check, run against the broken released template

The new script-agent-startup.sh was piped into the test workspace, which is still on the released version, and run there unmodified. It exits 1:

/tmp wiped on container start
ERROR: the Coder agent CLI is not on PATH.
       Nothing may remove the agent's own files from /tmp - see
       script-container-entrypoint.sh for why the wipe runs there.
       PATH=/tmp/coder-script-data/bin:/usr/local/sbin:...:/bin:/tmp/coder.atrw4f
REMOTE_EXIT=1

The trailing PATH entry is the agent's own binary directory, appended by the agent at startup; on the released template that directory no longer exists, and /proc/1/cwd there reads /tmp/coder.atrw4f (deleted).

Why the wipe is worth keeping rather than dropping

Container-only restarts are not hypothetical here: the longest-running workspace pod in the cluster shows 3 restarts over 19 days, each of which would otherwise inherit the previous container's scratch space on a fixed-size 20Gi volume.

Notes

• The test workspace was left untouched and is still on the released, broken version. It picks the fix up on the next coder update after this merges.
• An ad-hoc, non-activated template version pushed during investigation has been archived; the production template's active version was never changed.

[homelab-workflows-bot]

🎉 This PR is included in version 2.27.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀