feat: move vscode-server GC into the template as a mounted script

CLAUDE.md

@@ -77,6 +77,7 @@ Quick orientation map — for what each piece is *for* and the decisions behind
 | `script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/workspace-init.sh` — the wipe must precede the agent, see the gotcha below |
 | `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** (per-role PSS budgets, ten-minute dwell, per-role circuit breaker) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (helpers — the default) / `enforce-all` (helpers + editor) |
 | `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's arithmetic, process selection, budgets, dwell and circuit breaker. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite |
+| `script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) |
 
 **Image** (`images/homelab-workspace/Dockerfile`): three build stages — `base` (minimal bootstrap deps) → `system-base` (`unminimize` + full interactive toolset) → final stage (env vars into `/etc/environment`, fixed-UID/GID `coder` user, `USER coder`). All `apt`-touching `RUN` steps use BuildKit cache mounts — match that pattern when adding packages.
 
@@ -105,6 +106,7 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [
 - Adding a package/tool has three possible homes, and picking the wrong one is a real mistake, not a style choice — route by the rule in [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from): universal + stable → image (`Dockerfile`); occasionally-needed + apt-only + too heavy to bake in → the template's `system_packages` parameter; personal, fast-moving, or not an apt package → the operator's dotfiles (a *different* repo — see below), never this one.
 - `deployment.tf` mounts `/tmp` on its own ephemeral Longhorn volume, not the node's root filesystem and not the NFS-backed home PVC - see [DESIGN.md](DESIGN.md#design-tensions-and-decisions) for why both of those are wrong for it. Its lifecycle is per-Pod, the same as the `system` volume, so it is *not* wiped by a container-only restart within a live Pod - `script-container-entrypoint.sh` wipes it explicitly on every container start instead. Anything relying on `/tmp` persisting across a container restart was already wrong before this (the same was true for free when it was the container's writable overlay).
 - **The `/tmp` wipe belongs in the container entrypoint and nowhere later - this is a fixed regression, not a style preference.** The workspace container's `command` is `script-container-entrypoint.sh`, which wipes `/tmp` and then `exec`s Coder's generated `/workspace-init.sh`. That bootstrap unpacks the agent CLI into a per-boot `mktemp` directory under `/tmp`, chdirs into it, appends it to the PATH of every session and script the agent runs, and only then runs the agent startup script - so wiping `/tmp` from `script-agent-startup.sh` deletes the CLI the agent installed seconds earlier and every `coder stat` metadata panel reports `coder: command not found`. Do not "fix" a future collision here with an exclusion list: the agent's `/tmp` paths are version-dependent and not consistently prefixed (v2.35.3 owns `coder.XXXXXX/`, `coder-agent.sock`, `coder-agent*.log`, `coder-script-data/`, `coder-screen/` *and* `boundary-audit.sock`), so an allowlist rots silently. `script-agent-startup.sh` asserts the CLI resolves and runs, which is what makes a recurrence loud.
+- A `coder_script` in `scripts.tf` must never gate its actual work behind `[ -x <path> ] && ... || true` (or equivalent) when `<path>` is supplied by something outside the template, e.g. a dotfiles-installed binary. That guard makes "the payload isn't there" indistinguishable from "the payload ran and had nothing to do" — both report success on the schedule's weekly cron. This happened: `vscode_server_gc` was originally written that way, expecting `$HOME/.local/bin/vscode-server-gc` from dotfiles, and would have silently no-opped forever on any workspace dotfiles hadn't been applied to (confirmed on the `test` workspace, which reached ~11 GB of `~/.vscode-server` with dotfiles never applied). Fixed by making `script-vscode-server-gc.sh` a template-owned script mounted via `configmap.tf`/`deployment.tf`, so `scripts.tf` invokes it directly and an actual failure now surfaces as a failed run in the Coder UI instead of vanishing into `|| true`.
 - `deployment.tf`'s Deployment `metadata.name` (`local.workload_name` in `main.tf`) is not cosmetic: the cluster's Prometheus resolves pod → ReplicaSet → Deployment via an existing `kube_pod_owner` recording rule and exposes the result as a `workload` label with no other join needed, so whatever this Deployment is named *is* the identity CPU/memory/PSI/OOM metrics get attributed to. Don't revert it to an opaque identifier (e.g. the workspace UUID) without re-breaking that attribution — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions).
 
 ## Neighbouring repos

templates/kubernetes/homelab-workspace/configmap.tf

@@ -12,6 +12,7 @@ resource "kubernetes_config_map_v1" "workspace_scripts" {
     container_entrypoint_script = file("${path.cwd}/script-container-entrypoint.sh")
     memory_watchdog_script      = file("${path.cwd}/script-memory-watchdog.sh")
     prepare_workspace_script    = file("${path.cwd}/script-prepare-workspace.sh")
+    vscode_server_gc_script     = file("${path.cwd}/script-vscode-server-gc.sh")
     workspace_init_script       = coder_agent.main.init_script
   }
 }

templates/kubernetes/homelab-workspace/deployment.tf

@@ -147,6 +147,11 @@ resource "kubernetes_deployment_v1" "deployment" {
             name       = "coder-scripts"
             sub_path   = "memory_watchdog_script"
           }
+          volume_mount {
+            mount_path = "/vscode-server-gc.sh"
+            name       = "coder-scripts"
+            sub_path   = "vscode_server_gc_script"
+          }
           volume_mount {
             mount_path = "/workspace-init.sh"
             name       = "coder-scripts"

templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh

@@ -0,0 +1,206 @@
+#!/bin/bash
+set -euo pipefail
+
+# vscode-server-gc: reclaims space under ~/.vscode-server, which VS Code
+# Remote-SSH grows without bound -- interrupted downloads it never cleans up,
+# superseded extension versions, server versions it no longer considers live,
+# and the small CLI binaries that go with them.
+#
+# Every deletion is driven by a signal VS Code itself already writes to disk,
+# plus one extra safety check against currently-running processes -- never by
+# guessing at "newest N" from version numbers or mtimes:
+#   - cli/servers/*.staging           -> interrupted download, never usable
+#   - cli/servers/lru.json            -> server versions VS Code considers live
+#   - code-<hash> with no matching cli/servers/Stable-<hash>  -> orphaned binary
+#   - extensions/.obsolete            -> extension dirs VS Code already marked obsolete
+#   - extensions/extensions.json      -> cross-check: skip if still referenced
+# ~/.vscode-server/data/logs is deliberately left alone: nothing on disk marks
+# a dated log directory as safe to remove, and none of the above signals cover
+# it.
+#
+# This is template-owned rather than shipped from the operator's dotfiles
+# repo: ~/.vscode-server appears the moment VS Code connects to a workspace
+# whether or not dotfiles have ever been applied to it (confirmed on the
+# `test` workspace, which reached ~11 GB with dotfiles never applied), so a
+# cleanup script that only exists via a dotfiles deploy can't cover the case
+# that motivated writing it. See configmap.tf (mounts this into the
+# ConfigMap the pod reads from) and deployment.tf (mounts it into the
+# workspace container at /vscode-server-gc.sh); scripts.tf's coder_script
+# "vscode_server_gc" invokes it directly on its weekly cron -- no existence
+# check, because the ConfigMap mount guarantees it's there whenever the pod
+# is, and see that resource's comment for the silent-no-op failure mode this
+# replaced.
+
+usage() {
+  cat <<'EOF'
+Usage: vscode-server-gc.sh [--dry-run] [root-dir]
+
+  --dry-run   Print what would be removed without removing anything.
+  root-dir    Directory to operate on (default: $HOME/.vscode-server).
+              Exists for testing against a throwaway copy; production use
+              should never need to pass this.
+EOF
+}
+
+dry_run=0
+root=""
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --dry-run)
+      dry_run=1
+      shift
+      ;;
+    -h | --help)
+      usage
+      exit 0
+      ;;
+    -*)
+      echo "Error: unknown option $1" >&2
+      usage >&2
+      exit 1
+      ;;
+    *)
+      if [ -n "$root" ]; then
+        echo "Error: unexpected extra argument $1" >&2
+        exit 1
+      fi
+      root="$1"
+      shift
+      ;;
+  esac
+done
+
+root="${root:-$HOME/.vscode-server}"
+
+if [ ! -d "$root" ]; then
+  echo "no $root; nothing to do"
+  exit 0
+fi
+
+# Refuse to operate on something that doesn't look like a vscode-server data
+# directory, so a bad root-dir argument (or a future refactor) can't turn
+# this into rm -rf of something unrelated.
+if [ ! -d "$root/cli/servers" ] && [ ! -d "$root/extensions" ]; then
+  echo "Error: $root doesn't look like a vscode-server directory (no cli/servers or extensions); refusing to touch it" >&2
+  exit 1
+fi
+
+bytes_freed=0
+count_removed=0
+declare -A logically_removed
+
+# in_use PATH: true if a running process' command line references PATH. An
+# extra safety net on top of the disk-based signals below, so a stale or
+# lagging lru.json/.obsolete entry can never cause us to delete something
+# actually serving a live session.
+in_use() {
+  pgrep -f -- "$1" > /dev/null 2>&1
+}
+
+# path_gone PATH: true if PATH no longer exists, or was already removed
+# earlier in this run (tracked even under --dry-run, so later steps see a
+# consistent preview of the cascade rather than stale on-disk state).
+path_gone() {
+  [ -n "${logically_removed[$1]+x}" ] || [ ! -e "$1" ]
+}
+
+remove() {
+  local path="$1" reason="$2" size
+  size=$(du -sb "$path" 2> /dev/null | cut -f1) || true
+  size=${size:-0}
+  if [ "$dry_run" -eq 1 ]; then
+    echo "[dry-run] would remove $path ($reason, $((size / 1024 / 1024)) MB)"
+  else
+    rm -rf -- "$path"
+    echo "removed $path ($reason, $((size / 1024 / 1024)) MB)"
+  fi
+  logically_removed["$path"]=1
+  bytes_freed=$((bytes_freed + size))
+  count_removed=$((count_removed + 1))
+}
+
+echo "vscode-server-gc: scanning $root"
+
+# --- 1. cli/servers/*.staging: interrupted downloads, never usable. -------
+if [ -d "$root/cli/servers" ]; then
+  for dir in "$root"/cli/servers/*.staging; do
+    [ -d "$dir" ] || continue
+    remove "$dir" "interrupted download (.staging)"
+  done
+fi
+
+# --- 2. cli/servers/Stable-<hash>: prune anything lru.json doesn't list. --
+# lru.json is VS Code's own record of which server versions it considers
+# live. Skip this step entirely (fail safe) if it's missing or unparseable
+# rather than guess at what's current.
+lru_file="$root/cli/servers/lru.json"
+if [ -d "$root/cli/servers" ] && [ -f "$lru_file" ] && jq -e . "$lru_file" > /dev/null 2>&1; then
+  live=" $(jq -r '.[]' "$lru_file" | tr '\n' ' ') "
+  for dir in "$root"/cli/servers/Stable-*; do
+    [ -d "$dir" ] || continue
+    case "$dir" in *.staging) continue ;; esac
+    name=$(basename "$dir")
+    case "$live" in
+      *" $name "*) continue ;;
+    esac
+    if in_use "$dir"; then
+      echo "skipping $dir: not in lru.json but a running process references it"
+      continue
+    fi
+    remove "$dir" "not in lru.json"
+  done
+elif [ -d "$root/cli/servers" ]; then
+  echo "skipping server-version prune: $lru_file missing or not valid JSON"
+fi
+
+# --- 3. code-<hash> binaries orphaned by a removed server directory. ------
+shopt -s nullglob
+for bin in "$root"/code-*; do
+  if [ ! -f "$bin" ] || [ ! -x "$bin" ]; then
+    continue
+  fi
+  hash="${bin##*/code-}"
+  case "$hash" in
+    [0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;;
+    *) continue ;; # not a code-<40-hex-hash> binary; leave it alone
+  esac
+  if ! path_gone "$root/cli/servers/Stable-$hash"; then
+    continue
+  fi
+  if in_use "$bin"; then
+    echo "skipping $bin: no matching server directory but a running process references it"
+    continue
+  fi
+  remove "$bin" "no matching cli/servers/Stable-$hash"
+done
+shopt -u nullglob
+
+# --- 4. extensions/: obsolete versions per .obsolete, unless extensions.json still references them. ---
+obsolete_file="$root/extensions/.obsolete"
+extensions_json="$root/extensions/extensions.json"
+if [ -d "$root/extensions" ] && [ -f "$obsolete_file" ] && jq -e . "$obsolete_file" > /dev/null 2>&1; then
+  in_use_locations=""
+  if [ -f "$extensions_json" ] && jq -e . "$extensions_json" > /dev/null 2>&1; then
+    in_use_locations=" $(jq -r '.[].relativeLocation' "$extensions_json" | tr '\n' ' ') "
+  fi
+  while IFS= read -r name; do
+    [ -n "$name" ] || continue
+    dir="$root/extensions/$name"
+    [ -d "$dir" ] || continue
+    case "$in_use_locations" in
+      *" $name "*)
+        echo "skipping $dir: marked obsolete but extensions.json still references it"
+        continue
+        ;;
+    esac
+    remove "$dir" "marked obsolete in extensions/.obsolete"
+  done < <(jq -r 'keys[]' "$obsolete_file")
+elif [ -d "$root/extensions" ]; then
+  echo "skipping obsolete-extension prune: $obsolete_file missing or not valid JSON"
+fi
+
+summary="vscode-server-gc: done. removed $count_removed item(s), freed $((bytes_freed / 1024 / 1024)) MB"
+if [ "$dry_run" -eq 1 ]; then
+  summary="$summary (dry-run, nothing actually deleted)"
+fi
+echo "$summary"

templates/kubernetes/homelab-workspace/scripts.tf

@@ -30,23 +30,23 @@ resource "coder_script" "memory_watchdog" {
 # Weekly garbage collection of ~/.vscode-server, which grows without bound and
 # inflates the dentry/inode slab.
 #
-# The split is deliberate: the logic operates on a personal directory and lives
-# in the operator's dotfiles repo, while the schedule has to live here because
-# coder_script's cron is the only scheduler this pod has. Missing script => no-op,
-# so this resource is safe before the dotfiles side lands.
+# This used to expect $HOME/.local/bin/vscode-server-gc from the operator's
+# dotfiles, gated by `[ -x ... ] && ... || true`. That broke on any workspace
+# without dotfiles applied - confirmed on the `test` workspace, which reached
+# ~11 GB of ~/.vscode-server with dotfiles never applied to it - because the
+# guard made "the script isn't there" indistinguishable from "the script ran
+# and had nothing to do": both report success on this cron. script-vscode-
+# server-gc.sh is template-owned instead: mounted into the pod via
+# configmap.tf/deployment.tf like script-agent-startup.sh and
+# script-memory-watchdog.sh, so it is guaranteed present whenever this
+# resource's cron fires, and invoked directly below with no existence check -
+# an actual failure now surfaces as a failed run in the Coder UI instead of
+# vanishing into `|| true`.
 resource "coder_script" "vscode_server_gc" {
   agent_id     = coder_agent.main.id
   display_name = "vscode-server GC"
   icon         = "/icon/code.svg"
   # Coder's cron is 6-field (seconds first), not the usual 5. Sundays at 04:00.
   cron   = "0 0 4 * * 0"
-  script = <<-EOT
-    set -u
-    gc="$${HOME}/.local/bin/vscode-server-gc"
-    if [ -x "$${gc}" ]; then
-      "$${gc}"
-    else
-      echo "no $${gc}; skipping"
-    fi
-  EOT
+  script = "/bin/bash /vscode-server-gc.sh"
 }