From acdfa24f7a2f5b8906a3f7c34548f269482fbcbd Mon Sep 17 00:00:00 2001 From: Peter Pathirana Date: Tue, 18 Aug 2026 02:28:24 +0000 Subject: [PATCH] feat: move vscode-server GC into the template as a mounted script ~/.vscode-server grows without bound on any workspace VS Code connects to, whether or not the operator's dotfiles have ever been applied to it (confirmed on the `test` workspace, which reached ~11 GB with dotfiles never applied). The GC for it was previously scoped to the dotfiles repo, invoked from here via `[ -x $HOME/.local/bin/vscode-server-gc ] && ... || true` - a guard that makes "the script isn't there" indistinguishable from "the script ran and had nothing to do", so the weekly cron reported success on every workspace dotfiles hadn't reached. Adds script-vscode-server-gc.sh as a template-owned script alongside script-agent-startup.sh and script-memory-watchdog.sh: mounted into the ConfigMap (configmap.tf), mounted into the workspace container at /vscode-server-gc.sh (deployment.tf), and invoked directly by the existing coder_script.vscode_server_gc cron (scripts.tf) with no existence-check guard, so an actual failure now surfaces as a failed run in the Coder UI. The script's own logic is unchanged from what was verified against the dotfiles version: same deletion signals (cli/servers/*.staging, cli/servers/lru.json, orphaned code- binaries, extensions/.obsolete cross-checked against extensions.json), same pgrep-based in-use safety net, same decision to leave ~/.vscode-server/data/logs alone. Re-verified on the disposable `test` workspace via the new invocation path (/bin/bash /vscode-server-gc.sh): idempotent, zero items to remove on a second run. Ref: ppat/dotfiles#771 --- CLAUDE.md | 2 + .../kubernetes/homelab-workspace/configmap.tf | 1 + .../homelab-workspace/deployment.tf | 5 + .../script-vscode-server-gc.sh | 206 ++++++++++++++++++ .../kubernetes/homelab-workspace/scripts.tf | 26 +-- 5 files changed, 227 insertions(+), 13 deletions(-) create mode 100644 templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh diff --git a/CLAUDE.md b/CLAUDE.md index 7337f7d1..6f199375 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -77,6 +77,7 @@ Quick orientation map — for what each piece is *for* and the decisions behind | `script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/workspace-init.sh` — the wipe must precede the agent, see the gotcha below | | `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** (per-role PSS budgets, ten-minute dwell, per-role circuit breaker) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (helpers — the default) / `enforce-all` (helpers + editor) | | `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's arithmetic, process selection, budgets, dwell and circuit breaker. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite | +| `script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) | **Image** (`images/homelab-workspace/Dockerfile`): three build stages — `base` (minimal bootstrap deps) → `system-base` (`unminimize` + full interactive toolset) → final stage (env vars into `/etc/environment`, fixed-UID/GID `coder` user, `USER coder`). All `apt`-touching `RUN` steps use BuildKit cache mounts — match that pattern when adding packages. @@ -105,6 +106,7 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [ - Adding a package/tool has three possible homes, and picking the wrong one is a real mistake, not a style choice — route by the rule in [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from): universal + stable → image (`Dockerfile`); occasionally-needed + apt-only + too heavy to bake in → the template's `system_packages` parameter; personal, fast-moving, or not an apt package → the operator's dotfiles (a *different* repo — see below), never this one. - `deployment.tf` mounts `/tmp` on its own ephemeral Longhorn volume, not the node's root filesystem and not the NFS-backed home PVC - see [DESIGN.md](DESIGN.md#design-tensions-and-decisions) for why both of those are wrong for it. Its lifecycle is per-Pod, the same as the `system` volume, so it is *not* wiped by a container-only restart within a live Pod - `script-container-entrypoint.sh` wipes it explicitly on every container start instead. Anything relying on `/tmp` persisting across a container restart was already wrong before this (the same was true for free when it was the container's writable overlay). - **The `/tmp` wipe belongs in the container entrypoint and nowhere later - this is a fixed regression, not a style preference.** The workspace container's `command` is `script-container-entrypoint.sh`, which wipes `/tmp` and then `exec`s Coder's generated `/workspace-init.sh`. That bootstrap unpacks the agent CLI into a per-boot `mktemp` directory under `/tmp`, chdirs into it, appends it to the PATH of every session and script the agent runs, and only then runs the agent startup script - so wiping `/tmp` from `script-agent-startup.sh` deletes the CLI the agent installed seconds earlier and every `coder stat` metadata panel reports `coder: command not found`. Do not "fix" a future collision here with an exclusion list: the agent's `/tmp` paths are version-dependent and not consistently prefixed (v2.35.3 owns `coder.XXXXXX/`, `coder-agent.sock`, `coder-agent*.log`, `coder-script-data/`, `coder-screen/` *and* `boundary-audit.sock`), so an allowlist rots silently. `script-agent-startup.sh` asserts the CLI resolves and runs, which is what makes a recurrence loud. +- A `coder_script` in `scripts.tf` must never gate its actual work behind `[ -x ] && ... || true` (or equivalent) when `` is supplied by something outside the template, e.g. a dotfiles-installed binary. That guard makes "the payload isn't there" indistinguishable from "the payload ran and had nothing to do" — both report success on the schedule's weekly cron. This happened: `vscode_server_gc` was originally written that way, expecting `$HOME/.local/bin/vscode-server-gc` from dotfiles, and would have silently no-opped forever on any workspace dotfiles hadn't been applied to (confirmed on the `test` workspace, which reached ~11 GB of `~/.vscode-server` with dotfiles never applied). Fixed by making `script-vscode-server-gc.sh` a template-owned script mounted via `configmap.tf`/`deployment.tf`, so `scripts.tf` invokes it directly and an actual failure now surfaces as a failed run in the Coder UI instead of vanishing into `|| true`. - `deployment.tf`'s Deployment `metadata.name` (`local.workload_name` in `main.tf`) is not cosmetic: the cluster's Prometheus resolves pod → ReplicaSet → Deployment via an existing `kube_pod_owner` recording rule and exposes the result as a `workload` label with no other join needed, so whatever this Deployment is named *is* the identity CPU/memory/PSI/OOM metrics get attributed to. Don't revert it to an opaque identifier (e.g. the workspace UUID) without re-breaking that attribution — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). ## Neighbouring repos diff --git a/templates/kubernetes/homelab-workspace/configmap.tf b/templates/kubernetes/homelab-workspace/configmap.tf index 17f77635..554fed94 100644 --- a/templates/kubernetes/homelab-workspace/configmap.tf +++ b/templates/kubernetes/homelab-workspace/configmap.tf @@ -12,6 +12,7 @@ resource "kubernetes_config_map_v1" "workspace_scripts" { container_entrypoint_script = file("${path.cwd}/script-container-entrypoint.sh") memory_watchdog_script = file("${path.cwd}/script-memory-watchdog.sh") prepare_workspace_script = file("${path.cwd}/script-prepare-workspace.sh") + vscode_server_gc_script = file("${path.cwd}/script-vscode-server-gc.sh") workspace_init_script = coder_agent.main.init_script } } diff --git a/templates/kubernetes/homelab-workspace/deployment.tf b/templates/kubernetes/homelab-workspace/deployment.tf index 4debf653..4966ca14 100644 --- a/templates/kubernetes/homelab-workspace/deployment.tf +++ b/templates/kubernetes/homelab-workspace/deployment.tf @@ -147,6 +147,11 @@ resource "kubernetes_deployment_v1" "deployment" { name = "coder-scripts" sub_path = "memory_watchdog_script" } + volume_mount { + mount_path = "/vscode-server-gc.sh" + name = "coder-scripts" + sub_path = "vscode_server_gc_script" + } volume_mount { mount_path = "/workspace-init.sh" name = "coder-scripts" diff --git a/templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh b/templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh new file mode 100644 index 00000000..eafe758b --- /dev/null +++ b/templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh @@ -0,0 +1,206 @@ +#!/bin/bash +set -euo pipefail + +# vscode-server-gc: reclaims space under ~/.vscode-server, which VS Code +# Remote-SSH grows without bound -- interrupted downloads it never cleans up, +# superseded extension versions, server versions it no longer considers live, +# and the small CLI binaries that go with them. +# +# Every deletion is driven by a signal VS Code itself already writes to disk, +# plus one extra safety check against currently-running processes -- never by +# guessing at "newest N" from version numbers or mtimes: +# - cli/servers/*.staging -> interrupted download, never usable +# - cli/servers/lru.json -> server versions VS Code considers live +# - code- with no matching cli/servers/Stable- -> orphaned binary +# - extensions/.obsolete -> extension dirs VS Code already marked obsolete +# - extensions/extensions.json -> cross-check: skip if still referenced +# ~/.vscode-server/data/logs is deliberately left alone: nothing on disk marks +# a dated log directory as safe to remove, and none of the above signals cover +# it. +# +# This is template-owned rather than shipped from the operator's dotfiles +# repo: ~/.vscode-server appears the moment VS Code connects to a workspace +# whether or not dotfiles have ever been applied to it (confirmed on the +# `test` workspace, which reached ~11 GB with dotfiles never applied), so a +# cleanup script that only exists via a dotfiles deploy can't cover the case +# that motivated writing it. See configmap.tf (mounts this into the +# ConfigMap the pod reads from) and deployment.tf (mounts it into the +# workspace container at /vscode-server-gc.sh); scripts.tf's coder_script +# "vscode_server_gc" invokes it directly on its weekly cron -- no existence +# check, because the ConfigMap mount guarantees it's there whenever the pod +# is, and see that resource's comment for the silent-no-op failure mode this +# replaced. + +usage() { + cat <<'EOF' +Usage: vscode-server-gc.sh [--dry-run] [root-dir] + + --dry-run Print what would be removed without removing anything. + root-dir Directory to operate on (default: $HOME/.vscode-server). + Exists for testing against a throwaway copy; production use + should never need to pass this. +EOF +} + +dry_run=0 +root="" +while [ $# -gt 0 ]; do + case "$1" in + --dry-run) + dry_run=1 + shift + ;; + -h | --help) + usage + exit 0 + ;; + -*) + echo "Error: unknown option $1" >&2 + usage >&2 + exit 1 + ;; + *) + if [ -n "$root" ]; then + echo "Error: unexpected extra argument $1" >&2 + exit 1 + fi + root="$1" + shift + ;; + esac +done + +root="${root:-$HOME/.vscode-server}" + +if [ ! -d "$root" ]; then + echo "no $root; nothing to do" + exit 0 +fi + +# Refuse to operate on something that doesn't look like a vscode-server data +# directory, so a bad root-dir argument (or a future refactor) can't turn +# this into rm -rf of something unrelated. +if [ ! -d "$root/cli/servers" ] && [ ! -d "$root/extensions" ]; then + echo "Error: $root doesn't look like a vscode-server directory (no cli/servers or extensions); refusing to touch it" >&2 + exit 1 +fi + +bytes_freed=0 +count_removed=0 +declare -A logically_removed + +# in_use PATH: true if a running process' command line references PATH. An +# extra safety net on top of the disk-based signals below, so a stale or +# lagging lru.json/.obsolete entry can never cause us to delete something +# actually serving a live session. +in_use() { + pgrep -f -- "$1" > /dev/null 2>&1 +} + +# path_gone PATH: true if PATH no longer exists, or was already removed +# earlier in this run (tracked even under --dry-run, so later steps see a +# consistent preview of the cascade rather than stale on-disk state). +path_gone() { + [ -n "${logically_removed[$1]+x}" ] || [ ! -e "$1" ] +} + +remove() { + local path="$1" reason="$2" size + size=$(du -sb "$path" 2> /dev/null | cut -f1) || true + size=${size:-0} + if [ "$dry_run" -eq 1 ]; then + echo "[dry-run] would remove $path ($reason, $((size / 1024 / 1024)) MB)" + else + rm -rf -- "$path" + echo "removed $path ($reason, $((size / 1024 / 1024)) MB)" + fi + logically_removed["$path"]=1 + bytes_freed=$((bytes_freed + size)) + count_removed=$((count_removed + 1)) +} + +echo "vscode-server-gc: scanning $root" + +# --- 1. cli/servers/*.staging: interrupted downloads, never usable. ------- +if [ -d "$root/cli/servers" ]; then + for dir in "$root"/cli/servers/*.staging; do + [ -d "$dir" ] || continue + remove "$dir" "interrupted download (.staging)" + done +fi + +# --- 2. cli/servers/Stable-: prune anything lru.json doesn't list. -- +# lru.json is VS Code's own record of which server versions it considers +# live. Skip this step entirely (fail safe) if it's missing or unparseable +# rather than guess at what's current. +lru_file="$root/cli/servers/lru.json" +if [ -d "$root/cli/servers" ] && [ -f "$lru_file" ] && jq -e . "$lru_file" > /dev/null 2>&1; then + live=" $(jq -r '.[]' "$lru_file" | tr '\n' ' ') " + for dir in "$root"/cli/servers/Stable-*; do + [ -d "$dir" ] || continue + case "$dir" in *.staging) continue ;; esac + name=$(basename "$dir") + case "$live" in + *" $name "*) continue ;; + esac + if in_use "$dir"; then + echo "skipping $dir: not in lru.json but a running process references it" + continue + fi + remove "$dir" "not in lru.json" + done +elif [ -d "$root/cli/servers" ]; then + echo "skipping server-version prune: $lru_file missing or not valid JSON" +fi + +# --- 3. code- binaries orphaned by a removed server directory. ------ +shopt -s nullglob +for bin in "$root"/code-*; do + if [ ! -f "$bin" ] || [ ! -x "$bin" ]; then + continue + fi + hash="${bin##*/code-}" + case "$hash" in + [0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;; + *) continue ;; # not a code-<40-hex-hash> binary; leave it alone + esac + if ! path_gone "$root/cli/servers/Stable-$hash"; then + continue + fi + if in_use "$bin"; then + echo "skipping $bin: no matching server directory but a running process references it" + continue + fi + remove "$bin" "no matching cli/servers/Stable-$hash" +done +shopt -u nullglob + +# --- 4. extensions/: obsolete versions per .obsolete, unless extensions.json still references them. --- +obsolete_file="$root/extensions/.obsolete" +extensions_json="$root/extensions/extensions.json" +if [ -d "$root/extensions" ] && [ -f "$obsolete_file" ] && jq -e . "$obsolete_file" > /dev/null 2>&1; then + in_use_locations="" + if [ -f "$extensions_json" ] && jq -e . "$extensions_json" > /dev/null 2>&1; then + in_use_locations=" $(jq -r '.[].relativeLocation' "$extensions_json" | tr '\n' ' ') " + fi + while IFS= read -r name; do + [ -n "$name" ] || continue + dir="$root/extensions/$name" + [ -d "$dir" ] || continue + case "$in_use_locations" in + *" $name "*) + echo "skipping $dir: marked obsolete but extensions.json still references it" + continue + ;; + esac + remove "$dir" "marked obsolete in extensions/.obsolete" + done < <(jq -r 'keys[]' "$obsolete_file") +elif [ -d "$root/extensions" ]; then + echo "skipping obsolete-extension prune: $obsolete_file missing or not valid JSON" +fi + +summary="vscode-server-gc: done. removed $count_removed item(s), freed $((bytes_freed / 1024 / 1024)) MB" +if [ "$dry_run" -eq 1 ]; then + summary="$summary (dry-run, nothing actually deleted)" +fi +echo "$summary" diff --git a/templates/kubernetes/homelab-workspace/scripts.tf b/templates/kubernetes/homelab-workspace/scripts.tf index fc6f3cb4..c1ddf0ba 100644 --- a/templates/kubernetes/homelab-workspace/scripts.tf +++ b/templates/kubernetes/homelab-workspace/scripts.tf @@ -30,23 +30,23 @@ resource "coder_script" "memory_watchdog" { # Weekly garbage collection of ~/.vscode-server, which grows without bound and # inflates the dentry/inode slab. # -# The split is deliberate: the logic operates on a personal directory and lives -# in the operator's dotfiles repo, while the schedule has to live here because -# coder_script's cron is the only scheduler this pod has. Missing script => no-op, -# so this resource is safe before the dotfiles side lands. +# This used to expect $HOME/.local/bin/vscode-server-gc from the operator's +# dotfiles, gated by `[ -x ... ] && ... || true`. That broke on any workspace +# without dotfiles applied - confirmed on the `test` workspace, which reached +# ~11 GB of ~/.vscode-server with dotfiles never applied to it - because the +# guard made "the script isn't there" indistinguishable from "the script ran +# and had nothing to do": both report success on this cron. script-vscode- +# server-gc.sh is template-owned instead: mounted into the pod via +# configmap.tf/deployment.tf like script-agent-startup.sh and +# script-memory-watchdog.sh, so it is guaranteed present whenever this +# resource's cron fires, and invoked directly below with no existence check - +# an actual failure now surfaces as a failed run in the Coder UI instead of +# vanishing into `|| true`. resource "coder_script" "vscode_server_gc" { agent_id = coder_agent.main.id display_name = "vscode-server GC" icon = "/icon/code.svg" # Coder's cron is 6-field (seconds first), not the usual 5. Sundays at 04:00. cron = "0 0 4 * * 0" - script = <<-EOT - set -u - gc="$${HOME}/.local/bin/vscode-server-gc" - if [ -x "$${gc}" ]; then - "$${gc}" - else - echo "no $${gc}; skipping" - fi - EOT + script = "/bin/bash /vscode-server-gc.sh" }