diff --git a/CLAUDE.md b/CLAUDE.md index 7337f7d1..6f199375 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -77,6 +77,7 @@ Quick orientation map — for what each piece is *for* and the decisions behind | `script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/workspace-init.sh` — the wipe must precede the agent, see the gotcha below | | `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** (per-role PSS budgets, ten-minute dwell, per-role circuit breaker) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (helpers — the default) / `enforce-all` (helpers + editor) | | `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's arithmetic, process selection, budgets, dwell and circuit breaker. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite | +| `script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) | **Image** (`images/homelab-workspace/Dockerfile`): three build stages — `base` (minimal bootstrap deps) → `system-base` (`unminimize` + full interactive toolset) → final stage (env vars into `/etc/environment`, fixed-UID/GID `coder` user, `USER coder`). All `apt`-touching `RUN` steps use BuildKit cache mounts — match that pattern when adding packages. @@ -105,6 +106,7 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [ - Adding a package/tool has three possible homes, and picking the wrong one is a real mistake, not a style choice — route by the rule in [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from): universal + stable → image (`Dockerfile`); occasionally-needed + apt-only + too heavy to bake in → the template's `system_packages` parameter; personal, fast-moving, or not an apt package → the operator's dotfiles (a *different* repo — see below), never this one. - `deployment.tf` mounts `/tmp` on its own ephemeral Longhorn volume, not the node's root filesystem and not the NFS-backed home PVC - see [DESIGN.md](DESIGN.md#design-tensions-and-decisions) for why both of those are wrong for it. Its lifecycle is per-Pod, the same as the `system` volume, so it is *not* wiped by a container-only restart within a live Pod - `script-container-entrypoint.sh` wipes it explicitly on every container start instead. Anything relying on `/tmp` persisting across a container restart was already wrong before this (the same was true for free when it was the container's writable overlay). - **The `/tmp` wipe belongs in the container entrypoint and nowhere later - this is a fixed regression, not a style preference.** The workspace container's `command` is `script-container-entrypoint.sh`, which wipes `/tmp` and then `exec`s Coder's generated `/workspace-init.sh`. That bootstrap unpacks the agent CLI into a per-boot `mktemp` directory under `/tmp`, chdirs into it, appends it to the PATH of every session and script the agent runs, and only then runs the agent startup script - so wiping `/tmp` from `script-agent-startup.sh` deletes the CLI the agent installed seconds earlier and every `coder stat` metadata panel reports `coder: command not found`. Do not "fix" a future collision here with an exclusion list: the agent's `/tmp` paths are version-dependent and not consistently prefixed (v2.35.3 owns `coder.XXXXXX/`, `coder-agent.sock`, `coder-agent*.log`, `coder-script-data/`, `coder-screen/` *and* `boundary-audit.sock`), so an allowlist rots silently. `script-agent-startup.sh` asserts the CLI resolves and runs, which is what makes a recurrence loud. +- A `coder_script` in `scripts.tf` must never gate its actual work behind `[ -x ] && ... || true` (or equivalent) when `` is supplied by something outside the template, e.g. a dotfiles-installed binary. That guard makes "the payload isn't there" indistinguishable from "the payload ran and had nothing to do" — both report success on the schedule's weekly cron. This happened: `vscode_server_gc` was originally written that way, expecting `$HOME/.local/bin/vscode-server-gc` from dotfiles, and would have silently no-opped forever on any workspace dotfiles hadn't been applied to (confirmed on the `test` workspace, which reached ~11 GB of `~/.vscode-server` with dotfiles never applied). Fixed by making `script-vscode-server-gc.sh` a template-owned script mounted via `configmap.tf`/`deployment.tf`, so `scripts.tf` invokes it directly and an actual failure now surfaces as a failed run in the Coder UI instead of vanishing into `|| true`. - `deployment.tf`'s Deployment `metadata.name` (`local.workload_name` in `main.tf`) is not cosmetic: the cluster's Prometheus resolves pod → ReplicaSet → Deployment via an existing `kube_pod_owner` recording rule and exposes the result as a `workload` label with no other join needed, so whatever this Deployment is named *is* the identity CPU/memory/PSI/OOM metrics get attributed to. Don't revert it to an opaque identifier (e.g. the workspace UUID) without re-breaking that attribution — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). ## Neighbouring repos diff --git a/templates/kubernetes/homelab-workspace/configmap.tf b/templates/kubernetes/homelab-workspace/configmap.tf index 17f77635..554fed94 100644 --- a/templates/kubernetes/homelab-workspace/configmap.tf +++ b/templates/kubernetes/homelab-workspace/configmap.tf @@ -12,6 +12,7 @@ resource "kubernetes_config_map_v1" "workspace_scripts" { container_entrypoint_script = file("${path.cwd}/script-container-entrypoint.sh") memory_watchdog_script = file("${path.cwd}/script-memory-watchdog.sh") prepare_workspace_script = file("${path.cwd}/script-prepare-workspace.sh") + vscode_server_gc_script = file("${path.cwd}/script-vscode-server-gc.sh") workspace_init_script = coder_agent.main.init_script } } diff --git a/templates/kubernetes/homelab-workspace/deployment.tf b/templates/kubernetes/homelab-workspace/deployment.tf index 4debf653..4966ca14 100644 --- a/templates/kubernetes/homelab-workspace/deployment.tf +++ b/templates/kubernetes/homelab-workspace/deployment.tf @@ -147,6 +147,11 @@ resource "kubernetes_deployment_v1" "deployment" { name = "coder-scripts" sub_path = "memory_watchdog_script" } + volume_mount { + mount_path = "/vscode-server-gc.sh" + name = "coder-scripts" + sub_path = "vscode_server_gc_script" + } volume_mount { mount_path = "/workspace-init.sh" name = "coder-scripts" diff --git a/templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh b/templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh new file mode 100644 index 00000000..eafe758b --- /dev/null +++ b/templates/kubernetes/homelab-workspace/script-vscode-server-gc.sh @@ -0,0 +1,206 @@ +#!/bin/bash +set -euo pipefail + +# vscode-server-gc: reclaims space under ~/.vscode-server, which VS Code +# Remote-SSH grows without bound -- interrupted downloads it never cleans up, +# superseded extension versions, server versions it no longer considers live, +# and the small CLI binaries that go with them. +# +# Every deletion is driven by a signal VS Code itself already writes to disk, +# plus one extra safety check against currently-running processes -- never by +# guessing at "newest N" from version numbers or mtimes: +# - cli/servers/*.staging -> interrupted download, never usable +# - cli/servers/lru.json -> server versions VS Code considers live +# - code- with no matching cli/servers/Stable- -> orphaned binary +# - extensions/.obsolete -> extension dirs VS Code already marked obsolete +# - extensions/extensions.json -> cross-check: skip if still referenced +# ~/.vscode-server/data/logs is deliberately left alone: nothing on disk marks +# a dated log directory as safe to remove, and none of the above signals cover +# it. +# +# This is template-owned rather than shipped from the operator's dotfiles +# repo: ~/.vscode-server appears the moment VS Code connects to a workspace +# whether or not dotfiles have ever been applied to it (confirmed on the +# `test` workspace, which reached ~11 GB with dotfiles never applied), so a +# cleanup script that only exists via a dotfiles deploy can't cover the case +# that motivated writing it. See configmap.tf (mounts this into the +# ConfigMap the pod reads from) and deployment.tf (mounts it into the +# workspace container at /vscode-server-gc.sh); scripts.tf's coder_script +# "vscode_server_gc" invokes it directly on its weekly cron -- no existence +# check, because the ConfigMap mount guarantees it's there whenever the pod +# is, and see that resource's comment for the silent-no-op failure mode this +# replaced. + +usage() { + cat <<'EOF' +Usage: vscode-server-gc.sh [--dry-run] [root-dir] + + --dry-run Print what would be removed without removing anything. + root-dir Directory to operate on (default: $HOME/.vscode-server). + Exists for testing against a throwaway copy; production use + should never need to pass this. +EOF +} + +dry_run=0 +root="" +while [ $# -gt 0 ]; do + case "$1" in + --dry-run) + dry_run=1 + shift + ;; + -h | --help) + usage + exit 0 + ;; + -*) + echo "Error: unknown option $1" >&2 + usage >&2 + exit 1 + ;; + *) + if [ -n "$root" ]; then + echo "Error: unexpected extra argument $1" >&2 + exit 1 + fi + root="$1" + shift + ;; + esac +done + +root="${root:-$HOME/.vscode-server}" + +if [ ! -d "$root" ]; then + echo "no $root; nothing to do" + exit 0 +fi + +# Refuse to operate on something that doesn't look like a vscode-server data +# directory, so a bad root-dir argument (or a future refactor) can't turn +# this into rm -rf of something unrelated. +if [ ! -d "$root/cli/servers" ] && [ ! -d "$root/extensions" ]; then + echo "Error: $root doesn't look like a vscode-server directory (no cli/servers or extensions); refusing to touch it" >&2 + exit 1 +fi + +bytes_freed=0 +count_removed=0 +declare -A logically_removed + +# in_use PATH: true if a running process' command line references PATH. An +# extra safety net on top of the disk-based signals below, so a stale or +# lagging lru.json/.obsolete entry can never cause us to delete something +# actually serving a live session. +in_use() { + pgrep -f -- "$1" > /dev/null 2>&1 +} + +# path_gone PATH: true if PATH no longer exists, or was already removed +# earlier in this run (tracked even under --dry-run, so later steps see a +# consistent preview of the cascade rather than stale on-disk state). +path_gone() { + [ -n "${logically_removed[$1]+x}" ] || [ ! -e "$1" ] +} + +remove() { + local path="$1" reason="$2" size + size=$(du -sb "$path" 2> /dev/null | cut -f1) || true + size=${size:-0} + if [ "$dry_run" -eq 1 ]; then + echo "[dry-run] would remove $path ($reason, $((size / 1024 / 1024)) MB)" + else + rm -rf -- "$path" + echo "removed $path ($reason, $((size / 1024 / 1024)) MB)" + fi + logically_removed["$path"]=1 + bytes_freed=$((bytes_freed + size)) + count_removed=$((count_removed + 1)) +} + +echo "vscode-server-gc: scanning $root" + +# --- 1. cli/servers/*.staging: interrupted downloads, never usable. ------- +if [ -d "$root/cli/servers" ]; then + for dir in "$root"/cli/servers/*.staging; do + [ -d "$dir" ] || continue + remove "$dir" "interrupted download (.staging)" + done +fi + +# --- 2. cli/servers/Stable-: prune anything lru.json doesn't list. -- +# lru.json is VS Code's own record of which server versions it considers +# live. Skip this step entirely (fail safe) if it's missing or unparseable +# rather than guess at what's current. +lru_file="$root/cli/servers/lru.json" +if [ -d "$root/cli/servers" ] && [ -f "$lru_file" ] && jq -e . "$lru_file" > /dev/null 2>&1; then + live=" $(jq -r '.[]' "$lru_file" | tr '\n' ' ') " + for dir in "$root"/cli/servers/Stable-*; do + [ -d "$dir" ] || continue + case "$dir" in *.staging) continue ;; esac + name=$(basename "$dir") + case "$live" in + *" $name "*) continue ;; + esac + if in_use "$dir"; then + echo "skipping $dir: not in lru.json but a running process references it" + continue + fi + remove "$dir" "not in lru.json" + done +elif [ -d "$root/cli/servers" ]; then + echo "skipping server-version prune: $lru_file missing or not valid JSON" +fi + +# --- 3. code- binaries orphaned by a removed server directory. ------ +shopt -s nullglob +for bin in "$root"/code-*; do + if [ ! -f "$bin" ] || [ ! -x "$bin" ]; then + continue + fi + hash="${bin##*/code-}" + case "$hash" in + [0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;; + *) continue ;; # not a code-<40-hex-hash> binary; leave it alone + esac + if ! path_gone "$root/cli/servers/Stable-$hash"; then + continue + fi + if in_use "$bin"; then + echo "skipping $bin: no matching server directory but a running process references it" + continue + fi + remove "$bin" "no matching cli/servers/Stable-$hash" +done +shopt -u nullglob + +# --- 4. extensions/: obsolete versions per .obsolete, unless extensions.json still references them. --- +obsolete_file="$root/extensions/.obsolete" +extensions_json="$root/extensions/extensions.json" +if [ -d "$root/extensions" ] && [ -f "$obsolete_file" ] && jq -e . "$obsolete_file" > /dev/null 2>&1; then + in_use_locations="" + if [ -f "$extensions_json" ] && jq -e . "$extensions_json" > /dev/null 2>&1; then + in_use_locations=" $(jq -r '.[].relativeLocation' "$extensions_json" | tr '\n' ' ') " + fi + while IFS= read -r name; do + [ -n "$name" ] || continue + dir="$root/extensions/$name" + [ -d "$dir" ] || continue + case "$in_use_locations" in + *" $name "*) + echo "skipping $dir: marked obsolete but extensions.json still references it" + continue + ;; + esac + remove "$dir" "marked obsolete in extensions/.obsolete" + done < <(jq -r 'keys[]' "$obsolete_file") +elif [ -d "$root/extensions" ]; then + echo "skipping obsolete-extension prune: $obsolete_file missing or not valid JSON" +fi + +summary="vscode-server-gc: done. removed $count_removed item(s), freed $((bytes_freed / 1024 / 1024)) MB" +if [ "$dry_run" -eq 1 ]; then + summary="$summary (dry-run, nothing actually deleted)" +fi +echo "$summary" diff --git a/templates/kubernetes/homelab-workspace/scripts.tf b/templates/kubernetes/homelab-workspace/scripts.tf index fc6f3cb4..c1ddf0ba 100644 --- a/templates/kubernetes/homelab-workspace/scripts.tf +++ b/templates/kubernetes/homelab-workspace/scripts.tf @@ -30,23 +30,23 @@ resource "coder_script" "memory_watchdog" { # Weekly garbage collection of ~/.vscode-server, which grows without bound and # inflates the dentry/inode slab. # -# The split is deliberate: the logic operates on a personal directory and lives -# in the operator's dotfiles repo, while the schedule has to live here because -# coder_script's cron is the only scheduler this pod has. Missing script => no-op, -# so this resource is safe before the dotfiles side lands. +# This used to expect $HOME/.local/bin/vscode-server-gc from the operator's +# dotfiles, gated by `[ -x ... ] && ... || true`. That broke on any workspace +# without dotfiles applied - confirmed on the `test` workspace, which reached +# ~11 GB of ~/.vscode-server with dotfiles never applied to it - because the +# guard made "the script isn't there" indistinguishable from "the script ran +# and had nothing to do": both report success on this cron. script-vscode- +# server-gc.sh is template-owned instead: mounted into the pod via +# configmap.tf/deployment.tf like script-agent-startup.sh and +# script-memory-watchdog.sh, so it is guaranteed present whenever this +# resource's cron fires, and invoked directly below with no existence check - +# an actual failure now surfaces as a failed run in the Coder UI instead of +# vanishing into `|| true`. resource "coder_script" "vscode_server_gc" { agent_id = coder_agent.main.id display_name = "vscode-server GC" icon = "/icon/code.svg" # Coder's cron is 6-field (seconds first), not the usual 5. Sundays at 04:00. cron = "0 0 4 * * 0" - script = <<-EOT - set -u - gc="$${HOME}/.local/bin/vscode-server-gc" - if [ -x "$${gc}" ]; then - "$${gc}" - else - echo "no $${gc}; skipping" - fi - EOT + script = "/bin/bash /vscode-server-gc.sh" }