diff --git a/CLAUDE.md b/CLAUDE.md index d568fc7f..e292f4a2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,8 +75,8 @@ Quick orientation map — for what each piece is *for* and the decisions behind | `variables.tf` | `workspace_image`, `test_mode` — both supplied by the release workflow | | `script-agent-startup.sh` / `script-prepare-workspace.sh` | Scripts run on agent/workspace startup | | `script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/workspace-init.sh` — the wipe must precede the agent, see the gotcha below | -| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** (per-role PSS budgets, ten-minute dwell, per-role circuit breaker) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (helpers — the default) / `enforce-all` (helpers + editor + the uncalibrated `treeHelper` role) | -| `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's arithmetic, process selection, budgets, dwell and circuit breaker. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite | +| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** against a fixed **2048 MiB envelope** for the VS Code tree (per-role shares of it, PSS, ten-minute dwell) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (the default; arms every role). `enforce-all` is retired and honoured as `enforce` | +| `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's envelope and shares, process selection, dwell, the oversize rule and the kill-rate report. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite | | `script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) | **Image** (`images/homelab-workspace/Dockerfile`): three build stages — `base` (minimal bootstrap deps) → `system-base` (`unminimize` + full interactive toolset) → final stage (env vars into `/etc/environment`, fixed-UID/GID `coder` user, `USER coder`). All `apt`-touching `RUN` steps use BuildKit cache mounts — match that pattern when adding packages. @@ -93,12 +93,16 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [ - `script-memory-watchdog.sh` computes headroom as `memory.max − U`, where `U` sums only the *unreclaimable* fields of `memory.stat` (`anon`, `shmem`, `unevictable`, `slab_unreclaimable`, `kernel_stack`, `pagetables`, `sec_pagetables`, `percpu`, `sock`). Do not "simplify" it to `memory.current` or to `memory.stat`'s `kernel` roll-up: on the live pod those read 96% and 42% of the limit while true `U` is 28%. Nothing acts on this number any more — it is pod-level context for the per-process rows and the honest figure published in the workspace UI. - `calibration.csv`'s `du_bytes_per_s` column is a per-sample instantaneous rate (`(M_U - PREV_U) / SAMPLE_INTERVAL`, 10s by default), not a drift rate, and naively averaging it overstates drift roughly fourfold: ~19.8 kB/s (~68 MB/hour) naive mean on the live pod versus ~18.4 MB/hour computed from `u_mb` across two `event=census` rows 26.1h apart. The naive mean is dominated by a handful of 10s allocation/GC spikes (+102 to −58 MB/s) that an hourly comparison washes out. The column and its caveat are documented together at its declaration in the script; nothing currently reads this column programmatically, so the trap is for a human, not a bug. - **The watchdog is a drift policer, not an OOM preventer, and the difference is measured.** The graded L1–L4 shedding ladder that used to be here was removed, not tuned: the recorded kills are 70–220 MB/s spikes that go from idle to dead inside a minute, a live reproduction climbed the ladder correctly and logged `no-candidates` because the runaway was not in the tree it managed, and the entire editor tree it could shed is ~0.7 GiB — five seconds of that growth. Before re-adding anything reactive, establish that a poll loop can see the event at all. What the loop *is* good at is MB-per-minute growth in the standing population, which is what it now does. -- **Budgets are per role, in PSS, and never below a role's measured *resting* size — not its fresh size.** Fresh tree: extension host 471 MB PSS, serverMain 160 MB, ptyHost 36 MB, file watcher 34 MB. Resting tree (reconnected, idle, 8 GiB pod): extension host **713 MB**, file watcher 88 MB, language server 54 MB, tree total 1093 MB. A uniform 512 MB budget would sit 40 MB above where the extension host *starts* and 200 MB below where it *lives*, which is the same class of defect as the earlier `RLIMIT_DATA` ceiling that landed below what an idle file watcher already held — "calibrated against fresh, deployed against resting" has now appeared twice in this design, so `RESTING_ROLE` holds the resting figures and nothing else. Each budget is `max(min(role budget, memory.max / 8), resting × 1.5)`, and the derivation records which budgets the floor lifted (`floored=1`), which is documented to mean the pod is too small to bound that role at its intended share. **That reading only holds while every declared budget already clears its own resting floor, and it did not: `extensionHost` was declared at 1024 MiB against a 1069 MiB floor, so `floored=1` was true at 4, 8, 16 and 64 GiB alike — a property of the constants reported as a property of the pod.** The declared number is now 1069 MiB, the one that was always in force; no effective budget changed at any pod size, and the flag now flips off at 16 GiB. The tests assert the *properties* — no budget at or below resting, and no declared budget below its own floor, at any pod size — rather than the arithmetic. Override one role with `WATCHDOG_BUDGET_`; PSS (`smaps_rollup`) is the comparison, not RSS and not `VmData`. +- **The VS Code tree gets a fixed 2048 MiB envelope; each role holds a named share of it that does not vary with pod size; a process is killed when its PSS has been above its share for ten continuous minutes *and* it has been observed inside that share since it started.** That last clause is the whole of the `oversize` rule, and it is what replaced the `resting × 1.5` floor. Shares: `serverMain` 512 (the largest, allocated by *consequence* — it holds the remote connection, so a wrong kill costs a disconnect rather than a reload), `extensionHost` 480, `tsserver` 320, `treeHelper` 320, `fileWatcher` 160, `languageServer` 128, `extensionHelper` 128 — 2048 exactly. `claudeHelper` 512 sits **outside** the envelope (not VS Code; sized against the measured 1.66 GB python MCP server). Override one role with `WATCHDOG_BUDGET_`; PSS (`smaps_rollup`) is the comparison, not RSS and not `VmData`. The tests assert *properties* — the shares sum to the envelope and are identical at every `memory.max`, no share exceeds an eighth of the smallest supported pod, `serverMain` is strictly the largest, every role is armed under `enforce` — rather than one pod's arithmetic. +- **Three mechanisms were removed under the envelope, and none of them may come back piecemeal.** (1) The `resting × 1.5` **floor**: it stopped a share landing below what a healthy process already holds — a real scar, this design shipped that twice — but a floor can push the sum past the envelope, which is the envelope defeated by arithmetic. Its protection now comes from the `oversize` rule: a process never seen inside its share is reported forever and never signalled, which is measured on the running process rather than read from a hand-maintained constant that goes stale, and cannot inflate a budget. `RESTING_ROLE`, `RESTING_FACTOR_*` and `FLOORED` are gone with it; the measurements survive as comments beside the share each one argues about. (2) The `memory.max / 8` **divisor**: it was always per-role and never a total (#865's body and the code comment both say so), and it is meaningless against a fixed envelope; its protection survives because the largest share, 512 MiB, is exactly an eighth of the smallest pod this template offers — asserted at every pod size rather than trusted. (3) The **circuit breaker's permanent disarm**: under a deliberate cap, repeated kills mean the process does not fit, not that the number is wrong, and a mechanism that switches itself off after three kills is inert by mid-morning while arriving to look exactly like the watchdog working. It now emits `event=kill-rate ... enforcing=yes` and disarms nothing; the kill *loop* it guarded against cannot form, because the replacement of a killed process arrives over budget and is `oversize`. +- **The extension host does not fit today, and the answer is reporting rather than thrashing.** It arrived at 610 MB PSS at 11 seconds old and settled at 577 MB before doing any work, against a 480 MiB share; no apportionment of 2048 MiB can both keep `serverMain` the largest share and clear 610 MB, which is arithmetic rather than a preference. So the envelope presupposes a tree trimmed in the operator's *dotfiles* (capped launcher heaps, extensions moved client-side) — and because **a template can never depend on dotfiles having been applied** (the `vscode-server-gc` lesson), the untrimmed case is designed for and not merely reported: that one role degrades to a repeating `event=oversize` line while the other six go on being enforced. `tsserver` is the same shape from the other end: it runs with `--max-old-space-size=3072` (VS Code's shipped default, observed live on both instances), so without the dotfiles cap its 320 MiB share means "report a healthy tsserver on a large project", not "bound it". +- **A kill needs ten minutes of continuous over-budget dwell.** That separates drift from load — a language server that balloons while indexing and hands the memory back must survive. The fit evidence behind the `oversize` rule is only taken at or past the 300 s age floor, so the classification cannot become a function of which sweep happened to catch a process seconds after `exec`. +- **The envelope is enforced as an apportionment and *measured* as a total.** Comparison stays per process (a role-level total cannot say which sibling is the offender, and does not compose with a per-`pid:starttime` dwell clock), so N members of one role can hold N × their share. That hole is not fixed; it is published — `TREE_PSS` reaches the census as `tree_pss_mb`/`envelope_pct`, the `summary` file, and every sweep's `TOTAL` row. Acting on the total is deliberately not attempted: that is the graded ladder that was removed because a poll loop cannot see the events it was reacting to. - **A kill needs ten minutes of continuous over-budget dwell, and three kills of one role inside an hour disarm that role.** The dwell is what separates drift from load — a language server that balloons while indexing and hands the memory back must survive. The breaker is what stops the failure that would make this actively harmful: kill the extension host → VS Code restarts it → it reloads every extension → it exceeds again → kill, a loop that arrives looking exactly like the watchdog working. It disarms and reports rather than widening its own budget, because a mechanism that raises the limit it is enforcing has stopped enforcing. -- **Inside the server tree, an unrecognised process is *governed*, not invisible — that default is inverted on purpose, and its risk is bought off by arming rather than by hope.** `role_of` returned `other` for three real helpers on the live tree, and `compute_policed` skips `other`, so they had no budget, no dwell clock and no warning: a `tamasfe.even-better-toml` `server.js` at **280 MB PSS** (already above the 256 MiB `languageServer` budget a matching pattern would have given it), tsserver's `typingsInstaller.js` at 61 MB, and the built-in `markdown-language-features` server at 54 MB. Widening the pattern list is the easy fix and the wrong one: the set of launch shapes is the union of every extension's own choices, so a pattern list is a permanent race against a vendor-controlled vocabulary that fails *silently*. So anything in the tree, on VS Code's own runtime, unprotected and unnamed becomes role `treeHelper`. Two things make that safe rather than merely bold: (1) `treeHelper` is in `UNKNOWN_ROLES`, armed **only** under `enforce-all`, because its 512 MiB budget is a reasoned guess and no member has been measured at rest — under `enforce` (the template default) a coverage gap arrives as `event=would-kill armed=no` in the container log instead of as silence; (2) a **direct child of a server root is excluded structurally** — core forks (`fileWatcher`, `extensionHost`, `ptyHost`) are exactly the direct children of `server-main.js`, and no extension helper is, so a VS Code upgrade that renames `--type=extensionHost` makes that process unmanaged and loud rather than managed on a 512 MiB budget it holds 685 MB against. Its sweep-log identity is `/