From bb52cf2c193a71602c685a0b32451c47daff64cc Mon Sep 17 00:00:00 2001 From: Peter Pathirana Date: Fri, 21 Aug 2026 22:44:00 +0000 Subject: [PATCH] feat: give the VS Code tree one memory envelope and shares of it Budgets were max(min(role budget, memory.max / 8), resting x 1.5): three interacting terms, each patching the previous one's failure, with no term anywhere that bounded the total. They are now one sentence. The VS Code tree gets a fixed 2048 MiB envelope. Each role holds a named share of it that does not vary with pod size, and a process is killed when its PSS has been above its share for ten continuous minutes AND it has been observed inside that share at some point since it started. claudeHelper (512 MiB) sits outside the envelope: it is not VS Code, and it is anchored on a measured 1.66 GB adversary rather than on a share of anything. Shares, summing to 2048 exactly: serverMain 512, extensionHost 480, tsserver 320, treeHelper 320, fileWatcher 160, languageServer 128, extensionHelper 128. serverMain is the largest by consequence rather than by size - it holds the remote connection, so a wrong kill costs a disconnect rather than a reload. Every other figure is argued in the script against the maximum the sweep log recorded for that role over ~38 hours. Three mechanisms are removed rather than accommodated. The resting x 1.5 floor. It stopped a share landing below what a healthy process already holds - a real scar - but a floor can push the sum past the envelope, which is the envelope defeated by arithmetic. What replaces it is a rule about the process instead of the role: nothing that has never been seen inside its share is signalled. It is reported as event=oversize, forever, once per dwell period. That is a better guard, because it is measured on the process actually running rather than read from a constant that goes stale, and it cannot inflate a budget. The memory.max / 8 divisor. It was always per-role and never a total, and it is meaningless against a fixed envelope. Its protection survives by construction: the largest share is exactly an eighth of the smallest pod this template offers, asserted at every pod size rather than trusted. The circuit breaker's permanent disarm. Under a deliberate cap, repeated kills mean the process does not fit, not that the number is wrong, and a mechanism that switches itself off after three kills is inert by mid-morning while arriving to look exactly like the watchdog working. It now emits event=kill-rate ... enforcing=yes. The kill loop it guarded against cannot form: a killed process's replacement arrives over budget and is oversize. Every role is armed under enforce, so enforce-all has no superset left to name. It is retired from the parameter and honoured as enforce in both the Terraform local and normalise_mode, because a workspace carrying that stored value asked for more enforcement and must not silently get none. The extension host does not fit today: it arrived at 610 MB PSS at 11s old and settled at 577 MB before doing any work. No apportionment of 2048 MiB can both keep serverMain the largest share and clear that, so the envelope presupposes a tree trimmed in dotfiles - and because a template can never depend on dotfiles having been applied, the untrimmed case degrades to a repeating oversize line while the other six roles go on being enforced. The envelope is enforced as an apportionment and measured as a total: TREE_PSS is published in the census, the summary and every sweep's TOTAL row, so the claim is falsifiable from the log. Acting on the total is not attempted. Tests: 355 passed (332 at base). Thirteen mutations were applied to the production script and every one flips an assertion red, including reintroducing the divisor, restoring the disarm, deleting either half of the oversize rule, and moving the oversize branch ahead of the in-flight escalation. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0197hA8JPwGMX8wufiQiy6oz --- CLAUDE.md | 16 +- DESIGN.md | 16 +- TESTING.md | 8 +- templates/kubernetes/homelab-workspace/env.tf | 11 +- .../homelab-workspace/parameters.tf | 29 +- .../script-memory-watchdog-test.sh | 741 ++++++++++++------ .../script-memory-watchdog.sh | 541 +++++++------ 7 files changed, 856 insertions(+), 506 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d568fc7f..e292f4a2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,8 +75,8 @@ Quick orientation map — for what each piece is *for* and the decisions behind | `variables.tf` | `workspace_image`, `test_mode` — both supplied by the release workflow | | `script-agent-startup.sh` / `script-prepare-workspace.sh` | Scripts run on agent/workspace startup | | `script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/workspace-init.sh` — the wipe must precede the agent, see the gotcha below | -| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** (per-role PSS budgets, ten-minute dwell, per-role circuit breaker) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (helpers — the default) / `enforce-all` (helpers + editor + the uncalibrated `treeHelper` role) | -| `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's arithmetic, process selection, budgets, dwell and circuit breaker. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite | +| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** against a fixed **2048 MiB envelope** for the VS Code tree (per-role shares of it, PSS, ten-minute dwell) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (the default; arms every role). `enforce-all` is retired and honoured as `enforce` | +| `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's envelope and shares, process selection, dwell, the oversize rule and the kill-rate report. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite | | `script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) | **Image** (`images/homelab-workspace/Dockerfile`): three build stages — `base` (minimal bootstrap deps) → `system-base` (`unminimize` + full interactive toolset) → final stage (env vars into `/etc/environment`, fixed-UID/GID `coder` user, `USER coder`). All `apt`-touching `RUN` steps use BuildKit cache mounts — match that pattern when adding packages. @@ -93,12 +93,16 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [ - `script-memory-watchdog.sh` computes headroom as `memory.max − U`, where `U` sums only the *unreclaimable* fields of `memory.stat` (`anon`, `shmem`, `unevictable`, `slab_unreclaimable`, `kernel_stack`, `pagetables`, `sec_pagetables`, `percpu`, `sock`). Do not "simplify" it to `memory.current` or to `memory.stat`'s `kernel` roll-up: on the live pod those read 96% and 42% of the limit while true `U` is 28%. Nothing acts on this number any more — it is pod-level context for the per-process rows and the honest figure published in the workspace UI. - `calibration.csv`'s `du_bytes_per_s` column is a per-sample instantaneous rate (`(M_U - PREV_U) / SAMPLE_INTERVAL`, 10s by default), not a drift rate, and naively averaging it overstates drift roughly fourfold: ~19.8 kB/s (~68 MB/hour) naive mean on the live pod versus ~18.4 MB/hour computed from `u_mb` across two `event=census` rows 26.1h apart. The naive mean is dominated by a handful of 10s allocation/GC spikes (+102 to −58 MB/s) that an hourly comparison washes out. The column and its caveat are documented together at its declaration in the script; nothing currently reads this column programmatically, so the trap is for a human, not a bug. - **The watchdog is a drift policer, not an OOM preventer, and the difference is measured.** The graded L1–L4 shedding ladder that used to be here was removed, not tuned: the recorded kills are 70–220 MB/s spikes that go from idle to dead inside a minute, a live reproduction climbed the ladder correctly and logged `no-candidates` because the runaway was not in the tree it managed, and the entire editor tree it could shed is ~0.7 GiB — five seconds of that growth. Before re-adding anything reactive, establish that a poll loop can see the event at all. What the loop *is* good at is MB-per-minute growth in the standing population, which is what it now does. -- **Budgets are per role, in PSS, and never below a role's measured *resting* size — not its fresh size.** Fresh tree: extension host 471 MB PSS, serverMain 160 MB, ptyHost 36 MB, file watcher 34 MB. Resting tree (reconnected, idle, 8 GiB pod): extension host **713 MB**, file watcher 88 MB, language server 54 MB, tree total 1093 MB. A uniform 512 MB budget would sit 40 MB above where the extension host *starts* and 200 MB below where it *lives*, which is the same class of defect as the earlier `RLIMIT_DATA` ceiling that landed below what an idle file watcher already held — "calibrated against fresh, deployed against resting" has now appeared twice in this design, so `RESTING_ROLE` holds the resting figures and nothing else. Each budget is `max(min(role budget, memory.max / 8), resting × 1.5)`, and the derivation records which budgets the floor lifted (`floored=1`), which is documented to mean the pod is too small to bound that role at its intended share. **That reading only holds while every declared budget already clears its own resting floor, and it did not: `extensionHost` was declared at 1024 MiB against a 1069 MiB floor, so `floored=1` was true at 4, 8, 16 and 64 GiB alike — a property of the constants reported as a property of the pod.** The declared number is now 1069 MiB, the one that was always in force; no effective budget changed at any pod size, and the flag now flips off at 16 GiB. The tests assert the *properties* — no budget at or below resting, and no declared budget below its own floor, at any pod size — rather than the arithmetic. Override one role with `WATCHDOG_BUDGET_`; PSS (`smaps_rollup`) is the comparison, not RSS and not `VmData`. +- **The VS Code tree gets a fixed 2048 MiB envelope; each role holds a named share of it that does not vary with pod size; a process is killed when its PSS has been above its share for ten continuous minutes *and* it has been observed inside that share since it started.** That last clause is the whole of the `oversize` rule, and it is what replaced the `resting × 1.5` floor. Shares: `serverMain` 512 (the largest, allocated by *consequence* — it holds the remote connection, so a wrong kill costs a disconnect rather than a reload), `extensionHost` 480, `tsserver` 320, `treeHelper` 320, `fileWatcher` 160, `languageServer` 128, `extensionHelper` 128 — 2048 exactly. `claudeHelper` 512 sits **outside** the envelope (not VS Code; sized against the measured 1.66 GB python MCP server). Override one role with `WATCHDOG_BUDGET_`; PSS (`smaps_rollup`) is the comparison, not RSS and not `VmData`. The tests assert *properties* — the shares sum to the envelope and are identical at every `memory.max`, no share exceeds an eighth of the smallest supported pod, `serverMain` is strictly the largest, every role is armed under `enforce` — rather than one pod's arithmetic. +- **Three mechanisms were removed under the envelope, and none of them may come back piecemeal.** (1) The `resting × 1.5` **floor**: it stopped a share landing below what a healthy process already holds — a real scar, this design shipped that twice — but a floor can push the sum past the envelope, which is the envelope defeated by arithmetic. Its protection now comes from the `oversize` rule: a process never seen inside its share is reported forever and never signalled, which is measured on the running process rather than read from a hand-maintained constant that goes stale, and cannot inflate a budget. `RESTING_ROLE`, `RESTING_FACTOR_*` and `FLOORED` are gone with it; the measurements survive as comments beside the share each one argues about. (2) The `memory.max / 8` **divisor**: it was always per-role and never a total (#865's body and the code comment both say so), and it is meaningless against a fixed envelope; its protection survives because the largest share, 512 MiB, is exactly an eighth of the smallest pod this template offers — asserted at every pod size rather than trusted. (3) The **circuit breaker's permanent disarm**: under a deliberate cap, repeated kills mean the process does not fit, not that the number is wrong, and a mechanism that switches itself off after three kills is inert by mid-morning while arriving to look exactly like the watchdog working. It now emits `event=kill-rate ... enforcing=yes` and disarms nothing; the kill *loop* it guarded against cannot form, because the replacement of a killed process arrives over budget and is `oversize`. +- **The extension host does not fit today, and the answer is reporting rather than thrashing.** It arrived at 610 MB PSS at 11 seconds old and settled at 577 MB before doing any work, against a 480 MiB share; no apportionment of 2048 MiB can both keep `serverMain` the largest share and clear 610 MB, which is arithmetic rather than a preference. So the envelope presupposes a tree trimmed in the operator's *dotfiles* (capped launcher heaps, extensions moved client-side) — and because **a template can never depend on dotfiles having been applied** (the `vscode-server-gc` lesson), the untrimmed case is designed for and not merely reported: that one role degrades to a repeating `event=oversize` line while the other six go on being enforced. `tsserver` is the same shape from the other end: it runs with `--max-old-space-size=3072` (VS Code's shipped default, observed live on both instances), so without the dotfiles cap its 320 MiB share means "report a healthy tsserver on a large project", not "bound it". +- **A kill needs ten minutes of continuous over-budget dwell.** That separates drift from load — a language server that balloons while indexing and hands the memory back must survive. The fit evidence behind the `oversize` rule is only taken at or past the 300 s age floor, so the classification cannot become a function of which sweep happened to catch a process seconds after `exec`. +- **The envelope is enforced as an apportionment and *measured* as a total.** Comparison stays per process (a role-level total cannot say which sibling is the offender, and does not compose with a per-`pid:starttime` dwell clock), so N members of one role can hold N × their share. That hole is not fixed; it is published — `TREE_PSS` reaches the census as `tree_pss_mb`/`envelope_pct`, the `summary` file, and every sweep's `TOTAL` row. Acting on the total is deliberately not attempted: that is the graded ladder that was removed because a poll loop cannot see the events it was reacting to. - **A kill needs ten minutes of continuous over-budget dwell, and three kills of one role inside an hour disarm that role.** The dwell is what separates drift from load — a language server that balloons while indexing and hands the memory back must survive. The breaker is what stops the failure that would make this actively harmful: kill the extension host → VS Code restarts it → it reloads every extension → it exceeds again → kill, a loop that arrives looking exactly like the watchdog working. It disarms and reports rather than widening its own budget, because a mechanism that raises the limit it is enforcing has stopped enforcing. -- **Inside the server tree, an unrecognised process is *governed*, not invisible — that default is inverted on purpose, and its risk is bought off by arming rather than by hope.** `role_of` returned `other` for three real helpers on the live tree, and `compute_policed` skips `other`, so they had no budget, no dwell clock and no warning: a `tamasfe.even-better-toml` `server.js` at **280 MB PSS** (already above the 256 MiB `languageServer` budget a matching pattern would have given it), tsserver's `typingsInstaller.js` at 61 MB, and the built-in `markdown-language-features` server at 54 MB. Widening the pattern list is the easy fix and the wrong one: the set of launch shapes is the union of every extension's own choices, so a pattern list is a permanent race against a vendor-controlled vocabulary that fails *silently*. So anything in the tree, on VS Code's own runtime, unprotected and unnamed becomes role `treeHelper`. Two things make that safe rather than merely bold: (1) `treeHelper` is in `UNKNOWN_ROLES`, armed **only** under `enforce-all`, because its 512 MiB budget is a reasoned guess and no member has been measured at rest — under `enforce` (the template default) a coverage gap arrives as `event=would-kill armed=no` in the container log instead of as silence; (2) a **direct child of a server root is excluded structurally** — core forks (`fileWatcher`, `extensionHost`, `ptyHost`) are exactly the direct children of `server-main.js`, and no extension helper is, so a VS Code upgrade that renames `--type=extensionHost` makes that process unmanaged and loud rather than managed on a 512 MiB budget it holds 685 MB against. Its sweep-log identity is `/