diff --git a/CLAUDE.md b/CLAUDE.md index 6f199375..d568fc7f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,7 +75,7 @@ Quick orientation map — for what each piece is *for* and the decisions behind | `variables.tf` | `workspace_image`, `test_mode` — both supplied by the release workflow | | `script-agent-startup.sh` / `script-prepare-workspace.sh` | Scripts run on agent/workspace startup | | `script-container-entrypoint.sh` | The workspace container's `command`. Wipes `/tmp` and `exec`s Coder's generated `/workspace-init.sh` — the wipe must precede the agent, see the gotcha below | -| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** (per-role PSS budgets, ten-minute dwell, per-role circuit breaker) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (helpers — the default) / `enforce-all` (helpers + editor) | +| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). It bounds the **standing population of restartable helpers** (per-role PSS budgets, ten-minute dwell, per-role circuit breaker) and records every per-process sweep. It does **not** try to prevent an acute OOM. `memory_watchdog_mode` selects `observe` / `enforce` (helpers — the default) / `enforce-all` (helpers + editor + the uncalibrated `treeHelper` role) | | `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's arithmetic, process selection, budgets, dwell and circuit breaker. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml`. `kill` is shadowed by a function throughout — the fixture pids are real pids in whatever container runs the suite | | `script-vscode-server-gc.sh` | Weekly GC of `~/.vscode-server` (interrupted downloads, superseded server versions/extensions, orphaned CLI binaries — see the script's own header for the exact signal per class, and the `coder_script.vscode_server_gc` comment in `scripts.tf` for why it's template-owned rather than dotfiles-owned) | @@ -91,9 +91,16 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [ - The Dockerfile writes shared env vars to `/etc/environment` rather than using `ENV`, because `PATH` needs to be extended by a script running after the image is built, not fixed at build time. - `parameters.tf`'s `local.validated_*` allowlist is the only thing stopping `system_packages`/`preferred_nodes` from injecting shell metacharacters into the init container — any new parameter whose value reaches a shell must go through the same validate-then-use step. - `script-memory-watchdog.sh` computes headroom as `memory.max − U`, where `U` sums only the *unreclaimable* fields of `memory.stat` (`anon`, `shmem`, `unevictable`, `slab_unreclaimable`, `kernel_stack`, `pagetables`, `sec_pagetables`, `percpu`, `sock`). Do not "simplify" it to `memory.current` or to `memory.stat`'s `kernel` roll-up: on the live pod those read 96% and 42% of the limit while true `U` is 28%. Nothing acts on this number any more — it is pod-level context for the per-process rows and the honest figure published in the workspace UI. +- `calibration.csv`'s `du_bytes_per_s` column is a per-sample instantaneous rate (`(M_U - PREV_U) / SAMPLE_INTERVAL`, 10s by default), not a drift rate, and naively averaging it overstates drift roughly fourfold: ~19.8 kB/s (~68 MB/hour) naive mean on the live pod versus ~18.4 MB/hour computed from `u_mb` across two `event=census` rows 26.1h apart. The naive mean is dominated by a handful of 10s allocation/GC spikes (+102 to −58 MB/s) that an hourly comparison washes out. The column and its caveat are documented together at its declaration in the script; nothing currently reads this column programmatically, so the trap is for a human, not a bug. - **The watchdog is a drift policer, not an OOM preventer, and the difference is measured.** The graded L1–L4 shedding ladder that used to be here was removed, not tuned: the recorded kills are 70–220 MB/s spikes that go from idle to dead inside a minute, a live reproduction climbed the ladder correctly and logged `no-candidates` because the runaway was not in the tree it managed, and the entire editor tree it could shed is ~0.7 GiB — five seconds of that growth. Before re-adding anything reactive, establish that a poll loop can see the event at all. What the loop *is* good at is MB-per-minute growth in the standing population, which is what it now does. -- **Budgets are per role, in PSS, and never below a role's measured *resting* size — not its fresh size.** Fresh tree: extension host 471 MB PSS, serverMain 160 MB, ptyHost 36 MB, file watcher 34 MB. Resting tree (reconnected, idle, 8 GiB pod): extension host **713 MB**, file watcher 88 MB, language server 54 MB, tree total 1093 MB. A uniform 512 MB budget would sit 40 MB above where the extension host *starts* and 200 MB below where it *lives*, which is the same class of defect as the earlier `RLIMIT_DATA` ceiling that landed below what an idle file watcher already held — "calibrated against fresh, deployed against resting" has now appeared twice in this design, so `RESTING_ROLE` holds the resting figures and nothing else. Each budget is `max(min(role budget, memory.max / 8), resting × 1.5)`; on the 8 GiB pod that makes the extension host 1069 MiB rather than the 1024 MiB share, and the derivation records which budgets the floor lifted (`floored=1`), because that means the pod is too small to bound the role at its intended share. The tests assert the *property* — no budget at or below resting, at any pod size — rather than the arithmetic. Override one role with `WATCHDOG_BUDGET_`; PSS (`smaps_rollup`) is the comparison, not RSS and not `VmData`. +- **Budgets are per role, in PSS, and never below a role's measured *resting* size — not its fresh size.** Fresh tree: extension host 471 MB PSS, serverMain 160 MB, ptyHost 36 MB, file watcher 34 MB. Resting tree (reconnected, idle, 8 GiB pod): extension host **713 MB**, file watcher 88 MB, language server 54 MB, tree total 1093 MB. A uniform 512 MB budget would sit 40 MB above where the extension host *starts* and 200 MB below where it *lives*, which is the same class of defect as the earlier `RLIMIT_DATA` ceiling that landed below what an idle file watcher already held — "calibrated against fresh, deployed against resting" has now appeared twice in this design, so `RESTING_ROLE` holds the resting figures and nothing else. Each budget is `max(min(role budget, memory.max / 8), resting × 1.5)`, and the derivation records which budgets the floor lifted (`floored=1`), which is documented to mean the pod is too small to bound that role at its intended share. **That reading only holds while every declared budget already clears its own resting floor, and it did not: `extensionHost` was declared at 1024 MiB against a 1069 MiB floor, so `floored=1` was true at 4, 8, 16 and 64 GiB alike — a property of the constants reported as a property of the pod.** The declared number is now 1069 MiB, the one that was always in force; no effective budget changed at any pod size, and the flag now flips off at 16 GiB. The tests assert the *properties* — no budget at or below resting, and no declared budget below its own floor, at any pod size — rather than the arithmetic. Override one role with `WATCHDOG_BUDGET_`; PSS (`smaps_rollup`) is the comparison, not RSS and not `VmData`. - **A kill needs ten minutes of continuous over-budget dwell, and three kills of one role inside an hour disarm that role.** The dwell is what separates drift from load — a language server that balloons while indexing and hands the memory back must survive. The breaker is what stops the failure that would make this actively harmful: kill the extension host → VS Code restarts it → it reloads every extension → it exceeds again → kill, a loop that arrives looking exactly like the watchdog working. It disarms and reports rather than widening its own budget, because a mechanism that raises the limit it is enforcing has stopped enforcing. +- **Inside the server tree, an unrecognised process is *governed*, not invisible — that default is inverted on purpose, and its risk is bought off by arming rather than by hope.** `role_of` returned `other` for three real helpers on the live tree, and `compute_policed` skips `other`, so they had no budget, no dwell clock and no warning: a `tamasfe.even-better-toml` `server.js` at **280 MB PSS** (already above the 256 MiB `languageServer` budget a matching pattern would have given it), tsserver's `typingsInstaller.js` at 61 MB, and the built-in `markdown-language-features` server at 54 MB. Widening the pattern list is the easy fix and the wrong one: the set of launch shapes is the union of every extension's own choices, so a pattern list is a permanent race against a vendor-controlled vocabulary that fails *silently*. So anything in the tree, on VS Code's own runtime, unprotected and unnamed becomes role `treeHelper`. Two things make that safe rather than merely bold: (1) `treeHelper` is in `UNKNOWN_ROLES`, armed **only** under `enforce-all`, because its 512 MiB budget is a reasoned guess and no member has been measured at rest — under `enforce` (the template default) a coverage gap arrives as `event=would-kill armed=no` in the container log instead of as silence; (2) a **direct child of a server root is excluded structurally** — core forks (`fileWatcher`, `extensionHost`, `ptyHost`) are exactly the direct children of `server-main.js`, and no extension helper is, so a VS Code upgrade that renames `--type=extensionHost` makes that process unmanaged and loud rather than managed on a 512 MiB budget it holds 685 MB against. Its sweep-log identity is `/