Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ What this replaced was `max(min(role budget, memory.max / 8), resting × 1.5)`

**What replaced the floor is a rule about the process rather than about the role: nothing that has never fitted is ever signalled.** A process counts as having fitted only if it has been observed at or below its share while old enough to have finished starting up — without that age condition the classification would be decided by whether a sweep happened to catch it in the seconds after `exec`, when everything is small. A process that was inside its share and drifted out of it is killed, because restarting it demonstrably recovers the memory. A process that has been above its share for its whole observed life is recorded as `oversize` and left alone, because killing it accomplishes nothing: the replacement arrives over budget too, and that is a kill loop with a supervisor's restart in the middle of it. The report repeats once per dwell period, indefinitely — it cannot silence itself, it cannot widen a budget, and it re-arms by itself the moment a replacement is seen fitting. That is a better guard than the floor was, because it is measured on the process that is actually running rather than read from a hand-maintained constant that goes stale, and because it bounds nothing upward.

**The extension host does not fit today, and that is designed for rather than reported.** It arrived at 610 MB PSS at eleven seconds old and settled at 577 MB before doing any work, against a 480 MiB share; no apportionment of 2048 MiB can both keep `serverMain` the largest share and clear 610 MB, which is arithmetic rather than a preference. So the envelope presupposes a tree that has been trimmed — capped launcher heaps and extensions moved to the client side, which happens in the operator's dotfiles — and this template can never depend on dotfiles having been applied, a coupling this repo pair has already been burned by. If the trim has not landed, that one role degrades to a repeating `event=oversize` line while the other six go on being enforced, and the line is exactly the evidence that it has not landed. `tsserver` is the same shape from the other end: it is launched with `--max-old-space-size=3072`, VS Code's shipped default, observed live on both instances in this pod, so without the same dotfiles cap its 320 MiB share means "report a healthy tsserver on a large project" rather than "bound it".
**The extension host does not fit today, and that is designed for rather than reported.** It arrived at 610 MB PSS at eleven seconds old and settled at 577 MB before doing any work, against a 480 MiB share; no apportionment of 2048 MiB can both keep `serverMain` the largest share and clear 610 MB, which is arithmetic rather than a preference. So the envelope presupposes a tree that has been trimmed — capped launcher heaps and extensions moved to the client side, which happens in the operator's dotfiles — and this template can never depend on dotfiles having been applied, a coupling this repo pair has already been burned by. If the trim has not landed, that one role degrades to a repeating `event=oversize` line while the other six go on being enforced, and the line is exactly the evidence that it has not landed. `tsserver` is the same shape from the other end, and a harder case than the sentence that used to sit here allowed: it is launched with `--max-old-space-size=3072`, VS Code's shipped default, observed live on both instances in this pod, but unlike the extension host it *does* fit its 320 MiB share at rest — 111 and 141 MB PSS measured — so the oversize rule never covers it. On a project large enough to hold it above 320 MB for ten minutes, that share means "kill a healthy tsserver", not "report" one. The dotfiles heap cap does not change that, and the section below is why.

**The envelope is enforced as an apportionment and measured as a total, and the difference matters.** Each share is compared against one process at a time, so a role with three members can hold three times its share without any single process being over budget. That hole is not fixed — a role-level total cannot say which sibling is the offender, and it does not compose with a dwell clock keyed per `pid:starttime` precisely so that load can be told from drift — but it is no longer invisible: the tree's total PSS is computed every sweep and published in the census as `tree_pss_mb` and `envelope_pct`, in the summary file, and in the `TOTAL` row of every sweep, so the envelope is a claim a reviewer can falsify from the log. Acting on the total is deliberately not attempted; that is the graded shedding ladder this design already removed, because a poll loop cannot see the events it was reacting to.

Expand All @@ -101,6 +101,36 @@ What this replaced was `max(min(role budget, memory.max / 8), resting × 1.5)`

The trade is unchanged in shape: this is a userspace daemon in a pod with no supervisor, doing crudely what the kernel would do properly if it were allowed to, and it is built to be deleted in one step if that ever changes. It measures deliberately against every stock reading, including Coder's own — page cache and reclaimable slab make this pod look near death while it is idle — so the honest number is published beside the misleading one in the workspace UI rather than replacing it, next to the largest helper as a share of its budget. What remains unaddressed is the acute spike, and honestly so: that is the kernel's job, and with `singleProcessOOMKill` now set at the kubelet it does it properly — the runaway dies and the container survives. Runway is the second line of defence, not the first, and it is the line that matters if the first is ever lost: that kubelet setting is hand-applied node state rather than code, and if a rebuild or the Talos migration drops it the blast radius silently returns to all-or-nothing with nothing to say why.

### Ordering the heap ceiling against the share

A launcher heap ceiling and a PSS share look like two settings of the same dial and are not. `--max-old-space-size` bounds V8's old space; PSS additionally carries the node binary's share of file-backed pages, native allocations, external `ArrayBuffer`s, and V8's new/code/large-object spaces. Measured on this server's bundled node, the default 2048 MB old space corresponds to a 2240 MB total heap limit — ~192 MB of non-old-space on top — and the native and file-backed remainder sits outside even that. Crucially the remainder is roughly *constant per role* rather than proportional to the heap, so the two quantities relate by addition and not by a ratio:

```text
peak PSS ≲ resting PSS + ceiling
```

which gives the whole rule in one line. For a process to fail its own allocation *before* the watchdog signals it — the ordering that trades an external SIGKILL for a fatal heap error the editor itself can report — the ceiling must satisfy:

```text
ceiling ≤ share − resting PSS
```

**Applied to this envelope, not one role clears it.** Resting PSS below is `smaps_rollup` on the live 8 GiB workspace; the ceiling column is what the rule permits.

| Role | Share | Resting PSS | Ceiling the share permits | Knob? |
| --- | --- | --- | --- | --- |
| `serverMain` | 512 MiB | 146 MiB | 366 MiB | none |
| `extensionHost` | 480 MiB | 747 MiB | **negative** | none (see below) |
| `tsserver` | 320 MiB | 141 / 111 MiB | 178 MiB | yes — `js/ts.tsserver.maxMemory` |
| `treeHelper` | 320 MiB | 279 MiB | 41 MiB | none |
| `fileWatcher` | 160 MiB | 117 MiB | 43 MiB | none |
| `languageServer` | 128 MiB | 56 MiB | 72 MiB | none |
| `extensionHelper` | 128 MiB | 31 MiB | 97 MiB | none |

Six of the seven have no ceiling to set at any value. The seventh is arithmetically reachable and is still refused, for three independent reasons. The 178 MiB it would take is near the 128 MiB floor the TypeScript extension clamps the setting to — `Math.max(n, 128)` in the installed `dist/extension.js`. The setting is `scope: "window"` in the extension's own `package.json`, which is exactly the scope the remote Machine settings file does not parse, so it can only live in the operator's client-side User settings and therefore applies to *every* window they open, local Mac projects included, where a 178 MiB TypeScript heap is a crash on any real project. And the failure it buys is not the recoverable one the ordering was meant to deliver: a V8 fatal heap error and an external SIGKILL both arrive at the same handler, which surfaces "The JS/TS language service crashed." and, after five crashes in five minutes, "The service will not be restarted." A ten-minute dwell cannot reach that rate. A binding heap ceiling on a project that legitimately needs the memory reaches it on the first few edits. **So the ceiling stays high and inert on purpose, the watchdog stays first for `tsserver`, and that ordering is the decision rather than the defect it resembles.**

**The extension host has no supported knob, and the route previously assumed to exist does not.** The assumption was that `NODE_OPTIONS`, set from this template's `env.tf`, would reach the remote server tree. It does not: `server-main.js` deletes `NODE_OPTIONS` from the child environment immediately before every fork it performs — one helper, called on the extension-host path and on the shared path that covers the file watcher, pty host and agent host. Setting it pod-wide would therefore cap `server-main.js` itself, cap every mise-node process the operator and Claude Code run, and leave the extension host at its default — maximum blast radius for no effect on the target. The channel that *does* propagate is `process.execArgv`, which the extension host inherits verbatim from `server-main.js` minus `--inspect` flags; but the only place to inject it is the `bin/code-server` shell script inside `cli/servers/Stable-<commit>/`, a directory VS Code replaces wholesale on every server upgrade, and no share of a 2048 MiB envelope is above where the role already rests anyway. The language servers do not even inherit that: `vscode-languageclient` passes `execArgv: []` explicitly when it forks them. There is no setting, no server CLI flag, no `remote.*` property and no `server-env-setup` support in this build — all four checked against the server bundle on disk. The extension host is bounded by loading less, which is `remote.extensionKind` in dotfiles, and by nothing else.

## Outcomes targeted

- One operator can keep dependencies current and ship template/image changes at low ongoing effort, without a fleet of environments to maintain.
Expand Down
45 changes: 35 additions & 10 deletions templates/kubernetes/homelab-workspace/script-memory-watchdog.sh
Original file line number Diff line number Diff line change
Expand Up @@ -219,21 +219,46 @@ declare -gA BUDGET_ROLE=(
# from birth, is classified oversize, and is reported rather than killed. No
# apportionment of 2048 MiB can both keep serverMain the largest share and put
# this one above 610 MB - that is arithmetic, not a preference - so the
# envelope presupposes a tree that has been trimmed. The trimming happens in
# the operator's dotfiles (capped launcher heaps, extensions moved to the
# client side) and this template cannot depend on it having been applied. If it
# envelope presupposes a tree that has been trimmed. The only lever that trims
# it is loading less - remote.extensionKind in the operator's dotfiles, moving
# extensions to the client side. There is no heap ceiling for this role at any
# value: the server registers no such setting, exposes no CLI flag, and deletes
# NODE_OPTIONS from the child environment before every fork, so a variable set
# in env.tf would reach every mise-node process in the pod and not this one.
# The evidence is in DESIGN.md, "Ordering the heap ceiling against the share".
# This template cannot depend on the trim having been applied either. If it
# has not, this role degrades to reporting and the other six go on being
# enforced; the repeating event=oversize line is exactly the evidence that the
# trim has not landed.
[extensionHost]=503316480 # 480 MiB
# 1.5x the 208 MB maximum observed, on a repository containing no TypeScript at
# all - so that figure is a floor of evidence rather than a ceiling. The number
# worth quoting is the other one: tsserver is launched with
# --max-old-space-size=3072, VS Code's shipped default, observed live on both
# instances in this pod. Any share of a 2048 MiB envelope is a fraction of the
# heap its own launcher sanctions, which is why the concurrent dotfiles change
# caps that ceiling too. Without the cap this share means "report a healthy
# tsserver on a large project", not "bound it".
# all - not one .ts file and not one tsconfig.json anywhere under ~/code - so
# that figure is a floor of evidence rather than a ceiling. The number worth
# quoting is the other one: tsserver is launched with --max-old-space-size=3072,
# VS Code's shipped default, observed live on both instances in this pod.
#
# This is the one role where the operator's dotfiles hold a heap ceiling for the
# same process this share bounds, and the two are *deliberately* not ordered so
# that the ceiling binds first. A heap ceiling bounds V8's old space; PSS also
# carries the binary, native allocations, external buffers and V8's other
# spaces, and that remainder is near-constant per role rather than proportional,
# so the two relate by addition: peak PSS is about resting PSS plus the ceiling.
# For V8 to fail first the ceiling would have to be at or under share minus
# resting, which here is 320 - 141 = 179 MiB. It is not set there, on purpose:
# the setting is window-scoped, so it lives in the operator's client-side User
# settings and applies to every window they open including local ones, and the
# failure it would buy is not more recoverable than this one - a V8 fatal heap
# error and a SIGKILL reach the same handler in the TypeScript extension, which
# stops restarting the service after five crashes in five minutes. A ten-minute
# dwell cannot reach that rate; a binding ceiling on a large project reaches it
# immediately. See DESIGN.md, "Ordering the heap ceiling against the share".
#
# What that costs is stated rather than hidden: unlike the extension host this
# role *does* fit at rest, so the oversize rule never covers it, and on a
# project large enough to hold it above this share for ten minutes the share
# means "kill a healthy tsserver". That kill is recoverable - the extension
# restarts the service - and it is rate-limited by the dwell to well under the
# five-in-five-minutes that would make the extension give up.
[tsserver]=335544320 # 320 MiB
# Above the 284 MB maximum observed for the largest member of the role - a
# tamasfe.even-better-toml server - with 13% of headroom. The role's other two
Expand Down
Loading