Skip to content

feat: give the VS Code tree one memory envelope and shares of it - #884

Merged
ppat merged 1 commit into
mainfrom
watchdog-envelope-v2
Aug 22, 2026
Merged

feat: give the VS Code tree one memory envelope and shares of it#884
ppat merged 1 commit into
mainfrom
watchdog-envelope-v2

Conversation

@ppat

@ppat ppat commented Aug 21, 2026

Copy link
Copy Markdown
Owner

The construction, in one sentence

The VS Code tree gets a fixed 2048 MiB envelope, each role holds a named share of it that does not vary with pod size, and a process is killed when its PSS has been above its share for ten continuous minutes and it has been observed inside that share at some point since it started.

What it replaces is max(min(role budget, memory.max / 8), resting × 1.5) — three interacting terms, each patching the previous one's failure, and no term anywhere that bounded the total.

The apportionment

Role Share Max observed (38 h) Derivation
serverMain 512 MiB 233 MB The largest single share, allocated by consequence rather than size: it holds the remote connection, so a wrong kill costs a disconnect rather than a reload. That the biggest share goes to the role with the smallest measured appetite costs ~280 MiB and is paid on purpose.
extensionHost 480 MiB 738 MB (hourly minimum climbed 528 → 686 over 29 h) The residual, and deliberately second-largest, because it is the only role that measurably drifts. It does not fit today — see below.
tsserver 320 MiB 208 MB 1.5× observed, on a repository containing no TypeScript — so that figure is a floor of evidence, not a ceiling. The number that matters is the other one: --max-old-space-size=3072, VS Code's shipped default, observed live on both instances in this pod.
treeHelper 320 MiB 284 MB (a TOML server) Above the largest measured member with 13% headroom. 256 MiB was considered and rejected: it sits below a member measured on this very workspace, so it would have shipped a role that was oversize on day one.
fileWatcher 160 MiB 107 MB, flat to the megabyte for 28 straight hours 1.5×. Headroom is for repositories with more files than this one — the one role whose size is a direct function of what is watched.
languageServer 128 MiB 31 MB 4×.
extensionHelper 128 MiB 24 MB (terraform-ls) 5×.
2048 MiB
claudeHelper 512 MiB 1.66 GB (the adversary); 142 MB the largest healthy MCP server measured Outside the envelope. Not VS Code, not spawned by the editor, and anchored on a measured offender rather than on a share of anything. ~3.6× the honest case, ~3.2× smaller than the adversary. Tightening it would buy the editor nothing.

Where I argued with the starting split

  • treeHelper 256 → 320. 256 is below a member measured at 284 MB on this workspace: it would have been permanently oversize from the first sweep.
  • languageServer 192 → 128 and extensionHelper 128 → 128; fileWatcher 192 → 160. The starting split gave 512 MiB to three roles whose combined measured maximum is 162 MB, and in a zero-sum envelope every one of those MiB is taken from extensionHost and treeHelper, which have measured members at 738 and 284. The counter-argument (fix: declare what each watchdog budget is anchored on, and stop arming guesses #883's, and it is a fair one) is that gopls/rust-analyzer live in those classes and are far larger. What licenses the tighter number is the oversize rule: a share that turns out to be too small for a legitimate process now produces a repeating log line and no kill, so being wrong costs evidence rather than a restart loop.
  • extensionHost 384 → 480, tsserver 384 → 320. Same trade, in the direction of the role that actually drifts.

The measurements, and what they are worth

All figures are observed, from this watchdog's own sweep.log over 2026-08-20 07:24 → 2026-08-21 22:14 on the live 8 GiB workspace. It is one operator, one Terraform/YAML/shell repository, and eight distinct processes behind the whole table — a role with 1 762 rows is one process sampled 1 762 times. None of these is a distribution.

One number is new and load-bearing: the whole VS Code tree ranged 854 → 1 732 MB PSS over the window, mean 1 353 MB, and its hourly mean climbed 1 130 → 1 731 MB over the last 30 hours — roughly 20 MB/h in the standing population. So the envelope is not binding today, and is binding in about 15 more hours of continuous uptime at the observed rate. That is the drift this loop exists to police, seen as a total for the first time.

The three removed mechanisms — Chesterton, one at a time

1. The resting × 1.5 floor. Protected against: a share landing below what a healthy process already holds, which under enforcement is a kill loop with a supervisor's restart in the middle of it. A real scar — this design shipped it twice, most memorably an RLIMIT_DATA ceiling below what an idle file watcher held. Why it cannot stay: a floor and an envelope are incompatible instruments. Under a total, a process whose resting size does not fit means the process must be made smaller; a floor says the budget must grow to meet it, and enough of those push the sum past the envelope. What protects now: the oversize rule — a process that has never been observed inside its share since it started is never signalled, only reported. That is a better guard than the floor: it is measured on the process actually running rather than read from a hand-maintained constant that goes stale, and it bounds nothing upward. RESTING_ROLE, RESTING_FACTOR_* and FLOORED are deleted; the measurements survive as comments beside the share each argues about.

2. The memory.max / 8 divisor. Protected against: a budget reasoned about for the 8 GiB pod letting one helper own a quarter of the 4 GiB one. (Per-role, never a total — #865's body and the code comment both say so.) What protects now: construction. The largest single share, 512 MiB, is exactly an eighth of the smallest pod this template offers, and the suite asserts that property at every pod size rather than trusting the coincidence.

3. The circuit breaker's permanent disarm. Protected against: the kill loop — kill the extension host → VS Code restarts it → it reloads every extension → it exceeds again → kill — which arrives looking exactly like the watchdog working. Why it cannot stay: under a deliberate cap, repeated kills mean the process does not fit, which is a decision rather than a defect. A mechanism that switches itself off after three of them is inert by mid-morning — and that is the failure the breaker was built to prevent, now caused by the breaker. What protects now: the loop cannot form. A killed process's replacement arrives over budget, has never been seen fitting, and is oversize. The counter is kept and reports event=kill-rate role=… enforcing=yes, which is a sizing signal for a human while enforcement continues.

The structurally-over-budget case

There is a real difference between "this grew past its share" and "this never fitted", and the old logs could not tell them apart. They now can, and the two are treated differently:

  • Drift — observed at or below its share while old enough to have finished starting up, then above it for ten continuous minutes → killed. Restarting it demonstrably recovers the memory.
  • Oversize — above its share for its whole observed life → event=oversize, once per dwell period, indefinitely, and never signalled. Killing it accomplishes nothing: the replacement arrives over budget too.

It is per process (keyed pid:starttime), evidence-based rather than count-based, it re-arms by itself the moment a replacement is seen fitting, it cannot silence itself, and it cannot widen a budget. The fit evidence is only taken at or past the 300 s age floor, or the classification would be decided by whether a sweep happened to catch the process seconds after exec.

The known gap, stated rather than hidden: a process born far too big and then growing further is recorded and never signalled. That is accepted — a process born at ten times its share is not the drift this loop polices — and the sweep log names it either way.

What happens if the dotfiles trim has not landed

Observed: the live extension host arrived at 610 MB PSS at 11 seconds old and settled at 577 MB before doing any work. Not the 471 MB "fresh" figure the docs carried.

Therefore, and this is arithmetic rather than a preference: no apportionment of 2048 MiB can both keep serverMain the largest share and put extensionHost's share above 610 MB. Clearing 610 needs ≥ 640 for the extension host and ≥ 704 for serverMain; that is 1 344 of 2 048 for two roles, leaving 704 for five, of which treeHelper alone needs 320.

So the envelope presupposes a trimmed tree, and this template can never depend on dotfiles having been applied — the coupling this repo pair was already burned by, which is why vscode-server-gc is template-owned. If the trim has not landed, that one role degrades to a repeating event=oversize line and the other six go on being enforced. It does not thrash, it does not widen anything, and the line is exactly the evidence that the trim has not landed. The trim needs to bring the extension host's arrival size under 480 MiB — a ~21% reduction from today's 610 MB.

tsserver is the same shape from the other end: with --max-old-space-size=3072 still in force its 320 MiB share means "report a healthy tsserver on a large project", not "bound it".

The envelope is enforced as an apportionment and measured as a total

Comparison stays per process — a role-level total cannot say which sibling is the offender, and does not compose with a dwell clock keyed per pid:starttime precisely so load can be told from drift. So a role with three members can hold three times its share without any single process being over budget. That hole is not fixed and no longer invisible: TREE_PSS is computed every sweep and published in the census as tree_pss_mb/envelope_pct, in the summary file, and in every sweep's TOTAL row, so the envelope is a claim a reviewer can falsify from the log. Acting on the total is deliberately not attempted — that is the graded ladder this design already removed, because a poll loop cannot see the events it was reacting to.

Modes

enforce arms every role, so enforce-all has no superset left to name and a mode that does nothing different is the "indistinguishable from broken" harm. It is retired from the parameter and honoured as enforce in both the Terraform local and the script's new normalise_mode — a workspace still carrying that stored value asked for more enforcement and must not silently get none. Anything unrecognised still falls back to inert observe.

What I did about #883, and why

#883 is superseded and should be closed. It added BUDGET_BASIS (measured/adversary/guess) and derived arming from it, disarming guess-anchored roles under enforce. Two of its three load-bearing parts do not survive this change:

  • The arming derivation is overruled by the operator's decision that every role is armed.
  • The basis taxonomy loses its meaning. Under an envelope, a share is neither measured nor guessed — it is allocated. Every share has the same provenance: a division of a fixed total. A column that answers "measured or guess?" for a number that is by construction an apportionment decision would be answering the wrong question.
  • Its fileWatcher resting correction (88 → 107 MiB) and its RESTING_ROLE staleness reasoning are moot: RESTING_ROLE is deleted.

Salvaged here, with attribution in the code:

  • BUDGET_ORDER — a declared reporting order asserted to name every declared role and nothing else. Genuinely good and orthogonal to the envelope; it closes the place where a fully-enforced share could go unreported.
  • The --max-old-space-size=3072 observation, re-verified live in this pod (3072 on both tsserver instances). It is now the central argument in the tsserver share's comment.
  • The fileWatcher 107 MB measurement, which is what its 160 MiB share is argued against.

I have not touched #883's branch. Recommend closing it with a pointer here.

Tests

355 passed, 0 failed (332 at the base commit). The suite asserts properties, not one pod's arithmetic:

  • the shares in force add up to the envelope, at 2/4/8/16/64 GiB;
  • and are the same numbers at every pod size (summing correctly is not enough — two roles could trade as the pod changed);
  • no share exceeds an eighth of the smallest supported pod;
  • serverMain is strictly the largest;
  • BUDGET_ORDER names every declared role exactly once;
  • every role is armed under enforce, none under observe, and a role with no share is armed by nothing;
  • the guard-reachability test from before is unchanged.

Thirteen mutations were applied to the production script and every one flips assertions red (counts are failed assertions):

Mutation Flips
change one share (serverMain 512 → 576) 11
reintroduce a memory.max / 8 clamp 6
arm only the old helper set under enforce 8
delete the oversize branch 13
delete the FIT_SEEN assignment 19
drop age >= MIN_AGE from FIT_SEEN 1
move the oversize branch ahead of the in-flight escalation 1
restore the permanent disarm 2
drop a role from BUDGET_ORDER 2
swap the serverMain and extensionHost shares 7
one share above an eighth of a 4 GiB pod (sum still 2048) 8
drop envelope= from event=budget 2
enforce-all falls through to observe 1

The fixture's PSS figures were replaced with the maxima the sweep log actually recorded for each role. One matters more than the rest: the extension host is written at the 738 MB it really reached, above its share, so the fixture reproduces the structural case rather than a convenient one — and that is also what keeps every kill assertion in the suite about exactly one process.

Seam: load_watchdog sets WATCHDOG_STDOUT_PATH and the suite exits if it did not take; verified before any event was generated. No fixture line reached /proc/1/fd/1. kill remains shadowed throughout. Nothing in the live pod was signalled at any point while writing this.

Where a reviewer should look hardest

  1. Is the oversize rule a floor by the back door? It is the single judgement call here. It does not raise any budget and the process stays over budget in every report forever — but it does mean a process that never fits is never killed, and the operator said things that do not fit get killed. My argument is that killing an oversize process cannot make its replacement smaller, so the alternative is a restart every ten minutes of a process the operator is using, which the operator named as the mechanism becoming a nuisance. If that trade is wrong, the fix is one branch in sweep_once.
  2. The apportionment itself, especially languageServer 128 and extensionHelper 128. That is where I argued against the starting split, and where fix: declare what each watchdog budget is anchored on, and stop arming guesses #883's narrow-sample objection bites hardest. A gopls in either class is permanently oversize — reported, not killed.
  3. serverMain at 512 MiB against a 233 MB measured maximum. It follows the operator's rule directly, and it is ~280 MiB that extensionHost does not get. Worth confirming the rule was meant to be this expensive.
  4. The ladder ordering in sweep_once — escalation-in-flight, then oversize, then armed. The drill that found the original ordering bug is preserved as a test, rewritten against the new branch.

Verification

  • ./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh — 355 passed, 0 failed. Stderr noise is byte-identical to the base commit's (one pre-existing cat line, one deliberate unwritable-path line).
  • pre-commit run --all-files — green at the base commit and green here.
  • commitlint's local hook fails with ERR_MODULE_NOT_FOUND for @commitlint/ensure in this pod's Node env, reproducibly and including against already-merged commits. Header and body-line rules checked by hand (no body line > 120 chars, allowed type, no scope).

Budgets were max(min(role budget, memory.max / 8), resting x 1.5): three interacting terms, each patching the
previous one's failure, with no term anywhere that bounded the total. They are now one sentence.

The VS Code tree gets a fixed 2048 MiB envelope. Each role holds a named share of it that does not vary with pod
size, and a process is killed when its PSS has been above its share for ten continuous minutes AND it has been
observed inside that share at some point since it started. claudeHelper (512 MiB) sits outside the envelope: it is
not VS Code, and it is anchored on a measured 1.66 GB adversary rather than on a share of anything.

Shares, summing to 2048 exactly: serverMain 512, extensionHost 480, tsserver 320, treeHelper 320, fileWatcher 160,
languageServer 128, extensionHelper 128. serverMain is the largest by consequence rather than by size - it holds the
remote connection, so a wrong kill costs a disconnect rather than a reload. Every other figure is argued in the
script against the maximum the sweep log recorded for that role over ~38 hours.

Three mechanisms are removed rather than accommodated.

The resting x 1.5 floor. It stopped a share landing below what a healthy process already holds - a real scar - but a
floor can push the sum past the envelope, which is the envelope defeated by arithmetic. What replaces it is a rule
about the process instead of the role: nothing that has never been seen inside its share is signalled. It is
reported as event=oversize, forever, once per dwell period. That is a better guard, because it is measured on the
process actually running rather than read from a constant that goes stale, and it cannot inflate a budget.

The memory.max / 8 divisor. It was always per-role and never a total, and it is meaningless against a fixed
envelope. Its protection survives by construction: the largest share is exactly an eighth of the smallest pod this
template offers, asserted at every pod size rather than trusted.

The circuit breaker's permanent disarm. Under a deliberate cap, repeated kills mean the process does not fit, not
that the number is wrong, and a mechanism that switches itself off after three kills is inert by mid-morning while
arriving to look exactly like the watchdog working. It now emits event=kill-rate ... enforcing=yes. The kill loop it
guarded against cannot form: a killed process's replacement arrives over budget and is oversize.

Every role is armed under enforce, so enforce-all has no superset left to name. It is retired from the parameter and
honoured as enforce in both the Terraform local and normalise_mode, because a workspace carrying that stored value
asked for more enforcement and must not silently get none.

The extension host does not fit today: it arrived at 610 MB PSS at 11s old and settled at 577 MB before doing any
work. No apportionment of 2048 MiB can both keep serverMain the largest share and clear that, so the envelope
presupposes a tree trimmed in dotfiles - and because a template can never depend on dotfiles having been applied,
the untrimmed case degrades to a repeating oversize line while the other six roles go on being enforced.

The envelope is enforced as an apportionment and measured as a total: TREE_PSS is published in the census, the
summary and every sweep's TOTAL row, so the claim is falsifiable from the log. Acting on the total is not attempted.

Tests: 355 passed (332 at base). Thirteen mutations were applied to the production script and every one flips an
assertion red, including reintroducing the divisor, restoring the disarm, deleting either half of the oversize rule,
and moving the oversize branch ahead of the in-flight escalation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197hA8JPwGMX8wufiQiy6oz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant