feat: give the VS Code tree one memory envelope and shares of it - #884
Merged
Conversation
Budgets were max(min(role budget, memory.max / 8), resting x 1.5): three interacting terms, each patching the previous one's failure, with no term anywhere that bounded the total. They are now one sentence. The VS Code tree gets a fixed 2048 MiB envelope. Each role holds a named share of it that does not vary with pod size, and a process is killed when its PSS has been above its share for ten continuous minutes AND it has been observed inside that share at some point since it started. claudeHelper (512 MiB) sits outside the envelope: it is not VS Code, and it is anchored on a measured 1.66 GB adversary rather than on a share of anything. Shares, summing to 2048 exactly: serverMain 512, extensionHost 480, tsserver 320, treeHelper 320, fileWatcher 160, languageServer 128, extensionHelper 128. serverMain is the largest by consequence rather than by size - it holds the remote connection, so a wrong kill costs a disconnect rather than a reload. Every other figure is argued in the script against the maximum the sweep log recorded for that role over ~38 hours. Three mechanisms are removed rather than accommodated. The resting x 1.5 floor. It stopped a share landing below what a healthy process already holds - a real scar - but a floor can push the sum past the envelope, which is the envelope defeated by arithmetic. What replaces it is a rule about the process instead of the role: nothing that has never been seen inside its share is signalled. It is reported as event=oversize, forever, once per dwell period. That is a better guard, because it is measured on the process actually running rather than read from a constant that goes stale, and it cannot inflate a budget. The memory.max / 8 divisor. It was always per-role and never a total, and it is meaningless against a fixed envelope. Its protection survives by construction: the largest share is exactly an eighth of the smallest pod this template offers, asserted at every pod size rather than trusted. The circuit breaker's permanent disarm. Under a deliberate cap, repeated kills mean the process does not fit, not that the number is wrong, and a mechanism that switches itself off after three kills is inert by mid-morning while arriving to look exactly like the watchdog working. It now emits event=kill-rate ... enforcing=yes. The kill loop it guarded against cannot form: a killed process's replacement arrives over budget and is oversize. Every role is armed under enforce, so enforce-all has no superset left to name. It is retired from the parameter and honoured as enforce in both the Terraform local and normalise_mode, because a workspace carrying that stored value asked for more enforcement and must not silently get none. The extension host does not fit today: it arrived at 610 MB PSS at 11s old and settled at 577 MB before doing any work. No apportionment of 2048 MiB can both keep serverMain the largest share and clear that, so the envelope presupposes a tree trimmed in dotfiles - and because a template can never depend on dotfiles having been applied, the untrimmed case degrades to a repeating oversize line while the other six roles go on being enforced. The envelope is enforced as an apportionment and measured as a total: TREE_PSS is published in the census, the summary and every sweep's TOTAL row, so the claim is falsifiable from the log. Acting on the total is not attempted. Tests: 355 passed (332 at base). Thirteen mutations were applied to the production script and every one flips an assertion red, including reintroducing the divisor, restoring the disarm, deleting either half of the oversize rule, and moving the oversize branch ahead of the in-flight escalation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197hA8JPwGMX8wufiQiy6oz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The construction, in one sentence
The VS Code tree gets a fixed 2048 MiB envelope, each role holds a named share of it that does not vary with pod size, and a process is killed when its PSS has been above its share for ten continuous minutes and it has been observed inside that share at some point since it started.
What it replaces is
max(min(role budget, memory.max / 8), resting × 1.5)— three interacting terms, each patching the previous one's failure, and no term anywhere that bounded the total.The apportionment
serverMainextensionHosttsserver--max-old-space-size=3072, VS Code's shipped default, observed live on both instances in this pod.treeHelperfileWatcherlanguageServerextensionHelperterraform-ls)claudeHelperWhere I argued with the starting split
treeHelper256 → 320. 256 is below a member measured at 284 MB on this workspace: it would have been permanently oversize from the first sweep.languageServer192 → 128 andextensionHelper128 → 128;fileWatcher192 → 160. The starting split gave 512 MiB to three roles whose combined measured maximum is 162 MB, and in a zero-sum envelope every one of those MiB is taken fromextensionHostandtreeHelper, which have measured members at 738 and 284. The counter-argument (fix: declare what each watchdog budget is anchored on, and stop arming guesses #883's, and it is a fair one) is thatgopls/rust-analyzerlive in those classes and are far larger. What licenses the tighter number is the oversize rule: a share that turns out to be too small for a legitimate process now produces a repeating log line and no kill, so being wrong costs evidence rather than a restart loop.extensionHost384 → 480,tsserver384 → 320. Same trade, in the direction of the role that actually drifts.The measurements, and what they are worth
All figures are observed, from this watchdog's own
sweep.logover 2026-08-20 07:24 → 2026-08-21 22:14 on the live 8 GiB workspace. It is one operator, one Terraform/YAML/shell repository, and eight distinct processes behind the whole table — a role with 1 762 rows is one process sampled 1 762 times. None of these is a distribution.One number is new and load-bearing: the whole VS Code tree ranged 854 → 1 732 MB PSS over the window, mean 1 353 MB, and its hourly mean climbed 1 130 → 1 731 MB over the last 30 hours — roughly 20 MB/h in the standing population. So the envelope is not binding today, and is binding in about 15 more hours of continuous uptime at the observed rate. That is the drift this loop exists to police, seen as a total for the first time.
The three removed mechanisms — Chesterton, one at a time
1. The
resting × 1.5floor. Protected against: a share landing below what a healthy process already holds, which under enforcement is a kill loop with a supervisor's restart in the middle of it. A real scar — this design shipped it twice, most memorably anRLIMIT_DATAceiling below what an idle file watcher held. Why it cannot stay: a floor and an envelope are incompatible instruments. Under a total, a process whose resting size does not fit means the process must be made smaller; a floor says the budget must grow to meet it, and enough of those push the sum past the envelope. What protects now: the oversize rule — a process that has never been observed inside its share since it started is never signalled, only reported. That is a better guard than the floor: it is measured on the process actually running rather than read from a hand-maintained constant that goes stale, and it bounds nothing upward.RESTING_ROLE,RESTING_FACTOR_*andFLOOREDare deleted; the measurements survive as comments beside the share each argues about.2. The
memory.max / 8divisor. Protected against: a budget reasoned about for the 8 GiB pod letting one helper own a quarter of the 4 GiB one. (Per-role, never a total — #865's body and the code comment both say so.) What protects now: construction. The largest single share, 512 MiB, is exactly an eighth of the smallest pod this template offers, and the suite asserts that property at every pod size rather than trusting the coincidence.3. The circuit breaker's permanent disarm. Protected against: the kill loop — kill the extension host → VS Code restarts it → it reloads every extension → it exceeds again → kill — which arrives looking exactly like the watchdog working. Why it cannot stay: under a deliberate cap, repeated kills mean the process does not fit, which is a decision rather than a defect. A mechanism that switches itself off after three of them is inert by mid-morning — and that is the failure the breaker was built to prevent, now caused by the breaker. What protects now: the loop cannot form. A killed process's replacement arrives over budget, has never been seen fitting, and is
oversize. The counter is kept and reportsevent=kill-rate role=… enforcing=yes, which is a sizing signal for a human while enforcement continues.The structurally-over-budget case
There is a real difference between "this grew past its share" and "this never fitted", and the old logs could not tell them apart. They now can, and the two are treated differently:
event=oversize, once per dwell period, indefinitely, and never signalled. Killing it accomplishes nothing: the replacement arrives over budget too.It is per process (keyed
pid:starttime), evidence-based rather than count-based, it re-arms by itself the moment a replacement is seen fitting, it cannot silence itself, and it cannot widen a budget. The fit evidence is only taken at or past the 300 s age floor, or the classification would be decided by whether a sweep happened to catch the process seconds afterexec.The known gap, stated rather than hidden: a process born far too big and then growing further is recorded and never signalled. That is accepted — a process born at ten times its share is not the drift this loop polices — and the sweep log names it either way.
What happens if the dotfiles trim has not landed
Observed: the live extension host arrived at 610 MB PSS at 11 seconds old and settled at 577 MB before doing any work. Not the 471 MB "fresh" figure the docs carried.
Therefore, and this is arithmetic rather than a preference: no apportionment of 2048 MiB can both keep
serverMainthe largest share and putextensionHost's share above 610 MB. Clearing 610 needs ≥ 640 for the extension host and ≥ 704 for serverMain; that is 1 344 of 2 048 for two roles, leaving 704 for five, of whichtreeHelperalone needs 320.So the envelope presupposes a trimmed tree, and this template can never depend on dotfiles having been applied — the coupling this repo pair was already burned by, which is why
vscode-server-gcis template-owned. If the trim has not landed, that one role degrades to a repeatingevent=oversizeline and the other six go on being enforced. It does not thrash, it does not widen anything, and the line is exactly the evidence that the trim has not landed. The trim needs to bring the extension host's arrival size under 480 MiB — a ~21% reduction from today's 610 MB.tsserveris the same shape from the other end: with--max-old-space-size=3072still in force its 320 MiB share means "report a healthy tsserver on a large project", not "bound it".The envelope is enforced as an apportionment and measured as a total
Comparison stays per process — a role-level total cannot say which sibling is the offender, and does not compose with a dwell clock keyed per
pid:starttimeprecisely so load can be told from drift. So a role with three members can hold three times its share without any single process being over budget. That hole is not fixed and no longer invisible:TREE_PSSis computed every sweep and published in the census astree_pss_mb/envelope_pct, in thesummaryfile, and in every sweep'sTOTALrow, so the envelope is a claim a reviewer can falsify from the log. Acting on the total is deliberately not attempted — that is the graded ladder this design already removed, because a poll loop cannot see the events it was reacting to.Modes
enforcearms every role, soenforce-allhas no superset left to name and a mode that does nothing different is the "indistinguishable from broken" harm. It is retired from the parameter and honoured asenforcein both the Terraform local and the script's newnormalise_mode— a workspace still carrying that stored value asked for more enforcement and must not silently get none. Anything unrecognised still falls back to inertobserve.What I did about #883, and why
#883 is superseded and should be closed. It added
BUDGET_BASIS(measured/adversary/guess) and derived arming from it, disarming guess-anchored roles underenforce. Two of its three load-bearing parts do not survive this change:fileWatcherresting correction (88 → 107 MiB) and itsRESTING_ROLEstaleness reasoning are moot:RESTING_ROLEis deleted.Salvaged here, with attribution in the code:
BUDGET_ORDER— a declared reporting order asserted to name every declared role and nothing else. Genuinely good and orthogonal to the envelope; it closes the place where a fully-enforced share could go unreported.--max-old-space-size=3072observation, re-verified live in this pod (3072on both tsserver instances). It is now the central argument in thetsservershare's comment.fileWatcher107 MB measurement, which is what its 160 MiB share is argued against.I have not touched #883's branch. Recommend closing it with a pointer here.
Tests
355 passed, 0 failed(332 at the base commit). The suite asserts properties, not one pod's arithmetic:serverMainis strictly the largest;BUDGET_ORDERnames every declared role exactly once;enforce, none underobserve, and a role with no share is armed by nothing;Thirteen mutations were applied to the production script and every one flips assertions red (counts are failed assertions):
serverMain512 → 576)memory.max / 8clampenforceFIT_SEENassignmentage >= MIN_AGEfromFIT_SEENBUDGET_ORDERserverMainandextensionHostsharesenvelope=fromevent=budgetenforce-allfalls through toobserveThe fixture's PSS figures were replaced with the maxima the sweep log actually recorded for each role. One matters more than the rest: the extension host is written at the 738 MB it really reached, above its share, so the fixture reproduces the structural case rather than a convenient one — and that is also what keeps every kill assertion in the suite about exactly one process.
Seam:
load_watchdogsetsWATCHDOG_STDOUT_PATHand the suite exits if it did not take; verified before any event was generated. No fixture line reached/proc/1/fd/1.killremains shadowed throughout. Nothing in the live pod was signalled at any point while writing this.Where a reviewer should look hardest
sweep_once.languageServer128 andextensionHelper128. That is where I argued against the starting split, and where fix: declare what each watchdog budget is anchored on, and stop arming guesses #883's narrow-sample objection bites hardest. Agoplsin either class is permanently oversize — reported, not killed.serverMainat 512 MiB against a 233 MB measured maximum. It follows the operator's rule directly, and it is ~280 MiB thatextensionHostdoes not get. Worth confirming the rule was meant to be this expensive.sweep_once— escalation-in-flight, then oversize, then armed. The drill that found the original ordering bug is preserved as a test, rewritten against the new branch.Verification
./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh— 355 passed, 0 failed. Stderr noise is byte-identical to the base commit's (one pre-existingcatline, one deliberate unwritable-path line).pre-commit run --all-files— green at the base commit and green here.commitlint's local hook fails withERR_MODULE_NOT_FOUNDfor@commitlint/ensurein this pod's Node env, reproducibly and including against already-merged commits. Header and body-line rules checked by hand (no body line > 120 chars, allowed type, no scope).