fix: govern the in-tree helpers the watchdog could not name - #881
Merged
Conversation
role_of returned `other` for three real VS Code helpers on the live tree, and
compute_policed skips `other`, so they had no budget, no dwell clock, no signal
and no warning: a tamasfe.even-better-toml server.js at 280 MB PSS (already
above the 256 MiB languageServer budget a matching pattern would have given
it), tsserver's typingsInstaller.js at 61 MB, and the built-in
markdown-language-features server at 54 MB.
Widening the pattern list is the easy fix and the wrong one - launch shapes are
each extension's own choice, so the list is a permanent race against a
vendor-controlled vocabulary that fails silently, one extension at a time. The
default inside the tree is inverted instead: on VS Code's own runtime,
unprotected and unnamed becomes role `treeHelper`. Two things buy off the risk
that creates rather than assuming it away:
- treeHelper is in a new UNKNOWN_ROLES tier, armed only under enforce-all,
because its 512 MiB budget is a reasoned guess and no member has been
measured at rest. Under enforce - the template default - a coverage gap
arrives as `event=would-kill armed=no` in the container log, not as
silence. The axis is calibration, not blast radius.
- a direct child of a server root is excluded structurally. Core forks
(fileWatcher, extensionHost, ptyHost) are exactly the direct children of
server-main.js and no extension helper is, so a release that renames
--type=extensionHost yields a process that is unmanaged and loud rather
than one carrying a guessed 512 MiB budget against the 685 MB it holds.
Sweep-log identity for the role is <extension>/<script>, because a dozen
extensions all ship a file called server.js and `server` answers nothing.
Four smaller corrections, each with the assertion that pins it:
- extensionHost's declared budget was 1024 MiB against its own 1069 MiB
resting floor, so `floored=1` was true at 4, 8, 16 and 64 GiB alike -
a property of the constants reported as a property of the pod. The
declared number is now the one always in force: no effective budget
changes at any pod size, and the flag now flips off at 16 GiB. The suite
asserts the invariant (no declared budget below its own floor) rather
than the flag on one pod.
- disarming a role now finishes an escalation already in flight and starts
no new one. A drill found the opposite by accident: the breaker tripped
between SIGTERM and SIGKILL and the process was left alive, over budget
and half-signalled, recorded as neither killed nor spared. record_kill is
not called again - it is the same kill - and the line says after_disarm.
- the "nothing policed" warning is per episode of blindness rather than a
one-shot latch over a lifetime high-water mark, so a blind spot arriving
on day two of a pod's life is no longer silent.
- `2>/dev/null` now precedes the input redirect it silences on every
per-process /proc read. The old order opened the file first, so a process
exiting mid-sweep produced 160 of the 161 lines in boot.log. Harmless,
but it is noise that would hide a real error. memory.stat/memory.max are
deliberately left alone: a missing cgroup file is worth seeing.
Not done, deliberately: budgets stay per-process rather than per-role-aggregate.
The only multi-process role here is tsserver (two, at 112 + 141 MB against
768 MiB each); a role total makes "which sibling dies" arbitrary; and it does
not compose with a dwell clock keyed per pid:starttime, which is what separates
load from drift. The cheap half is done instead - role_pss_mb and role_peak_mb
in the hourly census - so the evasion, and a stale RESTING_ROLE, become visible
as data. RESTING_ROLE stays hand-measured: a watchdog that re-derived its floor
from observation would raise the limit it enforces exactly when the thing it
enforces against grew.
Every new assertion is paired with the mutation that flips it red, and all
eight were run: removing the treeHelper branch (10 fail), the direct-child
guard (3), the argv[0] condition (5), arming treeHelper under enforce (3),
reverting the declared budget (2), restoring the old disarm ordering (2), the
one-shot visibility latch (2), and the identity breadcrumb (3).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197hA8JPwGMX8wufiQiy6oz
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The watchdog's policed set now matches the set it believes it polices. Five
defects between belief and behaviour are closed; one is deliberately left open
with a reason.
1. In-tree helpers nothing could name (the substantive change)
role_of()returnedotherfor three genuine VS Code helpers on the live tree,and
compute_policed()skipsROLE==other— so they had no budget, no dwelltracking, no signal, and no warning:
.../extensions/tamasfe.even-better-toml-*/dist/server.js.../extensions/node_modules/typescript/lib/typingsInstaller.js.../extensions/markdown-language-features/dist/serverWorkerMainThe first is a language server already above the 256 MiB
languageServerbudget it would have received had any pattern matched it. None of
*yaml-language-server*,*jsonServerMain*,*-language-server*,*languageserver*ortsserver.jsreaches any of these launch shapes.Widening the pattern list was rejected. The set of shapes is the union of
every extension's own choices, so a pattern list is a permanent race against a
vendor-controlled vocabulary — and it fails silently, one extension at a time,
which is the single property this mechanism cannot afford.
Instead the default inside the tree is inverted: unknown means governed, not
invisible. In the server tree, on VS Code's own runtime, unprotected, not in
the ptyHost subtree and unnamed ⇒ role
treeHelper.Inverting is not free — a guessed budget applied to a process whose legitimate
resting size exceeds it kills something healthy, and this design has shipped
exactly that failure twice. So the risk is bought off rather than assumed away:
treeHelperis in a newUNKNOWN_ROLEStier, armed only underenforce-all. The axis is calibration, not blast radius: its 512 MiBbudget is a reasoned guess and no member of the role has been measured at
rest. Under
enforce(the template default) it is measured, budgeted,dwell-tracked and reported — a coverage gap arrives as
event=would-kill armed=no role=treeHelperin the container log instead of assilence.
direct children of
server-main.jsare exactlyfileWatcher,extensionHost,ptyHost; every language server, typings installer andextension binary hangs off the extension host or off another helper. So a VS
Code release that renames
--type=extensionHostproduces a process that isunmanaged and loud, never one carrying a guessed 512 MiB budget against the
685 MB it actually holds.
Sweep-log identity for the role is
<extension>/<script>(
tamasfe.even-better-toml-0.21.2/server,typescript/typingsInstaller),because a dozen extensions all ship a
server.jsandserveranswers nothingin a post-mortem.
2.
floored=1was a constant, not a diagnosticfloored=1is documented to mean "this pod is too small to bound this role atits intended share". That reading only holds while every declared budget
already clears its own resting floor — and
extensionHostwas declared at1024 MiB against a
713 MB × 1.5 = 1069 MiBfloor, so the floor wonunconditionally and the flag was set at 4, 8, 16 and 64 GiB alike. It
reported a property of the constants while claiming to report a property of the
pod.
The declared number is now
1121452032— the value that was already inforce. Effective budgets are bit-identical at every pod size; only the
diagnostic changes, and it now flips off at 16 GiB. The suite asserts the
invariant (no declared budget below its own floor, for any role) rather than
the flag's value on one pod size.
3. The breaker abandoned an in-flight escalation
Observed in a drill: once
DISARMED[role]was set, the disarm branchshort-circuited ahead of the
KILLED_ATescalation check, so a process thathad already received
SIGTERMnever receivedSIGKILLand was still alive andover budget at drill end. Undocumented either way.
Decided: disarming means "stop deciding to kill this role", not "abandon a
kill already decided." Finishing one escalation cannot start the loop the
breaker exists to stop, and a process left half-signalled is the worst of both —
a helper that caught
SIGTERMand began shutting down is not something to walkaway from, and one that ignored it goes on holding the memory with the watchdog
having decided not to look again. The escalation check now precedes the disarm
check,
record_killis deliberately not called again (it is the same kill,already counted), and the line carries
after_disarm=yes. A disarmed role stillbegins nothing new — both halves are asserted.
4. Two smaller defects
VISIBILITY_WARNEDwas a one-shot latch set at the warmup sweep whether ornot it warned, testing a lifetime high-water mark — so a watchdog that policed
something once in its first half hour could never warn again however blind it
went. A pod lives for days; the interesting blind spot is the one that arrives
on day two when an upgrade changes a launch shape. It is now per episode of
blindness, re-arming when anything is policed again.
read -r line <"$entry/stat" 2>/dev/null || continueopens the file before2>/dev/nullapplies, so a process exiting between the/proclisting and theread makes bash print its own error to inherited stderr — 160 of the 161 lines
in
boot.log.|| continuehandles it correctly, so this is pure noise thatwould hide a real error. Every per-process
/procread now orders itcorrectly.
memory.stat/memory.maxare deliberately left alone: a missingcgroup file is a genuine error worth seeing.
5. Not done, on purpose: per-role aggregation
Budgets stay per-process. Confirmed by reading
sweep_once()that nothing sums arole, so N same-role processes can each sit just under budget indefinitely.
Not fixed:
is
tsserver— two of them, 112 + 141 MB against a 768 MiB budget each.The sum is a third of one budget.
third, none is the offender and killing the largest is an arbitrary choice
dressed as a decision.
pid:starttime, which isprecisely what separates load from drift.
The cheap half is done instead:
role_pss_mbin the hourly census, so theevasion becomes visible as data the moment it starts, and a future decision to
act on it begins from a measurement rather than from this argument.
6.
RESTING_ROLEstaleness — reported, never derivedrole_peak_mb(census) andpeak_pss(summary) publish each role's peak, whichis the review trigger for a hand-measured constant that has no other way to say
it has gone stale. It is deliberately not fed back into the budgets: a
watchdog that re-derived its own floor from what it observed would raise the
limit it is enforcing exactly when the thing it enforces against grew — the move
the circuit breaker already refuses, one level removed.
Note the cited drift did not reproduce: the live extension host read 685 MB
PSS at 27.6 h uptime, i.e. below the recorded 713 MB reference. The growth is
not monotonic — V8 hands memory back — so the reference is not stale today.
Evidence
All figures above are measured on the live workspace pod (read-only
/procandps), not argued. Fixtures for the three previously-unclassified processes aretranscribed from captured
ps -eo pid,ppid,sid,comm,argsoutput plus theirsmaps_rollupPSS, including thecomm=MainThreadthat a hand-written fixturegets wrong.
Falsification
This suite's failure mode is not "test fails wrongly", it is "test passes
vacuously" — it has twice been green while asserting nothing. Every new
assertion was paired with the mutation that must flip it red, and all eight were
run against the production script:
treeHelperbranch removed (the previous classifier)argv[0]condition removed from the branchtreeHelperadded toHELPER_ROLES(armed underenforce)extensionHostbudget reverted to 1024 MiBOne existing test had to be repaired rather than merely extended: mutation 1
of
test_operator_runtime_is_never_a_helperstubbedis_vscode_binarytoreturn 0, which after this change falsifies the position rule and the rolerule at once and therefore proves neither. It now clears the two pids from
NOT_EDITORinstead. A mutation has to remove one rule or it is not evidenceabout which rule was carrying the weight.
Gotchas for a future reader
treeHelperis not armed by the mode the template runs. If you expect itto kill something under
enforce, it will not, and that is the design. Armingit means
enforce-all, which also arms the two editor roles.classification aid. Removing it looks harmless and is the single change that
would let a renamed core fork be killed on a guessed budget.
role_ofnow consultsis_vscode_binary. Any test that stubs thatpredicate globally is mutating two rules, not one.
that is a bug, not an intended side effect.
2>/dev/nullbefore the redirect it silences — not after.release_singletonalready carried a comment showing this was understood; the per-process reads
did not follow it.
Verification
./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh—332 passed, 0 failed (276 before).
pre-commit run --all-files— green. Baseline at the base commit was checkedfirst and was also green; the historically-red
end-of-file-fixerandhadolint SC3037items are already fixed onmain.executed in the pod, and the suite's
WATCHDOG_STDOUT_PATHseam keeps fixtureaction lines out of container stdout.
Docs corrected in the same change
CLAUDE.mdandDESIGN.mdboth asserted thefloored=1diagnostic that thisPR shows was a constant. Both are corrected, and both gain the reasoning for the
inverted in-tree default, the aggregation not-doing, the escalation decision and
the redirect-order rule.