Skip to content

fix: govern the in-tree helpers the watchdog could not name - #881

Merged
ppat merged 1 commit into
mainfrom
watchdog-effectiveness
Aug 21, 2026
Merged

fix: govern the in-tree helpers the watchdog could not name#881
ppat merged 1 commit into
mainfrom
watchdog-effectiveness

Conversation

@ppat

@ppat ppat commented Aug 21, 2026

Copy link
Copy Markdown
Owner

What this changes

The watchdog's policed set now matches the set it believes it polices. Five
defects between belief and behaviour are closed; one is deliberately left open
with a reason.

1. In-tree helpers nothing could name (the substantive change)

role_of() returned other for three genuine VS Code helpers on the live tree,
and compute_policed() skips ROLE==other — so they had no budget, no dwell
tracking, no signal, and no warning:

process PSS measured
.../extensions/tamasfe.even-better-toml-*/dist/server.js 280 MiB
.../extensions/node_modules/typescript/lib/typingsInstaller.js 61 MiB
.../extensions/markdown-language-features/dist/serverWorkerMain 54 MiB

The first is a language server already above the 256 MiB languageServer
budget
it would have received had any pattern matched it. None of
*yaml-language-server*, *jsonServerMain*, *-language-server*,
*languageserver* or tsserver.js reaches any of these launch shapes.

Widening the pattern list was rejected. The set of shapes is the union of
every extension's own choices, so a pattern list is a permanent race against a
vendor-controlled vocabulary — and it fails silently, one extension at a time,
which is the single property this mechanism cannot afford.

Instead the default inside the tree is inverted: unknown means governed, not
invisible.
In the server tree, on VS Code's own runtime, unprotected, not in
the ptyHost subtree and unnamed ⇒ role treeHelper.

Inverting is not free — a guessed budget applied to a process whose legitimate
resting size exceeds it kills something healthy, and this design has shipped
exactly that failure twice. So the risk is bought off rather than assumed away:

  • treeHelper is in a new UNKNOWN_ROLES tier, armed only under
    enforce-all.
    The axis is calibration, not blast radius: its 512 MiB
    budget is a reasoned guess and no member of the role has been measured at
    rest. Under enforce (the template default) it is measured, budgeted,
    dwell-tracked and reported — a coverage gap arrives as
    event=would-kill armed=no role=treeHelper in the container log instead of as
    silence.
  • A direct child of a server root is excluded structurally. Measured: the
    direct children of server-main.js are exactly fileWatcher,
    extensionHost, ptyHost; every language server, typings installer and
    extension binary hangs off the extension host or off another helper. So a VS
    Code release that renames --type=extensionHost produces a process that is
    unmanaged and loud, never one carrying a guessed 512 MiB budget against the
    685 MB it actually holds.

Sweep-log identity for the role is <extension>/<script>
(tamasfe.even-better-toml-0.21.2/server, typescript/typingsInstaller),
because a dozen extensions all ship a server.js and server answers nothing
in a post-mortem.

2. floored=1 was a constant, not a diagnostic

floored=1 is documented to mean "this pod is too small to bound this role at
its intended share". That reading only holds while every declared budget
already clears its own resting floor — and extensionHost was declared at
1024 MiB against a 713 MB × 1.5 = 1069 MiB floor, so the floor won
unconditionally and the flag was set at 4, 8, 16 and 64 GiB alike. It
reported a property of the constants while claiming to report a property of the
pod.

The declared number is now 1121452032the value that was already in
force
. Effective budgets are bit-identical at every pod size; only the
diagnostic changes, and it now flips off at 16 GiB. The suite asserts the
invariant (no declared budget below its own floor, for any role) rather than
the flag's value on one pod size.

3. The breaker abandoned an in-flight escalation

Observed in a drill: once DISARMED[role] was set, the disarm branch
short-circuited ahead of the KILLED_AT escalation check, so a process that
had already received SIGTERM never received SIGKILL and was still alive and
over budget at drill end. Undocumented either way.

Decided: disarming means "stop deciding to kill this role", not "abandon a
kill already decided."
Finishing one escalation cannot start the loop the
breaker exists to stop, and a process left half-signalled is the worst of both —
a helper that caught SIGTERM and began shutting down is not something to walk
away from, and one that ignored it goes on holding the memory with the watchdog
having decided not to look again. The escalation check now precedes the disarm
check, record_kill is deliberately not called again (it is the same kill,
already counted), and the line carries after_disarm=yes. A disarmed role still
begins nothing new — both halves are asserted.

4. Two smaller defects

  • VISIBILITY_WARNED was a one-shot latch set at the warmup sweep whether or
    not it warned, testing a lifetime high-water mark — so a watchdog that policed
    something once in its first half hour could never warn again however blind it
    went. A pod lives for days; the interesting blind spot is the one that arrives
    on day two when an upgrade changes a launch shape. It is now per episode of
    blindness, re-arming when anything is policed again.
  • read -r line <"$entry/stat" 2>/dev/null || continue opens the file before
    2>/dev/null applies, so a process exiting between the /proc listing and the
    read makes bash print its own error to inherited stderr — 160 of the 161 lines
    in boot.log. || continue handles it correctly, so this is pure noise that
    would hide a real error. Every per-process /proc read now orders it
    correctly. memory.stat/memory.max are deliberately left alone: a missing
    cgroup file is a genuine error worth seeing.

5. Not done, on purpose: per-role aggregation

Budgets stay per-process. Confirmed by reading sweep_once() that nothing sums a
role, so N same-role processes can each sit just under budget indefinitely.
Not fixed:

  • The evidence says the case is not occurring. The only multi-process role here
    is tsserver — two of them, 112 + 141 MB against a 768 MiB budget each.
    The sum is a third of one budget.
  • A role total changes what a kill means: if three siblings each contributed a
    third, none is the offender and killing the largest is an arbitrary choice
    dressed as a decision.
  • It does not compose with a dwell clock keyed per pid:starttime, which is
    precisely what separates load from drift.

The cheap half is done instead: role_pss_mb in the hourly census, so the
evasion becomes visible as data the moment it starts, and a future decision to
act on it begins from a measurement rather than from this argument.

6. RESTING_ROLE staleness — reported, never derived

role_peak_mb (census) and peak_pss (summary) publish each role's peak, which
is the review trigger for a hand-measured constant that has no other way to say
it has gone stale. It is deliberately not fed back into the budgets: a
watchdog that re-derived its own floor from what it observed would raise the
limit it is enforcing exactly when the thing it enforces against grew — the move
the circuit breaker already refuses, one level removed.

Note the cited drift did not reproduce: the live extension host read 685 MB
PSS at 27.6 h uptime
, i.e. below the recorded 713 MB reference. The growth is
not monotonic — V8 hands memory back — so the reference is not stale today.

Evidence

All figures above are measured on the live workspace pod (read-only /proc and
ps), not argued. Fixtures for the three previously-unclassified processes are
transcribed from captured ps -eo pid,ppid,sid,comm,args output plus their
smaps_rollup PSS, including the comm=MainThread that a hand-written fixture
gets wrong.

Falsification

This suite's failure mode is not "test fails wrongly", it is "test passes
vacuously" — it has twice been green while asserting nothing. Every new
assertion was paired with the mutation that must flip it red, and all eight were
run against the production script:

mutation assertions flipped
treeHelper branch removed (the previous classifier) 10
direct-child-of-root guard removed 3
argv[0] condition removed from the branch 5
treeHelper added to HELPER_ROLES (armed under enforce) 3
declared extensionHost budget reverted to 1024 MiB 2
disarm branch restored ahead of the escalation 2
visibility latch restored to one-shot 2
identity breadcrumb loses the extension name 3

One existing test had to be repaired rather than merely extended: mutation 1
of test_operator_runtime_is_never_a_helper stubbed is_vscode_binary to
return 0, which after this change falsifies the position rule and the role
rule at once and therefore proves neither. It now clears the two pids from
NOT_EDITOR instead. A mutation has to remove one rule or it is not evidence
about which rule was carrying the weight.

Gotchas for a future reader

  • treeHelper is not armed by the mode the template runs. If you expect it
    to kill something under enforce, it will not, and that is the design. Arming
    it means enforce-all, which also arms the two editor roles.
  • The direct-child-of-a-server-root exclusion is a safety guard, not a
    classification aid.
    Removing it looks harmless and is the single change that
    would let a renamed core fork be killed on a guessed budget.
  • role_of now consults is_vscode_binary. Any test that stubs that
    predicate globally is mutating two rules, not one.
  • No effective budget changed in this PR. If a review finds one that did,
    that is a bug, not an intended side effect.
  • 2>/dev/null before the redirect it silences — not after. release_singleton
    already carried a comment showing this was understood; the per-process reads
    did not follow it.

Verification

  • ./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh
    332 passed, 0 failed (276 before).
  • pre-commit run --all-files — green. Baseline at the base commit was checked
    first and was also green; the historically-red end-of-file-fixer and
    hadolint SC3037 items are already fixed on main.
  • Nothing was run against live processes: the watchdog daemon itself was never
    executed in the pod, and the suite's WATCHDOG_STDOUT_PATH seam keeps fixture
    action lines out of container stdout.

Docs corrected in the same change

CLAUDE.md and DESIGN.md both asserted the floored=1 diagnostic that this
PR shows was a constant. Both are corrected, and both gain the reasoning for the
inverted in-tree default, the aggregation not-doing, the escalation decision and
the redirect-order rule.

role_of returned `other` for three real VS Code helpers on the live tree, and
compute_policed skips `other`, so they had no budget, no dwell clock, no signal
and no warning: a tamasfe.even-better-toml server.js at 280 MB PSS (already
above the 256 MiB languageServer budget a matching pattern would have given
it), tsserver's typingsInstaller.js at 61 MB, and the built-in
markdown-language-features server at 54 MB.

Widening the pattern list is the easy fix and the wrong one - launch shapes are
each extension's own choice, so the list is a permanent race against a
vendor-controlled vocabulary that fails silently, one extension at a time. The
default inside the tree is inverted instead: on VS Code's own runtime,
unprotected and unnamed becomes role `treeHelper`. Two things buy off the risk
that creates rather than assuming it away:

  - treeHelper is in a new UNKNOWN_ROLES tier, armed only under enforce-all,
    because its 512 MiB budget is a reasoned guess and no member has been
    measured at rest. Under enforce - the template default - a coverage gap
    arrives as `event=would-kill armed=no` in the container log, not as
    silence. The axis is calibration, not blast radius.
  - a direct child of a server root is excluded structurally. Core forks
    (fileWatcher, extensionHost, ptyHost) are exactly the direct children of
    server-main.js and no extension helper is, so a release that renames
    --type=extensionHost yields a process that is unmanaged and loud rather
    than one carrying a guessed 512 MiB budget against the 685 MB it holds.

Sweep-log identity for the role is <extension>/<script>, because a dozen
extensions all ship a file called server.js and `server` answers nothing.

Four smaller corrections, each with the assertion that pins it:

  - extensionHost's declared budget was 1024 MiB against its own 1069 MiB
    resting floor, so `floored=1` was true at 4, 8, 16 and 64 GiB alike -
    a property of the constants reported as a property of the pod. The
    declared number is now the one always in force: no effective budget
    changes at any pod size, and the flag now flips off at 16 GiB. The suite
    asserts the invariant (no declared budget below its own floor) rather
    than the flag on one pod.
  - disarming a role now finishes an escalation already in flight and starts
    no new one. A drill found the opposite by accident: the breaker tripped
    between SIGTERM and SIGKILL and the process was left alive, over budget
    and half-signalled, recorded as neither killed nor spared. record_kill is
    not called again - it is the same kill - and the line says after_disarm.
  - the "nothing policed" warning is per episode of blindness rather than a
    one-shot latch over a lifetime high-water mark, so a blind spot arriving
    on day two of a pod's life is no longer silent.
  - `2>/dev/null` now precedes the input redirect it silences on every
    per-process /proc read. The old order opened the file first, so a process
    exiting mid-sweep produced 160 of the 161 lines in boot.log. Harmless,
    but it is noise that would hide a real error. memory.stat/memory.max are
    deliberately left alone: a missing cgroup file is worth seeing.

Not done, deliberately: budgets stay per-process rather than per-role-aggregate.
The only multi-process role here is tsserver (two, at 112 + 141 MB against
768 MiB each); a role total makes "which sibling dies" arbitrary; and it does
not compose with a dwell clock keyed per pid:starttime, which is what separates
load from drift. The cheap half is done instead - role_pss_mb and role_peak_mb
in the hourly census - so the evasion, and a stale RESTING_ROLE, become visible
as data. RESTING_ROLE stays hand-measured: a watchdog that re-derived its floor
from observation would raise the limit it enforces exactly when the thing it
enforces against grew.

Every new assertion is paired with the mutation that flips it red, and all
eight were run: removing the treeHelper branch (10 fail), the direct-child
guard (3), the argv[0] condition (5), arming treeHelper under enforce (3),
reverting the declared budget (2), restoring the old disarm ordering (2), the
one-shot visibility latch (2), and the identity breadcrumb (3).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197hA8JPwGMX8wufiQiy6oz
@ppat
ppat merged commit d349321 into main Aug 21, 2026
16 checks passed
@ppat
ppat deleted the watchdog-effectiveness branch August 21, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant