Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions CLAUDE.md

Large diffs are not rendered by default.

16 changes: 12 additions & 4 deletions DESIGN.md

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,13 @@ Because all three stages run for real in dry-run — just scoped away from produ
Everything else here is declarative and is covered by the stages above. `script-memory-watchdog.sh` decides at runtime whether to kill a process, so it gets two things neither lint nor a template push can provide:

- **Fixtures** — `./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh`, also run by the `watchdog` job in `.github/workflows/test.yaml`. The suite is built around negative assertions paired with the mutation that must flip them: "it did not kill the agent session" proves nothing unless removing one rule makes it kill the agent session. Two of its own safety properties matter as much as its assertions: `kill` is shadowed by a function throughout, because the fixture pids name real processes in whatever container runs the suite; and every load of the watchdog redirects its stdout emission to a temporary file, with the suite exiting outright if that seam did not take — the real default is the container's own stdout, and a run without the seam has already put fixture kill lines into a live workspace's log stream.
- **A live drill on a disposable `test_mode` workspace**, which has disposable storage and can be wrecked freely. Fixtures cannot answer whether a real process tree classifies correctly, whether a supervisor really does respawn what was killed, or whether the circuit breaker stops a loop rather than joining it. The drill that has been run: a stand-in session root (`exec -a claude bash`, to exercise the shebang branch of the claude-root guard rather than the trivial compiled-binary one) with an over-budget helper that a supervisor respawns, a second helper inside its budget, and a process detached into its own session with `setsid` — the same mechanism Claude Code uses to detach a Bash tool call — larger than both.
- **A live drill on a disposable `test_mode` workspace**, which has disposable storage and can be wrecked freely. Fixtures cannot answer whether a real process tree classifies correctly, whether a supervisor really does respawn what was killed, or whether the respawned process comes back inside its share (a drift the watchdog should keep policing) or above it (an `oversize` it should report and leave alone). The drill that has been run: a stand-in session root (`exec -a claude bash`, to exercise the shebang branch of the claude-root guard rather than the trivial compiled-binary one) with an over-budget helper that a supervisor respawns, a second helper inside its budget, and a process detached into its own session with `setsid` — the same mechanism Claude Code uses to detach a Bash tool call — larger than both.

Neither replaces the other, and the live one is where every defect that mattered in this component has been found.

### What the most recent drill actually observed

Run 2026-08-21 on a disposable `watchdog-drill` workspace, `enforce` mode, with every threshold compressed and wall-clock arithmetic left real (`WATCHDOG_NOW` was never set): dwell 600→20s, age floor 300→45s, kill grace 30→8s, breaker window 3600→180s, sweep cadence ~60→~3s, `WATCHDOG_BUDGET_claudeHelper` pinned 512→100 MiB. The code paths and the arithmetic are the production ones; production *timings* were never waited out, which is what compressing them costs. The synthetic workload was compiled C — no `python3` or `node` exists on `PATH` in a fresh workspace without dotfiles applied, itself worth knowing.
Run 2026-08-21 on a disposable `watchdog-drill` workspace, `enforce` mode, **against the pre-envelope construction** — per-role budgets clamped by `memory.max / 8` and a resting floor, with a circuit breaker that disarmed a role permanently. Everything below is what was observed then and is left as the record. Two things in it no longer exist: `event=disarmed` (the breaker now reports `event=kill-rate ... enforcing=yes` and disarms nothing), and the compressed `breaker window` is now only the window that report is rated over. The kills themselves would still happen — the hog was respawned inside its pinned budget each time and drifted out of it, which is the drift case, not the `oversize` one. It was run with every threshold compressed and wall-clock arithmetic left real (`WATCHDOG_NOW` was never set): dwell 600→20s, age floor 300→45s, kill grace 30→8s, breaker window 3600→180s, sweep cadence ~60→~3s, `WATCHDOG_BUDGET_claudeHelper` pinned 512→100 MiB. The code paths and the arithmetic are the production ones; production *timings* were never waited out, which is what compressing them costs. The synthetic workload was compiled C — no `python3` or `node` exists on `PATH` in a fresh workspace without dotfiles applied, itself worth knowing.

The daemon's own `actions.log` for the drill, re-fetched from Loki afterward and matched byte-for-byte against the pod's local copy:

Expand All @@ -46,7 +46,7 @@ event=kill sig=KILL pid=<p2> role=claudeHelper id=hog pss_mb=150 budget_mb=100 o
event=disarmed role=claudeHelper reason=kill-loop kills=3 window_s=180 budget_mb=100 detail="a role that has to be killed this often does not have a drift problem, it has a wrong budget; raise WATCHDOG_BUDGET_claudeHelper or accept the size, but this watchdog will not keep restarting it"
```

The two kills are gated by two different rules, not the same one twice: the first fired at `over_s=22` against a 20s dwell — one sweep late, proportional to the ~3s cadence, i.e. dwell-bound. The second's dwell was satisfied around `over_s≈20`, but the kill waited for `age_s` to clear its 45s floor, so `over_s` kept climbing to 43 before it fired — age-bound. A third respawn (not shown above) supplied the kill that tripped the breaker.
The two kills are gated by two different rules, not the same one twice: the first fired at `over_s=22` against a 20s dwell — one sweep late, proportional to the ~3s cadence, i.e. dwell-bound. The second's dwell was satisfied around `over_s≈20`, but the kill waited for `age_s` to clear its 45s floor, so `over_s` kept climbing to 43 before it fired — age-bound. A third respawn (not shown above) supplied the kill that tripped the breaker; under the current construction that same kill produces an `event=kill-rate` line and enforcement continues.

`sweep.log` also confirms a dip resets the dwell clock rather than just capping it, in consecutive rows for the same pid:

Expand All @@ -61,7 +61,7 @@ Not exercised by this drill, and not claimed to be: production timings (everythi

### Checking a claim like this against the record

An `event=kill`/`event=disarmed` line, unlike the sweep log, does leave the pod (see [DESIGN.md](DESIGN.md#design-tensions-and-decisions)), so a claim like the one above is checkable: `{namespace="coder"} |= "component=memory-watchdog" |~ "event=(kill|would-kill|refused|disarmed)"` against Loki finds exactly the six lines quoted above, timestamped and labelled with the `watchdog-drill` pod.
An `event=kill`/`event=oversize` line, unlike the sweep log, does leave the pod (see [DESIGN.md](DESIGN.md#design-tensions-and-decisions)), so a claim like the one above is checkable: `{namespace="coder"} |= "component=memory-watchdog" |~ "event=(kill|would-kill|oversize|refused|kill-rate)"` against Loki finds exactly the six lines quoted above, timestamped and labelled with the `watchdog-drill` pod.

The same query, run over the full retained history, is also what falsifies a different claim: PR [#865](https://github.com/ppat/coder/pull/865) (merged 2026-08-18), which introduced this drill practice, described one run in its body as killing "a drifted 739 MB helper... three times" with the role disarming "on the third kill," and stated "the daemon's own lines from the test workspace are in Loki." Neither `event=kill` nor `event=disarmed` nor the string `739` appears anywhere in the retained record for any workspace at any time before this run. The only kill/disarm lines anywhere near that PR's merge time are on the operator's live workspace at 2026-08-18T01:01–01:08, and the daemon logged its own correction for them at the time: `event=note reason=fixture-leak`, naming fixture pids 41/61 at 1907/1583 MB — the same incident this repo's docs already describe as the fixture suite's stdout seam leaking into production before it was guarded. No `event=kill` or `event=disarmed` line from a workspace named `test` exists in the record at all. `event=budget role=extensionHost budget_mb=1069 pod_share_mb=512 floored=1` — the figures the PR body also cites as drill evidence — is not distinguishing evidence either: every disposable `test_mode` workspace since has printed exactly that line at startup, regardless of whether anything was ever killed, because `floored=1` was a constant at that pod size — the bug PR #881 fixes.

Expand Down
11 changes: 6 additions & 5 deletions templates/kubernetes/homelab-workspace/env.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,16 @@ resource "coder_env" "welcome_message" {
}

# The switch that arms the memory watchdog. "observe" measures and records what
# it would have done; "enforce" kills helpers that have been over budget for ten
# minutes; "enforce-all" adds the two editor roles whose restart the operator can
# see.
# it would have done; "enforce" kills any process that has been over its share of
# the 2048 MiB VS Code envelope for ten minutes and had previously been seen
# inside it.
#
# Set from a mutable workspace parameter rather than hardcoded here, because it
# is the one control that decides whether the watchdog may signal anything, and
# because turning it off has to be a parameter change rather than a code change
# on a bad day. The budgets themselves are derived from the pod's own memory.max,
# so this value does not have to be reconsidered per pod size.
# on a bad day. The shares are constants that add up to the envelope and do not
# vary with pod size, so this value does not have to be reconsidered per pod
# size either.
resource "coder_env" "memory_watchdog_mode" {
agent_id = coder_agent.main.id
name = "WATCHDOG_MODE"
Expand Down
29 changes: 17 additions & 12 deletions templates/kubernetes/homelab-workspace/parameters.tf
Original file line number Diff line number Diff line change
Expand Up @@ -33,24 +33,19 @@ data "coder_parameter" "memory_watchdog_mode" {

default = "enforce"
display_name = "Memory Watchdog"
description = "What the memory watchdog may do about a helper process that has been over its budget for ten minutes"
description = "What the memory watchdog may do about a process that has been over its share of the 2048 MiB VS Code envelope for ten minutes"
icon = "/icon/memory.svg"
mutable = true

option {
name = "Observe only"
value = "observe"
description = "Measure, record every sweep, and log the kill it would have made. Sends no signals"
description = "Measure, record every sweep, and log the action it would have taken. Sends no signals"
}
option {
name = "Enforce (helpers)"
name = "Enforce"
value = "enforce"
description = "Also kill drifted helpers: language servers, the file watcher, native extension helpers, and MCP servers an agent session spawned. Each restarts invisibly"
}
option {
name = "Enforce (helpers and editor)"
value = "enforce-all"
description = "Also kill the VS Code extension host and server. These restart visibly, so they are armed separately"
description = "Also kill drifted processes, in every role. A process that has never fitted its share is reported rather than killed, so nothing is restarted on a loop"
}
}

Expand All @@ -62,9 +57,19 @@ locals {
# So it goes through the same validate-then-use step as the list parameters
# below, and anything unrecognised falls back to the inert mode rather than to
# whatever was supplied.
validated_watchdog_mode = contains(
["observe", "enforce", "enforce-all"], data.coder_parameter.memory_watchdog_mode.value
) ? data.coder_parameter.memory_watchdog_mode.value : "observe"
#
# "enforce-all" is the retired third mode: under a fixed envelope every role is
# armed, so there is no superset left for it to name. A workspace still
# carrying the stored value is mapped to "enforce" rather than dropped to
# "observe", because it had asked for more enforcement and must not silently
# get none.
validated_watchdog_mode = (
data.coder_parameter.memory_watchdog_mode.value == "enforce-all" ? "enforce" : (
contains(
["observe", "enforce"], data.coder_parameter.memory_watchdog_mode.value
) ? data.coder_parameter.memory_watchdog_mode.value : "observe"
)
)

validated_preferred_nodes = (data.coder_parameter.preferred_nodes.value != "") ? [
for str in jsondecode(data.coder_parameter.preferred_nodes.value) :
Expand Down
Loading
Loading