Skip to content

heavy: count a reservation's work once; machine tier without a user manager; ~/.local checks (#771, #844, #923) - #945

Merged
scuttletoby merged 5 commits into
mainfrom
fix/heavy-771-844-923
Nov 22, 2026
Merged

scuttletoby merged 5 commits into
mainfrom
fix/heavy-771-844-923

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

What and why

Three heavy kit fixes in one PR, plus two review rounds (see "Review round" and "Review round 2" below).

#923, a reservation was counted twice. When an agent or a job worked inside a reserved folder, its learned debt was counted as owed work and the reservation still held its full size, so free dropped twice for the same work. Now each reservation gets an owedMb, from the debt of the pieces whose processes (from cgroup.procs) have their cwd inside it. The rule gets useMb + owedMb for that reservation, so the reservation only holds what is not already counted. The TypeScript and Python capacity rule is unchanged. t3-capacity prints reserved= from the new number and shows holding without the owed part, plus owed N when there is one.

#844, no systemd user manager meant always NOT OK. With cgroup v2 and user.slice but no manager for this user (for example a login without lingering, a container, or a cron job), facts fell to the basic tier, which is never free. A new machine tier answers like full from machine-wide numbers. It counts every user's work and every person's reservations, and leaves out this user's tickets and agent scopes, because it cannot see them. The reason says machine-wide numbers: no user manager. With no /run/user/<uid>, the answer cache uses a private heavy-facts-<uid> folder (mode 0700) under $TMPDIR or /tmp. When that folder is a link, a file, or a folder others can open, or no temp folder can be written, there is no cache and a note says why. The level then starts at "elevated" on every call. The comment on #800 that repeats #844 is covered here. The rest of #800 is left alone.

#771, minors from the #770 reviews.

  1. heavy and the sweep's note writing and pruning refuse a ~/.local that others can write. Before, only owner and type were checked.
  2. The README repair line now also chowns ~/.local itself and runs chmod go-w ~/.local.
  3. Decision: a ~/.local that is a link stays fail-closed (no second notes folder). This is security code, and an existing test pins the fail-closed behaviour. The README already documents it.
  • Nit: the sweep copy of the folder rule now matches bin/heavy.
  • Nit: HOME is resolved with realpath before it is checked, so a linked HOME cannot be swapped after the check.
  • Nit: the untimed-call hygiene check also catches with Popen(...) and from os import waitpid forms, with cases that prove it.

Review round (code lens and state lens, both NOT SAFE on de3dc1c)

  1. MAJOR, a debt counted zero times. A piece with processes in two reserved folders, or in nested ones (/x and /x/y), had its whole debt taken off each reservation, so the debt was counted zero times. Now:

    • each process counts toward one reservation only: the deepest folder of its user's that holds it, then the first by file name;
    • a piece's debt is split over its processes by their use (equally when they use nothing);
    • so for every reservation, held + owed + used is exactly what is reserved.

    New tests: test_two_folders_nested_folders_and_work_across_them_count_each_debt_once checks that sum for two folders, a nested folder, and one agent and one job spanning both. test_never_looser_than_main_beyond_counting_each_debt_once works out main's answer for the same facts. The answer here may be looser than main's only by the debts of the pieces working in reserved folders, each counted once. That is heavy: reservation counted twice for agents under reserve.d #923's point: main counted those twice. With no piece working in a reserved folder, the answer is never looser than main's. On de3dc1c this test fails: free was 7565 MB where the bound was 6829 MB.

  2. MAJOR, the machine tier did not see the owner's reservations. A caller with another home (a CI runner, root) read only its own reserve.d. Now the machine tier reads every person's reserve.d. "Every person" means the users in /etc/passwd and the user.slice users from UID_MIN to UID_MAX; root and system users are not read. Another user's files are opened without following links and must be plain files, so a FIFO cannot stop the read, and at most 64 KiB is read. Fail closed: a folder or file it cannot read (a home of mode 750 or 700), or a user with a session but no home it can find, makes the tier Basic, which is never free, with a note. Tested with two users' reserve folders (test_the_machine_tier_sees_every_person_s_reservations), covering a FIFO, a closed folder and a session with no home.

  3. minor, ~/.local of mode 775 refused silently. heavy now says when it refuses the second notes folder, and the sweep adds a note when it leaves old notes. Both messages name chmod go-w <folder>. The README repair line adds chmod go-w ~/.local.

  4. minor, a cache of another version skipped the hold-down. A cache file that exists but cannot be used now starts "elevated", the same as having no cache folder. That covers another version, other settings, a broken file, a file from the future, and one that is not this user's. Only a missing file, or one older than the carry window, starts with nothing carried. Tested with an old-version cache, an other-settings cache and a broken cache. (Round 2 below gives each version and settings its own file, so callers no longer share one.)

  5. minor, cache in the working folder. tempfile.gettempdir() fell back to the working folder. Now only $TMPDIR, then /tmp, is used, and it must be an absolute folder this user can write. Otherwise there is no cache and a note says so. Tested.

  6. Follow-up only, no code: heavy facts: machine tier misses the caller's own cgroup cap; cache folder squat and TMPDIR nits #951. It covers four things:

    • the machine tier reads only machine-wide numbers, not the caller's own cgroup cap;
    • another user can take the /tmp/heavy-facts-<uid> name first (a squat), which makes every call run without a cache;
    • a check-then-use of the cache folder in a non-sticky $TMPDIR;
    • a resolved home inside a non-sticky folder that others can write.

Review round 2 (code lens and state lens, both SAFE on 90ad533, with minors)

Fixed in 4674bef:

  1. "Elevated" stall. Two callers of one user with different settings (HEAVY_SLICE, HEAVY_QUEUE_ROOT, T3_CAPACITY_RESERVE_DIR), or an old and a new kit, shared one cache file. Each read the other's file as unusable, so every call started "elevated" and t3-capacity said NOT OK on a calm machine for as long as they took turns. Now:

    • the cache file is heavy-facts-v<version>-<16 hex of the settings' hash>.json, one per version and settings, so no caller resets another and each one's own hold-down steps down;
    • the safety stays: an own cache file that is there but unusable starts "elevated", and a first call with no file of its own, beside another cache file of this user's written in the last 5 minutes (another version after an update, other settings), starts "elevated" too;
    • cache files of other settings or versions older than the 5-minute carry window are removed on the next write, so they do not pile up;
    • $HEAVY_FACTS_CACHE still names one file, as before.

    New test test_callers_with_other_settings_taking_turns_each_reach_ok: callers A and B (another reservations folder) take turns every 20 s on a calm machine. B's first call is "elevated", and every call from the seventh on is OK. On 90ad533 it fails: every call after the first stays (75, 'elevated'). RedCheck takes out the per-settings name, the "recent other file" check and the stale-file removal one at a time and sees the test fail.

  2. Repair message. When the refused ~/.local belongs to another user (a root-owned one from an older heavy run as root), bin/heavy now says sudo chown "$USER:" <folder>; chmod go-w only when it is ours but others can write it; a link or file there: replace it with a real folder. The sweep's note uses the same rule. Tested in heavy.test.py with a folder another user owns.

  3. README, machine tier limits named plainly: it reads only each person's default reserve.d (not their own T3_CAPACITY_RESERVE_DIR), only /etc/passwd users with a UID in UID_MIN..UID_MAX or a user.slice session (not root or system users, not NSS-only users without a session, not homes that are / or relative); .conf files must be plain files (a link makes the tier Basic); a non-root caller cannot see other people's working folders, so their reservations hold their whole size (stricter, never looser).

  4. heavy facts: machine tier misses the caller's own cgroup cap; cache folder squat and TMPDIR nits #951 got a comment with the two leftovers: NSS users without a session, and the listing of every passwd home on each machine-tier read (autofs).

Tests: each fix fails without it

The new tests were run against unchanged copies of the kit.

Against origin/main (first round):

  • facts.test.py: 9 failures, 2 errors.
  • sweep.test.py: 2 failures.
  • heavy.test.py: 1 failure.

Against de3dc1c (this round), with the new tests:

  • facts.test.py: 18 failures and 1 error. They come from the two-folder and nested-folder test, the never-looser-than-main test, the every-person reservations test, the old-version cache test, the no-temp-folder test, test_two_callers, the settings-change step of the Holds test, and the in-file RedCheck. RedCheck takes out each of the 7 new fix lines one at a time and checks that its test fails.
  • sweep.test.py: 1 failure (no chmod go-w note).
  • heavy.test.py: 1 failure (no chmod go-w message).

On this branch (4674bef, main merged in), every heavy kit test passes:

  • facts.test.py: 69 OK.
  • heavy.test.py: 68 OK, run twice (gated, unwrapped on purpose: inside heavy 55 of its tests skip; every command it runs gets its own heavy). The once-flaky test_leftover_that_ignores_term_is_killed_after_the_grace_time passed both times.
  • sweep.test.py: 84 OK, 1 skip.
  • capacity_rule, queue, install and measure tests: OK.

Other checks: the seam check (--config-ref origin/main origin/main HEAD) exits 0, and vp fmt --check passes on the README.

State question

  • Runs again: facts are read fresh each call. owedMb, use and other users' reservations are recomputed from live cgroup.procs, /proc and each reserve.d, so a second run gives the same answer for the same machine. The cache folder is created once (mkdir with mode 0700) and checked each time, and a folder that already exists and is safe is reused.
  • Stops half-way: nothing is written except the cache file. That file is written to a temp name and then renamed, so a partial file is never read. If the cache folder was made but nothing was written, the next run uses it. A ~/.local or state folder that fails the check is refused (and now said) before any mkdir under it.
  • Old or empty state:
    • The cache version goes 2 → 3, and the file name changes (one per version and settings). An old kit's heavy-facts.json is never read; a first call beside one written in the last 5 minutes starts "elevated" once, then its own file carries. An old and a new heavy taking turns no longer keep each other "elevated". Old files past the carry window are removed.
    • Old t3-capacity, new facts: every field it reads is still there, so it keeps working. Its reserved= still adds the reservations (reservedMb), so that one line keeps the old double count until the new t3-capacity is installed. The free/OK answer is right with either printer.
    • New t3-capacity, old facts (no per-reservation owedMb): it reads that field with .get, so holding falls back to size minus use, as before.
    • With no user manager, the answer comes from the machine tier (Basic when some person's reservations cannot be read) and the temp cache.
    • With no reservations or no matching processes, owedMb is 0 and the answer is the same as before.
    • With no cgroup v2, the tier is still basic (never free), as before.
    • No temp folder this user can write: no cache, never the working folder.

General vs project

Everything here is general. It concerns cgroup v2, systemd user managers, /etc/passwd, /run/user and the temp root. Nothing about a project, host or path is built in. No project config changes.

Fixes #771
Part of #844
Fixes #923

🤖 Generated with Claude Code

…anager; ~/.local checks (#771, #844, #923)

- #923: a reservation's owed part is the learned debt of the pieces
  working in its folder, added to its use, so free counts it once.
  t3-capacity shows holding without the owed part.
- #844: with cgroup v2 and user.slice but no manager of this user's,
  facts answer at the new machine tier from machine-wide numbers
  instead of always NOT OK. The answer cache falls back to a private
  0700 folder under the temp root when /run/user/<uid> is missing.
- #771: heavy and the sweep refuse a ~/.local others can write; HOME
  is resolved before it is checked; README repair line covers
  ~/.local; the timeout hygiene check knows more Popen/waitpid forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: code · reviewer: lens agent (Opus 5.5) · read at: de3dc1c
verdict: not safe to merge
covered: the whole diff (heavy_facts.py, t3-capacity, bin/heavy, heavy_sweep.py, README, the three test files), capacity_rule.py decision/_read_facts/_margin, capacity_readings.py read_machine/_container_memory, and issues #771, #844, #923
checks: ran facts.test.py (62 OK) and sweep.test.py (84 OK, 1 skip) under heavy, heavy.test.py gated and unwrapped (68 OK); a scratch test (not committed) on the PR's own test machine with one agent whose processes work in two reserved folders; read the cache folder, HOME and ~/.local checks for links, other-writable folders and check-then-use races
state: runs again: owedMb and the tier are read fresh every call, the same answer for the same machine; stopped half-way: only the cache file is written, through a temp file and a rename, and a made-but-empty cache folder is reused; old state: cache v2 is ignored (v3), an old t3-capacity still reads every field it needs and a new one falls back with .get when owedMb is missing; empty state: no reservations or no matching processes give owedMb 0, the same answer as before
blocker: none
major: a piece whose processes work in two reserved folders has its whole debt taken off both reservations, so that debt is counted zero times and the answer is looser than before (lib/heavy_facts.py:219 and :437)
minor: the machine tier reads only the machine's root numbers, never the caller's own cgroup (/proc/self/cgroup: a runner service or user slice with MemoryMax, or a container sharing the host's cgroup namespace), and sends pieceCaps true although the caller's own work cannot run in a heavy scope; it can say OK while the caller's own cgroup is full (lib/heavy_facts.py:396, :456). The full tier has the same blind spot for user-UID.slice, and #844 asks for the same answer, so this is a follow-up, not a fix here
minor: another local user can create /tmp/heavy-facts-UID first; the check refuses it (right), but then that user's machine tier never has a cache and every --mb ask stays NOT OK. This fails closed and matches the old behaviour; the note says why. A README line on how to clear it would help (lib/heavy_facts.py:577)
minor: the cache folder is checked with lstat and then used by path; in a temp root that others can write without the sticky bit (an odd TMPDIR), the folder could be renamed and swapped for a link after the check. /tmp is sticky, so this is a nit; opening the folder with O_DIRECTORY|O_NOFOLLOW and using dir_fd would close it (lib/heavy_facts.py:586)
minor: realpath closes the HOME-link swap (#771 nit), but a resolved home inside a folder others can write without the sticky bit can still be renamed after the check; very unusual, not in the issue's scope (bin/heavy:445, lib/heavy_sweep.py:1252)

Detail for the major. Measured with the PR's test machine: two reservations of 1536 MB (folders a and b), one agent scope with pid 4001 (300 MB, cwd in a) and pid 4002 (10 MB, cwd in b), writer cost 768:

owed 458   hold 1846   reservations: a useMb 300 owedMb 458; b useMb 10 owedMb 458
held in all: 1846 + 458 + 310 = 2614, meant 3072 (458 MB counted zero times)
before this PR: 1236 + 1526 + 458 + 310 = 3530 (458 counted twice)

The same happens with nested reservations (/x and /x/y) and with a heavy job whose processes cross two reserved folders. For a job the debt is its whole kind cost less its use, so the gap can be thousands of MB. It can only happen on a machine with two or more reservations, but that is the case #923 is about. Fix: give each piece's debt to one reservation only. For example, the deepest folder that holds one of its processes, then the first by name. Add a test with one piece in two folders and one in nested folders that checks hold + owed + use equals the reservations' sum. The use part of nested reservations was already credited to both before this PR. That is older behaviour: leave it, or file it.

Issue coverage: #923 is covered except the case above, and the display fix (reserved= is now only the debts) is done. #844 is covered: machine tier, note and reason, README, tests, cache fallback. #771 is covered: items 1 to 3 and all three nits, each with a test that fails without it (RedCheck for facts; the PR body's red runs for sweep and heavy).

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: state · reviewer: lens agent (Opus 5.5) · read at: de3dc1c
verdict: not safe to merge
covered: the whole diff (bin/heavy, bin/t3-capacity, lib/heavy_facts.py, lib/heavy_sweep.py, README, the three test files), issues #771, #844, #923, and capacity_rule.decision / _read_facts for how the rule reads useMb + owedMb
checks: ran facts.test.py (62 OK), sweep.test.py (84 OK, 1 skip) and heavy.test.py unwrapped (68 OK). Removed two fix lines and reran facts.test.py: dropping + r['owedMb'] in facts_for turns 4 tests red (the #923 counted-once test, the t3-capacity reserved test, two RedCheck lines); making the tier 'full' if mine else 'basic' turns the #844 machine-tier test and RedCheck red. Loaded the PR's and main's bin/heavy on a real host whose ~/.local is mode 775 (umask 0002): main gives ~/.local/state/heavy/labelled, the PR gives '' for state_marks_dir().
state: runs again: owedMb is recomputed from live cgroup.procs and /proc each read, same answer for the same machine; the temp cache folder is reused only when it is a real 0700 folder of this user's. Stopped half-way: only the cache file is written (temp name plus rename); a half-made temp folder is reused or refused, never trusted. Old state: a v2 cache is ignored (version 3), but its carry is dropped too, see the second minor. Old t3-capacity with new facts: works (reservedMb, holds still there). New t3-capacity with old facts: works (r.get('owedMb'), machine owedMb exists on main too). No /run/user/: temp-folder cache; a planted, linked or open folder means no cache and every call starts elevated (fails closed). Empty state: no reservations or no matching pids gives owedMb 0, the same answer as main. Fails open: the machine tier for another user (major below) and the carry reset (second minor).
blocker: none
major: the new machine tier reads only the caller's own reservations, so the #844 caller (a CI runner or root, another user with another HOME) never sees the owner's reservations and now gets OK where main always said NOT OK. reserveDir is $T3_CAPACITY_RESERVE_DIR else the caller's own ~/.config/t3-capacity/reserve.d (lib/heavy_facts.py:626), and the reservation's use counts only this user's processes (lib/heavy_facts.py:208). #844 asks for the machine-wide answer to include "holds, reservations", and MACHINE_WHY (lib/heavy_facts.py:64) and the reason say reservations are counted machine-wide. Example: the owner reserves 1536 MB for a project; a runner user without a user manager asks for 3072 MB on a machine with 4000 MB free after margin: the runner is told OK, so the project's room is gone. Users with a user manager already had this gap on main, but those callers were the reason the basic tier failed closed. Fix: read reservations from a machine-wide place for this tier (the machine file, like holds, or a shared reserve.d), or narrow the note and reason to "this user's reservations only" and keep #844 open (or file a follow-up) before G2 moves the shared t3-capacity onto the kit. Add a test with two users and one reserve.d.
minor: on a host with umask 0002 (Ubuntu's default for user-private groups), ~/.local is often 775, so the new parent check (bin/heavy:404, lib/heavy_sweep.py:949 and parent_ok) silently drops the second notes folder that main used. Verified on a real host: main returns the state folder, the PR returns ''. This fails closed (heavy asks Docker after every run in a sandbox with its own /run/user), but nothing says why, and the sweep then never prunes notes left there by earlier runs. Add chmod go-w ~/.local to the README repair line and a note when the folder is refused.
minor: a cache file that is there but unusable (another version, so once per user after this 2→3 bump, or alternate writes by an old and a new heavy sharing /run/user//heavy-facts.json) gives carry None (lib/heavy_facts.py:482), and cr.level with no previous state goes straight to the raw level, skipping the severe/elevated hold-down. With no cache folder at all, the PR starts at "elevated". This mechanism is older than the PR, but the version bump triggers it on every installed host. Treat "present but unusable" like "no folder" (carry elevated).
minor: the #923 attribution takes a piece's whole debt off a reservation when any one of its processes has its cwd in the folder (lib/heavy_facts.py:218). An agent or build of another project that briefly runs a shell cd'd into the folder empties that reservation's hold while its own debt is counted once. Overlapping reservations (nested folders, or one folder in two .conf files) each subtract the same debt. Both are the same class as the existing useMb counting, but now with whole-cost amounts. Consider counting a piece only through its scope's main process, or once across reservations.
minor: tempfile.gettempdir() (lib/heavy_facts.py:583) falls back to the current folder when no temp folder is writable, so a read-only sandbox gets a heavy-facts-<uid>/ folder in the job's working tree. Use only TMPDIR or /tmp and give no cache otherwise.

…very reserve.d (#945)

Review fixes for #945:
- A process counts toward one reservation only (the deepest folder that
  holds it), and a piece's debt is split over its processes by their
  use, so two or nested reserved folders never take a debt off twice.
- The machine tier reads every person's reserve.d (passwd and
  user.slice users from UID_MIN to UID_MAX). One it cannot read, or a
  user with a session but no home it can find, makes it Basic.
- A ~/.local others can write (umask 002) is said, not dropped
  silently, by heavy and the sweep; the README repair line adds
  chmod go-w ~/.local.
- A cache file that is there but cannot be used (another version,
  other settings, broken, future) starts "elevated", never ok.
- No temporary folder this user can write: no cache, never the
  working folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: code r2 · reviewer: lens agent (Opus 5.5) · read at: 90ad533
verdict: safe to merge
covered: the whole diff since main (lib/heavy_facts.py, bin/t3-capacity, bin/heavy, lib/heavy_sweep.py, README, the three test files), the fix commit 90ad533 against de3dc1c, capacity_rule.py decision/_read_facts/cost_of/next_pressure, issues #771, #844, #923, #951, and both round-1 lens comments
checks: ran facts.test.py (68 OK) and sweep.test.py (84 OK, 1 skip) under heavy, heavy.test.py gated and unwrapped (68 OK). Checked by hand that debt() gives the same number as the rule's owed part for each piece (same cost_of, same demand reading and bounds), and that, per reservation, the new use is at most main's (one reservation per process, same uid filter in the full tier), so the hold is never below main's minus that reservation's owedMb, and the owedMb shares of one piece add up to at most its debt. A scratch probe (not committed) on the PR's test machine: two callers of one user whose settings differ (T3_CAPACITY_RESERVE_DIR) call --mb 64 in turn every 20 s on a calm machine, against this head and against main
state: runs again: owedMb, use, the tier and every person's reservations are read fresh each call, the same answer for the same machine. Stopped half-way: only the cache file is written (temp name, then rename); a made but empty temp cache folder is reused when it is a 0700 folder of this user's. Old state: a v2 cache now starts "elevated" once; an old t3-capacity still reads every field, a new one falls back with .get when owedMb is missing. Empty state: no reservations, or no process in one, gives owedMb 0 and main's answer. Two callers of one user with different settings keep each other "elevated" (first minor)
round 1, fixed: a piece with processes in two or nested reserved folders had its whole debt taken off each, so it was counted zero times. Now best() gives each process one reservation (the deepest of its user's, then the first by name) and count_in_reservations splits a piece's debt over its processes by their use. The tests check held + owed + used against the sum for two folders, a nested one and the same folder in two files, and RedCheck turns each fix line back and sees the test fail
round 1, fixed (state lens): the machine tier read only the caller's own reservations. other_reserve_dirs reads every person's reserve.d (passwd and user.slice users in UID_MIN..UID_MAX), with O_NOFOLLOW, plain files only, 64 KiB, and a folder or file it cannot read, or a session user with no home, makes the tier basic. #844 stays open (Part of) for hosts with closed homes
round 1, fixed (state lens): the ~/.local mode 775 refusal is now said in bin/heavy and in the sweep, with chmod go-w, and the README repair line has it
round 1, fixed (state lens): a cache file that is there but unusable starts "elevated", and the temp root is only $TMPDIR, then /tmp, never the working folder
round 1, moved to #951: the machine tier's blind spot for the caller's own cgroup cap, the /tmp/heavy-facts-UID squat, the cache folder check-then-use in a non-sticky TMPDIR, and a resolved home inside a non-sticky folder others can write
blocker: none
major: none
minor: two callers of one user whose settings differ (another T3_CAPACITY_RESERVE_DIR, queue or slice: a different cache key) and that share /run/user/UID/heavy-facts.json now keep each other at "elevated" for as long as they take turns within the 60 s calm window, so every t3-capacity and --mb ask says NOT OK on a calm machine; on main the other key carried nothing and the level was the raw one. It fails closed and stops when one caller stops, and the test pins the other-settings case on purpose, but the PR body names only an old and a new heavy taking turns. Pressure is machine-wide except boxPsi, so carrying the pressure of a valid file of this user's whatever its key, or a cache file per key, would avoid it (lib/heavy_facts.py:595-628, read_cache; probe below)
minor: the machine tier leaves out some reservations without making the tier basic: those under another user's $T3_CAPACITY_RESERVE_DIR (only ~/.config/t3-capacity/reserve.d is read), root's and system users' (below UID_MIN), NSS users with no session now, and users whose home is "/" or not absolute. The default case is covered, and all of these are unusual, but the README says "every person's reserve.d" and "it never takes room it cannot see"; name these limits there (lib/heavy_facts.py:231-273, README Tiers)
minor: in the machine tier a caller that is not root cannot read another user's /proc/PID/cwd, so another person's reservation gets no use and no owedMb and holds its whole size while their agents' debts are counted too (main's #923 double count, for those only). It is on the safe side, but the README says the machine tier "gets the answer a caller with a user manager gets"; it can be stricter (lib/heavy_facts.py:310-314)
minor: bin/heavy's refusal message says chmod go-w <folder> also when the folder is refused because another user owns it (a root-owned ~/.local from the #760 heavy), where chown is the fix; the README repair line has both (bin/heavy:398-421)

Probe for the first minor: --mb 64 on a calm machine, callers A and B take turns every 20 s (exit code, level):

main:      (0,'ok') x 12
this head: (0,'ok'), then (75,'elevated') x 11
one key:   (0,'ok') x 12

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: state r2 · reviewer: lens agent (Opus 5.5) · read at: 90ad533
verdict: safe to merge
covered: the whole diff against main and the fix diff since de3dc1c (lib/heavy_facts.py, bin/heavy, bin/t3-capacity, lib/heavy_sweep.py, README, the three test files), capacity_rule.level/decision for the hold-down, free_contract for the new tier value, issues #771, #844, #923, #951, and both round-1 lens comments
checks: ran facts.test.py under heavy (68 OK) and sweep.test.py under heavy (84 OK, 1 skip). Ran heavy.test.py gated and unwrapped: 67 of 68 OK, with one red, test_leftover_that_ignores_term_is_killed_after_the_grace_time (the "(stubborn" text was missing at load 19/12 cores). It passed when run alone right after, so it is a timing flake under load and not this diff. Removed three fix lines that RedCheck does not cover, one at a time, in a copy of the kit, and each turned its test red: the age < 0 to no_carry() return (cache-version test: ok, not elevated), the blind note for a session with no home (every-person test: machine, not basic), and the 0o077 check on the temp cache folder (no-cache-folder test: ok, not elevated). Traced each old and new state through read_cache, cache_path, other_reserve_dirs, reservation_files and count_in_reservations
state: runs again: owedMb, use and every person's reservations are read fresh from cgroup.procs, /proc and each reserve.d, so the same machine gives the same answer, and the sweep's 775 note is said again each run without changing anything. Stopped half-way: only the cache file is written (temp name, then rename; stale temp files are removed after a minute); a half-written reserve.d line is ignored or holds its MB, never less. Old state: a v2 cache, a cache of other settings, a broken one or one from the future now starts "elevated" and steps down to ok after elevatedClearSeconds of calm (tested); an old heavy reads a v3 file as unusable and goes to the raw level, as it always did. Old t3-capacity with new facts: every field it reads is still there (reserved= keeps the old double count, display only). New t3-capacity with old facts: holding falls back through r.get('owedMb'). No reader checks the tier value, so "machine" breaks no older reader. A reservation folder that is gone holds nothing. A deleted user's leftover session, or a closed home, makes the machine tier Basic (fails closed, with a note). Empty state: a fresh install or a reboot has no cache file and starts at the raw level, as on main and as documented. Fails open: nothing found, except the narrow NSS gap in the second minor. "Elevated" sticks only while two callers with different settings or versions take turns on one cache file (first minor), or with no cache at all (#951).
round 1, fixed: a debt was counted zero times for work across two or nested reservations. Each process now counts toward one reservation (best(): the deepest folder, then the first by name) and a piece's debt is split by use, so held + owed + used equals what is reserved (lib/heavy_facts.py:276, :289; the two-folder/nested and never-looser-than-main tests)
round 1, fixed: the machine tier read only the caller's own reserve.d. It now reads every person's and fails closed to Basic on a folder or file it cannot read, or a session with no home (lib/heavy_facts.py:551; the every-person test covers a FIFO, a closed folder and a session with no home)
round 1, fixed: a ~/.local of mode 775 was refused silently. bin/heavy and the sweep now say so and name chmod go-w; the README repair line has it
round 1, fixed: an unusable cache skipped the hold-down. It now gets no_carry(); only a missing file or one past the carry window carries nothing (lib/heavy_facts.py:589-622)
round 1, fixed: the cache fell back to the working folder. temp_root() uses only $TMPDIR, then /tmp
round 1, fixed: one shell cd'd into a reserved folder took a piece's whole debt off it. The debt is now split by use, so a small process takes only its share
round 1, filed: the own-cgroup cap, the /tmp name squat, the check-then-use of the cache folder and the home's parents are in #951
blocker: none
major: none
minor: "elevated" can stick for as long as two callers take turns on one cache file. The key covers HEAVY_SLICE, HEAVY_QUEUE_ROOT, T3_CAPACITY_RESERVE_DIR, the user and the holds, and the version is checked too. Two callers of one user with different settings (or an old and a new heavy, as the PR body says) that each call within elevatedClearSeconds (60 s) of the other each read the other's file as unusable, so every call starts "elevated" and never reaches ok. t3-capacity needs level ok, so it says NOT OK to both for as long as they keep calling, and no note names the cause. This fails closed, but it can stall every gate on the machine. Example: a test harness that sets HEAVY_QUEUE_ROOT but not HEAVY_FACTS_CACHE, running beside live agents. Main went to the raw level there instead. Fix idea: one file per version and key, for example heavy-facts-v3-.json, which also ends the rollout case; or a note when the key or version differs. Add it to #951 (lib/heavy_facts.py:616)
minor: for people with no session, the machine tier reads only /etc/passwd. A person whose account comes from NSS (LDAP, sssd) and who has no session or linger is not read, so a reservation of theirs is not seen. This is the one fail-open gap left, and it is narrow. Each machine-tier call also lists every passwd home, which on a host with autofs homes mounts them all every 5 s. Fix idea: also read pwd.getpwall(), or say in the README that only local and logged-in people are read (lib/heavy_facts.py:251)
minor: a .conf in another person's reserve.d that is a link (stow-style dotfiles) is refused (O_NOFOLLOW), which makes every machine-tier caller Basic. That fails closed and a note names the file, but the README's Tiers paragraph does not say that another person's .conf files must be plain files (lib/heavy_facts.py:196, README.md:548)

scuttletoby and others added 3 commits November 22, 2026 12:07
…ssage; machine tier limits in README (#945)

- Two callers of one user with other settings (slice, queue, reservations folder) or another kit version shared
  one cache file and kept each other "elevated" (NOT OK on a calm machine). Each now has its own
  heavy-facts-v<version>-<hash>.json, so each one's hold-down steps down on its own. A first call beside another
  recent cache file of this user's still starts "elevated"; stale ones are removed after the carry window.
- bin/heavy and the sweep name chown when another user owns the refused folder, chmod go-w otherwise.
- README: the machine tier's limits named plainly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby
scuttletoby merged commit cce8005 into main Nov 22, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

heavy: reservation counted twice for agents under reserve.d heavy: minors from the #770 reviews (~/.local mode, README repair line, linked ~/.local)

1 participant