diff --git a/fork-kit/heavy/README.md b/fork-kit/heavy/README.md index 4d177b59484ab..098dd4d1a89a8 100644 --- a/fork-kit/heavy/README.md +++ b/fork-kit/heavy/README.md @@ -59,9 +59,11 @@ heavy asks Docker only after a run that labelled a container (`heavy docker-labe `/run/user` is full or a sandbox has a `/run/user` of its own, in `~/.local/state/heavy/labelled/`, which heavy reads too; the home is `$HOME`, else the password database's, and only one this user owns and no one else can write (not `/`), so heavy run as root with a user's `HOME` (`sudo -E heavy ...`) or `HOME=/tmp` uses root's own home -and never makes a root-owned folder in the user's; `~/.local` must be a real folder (not a link) of the user's; -an older heavy run as root with your `HOME` may have left a root-owned `~/.local/state` (heavy then skips it): -`sudo chown -R "$USER:" ~/.local/state` gives it back; +and never makes a root-owned folder in the user's; `~/.local` must be a real folder (not a link, even to a folder +of the user's: kept fail-closed) of the user's that no one else can write; an older heavy run as root with your +`HOME` may have left a root-owned `~/.local` or `~/.local/state`, and a umask of 002 a `~/.local` of mode 775 +(heavy then skips it and says so): +`sudo chown "$USER:" ~/.local && sudo chown -R "$USER:" ~/.local/state && chmod go-w ~/.local` gives them back; heavy tells `docker-label` which folders it reads in `HEAVY_MARKS`; without either folder heavy asks after every run), so a run with no containers costs no `docker ps`. No Docker, or none this user can reach (a rootless Docker through `docker context use` counts as reachable): nothing to do, nothing said. Docker down: one @@ -543,9 +545,25 @@ after another, up to the group's ceiling), the memory budget, the queue and note - **--kind K:** does the rule admit one piece of K (`agent-writer`, `job-build`, or heavy's own `build`). **--mb MB** alone: do MB fit the free budget at level "ok"; with --kind, the piece declares MB. - **T3 down or not installed:** no agent scopes, still an answer from the kernel. -- **Tiers:** full with cgroup v2 and a systemd user manager. Without them (Basic) agents and other users' work - cannot be counted, so the answer is never free and every `fits` is 0 (a note says why). An other user's folder - that exists but cannot be listed is Basic too. +- **Tiers:** full with cgroup v2 and a systemd user manager. Machine with cgroup v2 and `user.slice` but no user + manager of the caller's (a CI runner's user without lingering or a login session): it has no agents or heavy + jobs of its own, every user's are counted as usual, and so are other people's reservations (`tier: "machine"`, a + note, and the reason says "machine-wide numbers"). Its limits: + - It reads only each person's default `~/.config/t3-capacity/reserve.d`, not a `T3_CAPACITY_RESERVE_DIR` they + set for themselves. + - It reads the people of `/etc/passwd` with a UID from `UID_MIN` to `UID_MAX`, and anyone with a `user.slice` + session. Not read: root and system users (below `UID_MIN`), people known only through NSS (LDAP, sssd) who have + no session now, and homes that are `/` or not absolute. + - Each `.conf` there must be a plain file: a link (stow-style dotfiles) is refused and makes the tier Basic. + - A person's `reserve.d` it cannot read (a home closed to others, mode 750 or 700), or a user with a session but + no home it can find, makes it Basic. + - A caller that is not root cannot see another person's working folders, so their reservations hold their whole + size, and their agents' debts are counted too. That is stricter than a caller with a user manager, never looser. + + Without cgroup v2, or without both a user manager and `user.slice` (Basic), agents and other users' work cannot + be counted, so the answer is never free and every `fits` is 0 (a note says why). An other user's folder that + exists but cannot be listed is Basic too. + - **Unreadable signals** (PSI, a box with a live piece but without its `memory.pressure`, or with no folder at all) give "elevated", never "ok"; a missing MemAvailable or a container limit that cannot be read gives busy for every kind. A box with no live piece of its own passes no box pressure (as the server's observer). A piece whose @@ -553,10 +571,15 @@ after another, up to the group's ceiling), the memory budget, the queue and note - **Reservations:** files `~/.config/t3-capacity/reserve.d/*.conf` (or `$T3_CAPACITY_RESERVE_DIR`), lines ` [until=YYYY-MM-DD]`. The folder matches itself and what is below it, with or without a trailing `/`, never a sibling sharing its first letters (`/x/app` is not `/x/app-x`); its real path (links - resolved) matches too. A reservation holds back its MB less what this user's processes working in that folder - already use; a caller working inside the folder (its real working folder) is not held back by its own - reservation. Ended, broken or unreadable lines hold nothing and give a note. Limit: another user's reservations - are not seen. + resolved) matches too. A reservation holds back its MB less what its user's processes working in that folder + already use, and less what agents and heavy jobs with a process working there still owe the rule (their cost + less their demand, `owedMb`): that room is already held as their debt, so it is counted once (one idle 300 MB + agent under a 1536 MB reservation holds 1536 MB in all, not 1536 + 768 - 300). Each process counts toward one + reservation only (the deepest folder that holds it, then the first by file name), and a piece's debt is split + over its processes by their use, so with two or nested reserved folders nothing is taken off twice. A caller working + inside the folder (its real working folder) is not held back by its own reservation. Ended, broken or unreadable + lines hold nothing and give a note. Limit: in the full tier another user's reservations are not seen (the + machine tier reads them, above). - **Holds (machine file):** a cgroup outside the user managers, such as a CI runners' slice, is not in the ledger. `heavy.capacity.holds` names such cgroups; one that uses more than `inUseFromMb` (anon + shmem + swap) keeps its budget (its `memory.max`, or `budgetMb` when it has none) less the RAM it uses (anon + shmem: `memory.max` limits @@ -564,15 +587,23 @@ after another, up to the group's ceiling), the memory budget, the queue and note Not there: nothing runs in it. There without a memory controller (no `memory.stat`): its whole `budgetMb` is held, with a note. Its use cannot be read: the whole budget is held, with a note. A reservation file named like a hold (`