Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 53 additions & 18 deletions fork-kit/heavy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,11 @@ heavy asks Docker only after a run that labelled a container (`heavy docker-labe
`/run/user` is full or a sandbox has a `/run/user` of its own, in `~/.local/state/heavy/labelled/`, which heavy
reads too; the home is `$HOME`, else the password database's, and only one this user owns and no one else can
write (not `/`), so heavy run as root with a user's `HOME` (`sudo -E heavy ...`) or `HOME=/tmp` uses root's own home
and never makes a root-owned folder in the user's; `~/.local` must be a real folder (not a link) of the user's;
an older heavy run as root with your `HOME` may have left a root-owned `~/.local/state` (heavy then skips it):
`sudo chown -R "$USER:" ~/.local/state` gives it back;
and never makes a root-owned folder in the user's; `~/.local` must be a real folder (not a link, even to a folder
of the user's: kept fail-closed) of the user's that no one else can write; an older heavy run as root with your
`HOME` may have left a root-owned `~/.local` or `~/.local/state`, and a umask of 002 a `~/.local` of mode 775
(heavy then skips it and says so):
`sudo chown "$USER:" ~/.local && sudo chown -R "$USER:" ~/.local/state && chmod go-w ~/.local` gives them back;
heavy tells `docker-label` which folders it reads in `HEAVY_MARKS`; without either folder heavy asks after every
run), so a run with no containers costs no `docker ps`. No Docker, or none this user can reach (a
rootless Docker through `docker context use` counts as reachable): nothing to do, nothing said. Docker down: one
Expand Down Expand Up @@ -543,36 +545,65 @@ after another, up to the group's ceiling), the memory budget, the queue and note
- **--kind K:** does the rule admit one piece of K (`agent-writer`, `job-build`, or heavy's own `build`). **--mb
MB** alone: do MB fit the free budget at level "ok"; with --kind, the piece declares MB.
- **T3 down or not installed:** no agent scopes, still an answer from the kernel.
- **Tiers:** full with cgroup v2 and a systemd user manager. Without them (Basic) agents and other users' work
cannot be counted, so the answer is never free and every `fits` is 0 (a note says why). An other user's folder
that exists but cannot be listed is Basic too.
- **Tiers:** full with cgroup v2 and a systemd user manager. Machine with cgroup v2 and `user.slice` but no user
manager of the caller's (a CI runner's user without lingering or a login session): it has no agents or heavy
jobs of its own, every user's are counted as usual, and so are other people's reservations (`tier: "machine"`, a
note, and the reason says "machine-wide numbers"). Its limits:
- It reads only each person's default `~/.config/t3-capacity/reserve.d`, not a `T3_CAPACITY_RESERVE_DIR` they
set for themselves.
- It reads the people of `/etc/passwd` with a UID from `UID_MIN` to `UID_MAX`, and anyone with a `user.slice`
session. Not read: root and system users (below `UID_MIN`), people known only through NSS (LDAP, sssd) who have
no session now, and homes that are `/` or not absolute.
- Each `.conf` there must be a plain file: a link (stow-style dotfiles) is refused and makes the tier Basic.
- A person's `reserve.d` it cannot read (a home closed to others, mode 750 or 700), or a user with a session but
no home it can find, makes it Basic.
- A caller that is not root cannot see another person's working folders, so their reservations hold their whole
size, and their agents' debts are counted too. That is stricter than a caller with a user manager, never looser.

Without cgroup v2, or without both a user manager and `user.slice` (Basic), agents and other users' work cannot
be counted, so the answer is never free and every `fits` is 0 (a note says why). An other user's folder that
exists but cannot be listed is Basic too.

- **Unreadable signals** (PSI, a box with a live piece but without its `memory.pressure`, or with no folder at
all) give "elevated", never "ok"; a missing MemAvailable or a container limit that cannot be read gives busy for
every kind. A box with no live piece of its own passes no box pressure (as the server's observer). A piece whose
`memory.stat` is cut short or broken owes its whole cost.
- **Reservations:** files `~/.config/t3-capacity/reserve.d/*.conf` (or `$T3_CAPACITY_RESERVE_DIR`), lines
`<MB> <absolute folder> [until=YYYY-MM-DD]`. The folder matches itself and what is below it, with or without a
trailing `/`, never a sibling sharing its first letters (`/x/app` is not `/x/app-x`); its real path (links
resolved) matches too. A reservation holds back its MB less what this user's processes working in that folder
already use; a caller working inside the folder (its real working folder) is not held back by its own
reservation. Ended, broken or unreadable lines hold nothing and give a note. Limit: another user's reservations
are not seen.
resolved) matches too. A reservation holds back its MB less what its user's processes working in that folder
already use, and less what agents and heavy jobs with a process working there still owe the rule (their cost
less their demand, `owedMb`): that room is already held as their debt, so it is counted once (one idle 300 MB
agent under a 1536 MB reservation holds 1536 MB in all, not 1536 + 768 - 300). Each process counts toward one
reservation only (the deepest folder that holds it, then the first by file name), and a piece's debt is split
over its processes by their use, so with two or nested reserved folders nothing is taken off twice. A caller working
inside the folder (its real working folder) is not held back by its own reservation. Ended, broken or unreadable
lines hold nothing and give a note. Limit: in the full tier another user's reservations are not seen (the
machine tier reads them, above).
- **Holds (machine file):** a cgroup outside the user managers, such as a CI runners' slice, is not in the ledger.
`heavy.capacity.holds` names such cgroups; one that uses more than `inUseFromMb` (anon + shmem + swap) keeps its
budget (its `memory.max`, or `budgetMb` when it has none) less the RAM it uses (anon + shmem: `memory.max` limits
RAM only, so swap is left out), as a reservation named by its `label`, for every user except its own (`users`).
Not there: nothing runs in it. There without a memory controller (no `memory.stat`): its whole `budgetMb` is held,
with a note. Its use cannot be read: the whole budget is held, with a note. A reservation file named like a hold
(`<label>.conf`) gives a note; both hold.
- **Cache:** `/run/user/<uid>/heavy-facts.json` (or `$HEAVY_FACTS_CACHE`), 0600, read again after 5 s (`--fresh`:
now). Pressure levels carry from it (at most 5 minutes old) so calm must last before a level drops. Without a
cache folder nothing carries, so every call starts "elevated" (only the rule's floors admit; never straight to
"ok").
- **Cache:** `/run/user/<uid>/heavy-facts-v<version>-<hash>.json`, one file per kit version and settings (slice,
queue, reservations folder, user, holds), so two callers of one user with other settings, or an old and a new
kit, never share one and each one's hold-down steps down on its own (or one file, `$HEAVY_FACTS_CACHE`), 0600,
read again after 5 s (`--fresh`: now). A user without `/run/user/<uid>` (no user manager) uses
`heavy-facts-<uid>/` in the temporary folder (`$TMPDIR`, else `/tmp`, never the working folder), made 0700; one that is a link, another user's or open to
others, or no temporary folder this user can write, is no cache (a note). Pressure levels carry from it (at most
5 minutes old) so calm must last before a level drops. Without a cache folder, with a cache file that is there
but cannot be used (broken, from the future, other settings in `$HEAVY_FACTS_CACHE`), or with no file of its own
yet but another cache file of this user's written in the last 5 minutes beside it (another version, as once after
an update, or other settings), nothing carries, so the call starts "elevated" (only the rule's floors admit; never
straight to "ok"). Cache files of other settings or versions older than 5 minutes are removed.

Two thin callers come with it:

- `bin/t3-capacity [--kind KIND] [MB]`: the older host check's output (first line `available=..MB swapfree=..MB
load=..`, available less the rule's whole reservation hold, last line `OK` or `NOT OK: why`, exit 0 or 1), now
load=..`, available less the rule's whole reservation hold, `reserved=` what admitted jobs and agents may still
grow into (so `available - reserved` counts each once), last line `OK` or `NOT OK: why`, exit 0 or 1), now
from `heavy facts`. MB defaults to 3072. The installer links `/usr/local/bin/t3-capacity` to it, so the command
has one source, the release in use (a host's own plain copy there only with `--replace-t3-capacity`: Install,
below); `heavy --self-check` warns when the `t3-capacity` found on PATH is another copy (its readings can differ:
Expand All @@ -586,12 +617,15 @@ either caller.

### Facts run again, stop half-way, or meet old or empty state

- **No user manager** (no `/run/user/<uid>`): Machine tier, and the cache in the temporary folder, so the level
carries and can reach "ok" as with one.
- **Runs again:** within 5 s the cache answers; two callers in the same second both answer and leave one valid
cache (each writes its own temporary file and renames it).
- **Stops half-way:** a temporary cache file is left; the next run removes ones older than a minute.
- **Old, empty, broken or foreign cache** (another owner, group or other access, a link, a time in the future, an
unknown key or version, or written with other settings: slice, queue or reservations folder): read again from
the kernel. A cache that cannot be written gives a note.
unknown key or version (an older kit's cache, without each reservation's `owedMb`, is version 2), or written with
other settings: slice, queue or reservations folder): read again from the kernel. A cache that cannot be written
gives a note.
- **Cost file** (`peaks.json`) missing, empty or broken: the rule's start costs, with the queue's note.
- **Reservation file cut short** (its writer stopped half-way): its whole lines hold, the cut line is a note.
A reservation whose processes are all gone still holds all its MB (the room is kept for them), until its
Expand Down Expand Up @@ -663,7 +697,8 @@ limit options. A user's own unit in `~/.config/systemd/user/` wins over the one
| `/run/user/<uid>/heavy-queue/<slice>/` | the memory queue: its lock and one ticket per queued job (tmpfs) |
| `~/.local/state/heavy/` | the queue's wait log (`waits.log`) and peak history (`peaks.json`), |
| | measured runs for `heavy capacity` (`measured.jsonl`, `.1`, `.lock`) |
| `/run/user/<uid>/heavy-facts.json` | `heavy facts`' 5 s cache (tmpfs) |
| `/run/user/<uid>/heavy-facts-*.json` | `heavy facts`' 5 s cache (tmpfs), one per version and settings |
| `/tmp/heavy-facts-<uid>/heavy-facts-*.json` | the same, for a user without `/run/user/<uid>` (no user manager) |
| `~/.config/t3-capacity/reserve.d/*.conf` | each user's reservations for `heavy facts` and its callers |

## Install (owner, as root)
Expand Down
34 changes: 26 additions & 8 deletions fork-kit/heavy/bin/heavy
Original file line number Diff line number Diff line change
Expand Up @@ -386,13 +386,31 @@ def docker_context():
return name if isinstance(name, str) and name not in ('', 'default') else ''


def repair(folder, st, uid):
"""The command that makes a refused folder usable: chown when another user owns it (a root-owned ~/.local from an
older heavy run as root, #760), else chmod go-w (a ~/.local of mode 775, as umask 002 makes it). A link or a file
in its place is not a folder at all."""
if not stat.S_ISDIR(st.st_mode):
return f'replace {folder} with a real folder'
if st.st_uid != uid:
return f'sudo chown "$USER:" {folder}'
return f'chmod go-w {folder}'


def own_folder(base, make_base=False):
"""<base>/heavy/labelled, made if missing: '' unless it, its parent and base are folders this user owns and
only this user can write. make_base: base (~/.local/state) and its parent (~/.local) are made too, one at a
time, and the parent must be a real folder (no symlink) this user owns, so a planted ~/.local link never
leads heavy to make folders elsewhere."""
time, and the parent must be a real folder (no symlink) this user owns that no one else can write, so a planted
~/.local link never leads heavy to make folders elsewhere, and no one else can swap state for a link (#771). A
folder refused there is said (#945: a ~/.local of mode 775, as umask 002 makes it, would drop it silently)."""
uid = os.getuid()
path = os.path.join(base, 'heavy', 'labelled')

def refused(folder, st):
if make_base:
say(f'no second notes folder for docker-label: {folder} must be a real folder of yours that no one else '
f'can write (`{repair(folder, st, uid)}`; README, Docker leftovers)')
return ''
try:
if make_base:
parent = os.path.dirname(base)
Expand All @@ -401,8 +419,8 @@ def own_folder(base, make_base=False):
except FileExistsError:
pass
st = os.lstat(parent)
if not stat.S_ISDIR(st.st_mode) or st.st_uid != uid:
return ''
if not stat.S_ISDIR(st.st_mode) or st.st_uid != uid or st.st_mode & 0o022:
return refused(parent, st)
for folder in (base, os.path.dirname(path), path):
if folder != base or make_base:
try:
Expand All @@ -411,7 +429,7 @@ def own_folder(base, make_base=False):
pass
st = os.lstat(folder)
if not stat.S_ISDIR(st.st_mode) or st.st_uid != uid or st.st_mode & 0o022:
return ''
return refused(folder, st)
except OSError:
return ''
return path
Expand Down Expand Up @@ -442,12 +460,12 @@ def state_base():
for home in homes:
if not home or not os.path.isabs(home):
continue
home = os.path.realpath(home) # resolved once: a HOME link swapped after this check changes nothing (#771)
try:
st = os.stat(home)
st = os.lstat(home)
except OSError:
continue
if (stat.S_ISDIR(st.st_mode) and st.st_uid == os.geteuid() and not st.st_mode & 0o022
and os.path.realpath(home) != '/'):
if stat.S_ISDIR(st.st_mode) and st.st_uid == os.geteuid() and not st.st_mode & 0o022 and home != '/':
return os.path.join(home, '.local', 'state')
return ''

Expand Down
20 changes: 16 additions & 4 deletions fork-kit/heavy/bin/t3-capacity
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ piece of KIND (agent-writer, job-build, build, ...); with MB too, that piece dec
Output, kept as the older t3-capacity had it so hooks and scripts that read it keep working:
line 1 available=<MB>MB swapfree=<MB>MB load=<load>/<n>cores agents=<n> jobs=<n> waiting=<n> reserved=<MB>MB
need=<MB>MB free=<MB>MB level=<level> tier=<tier> (available: after the reservations that hold back
this caller; reserved: what admitted jobs and agents may still grow into, and those reservations)
then reservations: <name>:<MB>MB(using <MB>,holding <MB>) or <name>:own (a reservation of the caller's folder)
this caller; reserved: what admitted jobs and agents may still grow into, so available - reserved
counts each once; tier full, machine (no user manager: machine-wide numbers) or basic (never OK))
then reservations: <name>:<MB>MB(using <MB>,holding <MB>) or <name>:own (a reservation of the caller's folder;
holding is what is left after its use and the debts of agents and jobs working in it, ", owed <MB>")
note: <text>, fits: <kind>=<n> ..., and last "NOT OK: <why>" or "OK".
Exit 0 OK, 1 NOT OK (busy, or heavy facts did not answer within 5 s: never OK), 2 a wrong call. A NOT OK with stdout
not a terminal also says on stderr that a pipe would hide the exit status.
Expand Down Expand Up @@ -51,14 +53,14 @@ def main(argv):
avail = max(0, (m['availableMb'] or 0) - (m['holdMb'] or 0)) # the rule's hold (all of it, not the
# trimmed reservations list)
print(f'available={avail}MB swapfree={m["swapFreeMb"]}MB load={c["load1"]}/{c["cores"]}cores '
f'agents={q["agents"]} jobs={q["running"]} waiting={q["waiting"]} reserved={m["reservedMb"]}MB '
f'agents={q["agents"]} jobs={q["running"]} waiting={q["waiting"]} reserved={m["owedMb"]}MB '
f'need={need if need is not None else m["costMb"]}MB free={m["freeMb"]}MB level={d["level"]} '
f'tier={d["tier"]}')
lines = []
if d['reservations']:
lines.append('reservations: ' + ' '.join(
f'{r["name"]}:own' if r['own'] else
f'{r["name"]}:{r["mb"]}MB(using {r["useMb"]},holding {max(0, r["mb"] - r["useMb"])})'
f'{r["name"]}:{r["mb"]}MB(using {r["useMb"]},holding {held(r)}{owed(r)})'
for r in d['reservations']))
lines += [f'note: {n}' for n in d['notes']]
lines.append('fits: ' + ' '.join(f'{k}={n}' for k, n in d['fits'].items()))
Expand All @@ -72,6 +74,16 @@ def main(argv):
return 1


def held(r):
"""What a reservation holds back: its size less its use and the debts already counted (#923; an older heavy
facts has no owedMb)."""
return max(0, r['mb'] - r['useMb'] - (r.get('owedMb') or 0))


def owed(r):
return f',owed {r["owedMb"]}' if r.get('owedMb') else ''


def usage():
print('usage: t3-capacity [--kind KIND] [MB]', file=sys.stderr)
return 2
Expand Down
Loading
Loading