From 04a398d04e150f2031e7c75ab3e2cf7e427a385b Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 18:45:19 +0000 Subject: [PATCH 01/33] fork fleet: desk setup pack kit (D8) fork-kit/desk: bootstrap, install, PASS/FAIL check and uninstall for a desktop's side of a desk pool (runner users, launcher, walls, desk agent, gate hook, settings, key, units), from one release commit. Safe to run again and to stop at any step; refuses an install it did not make; uninstall removes only what it made. Co-Authored-By: Claude Opus 5.5 --- .github/fork-ci-tests.txt | 1 + .github/fork-fleet.md | 5 +- fork-kit/README.md | 1 + fork-kit/common/machine.example.json | 7 + fork-kit/desk/README.md | 68 ++++ fork-kit/desk/bootstrap.sh | 60 ++++ fork-kit/desk/sbin/desk-install | 475 ++++++++++++++++++++++++++ fork-kit/desk/tests/install.test.py | 479 +++++++++++++++++++++++++++ 8 files changed, 1094 insertions(+), 2 deletions(-) create mode 100644 fork-kit/desk/README.md create mode 100755 fork-kit/desk/bootstrap.sh create mode 100755 fork-kit/desk/sbin/desk-install create mode 100644 fork-kit/desk/tests/install.test.py diff --git a/.github/fork-ci-tests.txt b/.github/fork-ci-tests.txt index 58e58018f94aa..495cdceec99f6 100644 --- a/.github/fork-ci-tests.txt +++ b/.github/fork-ci-tests.txt @@ -42,6 +42,7 @@ fork-ci/actions/pr-queue/lens.test.py fork-ci/actions/pr-queue/pr_queue.test.py fork-ci/actions/pr-queue/tested-base/tested_base.test.py fork-kit/common/tests/machine_file.test.py +fork-kit/desk/tests/install.test.py fork-kit/golive/tests/check.test.py fork-kit/golive/tests/contract.test.py fork-kit/golive/tests/journal.test.py diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index a7be89e2d3339..a42c9379baa9f 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -642,8 +642,9 @@ beside the walls (one kit: the one those walls read), so `desk.conf` needs no `LAUNCH` line; an old one that names another launcher makes the agent refuse to run (exit 2) until the line is gone. The agent fixes the kit when it starts, in its real path: a link above it that an upgrade repoints changes nothing -until the agent restarts, so restart it after an upgrade. Then, once per -desktop: +until the agent restarts, so restart it after an upgrade. The desk kit +([fork-kit/desk](../fork-kit/desk/README.md)) does all of this, with a PASS/FAIL +check and an uninstall; by hand, once per desktop: ```sh install -d -m 755 /etc/fork-fleet diff --git a/fork-kit/README.md b/fork-kit/README.md index eac02830f821d..0f3680cd3d5c3 100644 --- a/fork-kit/README.md +++ b/fork-kit/README.md @@ -15,6 +15,7 @@ Every kit follows the same pattern: | Kit | What | | ---------------------------- | ------------------------------------------------------------------------------------------------------- | +| [desk](desk/README.md) | A desktop's side of a fork fleet desk pool: runner users, walls, desk agent; one folder (Linux, WSL2) | | [gh-apps](gh-apps/README.md) | GitHub App tokens for agents through `gh-as`, without giving agents the keys (Linux) | | [golive](golive/README.md) | Move a T3 boot service to a new build with a cold copy first; core and `check` so far (Linux) | | [heavy](heavy/README.md) | `heavy `: run builds and tests in a capped slice, stop what they leave behind (Linux, systemd) | diff --git a/fork-kit/common/machine.example.json b/fork-kit/common/machine.example.json index f2173a8f51d48..6251cd5cef9f2 100644 --- a/fork-kit/common/machine.example.json +++ b/fork-kit/common/machine.example.json @@ -25,5 +25,12 @@ "backupDir": "/srv/example/golive/backups", "stateDir": "/srv/example/golive/state", "freeFloorMb": 2048 + }, + "desk": { + "pool": "desktop", + "host": "fleet-host.example", + "hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "slots": 2, + "reserveMb": 4096 } } diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md new file mode 100644 index 0000000000000..b63fc5bc97788 --- /dev/null +++ b/fork-kit/desk/README.md @@ -0,0 +1,68 @@ +# desk: a desktop's side of a fork fleet desk pool + +Turns a Linux machine (or WSL2 on Windows) into one pool of the fork fleet: the desk agent dials out to the +always-on host over ssh, and the dispatcher there starts one-job runners on this machine through it. Everything +the fleet runbook ([.github/fork-fleet.md](../../.github/fork-fleet.md), "The desktop's side") lists by hand, +done by one installer, with a PASS/FAIL check and an uninstall. + +## The machine file + +The facts of this machine go in the `desk` section of `/etc/fork-kit/machine.json` (root-owned; see +[common](../common/README.md)). Made-up example: + +```json +{ + "version": 1, + "desk": { + "pool": "desktop", + "host": "fleet-host.example", + "hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...", + "slots": 2, + "reserveMb": 4096 + } +} +``` + +`pool` must be a pool in the fleet config; `host` and `hostKey` are the always-on host and its ssh host key +(on that host: `cat /etc/ssh/ssh_host_ed25519_key.pub`, without the comment). Optional: `port` (22), `user` (`fleet-desk`), `slots` (1, at most 8; one +runner user each), `reserveMb` (memory the launcher keeps free for the owner), `netBase` (`10.N`, the runners' network, +default `10.204`: pick another when this machine's network already uses it). + +## Install, check, uninstall + +As root, from the fork's protected release branch: + +```sh +sh bootstrap.sh --repo OWNER/REPO --sha RELEASE_SHA [--fleet-config /path/to/project/fork-fleet.json] +/opt/fork-fleet/desk-install check # every line PASS, then ALL PASS +/opt/fork-fleet/desk-install uninstall +``` + +`--fleet-config` is the project's fleet config (its pools); without it the release's own is used. The project's +is remembered, so a later install of a new release keeps it. Install prints the desk key's public half: add it on +the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner template is made in the background by +the refresh timer; `check` says FAIL for `template` until it is there. + +`check` only reads: the installed files against their manifest, every wall of this OS (`fork-fleet-walls.py +describe`), the agent's settings, the units, the template, the runner users and IPv4 forwarding. + +## Runs again, stops half-way, old or empty state + +- **Again:** files, settings and units that did not change are not rewritten, and the agent is not stopped. +- **Stopped half-way:** the next install ends it. A new release is staged in `/opt/fork-fleet/.kit-stage-*` and + checked first (gate loads with the fleet config, the pool is in it, `bash -n`, the walls know this OS); a release + that fails is never installed. Files are renamed in one by one while the agent is stopped; `SOURCE` is written + last, so `check` reports a half-done swap. Users are recorded in `/etc/fork-fleet/desk-kit.json` before they are + made, so uninstall removes them even after a stop. +- **Old install:** a launcher, `launch.conf`, `desk.conf`, desk key or unit that this kit did not write is refused + before anything changes. Remove the older (by hand) install first, the runbook's steps in reverse. +- **Empty:** `check` before an install says FAIL and writes nothing; `uninstall` with no record does nothing. + +Uninstall stops every runner, removes the namespaces and the `fork_fleet` nft table, the kit's units, files and +settings, and only the users it made itself. + +## Why one flat folder + +The other kits keep one folder per version behind one link. The desk agent refuses a launcher or walls file whose +folder path has a link in it (so nobody can point it elsewhere), and every fleet default names `/opt/fork-fleet`, so +this kit installs flat there: changed files are renamed in, a manifest records each hash. diff --git a/fork-kit/desk/bootstrap.sh b/fork-kit/desk/bootstrap.sh new file mode 100755 index 0000000000000..7bf30a1b8d38b --- /dev/null +++ b/fork-kit/desk/bootstrap.sh @@ -0,0 +1,60 @@ +#!/bin/sh +# desk bootstrap: installs the desk kit (a desktop's side of a fork fleet desk pool) of one release of this fork. +# Run as root on the desktop (on WSL2: inside the distro), after the owner wrote the machine file's `desk` section: +# +# sh desk-bootstrap.sh --repo FORK_OWNER/FORK --sha RELEASE_SHA [--fleet-config FILE] +# +# --fleet-config: the fleet config of the project whose jobs this desktop runs (that repository's file, at the commit +# its always-on host uses); without it, the release's own .github/fork-fleet.json (the fork's CI). +# It refuses a sha that is not a commit on the fork's protected release branch (ci-release), downloads exactly that +# commit, prints the sha and the hash of what it downloaded, and runs its installer (safe to run again). +# Needs: curl, tar, python3. Linux with systemd only (WSL2 with systemd on). +set -eu +umask 022 +PATH=/usr/sbin:/usr/bin:/sbin:/bin +export PATH +unset TMPDIR CURL_HOME SSL_CERT_FILE SSL_CERT_DIR || true + +die() { echo "desk-bootstrap: $*" >&2; exit 1; } + +repo='' sha='' branch='ci-release' api='https://api.github.com' codeload='https://codeload.github.com' +while [ $# -gt 0 ]; do + case "$1" in + --repo) repo="${2-}"; shift 2 ;; + --sha) sha="${2-}"; shift 2 ;; + --release-branch) branch="${2-}"; shift 2 ;; + --fleet-config) break ;; + *) die "unknown option $1 (see the top of this file)" ;; + esac +done +printf '%s\n' "$repo" | grep -Eqx '[A-Za-z0-9][A-Za-z0-9-]{0,38}/[A-Za-z0-9._-]{1,100}' || die "--repo must be OWNER/NAME" +printf '%s\n' "$sha" | grep -Eqx '[0-9a-f]{40}' || die "--sha must be the full 40-character release commit" +printf '%s\n' "$branch" | grep -Eqx '[A-Za-z0-9._/-]{1,100}' || die "bad --release-branch" +[ "$(id -u)" = 0 ] || die "run this as root" + +work="$(mktemp -d /tmp/desk-bootstrap.XXXXXXXX)" +trap 'rm -rf "${work:?}"' EXIT +trap 'exit 1' INT TERM HUP +fetch() { curl -fsSL --proto '=https' --tlsv1.2 --max-redirs 0 -H 'Accept: application/vnd.github+json' -o "$2" "$1"; } + +fetch "$api/repos/$repo/branches/$branch" "$work/branch.json" || die "cannot read branch $branch of $repo" +fetch "$api/repos/$repo/compare/$sha...$branch" "$work/compare.json" || die "cannot compare $sha with $branch" +python3 -I -B - "$work/branch.json" "$work/compare.json" "$sha" <<'PY' || die "$sha is not a release: it must be a commit on the protected branch $branch" +import json, sys +branch, compare = (json.load(open(p, encoding='utf-8')) for p in sys.argv[1:3]) +# base_commit is always the sha asked for; merge_base_commit is the newest commit both share: it is the sha itself +# only when the sha is on the branch +ok = (branch.get('protected') is True and compare.get('status') in ('ahead', 'identical') + and (compare.get('merge_base_commit') or {}).get('sha') == sys.argv[3]) +sys.exit(0 if ok else 1) +PY + +# codeload.github.com answers a commit tarball directly (no redirect) +fetch "$codeload/$repo/tar.gz/$sha" "$work/kit.tar.gz" || die "cannot download $repo at $sha" +mkdir "$work/src" +tar -xzf "$work/kit.tar.gz" -C "$work/src" --no-same-owner --no-same-permissions +top="$(find "$work/src" -mindepth 1 -maxdepth 1 -type d)" +[ "$(printf '%s\n' "$top" | wc -l)" = 1 ] && [ -n "$top" ] || die "the download does not have one top folder" +[ -f "$top/fork-kit/desk/sbin/desk-install" ] || die "release $sha has no fork-kit/desk (is it a release with the kit?)" +echo "desk-bootstrap: release $sha of $repo, download sha256 $(sha256sum "$work/kit.tar.gz" | cut -d' ' -f1)" +python3 -I -B "$top/fork-kit/desk/sbin/desk-install" install --source "$top" --sha "$sha" "$@" diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install new file mode 100755 index 0000000000000..0f71da2ff96df --- /dev/null +++ b/fork-kit/desk/sbin/desk-install @@ -0,0 +1,475 @@ +#!/usr/bin/python3 -IB +"""desk-install: a desktop's side of a fork fleet desk pool, as root (fork-kit/desk/README.md). + + desk-install install --source RELEASE_FOLDER --sha RELEASE_SHA [--fleet-config FILE] + desk-install check every part and wall, PASS or FAIL; reads only (safe while a job runs) + desk-install uninstall + +install reads the `desk` section of the machine file, makes the runner users, stages the fleet files of one release +(launcher, walls, desk agent, gate hook and the fleet config), checks the staged copy, stops the agent, moves each +file into /opt/fork-fleet, writes MANIFEST.sha256 and then SOURCE (last), writes the settings in /etc/fork-fleet and +the units, makes the desk key once, and starts the agent and the refresh timer. Safe to run again and to stop at +any step: a stopped run leaves SOURCE naming the old release (check says what differs) and the next run ends it. +""" +import argparse, contextlib, hashlib, importlib.util, json, os, re, shutil, subprocess, sys, tempfile + +PREFIX = '' # tests only: every fixed path below is read under this folder +ROOT_UID = 0 # tests only: who must run this and own the installed files +MACHINE = None # tests only: the machine file (None: the fixed path) +OPT, ETC, WORK = '/opt/fork-fleet', '/etc/fork-fleet', '/var/lib/fork-fleet' +STATE = ETC + '/desk-kit.json' +KEY = ETC + '/desk-key' +UNIT_DIR = '/etc/systemd/system' +SYSCTL = '/etc/sysctl.d/60-fork-fleet.conf' +HEADER = '# desk kit: written by desk-install. Do not edit: change the machine file and install again.\n' +FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh', + 'fork-fleet-job-paths.txt': '.github/scripts/fork-fleet-job-paths.txt', + 'fork-fleet-walls.py': '.github/scripts/fork-fleet-walls.py', + 'fork-fleet-desk-agent.py': '.github/scripts/fork-fleet-desk-agent.py', + 'fork-fleet-desk-wire.py': '.github/scripts/fork-fleet-desk-wire.py', + 'hook.sh': '.github/scripts/fork-fleet-hook.sh', 'fork-fleet.py': '.github/scripts/fork-fleet.py', + 'fork-fleet.json': '.github/fork-fleet.json', 'desk-install': 'fork-kit/desk/sbin/desk-install', + 'machine_file.py': 'fork-kit/common/machine_file.py', 'README.md': 'fork-kit/desk/README.md'} +WALLS = '.github/scripts/fork-fleet-walls' # copied whole: every OS's walls file and fixture +UNITS = {'fork-fleet-desk-agent.service': '.github/scripts/fork-fleet-desk-agent.service', + 't3code-fleet-refresh.service': '.github/scripts/fork-fleet-refresh.service', + 't3code-fleet-refresh.timer': '.github/scripts/fork-fleet-refresh.timer'} +AGENT, TIMER, REFRESH = 'fork-fleet-desk-agent.service', 't3code-fleet-refresh.timer', 't3code-fleet-refresh.service' +FOREIGN = (OPT + '/fork-fleet-launch.sh', ETC + '/launch.conf', ETC + '/desk.conf', KEY, + *(f'{UNIT_DIR}/{u}' for u in UNITS)) +CMD = {k: [k] for k in ('useradd', 'userdel', 'loginctl', 'systemctl', 'sysctl', 'ssh-keygen', 'ip', 'nft', 'bash', + 'id')} +CMD['python3'] = ['/usr/bin/python3'] +ENV = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'C.UTF-8'} +SHA = re.compile(r'[0-9a-f]{40}') +SHAPES = {'pool': r'[a-z0-9][a-z0-9_-]{0,39}', 'host': r'[A-Za-z0-9][A-Za-z0-9.-]{0,252}', + 'user': r'[a-z_][a-z0-9_-]{0,31}', 'netBase': r'10\.(0|[1-9][0-9]?|1[0-9]{2}|2[0-4][0-9]|25[0-5])', + 'hostKey': r'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI[A-Za-z0-9+/]{43}'} + + +class Refused(Exception): + pass + + +def p(path): + return PREFIX + path + + +def say(msg): + print(f'desk-install: {msg}', flush=True) + + +def run(name, *args, check=True, **kw): + r = subprocess.run([*CMD.get(name, [name]), *args], env=ENV, stdin=subprocess.DEVNULL, capture_output=True, + text=True, timeout=kw.pop('timeout', 120), **kw) + if check and r.returncode: + raise Refused(f'{name} {" ".join(args)} failed: {(r.stderr or r.stdout).strip()[:300]}') + return r + + +def py(script, *args): + return run('python3', '-I', '-B', script, *args, check=False) + + +def sha256(path): + with open(path, 'rb') as f: + return hashlib.sha256(f.read()).hexdigest() + + +def user_exists(name): + return run('id', '-u', name, check=False).returncode == 0 + + +def machine_section(): + here = os.path.dirname(os.path.realpath(__file__)) + found = [c for c in (os.path.join(here, 'machine_file.py'), os.path.join(here, '..', '..', 'common', + 'machine_file.py')) if os.path.isfile(c)] + spec = importlib.util.spec_from_file_location('machine_file', found[0]) + mf = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mf) + + def shape(key): + return lambda v: None if re.fullmatch(SHAPES[key], v) else f'not a {key} ({SHAPES[key]})' + + def within(lo, hi): + return lambda v: None if lo <= v <= hi else f'{lo} to {hi}' + try: + section = mf.read_section('desk', MACHINE, ROOT_UID) + return mf.fields(section, {'pool': (str, mf.REQUIRED, shape('pool')), 'host': (str, mf.REQUIRED, shape('host')), + 'port': (int, 22, within(1, 65535)), 'user': (str, 'fleet-desk', shape('user')), + 'hostKey': (str, mf.REQUIRED, shape('hostKey')), 'slots': (int, 1, within(1, 8)), + 'reserveMb': (int, None, within(0, 1000000)), + 'netBase': (str, None, shape('netBase'))}, 'desk') + except mf.MachineFileError as e: + raise Refused(f'machine file: {e}') from None + + +def read_state(): + try: + with open(p(STATE), encoding='utf-8') as f: + return json.load(f) + except FileNotFoundError: + return None + except (OSError, ValueError) as e: + raise Refused(f'{STATE} cannot be read ({e}): fix or remove it and run again') from None + + +def write_file(path, text, mode=0o644): + """Atomic (a temp file in the same folder, renamed over the old one); unchanged text is not rewritten.""" + real = p(path) + with contextlib.suppress(OSError): + with open(real, encoding='utf-8') as f: + if f.read() == text and os.stat(real).st_mode & 0o777 == mode: + return False + os.makedirs(os.path.dirname(real), mode=0o755, exist_ok=True) + fd, tmp = tempfile.mkstemp(dir=os.path.dirname(real), prefix='.' + os.path.basename(real) + '.') + try: + with os.fdopen(fd, 'w', encoding='utf-8') as f: + f.write(text) + f.flush() + os.fsync(f.fileno()) + os.chmod(tmp, mode) + os.replace(tmp, real) + except BaseException: + with contextlib.suppress(OSError): + os.unlink(tmp) + raise + return True + + +def save_state(state): + write_file(STATE, json.dumps(state, indent=2, sort_keys=True) + '\n', 0o600) + + +def ours(path): + try: + with open(p(path), encoding='utf-8') as f: + return f.readline() == HEADER + except (OSError, UnicodeDecodeError): + return False + + +def stage(source, sha, fleet_config, pool): + """The release's files in /opt/fork-fleet/.kit-stage-*, checked; returns (folder, {path: sha256}).""" + for name in os.listdir(p(OPT)): # a run that stopped while staging + if name.startswith('.kit-stage-'): + shutil.rmtree(os.path.join(p(OPT), name), ignore_errors=True) + tmp = tempfile.mkdtemp(prefix='.kit-stage-', dir=p(OPT)) + pairs = [(dst, os.path.join(source, src)) for dst, src in FILES.items()] + pairs += [(f'units/{u}', os.path.join(source, src)) for u, src in UNITS.items()] + pairs += [(f'fork-fleet-walls/{n}', os.path.join(source, WALLS, n)) for n in sorted(os.listdir(os.path.join( + source, WALLS)))] + manifest = {} + for rel, src in pairs: + src = fleet_config if rel == 'fork-fleet.json' and fleet_config else src + dst = os.path.join(tmp, rel) + os.makedirs(os.path.dirname(dst), mode=0o755, exist_ok=True) + shutil.copyfile(src, dst) + executable = rel in ('fork-fleet-launch.sh', 'hook.sh', 'desk-install') or ( + rel.startswith('fork-fleet-walls/') and not rel.endswith('.json')) + os.chmod(dst, 0o755 if executable else 0o644) + manifest[rel] = sha256(dst) + for d, _, _ in os.walk(tmp): + os.chmod(d, 0o755) + problems = [] + r = py(os.path.join(tmp, 'fork-fleet-walls.py'), 'os') + if r.returncode: + problems.append(f'fork-fleet-walls.py os: exit {r.returncode} {r.stderr.strip()[:200]}') + r = py(os.path.join(tmp, 'fork-fleet.py'), '--config', os.path.join(tmp, 'fork-fleet.json'), 'gate') + if 'no Runner.Worker above this hook' not in r.stdout + r.stderr: + problems.append(f'the gate does not load with this fleet config: {(r.stderr or r.stdout).strip()[:300]}') + with contextlib.suppress(OSError, ValueError, TypeError, AttributeError): + with open(os.path.join(tmp, 'fork-fleet.json'), encoding='utf-8') as f: + if pool not in [x.get('name') for x in json.load(f)['pools']]: + problems.append(f'the fleet config has no pool {pool!r} (the machine file\'s desk.pool)') + for rel in ('fork-fleet-launch.sh', 'hook.sh'): + if run('bash', '-n', os.path.join(tmp, rel), check=False).returncode: + problems.append(f'{rel}: bash -n fails') + if py(os.path.join(tmp, 'fork-fleet-desk-agent.py'), '--help').returncode: + problems.append('fork-fleet-desk-agent.py --help fails') + if problems: + shutil.rmtree(tmp, ignore_errors=True) + raise Refused(f'the release failed its self-test ({OPT} is unchanged):\n ' + '\n '.join(problems)) + return tmp, manifest + + +def read_manifest(): + with contextlib.suppress(OSError): + with open(p(OPT + '/MANIFEST.sha256'), encoding='utf-8') as f: + return dict(reversed(line.rstrip('\n').split(' ', 1)) for line in f if ' ' in line) + return {} + + +def swap(tmp, manifest, sha): + """Each changed file renamed into /opt/fork-fleet (the agent stopped meanwhile, so no verb runs from half-old, + half-new files); files of the old release that are gone, removed. Unchanged files are left as they are.""" + def same(rel): + with contextlib.suppress(OSError): + return sha256(p(f'{OPT}/{rel}')) == manifest[rel] + changed = [rel for rel in sorted(manifest) if not same(rel)] + gone = sorted(set(read_manifest()) - set(manifest)) + if changed or gone: + run('systemctl', 'stop', AGENT, check=False) + for rel in changed: + os.makedirs(os.path.dirname(p(f'{OPT}/{rel}')), mode=0o755, exist_ok=True) + os.replace(os.path.join(tmp, rel), p(f'{OPT}/{rel}')) + for rel in gone: + with contextlib.suppress(FileNotFoundError): + os.unlink(p(f'{OPT}/{rel}')) + write_file(OPT + '/MANIFEST.sha256', ''.join(f'{d} {r}\n' for r, d in sorted(manifest.items()))) + write_file(OPT + '/SOURCE', sha + '\n') + shutil.rmtree(tmp, ignore_errors=True) + + +def lines_of(path): + with contextlib.suppress(FileNotFoundError), open(p(path), encoding='utf-8') as f: + return f.read().splitlines() + return [] + + +def make_users(users, state): + for u in users: + if user_exists(u): + continue + if u not in state['createdUsers']: # recorded first: a run stopped here still removes it on uninstall + state['createdUsers'].append(u) + save_state(state) + run('useradd', '--no-create-home', '--home-dir', '/nonexistent', '--user-group', '--shell', + '/usr/sbin/nologin', u) + say(f'made runner user {u}') + for u in users: + run('loginctl', 'disable-linger', u, check=False) + for f in ('/etc/cron.deny', '/etc/at.deny'): + if os.path.exists(p(f.replace('.deny', '.allow'))): + continue + lines = lines_of(f) + if u not in lines: + write_file(f, '\n'.join(lines + [u]) + '\n') + + +def settings(m, users): + known = (m['host'] if m['port'] == 22 else f'[{m["host"]}]:{m["port"]}') + ' ' + m['hostKey'] + files = {ETC + '/launch.conf': (f'RUNNER_USERS={",".join(users)}\n' + + (f'RESERVE_MB={m["reserveMb"]}\n' if m['reserveMb'] is not None else '') + + (f'NET_BASE={m["netBase"]}\n' if m['netBase'] else ''), 0o600), + ETC + '/desk.conf': (f'POOL={m["pool"]}\nHOST={m["host"]}\nPORT={m["port"]}\nUSER={m["user"]}\n', 0o600), + ETC + '/desk-known_hosts': (known + '\n', 0o644), + ETC + '/resolv.conf': ('nameserver 1.1.1.1\nnameserver 9.9.9.9\n', 0o644), + SYSCTL: ('net.ipv4.ip_forward = 1\n', 0o644)} + return {path: (HEADER + text, mode) for path, (text, mode) in files.items()} + + +def make_key(pool): + """The desk key, once: made under a temp name, then renamed (the public half first).""" + if os.path.exists(p(KEY)) and os.path.exists(p(KEY + '.pub')): + return + new = p(ETC + '/.desk-key.new') + for f in (new, new + '.pub'): + with contextlib.suppress(FileNotFoundError): + os.unlink(f) + if os.path.exists(p(KEY)): + raise Refused(f'{KEY} is there without {KEY}.pub: remove both and run again (a new key)') + run('ssh-keygen', '-q', '-t', 'ed25519', '-N', '', '-C', f'desk-{pool}', '-f', new) + os.replace(new + '.pub', p(KEY + '.pub')) + os.replace(new, p(KEY)) + + +def install(a, stop_after=None): + def done(step): + if step == stop_after: + raise SystemExit(f'stopped after {step} (test)') + + if os.geteuid() != ROOT_UID: + raise Refused('run this as root') + if not SHA.fullmatch(a.sha or ''): + raise Refused('--sha must be the full 40-character release commit') + missing = [s for s in [*FILES.values(), *UNITS.values(), WALLS] if not os.path.exists(os.path.join(a.source, s))] + if missing: + raise Refused(f'--source {a.source}: {missing[0]} is missing (point it at the release\'s top folder)') + if a.fleet_config and not os.path.isfile(a.fleet_config): + raise Refused(f'--fleet-config {a.fleet_config}: no such file') + m = machine_section() + r = py(os.path.join(a.source, FILES['fork-fleet-walls.py']), 'os') + if r.returncode == 3: + raise Refused(f'this OS ({r.stdout.strip() or "unknown"}) is not supported yet: it has no walls file') + state = read_state() + if state is None: + there = [f for f in FOREIGN if os.path.lexists(p(f))] + if there: + raise Refused(f'{there[0]} is from an install this kit did not make: uninstall that first (README)') + state = {'version': 1, 'createdUsers': [], 'fleetConfig': 'release'} + for u in UNITS: + if os.path.lexists(p(f'{UNIT_DIR}/{u}')) and not ours(f'{UNIT_DIR}/{u}'): + raise Refused(f'{UNIT_DIR}/{u} was not written by this kit: remove it and run again') + config = os.path.realpath(a.fleet_config) if a.fleet_config else None + if config is None and state['fleetConfig'] == 'project': + config = p(OPT + '/fork-fleet.json') # the project's config from the last install, checked again below + users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] + done('check') + for d, mode in ((OPT, 0o755), (OPT + '/template', 0o755), (ETC, 0o700)): + os.makedirs(p(d), mode=mode, exist_ok=True) + os.chmod(p(d), mode) + save_state(state) + make_users(users, state) + done('users') + tmp, manifest = stage(a.source, a.sha, config, m['pool']) + done('stage') + swap(tmp, manifest, a.sha) + state.update(sha=a.sha, fleetConfig='project' if config else 'release') + save_state(state) + done('files') + for path, (text, mode) in settings(m, users).items(): + write_file(path, text, mode) + run('sysctl', '-q', '-p', p(SYSCTL)) + make_key(m['pool']) + done('settings') + for u in UNITS: + with open(p(f'{OPT}/units/{u}'), encoding='utf-8') as f: + write_file(f'{UNIT_DIR}/{u}', HEADER + f.read()) + r = py(p(OPT + '/fork-fleet-desk-agent.py'), 'check') + if r.returncode: + raise Refused(f'the desk agent refuses its settings (not started): {r.stderr.strip()[:300]}') + run('systemctl', 'daemon-reload') + run('systemctl', 'enable', '--now', TIMER) + if not os.path.exists(p(OPT + '/template/runner-template/.fork-fleet-version')): + run(p(OPT + '/fork-fleet-launch.sh'), 'prune', check=False) # makes the launch lock refresh opens + run('systemctl', 'start', '--no-block', REFRESH) # the first runner template, in the background + run('systemctl', 'enable', AGENT) + run('systemctl', 'restart', AGENT) + done('units') + with open(p(KEY + '.pub'), encoding='utf-8') as f: + pub = f.read().strip() + say(f'installed release {a.sha} for pool {m["pool"]} ({len(users)} slot(s)); the agent dials {m["host"]}') + say(f'its public key, for the always-on host (fork-fleet-desk-key.py add {m["pool"]}):\n{pub}') + say('then: desk-install check (every line PASS)') + return 0 + + +def check(): + rows = [] + + def row(ok, name, what): + rows.append(ok) + print(f'{"PASS" if ok else "FAIL"} {name} {what}', flush=True) + problems = installed_problems() + for x in problems: + row(False, 'kit', x) + if not problems: + row(True, 'kit', f'the installed files are release {read_state()["sha"]}\'s') + walls = py(p(OPT + '/fork-fleet-walls.py'), 'describe') + lines = [x for x in walls.stdout.splitlines() if x.startswith(('PASS ', 'FAIL '))] + for line in lines: + print(line) + rows.append(line.startswith('PASS')) + if walls.returncode and not any(x.startswith('FAIL') for x in lines): + row(False, 'walls', f'exit {walls.returncode}: {walls.stderr.strip()[:200]}') + r = py(p(OPT + '/fork-fleet-desk-agent.py'), 'check') + row(r.returncode == 0, 'agent-settings', (r.stdout or r.stderr).strip()[:200]) + for unit in (AGENT, TIMER): + on = run('systemctl', 'is-active', '--quiet', unit, check=False).returncode == 0 + row(on and run('systemctl', 'is-enabled', '--quiet', unit, check=False).returncode == 0, 'unit', + f'{unit} enabled and active') + row(os.path.exists(p(OPT + '/template/runner-template/.fork-fleet-version')), 'template', + 'the runner template is there (the refresh timer makes it)') + try: + m = machine_section() + users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] + row(True, 'machine-file', f'desk section: pool {m["pool"]}, {m["slots"]} slot(s), always-on host {m["host"]}') + except Refused as e: + users = [] + row(False, 'machine-file', str(e)[:300]) + for u in users: + row(user_exists(u), 'runner-user', f'{u} exists') + try: + with open(p('/proc/sys/net/ipv4/ip_forward'), encoding='ascii') as f: + forward = f.read().strip() == '1' + except OSError: + forward = False + row(forward, 'ip-forward', 'IPv4 forwarding is on (the runners\' namespaces need it)') + fails = rows.count(False) + print('ALL PASS' if not fails else f'{fails} FAIL') + return 0 if not fails else 1 + + +def installed_problems(): + """What differs from the install record: /opt/fork-fleet against its manifest, SOURCE, the units.""" + problems, state = [], read_state() or {} + manifest = read_manifest() + if not manifest or not state.get('sha'): + return [f'{OPT}/MANIFEST.sha256 or {STATE} is missing: not installed (or a first install stopped)'] + for rel, digest in manifest.items(): + try: + ok = sha256(p(f'{OPT}/{rel}')) == digest and os.stat(p(f'{OPT}/{rel}')).st_uid == ROOT_UID + except OSError: + ok = False + if not ok: + problems.append(f'{OPT}/{rel}: differs from what was installed') + try: + with open(p(OPT + '/SOURCE'), encoding='utf-8') as f: + source = f.read().strip() + except OSError: + source = None + if source != state['sha']: + problems.append(f'{OPT}/SOURCE names {source}, the record {state["sha"]}: install again') + problems += [f'{UNIT_DIR}/{u}: missing or not the kit\'s' for u in UNITS if not ours(f'{UNIT_DIR}/{u}')] + return problems + + +def uninstall(): + if os.geteuid() != ROOT_UID: + raise Refused('run this as root') + state = read_state() + if state is None: + say(f'no {STATE}: nothing here was installed by this kit (nothing changed)') + return 0 + for unit in (AGENT, TIMER, REFRESH): + run('systemctl', 'disable', '--now', unit, check=False) + launcher = p(OPT + '/fork-fleet-launch.sh') + if os.path.exists(launcher): + for name in run(launcher, 'active', check=False).stdout.split(): + run(launcher, 'stop', name, check=False) + run(launcher, 'prune', check=False) + for u in state['createdUsers']: + run('ip', 'netns', 'delete', f'fork-fleet-{u}', check=False) + run('nft', 'delete', 'table', 'inet', 'fork_fleet', check=False) + for u in UNITS: + if ours(f'{UNIT_DIR}/{u}'): + os.unlink(p(f'{UNIT_DIR}/{u}')) + run('systemctl', 'daemon-reload', check=False) + if ours(SYSCTL): + os.unlink(p(SYSCTL)) + for d in (OPT, WORK): + shutil.rmtree(p(d), ignore_errors=True) + for u in state['createdUsers']: + if user_exists(u): + run('userdel', u) + for f in ('/etc/cron.deny', '/etc/at.deny'): + if u in lines_of(f): + write_file(f, ''.join(x + '\n' for x in lines_of(f) if x != u)) + shutil.rmtree(p(ETC), ignore_errors=True) # the record (desk-kit.json) goes last, with the folder + say('uninstalled; on the always-on host remove its key: fork-fleet-desk-key.py remove POOL') + return 0 + + +def main(argv, stop_after=None): + ap = argparse.ArgumentParser(prog='desk-install', description=__doc__.split('\n\n')[0]) + sub = ap.add_subparsers(dest='cmd', required=True) + i = sub.add_parser('install') + i.add_argument('--source', required=True) + i.add_argument('--sha', required=True) + i.add_argument('--fleet-config') + for name in ('check', 'uninstall'): + sub.add_parser(name) + a = ap.parse_args(argv) + os.umask(0o022) + try: + if a.cmd == 'install': + return install(a, stop_after) + return check() if a.cmd == 'check' else uninstall() + except Refused as e: + print(f'desk-install: {e}', file=sys.stderr) + return 1 + + +if __name__ == '__main__': + sys.exit(main(sys.argv[1:])) diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py new file mode 100644 index 0000000000000..65673b609122f --- /dev/null +++ b/fork-kit/desk/tests/install.test.py @@ -0,0 +1,479 @@ +"""desk-install on a fake host in a temporary folder: every fixed path under PREFIX, the current user as root, and +every system command (useradd, systemctl, ssh-keygen, ...) a fake that records its arguments. A fake release folder +holds stubs of the fleet files. No root, no systemd, no network, no real user. + + python3 -I -B fork-kit/desk/tests/install.test.py +""" +import contextlib, importlib.machinery, importlib.util, io, json, os, shutil, stat, subprocess, sys, tempfile, unittest + +KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +REPO = os.path.dirname(os.path.dirname(KIT)) +SHA1, SHA2 = 'a' * 40, 'b' * 40 +HOST_KEY = 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI' + 'A' * 43 +MACHINE = {'version': 1, 'desk': {'pool': 'desktop', 'host': 'fleet-host.example', 'hostKey': HOST_KEY, 'slots': 2}} + +# One fake for every system command: argv[1] is the command's name. Its state lives in files beside it. +FAKE = r''' +import os, sys +here = os.path.dirname(os.path.abspath(__file__)) +name, args = sys.argv[1], sys.argv[2:] +with open(os.path.join(here, 'calls.log'), 'a') as f: + f.write(' '.join([name, *args]) + '\n') +def lines(n): + try: + return open(os.path.join(here, n)).read().split() + except FileNotFoundError: + return [] +def put(n, items): + open(os.path.join(here, n), 'w').write('\n'.join(items) + '\n') +fail = lines('fail') +if name in fail: + put('fail', [x for x in fail if x != name]) # fails once + sys.exit(9) +if name in lines('fail-after'): # does its work, then dies (a run stopped right after it) + put('fail-after', [x for x in lines('fail-after') if x != name]) + put('users', lines('users') + [args[-1]]) + sys.exit(9) +if name == 'id': + sys.exit(0 if args[-1] in lines('users') else 1) +if name == 'useradd': + put('users', lines('users') + [args[-1]]) +if name == 'userdel': + put('users', [u for u in lines('users') if u != args[-1]]) +if name == 'ssh-keygen': + key = args[args.index('-f') + 1] + open(key, 'w').write('PRIVATE\n'); os.chmod(key, 0o600) + open(key + '.pub', 'w').write('ssh-ed25519 AAAAfake desk-x\n') +if name == 'systemctl' and args[0] in ('is-active', 'is-enabled'): + sys.exit(1 if args[-1] in lines('down') else 0) +''' + +STUBS = { + '.github/scripts/fork-fleet-launch.sh': '#!/bin/bash\necho "launch $*" >> {log}\n' + '[ "$1" = active ] && cat {here}/active 2>/dev/null\nexit 0\n', + '.github/scripts/fork-fleet-walls.py': '#!/usr/bin/python3 -I\nimport sys, os\n' + 'v = sys.argv[1]\nif v == "os":\n' + ' print("linux"); sys.exit(int(open("{here}/os-exit").read()))\n' + 'bad = os.path.exists("{here}/wall-fails")\n' + 'print("PASS os linux")\nprint(("FAIL vsock closed: open" if bad else' + ' "PASS vsock closed"))\nsys.exit(1 if bad else 0)\n', + '.github/scripts/fork-fleet.py': 'import sys\nprint("::error::no Runner.Worker above this hook")\nsys.exit(1)\n', + '.github/scripts/fork-fleet-desk-agent.py': 'import os, sys\nif sys.argv[-1] == "check" and ' + 'os.path.exists("{here}/agent-bad"):\n' + ' print("bad settings", file=sys.stderr); sys.exit(2)\n' + 'print("settings ok")\n', + '.github/scripts/fork-fleet-desk-wire.py': '# wire\n', + '.github/scripts/fork-fleet-hook.sh': '#!/bin/bash\nexit 0\n', + '.github/scripts/fork-fleet-job-paths.txt': '/usr\n', + '.github/fork-fleet.json': json.dumps({'pools': [{'name': 'desktop'}, {'name': 'vm'}]}), + '.github/scripts/fork-fleet-walls/linux': '#!/bin/sh\n', + '.github/scripts/fork-fleet-walls/linux.fixture.json': '{}\n', + '.github/scripts/fork-fleet-walls/wsl2': '#!/bin/sh\n', + '.github/scripts/fork-fleet-walls/wsl2.fixture.json': '{}\n', + '.github/scripts/fork-fleet-desk-agent.service': '# agent unit\n[Service]\n', + '.github/scripts/fork-fleet-refresh.service': '# refresh unit\n[Service]\n', + '.github/scripts/fork-fleet-refresh.timer': '# refresh timer\n[Timer]\n', +} + + +def slurp(path, mode='r'): + with open(path, mode) as f: + return f.read() + + +def load(): + path = os.path.join(KIT, 'sbin', 'desk-install') + loader = importlib.machinery.SourceFileLoader('desk_install', path) + mod = importlib.util.module_from_spec(importlib.util.spec_from_loader('desk_install', loader)) + loader.exec_module(mod) + return mod + + +class Desk(unittest.TestCase): + def setUp(self): + self.umask = os.umask(0o022) + self.tmp = os.path.realpath(tempfile.mkdtemp(prefix='desk-install-test-')) + self.addCleanup(self.close) + self.root, self.fakes = os.path.join(self.tmp, 'root'), os.path.join(self.tmp, 'fakes') + os.makedirs(self.fakes) + with open(os.path.join(self.fakes, 'fake.py'), 'w') as f: + f.write(FAKE) + self.write(os.path.join(self.fakes, 'os-exit'), '0') + self.source = self.release() + self.m = load() + self.m.PREFIX, self.m.ROOT_UID = self.root, os.geteuid() + self.m.MACHINE = os.path.join(self.tmp, 'machine', 'machine.json') + fake = os.path.join(self.fakes, 'fake.py') + self.m.CMD = {k: [sys.executable, '-I', fake, k] for k in self.m.CMD} + self.m.CMD.update(python3=[sys.executable, '-I', '-B'], bash=['bash']) + self.machine(MACHINE) + self.write(self.path('/proc/sys/net/ipv4/ip_forward'), '1\n') + + def close(self): + os.umask(self.umask) + shutil.rmtree(self.tmp, ignore_errors=True) + + def write(self, path, text, mode=0o644): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, 'w') as f: + f.write(text) + os.chmod(path, mode) + + def release(self, name='release', extra=None): + top = os.path.join(self.tmp, name) + for rel, text in {**STUBS, **(extra or {})}.items(): + if text is not None: + self.write(os.path.join(top, rel), text.replace('{log}', os.path.join(self.fakes, 'calls.log')) + .replace('{here}', self.fakes), 0o755) + for rel in ('fork-kit/desk/sbin/desk-install', 'fork-kit/desk/README.md', 'fork-kit/common/machine_file.py'): + self.write(os.path.join(top, rel), slurp(os.path.join(REPO, rel)), 0o755) + return top + + def machine(self, data): + path = self.m.MACHINE + os.makedirs(os.path.dirname(path), mode=0o755, exist_ok=True) + self.write(path, json.dumps(data) if not isinstance(data, str) else data) + + def path(self, p): + return self.root + p + + def calls(self): + with contextlib.suppress(FileNotFoundError): + with open(os.path.join(self.fakes, 'calls.log')) as f: + return f.read().splitlines() + return [] + + def clear_calls(self): + with contextlib.suppress(FileNotFoundError): + os.unlink(os.path.join(self.fakes, 'calls.log')) + + def run_main(self, *argv, stop_after=None): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + try: + code = self.m.main(list(argv), stop_after=stop_after) + except SystemExit as e: + code = str(e) + return code, out.getvalue(), err.getvalue() + + def install(self, *extra, sha=SHA1, stop_after=None, source=None): + return self.run_main('install', '--source', source or self.source, '--sha', sha, *extra, + stop_after=stop_after) + + def read(self, p): + with open(self.path(p)) as f: + return f.read() + + def tree(self): + """Every path under the fake root with its kind, mode and contents (no times).""" + out = {} + for dirpath, dirnames, filenames in os.walk(self.root): + for name in dirnames + filenames: + full = os.path.join(dirpath, name) + st = os.lstat(full) + body = slurp(full, 'rb') if stat.S_ISREG(st.st_mode) else None + out[full[len(self.root):]] = (stat.S_IFMT(st.st_mode), st.st_mode & 0o777, body) + return out + + def template(self): + self.write(self.path('/opt/fork-fleet/template/runner-template/.fork-fleet-version'), '2.330.0\n') + + # ------------------------------------------------------------------------------------------------- install + + def test_fresh_install_files_settings_users_key_units(self): + code, out, err = self.install() + self.assertEqual(code, 0, err) + self.assertEqual(self.read('/opt/fork-fleet/SOURCE'), SHA1 + '\n') + for rel in ('fork-fleet-launch.sh', 'hook.sh', 'fork-fleet.py', 'fork-fleet.json', 'fork-fleet-walls.py', + 'fork-fleet-walls/linux', 'fork-fleet-walls/wsl2.fixture.json', 'desk-install', 'machine_file.py'): + self.assertTrue(os.path.isfile(self.path(f'/opt/fork-fleet/{rel}')), rel) + self.assertEqual(stat.S_IMODE(os.stat(self.path('/opt/fork-fleet/hook.sh')).st_mode), 0o755) + self.assertEqual(stat.S_IMODE(os.stat(self.path('/opt/fork-fleet/fork-fleet.json')).st_mode), 0o644) + self.assertFalse([n for n in os.listdir(self.path('/opt/fork-fleet')) if n.startswith('.kit-stage')]) + launch = self.read('/etc/fork-fleet/launch.conf') + self.assertTrue(launch.startswith(self.m.HEADER)) + self.assertIn('RUNNER_USERS=fork-runner-1,fork-runner-2\n', launch) + self.assertNotIn('RESERVE_MB', launch) + self.assertIn('POOL=desktop\nHOST=fleet-host.example\nPORT=22\nUSER=fleet-desk\n', + self.read('/etc/fork-fleet/desk.conf')) + self.assertTrue(self.read('/etc/fork-fleet/desk-known_hosts').endswith(f'fleet-host.example {HOST_KEY}\n')) + for f, mode in (('launch.conf', 0o600), ('desk.conf', 0o600), ('desk-key', 0o600), ('desk-kit.json', 0o600)): + self.assertEqual(stat.S_IMODE(os.stat(self.path(f'/etc/fork-fleet/{f}')).st_mode), mode, f) + self.assertEqual(stat.S_IMODE(os.stat(self.path('/etc/fork-fleet')).st_mode), 0o700) + self.assertIn('ssh-ed25519 AAAAfake desk-x', out) + self.assertEqual(self.read('/etc/sysctl.d/60-fork-fleet.conf'), self.m.HEADER + 'net.ipv4.ip_forward = 1\n') + for u in self.m.UNITS: + self.assertTrue(self.read(f'/etc/systemd/system/{u}').startswith(self.m.HEADER), u) + calls = self.calls() + self.assertEqual([c for c in calls if c.startswith('useradd')], [ + f'useradd --no-create-home --home-dir /nonexistent --user-group --shell /usr/sbin/nologin fork-runner-{i}' + for i in (1, 2)]) + self.assertEqual(self.read('/etc/cron.deny'), 'fork-runner-1\nfork-runner-2\n') + self.assertEqual(json.loads(self.read('/etc/fork-fleet/desk-kit.json'))['createdUsers'], + ['fork-runner-1', 'fork-runner-2']) + order = [c for c in calls if c.startswith(('systemctl', 'launch'))] + self.assertEqual(order, ['systemctl stop fork-fleet-desk-agent.service', 'systemctl daemon-reload', + 'systemctl enable --now t3code-fleet-refresh.timer', 'launch prune', + 'systemctl start --no-block t3code-fleet-refresh.service', + 'systemctl enable fork-fleet-desk-agent.service', + 'systemctl restart fork-fleet-desk-agent.service']) + self.template() + code, out, _ = self.run_main('check') + self.assertEqual(code, 0, out) + self.assertTrue(out.endswith('ALL PASS\n'), out) + self.assertIn('PASS vsock closed', out) + + def test_runs_again_and_changes_nothing(self): + self.assertEqual(self.install()[0], 0) + self.template() + before = self.tree() + self.clear_calls() + code, _, err = self.install() + self.assertEqual(code, 0, err) + self.assertEqual(self.tree(), before) + calls = self.calls() + self.assertFalse([c for c in calls if c.startswith(('useradd', 'ssh-keygen', 'systemctl stop'))], calls) + self.assertNotIn('systemctl start --no-block t3code-fleet-refresh.service', calls) # the template is there + + def test_stopped_at_any_step_the_next_run_ends_it(self): + self.assertEqual(self.install()[0], 0) + want = self.tree() + for step in ('check', 'users', 'stage', 'files', 'settings', 'units'): + with self.subTest(step=step): + shutil.rmtree(self.root) + with contextlib.suppress(FileNotFoundError): + os.unlink(os.path.join(self.fakes, 'users')) + self.write(self.path('/proc/sys/net/ipv4/ip_forward'), '1\n') + code, _, _ = self.install(stop_after=step) + self.assertIn('stopped after', str(code)) + if step in ('stage', 'files', 'settings'): + self.assertNotEqual(self.run_main('check')[0], 0) + code, _, err = self.install() + self.assertEqual(code, 0, err) + self.assertEqual(self.tree(), want) + + def test_upgrade_replaces_changed_files_and_removes_gone_ones(self): + self.assertEqual(self.install()[0], 0) + launcher = os.stat(self.path('/opt/fork-fleet/fork-fleet-launch.sh')).st_ino + new = self.release('release2', {'.github/scripts/fork-fleet-walls/wsl2': None, + '.github/scripts/fork-fleet-walls/wsl2.fixture.json': None, + '.github/scripts/fork-fleet-hook.sh': '#!/bin/bash\nexit 0 # new\n'}) + self.clear_calls() + code, _, err = self.install(sha=SHA2, source=new) + self.assertEqual(code, 0, err) + self.assertEqual(self.read('/opt/fork-fleet/SOURCE'), SHA2 + '\n') + self.assertFalse(os.path.exists(self.path('/opt/fork-fleet/fork-fleet-walls/wsl2'))) + self.assertIn('# new', self.read('/opt/fork-fleet/hook.sh')) + self.assertEqual(os.stat(self.path('/opt/fork-fleet/fork-fleet-launch.sh')).st_ino, launcher) # unchanged + calls = self.calls() + self.assertLess(calls.index('systemctl stop fork-fleet-desk-agent.service'), + calls.index('systemctl restart fork-fleet-desk-agent.service')) + self.assertEqual(json.loads(self.read('/etc/fork-fleet/desk-kit.json'))['sha'], SHA2) + + def test_project_fleet_config_is_used_and_kept(self): + project = os.path.join(self.tmp, 'project.json') + self.write(project, json.dumps({'pools': [{'name': 'desktop'}], 'project': True})) + code, _, err = self.install('--fleet-config', project) + self.assertEqual(code, 0, err) + self.assertIn('"project": true', self.read('/opt/fork-fleet/fork-fleet.json')) + code, _, err = self.install(sha=SHA2) # no --fleet-config: the project's stays + self.assertEqual(code, 0, err) + self.assertIn('"project": true', self.read('/opt/fork-fleet/fork-fleet.json')) + + def test_refusals_change_nothing(self): + cases = { + 'not a release sha': (lambda: None, ['--sha', 'abc'], '--sha must be'), + 'no machine file': (lambda: os.unlink(self.m.MACHINE), [], 'machine file'), + 'no desk section': (lambda: self.machine({'version': 1}), [], 'no "desk" section'), + 'bad host key': (lambda: self.machine({'version': 1, 'desk': {**MACHINE['desk'], + 'hostKey': 'ssh-rsa AAAA'}}), [], 'hostKey'), + 'too many slots': (lambda: self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'slots': 9}}), [], + 'slots'), + 'bad net base': (lambda: self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'netBase': '10.256'}}), + [], 'netBase'), + 'unknown key': (lambda: self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'token': 'x'}}), [], + 'token'), + 'os not supported': (lambda: self.write(os.path.join(self.fakes, 'os-exit'), '3'), [], + 'not supported yet'), + 'an install the kit did not make': ( + lambda: self.write(self.path('/opt/fork-fleet/fork-fleet-launch.sh'), 'old'), [], + 'this kit did not make'), + 'a foreign unit': (lambda: self.write(self.path('/etc/systemd/system/fork-fleet-desk-agent.service'), 'x'), + [], 'this kit did not make'), + 'no such fleet config': (lambda: None, ['--fleet-config', '/nonexistent/x.json'], 'no such file'), + } + for name, (setup, args, msg) in cases.items(): + with self.subTest(name): + shutil.rmtree(self.root, ignore_errors=True) + os.makedirs(self.root) + self.machine(MACHINE) + self.write(os.path.join(self.fakes, 'os-exit'), '0') + setup() + before = self.tree() + self.clear_calls() + argv = ['install', '--source', self.source, '--sha', SHA1, *args] + if '--sha' in args: + argv = ['install', '--source', self.source, *args] + code, _, err = self.run_main(*argv) + self.assertEqual(code, 1) + self.assertIn(msg, err) + self.assertEqual(self.tree(), before) + self.assertFalse([c for c in self.calls() if not c.startswith('id ')], self.calls()) + + def test_bad_release_is_not_installed(self): + self.assertEqual(self.install()[0], 0) + before = {k: v for k, v in self.tree().items() if k != '/etc/fork-fleet/desk-kit.json'} + for name, extra, msg in ( + ('no pool', {'.github/fork-fleet.json': json.dumps({'pools': [{'name': 'vm'}]})}, "no pool 'desktop'"), + ('gate does not load', {'.github/scripts/fork-fleet.py': 'raise SystemExit("bad config")\n'}, + 'the gate does not load'), + ('launcher syntax', {'.github/scripts/fork-fleet-launch.sh': '#!/bin/bash\nif then\n'}, 'bash -n')): + with self.subTest(name): + bad = self.release('bad-' + name.replace(' ', '-'), extra) + code, _, err = self.install(sha=SHA2, source=bad) + self.assertEqual(code, 1) + self.assertIn(msg, err) + after = {k: v for k, v in self.tree().items() if k != '/etc/fork-fleet/desk-kit.json'} + self.assertEqual(after, before) + + def test_key_is_made_once_and_a_half_key_is_refused(self): + self.assertEqual(self.install()[0], 0) + os.unlink(self.path('/etc/fork-fleet/desk-key.pub')) + code, _, err = self.install() + self.assertEqual(code, 1) + self.assertIn('without', err) + os.unlink(self.path('/etc/fork-fleet/desk-key')) + self.write(self.path('/etc/fork-fleet/.desk-key.new'), 'torn') # a keygen that stopped + code, _, err = self.install() + self.assertEqual(code, 0, err) + self.assertEqual(self.read('/etc/fork-fleet/desk-key'), 'PRIVATE\n') + self.assertFalse(os.path.exists(self.path('/etc/fork-fleet/.desk-key.new'))) + + def test_reserve_and_port_reach_the_settings(self): + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'reserveMb': 4096, 'port': 2222, 'slots': 1, + 'netBase': '10.77'}}) + self.assertEqual(self.install()[0], 0) + self.assertIn('RUNNER_USERS=fork-runner-1\nRESERVE_MB=4096\nNET_BASE=10.77\n', + self.read('/etc/fork-fleet/launch.conf')) + self.assertTrue(self.read('/etc/fork-fleet/desk-known_hosts').endswith( + f'[fleet-host.example]:2222 {HOST_KEY}\n')) + + def test_existing_user_is_used_not_recorded(self): + self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n') + self.assertEqual(self.install()[0], 0) + self.assertEqual(json.loads(self.read('/etc/fork-fleet/desk-kit.json'))['createdUsers'], ['fork-runner-2']) + self.assertEqual(self.run_main('uninstall')[0], 0) + self.assertEqual(slurp(os.path.join(self.fakes, 'users')).split(), ['fork-runner-1']) + + def test_a_user_made_by_a_run_that_died_is_removed_by_uninstall(self): + self.write(os.path.join(self.fakes, 'fail-after'), 'useradd\n') + code, _, _ = self.install() + self.assertEqual(code, 1) + self.assertEqual(slurp(os.path.join(self.fakes, 'users')).split(), ['fork-runner-1']) + self.assertEqual(self.run_main('uninstall')[0], 0) + self.assertEqual(slurp(os.path.join(self.fakes, 'users')).split(), []) + + def test_agent_refusing_its_settings_starts_nothing(self): + self.write(os.path.join(self.fakes, 'agent-bad'), '') + code, _, err = self.install() + self.assertEqual(code, 1) + self.assertIn('the desk agent refuses its settings', err) + self.assertFalse([c for c in self.calls() if c.startswith(('systemctl enable', 'systemctl restart'))]) + + # --------------------------------------------------------------------------------------------------- check + + def test_check_names_each_failure(self): + self.assertEqual(self.install()[0], 0) + self.write(os.path.join(self.fakes, 'wall-fails'), '') + self.write(os.path.join(self.fakes, 'agent-bad'), '') + self.write(os.path.join(self.fakes, 'down'), 'fork-fleet-desk-agent.service\n') + self.write(self.path('/opt/fork-fleet/hook.sh'), '#!/bin/bash\nevil\n', 0o755) + self.write(self.path('/proc/sys/net/ipv4/ip_forward'), '0\n') + os.unlink(os.path.join(self.fakes, 'users')) + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + for line in ('FAIL kit /opt/fork-fleet/hook.sh: differs', 'FAIL vsock closed: open', + 'FAIL agent-settings bad settings', 'FAIL unit fork-fleet-desk-agent.service', + 'PASS unit t3code-fleet-refresh.timer', 'FAIL template', 'FAIL runner-user fork-runner-1', + 'FAIL ip-forward'): + self.assertIn(line, out) + self.assertTrue(out.endswith('8 FAIL\n'), out) + + def test_check_before_install_fails_without_writing(self): + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + self.assertIn('FAIL kit', out) + self.assertFalse(os.path.exists(self.path('/opt'))) + + # ----------------------------------------------------------------------------------------------- uninstall + + def test_uninstall_leaves_nothing_of_its_own(self): + self.write(self.path('/etc/cron.deny'), 'someone\n') + self.write(self.path('/etc/systemd/system/other.service'), 'x') + self.write(os.path.join(self.fakes, 'users'), 'someone\n') + self.assertEqual(self.install()[0], 0) + self.write(os.path.join(self.fakes, 'active'), 'desktop-1\n') + self.write(self.path('/var/lib/fork-fleet/runners/x'), '') + self.clear_calls() + code, out, err = self.run_main('uninstall') + self.assertEqual(code, 0, err) + calls = self.calls() + for c in ('systemctl disable --now fork-fleet-desk-agent.service', 'launch active', 'launch stop desktop-1', + 'launch prune', 'ip netns delete fork-fleet-fork-runner-1', 'nft delete table inet fork_fleet', + 'userdel fork-runner-1', 'userdel fork-runner-2'): + self.assertIn(c, calls) + self.assertLess(calls.index('launch stop desktop-1'), calls.index('userdel fork-runner-1')) + for path in ('/opt/fork-fleet', '/etc/fork-fleet', '/var/lib/fork-fleet', '/etc/sysctl.d/60-fork-fleet.conf', + '/etc/systemd/system/fork-fleet-desk-agent.service'): + self.assertFalse(os.path.lexists(self.path(path)), path) + self.assertEqual(self.read('/etc/cron.deny'), 'someone\n') + self.assertTrue(os.path.exists(self.path('/etc/systemd/system/other.service'))) + self.assertEqual(slurp(os.path.join(self.fakes, 'users')).split(), ['someone']) + code, out, _ = self.run_main('uninstall') # again: nothing to do + self.assertEqual(code, 0) + self.assertIn('nothing here was installed by this kit', out) + + def test_uninstall_stopped_half_way_runs_again(self): + self.assertEqual(self.install()[0], 0) + self.write(os.path.join(self.fakes, 'fail'), 'userdel\n') + code, _, err = self.run_main('uninstall') + self.assertEqual(code, 1) + self.assertTrue(os.path.exists(self.path('/etc/fork-fleet/desk-kit.json'))) # the record stays + code, _, err = self.run_main('uninstall') + self.assertEqual(code, 0, err) + self.assertFalse(os.path.exists(self.path('/etc/fork-fleet'))) + self.assertEqual(slurp(os.path.join(self.fakes, 'users')).split(), []) + + def test_uninstall_of_a_foreign_install_changes_nothing(self): + self.write(self.path('/opt/fork-fleet/fork-fleet-launch.sh'), 'old') + before = self.tree() + code, out, _ = self.run_main('uninstall') + self.assertEqual(code, 0) + self.assertEqual(self.tree(), before) + self.assertEqual(self.calls(), []) + + +class Release(unittest.TestCase): + def test_every_file_the_kit_installs_is_in_this_repository(self): + m = load() + for src in [*m.FILES.values(), *m.UNITS.values(), m.WALLS]: + self.assertTrue(os.path.exists(os.path.join(REPO, src)), src) + self.assertTrue(os.listdir(os.path.join(REPO, m.WALLS))) + + def test_bootstrap_checks_its_arguments_before_anything(self): + boot = os.path.join(KIT, 'bootstrap.sh') + self.assertEqual(subprocess.run(['sh', '-n', boot]).returncode, 0) + env = {'PATH': '/usr/bin:/bin'} + for args, msg in ((['--repo', 'x'], '--repo must be'), (['--repo', 'a/b', '--sha', 'x'], '--sha must be'), + (['--nope'], 'unknown option')): + r = subprocess.run(['sh', boot, *args], capture_output=True, text=True, env=env) + self.assertEqual(r.returncode, 1) + self.assertIn(msg, r.stderr) + if os.geteuid() != 0: + r = subprocess.run(['sh', boot, '--repo', 'a/b', '--sha', 'a' * 40], capture_output=True, text=True, + env=env) + self.assertIn('run this as root', r.stderr) + + +if __name__ == '__main__': + unittest.main() From 3a91ca27b1f7b962093492a86b6249f7555bc300 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 18:51:19 +0000 Subject: [PATCH 02/33] fork fleet: Windows starter for a desk pool (D9) start-desk.ps1 sets up a WSL2 distro for the desk kit from Windows PowerShell 5.1: a memory and swap cap in .wslconfig, NAT, the WSL2 walls' settings inside the distro, the packages, the machine file, then the desk kit's bootstrap from one release commit, a logon task that keeps the distro running, and a PASS/FAIL check. -Uninstall removes it. Every step skips what is already right; WSL restarts only on a change. Co-Authored-By: Claude Opus 5.5 --- .github/fork-ci-tests.txt | 1 + fork-kit/desk/README.md | 20 ++++ fork-kit/desk/tests/windows.test.py | 140 ++++++++++++++++++++++ fork-kit/desk/windows/start-desk.ps1 | 170 +++++++++++++++++++++++++++ 4 files changed, 331 insertions(+) create mode 100644 fork-kit/desk/tests/windows.test.py create mode 100644 fork-kit/desk/windows/start-desk.ps1 diff --git a/.github/fork-ci-tests.txt b/.github/fork-ci-tests.txt index 495cdceec99f6..55c2903cd3e5e 100644 --- a/.github/fork-ci-tests.txt +++ b/.github/fork-ci-tests.txt @@ -43,6 +43,7 @@ fork-ci/actions/pr-queue/pr_queue.test.py fork-ci/actions/pr-queue/tested-base/tested_base.test.py fork-kit/common/tests/machine_file.test.py fork-kit/desk/tests/install.test.py +fork-kit/desk/tests/windows.test.py fork-kit/golive/tests/check.test.py fork-kit/golive/tests/contract.test.py fork-kit/golive/tests/journal.test.py diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index b63fc5bc97788..5e22993a032fd 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -46,6 +46,26 @@ the refresh timer; `check` says FAIL for `template` until it is there. `check` only reads: the installed files against their manifest, every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the runner users and IPv4 forwarding. +## Windows (WSL2) + +[windows/start-desk.ps1](windows/start-desk.ps1) does the Windows side and then this kit inside one WSL2 distro. +Save the machine file on Windows, then in a normal (not administrator) Windows PowerShell window: + +```powershell +powershell -ExecutionPolicy Bypass -File start-desk.ps1 -Repo OWNER/REPO -Sha RELEASE_SHA -MachineFile machine.json +``` + +It prints the PC's facts and asks before it changes anything. Then: the distro (`-Distro`, default `fork-desk`, +Ubuntu 24.04); `.wslconfig`'s `[wsl2]` memory cap (`-MemoryGB`, default three quarters of the RAM and at least 6 GB +left to Windows), swap (`-SwapGB`, 4), NAT and DNS tunnelling, for every WSL distro (a dated copy of the old file +is kept, other keys stay); the distro's own settings the WSL2 walls ask for (`/etc/wsl.conf`: systemd on, no +Windows drives, programs or PATH; its own `/etc/resolv.conf`; WSL's binfmt handler masked); one `wsl --shutdown` +only when one of these changed (every WSL distro and Docker Desktop stop); the packages the fleet needs; the +machine file and `-FleetConfig`, if given; this kit's bootstrap; a logon task that keeps the distro running while +the owner is logged in. Last, `desk-install check` and the Windows side's own PASS/FAIL lines. Safe to run again. +`-Uninstall` runs `desk-install uninstall` in the distro and removes the logon task; the distro and `.wslconfig` +stay (`wsl --unregister DISTRO` removes the distro). + ## Runs again, stops half-way, old or empty state - **Again:** files, settings and units that did not change are not rewritten, and the agent is not stopped. diff --git a/fork-kit/desk/tests/windows.test.py b/fork-kit/desk/tests/windows.test.py new file mode 100644 index 0000000000000..51e5f88bd674f --- /dev/null +++ b/fork-kit/desk/tests/windows.test.py @@ -0,0 +1,140 @@ +"""start-desk.ps1, without Windows: its in-distro part run on a fake root (twice) and read by the real WSL2 walls, +and the rules a script for Windows PowerShell 5.1 must keep. With pwsh on PATH, it is also parsed. + + python3 -I -B fork-kit/desk/tests/windows.test.py +""" +import importlib.machinery, importlib.util, os, re, shutil, subprocess, tempfile, unittest + +KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +REPO = os.path.dirname(os.path.dirname(KIT)) +PS1 = os.path.join(KIT, 'windows', 'start-desk.ps1') +WSL2 = os.path.join(REPO, '.github', 'scripts', 'fork-fleet-walls', 'wsl2') + + +def slurp(path): + with open(path) as f: + return f.read() + + +def put(path, s): + with open(path, 'w') as f: + f.write(s) + + +def text(): + with open(PS1, 'rb') as f: + return f.read() + + +def prepare(): + m = re.search(r"\$Prepare = @'\r?\n(.*?)\r?\n'@", text().decode('ascii'), re.S) + return m.group(1) + + +def walls(root): + os.environ['FORK_FLEET_WALLS_ROOT'] = root + try: + loader = importlib.machinery.SourceFileLoader('wsl2_walls', WSL2) + mod = importlib.util.module_from_spec(importlib.util.spec_from_loader('wsl2_walls', loader)) + loader.exec_module(mod) + return mod + finally: + del os.environ['FORK_FLEET_WALLS_ROOT'] + + +class Prepare(unittest.TestCase): + def setUp(self): + self.root = os.path.realpath(tempfile.mkdtemp(prefix='desk-windows-test-')) + self.addCleanup(shutil.rmtree, self.root, True) + os.makedirs(self.root + '/etc') + os.makedirs(self.root + '/proc/sys/fs/binfmt_misc') + put(self.root + '/proc/sys/fs/binfmt_misc/status', 'enabled\n') + + def run_prepare(self): + # as InDistro sends it: LF only, then '# end' and the CR LF Windows PowerShell adds + r = subprocess.run(['bash', '-s'], input=prepare() + '\n# end\r\n', text=True, capture_output=True, + env={'PATH': '/usr/bin:/bin', 'ROOT': self.root}) + self.assertEqual(r.returncode, 0, r.stderr) + return r.stdout + + def test_the_wsl2_walls_pass_after_it_and_a_second_run_changes_nothing(self): + # as WSL leaves a new distro: interop on, resolv.conf a link into /mnt/wsl + put(self.root + '/etc/wsl.conf', '[boot]\nsystemd=true\n[user]\ndefault=someone\n') + os.symlink('/mnt/wsl/resolv.conf', self.root + '/etc/resolv.conf') + w = walls(self.root) + self.assertIsNotNone(w.interop_wall()) + self.assertIsNotNone(w.resolv_wall()) + out = self.run_prepare() + self.assertIn('changed: /etc/wsl.conf', out) + self.assertEqual(slurp(self.root + '/etc/resolv.conf'), 'nameserver 10.255.255.254\n') + backups = [n for n in os.listdir(self.root + '/etc') if n.startswith('wsl.conf.bak-')] + self.assertEqual(len(backups), 1) + w = walls(self.root) + for wall in (w.interop_wall, w.path_wall, w.resolv_wall): + self.assertIsNone(wall(), wall.__name__) + self.assertEqual(w.wsl_conf('boot', 'systemd'), 'true') + self.assertEqual(w.wsl_conf('automount', 'enabled'), 'false') + before = self.etc() + self.assertEqual(self.run_prepare(), '') + self.assertEqual(self.etc(), before) + + def etc(self): + out = {} + for dirpath, _, files in os.walk(self.root + '/etc'): + for n in files: + full = os.path.join(dirpath, n) + out[full] = os.readlink(full) if os.path.islink(full) else slurp(full) + return out + + def test_an_empty_distro(self): + self.assertIn('changed: /etc/resolv.conf', self.run_prepare()) + self.assertFalse([n for n in os.listdir(self.root + '/etc') if '.bak-' in n]) + self.assertIsNone(walls(self.root).interop_wall()) + + def test_a_resolv_conf_of_its_own_is_kept(self): + put(self.root + '/etc/resolv.conf', 'nameserver 192.0.2.1\n') + self.run_prepare() + self.assertEqual(slurp(self.root + '/etc/resolv.conf'), 'nameserver 192.0.2.1\n') + + +class Script(unittest.TestCase): + def test_windows_powershell_reads_it(self): + raw = text() + self.assertTrue(raw.isascii(), 'Windows PowerShell 5.1 reads a file without a BOM as ANSI: ASCII only') + self.assertNotIn(b'\t', raw) + src = raw.decode('ascii') + for a, b in (('{', '}'), ('(', ')'), ('[', ']')): + self.assertEqual(src.count(a), src.count(b), a) + if shutil.which('pwsh'): + # the path inside the command: words after -Command join the command, they never reach $args + path = PS1.replace("'", "''") + r = subprocess.run(['pwsh', '-NoProfile', '-NonInteractive', '-Command', + f"$e=$null; [System.Management.Automation.Language.Parser]::ParseFile('{path}', " + "[ref]$null, [ref]$e) | Out-Null; if ($e) { $e | Out-String; exit 1 }"], + capture_output=True, text=True, timeout=120, stdin=subprocess.DEVNULL) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + + def test_it_asks_before_it_changes_anything(self): + src = text().decode('ascii') + body = src[src.index('try {'):] + first_change = min(body.index(x) for x in ('wsl.exe --install', 'Set-Content', 'Unregister-ScheduledTask', + 'InDistro', 'Register-ScheduledTask', 'wsl.exe --shutdown')) + self.assertIn("Read-Host", body[:first_change]) + + def test_text_piped_into_the_distro_ends_in_a_comment(self): + src = text().decode('ascii') + self.assertIn('''function Script([string]$s) { ($s -replace "`r", '') + "`n# end" }''', src) + for line in src.splitlines(): + if '| wsl.exe' in line and 'Script ' not in line: + self.assertIn('$text |', line) # Into: data (JSON), where a trailing CR LF is only white space + self.assertNotIn('`"', src.split('try {')[0].split('$Prepare')[0]) # no quotes inside native arguments + + def test_the_kit_it_runs_is_this_one(self): + src = text().decode('ascii') + self.assertIn('/fork-kit/desk/bootstrap.sh', src) + self.assertTrue(os.path.isfile(os.path.join(KIT, 'bootstrap.sh'))) + self.assertIn('/opt/fork-fleet/desk-install check', src) + + +if __name__ == '__main__': + unittest.main() diff --git a/fork-kit/desk/windows/start-desk.ps1 b/fork-kit/desk/windows/start-desk.ps1 new file mode 100644 index 0000000000000..4c925222ed016 --- /dev/null +++ b/fork-kit/desk/windows/start-desk.ps1 @@ -0,0 +1,170 @@ +# fork-kit/desk/windows/start-desk.ps1: a Windows PC as a fork fleet desk pool, through one WSL2 distro +# (fork-kit/desk/README.md, "Windows"). Run it in a normal Windows PowerShell window (not as administrator): +# powershell -ExecutionPolicy Bypass -File start-desk.ps1 -Repo OWNER/REPO -Sha RELEASE_SHA -MachineFile machine.json +# [-FleetConfig fork-fleet.json] [-Distro fork-desk] [-MemoryGB N] [-SwapGB 4] [-Uninstall] +# It prints this PC's facts and asks before it changes anything. Safe to run again: every step checks what is there. +# Steps: the distro (Ubuntu 24.04); .wslconfig's [wsl2] memory cap and NAT (a dated copy of the old file is kept; +# other keys stay); the distro's own settings (the WSL walls: no interop, no Windows PATH or drives, its own +# resolv.conf, systemd); one `wsl --shutdown`; the machine file; the desk kit (bootstrap.sh, in the distro); a logon +# task that keeps the distro running; then `desk-install check` and this side's checks. +param( + [Parameter(Mandatory)][ValidatePattern('^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$')][string]$Repo, + [Parameter(Mandatory)][ValidatePattern('^[0-9a-f]{40}$')][string]$Sha, + [string]$MachineFile, [string]$FleetConfig, + [ValidatePattern('^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$')][string]$Distro = 'fork-desk', + [ValidateRange(0, 4096)][int]$MemoryGB = 0, [ValidateRange(0, 4096)][int]$SwapGB = 4, + [switch]$Uninstall +) +$ErrorActionPreference = 'Stop' +$env:WSL_UTF8 = '1' # wsl.exe's own lines in UTF-8, not UTF-16 +$OutputEncoding = [Text.UTF8Encoding]::new($false) # what is piped into wsl.exe: UTF-8, not ASCII +$Task = "fork-desk-$Distro" +$WslConfig = Join-Path $env:USERPROFILE '.wslconfig' + +function Head([string]$t) { Write-Host "==== $t" -ForegroundColor Cyan } +function Run([string]$what, [scriptblock]$cmd) { + & $cmd + if ($LASTEXITCODE -ne 0) { throw "$what failed (exit $LASTEXITCODE)" } +} +# PowerShell ends what it pipes into a program with CR LF: the script ends in a comment line, so bash never sees a +# stray CR, and LF only before it. +function Script([string]$s) { ($s -replace "`r", '') + "`n# end" } +function InDistro([string]$what, [string]$script) { # bash as root in the distro, the script on stdin; its output + Script $script | wsl.exe -d $Distro -u root --cd / -- bash -s + if ($LASTEXITCODE -ne 0) { throw "$what failed (exit $LASTEXITCODE)" } +} +function Into([string]$file, [string]$dest) { # a Windows file into the distro, root-owned, mode 644 + if (-not (Test-Path -LiteralPath $file -PathType Leaf)) { throw "no such file: $file" } + $text = [IO.File]::ReadAllText((Resolve-Path -LiteralPath $file)) -replace "`r", '' + $dir = $dest.Substring(0, $dest.LastIndexOf('/')) # no quotes in the command: Windows PowerShell mangles them + $text | wsl.exe -d $Distro -u root --cd / -- sh -c "umask 022; mkdir -p $dir && cat > $dest.new && mv $dest.new $dest" + if ($LASTEXITCODE -ne 0) { throw "copying $file into the distro failed (exit $LASTEXITCODE)" } +} + +# The distro's own settings, the ones only a restart of WSL applies. ROOT is for the tests only. +$Prepare = @' +set -eu +ROOT=${ROOT:-} +want='[boot] +systemd=true +[automount] +enabled=false +mountFsTab=false +[interop] +enabled=false +appendWindowsPath=false +[network] +generateResolvConf=false' +mkdir -p "$ROOT/etc/binfmt.d" +if [ "$(cat "$ROOT/etc/wsl.conf" 2>/dev/null)" != "$want" ]; then + [ ! -e "$ROOT/etc/wsl.conf" ] || cp -p "$ROOT/etc/wsl.conf" "$ROOT/etc/wsl.conf.bak-$(date +%Y%m%d-%H%M%S)" + printf '%s\n' "$want" > "$ROOT/etc/wsl.conf.new" && mv "$ROOT/etc/wsl.conf.new" "$ROOT/etc/wsl.conf" + echo 'changed: /etc/wsl.conf' +fi +# WSL's resolv.conf is a link into /mnt/wsl; a file of the distro's own instead (dnsTunneling answers at this address) +if [ -L "$ROOT/etc/resolv.conf" ] || [ ! -f "$ROOT/etc/resolv.conf" ]; then + rm -f "$ROOT/etc/resolv.conf" + printf 'nameserver 10.255.255.254\n' > "$ROOT/etc/resolv.conf" + echo 'changed: /etc/resolv.conf' +fi +[ "$(readlink "$ROOT/etc/binfmt.d/WSLInterop.conf" 2>/dev/null)" = /dev/null ] || + { ln -sfn /dev/null "$ROOT/etc/binfmt.d/WSLInterop.conf"; echo 'changed: /etc/binfmt.d/WSLInterop.conf'; } +'@ +$Packages = 'set -eu; export DEBIAN_FRONTEND=noninteractive; need=; for p in python3 nftables iproute2 curl ca-certificates openssh-client tar; do dpkg -s $p >/dev/null 2>&1 || need="$need $p"; done; [ -z "$need" ] || { apt-get update -q && apt-get install -y -q $need; }' + +try { + Head 'This PC' + $os = Get-CimInstance Win32_OperatingSystem; $cs = Get-CimInstance Win32_ComputerSystem + $ram = [math]::Floor($cs.TotalPhysicalMemory / 1GB) + if ($MemoryGB -eq 0) { $MemoryGB = [math]::Max(4, [math]::Min([math]::Floor($ram * 3 / 4), $ram - 6)) } + "Windows $($os.Version) build $($os.BuildNumber); $($cs.NumberOfLogicalProcessors) threads; RAM $ram GB" + 'WSL:'; wsl.exe --version + $have = @(wsl.exe -l -q | ForEach-Object { $_.Trim([char]0, ' ') }) -contains $Distro + "distro ${Distro}: $(if ($have) { 'there' } else { 'not yet' })" + if (Test-Path $WslConfig) { ".wslconfig now:"; Get-Content $WslConfig | ForEach-Object { " $_" } } + + if ($Uninstall) { + if ((Read-Host "Remove the desk kit from $Distro and the logon task $Task? Type yes") -ne 'yes') { 'Stopped: nothing changed.'; return } + if ($have) { InDistro 'desk-install uninstall' '[ ! -x /opt/fork-fleet/desk-install ] || /opt/fork-fleet/desk-install uninstall' } + Unregister-ScheduledTask -TaskName $Task -Confirm:$false -ErrorAction SilentlyContinue + "Done. The distro and .wslconfig are kept; to remove the distro too: wsl --unregister $Distro" + return + } + if (-not $MachineFile) { throw '-MachineFile is required: the machine file with the desk section (README)' } + + Write-Host '' + Write-Host "Next: the distro $Distro; .wslconfig [wsl2] memory=${MemoryGB}GB swap=${SwapGB}GB networkingMode=nat" -ForegroundColor Yellow + Write-Host 'dnsTunneling=true guiApplications=false (for ALL your WSL distros; a dated copy is kept); then ONE' -ForegroundColor Yellow + Write-Host 'wsl --shutdown (every WSL distro and Docker Desktop stop), the desk kit, and a logon task.' -ForegroundColor Yellow + if ((Read-Host 'Type yes to go on') -ne 'yes') { Write-Host 'Stopped: nothing changed.'; return } + + Head 'the distro' + if (-not $have) { + wsl.exe --install -d Ubuntu-24.04 --name $Distro --no-launch + if ($LASTEXITCODE -ne 0) { throw "wsl --install failed (exit $LASTEXITCODE): run wsl --update (or restart Windows after WSL's own install), then this again" } + } + Run 'the first start of the distro' { wsl.exe -d $Distro -u root --cd / -- true } + + Head '.wslconfig' + $want = [ordered]@{ memory = "${MemoryGB}GB"; swap = "${SwapGB}GB"; networkingMode = 'nat'; dnsTunneling = 'true'; guiApplications = 'false' } + $lines = if (Test-Path $WslConfig) { @(Get-Content $WslConfig) } else { @() } + $out = [Collections.Generic.List[string]]::new(); $sec = ''; $done = @{} + function AddMissing { foreach ($k in $want.Keys) { if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 } } } + foreach ($l in $lines) { + if ($l -match '^\s*\[(.+)\]\s*$') { if ($sec -eq 'wsl2') { AddMissing }; $sec = $Matches[1].Trim().ToLower(); $out.Add($l); continue } + if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=' -and $want.Contains($Matches[1])) { + $k = $Matches[1]; if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 }; continue } + $out.Add($l) + } + if ($sec -eq 'wsl2') { AddMissing } + if ($done.Count -lt $want.Count) { $out.Add('[wsl2]'); AddMissing } + $restart = ($out -join "`n") -ne ($lines -join "`n") + if ($restart) { + if (Test-Path $WslConfig) { Copy-Item $WslConfig "$WslConfig.bak-$(Get-Date -Format yyyyMMdd-HHmmss)" } + Set-Content -Encoding ascii $WslConfig $out; 'changed: .wslconfig' + } + + Head "the distro's own settings" + $changed = InDistro "the distro's settings" $Prepare + $changed + $pid1 = (wsl.exe -d $Distro -u root --cd / -- ps -p 1 -o comm=) -join '' + if ($restart -or $changed -or $pid1.Trim() -ne 'systemd') { + Head 'wsl --shutdown (the new settings apply at the next start)' + Run 'wsl --shutdown' { wsl.exe --shutdown } + } + InDistro 'the packages the fleet needs' $Packages + + Head 'the machine file and the desk kit' + Into $MachineFile '/etc/fork-kit/machine.json' + $extra = '' + if ($FleetConfig) { Into $FleetConfig '/etc/fork-kit/desk-fleet.json'; $extra = ' --fleet-config /etc/fork-kit/desk-fleet.json' } + InDistro 'the desk kit (its FAIL line above says why)' @" +set -eu +curl -fsSL --proto =https https://raw.githubusercontent.com/$Repo/$Sha/fork-kit/desk/bootstrap.sh -o /root/desk-bootstrap.sh +sh /root/desk-bootstrap.sh --repo $Repo --sha $Sha$extra +"@ + + Head "the logon task $Task (keeps the distro running while you are logged in)" + $a = New-ScheduledTaskAction -Execute 'conhost.exe' -Argument "--headless wsl.exe -d $Distro -u root --cd / -- sleep infinity" + $me = [Security.Principal.WindowsIdentity]::GetCurrent().Name + Register-ScheduledTask -TaskName $Task -Action $a -Trigger (New-ScheduledTaskTrigger -AtLogOn -User $me) ` + -Settings (New-ScheduledTaskSettingsSet -ExecutionTimeLimit 0 -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries) -Force | Out-Null + if ((Get-ScheduledTask $Task).State -ne 'Running') { Start-ScheduledTask $Task } + + Head 'check' + $bad = 0 + wsl.exe -d $Distro -u root --cd / -- /opt/fork-fleet/desk-install check + if ($LASTEXITCODE -ne 0) { $bad++ } + $cfg = Get-Content $WslConfig -Raw + foreach ($k in $want.Keys) { + if ($cfg -match "(?m)^\s*$k\s*=\s*$([regex]::Escape($want[$k]))\s*$") { "PASS wslconfig $k=$($want[$k])" } else { "FAIL wslconfig no $k=$($want[$k])"; $bad++ } + } + $t = Get-ScheduledTask $Task -ErrorAction SilentlyContinue + if ($t -and $t.State -eq 'Running') { "PASS logon-task $Task running" } else { "FAIL logon-task $Task is $(if ($t) { $t.State } else { 'missing' })"; $bad++ } + if ($bad -eq 0) { Write-Host 'ALL PASS: this PC takes fleet jobs while you are logged in.' -ForegroundColor Green } + else { Write-Host 'Some FAIL lines above (a template FAIL clears by itself in a few minutes; run this again then).' -ForegroundColor Red } +} +catch { + Write-Host "FAIL: $($_.Exception.Message)" -ForegroundColor Red + Write-Host 'Nothing printed here is a secret. Fix it and run this again: every step checks what is there.' -ForegroundColor Red +} From f18fbda990825feb2a173468a519d1ab74606157 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 19:01:51 +0000 Subject: [PATCH 03/33] fork fleet: free/v1 "is this machine free?" contract and examples (D10) One small contract for every tool that asks whether a machine may take more work: no arguments, 3 s, exit 0 free / 75 busy, one JSON object on stdout. fork-kit/common/free_contract.py is the one reader (judge, ask, and a CLI for shell callers); free-fixtures.json holds the cases every reader must agree on. Two examples: memory (pressure, swap) and busy-while-running (named programs). The launcher's fits also asks FREE_COMMAND when launch.conf names one (default none: the memory rule alone). The desk kit installs the reader and the memory example and writes FREE_COMMAND from the machine file's desk.freeCommand. Co-Authored-By: Claude Opus 5.5 --- .github/fork-ci-tests.txt | 1 + .github/scripts/fork-fleet-launch.sh | 19 +- .github/scripts/fork-fleet-walls/wsl2 | 2 +- .github/scripts/fork-fleet.test.py | 35 +++- fork-kit/common/FREE.md | 69 ++++++++ fork-kit/common/README.md | 4 + fork-kit/common/free-fixtures.json | 159 +++++++++++++++++ fork-kit/common/free/busy-while-running | 16 ++ fork-kit/common/free/memory | 40 +++++ fork-kit/common/free_contract.py | 143 +++++++++++++++ fork-kit/common/machine.example.json | 3 +- fork-kit/common/tests/free_contract.test.py | 182 ++++++++++++++++++++ fork-kit/desk/README.md | 4 +- fork-kit/desk/sbin/desk-install | 15 +- fork-kit/desk/tests/install.test.py | 25 ++- 15 files changed, 702 insertions(+), 15 deletions(-) create mode 100644 fork-kit/common/FREE.md create mode 100644 fork-kit/common/free-fixtures.json create mode 100755 fork-kit/common/free/busy-while-running create mode 100755 fork-kit/common/free/memory create mode 100644 fork-kit/common/free_contract.py create mode 100644 fork-kit/common/tests/free_contract.test.py diff --git a/.github/fork-ci-tests.txt b/.github/fork-ci-tests.txt index 55c2903cd3e5e..ccedb1f29c7de 100644 --- a/.github/fork-ci-tests.txt +++ b/.github/fork-ci-tests.txt @@ -41,6 +41,7 @@ fork-ci/actions/pinned/pinned.test.py fork-ci/actions/pr-queue/lens.test.py fork-ci/actions/pr-queue/pr_queue.test.py fork-ci/actions/pr-queue/tested-base/tested_base.test.py +fork-kit/common/tests/free_contract.test.py fork-kit/common/tests/machine_file.test.py fork-kit/desk/tests/install.test.py fork-kit/desk/tests/windows.test.py diff --git a/.github/scripts/fork-fleet-launch.sh b/.github/scripts/fork-fleet-launch.sh index e3250885ba769..7ac6b6e6dd608 100755 --- a/.github/scripts/fork-fleet-launch.sh +++ b/.github/scripts/fork-fleet-launch.sh @@ -20,7 +20,8 @@ # TEMPLATE and swaps the two by one rename under the launch lock. Running jobs keep their copies; any # failure leaves the old template. Below. # active the NAME of every fork-fleet-* unit that is loaded (starting, running or stopping). -# fits MB exit 0 when MemAvailable minus RESERVE_MB leaves MB megabytes. +# fits MB exit 0 when MemAvailable minus RESERVE_MB leaves MB megabytes and, with FREE_COMMAND set, that free +# script answers free (free/v1, fork-kit/common/FREE.md: run by FREE_CHECK, its reader). # stop NAME stops that unit (a job on it fails: its runner is gone). # prune removes the copies and environment files whose unit is gone. # refresh-net every dispatcher pass: looks this host's own public address up again and, when what is blocked would @@ -61,7 +62,8 @@ # fork-fleet-job-paths.txt beside this launcher), ALLOW_PATHS (more paths jobs see read-only, comma-separated, say a # tool cache: empty by default), IP_FORWARD, SYSTEMD_RUN, SYSTEMCTL, RUNNER_RELEASE (refresh's release record, an https # URL), RUNNER_DOWNLOAD (the https base of its tarballs) and RUNNER_MIN (the oldest runner version refresh may install, -# empty for none) have defaults below. +# empty for none), FREE_COMMAND (a free/v1 script's absolute path, empty for none) and FREE_CHECK (its reader, +# free_contract.py beside this launcher) have defaults below. # HOOK (and the two files beside it), WORK_ROOT and RESOLV_CONF are bound into every job too: never under a folder that # holds data (/home, /root, /mnt, /media, /srv), also through a symlink (no_data). A bind mount of a data folder at # one of them is not seen (it has no link to resolve): that is the operator's to avoid (.github/fork-fleet.md, step 4). @@ -116,8 +118,8 @@ LINGER_DIR=/var/lib/systemd/linger RESERVE_MB=2048 MAX_SECONDS=3600 MEMINFO=/pro RESOLV_CONF=/etc/fork-fleet/resolv.conf IP_FORWARD=/proc/sys/net/ipv4/ip_forward SYSTEMD_RUN=systemd-run SYSTEMCTL=systemctl OWN_ADDRESS_URLS=https://api.ipify.org,https://checkip.amazonaws.com ALLOW_PATHS='' JOB_PATHS='' RUNNER_RELEASE=https://api.github.com/repos/actions/runner/releases/latest -RUNNER_DOWNLOAD=https://github.com/actions/runner/releases/download RUNNER_MIN='' -setting='^(RUNNER_USERS|TEMPLATE|WORK_ROOT|RUN_DIR|HOOK|LINGER_DIR|RESERVE_MB|MAX_SECONDS|MEMINFO|NET_BASE|RESOLV_CONF|IP_FORWARD|SYSTEMD_RUN|SYSTEMCTL|OWN_ADDRESS_URLS|JOB_PATHS|ALLOW_PATHS|RUNNER_RELEASE|RUNNER_DOWNLOAD|RUNNER_MIN)=([^[:space:]]*)$' +RUNNER_DOWNLOAD=https://github.com/actions/runner/releases/download RUNNER_MIN='' FREE_COMMAND='' FREE_CHECK='' +setting='^(RUNNER_USERS|TEMPLATE|WORK_ROOT|RUN_DIR|HOOK|LINGER_DIR|RESERVE_MB|MAX_SECONDS|MEMINFO|NET_BASE|RESOLV_CONF|IP_FORWARD|SYSTEMD_RUN|SYSTEMCTL|OWN_ADDRESS_URLS|JOB_PATHS|ALLOW_PATHS|RUNNER_RELEASE|RUNNER_DOWNLOAD|RUNNER_MIN|FREE_COMMAND|FREE_CHECK)=([^[:space:]]*)$' while IFS= read -r line || [[ -n $line ]]; do [[ $line =~ ^[[:space:]]*(#|$) ]] && continue [[ $line =~ $setting ]] || die "$conf: not a setting: ${line%%=*}" @@ -135,6 +137,9 @@ PLAIN_PATH='(/[A-Za-z0-9._@+-]+)+' if [[ -n $JOB_PATHS ]]; then JOB_PATHS=$ROOT$JOB_PATHS; else JOB_PATHS=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)/fork-fleet-job-paths.txt fi +[[ $FREE_COMMAND =~ ^($PLAIN_PATH)?$ && $FREE_CHECK =~ ^($PLAIN_PATH)?$ ]] || + die "$conf: FREE_COMMAND and FREE_CHECK are plain absolute paths" +[[ -n $FREE_CHECK ]] || FREE_CHECK=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)/free_contract.py NAME='^[a-z0-9][a-z0-9-]{0,62}-[0-9]{1,20}$' name_ok() { [[ $1 =~ $NAME ]] || die "not a runner name: $1"; } @@ -682,7 +687,11 @@ fits() { [[ $1 =~ ^[0-9]{1,7}$ ]] || die "fits: not a number of megabytes: $1" kb=$(awk '$1 == "MemAvailable:" { print $2 }' "$MEMINFO") [[ $kb =~ ^[0-9]+$ ]] || die "no MemAvailable in $MEMINFO" - ((kb / 1024 - RESERVE_MB >= $1)) + ((kb / 1024 - RESERVE_MB >= $1)) || return 1 + [[ -n $FREE_COMMAND ]] || return 0 + owned_safe "$FREE_CHECK" + # Its one line (free, busy: why, not available: why) for the dispatcher's log; any exit but 0 is not free. + /usr/bin/python3 -I "$FREE_CHECK" ask --need-mb "$1" "$FREE_COMMAND" >&2 } # The ranges no job reaches: the host itself is refused on its own (the input chain), these on the way out: RFC 1918, diff --git a/.github/scripts/fork-fleet-walls/wsl2 b/.github/scripts/fork-fleet-walls/wsl2 index 055c103ae4f15..e6bffaed078cb 100755 --- a/.github/scripts/fork-fleet-walls/wsl2 +++ b/.github/scripts/fork-fleet-walls/wsl2 @@ -36,7 +36,7 @@ ROOT = os.environ.get('FORK_FLEET_WALLS_ROOT', '') ME = os.path.realpath(__file__) # The sha256 of the launcher (fork-fleet-launch.sh) whose job units unit_props reads as safe: the one this file ships # with. A changed launcher must be read again and this pin changed with it (fork-fleet-walls-wsl2.test.py checks it). -LAUNCHER_SHA256 = 'a2f56e1fe49b485359d0bc1ecb6ee0ff78365afaddb715b3f7e717d6a542f38a' +LAUNCHER_SHA256 = 'aabfec798b7d0fda3986a2686bddc425a06c8b0246a016266d944bac748ed869' WSL_CONF = '/etc/wsl.conf' BINFMT = '/proc/sys/fs/binfmt_misc' MASK = '/etc/binfmt.d/WSLInterop.conf' # same name as WSL's /usr/lib/binfmt.d file: systemd-binfmt reads this one diff --git a/.github/scripts/fork-fleet.test.py b/.github/scripts/fork-fleet.test.py index 0494947aee1b5..6f0434f637854 100644 --- a/.github/scripts/fork-fleet.test.py +++ b/.github/scripts/fork-fleet.test.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Tests of fork-fleet.py, fork-fleet-launch.sh and fork-fleet-hook.sh, and of their wiring in fork-tests.yml, fork-fleet.yml and fork-release.yml. Run: python3 . Linux only (the gate and launcher tests run real processes).""" -import copy, http.client, importlib.util, io, json, os, pwd, re, shutil, signal, socket, stat, subprocess, sys +import copy, datetime, http.client, importlib.util, io, json, os, pwd, re, shutil, signal, socket, stat, subprocess, sys import tempfile import unittest, urllib.error, urllib.parse from contextlib import redirect_stdout @@ -2169,6 +2169,39 @@ def test_fits(self): self.write('meminfo', 'MemTotal: 1 kB\n', 0o644) self.assertIn('no MemAvailable', self.launch('fits', '1').stderr) + def test_fits_asks_the_free_script_too(self): + """FREE_COMMAND (free/v1, fork-kit/common/FREE.md): after the memory rule, its answer through FREE_CHECK.""" + with open(os.path.join(HERE, '..', '..', 'fork-kit', 'common', 'free_contract.py')) as f: + reader = self.write('free_contract.py', f.read(), 0o644) + script = self.path('free') + self.conf(FREE_COMMAND=script, FREE_CHECK=reader) + free = 8388608 // 1024 - 2048 + + def answer(code, **facts): + at = datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ') + body = json.dumps({'v': 1, 'free': code == 0, 'at': at, **facts}) + self.write('free', f"#!/bin/sh\necho '{body}'\n: > {self.rec}/free-ran\nexit {code}\n", 0o755) + answer(0) + self.assertEqual(self.launch('fits', str(free)).returncode, 0) + answer(75, reason='a game is running') + got = self.launch('fits', '1') + self.assertEqual(got.returncode, 75) + self.assertIn('busy: a game is running', got.stderr) + answer(0, memory={'freeMb': 1000}) + self.assertNotEqual(self.launch('fits', '1001').returncode, 0) + self.assertEqual(self.launch('fits', '1000').returncode, 0) + answer(3) + self.assertIn('not available: exit 3', self.launch('fits', '1').stderr) + answer(0) + os.remove(self.path('rec/free-ran')) + self.assertEqual(self.launch('fits', str(free + 1)).returncode, 1) + self.assertFalse(os.path.exists(self.path('rec/free-ran')), 'the memory rule says no first') + os.chmod(reader, 0o666) + self.assertIn('writable by its owner only', self.launch('fits', '1').stderr) + os.chmod(reader, 0o644) + self.conf(FREE_COMMAND='free') + self.assertIn('plain absolute paths', self.launch('fits', '1').stderr) + def test_stop(self): self.assertEqual(self.launch('stop', self.NAME).returncode, 0) self.assertEqual(self.rec_text('stopped'), f'stop -- fork-fleet-{self.NAME}.service\n') diff --git a/fork-kit/common/FREE.md b/fork-kit/common/FREE.md new file mode 100644 index 0000000000000..800793bfcd73f --- /dev/null +++ b/fork-kit/common/FREE.md @@ -0,0 +1,69 @@ +# free/v1: "is this machine free?" + +One small contract for every tool that asks whether a machine may take more work: a fleet desktop's `fits` +(the desk kit), the placement `factsCommand`, and `heavy facts` once it exists. A machine answers with one +**free script**; every consumer reads it the same way, so CI and agents never disagree about one machine. + +## The script + +- Called with **no arguments**, by its **absolute path**, stdin empty, a plain environment (`PATH`, `LANG`), + from `/`. It answers within **3 seconds** or counts as not available; its whole process group is then ended. +- It and every folder above it belong to root (or to the user the consumer runs as) and no one else can change + them (a sticky folder such as `/tmp` is fine). Otherwise it is not run. +- **The exit code is the verdict:** `0` free, `75` busy. Anything else, a timeout, output that is not one JSON + object, a higher `v`, a `free` that disagrees with the exit code, or facts more than 60 seconds from now = **not + available**. Busy and not available both count as **not free**. +- **stdout is the detail:** one JSON object of at most 4096 bytes. Unknown fields are ignored. + +| Field | Required | What | +| ----------- | -------------------- | ------------------------------------------------------------------------ | +| `v` | yes | `1` | +| `free` | yes | `true` or `false`, the same as the exit code | +| `at` | yes | when the facts were taken, ISO 8601 with a zone (`2026-11-15T05:00:00Z`) | +| `reason` | when `free` is false | short words for a person ("memory pressure 62%", "a game is running") | +| `labels` | no | names a placement rule can match (`["ci-runner"]`) | +| `memory` | no | `budgetMb`, `reservedMb`, `freeMb`, `pressureFull10`, `swapUsedPercent` | +| `queue` | no | `running`, `waiting` | +| `leftovers` | no | `[{"comm", "count", "cpuPercent"}]`, busiest first | + +```json +{ + "v": 1, + "free": false, + "reason": "memory pressure 62%", + "at": "2026-11-15T05:00:00Z", + "labels": ["ci-runner"], + "memory": { + "budgetMb": 4608, + "reservedMb": 3072, + "freeMb": 1536, + "pressureFull10": 62, + "swapUsedPercent": 80 + }, + "queue": { "running": 2, "waiting": 1 }, + "leftovers": [{ "comm": "python3", "count": 73, "cpuPercent": 380 }] +} +``` + +A consumer that needs a size (the desk `fits MB`) also treats a free answer whose `memory.freeMb` is below that +size as busy. An answer without `memory.freeMb` makes no memory claim; the consumer's own memory rule still holds. + +## Pieces + +- [free_contract.py](free_contract.py): the one Python reader (`judge` for an answer, `ask` to run a script, and + `python3 -I free_contract.py ask [--need-mb MB] /path` for shell callers: exit 0 free, 75 busy, 1 not available). +- [free-fixtures.json](free-fixtures.json): the cases every reader, in any language, must agree on. +- Examples in [free/](free/): + - [memory](free/memory): busy under memory pressure or with swap nearly full. The desk kit installs it as + `/opt/fork-fleet/free-memory`. + - [busy-while-running](free/busy-while-running): busy while a named program runs (a game, say). Copy it and put + your programs in it. + +Several conditions in one script: call the others from it and answer busy when one is busy. + +## Using it on a desk pool + +Default: no free script; `fits` uses its memory rule alone (MemAvailable minus `RESERVE_MB`). To add one, put +`"freeCommand": "/opt/fork-fleet/free-memory"` (or your own script's path) in the machine file's `desk` section +and install again: the kit writes `FREE_COMMAND` to `launch.conf`, `fits` then also needs a free answer, and +`desk-install check` shows what the script answers now. diff --git a/fork-kit/common/README.md b/fork-kit/common/README.md index 364979c395b6b..58217a014049c 100644 --- a/fork-kit/common/README.md +++ b/fork-kit/common/README.md @@ -48,6 +48,10 @@ install -d -m 755 /etc/fork-kit && install -m 644 /dev/stdin /etc/fork-kit/machi EOF ``` +## free/v1 + +The "is this machine free?" contract, its one reader and example scripts: [FREE.md](FREE.md). + ## Loader ```python diff --git a/fork-kit/common/free-fixtures.json b/fork-kit/common/free-fixtures.json new file mode 100644 index 0000000000000..f9c6b1573d04a --- /dev/null +++ b/fork-kit/common/free-fixtures.json @@ -0,0 +1,159 @@ +{ + "about": "free/v1 (FREE.md): the cases every parser must agree on. exit null = no answer in time; with padTo, the PAD in stdout is filled with x until stdout is padTo bytes long; now is the reader's clock; needMb is the size the reader asks for.", + "now": "2026-11-15T05:00:30Z", + "cases": [ + { + "name": "free", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "free" + }, + { + "name": "busy with a reason", + "exit": 75, + "stdout": "{\"v\": 1, \"free\": false, \"reason\": \"memory pressure 62%\", \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "busy" + }, + { + "name": "the full example", + "exit": 75, + "stdout": "{\"v\": 1, \"free\": false, \"reason\": \"memory pressure 62%\", \"at\": \"2026-11-15T05:00:00Z\", \"labels\": [\"ci-runner\"], \"memory\": {\"budgetMb\": 4608, \"reservedMb\": 3072, \"freeMb\": 1536, \"pressureFull10\": 62, \"swapUsedPercent\": 80}, \"queue\": {\"running\": 2, \"waiting\": 1}, \"leftovers\": [{\"comm\": \"python3\", \"count\": 73, \"cpuPercent\": 380}]}", + "verdict": "busy" + }, + { + "name": "unknown fields are ignored", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\", \"later\": {\"x\": 1}}", + "verdict": "free" + }, + { + "name": "an offset instead of Z", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T06:00:00.250+01:00\"}", + "verdict": "free" + }, + { + "name": "enough memory for the size asked", + "exit": 0, + "needMb": 2048, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\", \"memory\": {\"freeMb\": 2048}}", + "verdict": "free" + }, + { + "name": "too little memory for the size asked", + "exit": 0, + "needMb": 2049, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\", \"memory\": {\"freeMb\": 2048}}", + "verdict": "busy" + }, + { + "name": "no memory figure: a size asked is not judged", + "exit": 0, + "needMb": 99999, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "free" + }, + { "name": "no answer in time", "exit": null, "stdout": "", "verdict": "unavailable" }, + { + "name": "another exit code", + "exit": 1, + "stdout": "{\"v\": 1, \"free\": false, \"reason\": \"x\", \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "exit 0 but free is false", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": false, \"reason\": \"x\", \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "exit 75 but free is true", + "exit": 75, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "free is a string", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": \"true\", \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "busy without a reason", + "exit": 75, + "stdout": "{\"v\": 1, \"free\": false, \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "busy with a blank reason", + "exit": 75, + "stdout": "{\"v\": 1, \"free\": false, \"reason\": \" \", \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "a newer version", + "exit": 0, + "stdout": "{\"v\": 2, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "no version", + "exit": 0, + "stdout": "{\"free\": true, \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "version true", + "exit": 0, + "stdout": "{\"v\": true, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\"}", + "verdict": "unavailable" + }, + { + "name": "no time", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true}", + "verdict": "unavailable" + }, + { + "name": "a time without a zone", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00\"}", + "verdict": "unavailable" + }, + { + "name": "facts older than 60 seconds", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T04:59:29Z\"}", + "verdict": "unavailable" + }, + { + "name": "facts from 60 seconds ago", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T04:59:30Z\"}", + "verdict": "free" + }, + { + "name": "facts from the future", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:01:31Z\"}", + "verdict": "unavailable" + }, + { "name": "not JSON", "exit": 0, "stdout": "free", "verdict": "unavailable" }, + { "name": "a JSON list", "exit": 0, "stdout": "[1]", "verdict": "unavailable" }, + { "name": "nothing on stdout", "exit": 0, "stdout": "", "verdict": "unavailable" }, + { + "name": "4096 bytes is the most", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\", \"pad\": \"PAD\"}", + "padTo": 4096, + "verdict": "free" + }, + { + "name": "4097 bytes is too many", + "exit": 0, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\", \"pad\": \"PAD\"}", + "padTo": 4097, + "verdict": "unavailable" + } + ] +} diff --git a/fork-kit/common/free/busy-while-running b/fork-kit/common/free/busy-while-running new file mode 100755 index 0000000000000..3dd1b9b18c96d --- /dev/null +++ b/fork-kit/common/free/busy-while-running @@ -0,0 +1,16 @@ +#!/bin/sh +# A free/v1 script (FREE.md): busy while one of the programs in NAMES runs, say a game or a video editor, so the +# machine is left to its person. Copy it, put your programs' process names (as `ps -e -o comm=` shows them) in +# NAMES, make the copy root's and not writable by others, and name the copy as the free command. +# It sees the processes of this system only: in WSL, Windows programs are not visible (interop is off, on purpose). +NAMES='example-game example-editor' + +at=$(date -u +%Y-%m-%dT%H:%M:%SZ) +running=$(ps -e -o comm=) || { echo '{"v": 1, "free": false, "reason": "ps failed", "at": "'"$at"'"}'; exit 1; } +for name in $NAMES; do + if printf '%s\n' "$running" | grep -qxF -- "$name"; then + echo '{"v": 1, "free": false, "reason": "'"$name"' is running", "at": "'"$at"'"}' + exit 75 + fi +done +echo '{"v": 1, "free": true, "at": "'"$at"'"}' diff --git a/fork-kit/common/free/memory b/fork-kit/common/free/memory new file mode 100755 index 0000000000000..b94727ba8219c --- /dev/null +++ b/fork-kit/common/free/memory @@ -0,0 +1,40 @@ +#!/usr/bin/python3 -I +"""A free/v1 script (FREE.md): free while memory is not under pressure and swap is not nearly full. + +Busy when, over the last 10 seconds, all tasks waited on memory more than PRESSURE_FULL10 percent of the time +(/proc/pressure/memory, "full avg10"), or when more than SWAP_USED_PERCENT of swap is in use. Change the two +numbers in a copy of this file; the desk kit installs this one as /opt/fork-fleet/free-memory. +""" +import datetime, json, sys + +PRESSURE_FULL10, SWAP_USED_PERCENT = 10, 50 + + +def meminfo(): + with open('/proc/meminfo', encoding='ascii') as f: + return {k: int(v.split()[0]) for k, v in (line.split(':', 1) for line in f)} + + +def pressure_full10(): + try: + with open('/proc/pressure/memory', encoding='ascii') as f: + for line in f: + if line.startswith('full '): + return float(dict(x.split('=') for x in line.split()[1:])['avg10']) + except OSError: # a kernel without PSI: no pressure figure, the swap rule alone + return None + return None + + +m, full10 = meminfo(), pressure_full10() +swap = round(100 * (m['SwapTotal'] - m['SwapFree']) / m['SwapTotal']) if m.get('SwapTotal') else 0 +reason = (f'memory pressure {full10:g}%' if full10 is not None and full10 > PRESSURE_FULL10 else + f'swap {swap}% used' if swap > SWAP_USED_PERCENT else '') +memory = {'freeMb': m['MemAvailable'] // 1024, 'swapUsedPercent': swap} +if full10 is not None: + memory['pressureFull10'] = full10 +answer = {'v': 1, 'free': not reason, 'at': datetime.datetime.now(datetime.timezone.utc).isoformat(), 'memory': memory} +if reason: + answer['reason'] = reason +print(json.dumps(answer)) +sys.exit(75 if reason else 0) diff --git a/fork-kit/common/free_contract.py b/fork-kit/common/free_contract.py new file mode 100644 index 0000000000000..045153c71a94d --- /dev/null +++ b/fork-kit/common/free_contract.py @@ -0,0 +1,143 @@ +"""free/v1, "is this machine free?" (FREE.md beside this file): runs a free script and judges its answer. + +One parser for every Python consumer (the desk launcher's `fits`, the desk kit's check); free-fixtures.json beside +it holds the cases every parser, in any language, must agree on. + + python3 -I free_contract.py ask [--need-mb MB] /absolute/script + one line on stdout (free / busy: REASON / not available: REASON); + exit 0 free, 75 busy, 1 not available, 2 a wrong call +""" +import argparse, datetime, json, os, select, signal, stat, subprocess, sys, time + +VERSION, LIMIT_SECONDS, MAX_BYTES, MAX_AGE_SECONDS = 1, 3, 4096, 60 +FREE, BUSY = 0, 75 +ENV = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin', 'LANG': 'C.UTF-8'} + + +def number(x): + return isinstance(x, (int, float)) and not isinstance(x, bool) + + +def parse_at(at): + if not isinstance(at, str): + return None + try: + when = datetime.datetime.fromisoformat(at[:-1] + '+00:00' if at.endswith('Z') else at) + except ValueError: + return None + return when if when.tzinfo is not None else None + + +def judge(code, out, now, need_mb=None): + """(verdict, reason, facts) for one answer: verdict 'free', 'busy' or 'unavailable' (only 'free' is free). + + code: the exit code, None when the script gave no answer in time. out: its stdout (bytes). now: an aware + datetime. need_mb: when given, a `memory.freeMb` below it makes a free answer busy.""" + def no(why): + return 'unavailable', why, None + if len(out) > MAX_BYTES: + return no(f'more than {MAX_BYTES} bytes on stdout') + if code is None: + return no(f'no answer within {LIMIT_SECONDS} seconds') + if code not in (FREE, BUSY): + return no(f'exit {code}') + try: + facts = json.loads(out.decode('utf-8')) + except ValueError: # UnicodeDecodeError is one + return no('stdout is not JSON') + if not isinstance(facts, dict): + return no('stdout is not a JSON object') + v, free, reason = facts.get('v'), facts.get('free'), facts.get('reason') + if type(v) is not int or v < 1: + return no('no version "v"') + if v > VERSION: + return no(f'version {v}, this reader knows {VERSION}') + if type(free) is not bool or free != (code == FREE): + return no(f'"free" is {json.dumps(free)} but the exit code is {code}') + if not free and not (isinstance(reason, str) and reason.strip()): + return no('busy without a "reason"') + when = parse_at(facts.get('at')) + if when is None: + return no('no time "at" (ISO 8601 with a zone)') + if abs((now - when).total_seconds()) > MAX_AGE_SECONDS: + return no(f'"at" is {facts["at"]}, more than {MAX_AGE_SECONDS} seconds from now') + if not free: + return 'busy', ' '.join(reason.split())[:200], facts + memory = facts.get('memory') + free_mb = memory.get('freeMb') if isinstance(memory, dict) else None + if need_mb is not None and number(free_mb) and free_mb < need_mb: + return 'busy', f'{free_mb:g} MB free, {need_mb} MB needed', facts + return 'free', '', facts + + +def unsafe(command, uid=None): + """Why COMMAND must not run, or None: an absolute plain path to an executable file that it and every folder + above it belong to root or to this user and that no one else may change (a sticky folder like /tmp is fine).""" + uid = os.geteuid() if uid is None else uid + if not os.path.isabs(command) or os.path.normpath(command) != command: + return f'{command!r} is not an absolute plain path' + path = os.path.realpath(command) + try: + if not stat.S_ISREG(os.stat(path).st_mode) or not os.access(path, os.X_OK): + return f'{path} is not an executable file' + while True: + st = os.stat(path) + if st.st_uid not in (0, uid): + return f'{path} belongs to uid {st.st_uid}' + if st.st_mode & 0o022 and not (stat.S_ISDIR(st.st_mode) and st.st_mode & stat.S_ISVTX): + return f'{path} may be changed by others' + if path == '/': + return None + path = os.path.dirname(path) + except OSError as e: + return f'{path}: {e.strerror}' + + +def ask(command, need_mb=None, timeout=LIMIT_SECONDS): + """Runs the free script (no arguments, stdin empty, a plain environment, its whole process group ended after it) + and judges its answer: (verdict, reason, facts).""" + why = unsafe(command) + if why: + return 'unavailable', why, None + try: + p = subprocess.Popen([command], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, + env=ENV, cwd='/', start_new_session=True) + except OSError as e: + return 'unavailable', f'cannot run {command}: {e.strerror}', None + out, code, deadline = b'', None, time.monotonic() + timeout + with p: + try: + while len(out) <= MAX_BYTES: + left = deadline - time.monotonic() + if left <= 0 or not select.select([p.stdout], [], [], left)[0]: + break + chunk = os.read(p.stdout.fileno(), 65536) + if not chunk: + try: + code = p.wait(max(0.0, deadline - time.monotonic())) + except subprocess.TimeoutExpired: + pass + break + out += chunk + finally: + try: + os.killpg(p.pid, signal.SIGKILL) + except (ProcessLookupError, PermissionError): + pass + return judge(code, out, datetime.datetime.now(datetime.timezone.utc), need_mb) + + +def main(argv): + parser = argparse.ArgumentParser(prog='free_contract.py') + sub = parser.add_subparsers(dest='verb', required=True) + a = sub.add_parser('ask', help='run a free script and say what it answered') + a.add_argument('--need-mb', type=int) + a.add_argument('command') + args = parser.parse_args(argv) + verdict, reason, _ = ask(args.command, args.need_mb) + print({'free': 'free', 'busy': f'busy: {reason}'}.get(verdict, f'not available: {reason}')) + return {'free': 0, 'busy': BUSY}.get(verdict, 1) + + +if __name__ == '__main__': + sys.exit(main(sys.argv[1:])) diff --git a/fork-kit/common/machine.example.json b/fork-kit/common/machine.example.json index 6251cd5cef9f2..341478a759fd1 100644 --- a/fork-kit/common/machine.example.json +++ b/fork-kit/common/machine.example.json @@ -31,6 +31,7 @@ "host": "fleet-host.example", "hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "slots": 2, - "reserveMb": 4096 + "reserveMb": 4096, + "freeCommand": "/opt/fork-fleet/free-memory" } } diff --git a/fork-kit/common/tests/free_contract.test.py b/fork-kit/common/tests/free_contract.test.py new file mode 100644 index 0000000000000..6c0a0ffce2f43 --- /dev/null +++ b/fork-kit/common/tests/free_contract.test.py @@ -0,0 +1,182 @@ +"""Contract tests for free/v1 (fork-kit/common/FREE.md): free_contract.py against free-fixtures.json (the cases +every reader shares), running real scripts (time limit, output limit, process group, safety), its command line, +and the example scripts in free/. + + python3 -I -B fork-kit/common/tests/free_contract.test.py +""" +import datetime, importlib.machinery, importlib.util, json, os, shutil, subprocess, sys, tempfile, time, unittest + +COMMON = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +READER = os.path.join(COMMON, 'free_contract.py') + + +def load(): + loader = importlib.machinery.SourceFileLoader('free_contract', READER) + mod = importlib.util.module_from_spec(importlib.util.spec_from_loader('free_contract', loader)) + loader.exec_module(mod) + return mod + + +fc = load() + + +def now_text(delta=0): + t = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(seconds=delta) + return t.strftime('%Y-%m-%dT%H:%M:%SZ') + + +class Fixtures(unittest.TestCase): + def test_every_shared_case(self): + with open(os.path.join(COMMON, 'free-fixtures.json'), encoding='utf-8') as f: + data = json.load(f) + now = fc.parse_at(data['now']) + self.assertGreater(len(data['cases']), 20) + for case in data['cases']: + with self.subTest(case['name']): + out = case['stdout'] + if 'padTo' in case: + out = out.replace('PAD', 'x' * (case['padTo'] - len(out) + 3)) + self.assertEqual(len(out), case['padTo']) + verdict, reason, facts = fc.judge(case['exit'], out.encode(), now, case.get('needMb')) + self.assertEqual(verdict, case['verdict'], reason) + self.assertEqual(bool(reason), verdict != 'free') + self.assertEqual(facts is None, verdict == 'unavailable') + + +class Ask(unittest.TestCase): + def setUp(self): + self.d = os.path.realpath(tempfile.mkdtemp(prefix='free-contract-test-')) + self.addCleanup(shutil.rmtree, self.d, True) + + def script(self, body, name='free', mode=0o755): + path = os.path.join(self.d, name) + with open(path, 'w') as f: + f.write('#!/bin/sh\n' + body) + os.chmod(path, mode) + return path + + def test_free_busy_and_the_environment_it_gets(self): + s = self.script(f'env > {self.d}/env; pwd > {self.d}/pwd; echo "$#" > {self.d}/argc\n' + f'echo \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n') + self.assertEqual(fc.ask(s)[0], 'free') + with open(f'{self.d}/env') as f: + names = sorted(line.split('=', 1)[0] for line in f.read().splitlines()) + self.assertEqual([n for n in names if n not in ('PWD', 'SHLVL', '_', 'OLDPWD')], ['LANG', 'PATH']) + with open(f'{self.d}/pwd') as f, open(f'{self.d}/argc') as g: + self.assertEqual((f.read(), g.read()), ('/\n', '0\n')) + busy = f'{{"v": 1, "free": false, "reason": "a game\\nis on", "at": "{now_text()}"}}' + s = self.script(f'printf "%s\\n" \'{busy}\'\nexit 75\n') # not echo: in dash it turns \\n into a newline + self.assertEqual(fc.ask(s)[:2], ('busy', 'a game is on')) + + def test_no_answer_in_time_ends_its_whole_group(self): + s = self.script(f'sleep 30 & echo $! > {self.d}/child\nsleep 30\n') + t = time.monotonic() + verdict, reason, _ = fc.ask(s, timeout=1) + self.assertLess(time.monotonic() - t, 10) + self.assertEqual(verdict, 'unavailable') + self.assertIn('no answer', reason) + with open(f'{self.d}/child') as f: + child = int(f.read()) + for _ in range(50): + try: + with open(f'/proc/{child}/stat') as f: + if f.read().rsplit(')', 1)[1].split()[0] == 'Z': + break + except (FileNotFoundError, ProcessLookupError): # gone between the open and the read + break + time.sleep(0.1) + else: + self.fail('the script\'s background child still runs') + + def test_a_child_that_keeps_stdout_open_is_no_answer(self): + s = self.script(f'(sleep 30) &\necho \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n') + self.assertEqual(fc.ask(s, timeout=1)[0], 'unavailable') + + def test_endless_output_is_cut_off(self): + s = self.script('yes\n') + verdict, reason, _ = fc.ask(s, timeout=5) + self.assertEqual(verdict, 'unavailable') + self.assertIn('4096 bytes', reason) + + def test_unsafe_scripts_never_run(self): + mark = os.path.join(self.d, 'ran') + body = f': > {mark}\necho \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n' + cases = {'relative': lambda: 'free', + 'dot-dot': lambda: self.script(body).replace('/free', '/../' + os.path.basename(self.d) + '/free'), + 'writable by others': lambda: self.script(body, mode=0o757), + 'not executable': lambda: self.script(body, mode=0o644), + 'a folder': lambda: self.d, + 'missing': lambda: os.path.join(self.d, 'nothing')} + for name, make in cases.items(): + with self.subTest(name): + verdict, reason, _ = fc.ask(make()) + self.assertEqual(verdict, 'unavailable') + self.assertFalse(os.path.exists(mark), reason) + os.chmod(self.d, 0o777) # a folder others may change, no sticky bit + try: + self.assertIn('may be changed by others', fc.ask(self.script(body))[1]) + finally: + os.chmod(self.d, 0o700) + self.assertFalse(os.path.exists(mark)) + self.assertIn('belongs to uid', fc.unsafe(self.script(body), uid=os.geteuid() + 12345) or '') + os.chmod(self.d, 0o1777) # sticky, like /tmp: fine + self.assertEqual(fc.ask(self.script(body))[0], 'free') + + def test_a_link_is_judged_by_where_it_leads(self): + s = self.script(f'echo \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n') + os.symlink(s, os.path.join(self.d, 'link')) + self.assertEqual(fc.ask(os.path.join(self.d, 'link'))[0], 'free') + + def test_the_command_line(self): + def cli(*args): + return subprocess.run([sys.executable, '-I', '-B', READER, *args], capture_output=True, text=True, + timeout=30) + s = self.script(f'echo \'{{"v": 1, "free": true, "at": "{now_text()}", "memory": {{"freeMb": 500}}}}\'\n') + got = cli('ask', s) + self.assertEqual((got.returncode, got.stdout), (0, 'free\n')) + got = cli('ask', '--need-mb', '501', s) + self.assertEqual((got.returncode, got.stdout), (75, 'busy: 500 MB free, 501 MB needed\n')) + got = cli('ask', 'relative') + self.assertEqual(got.returncode, 1) + self.assertTrue(got.stdout.startswith('not available: ')) + self.assertEqual(cli('ask').returncode, 2) + + +class Examples(unittest.TestCase): + """Each example answers by the contract on this machine (free or busy, never not available). Run from a copy in + a folder of its own: a checkout's folders may be group-writable, and then the reader rightly refuses them.""" + + def copy(self, name, text=None): + d = os.path.realpath(tempfile.mkdtemp(prefix='free-contract-test-')) + self.addCleanup(shutil.rmtree, d, True) + if text is None: + with open(os.path.join(COMMON, 'free', name)) as f: + text = f.read() + path = os.path.join(d, name) + with open(path, 'w') as f: + f.write(text) + os.chmod(path, 0o755) + return path + + def test_memory(self): + verdict, reason, facts = fc.ask(self.copy('memory')) + self.assertIn(verdict, ('free', 'busy'), reason) + self.assertIsInstance(facts['memory']['freeMb'], int) + + def test_busy_while_running(self): + with open(os.path.join(COMMON, 'free', 'busy-while-running')) as f: + text = f.read() + self.assertEqual(fc.ask(self.copy('busy-while-running'))[0], 'free') + names = "NAMES='example-game example-editor'" + self.assertIn(names, text) + copy = self.copy('busy', text.replace(names, "NAMES='not-running sleep'")) + p = subprocess.Popen(['sleep', '30']) + try: + self.assertEqual(fc.ask(copy)[:2], ('busy', 'sleep is running')) + finally: + p.kill() + p.wait() + + +if __name__ == '__main__': + unittest.main() diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 5e22993a032fd..01e97eb99b5f9 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -26,7 +26,9 @@ The facts of this machine go in the `desk` section of `/etc/fork-kit/machine.jso `pool` must be a pool in the fleet config; `host` and `hostKey` are the always-on host and its ssh host key (on that host: `cat /etc/ssh/ssh_host_ed25519_key.pub`, without the comment). Optional: `port` (22), `user` (`fleet-desk`), `slots` (1, at most 8; one runner user each), `reserveMb` (memory the launcher keeps free for the owner), `netBase` (`10.N`, the runners' network, -default `10.204`: pick another when this machine's network already uses it). +default `10.204`: pick another when this machine's network already uses it), `freeCommand` (a free/v1 script, +[FREE.md](../common/FREE.md): `fits` then also needs it to answer free; the kit installs an example as +`/opt/fork-fleet/free-memory`; default none, the memory rule alone). ## Install, check, uninstall diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index 0f71da2ff96df..59e560a91f11c 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -29,7 +29,8 @@ FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh', 'fork-fleet-desk-wire.py': '.github/scripts/fork-fleet-desk-wire.py', 'hook.sh': '.github/scripts/fork-fleet-hook.sh', 'fork-fleet.py': '.github/scripts/fork-fleet.py', 'fork-fleet.json': '.github/fork-fleet.json', 'desk-install': 'fork-kit/desk/sbin/desk-install', - 'machine_file.py': 'fork-kit/common/machine_file.py', 'README.md': 'fork-kit/desk/README.md'} + 'machine_file.py': 'fork-kit/common/machine_file.py', 'README.md': 'fork-kit/desk/README.md', + 'free_contract.py': 'fork-kit/common/free_contract.py', 'free-memory': 'fork-kit/common/free/memory'} WALLS = '.github/scripts/fork-fleet-walls' # copied whole: every OS's walls file and fixture UNITS = {'fork-fleet-desk-agent.service': '.github/scripts/fork-fleet-desk-agent.service', 't3code-fleet-refresh.service': '.github/scripts/fork-fleet-refresh.service', @@ -44,6 +45,7 @@ ENV = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'C.UTF-8'} SHA = re.compile(r'[0-9a-f]{40}') SHAPES = {'pool': r'[a-z0-9][a-z0-9_-]{0,39}', 'host': r'[A-Za-z0-9][A-Za-z0-9.-]{0,252}', 'user': r'[a-z_][a-z0-9_-]{0,31}', 'netBase': r'10\.(0|[1-9][0-9]?|1[0-9]{2}|2[0-4][0-9]|25[0-5])', + 'freeCommand': r'(/[A-Za-z0-9._@+-]+)+', 'hostKey': r'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI[A-Za-z0-9+/]{43}'} @@ -99,7 +101,8 @@ def machine_section(): 'port': (int, 22, within(1, 65535)), 'user': (str, 'fleet-desk', shape('user')), 'hostKey': (str, mf.REQUIRED, shape('hostKey')), 'slots': (int, 1, within(1, 8)), 'reserveMb': (int, None, within(0, 1000000)), - 'netBase': (str, None, shape('netBase'))}, 'desk') + 'netBase': (str, None, shape('netBase')), + 'freeCommand': (str, None, shape('freeCommand'))}, 'desk') except mf.MachineFileError as e: raise Refused(f'machine file: {e}') from None @@ -165,7 +168,7 @@ def stage(source, sha, fleet_config, pool): dst = os.path.join(tmp, rel) os.makedirs(os.path.dirname(dst), mode=0o755, exist_ok=True) shutil.copyfile(src, dst) - executable = rel in ('fork-fleet-launch.sh', 'hook.sh', 'desk-install') or ( + executable = rel in ('fork-fleet-launch.sh', 'hook.sh', 'desk-install', 'free-memory') or ( rel.startswith('fork-fleet-walls/') and not rel.endswith('.json')) os.chmod(dst, 0o755 if executable else 0o644) manifest[rel] = sha256(dst) @@ -251,7 +254,8 @@ def settings(m, users): known = (m['host'] if m['port'] == 22 else f'[{m["host"]}]:{m["port"]}') + ' ' + m['hostKey'] files = {ETC + '/launch.conf': (f'RUNNER_USERS={",".join(users)}\n' + (f'RESERVE_MB={m["reserveMb"]}\n' if m['reserveMb'] is not None else '') - + (f'NET_BASE={m["netBase"]}\n' if m['netBase'] else ''), 0o600), + + (f'NET_BASE={m["netBase"]}\n' if m['netBase'] else '') + + (f'FREE_COMMAND={m["freeCommand"]}\n' if m['freeCommand'] else ''), 0o600), ETC + '/desk.conf': (f'POOL={m["pool"]}\nHOST={m["host"]}\nPORT={m["port"]}\nUSER={m["user"]}\n', 0o600), ETC + '/desk-known_hosts': (known + '\n', 0o644), ETC + '/resolv.conf': ('nameserver 1.1.1.1\nnameserver 9.9.9.9\n', 0o644), @@ -375,6 +379,9 @@ def check(): m = machine_section() users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] row(True, 'machine-file', f'desk section: pool {m["pool"]}, {m["slots"]} slot(s), always-on host {m["host"]}') + if m['freeCommand']: # busy is fine: the script answers by the contract (FREE.md) + r = py(p(OPT + '/free_contract.py'), 'ask', m['freeCommand']) + row(r.returncode in (0, 75), 'free-script', f'{m["freeCommand"]}: {(r.stdout or r.stderr).strip()[:200]}') except Refused as e: users = [] row(False, 'machine-file', str(e)[:300]) diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index 65673b609122f..2de38ec65819b 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -4,7 +4,8 @@ python3 -I -B fork-kit/desk/tests/install.test.py """ -import contextlib, importlib.machinery, importlib.util, io, json, os, shutil, stat, subprocess, sys, tempfile, unittest +import contextlib, datetime, importlib.machinery, importlib.util, io, json, os, shutil, stat, subprocess, sys +import tempfile, unittest KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) REPO = os.path.dirname(os.path.dirname(KIT)) @@ -125,7 +126,8 @@ def release(self, name='release', extra=None): if text is not None: self.write(os.path.join(top, rel), text.replace('{log}', os.path.join(self.fakes, 'calls.log')) .replace('{here}', self.fakes), 0o755) - for rel in ('fork-kit/desk/sbin/desk-install', 'fork-kit/desk/README.md', 'fork-kit/common/machine_file.py'): + for rel in ('fork-kit/desk/sbin/desk-install', 'fork-kit/desk/README.md', 'fork-kit/common/machine_file.py', + 'fork-kit/common/free_contract.py', 'fork-kit/common/free/memory'): self.write(os.path.join(top, rel), slurp(os.path.join(REPO, rel)), 0o755) return top @@ -358,6 +360,25 @@ def test_reserve_and_port_reach_the_settings(self): self.assertTrue(self.read('/etc/fork-fleet/desk-known_hosts').endswith( f'[fleet-host.example]:2222 {HOST_KEY}\n')) + def test_a_free_script_reaches_the_settings_and_the_check(self): + script = os.path.join(self.tmp, 'free') + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'freeCommand': script}}) + self.assertEqual(self.install()[0], 0) + self.assertIn(f'FREE_COMMAND={script}\n', self.read('/etc/fork-fleet/launch.conf')) + self.assertEqual(stat.S_IMODE(os.stat(self.path('/opt/fork-fleet/free-memory')).st_mode), 0o755) + self.template() + now = datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ') + for code, body, want in ((0, '"free": true', 'PASS free-script ' + script + ': free\n'), + (75, '"free": false, "reason": "a game"', 'busy: a game'), + (75, '"free": true', 'FAIL free-script')): + self.write(script, f"#!/bin/sh\necho '{{\"v\": 1, {body}, \"at\": \"{now}\"}}'\nexit {code}\n", 0o755) + out = self.run_main('check')[1] + self.assertIn(want, out) + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'freeCommand': 'free'}}) + code, _, err = self.install() + self.assertEqual(code, 1) + self.assertIn('freeCommand', err) + def test_existing_user_is_used_not_recorded(self): self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n') self.assertEqual(self.install()[0], 0) From 57e1cb53a060041f3535ab97a874fbe2f5a1c381 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 19:13:43 +0000 Subject: [PATCH 04/33] fork fleet: `seed` verb and `seeds` in the fleet config (D11) A desk pool may list warm caches ("seeds") in its fleet config entry. The always-on host's forced command answers a desktop's `seeds` (the pool's list) and `seed NAME` (that one ready-made tar, with a size and SHA-256 trailer) in an ssh session beside the agent's. Only names of the list, never a path; exit 3 while another stream for the pool runs, 4 when the file may not be sent. The desktop's fetcher follows in the next change. Co-Authored-By: Claude Opus 5.5 --- .github/fork-ci-tests.txt | 1 + .github/fork-fleet.md | 26 +++ .github/scripts/fork-fleet-desk-seed.py | 129 +++++++++++++ .github/scripts/fork-fleet-desk-seed.test.py | 186 +++++++++++++++++++ .github/scripts/fork-fleet-desk.py | 18 ++ .github/scripts/fork-fleet.py | 9 + 6 files changed, 369 insertions(+) create mode 100644 .github/scripts/fork-fleet-desk-seed.py create mode 100644 .github/scripts/fork-fleet-desk-seed.test.py diff --git a/.github/fork-ci-tests.txt b/.github/fork-ci-tests.txt index ccedb1f29c7de..e4cebd33243c1 100644 --- a/.github/fork-ci-tests.txt +++ b/.github/fork-ci-tests.txt @@ -9,6 +9,7 @@ .github/scripts/fork-contracts.test.py .github/scripts/fork-fleet-desk-agent.test.py .github/scripts/fork-fleet-desk-key.test.py +.github/scripts/fork-fleet-desk-seed.test.py .github/scripts/fork-fleet-desk.test.py .github/scripts/fork-fleet.test.py .github/scripts/fork-fleet-refresh.test.py diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index a42c9379baa9f..1a0b7c895f3a5 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -448,6 +448,7 @@ done files="fork-fleet.json:fork-fleet.json:644 scripts/fork-fleet.py:fork-fleet.py:644 scripts/fork-fleet-hook.sh:hook.sh:755 scripts/fork-fleet-launch.sh:fork-fleet-launch.sh:755 scripts/fork-fleet-desk.py:fork-fleet-desk.py:644 scripts/fork-fleet-desk-wire.py:fork-fleet-desk-wire.py:644 + scripts/fork-fleet-desk-seed.py:fork-fleet-desk-seed.py:644 scripts/fork-fleet-refresh.service:fork-fleet-refresh.service:644 scripts/fork-fleet-refresh.timer:fork-fleet-refresh.timer:644 scripts/fork-fleet-job-paths.txt:fork-fleet-job-paths.txt:644" @@ -683,6 +684,31 @@ Bun, Node and Rust are installed per job by the setup actions, into the job's own home, which goes with its copy. Leave a few tens of GB of disk for `/var/lib/fork-fleet/runners`. +#### Seeds (warm caches for a desk pool) + +A job's copy starts empty, so each job would download its tools and +dependencies again. A desk pool may list **seeds** in its fleet config entry, +ready-made caches the desktop keeps and its jobs start from: + +```json +"seeds": [{"name": "gradle", "every_hours": 6, "max_age_hours": 72, "max_mb": 4096}] +``` + +`every_hours` is how often a desktop takes a new one, `max_age_hours` (at +least `every_hours`) the age, from when it was made, at which it is dropped, +`max_mb` its largest size. Nothing in the fleet makes a seed: on the always-on +host the project's own timer or job writes a tar of a cache folder's contents +to `/var/lib/fork-fleet/seed-files/NAME.tar`, by a temp file and a rename (a +file changed while it is sent is refused). That folder and its files are +root's (or the project's), readable by the desk-link user and writable by no +one else; never put a secret in a seed: every job of the pool reads it. + +The desktop asks for them with the desk link's key, in an ssh session of its +own: the same forced command answers `seeds` with the pool's list and +`seed NAME` with that one file, only a name of that list, and a trailer with its +size and SHA-256 (`fork-fleet-desk-seed.py`, which step 3 installs; its header +has the exit codes). + ## Owner steps on the repository Run these from a machine where `gh` is logged in as the owner. None of them diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py new file mode 100644 index 0000000000000..2a68f886fc7df --- /dev/null +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Fork only: warm caches ("seeds") for a desk pool's jobs (.github/fork-fleet.md, "Seeds"). + +A project lists the caches its desk pool's jobs may start from in the pool's `seeds` (fork-fleet.json): a name, how +often a desktop takes a new one (every_hours), the age at which it is dropped (max_age_hours, from when it was made) +and its largest size (max_mb). On the always-on host something of the project's own (a timer, a job) makes each one: +a tar of a folder's contents at FILES/NAME.tar, written to a temp file and renamed. Nothing here makes one. The +desktop pulls them with the desk link's key, in an ssh session of its own beside the agent's: + + ssh ... seeds the forced command (fork-fleet-desk.py serve POOL, the desk-link user) sees it in + SSH_ORIGINAL_COMMAND and prints the pool's list (one JSON object; `bytes` and `made` of each + seed the host offers now, `why` for one it does not) + ssh ... seed NAME streams FILES/NAME.tar: only a name of that list, never a path the desktop names. The last + stderr line is the trailer `seed NAME BYTES SHA256 MADE` (MADE: the file's mtime, epoch seconds) + +The host side exits 2 for any other command, 3 while another stream for the pool runs, 4 when the file may not be +sent (missing, a link, in a folder or a file others may write, above max_mb, older than max_age_hours, or changed +while it was sent: made in place, not by rename). Standard library only. +""" +import fcntl, hashlib, importlib.util, json, os, re, stat, sys, time + +sys.dont_write_bytecode = True +HERE = os.path.dirname(os.path.realpath(__file__)) + + +def module(name, file): + spec = importlib.util.spec_from_file_location(name, os.path.join(HERE, file)) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +FILES = '/var/lib/fork-fleet/seed-files' # the always-on host's: NAME.tar, made by the project +CONFIG = os.path.join(HERE, 'fork-fleet.json') +PIECE = 1 << 20 +ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'} + + +class Refused(Exception): + def __init__(self, msg, code): + super().__init__(msg) + self.code = code + + +def pool_seeds(pool, config): + try: + cfg = module('fork_fleet', 'fork-fleet.py').load(config) + except (OSError, ValueError) as e: + raise Refused(f'the fleet config: {e}', 2) from None + for p in cfg['pools']: + if p['name'] == pool: + return {s['name']: s for s in p.get('seeds', [])} + raise Refused(f'pool {pool} is not in {config}', 2) + + +def offered(seed, files, now): + """An open fd and the stat of FILES/NAME.tar; Refused (4) when it may not be sent.""" + path = os.path.join(files, seed['name'] + '.tar') + try: + st = os.lstat(files) + if not stat.S_ISDIR(st.st_mode) or st.st_mode & 0o022: + raise Refused(f'{files} must be a folder (not a link) that only its owner may write', 4) + fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) + except OSError as e: + raise Refused(f'{path}: {"not made yet" if isinstance(e, FileNotFoundError) else e.strerror}', 4) from None + st = os.fstat(fd) + hours = seed['max_age_hours'] + why = ('not a plain file' if not stat.S_ISREG(st.st_mode) else 'others may write it' if st.st_mode & 0o022 + else f'above max_mb ({seed["max_mb"]})' if st.st_size > seed['max_mb'] << 20 + else f'older than max_age_hours ({hours})' if now - st.st_mtime > hours * 3600 else None) + if why: + os.close(fd) + raise Refused(f'{path}: {why}', 4) + return fd, st + + +def offer(pool, command, lock_dir, out=1, err=sys.stderr, config=CONFIG, files=FILES, now=time.time): + try: + if os.geteuid() == 0: + raise Refused('serve never runs as root', 2) + word, _, name = command.partition(' ') + seeds = pool_seeds(pool, config) + if command == 'seeds': + got = [] + for s in seeds.values(): + item = {'name': s['name'], **{k: s[v] for k, v in ITEM.items()}} + try: + fd, st = offered(s, files, now()) + os.close(fd) + item.update(bytes=st.st_size, made=int(st.st_mtime)) + except Refused as e: + item['why'] = str(e) + got.append(item) + os.write(out, (json.dumps({'v': 1, 'seeds': got}) + '\n').encode()) + return 0 + if word != 'seed' or name not in seeds: + raise Refused(f'refused: {command[:60]!r} (only `seeds`, or `seed NAME` of pool {pool}\'s seeds)', 2) + lock = os.open(os.path.join(lock_dir, f'{pool}.seed.lock'), os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600) + try: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + raise Refused(f'another seed stream for pool {pool} runs', 3) from None + fd, st = offered(seeds[name], files, now()) + try: + sha, sent = send(fd, out) + after = os.fstat(fd) + finally: + os.close(fd) + finally: + os.close(lock) + if sent != st.st_size or (after.st_size, after.st_mtime_ns) != (st.st_size, st.st_mtime_ns): + raise Refused(f'{name}: it changed while it was sent (made in place, not by rename?)', 4) + print(f'seed {name} {sent} {sha} {int(st.st_mtime)}', file=err, flush=True) + return 0 + except Refused as e: + print(f'fork-fleet-desk seed: {e}', file=err, flush=True) + return e.code + + +def send(fd, out): + sha, sent = hashlib.sha256(), 0 + while data := os.read(fd, PIECE): + sha.update(data) + sent += len(data) + view = memoryview(data) + while view: + view = view[os.write(out, view):] + return sha.hexdigest(), sent diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py new file mode 100644 index 0000000000000..6fb4307c5a897 --- /dev/null +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""Tests of fork-fleet-desk-seed.py (a desk pool's warm caches) and the `seeds` check of fork-fleet.json: the host's +offer in-process and the forced command's divert in fork-fleet-desk.py. Temp folders only; no root, no ssh, no +network. Run: python3 -I -B . Linux only.""" +import fcntl, hashlib, importlib.util, io, json, os, shutil, subprocess, sys, tarfile, tempfile, time, unittest +from unittest import mock + +sys.dont_write_bytecode = True +HERE = os.path.dirname(os.path.abspath(__file__)) + + +def load(name, file): + spec = importlib.util.spec_from_file_location(name, os.path.join(HERE, file)) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +seed, ff = load('fork_fleet_desk_seed', 'fork-fleet-desk-seed.py'), load('fork_fleet', 'fork-fleet.py') +with open(os.path.join(HERE, '..', 'fork-fleet.json'), encoding='utf-8') as _f: + CONFIG = json.load(_f) +POOL = CONFIG['pools'][0]['name'] +SEEDS = [{'name': 'm2', 'every_hours': 6, 'max_age_hours': 1000, 'max_mb': 2}, + {'name': 'npm', 'every_hours': 1, 'max_age_hours': 48, 'max_mb': 1}] + +class Base(unittest.TestCase): + def setUp(self): + self.t = os.path.realpath(tempfile.mkdtemp(prefix='seed-test-')) + self.addCleanup(shutil.rmtree, self.t, True) + for d in ('files', 'run', 'src'): + os.mkdir(os.path.join(self.t, d), 0o755) + self.config(SEEDS) + + def config(self, seeds): + cfg = json.loads(json.dumps(CONFIG)) + cfg['pools'][0]['seeds'] = seeds + with open(os.path.join(self.t, 'fork-fleet.json'), 'w', encoding='utf-8') as f: + json.dump(cfg, f) + + def make(self, name, members, mtime=None): + """FILES/NAME.tar of `members` ({path: bytes}), made as a project would: a temp file, then a rename.""" + tmp = os.path.join(self.t, 'files', f'.{name}.tmp') + with tarfile.open(tmp, 'w', format=tarfile.PAX_FORMAT) as tar: + for path, data in members.items(): + info = tarfile.TarInfo(path) + info.size, info.mode = len(data), 0o644 + tar.addfile(info, io.BytesIO(data)) + if mtime is not None: + os.utime(tmp, (mtime, mtime)) + os.chmod(tmp, 0o644) + os.replace(tmp, os.path.join(self.t, 'files', f'{name}.tar')) + with open(os.path.join(self.t, 'files', f'{name}.tar'), 'rb') as f: + return f.read() + + def offer(self, command, **kw): + r, w = os.pipe() + err = io.StringIO() + with os.fdopen(r, 'rb') as rf: + try: + code = seed.offer(POOL, command, os.path.join(self.t, 'run'), out=w, err=err, + config=os.path.join(self.t, 'fork-fleet.json'), files=os.path.join(self.t, 'files'), + **kw) + finally: + os.close(w) + return code, rf.read(), err.getvalue() + + +class Config(unittest.TestCase): + def test_seeds_are_checked(self): + def problems(seeds): + cfg = json.loads(json.dumps(CONFIG)) + cfg['pools'][0]['seeds'] = seeds + return [p for p in ff.check_config(cfg) if 'seed' in p] + self.assertEqual(problems(SEEDS), []) + self.assertEqual(problems([]), []) + good = SEEDS[0] + for bad in ({}, [{**good, 'name': 'M2'}], [{**good, 'name': '../x'}], [{**good, 'every_hours': 0}], + [{**good, 'max_mb': True}], [{**good, 'max_age_hours': 5}], [{**good, 'max_mb': '2'}], + [{k: v for k, v in good.items() if k != 'max_mb'}], ['m2'], [good, good]): + with self.subTest(bad): + self.assertEqual(len(problems(bad)), 1, bad) + + +class Host(Base): + def test_the_list_says_what_is_offered_and_why_not(self): + self.make('m2', {'a': b'x' * 100}) + code, out, err = self.offer('seeds') + self.assertEqual(code, 0, err) + got = json.loads(out) + self.assertEqual(got['v'], 1) + m2, npm = got['seeds'] + st = os.stat(os.path.join(self.t, 'files', 'm2.tar')) + self.assertEqual(m2, {'name': 'm2', 'everyHours': 6, 'maxAgeHours': 1000, 'maxMb': 2, 'bytes': st.st_size, + 'made': int(st.st_mtime)}) + self.assertNotIn('made', npm) + self.assertIn('not made yet', npm['why']) + + def test_a_seed_streams_whole_with_its_trailer(self): + data = self.make('m2', {'a/b': b'hello', 'c': b'x' * 5000}) + code, out, err = self.offer('seed m2') + self.assertEqual(code, 0, err) + self.assertEqual(out, data) + st = os.stat(os.path.join(self.t, 'files', 'm2.tar')) + self.assertEqual(err, f'seed m2 {len(data)} {hashlib.sha256(data).hexdigest()} {int(st.st_mtime)}\n') + + def test_other_commands_and_names_are_refused_and_send_nothing(self): + self.make('m2', {'a': b'x'}) + for command in ('seed', 'seed ../m2', 'seed /etc/passwd', 'seed m2 x', 'seed other', 'seeds x', 'serve x', + 'rm -rf /', 'seed M2'): + with self.subTest(command): + code, out, err = self.offer(command) + self.assertEqual((code, out), (2, b''), err) + + def test_a_file_that_may_not_be_sent_is_exit_4(self): + files = os.path.join(self.t, 'files') + now = time.time() + cases = {'missing': lambda: None, + 'a link': lambda: os.symlink(os.path.join(self.t, 'src', 'x.tar'), f'{files}/m2.tar'), + 'group-writable': lambda: (self.make('m2', {'a': b'x'}), os.chmod(f'{files}/m2.tar', 0o664)), + 'too big': lambda: self.make('m2', {'a': os.urandom(3 << 20)}), + 'too old': lambda: self.make('m2', {'a': b'x'}, mtime=now - 1001 * 3600), + 'a folder': lambda: os.mkdir(f'{files}/m2.tar'), + 'folder others write': lambda: (self.make('m2', {'a': b'x'}), os.chmod(files, 0o777))} + with open(os.path.join(self.t, 'src', 'x.tar'), 'wb') as f: + f.write(b'x') + os.chmod(os.path.join(self.t, 'src', 'x.tar'), 0o644) # only the link itself may be the reason + for name, make in cases.items(): + with self.subTest(name): + make() + code, out, err = self.offer('seed m2') + self.assertEqual((code, out), (4, b''), err) + os.chmod(files, 0o755) + p = f'{files}/m2.tar' + if os.path.isdir(p) and not os.path.islink(p): + os.rmdir(p) + elif os.path.lexists(p): + os.unlink(p) + + def test_a_second_stream_for_the_pool_is_exit_3(self): + self.make('m2', {'a': b'x'}) + with open(os.path.join(self.t, 'run', f'{POOL}.seed.lock'), 'a') as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + self.assertEqual(self.offer('seed m2')[:2], (3, b'')) + self.assertEqual(self.offer('seed m2')[0], 0) # the lock is gone with its holder + + def test_a_file_changed_while_sent_is_exit_4(self): + self.make('m2', {'a': b'x' * 3000}) + path, send = os.path.join(self.t, 'files', 'm2.tar'), seed.send + + def changing(fd, out): + with open(path, 'ab') as f: # made in place, not by rename + f.write(b'\0' * 512) + return send(fd, out) + with mock.patch.object(seed, 'send', changing): + code, _, err = self.offer('seed m2') + self.assertEqual(code, 4) + self.assertIn('changed while it was sent', err) + + def test_never_as_root_and_a_bad_config_is_exit_2(self): + with mock.patch.object(seed.os, 'geteuid', return_value=0): + self.assertEqual(self.offer('seeds')[:2], (2, b'')) + with open(os.path.join(self.t, 'fork-fleet.json'), 'w') as f: + f.write('{') + self.assertEqual(self.offer('seeds')[:2], (2, b'')) + + def test_the_forced_command_diverts_seed_requests(self): + opt = os.path.join(self.t, 'opt') # the host's install: the scripts and fork-fleet.json in one folder + os.mkdir(opt) + for f in ('fork-fleet-desk.py', 'fork-fleet-desk-wire.py', 'fork-fleet-desk-seed.py', 'fork-fleet.py'): + shutil.copy(os.path.join(HERE, f), opt) + shutil.copy(os.path.join(self.t, 'fork-fleet.json'), opt) + desk = os.path.join(opt, 'fork-fleet-desk.py') + base = {'PATH': os.environ['PATH'], 'FORK_FLEET_DESK_DIR': os.path.join(self.t, 'run2')} + got = subprocess.run([sys.executable, '-I', '-B', desk, 'serve', POOL], capture_output=True, text=True, + env={**base, 'SSH_ORIGINAL_COMMAND': 'seeds'}, timeout=30, stdin=subprocess.DEVNULL) + self.assertEqual(got.returncode, 0, got.stderr) + self.assertEqual([s['name'] for s in json.loads(got.stdout)['seeds']], ['m2', 'npm']) + got = subprocess.run([sys.executable, '-I', '-B', desk, 'serve', POOL], capture_output=True, text=True, + env={**base, 'SSH_ORIGINAL_COMMAND': 'seed nothing-like-this'}, timeout=30, + stdin=subprocess.DEVNULL) + self.assertEqual((got.returncode, got.stdout), (2, '')) + self.assertFalse(os.path.exists(os.path.join(self.t, 'run2', f'{POOL}.sock'))) # no session was served + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/scripts/fork-fleet-desk.py b/.github/scripts/fork-fleet-desk.py index 254aec7510e7f..eb08f20058a57 100755 --- a/.github/scripts/fork-fleet-desk.py +++ b/.github/scripts/fork-fleet-desk.py @@ -17,6 +17,8 @@ POOL]. Sends one launcher verb (`start` with its stdin) through the live session and prints the desktop's stdout and stderr and exits with its exit code. No live session: exit 3 at once, nothing sent. No answer in the verb's time (fork-fleet-desk-wire.py, "Time"): exit 75, the verb may or may not have run. + The same forced command asked `seeds` or `seed NAME` (SSH_ORIGINAL_COMMAND): a desktop's warm caches, not a + session (fork-fleet-desk-seed.py; its lock DIR/POOL.seed.lock). DIR is /run/fork-fleet-desk, made at boot by systemd-tmpfiles (runbook): owned by the desk-link user, its group the socket group (who may connect: the dispatcher's user, or root), mode 0710 or 0750. The socket is 0660 in that group, and serve lets in only root, its own user and members of that group (SO_PEERCRED). Not as root (tests), @@ -391,9 +393,25 @@ def launch(pool, verb, args, stdin, path, out=sys.stdout, err=sys.stderr, wait=N return got['exit'] +def seed_side(pool, asked, env): + """A desktop's seed session (fork-fleet-desk-seed.py): the same key and forced command, another command.""" + pool_ok(pool) + if os.geteuid() == 0: + raise Refused('serve never runs as root', 2) + path = sock_dir(env) + safe_dir(path, (os.geteuid(),), make=True) + spec = importlib.util.spec_from_file_location('fork_fleet_desk_seed', os.path.join(HERE, 'fork-fleet-desk-seed.py')) + seed = importlib.util.module_from_spec(spec) + spec.loader.exec_module(seed) + return seed.offer(pool, asked, path) + + def main(argv=None, env=os.environ): argv = sys.argv[1:] if argv is None else argv try: + if len(argv) == 2 and argv[0] == 'serve' and (env.get('SSH_ORIGINAL_COMMAND') == 'seeds' or + env.get('SSH_ORIGINAL_COMMAND', '').startswith('seed ')): + return seed_side(argv[1], env['SSH_ORIGINAL_COMMAND'], env) if len(argv) == 2 and argv[0] == 'serve': for sig in (signal.SIGTERM, signal.SIGHUP): signal.signal(sig, lambda *_: sys.exit(1)) # through serve's finally: the socket goes, clients learn diff --git a/.github/scripts/fork-fleet.py b/.github/scripts/fork-fleet.py index 95a1be08f69b6..bec1f34d7718d 100644 --- a/.github/scripts/fork-fleet.py +++ b/.github/scripts/fork-fleet.py @@ -139,6 +139,15 @@ def check_config(cfg): who = f'pool {p.get("name")!r}' if not (isinstance(p.get('name'), str) and POOL.fullmatch(p['name'])): out.append(f'{who}: name must be a-z, 0-9, _ and - (it names the dispatcher\'s lock file)') + seeds = p.get('seeds', []) # a desk pool's warm caches: fork-fleet-desk-seed.py + if not (isinstance(seeds, list) and all( + isinstance(x, dict) and isinstance(x.get('name'), str) and POOL.fullmatch(x['name']) + and all(positive(x.get(k)) for k in ('every_hours', 'max_age_hours', 'max_mb')) + and x['max_age_hours'] >= x['every_hours'] for x in seeds)): + out.append(f'{who}: seeds must list {{"name" (a-z, 0-9, _, -), "every_hours", "max_age_hours" (at least ' + 'every_hours), "max_mb"}, each a positive whole number') + elif len({x['name'] for x in seeds}) != len(seeds): + out.append(f'{who}: two seeds with one name') if not positive(p.get('slots')): out.append(f'{who}: slots must be a positive whole number') if not isinstance(p.get('waits'), bool): From ec41bd876207d62718749348df5273eae1148f42 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 19:14:00 +0000 Subject: [PATCH 05/33] fork fleet: a desktop fetches its desk pool's seeds (D11b) fork-fleet-desk-seed.py run, hourly from t3code-fleet-seed.timer (root, sandboxed), asks the always-on host for the pool's seed list over the desk link's key, fetches a seed only when it changed and every_hours passed, unpacks it with tarfile's data filter, checks the trailer's size and SHA-256 and swaps it in by rename. Jobs see /var/lib/fork-fleet/seeds read-only (ALLOW_PATHS). Torn downloads keep the old seed; old, future-dated or unlisted seeds are dropped; replaced versions stay a day for running jobs. The desk kit installs the script and the timer and removes the seeds on uninstall. Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 8 + .github/scripts/fork-fleet-desk-seed.py | 257 ++++++++++++++++++- .github/scripts/fork-fleet-desk-seed.test.py | 236 ++++++++++++++++- .github/scripts/fork-fleet-seed.service | 44 ++++ .github/scripts/fork-fleet-seed.timer | 13 + fork-kit/desk/README.md | 7 +- fork-kit/desk/sbin/desk-install | 19 +- fork-kit/desk/tests/install.test.py | 19 +- 8 files changed, 585 insertions(+), 18 deletions(-) create mode 100644 .github/scripts/fork-fleet-seed.service create mode 100644 .github/scripts/fork-fleet-seed.timer diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index 1a0b7c895f3a5..ae52011a38f6b 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -709,6 +709,14 @@ own: the same forced command answers `seeds` with the pool's list and size and SHA-256 (`fork-fleet-desk-seed.py`, which step 3 installs; its header has the exit codes). +The desktop's side is the same script's `run`, hourly from +`t3code-fleet-seed.timer` (the desk kit installs both). Each new seed is +unpacked with Python's tar data filter, checked against the trailer, and +swapped in by a rename; jobs see `/var/lib/fork-fleet/seeds/NAME` read-only +(the launcher's `ALLOW_PATHS`), copy what they need from it when it is there, +and run cold when it is not. The state table (a torn download, the host down, +clocks that disagree, a run that stopped half-way) is in the script's header. + ## Owner steps on the repository Run these from a machine where `gh` is logged in as the owner. None of them diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index 2a68f886fc7df..40a8622214178 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -12,12 +12,33 @@ seed the host offers now, `why` for one it does not) ssh ... seed NAME streams FILES/NAME.tar: only a name of that list, never a path the desktop names. The last stderr line is the trailer `seed NAME BYTES SHA256 MADE` (MADE: the file's mtime, epoch seconds) + python3 -I fork-fleet-desk-seed.py [--config FILE] run + the desktop, as root, hourly (t3code-fleet-seed.timer), with the agent's settings (FILE, default + its CONF). Each new seed is unpacked with tarfile's data filter into a folder of its own, checked + against the trailer, flushed, and swapped in by rename: SEEDS/NAME is a link to + .v/NAME.STAMP/data, root's (folders 0755, files no one else may write), unpacked in .new + (one folder: a rename never crosses a mount of the unit's sandbox). Jobs see SEEDS read-only + (the launcher's ALLOW_PATHS): a job uses /var/lib/fork-fleet/seeds/NAME when it is there and runs + cold when it is not. +Each run (one at a time: a lock) meets this state: + the same seed on the host (bytes and MADE) nothing is fetched + a new one under every_hours after the last nothing yet + a torn copy (the stream cut or past FETCH_WAIT, a trailer that does not match, more than max_mb, a tar member + that would leave its folder or is a device) the copy is removed, the old seed stays, the next hour tries again + no seed (never fetched, the host down) jobs run cold + older than max_age_hours, made over AHEAD in the future (the clocks disagree), or no longer in the pool's list + the link is removed: jobs run cold + a label that is missing or unreadable the link is removed and the seed fetched again in the same run + a run that stopped half-way SEEDS/.new is emptied by the next run; a link is the old one or the new + a replaced version kept KEEP_OLD after it was replaced, for a job still reading it (a job + runs at most the launcher's MAX_SECONDS, 1 h by default), then removed The host side exits 2 for any other command, 3 while another stream for the pool runs, 4 when the file may not be sent (missing, a link, in a folder or a file others may write, above max_mb, older than max_age_hours, or changed while it was sent: made in place, not by rename). Standard library only. """ -import fcntl, hashlib, importlib.util, json, os, re, stat, sys, time +import argparse, fcntl, hashlib, importlib.util, json, os, re, shutil, stat, subprocess, sys, tarfile, tempfile +import threading, time sys.dont_write_bytecode = True HERE = os.path.dirname(os.path.realpath(__file__)) @@ -31,7 +52,12 @@ def module(name, file): FILES = '/var/lib/fork-fleet/seed-files' # the always-on host's: NAME.tar, made by the project +SEEDS = '/var/lib/fork-fleet/seeds' # the desktop's: what jobs see CONFIG = os.path.join(HERE, 'fork-fleet.json') +NAME = re.compile(r'[a-z0-9][a-z0-9_-]{0,39}') +LINK = re.compile(r'\.v/((?P[a-z0-9][a-z0-9_-]{0,39})\.[0-9]{1,20})/data') +LIST_WAIT, FETCH_WAIT = 30, 30 * 60 # seconds for the list, and for one seed's whole stream +KEEP_OLD, AHEAD = 24 * 3600, 3600 PIECE = 1 << 20 ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'} @@ -42,6 +68,8 @@ def __init__(self, msg, code): self.code = code +# ---------------------------------------------------------------- the always-on host (serve's forced command) + def pool_seeds(pool, config): try: cfg = module('fork_fleet', 'fork-fleet.py').load(config) @@ -127,3 +155,230 @@ def send(fd, out): while view: view = view[os.write(out, view):] return sha.hexdigest(), sent + + +# ---------------------------------------------------------------- the desktop + +class Torn(Exception): + pass + + +class Summed: + """The ssh stream as tarfile reads it: counted, hashed, cut past `most` bytes.""" + + def __init__(self, raw, most): + self.raw, self.most, self.size, self.sha = raw, most, 0, hashlib.sha256() + + def read(self, n=PIECE): + data = self.raw.read(n if n is not None and n >= 0 else PIECE) + self.size += len(data) + if self.size > self.most: + raise Torn(f'more than {self.most >> 20} MB (max_mb)') + self.sha.update(data) + return data + + +def ssh(agent, conf, *words): + return agent.ssh_argv(conf)[:-2] + list(words) # the agent's ssh, another command than `serve POOL` + + +def listing(agent, conf): + """{name: item} of the host's list; Torn when it cannot be had or is not well formed.""" + with tempfile.TemporaryFile() as err: + try: + r = subprocess.run(ssh(agent, conf, 'seeds'), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=err, + env=agent.ENV, timeout=LIST_WAIT, start_new_session=True) + except subprocess.TimeoutExpired: + raise Torn(f'no list in {LIST_WAIT} s') from None + except OSError as e: + raise Torn(f'no ssh: {e}') from None + if r.returncode: + raise Torn(f'no list (exit {r.returncode}): {tail(err)}') + try: + got = json.loads(r.stdout) + assert isinstance(got, dict) and got.get('v') == 1 and isinstance(got.get('seeds'), list) + out = {} + for s in got['seeds']: + assert isinstance(s, dict) and isinstance(s.get('name'), str) and NAME.fullmatch(s['name']) + assert all(whole(s.get(k)) and s[k] > 0 for k in ITEM) and ('made' not in s or whole(s.get('bytes')) + and whole(s['made'])) + out[s['name']] = s + return out + except (ValueError, AssertionError): + raise Torn('the host\'s list is not well formed') from None + + +def whole(x): + return isinstance(x, int) and not isinstance(x, bool) and x >= 0 + + +def tail(f): + f.seek(0) + return ' '.join(f.read()[-2000:].decode('utf-8', 'replace').split())[-300:] or 'nothing on stderr' + + +def fetch(agent, conf, item, root, now): + """The seed `item` into a new version, swapped in; its label. Torn (or OSError, TarError) leaves the old one.""" + name, stamp = item['name'], time.time_ns() + new = os.path.join(root + '/.new', f'{name}.{stamp}') + os.mkdir(new, 0o755) + os.mkdir(os.path.join(new, 'data'), 0o755) + with tempfile.TemporaryFile() as err: + p = subprocess.Popen(ssh(agent, conf, 'seed', name), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, + stderr=err, env=agent.ENV, start_new_session=True) + timer = threading.Timer(FETCH_WAIT, p.kill) + timer.start() + try: + got = Summed(p.stdout, item['maxMb'] << 20) + with tarfile.open(fileobj=got, mode='r|') as t: + t.extractall(os.path.join(new, 'data'), filter='data') + while got.read(PIECE): # the tar's end blocks and padding count in the trailer's size and sha256 + pass + code = p.wait(timeout=30) + finally: + timer.cancel() + p.kill() + p.wait() + p.stdout.close() + err.seek(0) + last = (err.read()[-4096:].decode('utf-8', 'replace').strip().splitlines() or [''])[-1].split() + if code or last[:4] != ['seed', name, str(got.size), got.sha.hexdigest()] or len(last) != 5 or \ + not last[4].isdigit(): + raise Torn(f'exit {code}, {got.size} bytes, the trailer does not match: {tail(err)}') + label = {'v': 1, 'name': name, 'bytes': got.size, 'sha256': got.sha.hexdigest(), 'made': int(last[4]), + 'fetched': int(now()), 'maxAgeHours': item['maxAgeHours']} + with open(os.path.join(new, 'label.json'), 'x', encoding='utf-8') as f: + json.dump(label, f) + os.sync() # the new version is on disk before any link names it + version = os.path.join(root, '.v', os.path.basename(new)) + os.rename(new, version) + link = os.path.join(root, f'.{name}.link') + remove(link) + os.symlink(f'.v/{name}.{stamp}/data', link) + old = current(root, name)[0] + if old: + os.utime(old, (now(), now())) # replaced now: kept KEEP_OLD for a job still reading it + os.replace(link, os.path.join(root, name)) + return label + + +def current(root, name): + """(version folder or None, label or None) of the link SEEDS/NAME.""" + try: + m = LINK.fullmatch(os.readlink(os.path.join(root, name))) + except OSError: + return None, None + if not m or m['name'] != name: + return None, None + version = os.path.join(root, '.v', m[1]) + try: + with open(os.path.join(version, 'label.json'), encoding='utf-8') as f: + label = json.load(f) + ok = isinstance(label, dict) and label.get('name') == name and all( + whole(label.get(k)) for k in ('bytes', 'made', 'fetched', 'maxAgeHours')) + except (OSError, ValueError): + ok = False + return version, label if ok else None + + +def remove(path): + try: + st = os.lstat(path) + except FileNotFoundError: + return + if stat.S_ISDIR(st.st_mode): + shutil.rmtree(path) + else: + os.unlink(path) + + +def drop(root, name, version, now): + if version and os.path.isdir(version): + os.utime(version, (now(), now())) # first: a run that stops here leaves it linked, never pruned early + remove(os.path.join(root, name)) + + +def run(agent, conf, root=SEEDS, now=time.time, say=print): + if not hasattr(tarfile, 'data_filter'): + raise agent.Bad('this Python has no tarfile data filter (3.12, or 3.8.17 and later): no seeds') + os.umask(0o022) + for d in (root, root + '/.new', root + '/.v'): # root: made by the unit (StateDirectory) or here + os.makedirs(d, 0o755, exist_ok=True) + with open(os.path.join(root + '/.new', '.lock'), 'a') as lock: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + say('another seed run is running') + return 1 + for e in os.listdir(root + '/.new'): # a run that stopped half-way + if e != '.lock': + remove(os.path.join(root + '/.new', e)) + try: + items = listing(agent, conf) + except Torn as e: + items = None + say(f'seeds: {e}: each seed stays until its age drops it') + for name in sorted(n for n in os.listdir(root) if NAME.fullmatch(n)): + version, label = current(root, name) + if label is None: + why = 'its label is missing or cannot be read (fetched again now)' + elif items is not None and name not in items: + why = 'no longer one of the pool\'s seeds' + else: + age = now() - label['made'] + hours = items[name]['maxAgeHours'] if items else label['maxAgeHours'] + why = (f'older than {hours} h (max_age_hours)' if age > hours * 3600 + else 'made in the future (the clocks disagree)' if age < -AHEAD else None) + if why: + drop(root, name, version, now) + say(f'seed {name}: dropped, {why}; jobs run cold until a new one comes') + for name, item in sorted((items or {}).items()): + label = current(root, name)[1] + if 'made' not in item: + say(f'seed {name}: the host offers none ({item.get("why", "no reason")}): ' + f'{"the one here stays" if label else "jobs run cold"}') + continue + if label and (label['bytes'], label['made']) == (item['bytes'], item['made']): + continue + if label and 0 <= now() - label['fetched'] < item['everyHours'] * 3600: + continue + if not -AHEAD <= now() - item['made'] <= item['maxAgeHours'] * 3600: + say(f'seed {name}: the host\'s is too old or made in the future (the clocks disagree): not fetched') + continue + try: + got = fetch(agent, conf, item, root, now) + say(f'seed {name}: {got["bytes"] >> 20} MB, sha256 checked, swapped in') + except (Torn, OSError, tarfile.TarError, subprocess.SubprocessError) as e: + for e2 in os.listdir(root + '/.new'): + if e2 != '.lock': + remove(os.path.join(root + '/.new', e2)) + say(f'seed {name}: keeps {"the old one" if label else "none"}: {e.__class__.__name__}: ' + f'{str(e)[:300]}; tried again next hour') + prune(root, now) + return 0 + + +def prune(root, now): + """Versions no link names, KEEP_OLD after they were replaced or dropped.""" + linked = {current(root, n)[0] for n in os.listdir(root) if NAME.fullmatch(n)} + for e in os.listdir(root + '/.v'): + path = os.path.join(root, '.v', e) + if path not in linked and now() - os.lstat(path).st_mtime > KEEP_OLD: + remove(path) + + +def main(argv=None): + a = argparse.ArgumentParser(description='a desk pool\'s warm caches, the desktop\'s side (as root)') + agent = module('fork_fleet_desk_agent', 'fork-fleet-desk-agent.py') + a.add_argument('--config', default=agent.CONF) + a.add_argument('verb', choices=('run',)) + a = a.parse_args(argv) + try: + return run(agent, agent.load(a.config)) + except agent.Bad as e: + print(f'fork-fleet-desk-seed: {e}', file=sys.stderr) + return 2 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 6fb4307c5a897..9a13b16bfe108 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 """Tests of fork-fleet-desk-seed.py (a desk pool's warm caches) and the `seeds` check of fork-fleet.json: the host's -offer in-process and the forced command's divert in fork-fleet-desk.py. Temp folders only; no root, no ssh, no -network. Run: python3 -I -B . Linux only.""" -import fcntl, hashlib, importlib.util, io, json, os, shutil, subprocess, sys, tarfile, tempfile, time, unittest +offer in-process, the desktop's run end to end through a fake ssh that runs the host's side, and the forced command's +divert in fork-fleet-desk.py. Temp folders only; no root, no ssh, no network. Run: python3 -I -B . Linux only.""" +import fcntl, hashlib, importlib.util, io, json, os, shutil, stat, subprocess, sys, tarfile, tempfile, time, unittest from unittest import mock sys.dont_write_bytecode = True @@ -17,12 +17,49 @@ def load(name, file): seed, ff = load('fork_fleet_desk_seed', 'fork-fleet-desk-seed.py'), load('fork_fleet', 'fork-fleet.py') +agent = load('fork_fleet_desk_agent', 'fork-fleet-desk-agent.py') with open(os.path.join(HERE, '..', 'fork-fleet.json'), encoding='utf-8') as _f: CONFIG = json.load(_f) POOL = CONFIG['pools'][0]['name'] SEEDS = [{'name': 'm2', 'every_hours': 6, 'max_age_hours': 1000, 'max_mb': 2}, {'name': 'npm', 'every_hours': 1, 'max_age_hours': 48, 'max_mb': 1}] +FAKE_SSH = '''#!{python} -I +import io, os, sys +sys.dont_write_bytecode = True +import importlib.util +spec = importlib.util.spec_from_file_location('s', {module!r}) +seed = importlib.util.module_from_spec(spec) +spec.loader.exec_module(seed) +args = sys.argv[1:] +words = args[args.index('--') + 2:] +with open({t!r} + '/calls', 'a') as f: + f.write(' '.join(words) + '\\n') +try: + with open({t!r} + '/mode') as f: + mode = f.read().strip() +except FileNotFoundError: + mode = '' +if mode == 'down': + print('ssh: connect to host h port 22: No route to host', file=sys.stderr) + sys.exit(255) +kw = dict(config={t!r} + '/fork-fleet.json', files={t!r} + '/files') +if words[0] == 'seed' and mode in ('cut', 'trailer'): + with open({t!r} + '/sent', 'w+b') as out: + err = io.StringIO() + code = seed.offer({pool!r}, ' '.join(words), {t!r} + '/run', out=out.fileno(), err=err, **kw) + out.seek(0) + data = out.read() + if mode == 'cut': + os.write(1, data[:len(data) // 2]) + sys.exit(255) + os.write(1, data) + print(err.getvalue().rsplit(' ', 2)[0] + ' ' + '0' * 64 + ' 1', file=sys.stderr) # a wrong sha256 + sys.exit(code) +sys.exit(seed.offer({pool!r}, ' '.join(words), {t!r} + '/run', **kw)) +''' + + class Base(unittest.TestCase): def setUp(self): self.t = os.path.realpath(tempfile.mkdtemp(prefix='seed-test-')) @@ -182,5 +219,198 @@ def test_the_forced_command_diverts_seed_requests(self): self.assertFalse(os.path.exists(os.path.join(self.t, 'run2', f'{POOL}.sock'))) # no session was served +class Desktop(Base): + def setUp(self): + super().setUp() + self.root = os.path.join(self.t, 'seeds') + ssh = os.path.join(self.t, 'ssh') + with open(ssh, 'w') as f: + f.write(FAKE_SSH.format(python=sys.executable, module=seed.__file__, t=self.t, pool=POOL)) + os.chmod(ssh, 0o755) + self.conf = {'SSH': ssh, 'KEY': '/nonexistent/key', 'KNOWN_HOSTS': '/nonexistent/known', 'PORT': '22', + 'USER': 'fleet-desk', 'HOST': 'fleet-host.example', 'POOL': POOL} + self.said = [] + sync = mock.patch.object(seed.os, 'sync') + sync.start() + self.addCleanup(sync.stop) + + def run_seed(self, now=None): + self.said = [] + return seed.run(agent, self.conf, self.root, now=now or time.time, say=self.said.append) + + def mode(self, text): + with open(os.path.join(self.t, 'mode'), 'w') as f: + f.write(text) + + def fetches(self): + try: + with open(os.path.join(self.t, 'calls')) as f: + return [line for line in f.read().splitlines() if line.startswith('seed ')] + except FileNotFoundError: + return [] + + def files(self, name): + top = os.path.join(self.root, name) + if not os.path.exists(top): + return None + out = {} + for d, _, names in os.walk(top): + for n in names: + with open(os.path.join(d, n), 'rb') as f: + out[os.path.relpath(os.path.join(d, n), top)] = f.read() + return out + + def test_fetched_checked_and_swapped_in_then_not_again(self): + self.make('m2', {'repo/a.jar': b'jar', 'repo/b/c.pom': b'pom'}) + self.assertEqual(self.run_seed(), 0) + self.assertEqual(self.files('m2'), {'repo/a.jar': b'jar', 'repo/b/c.pom': b'pom'}) + self.assertTrue(os.readlink(os.path.join(self.root, 'm2')).startswith('.v/m2.')) + self.assertIsNone(self.files('npm')) + self.assertTrue(any('m2:' in s and 'swapped in' in s for s in self.said), self.said) + self.assertTrue(any('npm' in s and 'jobs run cold' in s for s in self.said), self.said) + for d, dirs, names in os.walk(self.root): + for n in dirs + names: + st = os.lstat(os.path.join(d, n)) + if not stat.S_ISLNK(st.st_mode): # a link's own mode is always 0777 and means nothing + self.assertFalse(st.st_mode & 0o022, os.path.join(d, n)) + label = json.load(open(os.path.join(seed.current(self.root, 'm2')[0], 'label.json'))) + self.assertEqual(label['bytes'], os.path.getsize(os.path.join(self.t, 'files', 'm2.tar'))) + self.assertEqual(self.run_seed(), 0) # the same seed on the host: nothing fetched + self.assertEqual(self.fetches(), ['seed m2']) + self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) + + def test_a_new_seed_waits_for_every_hours_and_the_old_stays_for_running_jobs(self): + self.make('m2', {'a': b'one'}) + self.run_seed() + old = seed.current(self.root, 'm2')[0] + self.make('m2', {'a': b'two'}, mtime=time.time() + 5) + self.run_seed() + self.assertEqual(self.files('m2'), {'a': b'one'}) # under every_hours since the last fetch + later = time.time() + seed.KEEP_OLD + 3600 # a swap long after the old one came: kept from the swap on + self.run_seed(now=lambda: later) + self.assertEqual(self.files('m2'), {'a': b'two'}) + self.assertTrue(os.path.isdir(old)) # a job still reading it keeps it + self.run_seed(now=lambda: later + seed.KEEP_OLD + 60) + self.assertFalse(os.path.exists(old)) + self.assertEqual(self.files('m2'), {'a': b'two'}) + self.assertEqual(os.listdir(os.path.join(self.root, '.v')), [os.path.basename(seed.current(self.root, + 'm2')[0])]) + + def torn(self, setup, want_in_say): + self.make('m2', {'a': b'one'}) + self.run_seed() + setup() + self.run_seed(now=lambda: time.time() + 7 * 3600) + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) + self.assertEqual(len(os.listdir(os.path.join(self.root, '.v'))), 1) + self.assertTrue(any('keeps the old one' in s and any(w in s for w in want_in_say.split('|')) + for s in self.said), self.said) + + def test_a_cut_stream_leaves_the_old_seed(self): + self.torn(lambda: (self.make('m2', {'a': os.urandom(200000)}, mtime=time.time() + 5), self.mode('cut')), + 'Torn|ReadError') # cut in the tar or after it + + def test_a_trailer_that_does_not_match_leaves_the_old_seed(self): + self.torn(lambda: (self.make('m2', {'a': b'two'}, mtime=time.time() + 5), self.mode('trailer')), + 'trailer does not match') + + def test_more_than_max_mb_leaves_the_old_seed(self): + def bigger(): + self.config([{**SEEDS[0], 'max_mb': 3}, SEEDS[1]]) # the host offers it; the list the desktop sees + self.make('m2', {'a': os.urandom(5 << 19)}, mtime=time.time() + 5) + with mock.patch.object(seed, 'listing', lambda a, c, real=seed.listing: { + k: {**v, 'maxMb': 2} for k, v in real(a, c).items()}): + self.run_seed(now=lambda: time.time() + 7 * 3600) + self.make('m2', {'a': b'one'}) + self.run_seed() + bigger() + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertTrue(any('max_mb' in s for s in self.said), self.said) + + def test_a_tar_that_would_leave_its_folder_writes_nothing_outside(self): + bad = {'a ../ path': ('../../escaped', tarfile.REGTYPE, ''), 'a device': ('escaped', tarfile.CHRTYPE, ''), + 'an absolute link': ('escaped', tarfile.SYMTYPE, '/'), + 'a link up': ('escaped', tarfile.SYMTYPE, '../../..'), + 'a hard link out': ('escaped', tarfile.LNKTYPE, '/etc/hostname')} + for why, (name, kind, target) in bad.items(): + def evil(): + tmp = os.path.join(self.t, 'files', '.m2.tmp') + with tarfile.open(tmp, 'w') as tar: # one bad member each: the filter stops at the first + ok = tarfile.TarInfo('ok') + ok.size = 1 + tar.addfile(ok, io.BytesIO(b'x')) + info = tarfile.TarInfo(name) + info.type, info.linkname = kind, target + tar.addfile(info) + os.utime(tmp, (time.time() + 5,) * 2) + os.replace(tmp, os.path.join(self.t, 'files', 'm2.tar')) + with self.subTest(why): + shutil.rmtree(self.root, True) + self.torn(evil, 'Error') + for d, dirs, names in os.walk(self.t): + self.assertNotIn('escaped', dirs + names, d) + + def test_dropped_when_no_longer_listed_or_too_old_or_from_the_future(self): + self.make('m2', {'a': b'one'}) + self.run_seed() + self.run_seed(now=lambda: time.time() - 2 * 3600) # the desktop's clock is behind: made in the future + self.assertIsNone(self.files('m2')) + self.assertTrue(any('clocks disagree' in s for s in self.said), self.said) + self.run_seed() + self.assertIsNotNone(self.files('m2')) + old = seed.current(self.root, 'm2')[0] + self.run_seed(now=lambda: time.time() + 1001 * 3600) # over max_age_hours, and the host's is too + self.assertIsNone(self.files('m2')) + self.assertTrue(os.path.isdir(old)) # dropped, but a job still reading it keeps it + self.assertTrue(any('older than 1000 h' in s for s in self.said), self.said) + self.run_seed() + self.config([SEEDS[1]]) + self.run_seed() + self.assertIsNone(self.files('m2')) + self.assertTrue(any('no longer one of the pool' in s for s in self.said), self.said) + + def test_host_down_keeps_each_seed_until_its_own_age_drops_it(self): + self.make('m2', {'a': b'one'}) + self.run_seed() + self.mode('down') + self.assertEqual(self.run_seed(now=lambda: time.time() + 999 * 3600), 0) + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertTrue(any('no list' in s for s in self.said), self.said) + self.run_seed(now=lambda: time.time() + 1001 * 3600) + self.assertIsNone(self.files('m2')) + + def test_a_run_that_stopped_half_way_and_bad_links_or_labels(self): + self.make('m2', {'a': b'one'}) + self.run_seed() + os.makedirs(os.path.join(self.root, '.new', 'm2.123', 'data')) # a torn unpack + os.symlink('.v/m2.999/data', os.path.join(self.root, '.m2.link')) # a link made, not yet swapped in + os.symlink('/etc', os.path.join(self.root, 'npm')) # not a link this program makes + os.unlink(os.path.join(seed.current(self.root, 'm2')[0], 'label.json')) + self.run_seed() + self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) + self.assertFalse(os.path.lexists(os.path.join(self.root, 'npm'))) + self.assertTrue(os.path.isdir('/etc')) + self.assertEqual(self.files('m2'), {'a': b'one'}) # fetched again in the same run + self.assertEqual(len(self.fetches()), 2) + self.assertTrue(any('label is missing' in s for s in self.said), self.said) + + def test_one_run_at_a_time(self): + os.makedirs(os.path.join(self.root, '.new')) + with open(os.path.join(self.root, '.new', '.lock'), 'a') as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + self.assertEqual(self.run_seed(), 1) + self.assertEqual(self.fetches(), []) + + def test_bad_settings_are_exit_2(self): + bad = os.path.join(self.t, 'desk.conf') + with open(bad, 'w') as f: + f.write('POOL=x\n') + os.chmod(bad, 0o666) + with mock.patch('sys.stderr', io.StringIO()) as err: + self.assertEqual(seed.main(['--config', bad, 'run']), 2) + self.assertIn('fork-fleet-desk-seed:', err.getvalue()) + + if __name__ == '__main__': unittest.main() diff --git a/.github/scripts/fork-fleet-seed.service b/.github/scripts/fork-fleet-seed.service new file mode 100644 index 0000000000000..901eee69a986b --- /dev/null +++ b/.github/scripts/fork-fleet-seed.service @@ -0,0 +1,44 @@ +# Fork only: a desk pool's warm caches, the desktop's side (fork-fleet-desk-seed.py run; .github/fork-fleet.md, +# "Seeds"). The desk kit installs it as /etc/systemd/system/t3code-fleet-seed.service with its timer; not as +# fork-fleet-*, the runner units' prefix. Safe to start by hand: a second run while one runs exits at once (a lock). +# +# Root, so the seeds are root's and no job can change one; sandboxed: it may write only the seeds' folder +# (StateDirectory makes it, 0755; new copies are unpacked inside it, so a rename never crosses a mount), reads the +# desk link's settings and key in /etc/fork-fleet, reaches the network (ssh to the always-on host only, with the +# agent's pinned host key) and has a memory, CPU and task limit. +[Unit] +Description=t3code fork fleet: warm caches for a desk pool's jobs +Wants=network-online.target +After=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/python3 -I /opt/fork-fleet/fork-fleet-desk-seed.py run +TimeoutStartSec=3h +Nice=10 +IOSchedulingClass=idle +UMask=0022 +StateDirectory=fork-fleet/seeds +StateDirectoryMode=0755 +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +PrivateDevices=yes +NoNewPrivileges=yes +RestrictSUIDSGID=yes +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +RestrictNamespaces=yes +CapabilityBoundingSet=CAP_DAC_OVERRIDE +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +LockPersonality=yes +RestrictRealtime=yes +SystemCallArchitectures=native +MemoryMax=512M +MemorySwapMax=0 +CPUQuota=100% +TasksMax=32 diff --git a/.github/scripts/fork-fleet-seed.timer b/.github/scripts/fork-fleet-seed.timer new file mode 100644 index 0000000000000..8cb4bfb1424f8 --- /dev/null +++ b/.github/scripts/fork-fleet-seed.timer @@ -0,0 +1,13 @@ +# Fork only: runs t3code-fleet-seed.service every hour, at a random minute in its first 10, and after a wake or a +# boot when a run was missed (a desktop that slept). A run with nothing new asks the always-on host for one short +# list and fetches nothing; each seed's every_hours (fork-fleet.json) paces the downloads. +[Unit] +Description=t3code fork fleet: hourly warm caches for a desk pool + +[Timer] +OnCalendar=hourly +RandomizedDelaySec=10min +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 01e97eb99b5f9..57f2268d951e0 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -48,6 +48,9 @@ the refresh timer; `check` says FAIL for `template` until it is there. `check` only reads: the installed files against their manifest, every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the runner users and IPv4 forwarding. +The seed timer (`t3code-fleet-seed.timer`) fetches the pool's warm caches, if its fleet config lists any, into +`/var/lib/fork-fleet/seeds`, which jobs see read-only ([fork-fleet.md](../../.github/fork-fleet.md), "Seeds"). + ## Windows (WSL2) [windows/start-desk.ps1](windows/start-desk.ps1) does the Windows side and then this kit inside one WSL2 distro. @@ -80,8 +83,8 @@ stay (`wsl --unregister DISTRO` removes the distro). before anything changes. Remove the older (by hand) install first, the runbook's steps in reverse. - **Empty:** `check` before an install says FAIL and writes nothing; `uninstall` with no record does nothing. -Uninstall stops every runner, removes the namespaces and the `fork_fleet` nft table, the kit's units, files and -settings, and only the users it made itself. +Uninstall stops every runner, removes the namespaces and the `fork_fleet` nft table, the kit's units, files, +settings and seeds, and only the users it made itself. ## Why one flat folder diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index 59e560a91f11c..ba3dede670ad5 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -8,8 +8,8 @@ install reads the `desk` section of the machine file, makes the runner users, stages the fleet files of one release (launcher, walls, desk agent, gate hook and the fleet config), checks the staged copy, stops the agent, moves each file into /opt/fork-fleet, writes MANIFEST.sha256 and then SOURCE (last), writes the settings in /etc/fork-fleet and -the units, makes the desk key once, and starts the agent and the refresh timer. Safe to run again and to stop at -any step: a stopped run leaves SOURCE naming the old release (check says what differs) and the next run ends it. +the units, makes the desk key once, and starts the agent and the refresh and seed timers. Safe to run again and to +stop at any step: a stopped run leaves SOURCE naming the old release (check says what differs) and the next run ends it. """ import argparse, contextlib, hashlib, importlib.util, json, os, re, shutil, subprocess, sys, tempfile @@ -27,6 +27,7 @@ FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh', 'fork-fleet-walls.py': '.github/scripts/fork-fleet-walls.py', 'fork-fleet-desk-agent.py': '.github/scripts/fork-fleet-desk-agent.py', 'fork-fleet-desk-wire.py': '.github/scripts/fork-fleet-desk-wire.py', + 'fork-fleet-desk-seed.py': '.github/scripts/fork-fleet-desk-seed.py', 'hook.sh': '.github/scripts/fork-fleet-hook.sh', 'fork-fleet.py': '.github/scripts/fork-fleet.py', 'fork-fleet.json': '.github/fork-fleet.json', 'desk-install': 'fork-kit/desk/sbin/desk-install', 'machine_file.py': 'fork-kit/common/machine_file.py', 'README.md': 'fork-kit/desk/README.md', @@ -34,8 +35,12 @@ FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh', WALLS = '.github/scripts/fork-fleet-walls' # copied whole: every OS's walls file and fixture UNITS = {'fork-fleet-desk-agent.service': '.github/scripts/fork-fleet-desk-agent.service', 't3code-fleet-refresh.service': '.github/scripts/fork-fleet-refresh.service', - 't3code-fleet-refresh.timer': '.github/scripts/fork-fleet-refresh.timer'} + 't3code-fleet-refresh.timer': '.github/scripts/fork-fleet-refresh.timer', + 't3code-fleet-seed.service': '.github/scripts/fork-fleet-seed.service', + 't3code-fleet-seed.timer': '.github/scripts/fork-fleet-seed.timer'} AGENT, TIMER, REFRESH = 'fork-fleet-desk-agent.service', 't3code-fleet-refresh.timer', 't3code-fleet-refresh.service' +SEED_TIMER, SEED = 't3code-fleet-seed.timer', 't3code-fleet-seed.service' +SEEDS = WORK + '/seeds' # the warm caches (fork-fleet-desk-seed.py); jobs see them read-only (ALLOW_PATHS) FOREIGN = (OPT + '/fork-fleet-launch.sh', ETC + '/launch.conf', ETC + '/desk.conf', KEY, *(f'{UNIT_DIR}/{u}' for u in UNITS)) CMD = {k: [k] for k in ('useradd', 'userdel', 'loginctl', 'systemctl', 'sysctl', 'ssh-keygen', 'ip', 'nft', 'bash', @@ -252,7 +257,7 @@ def make_users(users, state): def settings(m, users): known = (m['host'] if m['port'] == 22 else f'[{m["host"]}]:{m["port"]}') + ' ' + m['hostKey'] - files = {ETC + '/launch.conf': (f'RUNNER_USERS={",".join(users)}\n' + files = {ETC + '/launch.conf': (f'RUNNER_USERS={",".join(users)}\nALLOW_PATHS={SEEDS}\n' + (f'RESERVE_MB={m["reserveMb"]}\n' if m['reserveMb'] is not None else '') + (f'NET_BASE={m["netBase"]}\n' if m['netBase'] else '') + (f'FREE_COMMAND={m["freeCommand"]}\n' if m['freeCommand'] else ''), 0o600), @@ -334,7 +339,7 @@ def install(a, stop_after=None): if r.returncode: raise Refused(f'the desk agent refuses its settings (not started): {r.stderr.strip()[:300]}') run('systemctl', 'daemon-reload') - run('systemctl', 'enable', '--now', TIMER) + run('systemctl', 'enable', '--now', TIMER, SEED_TIMER) if not os.path.exists(p(OPT + '/template/runner-template/.fork-fleet-version')): run(p(OPT + '/fork-fleet-launch.sh'), 'prune', check=False) # makes the launch lock refresh opens run('systemctl', 'start', '--no-block', REFRESH) # the first runner template, in the background @@ -369,7 +374,7 @@ def check(): row(False, 'walls', f'exit {walls.returncode}: {walls.stderr.strip()[:200]}') r = py(p(OPT + '/fork-fleet-desk-agent.py'), 'check') row(r.returncode == 0, 'agent-settings', (r.stdout or r.stderr).strip()[:200]) - for unit in (AGENT, TIMER): + for unit in (AGENT, TIMER, SEED_TIMER): on = run('systemctl', 'is-active', '--quiet', unit, check=False).returncode == 0 row(on and run('systemctl', 'is-enabled', '--quiet', unit, check=False).returncode == 0, 'unit', f'{unit} enabled and active') @@ -429,7 +434,7 @@ def uninstall(): if state is None: say(f'no {STATE}: nothing here was installed by this kit (nothing changed)') return 0 - for unit in (AGENT, TIMER, REFRESH): + for unit in (AGENT, TIMER, REFRESH, SEED_TIMER, SEED): run('systemctl', 'disable', '--now', unit, check=False) launcher = p(OPT + '/fork-fleet-launch.sh') if os.path.exists(launcher): diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index 2de38ec65819b..614834371c4c1 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -64,6 +64,7 @@ def put(n, items): ' print("bad settings", file=sys.stderr); sys.exit(2)\n' 'print("settings ok")\n', '.github/scripts/fork-fleet-desk-wire.py': '# wire\n', + '.github/scripts/fork-fleet-desk-seed.py': '# seed\n', '.github/scripts/fork-fleet-hook.sh': '#!/bin/bash\nexit 0\n', '.github/scripts/fork-fleet-job-paths.txt': '/usr\n', '.github/fork-fleet.json': json.dumps({'pools': [{'name': 'desktop'}, {'name': 'vm'}]}), @@ -74,6 +75,8 @@ def put(n, items): '.github/scripts/fork-fleet-desk-agent.service': '# agent unit\n[Service]\n', '.github/scripts/fork-fleet-refresh.service': '# refresh unit\n[Service]\n', '.github/scripts/fork-fleet-refresh.timer': '# refresh timer\n[Timer]\n', + '.github/scripts/fork-fleet-seed.service': '# seed unit\n[Service]\n', + '.github/scripts/fork-fleet-seed.timer': '# seed timer\n[Timer]\n', } @@ -194,7 +197,8 @@ def test_fresh_install_files_settings_users_key_units(self): self.assertFalse([n for n in os.listdir(self.path('/opt/fork-fleet')) if n.startswith('.kit-stage')]) launch = self.read('/etc/fork-fleet/launch.conf') self.assertTrue(launch.startswith(self.m.HEADER)) - self.assertIn('RUNNER_USERS=fork-runner-1,fork-runner-2\n', launch) + self.assertIn('RUNNER_USERS=fork-runner-1,fork-runner-2\nALLOW_PATHS=/var/lib/fork-fleet/seeds\n', launch) + self.assertTrue(os.path.isfile(self.path('/opt/fork-fleet/fork-fleet-desk-seed.py'))) self.assertNotIn('RESERVE_MB', launch) self.assertIn('POOL=desktop\nHOST=fleet-host.example\nPORT=22\nUSER=fleet-desk\n', self.read('/etc/fork-fleet/desk.conf')) @@ -215,7 +219,8 @@ def test_fresh_install_files_settings_users_key_units(self): ['fork-runner-1', 'fork-runner-2']) order = [c for c in calls if c.startswith(('systemctl', 'launch'))] self.assertEqual(order, ['systemctl stop fork-fleet-desk-agent.service', 'systemctl daemon-reload', - 'systemctl enable --now t3code-fleet-refresh.timer', 'launch prune', + 'systemctl enable --now t3code-fleet-refresh.timer t3code-fleet-seed.timer', + 'launch prune', 'systemctl start --no-block t3code-fleet-refresh.service', 'systemctl enable fork-fleet-desk-agent.service', 'systemctl restart fork-fleet-desk-agent.service']) @@ -355,7 +360,8 @@ def test_reserve_and_port_reach_the_settings(self): self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'reserveMb': 4096, 'port': 2222, 'slots': 1, 'netBase': '10.77'}}) self.assertEqual(self.install()[0], 0) - self.assertIn('RUNNER_USERS=fork-runner-1\nRESERVE_MB=4096\nNET_BASE=10.77\n', + self.assertIn('RUNNER_USERS=fork-runner-1\nALLOW_PATHS=/var/lib/fork-fleet/seeds\nRESERVE_MB=4096\n' + 'NET_BASE=10.77\n', self.read('/etc/fork-fleet/launch.conf')) self.assertTrue(self.read('/etc/fork-fleet/desk-known_hosts').endswith( f'[fleet-host.example]:2222 {HOST_KEY}\n')) @@ -415,7 +421,8 @@ def test_check_names_each_failure(self): self.assertEqual(code, 1) for line in ('FAIL kit /opt/fork-fleet/hook.sh: differs', 'FAIL vsock closed: open', 'FAIL agent-settings bad settings', 'FAIL unit fork-fleet-desk-agent.service', - 'PASS unit t3code-fleet-refresh.timer', 'FAIL template', 'FAIL runner-user fork-runner-1', + 'PASS unit t3code-fleet-refresh.timer', 'PASS unit t3code-fleet-seed.timer', + 'FAIL template', 'FAIL runner-user fork-runner-1', 'FAIL ip-forward'): self.assertIn(line, out) self.assertTrue(out.endswith('8 FAIL\n'), out) @@ -435,11 +442,13 @@ def test_uninstall_leaves_nothing_of_its_own(self): self.assertEqual(self.install()[0], 0) self.write(os.path.join(self.fakes, 'active'), 'desktop-1\n') self.write(self.path('/var/lib/fork-fleet/runners/x'), '') + self.write(self.path('/var/lib/fork-fleet/seeds/.v/m2.1/data/x'), '') self.clear_calls() code, out, err = self.run_main('uninstall') self.assertEqual(code, 0, err) calls = self.calls() - for c in ('systemctl disable --now fork-fleet-desk-agent.service', 'launch active', 'launch stop desktop-1', + for c in ('systemctl disable --now fork-fleet-desk-agent.service', + 'systemctl disable --now t3code-fleet-seed.timer', 'launch active', 'launch stop desktop-1', 'launch prune', 'ip netns delete fork-fleet-fork-runner-1', 'nft delete table inet fork_fleet', 'userdel fork-runner-1', 'userdel fork-runner-2'): self.assertIn(c, calls) From 5f64eb96f409f605e2e7974edd398f5e95c2c3bf Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:45:10 +0000 Subject: [PATCH 06/33] fork kit desk: fix round 1 (drop-ins, uninstall, re-run, bootstrap) (D8) - install refuses a systemd drop-in it did not write for its units; check fails on one - re-run restarts the agent only when a file, setting or key changed - check compares the settings files with the machine file - uninstall stops every runner, keeps the nft table, users and record while one still runs, removes every fork-fleet-* namespace and ffh- link, removes only what the kit made (deny files it created, its users' deny lines), names what it leaves, and removes its record last - the record is read only when it has this kit's shape - --release-fleet-config goes back to the release's fleet config - bootstrap parses --fleet-config like the other flags Co-Authored-By: Claude Opus 5.5 --- fork-kit/common/machine.example.json | 2 +- fork-kit/desk/README.md | 32 +++-- fork-kit/desk/bootstrap.sh | 27 ++-- fork-kit/desk/sbin/desk-install | 184 +++++++++++++++++++++++---- fork-kit/desk/tests/install.test.py | 151 ++++++++++++++++++++-- 5 files changed, 340 insertions(+), 56 deletions(-) diff --git a/fork-kit/common/machine.example.json b/fork-kit/common/machine.example.json index 6251cd5cef9f2..aa2cb5e9e7a1f 100644 --- a/fork-kit/common/machine.example.json +++ b/fork-kit/common/machine.example.json @@ -27,7 +27,7 @@ "freeFloorMb": 2048 }, "desk": { - "pool": "desktop", + "pool": "pc", "host": "fleet-host.example", "hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "slots": 2, diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index b63fc5bc97788..5725bc916eb0a 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -14,7 +14,7 @@ The facts of this machine go in the `desk` section of `/etc/fork-kit/machine.jso { "version": 1, "desk": { - "pool": "desktop", + "pool": "pc", "host": "fleet-host.example", "hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...", "slots": 2, @@ -23,7 +23,8 @@ The facts of this machine go in the `desk` section of `/etc/fork-kit/machine.jso } ``` -`pool` must be a pool in the fleet config; `host` and `hostKey` are the always-on host and its ssh host key +`pool` must be a pool in the fleet config that is installed (the release's own has `pc`; a project's +`--fleet-config` has its own pools); `host` and `hostKey` are the always-on host and its ssh host key (on that host: `cat /etc/ssh/ssh_host_ed25519_key.pub`, without the comment). Optional: `port` (22), `user` (`fleet-desk`), `slots` (1, at most 8; one runner user each), `reserveMb` (memory the launcher keeps free for the owner), `netBase` (`10.N`, the runners' network, default `10.204`: pick another when this machine's network already uses it). @@ -34,32 +35,41 @@ As root, from the fork's protected release branch: ```sh sh bootstrap.sh --repo OWNER/REPO --sha RELEASE_SHA [--fleet-config /path/to/project/fork-fleet.json] +sh bootstrap.sh --repo OWNER/REPO --sha RELEASE_SHA --release-fleet-config # back to the release's own /opt/fork-fleet/desk-install check # every line PASS, then ALL PASS /opt/fork-fleet/desk-install uninstall ``` `--fleet-config` is the project's fleet config (its pools); without it the release's own is used. The project's -is remembered, so a later install of a new release keeps it. Install prints the desk key's public half: add it on +is remembered, so a later install of a new release keeps it; `--release-fleet-config` goes back to the +release's own. Install prints the desk key's public half: add it on the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner template is made in the background by the refresh timer; `check` says FAIL for `template` until it is there. -`check` only reads: the installed files against their manifest, every wall of this OS (`fork-fleet-walls.py -describe`), the agent's settings, the units, the template, the runner users and IPv4 forwarding. +`check` only reads: the installed files against their manifest, systemd drop-ins that change the kit's units, +every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings +files against the machine file, the runner users and IPv4 forwarding. ## Runs again, stops half-way, old or empty state -- **Again:** files, settings and units that did not change are not rewritten, and the agent is not stopped. +- **Again:** files, settings and units that did not change are not rewritten, and the agent is not stopped or + restarted (a restart would end a job it is starting). After a change it is restarted. - **Stopped half-way:** the next install ends it. A new release is staged in `/opt/fork-fleet/.kit-stage-*` and checked first (gate loads with the fleet config, the pool is in it, `bash -n`, the walls know this OS); a release - that fails is never installed. Files are renamed in one by one while the agent is stopped; `SOURCE` is written + that fails is never installed. Files are renamed in one by one while the agent and the refresh timer are stopped; + `SOURCE` is written last, so `check` reports a half-done swap. Users are recorded in `/etc/fork-fleet/desk-kit.json` before they are made, so uninstall removes them even after a stop. -- **Old install:** a launcher, `launch.conf`, `desk.conf`, desk key or unit that this kit did not write is refused - before anything changes. Remove the older (by hand) install first, the runbook's steps in reverse. +- **Old install:** a launcher, `launch.conf`, `desk.conf`, desk key, unit, or a systemd drop-in for one of the kit's + units (`UNIT.d/*.conf`, also the dash-prefix forms) that this kit did not write is refused before anything changes. Remove the older (by hand) install first, the runbook's steps in reverse. - **Empty:** `check` before an install says FAIL and writes nothing; `uninstall` with no record does nothing. -Uninstall stops every runner, removes the namespaces and the `fork_fleet` nft table, the kit's units, files and -settings, and only the users it made itself. +Uninstall stops every runner (through the launcher, then `systemctl stop` for any left). While one still runs it +stops there and changes nothing else: the `fork_fleet` nft table, users and record stay; run it again. Then it +removes every `fork-fleet-*` namespace and `ffh-*` link, the nft table, the kit's units, the files of its manifest, +the template and runner copies, its settings and key, the users it made, and `cron.deny`/`at.deny` when the kit +made them (their lines for its users otherwise). The record goes last. Anything else in its folders (a file the kit +did not make, a drop-in) is kept and named. IPv4 forwarding stays on until the next reboot. ## Why one flat folder diff --git a/fork-kit/desk/bootstrap.sh b/fork-kit/desk/bootstrap.sh index 7bf30a1b8d38b..7465eba5c60ae 100755 --- a/fork-kit/desk/bootstrap.sh +++ b/fork-kit/desk/bootstrap.sh @@ -2,10 +2,11 @@ # desk bootstrap: installs the desk kit (a desktop's side of a fork fleet desk pool) of one release of this fork. # Run as root on the desktop (on WSL2: inside the distro), after the owner wrote the machine file's `desk` section: # -# sh desk-bootstrap.sh --repo FORK_OWNER/FORK --sha RELEASE_SHA [--fleet-config FILE] +# sh desk-bootstrap.sh --repo FORK_OWNER/FORK --sha RELEASE_SHA [--fleet-config FILE | --release-fleet-config] # # --fleet-config: the fleet config of the project whose jobs this desktop runs (that repository's file, at the commit -# its always-on host uses); without it, the release's own .github/fork-fleet.json (the fork's CI). +# its always-on host uses); without it, the release's own .github/fork-fleet.json (the fork's CI), or the project's +# from the last install. --release-fleet-config: back to the release's own after a project's was used. # It refuses a sha that is not a commit on the fork's protected release branch (ci-release), downloads exactly that # commit, prints the sha and the hash of what it downloaded, and runs its installer (safe to run again). # Needs: curl, tar, python3. Linux with systemd only (WSL2 with systemd on). @@ -17,19 +18,26 @@ unset TMPDIR CURL_HOME SSL_CERT_FILE SSL_CERT_DIR || true die() { echo "desk-bootstrap: $*" >&2; exit 1; } -repo='' sha='' branch='ci-release' api='https://api.github.com' codeload='https://codeload.github.com' +repo='' sha='' branch='ci-release' fleet='' release_fleet='' api='https://api.github.com' +codeload='https://codeload.github.com' while [ $# -gt 0 ]; do case "$1" in - --repo) repo="${2-}"; shift 2 ;; - --sha) sha="${2-}"; shift 2 ;; - --release-branch) branch="${2-}"; shift 2 ;; - --fleet-config) break ;; + --repo | --sha | --release-branch | --fleet-config) [ $# -ge 2 ] || die "$1 needs a value" ;; + esac + case "$1" in + --repo) repo="$2"; shift 2 ;; + --sha) sha="$2"; shift 2 ;; + --release-branch) branch="$2"; shift 2 ;; + --fleet-config) fleet="$2"; shift 2 ;; + --release-fleet-config) release_fleet=1; shift ;; *) die "unknown option $1 (see the top of this file)" ;; esac done printf '%s\n' "$repo" | grep -Eqx '[A-Za-z0-9][A-Za-z0-9-]{0,38}/[A-Za-z0-9._-]{1,100}' || die "--repo must be OWNER/NAME" printf '%s\n' "$sha" | grep -Eqx '[0-9a-f]{40}' || die "--sha must be the full 40-character release commit" printf '%s\n' "$branch" | grep -Eqx '[A-Za-z0-9._/-]{1,100}' || die "bad --release-branch" +[ -z "$fleet" ] || [ -f "$fleet" ] || die "--fleet-config $fleet: no such file" +[ -z "$fleet" ] || [ -z "$release_fleet" ] || die "--fleet-config and --release-fleet-config: one or the other" [ "$(id -u)" = 0 ] || die "run this as root" work="$(mktemp -d /tmp/desk-bootstrap.XXXXXXXX)" @@ -57,4 +65,7 @@ top="$(find "$work/src" -mindepth 1 -maxdepth 1 -type d)" [ "$(printf '%s\n' "$top" | wc -l)" = 1 ] && [ -n "$top" ] || die "the download does not have one top folder" [ -f "$top/fork-kit/desk/sbin/desk-install" ] || die "release $sha has no fork-kit/desk (is it a release with the kit?)" echo "desk-bootstrap: release $sha of $repo, download sha256 $(sha256sum "$work/kit.tar.gz" | cut -d' ' -f1)" -python3 -I -B "$top/fork-kit/desk/sbin/desk-install" install --source "$top" --sha "$sha" "$@" +set -- install --source "$top" --sha "$sha" +[ -z "$fleet" ] || set -- "$@" --fleet-config "$fleet" +[ -z "$release_fleet" ] || set -- "$@" --release-fleet-config +python3 -I -B "$top/fork-kit/desk/sbin/desk-install" "$@" diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index 0f71da2ff96df..a70e3e5a6fe4c 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -1,15 +1,16 @@ #!/usr/bin/python3 -IB """desk-install: a desktop's side of a fork fleet desk pool, as root (fork-kit/desk/README.md). - desk-install install --source RELEASE_FOLDER --sha RELEASE_SHA [--fleet-config FILE] + desk-install install --source RELEASE_FOLDER --sha RELEASE_SHA [--fleet-config FILE | --release-fleet-config] desk-install check every part and wall, PASS or FAIL; reads only (safe while a job runs) desk-install uninstall install reads the `desk` section of the machine file, makes the runner users, stages the fleet files of one release (launcher, walls, desk agent, gate hook and the fleet config), checks the staged copy, stops the agent, moves each file into /opt/fork-fleet, writes MANIFEST.sha256 and then SOURCE (last), writes the settings in /etc/fork-fleet and -the units, makes the desk key once, and starts the agent and the refresh timer. Safe to run again and to stop at -any step: a stopped run leaves SOURCE naming the old release (check says what differs) and the next run ends it. +the units, makes the desk key once, and starts the agent and the refresh timer (restarting the agent only when +something changed). Safe to run again and to stop at any step: a stopped run leaves SOURCE naming the old release +(check says what differs) and the next run ends it. uninstall removes only what the kit made, its record last. """ import argparse, contextlib, hashlib, importlib.util, json, os, re, shutil, subprocess, sys, tempfile @@ -20,6 +21,10 @@ OPT, ETC, WORK = '/opt/fork-fleet', '/etc/fork-fleet', '/var/lib/fork-fleet' STATE = ETC + '/desk-kit.json' KEY = ETC + '/desk-key' UNIT_DIR = '/etc/systemd/system' +DROPIN_ROOTS = ('/etc/systemd/system', '/run/systemd/system', '/usr/local/lib/systemd/system', '/usr/lib/systemd/system', + '/lib/systemd/system') +DENY = ('/etc/cron.deny', '/etc/at.deny') +SETTINGS = ('launch.conf', 'desk.conf', 'desk-known_hosts', 'resolv.conf', 'desk-key', 'desk-key.pub') SYSCTL = '/etc/sysctl.d/60-fork-fleet.conf' HEADER = '# desk kit: written by desk-install. Do not edit: change the machine file and install again.\n' FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh', @@ -45,6 +50,7 @@ SHA = re.compile(r'[0-9a-f]{40}') SHAPES = {'pool': r'[a-z0-9][a-z0-9_-]{0,39}', 'host': r'[A-Za-z0-9][A-Za-z0-9.-]{0,252}', 'user': r'[a-z_][a-z0-9_-]{0,31}', 'netBase': r'10\.(0|[1-9][0-9]?|1[0-9]{2}|2[0-4][0-9]|25[0-5])', 'hostKey': r'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI[A-Za-z0-9+/]{43}'} +RUNNER_UNIT = re.compile(r'fork-fleet-[a-z0-9][a-z0-9-]{0,62}-[0-9]{1,20}\.service') # the launcher's runner units class Refused(Exception): @@ -107,11 +113,23 @@ def machine_section(): def read_state(): try: with open(p(STATE), encoding='utf-8') as f: - return json.load(f) + state = json.load(f) except FileNotFoundError: return None except (OSError, ValueError) as e: raise Refused(f'{STATE} cannot be read ({e}): fix or remove it and run again') from None + state = state if isinstance(state, dict) else {} + state.setdefault('createdDenyFiles', []) + + def names(key, shape): + return isinstance(state.get(key), list) and all(isinstance(x, str) and re.fullmatch(shape, x) + for x in state[key]) + if not (state.get('version') == 1 and names('createdUsers', SHAPES['user']) + and names('createdDenyFiles', '|'.join(map(re.escape, DENY))) + and state.get('fleetConfig') in ('release', 'project') + and (state.get('sha') is None or SHA.fullmatch(str(state['sha'])))): + raise Refused(f'{STATE} is not an install record of this kit (version 1): fix or remove it and run again') + return state def write_file(path, text, mode=0o644): @@ -149,6 +167,23 @@ def ours(path): return False +def drop_ins(unit): + """Drop-ins that change one of the kit's units: `UNIT.d/`, and the dash-prefix forms (`fork-fleet-.service.d/`) + in every unit folder, and whatever systemd itself lists. The type-wide `service.d/` (distribution defaults) is left + out: it is not aimed at the kit's units.""" + stem, kind = unit.rsplit('.', 1) + names = [unit] + [stem[:i + 1] + '.' + kind for i, c in enumerate(stem) if c == '-'] + found = set() + for root in DROPIN_ROOTS: + for n in names: + with contextlib.suppress(OSError): + found.update(f'{root}/{n}.d/{f}' for f in os.listdir(p(f'{root}/{n}.d')) if f.endswith('.conf')) + r = run('systemctl', 'show', '-p', 'DropInPaths', '--value', unit, check=False) + if r.returncode == 0: + found.update(x for x in r.stdout.split() if os.path.basename(os.path.dirname(x)) != kind + '.d') + return sorted(found) + + def stage(source, sha, fleet_config, pool): """The release's files in /opt/fork-fleet/.kit-stage-*, checked; returns (folder, {path: sha256}).""" for name in os.listdir(p(OPT)): # a run that stopped while staging @@ -178,10 +213,15 @@ def stage(source, sha, fleet_config, pool): r = py(os.path.join(tmp, 'fork-fleet.py'), '--config', os.path.join(tmp, 'fork-fleet.json'), 'gate') if 'no Runner.Worker above this hook' not in r.stdout + r.stderr: problems.append(f'the gate does not load with this fleet config: {(r.stderr or r.stdout).strip()[:300]}') - with contextlib.suppress(OSError, ValueError, TypeError, AttributeError): + try: with open(os.path.join(tmp, 'fork-fleet.json'), encoding='utf-8') as f: - if pool not in [x.get('name') for x in json.load(f)['pools']]: - problems.append(f'the fleet config has no pool {pool!r} (the machine file\'s desk.pool)') + pools = json.load(f).get('pools') + if not isinstance(pools, list) or not all(isinstance(x, dict) for x in pools): + problems.append('the fleet config\'s "pools" is not a list of pools') + elif pool not in [x.get('name') for x in pools]: + problems.append(f'the fleet config has no pool {pool!r} (the machine file\'s desk.pool)') + except (OSError, ValueError, AttributeError) as e: + problems.append(f'the fleet config cannot be read: {e}') for rel in ('fork-fleet-launch.sh', 'hook.sh'): if run('bash', '-n', os.path.join(tmp, rel), check=False).returncode: problems.append(f'{rel}: bash -n fails') @@ -202,14 +242,15 @@ def read_manifest(): def swap(tmp, manifest, sha): """Each changed file renamed into /opt/fork-fleet (the agent stopped meanwhile, so no verb runs from half-old, - half-new files); files of the old release that are gone, removed. Unchanged files are left as they are.""" + half-new files, and the refresh timer and service stopped too: refresh runs the launcher); files of the old + release that are gone, removed. Unchanged files are left as they are. True when anything changed.""" def same(rel): with contextlib.suppress(OSError): return sha256(p(f'{OPT}/{rel}')) == manifest[rel] changed = [rel for rel in sorted(manifest) if not same(rel)] gone = sorted(set(read_manifest()) - set(manifest)) if changed or gone: - run('systemctl', 'stop', AGENT, check=False) + run('systemctl', 'stop', TIMER, REFRESH, AGENT, check=False) for rel in changed: os.makedirs(os.path.dirname(p(f'{OPT}/{rel}')), mode=0o755, exist_ok=True) os.replace(os.path.join(tmp, rel), p(f'{OPT}/{rel}')) @@ -219,6 +260,7 @@ def swap(tmp, manifest, sha): write_file(OPT + '/MANIFEST.sha256', ''.join(f'{d} {r}\n' for r, d in sorted(manifest.items()))) write_file(OPT + '/SOURCE', sha + '\n') shutil.rmtree(tmp, ignore_errors=True) + return bool(changed or gone) def lines_of(path): @@ -239,9 +281,14 @@ def make_users(users, state): say(f'made runner user {u}') for u in users: run('loginctl', 'disable-linger', u, check=False) - for f in ('/etc/cron.deny', '/etc/at.deny'): + for f in DENY: if os.path.exists(p(f.replace('.deny', '.allow'))): continue + # A deny file that was not there opens cron or at to every other user (with neither file, only root may + # use at): recorded first, so uninstall removes the file again. + if not os.path.lexists(p(f)) and f not in state['createdDenyFiles']: + state['createdDenyFiles'].append(f) + save_state(state) lines = lines_of(f) if u not in lines: write_file(f, '\n'.join(lines + [u]) + '\n') @@ -262,7 +309,7 @@ def settings(m, users): def make_key(pool): """The desk key, once: made under a temp name, then renamed (the public half first).""" if os.path.exists(p(KEY)) and os.path.exists(p(KEY + '.pub')): - return + return False new = p(ETC + '/.desk-key.new') for f in (new, new + '.pub'): with contextlib.suppress(FileNotFoundError): @@ -272,6 +319,7 @@ def make_key(pool): run('ssh-keygen', '-q', '-t', 'ed25519', '-N', '', '-C', f'desk-{pool}', '-f', new) os.replace(new + '.pub', p(KEY + '.pub')) os.replace(new, p(KEY)) + return True def install(a, stop_after=None): @@ -281,6 +329,8 @@ def install(a, stop_after=None): if os.geteuid() != ROOT_UID: raise Refused('run this as root') + if a.fleet_config and a.release_fleet_config: + raise Refused('--fleet-config and --release-fleet-config: one or the other') if not SHA.fullmatch(a.sha or ''): raise Refused('--sha must be the full 40-character release commit') missing = [s for s in [*FILES.values(), *UNITS.values(), WALLS] if not os.path.exists(os.path.join(a.source, s))] @@ -297,12 +347,14 @@ def install(a, stop_after=None): there = [f for f in FOREIGN if os.path.lexists(p(f))] if there: raise Refused(f'{there[0]} is from an install this kit did not make: uninstall that first (README)') - state = {'version': 1, 'createdUsers': [], 'fleetConfig': 'release'} + state = {'version': 1, 'createdUsers': [], 'createdDenyFiles': [], 'fleetConfig': 'release'} for u in UNITS: if os.path.lexists(p(f'{UNIT_DIR}/{u}')) and not ours(f'{UNIT_DIR}/{u}'): raise Refused(f'{UNIT_DIR}/{u} was not written by this kit: remove it and run again') + if drop_ins(u): + raise Refused(f'{drop_ins(u)[0]} changes {u} and was not written by this kit: remove it and run again') config = os.path.realpath(a.fleet_config) if a.fleet_config else None - if config is None and state['fleetConfig'] == 'project': + if config is None and state['fleetConfig'] == 'project' and not a.release_fleet_config: config = p(OPT + '/fork-fleet.json') # the project's config from the last install, checked again below users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] done('check') @@ -314,18 +366,18 @@ def install(a, stop_after=None): done('users') tmp, manifest = stage(a.source, a.sha, config, m['pool']) done('stage') - swap(tmp, manifest, a.sha) + changed = swap(tmp, manifest, a.sha) state.update(sha=a.sha, fleetConfig='project' if config else 'release') save_state(state) done('files') for path, (text, mode) in settings(m, users).items(): - write_file(path, text, mode) + changed |= write_file(path, text, mode) run('sysctl', '-q', '-p', p(SYSCTL)) - make_key(m['pool']) + changed |= make_key(m['pool']) done('settings') for u in UNITS: with open(p(f'{OPT}/units/{u}'), encoding='utf-8') as f: - write_file(f'{UNIT_DIR}/{u}', HEADER + f.read()) + changed |= write_file(f'{UNIT_DIR}/{u}', HEADER + f.read()) r = py(p(OPT + '/fork-fleet-desk-agent.py'), 'check') if r.returncode: raise Refused(f'the desk agent refuses its settings (not started): {r.stderr.strip()[:300]}') @@ -335,7 +387,9 @@ def install(a, stop_after=None): run(p(OPT + '/fork-fleet-launch.sh'), 'prune', check=False) # makes the launch lock refresh opens run('systemctl', 'start', '--no-block', REFRESH) # the first runner template, in the background run('systemctl', 'enable', AGENT) - run('systemctl', 'restart', AGENT) + # restart only after a change: a restart ends the agent's whole control group, a `start` it is running with it + # (the always-on host would see that job fail). Unchanged: start is a no-op for a running agent. + run('systemctl', 'restart' if changed else 'start', AGENT) done('units') with open(p(KEY + '.pub'), encoding='utf-8') as f: pub = f.read().strip() @@ -345,6 +399,14 @@ def install(a, stop_after=None): return 0 +def same_text(path, text, mode): + try: + with open(p(path), encoding='utf-8') as f: + return f.read() == text and os.stat(p(path)).st_mode & 0o777 == mode + except (OSError, UnicodeDecodeError): + return False + + def check(): rows = [] @@ -375,6 +437,9 @@ def check(): m = machine_section() users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] row(True, 'machine-file', f'desk section: pool {m["pool"]}, {m["slots"]} slot(s), always-on host {m["host"]}') + stale = [path for path, (text, mode) in settings(m, users).items() if not same_text(path, text, mode)] + row(not stale, 'settings', f'{stale[0]} differs from the machine file: install again' if stale else + 'the written settings match the machine file') except Refused as e: users = [] row(False, 'machine-file', str(e)[:300]) @@ -393,7 +458,10 @@ def check(): def installed_problems(): """What differs from the install record: /opt/fork-fleet against its manifest, SOURCE, the units.""" - problems, state = [], read_state() or {} + try: + problems, state = [], read_state() or {} + except Refused as e: + return [str(e)] manifest = read_manifest() if not manifest or not state.get('sha'): return [f'{OPT}/MANIFEST.sha256 or {STATE} is missing: not installed (or a first install stopped)'] @@ -412,10 +480,42 @@ def installed_problems(): if source != state['sha']: problems.append(f'{OPT}/SOURCE names {source}, the record {state["sha"]}: install again') problems += [f'{UNIT_DIR}/{u}: missing or not the kit\'s' for u in UNITS if not ours(f'{UNIT_DIR}/{u}')] + problems += [f'{d}: a drop-in the kit did not write changes {u}' for u in UNITS for d in drop_ins(u)] return problems +def running(): + """The launcher's runner units still running (or starting or stopping).""" + r = run('systemctl', 'list-units', '--plain', '--no-legend', '--type=service', + '--state=active,activating,deactivating,reloading', 'fork-fleet-*', check=False) + return [x.split()[0] for x in r.stdout.splitlines() if x.split() and RUNNER_UNIT.fullmatch(x.split()[0])] + + +def remove(path): + """One path the kit made (a folder with all in it); absent is fine.""" + real = p(path) + if os.path.isdir(real) and not os.path.islink(real): + shutil.rmtree(real) + else: + with contextlib.suppress(FileNotFoundError): + os.unlink(real) + + +def rmdir_left(path): + """The folder if empty; otherwise what is left in it (not the kit's, so kept).""" + try: + os.rmdir(p(path)) + except FileNotFoundError: + pass + except OSError: + return [f'{path}/{n}' for n in sorted(os.listdir(p(path)))] + return [] + + def uninstall(): + """Removes only what the kit made: its units, the files of its manifest, its settings, the runner copies and + template its programs made, the users and deny files it made. Every runner is stopped first; while one still runs, + the network table, users and record stay (run it again). The record goes last, so a stopped uninstall runs again.""" if os.geteuid() != ROOT_UID: raise Refused('run this as root') state = read_state() @@ -428,9 +528,21 @@ def uninstall(): if os.path.exists(launcher): for name in run(launcher, 'active', check=False).stdout.split(): run(launcher, 'stop', name, check=False) + for unit in running(): # what the launcher could not stop (or no launcher is left to ask) + run('systemctl', 'stop', unit, check=False) + left = running() + if left: + raise Refused(f'runner {left[0]} is still running: the network table, users and record stay (its namespace ' + 'keeps routing through them). Stop it (systemctl stop) and run uninstall again') + if os.path.exists(launcher): run(launcher, 'prune', check=False) - for u in state['createdUsers']: - run('ip', 'netns', 'delete', f'fork-fleet-{u}', check=False) + for ns in run('ip', 'netns', 'list', check=False).stdout.splitlines(): + if ns.split() and ns.split()[0].startswith('fork-fleet-'): # every runner's, made by the kit or by hand + run('ip', 'netns', 'delete', ns.split()[0], check=False) + for line in run('ip', '-o', 'link', 'show', check=False).stdout.splitlines(): + link = re.match(r'[0-9]+: (ffh-[0-9]+)[@:]', line) + if link: + run('ip', 'link', 'delete', link.group(1), check=False) run('nft', 'delete', 'table', 'inet', 'fork_fleet', check=False) for u in UNITS: if ours(f'{UNIT_DIR}/{u}'): @@ -438,15 +550,31 @@ def uninstall(): run('systemctl', 'daemon-reload', check=False) if ours(SYSCTL): os.unlink(p(SYSCTL)) - for d in (OPT, WORK): - shutil.rmtree(p(d), ignore_errors=True) + for rel in [*read_manifest(), 'MANIFEST.sha256', 'SOURCE', 'template', + *(n for n in os.listdir(p(OPT)) if n.startswith('.kit-stage-'))] if os.path.isdir(p(OPT)) else []: + remove(f'{OPT}/{rel}') + remove(WORK + '/runners') for u in state['createdUsers']: if user_exists(u): run('userdel', u) - for f in ('/etc/cron.deny', '/etc/at.deny'): - if u in lines_of(f): - write_file(f, ''.join(x + '\n' for x in lines_of(f) if x != u)) - shutil.rmtree(p(ETC), ignore_errors=True) # the record (desk-kit.json) goes last, with the folder + for f in DENY: + lines = lines_of(f) + if any(x in state['createdUsers'] for x in lines): + write_file(f, ''.join(x + '\n' for x in lines if x not in state['createdUsers'])) + if f in state['createdDenyFiles'] and not [x for x in lines_of(f) if x.strip()]: + remove(f) + for name in SETTINGS: + remove(f'{ETC}/{name}') + for name in os.listdir(p(ETC)) if os.path.isdir(p(ETC)) else []: + if name.startswith(('.desk-key.new', '.desk-kit.json.', *(f'.{n}.' for n in SETTINGS))): # a stopped write or keygen + remove(f'{ETC}/{name}') + remove(STATE) # the record last + left = [] + for d in ([f'{OPT}/{x}' for x in ('units', 'fork-fleet-walls')] + [OPT, WORK, ETC]): + left += rmdir_left(d) + left += [f'{d} (a drop-in, not the kit\'s)' for u in UNITS for d in drop_ins(u)] + for x in left: + say(f'left {x}: the kit did not make it') say('uninstalled; on the always-on host remove its key: fork-fleet-desk-key.py remove POOL') return 0 @@ -458,6 +586,8 @@ def main(argv, stop_after=None): i.add_argument('--source', required=True) i.add_argument('--sha', required=True) i.add_argument('--fleet-config') + i.add_argument('--release-fleet-config', action='store_true', + help='back to the release\'s own fleet config after a project\'s was used') for name in ('check', 'uninstall'): sub.add_parser(name) a = ap.parse_args(argv) diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index 65673b609122f..3903cd1297170 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -46,11 +46,26 @@ def put(n, items): open(key + '.pub', 'w').write('ssh-ed25519 AAAAfake desk-x\n') if name == 'systemctl' and args[0] in ('is-active', 'is-enabled'): sys.exit(1 if args[-1] in lines('down') else 0) +if name == 'systemctl' and args[0] == 'list-units': # runner units: the file `units`; `stuck` ones never stop + print(''.join(f'{u} loaded active running runner\n' for u in lines('units')), end='') +if name == 'systemctl' and args[0] == 'stop': + put('units', [u for u in lines('units') if u not in args[1:] or u in lines('stuck')]) +if name == 'ip' and args[:2] == ['netns', 'list']: + print(''.join(f'{n} (id: 0)\n' for n in lines('netns')), end='') +if name == 'ip' and args[:2] == ['netns', 'delete']: + put('netns', [n for n in lines('netns') if n != args[2]]) + put('links', [n for n in lines('links') if n != 'ffh-' + args[2].rsplit('-', 1)[-1]]) +if name == 'ip' and args[:2] == ['-o', 'link']: + print(''.join(f'{i}: {n}@if2: mtu 1500\n' for i, n in enumerate(lines('links'), 5)), end='') +if name == 'ip' and args[:2] == ['link', 'delete']: + put('links', [n for n in lines('links') if n != args[2]]) ''' STUBS = { '.github/scripts/fork-fleet-launch.sh': '#!/bin/bash\necho "launch $*" >> {log}\n' - '[ "$1" = active ] && cat {here}/active 2>/dev/null\nexit 0\n', + '[ "$1" = active ] && cat {here}/active 2>/dev/null\n' + 'if [ "$1" = stop ]; then grep -vx "$2" {here}/active > {here}/a.new;' + ' mv {here}/a.new {here}/active; fi\nexit 0\n', '.github/scripts/fork-fleet-walls.py': '#!/usr/bin/python3 -I\nimport sys, os\n' 'v = sys.argv[1]\nif v == "os":\n' ' print("linux"); sys.exit(int(open("{here}/os-exit").read()))\n' @@ -211,8 +226,9 @@ def test_fresh_install_files_settings_users_key_units(self): self.assertEqual(self.read('/etc/cron.deny'), 'fork-runner-1\nfork-runner-2\n') self.assertEqual(json.loads(self.read('/etc/fork-fleet/desk-kit.json'))['createdUsers'], ['fork-runner-1', 'fork-runner-2']) - order = [c for c in calls if c.startswith(('systemctl', 'launch'))] - self.assertEqual(order, ['systemctl stop fork-fleet-desk-agent.service', 'systemctl daemon-reload', + order = [c for c in calls if c.startswith(('systemctl', 'launch')) and not c.startswith('systemctl show')] + self.assertEqual(order, ['systemctl stop t3code-fleet-refresh.timer t3code-fleet-refresh.service ' + 'fork-fleet-desk-agent.service', 'systemctl daemon-reload', 'systemctl enable --now t3code-fleet-refresh.timer', 'launch prune', 'systemctl start --no-block t3code-fleet-refresh.service', 'systemctl enable fork-fleet-desk-agent.service', @@ -232,9 +248,18 @@ def test_runs_again_and_changes_nothing(self): self.assertEqual(code, 0, err) self.assertEqual(self.tree(), before) calls = self.calls() - self.assertFalse([c for c in calls if c.startswith(('useradd', 'ssh-keygen', 'systemctl stop'))], calls) + self.assertFalse([c for c in calls if c.startswith(('useradd', 'ssh-keygen', 'systemctl stop', + 'systemctl restart'))], calls) + self.assertIn('systemctl start fork-fleet-desk-agent.service', calls) # a no-op for a running agent self.assertNotIn('systemctl start --no-block t3code-fleet-refresh.service', calls) # the template is there + def test_a_changed_setting_restarts_the_agent(self): + self.assertEqual(self.install()[0], 0) + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'reserveMb': 2048}}) + self.clear_calls() + self.assertEqual(self.install()[0], 0) + self.assertIn('systemctl restart fork-fleet-desk-agent.service', self.calls()) + def test_stopped_at_any_step_the_next_run_ends_it(self): self.assertEqual(self.install()[0], 0) want = self.tree() @@ -266,8 +291,8 @@ def test_upgrade_replaces_changed_files_and_removes_gone_ones(self): self.assertIn('# new', self.read('/opt/fork-fleet/hook.sh')) self.assertEqual(os.stat(self.path('/opt/fork-fleet/fork-fleet-launch.sh')).st_ino, launcher) # unchanged calls = self.calls() - self.assertLess(calls.index('systemctl stop fork-fleet-desk-agent.service'), - calls.index('systemctl restart fork-fleet-desk-agent.service')) + stop = 'systemctl stop t3code-fleet-refresh.timer t3code-fleet-refresh.service fork-fleet-desk-agent.service' + self.assertLess(calls.index(stop), calls.index('systemctl restart fork-fleet-desk-agent.service')) self.assertEqual(json.loads(self.read('/etc/fork-fleet/desk-kit.json'))['sha'], SHA2) def test_project_fleet_config_is_used_and_kept(self): @@ -279,6 +304,13 @@ def test_project_fleet_config_is_used_and_kept(self): code, _, err = self.install(sha=SHA2) # no --fleet-config: the project's stays self.assertEqual(code, 0, err) self.assertIn('"project": true', self.read('/opt/fork-fleet/fork-fleet.json')) + code, _, err = self.install('--release-fleet-config', sha=SHA2) # and back to the release's own + self.assertEqual(code, 0, err) + self.assertNotIn('"project": true', self.read('/opt/fork-fleet/fork-fleet.json')) + self.assertEqual(json.loads(self.read('/etc/fork-fleet/desk-kit.json'))['fleetConfig'], 'release') + code, _, err = self.install('--release-fleet-config', '--fleet-config', project) + self.assertEqual(code, 1) + self.assertIn('one or the other', err) def test_refusals_change_nothing(self): cases = { @@ -301,6 +333,13 @@ def test_refusals_change_nothing(self): 'a foreign unit': (lambda: self.write(self.path('/etc/systemd/system/fork-fleet-desk-agent.service'), 'x'), [], 'this kit did not make'), 'no such fleet config': (lambda: None, ['--fleet-config', '/nonexistent/x.json'], 'no such file'), + 'a drop-in for the agent': (lambda: self.write(self.path( + '/etc/systemd/system/fork-fleet-desk-agent.service.d/override.conf'), '[Service]\nExecStart=\n'), [], + 'override.conf changes fork-fleet-desk-agent.service'), + 'a dash-prefix drop-in': (lambda: self.write(self.path( + '/run/systemd/system/t3code-fleet-.timer.d/x.conf'), '[Timer]\n'), [], 'changes t3code-fleet'), + 'a record of another shape': (lambda: self.write(self.path('/etc/fork-fleet/desk-kit.json'), + '{"version": 1}'), [], 'not an install record'), } for name, (setup, args, msg) in cases.items(): with self.subTest(name): @@ -318,7 +357,8 @@ def test_refusals_change_nothing(self): self.assertEqual(code, 1) self.assertIn(msg, err) self.assertEqual(self.tree(), before) - self.assertFalse([c for c in self.calls() if not c.startswith('id ')], self.calls()) + self.assertFalse([c for c in self.calls() if not c.startswith(('id ', 'systemctl show'))], + self.calls()) def test_bad_release_is_not_installed(self): self.assertEqual(self.install()[0], 0) @@ -327,7 +367,9 @@ def test_bad_release_is_not_installed(self): ('no pool', {'.github/fork-fleet.json': json.dumps({'pools': [{'name': 'vm'}]})}, "no pool 'desktop'"), ('gate does not load', {'.github/scripts/fork-fleet.py': 'raise SystemExit("bad config")\n'}, 'the gate does not load'), - ('launcher syntax', {'.github/scripts/fork-fleet-launch.sh': '#!/bin/bash\nif then\n'}, 'bash -n')): + ('launcher syntax', {'.github/scripts/fork-fleet-launch.sh': '#!/bin/bash\nif then\n'}, 'bash -n'), + ('pools not a list', {'.github/fork-fleet.json': json.dumps({'pools': {'desktop': {}}})}, + 'not a list of pools')): with self.subTest(name): bad = self.release('bad-' + name.replace(' ', '-'), extra) code, _, err = self.install(sha=SHA2, source=bad) @@ -399,6 +441,21 @@ def test_check_names_each_failure(self): self.assertIn(line, out) self.assertTrue(out.endswith('8 FAIL\n'), out) + def test_check_fails_on_a_drop_in_and_on_settings_the_machine_file_no_longer_gives(self): + self.assertEqual(self.install()[0], 0) + self.template() + self.assertEqual(self.run_main('check')[0], 0) + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'slots': 1}}) + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + self.assertIn('FAIL settings /etc/fork-fleet/launch.conf differs from the machine file', out) + self.machine(MACHINE) + self.write(self.path('/etc/systemd/system/fork-fleet-desk-agent.service.d/override.conf'), + '[Service]\nExecStart=\nExecStart=/bin/sh -c evil\n') + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + self.assertIn('override.conf: a drop-in the kit did not write changes fork-fleet-desk-agent.service', out) + def test_check_before_install_fails_without_writing(self): code, out, _ = self.run_main('check') self.assertEqual(code, 1) @@ -433,6 +490,76 @@ def test_uninstall_leaves_nothing_of_its_own(self): self.assertEqual(code, 0) self.assertIn('nothing here was installed by this kit', out) + def test_uninstall_keeps_what_the_kit_did_not_make(self): + self.assertEqual(self.install()[0], 0) + for path in ('/etc/fork-fleet/token', '/opt/fork-fleet/notes.txt', '/var/lib/fork-fleet/other/x'): + self.write(self.path(path), 'not the kit\'s\n') + self.write(self.path('/var/lib/fork-fleet/runners/desktop-1/x'), '') + code, out, err = self.run_main('uninstall') + self.assertEqual(code, 0, err) + left = {k for k in self.tree() if k.startswith(('/etc/fork-fleet', '/opt/fork-fleet', '/var/lib/fork-fleet'))} + self.assertEqual(left, {'/etc/fork-fleet', '/etc/fork-fleet/token', '/opt/fork-fleet', + '/opt/fork-fleet/notes.txt', '/var/lib/fork-fleet', '/var/lib/fork-fleet/other', + '/var/lib/fork-fleet/other/x'}) + self.assertIn('left /etc/fork-fleet/token: the kit did not make it', out) + + def test_uninstall_removes_the_deny_files_it_made(self): + self.write(self.path('/etc/cron.deny'), 'someone\n') # there before: kept, without the kit's lines + self.assertEqual(self.install()[0], 0) + self.assertEqual(self.read('/etc/at.deny'), 'fork-runner-1\nfork-runner-2\n') + self.assertEqual(self.run_main('uninstall')[0], 0) + self.assertFalse(os.path.lexists(self.path('/etc/at.deny'))) # with none, only root may use at again + self.assertEqual(self.read('/etc/cron.deny'), 'someone\n') + + def test_uninstall_keeps_the_table_while_a_runner_still_runs(self): + self.assertEqual(self.install()[0], 0) + self.write(os.path.join(self.fakes, 'units'), 'fork-fleet-desktop-1.service\nfork-fleet-desktop-2.service\n') + self.write(os.path.join(self.fakes, 'stuck'), 'fork-fleet-desktop-2.service\n') + self.clear_calls() + code, _, err = self.run_main('uninstall') + self.assertEqual(code, 1) + self.assertIn('fork-fleet-desktop-2.service is still running', err) + calls = self.calls() + self.assertIn('systemctl stop fork-fleet-desktop-1.service', calls) # what the launcher did not stop + self.assertFalse([c for c in calls if c.startswith(('nft', 'userdel', 'ip netns delete'))], calls) + self.assertTrue(os.path.exists(self.path('/etc/fork-fleet/desk-kit.json'))) + os.unlink(os.path.join(self.fakes, 'stuck')) + self.assertEqual(self.run_main('uninstall')[0], 0) + self.assertIn('nft delete table inet fork_fleet', self.calls()) + + def test_uninstall_removes_every_runner_namespace(self): + self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n') # made by hand, not by the kit + self.assertEqual(self.install()[0], 0) + self.write(os.path.join(self.fakes, 'netns'), 'fork-fleet-fork-runner-1\nfork-fleet-fork-runner-2\nother\n') + self.write(os.path.join(self.fakes, 'links'), 'ffh-1001\neth0\nffh-1009\n') + self.assertEqual(self.run_main('uninstall')[0], 0) + self.assertEqual(slurp(os.path.join(self.fakes, 'netns')).split(), ['other']) + self.assertEqual(slurp(os.path.join(self.fakes, 'links')).split(), ['eth0']) + + def test_uninstall_removes_its_record_last(self): + self.assertEqual(self.install()[0], 0) + gone, unlink = [], os.unlink + + def record(path, *a, **kw): + gone.append(os.path.basename(path)) + return unlink(path, *a, **kw) + os.unlink = record + try: + self.assertEqual(self.run_main('uninstall')[0], 0) + finally: + os.unlink = unlink + self.assertEqual(gone[-1], 'desk-kit.json', gone) + + def test_uninstall_refuses_a_record_of_another_shape(self): + self.assertEqual(self.install()[0], 0) + for bad in ('[]', '{"version": 1, "createdUsers": "fork-runner-1", "fleetConfig": "release"}', + '{"version": 1, "createdUsers": ["root;x"], "fleetConfig": "release"}'): + with self.subTest(bad): + self.write(self.path('/etc/fork-fleet/desk-kit.json'), bad, 0o600) + code, _, err = self.run_main('uninstall') + self.assertEqual(code, 1) + self.assertIn('not an install record', err) + def test_uninstall_stopped_half_way_runs_again(self): self.assertEqual(self.install()[0], 0) self.write(os.path.join(self.fakes, 'fail'), 'userdel\n') @@ -465,7 +592,13 @@ def test_bootstrap_checks_its_arguments_before_anything(self): self.assertEqual(subprocess.run(['sh', '-n', boot]).returncode, 0) env = {'PATH': '/usr/bin:/bin'} for args, msg in ((['--repo', 'x'], '--repo must be'), (['--repo', 'a/b', '--sha', 'x'], '--sha must be'), - (['--nope'], 'unknown option')): + (['--nope'], 'unknown option'), + # flags after --fleet-config are checked too: a later --sha cannot slip past + (['--repo', 'a/b', '--fleet-config', boot, '--sha', 'x'], '--sha must be'), + (['--fleet-config', boot, '--repo', 'x'], '--repo must be'), + (['--repo', 'a/b', '--sha', 'a' * 40, '--fleet-config', '/nonexistent/f.json'], + 'no such file'), + (['--repo', 'a/b', '--sha', 'a' * 40, '--fleet-config'], 'needs a value')): r = subprocess.run(['sh', boot, *args], capture_output=True, text=True, env=env) self.assertEqual(r.returncode, 1) self.assertIn(msg, r.stderr) From 3e3e3140c8729c4cf13201f6817f3b7b10efa084 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:45:29 +0000 Subject: [PATCH 07/33] fork kit desk: Windows starter fix round 1 (D9) - .wslconfig is read and written as UTF-8 without a BOM (never ASCII), by a temp file and a rename, with a dated backup; only [wsl2] keys are judged - the restart decision reads the live state (PID 1, interop, wsl.conf newer than boot, memory, networking mode, .wslconfig newer than the boot) - a distro the starter did not make is named and must be typed to take over - MemoryGB above the PC's RAM less 2 GB is refused - exit 1 on a failed check; -Uninstall needs no -Repo/-Sha; UNC paths work Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 20 +++-- fork-kit/desk/tests/windows.test.py | 91 ++++++++++++++++++++-- fork-kit/desk/windows/start-desk.ps1 | 108 ++++++++++++++++++++------- 3 files changed, 180 insertions(+), 39 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 5e22993a032fd..858bdc0003070 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -55,15 +55,19 @@ Save the machine file on Windows, then in a normal (not administrator) Windows P powershell -ExecutionPolicy Bypass -File start-desk.ps1 -Repo OWNER/REPO -Sha RELEASE_SHA -MachineFile machine.json ``` -It prints the PC's facts and asks before it changes anything. Then: the distro (`-Distro`, default `fork-desk`, -Ubuntu 24.04); `.wslconfig`'s `[wsl2]` memory cap (`-MemoryGB`, default three quarters of the RAM and at least 6 GB -left to Windows), swap (`-SwapGB`, 4), NAT and DNS tunnelling, for every WSL distro (a dated copy of the old file -is kept, other keys stay); the distro's own settings the WSL2 walls ask for (`/etc/wsl.conf`: systemd on, no -Windows drives, programs or PATH; its own `/etc/resolv.conf`; WSL's binfmt handler masked); one `wsl --shutdown` -only when one of these changed (every WSL distro and Docker Desktop stop); the packages the fleet needs; the +It prints the PC's facts and asks before it changes anything. A distro of that name that the script did not make +is named in the question (its whole `/etc/wsl.conf` is replaced) and must be confirmed by typing its name. Then: the +distro (`-Distro`, default `fork-desk`, Ubuntu 24.04); `.wslconfig`'s `[wsl2]` memory cap (`-MemoryGB`, default +three quarters of the RAM, at most the RAM less 6 GB, never under 4 GB; one that leaves Windows under 2 GB is +refused), swap (`-SwapGB`, 4), NAT and DNS tunnelling, for every WSL distro (read and written as UTF-8 without a +BOM; a dated copy of the old file is kept, other keys and sections stay); the distro's own settings the WSL2 walls +ask for (`/etc/wsl.conf`: systemd on, no Windows drives, programs or PATH; its own `/etc/resolv.conf`; WSL's binfmt +handler masked); one `wsl --shutdown` when one of these changed or the running WSL does not have them yet (PID 1 not +systemd, interop live, `wsl.conf` or `.wslconfig` newer than the running one, more memory than the cap, not NAT), +so a run stopped before its shutdown restarts next time (every WSL distro and Docker Desktop stop); the packages the fleet needs; the machine file and `-FleetConfig`, if given; this kit's bootstrap; a logon task that keeps the distro running while -the owner is logged in. Last, `desk-install check` and the Windows side's own PASS/FAIL lines. Safe to run again. -`-Uninstall` runs `desk-install uninstall` in the distro and removes the logon task; the distro and `.wslconfig` +the owner is logged in. Last, `desk-install check` and the Windows side's own PASS/FAIL lines (the `[wsl2]` keys); +any FAIL or error exits 1. Safe to run again. `-Uninstall` (needs no `-Repo` or `-Sha`) runs `desk-install uninstall` in the distro and removes the logon task; the distro and `.wslconfig` stay (`wsl --unregister DISTRO` removes the distro). ## Runs again, stops half-way, old or empty state diff --git a/fork-kit/desk/tests/windows.test.py b/fork-kit/desk/tests/windows.test.py index 51e5f88bd674f..d36922dc5d0c2 100644 --- a/fork-kit/desk/tests/windows.test.py +++ b/fork-kit/desk/tests/windows.test.py @@ -3,7 +3,7 @@ python3 -I -B fork-kit/desk/tests/windows.test.py """ -import importlib.machinery, importlib.util, os, re, shutil, subprocess, tempfile, unittest +import importlib.machinery, importlib.util, os, re, shutil, subprocess, tempfile, time, unittest KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) REPO = os.path.dirname(os.path.dirname(KIT)) @@ -26,11 +26,16 @@ def text(): return f.read() -def prepare(): - m = re.search(r"\$Prepare = @'\r?\n(.*?)\r?\n'@", text().decode('ascii'), re.S) +def prepare(name='Prepare'): + m = re.search(r"\$" + name + r" = @'\r?\n(.*?)\r?\n'@", text().decode('ascii'), re.S) return m.group(1) +def wslconfig_part(): + src = text().decode('ascii') + return src[src.index('# --- wslconfig ---'):src.index('# --- end wslconfig ---')] + + def walls(root): os.environ['FORK_FLEET_WALLS_ROOT'] = root try: @@ -91,6 +96,37 @@ def test_an_empty_distro(self): self.assertFalse([n for n in os.listdir(self.root + '/etc') if '.bak-' in n]) self.assertIsNone(walls(self.root).interop_wall()) + def live(self, env=None): + r = subprocess.run(['bash', '-s'], input=prepare('Live') + '\n# end\r\n', text=True, capture_output=True, + env={'PATH': '/usr/bin:/bin', 'ROOT': self.root, **(env or {})}) + self.assertEqual(r.returncode, 0, r.stderr) + return r.stdout + + def test_the_restart_decision_reads_the_live_state(self): + # a run stopped after it wrote wsl.conf, before its `wsl --shutdown`: the next run changes nothing, but the + # running distro still has the old settings, so it restarts + os.makedirs(self.root + '/proc/1') + put(self.root + '/proc/1/comm', 'systemd\n') + put(self.root + '/proc/meminfo', 'MemTotal: 8000000 kB\n') + self.run_prepare() + past = time.time() - 3600 + os.utime(self.root + '/proc/1', (past, past)) + self.assertIn('restart: /etc/wsl.conf is newer than the running distro', self.live()) + os.utime(self.root + '/etc/wsl.conf', (past - 60, past - 60)) + self.assertEqual(self.live({'WANT_KB': '8388608'}), '') + put(self.root + '/proc/sys/fs/binfmt_misc/WSLInterop', 'enabled\n') + put(self.root + '/proc/1/comm', 'init\n') + out = self.live({'WANT_KB': '4194304'}) + for why in ('PID 1 is not systemd', 'WSL interop is live (WSLInterop)', 'above the cap'): + self.assertIn(why, out) + os.unlink(self.root + '/proc/sys/fs/binfmt_misc/WSLInterop') + put(self.root + '/proc/1/comm', 'systemd\n') + fake = os.path.join(self.root, 'bin') + os.makedirs(fake) + put(fake + '/wslinfo', '#!/bin/sh\necho mirrored\n') + os.chmod(fake + '/wslinfo', 0o755) + self.assertIn('the networking mode is mirrored, not nat', self.live({'PATH': fake + ':/usr/bin:/bin'})) + def test_a_resolv_conf_of_its_own_is_kept(self): put(self.root + '/etc/resolv.conf', 'nameserver 192.0.2.1\n') self.run_prepare() @@ -117,10 +153,55 @@ def test_windows_powershell_reads_it(self): def test_it_asks_before_it_changes_anything(self): src = text().decode('ascii') body = src[src.index('try {'):] - first_change = min(body.index(x) for x in ('wsl.exe --install', 'Set-Content', 'Unregister-ScheduledTask', - 'InDistro', 'Register-ScheduledTask', 'wsl.exe --shutdown')) + first_change = min(body.index(x) for x in ('wsl.exe --install', 'Merge-WslConfig $WslConfig', + 'Unregister-ScheduledTask', 'InDistro', 'Register-ScheduledTask', + 'wsl.exe --shutdown')) self.assertIn("Read-Host", body[:first_change]) + def test_a_distro_it_did_not_make_is_named_and_must_be_typed(self): + src = text().decode('ascii') + body = src[src.index('try {'):] + ask = body.index("if ($have -and -not $ours)") + self.assertLess(ask, body.index('wsl.exe --install')) + self.assertIn("-cne $Distro", body[ask:ask + 800]) + self.assertIn('/etc/wsl.conf is REPLACED', body[ask:ask + 800]) + self.assertIn('/etc/fork-kit/desk-distro', prepare()) # the mark of a distro it made or took over + + def test_wslconfig_is_utf8_without_a_bom_never_ascii(self): + src = text().decode('ascii') + self.assertNotRegex(src, r'(?i)-Encoding\s+ascii') + self.assertNotRegex(src, r'(?i)(Get|Set)-Content[^\n]*\$WslConfig') + self.assertIn('[Text.UTF8Encoding]::new($false)', wslconfig_part()) + + @unittest.skipUnless(shutil.which('pwsh'), 'pwsh is not installed') + def test_wslconfig_merge_keeps_every_other_letter_and_key(self): + d = tempfile.mkdtemp(prefix='desk-wslconfig-test-') + self.addCleanup(shutil.rmtree, d, True) + path = os.path.join(d, '.wslconfig') + old = '[wsl2]\nkernel=D:\\kernels\\Jos\u00e9\\bzImage\nmemory=2GB\n[experimental]\nsparseVhd=true\nmemory=9GB\n' + with open(path, 'w', encoding='utf-8', newline='\n') as f: + f.write(old) + script = os.path.join(d, 'merge.ps1') + with open(script, 'w', encoding='ascii') as f: + f.write(wslconfig_part() + '\n$want = [ordered]@{ memory = "6GB"; swap = "4GB" }\n' + '"changed=$(Merge-WslConfig $args[0] $want)"\n"again=$(Merge-WslConfig $args[0] $want)"\n' + '$k = Wsl2Keys (Read-WslConfig $args[0]); "memory=$($k[\'memory\'])"\n') + r = subprocess.run(['pwsh', '-NoProfile', '-NonInteractive', '-File', script, path], capture_output=True, + text=True, timeout=120, stdin=subprocess.DEVNULL) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn('changed=True', r.stdout) + self.assertIn('again=False', r.stdout) + self.assertIn('memory=6GB', r.stdout) # the [wsl2] one, not [experimental]'s + with open(path, 'rb') as f: + raw = f.read() + self.assertFalse(raw.startswith(b'\xef\xbb\xbf')) + new = raw.decode('utf-8').replace('\r\n', '\n') + self.assertIn('kernel=D:\\kernels\\Jos\u00e9\\bzImage\n', new) + self.assertIn('[wsl2]\nkernel=D:\\kernels\\Jos\u00e9\\bzImage\nmemory=6GB\nswap=4GB\n[experimental]', new) + self.assertIn('sparseVhd=true\nmemory=9GB', new) + backups = [n for n in os.listdir(d) if n.startswith('.wslconfig.bak-')] + self.assertEqual(len(backups), 1) + def test_text_piped_into_the_distro_ends_in_a_comment(self): src = text().decode('ascii') self.assertIn('''function Script([string]$s) { ($s -replace "`r", '') + "`n# end" }''', src) diff --git a/fork-kit/desk/windows/start-desk.ps1 b/fork-kit/desk/windows/start-desk.ps1 index 4c925222ed016..2d774f9e3f3ff 100644 --- a/fork-kit/desk/windows/start-desk.ps1 +++ b/fork-kit/desk/windows/start-desk.ps1 @@ -8,8 +8,8 @@ # resolv.conf, systemd); one `wsl --shutdown`; the machine file; the desk kit (bootstrap.sh, in the distro); a logon # task that keeps the distro running; then `desk-install check` and this side's checks. param( - [Parameter(Mandatory)][ValidatePattern('^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$')][string]$Repo, - [Parameter(Mandatory)][ValidatePattern('^[0-9a-f]{40}$')][string]$Sha, + [ValidatePattern('^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$')][string]$Repo, # both required, except with -Uninstall + [ValidatePattern('^[0-9a-f]{40}$')][string]$Sha, [string]$MachineFile, [string]$FleetConfig, [ValidatePattern('^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$')][string]$Distro = 'fork-desk', [ValidateRange(0, 4096)][int]$MemoryGB = 0, [ValidateRange(0, 4096)][int]$SwapGB = 4, @@ -35,7 +35,7 @@ function InDistro([string]$what, [string]$script) { # bash as root in the dist } function Into([string]$file, [string]$dest) { # a Windows file into the distro, root-owned, mode 644 if (-not (Test-Path -LiteralPath $file -PathType Leaf)) { throw "no such file: $file" } - $text = [IO.File]::ReadAllText((Resolve-Path -LiteralPath $file)) -replace "`r", '' + $text = [IO.File]::ReadAllText((Resolve-Path -LiteralPath $file).ProviderPath) -replace "`r", '' # UNC paths too $dir = $dest.Substring(0, $dest.LastIndexOf('/')) # no quotes in the command: Windows PowerShell mangles them $text | wsl.exe -d $Distro -u root --cd / -- sh -c "umask 022; mkdir -p $dir && cat > $dest.new && mv $dest.new $dest" if ($LASTEXITCODE -ne 0) { throw "copying $file into the distro failed (exit $LASTEXITCODE)" } @@ -69,7 +69,59 @@ if [ -L "$ROOT/etc/resolv.conf" ] || [ ! -f "$ROOT/etc/resolv.conf" ]; then fi [ "$(readlink "$ROOT/etc/binfmt.d/WSLInterop.conf" 2>/dev/null)" = /dev/null ] || { ln -sfn /dev/null "$ROOT/etc/binfmt.d/WSLInterop.conf"; echo 'changed: /etc/binfmt.d/WSLInterop.conf'; } +mkdir -p "$ROOT/etc/fork-kit" # this distro is the desk's now: a later run asks no takeover question +[ -e "$ROOT/etc/fork-kit/desk-distro" ] || echo 'made or taken over by start-desk.ps1' > "$ROOT/etc/fork-kit/desk-distro" '@ +# What the running WSL has now, whatever this run changed: a run stopped before its `wsl --shutdown` (or settings +# changed by hand) still restarts. One line per reason; nothing when the live state is what the walls need. +$Live = @' +set -eu +ROOT=${ROOT:-} +[ "$(cat "$ROOT/proc/1/comm" 2>/dev/null)" = systemd ] || echo 'restart: PID 1 is not systemd' +for h in "$ROOT"/proc/sys/fs/binfmt_misc/WSLInterop*; do + [ ! -e "$h" ] || { echo "restart: WSL interop is live (${h##*/})"; break; } +done +[ ! "$ROOT/etc/wsl.conf" -nt "$ROOT/proc/1" ] || echo 'restart: /etc/wsl.conf is newer than the running distro' +mem=$(sed -n 's/^MemTotal: *\([0-9]*\) kB$/\1/p' "$ROOT/proc/meminfo" 2>/dev/null || true) +[ -z "${WANT_KB:-}" ] || [ -z "$mem" ] || [ "$mem" -le "$WANT_KB" ] || echo "restart: WSL has $mem kB, above the cap" +mode=$(wslinfo --networking-mode 2>/dev/null || true) +[ -z "$mode" ] || [ "$mode" = nat ] || echo "restart: the networking mode is $mode, not nat" +'@ +# .wslconfig, read and written as UTF-8 without a BOM: Windows PowerShell 5.1's Get-Content reads a file without a +# BOM as ANSI and Set-Content -Encoding ascii turns every other letter into '?' (a kernel= path with an accent in it +# then stops every distro). The tests run this part alone (between the two marker lines) under pwsh. +# --- wslconfig --- +function Read-WslConfig([string]$path) { + if (-not (Test-Path -LiteralPath $path)) { return @() } + @([IO.File]::ReadAllLines($path, [Text.UTF8Encoding]::new($false))) # a BOM (UTF-8 or UTF-16) is still read +} +function Wsl2Keys([string[]]$lines) { # the keys of the [wsl2] section only + $keys = @{}; $sec = '' + foreach ($l in $lines) { + if ($l -match '^\s*\[(.+)\]\s*$') { $sec = $Matches[1].Trim().ToLower(); continue } + if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=\s*(.*?)\s*$') { $keys[$Matches[1]] = $Matches[2] } + } + $keys +} +function Merge-WslConfig([string]$path, $want) { # True when it changed the file (a dated copy of the old one kept) + $lines = Read-WslConfig $path + $out = [Collections.Generic.List[string]]::new(); $sec = ''; $done = @{} + function AddMissing { foreach ($k in $want.Keys) { if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 } } } + foreach ($l in $lines) { + if ($l -match '^\s*\[(.+)\]\s*$') { if ($sec -eq 'wsl2') { AddMissing }; $sec = $Matches[1].Trim().ToLower(); $out.Add($l); continue } + if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=' -and $want.Contains($Matches[1])) { + $k = $Matches[1]; if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 }; continue } + $out.Add($l) + } + if ($sec -eq 'wsl2') { AddMissing } + if ($done.Count -lt $want.Count) { $out.Add('[wsl2]'); AddMissing } + if (($out -join "`n") -eq ($lines -join "`n")) { return $false } + if (Test-Path -LiteralPath $path) { Copy-Item -LiteralPath $path "$path.bak-$(Get-Date -Format yyyyMMdd-HHmmss)" } + [IO.File]::WriteAllLines("$path.new", [string[]]$out, [Text.UTF8Encoding]::new($false)) + Move-Item -LiteralPath "$path.new" $path -Force + $true +} +# --- end wslconfig --- $Packages = 'set -eu; export DEBIAN_FRONTEND=noninteractive; need=; for p in python3 nftables iproute2 curl ca-certificates openssh-client tar; do dpkg -s $p >/dev/null 2>&1 || need="$need $p"; done; [ -z "$need" ] || { apt-get update -q && apt-get install -y -q $need; }' try { @@ -77,11 +129,15 @@ try { $os = Get-CimInstance Win32_OperatingSystem; $cs = Get-CimInstance Win32_ComputerSystem $ram = [math]::Floor($cs.TotalPhysicalMemory / 1GB) if ($MemoryGB -eq 0) { $MemoryGB = [math]::Max(4, [math]::Min([math]::Floor($ram * 3 / 4), $ram - 6)) } + if (-not $Uninstall -and $MemoryGB -gt $ram - 2) { throw "-MemoryGB $MemoryGB leaves Windows less than 2 GB of this PC's $ram GB" } "Windows $($os.Version) build $($os.BuildNumber); $($cs.NumberOfLogicalProcessors) threads; RAM $ram GB" 'WSL:'; wsl.exe --version $have = @(wsl.exe -l -q | ForEach-Object { $_.Trim([char]0, ' ') }) -contains $Distro "distro ${Distro}: $(if ($have) { 'there' } else { 'not yet' })" - if (Test-Path $WslConfig) { ".wslconfig now:"; Get-Content $WslConfig | ForEach-Object { " $_" } } + if (Test-Path $WslConfig) { ".wslconfig now:"; Read-WslConfig $WslConfig | ForEach-Object { " $_" } } + # A distro of that name that this script did not make: its /etc/wsl.conf is replaced below + $ours = $false + if ($have) { wsl.exe -d $Distro -u root --cd / -- test -e /etc/fork-kit/desk-distro; $ours = $LASTEXITCODE -eq 0 } if ($Uninstall) { if ((Read-Host "Remove the desk kit from $Distro and the logon task $Task? Type yes") -ne 'yes') { 'Stopped: nothing changed.'; return } @@ -91,12 +147,19 @@ try { return } if (-not $MachineFile) { throw '-MachineFile is required: the machine file with the desk section (README)' } + if (-not $Repo -or -not $Sha) { throw '-Repo and -Sha are required (the fork and its release commit)' } Write-Host '' Write-Host "Next: the distro $Distro; .wslconfig [wsl2] memory=${MemoryGB}GB swap=${SwapGB}GB networkingMode=nat" -ForegroundColor Yellow Write-Host 'dnsTunneling=true guiApplications=false (for ALL your WSL distros; a dated copy is kept); then ONE' -ForegroundColor Yellow Write-Host 'wsl --shutdown (every WSL distro and Docker Desktop stop), the desk kit, and a logon task.' -ForegroundColor Yellow - if ((Read-Host 'Type yes to go on') -ne 'yes') { Write-Host 'Stopped: nothing changed.'; return } + if ($have -and -not $ours) { + Write-Host "The distro $Distro is already there and this script did not make it. Its /etc/wsl.conf is REPLACED" -ForegroundColor Yellow + Write-Host '(its [user] default= and every other line go; no /mnt/c, no Windows programs or PATH), and it runs fleet' -ForegroundColor Yellow + Write-Host 'jobs from now on. Pick another -Distro to keep it as it is.' -ForegroundColor Yellow + if ((Read-Host "Type the distro's name ($Distro) to take it over") -cne $Distro) { Write-Host 'Stopped: nothing changed.'; return } + } + elseif ((Read-Host 'Type yes to go on') -ne 'yes') { Write-Host 'Stopped: nothing changed.'; return } Head 'the distro' if (-not $have) { @@ -107,28 +170,20 @@ try { Head '.wslconfig' $want = [ordered]@{ memory = "${MemoryGB}GB"; swap = "${SwapGB}GB"; networkingMode = 'nat'; dnsTunneling = 'true'; guiApplications = 'false' } - $lines = if (Test-Path $WslConfig) { @(Get-Content $WslConfig) } else { @() } - $out = [Collections.Generic.List[string]]::new(); $sec = ''; $done = @{} - function AddMissing { foreach ($k in $want.Keys) { if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 } } } - foreach ($l in $lines) { - if ($l -match '^\s*\[(.+)\]\s*$') { if ($sec -eq 'wsl2') { AddMissing }; $sec = $Matches[1].Trim().ToLower(); $out.Add($l); continue } - if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=' -and $want.Contains($Matches[1])) { - $k = $Matches[1]; if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 }; continue } - $out.Add($l) - } - if ($sec -eq 'wsl2') { AddMissing } - if ($done.Count -lt $want.Count) { $out.Add('[wsl2]'); AddMissing } - $restart = ($out -join "`n") -ne ($lines -join "`n") - if ($restart) { - if (Test-Path $WslConfig) { Copy-Item $WslConfig "$WslConfig.bak-$(Get-Date -Format yyyyMMdd-HHmmss)" } - Set-Content -Encoding ascii $WslConfig $out; 'changed: .wslconfig' - } + $restart = Merge-WslConfig $WslConfig $want + if ($restart) { 'changed: .wslconfig' } Head "the distro's own settings" $changed = InDistro "the distro's settings" $Prepare $changed - $pid1 = (wsl.exe -d $Distro -u root --cd / -- ps -p 1 -o comm=) -join '' - if ($restart -or $changed -or $pid1.Trim() -ne 'systemd') { + # The live state decides, not only what this run changed: a run stopped before its shutdown restarts next time + $live = @(InDistro 'the running settings' ("WANT_KB=$([long]$MemoryGB * 1048576)`n" + $Live)) + $up = [double]((wsl.exe -d $Distro -u root --cd / -- cat /proc/uptime) -split ' ')[0] + if ((Test-Path $WslConfig) -and ((Get-Date) - (Get-Item -LiteralPath $WslConfig).LastWriteTime).TotalSeconds -lt $up) { + $live += 'restart: .wslconfig is newer than the running WSL' + } + $live + if ($restart -or $changed -or $live) { Head 'wsl --shutdown (the new settings apply at the next start)' Run 'wsl --shutdown' { wsl.exe --shutdown } } @@ -155,16 +210,17 @@ sh /root/desk-bootstrap.sh --repo $Repo --sha $Sha$extra $bad = 0 wsl.exe -d $Distro -u root --cd / -- /opt/fork-fleet/desk-install check if ($LASTEXITCODE -ne 0) { $bad++ } - $cfg = Get-Content $WslConfig -Raw + $cfg = Wsl2Keys (Read-WslConfig $WslConfig) foreach ($k in $want.Keys) { - if ($cfg -match "(?m)^\s*$k\s*=\s*$([regex]::Escape($want[$k]))\s*$") { "PASS wslconfig $k=$($want[$k])" } else { "FAIL wslconfig no $k=$($want[$k])"; $bad++ } + if ($cfg[$k] -eq $want[$k]) { "PASS wslconfig [wsl2] $k=$($want[$k])" } else { "FAIL wslconfig [wsl2] no $k=$($want[$k])"; $bad++ } } $t = Get-ScheduledTask $Task -ErrorAction SilentlyContinue if ($t -and $t.State -eq 'Running') { "PASS logon-task $Task running" } else { "FAIL logon-task $Task is $(if ($t) { $t.State } else { 'missing' })"; $bad++ } if ($bad -eq 0) { Write-Host 'ALL PASS: this PC takes fleet jobs while you are logged in.' -ForegroundColor Green } - else { Write-Host 'Some FAIL lines above (a template FAIL clears by itself in a few minutes; run this again then).' -ForegroundColor Red } + else { Write-Host 'Some FAIL lines above (a template FAIL clears by itself in a few minutes; run this again then).' -ForegroundColor Red; exit 1 } } catch { Write-Host "FAIL: $($_.Exception.Message)" -ForegroundColor Red Write-Host 'Nothing printed here is a secret. Fix it and run this again: every step checks what is there.' -ForegroundColor Red + exit 1 } From 8a312aeae143270684cf0f92688090147845f6dd Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:45:39 +0000 Subject: [PATCH 08/33] fork kit: free/v1 fix round 1 (D10) - ask() runs the file unsafe() judged (the link's target), never the link - NaN and Infinity are not JSON: not available (two fixture cases) - a child that left the group with setsid and still holds stdout is ended; one that also closed stdout is the documented bound - poll() instead of select() (descriptors above 1024) - busy-while-running matches names longer than 15 characters - FREE.md: the bound, links, ACLs, the memory example's pressure caveat Co-Authored-By: Claude Opus 5.5 --- fork-kit/common/FREE.md | 16 ++++--- fork-kit/common/free-fixtures.json | 14 ++++++ fork-kit/common/free/busy-while-running | 3 +- fork-kit/common/free_contract.py | 51 ++++++++++++++++---- fork-kit/common/tests/free_contract.test.py | 52 +++++++++++++++++++++ 5 files changed, 120 insertions(+), 16 deletions(-) diff --git a/fork-kit/common/FREE.md b/fork-kit/common/FREE.md index 800793bfcd73f..acede94965da0 100644 --- a/fork-kit/common/FREE.md +++ b/fork-kit/common/FREE.md @@ -7,12 +7,15 @@ One small contract for every tool that asks whether a machine may take more work ## The script - Called with **no arguments**, by its **absolute path**, stdin empty, a plain environment (`PATH`, `LANG`), - from `/`. It answers within **3 seconds** or counts as not available; its whole process group is then ended. + from `/`. It answers within **3 seconds** or counts as not available; its whole process group is then ended, + and so is any process that still holds its stdout. It leaves nothing running: a child that leaves the group + (`setsid`) and closes stdout is not seen and not ended. - It and every folder above it belong to root (or to the user the consumer runs as) and no one else can change - them (a sticky folder such as `/tmp` is fine). Otherwise it is not run. + them (a sticky folder such as `/tmp` is fine). Otherwise it is not run. A link is judged and run by the file it + leads to. Only the mode bits are read: an ACL that lets others write is not seen, so do not set one. - **The exit code is the verdict:** `0` free, `75` busy. Anything else, a timeout, output that is not one JSON - object, a higher `v`, a `free` that disagrees with the exit code, or facts more than 60 seconds from now = **not - available**. Busy and not available both count as **not free**. + object (`NaN` and `Infinity` are not JSON), a higher `v`, a `free` that disagrees with the exit code, or facts + more than 60 seconds from now = **not available**. Busy and not available both count as **not free**. - **stdout is the detail:** one JSON object of at most 4096 bytes. Unknown fields are ignored. | Field | Required | What | @@ -55,9 +58,10 @@ size as busy. An answer without `memory.freeMb` makes no memory claim; the consu - [free-fixtures.json](free-fixtures.json): the cases every reader, in any language, must agree on. - Examples in [free/](free/): - [memory](free/memory): busy under memory pressure or with swap nearly full. The desk kit installs it as - `/opt/fork-fleet/free-memory`. + `/opt/fork-fleet/free-memory`. Pressure is "all tasks stalled" over 10 seconds, which can stay low while the + machine swaps hard; the swap rule is there for that. - [busy-while-running](free/busy-while-running): busy while a named program runs (a game, say). Copy it and put - your programs in it. + your programs in it. It sees the processes of its own system only: in WSL, Windows programs are not visible. Several conditions in one script: call the others from it and answer busy when one is busy. diff --git a/fork-kit/common/free-fixtures.json b/fork-kit/common/free-fixtures.json index f9c6b1573d04a..062358fc48486 100644 --- a/fork-kit/common/free-fixtures.json +++ b/fork-kit/common/free-fixtures.json @@ -53,6 +53,20 @@ "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\"}", "verdict": "free" }, + { + "name": "NaN is not JSON", + "exit": 0, + "needMb": 100, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\", \"memory\": {\"freeMb\": NaN}}", + "verdict": "unavailable" + }, + { + "name": "Infinity is not JSON", + "exit": 0, + "needMb": 100, + "stdout": "{\"v\": 1, \"free\": true, \"at\": \"2026-11-15T05:00:00Z\", \"memory\": {\"freeMb\": -Infinity}}", + "verdict": "unavailable" + }, { "name": "no answer in time", "exit": null, "stdout": "", "verdict": "unavailable" }, { "name": "another exit code", diff --git a/fork-kit/common/free/busy-while-running b/fork-kit/common/free/busy-while-running index 3dd1b9b18c96d..f5b5cd40723bd 100755 --- a/fork-kit/common/free/busy-while-running +++ b/fork-kit/common/free/busy-while-running @@ -8,7 +8,8 @@ NAMES='example-game example-editor' at=$(date -u +%Y-%m-%dT%H:%M:%SZ) running=$(ps -e -o comm=) || { echo '{"v": 1, "free": false, "reason": "ps failed", "at": "'"$at"'"}'; exit 1; } for name in $NAMES; do - if printf '%s\n' "$running" | grep -qxF -- "$name"; then + short=$(printf '%.15s' "$name") # the kernel keeps only the first 15 characters of a process name + if printf '%s\n' "$running" | grep -qxF -- "$short"; then echo '{"v": 1, "free": false, "reason": "'"$name"' is running", "at": "'"$at"'"}' exit 75 fi diff --git a/fork-kit/common/free_contract.py b/fork-kit/common/free_contract.py index 045153c71a94d..1df6c09274543 100644 --- a/fork-kit/common/free_contract.py +++ b/fork-kit/common/free_contract.py @@ -28,6 +28,10 @@ def parse_at(at): return when if when.tzinfo is not None else None +def no_constant(name): + raise ValueError(f'{name} is not JSON') + + def judge(code, out, now, need_mb=None): """(verdict, reason, facts) for one answer: verdict 'free', 'busy' or 'unavailable' (only 'free' is free). @@ -42,8 +46,8 @@ def no(why): if code not in (FREE, BUSY): return no(f'exit {code}') try: - facts = json.loads(out.decode('utf-8')) - except ValueError: # UnicodeDecodeError is one + facts = json.loads(out.decode('utf-8'), parse_constant=no_constant) + except ValueError: # UnicodeDecodeError is one, and so is NaN or Infinity (not JSON, and NaN < size is False) return no('stdout is not JSON') if not isinstance(facts, dict): return no('stdout is not a JSON object') @@ -72,7 +76,9 @@ def no(why): def unsafe(command, uid=None): """Why COMMAND must not run, or None: an absolute plain path to an executable file that it and every folder - above it belong to root or to this user and that no one else may change (a sticky folder like /tmp is fine).""" + above it belong to root or to this user and that no one else may change (a sticky folder like /tmp is fine). + A link is judged by the file it leads to, and ask() runs that file, never the link (it could be repointed). + Plain mode bits only: an ACL that lets others write is not seen.""" uid = os.geteuid() if uid is None else uid if not os.path.isabs(command) or os.path.normpath(command) != command: return f'{command!r} is not an absolute plain path' @@ -93,26 +99,47 @@ def unsafe(command, uid=None): return f'{path}: {e.strerror}' +def holders(pipe): + """The other processes that hold the pipe PIPE (a file object) open: a script's child that left its process + group (setsid) and kept stdout. Only processes this user may look at are seen (all of them for root).""" + name, found = f'pipe:[{os.fstat(pipe.fileno()).st_ino}]', [] + for pid in filter(str.isdigit, os.listdir('/proc')): + if int(pid) == os.getpid(): + continue + try: + if any(os.readlink(f'/proc/{pid}/fd/{fd}') == name for fd in os.listdir(f'/proc/{pid}/fd')): + found.append(int(pid)) + except OSError: # gone, or not ours to look at + pass + return found + + def ask(command, need_mb=None, timeout=LIMIT_SECONDS): - """Runs the free script (no arguments, stdin empty, a plain environment, its whole process group ended after it) - and judges its answer: (verdict, reason, facts).""" + """Runs the free script (no arguments, stdin empty, a plain environment) and judges its answer: (verdict, + reason, facts). After it, its whole process group is ended, and so is any other process that still holds its + stdout (a child that left the group with setsid). A child that leaves the group and closes stdout is not seen: + FREE.md asks scripts to leave nothing running.""" why = unsafe(command) if why: return 'unavailable', why, None + path = os.path.realpath(command) # what unsafe() judged; the link itself may be repointed after the check try: - p = subprocess.Popen([command], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, + p = subprocess.Popen([path], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=ENV, cwd='/', start_new_session=True) except OSError as e: - return 'unavailable', f'cannot run {command}: {e.strerror}', None - out, code, deadline = b'', None, time.monotonic() + timeout + return 'unavailable', f'cannot run {path}: {e.strerror}', None + out, code, closed, deadline = b'', None, False, time.monotonic() + timeout + poll = select.poll() # not select(): a long-lived consumer may hold descriptors above 1024 + poll.register(p.stdout, select.POLLIN) with p: try: while len(out) <= MAX_BYTES: left = deadline - time.monotonic() - if left <= 0 or not select.select([p.stdout], [], [], left)[0]: + if left <= 0 or not poll.poll(left * 1000): break chunk = os.read(p.stdout.fileno(), 65536) if not chunk: + closed = True try: code = p.wait(max(0.0, deadline - time.monotonic())) except subprocess.TimeoutExpired: @@ -124,6 +151,12 @@ def ask(command, need_mb=None, timeout=LIMIT_SECONDS): os.killpg(p.pid, signal.SIGKILL) except (ProcessLookupError, PermissionError): pass + if not closed: # someone still holds stdout open + for pid in holders(p.stdout): + try: + os.kill(pid, signal.SIGKILL) + except (ProcessLookupError, PermissionError): + pass return judge(code, out, datetime.datetime.now(datetime.timezone.utc), need_mb) diff --git a/fork-kit/common/tests/free_contract.test.py b/fork-kit/common/tests/free_contract.test.py index 6c0a0ffce2f43..2641736a197f1 100644 --- a/fork-kit/common/tests/free_contract.test.py +++ b/fork-kit/common/tests/free_contract.test.py @@ -127,6 +127,45 @@ def test_a_link_is_judged_by_where_it_leads(self): os.symlink(s, os.path.join(self.d, 'link')) self.assertEqual(fc.ask(os.path.join(self.d, 'link'))[0], 'free') + def test_a_link_repointed_after_the_check_does_not_change_what_runs(self): + good = self.script(f': > {self.d}/good-ran\necho \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n', + name='good') + bad = self.script(f': > {self.d}/bad-ran\necho \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n', + name='bad') + link = os.path.join(self.d, 'link') + os.symlink(good, link) + checked = fc.unsafe + + def check_then_repoint(command, uid=None): + why = checked(command, uid) + os.unlink(link) + os.symlink(bad, link) # what another user could do in a folder they may change + return why + fc.unsafe = check_then_repoint + try: + self.assertEqual(fc.ask(link)[0], 'free') + finally: + fc.unsafe = checked + self.assertEqual((os.path.exists(f'{self.d}/good-ran'), os.path.exists(f'{self.d}/bad-ran')), (True, False)) + + def test_a_child_that_left_the_group_and_holds_stdout_is_ended(self): + s = self.script(f'setsid sh -c \'echo $$ > {self.d}/child; exec sleep 30\' &\n' + f'echo \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n') + self.assertEqual(fc.ask(s, timeout=1)[0], 'unavailable') + with open(f'{self.d}/child') as f: + child = int(f.read()) + for _ in range(50): + try: + with open(f'/proc/{child}/stat') as f: + if f.read().rsplit(')', 1)[1].split()[0] == 'Z': + break + except (FileNotFoundError, ProcessLookupError): + break + time.sleep(0.1) + else: + os.kill(child, 9) + self.fail('the child that left the group with setsid still runs') + def test_the_command_line(self): def cli(*args): return subprocess.run([sys.executable, '-I', '-B', READER, *args], capture_output=True, text=True, @@ -177,6 +216,19 @@ def test_busy_while_running(self): p.kill() p.wait() + def test_busy_while_running_knows_names_longer_than_15_characters(self): + with open(os.path.join(COMMON, 'free', 'busy-while-running')) as f: + text = f.read() + copy = self.copy('busy', text.replace("NAMES='example-game example-editor'", "NAMES='sleeps-a-long-while'")) + d = os.path.dirname(copy) + shutil.copy(shutil.which('sleep'), os.path.join(d, 'sleeps-a-long-while')) + p = subprocess.Popen([os.path.join(d, 'sleeps-a-long-while'), '30']) + try: + self.assertEqual(fc.ask(copy)[:2], ('busy', 'sleeps-a-long-while is running')) + finally: + p.kill() + p.wait() + if __name__ == '__main__': unittest.main() From 7026afd9ace39bf0d86a2b1e6f2b41fe1ee418d7 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:46:19 +0000 Subject: [PATCH 09/33] fork fleet: seed fix round 1 (D11) - seeds live in FILES/POOL/NAME.tar: pools never share a file - a file dated more than five minutes in the future is refused (exit 4) - the stream stops at the size that was checked; ctime and inode join the changed-file check - the forced command sends every command that starts with `seed` to the seed handler, so `seeds x` and `seed` exit 2 and never reach a session - a desktop that takes nothing for STALL seconds is cut off (exit 1) and the pool's lock let go - the lock file must be this user's; a clear refusal otherwise Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 19 +++-- .github/scripts/fork-fleet-desk-seed.py | 89 +++++++++++++------- .github/scripts/fork-fleet-desk-seed.test.py | 88 ++++++++++++++++--- .github/scripts/fork-fleet-desk.py | 9 +- 4 files changed, 153 insertions(+), 52 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index 1a0b7c895f3a5..ca2414b079264 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -698,16 +698,21 @@ ready-made caches the desktop keeps and its jobs start from: least `every_hours`) the age, from when it was made, at which it is dropped, `max_mb` its largest size. Nothing in the fleet makes a seed: on the always-on host the project's own timer or job writes a tar of a cache folder's contents -to `/var/lib/fork-fleet/seed-files/NAME.tar`, by a temp file and a rename (a -file changed while it is sent is refused). That folder and its files are -root's (or the project's), readable by the desk-link user and writable by no -one else; never put a secret in a seed: every job of the pool reads it. +to `/var/lib/fork-fleet/seed-files/POOL/NAME.tar` (a folder per pool, so two +pools never share a file), by a temp file and a rename. A file changed while it +is sent (size, mtime or ctime), dated more than five minutes in the future, older +than `max_age_hours` or above `max_mb` is refused. Both folders and the files are +root's (or the project's), readable by the desk-link user and writable by no one +else (the owner itself is not checked, only that no one else may write); never +put a secret in a seed: every job of the pool reads it. The desktop asks for them with the desk link's key, in an ssh session of its -own: the same forced command answers `seeds` with the pool's list and +own: the same forced command sends every command that starts with `seed` to +`fork-fleet-desk-seed.py` (which step 3 installs; its header has the exit +codes), never to a session. It answers `seeds` with the pool's list and `seed NAME` with that one file, only a name of that list, and a trailer with its -size and SHA-256 (`fork-fleet-desk-seed.py`, which step 3 installs; its header -has the exit codes). +size and SHA-256; anything else is refused. One stream per pool at a time; a +desktop that takes nothing for two minutes is cut off and the next one may go. ## Owner steps on the repository diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index 2a68f886fc7df..8aebe9fbd05f3 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -4,20 +4,23 @@ A project lists the caches its desk pool's jobs may start from in the pool's `seeds` (fork-fleet.json): a name, how often a desktop takes a new one (every_hours), the age at which it is dropped (max_age_hours, from when it was made) and its largest size (max_mb). On the always-on host something of the project's own (a timer, a job) makes each one: -a tar of a folder's contents at FILES/NAME.tar, written to a temp file and renamed. Nothing here makes one. The -desktop pulls them with the desk link's key, in an ssh session of its own beside the agent's: +a tar of a folder's contents at FILES/POOL/NAME.tar (a folder per pool: pools never share a file), written to a temp +file and renamed. Nothing here makes one. The desktop pulls them with the desk link's key, in an ssh session of its +own beside the agent's: ssh ... seeds the forced command (fork-fleet-desk.py serve POOL, the desk-link user) sees it in SSH_ORIGINAL_COMMAND and prints the pool's list (one JSON object; `bytes` and `made` of each seed the host offers now, `why` for one it does not) - ssh ... seed NAME streams FILES/NAME.tar: only a name of that list, never a path the desktop names. The last + ssh ... seed NAME streams FILES/POOL/NAME.tar: only a name of that list, never a path the desktop names. The last stderr line is the trailer `seed NAME BYTES SHA256 MADE` (MADE: the file's mtime, epoch seconds) -The host side exits 2 for any other command, 3 while another stream for the pool runs, 4 when the file may not be -sent (missing, a link, in a folder or a file others may write, above max_mb, older than max_age_hours, or changed -while it was sent: made in place, not by rename). Standard library only. +The forced command sends every command that starts with `seed` here. The host side exits 2 for any other command, +3 while another stream for the pool runs, 4 when the file may not be sent (missing, a link, in a folder or a file +others may write, above max_mb, older than max_age_hours or dated in the future, or changed while it was sent: made +in place, not by rename), 1 when the desktop stops reading for STALL seconds (the pool's lock is then let go). +Standard library only. """ -import fcntl, hashlib, importlib.util, json, os, re, stat, sys, time +import fcntl, hashlib, importlib.util, json, os, signal, stat, sys, time sys.dont_write_bytecode = True HERE = os.path.dirname(os.path.realpath(__file__)) @@ -30,9 +33,9 @@ def module(name, file): return mod -FILES = '/var/lib/fork-fleet/seed-files' # the always-on host's: NAME.tar, made by the project +FILES = '/var/lib/fork-fleet/seed-files' # the always-on host's: POOL/NAME.tar, made by the project CONFIG = os.path.join(HERE, 'fork-fleet.json') -PIECE = 1 << 20 +PIECE, STALL, AHEAD = 1 << 20, 120, 300 # AHEAD: seconds a file's date may be ahead of now (a clock set back) ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'} @@ -53,13 +56,14 @@ def pool_seeds(pool, config): raise Refused(f'pool {pool} is not in {config}', 2) -def offered(seed, files, now): - """An open fd and the stat of FILES/NAME.tar; Refused (4) when it may not be sent.""" - path = os.path.join(files, seed['name'] + '.tar') +def offered(seed, folder, now): + """An open fd and the stat of FOLDER/NAME.tar (FOLDER: FILES/POOL); Refused (4) when it may not be sent.""" + path = os.path.join(folder, seed['name'] + '.tar') try: - st = os.lstat(files) - if not stat.S_ISDIR(st.st_mode) or st.st_mode & 0o022: - raise Refused(f'{files} must be a folder (not a link) that only its owner may write', 4) + for d in (os.path.dirname(folder), folder): + st = os.lstat(d) + if not stat.S_ISDIR(st.st_mode) or st.st_mode & 0o022: + raise Refused(f'{d} must be a folder (not a link) that only its owner may write', 4) fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) except OSError as e: raise Refused(f'{path}: {"not made yet" if isinstance(e, FileNotFoundError) else e.strerror}', 4) from None @@ -67,7 +71,8 @@ def offered(seed, files, now): hours = seed['max_age_hours'] why = ('not a plain file' if not stat.S_ISREG(st.st_mode) else 'others may write it' if st.st_mode & 0o022 else f'above max_mb ({seed["max_mb"]})' if st.st_size > seed['max_mb'] << 20 - else f'older than max_age_hours ({hours})' if now - st.st_mtime > hours * 3600 else None) + else f'older than max_age_hours ({hours})' if now - st.st_mtime > hours * 3600 + else 'dated in the future' if st.st_mtime > now + AHEAD else None) if why: os.close(fd) raise Refused(f'{path}: {why}', 4) @@ -79,13 +84,13 @@ def offer(pool, command, lock_dir, out=1, err=sys.stderr, config=CONFIG, files=F if os.geteuid() == 0: raise Refused('serve never runs as root', 2) word, _, name = command.partition(' ') - seeds = pool_seeds(pool, config) + seeds, folder = pool_seeds(pool, config), os.path.join(files, pool) if command == 'seeds': got = [] for s in seeds.values(): item = {'name': s['name'], **{k: s[v] for k, v in ITEM.items()}} try: - fd, st = offered(s, files, now()) + fd, st = offered(s, folder, now()) os.close(fd) item.update(bytes=st.st_size, made=int(st.st_mtime)) except Refused as e: @@ -95,21 +100,28 @@ def offer(pool, command, lock_dir, out=1, err=sys.stderr, config=CONFIG, files=F return 0 if word != 'seed' or name not in seeds: raise Refused(f'refused: {command[:60]!r} (only `seeds`, or `seed NAME` of pool {pool}\'s seeds)', 2) - lock = os.open(os.path.join(lock_dir, f'{pool}.seed.lock'), os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600) + lock_path = os.path.join(lock_dir, f'{pool}.seed.lock') try: + lock = os.open(lock_path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600) + except OSError as e: + raise Refused(f'{lock_path}: {e.strerror} (a leftover of another user? remove it)', 1) from None + try: + if os.fstat(lock).st_uid != os.geteuid(): + raise Refused(f'{lock_path} belongs to uid {os.fstat(lock).st_uid}, not this user: remove it', 1) try: fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) except OSError: raise Refused(f'another seed stream for pool {pool} runs', 3) from None - fd, st = offered(seeds[name], files, now()) + fd, st = offered(seeds[name], folder, now()) try: - sha, sent = send(fd, out) + sha, sent = send(fd, out, st.st_size) after = os.fstat(fd) finally: os.close(fd) finally: os.close(lock) - if sent != st.st_size or (after.st_size, after.st_mtime_ns) != (st.st_size, st.st_mtime_ns): + same = ('st_size', 'st_mtime_ns', 'st_ctime_ns', 'st_ino') + if sent != st.st_size or any(getattr(after, k) != getattr(st, k) for k in same): raise Refused(f'{name}: it changed while it was sent (made in place, not by rename?)', 4) print(f'seed {name} {sent} {sha} {int(st.st_mtime)}', file=err, flush=True) return 0 @@ -118,12 +130,31 @@ def offer(pool, command, lock_dir, out=1, err=sys.stderr, config=CONFIG, files=F return e.code -def send(fd, out): +class Stalled(Exception): + pass + + +def stalled(*_): + raise Stalled + + +def send(fd, out, size, stall=None): + """Sends at most SIZE bytes of FD (the size that was checked: a file that grows is cut there, and then fails the + changed-file check) to OUT; Refused (1) when OUT takes nothing for STALL seconds.""" + stall = STALL if stall is None else stall sha, sent = hashlib.sha256(), 0 - while data := os.read(fd, PIECE): - sha.update(data) - sent += len(data) - view = memoryview(data) - while view: - view = view[os.write(out, view):] + old = signal.signal(signal.SIGALRM, stalled) + try: + while sent < size and (data := os.read(fd, min(PIECE, size - sent))): + sha.update(data) + sent += len(data) + view = memoryview(data) + while view: + signal.alarm(stall) # each write that moves anything starts the wait again + view = view[os.write(out, view):] + except Stalled: + raise Refused(f'the desktop took nothing for {stall} seconds: stopped', 1) from None + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, old) return sha.hexdigest(), sent diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 6fb4307c5a897..d6fcba4cd80e8 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -27,7 +27,7 @@ class Base(unittest.TestCase): def setUp(self): self.t = os.path.realpath(tempfile.mkdtemp(prefix='seed-test-')) self.addCleanup(shutil.rmtree, self.t, True) - for d in ('files', 'run', 'src'): + for d in ('files', f'files/{POOL}', 'run', 'src'): os.mkdir(os.path.join(self.t, d), 0o755) self.config(SEEDS) @@ -38,8 +38,8 @@ def config(self, seeds): json.dump(cfg, f) def make(self, name, members, mtime=None): - """FILES/NAME.tar of `members` ({path: bytes}), made as a project would: a temp file, then a rename.""" - tmp = os.path.join(self.t, 'files', f'.{name}.tmp') + """FILES/POOL/NAME.tar of `members` ({path: bytes}), made as a project would: a temp file, then a rename.""" + tmp = os.path.join(self.t, 'files', POOL, f'.{name}.tmp') with tarfile.open(tmp, 'w', format=tarfile.PAX_FORMAT) as tar: for path, data in members.items(): info = tarfile.TarInfo(path) @@ -48,10 +48,13 @@ def make(self, name, members, mtime=None): if mtime is not None: os.utime(tmp, (mtime, mtime)) os.chmod(tmp, 0o644) - os.replace(tmp, os.path.join(self.t, 'files', f'{name}.tar')) - with open(os.path.join(self.t, 'files', f'{name}.tar'), 'rb') as f: + os.replace(tmp, self.tar(name)) + with open(self.tar(name), 'rb') as f: return f.read() + def tar(self, name): + return os.path.join(self.t, 'files', POOL, f'{name}.tar') + def offer(self, command, **kw): r, w = os.pipe() err = io.StringIO() @@ -89,7 +92,7 @@ def test_the_list_says_what_is_offered_and_why_not(self): got = json.loads(out) self.assertEqual(got['v'], 1) m2, npm = got['seeds'] - st = os.stat(os.path.join(self.t, 'files', 'm2.tar')) + st = os.stat(self.tar('m2')) self.assertEqual(m2, {'name': 'm2', 'everyHours': 6, 'maxAgeHours': 1000, 'maxMb': 2, 'bytes': st.st_size, 'made': int(st.st_mtime)}) self.assertNotIn('made', npm) @@ -100,7 +103,7 @@ def test_a_seed_streams_whole_with_its_trailer(self): code, out, err = self.offer('seed m2') self.assertEqual(code, 0, err) self.assertEqual(out, data) - st = os.stat(os.path.join(self.t, 'files', 'm2.tar')) + st = os.stat(self.tar('m2')) self.assertEqual(err, f'seed m2 {len(data)} {hashlib.sha256(data).hexdigest()} {int(st.st_mtime)}\n') def test_other_commands_and_names_are_refused_and_send_nothing(self): @@ -112,13 +115,15 @@ def test_other_commands_and_names_are_refused_and_send_nothing(self): self.assertEqual((code, out), (2, b''), err) def test_a_file_that_may_not_be_sent_is_exit_4(self): - files = os.path.join(self.t, 'files') + top, files = os.path.join(self.t, 'files'), os.path.join(self.t, 'files', POOL) now = time.time() cases = {'missing': lambda: None, 'a link': lambda: os.symlink(os.path.join(self.t, 'src', 'x.tar'), f'{files}/m2.tar'), 'group-writable': lambda: (self.make('m2', {'a': b'x'}), os.chmod(f'{files}/m2.tar', 0o664)), 'too big': lambda: self.make('m2', {'a': os.urandom(3 << 20)}), 'too old': lambda: self.make('m2', {'a': b'x'}, mtime=now - 1001 * 3600), + 'dated in the future': lambda: self.make('m2', {'a': b'x'}, mtime=now + 10 * 365 * 86400), + 'the files folder others write': lambda: (self.make('m2', {'a': b'x'}), os.chmod(top, 0o777)), 'a folder': lambda: os.mkdir(f'{files}/m2.tar'), 'folder others write': lambda: (self.make('m2', {'a': b'x'}), os.chmod(files, 0o777))} with open(os.path.join(self.t, 'src', 'x.tar'), 'wb') as f: @@ -130,6 +135,7 @@ def test_a_file_that_may_not_be_sent_is_exit_4(self): code, out, err = self.offer('seed m2') self.assertEqual((code, out), (4, b''), err) os.chmod(files, 0o755) + os.chmod(top, 0o755) p = f'{files}/m2.tar' if os.path.isdir(p) and not os.path.islink(p): os.rmdir(p) @@ -145,17 +151,69 @@ def test_a_second_stream_for_the_pool_is_exit_3(self): def test_a_file_changed_while_sent_is_exit_4(self): self.make('m2', {'a': b'x' * 3000}) - path, send = os.path.join(self.t, 'files', 'm2.tar'), seed.send + path, send = self.tar('m2'), seed.send - def changing(fd, out): + def changing(fd, out, size): with open(path, 'ab') as f: # made in place, not by rename f.write(b'\0' * 512) - return send(fd, out) + return send(fd, out, size) with mock.patch.object(seed, 'send', changing): + code, out, err = self.offer('seed m2') + self.assertEqual(code, 4) + self.assertIn('changed while it was sent', err) + self.assertEqual(len(out), os.stat(path).st_size - 512) # cut at the size that was checked + + def test_a_same_size_rewrite_with_its_old_date_put_back_is_exit_4(self): + self.make('m2', {'a': b'x' * 3000}) + path, send = self.tar('m2'), seed.send + + def rewriting(fd, out, size): + st = os.stat(path) + with open(path, 'r+b') as f: # in place, same size, the old mtime put back: only ctime tells + f.write(b'y' * 512) + os.utime(path, ns=(st.st_atime_ns, st.st_mtime_ns)) + return send(fd, out, size) + with mock.patch.object(seed, 'send', rewriting): code, _, err = self.offer('seed m2') self.assertEqual(code, 4) self.assertIn('changed while it was sent', err) + def test_a_desktop_that_stops_reading_lets_the_lock_go(self): + self.make('m2', {'a': os.urandom(1 << 20)}) + r, w = os.pipe() # nobody reads r: the pipe fills and the write waits + err = io.StringIO() + try: + with mock.patch.object(seed, 'STALL', 1): + t = time.monotonic() + code = seed.offer(POOL, 'seed m2', os.path.join(self.t, 'run'), out=w, err=err, + config=os.path.join(self.t, 'fork-fleet.json'), files=os.path.join(self.t, 'files')) + self.assertLess(time.monotonic() - t, 30) + finally: + os.close(r) + os.close(w) + self.assertEqual(code, 1) + self.assertIn('took nothing for 1 seconds', err.getvalue()) + self.assertEqual(self.offer('seed m2')[0], 0) # the pool's lock was let go + + def test_pools_never_share_a_seed_file(self): + cfg = json.loads(json.dumps(CONFIG)) + cfg['pools'][0]['seeds'] = SEEDS + cfg['pools'].append({**cfg['pools'][0], 'name': 'other-pool'}) + with open(os.path.join(self.t, 'fork-fleet.json'), 'w', encoding='utf-8') as f: + json.dump(cfg, f) + self.make('m2', {'a': b'x'}) + os.mkdir(os.path.join(self.t, 'files', 'other-pool'), 0o755) + r, w = os.pipe() + err = io.StringIO() + try: + code = seed.offer('other-pool', 'seed m2', os.path.join(self.t, 'run'), out=w, err=err, + config=os.path.join(self.t, 'fork-fleet.json'), files=os.path.join(self.t, 'files')) + finally: + os.close(w) + os.close(r) + self.assertEqual(code, 4) + self.assertIn('not made yet', err.getvalue()) + def test_never_as_root_and_a_bad_config_is_exit_2(self): with mock.patch.object(seed.os, 'geteuid', return_value=0): self.assertEqual(self.offer('seeds')[:2], (2, b'')) @@ -180,6 +238,14 @@ def test_the_forced_command_diverts_seed_requests(self): stdin=subprocess.DEVNULL) self.assertEqual((got.returncode, got.stdout), (2, '')) self.assertFalse(os.path.exists(os.path.join(self.t, 'run2', f'{POOL}.sock'))) # no session was served + for command in ('seeds x', 'seed', 'seedsx', 'seeds '): + with self.subTest(command): + got = subprocess.run([sys.executable, '-I', '-B', desk, 'serve', POOL], capture_output=True, + text=True, env={**base, 'SSH_ORIGINAL_COMMAND': command}, timeout=30, + stdin=subprocess.DEVNULL) + self.assertEqual((got.returncode, got.stdout), (2, ''), got.stderr) + self.assertIn('fork-fleet-desk seed: refused', got.stderr) + self.assertFalse(os.path.exists(os.path.join(self.t, 'run2', f'{POOL}.sock'))) if __name__ == '__main__': diff --git a/.github/scripts/fork-fleet-desk.py b/.github/scripts/fork-fleet-desk.py index eb08f20058a57..fd8ff12cb13d9 100755 --- a/.github/scripts/fork-fleet-desk.py +++ b/.github/scripts/fork-fleet-desk.py @@ -17,8 +17,8 @@ POOL]. Sends one launcher verb (`start` with its stdin) through the live session and prints the desktop's stdout and stderr and exits with its exit code. No live session: exit 3 at once, nothing sent. No answer in the verb's time (fork-fleet-desk-wire.py, "Time"): exit 75, the verb may or may not have run. - The same forced command asked `seeds` or `seed NAME` (SSH_ORIGINAL_COMMAND): a desktop's warm caches, not a - session (fork-fleet-desk-seed.py; its lock DIR/POOL.seed.lock). + The same forced command asked anything that starts with `seed` (SSH_ORIGINAL_COMMAND; `seeds`, `seed NAME`): a + desktop's warm caches, never a session (fork-fleet-desk-seed.py; its lock DIR/POOL.seed.lock). DIR is /run/fork-fleet-desk, made at boot by systemd-tmpfiles (runbook): owned by the desk-link user, its group the socket group (who may connect: the dispatcher's user, or root), mode 0710 or 0750. The socket is 0660 in that group, and serve lets in only root, its own user and members of that group (SO_PEERCRED). Not as root (tests), @@ -409,9 +409,8 @@ def seed_side(pool, asked, env): def main(argv=None, env=os.environ): argv = sys.argv[1:] if argv is None else argv try: - if len(argv) == 2 and argv[0] == 'serve' and (env.get('SSH_ORIGINAL_COMMAND') == 'seeds' or - env.get('SSH_ORIGINAL_COMMAND', '').startswith('seed ')): - return seed_side(argv[1], env['SSH_ORIGINAL_COMMAND'], env) + if len(argv) == 2 and argv[0] == 'serve' and env.get('SSH_ORIGINAL_COMMAND', '').startswith('seed'): + return seed_side(argv[1], env['SSH_ORIGINAL_COMMAND'], env) # every seed* command; it refuses the rest if len(argv) == 2 and argv[0] == 'serve': for sig in (signal.SIGTERM, signal.SIGHUP): signal.signal(sig, lambda *_: sys.exit(1)) # through serve's finally: the socket goes, clients learn From bfe2987dd50d1f627194b08275021e71a51e548e Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:50:22 +0000 Subject: [PATCH 10/33] fork fleet: desktop seed fetch fix round 1 (D11b) - the desktop takes only the names its own installed fleet config lists for its pool, and every_hours, max_age_hours and max_mb from it (max_mb: the smaller of its own and the host's); the host can no longer raise them - ssh's stderr keeps only its last 4 KB; the host's list is read up to 1 MB - after unpacking, a walk refuses a link that leads out of the seed and any entry that is not a folder, file or link, and makes everything readable by jobs (0600 cache files) - an odd trailer (a MADE int() cannot read) is a torn copy, never a crash - members' sizes and count limited (sparse members, many tiny ones) - a leftover .NAME.link is removed at the start of each run - seeds come from the host's per-pool folder (D11); SKEW names the desktop's clock tolerance - the unit's sandbox lines are marked safety-critical Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 19 +- .github/scripts/fork-fleet-desk-seed.py | 184 ++++++++++++++----- .github/scripts/fork-fleet-desk-seed.test.py | 112 ++++++++++- .github/scripts/fork-fleet-seed.service | 2 + 4 files changed, 255 insertions(+), 62 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index 68e82be6663db..77e778db8d2bc 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -715,11 +715,20 @@ size and SHA-256; anything else is refused. One stream per pool at a time; a desktop that takes nothing for two minutes is cut off and the next one may go. The desktop's side is the same script's `run`, hourly from -`t3code-fleet-seed.timer` (the desk kit installs both). Each new seed is -unpacked with Python's tar data filter, checked against the trailer, and -swapped in by a rename; jobs see `/var/lib/fork-fleet/seeds/NAME` read-only -(the launcher's `ALLOW_PATHS`), copy what they need from it when it is there, -and run cold when it is not. The state table (a torn download, the host down, +`t3code-fleet-seed.timer` (the desk kit installs both). It takes only the +names its own installed fleet config lists for its pool, and holds them to +its own `every_hours` and `max_age_hours` and the smaller of its own and the +host's `max_mb`: a host cannot make a desktop take more. Each new seed is +unpacked with Python's tar data filter (members' sizes and count limited +too), checked against the trailer, checked again for links that lead out of +it, made readable by every job (a cache's 0600 files too), and swapped in by a +rename; ssh's stderr and the list are read with a limit. Jobs see +`/var/lib/fork-fleet/seeds/NAME` read-only (the launcher's `ALLOW_PATHS`; +the unpacking and old copies beside it, `.new` and `.v`, are visible too and +hold the same data), copy what they need from it when it is there, and run +cold when it is not. A replaced copy is kept 24 hours for a job still reading +it; there is no free-space check before unpacking (a full disk ends that seed's +fetch and keeps the old one). The state table (a torn download, the host down, clocks that disagree, a run that stopped half-way) is in the script's header. ## Owner steps on the repository diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index f4b2a4327a266..15ad764cc0ccf 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -15,8 +15,10 @@ stderr line is the trailer `seed NAME BYTES SHA256 MADE` (MADE: the file's mtime, epoch seconds) python3 -I fork-fleet-desk-seed.py [--config FILE] run the desktop, as root, hourly (t3code-fleet-seed.timer), with the agent's settings (FILE, default - its CONF). Each new seed is unpacked with tarfile's data filter into a folder of its own, checked - against the trailer, flushed, and swapped in by rename: SEEDS/NAME is a link to + its CONF). Only names its own installed fleet config lists for its POOL, held to its own + every_hours and max_age_hours and the smaller of its own and the host's max_mb. Each new seed is + unpacked with tarfile's data filter into a folder of its own, checked against the trailer, walked + (a link out of it is refused; all made readable), flushed, and swapped in by rename: SEEDS/NAME is a link to .v/NAME.STAMP/data, root's (folders 0755, files no one else may write), unpacked in .new (one folder: a rename never crosses a mount of the unit's sandbox). Jobs see SEEDS read-only (the launcher's ALLOW_PATHS): a job uses /var/lib/fork-fleet/seeds/NAME when it is there and runs @@ -42,7 +44,7 @@ Standard library only. """ import argparse, fcntl, hashlib, importlib.util, json, os, re, shutil, signal, stat, subprocess, sys, tarfile -import tempfile, threading, time +import threading, time sys.dont_write_bytecode = True HERE = os.path.dirname(os.path.realpath(__file__)) @@ -61,6 +63,7 @@ def module(name, file): NAME = re.compile(r'[a-z0-9][a-z0-9_-]{0,39}') LINK = re.compile(r'\.v/((?P[a-z0-9][a-z0-9_-]{0,39})\.[0-9]{1,20})/data') LIST_WAIT, FETCH_WAIT = 30, 30 * 60 # seconds for the list, and for one seed's whole stream +LIST_MOST, ERR_KEEP, MEMBERS = 1 << 20, 4096, 200_000 # the list's bytes, ssh's stderr kept, a seed's tar members KEEP_OLD, SKEW = 24 * 3600, 3600 # SKEW: how far the host's clock may be ahead of the desktop's PIECE, STALL, AHEAD = 1 << 20, 120, 300 # AHEAD: seconds a file's date may be ahead of now (a clock set back) ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'} @@ -214,20 +217,52 @@ def ssh(agent, conf, *words): return agent.ssh_argv(conf)[:-2] + list(words) # the agent's ssh, another command than `serve POOL` +class Tail(threading.Thread): + """Reads a pipe to its end and keeps only its last ERR_KEEP bytes (ssh's stderr: the host may write without + end; the trailer is its last line).""" + + def __init__(self, f): + super().__init__(daemon=True) + self.f, self.data = f, b'' + self.start() + + def run(self): + while chunk := os.read(self.f.fileno(), 65536): + self.data = (self.data + chunk)[-ERR_KEEP:] + + def text(self): + self.join(5) + return ' '.join(self.data.decode('utf-8', 'replace').split())[-300:] or 'nothing on stderr' + + def listing(agent, conf): - """{name: item} of the host's list; Torn when it cannot be had or is not well formed.""" - with tempfile.TemporaryFile() as err: - try: - r = subprocess.run(ssh(agent, conf, 'seeds'), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=err, - env=agent.ENV, timeout=LIST_WAIT, start_new_session=True) - except subprocess.TimeoutExpired: - raise Torn(f'no list in {LIST_WAIT} s') from None - except OSError as e: - raise Torn(f'no ssh: {e}') from None - if r.returncode: - raise Torn(f'no list (exit {r.returncode}): {tail(err)}') + """{name: item} of the host's list (at most LIST_MOST bytes); Torn when it cannot be had or is not well formed.""" + try: + p = subprocess.Popen(ssh(agent, conf, 'seeds'), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, env=agent.ENV, start_new_session=True) + except OSError as e: + raise Torn(f'no ssh: {e}') from None + late = threading.Event() + timer = threading.Timer(LIST_WAIT, lambda: (late.set(), p.kill())) + timer.start() + out = b'' try: - got = json.loads(r.stdout) + err = Tail(p.stderr) + while len(out) <= LIST_MOST and (chunk := os.read(p.stdout.fileno(), 65536)): + out += chunk + finally: + timer.cancel() + p.kill() + code = p.wait() + p.stdout.close() + if late.is_set(): + raise Torn(f'no list in {LIST_WAIT} s') + if len(out) > LIST_MOST: + raise Torn(f'the host\'s list is above {LIST_MOST} bytes') + if code: + raise Torn(f'no list (exit {code}): {err.text()}') + try: + got = json.loads(out, parse_constant=no_constant) assert isinstance(got, dict) and got.get('v') == 1 and isinstance(got.get('seeds'), list) out = {} for s in got['seeds']: @@ -240,43 +275,78 @@ def listing(agent, conf): raise Torn('the host\'s list is not well formed') from None +def no_constant(name): + raise ValueError(f'{name} is not JSON') + + def whole(x): return isinstance(x, int) and not isinstance(x, bool) and x >= 0 -def tail(f): - f.seek(0) - return ' '.join(f.read()[-2000:].decode('utf-8', 'replace').split())[-300:] or 'nothing on stderr' +def guarded(most): + """tarfile's data filter, and a limit on what the tar says it holds: members' sizes together at most MOST + bytes (a sparse member is small in the stream, not on disk) and at most MEMBERS members (each is kept in memory).""" + seen = [0, 0] + + def check(member, path): + seen[0] += 1 + seen[1] += member.size + if seen[0] > MEMBERS or seen[1] > most: + raise Torn(f'the tar holds more than {MEMBERS} members or {most >> 20} MB (max_mb)') + return tarfile.data_filter(member, path) + return check + + +def settle(data): + """After unpacking: every entry a folder, a plain file or a link that stays inside DATA (a check of its own, + besides the data filter, which some Python releases let links slip past); folders and files readable by all + (jobs run as other users; a cache may hold 0600 files), writable by root only. Torn otherwise.""" + top = os.path.realpath(data) + for d, dirs, names in os.walk(data): + for n in dirs + names: + path = os.path.join(d, n) + st = os.lstat(path) + if stat.S_ISLNK(st.st_mode): + if os.path.commonpath([top, os.path.realpath(path)]) != top: + raise Torn(f'{os.path.relpath(path, data)} is a link that leads out of the seed') + elif stat.S_ISDIR(st.st_mode): + os.chmod(path, (st.st_mode & 0o755) | 0o555) + elif stat.S_ISREG(st.st_mode): + os.chmod(path, (st.st_mode & 0o755) | 0o444) + else: + raise Torn(f'{os.path.relpath(path, data)} is not a folder, a file or a link') def fetch(agent, conf, item, root, now): - """The seed `item` into a new version, swapped in; its label. Torn (or OSError, TarError) leaves the old one.""" - name, stamp = item['name'], time.time_ns() + """The seed `item` (its maxMb: the smaller of the host's and the desktop's own) into a new version, swapped in; + its label. Torn (or OSError, TarError) leaves the old one.""" + name, stamp, most = item['name'], time.time_ns(), item['maxMb'] << 20 new = os.path.join(root + '/.new', f'{name}.{stamp}') os.mkdir(new, 0o755) os.mkdir(os.path.join(new, 'data'), 0o755) - with tempfile.TemporaryFile() as err: - p = subprocess.Popen(ssh(agent, conf, 'seed', name), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, - stderr=err, env=agent.ENV, start_new_session=True) - timer = threading.Timer(FETCH_WAIT, p.kill) - timer.start() - try: - got = Summed(p.stdout, item['maxMb'] << 20) - with tarfile.open(fileobj=got, mode='r|') as t: - t.extractall(os.path.join(new, 'data'), filter='data') - while got.read(PIECE): # the tar's end blocks and padding count in the trailer's size and sha256 - pass - code = p.wait(timeout=30) - finally: - timer.cancel() - p.kill() - p.wait() - p.stdout.close() - err.seek(0) - last = (err.read()[-4096:].decode('utf-8', 'replace').strip().splitlines() or [''])[-1].split() - if code or last[:4] != ['seed', name, str(got.size), got.sha.hexdigest()] or len(last) != 5 or \ - not last[4].isdigit(): - raise Torn(f'exit {code}, {got.size} bytes, the trailer does not match: {tail(err)}') + p = subprocess.Popen(ssh(agent, conf, 'seed', name), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, env=agent.ENV, start_new_session=True) + timer = threading.Timer(FETCH_WAIT, p.kill) + timer.start() + try: + err = Tail(p.stderr) + got = Summed(p.stdout, most) + with tarfile.open(fileobj=got, mode='r|') as t: + t.extractall(os.path.join(new, 'data'), filter=guarded(most)) + while got.read(PIECE): # the tar's end blocks and padding count in the trailer's size and sha256 + pass + code = p.wait(timeout=30) + finally: + timer.cancel() + p.kill() + p.wait() + p.stdout.close() + err.join(5) + last = (err.data.decode('utf-8', 'replace').strip().splitlines() or [''])[-1].split() + if code or len(last) != 5 or last[:4] != ['seed', name, str(got.size), got.sha.hexdigest()] or \ + not re.fullmatch(r'[0-9]{1,20}', last[4]): + raise Torn(f'exit {code}, {got.size} bytes, the trailer does not match: {err.text()}') + settle(os.path.join(new, 'data')) label = {'v': 1, 'name': name, 'bytes': got.size, 'sha256': got.sha.hexdigest(), 'made': int(last[4]), 'fetched': int(now()), 'maxAgeHours': item['maxAgeHours']} with open(os.path.join(new, 'label.json'), 'x', encoding='utf-8') as f: @@ -330,9 +400,22 @@ def drop(root, name, version, now): remove(os.path.join(root, name)) -def run(agent, conf, root=SEEDS, now=time.time, say=print): - if not hasattr(tarfile, 'data_filter'): +def own_seeds(conf, config): + """{name: seed} of this desktop's pool in its own installed fleet config: the only names it takes, and the + every_hours, max_age_hours and max_mb it holds them to (the host's list never raises them).""" + try: + return pool_seeds(conf['POOL'], config) + except Refused as e: + raise Torn(f'its own {e}') from None + + +def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): + if not hasattr(tarfile, 'data_filter'): # releases with known filter bypasses pass this: settle() checks again raise agent.Bad('this Python has no tarfile data filter (3.12, or 3.8.17 and later): no seeds') + try: + own = own_seeds(conf, config) + except Torn as e: + raise agent.Bad(f'{e}: no seeds') from None os.umask(0o022) for d in (root, root + '/.new', root + '/.v'): # root: made by the unit (StateDirectory) or here os.makedirs(d, 0o755, exist_ok=True) @@ -345,8 +428,13 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print): for e in os.listdir(root + '/.new'): # a run that stopped half-way if e != '.lock': remove(os.path.join(root + '/.new', e)) + for e in os.listdir(root): # a link made but not yet swapped in + if e.startswith('.') and e.endswith('.link'): + remove(os.path.join(root, e)) try: - items = listing(agent, conf) + host = listing(agent, conf) + items = {n: {**host[n], 'everyHours': s['every_hours'], 'maxAgeHours': s['max_age_hours'], + 'maxMb': min(host[n]['maxMb'], s['max_mb'])} for n, s in own.items() if n in host} except Torn as e: items = None say(f'seeds: {e}: each seed stays until its age drops it') @@ -354,11 +442,11 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print): version, label = current(root, name) if label is None: why = 'its label is missing or cannot be read (fetched again now)' - elif items is not None and name not in items: + elif name not in own: why = 'no longer one of the pool\'s seeds' else: age = now() - label['made'] - hours = items[name]['maxAgeHours'] if items else label['maxAgeHours'] + hours = own[name]['max_age_hours'] why = (f'older than {hours} h (max_age_hours)' if age > hours * 3600 else 'made in the future (the clocks disagree)' if age < -SKEW else None) if why: @@ -380,7 +468,7 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print): try: got = fetch(agent, conf, item, root, now) say(f'seed {name}: {got["bytes"] >> 20} MB, sha256 checked, swapped in') - except (Torn, OSError, tarfile.TarError, subprocess.SubprocessError) as e: + except (Torn, OSError, ValueError, tarfile.TarError, subprocess.SubprocessError) as e: for e2 in os.listdir(root + '/.new'): if e2 != '.lock': remove(os.path.join(root + '/.new', e2)) diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 6ab15a8031804..5b05dbf3bf216 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -44,7 +44,12 @@ def load(name, file): print('ssh: connect to host h port 22: No route to host', file=sys.stderr) sys.exit(255) kw = dict(config={t!r} + '/fork-fleet.json', files={t!r} + '/files') -if words[0] == 'seed' and mode in ('cut', 'trailer'): +if words[0] == 'seeds' and mode == 'biglist': + os.write(1, b' ' * (2 << 20)) +if mode == 'noisy': + for _ in range(64): + os.write(2, b'x' * (1 << 16) + b'\n') +if words[0] == 'seed' and mode in ('cut', 'trailer', 'odd'): with open({t!r} + '/sent', 'w+b') as out: err = io.StringIO() code = seed.offer({pool!r}, ' '.join(words), {t!r} + '/run', out=out.fileno(), err=err, **kw) @@ -54,7 +59,10 @@ def load(name, file): os.write(1, data[:len(data) // 2]) sys.exit(255) os.write(1, data) - print(err.getvalue().rsplit(' ', 2)[0] + ' ' + '0' * 64 + ' 1', file=sys.stderr) # a wrong sha256 + if mode == 'odd': # a MADE that str.isdigit() takes but int() does not + print(err.getvalue().rsplit(' ', 1)[0] + ' \u00b2', file=sys.stderr) + else: + print(err.getvalue().rsplit(' ', 2)[0] + ' ' + '0' * 64 + ' 1', file=sys.stderr) # a wrong sha256 sys.exit(code) sys.exit(seed.offer({pool!r}, ' '.join(words), {t!r} + '/run', **kw)) ''' @@ -74,13 +82,13 @@ def config(self, seeds): with open(os.path.join(self.t, 'fork-fleet.json'), 'w', encoding='utf-8') as f: json.dump(cfg, f) - def make(self, name, members, mtime=None): + def make(self, name, members, mtime=None, mode=0o644): """FILES/POOL/NAME.tar of `members` ({path: bytes}), made as a project would: a temp file, then a rename.""" tmp = os.path.join(self.t, 'files', POOL, f'.{name}.tmp') with tarfile.open(tmp, 'w', format=tarfile.PAX_FORMAT) as tar: for path, data in members.items(): info = tarfile.TarInfo(path) - info.size, info.mode = len(data), 0o644 + info.size, info.mode = len(data), mode tar.addfile(info, io.BytesIO(data)) if mtime is not None: os.utime(tmp, (mtime, mtime)) @@ -296,13 +304,22 @@ def setUp(self): self.conf = {'SSH': ssh, 'KEY': '/nonexistent/key', 'KNOWN_HOSTS': '/nonexistent/known', 'PORT': '22', 'USER': 'fleet-desk', 'HOST': 'fleet-host.example', 'POOL': POOL} self.said = [] + self.own(SEEDS) sync = mock.patch.object(seed.os, 'sync') sync.start() self.addCleanup(sync.stop) + def own(self, seeds): + """The desktop's own installed fleet config (/opt/fork-fleet/fork-fleet.json); self.config is the host's.""" + cfg = json.loads(json.dumps(CONFIG)) + cfg['pools'][0]['seeds'] = seeds + with open(os.path.join(self.t, 'own.json'), 'w', encoding='utf-8') as f: + json.dump(cfg, f) + def run_seed(self, now=None): self.said = [] - return seed.run(agent, self.conf, self.root, now=now or time.time, say=self.said.append) + return seed.run(agent, self.conf, self.root, now=now or time.time, say=self.said.append, + config=os.path.join(self.t, 'own.json')) def mode(self, text): with open(os.path.join(self.t, 'mode'), 'w') as f: @@ -340,7 +357,7 @@ def test_fetched_checked_and_swapped_in_then_not_again(self): if not stat.S_ISLNK(st.st_mode): # a link's own mode is always 0777 and means nothing self.assertFalse(st.st_mode & 0o022, os.path.join(d, n)) label = json.load(open(os.path.join(seed.current(self.root, 'm2')[0], 'label.json'))) - self.assertEqual(label['bytes'], os.path.getsize(os.path.join(self.t, 'files', 'm2.tar'))) + self.assertEqual(label['bytes'], os.path.getsize(self.tar('m2'))) self.assertEqual(self.run_seed(), 0) # the same seed on the host: nothing fetched self.assertEqual(self.fetches(), ['seed m2']) self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) @@ -394,6 +411,78 @@ def bigger(): self.assertEqual(self.files('m2'), {'a': b'one'}) self.assertTrue(any('max_mb' in s for s in self.said), self.said) + def test_max_mb_is_the_desktops_not_the_hosts(self): + self.config([{**SEEDS[0], 'max_mb': 3}, SEEDS[1]]) # the host lists 3; the desktop's own copy says 2 + self.make('m2', {'a': os.urandom(5 << 19)}) # 2.5 MB + self.run_seed() + self.assertIsNone(self.files('m2'), self.said) + self.assertTrue(any('max_mb' in s for s in self.said), self.said) + + def test_every_and_max_age_hours_are_the_desktops_and_unknown_names_are_ignored(self): + self.config([{**SEEDS[0], 'every_hours': 1, 'max_age_hours': 100000}, SEEDS[1], + {'name': 'extra', 'every_hours': 1, 'max_age_hours': 2, 'max_mb': 1}]) + self.make('m2', {'a': b'one'}) + self.make('extra', {'a': b'x'}) + self.run_seed() + self.assertIsNone(self.files('extra')) # not in the desktop's own list: never fetched + self.assertNotIn('seed extra', self.fetches()) + self.make('m2', {'a': b'two'}, mtime=time.time() + 5) + self.run_seed(now=lambda: time.time() + 2 * 3600) # the host says every 1 h; the desktop's own, 6 h + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.mode('down') + self.run_seed(now=lambda: time.time() + 1001 * 3600) # the desktop's own max_age_hours (1000), not 100000 + self.assertIsNone(self.files('m2')) + + def test_stderr_is_never_kept_whole_and_a_huge_list_is_refused(self): + self.mode('noisy') # 4 MB on stderr before the trailer + self.make('m2', {'a': b'one'}) + with mock.patch('tempfile.TemporaryFile', side_effect=AssertionError('stderr kept on disk')): + self.assertEqual(self.run_seed(), 0) + self.assertEqual(self.files('m2'), {'a': b'one'}, self.said) + self.mode('biglist') + self.make('m2', {'a': b'two'}, mtime=time.time() + 5) + self.run_seed(now=lambda: time.time() + 7 * 3600) + self.assertTrue(any('above' in s and 'bytes' in s for s in self.said), self.said) + self.assertEqual(self.files('m2'), {'a': b'one'}) + + def test_a_0600_file_is_readable_by_jobs(self): + self.make('m2', {'cache/entry': b'x', 'cache/tool': b'#!/bin/sh\n'}, mode=0o600) + self.run_seed() + top = os.path.realpath(os.path.join(self.root, 'm2')) + for d, dirs, names in os.walk(top): + for n in dirs + names: + mode = os.lstat(os.path.join(d, n)).st_mode + self.assertEqual(mode & 0o444, 0o444, (n, oct(mode))) + self.assertFalse(mode & 0o022, (n, oct(mode))) + if stat.S_ISDIR(mode): + self.assertEqual(mode & 0o111, 0o111, (n, oct(mode))) + + def test_a_link_out_of_the_seed_is_refused_even_past_the_filter(self): + self.make('m2', {'a': b'one'}) + self.run_seed() + tmp = os.path.join(self.t, 'files', POOL, '.m2.tmp') + with tarfile.open(tmp, 'w') as tar: + for name, target in (('inside', 'a'), ('escaped', '/')): + info = tarfile.TarInfo(name) + info.type, info.linkname = tarfile.SYMTYPE, target + tar.addfile(info) + os.utime(tmp, (time.time() + 5,) * 2) + os.replace(tmp, self.tar('m2')) + with mock.patch.object(seed.tarfile, 'data_filter', lambda member, path: member): # a filter bypass + self.run_seed(now=lambda: time.time() + 7 * 3600) + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertTrue(any('leads out of the seed' in s for s in self.said), self.said) + for d, dirs, names in os.walk(self.root): + self.assertNotIn('escaped', dirs + names, d) + + def test_an_odd_trailer_keeps_the_old_seed_and_the_run_goes_on(self): + self.torn(lambda: (self.make('m2', {'a': b'two'}, mtime=time.time() + 5), self.mode('odd')), + 'trailer does not match') + + def test_too_many_members_keep_the_old_seed(self): + with mock.patch.object(seed, 'MEMBERS', 2): + self.torn(lambda: self.make('m2', {'a': b'1', 'b': b'2', 'c': b'3'}, mtime=time.time() + 5), 'members') + def test_a_tar_that_would_leave_its_folder_writes_nothing_outside(self): bad = {'a ../ path': ('../../escaped', tarfile.REGTYPE, ''), 'a device': ('escaped', tarfile.CHRTYPE, ''), 'an absolute link': ('escaped', tarfile.SYMTYPE, '/'), @@ -401,7 +490,7 @@ def test_a_tar_that_would_leave_its_folder_writes_nothing_outside(self): 'a hard link out': ('escaped', tarfile.LNKTYPE, '/etc/hostname')} for why, (name, kind, target) in bad.items(): def evil(): - tmp = os.path.join(self.t, 'files', '.m2.tmp') + tmp = os.path.join(self.t, 'files', POOL, '.m2.tmp') with tarfile.open(tmp, 'w') as tar: # one bad member each: the filter stops at the first ok = tarfile.TarInfo('ok') ok.size = 1 @@ -410,7 +499,7 @@ def evil(): info.type, info.linkname = kind, target tar.addfile(info) os.utime(tmp, (time.time() + 5,) * 2) - os.replace(tmp, os.path.join(self.t, 'files', 'm2.tar')) + os.replace(tmp, self.tar('m2')) with self.subTest(why): shutil.rmtree(self.root, True) self.torn(evil, 'Error') @@ -431,7 +520,10 @@ def test_dropped_when_no_longer_listed_or_too_old_or_from_the_future(self): self.assertTrue(os.path.isdir(old)) # dropped, but a job still reading it keeps it self.assertTrue(any('older than 1000 h' in s for s in self.said), self.said) self.run_seed() - self.config([SEEDS[1]]) + self.config([SEEDS[1]]) # the host no longer lists it: the desktop's own list still does, it stays + self.run_seed() + self.assertIsNotNone(self.files('m2')) + self.own([SEEDS[1]]) # the desktop's own list no longer does: dropped self.run_seed() self.assertIsNone(self.files('m2')) self.assertTrue(any('no longer one of the pool' in s for s in self.said), self.said) @@ -451,11 +543,13 @@ def test_a_run_that_stopped_half_way_and_bad_links_or_labels(self): self.run_seed() os.makedirs(os.path.join(self.root, '.new', 'm2.123', 'data')) # a torn unpack os.symlink('.v/m2.999/data', os.path.join(self.root, '.m2.link')) # a link made, not yet swapped in + os.symlink('.v/npm.999/data', os.path.join(self.root, '.npm.link')) # one of a seed not fetched now os.symlink('/etc', os.path.join(self.root, 'npm')) # not a link this program makes os.unlink(os.path.join(seed.current(self.root, 'm2')[0], 'label.json')) self.run_seed() self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) self.assertFalse(os.path.lexists(os.path.join(self.root, 'npm'))) + self.assertFalse(os.path.lexists(os.path.join(self.root, '.npm.link'))) self.assertTrue(os.path.isdir('/etc')) self.assertEqual(self.files('m2'), {'a': b'one'}) # fetched again in the same run self.assertEqual(len(self.fetches()), 2) diff --git a/.github/scripts/fork-fleet-seed.service b/.github/scripts/fork-fleet-seed.service index 901eee69a986b..e1380376cfaf2 100644 --- a/.github/scripts/fork-fleet-seed.service +++ b/.github/scripts/fork-fleet-seed.service @@ -6,6 +6,8 @@ # (StateDirectory makes it, 0755; new copies are unpacked inside it, so a rename never crosses a mount), reads the # desk link's settings and key in /etc/fork-fleet, reaches the network (ssh to the always-on host only, with the # agent's pinned host key) and has a memory, CPU and task limit. +# The sandbox lines below are safety-critical: they bound what a tar that slips past Python's data filter (some +# releases have known bypasses; the script also checks every link after unpacking) could reach. Do not loosen them. [Unit] Description=t3code fork fleet: warm caches for a desk pool's jobs Wants=network-online.target From 7e73543162676bff133c4265d817b4ab710ae641 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:20:14 +0000 Subject: [PATCH 11/33] fork kit desk: the uninstall test gives its namespace through the fake ip (D8) Uninstall now deletes the namespaces ip netns list shows, not one per recorded user; the older test listed none. Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/tests/install.test.py | 1 + 1 file changed, 1 insertion(+) diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index 3903cd1297170..d5069cfafc709 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -471,6 +471,7 @@ def test_uninstall_leaves_nothing_of_its_own(self): self.assertEqual(self.install()[0], 0) self.write(os.path.join(self.fakes, 'active'), 'desktop-1\n') self.write(self.path('/var/lib/fork-fleet/runners/x'), '') + self.write(os.path.join(self.fakes, 'netns'), 'fork-fleet-fork-runner-1\n') # the namespaces that are there self.clear_calls() code, out, err = self.run_main('uninstall') self.assertEqual(code, 0, err) From 99bf23565b0b3374358954f5da7d3083bbed30c3 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:22:30 +0000 Subject: [PATCH 12/33] fork kit desk: the record-last test lists folders by name (D8) A whole-folder removal took the record first only for some directory orders, so the test passed on the old code by luck. Listing by name puts the record first, which the old order fails every time. Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/tests/install.test.py | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index d5069cfafc709..2b68fdc57f035 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -539,16 +539,34 @@ def test_uninstall_removes_every_runner_namespace(self): def test_uninstall_removes_its_record_last(self): self.assertEqual(self.install()[0], 0) - gone, unlink = [], os.unlink + gone, unlink, scandir = [], os.unlink, os.scandir def record(path, *a, **kw): gone.append(os.path.basename(path)) return unlink(path, *a, **kw) - os.unlink = record + + class Sorted: + """Folders listed by name, so the record (desk-kit.json sorts before desk.conf and launch.conf) comes + first: a whole-folder removal would take it before the rest, whatever order the file system gives.""" + def __init__(self, *a, **kw): + self.it = scandir(*a, **kw) + + def __enter__(self): + return self + + def __exit__(self, *exc): + self.it.close() + + def __iter__(self): + return iter(sorted(self.it, key=lambda e: e.name)) + + def close(self): + self.it.close() + os.unlink, os.scandir = record, Sorted try: self.assertEqual(self.run_main('uninstall')[0], 0) finally: - os.unlink = unlink + os.unlink, os.scandir = unlink, scandir self.assertEqual(gone[-1], 'desk-kit.json', gone) def test_uninstall_refuses_a_record_of_another_shape(self): From a88492342618f206fc845bb8a09339aedcab4660 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:23:44 +0000 Subject: [PATCH 13/33] fork kit desk: the starter's comment no longer names the old encoding flag (D9) The text check that no ASCII encoding is left matched the comment that explains the bug. Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/windows/start-desk.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fork-kit/desk/windows/start-desk.ps1 b/fork-kit/desk/windows/start-desk.ps1 index 2d774f9e3f3ff..531b059c169f7 100644 --- a/fork-kit/desk/windows/start-desk.ps1 +++ b/fork-kit/desk/windows/start-desk.ps1 @@ -88,7 +88,7 @@ mode=$(wslinfo --networking-mode 2>/dev/null || true) [ -z "$mode" ] || [ "$mode" = nat ] || echo "restart: the networking mode is $mode, not nat" '@ # .wslconfig, read and written as UTF-8 without a BOM: Windows PowerShell 5.1's Get-Content reads a file without a -# BOM as ANSI and Set-Content -Encoding ascii turns every other letter into '?' (a kernel= path with an accent in it +# BOM as ANSI and Set-Content with the ASCII encoding turns every other letter into '?' (a kernel= path with an accent in it # then stops every distro). The tests run this part alone (between the two marker lines) under pwsh. # --- wslconfig --- function Read-WslConfig([string]$path) { From f706ea461c4acb39490ddfec0a63bb11b92e36c7 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:26:26 +0000 Subject: [PATCH 14/33] fork kit free: resolve the link once, then judge and run that file (D10) The round-1 fix resolved the link again after the check, so a link repointed between the two still changed what ran. The new test caught it. Co-Authored-By: Claude Opus 5.5 --- fork-kit/common/free_contract.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/fork-kit/common/free_contract.py b/fork-kit/common/free_contract.py index 1df6c09274543..e4e9ee18baaa4 100644 --- a/fork-kit/common/free_contract.py +++ b/fork-kit/common/free_contract.py @@ -77,7 +77,7 @@ def no(why): def unsafe(command, uid=None): """Why COMMAND must not run, or None: an absolute plain path to an executable file that it and every folder above it belong to root or to this user and that no one else may change (a sticky folder like /tmp is fine). - A link is judged by the file it leads to, and ask() runs that file, never the link (it could be repointed). + A link is judged by the file it leads to; ask() resolves the link once and both judges and runs that file. Plain mode bits only: an ACL that lets others write is not seen.""" uid = os.geteuid() if uid is None else uid if not os.path.isabs(command) or os.path.normpath(command) != command: @@ -119,10 +119,11 @@ def ask(command, need_mb=None, timeout=LIMIT_SECONDS): reason, facts). After it, its whole process group is ended, and so is any other process that still holds its stdout (a child that left the group with setsid). A child that leaves the group and closes stdout is not seen: FREE.md asks scripts to leave nothing running.""" - why = unsafe(command) + plain = os.path.isabs(command) and os.path.normpath(command) == command + path = os.path.realpath(command) if plain else command # resolved once: a link repointed later changes nothing + why = unsafe(path) if why: return 'unavailable', why, None - path = os.path.realpath(command) # what unsafe() judged; the link itself may be repointed after the check try: p = subprocess.Popen([path], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=ENV, cwd='/', start_new_session=True) From 099f4387de19348368a29818358a025ebeb07cf8 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:31:58 +0000 Subject: [PATCH 15/33] fork fleet: seed tests read the small stream and use a valid second pool (D11) The lock check after a stall sent 1 MB into a pipe nobody read until the call returned, and the second pool reused the first one's label and beacon, which the config check refuses. The pool test also puts a tar at the old shared path, so the old code fails it. Co-Authored-By: Claude Opus 5.5 --- .github/scripts/fork-fleet-desk-seed.test.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index d6fcba4cd80e8..c237f4173a99e 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -193,15 +193,19 @@ def test_a_desktop_that_stops_reading_lets_the_lock_go(self): os.close(w) self.assertEqual(code, 1) self.assertIn('took nothing for 1 seconds', err.getvalue()) + self.make('m2', {'a': b'x'}) # small enough for the pipe: offer() below has no reader until it returns self.assertEqual(self.offer('seed m2')[0], 0) # the pool's lock was let go def test_pools_never_share_a_seed_file(self): cfg = json.loads(json.dumps(CONFIG)) cfg['pools'][0]['seeds'] = SEEDS - cfg['pools'].append({**cfg['pools'][0], 'name': 'other-pool'}) + cfg['pools'].append({**cfg['pools'][0], 'name': 'other-pool', 'label': 'fleet-other-t3code', + 'beacon': 'FLEET_BEACON_OTHER'}) + self.assertEqual(ff.check_config(cfg), []) with open(os.path.join(self.t, 'fork-fleet.json'), 'w', encoding='utf-8') as f: json.dump(cfg, f) self.make('m2', {'a': b'x'}) + shutil.copy(self.tar('m2'), os.path.join(self.t, 'files', 'm2.tar')) # one file for all pools, as before os.mkdir(os.path.join(self.t, 'files', 'other-pool'), 0o755) r, w = os.pipe() err = io.StringIO() From 8b687649b3ec883bbe09b1fc1b10df48a618af87 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:35:19 +0000 Subject: [PATCH 16/33] fork fleet: seed tests: the noisy fake ssh compiles, the upgrade stops the seed units (D11b) The fake ssh template wrote a real newline inside a bytes literal, so every desktop run got a syntax error instead of a list. D8's upgrade test now names the stop call, which on this branch also stops the seed timer and service. Co-Authored-By: Claude Opus 5.5 --- .github/scripts/fork-fleet-desk-seed.test.py | 2 +- fork-kit/desk/tests/install.test.py | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index f6720da1d0cf9..a0e73e3598400 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -48,7 +48,7 @@ def load(name, file): os.write(1, b' ' * (2 << 20)) if mode == 'noisy': for _ in range(64): - os.write(2, b'x' * (1 << 16) + b'\n') + os.write(2, b'x' * (1 << 16) + b'\\n') if words[0] == 'seed' and mode in ('cut', 'trailer', 'odd'): with open({t!r} + '/sent', 'w+b') as out: err = io.StringIO() diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index 766954d627e1f..90109fcc9632b 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -298,7 +298,8 @@ def test_upgrade_replaces_changed_files_and_removes_gone_ones(self): self.assertIn('# new', self.read('/opt/fork-fleet/hook.sh')) self.assertEqual(os.stat(self.path('/opt/fork-fleet/fork-fleet-launch.sh')).st_ino, launcher) # unchanged calls = self.calls() - stop = 'systemctl stop t3code-fleet-refresh.timer t3code-fleet-refresh.service fork-fleet-desk-agent.service' + stop = ('systemctl stop t3code-fleet-refresh.timer t3code-fleet-refresh.service t3code-fleet-seed.timer ' + 't3code-fleet-seed.service fork-fleet-desk-agent.service') self.assertLess(calls.index(stop), calls.index('systemctl restart fork-fleet-desk-agent.service')) self.assertEqual(json.loads(self.read('/etc/fork-fleet/desk-kit.json'))['sha'], SHA2) From 4a13be544fe0133ad4f67be0579f4427054a4ee9 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:37:27 +0000 Subject: [PATCH 17/33] fork kit desk: fix round 2 (restart mark, unit check, deny lines, fleet config) (D8) - A restart-pending mark in the record, saved before the first change the agent reads (files, settings, key, units) and cleared after the restart: a run stopped in between restarts the agent on the next run; check names it. - check compares each installed unit with the header plus the release's /opt/fork-fleet/units/UNIT. - The record lists every deny line the kit added (also for runner users that were there before); uninstall removes them all. - The fleet config choice (and the project's copy in /etc/fork-fleet/fleet-config.json) is saved before the swap. - --fleet-config is refused in a folder others or a runner user may write. - check names a kit-made runner user that a smaller slots no longer uses. - The bootstrap test gives a good --sha/--repo before --fleet-config, so it fails on the round-0 bootstrap. Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 23 ++++-- fork-kit/desk/sbin/desk-install | 120 +++++++++++++++++++++++----- fork-kit/desk/tests/install.test.py | 91 ++++++++++++++++++++- 3 files changed, 205 insertions(+), 29 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 5725bc916eb0a..86a0fe756420e 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -40,20 +40,25 @@ sh bootstrap.sh --repo OWNER/REPO --sha RELEASE_SHA --release-fleet-config # b /opt/fork-fleet/desk-install uninstall ``` -`--fleet-config` is the project's fleet config (its pools); without it the release's own is used. The project's -is remembered, so a later install of a new release keeps it; `--release-fleet-config` goes back to the -release's own. Install prints the desk key's public half: add it on +`--fleet-config` is the project's fleet config (its pools); without it the release's own is used. It must be in a +folder only root (or you) can write. The project's is kept in `/etc/fork-fleet/fleet-config.json` and the choice is +recorded before any file is swapped, so a later install of a new release keeps it, even after a stopped run; +`--release-fleet-config` goes back to the release's own. Install prints the desk key's public half: add it on the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner template is made in the background by the refresh timer; `check` says FAIL for `template` until it is there. -`check` only reads: the installed files against their manifest, systemd drop-ins that change the kit's units, +`check` only reads: the installed files against their manifest, each unit against the release's (header plus +`/opt/fork-fleet/units/UNIT`), a restart a stopped install still owes, systemd drop-ins that change the kit's units, every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings -files against the machine file, the runner users and IPv4 forwarding. +files against the machine file, the runner users (and names a user the kit made that a smaller `slots` no longer +uses) and IPv4 forwarding. ## Runs again, stops half-way, old or empty state - **Again:** files, settings and units that did not change are not rewritten, and the agent is not stopped or - restarted (a restart would end a job it is starting). After a change it is restarted. + restarted (a restart would end a job it is starting). After a change it is restarted. The agent reads its settings + only when it starts, so before the first change the record gets a restart-pending mark, cleared after the + restart: a run stopped in between restarts the agent on the next run. - **Stopped half-way:** the next install ends it. A new release is staged in `/opt/fork-fleet/.kit-stage-*` and checked first (gate loads with the fleet config, the pool is in it, `bash -n`, the walls know this OS); a release that fails is never installed. Files are renamed in one by one while the agent and the refresh timer are stopped; @@ -61,14 +66,16 @@ files against the machine file, the runner users and IPv4 forwarding. last, so `check` reports a half-done swap. Users are recorded in `/etc/fork-fleet/desk-kit.json` before they are made, so uninstall removes them even after a stop. - **Old install:** a launcher, `launch.conf`, `desk.conf`, desk key, unit, or a systemd drop-in for one of the kit's - units (`UNIT.d/*.conf`, also the dash-prefix forms) that this kit did not write is refused before anything changes. Remove the older (by hand) install first, the runbook's steps in reverse. + units (`UNIT.d/*.conf`, also the dash-prefix forms) that this kit did not write is refused before anything changes. + The type-wide `service.d/` and `timer.d/` folders are not checked: they hold the distribution's defaults for every + unit (refusing them would refuse a stock system), and only root can write them. Remove the older (by hand) install first, the runbook's steps in reverse. - **Empty:** `check` before an install says FAIL and writes nothing; `uninstall` with no record does nothing. Uninstall stops every runner (through the launcher, then `systemctl stop` for any left). While one still runs it stops there and changes nothing else: the `fork_fleet` nft table, users and record stay; run it again. Then it removes every `fork-fleet-*` namespace and `ffh-*` link, the nft table, the kit's units, the files of its manifest, the template and runner copies, its settings and key, the users it made, and `cron.deny`/`at.deny` when the kit -made them (their lines for its users otherwise). The record goes last. Anything else in its folders (a file the kit +made them (otherwise the lines it added, for every runner user, also one that was there before). The record goes last. Anything else in its folders (a file the kit did not make, a drop-in) is kept and named. IPv4 forwarding stays on until the next reboot. ## Why one flat folder diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index a70e3e5a6fe4c..5ff5e9be200af 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -12,7 +12,7 @@ the units, makes the desk key once, and starts the agent and the refresh timer ( something changed). Safe to run again and to stop at any step: a stopped run leaves SOURCE naming the old release (check says what differs) and the next run ends it. uninstall removes only what the kit made, its record last. """ -import argparse, contextlib, hashlib, importlib.util, json, os, re, shutil, subprocess, sys, tempfile +import argparse, contextlib, hashlib, importlib.util, json, os, pwd, re, shutil, stat, subprocess, sys, tempfile PREFIX = '' # tests only: every fixed path below is read under this folder ROOT_UID = 0 # tests only: who must run this and own the installed files @@ -24,7 +24,9 @@ UNIT_DIR = '/etc/systemd/system' DROPIN_ROOTS = ('/etc/systemd/system', '/run/systemd/system', '/usr/local/lib/systemd/system', '/usr/lib/systemd/system', '/lib/systemd/system') DENY = ('/etc/cron.deny', '/etc/at.deny') -SETTINGS = ('launch.conf', 'desk.conf', 'desk-known_hosts', 'resolv.conf', 'desk-key', 'desk-key.pub') +PROJECT = ETC + '/fleet-config.json' # the project's fleet config (--fleet-config), kept for later installs +SETTINGS = ('launch.conf', 'desk.conf', 'desk-known_hosts', 'resolv.conf', 'desk-key', 'desk-key.pub', + 'fleet-config.json') SYSCTL = '/etc/sysctl.d/60-fork-fleet.conf' HEADER = '# desk kit: written by desk-install. Do not edit: change the machine file and install again.\n' FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh', @@ -86,6 +88,32 @@ def user_exists(name): return run('id', '-u', name, check=False).returncode == 0 +def job_writable(path): + """Why a job could change this file (it, or a folder above it, writable by others or by a runner user's group, or + owned by a runner user); None when only root or the caller can. A sticky folder (/tmp) is fine: nobody else may + rename or remove what is in it.""" + uids, gids = set(), set() + for i in range(1, 9): + with contextlib.suppress(KeyError): + pw = pwd.getpwnam(f'fork-runner-{i}') + uids.add(pw.pw_uid) + gids.add(pw.pw_gid) + path = os.path.abspath(path) + while True: + try: + st = os.stat(path) + except OSError as e: + return f'{path} cannot be read ({e.strerror})' + if st.st_uid in uids: + return f'{path} belongs to a runner user' + if (st.st_mode & 0o002 or st.st_mode & 0o020 and st.st_gid in gids) and not ( + stat.S_ISDIR(st.st_mode) and st.st_mode & stat.S_ISVTX): + return f'{path} can be written by other users' + if path == os.path.dirname(path): + return None + path = os.path.dirname(path) + + def machine_section(): here = os.path.dirname(os.path.realpath(__file__)) found = [c for c in (os.path.join(here, 'machine_file.py'), os.path.join(here, '..', '..', 'common', @@ -120,12 +148,19 @@ def read_state(): raise Refused(f'{STATE} cannot be read ({e}): fix or remove it and run again') from None state = state if isinstance(state, dict) else {} state.setdefault('createdDenyFiles', []) + state.setdefault('addedDenyLines', []) # [deny file, user]: lines the kit added (also for users it did not make) + deny = '|'.join(map(re.escape, DENY)) def names(key, shape): return isinstance(state.get(key), list) and all(isinstance(x, str) and re.fullmatch(shape, x) for x in state[key]) if not (state.get('version') == 1 and names('createdUsers', SHAPES['user']) - and names('createdDenyFiles', '|'.join(map(re.escape, DENY))) + and names('createdDenyFiles', deny) + and isinstance(state['addedDenyLines'], list) + and all(isinstance(x, list) and len(x) == 2 and all(isinstance(y, str) for y in x) + and re.fullmatch(deny, x[0]) and re.fullmatch(SHAPES['user'], x[1]) + for x in state['addedDenyLines']) + and isinstance(state.get('restartPending', False), bool) and state.get('fleetConfig') in ('release', 'project') and (state.get('sha') is None or SHA.fullmatch(str(state['sha'])))): raise Refused(f'{STATE} is not an install record of this kit (version 1): fix or remove it and run again') @@ -240,7 +275,7 @@ def read_manifest(): return {} -def swap(tmp, manifest, sha): +def swap(tmp, manifest, sha, before_change): """Each changed file renamed into /opt/fork-fleet (the agent stopped meanwhile, so no verb runs from half-old, half-new files, and the refresh timer and service stopped too: refresh runs the launcher); files of the old release that are gone, removed. Unchanged files are left as they are. True when anything changed.""" @@ -250,6 +285,7 @@ def swap(tmp, manifest, sha): changed = [rel for rel in sorted(manifest) if not same(rel)] gone = sorted(set(read_manifest()) - set(manifest)) if changed or gone: + before_change() run('systemctl', 'stop', TIMER, REFRESH, AGENT, check=False) for rel in changed: os.makedirs(os.path.dirname(p(f'{OPT}/{rel}')), mode=0o755, exist_ok=True) @@ -291,6 +327,9 @@ def make_users(users, state): save_state(state) lines = lines_of(f) if u not in lines: + if [f, u] not in state['addedDenyLines']: # recorded first: uninstall removes the line again + state['addedDenyLines'].append([f, u]) + save_state(state) write_file(f, '\n'.join(lines + [u]) + '\n') @@ -306,10 +345,11 @@ def settings(m, users): return {path: (HEADER + text, mode) for path, (text, mode) in files.items()} -def make_key(pool): +def make_key(pool, before_change): """The desk key, once: made under a temp name, then renamed (the public half first).""" if os.path.exists(p(KEY)) and os.path.exists(p(KEY + '.pub')): return False + before_change() new = p(ETC + '/.desk-key.new') for f in (new, new + '.pub'): with contextlib.suppress(FileNotFoundError): @@ -338,6 +378,9 @@ def install(a, stop_after=None): raise Refused(f'--source {a.source}: {missing[0]} is missing (point it at the release\'s top folder)') if a.fleet_config and not os.path.isfile(a.fleet_config): raise Refused(f'--fleet-config {a.fleet_config}: no such file') + if a.fleet_config and job_writable(a.fleet_config): + raise Refused(f'--fleet-config {a.fleet_config}: {job_writable(a.fleet_config)}; copy it to a folder only root ' + 'can write (for example /root) and run again') m = machine_section() r = py(os.path.join(a.source, FILES['fork-fleet-walls.py']), 'os') if r.returncode == 3: @@ -347,7 +390,8 @@ def install(a, stop_after=None): there = [f for f in FOREIGN if os.path.lexists(p(f))] if there: raise Refused(f'{there[0]} is from an install this kit did not make: uninstall that first (README)') - state = {'version': 1, 'createdUsers': [], 'createdDenyFiles': [], 'fleetConfig': 'release'} + state = {'version': 1, 'createdUsers': [], 'createdDenyFiles': [], 'addedDenyLines': [], + 'fleetConfig': 'release'} for u in UNITS: if os.path.lexists(p(f'{UNIT_DIR}/{u}')) and not ours(f'{UNIT_DIR}/{u}'): raise Refused(f'{UNIT_DIR}/{u} was not written by this kit: remove it and run again') @@ -355,7 +399,8 @@ def install(a, stop_after=None): raise Refused(f'{drop_ins(u)[0]} changes {u} and was not written by this kit: remove it and run again') config = os.path.realpath(a.fleet_config) if a.fleet_config else None if config is None and state['fleetConfig'] == 'project' and not a.release_fleet_config: - config = p(OPT + '/fork-fleet.json') # the project's config from the last install, checked again below + # the project's config from the last install (older records: the installed copy), checked again below + config = p(PROJECT) if os.path.isfile(p(PROJECT)) else p(OPT + '/fork-fleet.json') users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] done('check') for d, mode in ((OPT, 0o755), (OPT + '/template', 0o755), (ETC, 0o700)): @@ -366,18 +411,38 @@ def install(a, stop_after=None): done('users') tmp, manifest = stage(a.source, a.sha, config, m['pool']) done('stage') - changed = swap(tmp, manifest, a.sha) - state.update(sha=a.sha, fleetConfig='project' if config else 'release') + + def pending(): + """Recorded before anything the agent reads changes: the agent reads its settings only when it starts, so a + run stopped after a write (and before the restart) still restarts it on the next run.""" + if not state.get('restartPending'): + state['restartPending'] = True + save_state(state) + + def put(path, text, mode=0o644): + if not same_text(path, text, mode): + pending() + write_file(path, text, mode) + # the fleet config choice before the swap: a run stopped after it keeps what was asked for + if config: + with open(os.path.join(tmp, 'fork-fleet.json'), encoding='utf-8') as f: + write_file(PROJECT, f.read()) + state['fleetConfig'] = 'project' if config else 'release' + save_state(state) + if not config: + remove(PROJECT) + swap(tmp, manifest, a.sha, pending) + state['sha'] = a.sha save_state(state) done('files') for path, (text, mode) in settings(m, users).items(): - changed |= write_file(path, text, mode) + put(path, text, mode) run('sysctl', '-q', '-p', p(SYSCTL)) - changed |= make_key(m['pool']) + make_key(m['pool'], pending) done('settings') for u in UNITS: with open(p(f'{OPT}/units/{u}'), encoding='utf-8') as f: - changed |= write_file(f'{UNIT_DIR}/{u}', HEADER + f.read()) + put(f'{UNIT_DIR}/{u}', HEADER + f.read()) r = py(p(OPT + '/fork-fleet-desk-agent.py'), 'check') if r.returncode: raise Refused(f'the desk agent refuses its settings (not started): {r.stderr.strip()[:300]}') @@ -387,9 +452,11 @@ def install(a, stop_after=None): run(p(OPT + '/fork-fleet-launch.sh'), 'prune', check=False) # makes the launch lock refresh opens run('systemctl', 'start', '--no-block', REFRESH) # the first runner template, in the background run('systemctl', 'enable', AGENT) - # restart only after a change: a restart ends the agent's whole control group, a `start` it is running with it - # (the always-on host would see that job fail). Unchanged: start is a no-op for a running agent. - run('systemctl', 'restart' if changed else 'start', AGENT) + # restart only after a change (this run's or a stopped run's): a restart ends the agent's whole control group, a + # `start` it is running with it (the always-on host would see that job fail). Unchanged: start is a no-op. + run('systemctl', 'restart' if state.get('restartPending') else 'start', AGENT) + if state.pop('restartPending', None) is not None: + save_state(state) done('units') with open(p(KEY + '.pub'), encoding='utf-8') as f: pub = f.read().strip() @@ -445,6 +512,10 @@ def check(): row(False, 'machine-file', str(e)[:300]) for u in users: row(user_exists(u), 'runner-user', f'{u} exists') + with contextlib.suppress(Refused): # a smaller `slots` leaves the kit's extra users: named, not lost + for u in (read_state() or {}).get('createdUsers', []) if users else []: + if u not in users and user_exists(u): + row(True, 'runner-user', f'{u} is no longer a slot (no login, unused; uninstall removes it)') try: with open(p('/proc/sys/net/ipv4/ip_forward'), encoding='ascii') as f: forward = f.read().strip() == '1' @@ -479,7 +550,18 @@ def installed_problems(): source = None if source != state['sha']: problems.append(f'{OPT}/SOURCE names {source}, the record {state["sha"]}: install again') - problems += [f'{UNIT_DIR}/{u}: missing or not the kit\'s' for u in UNITS if not ours(f'{UNIT_DIR}/{u}')] + if state.get('restartPending'): + problems.append('a stopped install changed what the agent reads but did not restart it: install again') + for u in UNITS: + try: + with open(p(f'{OPT}/units/{u}'), encoding='utf-8') as f: + want = HEADER + f.read() + except OSError: + want = None + if not ours(f'{UNIT_DIR}/{u}'): + problems.append(f'{UNIT_DIR}/{u}: missing or not the kit\'s') + elif want is None or not same_text(f'{UNIT_DIR}/{u}', want, 0o644): + problems.append(f'{UNIT_DIR}/{u}: not the unit of release {state["sha"]} ({OPT}/units/{u}): install again') problems += [f'{d}: a drop-in the kit did not write changes {u}' for u in UNITS for d in drop_ins(u)] return problems @@ -559,8 +641,10 @@ def uninstall(): run('userdel', u) for f in DENY: lines = lines_of(f) - if any(x in state['createdUsers'] for x in lines): - write_file(f, ''.join(x + '\n' for x in lines if x not in state['createdUsers'])) + # the lines the kit added (for every runner user, made by it or not; older records: the users it made) + added = set(state['createdUsers']) | {u for g, u in state['addedDenyLines'] if g == f} + if any(x in added for x in lines): + write_file(f, ''.join(x + '\n' for x in lines if x not in added)) if f in state['createdDenyFiles'] and not [x for x in lines_of(f) if x.strip()]: remove(f) for name in SETTINGS: diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index 2b68fdc57f035..e00bef21edfe1 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -260,6 +260,32 @@ def test_a_changed_setting_restarts_the_agent(self): self.assertEqual(self.install()[0], 0) self.assertIn('systemctl restart fork-fleet-desk-agent.service', self.calls()) + def test_a_change_written_by_a_killed_run_still_restarts_the_agent(self): + self.assertEqual(self.install()[0], 0) + self.template() + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'host': 'new-host.example'}}) + self.assertIn('stopped after', str(self.install(stop_after='settings')[0])) + code, out, _ = self.run_main('check') # desk.conf is new, the running agent still has the old one + self.assertEqual(code, 1) + self.assertIn('did not restart it', out) + self.clear_calls() + self.assertEqual(self.install()[0], 0) + self.assertIn('systemctl restart fork-fleet-desk-agent.service', self.calls()) + self.assertNotIn('restartPending', self.read('/etc/fork-fleet/desk-kit.json')) + self.clear_calls() + self.assertEqual(self.install()[0], 0) # and once restarted, not again + self.assertNotIn('systemctl restart fork-fleet-desk-agent.service', self.calls()) + + def test_a_refused_agent_check_keeps_the_restart_for_the_next_run(self): + self.assertEqual(self.install()[0], 0) + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'port': 2222}}) + self.write(os.path.join(self.fakes, 'agent-bad'), '') + self.assertEqual(self.install()[0], 1) + os.unlink(os.path.join(self.fakes, 'agent-bad')) + self.clear_calls() + self.assertEqual(self.install()[0], 0) + self.assertIn('systemctl restart fork-fleet-desk-agent.service', self.calls()) + def test_stopped_at_any_step_the_next_run_ends_it(self): self.assertEqual(self.install()[0], 0) want = self.tree() @@ -312,6 +338,33 @@ def test_project_fleet_config_is_used_and_kept(self): self.assertEqual(code, 1) self.assertIn('one or the other', err) + def test_the_fleet_config_choice_outlives_a_run_killed_after_the_swap(self): + project = os.path.join(self.tmp, 'project.json') + self.write(project, json.dumps({'pools': [{'name': 'desktop'}], 'project': True})) + newer = self.release('release3', {'.github/fork-fleet.json': json.dumps({'pools': [{'name': 'desktop'}], + 'release3': True})}) + swap = self.m.swap + + def killed(*args): + swap(*args) + raise SystemExit('killed right after the swap') + for first, then, want, other in ((['--fleet-config', project], ['--release-fleet-config'], '"release3"', + '"project"'), + ([], ['--fleet-config', project], '"project"', '"release3"')): + with self.subTest(then[0]): + shutil.rmtree(self.root) + self.write(self.path('/proc/sys/net/ipv4/ip_forward'), '1\n') + self.assertEqual(self.install(*first)[0], 0) + self.m.swap = killed + try: + self.assertIn('killed', str(self.install(*then, sha=SHA2)[0])) + finally: + self.m.swap = swap + code, _, err = self.install(sha=SHA1, source=newer) # a plain later install keeps the last choice + self.assertEqual(code, 0, err) + self.assertIn(want, self.read('/opt/fork-fleet/fork-fleet.json')) + self.assertNotIn(other, self.read('/opt/fork-fleet/fork-fleet.json')) + def test_refusals_change_nothing(self): cases = { 'not a release sha': (lambda: None, ['--sha', 'abc'], '--sha must be'), @@ -340,6 +393,11 @@ def test_refusals_change_nothing(self): '/run/systemd/system/t3code-fleet-.timer.d/x.conf'), '[Timer]\n'), [], 'changes t3code-fleet'), 'a record of another shape': (lambda: self.write(self.path('/etc/fork-fleet/desk-kit.json'), '{"version": 1}'), [], 'not an install record'), + 'a fleet config others may write': (lambda: (os.makedirs(os.path.join(self.tmp, 'open'), exist_ok=True), + os.chmod(os.path.join(self.tmp, 'open'), 0o777), + self.write(os.path.join(self.tmp, 'open', 'f.json'), '{}')), + ['--fleet-config', os.path.join(self.tmp, 'open', 'f.json')], + 'can be written by other users'), } for name, (setup, args, msg) in cases.items(): with self.subTest(name): @@ -456,6 +514,23 @@ def test_check_fails_on_a_drop_in_and_on_settings_the_machine_file_no_longer_giv self.assertEqual(code, 1) self.assertIn('override.conf: a drop-in the kit did not write changes fork-fleet-desk-agent.service', out) + def test_check_fails_when_a_unit_is_not_the_release_unit(self): + self.assertEqual(self.install()[0], 0) + self.template() + new = self.release('release2', {'.github/scripts/fork-fleet-desk-agent.service': '# v2\n[Service]\n'}) + self.assertIn('stopped after', str(self.install(sha=SHA2, source=new, stop_after='settings')[0])) + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + self.assertIn('fork-fleet-desk-agent.service: not the unit of release', out) + self.assertEqual(self.install(sha=SHA2, source=new)[0], 0) + self.assertEqual(self.run_main('check')[0], 0) + + def test_check_names_a_runner_user_left_by_fewer_slots(self): + self.assertEqual(self.install()[0], 0) + self.machine({'version': 1, 'desk': {**MACHINE['desk'], 'slots': 1}}) + self.assertEqual(self.install()[0], 0) + self.assertIn('PASS runner-user fork-runner-2 is no longer a slot', self.run_main('check')[1]) + def test_check_before_install_fails_without_writing(self): code, out, _ = self.run_main('check') self.assertEqual(code, 1) @@ -512,6 +587,15 @@ def test_uninstall_removes_the_deny_files_it_made(self): self.assertFalse(os.path.lexists(self.path('/etc/at.deny'))) # with none, only root may use at again self.assertEqual(self.read('/etc/cron.deny'), 'someone\n') + def test_uninstall_removes_the_deny_lines_it_added_for_users_that_were_there(self): + self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\nfork-runner-2\n') # both made by hand + self.write(self.path('/etc/cron.deny'), 'someone\n') + self.assertEqual(self.install()[0], 0) + self.assertEqual(self.read('/etc/at.deny'), 'fork-runner-1\nfork-runner-2\n') + self.assertEqual(self.run_main('uninstall')[0], 0) + self.assertFalse(os.path.lexists(self.path('/etc/at.deny'))) # with none, only root may use at again + self.assertEqual(self.read('/etc/cron.deny'), 'someone\n') + def test_uninstall_keeps_the_table_while_a_runner_still_runs(self): self.assertEqual(self.install()[0], 0) self.write(os.path.join(self.fakes, 'units'), 'fork-fleet-desktop-1.service\nfork-fleet-desktop-2.service\n') @@ -612,9 +696,10 @@ def test_bootstrap_checks_its_arguments_before_anything(self): env = {'PATH': '/usr/bin:/bin'} for args, msg in ((['--repo', 'x'], '--repo must be'), (['--repo', 'a/b', '--sha', 'x'], '--sha must be'), (['--nope'], 'unknown option'), - # flags after --fleet-config are checked too: a later --sha cannot slip past - (['--repo', 'a/b', '--fleet-config', boot, '--sha', 'x'], '--sha must be'), - (['--fleet-config', boot, '--repo', 'x'], '--repo must be'), + # flags after --fleet-config are checked too: a later --sha cannot replace a good one + (['--repo', 'a/b', '--sha', 'a' * 40, '--fleet-config', boot, '--sha', 'x'], '--sha must be'), + (['--repo', 'a/b', '--sha', 'a' * 40, '--fleet-config', boot, '--repo', 'x'], + '--repo must be'), (['--repo', 'a/b', '--sha', 'a' * 40, '--fleet-config', '/nonexistent/f.json'], 'no such file'), (['--repo', 'a/b', '--sha', 'a' * 40, '--fleet-config'], 'needs a value')): From e88ff71f0b413678b0d88493a97881fb11141678 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:40:19 +0000 Subject: [PATCH 18/33] fork kit desk: Windows starter fix round 2 (WSL 2 only, inputs first, marks) (D9) - wsl --install with --version 2; a WSL 1 distro (wsl -l -v) is refused before the question and before any restart, a new one is checked after the install. - -MachineFile and -FleetConfig are tested before the first question; -Sha is matched case-sensitively; a PC too small for the default cap gets its own message. - A Windows-side mark for an owed wsl --shutdown (written before .wslconfig changes, removed after the shutdown: uptime skips sleep) and for a distro this script is installing (no takeover question after a stop); a failed first start names wsl --unregister. - A live check for Windows drives mounted under /mnt. - -Uninstall stops the logon task before removing it and names the first .wslconfig.bak-*. - The live tests put a wslinfo that answers nat first on PATH. Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 15 ++++-- fork-kit/desk/tests/windows.test.py | 70 +++++++++++++++++++++++++++- fork-kit/desk/windows/start-desk.ps1 | 65 +++++++++++++++++++++----- 3 files changed, 132 insertions(+), 18 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 3a21729e19f6d..7da0fa4d9985d 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -62,19 +62,26 @@ Save the machine file on Windows, then in a normal (not administrator) Windows P powershell -ExecutionPolicy Bypass -File start-desk.ps1 -Repo OWNER/REPO -Sha RELEASE_SHA -MachineFile machine.json ``` -It prints the PC's facts and asks before it changes anything. A distro of that name that the script did not make +It prints the PC's facts and asks before it changes anything. Before the question it checks the machine file and +`-FleetConfig` exist and refuses a WSL 1 distro of that name (`wsl -l -v`; WSL 1 runs no systemd: +`wsl --set-version DISTRO 2` first). A distro of that name that the script did not make is named in the question (its whole `/etc/wsl.conf` is replaced) and must be confirmed by typing its name. Then: the -distro (`-Distro`, default `fork-desk`, Ubuntu 24.04); `.wslconfig`'s `[wsl2]` memory cap (`-MemoryGB`, default +distro (`-Distro`, default `fork-desk`, Ubuntu 24.04, installed with `--version 2`; a run stopped while installing it +leaves a mark in `%LOCALAPPDATA%\fork-desk`, so the next run asks no takeover question, and a broken one names +`wsl --unregister DISTRO`); `.wslconfig`'s `[wsl2]` memory cap (`-MemoryGB`, default three quarters of the RAM, at most the RAM less 6 GB, never under 4 GB; one that leaves Windows under 2 GB is refused), swap (`-SwapGB`, 4), NAT and DNS tunnelling, for every WSL distro (read and written as UTF-8 without a BOM; a dated copy of the old file is kept, other keys and sections stay); the distro's own settings the WSL2 walls ask for (`/etc/wsl.conf`: systemd on, no Windows drives, programs or PATH; its own `/etc/resolv.conf`; WSL's binfmt handler masked); one `wsl --shutdown` when one of these changed or the running WSL does not have them yet (PID 1 not -systemd, interop live, `wsl.conf` or `.wslconfig` newer than the running one, more memory than the cap, not NAT), +systemd, interop live, Windows drives mounted, `wsl.conf` or `.wslconfig` newer than the running one, more memory +than the cap, not NAT, or a `.wslconfig` change whose shutdown is still owed: a mark in `%LOCALAPPDATA%\fork-desk`, +written before the change and removed after the shutdown, because WSL's uptime skips sleep), so a run stopped before its shutdown restarts next time (every WSL distro and Docker Desktop stop); the packages the fleet needs; the machine file and `-FleetConfig`, if given; this kit's bootstrap; a logon task that keeps the distro running while the owner is logged in. Last, `desk-install check` and the Windows side's own PASS/FAIL lines (the `[wsl2]` keys); -any FAIL or error exits 1. Safe to run again. `-Uninstall` (needs no `-Repo` or `-Sha`) runs `desk-install uninstall` in the distro and removes the logon task; the distro and `.wslconfig` +any FAIL or error exits 1. Safe to run again. `-Uninstall` (needs no `-Repo` or `-Sha`) runs `desk-install uninstall` in the distro, stops and removes the logon +task and names the first `.wslconfig.bak-*` (the file before the first run); the distro and `.wslconfig` stay (`wsl --unregister DISTRO` removes the distro). ## Runs again, stops half-way, old or empty state diff --git a/fork-kit/desk/tests/windows.test.py b/fork-kit/desk/tests/windows.test.py index d36922dc5d0c2..250005e009e9d 100644 --- a/fork-kit/desk/tests/windows.test.py +++ b/fork-kit/desk/tests/windows.test.py @@ -97,8 +97,14 @@ def test_an_empty_distro(self): self.assertIsNone(walls(self.root).interop_wall()) def live(self, env=None): + # a wslinfo that answers nat first on PATH: inside a WSL with mirrored networking the real one would answer + fake = os.path.join(self.root, 'nat-bin') + if not os.path.exists(fake): + os.makedirs(fake) + put(fake + '/wslinfo', '#!/bin/sh\necho nat\n') + os.chmod(fake + '/wslinfo', 0o755) r = subprocess.run(['bash', '-s'], input=prepare('Live') + '\n# end\r\n', text=True, capture_output=True, - env={'PATH': '/usr/bin:/bin', 'ROOT': self.root, **(env or {})}) + env={'PATH': fake + ':/usr/bin:/bin', 'ROOT': self.root, **(env or {})}) self.assertEqual(r.returncode, 0, r.stderr) return r.stdout @@ -127,6 +133,14 @@ def test_the_restart_decision_reads_the_live_state(self): os.chmod(fake + '/wslinfo', 0o755) self.assertIn('the networking mode is mirrored, not nat', self.live({'PATH': fake + ':/usr/bin:/bin'})) + def test_a_mounted_windows_drive_restarts(self): + os.makedirs(self.root + '/proc/1') + put(self.root + '/proc/1/comm', 'systemd\n') + put(self.root + '/proc/mounts', 'none /usr/lib/wsl/drivers 9p ro 0 0\n/dev/sdc / ext4 rw 0 0\n') + self.assertEqual(self.live(), '') # WSL's own driver share is not a Windows drive + put(self.root + '/proc/mounts', 'C:\\134 /mnt/c 9p rw 0 0\n') + self.assertIn('restart: Windows drives are mounted under /mnt', self.live()) + def test_a_resolv_conf_of_its_own_is_kept(self): put(self.root + '/etc/resolv.conf', 'nameserver 192.0.2.1\n') self.run_prepare() @@ -161,12 +175,64 @@ def test_it_asks_before_it_changes_anything(self): def test_a_distro_it_did_not_make_is_named_and_must_be_typed(self): src = text().decode('ascii') body = src[src.index('try {'):] - ask = body.index("if ($have -and -not $ours)") + ask = body.index("if ($have -and -not $ours) {\n Write-Host") self.assertLess(ask, body.index('wsl.exe --install')) self.assertIn("-cne $Distro", body[ask:ask + 800]) self.assertIn('/etc/wsl.conf is REPLACED', body[ask:ask + 800]) self.assertIn('/etc/fork-kit/desk-distro', prepare()) # the mark of a distro it made or took over + def test_wsl1_is_refused_before_anything_changes_and_a_new_distro_is_wsl2(self): + src = text().decode('ascii') + body = src[src.index('try {'):] + install = [x for x in body.splitlines() if 'wsl.exe --install' in x] + self.assertEqual(len(install), 1) + self.assertIn('--version 2', install[0]) + refuse = body.index('if ($have -and $version -ne 2) { throw') + for later in ("Read-Host 'Type yes", 'Merge-WslConfig $WslConfig', 'wsl.exe --shutdown', 'Mark $Made', + "InDistro \"the distro's settings\""): + self.assertLess(refuse, body.index(later), later) + self.assertLess(body.index('wsl.exe --install'), body.index('if ($version -ne 2) { throw')) + self.assertLess(body.index('if ($version -ne 2) { throw'), body.index('wsl.exe --shutdown')) + self.assertIn('WslVersion @(wsl.exe -l -v) $Distro', body) + self.assertNotIn('wsl.exe -l -q', body) + + def test_inputs_are_checked_before_the_question(self): + src = text().decode('ascii') + body = src[src.index('if (-not $MachineFile)'):] + ask = min(body.index("Read-Host 'Type yes"), body.index('Read-Host "Type the distro')) + self.assertLess(body.index('Test-Path -LiteralPath $MachineFile -PathType Leaf'), ask) + self.assertLess(body.index('Test-Path -LiteralPath $FleetConfig -PathType Leaf'), ask) + self.assertIn("-cmatch '^[0-9a-f]{40}$'", src.split('try {')[0]) # an uppercase -Sha is refused too + + def test_a_stopped_run_keeps_its_marks(self): + src = text().decode('ascii') + body = src[src.index('try {'):] + self.assertLess(body.index('Mark $Made'), body.index('wsl.exe --install')) + self.assertIn('Merge-WslConfig $WslConfig $want { Mark $Owed }', body) + self.assertIn('& $before', wslconfig_part()) + self.assertLess(wslconfig_part().index('& $before'), wslconfig_part().index('WriteAllLines')) + shutdown = body.index('Run \'wsl --shutdown\'') + self.assertLess(body.index('if (Test-Path -LiteralPath $Owed)'), shutdown) + self.assertGreater(body.index('Remove-Item -LiteralPath $Owed -ErrorAction'), shutdown) + self.assertIn('wsl --unregister $Distro (removes it and all in it)', body) + uninstall = body[body.index('if ($Uninstall)'):] + self.assertLess(uninstall.index('Stop-ScheduledTask'), uninstall.index('Unregister-ScheduledTask')) + self.assertIn('.wslconfig as it was before the first run', uninstall) + + @unittest.skipUnless(shutil.which('pwsh'), 'pwsh is not installed') + def test_wsl_version_is_read_from_the_list(self): + d = tempfile.mkdtemp(prefix='desk-wslversion-test-') + self.addCleanup(shutil.rmtree, d, True) + script = os.path.join(d, 'v.ps1') + with open(script, 'w', encoding='ascii') as f: + f.write(wslconfig_part() + '\n$l = @(" NAME STATE VERSION", "* Ubuntu Running 2", ' + '" fork-desk Stopped 1")\n"$(WslVersion $l fork-desk) $(WslVersion $l Ubuntu) ' + '$(WslVersion $l none)"\n') + r = subprocess.run(['pwsh', '-NoProfile', '-NonInteractive', '-File', script], capture_output=True, + text=True, timeout=120, stdin=subprocess.DEVNULL) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertEqual(r.stdout.strip(), '1 2 0') + def test_wslconfig_is_utf8_without_a_bom_never_ascii(self): src = text().decode('ascii') self.assertNotRegex(src, r'(?i)-Encoding\s+ascii') diff --git a/fork-kit/desk/windows/start-desk.ps1 b/fork-kit/desk/windows/start-desk.ps1 index 531b059c169f7..22630df49f0f8 100644 --- a/fork-kit/desk/windows/start-desk.ps1 +++ b/fork-kit/desk/windows/start-desk.ps1 @@ -9,7 +9,7 @@ # task that keeps the distro running; then `desk-install check` and this side's checks. param( [ValidatePattern('^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$')][string]$Repo, # both required, except with -Uninstall - [ValidatePattern('^[0-9a-f]{40}$')][string]$Sha, + [ValidateScript({ $_ -cmatch '^[0-9a-f]{40}$' })][string]$Sha, # -cmatch: ValidatePattern ignores case [string]$MachineFile, [string]$FleetConfig, [ValidatePattern('^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$')][string]$Distro = 'fork-desk', [ValidateRange(0, 4096)][int]$MemoryGB = 0, [ValidateRange(0, 4096)][int]$SwapGB = 4, @@ -20,6 +20,11 @@ $env:WSL_UTF8 = '1' # wsl.exe's own lines in UTF-8, not UTF $OutputEncoding = [Text.UTF8Encoding]::new($false) # what is piped into wsl.exe: UTF-8, not ASCII $Task = "fork-desk-$Distro" $WslConfig = Join-Path $env:USERPROFILE '.wslconfig' +# Marks on the Windows side, for a run that stops: `made` (this script is installing the distro: a later run asks no +# takeover question), `restart` (.wslconfig changed and its `wsl --shutdown` is still owed: VM uptime skips sleep) +$Marks = Join-Path $env:LOCALAPPDATA 'fork-desk' +$Made = Join-Path $Marks "$Distro.made"; $Owed = Join-Path $Marks "$Distro.restart" +function Mark([string]$path) { New-Item -ItemType Directory -Force -Path $Marks | Out-Null; Set-Content -LiteralPath $path 'start-desk.ps1' } function Head([string]$t) { Write-Host "==== $t" -ForegroundColor Cyan } function Run([string]$what, [scriptblock]$cmd) { @@ -86,6 +91,10 @@ mem=$(sed -n 's/^MemTotal: *\([0-9]*\) kB$/\1/p' "$ROOT/proc/meminfo" 2>/dev/nul [ -z "${WANT_KB:-}" ] || [ -z "$mem" ] || [ "$mem" -le "$WANT_KB" ] || echo "restart: WSL has $mem kB, above the cap" mode=$(wslinfo --networking-mode 2>/dev/null || true) [ -z "$mode" ] || [ "$mode" = nat ] || echo "restart: the networking mode is $mode, not nat" +# automount=false: no Windows drive mounted under /mnt +if awk '$2 ~ /^\/mnt\/[a-z]$/ { found = 1 } END { exit !found }' "$ROOT/proc/mounts" 2>/dev/null; then + echo 'restart: Windows drives are mounted under /mnt' +fi '@ # .wslconfig, read and written as UTF-8 without a BOM: Windows PowerShell 5.1's Get-Content reads a file without a # BOM as ANSI and Set-Content with the ASCII encoding turns every other letter into '?' (a kernel= path with an accent in it @@ -103,7 +112,7 @@ function Wsl2Keys([string[]]$lines) { # the keys of the [wsl2] section only } $keys } -function Merge-WslConfig([string]$path, $want) { # True when it changed the file (a dated copy of the old one kept) +function Merge-WslConfig([string]$path, $want, [scriptblock]$before = {}) { # True when it changed the file (a dated copy of the old one kept; $before runs first) $lines = Read-WslConfig $path $out = [Collections.Generic.List[string]]::new(); $sec = ''; $done = @{} function AddMissing { foreach ($k in $want.Keys) { if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 } } } @@ -116,11 +125,19 @@ function Merge-WslConfig([string]$path, $want) { # True when it changed the fi if ($sec -eq 'wsl2') { AddMissing } if ($done.Count -lt $want.Count) { $out.Add('[wsl2]'); AddMissing } if (($out -join "`n") -eq ($lines -join "`n")) { return $false } + & $before if (Test-Path -LiteralPath $path) { Copy-Item -LiteralPath $path "$path.bak-$(Get-Date -Format yyyyMMdd-HHmmss)" } [IO.File]::WriteAllLines("$path.new", [string[]]$out, [Text.UTF8Encoding]::new($false)) Move-Item -LiteralPath "$path.new" $path -Force $true } +function WslVersion([string[]]$lines, [string]$name) { # the WSL version of one distro in `wsl -l -v`; 0: not there + foreach ($l in $lines) { + $f = @(($l.Trim([char]0, ' ', '*', "`t") -replace [char]0, '') -split '\s+') + if ($f.Count -ge 3 -and $f[0] -eq $name -and $f[-1] -match '^[0-9]+$') { return [int]$f[-1] } + } + 0 +} # --- end wslconfig --- $Packages = 'set -eu; export DEBIAN_FRONTEND=noninteractive; need=; for p in python3 nftables iproute2 curl ca-certificates openssh-client tar; do dpkg -s $p >/dev/null 2>&1 || need="$need $p"; done; [ -z "$need" ] || { apt-get update -q && apt-get install -y -q $need; }' @@ -128,26 +145,41 @@ try { Head 'This PC' $os = Get-CimInstance Win32_OperatingSystem; $cs = Get-CimInstance Win32_ComputerSystem $ram = [math]::Floor($cs.TotalPhysicalMemory / 1GB) - if ($MemoryGB -eq 0) { $MemoryGB = [math]::Max(4, [math]::Min([math]::Floor($ram * 3 / 4), $ram - 6)) } - if (-not $Uninstall -and $MemoryGB -gt $ram - 2) { throw "-MemoryGB $MemoryGB leaves Windows less than 2 GB of this PC's $ram GB" } + $asked = $MemoryGB -ne 0 + if (-not $asked) { $MemoryGB = [math]::Max(4, [math]::Min([math]::Floor($ram * 3 / 4), $ram - 6)) } + if (-not $Uninstall -and $MemoryGB -gt $ram - 2) { + if ($asked) { throw "-MemoryGB $MemoryGB leaves Windows less than 2 GB of this PC's $ram GB" } + throw "this PC has $ram GB of RAM: a desk needs at least 4 GB for WSL and 2 GB for Windows" + } "Windows $($os.Version) build $($os.BuildNumber); $($cs.NumberOfLogicalProcessors) threads; RAM $ram GB" 'WSL:'; wsl.exe --version - $have = @(wsl.exe -l -q | ForEach-Object { $_.Trim([char]0, ' ') }) -contains $Distro - "distro ${Distro}: $(if ($have) { 'there' } else { 'not yet' })" + $version = WslVersion @(wsl.exe -l -v) $Distro # read without starting any distro + $have = $version -ne 0 + "distro ${Distro}: $(if ($have) { "there (WSL $version)" } else { 'not yet' })" if (Test-Path $WslConfig) { ".wslconfig now:"; Read-WslConfig $WslConfig | ForEach-Object { " $_" } } - # A distro of that name that this script did not make: its /etc/wsl.conf is replaced below - $ours = $false - if ($have) { wsl.exe -d $Distro -u root --cd / -- test -e /etc/fork-kit/desk-distro; $ours = $LASTEXITCODE -eq 0 } if ($Uninstall) { if ((Read-Host "Remove the desk kit from $Distro and the logon task $Task? Type yes") -ne 'yes') { 'Stopped: nothing changed.'; return } if ($have) { InDistro 'desk-install uninstall' '[ ! -x /opt/fork-fleet/desk-install ] || /opt/fork-fleet/desk-install uninstall' } + Stop-ScheduledTask -TaskName $Task -ErrorAction SilentlyContinue # its `sleep infinity` too Unregister-ScheduledTask -TaskName $Task -Confirm:$false -ErrorAction SilentlyContinue + Remove-Item -LiteralPath $Owed, $Made -ErrorAction SilentlyContinue + $bak = @(Get-ChildItem -Path "$WslConfig.bak-*" -ErrorAction SilentlyContinue | Sort-Object Name) "Done. The distro and .wslconfig are kept; to remove the distro too: wsl --unregister $Distro" + if ($bak) { ".wslconfig as it was before the first run: $($bak[0].FullName) (copy it back over .wslconfig to undo)" } return } if (-not $MachineFile) { throw '-MachineFile is required: the machine file with the desk section (README)' } if (-not $Repo -or -not $Sha) { throw '-Repo and -Sha are required (the fork and its release commit)' } + # every input before the question, so a typo is found before anything changes + if (-not (Test-Path -LiteralPath $MachineFile -PathType Leaf)) { throw "-MachineFile ${MachineFile}: no such file" } + if ($FleetConfig -and -not (Test-Path -LiteralPath $FleetConfig -PathType Leaf)) { throw "-FleetConfig ${FleetConfig}: no such file" } + # WSL1 runs no systemd: every run would restart WSL and check would never pass + if ($have -and $version -ne 2) { throw "the distro $Distro is WSL $version; the desk needs WSL 2: wsl --set-version $Distro 2 (or pick another -Distro), then this again" } + # A distro of that name that this script did not make: its /etc/wsl.conf is replaced below. (This starts it; it + # changes nothing in it.) A run stopped while installing it left the Windows-side mark. + $ours = Test-Path -LiteralPath $Made + if ($have -and -not $ours) { wsl.exe -d $Distro -u root --cd / -- test -e /etc/fork-kit/desk-distro; $ours = $LASTEXITCODE -eq 0 } Write-Host '' Write-Host "Next: the distro $Distro; .wslconfig [wsl2] memory=${MemoryGB}GB swap=${SwapGB}GB networkingMode=nat" -ForegroundColor Yellow @@ -163,29 +195,38 @@ try { Head 'the distro' if (-not $have) { - wsl.exe --install -d Ubuntu-24.04 --name $Distro --no-launch + Mark $Made # before the install: a run stopped inside it asks no takeover question next time + wsl.exe --install -d Ubuntu-24.04 --name $Distro --no-launch --version 2 if ($LASTEXITCODE -ne 0) { throw "wsl --install failed (exit $LASTEXITCODE): run wsl --update (or restart Windows after WSL's own install), then this again" } + $version = WslVersion @(wsl.exe -l -v) $Distro + if ($version -ne 2) { throw "the new distro $Distro is WSL $version, not 2: wsl --unregister $Distro, then wsl --set-default-version 2 and this again" } + } + wsl.exe -d $Distro -u root --cd / -- true + if ($LASTEXITCODE -ne 0) { + throw "the first start of the distro failed (exit $LASTEXITCODE). If a run was stopped while installing it: wsl --unregister $Distro (removes it and all in it), then this again" } - Run 'the first start of the distro' { wsl.exe -d $Distro -u root --cd / -- true } Head '.wslconfig' $want = [ordered]@{ memory = "${MemoryGB}GB"; swap = "${SwapGB}GB"; networkingMode = 'nat'; dnsTunneling = 'true'; guiApplications = 'false' } - $restart = Merge-WslConfig $WslConfig $want + $restart = Merge-WslConfig $WslConfig $want { Mark $Owed } # the mark first: a run stopped after the write still restarts if ($restart) { 'changed: .wslconfig' } Head "the distro's own settings" $changed = InDistro "the distro's settings" $Prepare $changed + Remove-Item -LiteralPath $Made -ErrorAction SilentlyContinue # the distro's own mark is there now # The live state decides, not only what this run changed: a run stopped before its shutdown restarts next time $live = @(InDistro 'the running settings' ("WANT_KB=$([long]$MemoryGB * 1048576)`n" + $Live)) $up = [double]((wsl.exe -d $Distro -u root --cd / -- cat /proc/uptime) -split ' ')[0] if ((Test-Path $WslConfig) -and ((Get-Date) - (Get-Item -LiteralPath $WslConfig).LastWriteTime).TotalSeconds -lt $up) { $live += 'restart: .wslconfig is newer than the running WSL' } + if (Test-Path -LiteralPath $Owed) { $live += 'restart: .wslconfig changed and its wsl --shutdown is still owed' } $live if ($restart -or $changed -or $live) { Head 'wsl --shutdown (the new settings apply at the next start)' Run 'wsl --shutdown' { wsl.exe --shutdown } + Remove-Item -LiteralPath $Owed -ErrorAction SilentlyContinue } InDistro 'the packages the fleet needs' $Packages From 83849bcb5f9f2acaefd5fc3c447760e9d25360dd Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:48:19 +0000 Subject: [PATCH 19/33] fork kit free/v1: judge the path as written, bounded wait (D10 fix round 2) - free_contract.resolve() follows freeCommand one part at a time, as the kernel does: every folder on the path as written must belong to root or the caller and not be changeable by others (sticky is fine), and every link on the way must belong to root or the caller. Another user's link in a sticky folder is no longer followed to a root program (what fs.protected_symlinks refuses). Loops stop after 40 links. - ask(): a process group it may not signal no longer makes it wait without a limit; it ends the script itself when it can and waits at most one more second. - Tests: the lens test (another user's link in a sticky folder), folders of the path as written, relative links, chains and loops, and the bounded wait. FREE.md says the same. Co-Authored-By: Claude Opus 5.5 --- fork-kit/common/FREE.md | 8 +- fork-kit/common/free_contract.py | 131 ++++++++++++++------ fork-kit/common/tests/free_contract.test.py | 65 ++++++++++ 3 files changed, 163 insertions(+), 41 deletions(-) diff --git a/fork-kit/common/FREE.md b/fork-kit/common/FREE.md index acede94965da0..95eebbe60cedc 100644 --- a/fork-kit/common/FREE.md +++ b/fork-kit/common/FREE.md @@ -11,8 +11,12 @@ One small contract for every tool that asks whether a machine may take more work and so is any process that still holds its stdout. It leaves nothing running: a child that leaves the group (`setsid`) and closes stdout is not seen and not ended. - It and every folder above it belong to root (or to the user the consumer runs as) and no one else can change - them (a sticky folder such as `/tmp` is fine). Otherwise it is not run. A link is judged and run by the file it - leads to. Only the mode bits are read: an ACL that lets others write is not seen, so do not set one. + them (a sticky folder such as `/tmp` is fine). Otherwise it is not run. The path is followed one part at a + time: the folders of the path as written are judged too, and every link on the way must belong to root or to + that user (another user's link in `/tmp` is refused, as the kernel's `fs.protected_symlinks` would). The script + is judged and run by the file the path leads to, resolved once. Only the mode bits are read: an ACL that lets + others write is not seen, so do not set one. A script that could not be ended (a group the consumer may not + signal) is waited for at most one more second, then left to finish on its own. - **The exit code is the verdict:** `0` free, `75` busy. Anything else, a timeout, output that is not one JSON object (`NaN` and `Infinity` are not JSON), a higher `v`, a `free` that disagrees with the exit code, or facts more than 60 seconds from now = **not available**. Busy and not available both count as **not free**. diff --git a/fork-kit/common/free_contract.py b/fork-kit/common/free_contract.py index e4e9ee18baaa4..f0216467c9073 100644 --- a/fork-kit/common/free_contract.py +++ b/fork-kit/common/free_contract.py @@ -74,26 +74,71 @@ def no(why): return 'free', '', facts +def kept(path, st, uid): + """Why the folder PATH (its stat ST) may not be trusted to hold what we run, or None.""" + if st.st_uid not in (0, uid): + return f'{path} belongs to uid {st.st_uid}' + if st.st_mode & 0o022 and not (stat.S_ISDIR(st.st_mode) and st.st_mode & stat.S_ISVTX): + return f'{path} may be changed by others' + return None + + +def resolve(command, uid=None): + """(the file COMMAND leads to, None), or (None, why not). Follows the path one part at a time, as the kernel + does, so the folders of the path as written are judged, not only those of the file it leads to: every folder + on the way belongs to root or to this user and no one else may change it (a sticky folder like /tmp is fine), + and every link on the way belongs to root or to this user. The kernel's fs.protected_symlinks would refuse + another user's link in a sticky folder; resolving here must not quietly follow one.""" + uid = os.geteuid() if uid is None else uid + if not os.path.isabs(command) or os.path.normpath(command) != command: + return None, f'{command!r} is not an absolute plain path' + parts, path, links = command.split('/')[1:], '/', 0 + try: + while parts: + name = parts.pop(0) + if name in ('', '.'): + continue + if name == '..': + path = os.path.dirname(path) + continue + why = kept(path, os.stat(path), uid) + if why: + return None, why + here = os.path.join(path, name) + st = os.lstat(here) + if not stat.S_ISLNK(st.st_mode): + path = here + continue + if st.st_uid not in (0, uid): + return None, f'{here} is a link of uid {st.st_uid}' + links += 1 + if links > 40: + return None, f'{command}: too many levels of links' + target = os.readlink(here) + parts = target.split('/') + parts + if target.startswith('/'): + path = '/' + except OSError as e: + return None, f'{os.path.join(path, name)}: {e.strerror}' + return path, None + + def unsafe(command, uid=None): - """Why COMMAND must not run, or None: an absolute plain path to an executable file that it and every folder - above it belong to root or to this user and that no one else may change (a sticky folder like /tmp is fine). - A link is judged by the file it leads to; ask() resolves the link once and both judges and runs that file. + """Why COMMAND must not run, or None: an absolute plain path that resolve() accepts, to an executable file + that it and every folder above it belong to root or to this user and that no one else may change (a sticky + folder like /tmp is fine). ask() resolves a link once and both judges and runs the file it leads to. Plain mode bits only: an ACL that lets others write is not seen.""" uid = os.geteuid() if uid is None else uid - if not os.path.isabs(command) or os.path.normpath(command) != command: - return f'{command!r} is not an absolute plain path' - path = os.path.realpath(command) + path, why = resolve(command, uid) + if why: + return why try: if not stat.S_ISREG(os.stat(path).st_mode) or not os.access(path, os.X_OK): return f'{path} is not an executable file' while True: - st = os.stat(path) - if st.st_uid not in (0, uid): - return f'{path} belongs to uid {st.st_uid}' - if st.st_mode & 0o022 and not (stat.S_ISDIR(st.st_mode) and st.st_mode & stat.S_ISVTX): - return f'{path} may be changed by others' - if path == '/': - return None + why = kept(path, os.stat(path), uid) + if why or path == '/': + return why path = os.path.dirname(path) except OSError as e: return f'{path}: {e.strerror}' @@ -119,9 +164,8 @@ def ask(command, need_mb=None, timeout=LIMIT_SECONDS): reason, facts). After it, its whole process group is ended, and so is any other process that still holds its stdout (a child that left the group with setsid). A child that leaves the group and closes stdout is not seen: FREE.md asks scripts to leave nothing running.""" - plain = os.path.isabs(command) and os.path.normpath(command) == command - path = os.path.realpath(command) if plain else command # resolved once: a link repointed later changes nothing - why = unsafe(path) + path, why = resolve(command) # resolved once: a link repointed later changes nothing + why = why or unsafe(path) if why: return 'unavailable', why, None try: @@ -132,32 +176,41 @@ def ask(command, need_mb=None, timeout=LIMIT_SECONDS): out, code, closed, deadline = b'', None, False, time.monotonic() + timeout poll = select.poll() # not select(): a long-lived consumer may hold descriptors above 1024 poll.register(p.stdout, select.POLLIN) - with p: + try: + while len(out) <= MAX_BYTES: + left = deadline - time.monotonic() + if left <= 0 or not poll.poll(left * 1000): + break + chunk = os.read(p.stdout.fileno(), 65536) + if not chunk: + closed = True + try: + code = p.wait(max(0.0, deadline - time.monotonic())) + except subprocess.TimeoutExpired: + pass + break + out += chunk + finally: try: - while len(out) <= MAX_BYTES: - left = deadline - time.monotonic() - if left <= 0 or not poll.poll(left * 1000): - break - chunk = os.read(p.stdout.fileno(), 65536) - if not chunk: - closed = True - try: - code = p.wait(max(0.0, deadline - time.monotonic())) - except subprocess.TimeoutExpired: - pass - break - out += chunk - finally: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + except PermissionError: # a group we may not signal (a set-uid script): end at least the script try: - os.killpg(p.pid, signal.SIGKILL) - except (ProcessLookupError, PermissionError): + p.kill() + except OSError: pass - if not closed: # someone still holds stdout open - for pid in holders(p.stdout): - try: - os.kill(pid, signal.SIGKILL) - except (ProcessLookupError, PermissionError): - pass + if not closed: # someone still holds stdout open + for pid in holders(p.stdout): + try: + os.kill(pid, signal.SIGKILL) + except (ProcessLookupError, PermissionError): + pass + p.stdout.close() + try: # never a wait without a limit: a script we could not end is left to finish on its own + p.wait(1) + except subprocess.TimeoutExpired: + pass return judge(code, out, datetime.datetime.now(datetime.timezone.utc), need_mb) diff --git a/fork-kit/common/tests/free_contract.test.py b/fork-kit/common/tests/free_contract.test.py index 2641736a197f1..8423a0ffd31be 100644 --- a/fork-kit/common/tests/free_contract.test.py +++ b/fork-kit/common/tests/free_contract.test.py @@ -127,6 +127,71 @@ def test_a_link_is_judged_by_where_it_leads(self): os.symlink(s, os.path.join(self.d, 'link')) self.assertEqual(fc.ask(os.path.join(self.d, 'link'))[0], 'free') + def test_a_link_another_user_owns_in_a_sticky_folder_is_not_followed(self): + # what fs.protected_symlinks refuses; resolving in Python must not quietly follow it to a root program + real, geteuid = os.geteuid(), os.geteuid + with tempfile.TemporaryDirectory() as d: + os.chmod(d, 0o1777) + link = os.path.join(d, 'free') + os.symlink(os.path.realpath(shutil.which('true')), link) + os.geteuid = lambda: real + 4242 + try: + verdict, reason, _ = fc.ask(link) + finally: + os.geteuid = geteuid + self.assertEqual(verdict, 'unavailable') + self.assertNotEqual(reason, 'stdout is not JSON') # true ran: the link was followed + self.assertIn(f'uid {real}', reason) + + def test_the_folders_of_the_path_as_written_are_judged_too(self): + body = f': > {self.d}/ran\necho \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n' + good = self.script(body) + open_ = os.path.join(self.d, 'open') + os.mkdir(open_) + os.symlink(good, os.path.join(open_, 'free')) + os.chmod(open_, 0o777) # others may swap the link for one to any program they like + verdict, reason, _ = fc.ask(os.path.join(open_, 'free')) + self.assertEqual((verdict, reason), ('unavailable', f'{open_} may be changed by others')) + self.assertFalse(os.path.exists(f'{self.d}/ran')) + os.chmod(open_, 0o755) + self.assertEqual(fc.ask(os.path.join(open_, 'free'))[0], 'free') + + def test_relative_links_chains_and_loops(self): + good = self.script(f'echo \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n') + os.mkdir(os.path.join(self.d, 'sub'), 0o755) + os.chmod(os.path.join(self.d, 'sub'), 0o755) # whatever the umask + os.symlink('../free', os.path.join(self.d, 'sub', 'one')) + os.symlink('one', os.path.join(self.d, 'sub', 'two')) + os.symlink(os.path.join(self.d, 'sub'), os.path.join(self.d, 'dir')) + self.assertEqual(fc.resolve(os.path.join(self.d, 'dir', 'two')), (good, None)) + self.assertEqual(fc.ask(os.path.join(self.d, 'dir', 'two'))[0], 'free') + os.symlink('b', os.path.join(self.d, 'a')) + os.symlink('a', os.path.join(self.d, 'b')) + verdict, reason, _ = fc.ask(os.path.join(self.d, 'a')) + self.assertEqual(verdict, 'unavailable') + self.assertIn('too many levels of links', reason) + + def test_a_script_it_may_not_end_is_not_waited_for_without_a_limit(self): + s = self.script(f'echo $$ > {self.d}/pid\necho \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n' + 'exec sleep 30 >&-\n') + def refused(*_): + raise PermissionError(1, 'Operation not permitted') + killpg, kill = os.killpg, os.kill + os.killpg = os.kill = refused # a set-uid script: its group is not ours to signal + t = time.monotonic() + try: + verdict = fc.ask(s, timeout=1)[0] + finally: + os.killpg, os.kill = killpg, kill + with open(f'{self.d}/pid') as f: + pid = int(f.read()) + try: + os.kill(pid, 9) + except ProcessLookupError: + pass + self.assertLess(time.monotonic() - t, 10) + self.assertEqual(verdict, 'unavailable') + def test_a_link_repointed_after_the_check_does_not_change_what_runs(self): good = self.script(f': > {self.d}/good-ran\necho \'{{"v": 1, "free": true, "at": "{now_text()}"}}\'\n', name='good') From 3fb86174250b9b0502709133371b60a09ab04b63 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:53:05 +0000 Subject: [PATCH 20/33] fork fleet: seed fix round 2 (rename while sent, deadline, max_mb cap) (D11) - A seed replaced by rename while it streams is sent whole: when the open file has no links left (st_nlink 0) it was replaced, not rewritten, so its ctime is not compared; size and mtime still are. The st_ino compare (fstat of the same fd, it never differed) is gone. - The stall wait restarts per 64 KiB piece written, not per 1 MiB, and a stream has an overall deadline: 600 s plus one second per 256 KiB. - max_mb is at most 16384 (fork-fleet.py SEED_MAX_MB). - Tests: the lens rename test, the lock-owner refusal, every refused file must say its own reason (the future-dated case now fails on round-0's path), the deadline, and a slow but steady reader that is kept. Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 10 ++- .github/scripts/fork-fleet-desk-seed.py | 29 +++++-- .github/scripts/fork-fleet-desk-seed.test.py | 82 +++++++++++++++++--- .github/scripts/fork-fleet.py | 5 +- 4 files changed, 102 insertions(+), 24 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index ca2414b079264..bc28a77eaa928 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -696,11 +696,12 @@ ready-made caches the desktop keeps and its jobs start from: `every_hours` is how often a desktop takes a new one, `max_age_hours` (at least `every_hours`) the age, from when it was made, at which it is dropped, -`max_mb` its largest size. Nothing in the fleet makes a seed: on the always-on +`max_mb` its largest size (at most 16384). Nothing in the fleet makes a seed: on the always-on host the project's own timer or job writes a tar of a cache folder's contents to `/var/lib/fork-fleet/seed-files/POOL/NAME.tar` (a folder per pool, so two -pools never share a file), by a temp file and a rename. A file changed while it -is sent (size, mtime or ctime), dated more than five minutes in the future, older +pools never share a file), by a temp file and a rename. A file changed in place +while it is sent (size, mtime or ctime) is refused; one replaced by a rename while +it is sent is fine (the old one is sent whole). A file dated more than five minutes in the future, older than `max_age_hours` or above `max_mb` is refused. Both folders and the files are root's (or the project's), readable by the desk-link user and writable by no one else (the owner itself is not checked, only that no one else may write); never @@ -712,7 +713,8 @@ own: the same forced command sends every command that starts with `seed` to codes), never to a session. It answers `seeds` with the pool's list and `seed NAME` with that one file, only a name of that list, and a trailer with its size and SHA-256; anything else is refused. One stream per pool at a time; a -desktop that takes nothing for two minutes is cut off and the next one may go. +desktop that takes less than 64 KiB in two minutes, or not the whole file within +ten minutes plus one second per 256 KiB, is cut off and the next one may go. ## Owner steps on the repository diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index 8aebe9fbd05f3..b24b98ea9fed3 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -17,10 +17,12 @@ The forced command sends every command that starts with `seed` here. The host side exits 2 for any other command, 3 while another stream for the pool runs, 4 when the file may not be sent (missing, a link, in a folder or a file others may write, above max_mb, older than max_age_hours or dated in the future, or changed while it was sent: made -in place, not by rename), 1 when the desktop stops reading for STALL seconds (the pool's lock is then let go). +in place, not by rename; a file replaced by rename while it is sent is fine, the old one is sent whole), 1 when the +desktop takes less than WRITE bytes in STALL seconds or the whole file not by its deadline (GRACE seconds plus +MIN_RATE bytes a second): the pool's lock is then let go. Standard library only. """ -import fcntl, hashlib, importlib.util, json, os, signal, stat, sys, time +import fcntl, hashlib, importlib.util, json, math, os, signal, stat, sys, time sys.dont_write_bytecode = True HERE = os.path.dirname(os.path.realpath(__file__)) @@ -35,7 +37,8 @@ def module(name, file): FILES = '/var/lib/fork-fleet/seed-files' # the always-on host's: POOL/NAME.tar, made by the project CONFIG = os.path.join(HERE, 'fork-fleet.json') -PIECE, STALL, AHEAD = 1 << 20, 120, 300 # AHEAD: seconds a file's date may be ahead of now (a clock set back) +PIECE, WRITE, STALL, AHEAD = 1 << 20, 64 << 10, 120, 300 # AHEAD: seconds a file's date may be ahead of now +GRACE, MIN_RATE = 600, 256 << 10 # a stream's deadline: GRACE seconds plus one second per MIN_RATE bytes ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'} @@ -120,7 +123,9 @@ def offer(pool, command, lock_dir, out=1, err=sys.stderr, config=CONFIG, files=F os.close(fd) finally: os.close(lock) - same = ('st_size', 'st_mtime_ns', 'st_ctime_ns', 'st_ino') + # A rename over NAME.tar unlinks the open file (st_nlink 0), which changes its ctime but not its bytes: + # only a file still in place must keep its ctime (a rewrite with the old mtime put back). + same = ('st_size', 'st_mtime_ns') + (('st_ctime_ns',) if after.st_nlink else ()) if sent != st.st_size or any(getattr(after, k) != getattr(st, k) for k in same): raise Refused(f'{name}: it changed while it was sent (made in place, not by rename?)', 4) print(f'seed {name} {sent} {sha} {int(st.st_mtime)}', file=err, flush=True) @@ -140,9 +145,11 @@ def stalled(*_): def send(fd, out, size, stall=None): """Sends at most SIZE bytes of FD (the size that was checked: a file that grows is cut there, and then fails the - changed-file check) to OUT; Refused (1) when OUT takes nothing for STALL seconds.""" + changed-file check) to OUT; Refused (1) when OUT takes less than WRITE bytes in STALL seconds, or not all of it + within GRACE seconds plus SIZE / MIN_RATE.""" stall = STALL if stall is None else stall sha, sent = hashlib.sha256(), 0 + deadline = time.monotonic() + GRACE + size / MIN_RATE old = signal.signal(signal.SIGALRM, stalled) try: while sent < size and (data := os.read(fd, min(PIECE, size - sent))): @@ -150,10 +157,16 @@ def send(fd, out, size, stall=None): sent += len(data) view = memoryview(data) while view: - signal.alarm(stall) # each write that moves anything starts the wait again - view = view[os.write(out, view):] + left = deadline - time.monotonic() + if left <= 0: + raise Stalled + signal.alarm(max(1, min(stall, math.ceil(left)))) # each piece that moves starts the wait again + view = view[os.write(out, view[:WRITE]):] except Stalled: - raise Refused(f'the desktop took nothing for {stall} seconds: stopped', 1) from None + if time.monotonic() >= deadline: + raise Refused(f'the desktop did not take {size} bytes within {GRACE + size / MIN_RATE:.0f} seconds ' + f'(at least {MIN_RATE >> 10} KiB a second): stopped', 1) from None + raise Refused(f'the desktop took less than {WRITE >> 10} KiB in {stall} seconds: stopped', 1) from None finally: signal.alarm(0) signal.signal(signal.SIGALRM, old) diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index c237f4173a99e..8171f0d8ca14e 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -2,7 +2,8 @@ """Tests of fork-fleet-desk-seed.py (a desk pool's warm caches) and the `seeds` check of fork-fleet.json: the host's offer in-process and the forced command's divert in fork-fleet-desk.py. Temp folders only; no root, no ssh, no network. Run: python3 -I -B . Linux only.""" -import fcntl, hashlib, importlib.util, io, json, os, shutil, subprocess, sys, tarfile, tempfile, time, unittest +import fcntl, hashlib, importlib.util, io, json, os, shutil, subprocess, sys, tarfile, tempfile, threading, time +import unittest from unittest import mock sys.dont_write_bytecode = True @@ -76,9 +77,11 @@ def problems(seeds): return [p for p in ff.check_config(cfg) if 'seed' in p] self.assertEqual(problems(SEEDS), []) self.assertEqual(problems([]), []) + self.assertEqual(problems([{**SEEDS[0], 'max_mb': ff.SEED_MAX_MB}]), []) good = SEEDS[0] for bad in ({}, [{**good, 'name': 'M2'}], [{**good, 'name': '../x'}], [{**good, 'every_hours': 0}], [{**good, 'max_mb': True}], [{**good, 'max_age_hours': 5}], [{**good, 'max_mb': '2'}], + [{**good, 'max_mb': ff.SEED_MAX_MB + 1}], [{k: v for k, v in good.items() if k != 'max_mb'}], ['m2'], [good, good]): with self.subTest(bad): self.assertEqual(len(problems(bad)), 1, bad) @@ -117,15 +120,16 @@ def test_other_commands_and_names_are_refused_and_send_nothing(self): def test_a_file_that_may_not_be_sent_is_exit_4(self): top, files = os.path.join(self.t, 'files'), os.path.join(self.t, 'files', POOL) now = time.time() - cases = {'missing': lambda: None, - 'a link': lambda: os.symlink(os.path.join(self.t, 'src', 'x.tar'), f'{files}/m2.tar'), - 'group-writable': lambda: (self.make('m2', {'a': b'x'}), os.chmod(f'{files}/m2.tar', 0o664)), - 'too big': lambda: self.make('m2', {'a': os.urandom(3 << 20)}), - 'too old': lambda: self.make('m2', {'a': b'x'}, mtime=now - 1001 * 3600), + cases = {'not made yet': lambda: None, + 'Too many levels of symbolic links': lambda: os.symlink(os.path.join(self.t, 'src', 'x.tar'), + f'{files}/m2.tar'), + 'others may write it': lambda: (self.make('m2', {'a': b'x'}), os.chmod(f'{files}/m2.tar', 0o664)), + 'above max_mb': lambda: self.make('m2', {'a': os.urandom(3 << 20)}), + 'older than max_age_hours': lambda: self.make('m2', {'a': b'x'}, mtime=now - 1001 * 3600), 'dated in the future': lambda: self.make('m2', {'a': b'x'}, mtime=now + 10 * 365 * 86400), - 'the files folder others write': lambda: (self.make('m2', {'a': b'x'}), os.chmod(top, 0o777)), - 'a folder': lambda: os.mkdir(f'{files}/m2.tar'), - 'folder others write': lambda: (self.make('m2', {'a': b'x'}), os.chmod(files, 0o777))} + f'{top} must be a folder': lambda: (self.make('m2', {'a': b'x'}), os.chmod(top, 0o777)), + 'not a plain file': lambda: os.mkdir(f'{files}/m2.tar'), + f'{files} must be a folder': lambda: (self.make('m2', {'a': b'x'}), os.chmod(files, 0o777))} with open(os.path.join(self.t, 'src', 'x.tar'), 'wb') as f: f.write(b'x') os.chmod(os.path.join(self.t, 'src', 'x.tar'), 0o644) # only the link itself may be the reason @@ -134,6 +138,7 @@ def test_a_file_that_may_not_be_sent_is_exit_4(self): make() code, out, err = self.offer('seed m2') self.assertEqual((code, out), (4, b''), err) + self.assertIn(name, err) # refused for this reason, read where the pool's files are os.chmod(files, 0o755) os.chmod(top, 0o755) p = f'{files}/m2.tar' @@ -163,6 +168,25 @@ def changing(fd, out, size): self.assertIn('changed while it was sent', err) self.assertEqual(len(out), os.stat(path).st_size - 512) # cut at the size that was checked + def test_a_rename_while_sent_is_fine(self): + old, send = self.make('m2', {'a': b'x' * 3000}), seed.send + + def renaming(fd, out, size): + self.make('m2', {'a': b'y' * 3000}) # the documented refresh: a temp file, then a rename + return send(fd, out, size) + with mock.patch.object(seed, 'send', renaming): + code, out, err = self.offer('seed m2') + self.assertEqual((code, out), (0, old), err) + self.assertTrue(err.startswith(f'seed m2 {len(old)} {hashlib.sha256(old).hexdigest()} '), err) + + def test_a_lock_of_another_user_is_refused(self): + self.make('m2', {'a': b'x'}) + open(os.path.join(self.t, 'run', f'{POOL}.seed.lock'), 'a').close() + with mock.patch.object(seed.os, 'geteuid', return_value=os.geteuid() + 4242): + code, out, err = self.offer('seed m2') + self.assertEqual((code, out), (1, b''), err) + self.assertIn(f'belongs to uid {os.geteuid()}', err) + def test_a_same_size_rewrite_with_its_old_date_put_back_is_exit_4(self): self.make('m2', {'a': b'x' * 3000}) path, send = self.tar('m2'), seed.send @@ -192,10 +216,48 @@ def test_a_desktop_that_stops_reading_lets_the_lock_go(self): os.close(r) os.close(w) self.assertEqual(code, 1) - self.assertIn('took nothing for 1 seconds', err.getvalue()) + self.assertIn('took less than 64 KiB in 1 seconds', err.getvalue()) self.make('m2', {'a': b'x'}) # small enough for the pipe: offer() below has no reader until it returns self.assertEqual(self.offer('seed m2')[0], 0) # the pool's lock was let go + def test_a_slow_desktop_meets_the_deadline(self): + self.make('m2', {'a': os.urandom(1 << 20)}) + r, w = os.pipe() # nobody reads r; the stall wait alone (100 s) would hold the lock much longer + err = io.StringIO() + try: + with mock.patch.multiple(seed, STALL=100, GRACE=1, MIN_RATE=1 << 40): + t = time.monotonic() + code = seed.offer(POOL, 'seed m2', os.path.join(self.t, 'run'), out=w, err=err, + config=os.path.join(self.t, 'fork-fleet.json'), files=os.path.join(self.t, 'files')) + self.assertLess(time.monotonic() - t, 30) + finally: + os.close(r) + os.close(w) + self.assertEqual(code, 1, err.getvalue()) + self.assertIn('within 1 seconds', err.getvalue()) + + def test_a_desktop_reading_slowly_but_steadily_is_not_dropped(self): + data = self.make('m2', {'a': os.urandom(300 << 10)}) + r, w = os.pipe() + got = bytearray() + + def reader(): + with os.fdopen(r, 'rb', buffering=0) as f: + while chunk := f.read(16 << 10): # 16 KiB, then a pause: under a stall wait per 64 KiB piece + got.extend(chunk) + time.sleep(0.3) + th = threading.Thread(target=reader) + th.start() + err = io.StringIO() + try: + with mock.patch.object(seed, 'STALL', 2): + code = seed.offer(POOL, 'seed m2', os.path.join(self.t, 'run'), out=w, err=err, + config=os.path.join(self.t, 'fork-fleet.json'), files=os.path.join(self.t, 'files')) + finally: + os.close(w) + th.join(60) + self.assertEqual((code, bytes(got)), (0, data), err.getvalue()) + def test_pools_never_share_a_seed_file(self): cfg = json.loads(json.dumps(CONFIG)) cfg['pools'][0]['seeds'] = SEEDS diff --git a/.github/scripts/fork-fleet.py b/.github/scripts/fork-fleet.py index bec1f34d7718d..2581a950e3cfb 100644 --- a/.github/scripts/fork-fleet.py +++ b/.github/scripts/fork-fleet.py @@ -55,6 +55,7 @@ VARIABLE = re.compile(r'(?!GITHUB_)[A-Z][A-Z0-9_]{0,99}') JOB_ID = re.compile(r'[a-z0-9_]{1,40}') POOL = re.compile(r'[a-z0-9][a-z0-9_-]{0,39}') +SEED_MAX_MB = 16384 # a seed's largest max_mb: 16 GiB KINDS = ('push', 'draft', 'ready') # a pool's `takes`: pushes, draft pull requests, pull requests ready for review # Events whose code and workflow are this repository's own (a pull request event: only when head and base are too). OWN_EVENTS = ('push', 'workflow_dispatch', 'schedule') @@ -143,9 +144,9 @@ def check_config(cfg): if not (isinstance(seeds, list) and all( isinstance(x, dict) and isinstance(x.get('name'), str) and POOL.fullmatch(x['name']) and all(positive(x.get(k)) for k in ('every_hours', 'max_age_hours', 'max_mb')) - and x['max_age_hours'] >= x['every_hours'] for x in seeds)): + and x['max_age_hours'] >= x['every_hours'] and x['max_mb'] <= SEED_MAX_MB for x in seeds)): out.append(f'{who}: seeds must list {{"name" (a-z, 0-9, _, -), "every_hours", "max_age_hours" (at least ' - 'every_hours), "max_mb"}, each a positive whole number') + f'every_hours), "max_mb" (at most {SEED_MAX_MB})}}, each a positive whole number') elif len({x['name'] for x in seeds}) != len(seeds): out.append(f'{who}: two seeds with one name') if not positive(p.get('slots')): From 223795c267837ca1bbddb9d4ce22acb44508f25c Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:05:59 +0000 Subject: [PATCH 21/33] fork fleet: seed fetch round 2: entries count against max_mb, one seed never stops the run (D11b) - guarded(): each member's size plus 4 KiB for it and each folder it makes, within max_mb; at most 200,000 members and folders; no member over 64 folders deep or with a name/link over 1 KiB - run(): any error in one seed is said and the run goes on; prune runs in a finally, per version - remove(): folder by folder through dir fds, no recursion (a tree of any depth) - the trailer's bytes and MADE must be the list's - a broken or pool-less own config still drops seeds past their label's maxAgeHours, then exits 2 - an older or half-installed host (no list) means no seeds: tested - desk-install check: a row for the seed service's last result - ssh's stderr pipe closed; the 0600 test fails when nothing was fetched Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 16 +- .github/scripts/fork-fleet-desk-seed.py | 247 +++++++++++++------ .github/scripts/fork-fleet-desk-seed.test.py | 169 ++++++++++++- fork-kit/desk/README.md | 3 +- fork-kit/desk/sbin/desk-install | 3 + fork-kit/desk/tests/install.test.py | 9 +- 6 files changed, 364 insertions(+), 83 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index dd94a0ce8debf..819c8a6fa4c6d 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -721,10 +721,20 @@ The desktop's side is the same script's `run`, hourly from names its own installed fleet config lists for its pool, and holds them to its own `every_hours` and `max_age_hours` and the smaller of its own and the host's `max_mb`: a host cannot make a desktop take more. Each new seed is -unpacked with Python's tar data filter (members' sizes and count limited -too), checked against the trailer, checked again for links that lead out of +unpacked with Python's tar data filter and a limit on what it would make on +disk (each member's size plus 4 KiB for it and for each folder it makes, all +within `max_mb`; at most 200,000 members and folders; no member more than 64 +folders deep or with a name over 1 KiB), checked against the trailer (its size +and date must be the list's too), checked again for links that lead out of it, made readable by every job (a cache's 0600 files too), and swapped in by a -rename; ssh's stderr and the list are read with a limit. Jobs see +rename; ssh's stderr and the list are read with a limit. A seed that fails in +any way is named in the log and the run goes on with the next; old copies +are pruned in every run. A host that gives no list (down, an older kit, a +half-done install) means no new seeds: each one here stays until its age +drops it. A desktop whose own fleet config is broken or lacks its pool fetches +nothing and exits 2 (`desk-install check` shows the seed service's last +result), but still drops each seed past the `max_age_hours` it was fetched +with. Jobs see `/var/lib/fork-fleet/seeds/NAME` read-only (the launcher's `ALLOW_PATHS`; the unpacking and old copies beside it, `.new` and `.v`, are visible too and hold the same data), copy what they need from it when it is there, and run diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index 5df3b3364ae36..7eac817cb4f03 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -17,7 +17,9 @@ the desktop, as root, hourly (t3code-fleet-seed.timer), with the agent's settings (FILE, default its CONF). Only names its own installed fleet config lists for its POOL, held to its own every_hours and max_age_hours and the smaller of its own and the host's max_mb. Each new seed is - unpacked with tarfile's data filter into a folder of its own, checked against the trailer, walked + unpacked with tarfile's data filter into a folder of its own (what it would make on disk held + to max_mb, ENTRY bytes counted for each member and folder; no member over DEPTH folders deep or + a name over NAME_MOST bytes), checked against the trailer (bytes and MADE the list's too), walked (a link out of it is refused; all made readable), flushed, and swapped in by rename: SEEDS/NAME is a link to .v/NAME.STAMP/data, root's (folders 0755, files no one else may write), unpacked in .new (one folder: a rename never crosses a mount of the unit's sandbox). Jobs see SEEDS read-only @@ -27,13 +29,20 @@ Each run (one at a time: a lock) meets this state: the same seed on the host (bytes and MADE) nothing is fetched a new one under every_hours after the last nothing yet - a torn copy (the stream cut or past FETCH_WAIT, a trailer that does not match, more than max_mb, a tar member - that would leave its folder or is a device) the copy is removed, the old seed stays, the next hour tries again + a torn copy (the stream cut or past FETCH_WAIT, a trailer that does not match it or the list, more than max_mb, + a tar member that would leave its folder, is a device, too deep or too long, or any other error) + the copy is removed, the old seed stays, the next hour tries again; + the other seeds go on, old versions are still pruned no seed (never fetched, the host down) jobs run cold + no list (the host down, an older kit that answers with a session or not JSON, exit 1 of a half-done install) + nothing fetched; each seed stays until its age drops it + its own fleet config broken or without POOL nothing fetched, exit 2; a seed older than its label's maxAgeHours + (its own max_age_hours when it was fetched) is still dropped older than max_age_hours, made over SKEW in the future (the clocks disagree), or no longer in the pool's list the link is removed: jobs run cold a label that is missing or unreadable the link is removed and the seed fetched again in the same run - a run that stopped half-way SEEDS/.new is emptied by the next run; a link is the old one or the new + a run that stopped half-way SEEDS/.new is emptied by the next run (folder by folder, a tree of + any depth); a link is the old one or the new a replaced version kept KEEP_OLD after it was replaced, for a job still reading it (a job runs at most the launcher's MAX_SECONDS, 1 h by default), then removed @@ -45,7 +54,7 @@ MIN_RATE bytes a second): the pool's lock is then let go. Standard library only. """ -import argparse, fcntl, hashlib, importlib.util, json, math, os, re, shutil, signal, stat, subprocess, sys +import argparse, fcntl, hashlib, importlib.util, json, math, os, re, signal, stat, subprocess, sys import tarfile, threading, time sys.dont_write_bytecode = True @@ -69,6 +78,7 @@ def module(name, file): KEEP_OLD, SKEW = 24 * 3600, 3600 # SKEW: how far the host's clock may be ahead of the desktop's PIECE, WRITE, STALL, AHEAD = 1 << 20, 64 << 10, 120, 300 # AHEAD: seconds a file's date may be ahead of now GRACE, MIN_RATE = 600, 256 << 10 # a stream's deadline: GRACE seconds plus one second per MIN_RATE bytes +DEPTH, NAME_MOST, ENTRY = 64, 1024, 4096 # a member's most folders deep and name bytes; the disk an entry costs ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'} @@ -243,6 +253,12 @@ def run(self): while chunk := os.read(self.f.fileno(), 65536): self.data = (self.data + chunk)[-ERR_KEEP:] + def close(self): + """The pipe closed once its reader is done (a child that keeps it open past 5 s keeps it, and the thread).""" + self.join(5) + if not self.is_alive(): + self.f.close() + def text(self): self.join(5) return ' '.join(self.data.decode('utf-8', 'replace').split())[-300:] or 'nothing on stderr' @@ -259,8 +275,8 @@ def listing(agent, conf): timer = threading.Timer(LIST_WAIT, lambda: (late.set(), p.kill())) timer.start() out = b'' + err = Tail(p.stderr) try: - err = Tail(p.stderr) while len(out) <= LIST_MOST and (chunk := os.read(p.stdout.fileno(), 65536)): out += chunk finally: @@ -268,6 +284,7 @@ def listing(agent, conf): p.kill() code = p.wait() p.stdout.close() + err.close() if late.is_set(): raise Torn(f'no list in {LIST_WAIT} s') if len(out) > LIST_MOST: @@ -297,15 +314,31 @@ def whole(x): def guarded(most): - """tarfile's data filter, and a limit on what the tar says it holds: members' sizes together at most MOST - bytes (a sparse member is small in the stream, not on disk) and at most MEMBERS members (each is kept in memory).""" - seen = [0, 0] + """tarfile's data filter, and a limit on what the tar would make on disk: each member's size (a sparse member is + small in the stream, not on disk) plus ENTRY bytes for it and for each folder it makes, together at most MOST + bytes (a stream of empty files or deep paths is small but would fill the disk with entries); at most MEMBERS + members and MEMBERS folders (each is kept in memory); no member more than DEPTH folders deep (unpacking and + removing a tree goes folder by folder) or with a name or link target over NAME_MOST bytes.""" + seen, folders = [0, 0], {} # folders: {(its parent's number, its name): its number}, a name kept once def check(member, path): + parts = [x for x in member.name.split('/') if x not in ('', '.')] + if len(parts) > DEPTH: + raise Torn(f'{"/".join(parts[:3])}/... is more than {DEPTH} folders deep') + for text in (member.name, member.linkname or ''): + if len(text.encode('utf-8', 'surrogateescape')) > NAME_MOST: + raise Torn(f'a member\'s name or link target is over {NAME_MOST} bytes: {text[:80]!r}...') + new = up = 0 + for i, part in enumerate(parts if member.isdir() else parts[:-1], 1): # the folders it makes + if (up, part) not in folders: + folders[up, part] = len(folders) + 1 + new += i < len(parts) # a folder member pays ENTRY once, as a member + up = folders[up, part] seen[0] += 1 - seen[1] += member.size - if seen[0] > MEMBERS or seen[1] > most: - raise Torn(f'the tar holds more than {MEMBERS} members or {most >> 20} MB (max_mb)') + seen[1] += member.size + ENTRY * (1 + new) + if seen[0] > MEMBERS or len(folders) > MEMBERS or seen[1] > most: + raise Torn(f'the tar holds more than {MEMBERS} members or folders, or more than {most >> 20} MB ' + f'(max_mb; {ENTRY} bytes counted for each member and folder)') return tarfile.data_filter(member, path) return check @@ -341,8 +374,8 @@ def fetch(agent, conf, item, root, now): stderr=subprocess.PIPE, env=agent.ENV, start_new_session=True) timer = threading.Timer(FETCH_WAIT, p.kill) timer.start() + err = Tail(p.stderr) try: - err = Tail(p.stderr) got = Summed(p.stdout, most) with tarfile.open(fileobj=got, mode='r|') as t: t.extractall(os.path.join(new, 'data'), filter=guarded(most)) @@ -354,11 +387,14 @@ def fetch(agent, conf, item, root, now): p.kill() p.wait() p.stdout.close() - err.join(5) + err.close() last = (err.data.decode('utf-8', 'replace').strip().splitlines() or [''])[-1].split() if code or len(last) != 5 or last[:4] != ['seed', name, str(got.size), got.sha.hexdigest()] or \ not re.fullmatch(r'[0-9]{1,20}', last[4]): raise Torn(f'exit {code}, {got.size} bytes, the trailer does not match: {err.text()}') + if (got.size, int(last[4])) != (item['bytes'], item['made']): # MADE decides the age drop and every_hours + raise Torn(f'the trailer does not match the list ({got.size} bytes made {last[4]}, the list said ' + f'{item["bytes"]} made {item["made"]}): the host made a new one since') settle(os.path.join(new, 'data')) label = {'v': 1, 'name': name, 'bytes': got.size, 'sha256': got.sha.hexdigest(), 'made': int(last[4]), 'fetched': int(now()), 'maxAgeHours': item['maxAgeHours']} @@ -397,14 +433,41 @@ def current(root, name): def remove(path): + """PATH, and when it is a folder all it holds, folder by folder without recursion: a tree of any depth (one an + older release unpacked, before DEPTH) never stops a run with RecursionError. Links are removed, never followed.""" try: st = os.lstat(path) except FileNotFoundError: return - if stat.S_ISDIR(st.st_mode): - shutil.rmtree(path) - else: + if not stat.S_ISDIR(st.st_mode): os.unlink(path) + return + flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_CLOEXEC + below, fd = [], os.open(path, flags) + try: + while True: + sub = None + with os.scandir(fd) as entries: + for e in entries: + if e.is_dir(follow_symlinks=False): + sub = e.name + break + os.unlink(e.name, dir_fd=fd) + if sub is not None: # down into it; back up once it is empty + child = os.open(sub, flags, dir_fd=fd) + os.close(fd) + fd = child + below.append(sub) + elif below: + up = os.open('..', flags, dir_fd=fd) + os.close(fd) + fd = up + os.rmdir(below.pop(), dir_fd=fd) + else: + break + finally: + os.close(fd) + os.rmdir(path) def drop(root, name, version, now): @@ -420,15 +483,22 @@ def own_seeds(conf, config): return pool_seeds(conf['POOL'], config) except Refused as e: raise Torn(f'its own {e}') from None + except Exception as e: # a config the fleet's own loader trips on + raise Torn(f'its own fleet config: {e.__class__.__name__}: {e}') from None def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): + """One run (the module's docstring). A seed that fails in any way, a deep tree included, is said and the run + goes on with the next; old versions are pruned in every run. When it may fetch nothing (no data filter, its own + fleet config broken or without its pool) it still drops each seed past its age, by its label's max_age_hours, + and then says why (agent.Bad: exit 2, the unit's last result shows it).""" + cannot, own = None, None if not hasattr(tarfile, 'data_filter'): # releases with known filter bypasses pass this: settle() checks again - raise agent.Bad('this Python has no tarfile data filter (3.12, or 3.8.17 and later): no seeds') + cannot = 'this Python has no tarfile data filter (3.12, or 3.8.17 and later)' try: own = own_seeds(conf, config) except Torn as e: - raise agent.Bad(f'{e}: no seeds') from None + cannot = cannot or str(e) os.umask(0o022) for d in (root, root + '/.new', root + '/.v'): # root: made by the unit (StateDirectory) or here os.makedirs(d, 0o755, exist_ok=True) @@ -438,66 +508,99 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): except OSError: say('another seed run is running') return 1 - for e in os.listdir(root + '/.new'): # a run that stopped half-way - if e != '.lock': - remove(os.path.join(root + '/.new', e)) - for e in os.listdir(root): # a link made but not yet swapped in - if e.startswith('.') and e.endswith('.link'): - remove(os.path.join(root, e)) try: - host = listing(agent, conf) - items = {n: {**host[n], 'everyHours': s['every_hours'], 'maxAgeHours': s['max_age_hours'], - 'maxMb': min(host[n]['maxMb'], s['max_mb'])} for n, s in own.items() if n in host} - except Torn as e: - items = None - say(f'seeds: {e}: each seed stays until its age drops it') - for name in sorted(n for n in os.listdir(root) if NAME.fullmatch(n)): - version, label = current(root, name) - if label is None: - why = 'its label is missing or cannot be read (fetched again now)' - elif name not in own: - why = 'no longer one of the pool\'s seeds' - else: - age = now() - label['made'] - hours = own[name]['max_age_hours'] - why = (f'older than {hours} h (max_age_hours)' if age > hours * 3600 - else 'made in the future (the clocks disagree)' if age < -SKEW else None) - if why: - drop(root, name, version, now) - say(f'seed {name}: dropped, {why}; jobs run cold until a new one comes') - for name, item in sorted((items or {}).items()): - label = current(root, name)[1] - if 'made' not in item: - say(f'seed {name}: the host offers none ({item.get("why", "no reason")}): ' - f'{"the one here stays" if label else "jobs run cold"}') - continue - if label and (label['bytes'], label['made']) == (item['bytes'], item['made']): - continue - if label and 0 <= now() - label['fetched'] < item['everyHours'] * 3600: - continue - if not -SKEW <= now() - item['made'] <= item['maxAgeHours'] * 3600: - say(f'seed {name}: the host\'s is too old or made in the future (the clocks disagree): not fetched') - continue + clean(root, say) + for e in os.listdir(root): # a link made but not yet swapped in + if e.startswith('.') and e.endswith('.link'): + each(say, f'seed link {e}', remove, os.path.join(root, e)) + for name in sorted(n for n in os.listdir(root) if NAME.fullmatch(n)): + each(say, f'seed {name}', age_out, root, name, own, now, say) + if cannot: + raise agent.Bad(f'{cannot}: no seeds fetched (each seed here stays until its age drops it)') try: - got = fetch(agent, conf, item, root, now) - say(f'seed {name}: {got["bytes"] >> 20} MB, sha256 checked, swapped in') - except (Torn, OSError, ValueError, tarfile.TarError, subprocess.SubprocessError) as e: - for e2 in os.listdir(root + '/.new'): - if e2 != '.lock': - remove(os.path.join(root + '/.new', e2)) - say(f'seed {name}: keeps {"the old one" if label else "none"}: {e.__class__.__name__}: ' - f'{str(e)[:300]}; tried again next hour') - prune(root, now) + host = listing(agent, conf) + items = {n: {**host[n], 'everyHours': s['every_hours'], 'maxAgeHours': s['max_age_hours'], + 'maxMb': min(host[n]['maxMb'], s['max_mb'])} for n, s in own.items() if n in host} + except Torn as e: # the host down, or a host on an older kit (no list, or not one this reads) + items = {} + say(f'seeds: {e}: each seed stays until its age drops it') + for name, item in sorted(items.items()): + if not each(say, f'seed {name}', take, agent, conf, item, root, now, say): + clean(root, say) + finally: + prune(root, now, say) return 0 -def prune(root, now): - """Versions no link names, KEEP_OLD after they were replaced or dropped.""" - linked = {current(root, n)[0] for n in os.listdir(root) if NAME.fullmatch(n)} - for e in os.listdir(root + '/.v'): - path = os.path.join(root, '.v', e) +def each(say, what, step, *args): + """step(*args); any Exception is said, not raised: one seed never stops the others. False when it failed.""" + try: + return step(*args) is not False + except Exception as e: + say(f'{what}: {e.__class__.__name__}: {str(e)[:300]}; tried again next hour') + return False + + +def clean(root, say): + """SEEDS/.new emptied but for the lock: a run (or a fetch) that stopped half-way.""" + for e in os.listdir(root + '/.new'): + if e != '.lock': + each(say, f'seeds: removing .new/{e}', remove, os.path.join(root + '/.new', e)) + + +def age_out(root, name, own, now, say): + version, label = current(root, name) + if label is None: + why = 'its label is missing or cannot be read (fetched again now)' + elif own is not None and name not in own: + why = 'no longer one of the pool\'s seeds' + else: + age = now() - label['made'] + hours = own[name]['max_age_hours'] if own is not None else label['maxAgeHours'] # own config broken + why = (f'older than {hours} h (max_age_hours)' if age > hours * 3600 + else 'made in the future (the clocks disagree)' if age < -SKEW else None) + if why: + drop(root, name, version, now) + say(f'seed {name}: dropped, {why}; jobs run cold until a new one comes') + + +def take(agent, conf, item, root, now, say): + """The listed seed `item`, fetched when it is new and due. False when its fetch failed (the old one stays).""" + name = item['name'] + label = current(root, name)[1] + if 'made' not in item: + say(f'seed {name}: the host offers none ({item.get("why", "no reason")}): ' + f'{"the one here stays" if label else "jobs run cold"}') + return + if label and (label['bytes'], label['made']) == (item['bytes'], item['made']): + return + if label and 0 <= now() - label['fetched'] < item['everyHours'] * 3600: + return + if not -SKEW <= now() - item['made'] <= item['maxAgeHours'] * 3600: + say(f'seed {name}: the host\'s is too old or made in the future (the clocks disagree): not fetched') + return + try: + got = fetch(agent, conf, item, root, now) + except Exception as e: # any: a torn stream, a full disk, a tar this Python cannot unpack + say(f'seed {name}: keeps {"the old one" if label else "none"}: {e.__class__.__name__}: {str(e)[:300]}; ' + f'tried again next hour') + return False + say(f'seed {name}: {got["bytes"] >> 20} MB, sha256 checked, swapped in') + + +def prune(root, now, say=print): + """Versions no link names, KEEP_OLD after they were replaced or dropped; one that cannot go is said.""" + def old(path): if path not in linked and now() - os.lstat(path).st_mtime > KEEP_OLD: remove(path) + try: + linked = {current(root, n)[0] for n in os.listdir(root) if NAME.fullmatch(n)} + names = os.listdir(root + '/.v') + except OSError as e: + say(f'seeds: no prune: {e}') + return + for e in names: + each(say, f'seeds: old version {e}', old, os.path.join(root, '.v', e)) def main(argv=None): diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 822f4f33e85b2..5aa1464ed2a58 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -2,7 +2,7 @@ """Tests of fork-fleet-desk-seed.py (a desk pool's warm caches) and the `seeds` check of fork-fleet.json: the host's offer in-process, the desktop's run end to end through a fake ssh that runs the host's side, and the forced command's divert in fork-fleet-desk.py. Temp folders only; no root, no ssh, no network. Run: python3 -I -B . Linux only.""" -import fcntl, hashlib, importlib.util, io, json, os, shutil, stat, subprocess, sys, tarfile, tempfile, threading +import contextlib, fcntl, hashlib, importlib.util, io, json, os, shutil, stat, subprocess, sys, tarfile, tempfile, threading import time, unittest from unittest import mock @@ -44,13 +44,19 @@ def load(name, file): if mode == 'down': print('ssh: connect to host h port 22: No route to host', file=sys.stderr) sys.exit(255) +if words[0] == 'seeds' and mode == 'oldhost': # a host on an older kit: a serve session, not a list + print('fork-fleet-desk: serving') + sys.exit(0) +if words[0] == 'seeds' and mode == 'exit1': # a host stopped half-way through its install + print('fork-fleet-desk: no config', file=sys.stderr) + sys.exit(1) kw = dict(config={t!r} + '/fork-fleet.json', files={t!r} + '/files') if words[0] == 'seeds' and mode == 'biglist': os.write(1, b' ' * (2 << 20)) if mode == 'noisy': for _ in range(64): os.write(2, b'x' * (1 << 16) + b'\\n') -if words[0] == 'seed' and mode in ('cut', 'trailer', 'odd'): +if words[0] == 'seed' and mode in ('cut', 'trailer', 'odd', 'made1'): with open({t!r} + '/sent', 'w+b') as out: err = io.StringIO() code = seed.offer({pool!r}, ' '.join(words), {t!r} + '/run', out=out.fileno(), err=err, **kw) @@ -62,6 +68,8 @@ def load(name, file): os.write(1, data) if mode == 'odd': # a MADE that str.isdigit() takes but int() does not print(err.getvalue().rsplit(' ', 1)[0] + ' \u00b2', file=sys.stderr) + elif mode == 'made1': # bytes and sha256 right, MADE not the list's + print(err.getvalue().rsplit(' ', 1)[0] + ' 1', file=sys.stderr) else: print(err.getvalue().rsplit(' ', 2)[0] + ' ' + '0' * 64 + ' 1', file=sys.stderr) # a wrong sha256 sys.exit(code) @@ -72,7 +80,7 @@ def load(name, file): class Base(unittest.TestCase): def setUp(self): self.t = os.path.realpath(tempfile.mkdtemp(prefix='seed-test-')) - self.addCleanup(shutil.rmtree, self.t, True) + self.addCleanup(seed.remove, self.t) # a tree of any depth for d in ('files', f'files/{POOL}', 'run', 'src'): os.mkdir(os.path.join(self.t, d), 0o755) self.config(SEEDS) @@ -422,7 +430,8 @@ def test_fetched_checked_and_swapped_in_then_not_again(self): st = os.lstat(os.path.join(d, n)) if not stat.S_ISLNK(st.st_mode): # a link's own mode is always 0777 and means nothing self.assertFalse(st.st_mode & 0o022, os.path.join(d, n)) - label = json.load(open(os.path.join(seed.current(self.root, 'm2')[0], 'label.json'))) + with open(os.path.join(seed.current(self.root, 'm2')[0], 'label.json')) as f: + label = json.load(f) self.assertEqual(label['bytes'], os.path.getsize(self.tar('m2'))) self.assertEqual(self.run_seed(), 0) # the same seed on the host: nothing fetched self.assertEqual(self.fetches(), ['seed m2']) @@ -514,6 +523,7 @@ def test_stderr_is_never_kept_whole_and_a_huge_list_is_refused(self): def test_a_0600_file_is_readable_by_jobs(self): self.make('m2', {'cache/entry': b'x', 'cache/tool': b'#!/bin/sh\n'}, mode=0o600) self.run_seed() + self.assertEqual(self.files('m2'), {'cache/entry': b'x', 'cache/tool': b'#!/bin/sh\n'}, self.said) top = os.path.realpath(os.path.join(self.root, 'm2')) for d, dirs, names in os.walk(top): for n in dirs + names: @@ -572,6 +582,157 @@ def evil(): for d, dirs, names in os.walk(self.t): self.assertNotIn('escaped', dirs + names, d) + def deep_tar(self, name, depth, count=1, size=0): + """FILES/POOL/NAME.tar of COUNT members d{i}/x/x/.../f, DEPTH folders deep, newer than any made before.""" + tmp = os.path.join(self.t, 'files', POOL, f'.{name}.tmp') + with tarfile.open(tmp, 'w', format=tarfile.GNU_FORMAT) as tar: + for i in range(count): + info = tarfile.TarInfo(f'd{i}/' + 'x/' * depth + 'f') + info.size = size + tar.addfile(info, io.BytesIO(b'z' * size)) + os.utime(tmp, (time.time() + 5,) * 2) + os.chmod(tmp, 0o644) + os.replace(tmp, self.tar(name)) + + def deep_folder(self, top, depth): + """TOP/x/x/.../f, DEPTH folders deep, made folder by folder (a path that long is past PATH_MAX).""" + os.makedirs(top) + fd = os.open(top, os.O_RDONLY | os.O_DIRECTORY) + try: + for _ in range(depth): + os.mkdir('x', dir_fd=fd) + child = os.open('x', os.O_RDONLY | os.O_DIRECTORY, dir_fd=fd) + os.close(fd) + fd = child + os.close(os.open('f', os.O_WRONLY | os.O_CREAT, 0o600, dir_fd=fd)) + finally: + os.close(fd) + + def test_a_deep_path_does_not_stop_the_run(self): + self.make('npm', {'a': b'n'}) + self.deep_tar('m2', 1500) + self.assertEqual(self.run_seed(), 0) + self.assertEqual(self.files('npm'), {'a': b'n'}) + self.assertIsNone(self.files('m2')) + self.assertTrue(any('m2' in s and 'folders deep' in s for s in self.said), self.said) + self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) + + def test_any_failure_of_one_seed_never_stops_the_run_and_old_versions_are_still_pruned(self): + os.makedirs(os.path.join(self.root, '.v', 'm2.1', 'data')) # a version no link names, long replaced + os.utime(os.path.join(self.root, '.v', 'm2.1'), (1, 1)) + self.make('npm', {'a': b'n'}) + self.deep_tar('m2', 1500) + real = seed.fetch + + def broken(agent_, conf, item, root, now): + if item['name'] == 'm2': + raise RuntimeError('anything at all') + return real(agent_, conf, item, root, now) + with mock.patch.object(seed, 'fetch', broken): + self.assertEqual(self.run_seed(), 0) + self.assertEqual(self.files('npm'), {'a': b'n'}) + self.assertTrue(any('m2' in s and 'RuntimeError' in s for s in self.said), self.said) + self.assertFalse(os.path.exists(os.path.join(self.root, '.v', 'm2.1'))) + # a path deeper than Python's recursion allows, past the depth check: said, cleaned up, the run goes on + shutil.rmtree(self.root) + with mock.patch.object(seed, 'DEPTH', 10 ** 6), mock.patch.object(seed, 'NAME_MOST', 10 ** 6), \ + mock.patch.object(seed, 'ENTRY', 0): + self.assertEqual(self.run_seed(), 0) + self.assertTrue(any('m2' in s and 'RecursionError' in s for s in self.said), self.said) + self.assertEqual(self.files('npm'), {'a': b'n'}) + self.assertIsNone(self.files('m2')) + self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) + + def test_a_tree_deeper_than_recursion_left_by_an_older_run_is_removed(self): + self.make('m2', {'a': b'one'}) + self.run_seed() + self.deep_folder(os.path.join(self.root, '.new', 'm2.5'), 1500) # a torn unpack of an older release + self.deep_folder(os.path.join(self.root, '.v', 'm2.6', 'data'), 1500) # a version it left, long replaced + os.utime(os.path.join(self.root, '.v', 'm2.6'), (1, 1)) + self.assertEqual(self.run_seed(), 0) + self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) + self.assertEqual(os.listdir(os.path.join(self.root, '.v')), + [os.path.basename(seed.current(self.root, 'm2')[0])]) + self.assertEqual(self.files('m2'), {'a': b'one'}) + + def test_folders_count_against_max_mb(self): + self.deep_tar('m2', 900, count=150) + self.run_seed() + self.assertLess(sum(len(d) for _, d, _ in os.walk(self.root)), 100000) + self.assertIsNone(self.files('m2')) + cases = [('1220 folders', 'max_mb', lambda: self.deep_tar('m2', 60, count=20)), + ('600 empty files', 'max_mb', lambda: self.make('m2', {f'f{i}': b'' for i in range(600)}, + mtime=time.time() + 5)), + ('a long name', 'over 1024 bytes', lambda: self.make('m2', {'/'.join(['y' * 120] * 10): b'x'}, + mtime=time.time() + 5))] + for case, why, setup in cases: + with self.subTest(case): + shutil.rmtree(self.root, True) + self.torn(setup, why) + + def test_the_trailers_bytes_and_made_must_be_the_lists(self): + self.torn(lambda: (self.make('m2', {'a': b'two'}, mtime=time.time() + 5), self.mode('made1')), + 'does not match the list') + t0 = time.time() + 7 * 3600 + for h in range(5): # each hour tries again; a copy dated 1 is never swapped in, so none piles up in .v + self.run_seed(now=lambda: t0 + h * 3600) + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertEqual(len(os.listdir(os.path.join(self.root, '.v'))), 1) + os.unlink(os.path.join(self.t, 'mode')) + shutil.rmtree(self.root) + real = seed.listing + + def lying(*a): + out = real(*a) + out['m2'] = {**out['m2'], 'bytes': out['m2']['bytes'] + 512} + return out + lie = mock.patch.object(seed, 'listing', lying) + self.addCleanup(lie.stop) + self.torn(lambda: (self.make('m2', {'a': b'two'}, mtime=time.time() + 5), lie.start()), + 'does not match the list') + + def test_a_broken_or_pool_less_own_config_still_drops_a_too_old_seed(self): + for why in ('broken', 'pool-less'): + with self.subTest(why): + shutil.rmtree(self.root, True) + self.own(SEEDS) + self.make('m2', {'a': b'one'}) + self.run_seed() + with open(os.path.join(self.t, 'own.json'), 'w') as f: + if why == 'broken': + f.write('{') + else: + cfg = json.loads(json.dumps(CONFIG)) + cfg['pools'][0]['name'] = 'not-this-pool' + json.dump(cfg, f) + fetched = len(self.fetches()) + with self.assertRaisesRegex(agent.Bad, 'no seeds fetched'): + self.run_seed(now=lambda: time.time() + 999 * 3600) + self.assertEqual(self.files('m2'), {'a': b'one'}) # under its label's max_age_hours: kept + with self.assertRaises(agent.Bad): + self.run_seed(now=lambda: time.time() + 1001 * 3600) + self.assertIsNone(self.files('m2')) + self.assertTrue(any('older than 1000 h' in s for s in self.said), self.said) + self.assertEqual(len(self.fetches()), fetched) + + def test_an_older_or_half_installed_host_means_no_seeds(self): + for mode in ('oldhost', 'exit1'): + with self.subTest(mode): + shutil.rmtree(self.root, True) + with contextlib.suppress(FileNotFoundError): + os.unlink(os.path.join(self.t, 'mode')) + self.make('m2', {'a': b'one'}) + self.run_seed() + fetched = len(self.fetches()) + self.mode(mode) + self.make('m2', {'a': b'two'}, mtime=time.time() + 5) + self.assertEqual(self.run_seed(now=lambda: time.time() + 999 * 3600), 0) + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertTrue(any('stays until its age drops it' in s for s in self.said), self.said) + self.assertEqual(self.run_seed(now=lambda: time.time() + 1001 * 3600), 0) + self.assertIsNone(self.files('m2')) + self.assertEqual(len(self.fetches()), fetched) + def test_dropped_when_no_longer_listed_or_too_old_or_from_the_future(self): self.make('m2', {'a': b'one'}) self.run_seed() diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 6e23a0e7d88ee..acd39316b7889 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -51,7 +51,8 @@ the refresh timer; `check` says FAIL for `template` until it is there. `check` only reads: the installed files against their manifest, each unit against the release's (header plus `/opt/fork-fleet/units/UNIT`), a restart a stopped install still owes, systemd drop-ins that change the kit's units, -every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings +every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units (and the seed service's +last result), the template, the settings files against the machine file, the runner users (and names a user the kit made that a smaller `slots` no longer uses) and IPv4 forwarding. diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index 1ad128b9a7a18..495feb15643d1 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -507,6 +507,9 @@ def check(): on = run('systemctl', 'is-active', '--quiet', unit, check=False).returncode == 0 row(on and run('systemctl', 'is-enabled', '--quiet', unit, check=False).returncode == 0, 'unit', f'{unit} enabled and active') + last = run('systemctl', 'show', '-p', 'Result', '--value', SEED, check=False).stdout.strip() or 'unknown' + row(last == 'success', 'unit', f'{SEED} last run: {last}' + ('' if last == 'success' else + f' (journalctl -u {SEED} says why)')) row(os.path.exists(p(OPT + '/template/runner-template/.fork-fleet-version')), 'template', 'the runner template is there (the refresh timer makes it)') try: diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index 2fbe79f593d62..5626fd96aa021 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -47,6 +47,8 @@ def put(n, items): open(key + '.pub', 'w').write('ssh-ed25519 AAAAfake desk-x\n') if name == 'systemctl' and args[0] in ('is-active', 'is-enabled'): sys.exit(1 if args[-1] in lines('down') else 0) +if name == 'systemctl' and args[:3] == ['show', '-p', 'Result']: # a unit's last result: the file `result` + print((lines('result') or ['success'])[0]) if name == 'systemctl' and args[0] == 'list-units': # runner units: the file `units`; `stuck` ones never stop print(''.join(f'{u} loaded active running runner\n' for u in lines('units')), end='') if name == 'systemctl' and args[0] == 'stop': @@ -515,6 +517,7 @@ def test_check_names_each_failure(self): self.write(os.path.join(self.fakes, 'wall-fails'), '') self.write(os.path.join(self.fakes, 'agent-bad'), '') self.write(os.path.join(self.fakes, 'down'), 'fork-fleet-desk-agent.service\n') + self.write(os.path.join(self.fakes, 'result'), 'exit-code\n') # the seed service failed its last run self.write(self.path('/opt/fork-fleet/hook.sh'), '#!/bin/bash\nevil\n', 0o755) self.write(self.path('/proc/sys/net/ipv4/ip_forward'), '0\n') os.unlink(os.path.join(self.fakes, 'users')) @@ -523,10 +526,10 @@ def test_check_names_each_failure(self): for line in ('FAIL kit /opt/fork-fleet/hook.sh: differs', 'FAIL vsock closed: open', 'FAIL agent-settings bad settings', 'FAIL unit fork-fleet-desk-agent.service', 'PASS unit t3code-fleet-refresh.timer', 'PASS unit t3code-fleet-seed.timer', - 'FAIL template', 'FAIL runner-user fork-runner-1', - 'FAIL ip-forward'): + 'FAIL unit t3code-fleet-seed.service last run: exit-code', 'FAIL template', + 'FAIL runner-user fork-runner-1', 'FAIL ip-forward'): self.assertIn(line, out) - self.assertTrue(out.endswith('8 FAIL\n'), out) + self.assertTrue(out.endswith('9 FAIL\n'), out) def test_check_fails_on_a_drop_in_and_on_settings_the_machine_file_no_longer_gives(self): self.assertEqual(self.install()[0], 0) From 8c7524fe1b5c5efd55600effc0a0ed7076fea869 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 00:57:01 +0000 Subject: [PATCH 22/33] fork kit desk: fix round 3 (link check, locked runner users, modes, lock, stopped choice) (D8) - --fleet-config: the folders of the link and of the file it names are both checked. - A runner user made by hand must have no login shell, a locked password and no cron.allow/at.allow line: install refuses otherwise, check FAILs. - check looks at modes (files, folders, desk key); install puts a changed mode or owner back from the release. - One install or uninstall at a time (flock on /opt/fork-fleet); a second is refused and does not sweep the first one's staging folder. - The restart mark is set before a new fleet config choice is saved, and check compares the project's copy with the installed one: a run killed between the choice and the swap fails check. - An old record that names a project config that is gone: a readable refusal. - Tests for the runner-user branches of the ownership check. Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 24 +++-- fork-kit/desk/sbin/desk-install | 126 +++++++++++++++++++---- fork-kit/desk/tests/install.test.py | 153 +++++++++++++++++++++++++++- 3 files changed, 276 insertions(+), 27 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 86a0fe756420e..9a18bbbc1d5a2 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -41,30 +41,42 @@ sh bootstrap.sh --repo OWNER/REPO --sha RELEASE_SHA --release-fleet-config # b ``` `--fleet-config` is the project's fleet config (its pools); without it the release's own is used. It must be in a -folder only root (or you) can write. The project's is kept in `/etc/fork-fleet/fleet-config.json` and the choice is +folder only root (or you) can write; a link is followed, and the folders of both the link and the file it names are +checked. The project's is kept in `/etc/fork-fleet/fleet-config.json` and the choice is recorded before any file is swapped, so a later install of a new release keeps it, even after a stopped run; `--release-fleet-config` goes back to the release's own. Install prints the desk key's public half: add it on the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner template is made in the background by the refresh timer; `check` says FAIL for `template` until it is there. -`check` only reads: the installed files against their manifest, each unit against the release's (header plus +`check` only reads: the installed files against their manifest (text, root owner and mode), the folders' and the +desk key's modes, the project's fleet config against the installed one, each unit against the release's (header plus `/opt/fork-fleet/units/UNIT`), a restart a stopped install still owes, systemd drop-ins that change the kit's units, every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings -files against the machine file, the runner users (and names a user the kit made that a smaller `slots` no longer -uses) and IPv4 forwarding. +files against the machine file, the runner users (each must exist, have no login shell, a locked password and no +line in `cron.allow`/`at.allow`; and it names a user the kit made that a smaller `slots` no longer uses) and IPv4 +forwarding. + +A `fork-runner-N` user made by hand before the install is used, but only when it is as locked as one the kit makes +(no login shell, locked password, not in an allow file): otherwise install refuses before changing anything and +names the fix (`usermod -s /usr/sbin/nologin`, `passwd -l`). One install or uninstall runs at a time (a lock on +`/opt/fork-fleet`); a second one is refused. ## Runs again, stops half-way, old or empty state -- **Again:** files, settings and units that did not change are not rewritten, and the agent is not stopped or +- **Again:** files, settings and units that did not change are not rewritten (a file whose mode or owner was + changed is put back from the release), and the agent is not stopped or restarted (a restart would end a job it is starting). After a change it is restarted. The agent reads its settings only when it starts, so before the first change the record gets a restart-pending mark, cleared after the - restart: a run stopped in between restarts the agent on the next run. + restart: a run stopped in between restarts the agent on the next run. The mark is also set before a new fleet + config choice is saved, so a run stopped between saving the choice and the swap fails `check` until the next run. - **Stopped half-way:** the next install ends it. A new release is staged in `/opt/fork-fleet/.kit-stage-*` and checked first (gate loads with the fleet config, the pool is in it, `bash -n`, the walls know this OS); a release that fails is never installed. Files are renamed in one by one while the agent and the refresh timer are stopped; `SOURCE` is written last, so `check` reports a half-done swap. Users are recorded in `/etc/fork-fleet/desk-kit.json` before they are made, so uninstall removes them even after a stop. +- **Old record:** a record that names a project's fleet config while neither `/etc/fork-fleet/fleet-config.json` + nor the installed copy is there is refused with the fix (`--fleet-config FILE` or `--release-fleet-config`). - **Old install:** a launcher, `launch.conf`, `desk.conf`, desk key, unit, or a systemd drop-in for one of the kit's units (`UNIT.d/*.conf`, also the dash-prefix forms) that this kit did not write is refused before anything changes. The type-wide `service.d/` and `timer.d/` folders are not checked: they hold the distribution's defaults for every diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index 5ff5e9be200af..e444030744255 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -12,7 +12,7 @@ the units, makes the desk key once, and starts the agent and the refresh timer ( something changed). Safe to run again and to stop at any step: a stopped run leaves SOURCE naming the old release (check says what differs) and the next run ends it. uninstall removes only what the kit made, its record last. """ -import argparse, contextlib, hashlib, importlib.util, json, os, pwd, re, shutil, stat, subprocess, sys, tempfile +import argparse, contextlib, fcntl, hashlib, importlib.util, json, os, pwd, re, shutil, stat, subprocess, sys, tempfile PREFIX = '' # tests only: every fixed path below is read under this folder ROOT_UID = 0 # tests only: who must run this and own the installed files @@ -45,7 +45,8 @@ AGENT, TIMER, REFRESH = 'fork-fleet-desk-agent.service', 't3code-fleet-refresh.t FOREIGN = (OPT + '/fork-fleet-launch.sh', ETC + '/launch.conf', ETC + '/desk.conf', KEY, *(f'{UNIT_DIR}/{u}' for u in UNITS)) CMD = {k: [k] for k in ('useradd', 'userdel', 'loginctl', 'systemctl', 'sysctl', 'ssh-keygen', 'ip', 'nft', 'bash', - 'id')} + 'id', 'getent')} +NOLOGIN = ('/usr/sbin/nologin', '/sbin/nologin', '/bin/false', '/usr/bin/false') CMD['python3'] = ['/usr/bin/python3'] ENV = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'C.UTF-8'} SHA = re.compile(r'[0-9a-f]{40}') @@ -88,17 +89,47 @@ def user_exists(name): return run('id', '-u', name, check=False).returncode == 0 +def user_problem(u): + """Why runner user u (made by the kit or by hand) could be used for more than one-job runners: a login shell, a + usable password (or none), a line in cron.allow/at.allow; None when it is fine. Read through getent (as root).""" + r = run('getent', 'passwd', u, check=False) + fields = r.stdout.strip().split(':') + if r.returncode or len(fields) < 7: + return f'{u}: its passwd entry cannot be read' + if fields[6] not in NOLOGIN: + return f'{u} has the login shell {fields[6]!r} (usermod -s /usr/sbin/nologin {u})' + r = run('getent', 'shadow', u, check=False) + fields = r.stdout.strip().split(':') + if r.returncode or len(fields) < 2: + return f'{u}: its shadow entry cannot be read' + if not fields[1].startswith(('!', '*')): + return f'{u} has a usable password or none (passwd -l {u})' + for f in DENY: + allow = f.replace('.deny', '.allow') + if u in lines_of(allow): + return f'{u} is in {allow} (remove that line)' + return None + + def job_writable(path): """Why a job could change this file (it, or a folder above it, writable by others or by a runner user's group, or owned by a runner user); None when only root or the caller can. A sticky folder (/tmp) is fine: nobody else may - rename or remove what is in it.""" + rename or remove what is in it. A link is followed: the folders of the path as given and of the file it names are + both checked (the install reads the file it names).""" uids, gids = set(), set() for i in range(1, 9): with contextlib.suppress(KeyError): pw = pwd.getpwnam(f'fork-runner-{i}') uids.add(pw.pw_uid) gids.add(pw.pw_gid) - path = os.path.abspath(path) + for path in dict.fromkeys((os.path.abspath(path), os.path.realpath(path))): + why = writable_by_job(path, uids, gids) + if why: + return why + return None + + +def writable_by_job(path, uids, gids): while True: try: st = os.stat(path) @@ -235,9 +266,7 @@ def stage(source, sha, fleet_config, pool): dst = os.path.join(tmp, rel) os.makedirs(os.path.dirname(dst), mode=0o755, exist_ok=True) shutil.copyfile(src, dst) - executable = rel in ('fork-fleet-launch.sh', 'hook.sh', 'desk-install') or ( - rel.startswith('fork-fleet-walls/') and not rel.endswith('.json')) - os.chmod(dst, 0o755 if executable else 0o644) + os.chmod(dst, file_mode(rel)) manifest[rel] = sha256(dst) for d, _, _ in os.walk(tmp): os.chmod(d, 0o755) @@ -268,6 +297,13 @@ def stage(source, sha, fleet_config, pool): return tmp, manifest +def file_mode(rel): + """The mode of an installed file: programs 0755, the rest 0644.""" + executable = rel in ('fork-fleet-launch.sh', 'hook.sh', 'desk-install') or ( + rel.startswith('fork-fleet-walls/') and not rel.endswith('.json')) + return 0o755 if executable else 0o644 + + def read_manifest(): with contextlib.suppress(OSError): with open(p(OPT + '/MANIFEST.sha256'), encoding='utf-8') as f: @@ -278,10 +314,13 @@ def read_manifest(): def swap(tmp, manifest, sha, before_change): """Each changed file renamed into /opt/fork-fleet (the agent stopped meanwhile, so no verb runs from half-old, half-new files, and the refresh timer and service stopped too: refresh runs the launcher); files of the old - release that are gone, removed. Unchanged files are left as they are. True when anything changed.""" + release that are gone, removed. Unchanged files (same text, mode and owner root) are left as they are; one whose + mode or owner was changed is replaced by the staged copy. True when anything changed.""" def same(rel): with contextlib.suppress(OSError): - return sha256(p(f'{OPT}/{rel}')) == manifest[rel] + st = os.lstat(p(f'{OPT}/{rel}')) + return (stat.S_ISREG(st.st_mode) and st.st_uid == ROOT_UID and stat.S_IMODE(st.st_mode) == file_mode(rel) + and sha256(p(f'{OPT}/{rel}')) == manifest[rel]) changed = [rel for rel in sorted(manifest) if not same(rel)] gone = sorted(set(read_manifest()) - set(manifest)) if changed or gone: @@ -290,6 +329,8 @@ def swap(tmp, manifest, sha, before_change): for rel in changed: os.makedirs(os.path.dirname(p(f'{OPT}/{rel}')), mode=0o755, exist_ok=True) os.replace(os.path.join(tmp, rel), p(f'{OPT}/{rel}')) + for d in sorted({os.path.dirname(rel) for rel in manifest} - {''}): + os.chmod(p(f'{OPT}/{d}'), 0o755) for rel in gone: with contextlib.suppress(FileNotFoundError): os.unlink(p(f'{OPT}/{rel}')) @@ -362,7 +403,18 @@ def make_key(pool, before_change): return True -def install(a, stop_after=None): +def lock(stack): + """One install or uninstall at a time: a lock on the /opt/fork-fleet folder itself (nothing to clean up), held + until the run ends. A second run is refused, not queued: it would sweep the first one's staging folder.""" + fd = os.open(p(OPT), os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC) + stack.callback(os.close, fd) + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + raise Refused('another desk-install install or uninstall is running: wait for it and run again') from None + + +def install(a, stop_after=None, stack=None): def done(step): if step == stop_after: raise SystemExit(f'stopped after {step} (test)') @@ -378,7 +430,7 @@ def install(a, stop_after=None): raise Refused(f'--source {a.source}: {missing[0]} is missing (point it at the release\'s top folder)') if a.fleet_config and not os.path.isfile(a.fleet_config): raise Refused(f'--fleet-config {a.fleet_config}: no such file') - if a.fleet_config and job_writable(a.fleet_config): + if a.fleet_config and job_writable(a.fleet_config): # the file it names and every folder above both raise Refused(f'--fleet-config {a.fleet_config}: {job_writable(a.fleet_config)}; copy it to a folder only root ' 'can write (for example /root) and run again') m = machine_section() @@ -401,11 +453,19 @@ def install(a, stop_after=None): if config is None and state['fleetConfig'] == 'project' and not a.release_fleet_config: # the project's config from the last install (older records: the installed copy), checked again below config = p(PROJECT) if os.path.isfile(p(PROJECT)) else p(OPT + '/fork-fleet.json') + if not os.path.isfile(config): + raise Refused(f'the install record says a project\'s fleet config, but neither {PROJECT} nor ' + f'{OPT}/fork-fleet.json is there: run again with --fleet-config FILE or ' + '--release-fleet-config') users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] + for u in users: # a runner user made by hand must be as locked as one the kit makes + if user_exists(u) and user_problem(u): + raise Refused(f'runner user {user_problem(u)}: fix it and run again') done('check') for d, mode in ((OPT, 0o755), (OPT + '/template', 0o755), (ETC, 0o700)): os.makedirs(p(d), mode=mode, exist_ok=True) os.chmod(p(d), mode) + lock(stack if stack is not None else contextlib.ExitStack()) save_state(state) make_users(users, state) done('users') @@ -423,7 +483,10 @@ def install(a, stop_after=None): if not same_text(path, text, mode): pending() write_file(path, text, mode) - # the fleet config choice before the swap: a run stopped after it keeps what was asked for + # the fleet config choice before the swap: a run stopped after it keeps what was asked for. The restart mark goes + # first when the swap will change the installed config, so `check` fails until a run ends the swap. + if manifest['fork-fleet.json'] != read_manifest().get('fork-fleet.json'): + pending() if config: with open(os.path.join(tmp, 'fork-fleet.json'), encoding='utf-8') as f: write_file(PROJECT, f.read()) @@ -439,6 +502,7 @@ def install(a, stop_after=None): put(path, text, mode) run('sysctl', '-q', '-p', p(SYSCTL)) make_key(m['pool'], pending) + os.chmod(p(KEY), 0o600) # `check` wants it so; an older key made with another mode is set right done('settings') for u in UNITS: with open(p(f'{OPT}/units/{u}'), encoding='utf-8') as f: @@ -511,7 +575,8 @@ def check(): users = [] row(False, 'machine-file', str(e)[:300]) for u in users: - row(user_exists(u), 'runner-user', f'{u} exists') + why = f'{u} does not exist' if not user_exists(u) else user_problem(u) + row(not why, 'runner-user', why or f'{u} exists: no login shell, no password, not in cron.allow/at.allow') with contextlib.suppress(Refused): # a smaller `slots` leaves the kit's extra users: named, not lost for u in (read_state() or {}).get('createdUsers', []) if users else []: if u not in users and user_exists(u): @@ -538,11 +603,31 @@ def installed_problems(): return [f'{OPT}/MANIFEST.sha256 or {STATE} is missing: not installed (or a first install stopped)'] for rel, digest in manifest.items(): try: - ok = sha256(p(f'{OPT}/{rel}')) == digest and os.stat(p(f'{OPT}/{rel}')).st_uid == ROOT_UID + st = os.lstat(p(f'{OPT}/{rel}')) + ok = stat.S_ISREG(st.st_mode) and sha256(p(f'{OPT}/{rel}')) == digest and st.st_uid == ROOT_UID + mode_ok = stat.S_IMODE(st.st_mode) == file_mode(rel) except OSError: - ok = False + ok = mode_ok = False if not ok: problems.append(f'{OPT}/{rel}: differs from what was installed') + elif not mode_ok: + problems.append(f'{OPT}/{rel}: mode {stat.S_IMODE(st.st_mode):o}, not {file_mode(rel):o}: install again') + folders = {OPT: 0o755, OPT + '/template': 0o755, ETC: 0o700, KEY: 0o600} + folders.update({f'{OPT}/{os.path.dirname(rel)}': 0o755 for rel in manifest if os.path.dirname(rel)}) + for path, mode in sorted(folders.items()): + try: + st = os.lstat(p(path)) + except OSError: + continue # a missing key or folder is named by its own check (agent settings, template) + if stat.S_ISLNK(st.st_mode) or st.st_uid != ROOT_UID or stat.S_IMODE(st.st_mode) != mode: + problems.append(f'{path}: not root-owned with mode {mode:o}: install again') + if state.get('fleetConfig') == 'project': + try: + ok = sha256(p(PROJECT)) == manifest.get('fork-fleet.json') + except OSError: + ok = False + if not ok: + problems.append(f'{PROJECT} (the project\'s fleet config) is not the installed one: install again') try: with open(p(OPT + '/SOURCE'), encoding='utf-8') as f: source = f.read().strip() @@ -594,7 +679,7 @@ def rmdir_left(path): return [] -def uninstall(): +def uninstall(stack=None): """Removes only what the kit made: its units, the files of its manifest, its settings, the runner copies and template its programs made, the users and deny files it made. Every runner is stopped first; while one still runs, the network table, users and record stay (run it again). The record goes last, so a stopped uninstall runs again.""" @@ -604,6 +689,8 @@ def uninstall(): if state is None: say(f'no {STATE}: nothing here was installed by this kit (nothing changed)') return 0 + if os.path.isdir(p(OPT)): + lock(stack if stack is not None else contextlib.ExitStack()) for unit in (AGENT, TIMER, REFRESH): run('systemctl', 'disable', '--now', unit, check=False) launcher = p(OPT + '/fork-fleet-launch.sh') @@ -677,9 +764,10 @@ def main(argv, stop_after=None): a = ap.parse_args(argv) os.umask(0o022) try: - if a.cmd == 'install': - return install(a, stop_after) - return check() if a.cmd == 'check' else uninstall() + with contextlib.ExitStack() as stack: + if a.cmd == 'install': + return install(a, stop_after, stack) + return check() if a.cmd == 'check' else uninstall(stack) except Refused as e: print(f'desk-install: {e}', file=sys.stderr) return 1 diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index e00bef21edfe1..abde4a69192b3 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -4,7 +4,8 @@ python3 -I -B fork-kit/desk/tests/install.test.py """ -import contextlib, importlib.machinery, importlib.util, io, json, os, shutil, stat, subprocess, sys, tempfile, unittest +import contextlib, fcntl, importlib.machinery, importlib.util, io, json, os, shutil, stat, subprocess, sys, tempfile +import types, unittest KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) REPO = os.path.dirname(os.path.dirname(KIT)) @@ -36,6 +37,19 @@ def put(n, items): sys.exit(9) if name == 'id': sys.exit(0 if args[-1] in lines('users') else 1) +if name == 'getent': # a user's login shell: the file shell-USER (nologin); its password field: pw-USER (!) + u = args[-1] + if u not in lines('users'): + sys.exit(2) + def one(n, default): + try: + return open(os.path.join(here, n + '-' + u)).read().strip() + except FileNotFoundError: + return default + if args[0] == 'passwd': + print(f'{u}:x:1001:1001::/nonexistent:' + one('shell', '/usr/sbin/nologin')) + else: + print(f'{u}:' + one('pw', '!') + ':19000:0:99999:7:::') if name == 'useradd': put('users', lines('users') + [args[-1]]) if name == 'userdel': @@ -365,6 +379,123 @@ def killed(*args): self.assertIn(want, self.read('/opt/fork-fleet/fork-fleet.json')) self.assertNotIn(other, self.read('/opt/fork-fleet/fork-fleet.json')) + def test_a_run_killed_between_the_choice_and_the_swap_fails_check(self): + a, b = os.path.join(self.tmp, 'a.json'), os.path.join(self.tmp, 'b.json') + self.write(a, json.dumps({'pools': [{'name': 'desktop'}], 'which': 'a'})) + self.write(b, json.dumps({'pools': [{'name': 'desktop'}], 'which': 'b'})) + for first, then in ((['--fleet-config', a], ['--fleet-config', b]), + (['--fleet-config', a], ['--release-fleet-config']), + ([], ['--fleet-config', b])): + with self.subTest(then=then): + shutil.rmtree(self.root) + self.write(self.path('/proc/sys/net/ipv4/ip_forward'), '1\n') + self.assertEqual(self.install(*first)[0], 0) + self.template() + self.assertEqual(self.run_main('check')[0], 0) + swap = self.m.swap + + def killed(*args): + raise SystemExit('killed before the swap') + self.m.swap = killed + try: + self.assertIn('killed', str(self.install(*then)[0])) + finally: + self.m.swap = swap + code, out, _ = self.run_main('check') # the old config is still the installed one + self.assertEqual(code, 1, out) + self.assertEqual(self.install()[0], 0) + self.assertEqual(self.run_main('check')[0], 0) + + def test_a_fleet_config_link_into_an_open_folder_is_refused(self): + open_dir, safe = os.path.join(self.tmp, 'open'), os.path.join(self.tmp, 'safe') + os.makedirs(open_dir) + os.chmod(open_dir, 0o777) + self.write(os.path.join(open_dir, 'f.json'), json.dumps({'pools': [{'name': 'desktop'}]})) + os.makedirs(safe) + os.symlink(os.path.join(open_dir, 'f.json'), os.path.join(safe, 'link.json')) + code, _, err = self.install('--fleet-config', os.path.join(safe, 'link.json')) + self.assertEqual(code, 1) + self.assertIn(f'{open_dir} can be written by other users', err) + self.assertFalse(os.path.exists(self.path('/opt'))) + + def test_a_fleet_config_a_runner_user_owns_or_its_group_may_write_is_refused(self): + folder = os.path.join(self.tmp, 'cfg') + os.makedirs(folder) + config = os.path.join(folder, 'f.json') + self.write(config, json.dumps({'pools': [{'name': 'desktop'}]})) + me, group, other = os.geteuid(), os.getegid(), 2 ** 31 - 7 + + def runner(uid, gid): + def getpwnam(name): + if name != 'fork-runner-3': + raise KeyError(name) + return types.SimpleNamespace(pw_uid=uid, pw_gid=gid) + return types.SimpleNamespace(getpwnam=getpwnam) + real = self.m.pwd + try: + self.m.pwd = runner(me, other) # the file belongs to a runner user + self.assertEqual(self.m.job_writable(config), f'{config} belongs to a runner user') + self.m.pwd = runner(other, group) # its group is a runner user's, and the group may write it + os.chmod(config, 0o664) + self.assertEqual(self.m.job_writable(config), f'{config} can be written by other users') + os.chmod(config, 0o644) # the group may only read it: the file passes (a folder above may not) + self.assertNotIn(config + ' ', self.m.job_writable(config) or '') + finally: + self.m.pwd = real + + def test_install_sets_back_modes_check_names(self): + self.assertEqual(self.install()[0], 0) + self.template() + self.assertEqual(self.run_main('check')[0], 0) + os.chmod(self.path('/opt/fork-fleet/fork-fleet-launch.sh'), 0o777) + os.chmod(self.path('/opt/fork-fleet/fork-fleet-walls'), 0o777) + os.chmod(self.path('/etc/fork-fleet/desk-key'), 0o644) + os.chmod(self.path('/etc/fork-fleet'), 0o755) + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + for line in ('FAIL kit /opt/fork-fleet/fork-fleet-launch.sh: mode 777, not 755', + 'FAIL kit /opt/fork-fleet/fork-fleet-walls: not root-owned with mode 755', + 'FAIL kit /etc/fork-fleet/desk-key: not root-owned with mode 600', + 'FAIL kit /etc/fork-fleet: not root-owned with mode 700'): + self.assertIn(line, out) + self.assertEqual(self.install()[0], 0) + self.assertEqual(stat.S_IMODE(os.stat(self.path('/opt/fork-fleet/fork-fleet-launch.sh')).st_mode), 0o755) + code, out, _ = self.run_main('check') + self.assertEqual(code, 0, out) + + def test_a_second_install_or_uninstall_at_the_same_time_is_refused(self): + self.assertEqual(self.install()[0], 0) + fd = os.open(self.path('/opt/fork-fleet'), os.O_RDONLY | os.O_DIRECTORY) + try: + fcntl.flock(fd, fcntl.LOCK_EX) + stage = os.path.join(self.path('/opt/fork-fleet'), '.kit-stage-other') + os.makedirs(stage) # the other run's staging folder: not swept + for argv in (['install', '--source', self.source, '--sha', SHA2], ['uninstall']): + with self.subTest(argv[0]): + code, _, err = self.run_main(*argv) + self.assertEqual(code, 1) + self.assertIn('another desk-install install or uninstall is running', err) + self.assertTrue(os.path.isdir(stage)) + self.assertEqual(self.read('/opt/fork-fleet/SOURCE'), SHA1 + '\n') + finally: + os.close(fd) + self.assertEqual(self.install(sha=SHA2)[0], 0) + + def test_check_fails_on_a_runner_user_that_can_log_in(self): + self.assertEqual(self.install()[0], 0) + self.template() + self.assertEqual(self.run_main('check')[0], 0) + for name, text, want in (('shell-fork-runner-1', '/bin/bash', "fork-runner-1 has the login shell '/bin/bash'"), + ('pw-fork-runner-2', '', 'fork-runner-2 has a usable password or none')): + with self.subTest(name): + self.write(os.path.join(self.fakes, name), text + '\n') + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + self.assertIn('FAIL runner-user ' + want, out) + os.unlink(os.path.join(self.fakes, name)) + self.write(self.path('/etc/at.allow'), 'fork-runner-2\n') + self.assertIn('FAIL runner-user fork-runner-2 is in /etc/at.allow', self.run_main('check')[1]) + def test_refusals_change_nothing(self): cases = { 'not a release sha': (lambda: None, ['--sha', 'abc'], '--sha must be'), @@ -393,6 +524,21 @@ def test_refusals_change_nothing(self): '/run/systemd/system/t3code-fleet-.timer.d/x.conf'), '[Timer]\n'), [], 'changes t3code-fleet'), 'a record of another shape': (lambda: self.write(self.path('/etc/fork-fleet/desk-kit.json'), '{"version": 1}'), [], 'not an install record'), + 'a hand-made runner user with a login shell': ( + lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n'), + self.write(os.path.join(self.fakes, 'shell-fork-runner-1'), '/bin/bash\n')), [], + "login shell '/bin/bash'"), + 'a hand-made runner user with a password': ( + lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-2\n'), + self.write(os.path.join(self.fakes, 'pw-fork-runner-2'), '$6$salt$hash\n')), [], + 'usable password'), + 'a hand-made runner user in cron.allow': ( + lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n'), + self.write(self.path('/etc/cron.allow'), 'fork-runner-1\n')), [], 'is in /etc/cron.allow'), + 'a record that names a project config that is gone': ( + lambda: (self.write(self.path('/etc/fork-fleet/desk-kit.json'), json.dumps( + {'version': 1, 'createdUsers': [], 'fleetConfig': 'project', 'sha': SHA1}), 0o600)), [], + 'run again with --fleet-config FILE or --release-fleet-config'), 'a fleet config others may write': (lambda: (os.makedirs(os.path.join(self.tmp, 'open'), exist_ok=True), os.chmod(os.path.join(self.tmp, 'open'), 0o777), self.write(os.path.join(self.tmp, 'open', 'f.json'), '{}')), @@ -403,6 +549,9 @@ def test_refusals_change_nothing(self): with self.subTest(name): shutil.rmtree(self.root, ignore_errors=True) os.makedirs(self.root) + for n in os.listdir(self.fakes): # the users of the case before + if n == 'users' or n.startswith(('shell-', 'pw-')): + os.unlink(os.path.join(self.fakes, n)) self.machine(MACHINE) self.write(os.path.join(self.fakes, 'os-exit'), '0') setup() @@ -415,7 +564,7 @@ def test_refusals_change_nothing(self): self.assertEqual(code, 1) self.assertIn(msg, err) self.assertEqual(self.tree(), before) - self.assertFalse([c for c in self.calls() if not c.startswith(('id ', 'systemctl show'))], + self.assertFalse([c for c in self.calls() if not c.startswith(('id ', 'getent ', 'systemctl show'))], self.calls()) def test_bad_release_is_not_installed(self): From f211bcd66cc23a2ecc55317f38bc7ec03e31aa9b Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 00:59:35 +0000 Subject: [PATCH 23/33] fork kit desk: Windows starter fix round 3 ([experimental] keys, changes listed, PID 1 start) (D9) - networkingMode/dnsTunneling under [experimental] (the pre-2.0 form that can still win over [wsl2]) are removed by the merge, listed in the question, and the Windows-side check FAILs while one is there. - The question lists each .wslconfig key it changes, old -> new. - The owed-shutdown mark is one for every distro, like .wslconfig; uninstall keeps it. - wsl.conf is compared with PID 1's start (/proc/1/stat field 22 + btime), not the mtime of /proc/1; unreadable means restart. Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 21 ++++++--- fork-kit/desk/tests/windows.test.py | 70 +++++++++++++++++++++++++++- fork-kit/desk/windows/start-desk.ps1 | 47 ++++++++++++++----- 3 files changed, 119 insertions(+), 19 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 079d9c1a72362..00c6641d17340 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -70,7 +70,8 @@ Save the machine file on Windows, then in a normal (not administrator) Windows P powershell -ExecutionPolicy Bypass -File start-desk.ps1 -Repo OWNER/REPO -Sha RELEASE_SHA -MachineFile machine.json ``` -It prints the PC's facts and asks before it changes anything. Before the question it checks the machine file and +It prints the PC's facts and asks before it changes anything; the question lists each `.wslconfig` key it will +change, old value -> new. Before the question it checks the machine file and `-FleetConfig` exist and refuses a WSL 1 distro of that name (`wsl -l -v`; WSL 1 runs no systemd: `wsl --set-version DISTRO 2` first). A distro of that name that the script did not make is named in the question (its whole `/etc/wsl.conf` is replaced) and must be confirmed by typing its name. Then: the @@ -79,18 +80,26 @@ leaves a mark in `%LOCALAPPDATA%\fork-desk`, so the next run asks no takeover qu `wsl --unregister DISTRO`); `.wslconfig`'s `[wsl2]` memory cap (`-MemoryGB`, default three quarters of the RAM, at most the RAM less 6 GB, never under 4 GB; one that leaves Windows under 2 GB is refused), swap (`-SwapGB`, 4), NAT and DNS tunnelling, for every WSL distro (read and written as UTF-8 without a -BOM; a dated copy of the old file is kept, other keys and sections stay); the distro's own settings the WSL2 walls +BOM; a dated copy of the old file is kept, other keys and sections stay; an old `networkingMode` or `dnsTunneling` +line under `[experimental]`, the form before WSL 2.0 that can still win over `[wsl2]`, is removed, and the check +FAILs while one is there); the distro's own settings the WSL2 walls ask for (`/etc/wsl.conf`: systemd on, no Windows drives, programs or PATH; its own `/etc/resolv.conf`; WSL's binfmt handler masked); one `wsl --shutdown` when one of these changed or the running WSL does not have them yet (PID 1 not -systemd, interop live, Windows drives mounted, `wsl.conf` or `.wslconfig` newer than the running one, more memory -than the cap, not NAT, or a `.wslconfig` change whose shutdown is still owed: a mark in `%LOCALAPPDATA%\fork-desk`, -written before the change and removed after the shutdown, because WSL's uptime skips sleep), +systemd, interop live, Windows drives mounted, `wsl.conf` newer than PID 1's start (read from `/proc/1/stat` and +`btime`; unreadable means restart), `.wslconfig` newer than the running WSL, more memory than the cap, not NAT, or a +`.wslconfig` change whose shutdown is still owed: one mark in `%LOCALAPPDATA%\fork-desk` for every distro, like +`.wslconfig` itself, written before the change and removed after the shutdown, because WSL's uptime skips sleep), so a run stopped before its shutdown restarts next time (every WSL distro and Docker Desktop stop); the packages the fleet needs; the machine file and `-FleetConfig`, if given; this kit's bootstrap; a logon task that keeps the distro running while the owner is logged in. Last, `desk-install check` and the Windows side's own PASS/FAIL lines (the `[wsl2]` keys); any FAIL or error exits 1. Safe to run again. `-Uninstall` (needs no `-Repo` or `-Sha`) runs `desk-install uninstall` in the distro, stops and removes the logon task and names the first `.wslconfig.bak-*` (the file before the first run); the distro and `.wslconfig` -stay (`wsl --unregister DISTRO` removes the distro). +stay (`wsl --unregister DISTRO` removes the distro), and so does an owed shutdown's mark. + +Known limits: the `DISTRO.made` mark is per distro name; a run stopped inside `wsl --install`, then that distro +removed by hand and a distro of the same name made by hand, would skip the takeover question once (the mark is gone +after the next run's settings step). The PowerShell parts are tested only where `pwsh` is installed; the in-distro +parts run in the tests on a fake root. ## Runs again, stops half-way, old or empty state diff --git a/fork-kit/desk/tests/windows.test.py b/fork-kit/desk/tests/windows.test.py index 250005e009e9d..dc7f534b56272 100644 --- a/fork-kit/desk/tests/windows.test.py +++ b/fork-kit/desk/tests/windows.test.py @@ -108,6 +108,15 @@ def live(self, env=None): self.assertEqual(r.returncode, 0, r.stderr) return r.stdout + def started(self, at): + """PID 1 started at `at` (seconds since the epoch): /proc/stat's btime plus field 22 of /proc/1/stat.""" + os.makedirs(self.root + '/proc/1', exist_ok=True) + tick, boot = os.sysconf('SC_CLK_TCK'), int(at) - 500 + put(self.root + '/proc/stat', f'cpu 1 2 3 4\nbtime {boot}\nprocesses 9\n') + fields = ['1', '(sys temd)', 'S'] + ['0'] * 49 # a name with a space: the fields count from its ')' + fields[21] = str(500 * tick) + put(self.root + '/proc/1/stat', ' '.join(fields) + '\n') + def test_the_restart_decision_reads_the_live_state(self): # a run stopped after it wrote wsl.conf, before its `wsl --shutdown`: the next run changes nothing, but the # running distro still has the old settings, so it restarts @@ -116,7 +125,7 @@ def test_the_restart_decision_reads_the_live_state(self): put(self.root + '/proc/meminfo', 'MemTotal: 8000000 kB\n') self.run_prepare() past = time.time() - 3600 - os.utime(self.root + '/proc/1', (past, past)) + self.started(past) self.assertIn('restart: /etc/wsl.conf is newer than the running distro', self.live()) os.utime(self.root + '/etc/wsl.conf', (past - 60, past - 60)) self.assertEqual(self.live({'WANT_KB': '8388608'}), '') @@ -133,9 +142,25 @@ def test_the_restart_decision_reads_the_live_state(self): os.chmod(fake + '/wslinfo', 0o755) self.assertIn('the networking mode is mirrored, not nat', self.live({'PATH': fake + ':/usr/bin:/bin'})) + def test_the_start_of_pid_1_is_read_from_proc_stat_not_the_mtime_of_proc_1(self): + os.makedirs(self.root + '/proc/1') + put(self.root + '/proc/1/comm', 'systemd\n') + self.run_prepare() + now = time.time() + os.utime(self.root + '/etc/wsl.conf', (now - 600, now - 600)) + self.started(now - 300) # PID 1 started after the write ... + os.utime(self.root + '/proc/1', (now - 3600, now - 3600)) # ... whatever the mtime of /proc/1 says + self.assertEqual(self.live(), '') + self.started(now - 900) # PID 1 older than the write: it still runs the old settings + os.utime(self.root + '/proc/1', (now, now)) + self.assertIn('restart: /etc/wsl.conf is newer than the running distro', self.live()) + os.unlink(self.root + '/proc/stat') # unreadable: restart, never a guess + self.assertIn('restart: the start time of PID 1 cannot be read', self.live()) + def test_a_mounted_windows_drive_restarts(self): os.makedirs(self.root + '/proc/1') put(self.root + '/proc/1/comm', 'systemd\n') + self.started(time.time()) put(self.root + '/proc/mounts', 'none /usr/lib/wsl/drivers 9p ro 0 0\n/dev/sdc / ext4 rw 0 0\n') self.assertEqual(self.live(), '') # WSL's own driver share is not a Windows drive put(self.root + '/proc/mounts', 'C:\\134 /mnt/c 9p rw 0 0\n') @@ -233,6 +258,49 @@ def test_wsl_version_is_read_from_the_list(self): self.assertEqual(r.returncode, 0, r.stdout + r.stderr) self.assertEqual(r.stdout.strip(), '1 2 0') + def test_old_experimental_networking_keys_are_removed_named_and_checked(self): + part = wslconfig_part() + self.assertIn("$Experimental = @('networkingMode', 'dnsTunneling')", part) + merge = part[part.index('function Merge-WslConfig'):part.index('function WslVersion')] + self.assertIn("$sec -eq 'experimental'", merge) + self.assertIn('$Matches[1] -in $Experimental) { continue }', merge) + src = text().decode('ascii') + body = src[src.index('try {'):] + ask = min(body.index("Read-Host 'Type yes"), body.index('Read-Host "Type the distro')) + self.assertLess(body.index('$changes = @(Changes (Read-WslConfig $WslConfig) $want)'), ask) + check = body[body.index("Head 'check'"):] + self.assertIn("SectionKeys (Read-WslConfig $WslConfig) 'experimental'", check) + self.assertIn('FAIL wslconfig [experimental] $k=', check) + + def test_the_owed_shutdown_mark_is_shared_like_wslconfig(self): + src = text().decode('ascii') + self.assertIn("$Owed = Join-Path $Marks 'wslconfig.restart'", src) + uninstall = src[src.index('if ($Uninstall)'):src.index("if (-not $MachineFile)")] + self.assertNotIn('$Owed', uninstall) # another distro's run still owes the shutdown + + @unittest.skipUnless(shutil.which('pwsh'), 'pwsh is not installed') + def test_wslconfig_merge_removes_old_experimental_networking_keys(self): + d = tempfile.mkdtemp(prefix='desk-wslconfig-test-') + self.addCleanup(shutil.rmtree, d, True) + path = os.path.join(d, '.wslconfig') + with open(path, 'w', encoding='utf-8', newline='\n') as f: + f.write('[wsl2]\nmemory=24GB\nguiApplications=true\n[experimental]\nnetworkingMode=mirrored\n' + 'sparseVhd=true\ndnsTunneling=false\n') + script = os.path.join(d, 'merge.ps1') + with open(script, 'w', encoding='ascii') as f: + f.write(wslconfig_part() + '\n$want = [ordered]@{ memory = "9GB"; networkingMode = "nat"; ' + 'guiApplications = "false" }\nChanges (Read-WslConfig $args[0]) $want\n' + '"changed=$(Merge-WslConfig $args[0] $want)"\n' + '"left=$((SectionKeys (Read-WslConfig $args[0]) \'experimental\').Keys -join \',\')"\n' + '"after=$(@(Changes (Read-WslConfig $args[0]) $want).Count)"\n') + r = subprocess.run(['pwsh', '-NoProfile', '-NonInteractive', '-File', script, path], capture_output=True, + text=True, timeout=120, stdin=subprocess.DEVNULL) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + for line in ('[wsl2] memory: 24GB -> 9GB', '[wsl2] networkingMode: (not set) -> nat', + '[wsl2] guiApplications: true -> false', '[experimental] networkingMode=mirrored: removed', + '[experimental] dnsTunneling=false: removed', 'changed=True', 'left=sparseVhd', 'after=0'): + self.assertIn(line, r.stdout) + def test_wslconfig_is_utf8_without_a_bom_never_ascii(self): src = text().decode('ascii') self.assertNotRegex(src, r'(?i)-Encoding\s+ascii') diff --git a/fork-kit/desk/windows/start-desk.ps1 b/fork-kit/desk/windows/start-desk.ps1 index 22630df49f0f8..a0921439a25fc 100644 --- a/fork-kit/desk/windows/start-desk.ps1 +++ b/fork-kit/desk/windows/start-desk.ps1 @@ -20,10 +20,11 @@ $env:WSL_UTF8 = '1' # wsl.exe's own lines in UTF-8, not UTF $OutputEncoding = [Text.UTF8Encoding]::new($false) # what is piped into wsl.exe: UTF-8, not ASCII $Task = "fork-desk-$Distro" $WslConfig = Join-Path $env:USERPROFILE '.wslconfig' -# Marks on the Windows side, for a run that stops: `made` (this script is installing the distro: a later run asks no -# takeover question), `restart` (.wslconfig changed and its `wsl --shutdown` is still owed: VM uptime skips sleep) +# Marks on the Windows side, for a run that stops: `DISTRO.made` (this script is installing the distro: a later run +# asks no takeover question), `wslconfig.restart` (.wslconfig changed and its `wsl --shutdown` is still owed: VM +# uptime skips sleep). .wslconfig is shared by every distro, so its mark is too: a run for another -Distro still owes it. $Marks = Join-Path $env:LOCALAPPDATA 'fork-desk' -$Made = Join-Path $Marks "$Distro.made"; $Owed = Join-Path $Marks "$Distro.restart" +$Made = Join-Path $Marks "$Distro.made"; $Owed = Join-Path $Marks 'wslconfig.restart' function Mark([string]$path) { New-Item -ItemType Directory -Force -Path $Marks | Out-Null; Set-Content -LiteralPath $path 'start-desk.ps1' } function Head([string]$t) { Write-Host "==== $t" -ForegroundColor Cyan } @@ -86,7 +87,15 @@ ROOT=${ROOT:-} for h in "$ROOT"/proc/sys/fs/binfmt_misc/WSLInterop*; do [ ! -e "$h" ] || { echo "restart: WSL interop is live (${h##*/})"; break; } done -[ ! "$ROOT/etc/wsl.conf" -nt "$ROOT/proc/1" ] || echo 'restart: /etc/wsl.conf is newer than the running distro' +# when PID 1 started (boot time + its start in clock ticks; the mtime of /proc/1 is only when that entry was made) +tick=$(getconf CLK_TCK 2>/dev/null || echo 100) +since=$(sed -n 's/.*\x29 //p' "$ROOT/proc/1/stat" 2>/dev/null | cut -d' ' -f20) # after the name: field 22 +boot=$(sed -n 's/^btime \([0-9]*\)$/\1/p' "$ROOT/proc/stat" 2>/dev/null || true) +conf=$(stat -c %Y "$ROOT/etc/wsl.conf" 2>/dev/null || echo 0) +case "$since" in (''|*[!0-9]*) since= ;; esac +case "$boot" in (''|*[!0-9]*) since= ;; esac +if [ -z "$since" ]; then echo 'restart: the start time of PID 1 cannot be read' +elif [ "$conf" -ge $((boot + since / tick)) ]; then echo 'restart: /etc/wsl.conf is newer than the running distro'; fi mem=$(sed -n 's/^MemTotal: *\([0-9]*\) kB$/\1/p' "$ROOT/proc/meminfo" 2>/dev/null || true) [ -z "${WANT_KB:-}" ] || [ -z "$mem" ] || [ "$mem" -le "$WANT_KB" ] || echo "restart: WSL has $mem kB, above the cap" mode=$(wslinfo --networking-mode 2>/dev/null || true) @@ -104,14 +113,25 @@ function Read-WslConfig([string]$path) { if (-not (Test-Path -LiteralPath $path)) { return @() } @([IO.File]::ReadAllLines($path, [Text.UTF8Encoding]::new($false))) # a BOM (UTF-8 or UTF-16) is still read } -function Wsl2Keys([string[]]$lines) { # the keys of the [wsl2] section only +# networkingMode and dnsTunneling were [experimental] keys before WSL 2.0; an old line there can still win over +# [wsl2], so Merge-WslConfig removes them there (the [wsl2] ones are what the desk needs) and the check FAILs on them. +$Experimental = @('networkingMode', 'dnsTunneling') +function SectionKeys([string[]]$lines, [string]$section) { # the keys of one section only $keys = @{}; $sec = '' foreach ($l in $lines) { if ($l -match '^\s*\[(.+)\]\s*$') { $sec = $Matches[1].Trim().ToLower(); continue } - if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=\s*(.*?)\s*$') { $keys[$Matches[1]] = $Matches[2] } + if ($sec -eq $section -and $l -match '^\s*([^=#;\s]+)\s*=\s*(.*?)\s*$') { $keys[$Matches[1]] = $Matches[2] } } $keys } +function Wsl2Keys([string[]]$lines) { SectionKeys $lines 'wsl2' } +function Changes([string[]]$lines, $want) { # what Merge-WslConfig would change, one line each: old -> new + $now = Wsl2Keys $lines; $old = SectionKeys $lines 'experimental' + foreach ($k in $want.Keys) { + if ($now[$k] -ne $want[$k]) { " [wsl2] ${k}: $(if ($null -ne $now[$k]) { $now[$k] } else { '(not set)' }) -> $($want[$k])" } + } + foreach ($k in $Experimental) { if ($old.ContainsKey($k)) { " [experimental] $k=$($old[$k]): removed (the [wsl2] one is used)" } } +} function Merge-WslConfig([string]$path, $want, [scriptblock]$before = {}) { # True when it changed the file (a dated copy of the old one kept; $before runs first) $lines = Read-WslConfig $path $out = [Collections.Generic.List[string]]::new(); $sec = ''; $done = @{} @@ -120,6 +140,7 @@ function Merge-WslConfig([string]$path, $want, [scriptblock]$before = {}) { # if ($l -match '^\s*\[(.+)\]\s*$') { if ($sec -eq 'wsl2') { AddMissing }; $sec = $Matches[1].Trim().ToLower(); $out.Add($l); continue } if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=' -and $want.Contains($Matches[1])) { $k = $Matches[1]; if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 }; continue } + if ($sec -eq 'experimental' -and $l -match '^\s*([^=#;\s]+)\s*=' -and $Matches[1] -in $Experimental) { continue } $out.Add($l) } if ($sec -eq 'wsl2') { AddMissing } @@ -163,7 +184,7 @@ try { if ($have) { InDistro 'desk-install uninstall' '[ ! -x /opt/fork-fleet/desk-install ] || /opt/fork-fleet/desk-install uninstall' } Stop-ScheduledTask -TaskName $Task -ErrorAction SilentlyContinue # its `sleep infinity` too Unregister-ScheduledTask -TaskName $Task -Confirm:$false -ErrorAction SilentlyContinue - Remove-Item -LiteralPath $Owed, $Made -ErrorAction SilentlyContinue + Remove-Item -LiteralPath $Made -ErrorAction SilentlyContinue # an owed shutdown is kept: .wslconfig is shared $bak = @(Get-ChildItem -Path "$WslConfig.bak-*" -ErrorAction SilentlyContinue | Sort-Object Name) "Done. The distro and .wslconfig are kept; to remove the distro too: wsl --unregister $Distro" if ($bak) { ".wslconfig as it was before the first run: $($bak[0].FullName) (copy it back over .wslconfig to undo)" } @@ -181,10 +202,12 @@ try { $ours = Test-Path -LiteralPath $Made if ($have -and -not $ours) { wsl.exe -d $Distro -u root --cd / -- test -e /etc/fork-kit/desk-distro; $ours = $LASTEXITCODE -eq 0 } + $want = [ordered]@{ memory = "${MemoryGB}GB"; swap = "${SwapGB}GB"; networkingMode = 'nat'; dnsTunneling = 'true'; guiApplications = 'false' } + $changes = @(Changes (Read-WslConfig $WslConfig) $want) Write-Host '' - Write-Host "Next: the distro $Distro; .wslconfig [wsl2] memory=${MemoryGB}GB swap=${SwapGB}GB networkingMode=nat" -ForegroundColor Yellow - Write-Host 'dnsTunneling=true guiApplications=false (for ALL your WSL distros; a dated copy is kept); then ONE' -ForegroundColor Yellow - Write-Host 'wsl --shutdown (every WSL distro and Docker Desktop stop), the desk kit, and a logon task.' -ForegroundColor Yellow + Write-Host "Next: the distro $Distro; .wslconfig (for ALL your WSL distros; a dated copy is kept):" -ForegroundColor Yellow + if ($changes) { $changes | ForEach-Object { Write-Host $_ -ForegroundColor Yellow } } else { Write-Host ' no change' -ForegroundColor Yellow } + Write-Host 'then ONE wsl --shutdown (every WSL distro and Docker Desktop stop), the desk kit, and a logon task.' -ForegroundColor Yellow if ($have -and -not $ours) { Write-Host "The distro $Distro is already there and this script did not make it. Its /etc/wsl.conf is REPLACED" -ForegroundColor Yellow Write-Host '(its [user] default= and every other line go; no /mnt/c, no Windows programs or PATH), and it runs fleet' -ForegroundColor Yellow @@ -207,7 +230,6 @@ try { } Head '.wslconfig' - $want = [ordered]@{ memory = "${MemoryGB}GB"; swap = "${SwapGB}GB"; networkingMode = 'nat'; dnsTunneling = 'true'; guiApplications = 'false' } $restart = Merge-WslConfig $WslConfig $want { Mark $Owed } # the mark first: a run stopped after the write still restarts if ($restart) { 'changed: .wslconfig' } @@ -251,10 +273,11 @@ sh /root/desk-bootstrap.sh --repo $Repo --sha $Sha$extra $bad = 0 wsl.exe -d $Distro -u root --cd / -- /opt/fork-fleet/desk-install check if ($LASTEXITCODE -ne 0) { $bad++ } - $cfg = Wsl2Keys (Read-WslConfig $WslConfig) + $cfg = Wsl2Keys (Read-WslConfig $WslConfig); $old = SectionKeys (Read-WslConfig $WslConfig) 'experimental' foreach ($k in $want.Keys) { if ($cfg[$k] -eq $want[$k]) { "PASS wslconfig [wsl2] $k=$($want[$k])" } else { "FAIL wslconfig [wsl2] no $k=$($want[$k])"; $bad++ } } + foreach ($k in $Experimental) { if ($old.ContainsKey($k)) { "FAIL wslconfig [experimental] $k=$($old[$k]) is still there"; $bad++ } } $t = Get-ScheduledTask $Task -ErrorAction SilentlyContinue if ($t -and $t.State -eq 'Running') { "PASS logon-task $Task running" } else { "FAIL logon-task $Task is $(if ($t) { $t.State } else { 'missing' })"; $bad++ } if ($bad -eq 0) { Write-Host 'ALL PASS: this PC takes fleet jobs while you are logged in.' -ForegroundColor Green } From de6837e5b0016f73c85a2862e7482e4158591518 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 01:02:16 +0000 Subject: [PATCH 24/33] free/v1: say busy-while-running matches the process name; note the trailing-/ link case Co-Authored-By: Claude Opus 5.5 --- fork-kit/common/FREE.md | 12 +++++++++--- fork-kit/common/free/busy-while-running | 1 + 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/fork-kit/common/FREE.md b/fork-kit/common/FREE.md index 95eebbe60cedc..6339a4d24e6f6 100644 --- a/fork-kit/common/FREE.md +++ b/fork-kit/common/FREE.md @@ -14,9 +14,11 @@ One small contract for every tool that asks whether a machine may take more work them (a sticky folder such as `/tmp` is fine). Otherwise it is not run. The path is followed one part at a time: the folders of the path as written are judged too, and every link on the way must belong to root or to that user (another user's link in `/tmp` is refused, as the kernel's `fs.protected_symlinks` would). The script - is judged and run by the file the path leads to, resolved once. Only the mode bits are read: an ACL that lets - others write is not seen, so do not set one. A script that could not be ended (a group the consumer may not - signal) is waited for at most one more second, then left to finish on its own. + is judged and run by the file the path leads to, resolved once. (A link whose target ends in `/` is accepted + by the reader where the kernel would refuse it; it names the same, fully checked file, so it does no harm.) + Only the mode bits are read: an ACL that lets others write is not seen, so do not set one. A script that could + not be ended (a group the consumer may not signal) is waited for at most one more second, then left to finish + on its own. - **The exit code is the verdict:** `0` free, `75` busy. Anything else, a timeout, output that is not one JSON object (`NaN` and `Infinity` are not JSON), a higher `v`, a `free` that disagrees with the exit code, or facts more than 60 seconds from now = **not available**. Busy and not available both count as **not free**. @@ -66,6 +68,10 @@ size as busy. An answer without `memory.freeMb` makes no memory claim; the consu machine swaps hard; the swap rule is there for that. - [busy-while-running](free/busy-while-running): busy while a named program runs (a game, say). Copy it and put your programs in it. It sees the processes of its own system only: in WSL, Windows programs are not visible. + It matches the process name (`comm`), not the command line: a program started through an interpreter or a + wrapper (`python3 game.py`, `wine game.exe`, a shell script that runs the game) shows as `python3`, `wine` or + the program the wrapper starts, so name that. Naming an interpreter makes the machine busy whenever any script + of that interpreter runs. Several conditions in one script: call the others from it and answer busy when one is busy. diff --git a/fork-kit/common/free/busy-while-running b/fork-kit/common/free/busy-while-running index f5b5cd40723bd..a71cd5fefdc38 100755 --- a/fork-kit/common/free/busy-while-running +++ b/fork-kit/common/free/busy-while-running @@ -2,6 +2,7 @@ # A free/v1 script (FREE.md): busy while one of the programs in NAMES runs, say a game or a video editor, so the # machine is left to its person. Copy it, put your programs' process names (as `ps -e -o comm=` shows them) in # NAMES, make the copy root's and not writable by others, and name the copy as the free command. +# It matches the process name, not the command line: `python3 game.py` or `wine game.exe` shows as python3 or wine. # It sees the processes of this system only: in WSL, Windows programs are not visible (interop is off, on purpose). NAMES='example-game example-editor' From e5e00004f7c3ff2149d4a16be359b8ab56784976 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 01:05:06 +0000 Subject: [PATCH 25/33] fleet seeds: a last stall alarm after the last write is not a stall; no-seeds test; document trust, rate and links The stall alarm could go off between the end of the send loop and alarm(0), so Stalled escaped as a traceback. Now alarm(0) runs inside the try and a late alarm after everything read was written counts as done. Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 15 ++++++++- .github/scripts/fork-fleet-desk-seed.py | 34 ++++++++++++-------- .github/scripts/fork-fleet-desk-seed.test.py | 27 ++++++++++++++++ 3 files changed, 62 insertions(+), 14 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index bc28a77eaa928..90c774fbb2f5a 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -705,7 +705,12 @@ it is sent is fine (the old one is sent whole). A file dated more than five minu than `max_age_hours` or above `max_mb` is refused. Both folders and the files are root's (or the project's), readable by the desk-link user and writable by no one else (the owner itself is not checked, only that no one else may write); never -put a secret in a seed: every job of the pool reads it. +put a secret in a seed: every job of the pool reads it. **Make a seed only from +trusted code (the default branch), never in a job that runs pull request code:** +every later job of the pool starts from it, so a poisoned seed reaches them all. +A file replaced by a rename while another hard link to the old one stays (a +backup copy, say) is still "in place" for this check and is refused (exit 4): +keep no second link to a seed file. The desktop asks for them with the desk link's key, in an ssh session of its own: the same forced command sends every command that starts with `seed` to @@ -715,6 +720,14 @@ codes), never to a session. It answers `seeds` with the pool's list and size and SHA-256; anything else is refused. One stream per pool at a time; a desktop that takes less than 64 KiB in two minutes, or not the whole file within ten minutes plus one second per 256 KiB, is cut off and the next one may go. +So a link must carry at least about 256 KiB a second (2 Mbit/s) for a large +seed: at 100 KiB a second, a seed above about 96 MiB is always cut off. Size +`max_mb` to the desktop's link. A desktop just fast enough not to stall can hold +its pool's lock for a long time (16 GiB at 256 KiB a second is about 18 hours); +only that pool's own seed streams wait, nothing else. The host does not limit +how often a desktop asks: the key belongs to the pool's own desktop, one stream +per pool runs at a time, and each is cut off at its deadline, so the most it +costs is one file's reads and upload at a time. ## Owner steps on the repository diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index b24b98ea9fed3..2f28973d82ee7 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -150,24 +150,32 @@ def send(fd, out, size, stall=None): stall = STALL if stall is None else stall sha, sent = hashlib.sha256(), 0 deadline = time.monotonic() + GRACE + size / MIN_RATE + wrote = 0 old = signal.signal(signal.SIGALRM, stalled) try: - while sent < size and (data := os.read(fd, min(PIECE, size - sent))): - sha.update(data) - sent += len(data) - view = memoryview(data) - while view: - left = deadline - time.monotonic() - if left <= 0: - raise Stalled - signal.alarm(max(1, min(stall, math.ceil(left)))) # each piece that moves starts the wait again - view = view[os.write(out, view[:WRITE]):] + try: + while sent < size and (data := os.read(fd, min(PIECE, size - sent))): + sha.update(data) + sent += len(data) + view = memoryview(data) + while view: + left = deadline - time.monotonic() + if left <= 0: + raise Stalled + signal.alarm(max(1, min(stall, math.ceil(left)))) # each piece that moves starts the wait again + n = os.write(out, view[:WRITE]) + wrote += n + view = view[n:] + finally: + signal.alarm(0) # an alarm already on its way still arrives here, inside the `except Stalled` below except Stalled: - if time.monotonic() >= deadline: + if wrote == sent and not (sent < size and os.fstat(fd).st_size > sent): + pass # the last alarm came after the last write: everything read was sent + elif time.monotonic() >= deadline: raise Refused(f'the desktop did not take {size} bytes within {GRACE + size / MIN_RATE:.0f} seconds ' f'(at least {MIN_RATE >> 10} KiB a second): stopped', 1) from None - raise Refused(f'the desktop took less than {WRITE >> 10} KiB in {stall} seconds: stopped', 1) from None + else: + raise Refused(f'the desktop took less than {WRITE >> 10} KiB in {stall} seconds: stopped', 1) from None finally: - signal.alarm(0) signal.signal(signal.SIGALRM, old) return sha.hexdigest(), sent diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 8171f0d8ca14e..2fc94fe8ccefb 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -258,6 +258,33 @@ def reader(): th.join(60) self.assertEqual((code, bytes(got)), (0, data), err.getvalue()) + def test_an_alarm_that_comes_after_the_last_write_is_not_a_stall(self): + data = self.make('m2', {'a': b'y' * 3000}) + real = seed.signal + fired = [] + + def alarm(n): # the last stall alarm goes off just as the stream ends: the race between the loop and alarm(0) + got = real.alarm(n) + if n == 0 and not fired: + fired.append(True) + os.kill(os.getpid(), real.SIGALRM) + return got + fake = type('signal', (), {'SIGALRM': real.SIGALRM, 'signal': staticmethod(real.signal), + 'alarm': staticmethod(alarm)}) + with mock.patch.object(seed, 'signal', fake): + code, out, err = self.offer('seed m2') + self.assertTrue(fired) + self.assertEqual((code, out), (0, data), err) + + def test_a_pool_without_seeds_lists_nothing(self): + cfg = json.loads(json.dumps(CONFIG)) + cfg['pools'][0].pop('seeds', None) + with open(os.path.join(self.t, 'fork-fleet.json'), 'w', encoding='utf-8') as f: + json.dump(cfg, f) + code, out, err = self.offer('seeds') + self.assertEqual((code, json.loads(out)), (0, {'v': 1, 'seeds': []}), err) + self.assertEqual(self.offer('seed m2')[:2], (2, b'')) + def test_pools_never_share_a_seed_file(self): cfg = json.loads(json.dumps(CONFIG)) cfg['pools'][0]['seeds'] = SEEDS From 4812b17d85334efa4c974beee5e29fbc2e3d3b04 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 01:18:53 +0000 Subject: [PATCH 26/33] fleet seeds: prune first, bound unpack memory, back off torn seeds - Prune old versions right after the start-of-run clean, before any fetch, so a run killed in every fetch (MemoryMax, TimeoutStartSec) cannot let .v fill up. - Unpack member by member and let each go; extended (pax, GNU long name) headers at most 1 MiB per member and globally; sparse members refused; folder-name bytes capped; a hard link must name a regular file already unpacked. - A deeply nested list is no list (RecursionError); a folder that cannot be listed tears the seed. - One run fetches for at most RUN_WAIT (2 h), each seed at most the time left, well inside the unit's 3 h TimeoutStartSec. - .torn.json: a torn seed (or a fetch the run was killed in) is tried again after 1, 2, 4, 8, 16, then 24 h; torn on every try for over a day makes the run exit 1, so desk-install check shows FAIL. - Docs: memory limits, prune order, run deadline, backoff, disk budget per seed. Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 21 +- .github/scripts/fork-fleet-desk-seed.py | 190 ++++++++++++++++-- .github/scripts/fork-fleet-desk-seed.test.py | 200 ++++++++++++++++++- fork-kit/desk/README.md | 3 + 4 files changed, 386 insertions(+), 28 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index 2a7f3780f90cf..05f6a1f363988 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -737,12 +737,23 @@ host's `max_mb`: a host cannot make a desktop take more. Each new seed is unpacked with Python's tar data filter and a limit on what it would make on disk (each member's size plus 4 KiB for it and for each folder it makes, all within `max_mb`; at most 200,000 members and folders; no member more than 64 -folders deep or with a name over 1 KiB), checked against the trailer (its size +folders deep or with a name over 1 KiB) and on memory (members are let go as +they are unpacked; the extended pax or long-name headers before one member, and +the global ones of the whole tar, at most 1 MiB; sparse members are refused, so +make a seed without `--sparse`; a hard link must name a file unpacked before +it), checked against the trailer (its size and date must be the list's too), checked again for links that lead out of it, made readable by every job (a cache's 0600 files too), and swapped in by a rename; ssh's stderr and the list are read with a limit. A seed that fails in any way is named in the log and the run goes on with the next; old copies -are pruned in every run. A host that gives no list (down, an older kit, a +are pruned at the start and the end of every run (so a run killed by its unit's +memory or time limit cannot pile them up). A run fetches for at most two hours +in all (the unit allows three), each seed at most 30 minutes. A seed torn again +and again is tried again after 1, 2, 4, 8, 16 and then every 24 hours (a link +too slow for it does not spend every hour on it; a fetch that kills the run +counts as torn), and one torn on every try for more than a day makes the run +exit 1, so `desk-install check` shows FAIL until it is fetched +(`/var/lib/fork-fleet/seeds/.torn.json` names it and why). A host that gives no list (down, an older kit, a half-done install) means no new seeds: each one here stays until its age drops it. A desktop whose own fleet config is broken or lacks its pool fetches nothing and exits 2 (`desk-install check` shows the seed service's last @@ -753,7 +764,11 @@ the unpacking and old copies beside it, `.new` and `.v`, are visible too and hold the same data), copy what they need from it when it is there, and run cold when it is not. A replaced copy is kept 24 hours for a job still reading it; there is no free-space check before unpacking (a full disk ends that seed's -fetch and keeps the old one). The state table (a torn download, the host down, +fetch and keeps the old one). **Disk budget per seed:** up to `max_mb` x +(24 / `every_hours` + 2): the copy in use, the copies replaced in the last 24 +hours, and one being unpacked (`every_hours` 1 and `max_mb` 4096: up to about +104 GiB; `every_hours` 6: about 24 GiB). Size `max_mb` and `every_hours` to the +desktop's free disk. The state table (a torn download, the host down, clocks that disagree, a run that stopped half-way) is in the script's header. ## Owner steps on the repository diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index 563594ba3d3c4..432426950943b 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -19,8 +19,9 @@ every_hours and max_age_hours and the smaller of its own and the host's max_mb. Each new seed is unpacked with tarfile's data filter into a folder of its own (what it would make on disk held to max_mb, ENTRY bytes counted for each member and folder; no member over DEPTH folders deep or - a name over NAME_MOST bytes), checked against the trailer (bytes and MADE the list's too), walked - (a link out of it is refused; all made readable), flushed, and swapped in by rename: SEEDS/NAME is a link to + a name over NAME_MOST bytes; memory held too: see Info and unpack), checked against the + trailer (bytes and MADE the list's too), walked (a link out of it is refused; all made + readable), flushed, and swapped in by rename: SEEDS/NAME is a link to .v/NAME.STAMP/data, root's (folders 0755, files no one else may write), unpacked in .new (one folder: a rename never crosses a mount of the unit's sandbox). Jobs see SEEDS read-only (the launcher's ALLOW_PATHS): a job uses /var/lib/fork-fleet/seeds/NAME when it is there and runs @@ -31,8 +32,12 @@ a new one under every_hours after the last nothing yet a torn copy (the stream cut or past FETCH_WAIT, a trailer that does not match it or the list, more than max_mb, a tar member that would leave its folder, is a device, too deep or too long, or any other error) - the copy is removed, the old seed stays, the next hour tries again; - the other seeds go on, old versions are still pruned + the copy is removed, the old seed stays, the other seeds go on, old + versions are still pruned; tried again after 1, 2, 4, 8, 16, then 24 + hours (TornLog); torn on every try for over TORN_DAY: the run exits 1 + a fetch that kills the run (MemoryMax, TimeoutStartSec) + counted as torn before it began; the next run prunes first + the run's RUN_WAIT used the seeds left are tried next hour no seed (never fetched, the host down) jobs run cold no list (the host down, an older kit that answers with a session or not JSON, exit 1 of a half-done install) nothing fetched; each seed stays until its age drops it @@ -74,11 +79,14 @@ def module(name, file): NAME = re.compile(r'[a-z0-9][a-z0-9_-]{0,39}') LINK = re.compile(r'\.v/((?P[a-z0-9][a-z0-9_-]{0,39})\.[0-9]{1,20})/data') LIST_WAIT, FETCH_WAIT = 30, 30 * 60 # seconds for the list, and for one seed's whole stream +RUN_WAIT = 2 * 3600 # seconds a run may spend fetching, all seeds together (the unit's TimeoutStartSec is 3 h) +TORN_DAY = 24 * 3600 # a seed torn on every try for this long makes the run exit 1 (the unit's result: `check` FAIL) LIST_MOST, ERR_KEEP, MEMBERS = 1 << 20, 4096, 200_000 # the list's bytes, ssh's stderr kept, a seed's tar members KEEP_OLD, SKEW = 24 * 3600, 3600 # SKEW: how far the host's clock may be ahead of the desktop's PIECE, WRITE, STALL, AHEAD = 1 << 20, 64 << 10, 120, 300 # AHEAD: seconds a file's date may be ahead of now GRACE, MIN_RATE = 600, 256 << 10 # a stream's deadline: GRACE seconds plus one second per MIN_RATE bytes DEPTH, NAME_MOST, ENTRY = 64, 1024, 4096 # a member's most folders deep and name bytes; the disk an entry costs +HEADER_MOST, FOLDER_NAMES = 1 << 20, 16 << 20 # extended header bytes (see Info); folder name bytes kept ITEM = {'everyHours': 'every_hours', 'maxAgeHours': 'max_age_hours', 'maxMb': 'max_mb'} @@ -309,7 +317,7 @@ def listing(agent, conf): and whole(s['made'])) out[s['name']] = s return out - except (ValueError, AssertionError): + except (ValueError, AssertionError, RecursionError): # RecursionError: a list nested deeper than json can go raise Torn('the host\'s list is not well formed') from None @@ -321,13 +329,65 @@ def whole(x): return isinstance(x, int) and not isinstance(x, bool) and x >= 0 +EXTENDED = (tarfile.XHDTYPE, tarfile.XGLTYPE, tarfile.SOLARIS_XHDTYPE, tarfile.GNUTYPE_LONGNAME, + tarfile.GNUTYPE_LONGLINK) + + +class Info(tarfile.TarInfo): + """A tar header read with limits tarfile has not: it reads an extended header (pax, or a GNU long name) into + memory whole and its parsed form takes many times its size, and a sparse member's map can grow without end. So the + extended headers before one member may hold at most HEADER_MOST bytes together, the global pax headers of the + whole tar too, and a sparse member is refused (make a seed without --sparse). Torn before tarfile reads one.""" + + def _proc_member(self, tarfile_): + used = tarfile_.__dict__.setdefault('fork_headers', [0, 0]) # [before this member, global in all] + if self.type in EXTENDED: + used[0] += self.size + used[1] += self.size if self.type == tarfile.XGLTYPE else 0 + if max(used) > HEADER_MOST: + raise Torn(f'the tar\'s extended (pax or long-name) headers hold more than {HEADER_MOST >> 20} MB') + else: + used[0] = 0 + if self.type == tarfile.GNUTYPE_SPARSE: + self.sparse_refused() + return super()._proc_member(tarfile_) + + def sparse_refused(self, member=None, *_): + raise Torn(f'{(member or self).name[:80]!r} is a sparse member (make the seed without --sparse)') + + _proc_gnusparse_00 = _proc_gnusparse_01 = _proc_gnusparse_10 = sparse_refused # pax sparse forms + + +def unpack(t, data, most): + """The members of the open stream T into DATA, one at a time: tarfile keeps every member it has read in + `members` (and extractall each folder besides), so they are let go as they are made; only `guarded`'s count of + folders stays. A hard link must name a file already unpacked (otherwise tarfile would read the rest of the + stream into memory to find it).""" + check = guarded(most) + + def linked(member, path): + got = check(member, path) + if got is not None and got.islnk(): + try: + plain = stat.S_ISREG(os.lstat(os.path.join(path, got.linkname)).st_mode) + except OSError: + plain = False + if not plain: + raise Torn(f'{got.name[:80]!r} is a hard link to {got.linkname[:80]!r}, not a file unpacked before it') + return got + while (member := t.next()) is not None: + t.members.clear() + t.extract(member, data, set_attrs=not member.isdir(), filter=linked) # data_filter keeps no folder mode + + def guarded(most): """tarfile's data filter, and a limit on what the tar would make on disk: each member's size (a sparse member is small in the stream, not on disk) plus ENTRY bytes for it and for each folder it makes, together at most MOST bytes (a stream of empty files or deep paths is small but would fill the disk with entries); at most MEMBERS members and MEMBERS folders (each is kept in memory); no member more than DEPTH folders deep (unpacking and - removing a tree goes folder by folder) or with a name or link target over NAME_MOST bytes.""" - seen, folders = [0, 0], {} # folders: {(its parent's number, its name): its number}, a name kept once + removing a tree goes folder by folder) or with a name or link target over NAME_MOST bytes; the folder names + kept to count them at most FOLDER_NAMES bytes.""" + seen, folders = [0, 0, 0], {} # folders: {(its parent's number, its name): its number}, a name kept once def check(member, path): parts = [x for x in member.name.split('/') if x not in ('', '.')] @@ -340,13 +400,15 @@ def check(member, path): for i, part in enumerate(parts if member.isdir() else parts[:-1], 1): # the folders it makes if (up, part) not in folders: folders[up, part] = len(folders) + 1 + seen[2] += len(part) new += i < len(parts) # a folder member pays ENTRY once, as a member up = folders[up, part] seen[0] += 1 seen[1] += member.size + ENTRY * (1 + new) - if seen[0] > MEMBERS or len(folders) > MEMBERS or seen[1] > most: - raise Torn(f'the tar holds more than {MEMBERS} members or folders, or more than {most >> 20} MB ' - f'(max_mb; {ENTRY} bytes counted for each member and folder)') + if seen[0] > MEMBERS or len(folders) > MEMBERS or seen[1] > most or seen[2] > FOLDER_NAMES: + raise Torn(f'the tar holds more than {MEMBERS} members or folders, folder names over ' + f'{FOLDER_NAMES >> 20} MB, or more than {most >> 20} MB (max_mb; {ENTRY} bytes counted for ' + f'each member and folder)') return tarfile.data_filter(member, path) return check @@ -356,7 +418,10 @@ def settle(data): besides the data filter, which some Python releases let links slip past); folders and files readable by all (jobs run as other users; a cache may hold 0600 files), writable by root only. Torn otherwise.""" top = os.path.realpath(data) - for d, dirs, names in os.walk(data): + + def unlisted(e): # a folder os.walk cannot list (a path past PATH_MAX, say) would go unchecked + raise Torn(f'{os.path.relpath(e.filename or data, data)[:80]} cannot be listed: {e.strerror}') + for d, dirs, names in os.walk(data, onerror=unlisted): for n in dirs + names: path = os.path.join(d, n) st = os.lstat(path) @@ -371,22 +436,22 @@ def settle(data): raise Torn(f'{os.path.relpath(path, data)} is not a folder, a file or a link') -def fetch(agent, conf, item, root, now): +def fetch(agent, conf, item, root, now, wait=FETCH_WAIT): """The seed `item` (its maxMb: the smaller of the host's and the desktop's own) into a new version, swapped in; - its label. Torn (or OSError, TarError) leaves the old one.""" + its label. The stream is cut after WAIT seconds. Torn (or OSError, TarError) leaves the old one.""" name, stamp, most = item['name'], time.time_ns(), item['maxMb'] << 20 new = os.path.join(root + '/.new', f'{name}.{stamp}') os.mkdir(new, 0o755) os.mkdir(os.path.join(new, 'data'), 0o755) p = subprocess.Popen(ssh(agent, conf, 'seed', name), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=agent.ENV, start_new_session=True) - timer = threading.Timer(FETCH_WAIT, p.kill) + timer = threading.Timer(wait, p.kill) timer.start() err = Tail(p.stderr) try: got = Summed(p.stdout, most) - with tarfile.open(fileobj=got, mode='r|') as t: - t.extractall(os.path.join(new, 'data'), filter=guarded(most)) + with tarfile.open(fileobj=got, mode='r|', tarinfo=Info) as t: + unpack(t, os.path.join(new, 'data'), most) while got.read(PIECE): # the tar's end blocks and padding count in the trailer's size and sha256 pass code = p.wait(timeout=30) @@ -521,6 +586,7 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): for e in os.listdir(root): # a link made but not yet swapped in if e.startswith('.') and e.endswith('.link'): each(say, f'seed link {e}', remove, os.path.join(root, e)) + prune(root, now, say) # first too: a run killed later (its memory or time limit) skips the finally for name in sorted(n for n in os.listdir(root) if NAME.fullmatch(n)): each(say, f'seed {name}', age_out, root, name, own, now, say) if cannot: @@ -532,14 +598,78 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): except Torn as e: # the host down, or a host on an older kit (no list, or not one this reads) items = {} say(f'seeds: {e}: each seed stays until its age drops it') + torn, end = TornLog(root, now), time.monotonic() + RUN_WAIT + torn.keep(items) for name, item in sorted(items.items()): - if not each(say, f'seed {name}', take, agent, conf, item, root, now, say): + if not each(say, f'seed {name}', take, agent, conf, item, root, now, say, torn, end): clean(root, say) + each(say, 'seeds: .torn.json', torn.save) + if late := torn.late(): + say(f'seeds: {", ".join(late)} torn on every try for more than {TORN_DAY // 3600} h: exit 1 ' + f'(desk-install check shows it)') + return 1 finally: prune(root, now, say) return 0 +class TornLog: + """SEEDS/.torn.json, {name: {"first", "last", "tries", "why"}}: the seeds whose last fetch was torn, written + before each fetch (so a fetch that kills the run, by its memory or time limit, counts as torn too) and cleared + when the seed is fetched or the same as the host's. A seed torn TRIES times in a row is tried again only + min(2 ** (TRIES - 1), 24) hours after the last try (a link too slow for it, a host that cuts it: not every + hour); one torn on every try since more than TORN_DAY is `late`. Only seeds in the host's list are kept. A + missing or unreadable file is an empty record.""" + + def __init__(self, root, now): + self.path, self.now, self.seeds = root and os.path.join(root, '.torn.json'), now, {} + try: + with open(self.path, encoding='utf-8') as f: + got = json.load(f) + self.seeds = {n: r for n, r in got.items() if NAME.fullmatch(n) and isinstance(r, dict) and all( + whole(r.get(k)) for k in ('first', 'last', 'tries'))} + except (OSError, TypeError, ValueError, AttributeError, RecursionError): # TypeError: no path + pass + + def keep(self, items): + self.seeds = {n: r for n, r in self.seeds.items() if n in items} + + def due(self, name): + """None when the seed may be tried now, else the hours until it may.""" + r = self.seeds.get(name) + if not r: + return None + wait = min(2 ** (min(r['tries'], 6) - 1), 24) * 3600 - 300 # 5 min early: the hourly timer drifts + left = r['last'] + wait - self.now() + return math.ceil(left / 3600) if 0 < left <= wait else None # a date ahead of the clock: try now + + def trying(self, name): + """Counted as torn before the fetch, saved: `why` says so until the fetch ends.""" + r = self.seeds.get(name) or {'first': int(self.now()), 'tries': 0} + self.seeds[name] = {**r, 'last': int(self.now()), 'tries': r['tries'] + 1, + 'why': 'the run stopped during the fetch (its memory or time limit?)'} + self.save() + + def torn(self, name, why): + self.seeds[name]['why'] = why[:300] + self.save() + + def fine(self, name): + if self.seeds.pop(name, None): + self.save() + + def late(self): + return sorted(n for n, r in self.seeds.items() if self.now() - r['first'] > TORN_DAY) + + def save(self): + if self.path is None: + return + tmp = self.path + '.tmp' + with open(tmp, 'w', encoding='utf-8') as f: + json.dump(self.seeds, f) + os.replace(tmp, self.path) + + def each(say, what, step, *args): """step(*args); any Exception is said, not raised: one seed never stops the others. False when it failed.""" try: @@ -572,27 +702,43 @@ def age_out(root, name, own, now, say): say(f'seed {name}: dropped, {why}; jobs run cold until a new one comes') -def take(agent, conf, item, root, now, say): - """The listed seed `item`, fetched when it is new and due. False when its fetch failed (the old one stays).""" +def take(agent, conf, item, root, now, say, torn=None, end=None): + """The listed seed `item`, fetched when it is new and due (and not held back after torn tries, and while the + run has time left: END, monotonic). False when its fetch failed (the old one stays).""" name = item['name'] + torn = torn or TornLog(None, now) # one take() alone: nothing saved + end = time.monotonic() + RUN_WAIT if end is None else end label = current(root, name)[1] if 'made' not in item: say(f'seed {name}: the host offers none ({item.get("why", "no reason")}): ' f'{"the one here stays" if label else "jobs run cold"}') + torn.fine(name) # nothing to fetch is not a torn fetch return if label and (label['bytes'], label['made']) == (item['bytes'], item['made']): + torn.fine(name) return if label and 0 <= now() - label['fetched'] < item['everyHours'] * 3600: return if not -SKEW <= now() - item['made'] <= item['maxAgeHours'] * 3600: say(f'seed {name}: the host\'s is too old or made in the future (the clocks disagree): not fetched') return + if (hours := torn.due(name)) is not None: + say(f'seed {name}: torn {torn.seeds[name]["tries"]} times in a row; tried again in about {hours} h') + return + left = end - time.monotonic() + if left < 60: + say(f'seed {name}: this run has used its {RUN_WAIT // 60} minutes; tried again next hour') + return + torn.trying(name) try: - got = fetch(agent, conf, item, root, now) + got = fetch(agent, conf, item, root, now, wait=min(FETCH_WAIT, left)) except Exception as e: # any: a torn stream, a full disk, a tar this Python cannot unpack - say(f'seed {name}: keeps {"the old one" if label else "none"}: {e.__class__.__name__}: {str(e)[:300]}; ' - f'tried again next hour') + why = f'{e.__class__.__name__}: {str(e)[:300]}' + torn.torn(name, why) + hours = min(2 ** (min(torn.seeds[name]['tries'], 6) - 1), 24) + say(f'seed {name}: keeps {"the old one" if label else "none"}: {why}; tried again in {hours} h') return False + torn.fine(name) say(f'seed {name}: {got["bytes"] >> 20} MB, sha256 checked, swapped in') diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 6ac1e5e919e47..60c585a60b64d 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -3,7 +3,7 @@ offer in-process, the desktop's run end to end through a fake ssh that runs the host's side, and the forced command's divert in fork-fleet-desk.py. Temp folders only; no root, no ssh, no network. Run: python3 -I -B . Linux only.""" import contextlib, fcntl, hashlib, importlib.util, io, json, os, shutil, stat, subprocess, sys, tarfile, tempfile, threading -import time, unittest +import signal, time, unittest from unittest import mock sys.dont_write_bytecode = True @@ -47,6 +47,9 @@ def load(name, file): if words[0] == 'seeds' and mode == 'oldhost': # a host on an older kit: a serve session, not a list print('fork-fleet-desk: serving') sys.exit(0) +if words[0] == 'seeds' and mode == 'deep': # a list nested deeper than json can go + print('[' * 200000) + sys.exit(0) if words[0] == 'seeds' and mode == 'exit1': # a host stopped half-way through its install print('fork-fleet-desk: no config', file=sys.stderr) sys.exit(1) @@ -651,10 +654,10 @@ def test_any_failure_of_one_seed_never_stops_the_run_and_old_versions_are_still_ self.deep_tar('m2', 1500) real = seed.fetch - def broken(agent_, conf, item, root, now): + def broken(agent_, conf, item, root, now, **kw): if item['name'] == 'm2': raise RuntimeError('anything at all') - return real(agent_, conf, item, root, now) + return real(agent_, conf, item, root, now, **kw) with mock.patch.object(seed, 'fetch', broken): self.assertEqual(self.run_seed(), 0) self.assertEqual(self.files('npm'), {'a': b'n'}) @@ -670,6 +673,197 @@ def broken(agent_, conf, item, root, now): self.assertIsNone(self.files('m2')) self.assertEqual(os.listdir(os.path.join(self.root, '.new')), ['.lock']) + def raw(self, name, blob): + """FILES/POOL/NAME.tar holding BLOB (tar blocks made by hand), newer than any made before.""" + tmp = os.path.join(self.t, 'files', POOL, f'.{name}.tmp') + with open(tmp, 'wb') as f: + f.write(blob + b'\0' * 1024) + os.chmod(tmp, 0o644) + os.utime(tmp, (time.time() + 5,) * 2) + os.replace(tmp, self.tar(name)) + + @staticmethod + def block(info, data=b'', fmt=tarfile.GNU_FORMAT): + """One member: INFO's header (its size is DATA's) and DATA padded to whole blocks.""" + info.size = len(data) + return info.tobuf(format=fmt) + data + b'\0' * (-len(data) % 512) + + @staticmethod + def pax(**records): + out = b'' + for k, v in records.items(): + line = f' {k}={v}\n'.encode() + n = len(line) + 1 + while n != len(line) + len(str(n)): # the length counts its own digits + n = len(line) + len(str(n)) + out += str(n).encode() + line + return out + + def test_a_run_killed_in_every_fetch_still_prunes(self): # lens237sr2: the MUST + os.makedirs(os.path.join(self.root, '.v', 'm2.1', 'data')) # long replaced, no link names it + os.utime(os.path.join(self.root, '.v', 'm2.1'), (1, 1)) + self.make('m2', {'a': b'm'}) + self.make('npm', {'a': b'n'}) + real = seed.fetch + + def oom(agent_, conf, item, root, now, **kw): # MemoryMax or TimeoutStartSec: SIGKILL, no finally runs + if item['name'] == 'npm': + os.kill(os.getpid(), signal.SIGKILL) + return real(agent_, conf, item, root, now, **kw) + pid = os.fork() + if pid == 0: + try: + with mock.patch.object(seed, 'fetch', oom): + self.run_seed() + finally: + os._exit(0) + self.assertEqual(os.waitpid(pid, 0)[1] & 0x7f, signal.SIGKILL) + self.assertEqual(self.files('m2'), {'a': b'm'}) + self.assertFalse(os.path.exists(os.path.join(self.root, '.v', 'm2.1'))) + with open(os.path.join(self.root, '.torn.json')) as f: # the killed fetch counts as torn + self.assertEqual(json.load(f)['npm']['tries'], 1) + + def test_a_pax_header_seed_stays_under_the_units_memory(self): # lens237sr2: the MUST + big = [{**SEEDS[0], 'max_mb': 64}, SEEDS[1]] # a realistic max_mb (fork-fleet.md shows 4096) + self.config(big) + self.own(big) + body = b''.join(b'%d a.%07d=x\n' % (len(b' a.0000000=x\n') + 2, i) for i in range(2_500_000)) + info = tarfile.TarInfo('pax') + info.type = tarfile.XGLTYPE + self.raw('m2', self.block(info, body, tarfile.USTAR_FORMAT)) + del body + r, w = os.pipe() + pid = os.fork() + if pid == 0: + import resource + code = 2 + try: + os.close(r) + self.run_seed() + os.write(w, json.dumps(self.said).encode()) + code = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss >= 512 << 10 # MemoryMax=512M, in KB + finally: + os._exit(code) + os.close(w) + with os.fdopen(r) as f: + said = f.read() + self.assertEqual(os.waitpid(pid, 0)[1] >> 8, 0, said) # a ~35 MB stream (under max_mb) within 512 MB + self.assertIn('extended (pax or long-name) headers hold more than 1 MB', said) + + def test_extended_headers_and_sparse_members_are_held_to_limits(self): + long_name = tarfile.TarInfo('././@LongLink') + long_name.type = tarfile.GNUTYPE_LONGNAME + chain = tarfile.TarInfo('pax') + chain.type = tarfile.XHDTYPE + old_sparse = tarfile.TarInfo('s') + old_sparse.type = tarfile.GNUTYPE_SPARSE + pax_sparse = tarfile.TarInfo('pax') + pax_sparse.type = tarfile.XHDTYPE + cases = { + 'a GNU long name of 1.5 MB': (self.block(long_name, b'n' * (3 << 19) + b'\0') + + self.block(tarfile.TarInfo('f'), b'x'), 'extended'), + 'pax headers of 400 KB each, three before one member': ( + self.block(chain, self.pax(comment='c' * (400 << 10))) * 3 + self.block(tarfile.TarInfo('f'), b'x'), + 'extended'), + 'an old GNU sparse member': (self.block(old_sparse), 'sparse member'), + 'a pax (1.0) sparse member': ( + self.block(pax_sparse, self.pax(**{'GNU.sparse.major': 1, 'GNU.sparse.minor': 0, + 'GNU.sparse.name': 's', 'GNU.sparse.realsize': 0})) + + self.block(tarfile.TarInfo('s'), b'0\n'.ljust(512, b'\0')), 'sparse member'), + } + for what, (blob, why) in cases.items(): + with self.subTest(what): + self.raw('m2', blob) + self.run_seed() + self.assertIsNone(self.files('m2')) + self.assertTrue(any('m2' in x and why in x for x in self.said), self.said) + with contextlib.suppress(FileNotFoundError): # each case tried at once, not after a wait + os.unlink(os.path.join(self.root, '.torn.json')) + for k in ('_proc_member', '_proc_sparse', '_proc_gnusparse_00', '_proc_gnusparse_01', '_proc_gnusparse_10'): + self.assertTrue(callable(getattr(tarfile.TarInfo, k, None)), f'tarfile.TarInfo.{k} is gone: see Info') + + def test_members_are_let_go_as_they_are_unpacked(self): + self.make('m2', {f'd/{i}': b'x' for i in range(300)}) + most, real = [0], tarfile.TarFile.next + + def counted(t): + got = real(t) + most[0] = max(most[0], len(t.members)) + return got + with mock.patch.object(tarfile.TarFile, 'next', counted): + self.run_seed() + self.assertEqual(len(self.files('m2')), 300) + self.assertLessEqual(most[0], 2) + + def test_a_hard_link_must_name_a_file_unpacked_before_it(self): + def hard(name, target): + info = tarfile.TarInfo(name) + info.type, info.linkname = tarfile.LNKTYPE, target + return self.block(info) + self.raw('m2', self.block(tarfile.TarInfo('a'), b'one') + hard('b', 'a')) + self.run_seed() + self.assertEqual(self.files('m2'), {'a': b'one', 'b': b'one'}) + self.raw('m2', self.block(tarfile.TarInfo('a'), b'two') + hard('b', 'later') + + self.block(tarfile.TarInfo('later'), b'x')) + self.run_seed(now=lambda: time.time() + 7 * 3600) + self.assertEqual(self.files('m2'), {'a': b'one', 'b': b'one'}) # the old one stays + self.assertTrue(any('hard link' in x for x in self.said), self.said) + + def test_a_deeply_nested_list_means_no_list(self): # lens237cr2 + self.make('m2', {'a': b'one'}) + self.run_seed() + self.mode('deep') + self.assertEqual(self.run_seed(now=lambda: time.time() + 7 * 3600), 0) + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertTrue(any('not well formed' in x for x in self.said), self.said) + + def test_a_folder_that_cannot_be_listed_tears_the_seed(self): + self.make('m2', {'a/b/c': b'one'}) + real = os.scandir + + def scandir(path='.'): + if isinstance(path, str) and path.endswith(os.sep + 'b'): + raise OSError(36, 'File name too long', path) + return real(path) + with mock.patch.object(os, 'scandir', scandir): + self.run_seed() + self.assertIsNone(self.files('m2')) + self.assertTrue(any('cannot be listed' in x for x in self.said), self.said) + + def test_a_run_stops_fetching_when_its_time_is_used(self): + self.make('m2', {'a': b'one'}) + with mock.patch.object(seed, 'RUN_WAIT', 30): + self.assertEqual(self.run_seed(), 0) + self.assertIsNone(self.files('m2')) + self.assertEqual(self.fetches(), []) + self.assertTrue(any('this run has used its' in x for x in self.said), self.said) + waits = [] + real = seed.fetch + + def timed(agent_, conf, item, root, now, wait=None): + waits.append(wait) + return real(agent_, conf, item, root, now, wait=wait) + with mock.patch.object(seed, 'RUN_WAIT', 600), mock.patch.object(seed, 'fetch', timed): + self.run_seed() + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertTrue(waits and all(500 < w <= 600 for w in waits), waits) # each fetch within what is left + + def test_a_seed_torn_every_time_backs_off_then_fails_the_run_after_a_day(self): + self.make('m2', {'a': b'one'}) + start, tried = int(time.time()), [] + + def torn(agent_, conf, item, root, now, **kw): + tried.append(now() - start) + raise seed.Torn('cut off') + with mock.patch.object(seed, 'fetch', torn): + codes = {h: self.run_seed(now=lambda h=h: start + h * 3600) for h in range(0, 33)} + self.assertEqual([round(x / 3600) for x in tried], [0, 1, 3, 7, 15, 31]) # 1, 2, 4, 8, 16 h apart + self.assertEqual({h for h, c in codes.items() if c}, set(range(25, 33))) # torn for more than 24 h: exit 1 + self.assertTrue(any('torn on every try' in x for x in self.said), self.said) + self.assertEqual(self.run_seed(now=lambda: start + 63 * 3600), 0) # fetched at last: the record goes + self.assertEqual(self.files('m2'), {'a': b'one'}) + self.assertEqual(self.run_seed(now=lambda: start + 64 * 3600), 0) + def test_a_tree_deeper_than_recursion_left_by_an_older_run_is_removed(self): self.make('m2', {'a': b'one'}) self.run_seed() diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index b5c4f9c476baa..17e58d046b3aa 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -65,6 +65,9 @@ names the fix (`usermod -s /usr/sbin/nologin`, `passwd -l`). One install or unin The seed timer (`t3code-fleet-seed.timer`) fetches the pool's warm caches, if its fleet config lists any, into `/var/lib/fork-fleet/seeds`, which jobs see read-only ([fork-fleet.md](../../.github/fork-fleet.md), "Seeds"). +A torn seed is tried again after 1, 2, 4, 8, 16, then 24 hours; one torn on every try for more than a day makes each +run exit 1, so `check` shows the seed service's last run as failed. `/var/lib/fork-fleet/seeds/.torn.json` names it +and the host's reason (jobs can read that file too). ## Windows (WSL2) From 1ccced037826fcfcac283531f5a6f55f60dbd835 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 01:53:33 +0000 Subject: [PATCH 27/33] fork kit desk: fix round 4 (linked or foreign folders and key refused, runner users as strict as the launcher, lock in /etc/fork-fleet, owner of units and settings) (D8) Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 25 +++-- fork-kit/desk/sbin/desk-install | 82 +++++++++++++---- fork-kit/desk/tests/install.test.py | 138 +++++++++++++++++++++++++--- 3 files changed, 200 insertions(+), 45 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 9a18bbbc1d5a2..703577dcd3c4c 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -49,22 +49,27 @@ the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner temp the refresh timer; `check` says FAIL for `template` until it is there. `check` only reads: the installed files against their manifest (text, root owner and mode), the folders' and the -desk key's modes, the project's fleet config against the installed one, each unit against the release's (header plus -`/opt/fork-fleet/units/UNIT`), a restart a stopped install still owes, systemd drop-ins that change the kit's units, +desk key's modes and root owner (not a link), the project's fleet config against the installed one, each unit against +the release's (header plus `/opt/fork-fleet/units/UNIT`, mode and root owner), a restart a stopped install still owes, systemd drop-ins that change the kit's units, every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings -files against the machine file, the runner users (each must exist, have no login shell, a locked password and no -line in `cron.allow`/`at.allow`; and it names a user the kit made that a smaller `slots` no longer uses) and IPv4 +files against the machine file (text, mode and root owner), the runner users (each must exist, have no login shell, +a locked password, a uid other than 0, its own group only, no home that is there (`/nonexistent`) and no line in +`cron.allow`/`at.allow`; and it names a user the kit made that a smaller `slots` no longer uses) and IPv4 forwarding. -A `fork-runner-N` user made by hand before the install is used, but only when it is as locked as one the kit makes -(no login shell, locked password, not in an allow file): otherwise install refuses before changing anything and -names the fix (`usermod -s /usr/sbin/nologin`, `passwd -l`). One install or uninstall runs at a time (a lock on -`/opt/fork-fleet`); a second one is refused. +A `fork-runner-N` user made by hand before the install is used, but only when it passes the same checks (no login +shell, locked password, uid not 0, its own group only, no home that is there, not in an allow file; the launcher +refuses the same uid and group cases at each job start): otherwise install refuses before changing anything and +names the fix (`usermod -s /usr/sbin/nologin`, `passwd -l`, `gpasswd -d`, `usermod -d /nonexistent`). A +`/opt/fork-fleet`, `/opt/fork-fleet/template`, `/etc/fork-fleet` or desk key that is a link or not owned by root (a +hand setup) is refused the same way, not followed or fixed. One install or uninstall runs at a time (a lock on +`/etc/fork-fleet`, which only root can open); a second one is refused. ## Runs again, stops half-way, old or empty state -- **Again:** files, settings and units that did not change are not rewritten (a file whose mode or owner was - changed is put back from the release), and the agent is not stopped or +- **Again:** files, settings and units that did not change are not rewritten (a file in `/opt/fork-fleet`, a + setting or a unit whose mode or owner was changed, or that is a link, is written again as root's; the folders' + modes are set back; a linked or foreign-owned folder or key is refused, above), and the agent is not stopped or restarted (a restart would end a job it is starting). After a change it is restarted. The agent reads its settings only when it starts, so before the first change the record gets a restart-pending mark, cleared after the restart: a run stopped in between restarts the agent on the next run. The mark is also set before a new fleet diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index e444030744255..8a24cd7cbe214 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -91,13 +91,22 @@ def user_exists(name): def user_problem(u): """Why runner user u (made by the kit or by hand) could be used for more than one-job runners: a login shell, a - usable password (or none), a line in cron.allow/at.allow; None when it is fine. Read through getent (as root).""" + usable password (or none), uid 0, a group besides its own, a home that is there, a line in cron.allow/at.allow; + None when it is fine. The launcher's check_users refuses the same users at each job start (uid, groups). Read + through getent and id (as root).""" r = run('getent', 'passwd', u, check=False) fields = r.stdout.strip().split(':') if r.returncode or len(fields) < 7: return f'{u}: its passwd entry cannot be read' if fields[6] not in NOLOGIN: return f'{u} has the login shell {fields[6]!r} (usermod -s /usr/sbin/nologin {u})' + if fields[2] == '0' or not fields[2].isdigit(): + return f'{u} has uid {fields[2]}: a runner user is a user of its own, not root (userdel {u}; install makes it)' + gid, groups = run('id', '-g', u, check=False), run('id', '-G', u, check=False) + if gid.returncode or groups.returncode or groups.stdout.split() != gid.stdout.split(): + return f'{u} is in other groups ({groups.stdout.strip()}): a runner user is in its own group only (gpasswd -d)' + if fields[5] not in ('', '/nonexistent') and os.path.lexists(p(fields[5])): + return f'{u} has the home {fields[5]} (usermod -d /nonexistent {u}): a home could hold its ssh keys' r = run('getent', 'shadow', u, check=False) fields = r.stdout.strip().split(':') if r.returncode or len(fields) < 2: @@ -111,6 +120,33 @@ def user_problem(u): return None +def not_root_owned(path, kind): + """Why a folder or file the root-run kit trusts is not safe (a link, another kind, not owned by root); None when it + is, or absent. Install refuses such a path rather than follow it or fix it (chown would follow a link).""" + try: + st = os.lstat(p(path)) + except FileNotFoundError: + return None + if stat.S_ISLNK(st.st_mode): + return f'{path} is a link (a hand setup?): remove the link and run again' + if not kind(st.st_mode): + return f'{path} is not a {"folder" if kind is stat.S_ISDIR else "plain file"}: remove it and run again' + if st.st_uid != ROOT_UID: + return f'{path} is not owned by root (uid {st.st_uid}): chown root {path} (or remove it) and run again' + return None + + +def same_file(real, text, mode): + """The file (not a link) holds text, has mode and is owned by root.""" + try: + st = os.lstat(real) + with open(real, encoding='utf-8') as f: + return (stat.S_ISREG(st.st_mode) and st.st_uid == ROOT_UID and stat.S_IMODE(st.st_mode) == mode + and f.read() == text) + except (OSError, UnicodeDecodeError): + return False + + def job_writable(path): """Why a job could change this file (it, or a folder above it, writable by others or by a runner user's group, or owned by a runner user); None when only root or the caller can. A sticky folder (/tmp) is fine: nobody else may @@ -199,12 +235,11 @@ def read_state(): def write_file(path, text, mode=0o644): - """Atomic (a temp file in the same folder, renamed over the old one); unchanged text is not rewritten.""" + """Atomic (a temp file in the same folder, renamed over the old one); unchanged text is not rewritten (a link, a + changed mode or an owner other than root is: the new file is root's).""" real = p(path) - with contextlib.suppress(OSError): - with open(real, encoding='utf-8') as f: - if f.read() == text and os.stat(real).st_mode & 0o777 == mode: - return False + if same_file(real, text, mode): + return False os.makedirs(os.path.dirname(real), mode=0o755, exist_ok=True) fd, tmp = tempfile.mkstemp(dir=os.path.dirname(real), prefix='.' + os.path.basename(real) + '.') try: @@ -404,9 +439,13 @@ def make_key(pool, before_change): def lock(stack): - """One install or uninstall at a time: a lock on the /opt/fork-fleet folder itself (nothing to clean up), held - until the run ends. A second run is refused, not queued: it would sweep the first one's staging folder.""" - fd = os.open(p(OPT), os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC) + """One install or uninstall at a time: a lock on the /etc/fork-fleet folder itself (0700, so only root can open it + and hold it; nothing to clean up), held until the run ends. A second run is refused, not queued: it would sweep the + first one's staging folder.""" + try: + fd = os.open(p(ETC), os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_CLOEXEC) + except OSError as e: + raise Refused(f'{ETC} cannot be locked ({e.strerror}): it must be a folder, not a link') from None stack.callback(os.close, fd) try: fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) @@ -457,6 +496,10 @@ def install(a, stop_after=None, stack=None): raise Refused(f'the install record says a project\'s fleet config, but neither {PROJECT} nor ' f'{OPT}/fork-fleet.json is there: run again with --fleet-config FILE or ' '--release-fleet-config') + for path, kind in ((OPT, stat.S_ISDIR), (OPT + '/template', stat.S_ISDIR), (ETC, stat.S_ISDIR), + (KEY, stat.S_ISREG), (KEY + '.pub', stat.S_ISREG)): + if not_root_owned(path, kind): # a hand setup: the root-run launcher must not sit where a user can write + raise Refused(not_root_owned(path, kind)) users = [f'fork-runner-{i}' for i in range(1, m['slots'] + 1)] for u in users: # a runner user made by hand must be as locked as one the kit makes if user_exists(u) and user_problem(u): @@ -531,11 +574,7 @@ def install(a, stop_after=None, stack=None): def same_text(path, text, mode): - try: - with open(p(path), encoding='utf-8') as f: - return f.read() == text and os.stat(p(path)).st_mode & 0o777 == mode - except (OSError, UnicodeDecodeError): - return False + return same_file(p(path), text, mode) def check(): @@ -576,7 +615,8 @@ def check(): row(False, 'machine-file', str(e)[:300]) for u in users: why = f'{u} does not exist' if not user_exists(u) else user_problem(u) - row(not why, 'runner-user', why or f'{u} exists: no login shell, no password, not in cron.allow/at.allow') + row(not why, 'runner-user', why or f'{u} exists: no login shell, no password, own group only, no home, ' + 'not in cron.allow/at.allow') with contextlib.suppress(Refused): # a smaller `slots` leaves the kit's extra users: named, not lost for u in (read_state() or {}).get('createdUsers', []) if users else []: if u not in users and user_exists(u): @@ -619,8 +659,10 @@ def installed_problems(): st = os.lstat(p(path)) except OSError: continue # a missing key or folder is named by its own check (agent settings, template) - if stat.S_ISLNK(st.st_mode) or st.st_uid != ROOT_UID or stat.S_IMODE(st.st_mode) != mode: - problems.append(f'{path}: not root-owned with mode {mode:o}: install again') + if stat.S_ISLNK(st.st_mode) or st.st_uid != ROOT_UID: + problems.append(f'{path}: a link or not owned by root: remove the link (or chown root), then install again') + elif stat.S_IMODE(st.st_mode) != mode: + problems.append(f'{path}: mode {stat.S_IMODE(st.st_mode):o}, not {mode:o}: install again') if state.get('fleetConfig') == 'project': try: ok = sha256(p(PROJECT)) == manifest.get('fork-fleet.json') @@ -646,7 +688,8 @@ def installed_problems(): if not ours(f'{UNIT_DIR}/{u}'): problems.append(f'{UNIT_DIR}/{u}: missing or not the kit\'s') elif want is None or not same_text(f'{UNIT_DIR}/{u}', want, 0o644): - problems.append(f'{UNIT_DIR}/{u}: not the unit of release {state["sha"]} ({OPT}/units/{u}): install again') + problems.append(f'{UNIT_DIR}/{u}: not the unit of release {state["sha"]} ({OPT}/units/{u}), or its mode or ' + 'owner changed: install again') problems += [f'{d}: a drop-in the kit did not write changes {u}' for u in UNITS for d in drop_ins(u)] return problems @@ -689,8 +732,7 @@ def uninstall(stack=None): if state is None: say(f'no {STATE}: nothing here was installed by this kit (nothing changed)') return 0 - if os.path.isdir(p(OPT)): - lock(stack if stack is not None else contextlib.ExitStack()) + lock(stack if stack is not None else contextlib.ExitStack()) # the record is in /etc/fork-fleet: it is there for unit in (AGENT, TIMER, REFRESH): run('systemctl', 'disable', '--now', unit, check=False) launcher = p(OPT + '/fork-fleet-launch.sh') diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index abde4a69192b3..a5b813f994482 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -5,7 +5,7 @@ python3 -I -B fork-kit/desk/tests/install.test.py """ import contextlib, fcntl, importlib.machinery, importlib.util, io, json, os, shutil, stat, subprocess, sys, tempfile -import types, unittest +import types, unittest, unittest.mock KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) REPO = os.path.dirname(os.path.dirname(KIT)) @@ -35,19 +35,22 @@ def put(n, items): put('fail-after', [x for x in lines('fail-after') if x != name]) put('users', lines('users') + [args[-1]]) sys.exit(9) -if name == 'id': - sys.exit(0 if args[-1] in lines('users') else 1) -if name == 'getent': # a user's login shell: the file shell-USER (nologin); its password field: pw-USER (!) +def one(n, default): # a per-user setting: the file N-USER + try: + return open(os.path.join(here, n + '-' + args[-1])).read().strip() + except FileNotFoundError: + return default +if name == 'id': # its groups: the file groups-USER (its own group only) + if args[-1] not in lines('users'): + sys.exit(1) + print({'-u': one('uid', '1001'), '-g': '1001', '-G': one('groups', '1001')}.get(args[0], '')) +if name == 'getent': # login shell: shell-USER (nologin); password field: pw-USER (!); uid-USER; home-USER u = args[-1] if u not in lines('users'): sys.exit(2) - def one(n, default): - try: - return open(os.path.join(here, n + '-' + u)).read().strip() - except FileNotFoundError: - return default if args[0] == 'passwd': - print(f'{u}:x:1001:1001::/nonexistent:' + one('shell', '/usr/sbin/nologin')) + print(f'{u}:x:' + one('uid', '1001') + ':1001::' + one('home', '/nonexistent') + ':' + + one('shell', '/usr/sbin/nologin')) else: print(f'{u}:' + one('pw', '!') + ':19000:0:99999:7:::') if name == 'useradd': @@ -454,9 +457,9 @@ def test_install_sets_back_modes_check_names(self): code, out, _ = self.run_main('check') self.assertEqual(code, 1) for line in ('FAIL kit /opt/fork-fleet/fork-fleet-launch.sh: mode 777, not 755', - 'FAIL kit /opt/fork-fleet/fork-fleet-walls: not root-owned with mode 755', - 'FAIL kit /etc/fork-fleet/desk-key: not root-owned with mode 600', - 'FAIL kit /etc/fork-fleet: not root-owned with mode 700'): + 'FAIL kit /opt/fork-fleet/fork-fleet-walls: mode 777, not 755', + 'FAIL kit /etc/fork-fleet/desk-key: mode 644, not 600', + 'FAIL kit /etc/fork-fleet: mode 755, not 700'): self.assertIn(line, out) self.assertEqual(self.install()[0], 0) self.assertEqual(stat.S_IMODE(os.stat(self.path('/opt/fork-fleet/fork-fleet-launch.sh')).st_mode), 0o755) @@ -465,7 +468,7 @@ def test_install_sets_back_modes_check_names(self): def test_a_second_install_or_uninstall_at_the_same_time_is_refused(self): self.assertEqual(self.install()[0], 0) - fd = os.open(self.path('/opt/fork-fleet'), os.O_RDONLY | os.O_DIRECTORY) + fd = os.open(self.path('/etc/fork-fleet'), os.O_RDONLY | os.O_DIRECTORY) try: fcntl.flock(fd, fcntl.LOCK_EX) stage = os.path.join(self.path('/opt/fork-fleet'), '.kit-stage-other') @@ -481,6 +484,83 @@ def test_a_second_install_or_uninstall_at_the_same_time_is_refused(self): os.close(fd) self.assertEqual(self.install(sha=SHA2)[0], 0) + def test_the_lock_is_in_the_root_only_folder(self): + """lens225r3 nit 4: a lock on the world-readable /opt/fork-fleet could be held by any local user; the lock is + on /etc/fork-fleet (0700), which only root can open.""" + self.assertEqual(self.install()[0], 0) + fd = os.open(self.path('/opt/fork-fleet'), os.O_RDONLY | os.O_DIRECTORY) + try: + fcntl.flock(fd, fcntl.LOCK_EX) # what any local user could do: install is not blocked by it + self.assertEqual(self.install(sha=SHA2)[0], 0) + self.assertEqual(self.run_main('uninstall')[0], 0) + finally: + os.close(fd) + + def test_a_linked_or_foreign_owned_folder_or_key_is_refused(self): + """lens225r3 nit 1 (probe): install neither followed nor fixed a linked or non-root /opt/fork-fleet, + /etc/fork-fleet or desk key, so `check` FAILed forever. Install now refuses before changing anything.""" + other = os.path.join(self.tmp, 'elsewhere') + os.makedirs(other) + os.makedirs(os.path.dirname(self.path('/opt/fork-fleet')), exist_ok=True) + os.symlink(other, self.path('/opt/fork-fleet')) + code, _, err = self.install() + self.assertEqual(code, 1) + self.assertIn('/opt/fork-fleet is a link', err) + self.assertEqual(os.listdir(other), []) + os.unlink(self.path('/opt/fork-fleet')) + self.assertEqual(self.install()[0], 0) + self.template() + self.assertEqual(self.run_main('check')[0], 0) + real_lstat, opt = os.lstat, self.path('/opt/fork-fleet') + + def lstat(path, *a, **kw): # /opt/fork-fleet owned by another user (a test cannot chown) + st = real_lstat(path, *a, **kw) + if os.fspath(path) != opt: + return st + return os.stat_result((st.st_mode, st.st_ino, st.st_dev, st.st_nlink, st.st_uid + 1, *st[5:])) + before = self.tree() + with unittest.mock.patch.object(self.m.os, 'lstat', lstat): + code, _, err = self.install(sha=SHA2) + self.assertIn('FAIL kit /opt/fork-fleet: a link or not owned by root', self.run_main('check')[1]) + self.assertEqual(code, 1) + self.assertIn('/opt/fork-fleet is not owned by root', err) + self.assertEqual(self.tree(), before) + + def test_a_unit_or_setting_with_another_owner_fails_check_and_install_puts_it_back(self): + """lens225r3 nit 5: units were checked for text and mode, not owner. A unit or setting that is a link or not + root's FAILs check and is written again (as root) by install.""" + self.assertEqual(self.install()[0], 0) + self.template() + unit = self.path('/etc/systemd/system/fork-fleet-desk-agent.service') + text = slurp(unit) + real = os.path.join(self.tmp, 'unit-elsewhere') + self.write(real, text) + os.unlink(unit) + os.symlink(real, unit) # same text through a link: not root's file + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + self.assertIn('fork-fleet-desk-agent.service: not the unit of release', out) + self.assertEqual(self.install()[0], 0) + self.assertFalse(os.path.islink(unit)) + self.assertEqual(self.run_main('check')[0], 0) + + def test_a_hand_made_user_in_other_groups_or_with_a_home_fails_check(self): + """lens225r3 nit 3: user_problem now matches the launcher's check_users (uid not 0, own group only) and wants + no home that is there.""" + self.assertEqual(self.install()[0], 0) + self.template() + os.makedirs(self.path('/home/fork-runner-1')) + for name, text, want in (('groups-fork-runner-1', '1001 27', 'fork-runner-1 is in other groups (1001 27)'), + ('uid-fork-runner-2', '0', 'fork-runner-2 has uid 0'), + ('home-fork-runner-1', '/home/fork-runner-1', 'fork-runner-1 has the home')): + with self.subTest(name): + self.write(os.path.join(self.fakes, name), text + '\n') + code, out, _ = self.run_main('check') + self.assertEqual(code, 1) + self.assertIn('FAIL runner-user ' + want, out) + os.unlink(os.path.join(self.fakes, name)) + self.assertEqual(self.run_main('check')[0], 0) + def test_check_fails_on_a_runner_user_that_can_log_in(self): self.assertEqual(self.install()[0], 0) self.template() @@ -532,6 +612,34 @@ def test_refusals_change_nothing(self): lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-2\n'), self.write(os.path.join(self.fakes, 'pw-fork-runner-2'), '$6$salt$hash\n')), [], 'usable password'), + 'a hand-made runner user with uid 0': ( + lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n'), + self.write(os.path.join(self.fakes, 'uid-fork-runner-1'), '0\n')), [], 'has uid 0'), + 'a hand-made runner user in other groups': ( + lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-2\n'), + self.write(os.path.join(self.fakes, 'groups-fork-runner-2'), '1001 27\n')), [], + 'is in other groups (1001 27)'), + 'a hand-made runner user with a home': ( + lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n'), + os.makedirs(self.path('/home/fork-runner-1')), + self.write(os.path.join(self.fakes, 'home-fork-runner-1'), '/home/fork-runner-1\n')), [], + 'has the home /home/fork-runner-1'), + 'a linked /opt/fork-fleet': ( + lambda: (os.makedirs(os.path.join(self.tmp, 'elsewhere'), exist_ok=True), + os.makedirs(self.path('/opt')), + os.symlink(os.path.join(self.tmp, 'elsewhere'), self.path('/opt/fork-fleet'))), [], + '/opt/fork-fleet is a link'), + 'a linked /etc/fork-fleet': ( + lambda: (os.makedirs(os.path.join(self.tmp, 'elsewhere'), exist_ok=True), + os.makedirs(self.path('/etc')), + os.symlink(os.path.join(self.tmp, 'elsewhere'), self.path('/etc/fork-fleet'))), [], + '/etc/fork-fleet is a link'), + 'a linked desk key': ( + lambda: (self.write(self.path('/etc/fork-fleet/desk-kit.json'), json.dumps( + {'version': 1, 'createdUsers': [], 'fleetConfig': 'release', 'sha': SHA1}), 0o600), + self.write(os.path.join(self.tmp, 'elsewhere-key'), 'PRIVATE\n', 0o600), + os.symlink(os.path.join(self.tmp, 'elsewhere-key'), self.path('/etc/fork-fleet/desk-key'))), + [], '/etc/fork-fleet/desk-key is a link'), 'a hand-made runner user in cron.allow': ( lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n'), self.write(self.path('/etc/cron.allow'), 'fork-runner-1\n')), [], 'is in /etc/cron.allow'), @@ -550,7 +658,7 @@ def test_refusals_change_nothing(self): shutil.rmtree(self.root, ignore_errors=True) os.makedirs(self.root) for n in os.listdir(self.fakes): # the users of the case before - if n == 'users' or n.startswith(('shell-', 'pw-')): + if n == 'users' or n.startswith(('shell-', 'pw-', 'uid-', 'groups-', 'home-')): os.unlink(os.path.join(self.fakes, n)) self.machine(MACHINE) self.write(os.path.join(self.fakes, 'os-exit'), '0') From b59601fb71a323d17159f283180174f1ab8d0dc4 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 01:56:11 +0000 Subject: [PATCH 28/33] fork kit desk windows: fix round 4 (owed mark before wsl.conf changes, question and merge decide the same way) (D9) Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 12 ++--- fork-kit/desk/tests/windows.test.py | 65 +++++++++++++++++++++++++++- fork-kit/desk/windows/start-desk.ps1 | 50 +++++++++++++-------- 3 files changed, 101 insertions(+), 26 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 70bd5b1a46c2a..35e39e3db938e 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -86,13 +86,15 @@ three quarters of the RAM, at most the RAM less 6 GB, never under 4 GB; one that refused), swap (`-SwapGB`, 4), NAT and DNS tunnelling, for every WSL distro (read and written as UTF-8 without a BOM; a dated copy of the old file is kept, other keys and sections stay; an old `networkingMode` or `dnsTunneling` line under `[experimental]`, the form before WSL 2.0 that can still win over `[wsl2]`, is removed, and the check -FAILs while one is there); the distro's own settings the WSL2 walls +FAILs while one is there; the question lists each change, old -> new, and the file is written only when it lists one, +so "no change" means the file is left alone); the distro's own settings the WSL2 walls ask for (`/etc/wsl.conf`: systemd on, no Windows drives, programs or PATH; its own `/etc/resolv.conf`; WSL's binfmt handler masked); one `wsl --shutdown` when one of these changed or the running WSL does not have them yet (PID 1 not -systemd, interop live, Windows drives mounted, `wsl.conf` newer than PID 1's start (read from `/proc/1/stat` and -`btime`; unreadable means restart), `.wslconfig` newer than the running WSL, more memory than the cap, not NAT, or a -`.wslconfig` change whose shutdown is still owed: one mark in `%LOCALAPPDATA%\fork-desk` for every distro, like -`.wslconfig` itself, written before the change and removed after the shutdown, because WSL's uptime skips sleep), +systemd, interop live, Windows drives mounted, `wsl.conf` changed after PID 1 started as far as the VM's clock tells +(`/proc/1/stat` and `btime`; unreadable means restart; the clock skips the PC's sleep, so this alone can miss a change, +hence the mark below), `.wslconfig` newer than the running WSL, more memory than the cap, not NAT, or a `.wslconfig` +or `wsl.conf` change whose shutdown is still owed: one mark in `%LOCALAPPDATA%\fork-desk` for every distro, like +`.wslconfig` itself, written before either change and removed after the shutdown, because WSL's uptime skips sleep), so a run stopped before its shutdown restarts next time (every WSL distro and Docker Desktop stop); the packages the fleet needs; the machine file and `-FleetConfig`, if given; this kit's bootstrap; a logon task that keeps the distro running while the owner is logged in. Last, `desk-install check` and the Windows side's own PASS/FAIL lines (the `[wsl2]` keys); diff --git a/fork-kit/desk/tests/windows.test.py b/fork-kit/desk/tests/windows.test.py index dc7f534b56272..1863e6b59b3bc 100644 --- a/fork-kit/desk/tests/windows.test.py +++ b/fork-kit/desk/tests/windows.test.py @@ -91,6 +91,19 @@ def etc(self): out[full] = os.readlink(full) if os.path.islink(full) else slurp(full) return out + def test_check_only_says_whether_wsl_conf_changes_and_writes_nothing(self): + def check_only(): + r = subprocess.run(['bash', '-s'], input='CHECK_ONLY=1\n' + prepare() + '\n# end\r\n', text=True, + capture_output=True, env={'PATH': '/usr/bin:/bin', 'ROOT': self.root}) + self.assertEqual(r.returncode, 0, r.stderr) + return r.stdout + before = self.etc() + self.assertEqual(check_only(), 'differs: /etc/wsl.conf\n') + self.assertEqual(self.etc(), before) + self.assertFalse(os.path.exists(self.root + '/etc/binfmt.d')) + self.run_prepare() + self.assertEqual(check_only(), '') + def test_an_empty_distro(self): self.assertIn('changed: /etc/resolv.conf', self.run_prepare()) self.assertFalse([n for n in os.listdir(self.root + '/etc') if '.bak-' in n]) @@ -261,9 +274,10 @@ def test_wsl_version_is_read_from_the_list(self): def test_old_experimental_networking_keys_are_removed_named_and_checked(self): part = wslconfig_part() self.assertIn("$Experimental = @('networkingMode', 'dnsTunneling')", part) - merge = part[part.index('function Merge-WslConfig'):part.index('function WslVersion')] + merge = part[part.index('function MergeLines'):part.index('function WslVersion')] self.assertIn("$sec -eq 'experimental'", merge) - self.assertIn('$Matches[1] -in $Experimental) { continue }', merge) + self.assertIn('$Matches[1] -in $Experimental) {', merge) + self.assertIn(': removed (the [wsl2] one is used)"); continue }', merge) src = text().decode('ascii') body = src[src.index('try {'):] ask = min(body.index("Read-Host 'Type yes"), body.index('Read-Host "Type the distro')) @@ -272,6 +286,53 @@ def test_old_experimental_networking_keys_are_removed_named_and_checked(self): self.assertIn("SectionKeys (Read-WslConfig $WslConfig) 'experimental'", check) self.assertIn('FAIL wslconfig [experimental] $k=', check) + def test_the_question_and_the_merge_decide_the_same_way(self): + """lens d8min r3 N2: the question compared trimmed values, the merge whole text, so "no change" could be shown + while the file was rewritten. Both now come from MergeLines: the file is written only when it has a note.""" + part = wslconfig_part() + self.assertIn('function Changes([string[]]$lines, $want) { (MergeLines $lines $want).Notes }', part) + merge = part[part.index('function Merge-WslConfig'):part.index('function WslVersion')] + self.assertIn('$merged = MergeLines (Read-WslConfig $path) $want', merge) + self.assertIn('if ($merged.Notes.Count -eq 0) { return $false }', merge) + self.assertNotIn('-join', merge) # no second, text-wide test + + @unittest.skipUnless(shutil.which('pwsh'), 'pwsh is not installed') + def test_no_change_means_the_file_is_left_alone(self): + d = tempfile.mkdtemp(prefix='desk-wslconfig-test-') + self.addCleanup(shutil.rmtree, d, True) + path = os.path.join(d, '.wslconfig') + for old, notes in (('[wsl2]\nmemory = 9GB\nNetworkingMode=NAT\n', []), + ('[wsl2]\nmemory=9GB\nnetworkingMode=nat\nmemory=12GB\n', + ['[wsl2] memory=12GB: a repeated line, removed'])): + with self.subTest(old): + with open(path, 'w', encoding='utf-8', newline='\n') as f: + f.write(old) + script = os.path.join(d, 'merge.ps1') + with open(script, 'w', encoding='ascii') as f: + f.write(wslconfig_part() + '\n$want = [ordered]@{ memory = "9GB"; networkingMode = "nat" }\n' + '"notes=$(@(Changes (Read-WslConfig $args[0]) $want) -join \'|\')"\n' + '"changed=$(Merge-WslConfig $args[0] $want)"\n') + r = subprocess.run(['pwsh', '-NoProfile', '-NonInteractive', '-File', script, path], + capture_output=True, text=True, timeout=120, stdin=subprocess.DEVNULL) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn(f'changed={bool(notes)}', r.stdout) + for n in notes: + self.assertIn(n, r.stdout) + if not notes: + self.assertIn('notes=\n', r.stdout) + self.assertEqual(slurp(path), old) + for n in os.listdir(d): + if n.startswith('.wslconfig.bak-'): + os.unlink(os.path.join(d, n)) + + def test_the_owed_mark_goes_before_wsl_conf_changes(self): + """lens d8min r3 N1: the start of PID 1 is read from the VM's clock, which skips the PC's sleep, so a run + stopped after writing wsl.conf could look restarted. The owed mark now goes first when wsl.conf will change.""" + src = text().decode('ascii') + body = src[src.index("Head \"the distro's own settings\""):] + self.assertLess(body.index('if (InDistro \'reading /etc/wsl.conf\' ("CHECK_ONLY=1`n" + $Prepare)) { Mark $Owed }'), + body.index("$changed = InDistro \"the distro's settings\" $Prepare")) + def test_the_owed_shutdown_mark_is_shared_like_wslconfig(self): src = text().decode('ascii') self.assertIn("$Owed = Join-Path $Marks 'wslconfig.restart'", src) diff --git a/fork-kit/desk/windows/start-desk.ps1 b/fork-kit/desk/windows/start-desk.ps1 index a0921439a25fc..47bce128bf26a 100644 --- a/fork-kit/desk/windows/start-desk.ps1 +++ b/fork-kit/desk/windows/start-desk.ps1 @@ -21,8 +21,8 @@ $OutputEncoding = [Text.UTF8Encoding]::new($false) # what is piped into wsl.ex $Task = "fork-desk-$Distro" $WslConfig = Join-Path $env:USERPROFILE '.wslconfig' # Marks on the Windows side, for a run that stops: `DISTRO.made` (this script is installing the distro: a later run -# asks no takeover question), `wslconfig.restart` (.wslconfig changed and its `wsl --shutdown` is still owed: VM -# uptime skips sleep). .wslconfig is shared by every distro, so its mark is too: a run for another -Distro still owes it. +# asks no takeover question), `wslconfig.restart` (.wslconfig or the distro's wsl.conf changed and its `wsl --shutdown` +# is still owed: VM uptime and clock skip sleep). .wslconfig is shared by every distro, so its mark is too: a run for another -Distro still owes it. $Marks = Join-Path $env:LOCALAPPDATA 'fork-desk' $Made = Join-Path $Marks "$Distro.made"; $Owed = Join-Path $Marks 'wslconfig.restart' function Mark([string]$path) { New-Item -ItemType Directory -Force -Path $Marks | Out-Null; Set-Content -LiteralPath $path 'start-desk.ps1' } @@ -61,6 +61,10 @@ enabled=false appendWindowsPath=false [network] generateResolvConf=false' +if [ -n "${CHECK_ONLY:-}" ]; then # only say whether wsl.conf will change: the owed mark is written before it is + [ "$(cat "$ROOT/etc/wsl.conf" 2>/dev/null)" = "$want" ] || echo 'differs: /etc/wsl.conf' + exit 0 +fi mkdir -p "$ROOT/etc/binfmt.d" if [ "$(cat "$ROOT/etc/wsl.conf" 2>/dev/null)" != "$want" ]; then [ ! -e "$ROOT/etc/wsl.conf" ] || cp -p "$ROOT/etc/wsl.conf" "$ROOT/etc/wsl.conf.bak-$(date +%Y%m%d-%H%M%S)" @@ -125,30 +129,35 @@ function SectionKeys([string[]]$lines, [string]$section) { # the keys of one s $keys } function Wsl2Keys([string[]]$lines) { SectionKeys $lines 'wsl2' } -function Changes([string[]]$lines, $want) { # what Merge-WslConfig would change, one line each: old -> new - $now = Wsl2Keys $lines; $old = SectionKeys $lines 'experimental' - foreach ($k in $want.Keys) { - if ($now[$k] -ne $want[$k]) { " [wsl2] ${k}: $(if ($null -ne $now[$k]) { $now[$k] } else { '(not set)' }) -> $($want[$k])" } - } - foreach ($k in $Experimental) { if ($old.ContainsKey($k)) { " [experimental] $k=$($old[$k]): removed (the [wsl2] one is used)" } } -} -function Merge-WslConfig([string]$path, $want, [scriptblock]$before = {}) { # True when it changed the file (a dated copy of the old one kept; $before runs first) - $lines = Read-WslConfig $path - $out = [Collections.Generic.List[string]]::new(); $sec = ''; $done = @{} - function AddMissing { foreach ($k in $want.Keys) { if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 } } } +# The merged lines and one note per change (old -> new). The question shows these notes and the file is written only +# when there is one, so "no change" always means the file is left alone. A [wsl2] line whose value already matches +# (trimmed, any case: as the final check compares) is kept as it is; a repeated key's later lines are removed. +function MergeLines([string[]]$lines, $want) { + $out = [Collections.Generic.List[string]]::new(); $notes = [Collections.Generic.List[string]]::new(); $sec = ''; $done = @{} + function AddMissing { foreach ($k in $want.Keys) { if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1; $notes.Add(" [wsl2] ${k}: (not set) -> $($want[$k])") } } } foreach ($l in $lines) { if ($l -match '^\s*\[(.+)\]\s*$') { if ($sec -eq 'wsl2') { AddMissing }; $sec = $Matches[1].Trim().ToLower(); $out.Add($l); continue } - if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=' -and $want.Contains($Matches[1])) { - $k = $Matches[1]; if (-not $done[$k]) { $out.Add("$k=$($want[$k])"); $done[$k] = 1 }; continue } - if ($sec -eq 'experimental' -and $l -match '^\s*([^=#;\s]+)\s*=' -and $Matches[1] -in $Experimental) { continue } + if ($sec -eq 'wsl2' -and $l -match '^\s*([^=#;\s]+)\s*=\s*(.*?)\s*$' -and $want.Contains($Matches[1])) { + $k = $Matches[1]; $v = $Matches[2] + if ($done[$k]) { $notes.Add(" [wsl2] $k=${v}: a repeated line, removed"); continue } + $done[$k] = 1 + if ($v -eq $want[$k]) { $out.Add($l) } else { $out.Add("$k=$($want[$k])"); $notes.Add(" [wsl2] ${k}: $v -> $($want[$k])") } + continue } + if ($sec -eq 'experimental' -and $l -match '^\s*([^=#;\s]+)\s*=\s*(.*?)\s*$' -and $Matches[1] -in $Experimental) { + $notes.Add(" [experimental] $($Matches[1])=$($Matches[2]): removed (the [wsl2] one is used)"); continue } $out.Add($l) } if ($sec -eq 'wsl2') { AddMissing } if ($done.Count -lt $want.Count) { $out.Add('[wsl2]'); AddMissing } - if (($out -join "`n") -eq ($lines -join "`n")) { return $false } + [pscustomobject]@{ Out = $out; Notes = $notes } +} +function Changes([string[]]$lines, $want) { (MergeLines $lines $want).Notes } # what Merge-WslConfig changes +function Merge-WslConfig([string]$path, $want, [scriptblock]$before = {}) { # True when it changed the file (a dated copy of the old one kept; $before runs first) + $merged = MergeLines (Read-WslConfig $path) $want + if ($merged.Notes.Count -eq 0) { return $false } # the same test as the question's & $before if (Test-Path -LiteralPath $path) { Copy-Item -LiteralPath $path "$path.bak-$(Get-Date -Format yyyyMMdd-HHmmss)" } - [IO.File]::WriteAllLines("$path.new", [string[]]$out, [Text.UTF8Encoding]::new($false)) + [IO.File]::WriteAllLines("$path.new", [string[]]$merged.Out, [Text.UTF8Encoding]::new($false)) Move-Item -LiteralPath "$path.new" $path -Force $true } @@ -234,6 +243,9 @@ try { if ($restart) { 'changed: .wslconfig' } Head "the distro's own settings" + # The owed mark before wsl.conf changes too: a run stopped before its shutdown restarts next time even when the + # start of PID 1 (read from the VM's clock, which skips sleep) looks later than the change. + if (InDistro 'reading /etc/wsl.conf' ("CHECK_ONLY=1`n" + $Prepare)) { Mark $Owed } $changed = InDistro "the distro's settings" $Prepare $changed Remove-Item -LiteralPath $Made -ErrorAction SilentlyContinue # the distro's own mark is there now @@ -243,7 +255,7 @@ try { if ((Test-Path $WslConfig) -and ((Get-Date) - (Get-Item -LiteralPath $WslConfig).LastWriteTime).TotalSeconds -lt $up) { $live += 'restart: .wslconfig is newer than the running WSL' } - if (Test-Path -LiteralPath $Owed) { $live += 'restart: .wslconfig changed and its wsl --shutdown is still owed' } + if (Test-Path -LiteralPath $Owed) { $live += 'restart: a change (.wslconfig or wsl.conf) still owes its wsl --shutdown' } $live if ($restart -or $changed -or $live) { Head 'wsl --shutdown (the new settings apply at the next start)' From 297a2e2db687674affa2c9e87bde9b66cbafea55 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 01:58:25 +0000 Subject: [PATCH 29/33] free/v1: Wine names its process after the .exe, not wine (D10 fix round 4) Co-Authored-By: Claude Opus 5.5 --- fork-kit/common/FREE.md | 7 ++++--- fork-kit/common/free/busy-while-running | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/fork-kit/common/FREE.md b/fork-kit/common/FREE.md index 6339a4d24e6f6..ef502b34b8d79 100644 --- a/fork-kit/common/FREE.md +++ b/fork-kit/common/FREE.md @@ -69,9 +69,10 @@ size as busy. An answer without `memory.freeMb` makes no memory claim; the consu - [busy-while-running](free/busy-while-running): busy while a named program runs (a game, say). Copy it and put your programs in it. It sees the processes of its own system only: in WSL, Windows programs are not visible. It matches the process name (`comm`), not the command line: a program started through an interpreter or a - wrapper (`python3 game.py`, `wine game.exe`, a shell script that runs the game) shows as `python3`, `wine` or - the program the wrapper starts, so name that. Naming an interpreter makes the machine busy whenever any script - of that interpreter runs. + wrapper (`python3 game.py`, a shell script that runs the game) shows as `python3` or the program the wrapper + starts, so name that. Wine usually names its process after the `.exe` (`game.exe`, cut to 15 characters, as + `ps -e -o comm=` shows it), not `wine`. Naming an interpreter makes the machine busy whenever any script of + that interpreter runs. Several conditions in one script: call the others from it and answer busy when one is busy. diff --git a/fork-kit/common/free/busy-while-running b/fork-kit/common/free/busy-while-running index a71cd5fefdc38..bc612e9c054e3 100755 --- a/fork-kit/common/free/busy-while-running +++ b/fork-kit/common/free/busy-while-running @@ -2,7 +2,8 @@ # A free/v1 script (FREE.md): busy while one of the programs in NAMES runs, say a game or a video editor, so the # machine is left to its person. Copy it, put your programs' process names (as `ps -e -o comm=` shows them) in # NAMES, make the copy root's and not writable by others, and name the copy as the free command. -# It matches the process name, not the command line: `python3 game.py` or `wine game.exe` shows as python3 or wine. +# It matches the process name, not the command line: `python3 game.py` shows as python3. Wine usually names its +# process after the .exe (`game.exe`, cut to 15 characters), not wine. # It sees the processes of this system only: in WSL, Windows programs are not visible (interop is off, on purpose). NAMES='example-game example-editor' From 70822b5e53236c1919adc51976bebec45e0c731d Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 02:00:46 +0000 Subject: [PATCH 30/33] fork fleet desk seed: test the one-piece stall guard (D11 fix round 4) Co-Authored-By: Claude Opus 5.5 --- .github/scripts/fork-fleet-desk-seed.test.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 2fc94fe8ccefb..6ef9c3912258e 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -220,6 +220,22 @@ def test_a_desktop_that_stops_reading_lets_the_lock_go(self): self.make('m2', {'a': b'x'}) # small enough for the pipe: offer() below has no reader until it returns self.assertEqual(self.offer('seed m2')[0], 0) # the pool's lock was let go + def test_a_desktop_that_stops_reading_a_one_piece_seed_is_exit_1(self): + """lens236r3: the guard's `wrote == sent` half was untested. A seed of one piece (under PIECE) has + sent == size at the stall, so only that half refuses the cut stream.""" + self.make('m2', {'a': os.urandom(512 << 10)}) + r, w = os.pipe() # nobody reads r + err = io.StringIO() + try: + with mock.patch.object(seed, 'STALL', 1): + code = seed.offer(POOL, 'seed m2', os.path.join(self.t, 'run'), out=w, err=err, + config=os.path.join(self.t, 'fork-fleet.json'), files=os.path.join(self.t, 'files')) + finally: + os.close(r) + os.close(w) + self.assertEqual(code, 1, err.getvalue()) + self.assertNotIn('seed m2 ', err.getvalue()) # no trailer for a cut stream + def test_a_slow_desktop_meets_the_deadline(self): self.make('m2', {'a': os.urandom(1 << 20)}) r, w = os.pipe() # nobody reads r; the stall wait alone (100 s) would hold the lock much longer From c68366bb302ea096e3338c5d34ba82789e434dc8 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 02:09:24 +0000 Subject: [PATCH 31/33] fleet seeds: links never load the stream, sizes never lower the count, torn record kept - A link member whose path is already there is refused, and a Tar subclass turns tarfile's fallback (_find_link_target, which reads the rest of the stream into memory) into Torn: a link that cannot be made never copies its target. - A member with a size below 0, or a non-file with a size, is refused: the disk count can no longer be lowered. - A run without the host's list leaves .torn.json as it is (exit 0). - A torn record that cannot be written (full disk) never stops the fetch: the try counts in memory and the run exits 1. - A torn seed is tried at once when the host offers another copy. - Docs: retry times are approximate; a replaced copy lives about 25 h; budget. - Test that tarfile's private parts the limits rely on are there. Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 33 +++-- .github/scripts/fork-fleet-desk-seed.py | 84 +++++++---- .github/scripts/fork-fleet-desk-seed.test.py | 146 ++++++++++++++++++- fork-kit/desk/README.md | 8 +- 4 files changed, 228 insertions(+), 43 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index 05f6a1f363988..8e2d5e2c0477b 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -736,12 +736,15 @@ its own `every_hours` and `max_age_hours` and the smaller of its own and the host's `max_mb`: a host cannot make a desktop take more. Each new seed is unpacked with Python's tar data filter and a limit on what it would make on disk (each member's size plus 4 KiB for it and for each folder it makes, all -within `max_mb`; at most 200,000 members and folders; no member more than 64 +within `max_mb`; a size below 0, or a size on a member that is not a file, is +refused; at most 200,000 members and folders; no member more than 64 folders deep or with a name over 1 KiB) and on memory (members are let go as they are unpacked; the extended pax or long-name headers before one member, and the global ones of the whole tar, at most 1 MiB; sparse members are refused, so make a seed without `--sparse`; a hard link must name a file unpacked before -it), checked against the trailer (its size +it, and a link whose path is already there, or that cannot be made, is refused, +never copied from its target, which tarfile would look for in the rest of the +stream), checked against the trailer (its size and date must be the list's too), checked again for links that lead out of it, made readable by every job (a cache's 0600 files too), and swapped in by a rename; ssh's stderr and the list are read with a limit. A seed that fails in @@ -749,13 +752,16 @@ any way is named in the log and the run goes on with the next; old copies are pruned at the start and the end of every run (so a run killed by its unit's memory or time limit cannot pile them up). A run fetches for at most two hours in all (the unit allows three), each seed at most 30 minutes. A seed torn again -and again is tried again after 1, 2, 4, 8, 16 and then every 24 hours (a link -too slow for it does not spend every hour on it; a fetch that kills the run -counts as torn), and one torn on every try for more than a day makes the run -exit 1, so `desk-install check` shows FAIL until it is fetched -(`/var/lib/fork-fleet/seeds/.torn.json` names it and why). A host that gives no list (down, an older kit, a -half-done install) means no new seeds: each one here stays until its age -drops it. A desktop whose own fleet config is broken or lacks its pool fetches +and again is tried again after about 1, 2, 4, 8, 16 and then every 24 hours (a +link too slow for it does not spend every hour on it; a fetch that kills the +run counts as torn; the times are approximate: the timer's 10 minute random +delay and a 5 minute margin can make a wait up to an hour longer), and at once +when the host offers another copy of it. One torn on every try for more than a +day makes the run exit 1, so `desk-install check` shows FAIL until it is +fetched (`/var/lib/fork-fleet/seeds/.torn.json` names it and why); so does a +run that cannot write that file (a full disk). A host that gives no list (down, +an older kit, a half-done install) means no new seeds: each one here stays until +its age drops it, and the torn record is left as it is. A desktop whose own fleet config is broken or lacks its pool fetches nothing and exits 2 (`desk-install check` shows the seed service's last result), but still drops each seed past the `max_age_hours` it was fetched with. Jobs see @@ -763,11 +769,12 @@ with. Jobs see the unpacking and old copies beside it, `.new` and `.v`, are visible too and hold the same data), copy what they need from it when it is there, and run cold when it is not. A replaced copy is kept 24 hours for a job still reading -it; there is no free-space check before unpacking (a full disk ends that seed's +it and removed by the next hourly prune, so it lives up to about 25 hours; +there is no free-space check before unpacking (a full disk ends that seed's fetch and keeps the old one). **Disk budget per seed:** up to `max_mb` x -(24 / `every_hours` + 2): the copy in use, the copies replaced in the last 24 -hours, and one being unpacked (`every_hours` 1 and `max_mb` 4096: up to about -104 GiB; `every_hours` 6: about 24 GiB). Size `max_mb` and `every_hours` to the +(25 / `every_hours`, rounded up, + 2): the copy in use, the copies replaced in +the last 25 hours, and one being unpacked (`every_hours` 1 and `max_mb` 4096: +up to about 108 GiB; `every_hours` 6: about 28 GiB). Size `max_mb` and `every_hours` to the desktop's free disk. The state table (a torn download, the host down, clocks that disagree, a run that stopped half-way) is in the script's header. diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index 432426950943b..f4aa3f3bf9478 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -31,16 +31,21 @@ the same seed on the host (bytes and MADE) nothing is fetched a new one under every_hours after the last nothing yet a torn copy (the stream cut or past FETCH_WAIT, a trailer that does not match it or the list, more than max_mb, - a tar member that would leave its folder, is a device, too deep or too long, or any other error) + a tar member that would leave its folder, is a device, too deep or too long, has a size below 0 (or one that is + not a file has a size), a link whose path is already there or that cannot be made, or any other error) the copy is removed, the old seed stays, the other seeds go on, old - versions are still pruned; tried again after 1, 2, 4, 8, 16, then 24 - hours (TornLog); torn on every try for over TORN_DAY: the run exits 1 + versions are still pruned; tried again after about 1, 2, 4, 8, 16, then + 24 hours (TornLog; the hourly timer's RandomizedDelaySec and a 5 min + margin make each wait up to an hour longer), at once when the host + offers another copy; torn on every try for over TORN_DAY: exit 1 + .torn.json cannot be written (a full disk) the run goes on, counts its tries in memory, and exits 1 a fetch that kills the run (MemoryMax, TimeoutStartSec) counted as torn before it began; the next run prunes first the run's RUN_WAIT used the seeds left are tried next hour no seed (never fetched, the host down) jobs run cold no list (the host down, an older kit that answers with a session or not JSON, exit 1 of a half-done install) - nothing fetched; each seed stays until its age drops it + nothing fetched, exit 0; each seed stays until its age drops it; the + torn record (.torn.json) is left as it is its own fleet config broken or without POOL nothing fetched, exit 2; a seed older than its label's maxAgeHours (its own max_age_hours when it was fetched) is still dropped older than max_age_hours, made over SKEW in the future (the clocks disagree), or no longer in the pool's list @@ -50,6 +55,7 @@ any depth); a link is the old one or the new a replaced version kept KEEP_OLD after it was replaced, for a job still reading it (a job runs at most the launcher's MAX_SECONDS, 1 h by default), then removed + by the next hourly prune: it lives up to about 25 h The forced command sends every command that starts with `seed` here. The host side exits 2 for any other command, 3 while another stream for the pool runs, 4 when the file may not be sent (missing, a link, in a folder or a file @@ -358,15 +364,25 @@ def sparse_refused(self, member=None, *_): _proc_gnusparse_00 = _proc_gnusparse_01 = _proc_gnusparse_10 = sparse_refused # pax sparse forms +class Tar(tarfile.TarFile): + """A link that cannot be made (its path is there, or os.link fails) is never copied from its target: tarfile + would read the whole rest of the stream into memory to find it (_find_link_target), past every limit here.""" + + def _find_link_target(self, tarinfo): + raise Torn(f'{tarinfo.name[:80]!r} is a link that cannot be made here (its target is not copied instead)') + + def unpack(t, data, most): - """The members of the open stream T into DATA, one at a time: tarfile keeps every member it has read in - `members` (and extractall each folder besides), so they are let go as they are made; only `guarded`'s count of - folders stays. A hard link must name a file already unpacked (otherwise tarfile would read the rest of the - stream into memory to find it).""" + """The members of the open stream T (a Tar) into DATA, one at a time: tarfile keeps every member it has read + in `members` (and extractall each folder besides), so they are let go as they are made; only `guarded`'s count + of folders stays. A link's path must not be there yet (tarfile would unlink a file, fail on a folder) and a hard + link must name a file already unpacked (otherwise tarfile would look for it in the rest of the stream).""" check = guarded(most) def linked(member, path): got = check(member, path) + if got is not None and (got.islnk() or got.issym()) and os.path.lexists(os.path.join(path, got.name)): + raise Torn(f'{got.name[:80]!r} is a link to {got.linkname[:80]!r} where a member is already there') if got is not None and got.islnk(): try: plain = stat.S_ISREG(os.lstat(os.path.join(path, got.linkname)).st_mode) @@ -390,6 +406,8 @@ def guarded(most): seen, folders = [0, 0, 0], {} # folders: {(its parent's number, its name): its number}, a name kept once def check(member, path): + if member.size < 0 or (member.size and not member.isreg()): # a size below 0 would lower the count + raise Torn(f'{member.name[:80]!r} has the size {member.size} (a member that is not a file has 0)') parts = [x for x in member.name.split('/') if x not in ('', '.')] if len(parts) > DEPTH: raise Torn(f'{"/".join(parts[:3])}/... is more than {DEPTH} folders deep') @@ -450,7 +468,7 @@ def fetch(agent, conf, item, root, now, wait=FETCH_WAIT): err = Tail(p.stderr) try: got = Summed(p.stdout, most) - with tarfile.open(fileobj=got, mode='r|', tarinfo=Info) as t: + with Tar.open(fileobj=got, mode='r|', tarinfo=Info) as t: unpack(t, os.path.join(new, 'data'), most) while got.read(PIECE): # the tar's end blocks and padding count in the trailer's size and sha256 pass @@ -596,17 +614,21 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): items = {n: {**host[n], 'everyHours': s['every_hours'], 'maxAgeHours': s['max_age_hours'], 'maxMb': min(host[n]['maxMb'], s['max_mb'])} for n, s in own.items() if n in host} except Torn as e: # the host down, or a host on an older kit (no list, or not one this reads) - items = {} say(f'seeds: {e}: each seed stays until its age drops it') + return 0 # the torn record stays as it is: no list says nothing about a seed torn, end = TornLog(root, now), time.monotonic() + RUN_WAIT torn.keep(items) for name, item in sorted(items.items()): if not each(say, f'seed {name}', take, agent, conf, item, root, now, say, torn, end): clean(root, say) - each(say, 'seeds: .torn.json', torn.save) + torn.save() + if torn.unsaved: + say(f'seeds: .torn.json cannot be written ({torn.unsaved}; a full disk?): exit 1 (desk-install ' + f'check shows it; the tries of this run are not counted for the next)') if late := torn.late(): say(f'seeds: {", ".join(late)} torn on every try for more than {TORN_DAY // 3600} h: exit 1 ' f'(desk-install check shows it)') + if torn.unsaved or late: return 1 finally: prune(root, now, say) @@ -614,15 +636,18 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): class TornLog: - """SEEDS/.torn.json, {name: {"first", "last", "tries", "why"}}: the seeds whose last fetch was torn, written - before each fetch (so a fetch that kills the run, by its memory or time limit, counts as torn too) and cleared - when the seed is fetched or the same as the host's. A seed torn TRIES times in a row is tried again only + """SEEDS/.torn.json, {name: {"first", "last", "tries", "why", "copy"}}: the seeds whose last fetch was torn, + written before each fetch (so a fetch that kills the run, by its memory or time limit, counts as torn too) and + cleared when the seed is fetched or the same as the host's. A seed torn TRIES times in a row is tried again only min(2 ** (TRIES - 1), 24) hours after the last try (a link too slow for it, a host that cuts it: not every - hour); one torn on every try since more than TORN_DAY is `late`. Only seeds in the host's list are kept. A - missing or unreadable file is an empty record.""" + hour), or at once when the host offers another copy (`copy`: its [bytes, made] when last tried); one torn on + every try since more than TORN_DAY is `late`. Only seeds in the host's list are kept, and only in a run that + read the list. A missing or unreadable file is an empty record; one that cannot be written (a full disk) is + kept in memory for this run and said in `unsaved`.""" def __init__(self, root, now): self.path, self.now, self.seeds = root and os.path.join(root, '.torn.json'), now, {} + self.unsaved = None try: with open(self.path, encoding='utf-8') as f: got = json.load(f) @@ -634,20 +659,23 @@ def __init__(self, root, now): def keep(self, items): self.seeds = {n: r for n, r in self.seeds.items() if n in items} - def due(self, name): - """None when the seed may be tried now, else the hours until it may.""" + def due(self, name, item=None): + """None when the seed may be tried now (ITEM, the host's offer, another copy than the one last tried + included), else the hours until it may.""" r = self.seeds.get(name) - if not r: + if not r or item and r.get('copy') not in (None, [item['bytes'], item['made']]): return None wait = min(2 ** (min(r['tries'], 6) - 1), 24) * 3600 - 300 # 5 min early: the hourly timer drifts left = r['last'] + wait - self.now() return math.ceil(left / 3600) if 0 < left <= wait else None # a date ahead of the clock: try now - def trying(self, name): + def trying(self, name, item=None): """Counted as torn before the fetch, saved: `why` says so until the fetch ends.""" r = self.seeds.get(name) or {'first': int(self.now()), 'tries': 0} self.seeds[name] = {**r, 'last': int(self.now()), 'tries': r['tries'] + 1, 'why': 'the run stopped during the fetch (its memory or time limit?)'} + if item: + self.seeds[name]['copy'] = [item['bytes'], item['made']] self.save() def torn(self, name, why): @@ -662,12 +690,18 @@ def late(self): return sorted(n for n, r in self.seeds.items() if self.now() - r['first'] > TORN_DAY) def save(self): + """Written by tmp + rename; an OSError (a full disk) is kept in `unsaved` (cleared by a save that works), + never raised: the run goes on, counts in memory and exits 1.""" if self.path is None: return tmp = self.path + '.tmp' - with open(tmp, 'w', encoding='utf-8') as f: - json.dump(self.seeds, f) - os.replace(tmp, self.path) + try: + with open(tmp, 'w', encoding='utf-8') as f: + json.dump(self.seeds, f) + os.replace(tmp, self.path) + self.unsaved = None + except OSError as e: + self.unsaved = f'{e.__class__.__name__}: {e.strerror or e}' def each(say, what, step, *args): @@ -722,14 +756,14 @@ def take(agent, conf, item, root, now, say, torn=None, end=None): if not -SKEW <= now() - item['made'] <= item['maxAgeHours'] * 3600: say(f'seed {name}: the host\'s is too old or made in the future (the clocks disagree): not fetched') return - if (hours := torn.due(name)) is not None: + if (hours := torn.due(name, item)) is not None: say(f'seed {name}: torn {torn.seeds[name]["tries"]} times in a row; tried again in about {hours} h') return left = end - time.monotonic() if left < 60: say(f'seed {name}: this run has used its {RUN_WAIT // 60} minutes; tried again next hour') return - torn.trying(name) + torn.trying(name, item) try: got = fetch(agent, conf, item, root, now, wait=min(FETCH_WAIT, left)) except Exception as e: # any: a torn stream, a full disk, a tar this Python cannot unpack diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index f5607843f7e3a..53982322e9844 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -795,8 +795,6 @@ def test_extended_headers_and_sparse_members_are_held_to_limits(self): self.assertTrue(any('m2' in x and why in x for x in self.said), self.said) with contextlib.suppress(FileNotFoundError): # each case tried at once, not after a wait os.unlink(os.path.join(self.root, '.torn.json')) - for k in ('_proc_member', '_proc_sparse', '_proc_gnusparse_00', '_proc_gnusparse_01', '_proc_gnusparse_10'): - self.assertTrue(callable(getattr(tarfile.TarInfo, k, None)), f'tarfile.TarInfo.{k} is gone: see Info') def test_members_are_let_go_as_they_are_unpacked(self): self.make('m2', {f'd/{i}': b'x' for i in range(300)}) @@ -825,6 +823,150 @@ def hard(name, target): self.assertEqual(self.files('m2'), {'a': b'one', 'b': b'one'}) # the old one stays self.assertTrue(any('hard link' in x for x in self.said), self.said) + def test_the_private_tarfile_parts_the_limits_rely_on_are_there(self): # lens237sr3: fail loud, not silently + for k in ('_proc_member', '_proc_sparse', '_proc_gnusparse_00', '_proc_gnusparse_01', '_proc_gnusparse_10'): + self.assertTrue(callable(getattr(tarfile.TarInfo, k, None)), f'tarfile.TarInfo.{k} is gone: see Info') + self.assertTrue(callable(getattr(tarfile.TarFile, '_find_link_target', None)), + 'tarfile.TarFile._find_link_target is gone: see Tar') + self.assertIs(seed.Tar._find_link_target, seed.Tar.__dict__['_find_link_target']) + + def forked(self): + """run_seed() in a child: 0 when it stayed under the unit's MemoryMax=512M, 1 when not (its peak RSS, or a + MemoryError: the child may map 1 GB at most, so the test never takes the machine's memory).""" + r, w = os.pipe() + pid = os.fork() + if pid == 0: + import resource + code = 2 + try: + os.close(r) + resource.setrlimit(resource.RLIMIT_AS, (1 << 30, 1 << 30)) + self.run_seed() + os.write(w, json.dumps(self.said).encode()) + code = int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss >= 512 << 10 + or any('MemoryError' in x for x in self.said)) + finally: + os._exit(code) + os.close(w) + with os.fdopen(r) as f: + self.said = json.loads(f.read() or '[]') + return os.waitpid(pid, 0)[1] >> 8 + + def test_a_link_that_cannot_be_made_never_loads_the_stream(self): # lens237sr3: the MUST + big = [{**SEEDS[0], 'max_mb': 128}, SEEDS[1]] + self.config(big) + self.own(big) + x = tarfile.TarInfo('pax') + x.type = tarfile.XHDTYPE + body = b''.join(b'15 a.%07d=x\n' % i for i in range(65_000)) # ~0.98 MB, under HEADER_MOST + rest = b''.join(self.block(x, body) + self.block(tarfile.TarInfo(f'f{i}')) for i in range(110)) # ~108 MB + del body + with open(os.path.join(self.t, 'rest'), 'wb') as f: # kept on disk: the child must not inherit it + f.write(rest) + del rest + + def rest(): + with open(os.path.join(self.t, 'rest'), 'rb') as f: + return f.read() + + def link(name, target, kind): + info = tarfile.TarInfo(name) + info.type, info.linkname = kind, target + return self.block(info) + cases = { # a link whose path is there: os.link fails; os.unlink of a folder fails + 'a hard link over a file': self.block(tarfile.TarInfo('a'), b'1') + self.block(tarfile.TarInfo('b'), b'2') + + link('b', 'a', tarfile.LNKTYPE), + 'a symlink over a folder': self.block(tarfile.TarInfo('d/x'), b'1') + link('d', 'x', tarfile.SYMTYPE), + } + for i, (what, head) in enumerate(cases.items()): + with self.subTest(what): + self.raw('m2', head + rest()) + self.assertEqual(self.forked(), 0, self.said[-3:]) # hard link: over 512 MB before round 4 + self.assertIsNone(self.files('m2')) + self.assertTrue(any('m2' in x and 'already there' in x for x in self.said), self.said) + with contextlib.suppress(FileNotFoundError): + os.unlink(os.path.join(self.root, '.torn.json')) + self.raw('m2', self.block(tarfile.TarInfo('a'), b'1') + link('b', 'a', tarfile.LNKTYPE)) + with mock.patch.object(os, 'link', side_effect=OSError(18, 'Invalid cross-device link')): # any other + self.run_seed() # link failure + self.assertIsNone(self.files('m2')) + self.assertTrue(any('m2' in x and 'cannot be made' in x for x in self.said), self.said) + + def test_a_negative_size_never_lowers_the_disk_count(self): # lens237sr3: the MUST + d = bytearray(tarfile.TarInfo('z').tobuf(format=tarfile.GNU_FORMAT)) + d[156:157] = tarfile.DIRTYPE + d[124:136] = b'\xff' + b'\0' * 11 # a base-256 size below zero + d[148:156] = b' ' * 8 + d[148:156] = b'%06o\0 ' % sum(d[:512]) + self.raw('m2', bytes(d) + b''.join(self.block(tarfile.TarInfo(f'e{i}')) for i in range(1500))) + self.run_seed() + self.assertIsNone(self.files('m2')) # before round 4: 1,500 files (6 MB of entries) under max_mb 2 + self.assertTrue(any('m2' in x and 'size' in x for x in self.said), self.said) + f = bytearray(tarfile.TarInfo('y').tobuf(format=tarfile.GNU_FORMAT)) # a folder with a size: refused too + f[156:157] = tarfile.DIRTYPE + f[124:136] = b'%011o\0' % 512 + f[148:156] = b' ' * 8 + f[148:156] = b'%06o\0 ' % sum(f[:512]) + self.raw('m2', bytes(f) + b'\0' * 512 + self.block(tarfile.TarInfo('e'), b'x')) + self.run_seed(now=lambda: time.time() + 2 * 3600) + self.assertIsNone(self.files('m2')) + self.assertTrue(any('m2' in x and 'size' in x for x in self.said), self.said) + + def test_one_lost_list_does_not_reset_the_torn_record(self): # lens237cr3 + self.make('m2', {'a': b'one'}) + start, codes, real, tried = int(time.time()), {}, seed.listing, [] + + def torn(agent_, conf, item, root, now, **kw): + tried.append(round((now() - start) / 3600)) + raise seed.Torn('cut off') + + def lost(h): + def listing(a, c): + if h % 12 == 11: # the 30 s list lost twice a day + raise seed.Torn('no list in 30 s') + return real(a, c) + return listing + with mock.patch.object(seed, 'fetch', torn): + for h in range(49): + with mock.patch.object(seed, 'listing', lost(h)): + codes[h] = self.run_seed(now=lambda h=h: start + h * 3600) + self.assertEqual(tried, [0, 1, 3, 7, 15, 31]) # the back-off goes on + self.assertEqual({h for h, c in codes.items() if c}, set(range(25, 49)) - {35, 47}) # late, list or not + + def test_a_torn_record_that_cannot_be_saved_still_counts_and_fails_the_run(self): # lens237cr3: disk full + self.make('m2', {'a': b'one'}) + os.makedirs(os.path.join(self.root, '.torn.json.tmp')) # the record cannot be written, as on a full disk + tried = [] + + def torn(agent_, conf, item, root, now, **kw): + tried.append(item['name']) + raise seed.Torn('No space left on device') + with mock.patch.object(seed, 'fetch', torn): + self.assertEqual(self.run_seed(), 1) + self.assertEqual(tried, ['m2']) # tried, and the try counted in memory + self.assertTrue(any('.torn.json' in x and 'cannot be written' in x for x in self.said), self.said) + os.rmdir(os.path.join(self.root, '.torn.json.tmp')) + with mock.patch.object(seed, 'fetch', torn): + self.assertEqual(self.run_seed(now=lambda: time.time() + 3600), 0) + with open(os.path.join(self.root, '.torn.json')) as f: + self.assertEqual(json.load(f)['m2']['tries'], 1) # the lost record starts again, no worse + + def test_a_new_copy_on_the_host_is_tried_at_once_after_torn_tries(self): # lens237cr3 nit + start, tried = int(time.time()), [] + self.make('m2', {'a': b'one'}, mtime=start - 3600) + + def torn(agent_, conf, item, root, now, **kw): + tried.append(round((now() - start) / 3600)) + raise seed.Torn('cut off') + with mock.patch.object(seed, 'fetch', torn): + for h in (0, 1, 2): + self.run_seed(now=lambda h=h: start + h * 3600) + self.make('m2', {'a': b'two!'}, mtime=start - 60) # the host made a fixed one + self.run_seed(now=lambda: start + 2 * 3600 + 120) + self.run_seed(now=lambda: start + 2 * 3600 + 180) + self.assertEqual(tried, [0, 1, 2]) # at once, then the back-off again (now 4 h) + self.assertTrue(any('torn 3 times' in x for x in self.said), self.said) + def test_a_deeply_nested_list_means_no_list(self): # lens237cr2 self.make('m2', {'a': b'one'}) self.run_seed() diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 9e0b34c37a557..67b78733d8356 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -69,9 +69,11 @@ hand setup) is refused the same way, not followed or fixed. One install or unins The seed timer (`t3code-fleet-seed.timer`) fetches the pool's warm caches, if its fleet config lists any, into `/var/lib/fork-fleet/seeds`, which jobs see read-only ([fork-fleet.md](../../.github/fork-fleet.md), "Seeds"). -A torn seed is tried again after 1, 2, 4, 8, 16, then 24 hours; one torn on every try for more than a day makes each -run exit 1, so `check` shows the seed service's last run as failed. `/var/lib/fork-fleet/seeds/.torn.json` names it -and the host's reason (jobs can read that file too). +A torn seed is tried again after about 1, 2, 4, 8, 16, then 24 hours (up to an hour later: the timer's random delay), +or at once when the host offers another copy; one torn on every try for more than a day makes each run that reads the +host's list exit 1, so `check` shows the seed service's last run as failed (a run that cannot write the record, a full +disk, exits 1 too). `/var/lib/fork-fleet/seeds/.torn.json` names it and the host's reason (jobs can read that file +too); a run without the host's list leaves it as it is. ## Windows (WSL2) From d4184a9b2d08af682f3ea88a6f33fbbeb146fba7 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 02:32:16 +0000 Subject: [PATCH 32/33] fork kit desk: fix round 5 (subfolders refused like the top folders, main group named after the user, public key in check, no FIFO opened) (D8) Co-Authored-By: Claude Opus 5.5 --- fork-kit/desk/README.md | 18 +++-- fork-kit/desk/sbin/desk-install | 35 ++++++--- fork-kit/desk/tests/install.test.py | 106 ++++++++++++++++++++++++++-- 3 files changed, 137 insertions(+), 22 deletions(-) diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index 703577dcd3c4c..8df2dd1c49e34 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -49,26 +49,32 @@ the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner temp the refresh timer; `check` says FAIL for `template` until it is there. `check` only reads: the installed files against their manifest (text, root owner and mode), the folders' and the -desk key's modes and root owner (not a link), the project's fleet config against the installed one, each unit against +desk key's (both halves) modes and root owner (not a link), the project's fleet config against the installed one, each unit against the release's (header plus `/opt/fork-fleet/units/UNIT`, mode and root owner), a restart a stopped install still owes, systemd drop-ins that change the kit's units, every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings files against the machine file (text, mode and root owner), the runner users (each must exist, have no login shell, -a locked password, a uid other than 0, its own group only, no home that is there (`/nonexistent`) and no line in +a locked password, a uid other than 0, its own group only (the main group is the group named after the user, and +no other), no home that is there (`/nonexistent`) and no line in `cron.allow`/`at.allow`; and it names a user the kit made that a smaller `slots` no longer uses) and IPv4 forwarding. A `fork-runner-N` user made by hand before the install is used, but only when it passes the same checks (no login shell, locked password, uid not 0, its own group only, no home that is there, not in an allow file; the launcher -refuses the same uid and group cases at each job start): otherwise install refuses before changing anything and +refuses uid 0 and more than one group at each job start, and install and `check` also refuse a main group that is +not the user's own, such as gid 0 from `useradd -g 0`): otherwise install refuses before changing anything and names the fix (`usermod -s /usr/sbin/nologin`, `passwd -l`, `gpasswd -d`, `usermod -d /nonexistent`). A -`/opt/fork-fleet`, `/opt/fork-fleet/template`, `/etc/fork-fleet` or desk key that is a link or not owned by root (a -hand setup) is refused the same way, not followed or fixed. One install or uninstall runs at a time (a lock on +`/opt/fork-fleet`, a folder in it (`template`, `units`, `fork-fleet-walls`, and any other the installed manifest +names), `/etc/fork-fleet` or desk key (either half) that is a link or not owned by root (a hand setup) is refused the +same way, not followed or fixed. The folders above them (`/opt`, `/etc`, `/etc/systemd/system`) are not checked: +they are the system's, root-owned on a stock system, and a user who could write them could already change what root +runs; keep them root's. One install or uninstall runs at a time (a lock on `/etc/fork-fleet`, which only root can open); a second one is refused. ## Runs again, stops half-way, old or empty state - **Again:** files, settings and units that did not change are not rewritten (a file in `/opt/fork-fleet`, a - setting or a unit whose mode or owner was changed, or that is a link, is written again as root's; the folders' + setting or a unit whose mode or owner was changed, or that is a link or a FIFO, is written again as root's (a FIFO is never opened; one at a unit + path is refused as not the kit's); the folders' modes are set back; a linked or foreign-owned folder or key is refused, above), and the agent is not stopped or restarted (a restart would end a job it is starting). After a change it is restarted. The agent reads its settings only when it starts, so before the first change the record gets a restart-pending mark, cleared after the diff --git a/fork-kit/desk/sbin/desk-install b/fork-kit/desk/sbin/desk-install index 8a24cd7cbe214..af56f043ead18 100755 --- a/fork-kit/desk/sbin/desk-install +++ b/fork-kit/desk/sbin/desk-install @@ -38,6 +38,7 @@ FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh', 'fork-fleet.json': '.github/fork-fleet.json', 'desk-install': 'fork-kit/desk/sbin/desk-install', 'machine_file.py': 'fork-kit/common/machine_file.py', 'README.md': 'fork-kit/desk/README.md'} WALLS = '.github/scripts/fork-fleet-walls' # copied whole: every OS's walls file and fixture +SUBDIRS = ('units', 'fork-fleet-walls') # the folders the manifest makes inside /opt/fork-fleet UNITS = {'fork-fleet-desk-agent.service': '.github/scripts/fork-fleet-desk-agent.service', 't3code-fleet-refresh.service': '.github/scripts/fork-fleet-refresh.service', 't3code-fleet-refresh.timer': '.github/scripts/fork-fleet-refresh.timer'} @@ -91,9 +92,10 @@ def user_exists(name): def user_problem(u): """Why runner user u (made by the kit or by hand) could be used for more than one-job runners: a login shell, a - usable password (or none), uid 0, a group besides its own, a home that is there, a line in cron.allow/at.allow; - None when it is fine. The launcher's check_users refuses the same users at each job start (uid, groups). Read - through getent and id (as root).""" + usable password (or none), uid 0, a main group that is not the group named after it (gid 0 from `useradd -g 0`), + another group besides it, a home that is there, a line in cron.allow/at.allow; None when it is fine. The + launcher's check_users refuses uid 0 and more than one group at each job start. Read through getent and id (as + root).""" r = run('getent', 'passwd', u, check=False) fields = r.stdout.strip().split(':') if r.returncode or len(fields) < 7: @@ -102,6 +104,10 @@ def user_problem(u): return f'{u} has the login shell {fields[6]!r} (usermod -s /usr/sbin/nologin {u})' if fields[2] == '0' or not fields[2].isdigit(): return f'{u} has uid {fields[2]}: a runner user is a user of its own, not root (userdel {u}; install makes it)' + own = run('getent', 'group', u, check=False).stdout.strip().split(':') + if len(own) < 3 or own[0] != u or own[2] != fields[3]: + return (f'{u} has the main group {fields[3]}, not its own group {u}: a runner user is in its own group only ' + f'(userdel {u}; install makes it with its own group)') gid, groups = run('id', '-g', u, check=False), run('id', '-G', u, check=False) if gid.returncode or groups.returncode or groups.stdout.split() != gid.stdout.split(): return f'{u} is in other groups ({groups.stdout.strip()}): a runner user is in its own group only (gpasswd -d)' @@ -137,12 +143,14 @@ def not_root_owned(path, kind): def same_file(real, text, mode): - """The file (not a link) holds text, has mode and is owned by root.""" + """The file (not a link) holds text, has mode and is owned by root. Not a plain file: not opened (a FIFO would + hang the run).""" try: st = os.lstat(real) + if not (stat.S_ISREG(st.st_mode) and st.st_uid == ROOT_UID and stat.S_IMODE(st.st_mode) == mode): + return False with open(real, encoding='utf-8') as f: - return (stat.S_ISREG(st.st_mode) and st.st_uid == ROOT_UID and stat.S_IMODE(st.st_mode) == mode - and f.read() == text) + return f.read() == text except (OSError, UnicodeDecodeError): return False @@ -262,6 +270,8 @@ def save_state(state): def ours(path): try: + if not stat.S_ISREG(os.stat(p(path)).st_mode): # a FIFO (or a link to one) is not opened: it would hang + return False with open(p(path), encoding='utf-8') as f: return f.readline() == HEADER except (OSError, UnicodeDecodeError): @@ -496,7 +506,9 @@ def install(a, stop_after=None, stack=None): raise Refused(f'the install record says a project\'s fleet config, but neither {PROJECT} nor ' f'{OPT}/fork-fleet.json is there: run again with --fleet-config FILE or ' '--release-fleet-config') - for path, kind in ((OPT, stat.S_ISDIR), (OPT + '/template', stat.S_ISDIR), (ETC, stat.S_ISDIR), + subdirs = sorted({*SUBDIRS, *(os.path.dirname(r) for r in read_manifest() if os.path.dirname(r))}) + for path, kind in ((OPT, stat.S_ISDIR), (OPT + '/template', stat.S_ISDIR), + *((f'{OPT}/{d}', stat.S_ISDIR) for d in subdirs), (ETC, stat.S_ISDIR), (KEY, stat.S_ISREG), (KEY + '.pub', stat.S_ISREG)): if not_root_owned(path, kind): # a hand setup: the root-run launcher must not sit where a user can write raise Refused(not_root_owned(path, kind)) @@ -546,6 +558,7 @@ def install(a, stop_after=None, stack=None): run('sysctl', '-q', '-p', p(SYSCTL)) make_key(m['pool'], pending) os.chmod(p(KEY), 0o600) # `check` wants it so; an older key made with another mode is set right + os.chmod(p(KEY + '.pub'), 0o644) done('settings') for u in UNITS: with open(p(f'{OPT}/units/{u}'), encoding='utf-8') as f: @@ -615,8 +628,8 @@ def check(): row(False, 'machine-file', str(e)[:300]) for u in users: why = f'{u} does not exist' if not user_exists(u) else user_problem(u) - row(not why, 'runner-user', why or f'{u} exists: no login shell, no password, own group only, no home, ' - 'not in cron.allow/at.allow') + row(not why, 'runner-user', why or f'{u} exists: no login shell, no password, uid not 0, only its own group ' + f'({u}), no home, not in cron.allow/at.allow') with contextlib.suppress(Refused): # a smaller `slots` leaves the kit's extra users: named, not lost for u in (read_state() or {}).get('createdUsers', []) if users else []: if u not in users and user_exists(u): @@ -652,7 +665,7 @@ def installed_problems(): problems.append(f'{OPT}/{rel}: differs from what was installed') elif not mode_ok: problems.append(f'{OPT}/{rel}: mode {stat.S_IMODE(st.st_mode):o}, not {file_mode(rel):o}: install again') - folders = {OPT: 0o755, OPT + '/template': 0o755, ETC: 0o700, KEY: 0o600} + folders = {OPT: 0o755, OPT + '/template': 0o755, ETC: 0o700, KEY: 0o600, KEY + '.pub': 0o644} folders.update({f'{OPT}/{os.path.dirname(rel)}': 0o755 for rel in manifest if os.path.dirname(rel)}) for path, mode in sorted(folders.items()): try: @@ -783,7 +796,7 @@ def uninstall(stack=None): remove(f'{ETC}/{name}') remove(STATE) # the record last left = [] - for d in ([f'{OPT}/{x}' for x in ('units', 'fork-fleet-walls')] + [OPT, WORK, ETC]): + for d in ([f'{OPT}/{x}' for x in SUBDIRS] + [OPT, WORK, ETC]): left += rmdir_left(d) left += [f'{d} (a drop-in, not the kit\'s)' for u in UNITS for d in drop_ins(u)] for x in left: diff --git a/fork-kit/desk/tests/install.test.py b/fork-kit/desk/tests/install.test.py index a5b813f994482..859467acd5595 100644 --- a/fork-kit/desk/tests/install.test.py +++ b/fork-kit/desk/tests/install.test.py @@ -40,17 +40,19 @@ def one(n, default): # a per-user setting: the file N-USER return open(os.path.join(here, n + '-' + args[-1])).read().strip() except FileNotFoundError: return default -if name == 'id': # its groups: the file groups-USER (its own group only) +if name == 'id': # its main group: gid-USER (1001, its own group's); its groups: groups-USER (its main group only) if args[-1] not in lines('users'): sys.exit(1) - print({'-u': one('uid', '1001'), '-g': '1001', '-G': one('groups', '1001')}.get(args[0], '')) -if name == 'getent': # login shell: shell-USER (nologin); password field: pw-USER (!); uid-USER; home-USER + print({'-u': one('uid', '1001'), '-g': one('gid', '1001'), '-G': one('groups', one('gid', '1001'))}.get(args[0], '')) +if name == 'getent': # login shell: shell-USER (nologin); password field: pw-USER (!); uid-USER; home-USER; gid-USER u = args[-1] if u not in lines('users'): sys.exit(2) if args[0] == 'passwd': - print(f'{u}:x:' + one('uid', '1001') + ':1001::' + one('home', '/nonexistent') + ':' + print(f'{u}:x:' + one('uid', '1001') + ':' + one('gid', '1001') + '::' + one('home', '/nonexistent') + ':' + one('shell', '/usr/sbin/nologin')) + elif args[0] == 'group': # the group named after the user: gid 1001 + print(f'{u}:x:1001:') else: print(f'{u}:' + one('pw', '!') + ':19000:0:99999:7:::') if name == 'useradd': @@ -526,6 +528,84 @@ def lstat(path, *a, **kw): # /opt/fork-fleet owned by another user (a test can self.assertIn('/opt/fork-fleet is not owned by root', err) self.assertEqual(self.tree(), before) + def test_a_linked_or_foreign_owned_subfolder_is_refused(self): + """lens225r4 finding 1 (probe): a link or foreign owner one level down (`units`, `fork-fleet-walls`) was + followed by install (the root-run walls and unit texts written into the link's target) and FAILed check + forever. Install now refuses it before changing anything, on a first install and on a later one.""" + other = os.path.join(self.tmp, 'elsewhere') + os.makedirs(other) + for sub in ('units', 'fork-fleet-walls'): + with self.subTest(sub): + shutil.rmtree(self.root) + self.write(self.path('/proc/sys/net/ipv4/ip_forward'), '1\n') + os.makedirs(self.path('/opt/fork-fleet')) + os.symlink(other, self.path(f'/opt/fork-fleet/{sub}')) + code, _, err = self.install() + self.assertEqual(code, 1, err) + self.assertIn(f'/opt/fork-fleet/{sub} is a link', err) + self.assertEqual(os.listdir(other), []) + os.unlink(self.path(f'/opt/fork-fleet/{sub}')) + self.assertEqual(self.install()[0], 0) + self.template() + self.assertNotIn('FAIL', self.run_main('check')[1]) + real_lstat, folder = os.lstat, self.path(f'/opt/fork-fleet/{sub}') + + def lstat(path, *a, **kw): # the subfolder owned by another user (a test cannot chown) + st = real_lstat(path, *a, **kw) + if os.fspath(path) != folder: + return st + return os.stat_result((st.st_mode, st.st_ino, st.st_dev, st.st_nlink, st.st_uid + 1, *st[5:])) + before = self.tree() + with unittest.mock.patch.object(self.m.os, 'lstat', lstat): + code, _, err = self.install(sha=SHA2) + self.assertEqual(code, 1) + self.assertIn(f'/opt/fork-fleet/{sub} is not owned by root', err) + self.assertEqual(self.tree(), before) + + def test_check_names_a_linked_public_key(self): + """lens225r4 nit 3: install refuses a linked desk-key.pub, but check only looked at the private key.""" + self.assertEqual(self.install()[0], 0) + self.template() + pub = self.path('/etc/fork-fleet/desk-key.pub') + self.write(os.path.join(self.tmp, 'pub-elsewhere'), slurp(pub)) + os.unlink(pub) + os.symlink(os.path.join(self.tmp, 'pub-elsewhere'), pub) + code, out, _ = self.run_main('check') + self.assertEqual(code, 1, out) + self.assertIn('FAIL kit /etc/fork-fleet/desk-key.pub: a link or not owned by root', out) + + def test_a_fifo_at_a_unit_or_setting_does_not_hang(self): + """lens225r4 nit 4: same_file (and ours) opened the path before testing it is a plain file; a FIFO hung + install and check. Now a FIFO is not the kit's file: check FAILs, install refuses (unit) or writes it (setting).""" + import signal + self.assertEqual(self.install()[0], 0) + self.template() + + def hung(*_): + raise AssertionError('hung on a FIFO') + old = signal.signal(signal.SIGALRM, hung) + self.addCleanup(signal.signal, signal.SIGALRM, old) + self.addCleanup(signal.alarm, 0) + conf = self.path('/etc/fork-fleet/launch.conf') + os.unlink(conf) + os.mkfifo(conf, 0o600) + signal.alarm(20) + code, out, _ = self.run_main('check') + self.assertEqual(code, 1, out) + self.assertIn('FAIL settings /etc/fork-fleet/launch.conf differs', out) + self.assertEqual(self.install()[0], 0) # the setting is written again, as a plain file + self.assertTrue(stat.S_ISREG(os.lstat(conf).st_mode)) + unit = self.path('/etc/systemd/system/fork-fleet-desk-agent.service') + os.unlink(unit) + os.mkfifo(unit, 0o644) + code, out, _ = self.run_main('check') + self.assertEqual(code, 1, out) + self.assertIn('fork-fleet-desk-agent.service: missing or not the kit\'s', out) + code, _, err = self.install() + self.assertEqual(code, 1) + self.assertIn('fork-fleet-desk-agent.service was not written by this kit', err) + signal.alarm(0) + def test_a_unit_or_setting_with_another_owner_fails_check_and_install_puts_it_back(self): """lens225r3 nit 5: units were checked for text and mode, not owner. A unit or setting that is a link or not root's FAILs check and is written again (as root) by install.""" @@ -551,6 +631,7 @@ def test_a_hand_made_user_in_other_groups_or_with_a_home_fails_check(self): self.template() os.makedirs(self.path('/home/fork-runner-1')) for name, text, want in (('groups-fork-runner-1', '1001 27', 'fork-runner-1 is in other groups (1001 27)'), + ('gid-fork-runner-1', '0', 'fork-runner-1 has the main group 0, not its own group'), ('uid-fork-runner-2', '0', 'fork-runner-2 has uid 0'), ('home-fork-runner-1', '/home/fork-runner-1', 'fork-runner-1 has the home')): with self.subTest(name): @@ -615,6 +696,21 @@ def test_refusals_change_nothing(self): 'a hand-made runner user with uid 0': ( lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n'), self.write(os.path.join(self.fakes, 'uid-fork-runner-1'), '0\n')), [], 'has uid 0'), + 'a hand-made runner user in group 0 only (useradd -g 0)': ( + lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-1\n'), + self.write(os.path.join(self.fakes, 'gid-fork-runner-1'), '0\n')), [], + 'has the main group 0, not its own group fork-runner-1'), + 'a linked /opt/fork-fleet/units': ( + lambda: (os.makedirs(os.path.join(self.tmp, 'elsewhere'), exist_ok=True), + os.makedirs(self.path('/opt/fork-fleet')), + os.symlink(os.path.join(self.tmp, 'elsewhere'), self.path('/opt/fork-fleet/units'))), [], + '/opt/fork-fleet/units is a link'), + 'a linked /opt/fork-fleet/fork-fleet-walls': ( + lambda: (os.makedirs(os.path.join(self.tmp, 'elsewhere'), exist_ok=True), + os.makedirs(self.path('/opt/fork-fleet')), + os.symlink(os.path.join(self.tmp, 'elsewhere'), + self.path('/opt/fork-fleet/fork-fleet-walls'))), [], + '/opt/fork-fleet/fork-fleet-walls is a link'), 'a hand-made runner user in other groups': ( lambda: (self.write(os.path.join(self.fakes, 'users'), 'fork-runner-2\n'), self.write(os.path.join(self.fakes, 'groups-fork-runner-2'), '1001 27\n')), [], @@ -658,7 +754,7 @@ def test_refusals_change_nothing(self): shutil.rmtree(self.root, ignore_errors=True) os.makedirs(self.root) for n in os.listdir(self.fakes): # the users of the case before - if n == 'users' or n.startswith(('shell-', 'pw-', 'uid-', 'groups-', 'home-')): + if n == 'users' or n.startswith(('shell-', 'pw-', 'uid-', 'gid-', 'groups-', 'home-')): os.unlink(os.path.join(self.fakes, n)) self.machine(MACHINE) self.write(os.path.join(self.fakes, 'os-exit'), '0') From 5be550a20f85fd0314985b45ee5c9b4091dc0958 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Mon, 16 Nov 2026 02:39:06 +0000 Subject: [PATCH 33/33] fork fleet seed round 5: once-per-wait skip for a new host copy, negative extended header, slash links, tarfile parts at run time (D11b) - Info._proc_member refuses a size below 0 before it is counted (Pythons without the CVE-2025-8194 fix). - A new copy on the host skips the back-off only once per wait (`early` in .torn.json); the log line says "about N h" and the at-once note only when it holds. - A link name ending in / is checked against its path without the slash. - run() refuses to fetch (exit 2) when tarfile lacks a private part the limits rely on. - Docs: the once-per-wait rule; a host down for good shows only as seeds that age out (documented, not changed). Co-Authored-By: Claude Opus 5.5 --- .github/fork-fleet.md | 17 +++-- .github/scripts/fork-fleet-desk-seed.py | 74 +++++++++++++------- .github/scripts/fork-fleet-desk-seed.test.py | 54 ++++++++++++++ fork-kit/desk/README.md | 10 +-- 4 files changed, 120 insertions(+), 35 deletions(-) diff --git a/.github/fork-fleet.md b/.github/fork-fleet.md index 8e2d5e2c0477b..9c50bb64e4699 100644 --- a/.github/fork-fleet.md +++ b/.github/fork-fleet.md @@ -755,16 +755,19 @@ in all (the unit allows three), each seed at most 30 minutes. A seed torn again and again is tried again after about 1, 2, 4, 8, 16 and then every 24 hours (a link too slow for it does not spend every hour on it; a fetch that kills the run counts as torn; the times are approximate: the timer's 10 minute random -delay and a 5 minute margin can make a wait up to an hour longer), and at once -when the host offers another copy of it. One torn on every try for more than a -day makes the run exit 1, so `desk-install check` shows FAIL until it is +delay and a 5 minute margin can make a wait up to an hour longer); another +copy on the host is tried at once, but only once per wait (the try counts, so a +host that remakes the seed every hour still meets the back-off). One torn on +every try for more than a day makes the run exit 1, so `desk-install check` shows FAIL until it is fetched (`/var/lib/fork-fleet/seeds/.torn.json` names it and why); so does a run that cannot write that file (a full disk). A host that gives no list (down, an older kit, a half-done install) means no new seeds: each one here stays until -its age drops it, and the torn record is left as it is. A desktop whose own fleet config is broken or lacks its pool fetches -nothing and exits 2 (`desk-install check` shows the seed service's last -result), but still drops each seed past the `max_age_hours` it was fetched -with. Jobs see +its age drops it, the torn record is left as it is, and the run exits 0 (a host +down for good shows in `desk-install check` only as seeds that age out). A +desktop whose own fleet config is broken or lacks its pool, or whose Python's +tarfile lacks the private parts the limits above rely on, fetches nothing and +exits 2 (`desk-install check` shows the seed service's last result), but still +drops each seed past the `max_age_hours` it was fetched with. Jobs see `/var/lib/fork-fleet/seeds/NAME` read-only (the launcher's `ALLOW_PATHS`; the unpacking and old copies beside it, `.new` and `.v`, are visible too and hold the same data), copy what they need from it when it is there, and run diff --git a/.github/scripts/fork-fleet-desk-seed.py b/.github/scripts/fork-fleet-desk-seed.py index f4aa3f3bf9478..451df194c1e02 100644 --- a/.github/scripts/fork-fleet-desk-seed.py +++ b/.github/scripts/fork-fleet-desk-seed.py @@ -36,8 +36,10 @@ the copy is removed, the old seed stays, the other seeds go on, old versions are still pruned; tried again after about 1, 2, 4, 8, 16, then 24 hours (TornLog; the hourly timer's RandomizedDelaySec and a 5 min - margin make each wait up to an hour longer), at once when the host - offers another copy; torn on every try for over TORN_DAY: exit 1 + margin make each wait up to an hour longer); another copy on the host + skips the wait once per back-off step (the try still counts, so a + host that remakes it hourly still backs off); torn on every try for + over TORN_DAY: exit 1 .torn.json cannot be written (a full disk) the run goes on, counts its tries in memory, and exits 1 a fetch that kills the run (MemoryMax, TimeoutStartSec) counted as torn before it began; the next run prunes first @@ -45,7 +47,8 @@ no seed (never fetched, the host down) jobs run cold no list (the host down, an older kit that answers with a session or not JSON, exit 1 of a half-done install) nothing fetched, exit 0; each seed stays until its age drops it; the - torn record (.torn.json) is left as it is + torn record (.torn.json) is left as it is (a host down for good shows + in check only as seeds that age out) its own fleet config broken or without POOL nothing fetched, exit 2; a seed older than its label's maxAgeHours (its own max_age_hours when it was fetched) is still dropped older than max_age_hours, made over SKEW in the future (the clocks disagree), or no longer in the pool's list @@ -339,6 +342,10 @@ def whole(x): tarfile.GNUTYPE_LONGLINK) +PRIVATE = ('_proc_member', '_proc_sparse', '_proc_gnusparse_00', '_proc_gnusparse_01', '_proc_gnusparse_10', + '_find_link_target') # the private tarfile parts Info's and Tar's limits rely on: without one, no fetch + + class Info(tarfile.TarInfo): """A tar header read with limits tarfile has not: it reads an extended header (pax, or a GNU long name) into memory whole and its parsed form takes many times its size, and a sparse member's map can grow without end. So the @@ -346,6 +353,8 @@ class Info(tarfile.TarInfo): whole tar too, and a sparse member is refused (make a seed without --sparse). Torn before tarfile reads one.""" def _proc_member(self, tarfile_): + if self.size < 0: # tarfile refuses it only from the CVE-2025-8194 fix on; here it would lower `used` + raise Torn(f'{self.name[:80]!r} has a size below 0') used = tarfile_.__dict__.setdefault('fork_headers', [0, 0]) # [before this member, global in all] if self.type in EXTENDED: used[0] += self.size @@ -381,7 +390,8 @@ def unpack(t, data, most): def linked(member, path): got = check(member, path) - if got is not None and (got.islnk() or got.issym()) and os.path.lexists(os.path.join(path, got.name)): + if got is not None and (got.islnk() or got.issym()) and os.path.lexists( + os.path.join(path, got.name.rstrip('/'))): # 'a/' names the path 'a' too raise Torn(f'{got.name[:80]!r} is a link to {got.linkname[:80]!r} where a member is already there') if got is not None and got.islnk(): try: @@ -580,12 +590,15 @@ def own_seeds(conf, config): def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): """One run (the module's docstring). A seed that fails in any way, a deep tree included, is said and the run - goes on with the next; old versions are pruned in every run. When it may fetch nothing (no data filter, its own - fleet config broken or without its pool) it still drops each seed past its age, by its label's max_age_hours, - and then says why (agent.Bad: exit 2, the unit's last result shows it).""" + goes on with the next; old versions are pruned in every run. When it may fetch nothing (no data filter or no + private tarfile part in PRIVATE, its own fleet config broken or without its pool) it still drops each seed past + its age, by its label's max_age_hours, and then says why (agent.Bad: exit 2, the unit's last result shows it).""" cannot, own = None, None if not hasattr(tarfile, 'data_filter'): # releases with known filter bypasses pass this: settle() checks again cannot = 'this Python has no tarfile data filter (3.12, or 3.8.17 and later)' + elif gone := [k for k in PRIVATE if not callable(getattr(tarfile.TarFile if k == '_find_link_target' + else tarfile.TarInfo, k, None))]: + cannot = f'this Python\'s tarfile has no {", ".join(gone)} (the limits of Info and Tar need them)' try: own = own_seeds(conf, config) except Torn as e: @@ -636,14 +649,15 @@ def run(agent, conf, root=SEEDS, now=time.time, say=print, config=CONFIG): class TornLog: - """SEEDS/.torn.json, {name: {"first", "last", "tries", "why", "copy"}}: the seeds whose last fetch was torn, - written before each fetch (so a fetch that kills the run, by its memory or time limit, counts as torn too) and - cleared when the seed is fetched or the same as the host's. A seed torn TRIES times in a row is tried again only - min(2 ** (TRIES - 1), 24) hours after the last try (a link too slow for it, a host that cuts it: not every - hour), or at once when the host offers another copy (`copy`: its [bytes, made] when last tried); one torn on - every try since more than TORN_DAY is `late`. Only seeds in the host's list are kept, and only in a run that - read the list. A missing or unreadable file is an empty record; one that cannot be written (a full disk) is - kept in memory for this run and said in `unsaved`.""" + """SEEDS/.torn.json, {name: {"first", "last", "tries", "why", "copy", "early"}}: the seeds whose last fetch was + torn, written before each fetch (so a fetch that kills the run, by its memory or time limit, counts as torn too) + and cleared when the seed is fetched or the same as the host's. A seed torn TRIES times in a row is tried again + only min(2 ** (TRIES - 1), 24) hours after the last try (a link too slow for it, a host that cuts it: not every + hour), or at once when the host offers another copy (`copy`: its [bytes, made] when last tried), but that only + once per back-off step (`early`: the last try was such a one); one torn on every try since more than TORN_DAY + is `late`. Only seeds in the host's list are kept, and only in a run that read the list. A missing or unreadable + file is an empty record; one that cannot be written (a full disk) is kept in memory for this run and said in + `unsaved`.""" def __init__(self, root, now): self.path, self.now, self.seeds = root and os.path.join(root, '.torn.json'), now, {} @@ -659,20 +673,30 @@ def __init__(self, root, now): def keep(self, items): self.seeds = {n: r for n, r in self.seeds.items() if n in items} - def due(self, name, item=None): - """None when the seed may be tried now (ITEM, the host's offer, another copy than the one last tried - included), else the hours until it may.""" + def left(self, name): + """The seconds until the seed's back-off is over, None when it is (or it has no record).""" r = self.seeds.get(name) - if not r or item and r.get('copy') not in (None, [item['bytes'], item['made']]): + if not r: return None wait = min(2 ** (min(r['tries'], 6) - 1), 24) * 3600 - 300 # 5 min early: the hourly timer drifts left = r['last'] + wait - self.now() - return math.ceil(left / 3600) if 0 < left <= wait else None # a date ahead of the clock: try now + return left if 0 < left <= wait else None # a date ahead of the clock: try now + + def due(self, name, item=None): + """None when the seed may be tried now, else the hours until it may. ITEM, the host's offer, may skip the + wait when it is another copy than the one last tried, but only once per back-off step (`early`: the last + try skipped it), so a host that remakes the seed every hour still meets the back-off.""" + r, left = self.seeds.get(name), self.left(name) + if left is None or item and not r.get('early') and r.get('copy') not in (None, [item['bytes'], item['made']]): + return None + return math.ceil(left / 3600) def trying(self, name, item=None): - """Counted as torn before the fetch, saved: `why` says so until the fetch ends.""" + """Counted as torn before the fetch, saved: `why` says so until the fetch ends; `early` when it is tried + before its back-off is over (a new copy).""" + early = self.left(name) is not None r = self.seeds.get(name) or {'first': int(self.now()), 'tries': 0} - self.seeds[name] = {**r, 'last': int(self.now()), 'tries': r['tries'] + 1, + self.seeds[name] = {**r, 'last': int(self.now()), 'tries': r['tries'] + 1, 'early': early, 'why': 'the run stopped during the fetch (its memory or time limit?)'} if item: self.seeds[name]['copy'] = [item['bytes'], item['made']] @@ -769,8 +793,10 @@ def take(agent, conf, item, root, now, say, torn=None, end=None): except Exception as e: # any: a torn stream, a full disk, a tar this Python cannot unpack why = f'{e.__class__.__name__}: {str(e)[:300]}' torn.torn(name, why) - hours = min(2 ** (min(torn.seeds[name]['tries'], 6) - 1), 24) - say(f'seed {name}: keeps {"the old one" if label else "none"}: {why}; tried again in {hours} h') + r = torn.seeds[name] + hours = min(2 ** (min(r['tries'], 6) - 1), 24) + say(f'seed {name}: keeps {"the old one" if label else "none"}: {why}; tried again in about {hours} h' + + ('' if r['early'] else ' (at once if the host makes another copy before then)')) return False torn.fine(name) say(f'seed {name}: {got["bytes"] >> 20} MB, sha256 checked, swapped in') diff --git a/.github/scripts/fork-fleet-desk-seed.test.py b/.github/scripts/fork-fleet-desk-seed.test.py index 53982322e9844..1553a52683f9e 100644 --- a/.github/scripts/fork-fleet-desk-seed.test.py +++ b/.github/scripts/fork-fleet-desk-seed.test.py @@ -912,6 +912,60 @@ def test_a_negative_size_never_lowers_the_disk_count(self): # lens237sr3: the self.assertIsNone(self.files('m2')) self.assertTrue(any('m2' in x and 'size' in x for x in self.said), self.said) + def test_a_negative_extended_header_never_lowers_the_header_count(self): # lens237sr4: the MUST + def hdr(name, typ, size): + d = bytearray(tarfile.TarInfo(name).tobuf(format=tarfile.GNU_FORMAT)) + d[156:157], d[124:136] = typ, size + d[148:156] = b' ' * 8 + d[148:156] = b'%06o\0 ' % sum(d[:512]) + return bytes(d) + neg = b'\xff' + ((1 << 88) - (1 << 20)).to_bytes(11, 'big') # -1 MB, base-256 + body = b'A' * ((2 << 20) - 1) + b'\0' # a 2 MB long name, over HEADER_MOST + blob = hdr('n', tarfile.GNUTYPE_LONGNAME, neg) + hdr('b', tarfile.GNUTYPE_LONGNAME, b'%011o\0' % len(body)) \ + + body + tarfile.TarInfo('f').tobuf(format=tarfile.GNU_FORMAT) + b'\0' * 1024 + old_block = lambda self_, n: -(-n // 512) * 512 # tarfile before the CVE-2025-8194 fix: no size check + with mock.patch.object(tarfile.TarInfo, '_block', old_block), self.assertRaisesRegex(seed.Torn, 'below 0'): + with seed.Tar.open(fileobj=io.BytesIO(blob), mode='r|', tarinfo=seed.Info) as t: + t.next() # before round 5: the 2 MB name read whole, the count at [0, 0] + + def test_a_link_named_with_a_slash_over_a_file_is_refused(self): # lens237sr4 nit + link = tarfile.TarInfo('a/') + link.type, link.linkname = tarfile.SYMTYPE, 'x' + self.raw('m2', self.block(tarfile.TarInfo('a'), b'1') + self.block(tarfile.TarInfo('x'), b'2') + + self.block(link)) + self.run_seed() + self.assertIsNone(self.files('m2')) # before round 5: the symlink replaced the file a + self.assertTrue(any('m2' in x and 'already there' in x for x in self.said), self.said) + + def test_a_python_without_the_private_tarfile_parts_fetches_nothing(self): # lens237sr4 nit: exit 2 + self.make('m2', {'a': b'one'}) + for owner, k in ((tarfile.TarInfo, '_proc_member'), (tarfile.TarInfo, '_proc_gnusparse_01'), + (tarfile.TarFile, '_find_link_target')): + with self.subTest(k), mock.patch.object(owner, k, None): + with self.assertRaisesRegex(agent.Bad, k): + self.run_seed() + self.assertEqual(self.fetches(), []) + self.assertEqual(self.run_seed(), 0) + self.assertEqual(self.files('m2'), {'a': b'one'}) + + def test_a_host_that_remakes_each_hour_never_meets_the_back_off(self): # lens237cr4 + sr4: the MUST + start, tried = int(time.time()) - 13 * 3600, [] + + def torn(agent_, conf, item, root, now, **kw): + tried.append(round((now() - start) / 3600)) + raise seed.Torn('cut off (slow link)') + said = [] + with mock.patch.object(seed, 'fetch', torn): + for h in range(12): + self.make('m2', {'a': b'%d' % h}, mtime=start + h * 3600 - 60) + self.run_seed(now=lambda h=h: start + h * 3600) + said += self.said + self.assertEqual(tried, [0, 1, 2, 6, 7]) # a new copy skips a wait once per step; before round 5: 0..11 + fails = [x for x in said if 'keeps' in x] + self.assertEqual(len(fails), 5, said) + self.assertTrue(all('tried again in about' in x for x in fails), fails) + self.assertEqual(['at once if' in x for x in fails], [True, True, False, True, False]) + def test_one_lost_list_does_not_reset_the_torn_record(self): # lens237cr3 self.make('m2', {'a': b'one'}) start, codes, real, tried = int(time.time()), {}, seed.listing, [] diff --git a/fork-kit/desk/README.md b/fork-kit/desk/README.md index d99ee0d57ef77..98f8a86b66a55 100644 --- a/fork-kit/desk/README.md +++ b/fork-kit/desk/README.md @@ -75,10 +75,12 @@ runs; keep them root's. One install or uninstall runs at a time (a lock on The seed timer (`t3code-fleet-seed.timer`) fetches the pool's warm caches, if its fleet config lists any, into `/var/lib/fork-fleet/seeds`, which jobs see read-only ([fork-fleet.md](../../.github/fork-fleet.md), "Seeds"). A torn seed is tried again after about 1, 2, 4, 8, 16, then 24 hours (up to an hour later: the timer's random delay), -or at once when the host offers another copy; one torn on every try for more than a day makes each run that reads the -host's list exit 1, so `check` shows the seed service's last run as failed (a run that cannot write the record, a full -disk, exits 1 too). `/var/lib/fork-fleet/seeds/.torn.json` names it and the host's reason (jobs can read that file -too); a run without the host's list leaves it as it is. +or at once when the host offers another copy, but that only once per wait (a host that remakes the seed every hour +still meets the back-off); one torn on every try for more than a day makes each run that reads the host's list exit +1, so `check` shows the seed service's last run as failed (a run that cannot write the record, a full disk, exits 1 +too). `/var/lib/fork-fleet/seeds/.torn.json` names it and the host's reason (jobs can read that file +too); a run without the host's list leaves it as it is and exits 0, so a host that stays down shows in `check` only +as seeds that age out. ## Windows (WSL2)