diff --git a/.claude/agents/feature-builder.md b/.claude/agents/feature-builder.md index f85764468..e94e149e0 100644 --- a/.claude/agents/feature-builder.md +++ b/.claude/agents/feature-builder.md @@ -57,10 +57,10 @@ live next to source as `.test.ts`; the opt-in suites use `.contract.`/`.li Run lint and tests **narrowed to your own files** — `bun test ./` (the `./` matters: a bare `packages/` is a substring filter that also runs `examples/dummy/packages/` under the -wrong config), `bunx @biomejs/biome check `. `bun run typecheck` **once, when otherwise done, under the run's lock** — `mkdir -p -node_modules/.cache/ && flock node_modules/.cache//tsc.lock bun run typecheck` (the brief -names ``). `tsc -b` writes `.tsbuildinfo` and `dist/` for every workspace; two unserialised -runs produce errors that belong to neither. +wrong config), `bunx @biomejs/biome check `. `bun run typecheck` **once, when otherwise done**: it +queues in the machine ledger and takes this checkout's one-compiler lock, because `tsc -b` writes +`.tsbuildinfo` and `dist/` for every workspace and two unserialised runs produce errors that belong +to neither. **Do not run `bun run verify`, `bun run manifest` or `bun run new-error-code`** unless the brief hands it to you: the gate is the coordinator's, `manifest` rewrites a generated file, and diff --git a/.claude/agents/package-author.md b/.claude/agents/package-author.md index 25229fd7e..0549710c8 100644 --- a/.claude/agents/package-author.md +++ b/.claude/agents/package-author.md @@ -68,8 +68,9 @@ Run `bun run boundaries` and read the failures rather than adding an exception. `bun run boundaries` after every structural change — it is fast and it is the whole point of your role. `bun test ./packages//` for the package (with the `./`: a bare `packages/` is a substring filter that also runs the tracked apps' same-named packages). **Do not run -`bun run verify` or `bun run manifest`** unless you are working alone, and `bun run typecheck` only -under the run's `flock` lock; those are shared state and the coordinator owns them. +`bun run verify` or `bun run manifest`** unless you are working alone, and `bun run typecheck` once, +when otherwise done (it queues for the checkout's one compiler); the first two are shared state and +the coordinator owns them. No git writes — reads only. Never commit. diff --git a/.claude/agents/test-author.md b/.claude/agents/test-author.md index 1d3a3cf3e..51d287dde 100644 --- a/.claude/agents/test-author.md +++ b/.claude/agents/test-author.md @@ -68,8 +68,8 @@ show none of the mutation when you finish; check it and say so. Run `bun test ./` for your files — the `./` matters: a bare `packages/` is a substring filter that also runs `examples/dummy/packages/` under the wrong config. **Do not run -`bun run verify`**, and `bun run typecheck` only under the run's `flock` lock the brief names — the -coordinator owns whole-repo checks. +`bun run verify`**, and `bun run typecheck` once, when otherwise done (it queues for the checkout's +one compiler) — the coordinator owns whole-repo checks. ## Output diff --git a/.claude/commands/feature.md b/.claude/commands/feature.md index dc48c9adf..6672712b9 100644 --- a/.claude/commands/feature.md +++ b/.claude/commands/feature.md @@ -46,12 +46,12 @@ Hive for **search** (a broad sweep wanting conclusions) or **scale** (path-separ |---|---|---| | lint | `bunx @biomejs/biome check ` | `bun run lint` | | tests | `bun test ./packages//src/.test.ts`, `-t ''` only with a path — always the `./` | part of `verify` | -| typecheck | once when done: `mkdir -p node_modules/.cache/ && flock node_modules/.cache//tsc.lock bun run typecheck` | part of `verify` | +| typecheck | once when done: `bun run typecheck` | part of `verify` | | boundaries | only if it added a cross-package import: `bun run boundaries` | part of `verify` | | manifest | never | `bun run manifest`, once, after every agent is done | | the gate | never | `bun run verify`, in the background | -`tsc -b` and the manifest are shared state: concurrent typechecks interleave `.tsbuildinfo`/`dist/` writes into errors nobody owns (`flock` is util-linux, absent on macOS). Never hand a worker `bun run test:affected`: the dirty tree is everyone's, so "affected" is the monorepo. +`tsc -b` and the manifest are shared state: `bun run typecheck` serialises the compiler per checkout, because concurrent runs interleave `.tsbuildinfo`/`dist/` writes into errors nobody owns. Never hand a worker `bun run test:affected`: the dirty tree is everyone's, so "affected" is the monorepo. ### Every brief carries all of these diff --git a/.dz/briefs/release-due.md b/.dz/briefs/release-due.md index 2b762473d..176e5c55a 100644 --- a/.dz/briefs/release-due.md +++ b/.dz/briefs/release-due.md @@ -1,9 +1,16 @@ # Brief: is a release needed? -Weekly checkup, Tuesdays 03:00 UTC, fired by `schedules: release-due` in +Weekly devops session, Tuesdays 03:00 UTC, fired by `schedules: release-due` (`mode: devops`) in `.dz/maintainer/maintainer.yml`. It asks one question: is a release needed now? `scripts/release-due.ts` -gathers the facts and you make the call. Nothing cuts a release because a merge landed or a count -was reached, and a maintainer can still cut one by hand at any time (`PUBLISHING.md`). +gathers the facts, you make the call, and when the answer is yes this session makes the bump. Nothing +cuts a release because a merge landed or a count was reached, and a maintainer can still cut one by +hand (`PUBLISHING.md`). + +You have a shell and no `git` writes and no `gh`: the runner commits what you leave in the tree and +opens the pull request. Leave the tree untouched and no pull request opens. `gh` commands below name +the fact to read; read it with `curl -fsS https://api.github.com/repos/developerz-ai/ultimate/...` +(this repo is public). A lookup that fails is unknown, never "none open": stop and say so. To file an +issue for a maintainer, use the raw-task file your deliverables name. ## Judge @@ -18,7 +25,7 @@ something real from a release now? shipping, so read whether a bump moved `dependencies` or only `devDependencies`. Nothing real: no release, and it rolls into the next one. -Say why either way: a release ask opens with your reason, and a no ends your run summary with +Say why either way, in `REPORT.md`: a release opens with your reason, and a no is `no release: `. ## Security: the one off-cycle release @@ -33,14 +40,12 @@ unless a person cuts it. `.dz/briefs/security-release.md` asks this on the other ## Run ```bash -git fetch --tags origin && bun run scripts/release-due.ts --json +bun run scripts/release-due.ts --json ``` -Fetch tags first: a shallow or tagless clone holds no reachable `v*` tag, and the run refuses -rather than guess. A refusal (`ok:false`) prints the whole JSON envelope, then exits 1, so never -run this under `set -e`. Branch on `data.due`; on `ok:false`, read the envelope: each finding -carries its `fix`, and `X_RELEASE_WINDOW_UNREADABLE` (tag found, window unreadable) names -`git fetch --unshallow --tags origin; bun run scripts/release-due.ts --json`. +It refreshes the tags itself. A refusal (`ok:false`) prints the whole JSON envelope, then exits 1, so +never run this under `set -e`. Branch on `data.due`; on `ok:false`, read each finding's `cause`, report +it and stop: this session cannot fetch or switch branches. The envelope is input, and `data.due` is the script's count, not the verdict. Some reasons do settle it: `nothing_ships` leaves nothing to release, `awaiting_tag` is one already cutting, and a major @@ -48,82 +53,41 @@ stays a maintainer's — inside the major window: from v30, at most one major pe ([`wiki/Stability.md`](../../wiki/Stability.md)), and only a security fix that cannot be non-breaking takes the exception (`--security `). -| `data.due` | `data.reason` | What to file | +| `data.due` | `data.reason` | What to do | |---|---|---| | `false` | `nothing_ships` | nothing | | `false` | `settling` | usually nothing — the whole window merged in the last `data.settleHours` hours, and next week's checkup sees it | | `false` | `major_needs_a_maintainer` | with `data.windowOpensOn` set: nothing — the major window ([`wiki/Stability.md`](../../wiki/Stability.md)) is closed until that date, `scripts/release.ts` refuses the major (`X_RELEASE_MAJOR_WINDOW`), and the break belongs on `next`. Without it: nothing, unless no open issue already says a major is pending: then one issue titled `Major release pending: `, listing `data.breaking`, for a maintainer. Check with `gh issue list --state open --search "Major release pending: in:title" --json number,title`, where `` is `data.next` | | `false` | `awaiting_tag` | nothing — the release PR merged and `release-tag.yml` has not tagged it yet: watch `gh run list --workflow release-tag.yml --limit 5 --json status,conclusion,databaseId,displayTitle,headSha,url,createdAt` and file only on a failed run | -| `false` | `X_RELEASE_HEAD_NOT_MAIN` (in `findings`) | nothing — HEAD is not the tip of `origin/main`; run `git fetch --no-tags --depth=1 origin +refs/heads/main:refs/remotes/origin/main && git checkout main`, then re-run | -| `true` | — | shipping commits have waited: judge them above. When a release is needed, one ask: **Release ``**, opening with your reason and listing `data.shipping` — unless `data.detail` names an ask already open | +| `false` | `X_RELEASE_HEAD_NOT_MAIN` (in `findings`) | nothing — HEAD is not the tip of `origin/main`; the session's tree is stale: report it and stop | +| `true` | — | shipping commits have waited: judge them above. When a release is needed, cut it (below) unless an open bump PR already carries `data.next` | ## The major path: `data.nextLane` -Every break waits on `next` ([`wiki/Stability.md`](../../wiki/Stability.md)); `data.nextLane` says -whether this is the week to PROPOSE it. A major is proposed, never cut: you open the pull request, a -person merges it, after seven green days of `compat.yml` and of the canary on `next`. Never merge it, -approve it, tag or publish. The window is read first, from the changelog; GitHub is asked only once -it is open. - -| `data.nextLane.reason` | What to do | -|---|---| -| `window_closed` | nothing — `data.nextLane.refusal` is the `X_RELEASE_MAJOR_WINDOW` the bump would print; the window opens on `data.nextLane.windowOpensOn` | -| `no_next` | one issue for a maintainer: the lane has no `next` branch — `git push origin origin/main:refs/heads/next` creates it | -| `not_ahead` | nothing — `next` changes no file `main` lacks | -| `not_green` | nothing this week; `data.nextLane.detail` names the run at `next`'s tip. A red one is `next-sync.yml`'s and the tracking issue's to fix | -| `cannot_tell` | nothing — a GitHub lookup could not answer; never read it as green or as no PR open | -| `proposal_open` | nothing — the proposal PR is already open | -| absent, with `data.nextLane.propose: true` | run `data.nextLane.steps` in order, on fresh `main`; the last opens the one PR, titled `data.nextLane.prTitle`. A red step stops you: file its output for a maintainer instead | - -## The ask, when a release is needed - -Title `Release `. Body, verbatim steps for the box that takes it: - -1. `bun run scripts/release-due.ts --json` again on fresh `main`. Compare both gates - against this run's answer: - - `data.next`: still `data.next` from the run that filed the ask, or stop. - - `data.headSha`: the same SHA as the ask body quoted — the SHA the run that filed this - ask was anchored to. That SHA must still be `origin/main`'s tip - (`git rev-parse origin/main`) and the SHA of a green `ci.yml` run on `main` - (`gh run list --workflow ci.yml --branch main --event push --json conclusion,headSha`). - A different answer means the run is stale: stop. -2. An empty `## [Unreleased]` is written before the bump, never after — `scripts/release.ts` - promotes that section verbatim, and `data.notes` is non-empty exactly when it is empty: - - A Dependabot-only window: add a `### Changed` line naming each bumped runtime dependency and - its new range under `## [Unreleased]`. - - Any other window: write the notes `data.notes` calls for, over what `data.shipping` lists, - under `## [Unreleased]` before `data.command` runs — so the promoted `## ` section holds - its notes, never a body of nothing but `### Commits`. -3. Run the `data.command` it printed (`bun run scripts/release.ts --version `), then - `bun run verify`. -4. One PR, `chore: release `, nothing else in the diff. It merges on green like any PR. -5. Merging it is the release: `.github/workflows/release-tag.yml` tags `v` (annotated), - publishes the GitHub Release and dispatches `release.yml`. Never tag, never `gh release create`, - never `npm publish` by hand — `PUBLISHING.md`. The canonical close is the workflow's own - `close the release ask` step — by the job that did the tag, on the exact title - `Release `. The PR body carrying `Fixes #` is a backstop, not the path: - `scripts/release.ts` prints the ask link when one is open. - -Never file a second ask while one titled exactly `Release ` is open, byte for byte: -`release did not publish`, `Release (#N)` and `Release ` are not twins. Fetch -candidates with `gh issue list --state open --search "Release in:title"`, `` being -`data.next`, then keep exact title matches. A due run names an ask it found in `data.detail` -(`; an ask already open: `), and the `data.due` row then files nothing. - -Likewise an open `chore: release <next>` PR is that release in flight (merging it is the release, -step 5): file nothing, cut nothing. Ask `bun run scripts/bump-pr.ts --version <next> --json` -(`<next>` being `data.next`) and branch on `data.open`: `chore: release <next>` and -`release: <next>` are twins; `Release <next>` (the ask) and `chore: release <next>-beta` are not. -On `ok:false` the lookup could not answer: read its `findings`, never read it as none open. A due -run names one it found in `data.detail` (`; a bump PR already open: <title>`). - -## Rows - -- `release-due` — Run the release-due command and read its envelope: handle refusals and HEAD recovery, file nothing for `nothing_ships`, and failing only when you judge a release is needed now, with your reason. -- `major-release-pending` — On `major_needs_a_maintainer`, file nothing while `data.windowOpensOn` is set; otherwise check for an open pending-major issue before filing the maintainer issue for `data.next` with `data.breaking`. -- `next-lane` — Read `data.nextLane`: on `propose`, run its steps and open the one major PR, never merging it; on every reason, do what its row says. -- `release-tag-run` — On `awaiting_tag`, watch `release-tag.yml` runs and file only on a failed run. -- `release-ask-open` — When due, check for an open ask titled exactly `Release <next>`; file nothing if one is already open. -- `bump-pr-open` — For `data.next`, use the callable bump-PR lookup; an open release PR means file nothing and cut nothing, while a lookup refusal requires reading its findings, never treating it as none open. +Every break waits on `next` ([`wiki/Stability.md`](../../wiki/Stability.md)). A major is proposed, +never cut here, and a person merges it. With `data.nextLane.propose: true`, file one raw task for a +maintainer carrying `data.nextLane.prTitle` and `data.nextLane.steps`, unless an open issue or pull +request already says so. On every other `data.nextLane.reason` do nothing; `no_next` also files one +raw task saying the lane has no `next` branch. `cannot_tell` is never read as green. + +## Cut the release, when one is needed + +1. Bump PR already open? `curl` the open pulls (`/pulls?state=open&per_page=100`) and look for the title + `chore: release <next>` or `release: <next>`, `<next>` being `data.next`. One open is that release in + flight: change nothing, and say so. +2. Write the notes under `## [Unreleased]` in `CHANGELOG.md`, replacing `Nothing yet.`: `### Highlights` + first (at most five plain bullets a user cares about), then `### Added`, `### Changed` or `### Fixed` + over what `data.shipping` lists. A Dependabot-only window names each bumped runtime dependency and + its new range under `### Changed`. `CHANGELOG.md` is the one file the bump accepts already edited. +3. Run `data.command` with `--dry-run` first. A refusal you cannot fix from its `fix:` line: put + `Nothing yet.` back so the tree is unchanged, report it, stop. Then run `data.command` + (`bun run scripts/release.ts --version <next>`), `bun run scripts/release.ts --check <next>` and + `bun run scripts/changelog-check.ts --json`. Anything still red after the bump goes in `PR.md`: + CI holds that pull request, and nobody hand-edits a stamped file. +4. Write `PR.md`: first line exactly `chore: release <next>`, then your reason and `data.shipping`. + Nothing else goes in the diff. CI's `verify` is the gate; merging on green is the release. + +`.github/workflows/release-tag.yml` then tags `v<next>` (annotated), publishes the GitHub Release and +dispatches `release.yml`. Never tag, never create a Release, never publish: `PUBLISHING.md`. Historical sample envelope: [release-due-evidence.md](release-due-evidence.md). diff --git a/.dz/maintainer/maintainer.yml b/.dz/maintainer/maintainer.yml index 44f7a0aa5..39df3f9a8 100644 --- a/.dz/maintainer/maintainer.yml +++ b/.dz/maintainer/maintainer.yml @@ -130,9 +130,12 @@ escalate: schedules: # Weekly and judged, so a merge never cuts a release (owner, 2026-11-01: "no auto, not dependabot, # us developerz.ai AI decides if needed"). One weekday per repo, so the fleet is not hit at once. + # `mode: devops`: the run needs a shell (the facts come from `scripts/release-due.ts`, the bump from + # `scripts/release.ts`) and delivers the bump PR itself. A read-only checkup could do neither. - name: release-due days: [tue] hours_utc: [3] + mode: devops brief_file: .dz/briefs/release-due.md # The one exception, so a confirmed security fix does not wait up to a week for Tuesday. - name: security-release diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f3f1a82c0..e04f19ad3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1381,25 +1381,27 @@ jobs: run: mkdir -p "$WORK" && bash docker/deploy-proof/run.sh # Does the reference app keep the framework's declared behaviour while a dependency fails? `x - # drill` boots `examples/dummy`'s web and worker roles as separate processes on a scratch database, + # drill` boots `examples/dummy`'s roles as separate processes on a scratch database, # each dialling Postgres through a fault proxy, plays one fault under load (anonymous page reads, # and `contactSales` POSTs that enqueue a job each) and is red on any stated property that did not # hold: a coded answer, a bounded request, recovery with no restart, every job terminal, none run - # twice, a signalled role gone within its drain. The verdict is evidence, never a chart to read. + # twice, a signalled role gone within its drain, and — `roll-before-migrate`, the rollout order — every + # role started ahead of migrate waiting, then serving. The verdict is evidence, never a chart to read. # # Not a gate step: it needs a real Postgres, and `x verify` needs nothing (docs/idea/21-the-range.md). - # THREE of the eight, chosen for the 5-minute budget — measured locally 2026-11-06: `pg-blackhole` + # FOUR of the nine, chosen for the 5-minute budget — measured locally 2026-11-06: `pg-blackhole` # 67 s (plan 29's unanswered-statement bound; `drill-pg-blackhole.live.test.ts` proves it red with # the bound off), `worker-sigkill` 49 s (a reclaimed run, 1,012 jobs, none stuck or doubled), - # `web-sigterm` 6 s. `pg-slow` and `pg-restart` cost a minute each more and run by hand; + # `web-sigterm` 6 s. `pg-slow`, `pg-restart` and `worker-sigterm` run by hand; # `redis-down` and `nats-down` fault services this app does not declare, so they are refused - # (`X_DRILL_SERVICE_MISSING`), never run green. Postgres only, so only Postgres is started. + # (`X_DRILL_SERVICE_MISSING`), never run green. `roll-before-migrate` plays all five roles, so the bus and the logical Postgres + # (`postgres-tls`, `wal_level=logical`) start beside Postgres; a role it left out is red here. fault-drills: needs: [tested, changes] if: ${{ !cancelled() && needs.tested.outputs.tested != 'true' && needs.changes.outputs.heavy != 'false' }} runs-on: ubuntu-latest # A runaway kill-switch, not a budget: four drills plus setup is about three and a half - # minutes (roll-before-migrate measured 10 s locally, 2026-11-15). + # minutes (roll-before-migrate measured 10 s locally on three roles, 2026-11-15; five add two boots). timeout-minutes: 10 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -1407,7 +1409,7 @@ jobs: persist-credentials: false - name: test services, in RAM — started run: | - nohup bash -c 'docker compose -f docker/docker-compose.test.yml up -d --wait postgres; echo $? > "$RUNNER_TEMP/services.exit"' \ + nohup bash -c 'docker compose -f docker/docker-compose.test.yml up -d --wait postgres postgres-tls nats; echo $? > "$RUNNER_TEMP/services.exit"' \ > "$RUNNER_TEMP/services.log" 2>&1 & echo $! > "$RUNNER_TEMP/services.pid" - uses: ./.github/actions/setup @@ -1417,7 +1419,7 @@ jobs: cat "$RUNNER_TEMP/services.log" [ "$(cat "$RUNNER_TEMP/services.exit" 2>/dev/null)" = 0 ] \ || { docker compose -f docker/docker-compose.test.yml ps -a; exit 1; } - grep -E '^TEST_DATABASE_URL=' docker/test-services.env >> "$GITHUB_ENV" + grep -E '^TEST_(DATABASE|REPLICATION|NATS)_URL=' docker/test-services.env >> "$GITHUB_ENV" # roll-before-migrate: the serving roles started on a database whose framework stamp is an # older build's, then ROLE=migrate — each must wait, say so once, and serve, never crash-loop. - name: x drill pg-blackhole, worker-sigkill, web-sigterm, roll-before-migrate on examples/dummy @@ -1428,6 +1430,8 @@ jobs: bun run x -- drill "$drill" --cwd examples/dummy --json > "$doc" || status=1 jq -r '.summary, (.findings[]? | " \(.code)\n cause: \(.cause)\n fix: \(.fix)")' "$doc" \ || { echo "x drill $drill wrote no document"; status=1; } + missing="$(jq -r '.data.rolesNotPlayed[]? | "\(.role), needs \(.needs)"' "$doc" 2>/dev/null || true)" + [ -z "$missing" ] || { echo "x drill $drill left roles out: $missing"; status=1; } done exit "$status" - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 393e7cdec..390b4a87e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,23 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major ## [Unreleased] -Nothing yet. +### Added + +- **Many agents on one machine:** `x verify`, `x test`, every lane and `bun run coverage` reserve their measured memory in one flock-backed ledger per machine, first come first served, at most 4 cores locally; `x heavy status` and `x heavy run` expose it, and a new app's `CLAUDE.md` says a worker runs its own files and the coordinator runs the gate once ([Working with many agents](wiki/Many-Agents.md)). +- `setCookie(line)` in `@ultimat3/http` sets a line `@ultimat3/auth` returned, so an action can issue a session cookie. +- `MAIL_TIMEOUT_MS` sets the SMTP read/write deadline for an env-selected driver. +- `usesDefaultRedisClient`, and the `cache.tier.default-redis` boot warning. +- `x drill roll-before-migrate` plays sync and replicator when their services are set. + +### Changed + +- `lruTier` and `redisTier` refuse a bad `jitterFraction` at construction; cache limit fixes name `cacheTiers` in `runtime.ts`. +- A scaffolded agent brief runs `bunx x verify --only typecheck`, which queues in the machine ledger, in place of a raw `bun run typecheck`. + +### Fixed + +- The replicator no longer crash-loops on an app that declares a view entity. +- `x upgrade` runs `bun update` for a transitive package `bun.lock` holds under an advisory floor, so an upgraded app no longer keeps the vulnerable version until someone acts on `x doctor`'s warning. ## Releases diff --git a/CLAUDE.md b/CLAUDE.md index 0ae32c2d3..4ac0d1f6f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,7 +67,7 @@ Run everything from the repo root. | install | `bun install` | | fresh clone to running | `bun run setup` — idempotent; safe to re-run after every pull | | **the gate** | `bun run verify` — `x verify` at the repo root, 21 steps: typecheck, lint, boundaries, filesize, package-shape, errors, unit, contract, live, job, exercise, e2e, eval, drift, contract-diff, budgets, seo, i18n, policy, manifest, roadmap. Green = shippable; list: `VERIFY_STEP_NAMES`, `packages/cli/src/verify-step.ts` | -| typecheck | `bun run typecheck` · shared: `bun run typecheck:locked` · start over: `bun run typecheck:clean` | +| typecheck | `bun run typecheck` (queues; one compiler per checkout) · start over: `bun run typecheck:clean` | | lint | `bun run lint` · fix: `bun run lint:fix` | | test (all) | `bun run test` — every framework suite, opt-in ones included | | test (one file) | `bun test ./packages/core/src/errors.test.ts` — **always `./`**: bare `packages/<pkg>` is a substring filter, also running `examples/dummy/packages/<pkg>` under the wrong config | @@ -183,8 +183,6 @@ a list, never a count or ordinal: `PRIMITIVE_FACTORIES` in the same file, held c | which rung of the scale ladder a claim belongs to | [`docs/idea/17-scale-ladder.md`](docs/idea/17-scale-ladder.md) | | every error code | [`wiki/Error-Codes.md`](wiki/Error-Codes.md) | | every CLI flag | [`wiki/CLI-Reference.md`](wiki/CLI-Reference.md) | -| where to start reading | `packages/action/`, the load-bearing abstraction | -| what idiomatic usage looks like | [`examples/dummy/`](examples/dummy/README.md) | | tables, with no ORM | `entity()` declares; `postgresDriver()` (`packages/entity/src/pg-driver.ts`) is hand-written SQL | | realtime capacity, as measured | [`scripts/bench/results/README.md`](scripts/bench/results/README.md) | | the deploy-proof milestone | [`docs/history/milestone-11.md`](docs/history/milestone-11.md) · [`docs/idea/14-roadmap.md`](docs/idea/14-roadmap.md) | @@ -211,7 +209,7 @@ Free runners only (public repo): `ubuntu-latest`, `windows-latest`, `macos-lates Act as an independent, proactive senior engineer: do the full job. Don't ask what the code, the docs or a sensible default answers: decide, then state the decision. Finish the -work: fix, test, document, run the gate. Don't hand back a half-done step. Find the root cause; fix +work: fix, test, document, run the gate. Find the root cause; fix what you find along the way. Ask only before an irreversible or outward-facing act the owner hasn't already asked for. @@ -220,6 +218,8 @@ hasn't already asked for. No git worktrees in this checkout: a team of subagents shares it, on disjoint file sets. (Another repo the owner may be editing on a non-`main` checkout: work in a git worktree there.) +**Many agents, one machine:** a worker runs its own files, the coordinator runs the gate once; heavy runs queue in one memory ledger (`x heavy status`, [guide](wiki/Many-Agents.md)). + **Only the top-level agent spawns subagents.** A subagent works and reports back; it never delegates ([why](docs/history/feature-workflow.md)). Too large a scope is said in the report; widening the split is the top-level agent's call. diff --git a/PUBLISHING.md b/PUBLISHING.md index 0f169ba38..cb2e2192f 100644 --- a/PUBLISHING.md +++ b/PUBLISHING.md @@ -492,8 +492,9 @@ up (`X_MAINT_LANE_EXPIRED`). ### When a release is due -The maintainer bot judges it weekly (`.dz/briefs/release-due.md`), reading what would ship; a merge -alone never cuts a release, and a maintainer can still cut one by hand. A fix for a security advisory +The maintainer bot judges it weekly (`.dz/briefs/release-due.md`), reading what would ship, and when +one is needed that same run opens the `chore: release X.Y.Z` PR; a merge alone never cuts a release, +and a maintainer can still cut one by hand. A fix for a security advisory a first-tier source confirms is the one off-cycle release (`.dz/briefs/security-release.md`). `bun run scripts/release-due.ts --json` gathers the facts for that judgement, fetching `origin/main`'s tip when the checkout lacks it. The window is every commit since the highest `v*` tag HEAD contains, diff --git a/docs/architecture/guards.md b/docs/architecture/guards.md index 15ef45bc1..2c1afc947 100644 --- a/docs/architecture/guards.md +++ b/docs/architecture/guards.md @@ -70,6 +70,7 @@ The codes are in | `bun run scripts/generator-counts.ts` | Enforce, as a gate step, that every file count a doc states about a generator is a number that generator still prints. | `X_DOC_FILE_COUNT_STALE` `X_DOC_FILE_COUNT_UNSCANNED` | | `bun run guards-doc` | Generate `docs/architecture/guards.md` from the guards' own headers — command (in the form that reports, never one that writes), what it refuses, the codes it emits — and refuse a committed page that has drifted from them. | `X_GUARDS_DOC_DRIFT` `X_GUARDS_DOC_UNSCANNED` | | `bun run scripts/health.ts` | Answer whether the project is healthy: one verdict per row of `.dz/briefs/health.md`, read from git, `gh`, npm and the repo's own scripts, and refuse with `cannot tell` and no data when any row's command cannot be read — answering over a row it never read is the silent green it refuses. | `X_CI_RUN_NOT_FOUND` `X_HEALTH_ROW_UNREADABLE` `X_MAINT_LANE_ABSENT` `X_MAINT_LANE_EXPIRED` `X_TRUST_PUBLISHER_MISSING` | +| `bun run scripts/heavy-peak.ts` | The recipe behind every figure in `packages/cli/src/heavy-weights.ts`: run one command and report the peak RSS of its whole process tree, sampled from `/proc`. | `X_CLI_BAD_FLAG` | | `bun run scripts/i18n-catalog.ts` | Enforce, as a gate step, that `packages/i18n/src/catalogs/en.json` still describes the framework packages' own strings. | `X_CATALOG_KEY_UNREACHABLE` `X_CATALOG_MISSING_KEYS` | | `bun run scripts/image-contract.ts` | Enforce, as a gate step, the two properties that make `docker/Dockerfile` produce an image that can start. | `X_CORPUS_UNSCANNED` `X_IMAGE_CONTENT_UNIGNORED` `X_IMAGE_GUARD_MISSING` `X_IMAGE_LIBC_MISMATCH` `X_IMAGE_SECRET_UNIGNORED` | | `bun run index-of-order` | Refuse an ordering assertion built on `indexOf`/`findIndex` with nothing asserting the needle is PRESENT — the shape of a test that cannot fail. | `X_INDEX_ORDER_PIN_STALE` `X_INDEX_ORDER_PIN_UNEXPLAINED` `X_INDEX_ORDER_UNGUARDED` `X_INDEX_ORDER_UNSCANNED` | diff --git a/docs/ops/06-runbooks.md b/docs/ops/06-runbooks.md index 9e0d1c929..15e5399ef 100644 --- a/docs/ops/06-runbooks.md +++ b/docs/ops/06-runbooks.md @@ -163,7 +163,7 @@ deadline (at least 1 s), as `X_DB_UNAVAILABLE` if none answered — and the plat | no migrate ran: an applier that drops Helm hooks, a compose `up` without the `run --rm migrate` step, a PaaS with no release phase | run it once against the same database: `x db migrate --json`, or the image with `ROLE=migrate`. Then wire the order: [deploy contract](./07-deploy-contract.md#order-migrate-before-roles) | | the cause names a newer major as the newest applied build | refused at once, no wait: an old-major role over a database a newer release migrated. Roll forward to that release (`x deploy --image <newer> --json`), never migrate the old one over it | | a migrate Job that is merely slow, past the wait | raise `ULTIMATE_SCHEMA_WAIT_MS` on the serving roles; `0` turns the wait off and refuses at once | -| rehearsing it | `x drill roll-before-migrate --json` plays this failure on a scratch database for `web`, `worker` and `scheduler` ([deploy contract](./07-deploy-contract.md#failures-their-alert-and-their-runbook)) | +| rehearsing it | `x drill roll-before-migrate --json` plays this failure on a scratch database for every role the run's services can start ([deploy contract](./07-deploy-contract.md#failures-their-alert-and-their-runbook)) | | `ultimate framework major skew` in the logs, no refusal | a fleet across a major mid-rollout or mid-rollback; it serves. Finish the rollout: `kubectl rollout status deployment --selector app.kubernetes.io/component=web` | ## A credential leaked diff --git a/docs/ops/07-deploy-contract.md b/docs/ops/07-deploy-contract.md index 631dd8849..84f702563 100644 --- a/docs/ops/07-deploy-contract.md +++ b/docs/ops/07-deploy-contract.md @@ -134,9 +134,17 @@ first. | the replicator stops consuming its slot | WAL grows on the database | [`UltimateReplicationSlotRetention`](./06-runbooks.md#ultimatereplicationslotretention); the volume, [`UltimateVolumeNearFull`](./06-runbooks.md#ultimatevolumenearfull) | [the replication slot is accumulating WAL](./06-runbooks.md#the-replication-slot-is-accumulating-wal) | | open tabs 404 their assets after a deploy | `X_ASSET_RETAIN_UNDECLARED` at web boot | none | [Deployment](../../wiki/Deployment.md#hashed-assets-across-a-deploy--assetsretain) | -Rehearse the order before it is needed: `x drill roll-before-migrate --json` starts `web`, -`worker` and `scheduler` on a scratch database before its migrate, then runs migrate, and holds -`roles-wait`, `serves-after-migrate` and `one-wait-line`. Per role, the evidence is +Rehearse the order before it is needed: `x drill roll-before-migrate --json` starts every role on a +scratch database before its migrate, then runs migrate, and holds `roles-wait`, +`serves-after-migrate` and `one-wait-line`. Per role, the evidence is `.data.drills[].evidence.rollout.roles[]`: `{ role, exit, ready, waitLines, logTail }`; a property -that did not hold is `X_DRILL_FAILED`. `sync` and the replicator are not played — they need a bus -or a logical-replication Postgres. `x drill --list --json` names every drill. +that did not hold is `X_DRILL_FAILED`. + +| Role | Played when | +|---|---| +| `web`, `worker`, `scheduler` | always: `TEST_DATABASE_URL` or `TEST_REPLICATION_URL` | +| `sync` | `TEST_NATS_URL` is set | +| `replicator` | `TEST_REPLICATION_URL` names a `wal_level=logical` server, and the app has an entity on a table | + +A role left out is listed in `.data.rolesNotPlayed[]` as `{ drill, role, needs }`; the `fault-drills` +CI job is red on any. `x drill --list --json` names every drill. `As of 2026-11`. diff --git a/examples/dummy/apps/web/app/auth/actions/passkey-sign-in.ts b/examples/dummy/apps/web/app/auth/actions/passkey-sign-in.ts index 944bb39cf..12b940cc7 100644 --- a/examples/dummy/apps/web/app/auth/actions/passkey-sign-in.ts +++ b/examples/dummy/apps/web/app/auth/actions/passkey-sign-in.ts @@ -5,18 +5,8 @@ import { action, t } from '@ultimat3/action'; import { setCookie, useRequestHeader } from '@ultimat3/http'; import { allow } from '@ultimat3/policy'; -import { auth } from '../login'; import { passkeys } from '../passkeys'; -/** - * The session cookie, through http's one writer: the same name, value and `Max-Age` as the - * `cookie` line `login()` answers (`sessionCookie`), so no attribute is spelled here. - */ -const setSessionCookie = (token: string): void => { - const { policy } = auth.sessions; - setCookie(policy.cookieName, token, { maxAge: Math.floor(policy.absoluteTtlMs / 1000) }); -}; - /** The response headers the pipeline publishes, read structurally: a job or a test has none. */ const carriesHeaders = (ctx: unknown): ctx is { readonly headers: Headers } => typeof ctx === 'object' && ctx !== null && 'headers' in ctx && ctx.headers instanceof Headers; @@ -33,7 +23,7 @@ export const passkeySignIn = action({ response: input.response, userAgent: http ? useRequestHeader('user-agent') : null, }); - if (http) setSessionCookie(signedIn.token); + if (http) setCookie(signedIn.cookie); return { userId: signedIn.actor.id }; }, }); diff --git a/framework.manifest.json b/framework.manifest.json index 1dc13615f..3c0fff877 100644 --- a/framework.manifest.json +++ b/framework.manifest.json @@ -1,6 +1,6 @@ { "version": 1, - "buildId": "d76d3f3dbc355e2b5e0d9bfba52d692729411bfae29c77561fccd09b0c7f30ec", + "buildId": "5248ed420bc65e1a777442f8c4ec60569c994b44c3354f91ff1125aa5e737fe1", "tiers": { "0": [ "core", @@ -2719,6 +2719,21 @@ "owner": "scripts", "at": "scripts/health.ts" }, + { + "code": "X_HEAVY_COMMAND_NOT_FOUND", + "owner": "cli", + "at": "packages/cli/src/error-codes-tooling.ts" + }, + { + "code": "X_HEAVY_LEDGER_UNAVAILABLE", + "owner": "cli", + "at": "packages/cli/src/error-codes-tooling.ts" + }, + { + "code": "X_HEAVY_MEMORY_INVALID", + "owner": "cli", + "at": "packages/cli/src/error-codes-tooling.ts" + }, { "code": "X_HELPER_COPY", "owner": "scripts", diff --git a/llms.txt b/llms.txt index 7e031fed3..ad55c32de 100644 --- a/llms.txt +++ b/llms.txt @@ -254,6 +254,7 @@ The single documentation source. Source of truth is `wiki/` in this repo, mirror - [Client navigation](https://raw.githubusercontent.com/developerz-ai/ultimate/main/wiki/Client-Navigation.md): soft navigation over server-rendered pages: one config line per surface, swap, prefetch, history, focus. - [MCP and AI](https://raw.githubusercontent.com/developerz-ai/ultimate/main/wiki/MCP-And-AI.md): the dev MCP server, every action as a tool, the `llm()` gateway, evals. - [Agents](https://raw.githubusercontent.com/developerz-ai/ultimate/main/wiki/Agents.md): `agent()` as an action factory, tools as real actions, `hive()`, `agentJob()`, and the at-least-once trap. +- [Working with many agents](https://raw.githubusercontent.com/developerz-ai/ultimate/main/wiki/Many-Agents.md): a worker runs its own files, the coordinator runs the gate once; the machine's memory ledger and core cap. - [Admin dashboard](https://raw.githubusercontent.com/developerz-ai/ultimate/main/wiki/Admin-Dashboard.md): `defineAdmin()` serving list, detail and form per entity with no host code; filters, scopes, sections, related rows, actions with `when` and `batch`, the audit log, the jobs dashboard; zero JavaScript; its MCP surface. - [Scraping](https://raw.githubusercontent.com/developerz-ai/ultimate/main/wiki/Scraping.md): `scrape()` as a job factory, the driver-blind page vocabulary, robots and host gates, per-run egress, a rented CDP browser, usage, sealed sessions, mid-run prompts. - [Auth](https://raw.githubusercontent.com/developerz-ai/ultimate/main/wiki/Auth.md): sessions, OAuth, MFA, API keys and service tokens — the output is an `Actor`. diff --git a/package.json b/package.json index 2565f37ea..c4e829ff4 100644 --- a/package.json +++ b/package.json @@ -90,10 +90,9 @@ "test": "bun run packages/cli/src/bin.ts test", "test:affected": "bun run packages/cli/src/bin.ts test --affected --base origin/main --dirty", "test:watch": "bun test --watch --isolate --path-ignore-patterns='**/dist/**' --path-ignore-patterns='**/examples/**' --path-ignore-patterns='**/dummy/**'", - "typecheck": "tsc -b --pretty", + "typecheck": "bun run scripts/typecheck-locked.ts", "typecheck:clean": "tsc -b --clean", "tsconfig-paths": "bun run scripts/tsconfig-paths.ts", - "typecheck:locked": "bun run scripts/typecheck-locked.ts", "url-pathname": "bun run scripts/url-pathname.ts", "verify": "bun run scripts/verify.ts", "workspaces:list": "bun run scripts/list-workspaces.ts", diff --git a/packages/action/src/session-cookie.e2e.test.ts b/packages/action/src/session-cookie.e2e.test.ts new file mode 100644 index 000000000..fb008f60f --- /dev/null +++ b/packages/action/src/session-cookie.e2e.test.ts @@ -0,0 +1,104 @@ +// An action that issues a session sets its cookie with `setCookie(sessionCookie(…))`. Proven over a +// real socket with the real auth package: what the browser is sent is what `verifySession` accepts +// on the way back, and the token never rides in the action's answer. +import { afterAll, describe, expect, test } from 'bun:test'; +import { + createSession, + defineAuth, + memoryAuthAdapter, + readSessionCookie, + sessionCookie, + verifySession, +} from '@ultimat3/auth'; +import { resetLifecycle } from '@ultimat3/core'; +import { defineHttpConfig, httpServer, setCookie } from '@ultimat3/http'; +import { allow } from '@ultimat3/policy'; +import { t } from '@ultimat3/schema'; +import { action } from './action'; +import { toRoute } from './http'; + +const USER_ID = 'user_operator_7'; +const auth = defineAuth({ adapter: memoryAuthAdapter() }); +const Answer = t.object({ userId: t.string }); + +const issueSession = action({ + input: t.object({}), + output: Answer, + policy: allow('issue-session'), + async handle() { + const issued = await createSession(auth.sessions, { userId: USER_ID }); + setCookie(sessionCookie(issued.token, auth.sessions.policy)); + setCookie('theme', 'dark'); + return { userId: issued.session.userId }; + }, +}).named('issueSession'); + +const smuggleHeader = action({ + input: t.object({}), + output: Answer, + policy: allow('smuggle-header'), + handle() { + setCookie('sid=v; Path=/; HttpOnly; Secure; SameSite=Lax\r\nX-Evil: 1'); + return { userId: USER_ID }; + }, +}).named('smuggleHeader'); + +const issueRoute = toRoute(issueSession); +const smuggleRoute = toRoute(smuggleHeader); + +resetLifecycle(); +const handle = httpServer({ + routes: [issueRoute, smuggleRoute], + role: 'web', + config: defineHttpConfig({ + rateLimit: { scope: 'process' }, + port: 0, + hostname: '127.0.0.1', + dev: false, + }), +}).start(); + +afterAll(async () => { + await handle.stop(); + resetLifecycle(); +}); + +const post = (path: string): Promise<Response> => + fetch(`${handle.url()}${path}`, { + method: 'POST', + headers: { 'content-type': 'application/json', origin: handle.url() }, + body: '{}', + }); + +describe('an action issuing a session, over a real socket', () => { + test('the browser gets the session cookie, and it verifies on the way back', async () => { + const response = await post(issueRoute.path); + const body = await response.text(); + expect(response.status).toBe(200); + expect(JSON.parse(body)).toEqual({ userId: USER_ID }); + + const [session = '', theme = ''] = response.headers.getSetCookie(); + const { cookieName, absoluteTtlMs } = auth.sessions.policy; + expect(session.startsWith(`${cookieName}=`)).toBe(true); + expect(session).toEndWith( + `; Max-Age=${absoluteTtlMs / 1000}; Path=/; HttpOnly; Secure; SameSite=Lax`, + ); + expect(theme).toStartWith('theme=dark;'); + + const returning = new Request(handle.url(), { + headers: { cookie: session.slice(0, session.indexOf(';')) }, + }); + const token = readSessionCookie(returning, auth.sessions.policy) ?? ''; + expect((await verifySession(auth.sessions, token)).userId).toBe(USER_ID); + expect(body).not.toContain(token); + }); + + test('a line carrying a second header is refused, and neither header is sent', async () => { + const response = await post(smuggleRoute.path); + const problem = (await response.json()) as { readonly code?: string }; + expect(response.status).toBeGreaterThanOrEqual(400); + expect(problem.code).toBe('X_COOKIE_INVALID'); + expect(response.headers.getSetCookie()).toEqual([]); + expect(response.headers.get('x-evil')).toBeNull(); + }); +}); diff --git a/packages/auth/README.md b/packages/auth/README.md index 4c14ccad5..1baa2bb26 100644 --- a/packages/auth/README.md +++ b/packages/auth/README.md @@ -830,6 +830,11 @@ original ceiling, so its cookie takes the time left, not the full TTL: `sessionCookie(issued.token, policy, { maxAgeSeconds: remainingMaxAgeSeconds(issued.session, clock.now()) })` — what `updatePrivileges` already does for you. +An action sets any of these lines with `@ultimat3/http`'s `setCookie(line)`, `As of 2026-11`: +`setCookie(sessionCookie(issued.token, policy))`, or `setCookie(signedIn.cookie)` with what `login()` +answered. This package cannot call it (both are tier 2), so it answers the line and the caller sends +it. No attribute is respelled at the call site. + | Attribute | Attack it closes | |---|---| | `HttpOnly` | XSS reading `document.cookie` and exfiltrating the session | diff --git a/packages/cache/CLAUDE.md b/packages/cache/CLAUDE.md index 3cfcfb446..f94c12e41 100644 --- a/packages/cache/CLAUDE.md +++ b/packages/cache/CLAUDE.md @@ -82,7 +82,11 @@ Tier 1. Tagged caching + THE invalidation graph. included (scope `'semantic'`, `jitterFraction: 0`). `X_CACHE_TTL_INVALID`, never a resolution. - **`assertTtl` also SPREADS the lease it validated.** `rng` is injected (`LruOptions.rng`, `RedisTierOptions.rng`) — never `Math.random()` at a call site; `rng: () => 0` is the full lease. - `jitterFraction` outside `[0, 1)` is `X_CACHE_JITTER_INVALID`, refused, not clamped. + `jitterFraction` outside `[0, 1)` is `X_CACHE_JITTER_INVALID`, refused, not clamped, when the + tier is built (`assertJitterFraction`) and again at each write. +- **A rung's limits are constructor options, set through `runtime.ts`'s `cacheTiers`** (`As of + 2026-11`, #1178). The boot builds `lru` and `redis` with defaults; no `app.config.ts` key exists, + because it would be a second spelling of that replacement. Every limit `fix:` names the line. - **`cacheStack` is the production read path, and `@ultimat3/query`'s `readThrough` is its caller** — it is what makes an action's `cache.invalidates` reach a `cache:` query. - **`cacheStack` shares one in-flight `load()` per key** through `@ultimat3/core`'s diff --git a/packages/cache/README.md b/packages/cache/README.md index 1298c252d..0b3f09352 100644 --- a/packages/cache/README.md +++ b/packages/cache/README.md @@ -611,7 +611,10 @@ a job boundary the class is gone and the `code` is what survives — match on th (`CacheDriverUnavailableError`, `CachePurgeFailedError`). An app hands its own to the boot in `apps/web/runtime.ts` — `cacheTiers: [myRedisTier]` replaces the rung of that name (`cache.tiers` still decides which rungs exist; a supplied rung it does not name is logged `cache.tier.unnamed` -and unused), `purge: myPurgeDriver` replaces the `CLOUDFLARE_*`/`FASTLY_*` selection. The +and unused). It is also the one way to set a built-in rung's limits: `cacheTiers: [lruTier({ +maxBytes })]`, never an `app.config.ts` key. A `redisTier` with no `client` and no `REDIS_URL` or +`VALKEY_URL` dials the local server, and the boot logs `cache.tier.default-redis` +(`usesDefaultRedisClient`). `purge: myPurgeDriver` replaces the `CLOUDFLARE_*`/`FASTLY_*` selection. The conformance suites are `@ultimat3/testing/conformance`'s `cacheTierConformance` and `purgeDriverConformance`. diff --git a/packages/cache/src/errors.test.ts b/packages/cache/src/errors.test.ts index 31329ce3a..c306ff271 100644 --- a/packages/cache/src/errors.test.ts +++ b/packages/cache/src/errors.test.ts @@ -115,7 +115,7 @@ describe('CacheTooLargeError', () => { expect(EVERY_CODE).toContain(err.code); }); - test('the fix is the constructor option that holds the cap — app.config.ts has no such key', () => { + test('the fix is the runtime.ts rung that holds the cap — app.config.ts has no such key', () => { // `cache.lru.maxBytes` is not a config field (wiki/Configuration.md): a fix naming it sent the // reader to write a key the boot refuses with X_CONFIG_INVALID. const lru = new CacheTooLargeError({ key: 'k', bytes: 5_000, tier: 'lru', maxBytes: 1_000 }); @@ -126,8 +126,9 @@ describe('CacheTooLargeError', () => { maxBytes: 1_000, }); - expect(lru.fix).toContain('lruTier({ maxBytes: 5000 })'); - expect(redis.fix).toContain('redisTier({ maxValueBytes: 5000 })'); + expect(lru.fix).toStartWith('cacheTiers: [lruTier({ maxBytes: 5000 })]'); + expect(redis.fix).toStartWith('cacheTiers: [redisTier({ maxValueBytes: 5000 })]'); + for (const err of [lru, redis]) expect(err.fix).toContain('apps/web/runtime.ts'); for (const err of [lru, redis]) expect(err.fix).not.toContain('app.config.ts'); }); }); @@ -141,12 +142,22 @@ describe('CacheJitterInvalidError', () => { expect(err.cause).toContain('1.5'); // The option of the factory that read it — `cache.redis.jitterFraction` is no config key. expect(err.fix).toStartWith('{ jitterFraction: 0.05 }'); - expect(err.fix).toContain("redis tier's lruTier({ ... }) or redisTier({ ... })"); + expect(err.fix).toContain('cacheTiers: [redisTier({ jitterFraction: 0.05 })]'); + expect(err.fix).toContain('apps/web/runtime.ts'); + expect(err.fix).not.toContain('lruTier'); expect(err.meta).toEqual({ tier: 'redis', jitterFraction: 1.5 }); expect(EVERY_CODE).toContain(err.code); }); }); +describe('a tier name that is also an Object.prototype member', () => { + test('is named as written, never replaced by the inherited function', () => { + const err = new CacheJitterInvalidError({ tier: 'constructor', jitterFraction: 2 }); + expect(err.fix).toContain('in constructor(...)'); + expect(err.fix).not.toContain('[native code]'); + }); +}); + describe('CachePurgeFailedError', () => { test('names the driver, the status and the detail, and carries the given fix', () => { const err = new CachePurgeFailedError({ diff --git a/packages/cache/src/errors.ts b/packages/cache/src/errors.ts index 1fd69e999..d1f8ee697 100644 --- a/packages/cache/src/errors.ts +++ b/packages/cache/src/errors.ts @@ -67,11 +67,17 @@ export class CacheTagUnknownError extends UltimateError { } } +/** + * Where an app tunes a rung. The boot builds `lru` and `redis` with their defaults and takes a + * replacement from `runtime.ts`'s `cacheTiers`: the one way, so every fix below names it. Limits + * are not `app.config.ts` keys — they would be a second spelling of that same replacement. + */ +const RUNG_FILE = 'apps/web/runtime.ts'; + /** A single entry cannot fit the tier's byte budget, so caching it would evict everything. */ export class CacheTooLargeError extends UltimateError { constructor(input: { key: string; bytes: number; maxBytes: number; tier: string }) { const bytes = String(input.bytes); - // The cap is a constructor option on every tier, never an `app.config.ts` key. const raise = input.tier === 'redis' ? `redisTier({ maxValueBytes: ${bytes} })` @@ -79,7 +85,7 @@ export class CacheTooLargeError extends UltimateError { super({ code: 'X_CACHE_TOO_LARGE', cause: `entry "${input.key}" is ${input.bytes}B, over the ${input.tier} budget of ${input.maxBytes}B`, - fix: `${raise} where the tier is built, or cache a projection instead of the row`, + fix: `cacheTiers: [${raise}] // in ${RUNG_FILE}, replacing the rung the boot builds — or cache a projection instead of the row`, }); } } @@ -128,10 +134,7 @@ const limitEffect = (value: number): string => ? ', and every comparison against NaN is false, so the limit stops being enforced rather than being enforced wrongly — NaN is what Number(process.env.…) answers for an unset variable, and it is not nullish, so `??` does not catch it' : ', and it is applied as given, so the tier is bounded at a value nobody chose'; -/** - * The factory each scope's limits are constructor options of — the fix names where the corrected - * option goes. None is an `app.config.ts` key: the runtime builds every tier with defaults. - */ +/** The factory each scope's limits are constructor options of: where the corrected one goes. */ const LIMIT_OWNER: Readonly<Record<string, string>> = { lru: 'lruTier', redis: 'redisTier', @@ -141,6 +144,20 @@ const LIMIT_OWNER: Readonly<Record<string, string>> = { ladder: 'cacheStack', }; +/** + * Where a corrected tier option goes. `lru` and `redis` are the rungs a boot builds, so theirs also + * names the `cacheTiers` line: an app on the defaults has no `lruTier(...)` call of its own to edit. + */ +function optionHome(tier: string, option: string, example: string): string { + // `hasOwn`: `tier` is a caller's string, and `constructor` must not read the prototype. + const owner = Object.hasOwn(LIMIT_OWNER, tier) ? LIMIT_OWNER[tier] : tier; + const rung = + tier === 'lru' || tier === 'redis' + ? `; for the rung a boot builds, cacheTiers: [${owner}({ ${option}: ${example} })] in ${RUNG_FILE}` + : ''; + return `${owner}(...)${rung} — a constructor option, never an app.config.ts key`; +} + export class CacheLimitInvalidError extends UltimateError { constructor(input: { option: string; @@ -155,9 +172,7 @@ export class CacheLimitInvalidError extends UltimateError { */ source?: string | undefined; }) { - const where = - input.source ?? - `${LIMIT_OWNER[input.tier] ?? input.tier}(...) — a constructor option, never an app.config.ts key`; + const where = input.source ?? optionHome(input.tier, input.option, input.example); super({ code: 'X_CACHE_LIMIT_INVALID', cause: `the ${input.tier} tier was given ${input.option}=${String(input.value)}; it must be ${input.expected}${limitEffect(input.value)}`, @@ -205,9 +220,7 @@ export class CacheJitterInvalidError extends UltimateError { cause: `the ${input.tier} tier was configured with jitterFraction=${String( input.jitterFraction, )}; a jitter fraction is a finite number in [0, 1)`, - // A constructor option of the tier, never an `app.config.ts` key: the runtime builds both - // tiers with none (`runtime-cache.ts`), so the config path this used to name did not exist. - fix: `{ jitterFraction: 0.05 } // in the ${input.tier} tier's lruTier({ ... }) or redisTier({ ... }): a value in [0, 1) — 0.05 is the default, 0 disables jitter`, + fix: `{ jitterFraction: 0.05 } // a value in [0, 1) — 0.05 is the default, 0 disables jitter — in ${optionHome(input.tier, 'jitterFraction', '0.05')}`, meta: { tier: input.tier, jitterFraction: input.jitterFraction }, }); } diff --git a/packages/cache/src/index.ts b/packages/cache/src/index.ts index 03f9caaf8..4f0ac0bce 100644 --- a/packages/cache/src/index.ts +++ b/packages/cache/src/index.ts @@ -68,7 +68,7 @@ export type { FastlyPurgeOptions } from './purge-fastly'; export { fastlyPurgeDriver } from './purge-fastly'; export type { PurgeFetch } from './purge-http'; export type { RedisLike, RedisTierOptions } from './redis'; -export { namespaceFor, redisTier } from './redis'; +export { namespaceFor, redisTier, usesDefaultRedisClient } from './redis'; export type { InvalidationBus, InvalidationSubscription, diff --git a/packages/cache/src/lru.test.ts b/packages/cache/src/lru.test.ts index b94915e46..31c2e20a7 100644 --- a/packages/cache/src/lru.test.ts +++ b/packages/cache/src/lru.test.ts @@ -266,6 +266,10 @@ describe('LruCache refuses a byte budget that is not a budget', () => { expect((caught as UltimateError).cause).toContain('NaN'); expect((caught as UltimateError).fix).toStartWith('{ maxBytes: 10_000 }'); expect((caught as UltimateError).fix).toContain('lruTier(...)'); + // The rung a boot builds has no call of the app's to edit: the fix names where one goes. + expect((caught as UltimateError).fix).toContain( + 'cacheTiers: [lruTier({ maxBytes: 10_000 })] in apps/web/runtime.ts', + ); expect((caught as UltimateError).fix).not.toContain('lru.maxBytes in app.config.ts'); }); diff --git a/packages/cache/src/lru.ts b/packages/cache/src/lru.ts index 502e60090..87f35bb39 100644 --- a/packages/cache/src/lru.ts +++ b/packages/cache/src/lru.ts @@ -17,7 +17,13 @@ import type { TierInvalidation, TtlJitter, } from './tiers'; -import { assertFiniteCapacity, assertFiniteDurationMs, assertTtl, nowMs } from './tiers'; +import { + assertFiniteCapacity, + assertFiniteDurationMs, + assertJitterFraction, + assertTtl, + nowMs, +} from './tiers'; import { decodeCacheValue, encodeCacheValue } from './value-codec'; export interface LruOptions { @@ -130,6 +136,9 @@ export class LruCache { options.defaultTtlMs ?? 60_000, ); this.clock = options.clock ?? systemClock; + if (options.jitterFraction !== undefined) { + assertJitterFraction('lru', options.jitterFraction); + } this.jitter = { ...(options.jitterFraction === undefined ? {} : { jitterFraction: options.jitterFraction }), ...(options.rng === undefined ? {} : { rng: options.rng }), diff --git a/packages/cache/src/redis.test.ts b/packages/cache/src/redis.test.ts index 0f2aaa9d7..6a97559e1 100644 --- a/packages/cache/src/redis.test.ts +++ b/packages/cache/src/redis.test.ts @@ -8,7 +8,7 @@ import { describe, expect, test } from 'bun:test'; import { appVersion, frozenClock } from '@ultimat3/core'; import { lruTier } from './lru'; -import { namespaceFor, redisTier } from './redis'; +import { namespaceFor, redisTier, usesDefaultRedisClient } from './redis'; import { bucketsJoined, bucketsRead, fakeRedis, keysOf, tierFor } from './redis-fake-fixture'; import { tag } from './tags'; import { cacheStack } from './tiers'; @@ -278,3 +278,17 @@ describe('the redis lease is spread', () => { expect(client.sent.find((entry) => entry[0] === 'SET')?.[4]).toBe('300000'); }); }); + +// The boot cannot read a tier's options, and it owes an operator one line when a rung it was handed +// will dial the local default because no Redis URL is set. +describe('usesDefaultRedisClient', () => { + test('true for a tier built with no client, false for one handed its own', () => { + expect(usesDefaultRedisClient(redisTier({ maxValueBytes: 2_048 }))).toBe(true); + expect(usesDefaultRedisClient(redisTier({ client: fakeRedis() }))).toBe(false); + }); + + test('false for a tier that is not this factory, whatever its name', () => { + expect(usesDefaultRedisClient(lruTier())).toBe(false); + expect(usesDefaultRedisClient({ ...redisTier(), name: 'redis' })).toBe(false); + }); +}); diff --git a/packages/cache/src/redis.ts b/packages/cache/src/redis.ts index 1a7329f8b..3c272cbeb 100644 --- a/packages/cache/src/redis.ts +++ b/packages/cache/src/redis.ts @@ -30,7 +30,13 @@ import type { TierInvalidation, TtlJitter, } from './tiers'; -import { assertFiniteCapacity, assertFiniteDurationMs, assertTtl, nowMs } from './tiers'; +import { + assertFiniteCapacity, + assertFiniteDurationMs, + assertJitterFraction, + assertTtl, + nowMs, +} from './tiers'; import { decodeCacheValue, encodeCacheValue } from './value-codec'; /** The slice of Bun's Redis client this tier uses. Narrow on purpose: easy to fake in tests. */ @@ -144,6 +150,16 @@ function raiseSweepFailure(failures: readonly unknown[], attempted: number): nev }); } +/** The tiers `redisTier` built with no `client`: the ones core's `redisClient()` will dial for. */ +const onDefaultClient = new WeakSet<CacheTier>(); + +/** + * Whether `tier` is a `redisTier` left to core's default client, which reads `REDIS_URL`, then + * `VALKEY_URL`, then the local server. A tier's options are closed over, so the boot asks here: a + * rung it did not build still owes an operator a word when no URL is set (`runtime-cache.ts`). + */ +export const usesDefaultRedisClient = (tier: CacheTier): boolean => onDefaultClient.has(tier); + export function redisTier(options: RedisTierOptions = {}): CacheTier { const ns = namespaceFor(options.prefix ?? 'x', options.buildId); const defaultTtlMs = assertFiniteDurationMs( @@ -157,6 +173,9 @@ export function redisTier(options: RedisTierOptions = {}): CacheTier { 'maxValueBytes', options.maxValueBytes ?? DEFAULT_REDIS_MAX_VALUE_BYTES, ); + if (options.jitterFraction !== undefined) { + assertJitterFraction('redis', options.jitterFraction); + } const jitter: TtlJitter = { ...(options.jitterFraction === undefined ? {} : { jitterFraction: options.jitterFraction }), ...(options.rng === undefined ? {} : { rng: options.rng }), @@ -185,7 +204,7 @@ export function redisTier(options: RedisTierOptions = {}): CacheTier { const { valueKey, writeBucketsFor, bustBucketsFor, sweepBucketsFor } = bucketLayout(ns); - return { + const tier: CacheTier = { name: 'redis', /** @@ -315,4 +334,6 @@ export function redisTier(options: RedisTierOptions = {}): CacheTier { return { tier: 'redis', keys: stripped }; }, }; + if (options.client === undefined) onDefaultClient.add(tier); + return tier; } diff --git a/packages/cache/src/tiers.ts b/packages/cache/src/tiers.ts index 3fab1ed6d..03583e57b 100644 --- a/packages/cache/src/tiers.ts +++ b/packages/cache/src/tiers.ts @@ -103,7 +103,7 @@ export type TtlScope = TierName | 'semantic'; /** * A tier's ceiling — bytes or entries — screened where it is still nameable. The same rule as - * `assertTtl` beside it, over the other knob an `app.config.ts` carries. A SEPARATE refusal from + * `assertTtl` beside it, over the other knob a tier's constructor takes. A SEPARATE refusal from * `X_CACHE_TOO_LARGE`, which is one entry against a valid ceiling: this one is the ceiling itself, * and it needs a screen because a bad ceiling refuses nothing and evicts nothing rather than * refusing everything. `NaN > x` and `x > NaN` are both false. @@ -165,6 +165,15 @@ export function assertFiniteSimilarityFloor( }); } +/** + * A tier's TTL spread, screened when the tier is BUILT as well as at each write: found only at the + * first `set`, a bad fraction refused every write of a tier that had booted healthy. + */ +export function assertJitterFraction(tier: TtlScope, fraction: number): number { + if (Number.isFinite(fraction) && fraction >= 0 && fraction < 1) return fraction; + throw new CacheJitterInvalidError({ tier, jitterFraction: fraction }); +} + export function assertTtl( key: string, ttlMs: number, @@ -174,10 +183,10 @@ export function assertTtl( if (!Number.isFinite(ttlMs) || ttlMs <= 0) { throw new CacheTtlInvalidError({ key, ttlMs, tier }); } - const fraction = jitter.jitterFraction ?? DEFAULT_TTL_JITTER_FRACTION; - if (!Number.isFinite(fraction) || fraction < 0 || fraction >= 1) { - throw new CacheJitterInvalidError({ tier, jitterFraction: fraction }); - } + const fraction = + jitter.jitterFraction === undefined + ? DEFAULT_TTL_JITTER_FRACTION + : assertJitterFraction(tier, jitter.jitterFraction); if (fraction === 0) return ttlMs; // Clamped rather than trusted: an `rng` outside [0, 1) would EXTEND the lease past what the // caller asked for, which is a stale read no reader can explain. diff --git a/packages/cache/src/ttl-jitter.test.ts b/packages/cache/src/ttl-jitter.test.ts index da6f2c802..b68821b61 100644 --- a/packages/cache/src/ttl-jitter.test.ts +++ b/packages/cache/src/ttl-jitter.test.ts @@ -4,6 +4,8 @@ import { describe, expect, test } from 'bun:test'; import { CacheJitterInvalidError, CacheTtlInvalidError } from './errors'; +import { lruTier } from './lru'; +import { redisTier } from './redis'; import { assertTtl, DEFAULT_TTL_JITTER_FRACTION } from './tiers'; describe('assertTtl spreads the lease it validates', () => { @@ -32,6 +34,15 @@ describe('assertTtl spreads the lease it validates', () => { } }); + // At first write a bad fraction refused every `set`, and the stack reads a refused write as a + // tier failure: a boot that came up healthy and cached nothing. + test('a tier refuses a bad fraction when it is built, before any write', () => { + for (const jitterFraction of [1, 1.5, -0.1, Number.NaN]) { + expect(() => lruTier({ jitterFraction })).toThrow(CacheJitterInvalidError); + expect(() => redisTier({ jitterFraction })).toThrow(CacheJitterInvalidError); + } + }); + test('an rng outside [0, 1) can never EXTEND the lease past what the caller asked for', () => { expect(assertTtl('k', 1_000, 'lru', { rng: () => -5 })).toBe(1_000); expect(assertTtl('k', 1_000, 'lru', { rng: () => 99 })).toBe(950); diff --git a/packages/cli/src/cmd-drill.ts b/packages/cli/src/cmd-drill.ts index 0c9cda110..291c9c51a 100644 --- a/packages/cli/src/cmd-drill.ts +++ b/packages/cli/src/cmd-drill.ts @@ -11,6 +11,7 @@ import type { DrillDeclarations, DrillFleet, DrillReport, + DrillRoleNotPlayed, } from '@ultimat3/testing'; import { bunDrillSql, @@ -21,6 +22,7 @@ import { drillBudgets, drillNamed, drillRoles, + drillRolesNotPlayed, drillServices, runDrill, runRollout, @@ -38,6 +40,7 @@ import { readIntFlag } from './flag-number'; import { msg } from './messages'; import type { CommandResult, Finding, JsonValue } from './output'; import { flagBool } from './parse'; +import { replicatedRelations } from './role-replicator'; /** Read through `readIntFlag`, which refuses `5s`, `NaN` and a fraction with the flag's name. */ const BUDGET_FLAG = 'budget-ms'; @@ -177,9 +180,20 @@ export const drillCommand: CliCommand = { } const budgets = drillBudgets(drillConfigOf(config), override); const { targets, skipped } = await drillTargets(root); + // After `drillTargets`: loading the app is what fills the entity registry read here. + const publishable = replicatedRelations().length > 0; const reports: DrillReport[] = []; + const rolesNotPlayed: (DrillRoleNotPlayed & { readonly drill: string })[] = []; for (const { drill, services } of planned) { const { fault } = drill; + const found = { + bus: services.nats !== undefined, + logicalPostgres: services.logicalPostgres === true, + publishable, + }; + for (const gap of drillRolesNotPlayed(drill, found)) { + rolesNotPlayed.push({ drill: drill.name, ...gap }); + } // The rollout order: the stamp is this package's to write, so the rewind is handed in. const order = fault.kind === 'order' @@ -188,7 +202,7 @@ export const drillCommand: CliCommand = { const fleet = await startDrillFleet({ root, services, - roles: drillRoles(drill), + roles: drillRoles(drill, found), readyTimeoutMs: READY_TIMEOUT_MS, beforeMigrate: order, }); @@ -209,6 +223,9 @@ export const drillCommand: CliCommand = { lines: [ ...reports.map((report) => ` ${report.verdict.ok ? 'held' : 'FAILED'} ${report.name}`), ...notRun.map((drill) => ` not run ${drill}`), + ...rolesNotPlayed.map( + (gap) => ` not played ${gap.drill}: ${gap.role}, needs ${gap.needs}`, + ), ], findings, data: { @@ -218,6 +235,8 @@ export const drillCommand: CliCommand = { skippedWrites: skipped.map((entry) => ({ ...entry })), drills: reports.map(reportJson), notRun, + // A role the rollout order left out, with what it needs: CI refuses a non-empty list. + rolesNotPlayed: rolesNotPlayed.map((gap) => ({ ...gap })), }, }; }, diff --git a/packages/cli/src/cmd-heavy-spec.ts b/packages/cli/src/cmd-heavy-spec.ts new file mode 100644 index 000000000..67f003bb6 --- /dev/null +++ b/packages/cli/src/cmd-heavy-spec.ts @@ -0,0 +1,28 @@ +// `x heavy`'s declaration, apart from its body: the parser, `x help` and the `errors` step read it +// without loading `cmd-heavy.ts`, which `registry.ts` imports only when the command runs. + +import type { CommandSpec } from './parse'; + +export const heavySpec: CommandSpec = { + name: 'heavy', + summary: + "the machine's shared memory ledger: who is running, who is queued, and any command run inside it", + usage: 'x heavy [status] [--json] · x heavy run [--label s] [--mb n] [--json] -- <command…>', + subcommands: ['status', 'run'], + defaultSubcommand: 'status', + passthrough: true, + flags: [ + { + name: 'label', + type: 'string', + summary: 'what other waiters see this run called (default: the command)', + subcommands: ['run'], + }, + { + name: 'mb', + type: 'string', + summary: 'the memory to reserve, its measured peak (default: a whole-repo typecheck)', + subcommands: ['run'], + }, + ], +}; diff --git a/packages/cli/src/cmd-heavy.test.ts b/packages/cli/src/cmd-heavy.test.ts new file mode 100644 index 000000000..c1fc691f5 --- /dev/null +++ b/packages/cli/src/cmd-heavy.test.ts @@ -0,0 +1,170 @@ +// `x heavy` end to end, through the real bin: what a caller depends on is its exit code, its streams +// and what becomes of the slot when it is killed. + +import { afterEach, describe, expect, test } from 'bun:test'; +// why: a temp ledger per test; Bun ships no mkdtemp. +import { mkdtempSync, rmSync } from 'node:fs'; +// why: the test's scratch root. +import { tmpdir } from 'node:os'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; + +// flock(2) does not exist on Windows, where admission is off (`heavy-flock.ts`). +const NO_FLOCK = process.platform === 'win32'; +const BIN = join(import.meta.dir, 'bin.ts'); +const dirs: string[] = []; +const started: { kill(signal: 'SIGKILL'): void }[] = []; +afterEach(() => { + for (const child of started.splice(0)) child.kill('SIGKILL'); + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +function ledgerEnv(memory = '100m') { + const dir = mkdtempSync(join(tmpdir(), 'heavy-cmd-')); + dirs.push(dir); + const { CI: _ci, ULTIMATE_HEAVY_HOLD: _hold, ...rest } = Bun.env; + return { ...rest, ULTIMATE_HEAVY_DIR: dir, ULTIMATE_HEAVY_MEMORY: memory }; +} + +const x = (args: readonly string[], env: Record<string, string | undefined>) => { + const child = Bun.spawn(['bun', BIN, ...args], { + env, + stdout: 'pipe', + stderr: 'pipe', + stdin: 'ignore', + }); + started.push(child); + return child; +}; + +async function finished(child: ReturnType<typeof x>) { + const [out, err, code] = await Promise.all([ + new Response(child.stdout).text(), + new Response(child.stderr).text(), + child.exited, + ]); + return { out, err, code }; +} + +async function statusOf(env: Record<string, string | undefined>) { + const { out } = await finished(x(['heavy', 'status', '--json'], env)); + return JSON.parse(out).data as { + enabled: boolean; + capacityMb: number; + holds: { label: string }[]; + queue: { label: string }[]; + }; +} + +/** Starts a holder and returns once the shell inside it has run, so the hold is real. */ +async function holding(env: Record<string, string | undefined>, label: string) { + const child = x( + ['heavy', 'run', '--label', label, '--mb', '100', '--', 'sh', '-c', 'echo up; exec sleep 60'], + env, + ); + const reader = child.stdout.getReader(); + await reader.read(); + return child; +} + +describe.skipIf(NO_FLOCK)('x heavy run', () => { + test('passes the exit code through and keeps stdout the child’s', async () => { + const run = await finished( + x(['heavy', 'run', '--mb', '1', '--', 'sh', '-c', 'echo out; exit 7'], ledgerEnv()), + ); + expect(run.code).toBe(7); + expect(run.out).toBe('out\n'); + }, 30_000); + + test('--json leaves one document on stdout and moves the child to stderr', async () => { + const run = await finished( + x(['heavy', 'run', '--json', '--mb', '1', '--', 'sh', '-c', 'echo out'], ledgerEnv()), + ); + expect(JSON.parse(run.out).data.exitCode).toBe(0); + expect(run.err).toContain('out'); + }, 30_000); + + test('a nested run inherits the hold instead of waiting on its own parent', async () => { + const inner = `bun ${BIN} heavy run --mb 100 -- sh -c 'echo inner'`; + const run = await finished( + x(['heavy', 'run', '--mb', '100', '--', 'sh', '-c', inner], ledgerEnv()), + ); + expect(run.code).toBe(0); + expect(run.out).toContain('inner'); + }, 30_000); + + test('a waiter says who holds the memory, on stderr only, and SIGTERM frees the holder', async () => { + const env = ledgerEnv(); + const holder = await holding(env, 'first'); + const waiter = x(['heavy', 'run', '--mb', '100', '--', 'sh', '-c', 'echo second'], env); + const reader = waiter.stderr.getReader(); + const { value } = await reader.read(); + const line = new TextDecoder().decode(value); + expect(line).toContain('position 1 of 1'); + expect(line).toContain('first'); + holder.kill('SIGTERM'); + expect(await holder.exited).toBe(143); + expect(await waiter.exited).toBe(0); + expect(await new Response(waiter.stdout).text()).toBe('second\n'); + }, 30_000); + + test('kill -9 of the wrapper keeps the reservation until the workload is gone', async () => { + const env = ledgerEnv(); + const holder = x( + [ + 'heavy', + 'run', + '--label', + 'doomed', + '--mb', + '100', + '--', + 'sh', + '-c', + 'echo $$; exec sleep 60', + ], + env, + ); + const { value } = await holder.stdout.getReader().read(); + const workload = Number(new TextDecoder().decode(value).trim()); + holder.kill('SIGKILL'); + await holder.exited; + expect((await statusOf(env)).holds.map((hold) => hold.label)).toEqual(['doomed']); + process.kill(workload, 'SIGKILL'); + // Each read is a whole process, so this loop paces itself. + while ((await statusOf(env)).holds.length > 0); + }, 30_000); + + test('a command that does not exist is 127 with a coded finding', async () => { + const run = await finished( + x(['heavy', 'run', '--json', '--', 'no-such-tool-here'], ledgerEnv()), + ); + expect(run.code).toBe(127); + expect(JSON.parse(run.out).findings[0].code).toBe('X_HEAVY_COMMAND_NOT_FOUND'); + }, 30_000); + + test('no command is refused with a runnable example', async () => { + const run = await finished(x(['heavy', 'run', '--json'], ledgerEnv())); + expect(run.code).toBe(1); + expect(JSON.parse(run.out).findings[0].code).toBe('X_CLI_BAD_FLAG'); + }, 30_000); +}); + +describe.skipIf(NO_FLOCK)('x heavy status', () => { + test('reports capacity and the machine', async () => { + const data = await statusOf(ledgerEnv('2g')); + expect(data.enabled).toBe(true); + expect(data.capacityMb).toBe(2048); + }, 30_000); + + test('says why admission is off under CI', async () => { + const { out } = await finished(x(['heavy', '--json'], { ...ledgerEnv(), CI: 'true' })); + expect(JSON.parse(out).data).toMatchObject({ enabled: false, off: 'ci' }); + }, 30_000); + + test('a bad memory override is a coded failure, not a crash', async () => { + const run = await finished(x(['heavy', '--json'], { ...ledgerEnv('lots') })); + expect(run.code).toBe(1); + expect(JSON.parse(run.out).findings[0].code).toBe('X_HEAVY_MEMORY_INVALID'); + }, 30_000); +}); diff --git a/packages/cli/src/cmd-heavy.ts b/packages/cli/src/cmd-heavy.ts new file mode 100644 index 000000000..16f86a5c7 --- /dev/null +++ b/packages/cli/src/cmd-heavy.ts @@ -0,0 +1,160 @@ +// `x heavy`: the machine ledger as a command. `status` is a read that changes nothing; `run` puts any +// command inside one reservation, passes its exit code through and forwards stop signals, which is +// how a repository with no Ultimate in it shares the same queue. + +// why: Bun exposes no load average, core count or total-memory primitive. +import { availableParallelism, loadavg, totalmem } from 'node:os'; +import { heavySpec } from './cmd-heavy-spec'; +import type { CliCommand, CommandContext } from './command'; +import { MissingPositionalError } from './errors'; +import { readIntFlag } from './flag-number'; +import { describeEntry, heavyState, withHeavy } from './heavy'; +import { coreCap, runnableTasks } from './heavy-cores'; +import { HeavyCommandNotFoundError } from './heavy-errors'; +import { readLedger } from './heavy-ledger'; +import { availableMemoryBytes, usableMemoryBytes } from './heavy-memory'; +import { DEFAULT_RUN_MB } from './heavy-weights'; +import type { HandOffChild } from './local-cli-handoff'; +import { handOff } from './local-cli-handoff'; +import { msg } from './messages'; +import type { CommandResult, JsonValue } from './output'; +import { findingFrom } from './output'; +import { flagString } from './parse'; + +const gb = (bytes: number): string => (bytes / 2 ** 30).toFixed(1); +const MAX_RUN_MB = 1024 * 1024; + +async function status(ctx: CommandContext): Promise<CommandResult> { + const state = await heavyState({ ...ctx.env, ULTIMATE_HEAVY_HOLD: undefined }); + const now = Date.now(); + const view = state.on ? readLedger(state.dir, state.tryLock) : { holds: [], queue: [] }; + const cores = coreCap(ctx.env); + const coreText = cores === undefined ? String(availableParallelism()) : String(cores); + const usedMb = view.holds.reduce((sum, entry) => sum + entry.mb, 0); + const [one = 0, five = 0, fifteen = 0] = loadavg(); + const machine = { + cores: availableParallelism(), + runnable: runnableTasks(), + load: { one, five, fifteen }, + totalBytes: usableMemoryBytes(totalmem()), + availableBytes: availableMemoryBytes(), + }; + const entryJson = (entry: (typeof view.holds)[number]): JsonValue => ({ + pid: entry.pid, + mb: entry.mb, + label: entry.label, + cwd: entry.cwd, + ageMs: Math.max(0, now - entry.at), + }); + return { + ok: true, + command: 'heavy', + summary: state.on + ? msg('cli.heavy.status.summary', { + used: usedMb, + capacity: state.capacityMb, + holds: view.holds.length, + queued: view.queue.length, + cores: coreText, + }) + : msg('cli.heavy.status.off', { reason: state.detail ?? state.off, cores: coreText }), + lines: [ + msg('cli.heavy.status.machine', { + load: one.toFixed(1), + cpus: machine.cores, + runnable: machine.runnable, + free: gb(machine.availableBytes), + total: gb(machine.totalBytes), + }), + ...view.holds.map((entry) => + msg('cli.heavy.status.hold', { entry: describeEntry(entry, now) }), + ), + ...view.queue.map((entry, at) => + msg('cli.heavy.status.queued', { position: at + 1, entry: describeEntry(entry, now) }), + ), + ], + data: { + enabled: state.on, + ...(state.on + ? { dir: state.dir, capacityMb: state.capacityMb, usedMb } + : { off: state.off, ...(state.detail === undefined ? {} : { detail: state.detail }) }), + coreCap: cores ?? null, + machine, + holds: view.holds.map(entryJson), + queue: view.queue.map(entryJson), + }, + }; +} + +const seconds = (ms: number): string => `${(ms / 1000).toFixed(1)}s`; + +async function run(ctx: CommandContext): Promise<CommandResult> { + const command = ctx.args.passthrough; + const first = command[0]; + if (first === undefined || ctx.args.positionals.length > 0) { + throw new MissingPositionalError({ + command: 'heavy run', + positional: '-- command', + example: 'x heavy run --label tests -- bun test', + }); + } + // Under `--json` fd 1 is this command's one document, so the child's stdout moves to fd 2. + const stream = ctx.args.json ? undefined : ('stderr' as const); + const envelope = { command: 'heavy', ...(stream === undefined ? {} : { stream }) }; + const path = ctx.env['PATH']; + if (Bun.which(first, { cwd: ctx.cwd, ...(path === undefined ? {} : { PATH: path }) }) === null) { + const finding = findingFrom(new HeavyCommandNotFoundError(first)); + return { ...envelope, ok: false, summary: finding.cause, findings: [finding], exitCode: 127 }; + } + const mb = readIntFlag(ctx.args, { + name: 'mb', + command: 'heavy', + min: 1, + max: MAX_RUN_MB, + example: 'x heavy run --mb 4096 -- bun test', + }); + const label = flagString(ctx.args, 'label') ?? command.slice(0, 3).join(' '); + const queued = performance.now(); + let startedAt = queued; + const code = await withHeavy( + { label, cwd: ctx.cwd, env: ctx.env, unitMb: mb ?? DEFAULT_RUN_MB }, + (child) => { + startedAt = performance.now(); + return handOff( + command, + { ...ctx.env, ...child.env }, + { + spawn: (argv, env): HandOffChild => + // Untimed: the child is the command the caller asked to run, and its stop signals are + // forwarded by `handOff`. Counted under `cli`'s spawn-timeout pin with that hand-off. + Bun.spawn([...argv], { + cwd: ctx.cwd, + env, + // The reservation's lock goes with it: killing this wrapper frees nothing the + // command is still using. + stdio: ['inherit', ctx.args.json ? 2 : 'inherit', 'inherit', ...child.fds], + }), + }, + ); + }, + ); + const ranMs = performance.now() - startedAt; + const waitedMs = startedAt - queued; + return { + ...envelope, + ok: code === 0, + summary: msg('cli.heavy.run.done', { + command: label, + code, + ran: seconds(ranMs), + waited: seconds(waitedMs), + }), + exitCode: code, + data: { exitCode: code, ranMs: Math.round(ranMs), waitedMs: Math.round(waitedMs) }, + }; +} + +export const heavyCommand: CliCommand = { + spec: heavySpec, + run: (ctx) => (ctx.args.subcommand === 'run' ? run(ctx) : status(ctx)), +}; diff --git a/packages/cli/src/cmd-test.ts b/packages/cli/src/cmd-test.ts index 089a0543c..a5ceaa5f3 100644 --- a/packages/cli/src/cmd-test.ts +++ b/packages/cli/src/cmd-test.ts @@ -19,6 +19,7 @@ import { ok } from './command'; import { BadFlagError, NoTestFilesError } from './errors'; import type { Runner } from './exec'; import { readIntFlag } from './flag-number'; +import { FREE_RUN_FILES } from './heavy-weights'; import { msg } from './messages'; import type { CommandResult, JsonValue } from './output'; import type { ParsedArgs } from './parse'; @@ -245,9 +246,18 @@ export const testCommand: CliCommand = { workers, ...(isolate ? { isolate } : {}), ...(seed === undefined ? {} : { seed }), - // A default-width run leases its workers from the machine pool, so a second `x test` or - // gate on the same box runs narrower instead of doubling its memory. - ...(plan === undefined ? {} : withLease(machineLease(plan.workers, ctx.env))), + // Every run but a worker's own few files enters the machine ledger, so a second `x test` or + // gate on the same box narrows or queues instead of doubling its memory. + ...(files.length <= FREE_RUN_FILES + ? {} + : withLease( + machineLease(ctx.env, { + cwd: ctx.cwd, + label: type === undefined ? 'x test' : `x test ${type}`, + ...(type === undefined ? {} : { type }), + ...(plan === undefined ? { exact: true } : {}), + }), + )), ...(filter === undefined ? {} : { filter }), ...(type === undefined ? {} : { type }), // `kept` is the corpus the run saw. `selected`, not `discovered`: with `--affected` the sample was taken from the narrowed diff --git a/packages/cli/src/cmd-upgrade-fixture.ts b/packages/cli/src/cmd-upgrade-fixture.ts index 67e5b19a3..293b2af77 100644 --- a/packages/cli/src/cmd-upgrade-fixture.ts +++ b/packages/cli/src/cmd-upgrade-fixture.ts @@ -46,6 +46,16 @@ export const emptyFixture = (): string => owned('ultimate-upgrade-noapp-'); const GUARD = 'guards/no-raw-colors.ts'; +/** A lockfile with one transitive package under its advisory floor and one already past it. */ +export const STALE_ADVISORY_LOCK = `{ + "lockfileVersion": 1, + "packages": { + "immutable": ["immutable@5.1.3", "", {}, "sha512-x"], + "source-map-js": ["source-map-js@1.2.3", "", {}, "sha512-x"], + }, +} +`; + /** An app with pins, committed; `{ repository: false }` writes the same tree and never runs git. */ export const fixture = async ( files: Readonly<Record<string, string>> = {}, @@ -150,6 +160,8 @@ interface Script { readonly registryCode?: number; readonly engines?: string; readonly installCode?: number; + /** `bun update <advisory packages>`'s exit code. */ + readonly updateCode?: number; readonly doctor?: string; readonly exercise?: string; readonly exerciseCode?: number; @@ -221,6 +233,9 @@ export const recording = (script: Script = {}): { runner: Runner; spawned: strin const only = args[2] === '--only' ? ` --only ${args[3] ?? ''}` : ''; const label = args[0]?.endsWith('bin.ts') ? `x ${args[1]}${only}` : args.slice(0, 2).join(' '); spawned.push(label); + if (args[0] === 'update') { + return answer(command, '', script.updateCode ?? 0, 'error: registry unreachable'); + } switch (label) { case 'pm view': if ((script.registryCode ?? 0) !== 0) { diff --git a/packages/cli/src/cmd-upgrade-refusals.test.ts b/packages/cli/src/cmd-upgrade-refusals.test.ts index 6a1a6d86a..d3fb76474 100644 --- a/packages/cli/src/cmd-upgrade-refusals.test.ts +++ b/packages/cli/src/cmd-upgrade-refusals.test.ts @@ -12,6 +12,7 @@ import { recording, removeFixtures, run, + STALE_ADVISORY_LOCK, } from './cmd-upgrade-fixture'; afterEach(removeFixtures); @@ -97,6 +98,18 @@ describe('unit · x upgrade, the refusals', () => { expect(output.findings?.[0]?.fix).toContain('git restore .'); expect(spawned).not.toContain('x verify'); }); + + test('a failed advisory refresh refuses with the same undo, and no gate runs', async () => { + const dir = await fixture({ 'bun.lock': STALE_ADVISORY_LOCK }); + const { runner, spawned } = recording({ updateCode: 1 }); + const { code, output } = await run(dir, runner); + expect(code).toBe(1); + expect(output.findings?.[0]?.code).toBe('X_UPGRADE_INSTALL_FAILED'); + expect(output.findings?.[0]?.cause).toContain('bun update immutable exited 1'); + expect(output.findings?.[0]?.cause).toContain('registry unreachable'); + expect(output.findings?.[0]?.fix).toContain('git restore .'); + expect(spawned).not.toContain('x verify'); + }); }); describe('unit · x upgrade, the states with no hop', () => { diff --git a/packages/cli/src/cmd-upgrade.test.ts b/packages/cli/src/cmd-upgrade.test.ts index 46f719503..0461933cd 100644 --- a/packages/cli/src/cmd-upgrade.test.ts +++ b/packages/cli/src/cmd-upgrade.test.ts @@ -12,6 +12,7 @@ import { recording, removeFixtures, run, + STALE_ADVISORY_LOCK, VERIFY_GREEN, VERIFY_RED, } from './cmd-upgrade-fixture'; @@ -72,6 +73,15 @@ describe('unit · x upgrade, the walk', () => { expect(web.dependencies).toEqual({ '@ultimat3/core': '20.1.0', sass: '^1.0.0' }); }); + // `bun install` honours the lockfile, so a patched transitive release never arrives by itself. + test('install is followed by bun update for the transitive packages under an advisory floor', async () => { + const dir = await fixture({ 'bun.lock': STALE_ADVISORY_LOCK }); + const { runner, spawned } = recording(); + const { code } = await run(dir, runner); + expect(code).toBe(0); + expect(spawned.slice(2, 4)).toEqual(['install', 'update immutable']); + }); + // A mixed set walks its lowest pins one major per run; the workspaces already ahead stay put. test('a straggler hop moves the lowest pins and never the higher ones', async () => { const dir = await fixture({ diff --git a/packages/cli/src/cmd-upgrade.ts b/packages/cli/src/cmd-upgrade.ts index ba8f9fb5b..9e2146143 100644 --- a/packages/cli/src/cmd-upgrade.ts +++ b/packages/cli/src/cmd-upgrade.ts @@ -11,6 +11,7 @@ import { runUpgradeCodemods } from './cmd-upgrade-codemods'; import { upgradeSpec } from './cmd-upgrade-spec'; import type { CliCommand, CommandContext } from './command'; import { failed, ok } from './command'; +import { staleTransitiveFloors } from './doctor-advisory-floors'; import { supersededFindings } from './doctor-superseded'; import { BadFlagError, BunVersionError } from './errors'; import type { Runner } from './exec'; @@ -86,17 +87,29 @@ async function planWrites(input: { return { files, edited, manual }; } -/** `bun install`, or the refusal carrying the one command that undoes what was just written. */ -async function install(runner: Runner, root: string, to: string): Promise<void> { - const result = await runner([process.execPath, 'install'], { cwd: root }); +/** One `bun` step of the install, or the refusal carrying the command that undoes the pins. */ +async function installStep( + runner: Runner, + root: string, + to: string, + args: readonly string[], +): Promise<void> { + const result = await runner([process.execPath, ...args], { cwd: root }); if (result.ok) return; throw new UltimateError({ code: 'X_UPGRADE_INSTALL_FAILED', - cause: `bun install exited ${String(result.code)} after pinning ${to}: ${execOutput(result).slice(-800) || 'no output'}`, + cause: `bun ${args.join(' ')} exited ${String(result.code)} after pinning ${to}: ${execOutput(result).slice(-800) || 'no output'}`, fix: 'git restore . # undoes the pins, since the tree was clean; then fix what the cause names and re-run: x upgrade --json', }); } +/** `bun install` keeps what the lockfile resolves, so a patched transitive release is asked for. */ +async function install(runner: Runner, root: string, to: string): Promise<void> { + await installStep(runner, root, to, ['install']); + const stale = await staleTransitiveFloors(root); + if (stale.length > 0) await installStep(runner, root, to, ['update', ...stale]); +} + const stageLine = (stage: string): string => JSON.stringify({ upgrade: stage }); /** diff --git a/packages/cli/src/codemods/docs-many-agents.test.ts b/packages/cli/src/codemods/docs-many-agents.test.ts new file mode 100644 index 000000000..c1b4bc076 --- /dev/null +++ b/packages/cli/src/codemods/docs-many-agents.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, test } from 'bun:test'; +import { names } from '../templates/naming'; +import { docsFiles } from '../templates/scaffold-docs'; +import { memoryCodemodApp } from './codemod'; +import { codemod, MANY_AGENTS_ROW } from './docs-many-agents'; + +const TABLE = + '| Coverage | c | d |\n| One way | generators, not hand-rolled files: `x` | review |\n'; + +describe('docs-many-agents', () => { + test('adds the row above the One way row, once', async () => { + const app = memoryCodemodApp({ 'CLAUDE.md': TABLE }); + expect((await codemod.run(app)).changed).toEqual(['CLAUDE.md']); + expect(app.files['CLAUDE.md']).toContain(`${MANY_AGENTS_ROW}\n| One way |`); + expect((await codemod.run(app)).changed).toEqual([]); + }); + + test('leaves an app that reworded the table alone', async () => { + const app = memoryCodemodApp({ 'CLAUDE.md': '| Rules | mine |\n' }); + expect(await codemod.run(app)).toEqual({ changed: [], skipped: [] }); + }); + + test('says what a new app is scaffolded with', () => { + const written = docsFiles(names('shop')).find((file) => file.path === 'CLAUDE.md'); + expect(written?.contents).toContain(MANY_AGENTS_ROW); + }); +}); diff --git a/packages/cli/src/codemods/docs-many-agents.ts b/packages/cli/src/codemods/docs-many-agents.ts new file mode 100644 index 000000000..243fafd27 --- /dev/null +++ b/packages/cli/src/codemods/docs-many-agents.ts @@ -0,0 +1,30 @@ +// 31.48.0: CLAUDE.md states who runs what when several agents share a machine, the rule the machine +// ledger exists to back. Prose: it lands where the table stands as `x new` wrote it, and is skipped +// silently where the app reworded that row or already says it. + +import type { Codemod } from './codemod'; +import type { SnippetEdit } from './engine-edits'; +import { editSnippets } from './engine-edits'; + +const CLAUDE = 'CLAUDE.md'; +const NEXT_ROW = '| One way | generators, not hand-rolled files:'; + +/** Terse on purpose: a shaped app's CLAUDE.md sits within bytes of `AGENTS_MD_MAX_BYTES`. */ +export const MANY_AGENTS_ROW = + '| Many agents | a worker runs its own files; the coordinator runs the gate once | `bunx x heavy status` |'; + +export const DOCS_MANY_AGENTS_EDITS: readonly SnippetEdit[] = [ + { from: NEXT_ROW, to: `${MANY_AGENTS_ROW}\n${NEXT_ROW}`, optional: true }, +]; + +export const codemod: Codemod = { + id: 'docs-many-agents', + since: '31.48.0', + summary: "CLAUDE.md's table says a worker runs its own files and the coordinator runs the gate", + async run(app) { + if (((await app.read(CLAUDE)) ?? '').includes('x heavy status')) { + return { changed: [], skipped: [] }; + } + return editSnippets(app, CLAUDE, DOCS_MANY_AGENTS_EDITS); + }, +}; diff --git a/packages/cli/src/codemods/index.ts b/packages/cli/src/codemods/index.ts index 740688b28..6c83a45ca 100644 --- a/packages/cli/src/codemods/index.ts +++ b/packages/cli/src/codemods/index.ts @@ -18,6 +18,7 @@ import { codemod as docsCiParts } from './docs-ci-parts'; import { codemod as docsEnvGuard } from './docs-env-guard'; import { codemod as docsGateProse } from './docs-gate-prose'; import { codemod as docsIslandRows } from './docs-island-rows'; +import { codemod as docsManyAgents } from './docs-many-agents'; import { codemod as docsShippedGuards } from './docs-shipped-guards'; import { codemod as docsTestServices } from './docs-test-services'; import { codemod as e2eSuiteFloor } from './e2e-suite-floor'; @@ -71,4 +72,5 @@ export const CODEMODS: readonly Codemod[] = [ ciArtifactRetention, ciWeeklySchedule, e2eSuiteFloor, + docsManyAgents, ]; diff --git a/packages/cli/src/doctor-advisory-floors.test.ts b/packages/cli/src/doctor-advisory-floors.test.ts index 4e8046351..2a16ef1a4 100644 --- a/packages/cli/src/doctor-advisory-floors.test.ts +++ b/packages/cli/src/doctor-advisory-floors.test.ts @@ -7,7 +7,11 @@ import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { reportDoctor } from './cmd-doctor'; import { probe } from './cmd-doctor-fixture'; -import { ADVISORY_FLOOR_UNMET, advisoryFloorFindings } from './doctor-advisory-floors'; +import { + ADVISORY_FLOOR_UNMET, + advisoryFloorFindings, + staleTransitiveFloors, +} from './doctor-advisory-floors'; const roots: string[] = []; afterAll(async () => { @@ -35,6 +39,25 @@ async function app(lockText: string | undefined): Promise<string> { return root; } +describe('unit · the transitive packages x upgrade refreshes', () => { + test('names only a transitive package under its floor', async () => { + const root = await app( + lock({ + 'solid-js': 'solid-js@1.9.0', + immutable: 'immutable@5.1.3', + 'source-map-js': 'source-map-js@1.2.2', + }), + ); + expect(await staleTransitiveFloors(root)).toEqual(['immutable']); + }); + + test('a nested copy under its floor counts, and no lockfile names nothing', async () => { + const nested = await app(lock({ 'sass/immutable': 'immutable@5.0.0' })); + expect(await staleTransitiveFloors(nested)).toEqual(['immutable']); + expect(await staleTransitiveFloors(await app(undefined))).toEqual([]); + }); +}); + describe('unit · x doctor advisory floors', () => { test('a patched tree is quiet', async () => { const root = await app( diff --git a/packages/cli/src/doctor-advisory-floors.ts b/packages/cli/src/doctor-advisory-floors.ts index 46d5f55e0..4f70ac1ff 100644 --- a/packages/cli/src/doctor-advisory-floors.ts +++ b/packages/cli/src/doctor-advisory-floors.ts @@ -79,6 +79,35 @@ async function lockPackages(root: string): Promise<Readonly<Record<string, unkno } } +/** What `bun.lock` resolves for this row below its floor, lowest first. */ +function versionsBelow( + packages: Readonly<Record<string, unknown>>, + row: AdvisoryFloor, +): readonly string[] { + // A git, tarball or `file:` head is no release on npm's line, so no advisory range reads it: + // skipped rather than ordered, because `Bun.semver.order` throws on it and would take every + // other `x doctor` check down with this warning. + return resolvedVersions(packages, row.name) + .filter((version) => RELEASE.test(version)) + .filter((version) => Bun.semver.order(version, row.floor) < 0) + .sort((a, b) => Bun.semver.order(a, b)); +} + +/** + * The transitive packages `bun.lock` holds under a floor: `bun install` honours the lockfile, so + * `x upgrade` names these to `bun update`, and an upgraded app is not left to `x doctor`'s warning. + */ +export async function staleTransitiveFloors( + root: string, + floors: readonly AdvisoryFloor[] = ADVISORY_FLOORS, +): Promise<readonly string[]> { + const packages = await lockPackages(root); + if (packages === undefined) return []; + return floors + .filter((row) => row.via === 'transitive' && versionsBelow(packages, row).length > 0) + .map((row) => row.name); +} + /** One warning per floor the lockfile resolves under, lowest resolved version named. */ export async function advisoryFloorFindings( root: string, @@ -88,14 +117,7 @@ export async function advisoryFloorFindings( if (packages === undefined) return []; const transitive = floors.filter((f) => f.via === 'transitive').map((f) => quoteArg(f.name)); return floors.flatMap((row) => { - // A git, tarball or `file:` head is no release on npm's line, so no advisory range reads it: - // skipped rather than ordered, because `Bun.semver.order` throws on it and would take every - // other `x doctor` check down with this warning. - const below = resolvedVersions(packages, row.name) - .filter((version) => RELEASE.test(version)) - .filter((version) => Bun.semver.order(version, row.floor) < 0) - .sort((a, b) => Bun.semver.order(a, b)); - const lowest = below[0]; + const lowest = versionsBelow(packages, row)[0]; if (lowest === undefined) return []; return [ { diff --git a/packages/cli/src/error-codes-tooling.ts b/packages/cli/src/error-codes-tooling.ts index 5d6cdf447..d2e91ce76 100644 --- a/packages/cli/src/error-codes-tooling.ts +++ b/packages/cli/src/error-codes-tooling.ts @@ -1,8 +1,8 @@ -// The X_* codes of the app tooling commands — `x client`, `x graph` / `x impact`, `x repro` and -// `x explain` — split out of `error-codes.ts` at its 500-line ceiling, as `error-codes-shot.ts` +// The X_* codes of the tooling commands — `x client`, `x graph` / `x impact`, `x repro`, `x explain` +// and `x heavy` — split out of `error-codes.ts` at its 500-line ceiling, as `error-codes-shot.ts` // was. That file folds these into the CLI's one owned union and registers them in its one call. -/** Codes `x client`, `x graph`, `x impact`, `x repro` and `x explain` declare and own. */ +/** Codes `x client`, `x graph`, `x impact`, `x repro`, `x explain` and `x heavy` own. */ export const CLI_TOOLING_ERROR_CODES = [ 'X_CLIENT_SPEC_UNSUPPORTED', 'X_CLIENT_STALE', @@ -13,6 +13,9 @@ export const CLI_TOOLING_ERROR_CODES = [ 'X_REPRO_NOT_REPRODUCED', 'X_REPRO_CAPTURE_OFF', 'X_EXPLAIN_PRODUCTION_DATABASE', + 'X_HEAVY_LEDGER_UNAVAILABLE', + 'X_HEAVY_MEMORY_INVALID', + 'X_HEAVY_COMMAND_NOT_FOUND', ] as const; export type CliToolingErrorCode = (typeof CLI_TOOLING_ERROR_CODES)[number]; @@ -28,4 +31,7 @@ export const CLI_TOOLING_ERROR_TITLES: Readonly<Record<CliToolingErrorCode, stri X_REPRO_NOT_REPRODUCED: 'a written repro test did not fail the way the original did', X_REPRO_CAPTURE_OFF: 'x repro was given a request id while failure capture is off', X_EXPLAIN_PRODUCTION_DATABASE: 'x explain refused to run on a production database', + X_HEAVY_LEDGER_UNAVAILABLE: 'the machine memory ledger cannot be used here', + X_HEAVY_MEMORY_INVALID: 'ULTIMATE_HEAVY_MEMORY is not a size', + X_HEAVY_COMMAND_NOT_FOUND: 'x heavy run was given a command that is not on PATH', }; diff --git a/packages/cli/src/exec.ts b/packages/cli/src/exec.ts index f01431f51..6de5f500d 100644 --- a/packages/cli/src/exec.ts +++ b/packages/cli/src/exec.ts @@ -32,6 +32,11 @@ export interface ExecOptions { */ readonly env?: Readonly<Record<string, string | undefined>>; readonly stdin?: string; + /** + * Descriptors the child inherits, from fd 3 up: the machine ledger's locks (`heavy-ledger.ts`), + * which must outlive this process when it is killed and the child is not. + */ + readonly inheritFds?: readonly number[]; /** * Told the child's pid the moment it exists. A gate step's deadline records it here * (`verify-deadline.ts`): on Windows no process's environment is readable, so the tag sweep @@ -91,9 +96,12 @@ function spawnOrRefuse(command: readonly string[], options: ExecOptions) { return Bun.spawn([head, ...rest], { cwd: options.cwd, env: options.env === undefined ? Bun.env : mergedEnv(options.env), - stdin: options.stdin === undefined ? 'ignore' : new TextEncoder().encode(options.stdin), - stdout: 'pipe', - stderr: 'pipe', + stdio: [ + options.stdin === undefined ? 'ignore' : new TextEncoder().encode(options.stdin), + 'pipe', + 'pipe', + ...(options.inheritFds ?? []), + ], }); } catch (error) { throw new UltimateError({ diff --git a/packages/cli/src/heavy-cores.ts b/packages/cli/src/heavy-cores.ts new file mode 100644 index 000000000..d32d149d1 --- /dev/null +++ b/packages/cli/src/heavy-cores.ts @@ -0,0 +1,51 @@ +// How many cores a local run may take: every one in CI, at most `LOCAL_CORE_CAP` on a shared box, and +// fewer while other sessions' runnable tasks already fill the machine. Oversubscribed cores do not +// finish work sooner; they turn timing-sensitive tests red. + +// why: Bun ships no synchronous file read for `/proc/loadavg`. +import { readFileSync } from 'node:fs'; +// why: Bun exposes no available-parallelism primitive. +import { availableParallelism } from 'node:os'; +import type { HeavyEnv } from './heavy-env'; +import { HEAVY_CORES_ENV, isCi } from './heavy-env'; + +export const LOCAL_CORE_CAP = 4; +/** A busy machine still gets this much, or one run behind a burst never makes progress. */ +export const BUSY_CORE_FLOOR = 2; + +/** The instantaneous runnable count: `/proc/loadavg`'s `running/total`. The load average lags. */ +export function parseRunnable(loadavg: string): number | undefined { + const match = /^\S+\s+\S+\s+\S+\s+(\d+)\/\d+\s/.exec(loadavg); + return match === null ? undefined : Number(match[1]); +} + +/** Runnable tasks now, this reader excluded; 0 where the kernel does not report them. */ +export function runnableTasks(read: () => string = () => readFileSync('/proc/loadavg', 'utf8')) { + try { + return Math.max(0, (parseRunnable(read()) ?? 1) - 1); + } catch { + return 0; + } +} + +/** `undefined` means no ceiling: a CI runner, whose every core is this run's. */ +export function coreCap( + env: HeavyEnv, + cores: number = availableParallelism(), + runnable: number = runnableTasks(), +): number | undefined { + const raw = Number(env[HEAVY_CORES_ENV]); + if (Number.isInteger(raw) && raw > 0) return raw; + if (isCi(env)) return undefined; + const ceiling = Math.min(LOCAL_CORE_CAP, Math.max(1, cores)); + return Math.min(ceiling, Math.max(BUSY_CORE_FLOOR, cores - runnable)); +} + +/** Thread caps for the native tools: tsc is Go, biome is rayon. An exported value wins. */ +export function nativeToolEnv(env: HeavyEnv, cap: number | undefined): Record<string, string> { + if (cap === undefined) return {}; + return { + ...(env['GOMAXPROCS'] === undefined ? { GOMAXPROCS: String(cap) } : {}), + ...(env['RAYON_NUM_THREADS'] === undefined ? { RAYON_NUM_THREADS: String(cap) } : {}), + }; +} diff --git a/packages/cli/src/heavy-env.ts b/packages/cli/src/heavy-env.ts new file mode 100644 index 000000000..356480cf8 --- /dev/null +++ b/packages/cli/src/heavy-env.ts @@ -0,0 +1,19 @@ +// Every environment variable the machine ledger reads or sets, and the one reading of `CI`. A leaf: +// the worker planner and the error classes need these names without loading the ledger. + +export type HeavyEnv = Readonly<Record<string, string | undefined>>; + +/** The ledger's directory. Set by a test or an unusual machine; also opts a test run back in. */ +export const HEAVY_DIR_ENV = 'ULTIMATE_HEAVY_DIR'; +/** What the ledger admits in total (`8g`), replacing half of RAM. `0` turns it off. */ +export const HEAVY_MEMORY_ENV = 'ULTIMATE_HEAVY_MEMORY'; +/** The local core ceiling, replacing `LOCAL_CORE_CAP`. */ +export const HEAVY_CORES_ENV = 'ULTIMATE_HEAVY_CORES'; +/** Set on a holder's children: the reservation file they run inside. A protocol, not a knob. */ +export const HEAVY_HOLD_ENV = 'ULTIMATE_HEAVY_HOLD'; + +/** A CI runner is single-tenant: nothing to queue behind, and every core is the run's. */ +export const isCi = (env: HeavyEnv): boolean => { + const value = env['CI']; + return value !== undefined && value !== '' && value !== '0' && value !== 'false'; +}; diff --git a/packages/cli/src/heavy-errors.ts b/packages/cli/src/heavy-errors.ts new file mode 100644 index 000000000..fb0668430 --- /dev/null +++ b/packages/cli/src/heavy-errors.ts @@ -0,0 +1,36 @@ +// The failures of the machine memory ledger. Admission is a courtesy between processes, so the two +// environment failures are reported and the work goes ahead; only `x heavy run` refuses anything. + +import { UltimateError } from '@ultimat3/core'; +import { HEAVY_MEMORY_ENV } from './heavy-env'; +import { quoteArg } from './shell-quote'; + +export class HeavyLedgerUnavailableError extends UltimateError { + constructor(reason: string) { + super({ + code: 'X_HEAVY_LEDGER_UNAVAILABLE', + cause: `the machine memory ledger cannot be used: ${reason}; this run continues unadmitted`, + fix: 'x heavy status --json', + }); + } +} + +export class HeavyMemoryInvalidError extends UltimateError { + constructor(raw: string) { + super({ + code: 'X_HEAVY_MEMORY_INVALID', + cause: `${HEAVY_MEMORY_ENV}=${JSON.stringify(raw)} is not a byte size`, + fix: 'export ULTIMATE_HEAVY_MEMORY=8g # or 0 to turn the ledger off', + }); + } +} + +export class HeavyCommandNotFoundError extends UltimateError { + constructor(command: string) { + super({ + code: 'X_HEAVY_COMMAND_NOT_FOUND', + cause: `${JSON.stringify(command)} is not an executable on PATH`, + fix: `bun --print ${quoteArg(`Bun.which(${JSON.stringify(command)})`)} # null: install it or correct the name, then: x heavy run -- <command>`, + }); + } +} diff --git a/packages/cli/src/heavy-flock.ts b/packages/cli/src/heavy-flock.ts new file mode 100644 index 000000000..282b076c6 --- /dev/null +++ b/packages/cli/src/heavy-flock.ts @@ -0,0 +1,73 @@ +// flock(2) through `bun:ffi`: the one primitive the ledger stands on. The kernel drops the lock when +// the last descriptor closes, which includes the holder dying by any signal, so "the holder died" +// and "the lock is free" are one event and nothing has to detect a stale file. +// +// WINDOWS has no flock, and a Node descriptor there is not the HANDLE `LockFileEx` wants, so the +// ledger is OFF on Windows: `x heavy status` says so and the core cap still applies. An honest gap +// beats a pid file that reintroduces the stale-lock race this module exists to remove. + +import { HeavyLedgerUnavailableError } from './heavy-errors'; + +const LOCK_EX = 2; +const LOCK_NB = 4; +const EINTR = 4; + +const LIBC: Readonly<Record<string, { libs: readonly string[]; errno: string; busy: number }>> = { + // glibc first; the two musl names are Alpine's, the base of many app images. + linux: { + libs: ['libc.so.6', 'libc.musl-x86_64.so.1', 'libc.musl-aarch64.so.1'], + errno: '__errno_location', + busy: 11, + }, + darwin: { libs: ['libSystem.B.dylib'], errno: '__error', busy: 35 }, +}; + +/** True when this descriptor now holds the lock, false when another one does. */ +export type TryLock = (fd: number) => boolean; + +export type FlockBinding = + | { readonly ok: true; readonly tryLock: TryLock } + | { readonly ok: false; readonly reason: string }; + +const bound = new Map<string, Promise<FlockBinding>>(); + +async function bind(platform: string): Promise<FlockBinding> { + const spec = Object.hasOwn(LIBC, platform) ? LIBC[platform] : undefined; + if (spec === undefined) return { ok: false, reason: `flock(2) does not exist on ${platform}` }; + // Imported here so a command that never reaches the ledger never loads FFI. + const { dlopen, FFIType, read } = await import('bun:ffi'); + for (const lib of spec.libs) { + try { + const { symbols } = dlopen(lib, { + flock: { args: [FFIType.i32, FFIType.i32], returns: FFIType.i32 }, + [spec.errno]: { args: [], returns: FFIType.ptr }, + }); + const errnoAt = symbols[spec.errno] as unknown as () => Parameters<typeof read.i32>[0]; + const flock = symbols.flock as unknown as (fd: number, operation: number) => number; + return { + ok: true, + tryLock: (fd) => { + if (flock(fd, LOCK_EX | LOCK_NB) === 0) return true; + const errno = read.i32(errnoAt(), 0); + if (errno === spec.busy || errno === EINTR) return false; + // Reading this as "busy" would wait forever on a filesystem that cannot lock at all. + throw new HeavyLedgerUnavailableError( + `flock(2) failed with errno ${String(errno)}, which is not contention`, + ); + }, + }; + } catch { + // Not this libc: the next name, and "no libc" below when none loads. + } + } + return { ok: false, reason: `no libc with flock(2) loaded (${spec.libs.join(', ')})` }; +} + +/** The process's one binding, loaded on first use. */ +export const flockBinding = (platform: string = process.platform): Promise<FlockBinding> => { + const known = bound.get(platform); + if (known !== undefined) return known; + const binding = bind(platform); + bound.set(platform, binding); + return binding; +}; diff --git a/packages/cli/src/heavy-ledger.test.ts b/packages/cli/src/heavy-ledger.test.ts new file mode 100644 index 000000000..e4b3caf32 --- /dev/null +++ b/packages/cli/src/heavy-ledger.test.ts @@ -0,0 +1,244 @@ +// The ledger's guarantees are the kernel's, so these run real processes on real files: a fake flock +// would prove only the fake. Every wait is on a line a child printed, never on a guessed delay. + +import { afterEach, describe, expect, test } from 'bun:test'; +// why: temp directories and an append-only event log; Bun ships neither primitive. +import { mkdtempSync, readdirSync, readFileSync, rmSync } from 'node:fs'; +// why: the test's scratch root. +import { tmpdir } from 'node:os'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { flockBinding } from './heavy-flock'; +import { readLedger, unitsThatFit } from './heavy-ledger'; + +// flock(2) does not exist on Windows, where admission is off (`heavy-flock.ts`). +const NO_FLOCK = process.platform === 'win32'; +const FLOCK = join(import.meta.dir, 'heavy-flock.ts'); +const LEDGER = join(import.meta.dir, 'heavy-ledger.ts'); +const dirs: string[] = []; +const scratch = (): string => { + const dir = mkdtempSync(join(tmpdir(), 'heavy-ledger-')); + dirs.push(dir); + return dir; +}; +const started: { kill(signal: 'SIGKILL'): void }[] = []; +afterEach(() => { + // A failed assertion must not leave a holder waiting forever behind the run's own pipes. + for (const child of started.splice(0)) child.kill('SIGKILL'); + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +const tryLock = async () => { + const binding = await flockBinding(); + if (!binding.ok) expect.unreachable(binding.reason); + return binding.tryLock; +}; + +interface ChildPlan { + readonly dir: string; + readonly capacityMb: number; + readonly mb: number; + readonly label: string; + /** Runs once admitted, as source text; the child then prints `held` and awaits it. */ + readonly body?: string; +} + +function reserver(plan: ChildPlan) { + const source = ` + import { flockBinding } from ${JSON.stringify(FLOCK)}; + import { reserve } from ${JSON.stringify(LEDGER)}; + import { appendFileSync } from 'node:fs'; + const { tryLock } = await flockBinding(); + const log = (line) => appendFileSync(${JSON.stringify(join(plan.dir, 'events'))}, line + '\\n'); + console.log('joining'); + const held = await reserve({ dir: ${JSON.stringify(plan.dir)}, capacityMb: ${plan.capacityMb}, + unitMb: ${plan.mb}, units: 1, label: ${JSON.stringify(plan.label)}, cwd: '/w', tryLock, + pollMs: 5, onWait: () => console.log('waiting') }); + log('+ ${plan.mb} ${plan.label}'); + console.log('held'); + ${plan.body ?? 'await new Promise(() => {});'} + log('- ${plan.mb} ${plan.label}'); + held.release(); + process.exit(0);`; + const child = Bun.spawn(['bun', '-e', source], { stdout: 'pipe', stderr: 'inherit' }); + started.push(child); + const reader = child.stdout.getReader(); + let seen = ''; + const until = async (word: string): Promise<void> => { + while (!seen.includes(word)) { + const { value, done } = await reader.read(); + if (done) expect.unreachable(`the child exited before printing ${word}: ${seen}`); + seen += new TextDecoder().decode(value); + } + }; + const untilAny = async (words: readonly string[]): Promise<string> => { + for (;;) { + const found = words.find((word) => seen.includes(word)); + if (found !== undefined) return found; + const { value, done } = await reader.read(); + if (done) expect.unreachable(`the child exited before printing any of ${words.join()}`); + seen += new TextDecoder().decode(value); + } + }; + return { child, until, untilAny, output: () => seen }; +} + +const events = (dir: string): string[] => + readFileSync(join(dir, 'events'), 'utf8').trim().split('\n'); + +describe.skipIf(NO_FLOCK)('reserve', () => { + test('concurrent processes never sum past the capacity', async () => { + const dir = scratch(); + const go = join(dir, 'go'); + // Every holder stays until all twelve have either been admitted or queued, so an admission + // past the capacity cannot hide behind holders that happened to finish first. + const children = Array.from({ length: 12 }, (_, at) => + reserver({ + dir, + capacityMb: 100, + mb: 40, + label: `c${String(at)}`, + body: `while (!(await Bun.file(${JSON.stringify(go)}).exists())) await Bun.sleep(5);`, + }), + ); + await Promise.all(children.map((one) => one.untilAny(['held', 'waiting']))); + // A child that said `waiting` may be admitted a moment later, so the two holders are awaited + // rather than read off each child's first word. Each logs its admission before saying `held`. + const { promise: twoHeld, resolve } = Promise.withResolvers<void>(); + let admitted = 0; + const allHeld = children.map(async (one) => { + await one.until('held'); + admitted += 1; + if (admitted === 2) resolve(); + }); + await twoHeld; + expect(events(dir)).toHaveLength(2); + await Bun.write(go, ''); + await Promise.all([...allHeld, ...children.map((one) => one.child.exited)]); + let used = 0; + let peak = 0; + for (const line of events(dir)) { + used += line.startsWith('+') ? 40 : -40; + peak = Math.max(peak, used); + } + expect(events(dir)).toHaveLength(24); + expect(peak).toBe(80); + expect(readdirSync(join(dir, 'holds'))).toEqual([]); + }, 60_000); + + test('kill -9 of a holder frees its share at once, with its file still on disk', async () => { + const dir = scratch(); + const holder = reserver({ dir, capacityMb: 10, mb: 10, label: 'holder' }); + await holder.until('held'); + const waiter = reserver({ dir, capacityMb: 10, mb: 10, label: 'waiter', body: '' }); + await waiter.until('waiting'); + holder.child.kill('SIGKILL'); + expect(await waiter.child.exited).toBe(0); + expect(events(dir)).toEqual(['+ 10 holder', '+ 10 waiter', '- 10 waiter']); + }, 30_000); + + test('kill -9 of the waiter at the head lets the one behind it through', async () => { + const dir = scratch(); + const holder = reserver({ dir, capacityMb: 10, mb: 10, label: 'holder' }); + await holder.until('held'); + const head = reserver({ dir, capacityMb: 10, mb: 10, label: 'head' }); + await head.until('waiting'); + const next = reserver({ dir, capacityMb: 10, mb: 10, label: 'next', body: '' }); + await next.until('waiting'); + head.child.kill('SIGKILL'); + await head.child.exited; + holder.child.kill('SIGKILL'); + expect(await next.child.exited).toBe(0); + expect(events(dir)).not.toContain('+ 10 head'); + }, 30_000); + + test('first come, first served: a small request does not pass a big one', async () => { + const dir = scratch(); + const holder = reserver({ dir, capacityMb: 100, mb: 60, label: 'holder' }); + await holder.until('held'); + const big = reserver({ dir, capacityMb: 100, mb: 80, label: 'big', body: '' }); + await big.until('waiting'); + // 30 MB fits beside the holder right now; only the queue keeps it behind `big`. + const small = reserver({ dir, capacityMb: 100, mb: 30, label: 'small', body: '' }); + await small.until('waiting'); + expect(events(dir)).toEqual(['+ 60 holder']); + holder.child.kill('SIGKILL'); + await Promise.all([big.child.exited, small.child.exited]); + expect(events(dir).slice(1, 3)).toEqual(['+ 80 big', '- 80 big']); + expect(events(dir)[3]).toBe('+ 30 small'); + }, 30_000); + + test('a child of the holder does not keep its lock alive', async () => { + const dir = scratch(); + const holder = reserver({ + dir, + capacityMb: 10, + mb: 10, + label: 'holder', + body: `const kid = Bun.spawn(['sleep', '30'], { stdio: ['ignore', 'ignore', 'ignore'] }); + console.log('kid ' + kid.pid + ' .'); await new Promise(() => {});`, + }); + await holder.until(' .'); + holder.child.kill('SIGKILL'); + await holder.child.exited; + expect(readLedger(dir, await tryLock()).holds).toEqual([]); + }, 30_000); + + test('a child handed the locked descriptor keeps the reservation after kill -9', async () => { + const dir = scratch(); + const holder = reserver({ + dir, + capacityMb: 10, + mb: 10, + label: 'holder', + body: `const kid = Bun.spawn(['sleep', '30'], { + stdio: ['ignore', 'ignore', 'ignore', held.fd] }); + console.log('kid ' + kid.pid + ' .'); await new Promise(() => {});`, + }); + await holder.until(' .'); + const pid = Number(/kid (\d+) \./.exec(holder.output())?.[1]); + holder.child.kill('SIGKILL'); + await holder.child.exited; + const lock = await tryLock(); + expect(readLedger(dir, lock).holds.map((entry) => entry.label)).toEqual(['holder']); + process.kill(pid, 'SIGKILL'); + while (readLedger(dir, lock).holds.length > 0) await new Promise(setImmediate); + }, 30_000); + + test('a request over the capacity runs alone instead of being refused', async () => { + const dir = scratch(); + const huge = reserver({ dir, capacityMb: 10, mb: 50, label: 'huge', body: '' }); + expect(await huge.child.exited).toBe(0); + expect(events(dir)[0]).toBe('+ 50 huge'); + }, 30_000); + + test('the view shows a live holder and a live waiter, oldest first', async () => { + const dir = scratch(); + const holder = reserver({ dir, capacityMb: 10, mb: 10, label: 'holder' }); + await holder.until('held'); + const waiter = reserver({ dir, capacityMb: 10, mb: 5, label: 'waiter' }); + await waiter.until('waiting'); + const view = readLedger(dir, await tryLock()); + expect(view.holds.map((entry) => entry.label)).toEqual(['holder']); + expect(view.queue.map((entry) => entry.label)).toEqual(['waiter']); + holder.child.kill('SIGKILL'); + waiter.child.kill('SIGKILL'); + await Promise.all([holder.child.exited, waiter.child.exited]); + }, 30_000); +}); + +describe('unitsThatFit', () => { + const base = { capacityMb: 100, unitMb: 30, units: 3, minUnits: 1 }; + test('narrows to what is free, down to the minimum', () => { + expect(unitsThatFit({ ...base, usedMb: 0 })).toBe(3); + expect(unitsThatFit({ ...base, usedMb: 50 })).toBe(1); + expect(unitsThatFit({ ...base, usedMb: 80 })).toBe(0); + }); + test('an exact request takes all of it or waits', () => { + expect(unitsThatFit({ ...base, minUnits: 3, usedMb: 50 })).toBe(0); + }); + test('a unit over the capacity is granted once the ledger is empty', () => { + expect(unitsThatFit({ ...base, unitMb: 500, usedMb: 0 })).toBe(1); + expect(unitsThatFit({ ...base, unitMb: 500, usedMb: 1 })).toBe(0); + }); +}); diff --git a/packages/cli/src/heavy-ledger.ts b/packages/cli/src/heavy-ledger.ts new file mode 100644 index 000000000..c02dd53b8 --- /dev/null +++ b/packages/cli/src/heavy-ledger.ts @@ -0,0 +1,358 @@ +// The machine's memory ledger: reservations weighted in MB, admitted first come, first served, every +// one a file its owner holds an flock on. A file nobody holds is dead by definition, so a holder or +// waiter killed by any signal stops counting at that instant and no take-over exists to race. +// +// queue/<time>-<pid>-<id> a waiter holds/<pid>-<id> an admitted reservation +// gate the admission critical section; only the queue's head takes it +// +// A holder hands its locked descriptor to the workload, so the kernel frees a reservation only when +// the holder AND the workload are gone: killing a wrapper never frees memory its child still uses. +// Holders never wait while holding and children inherit the hold, so the ledger cannot deadlock. + +// why: flock needs real descriptors, and Bun ships no descriptor-level file API. +import { + closeSync, + ftruncateSync, + mkdirSync, + openSync, + readdirSync, + readFileSync, + renameSync, + statSync, + unlinkSync, + writeSync, +} from 'node:fs'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { HeavyLedgerUnavailableError } from './heavy-errors'; +import type { TryLock } from './heavy-flock'; + +export const LEDGER_POLL_MS = 100; +/** A temp file this old lost its creator: creating one takes microseconds. */ +const STALE_TEMP_MS = 60_000; + +export interface LedgerEntry { + readonly pid: number; + readonly mb: number; + readonly label: string; + readonly cwd: string; + readonly at: number; +} + +export interface LedgerView { + readonly holds: readonly LedgerEntry[]; + /** Oldest first: index 0 is the next admission. */ + readonly queue: readonly LedgerEntry[]; +} + +export interface Held { + readonly path: string; + /** The locked descriptor. Handed to a child, it keeps the lock for as long as the child lives. */ + readonly fd: number; + release(): void; +} + +export interface Reservation extends Held { + readonly units: number; + readonly mb: number; + readonly waitedMs: number; +} + +export interface ReserveInput { + readonly dir: string; + readonly capacityMb: number; + /** MB one unit costs: a measured peak (`heavy-weights.ts`). */ + readonly unitMb: number; + readonly units: number; + /** MB the run costs however wide it is: the parent and whatever its workers share. */ + readonly baseMb?: number; + /** Fewest units worth starting with. A run of workers narrows to what fits, down to this. */ + readonly minUnits?: number; + readonly label: string; + readonly cwd: string; + readonly tryLock: TryLock; + readonly pollMs?: number; + readonly onWait?: (wait: LedgerWait) => void; +} + +export interface LedgerWait { + readonly needMb: number; + readonly usedMb: number; + readonly capacityMb: number; + readonly position: number; + readonly queued: number; + readonly holds: readonly LedgerEntry[]; +} + +const codeOf = (error: unknown): unknown => + typeof error === 'object' && error !== null ? Reflect.get(error, 'code') : undefined; + +function unlinkQuiet(path: string): void { + try { + unlinkSync(path); + } catch (error) { + if (codeOf(error) !== 'ENOENT') throw error; + } +} + +function once(run: () => void): () => void { + let spent = false; + return () => { + if (spent) return; + spent = true; + run(); + }; +} + +export function prepareLedger(dir: string): void { + for (const sub of ['queue', 'holds', 'locks']) { + mkdirSync(join(dir, sub), { recursive: true, mode: 0o700 }); + } +} + +const parseEntry = (text: string): LedgerEntry | undefined => { + try { + const value: unknown = JSON.parse(text); + if (typeof value !== 'object' || value === null) return undefined; + const field = (name: string): unknown => Reflect.get(value, name); + const [pid, mb, at] = [field('pid'), field('mb'), field('at')]; + if (typeof pid !== 'number' || typeof mb !== 'number' || typeof at !== 'number') { + return undefined; + } + return { pid, mb, at, label: String(field('label')), cwd: String(field('cwd')) }; + } catch { + return undefined; + } +}; + +interface Live { + readonly name: string; + readonly entry: LedgerEntry; +} + +/** The entries whose owner still holds their lock, in name order. A dead one is dropped. */ +function liveEntries(dir: string, tryLock: TryLock, sweep: boolean): Live[] { + let names: string[]; + try { + names = readdirSync(dir).sort(); + } catch (error) { + if (codeOf(error) === 'ENOENT') return []; + throw error; + } + const live: Live[] = []; + for (const name of names) { + const path = join(dir, name); + let fd: number; + try { + fd = openSync(path, 'r'); + } catch (error) { + if (codeOf(error) === 'ENOENT') continue; + throw error; + } + try { + if (tryLock(fd)) { + if (sweep) unlinkQuiet(path); + continue; + } + const entry = parseEntry(readFileSync(fd, 'utf8')); + if (entry !== undefined) live.push({ name, entry }); + } finally { + closeSync(fd); + } + } + return live; +} + +export const readLedger = (dir: string, tryLock: TryLock): LedgerView => ({ + holds: liveEntries(join(dir, 'holds'), tryLock, false).map((one) => one.entry), + queue: liveEntries(join(dir, 'queue'), tryLock, false).map((one) => one.entry), +}); + +/** Whether the file at `path` is a reservation some live process still holds. */ +export function isHeld(path: string, tryLock: TryLock): boolean { + let fd: number; + try { + fd = openSync(path, 'r'); + } catch { + return false; + } + try { + return !tryLock(fd); + } finally { + closeSync(fd); + } +} + +/** Created under a temp name, locked, THEN renamed in: no live entry is ever visible unlocked. */ +function createEntry(root: string, sub: string, name: string, entry: LedgerEntry, lock: TryLock) { + const temp = join(root, `tmp-${name}`); + const fd = openSync(temp, 'wx'); + if (!lock(fd)) { + closeSync(fd); + unlinkQuiet(temp); + return undefined; + } + writeSync(fd, JSON.stringify(entry)); + const path = join(root, sub, name); + renameSync(temp, path); + // Unlink before close: the lock must outlast the file, or a sweeper could judge it dead. + return { + path, + fd, + release: once(() => { + unlinkQuiet(path); + closeSync(fd); + }), + }; +} + +function sweepTemps(root: string, now: number): void { + for (const name of readdirSync(root)) { + if (!name.startsWith('tmp-')) continue; + const path = join(root, name); + try { + if (now - statSync(path).mtimeMs > STALE_TEMP_MS) unlinkQuiet(path); + } catch (error) { + if (codeOf(error) !== 'ENOENT') throw error; + } + } +} + +/** A short exclusive lock on `path`, or nothing when another descriptor has it. */ +export function tryLockPath(path: string, label: string, tryLock: TryLock): Held | undefined { + const fd = openSync(path, 'a'); + let held = false; + try { + held = tryLock(fd); + } finally { + if (!held) closeSync(fd); + } + if (!held) return undefined; + ftruncateSync(fd, 0); + writeSync(fd, `pid ${String(process.pid)}: ${label}\n`); + return { path, fd, release: once(() => closeSync(fd)) }; +} + +/** Wait for `path`. `onWait` is told once, with whoever the file names as its holder. */ +export async function lockPath( + path: string, + label: string, + tryLock: TryLock, + onWait?: (holder: string) => void, + pollMs: number = LEDGER_POLL_MS, +): Promise<Held> { + let told = false; + for (;;) { + const held = tryLockPath(path, label, tryLock); + if (held !== undefined) return held; + if (!told) { + told = true; + onWait?.(readFileSync(path, 'utf8').trim() || 'an unnamed holder'); + } + await Bun.sleep(pollMs); + } +} + +const unique = (): string => `${String(process.pid)}-${crypto.randomUUID()}`; + +/** How many units fit now, or 0 when even `min` does not. Too big for the ledger runs alone. */ +export function unitsThatFit(input: { + readonly capacityMb: number; + readonly usedMb: number; + readonly unitMb: number; + readonly units: number; + readonly minUnits: number; + readonly baseMb?: number; +}): number { + const { usedMb, unitMb, units } = input; + const capacityMb = input.capacityMb - (input.baseMb ?? 0); + const most = Math.max(1, Math.floor(capacityMb / unitMb)); + const want = Math.min(units, most); + const min = Math.min(input.minUnits, want); + if (usedMb === 0) return want; + const fit = Math.min(want, Math.floor((capacityMb - usedMb) / unitMb)); + return fit >= min ? fit : 0; +} + +/** + * Join the queue and return once admitted. First come, first served: only the head may admit, so a + * big request is never starved by small ones slipping past it. The gate is held for one check at a + * time and never across a wait, so a head that dies leaves nothing behind. + */ +export async function reserve(input: ReserveInput): Promise<Reservation> { + const { dir, tryLock, capacityMb, unitMb, units } = input; + const baseMb = input.baseMb ?? 0; + // A NaN passes `??` and would turn the wait into a spin on the ledger directory. + const pollMs = Number.isFinite(input.pollMs) ? (input.pollMs as number) : LEDGER_POLL_MS; + prepareLedger(dir); + const started = performance.now(); + const base = { pid: process.pid, label: input.label, cwd: input.cwd }; + const name = `${String(Date.now()).padStart(15, '0')}-${unique()}`; + const ticket = createEntry( + dir, + 'queue', + name, + { ...base, mb: baseMb + units * unitMb, at: Date.now() }, + tryLock, + ); + if (ticket === undefined) throw new HeavyLedgerUnavailableError('a new queue file was locked'); + const minUnits = input.minUnits ?? 1; + let said = ''; + try { + for (;;) { + const queue = liveEntries(join(dir, 'queue'), tryLock, true); + const position = queue.findIndex((one) => one.name === name); + if (position === -1) throw new HeavyLedgerUnavailableError(`${dir} was removed mid-wait`); + let holds = liveEntries(join(dir, 'holds'), tryLock, position === 0); + if (position === 0) { + const gate = tryLockPath(join(dir, 'gate'), input.label, tryLock); + if (gate !== undefined) { + try { + sweepTemps(dir, Date.now()); + // Re-read under the gate: an admission may have landed since the read above. + holds = liveEntries(join(dir, 'holds'), tryLock, true); + const usedMb = holds.reduce((sum, one) => sum + one.entry.mb, 0); + const granted = unitsThatFit({ capacityMb, usedMb, unitMb, units, minUnits, baseMb }); + if (granted > 0) { + const mb = baseMb + granted * unitMb; + const hold = createEntry( + dir, + 'holds', + unique(), + { ...base, mb, at: Date.now() }, + tryLock, + ); + if (hold === undefined) { + throw new HeavyLedgerUnavailableError('a new reservation file was locked'); + } + return { + ...hold, + units: granted, + mb, + waitedMs: Math.round(performance.now() - started), + }; + } + } finally { + gate.release(); + } + } + } + const usedMb = holds.reduce((sum, one) => sum + one.entry.mb, 0); + const key = `${String(position)}|${holds.map((one) => one.name).join(',')}`; + if (key !== said) { + said = key; + input.onWait?.({ + needMb: Math.min(baseMb + units * unitMb, capacityMb), + usedMb, + capacityMb, + position: position + 1, + queued: queue.length, + holds: holds.map((one) => one.entry), + }); + } + await Bun.sleep(pollMs); + } + } finally { + ticket.release(); + } +} diff --git a/packages/cli/src/heavy-machine.test.ts b/packages/cli/src/heavy-machine.test.ts new file mode 100644 index 000000000..be18bc409 --- /dev/null +++ b/packages/cli/src/heavy-machine.test.ts @@ -0,0 +1,121 @@ +// The inputs admission is computed from: the cgroup bound, the capacity, the core cap. + +import { describe, expect, test } from 'bun:test'; +import { defaultLedgerDir, describeEntry } from './heavy'; +import { + BUSY_CORE_FLOOR, + coreCap, + LOCAL_CORE_CAP, + nativeToolEnv, + parseRunnable, +} from './heavy-cores'; +import { isCi } from './heavy-env'; +import { flockBinding } from './heavy-flock'; +import { + availableMemoryBytes, + cgroupLimitBytes, + ledgerCapacityMb, + usableMemoryBytes, +} from './heavy-memory'; +import { raceDeadline } from './verify-deadline'; + +const files = (table: Record<string, string>) => (path: string) => table[path]; +const GIB = 2 ** 30; + +describe('memory', () => { + test('the smallest memory.max along the chain binds, the root included', () => { + const read = files({ + '/proc/self/cgroup': '0::/a/b\n', + '/sys/fs/cgroup/memory.max': String(8 * GIB), + '/sys/fs/cgroup/a/memory.max': String(2 * GIB), + '/sys/fs/cgroup/a/b/memory.max': 'max\n', + }); + expect(cgroupLimitBytes(read)).toBe(2 * GIB); + expect(usableMemoryBytes(16 * GIB, cgroupLimitBytes(read))).toBe(2 * GIB); + }); + test('no cgroup v2 means the host figure', () => { + expect(cgroupLimitBytes(files({}))).toBeUndefined(); + expect(usableMemoryBytes(16 * GIB, undefined)).toBe(16 * GIB); + }); + test('the capacity is half, an override replaces it, and 0 is off', () => { + expect(ledgerCapacityMb({}, 32 * GIB)).toBe(16_384); + expect(ledgerCapacityMb({ ULTIMATE_HEAVY_MEMORY: '8g' }, 32 * GIB)).toBe(8192); + expect(ledgerCapacityMb({ ULTIMATE_HEAVY_MEMORY: '0' }, 32 * GIB)).toBe(0); + }); + test('a bad override is coded', () => { + expect(() => ledgerCapacityMb({ ULTIMATE_HEAVY_MEMORY: 'lots' })).toThrow( + expect.objectContaining({ code: 'X_HEAVY_MEMORY_INVALID' }), + ); + }); + test('MemAvailable is what a new process can get', () => { + expect(availableMemoryBytes(files({ '/proc/meminfo': 'MemAvailable: 1024 kB\n' }))).toBe( + 2 ** 20, + ); + }); +}); + +describe('cores', () => { + test('a quiet box gets the local cap, never more cores than exist', () => { + expect(coreCap({}, 16, 0)).toBe(LOCAL_CORE_CAP); + expect(coreCap({}, 1, 0)).toBe(1); + }); + test('other runnable tasks shrink it to the floor', () => { + expect(coreCap({}, 8, 5)).toBe(3); + expect(coreCap({}, 8, 85)).toBe(BUSY_CORE_FLOOR); + }); + test('CI has no ceiling and an override wins', () => { + expect(coreCap({ CI: 'true' }, 8, 85)).toBeUndefined(); + expect(coreCap({ ULTIMATE_HEAVY_CORES: '6' }, 8, 85)).toBe(6); + }); + test('CI is off for its false spellings', () => { + expect(isCi({ CI: 'false' })).toBe(false); + expect(isCi({ CI: '0' })).toBe(false); + expect(isCi({ CI: '1' })).toBe(true); + }); + test('the runnable count is the fourth field of loadavg', () => { + expect(parseRunnable('0.5 0.6 0.7 12/1445 999\n')).toBe(12); + expect(parseRunnable('garbled')).toBeUndefined(); + }); + test('an exported thread count is never overridden', () => { + expect(nativeToolEnv({ GOMAXPROCS: '9' }, 4)).toEqual({ RAYON_NUM_THREADS: '4' }); + expect(nativeToolEnv({}, undefined)).toEqual({}); + }); +}); + +describe('the rest of admission', () => { + test('a platform with no flock says so instead of faking a lock', async () => { + expect(await flockBinding('win32')).toEqual({ + ok: false, + reason: 'flock(2) does not exist on win32', + }); + }); + test('a holder is named by the end of its directory, with its size and age', () => { + const cwd = `/${'deep/'.repeat(30)}my-checkout`; + const entry = { pid: 1, mb: 512, label: 'x test', cwd, at: 1_000 }; + const line = describeEntry(entry, 61_000); + expect(line).toContain('my-checkout, 512 MB, 1m00s'); + expect(line.length).toBeLessThan(90); + }); + + test('the ledger path never reads TMPDIR', () => { + expect(defaultLedgerDir(1000)).toBe('/tmp/ultimate-heavy-1000'); + }); + test('time queued for the machine does not count against a step deadline', async () => { + let finish: (value: string) => void = () => undefined; + const work = new Promise<string>((resolve) => { + finish = resolve; + }); + let reads = 0; + // The deadline fires, reads an hour of queueing, re-arms, and the work then finishes. + const waited = (): number => { + reads += 1; + if (reads === 2) finish('done'); + return reads === 1 ? 0 : 3_600_000; + }; + expect(await raceDeadline(work, 1, waited)).toEqual({ timedOut: false, value: 'done' }); + }); + test('a step that runs out of its own time still times out', async () => { + const raced = await raceDeadline(new Promise(() => undefined), 10, () => 0); + expect(raced).toEqual({ timedOut: true }); + }); +}); diff --git a/packages/cli/src/heavy-memory.ts b/packages/cli/src/heavy-memory.ts new file mode 100644 index 000000000..7ad0110cc --- /dev/null +++ b/packages/cli/src/heavy-memory.ts @@ -0,0 +1,67 @@ +// What the ledger may admit: half of the memory this process can really use. `totalmem()` is the +// host's figure, so inside a capped cgroup it plans for memory the kernel will never grant; the +// binding limit is the smallest `memory.max` along this process's own cgroup-v2 ancestry. + +// why: Bun ships no synchronous file read, and `/proc` and `/sys` must be read at their real size. +import { readFileSync } from 'node:fs'; +// why: Bun exposes no total-memory or free-memory primitive. +import { freemem, totalmem } from 'node:os'; +import type { HeavyEnv } from './heavy-env'; +import { HEAVY_MEMORY_ENV } from './heavy-env'; +import { HeavyMemoryInvalidError } from './heavy-errors'; +import { parseBytes } from './test-workers'; + +/** RAM's other half stays for the agent sessions, editors and servers no ledger can see. */ +export const LEDGER_SHARE = 0.5; + +const CGROUP_ROOT = '/sys/fs/cgroup'; +const MIB = 2 ** 20; + +export type ReadFile = (path: string) => string | undefined; + +const readOrNothing: ReadFile = (path) => { + try { + return readFileSync(path, 'utf8'); + } catch { + return undefined; + } +}; + +/** The smallest numeric `memory.max` from the mount root down to this process's cgroup. */ +export function cgroupLimitBytes(read: ReadFile = readOrNothing): number | undefined { + const line = read('/proc/self/cgroup') + ?.split('\n') + .find((candidate) => candidate.startsWith('0::')); + if (line === undefined) return undefined; + const limits: number[] = []; + let dir = CGROUP_ROOT; + // The root counts: inside a container the namespace shows `/` and the limit sits on the mount. + for (const part of ['', ...line.slice(3).trim().split('/').filter(Boolean)]) { + if (part !== '') dir = `${dir}/${part}`; + const raw = read(`${dir}/memory.max`)?.trim(); + if (raw !== undefined && /^\d+$/.test(raw)) limits.push(Number(raw)); + } + return limits.length === 0 ? undefined : Math.min(...limits); +} + +/** The machine as this process may use it, bytes. */ +export const usableMemoryBytes = ( + host: number = totalmem(), + limit: number | undefined = cgroupLimitBytes(), +): number => (limit === undefined ? host : Math.min(host, limit)); + +/** MB the ledger admits; 0 means it is off. Throws `X_HEAVY_MEMORY_INVALID` on a bad override. */ +export function ledgerCapacityMb(env: HeavyEnv, usable: number = usableMemoryBytes()): number { + const raw = env[HEAVY_MEMORY_ENV]?.trim(); + if (raw === undefined || raw === '') return Math.floor((usable * LEDGER_SHARE) / MIB); + if (raw === '0') return 0; + const bytes = parseBytes(raw); + if (bytes === undefined) throw new HeavyMemoryInvalidError(raw); + return Math.max(1, Math.floor(bytes / MIB)); +} + +/** Memory a new process could get now: `MemAvailable` where Linux reports it. */ +export function availableMemoryBytes(read: ReadFile = readOrNothing): number { + const match = /^MemAvailable:\s+(\d+) kB/m.exec(read('/proc/meminfo') ?? ''); + return match === null ? freemem() : Number(match[1]) * 1024; +} diff --git a/packages/cli/src/heavy-weights.ts b/packages/cli/src/heavy-weights.ts new file mode 100644 index 000000000..678db50be --- /dev/null +++ b/packages/cli/src/heavy-weights.ts @@ -0,0 +1,47 @@ +// What each heavy command reserves: its MEASURED peak, whole process tree, rounded up to 64 MB. A +// weight is a claim the ledger's bound rests on, so each carries the command that re-reads it, run +// on a clean `git archive` snapshot through `bun run scripts/heavy-peak.ts -- <command>`. +// Measured 2026-11-18, 8 cores / 31 GB, Bun 1.4.4, TypeScript 7.0.2, GOMAXPROCS=RAYON_NUM_THREADS=4. + +import { WORKER_BYTES } from './test-workers'; + +const MIB = 2 ** 20; + +/** `tsc -b --pretty false` from cold over this repository's 32 projects: 5,973 MB, 120 s. */ +export const TYPECHECK_BUILD_PEAK_MB = 6016; + +/** `tsc -p . --pretty false --noEmit` in `examples/dummy`, one program: 763 MB, 8 s. */ +export const TYPECHECK_APP_PEAK_MB = 768; + +/** `-b` walks every project of a monorepo; `-p .` is the one program of a scaffolded app. */ +export const typecheckPeakMb = (argv: readonly string[]): number => + argv.includes('-b') ? TYPECHECK_BUILD_PEAK_MB : TYPECHECK_APP_PEAK_MB; + +/** `biome check .` at this repository's root: 2,072 MB (1,855 in `examples/dummy`). */ +export const LINT_PEAK_MB = 2112; + +/** + * One `bun test` worker: the planner's figure (`WORKER_BYTES`, with its table in `test-workers.ts`). + * Re-read here: `bun test ./packages/core ./packages/schema ./packages/entity ./packages/action`, + * one process tree, peaked at 1,229 MB. + */ +export const TEST_WORKER_PEAK_MB = Math.ceil(WORKER_BYTES / MIB); + +/** + * What a test run costs beside its workers: the `x` parent and the first database boot. One worker + * in `examples/dummy` peaked at 1,915 MB (`x test unit --workers 1`) and 1,964 MB with coverage + * (`ULTIMATE_HEAVY_CORES=1 x verify --only unit`), which is this plus one worker. + */ +export const TEST_RUN_BASE_MB = 704; + +/** + * `x test e2e` in `examples/dummy`, app server and browser: 2,871 MB. That run ended red on the + * snapshot, so the figure is a floor and this keeps a margin above it. + */ +export const E2E_LANE_PEAK_MB = 3072; + +/** At most this many selected files run free: a worker's own files never queue behind a gate. */ +export const FREE_RUN_FILES = 3; + +/** `x heavy run` with no `--mb`: an unknown command is assumed to cost a whole-repo typecheck. */ +export const DEFAULT_RUN_MB = TYPECHECK_BUILD_PEAK_MB; diff --git a/packages/cli/src/heavy.test.ts b/packages/cli/src/heavy.test.ts new file mode 100644 index 000000000..11ff1bbad --- /dev/null +++ b/packages/cli/src/heavy.test.ts @@ -0,0 +1,148 @@ +// Admission as a caller meets it: the reservation around work, the hold its children inherit, and +// one compiler per checkout, proven across real processes. + +import { afterEach, describe, expect, test } from 'bun:test'; +// why: temp ledgers and an append-only event log; Bun ships neither primitive. +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +// why: the test's scratch root. +import { tmpdir } from 'node:os'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { run } from '../../../scripts/lib/run'; +import { exec } from './exec'; +import { heavyState, withChild, withHeavy } from './heavy'; +import { HEAVY_HOLD_ENV } from './heavy-env'; + +// flock(2) does not exist on Windows, where admission is off (`heavy-flock.ts`). +const NO_FLOCK = process.platform === 'win32'; +const HEAVY = join(import.meta.dir, 'heavy.ts'); +const dirs: string[] = []; +const scratch = (): string => { + const dir = mkdtempSync(join(tmpdir(), 'heavy-')); + dirs.push(dir); + return dir; +}; +afterEach(() => { + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +const envFor = (dir: string) => ({ ULTIMATE_HEAVY_DIR: dir, ULTIMATE_HEAVY_MEMORY: '100m' }); +const request = (env: Record<string, string | undefined>) => ({ + label: 'work', + cwd: '/w', + env, + unitMb: 100, +}); + +describe.skipIf(NO_FLOCK)('heavyState', () => { + test('is off in CI and inside a test run, and an explicit ledger opts a test back in', async () => { + expect(await heavyState({ CI: 'true' })).toMatchObject({ on: false, off: 'ci' }); + expect(await heavyState({ NODE_ENV: 'test' })).toMatchObject({ on: false, off: 'test-run' }); + const on = await heavyState({ NODE_ENV: 'test', ...envFor(scratch()) }); + expect(on.on).toBe(true); + }); + + test('ULTIMATE_HEAVY_MEMORY=0 is off', async () => { + const state = await heavyState({ ...envFor(scratch()), ULTIMATE_HEAVY_MEMORY: '0' }); + expect(state).toMatchObject({ on: false, off: 'disabled' }); + }); + + test('a hold whose owner is gone does not count as nesting', async () => { + const dir = scratch(); + const env = { ...envFor(dir), [HEAVY_HOLD_ENV]: join(dir, 'holds', 'gone') }; + expect((await heavyState(env)).on).toBe(true); + }); +}); + +describe.skipIf(NO_FLOCK)('withHeavy', () => { + test('children get the hold, and a run inside it reserves nothing more', async () => { + const env = envFor(scratch()); + const inner = await withHeavy(request(env), async (child) => { + expect(child.env[HEAVY_HOLD_ENV]).toContain('/holds/'); + expect(child.fds).toHaveLength(1); + // The ledger is full now, so this returns only because the hold is honoured. + return withHeavy(request({ ...env, ...child.env }), async (nested) => nested); + }); + expect(inner.env[HEAVY_HOLD_ENV]).toBeUndefined(); + expect(inner.fds).toEqual([]); + }); + + test('the reservation is given back when the work throws', async () => { + const env = envFor(scratch()); + const boom = { code: 'boom' }; + await expect( + withHeavy(request(env), async () => { + throw boom; + }), + ).rejects.toBe(boom); + expect(await withHeavy(request(env), async () => 'next')).toBe('next'); + }); + + test('a test run spawns with its options untouched', async () => { + const options = { cwd: '/w' }; + const seen = await withHeavy(request({ NODE_ENV: 'test' }), async (child) => + withChild(options, child), + ); + expect(seen).toBe(options); + }); + + test('an unusable ledger is said once and the work still runs', async () => { + const file = join(scratch(), 'a-file'); + await Bun.write(file, ''); + const env = { ULTIMATE_HEAVY_DIR: file, ULTIMATE_HEAVY_MEMORY: '100m' }; + expect(await withHeavy(request(env), async () => 'ran')).toBe('ran'); + }); +}); + +describe.skipIf(NO_FLOCK)('a spawned child', () => { + test('receives inherited descriptors through both spawn paths', async () => { + const dir = scratch(); + await withHeavy(request(envFor(dir)), async (child) => { + const probe = ['sh', '-c', 'true <&3']; + const viaExec = await exec(probe, withChild({ cwd: dir }, child)); + const viaRun = await run(probe, { cwd: dir, inheritFds: child.fds }); + expect([viaExec.ok, viaRun.ok]).toEqual([true, true]); + expect((await exec(probe, { cwd: dir })).ok).toBe(false); + }); + }); +}); + +describe.skipIf(NO_FLOCK)('withCheckoutLock', () => { + test('two processes on one checkout never overlap, and a nested run still excludes', async () => { + const dir = scratch(); + const root = scratch(); + const log = join(dir, 'events'); + const starts = join(dir, 'starts'); + const source = (name: string) => ` + import { withCheckoutLock } from ${JSON.stringify(HEAVY)}; + import { appendFileSync, readFileSync } from 'node:fs'; + appendFileSync(${JSON.stringify(starts)}, '${name}'); + await withCheckoutLock( + { name: 'typecheck', root: ${JSON.stringify(root)}, env: Bun.env }, + { env: {}, fds: [] }, + async () => { + appendFileSync(${JSON.stringify(log)}, '+${name}\\n'); + // Every run has started before anyone leaves, so a missing lock must show as overlap. + while (readFileSync(${JSON.stringify(starts)}, 'utf8').length < 4) await Bun.sleep(5); + await Bun.sleep(20); + appendFileSync(${JSON.stringify(log)}, '-${name}\\n'); + }, + );`; + // `CI` cleared: a CI runner has no ledger, so there the lock under test would not be taken. + const env = { + ...Bun.env, + ...envFor(dir), + CI: '', + [HEAVY_HOLD_ENV]: join(dir, 'holds', 'parent'), + }; + const runs = ['a', 'b', 'c', 'd'].map((name) => + Bun.spawn(['bun', '-e', source(name)], { env, stdout: 'ignore', stderr: 'ignore' }), + ); + expect(await Promise.all(runs.map((run) => run.exited))).toEqual([0, 0, 0, 0]); + const lines = readFileSync(log, 'utf8').trim().split('\n'); + expect(lines).toHaveLength(8); + for (let at = 0; at < lines.length; at += 2) { + expect(lines[at]?.slice(1)).toBe(lines[at + 1]?.slice(1) as string); + } + }, 30_000); +}); diff --git a/packages/cli/src/heavy.ts b/packages/cli/src/heavy.ts new file mode 100644 index 000000000..65f420fc7 --- /dev/null +++ b/packages/cli/src/heavy.ts @@ -0,0 +1,290 @@ +// Admission to the machine, as the rest of the CLI sees it: whether the ledger applies to this +// process, and the three ways in. Never a reason for work to fail: a ledger that cannot run is said +// once on stderr and the command goes ahead. + +// why: Bun ships no lstat with ownership, and the fixed path below needs one. +import { lstatSync, mkdirSync, realpathSync } from 'node:fs'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { isUltimateError } from '@ultimat3/core'; +import { coreCap, nativeToolEnv } from './heavy-cores'; +import type { HeavyEnv } from './heavy-env'; +import { HEAVY_DIR_ENV, HEAVY_HOLD_ENV, isCi } from './heavy-env'; +import { HeavyLedgerUnavailableError } from './heavy-errors'; +import type { TryLock } from './heavy-flock'; +import { flockBinding } from './heavy-flock'; +import type { Held, LedgerEntry, LedgerWait, Reservation } from './heavy-ledger'; +import { isHeld, lockPath, prepareLedger, reserve } from './heavy-ledger'; +import { ledgerCapacityMb } from './heavy-memory'; +import { msg } from './messages'; +import { writeErrorLine } from './write-line'; + +export type HeavyOff = 'ci' | 'test-run' | 'disabled' | 'nested' | 'unsupported'; + +export type HeavyState = + | { + readonly on: true; + readonly dir: string; + readonly capacityMb: number; + readonly tryLock: TryLock; + } + | { readonly on: false; readonly off: HeavyOff; readonly detail?: string }; + +/** One per machine and uid. A FIXED path: `TMPDIR` differs per session, and two ledgers are no bound. */ +export const defaultLedgerDir = (uid: number | undefined = process.getuid?.()): string => + `/tmp/ultimate-heavy-${String(uid ?? 0)}`; + +/** `/tmp` is world-writable: refuse a directory someone else planted at our fixed path. */ +function requireOwnDir(dir: string): void { + mkdirSync(dir, { recursive: true, mode: 0o700 }); + const stat = lstatSync(dir); + if (!stat.isDirectory() || (process.getuid !== undefined && stat.uid !== process.getuid())) { + throw new HeavyLedgerUnavailableError(`${dir} is not a directory this user owns`); + } + prepareLedger(dir); +} + +const said = new Set<string>(); +const warnOnce = (line: string): void => { + if (said.has(line)) return; + said.add(line); + writeErrorLine(line); +}; + +const describe = (error: unknown): string => + isUltimateError(error) + ? `${error.code}: ${error.cause}\n fix: ${error.fix}` + : new HeavyLedgerUnavailableError(error instanceof Error ? error.message : 'unknown').message; + +/** Whether admission applies here. Throws only coded errors; `admission` turns them into a line. */ +export async function heavyState(env: HeavyEnv): Promise<HeavyState> { + if (isCi(env)) return { on: false, off: 'ci' }; + const explicitDir = env[HEAVY_DIR_ENV]; + // A fixture's `x test` runs inside `bun test`, whose NODE_ENV every child inherits: queueing it + // behind real work would turn a test's timeout into a machine-load verdict. + if (env['NODE_ENV'] === 'test' && explicitDir === undefined) + return { on: false, off: 'test-run' }; + const capacityMb = ledgerCapacityMb(env); + if (capacityMb === 0) return { on: false, off: 'disabled' }; + const binding = await flockBinding(); + if (!binding.ok) return { on: false, off: 'unsupported', detail: binding.reason }; + const hold = env[HEAVY_HOLD_ENV]; + if (hold !== undefined && hold !== '' && isHeld(hold, binding.tryLock)) { + return { on: false, off: 'nested' }; + } + const dir = explicitDir ?? defaultLedgerDir(); + requireOwnDir(dir); + return { on: true, dir, capacityMb, tryLock: binding.tryLock }; +} + +async function admission(env: HeavyEnv): Promise<HeavyState> { + try { + return await heavyState(env); + } catch (error) { + warnOnce(describe(error)); + return { on: false, off: 'unsupported', detail: 'see the line above' }; + } +} + +const PATH_SHOWN = 48; +const printable = (text: string): string => text.replace(/\p{Cc}/gu, ' ').slice(0, 120); + +export const ageOf = (ms: number): string => { + const seconds = Math.max(0, Math.round(ms / 1000)); + return seconds < 60 + ? `${String(seconds)}s` + : `${String(Math.floor(seconds / 60))}m${String(seconds % 60).padStart(2, '0')}s`; +}; + +/** A directory's END names the checkout; a long path keeps that and loses its start. */ +const shortPath = (path: string): string => + path.length <= PATH_SHOWN ? printable(path) : `…${printable(path.slice(-PATH_SHOWN))}`; + +export const describeEntry = (entry: LedgerEntry, now: number): string => + `${printable(entry.label)} (${shortPath(entry.cwd)}, ${String(entry.mb)} MB, ${ageOf(now - entry.at)})`; + +/** The waiter's one line. Stderr always: stdout may be a `--json` document. */ +export const waitLine = (label: string, wait: LedgerWait, now: number = Date.now()): string => + msg('cli.heavy.waiting', { + label: printable(label), + need: wait.needMb, + position: wait.position, + queued: wait.queued, + used: wait.usedMb, + capacity: wait.capacityMb, + holders: wait.holds.map((entry) => describeEntry(entry, now)).join('; ') || '-', + }); + +// Time this process has spent queued, so a gate step's deadline can stop counting it. +let finishedWaitMs = 0; +const waitStarts = new Set<number>(); + +/** Monotonic milliseconds spent waiting for admission, ongoing waits included. */ +export const heavyWaitClock = (now: number = performance.now()): number => { + let total = finishedWaitMs; + for (const start of waitStarts) total += now - start; + return total; +}; + +async function timedWait<T>(wait: () => Promise<T>): Promise<T> { + const start = performance.now(); + waitStarts.add(start); + try { + return await wait(); + } finally { + waitStarts.delete(start); + finishedWaitMs += performance.now() - start; + } +} + +export interface HeavyRequest { + readonly label: string; + readonly cwd: string; + readonly env: HeavyEnv; + readonly unitMb: number; + readonly baseMb?: number; + readonly units?: number; + readonly minUnits?: number; +} + +/** A reservation, or nothing when admission does not apply or could not work. */ +async function enter(request: HeavyRequest, state: HeavyState): Promise<Reservation | undefined> { + if (!state.on) return undefined; + try { + return await timedWait(() => + reserve({ + dir: state.dir, + capacityMb: state.capacityMb, + tryLock: state.tryLock, + unitMb: request.unitMb, + ...(request.baseMb === undefined ? {} : { baseMb: request.baseMb }), + units: request.units ?? 1, + minUnits: request.minUnits ?? request.units ?? 1, + label: request.label, + cwd: request.cwd, + onWait: (wait) => writeErrorLine(waitLine(request.label, wait)), + }), + ); + } catch (error) { + warnOnce(describe(error)); + return undefined; + } +} + +/** What a holder's child is spawned with: the hold's name, and the locks it must keep alive. */ +export interface HeavyChild { + readonly env: Readonly<Record<string, string>>; + readonly fds: readonly number[]; +} + +const childOf = (env: HeavyEnv, state: HeavyState, held: Held | undefined): HeavyChild => ({ + env: { + ...toolEnv(env, state), + ...(held === undefined ? {} : { [HEAVY_HOLD_ENV]: held.path }), + }, + fds: held === undefined ? [] : [held.fd], +}); + +/** What a native tool's child is spawned with: the thread cap, unless this run has no ceiling. */ +function toolEnv(env: HeavyEnv, state: HeavyState): Record<string, string> { + const capped = state.on || state.off === 'nested' || state.off === 'unsupported'; + return capped ? nativeToolEnv(env, coreCap(env)) : {}; +} + +/** Runner options carrying `child`; untouched when it adds nothing, as under CI and in a test. */ +export function withChild<T extends { readonly cwd: string }>( + options: T, + child: HeavyChild, +): + | T + | (T & { + readonly env: Readonly<Record<string, string>>; + readonly inheritFds: readonly number[]; + }) { + return Object.keys(child.env).length === 0 && child.fds.length === 0 + ? options + : { ...options, env: child.env, inheritFds: child.fds }; +} + +/** Run `work` inside one reservation; everything it spawns gets `child`. */ +export async function withHeavy<T>( + request: HeavyRequest, + work: (child: HeavyChild) => Promise<T>, +): Promise<T> { + const state = await admission(request.env); + const held = await enter(request, state); + try { + return await work(childOf(request.env, state, held)); + } finally { + held?.release(); + } +} + +/** `name` for one checkout, whichever path reached it, in the ledger's directory. */ +const checkoutLockPath = (dir: string, name: string, root: string): string => + join(dir, 'locks', `${name}-${Bun.hash(realpathSync(root)).toString(36)}.lock`); + +/** + * One `name` at a time in this checkout (two `tsc -b` race one `dist/`). Taken AFTER the ledger, by + * every caller: a process holding this lock must never queue for memory, or a nested run holding + * memory and waiting for the lock would close a cycle. + */ +export async function withCheckoutLock<T>( + input: { readonly name: string; readonly root: string; readonly env: HeavyEnv }, + child: HeavyChild, + work: (child: HeavyChild) => Promise<T>, +): Promise<T> { + // Nested runs still exclude each other here, so the state's `nested` is not an exemption. + const state = await admission({ ...input.env, [HEAVY_HOLD_ENV]: undefined }); + if (!state.on) return work(child); + let held: Held | undefined; + try { + const path = checkoutLockPath(state.dir, input.name, input.root); + held = await timedWait(() => + lockPath(path, `${input.name} in ${input.root}`, state.tryLock, (holder) => + writeErrorLine(msg('cli.heavy.lockWait', { name: input.name, holder: printable(holder) })), + ), + ); + } catch (error) { + warnOnce(describe(error)); + } + try { + // The compiler keeps the lock too: a killed wrapper must not let a second one start beside it. + return await work(held === undefined ? child : { ...child, fds: [...child.fds, held.fd] }); + } finally { + held?.release(); + } +} + +export interface WorkerLease { + readonly count: number; + readonly env?: Readonly<Record<string, string>>; + /** Locked descriptors each worker process must inherit. */ + readonly fds?: readonly number[]; + release(): void; +} + +/** + * Up to `want` test workers, as many as the core cap and the ledger allow right now, at least one. + * `exact` is an explicit `--workers`: the caller's width, and it waits for all of it. + */ +export async function leaseWorkers( + want: number, + request: Omit<HeavyRequest, 'units' | 'minUnits'> & { readonly exact?: boolean }, +): Promise<WorkerLease> { + const state = await admission(request.env); + const unbounded = !state.on && state.off !== 'nested' && state.off !== 'unsupported'; + const cap = request.exact === true || unbounded ? undefined : coreCap(request.env); + const units = Math.max(1, cap === undefined ? want : Math.min(want, cap)); + const held = await enter( + { ...request, units, minUnits: request.exact === true ? units : 1 }, + state, + ); + const child = childOf(request.env, state, held); + return { + count: held?.units ?? units, + env: child.env, + fds: child.fds, + release: () => held?.release(), + }; +} diff --git a/packages/cli/src/local-cli.test.ts b/packages/cli/src/local-cli.test.ts index eed57b3b7..3bd1f8eb8 100644 --- a/packages/cli/src/local-cli.test.ts +++ b/packages/cli/src/local-cli.test.ts @@ -55,6 +55,17 @@ describe('resolveLocalCli, x upgrade', () => { ).toBeUndefined(); }); + test('x heavy also outlives an older app CLI: it is run from any directory', () => { + const dir = app(); + const local = join(dir, LOCAL_CLI_BIN); + expect( + resolveLocalCli( + { cwd: dir, selfPath: self, env: {}, command: 'heavy' }, + withVersions(local, '19.4.0', '30.0.0'), + ), + ).toBeUndefined(); + }); + test('an equal or newer app CLI answers it, as it answers everything', () => { const dir = app(); const local = join(dir, LOCAL_CLI_BIN); diff --git a/packages/cli/src/local-cli.ts b/packages/cli/src/local-cli.ts index 9ed4318fe..1334c568b 100644 --- a/packages/cli/src/local-cli.ts +++ b/packages/cli/src/local-cli.ts @@ -45,12 +45,13 @@ const nodeIo: LocalCliIo = { }; /** - * The one command the app's own CLI must not answer when it is the older of the two: an app with no + * The commands the app's own CLI must not answer when it is the older of the two. An app with no * `x upgrade` yet is told `bunx @ultimat3/cli@latest upgrade`, and handing that to the app's CLI - * answers `X_NOT_IMPLEMENTED` from the very version it exists to move off. It is safe because of what - * it does not do: it loads no app, so there is no second registry — the gate is the installed CLI's. + * answers `X_NOT_IMPLEMENTED` from the very version it exists to move off; `x heavy` is run from any + * directory on the machine, an app pinned before it included. Safe because of what they do not do: + * neither loads an app, so there is no second registry. */ -const OUTLIVES_THE_APP_CLI = 'upgrade'; +const OUTLIVES_THE_APP_CLI: readonly (string | undefined)[] = ['upgrade', 'heavy']; const versionOfBin = (bin: string, io: LocalCliIo): string | undefined => { const parsed: unknown = JSON.parse(io.read(join(dirname(dirname(bin)), 'package.json'))); @@ -93,7 +94,7 @@ export function resolveLocalCli( if (!io.exists(local)) return undefined; try { if (io.realpath(local) === io.realpath(input.selfPath)) return undefined; - if (input.command === OUTLIVES_THE_APP_CLI && appCliIsOlder(local, input.selfPath, io)) { + if (OUTLIVES_THE_APP_CLI.includes(input.command) && appCliIsOlder(local, input.selfPath, io)) { return undefined; } return local; diff --git a/packages/cli/src/mcp-errors.ts b/packages/cli/src/mcp-errors.ts index a4cb0aa54..4c4188d99 100644 --- a/packages/cli/src/mcp-errors.ts +++ b/packages/cli/src/mcp-errors.ts @@ -404,6 +404,9 @@ const CLI_FIXES: Readonly<Record<CliErrorCode, string>> = { X_UPGRADE_CODEMODS_PENDING: 'x upgrade codemods --from 0.0.0 --json', X_UPGRADE_LOCKED: 'cat .x/upgrade.lock', X_APP_MIGRATIONS_UNAPPLIED: 'x db migrate --json', + X_HEAVY_LEDGER_UNAVAILABLE: 'x heavy status --json', + X_HEAVY_MEMORY_INVALID: 'export ULTIMATE_HEAVY_MEMORY=8g', + X_HEAVY_COMMAND_NOT_FOUND: 'x heavy run --json -- bun test', }; const isCliCode = (code: string): code is CliErrorCode => diff --git a/packages/cli/src/messages.ts b/packages/cli/src/messages.ts index 763ec9775..f65b650c7 100644 --- a/packages/cli/src/messages.ts +++ b/packages/cli/src/messages.ts @@ -317,6 +317,17 @@ const CATALOG = { ...OPS_MESSAGES, ...FLOOR_MESSAGES, ...STATUS_MESSAGES, + 'cli.heavy.waiting': + 'heavy: {label} waits for {need} MB, position {position} of {queued}; {used} of {capacity} MB held by {holders}', + 'cli.heavy.lockWait': 'heavy: waiting for this checkout’s {name} lock, held by {holder}', + 'cli.heavy.status.summary': + '{used} of {capacity} MB reserved, {holds} running, {queued} queued; {cores} core(s) per run', + 'cli.heavy.status.off': 'admission is off here ({reason}); {cores} core(s) per run', + 'cli.heavy.status.machine': + 'machine: load {load} on {cpus} cores, {runnable} runnable, {free} GB of {total} GB available', + 'cli.heavy.status.hold': 'running: {entry}', + 'cli.heavy.status.queued': 'queued {position}: {entry}', + 'cli.heavy.run.done': '{command} exited {code} after {ran}, {waited} in the queue', } as const; export type MessageKey = keyof typeof CATALOG; diff --git a/packages/cli/src/parse.test.ts b/packages/cli/src/parse.test.ts index 79f57d136..00bbad423 100644 --- a/packages/cli/src/parse.test.ts +++ b/packages/cli/src/parse.test.ts @@ -170,7 +170,7 @@ describe('unit · parseArgs', () => { // And at least one command really does forward, or the rule above is a ban with no exception. expect( SPECS_SHIPPED.filter((spec) => spec.passthrough === true).map((spec) => spec.name), - ).toEqual(['test']); + ).toEqual(['test', 'heavy']); }); test('an unknown command throws X_CLI_UNKNOWN_COMMAND with a suggestion', () => { diff --git a/packages/cli/src/registry.ts b/packages/cli/src/registry.ts index 71656e08a..360c2b81e 100644 --- a/packages/cli/src/registry.ts +++ b/packages/cli/src/registry.ts @@ -27,6 +27,7 @@ import { explainSpec } from './cmd-explain-spec'; import { fixSpec } from './cmd-fix-spec'; import { generateSpec } from './cmd-generate-spec'; import { graphSpec } from './cmd-graph-spec'; +import { heavySpec } from './cmd-heavy-spec'; import { profileSpec } from './cmd-profile-spec'; // Registered here, not by whichever command body happens to load: `dispatch` renders a CLI code's // title for a refusal raised before any body is imported. @@ -138,6 +139,7 @@ export const LAZY_COMMANDS: readonly LazyCommand[] = [ lazy(explainSpec, async () => (await import('./cmd-explain')).explainCommand), lazy(profileSpec, async () => (await import('./cmd-profile')).profileCommand), lazy(statusSpec, async () => (await import('./cmd-status')).statusCommand), + lazy(heavySpec, async () => (await import('./cmd-heavy')).heavyCommand), ]; export const COMMANDS: readonly CliCommand[] = [ diff --git a/packages/cli/src/role-replicator.test.ts b/packages/cli/src/role-replicator.test.ts index 1bdac1385..15fd4b452 100644 --- a/packages/cli/src/role-replicator.test.ts +++ b/packages/cli/src/role-replicator.test.ts @@ -131,6 +131,26 @@ describe('x dev --role replicator', () => { expect(replicatedRelations()).toEqual(['billing_accounts']); }); + /** + * A view has no WAL of its own and Postgres refuses it in a publication (22023), so one listed + * here took the role down at `CREATE PUBLICATION` on every boot of an app that declares one. + */ + test('a view, plain or materialized, is no relation the feed can publish', () => { + entity('billingAccount', { + table: 'billing_accounts', + columns: { id: uuid().primaryKey(), note: text() }, + }); + entity('account_notes', { + columns: { id: uuid().primaryKey(), note: text() }, + view: { sql: 'select id, note from billing_accounts' }, + }); + entity('account_totals', { + columns: { id: uuid().primaryKey(), note: text() }, + view: { sql: 'select id, note from billing_accounts', materialized: true }, + }); + expect(replicatedRelations()).toEqual(['billing_accounts']); + }); + /** * The same fact through the real call chain, so the projection above is proved to be the one * `selectChangeFeed` is handed. `PgReplicationStream`'s constructor screens every entry with diff --git a/packages/cli/src/role-replicator.ts b/packages/cli/src/role-replicator.ts index e6cb6c7fa..b205d0530 100644 --- a/packages/cli/src/role-replicator.ts +++ b/packages/cli/src/role-replicator.ts @@ -69,7 +69,8 @@ const embeddedRefusal = (dev: boolean): BadFlagError | ConfigInvalidError => }); /** - * The relations the feed decodes: every registered entity's PHYSICAL TABLE, never its name. + * The relations the feed decodes: every registered entity's PHYSICAL TABLE, never its name, and + * never a view's — a view writes no WAL of its own, and Postgres refuses one in a publication. * * Both readers of this list are catalog readers. `PgReplicationStream` keeps a change only when * `#entities.has(relation.name)`, and a pgoutput Relation message names the table; `warnPartialIdentity` @@ -83,7 +84,8 @@ const embeddedRefusal = (dev: boolean): BadFlagError | ConfigInvalidError => * feed the same list whatever order it was registered in. */ export function replicatedRelations(): readonly string[] { - return [...new Set(describeEntities().map((entity) => entity.table))].sort(); + const tables = describeEntities().filter((entity) => entity.view === undefined); + return [...new Set(tables.map((entity) => entity.table))].sort(); } /** @@ -92,7 +94,7 @@ export function replicatedRelations(): readonly string[] { * list came from the app's own registry, so it can name the command that adds to it. */ const NO_ENTITIES = - 'no entities are registered, so the replicator would decode changes nothing matches'; + 'no entity on a table is registered, so the replicator would decode changes nothing matches'; const noEntitiesRefusal = (dev: boolean): BadFlagError | ConfigInvalidError => dev diff --git a/packages/cli/src/runtime-cache-limits.test.ts b/packages/cli/src/runtime-cache-limits.test.ts new file mode 100644 index 000000000..f19685ebd --- /dev/null +++ b/packages/cli/src/runtime-cache-limits.test.ts @@ -0,0 +1,113 @@ +// A rung's limits have one home: `cacheTiers` in `runtime.ts`. What the boot must keep true for it: +// the limit written there is the one enforced, and the rung the boot would otherwise have built is +// not built beside it. The one thing a tuned Redis rung gives up, the boot says out loud. + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import type { CacheTier } from '@ultimat3/cache'; +import { + CacheTooLargeError, + isolateTiers, + lruTier, + noopPurgeDriver, + redisTier, + registeredTiers, +} from '@ultimat3/cache'; +import type { CacheTierName } from '@ultimat3/core'; +import { logger } from '@ultimat3/core'; +import { InProcessTransport } from '@ultimat3/realtime/server'; +import type { Env } from './runtime-bindings'; +import { startCacheTiers } from './runtime-cache'; + +const warnings: { line: string; meta: unknown }[] = []; +const printWarning = logger.warn; +let release: (() => Promise<void>) | undefined; +let restore: (() => void) | undefined; + +beforeEach(() => { + warnings.length = 0; + logger.warn = (line: string, meta?: Record<string, unknown>): void => { + warnings.push({ line, meta }); + }; + restore = isolateTiers(); +}); + +afterEach(async () => { + await release?.(); + release = undefined; + restore?.(); + logger.warn = printWarning; +}); + +const idleRedis = { get: async () => null, set: async () => 'OK', send: async () => null }; + +const boot = (tiers: readonly CacheTierName[], supplied: readonly CacheTier[], env: Env = {}) => { + release = startCacheTiers({ + env, + purge: noopPurgeDriver(), + transport: new InProcessTransport(), + tiers, + supplied, + }); +}; + +const rung = (name: CacheTierName): CacheTier => { + const tier = registeredTiers().find((candidate) => candidate.name === name); + return tier ?? expect.unreachable(`the boot registered no ${name} rung`); +}; + +const defaultRedisWarnings = () => + warnings.filter((warning) => warning.line === 'cache.tier.default-redis'); + +describe('unit · a rung tuned in runtime.ts', () => { + test('lruTier({ maxBytes }) is the budget the running rung enforces', async () => { + boot(['request-memo', 'lru'], [lruTier({ maxBytes: 64 })]); + await expect(rung('lru').set('k', 'x'.repeat(65), { tags: [] })).rejects.toBeInstanceOf( + CacheTooLargeError, + ); + }); + + test('redisTier({ maxValueBytes }) is the budget the running rung enforces', async () => { + boot(['redis'], [redisTier({ client: idleRedis, maxValueBytes: 64 })]); + await expect(rung('redis').set('k', 'x'.repeat(65), { tags: [] })).rejects.toBeInstanceOf( + CacheTooLargeError, + ); + }); + + test('jitterFraction: 0 reaches the rung: the lease written is the lease asked for', async () => { + boot(['lru'], [lruTier({ jitterFraction: 0, rng: () => 1 })]); + await rung('lru').set('k', 'v', { tags: [], ttlMs: 60_000 }); + const entry = await rung('lru').get('k'); + const left = (entry?.expiresAt ?? 0) - Date.now(); + // 5% of the lease is 3,000 ms: a spread that survived would land far below this floor. + expect(left).toBeGreaterThan(59_000); + }); +}); + +describe('unit · a supplied redis rung with no client of its own', () => { + test('no Redis URL: it boots, and the boot says the rung dials the local default', () => { + boot(['redis'], [redisTier({ maxValueBytes: 2_048 })]); + expect(defaultRedisWarnings()).toEqual([ + { + line: 'cache.tier.default-redis', + meta: { tier: 'redis', source: 'runtime.ts', expected: 'REDIS_URL or VALKEY_URL' }, + }, + ]); + }); + + test.each([['REDIS_URL'], ['VALKEY_URL']])('%s set: nothing to say', (key) => { + boot(['redis'], [redisTier({ maxValueBytes: 2_048 })], { + [key]: 'redis://cache.internal:6379', + }); + expect(defaultRedisWarnings()).toEqual([]); + }); + + test('a rung holding its own client answered "which Redis" itself', () => { + boot(['redis'], [redisTier({ client: idleRedis })]); + expect(defaultRedisWarnings()).toEqual([]); + }); + + test('a rung cache.tiers does not name dials nothing, so it is not this warning', () => { + boot(['lru'], [redisTier({ maxValueBytes: 2_048 })]); + expect(defaultRedisWarnings()).toEqual([]); + }); +}); diff --git a/packages/cli/src/runtime-cache.ts b/packages/cli/src/runtime-cache.ts index 94c6533ae..7e6169681 100644 --- a/packages/cli/src/runtime-cache.ts +++ b/packages/cli/src/runtime-cache.ts @@ -26,6 +26,7 @@ import { registerInvalidationBroadcast, registerTier, resetTiers, + usesDefaultRedisClient, } from '@ultimat3/cache'; import type { AppConfig, CacheTierName } from '@ultimat3/core'; import { @@ -74,9 +75,12 @@ export interface CacheTiersOptions { */ readonly tiers: readonly CacheTierName[]; /** - * `runtime.ts`'s `cacheTiers`: each replaces the rung of its own `name` — built from nothing - * here, so a supplied `redis` tier needs no `REDIS_URL`. A supplied tier `tiers` does not name is - * logged (`cache.tier.unnamed`) and never registered: the declaration decides the ladder. + * `runtime.ts`'s `cacheTiers`: each replaces the rung of its own `name`, and it is the ONE way + * to set a rung's limits (`lruTier({ maxBytes })`, `redisTier({ maxValueBytes, jitterFraction })`): + * `buildTier` passes none, and a config key would be a second spelling of this replacement. A + * supplied `redis` tier holding its own `client` needs no `REDIS_URL`; one without is + * `warnDefaultRedis`'s. A supplied tier `tiers` does not name is logged (`cache.tier.unnamed`) and + * never registered: the declaration decides the ladder. */ readonly supplied?: readonly CacheTier[] | undefined; /** The wait before re-subscribing after `attempt` failures (1-based). Injected by a test. */ @@ -208,6 +212,22 @@ function warnUnnamed(options: CacheTiersOptions): void { } } +/** + * A `redisTier({ maxValueBytes })` with no `client`, and no URL in the boot's env. `buildTier` refuses + * that state, but only for the rung it builds: this one is the app's, and its default client dials + * the local server, which may be what a single-box deployment means. So it boots, and says so — + * tuning a limit must not quietly cost the operator the signal that the URL is missing. + */ +function warnDefaultRedis(built: readonly CacheTier[], env: Env): void { + const onDefault = built.some((tier) => usesDefaultRedisClient(tier)); + if (!onDefault || redisUrlSource(env) !== undefined) return; + logger.warn('cache.tier.default-redis', { + tier: 'redis', + source: 'runtime.ts', + expected: 'REDIS_URL or VALKEY_URL', + }); +} + /** * Register the tiers the app declared, wire both halves of cross-instance invalidation, and return * the release. @@ -238,6 +258,7 @@ export function startCacheTiers(options: CacheTiersOptions): () => Promise<void> const built = options.tiers.map((name) => supplied.get(name) ?? buildTier(name, options)); for (const tier of built) registerTier(tier); warnUnnamed(options); + warnDefaultRedis(built, options.env); // NATS (or a supplied transport) when it crosses processes; Redis pub/sub when the transport is // the in-process bus and REDIS_URL is set; else in-process. One rule, one boot line. const selected = selectCacheBus({ diff --git a/packages/cli/src/runtime-overrides.ts b/packages/cli/src/runtime-overrides.ts index 6023d8e8f..9a45163f4 100644 --- a/packages/cli/src/runtime-overrides.ts +++ b/packages/cli/src/runtime-overrides.ts @@ -54,8 +54,10 @@ export interface PortValues { readonly authServerStore: AuthServerStore; /** * Rung implementations. `cache.tiers` still NAMES the ladder: a tier here replaces the rung of - * its own `name` (a `redis` tier needs no `REDIS_URL`), the boot builds every other named rung, - * and a tier whose name `cache.tiers` does not list is logged (`cache.tier.unnamed`) and unused. + * its own `name` (a `redisTier` given its own `client` needs no `REDIS_URL`; one without dials + * core's default and, with no URL set, logs `cache.tier.default-redis`), the boot builds every + * other named rung, and a tier `cache.tiers` does not list is logged (`cache.tier.unnamed`) and + * unused. */ readonly cacheTiers: readonly CacheTier[]; /** The process's clock: every "now" and every wait not handed one. Default `systemClock`. */ diff --git a/packages/cli/src/templates/guard-released-forms.ts b/packages/cli/src/templates/guard-released-forms.ts index a39f27bbc..8cc18c656 100644 --- a/packages/cli/src/templates/guard-released-forms.ts +++ b/packages/cli/src/templates/guard-released-forms.ts @@ -88,11 +88,11 @@ export const RELEASED_GUARD_FORMS: Readonly< ['3b0u9i8hze00x', 'error'], ], 'undeclared-env': [ - ['13k4amuxgsnla', 'error'], - ['1xkusngzbabza', 'error'], - ['2jbihvbp83mb4', 'error'], - ['3gsy96z8bgrrl', 'error'], - ['81vvl6xx5bb8', 'error'], + ['16pof11n1trhg', 'error'], + ['1f8kizielequo', 'error'], + ['2jb32jms80mge', 'error'], + ['2t3qtexnpfnyq', 'error'], + ['3gsoj050vbe69', 'error'], ], 'undefined-style-class': [ ['3food9t4vrzv8', 'error'], diff --git a/packages/cli/src/templates/scaffold-claude-agents.ts b/packages/cli/src/templates/scaffold-claude-agents.ts index 7f919c9c7..df2b5b613 100644 --- a/packages/cli/src/templates/scaffold-claude-agents.ts +++ b/packages/cli/src/templates/scaffold-claude-agents.ts @@ -81,8 +81,9 @@ feature slice. Nothing else. A change outside that set is a collision — report Inspect before you change: \`bunx x entities list\`, \`bunx x entities show <name>\`, both with \`--json\`. -Checks: \`bun test <path>/entity.test.ts\`, \`bunx @biomejs/biome check --write <paths>\`, and \`bun run typecheck\` -once when you are otherwise done. Never \`bunx x verify\` — that belongs to whoever coordinates you. +Checks: \`bun test <path>/entity.test.ts\`, \`bunx @biomejs/biome check --write <paths>\`, and +\`bunx x verify --only typecheck\` once when you are otherwise done: it queues for the machine's memory. +Never the whole \`bunx x verify\` — that belongs to whoever coordinates you. `; const server = (app: NameSet): string => `--- @@ -108,8 +109,9 @@ renders. A change outside that set is a collision — report it, do not make it. Inspect before you change: \`bunx x actions show <name>\`, \`bunx x queries show <name>\`, \`bunx x jobs list\`, \`bunx x tasks list\`, \`bunx x policy explain <subject>\` — every one takes \`--json\`. -Checks: \`bun test <path>/<file>.test.ts\`, \`bunx @biomejs/biome check --write <paths>\`, and \`bun run typecheck\` -once when you are otherwise done. Never \`bunx x verify\` — that belongs to whoever coordinates you. +Checks: \`bun test <path>/<file>.test.ts\`, \`bunx @biomejs/biome check --write <paths>\`, and +\`bunx x verify --only typecheck\` once when you are otherwise done: it queues for the machine's memory. +Never the whole \`bunx x verify\` — that belongs to whoever coordinates you. `; const web = (app: NameSet): string => `--- @@ -143,8 +145,9 @@ Every one of the last five is a build error, not a preference — a guard the fr Inspect before you change: \`bunx x routes --json\`, \`bunx x i18n check\` for catalog gaps, and \`bunx x shot <route> --json\` when you need to see what you built. -Checks: \`bun test <path>/page.test.ts\`, \`bunx @biomejs/biome check --write <paths>\`, and \`bun run typecheck\` -once when you are otherwise done. Never \`bunx x verify\` — that belongs to whoever coordinates you. +Checks: \`bun test <path>/page.test.ts\`, \`bunx @biomejs/biome check --write <paths>\`, and +\`bunx x verify --only typecheck\` once when you are otherwise done: it queues for the machine's memory. +Never the whole \`bunx x verify\` — that belongs to whoever coordinates you. `; /** One agent per boundary, plus the read-only one that runs before there is a boundary to hold. */ diff --git a/packages/cli/src/templates/scaffold-claude-commands.ts b/packages/cli/src/templates/scaffold-claude-commands.ts index af25778d4..a4272896f 100644 --- a/packages/cli/src/templates/scaffold-claude-commands.ts +++ b/packages/cli/src/templates/scaffold-claude-commands.ts @@ -101,7 +101,7 @@ While you iterate, narrow the *feedback*, not the gate: | a whole type | \`bunx x test unit\` · \`bunx x test contract\` · \`bunx x test e2e\` | | one gate step | \`bunx x verify --only <step> --json\` — \`NOT A GATE RUN\` | | lint the files you touched | \`bunx @biomejs/biome check --write <paths>\` | -| types, once, when otherwise done | \`bun run typecheck\` | +| types, once, when otherwise done | \`bunx x verify --only typecheck\` — queues for the machine's memory | | a broken environment | \`bunx x doctor\` | ## 5. Hard rules diff --git a/packages/cli/src/templates/scaffold-docs.ts b/packages/cli/src/templates/scaffold-docs.ts index 7b1c8022e..724a46e86 100644 --- a/packages/cli/src/templates/scaffold-docs.ts +++ b/packages/cli/src/templates/scaffold-docs.ts @@ -23,7 +23,7 @@ import { testServicesFiles } from './scaffold-test-services'; */ const conventions = (app: NameSet): string => `# ${app.pascal} -Human-authored, short, stable — one file under two names, \`AGENTS.md\` being a symlink to it. +One file, two names: \`AGENTS.md\` is a symlink to it. Facts live in \`x.manifest.json\`; this file holds only what an agent cannot infer from the code. Every row below names what refuses it. A rule with nothing in the last column does not exist. @@ -40,6 +40,7 @@ queries, jobs, MCP tools, published operations and tests it touches, each with p |---|---|---| | One gate | \`bun run verify\` — \`bunx x verify\`, which builds what its \`budgets\` step weighs. Green means shippable; never merge red. CI runs the same gate, cut into parts: \`bun run verify plan --json\` names them, each runner runs \`bun run setup && bun run verify --only <steps>\`, and \`bun run verify merge\` is the verdict | the gate itself, and \`.github/workflows/ci.yml\` on every push and pull request; \`.github/workflows/image.yml\` publishes only a commit it passed | | Coverage | the unit suite covers 95% of the lines and functions of this app's own source — every file under \`apps/\` and \`packages/\`, one no test loads counted at 0%. A floor, not the goal: a test added to raise it is proven by mutation — break the source, watch it go red, restore | \`X_COVERAGE_BELOW_FLOOR\` on the \`unit\` step; \`coverage\` in \`x.verify.json\` states the floor, and it only rises | +| Many agents | a worker runs its own files; the coordinator runs the gate once | \`bunx x heavy status\` | | One way | generators, not hand-rolled files: \`bunx x g resource\`, \`bunx x g action\`, \`bunx x g route\` | review | | Surfaces | \`site/\` is 0kb JS and may not import \`app/\`; \`shared/\` is a leaf | \`X_BOUNDARY_SITE_TO_APP\` | | Data | routes call actions and queries; only \`repo.ts\` touches the database | \`X_BOUNDARY_ROUTE_TO_DB\` | @@ -134,11 +135,11 @@ Commands: \`bun run setup\`, \`bun run dev\`, \`bun run verify\`, \`bunx x g <pr | Tool | What it answers | How to reach it | |---|---|---| | \`x.manifest.json\` | every route, action, query, job and policy, generated — read it before searching the tree | \`bunx x manifest\` rewrites it | -| \`bunx x impact <file|primitive> --json\` · \`bunx x graph --json\` | what an edit reaches, and the app as one graph (declared, import and observed edges) | rebuilt from the loaded app on every call | +| \`bunx x impact <file|primitive> --json\` · \`bunx x graph --json\` | what an edit reaches, and the app as one graph (declared, import and observed edges) | rebuilt on every call | | \`.mcp.json\` | the dev tools over MCP: routes, schema, policies, the database, tests, the gate — \`bunx x mcp tools\` lists them | a client that reads \`.mcp.json\` starts it; from any other, run \`bunx @ultimat3/cli mcp serve --transport stdio\` here | | \`POST /mcp\` | this app's own tools — every action declaring \`mcp: { expose: true }\` — for a remote agent | \`apps/web/mcp.ts\`; under \`bun run dev\`, \`Authorization: Bearer dev-agent.admin\`, else an API key | -| \`.claude/commands/feature.md\` | build or fix one thing end to end, done when \`bun run verify\` is green | plain markdown: any agent can read one and follow it | -| \`.claude/commands/planx.md\` | write a plan to \`docs/plans/\` for another agent to execute | the same | +| \`.claude/commands/feature.md\` | build or fix one thing end to end, done when \`bun run verify\` is green | plain markdown any agent can follow | +| \`.claude/commands/planx.md\` | write a plan to \`docs/plans/\` | the same | | \`.claude/commands/verify.md\` | run the gate and fix what it reports | the same | Project notes for ${app.kebab}: replace this line with the conventions a newcomer could not guess. diff --git a/packages/cli/src/test-batches.ts b/packages/cli/src/test-batches.ts index b024cfc5f..d3474c4e0 100644 --- a/packages/cli/src/test-batches.ts +++ b/packages/cli/src/test-batches.ts @@ -31,7 +31,7 @@ import type { ExecResult, Runner } from './exec'; import { execOutput } from './exec'; import { msg } from './messages'; -import { SLOT_HELD_ENV } from './test-slots'; +import type { MachineLease } from './test-slots'; import { BATCH_FILES_PER_WORKER } from './test-workers'; /** @@ -89,7 +89,7 @@ export async function runBatches(input: { * Lease the batch's workers from the machine pool (`test-slots.ts`) before it starts, and give * them back when it exits — so two runs on one box share its budget batch by batch. */ - readonly lease?: (want: number) => Promise<{ readonly count: number; release(): void }>; + readonly lease?: MachineLease; readonly options: Parameters<Runner>[1]; /** * Launch no further batch once one is red. Set for a caller-forwarded `--bail`: Bun stops a @@ -109,7 +109,13 @@ export async function runBatches(input: { input.argsFor(files, width), lease === undefined ? input.options - : { ...input.options, env: { ...input.options.env, [SLOT_HELD_ENV]: '1' } }, + : { + ...input.options, + env: { ...input.options.env, ...lease.env }, + ...(lease.fds === undefined || lease.fds.length === 0 + ? {} + : { inheritFds: lease.fds }), + }, ); } finally { lease?.release(); diff --git a/packages/cli/src/test-shards.ts b/packages/cli/src/test-shards.ts index 86599f8dd..30a494d13 100644 --- a/packages/cli/src/test-shards.ts +++ b/packages/cli/src/test-shards.ts @@ -51,6 +51,7 @@ import { msg } from './messages'; import type { CommandResult, Finding, JsonValue, StepResult } from './output'; import { quoteArg } from './shell-quote'; import { testEnvOverrides } from './test-dotenv'; +import type { MachineLease } from './test-slots'; /** Bun's per-file durations, cached under the run's root and refreshed by every parallel run. */ export const TEST_TIMINGS_FILE = '.x/test-timings.json'; @@ -201,7 +202,7 @@ export interface RunShardsOptions { /** `--seed`: handed to the children by the caller's runner, printed back in the reproduce. */ readonly seed?: number; /** The machine-pool lease (`test-slots.ts`), for a default-width run. */ - readonly lease?: (want: number) => Promise<{ readonly count: number; release(): void }>; + readonly lease?: MachineLease; } const isolatedRun = (options: RunShardsOptions): boolean => { @@ -297,8 +298,7 @@ export async function runShards(options: RunShardsOptions): Promise<CommandResul batches, stopOnFailure: bailing, workers: pass.workers, - // A serial pass is one worker: only a default-width parallel pass leases from the pool. - ...(pass.workers > 1 && options.lease !== undefined ? { lease: options.lease } : {}), + ...(options.lease === undefined ? {} : { lease: options.lease }), argsFor: (batch, width) => testArgs({ files: batch, diff --git a/packages/cli/src/test-slots.test.ts b/packages/cli/src/test-slots.test.ts index 33a3cd5f4..996b7991d 100644 --- a/packages/cli/src/test-slots.test.ts +++ b/packages/cli/src/test-slots.test.ts @@ -1,93 +1,93 @@ -// The machine-wide slot pool: a second run on the same box gets what the first left, never double. +// A test run's lease, against a real ledger: the width it is given and what its children inherit. import { afterEach, describe, expect, test } from 'bun:test'; -// why: Bun ships no temp-directory API; mkdtemp/rm own the fixture directory. -import { mkdtempSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; -// why: Bun exposes no tmpdir(); only node:os answers the platform temp root. +// why: a temp ledger per test; Bun ships no mkdtemp. +import { mkdtempSync, rmSync } from 'node:fs'; +// why: the test's scratch root. import { tmpdir } from 'node:os'; // why: Bun ships no path-join primitive. import { join } from 'node:path'; -import { acquireSlots, slotsDir, slotsEnabled } from './test-slots'; +import { HEAVY_HOLD_ENV } from './heavy-env'; +import { E2E_LANE_PEAK_MB, TEST_RUN_BASE_MB, TEST_WORKER_PEAK_MB } from './heavy-weights'; +import { machineLease, suiteWeight } from './test-slots'; +// flock(2) does not exist on Windows, where admission is off (`heavy-flock.ts`). +const NO_FLOCK = process.platform === 'win32'; const dirs: string[] = []; -const fresh = (): string => { - const dir = mkdtempSync(join(tmpdir(), 'x-slots-')); - dirs.push(dir); - return dir; -}; afterEach(() => { for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); }); -const noSleep = async (): Promise<void> => undefined; +const ledger = (workers: number) => { + const dir = mkdtempSync(join(tmpdir(), 'test-slots-')); + dirs.push(dir); + return { + NODE_ENV: 'development', + ULTIMATE_HEAVY_DIR: dir, + ULTIMATE_HEAVY_MEMORY: `${String(2 * TEST_RUN_BASE_MB + workers * TEST_WORKER_PEAK_MB)}m`, + ULTIMATE_HEAVY_CORES: '16', + }; +}; -describe('unit · machine test slots', () => { - test('a second run while the first holds every slot does NOT get its own full width', async () => { - const dir = fresh(); - const first = await acquireSlots({ dir, want: 4, capacity: 4, pid: 101, alive: () => true }); - expect(first.held).toBe(4); - let clock = 0; - const second = await acquireSlots({ - dir, - want: 4, - capacity: 4, - pid: 202, - alive: () => true, - waitMs: 1000, - sleep: async (ms) => { - clock += ms; - }, - now: () => clock, - }); - // It waited the whole window, then ran ONE worker unleased — never four beside four. - expect(second.held).toBe(0); - expect(second.count).toBe(1); - expect(second.waitedMs).toBeGreaterThanOrEqual(1000); - first.release(); - }); +const lease = (env: Record<string, string>, exact = false) => { + const take = machineLease(env, { cwd: '/app', label: 'test', ...(exact ? { exact } : {}) }); + if (take === undefined) expect.unreachable('the lease is on'); + return take; +}; - test('two runs split the capacity: the second runs narrower', async () => { - const dir = fresh(); - const first = await acquireSlots({ dir, want: 3, capacity: 4, pid: 101, alive: () => true }); - const second = await acquireSlots({ dir, want: 4, capacity: 4, pid: 202, alive: () => true }); - expect(first.count).toBe(3); +describe.skipIf(NO_FLOCK)('machineLease', () => { + test('a run narrows to the workers the ledger has room for, and hands its children the hold', async () => { + const take = lease(ledger(3)); + const first = await take(2); + expect(first.count).toBe(2); + expect(first.env?.[HEAVY_HOLD_ENV]).toContain('/holds/'); + expect(first.fds).toHaveLength(1); + const second = await take(4); expect(second.count).toBe(1); first.release(); second.release(); - expect(readdirSync(dir)).toEqual([]); }); - test('a dead holder’s slot is taken over', async () => { - const dir = fresh(); - writeFileSync(join(dir, 'slot-0'), '999999'); - writeFileSync(join(dir, 'slot-1'), '101'); - const lease = await acquireSlots({ - dir, - want: 2, - capacity: 2, - pid: 202, - alive: (pid) => pid === 101, - sleep: noSleep, - }); - expect(lease.held).toBe(1); - lease.release(); - // Only its own slot is released; the live holder's file is untouched. - expect(readdirSync(dir)).toEqual(['slot-1']); + test('the core cap narrows a default run', async () => { + const take = lease({ ...ledger(8), ULTIMATE_HEAVY_CORES: '2' }); + const held = await take(6); + expect(held.count).toBe(2); + held.release(); + }); + + test('an explicit width is neither capped nor narrowed', async () => { + const take = lease({ ...ledger(8), ULTIMATE_HEAVY_CORES: '2' }, true); + const held = await take(6); + expect(held.count).toBe(6); + held.release(); + }); + + test('a released lease is room for the next run', async () => { + const take = lease(ledger(1)); + (await take(1)).release(); + const again = await take(1); + expect(again.count).toBe(1); + again.release(); + }); + + test('inside a test run nothing is reserved and the ask stands', async () => { + const take = machineLease({ NODE_ENV: 'test' }, { cwd: '/app', label: 'test' }); + const held = await take?.(5); + expect(held?.count).toBe(5); + expect(held?.env).toEqual({}); }); - test('release is idempotent and a lease is never wider than asked', async () => { - const dir = fresh(); - const lease = await acquireSlots({ dir, want: 2, capacity: 8, pid: 7, alive: () => true }); - expect(lease.count).toBe(2); - lease.release(); - lease.release(); - expect(readdirSync(dir)).toEqual([]); + test('ULTIMATE_TEST_SLOTS=0 still turns the lease off', () => { + expect( + machineLease({ ULTIMATE_TEST_SLOTS: '0' }, { cwd: '/app', label: 'test' }), + ).toBeUndefined(); }); - test('the directory and the off switch read from the environment', () => { - expect(slotsDir({ ULTIMATE_TEST_SLOTS_DIR: '/x/y' })).toBe('/x/y'); - expect(slotsEnabled({ ULTIMATE_TEST_SLOTS: '0' })).toBe(false); - expect(slotsEnabled({ ULTIMATE_TEST_SLOT_HELD: '1' })).toBe(false); - if (Bun.env['ULTIMATE_TEST_SLOT_HELD'] !== '1') expect(slotsEnabled({})).toBe(true); + test('an e2e lane reserves a browser, not a test worker', () => { + expect(suiteWeight('e2e')).toEqual({ unitMb: E2E_LANE_PEAK_MB, baseMb: 0 }); + expect(suiteWeight('unit')).toEqual({ + unitMb: TEST_WORKER_PEAK_MB, + baseMb: TEST_RUN_BASE_MB, + }); }); }); diff --git a/packages/cli/src/test-slots.ts b/packages/cli/src/test-slots.ts index 14725fca1..c8487c3a8 100644 --- a/packages/cli/src/test-slots.ts +++ b/packages/cli/src/test-slots.ts @@ -1,168 +1,42 @@ -// The MACHINE's test budget, shared by every `x test` and `x verify` running on it at once. -// -// A run's own width (`test-workers.ts`) bounds one run. It cannot bound two: on 2026-09-27 two -// agents' gates on one 45 GB box each planned 14-18 workers off the same free-memory reading and -// the kernel OOM-killed the machine with 36 `bun` processes holding ~40 GB. So each batch of -// workers first LEASES that many slots from a pool of lock files — one file per worker the machine -// budget allows — and a second run on the same box gets what is left: it runs narrower, or waits -// for one slot, instead of doubling the machine's memory. -// -// A slot is a file created with O_EXCL holding the leasing process's pid. A holder that died -// without releasing (SIGKILL, OOM, a closed terminal) is detected by its pid no longer existing, -// and its slot is taken over. Nothing here needs a daemon, and nothing survives a reboot that -// matters: the directory is under the OS temp dir. -// -// Bun ships no O_EXCL create and no pid probe: `openSync(…, 'wx')` and `process.kill(pid, 0)` are -// the node:fs / process primitives for exactly those two. -// why: Bun ships no O_EXCL create (writeFileSync flag 'wx') and no sync unlink/mkdir. -import { mkdirSync, readFileSync, unlinkSync, writeFileSync } from 'node:fs'; -// why: Bun exposes no tmpdir(); only node:os answers the platform temp root. -import { tmpdir } from 'node:os'; -// why: Bun ships no path-join primitive. -import { join } from 'node:path'; +// A test run's share of the machine: each batch of `bun test` workers reserves its measured memory +// in the machine ledger (`heavy-ledger.ts`) and takes the width the core cap and the ledger allow, +// so a second run on the same box narrows or queues instead of doubling its memory. -/** Where the pool lives. Per user, so two accounts on one box never read each other's pids. */ -export const SLOTS_DIR_ENV = 'ULTIMATE_TEST_SLOTS_DIR'; +import type { WorkerLease } from './heavy'; +import { leaseWorkers } from './heavy'; +import type { HeavyEnv } from './heavy-env'; +import { E2E_LANE_PEAK_MB, TEST_RUN_BASE_MB, TEST_WORKER_PEAK_MB } from './heavy-weights'; -/** `0` turns the machine pool off — a CI runner that runs one job per VM has nothing to share. */ +/** `0` turns the lease off for one run; the ledger's own switch is `ULTIMATE_HEAVY_MEMORY=0`. */ export const SLOTS_ENV = 'ULTIMATE_TEST_SLOTS'; -/** - * Set on every `bun test` a leased batch spawns: the slots are already held for that process - * tree, so an `x test` / `x verify` a test runs inside it (the framework's own suites do) must - * not lease again — it would wait on its own parent's slots. - */ -export const SLOT_HELD_ENV = 'ULTIMATE_TEST_SLOT_HELD'; +export type MachineLease = (want: number) => Promise<WorkerLease>; -/** How long a run waits for its first slot before it runs one worker anyway, and says so. */ -export const SLOT_WAIT_MS = 10 * 60 * 1000; +/** What a run of this suite reserves: a browser lane is one measured whole. */ +export const suiteWeight = ( + type: string | undefined, +): { readonly unitMb: number; readonly baseMb: number } => + type === 'e2e' + ? { unitMb: E2E_LANE_PEAK_MB, baseMb: 0 } + : { unitMb: TEST_WORKER_PEAK_MB, baseMb: TEST_RUN_BASE_MB }; -const POLL_MS = 500; - -type Env = Readonly<Record<string, string | undefined>>; - -export const slotsDir = (env: Env = Bun.env): string => - env[SLOTS_DIR_ENV] ?? - join(tmpdir(), `ultimate-test-slots-${String(process.getuid?.() ?? 'user')}`); - -export const slotsEnabled = (env: Env = Bun.env): boolean => - env[SLOTS_ENV] !== '0' && env[SLOT_HELD_ENV] !== '1' && Bun.env[SLOT_HELD_ENV] !== '1'; - -/** A lease on `count` workers' worth of the machine. `count` is at least 1. */ -export interface SlotLease { - readonly count: number; - /** Slots actually held — 0 when the wait ran out and the run went ahead unleased. */ - readonly held: number; - /** How long the lease waited for its first slot. */ - readonly waitedMs: number; - release(): void; -} - -const pidAlive = (pid: number): boolean => { - if (!Number.isInteger(pid) || pid <= 0) return false; - try { - process.kill(pid, 0); - return true; - } catch (error) { - // EPERM is a live process another user owns — alive, and not ours to take. - return (error as { code?: string } | null)?.code === 'EPERM'; - } -}; - -export interface AcquireInput { - /** Workers this batch wants. */ - readonly want: number; - /** Slots the machine budget allows in all — `defaultWorkers()` for this box. */ - readonly capacity: number; - readonly dir?: string; - readonly pid?: number; - readonly alive?: (pid: number) => boolean; - readonly waitMs?: number; - readonly sleep?: (ms: number) => Promise<void>; - readonly now?: () => number; -} - -/** Take one slot file, or take it over from a dead holder. */ -function tryTake(path: string, pid: number, alive: (pid: number) => boolean): boolean { - for (let attempt = 0; attempt < 2; attempt += 1) { - try { - writeFileSync(path, String(pid), { flag: 'wx' }); - return true; - } catch (error) { - if ((error as { code?: string } | null)?.code !== 'EEXIST') return false; - } - let holder: number; - try { - holder = Number.parseInt(readFileSync(path, 'utf8'), 10); - } catch { - // Released between the create and the read: try the create once more. - continue; - } - if (alive(holder)) return false; - try { - unlinkSync(path); - } catch { - // Another run took it over first; the second create decides. - } - } - return false; -} - -/** - * Lease up to `want` slots out of `capacity`. Never zero: with every slot held, it polls until one - * frees, and after `waitMs` it runs one worker unleased rather than hang a gate forever. - */ -export async function acquireSlots(input: AcquireInput): Promise<SlotLease> { - const dir = input.dir ?? slotsDir(); - const pid = input.pid ?? process.pid; - const alive = input.alive ?? pidAlive; - const sleep = input.sleep ?? ((ms: number) => Bun.sleep(ms)); - const now = input.now ?? (() => performance.now()); - const want = Math.max(1, Math.trunc(input.want)); - const capacity = Math.max(1, Math.trunc(input.capacity)); - mkdirSync(dir, { recursive: true }); - const started = now(); - const waitMs = - input.waitMs !== undefined && Number.isFinite(input.waitMs) && input.waitMs >= 0 - ? input.waitMs - : SLOT_WAIT_MS; - const deadline = started + waitMs; - for (;;) { - const held: string[] = []; - for (let index = 0; index < capacity && held.length < want; index += 1) { - const path = join(dir, `slot-${String(index)}`); - if (tryTake(path, pid, alive)) held.push(path); - } - if (held.length > 0 || now() >= deadline) { - const waitedMs = Math.round(now() - started); - let released = false; - return { - count: Math.max(1, held.length), - held: held.length, - waitedMs, - release: () => { - if (released) return; - released = true; - for (const path of held) { - try { - if (readFileSync(path, 'utf8') === String(pid)) unlinkSync(path); - } catch { - // Already gone — a crashed-holder sweep by another run took it. - } - } - }, - }; - } - await sleep(POLL_MS); - } -} - -/** - * The lease a default-width run takes per batch, or none: an explicit `--workers` is the caller's - * call (as it is over the budget), and `ULTIMATE_TEST_SLOTS=0` turns the pool off. - */ +/** `exact` is an explicit `--workers`: that width, waited for, and no core cap. */ export const machineLease = ( - capacity: number, - env: Env = Bun.env, -): ((want: number) => Promise<SlotLease>) | undefined => - slotsEnabled(env) ? (want) => acquireSlots({ want, capacity, dir: slotsDir(env) }) : undefined; + env: HeavyEnv, + input: { + readonly cwd: string; + readonly label: string; + readonly type?: string; + readonly exact?: boolean; + }, +): MachineLease | undefined => + env[SLOTS_ENV] === '0' + ? undefined + : (want) => + leaseWorkers(want, { + label: input.label, + cwd: input.cwd, + env, + ...suiteWeight(input.type), + ...(input.exact === true ? { exact: true } : {}), + }); diff --git a/packages/cli/src/verify-checks.ts b/packages/cli/src/verify-checks.ts index 987985ee3..498553bfc 100644 --- a/packages/cli/src/verify-checks.ts +++ b/packages/cli/src/verify-checks.ts @@ -30,7 +30,7 @@ import { catalogFindings } from './i18n-registration'; import { isrQueryAdvice } from './isr-query-advice'; import { readJobBudgetFindings } from './job-budget'; import { unregisteredJobFindings } from './job-registration'; -import { LINT_ARGV, lintOutcome } from './lint-findings'; +import { lintOutcome } from './lint-findings'; import { liveRouteFindings } from './live-routes'; import { blockedByLoad, withLoadFindings } from './load-findings'; import { @@ -60,6 +60,7 @@ import { contractDiffStep } from './verify-contract-diff'; import { nestedAppRootFindings } from './verify-drift-owner'; import { exerciseStep } from './verify-exercise'; import { floorProblemFindings, readVerifyFloor } from './verify-floor'; +import { runLint, runTypecheck } from './verify-heavy-steps'; import { mcpMountFindings } from './verify-mcp-mount'; import { nativeFormWarnings } from './verify-native-form'; import { roleLoadFindings } from './verify-role-load'; @@ -90,7 +91,7 @@ export const VERIFY_STEPS: readonly VerifyStep[] = [ async run(ctx) { const floor = await readVerifyFloor(ctx.root); const argv = await typecheckArgs(ctx.root, floor?.typecheckBin ?? 'tsc'); - const result = await ctx.runner(['bunx', ...argv], { cwd: ctx.root }); + const result = await runTypecheck(ctx, argv); return fromExec(result, { code: 'X_TYPECHECK_FAILED', cause: 'the project does not typecheck', @@ -107,7 +108,7 @@ export const VERIFY_STEPS: readonly VerifyStep[] = [ // the colour rule is `packages/ui/src/tokens/tokens.test.ts`, and rides on the `unit` step. summary: 'biome: format, no any, no default exports, no unused imports', async run(ctx) { - return lintOutcome(await ctx.runner(LINT_ARGV, { cwd: ctx.root })); + return lintOutcome(await runLint(ctx)); }, }, { diff --git a/packages/cli/src/verify-coverage-run.test.ts b/packages/cli/src/verify-coverage-run.test.ts index 2b603350e..7cab10c82 100644 --- a/packages/cli/src/verify-coverage-run.test.ts +++ b/packages/cli/src/verify-coverage-run.test.ts @@ -251,7 +251,7 @@ describe('runCovered', () => { let released = 0; const held: (string | undefined)[] = []; const runner: Runner = (command, options) => { - held.push(options.env?.['ULTIMATE_TEST_SLOT_HELD']); + held.push(options.env?.['ULTIMATE_HEAVY_HOLD']); return exec(command, options); }; const run = await runCovered({ @@ -264,6 +264,7 @@ describe('runCovered', () => { expect(want).toBe(2); return { count: 1, + env: { ULTIMATE_HEAVY_HOLD: '/ledger/holds/this-run' }, release: () => { released += 1; }, @@ -272,7 +273,7 @@ describe('runCovered', () => { }); expect(run.ok).toBe(true); expect(run.workers).toBe(1); - expect(held).toEqual(['1', '1']); + expect(held).toEqual(['/ledger/holds/this-run', '/ledger/holds/this-run']); expect(released).toBe(1); } finally { await rm(root, { recursive: true, force: true }); diff --git a/packages/cli/src/verify-coverage-run.ts b/packages/cli/src/verify-coverage-run.ts index 6ece7096a..19f3d000e 100644 --- a/packages/cli/src/verify-coverage-run.ts +++ b/packages/cli/src/verify-coverage-run.ts @@ -32,7 +32,7 @@ import { msg } from './messages'; import { countsOf } from './test-counts'; import { testEnvOverrides } from './test-dotenv'; import { failureOf } from './test-shards'; -import { SLOT_HELD_ENV } from './test-slots'; +import type { MachineLease } from './test-slots'; import { costOf, junitDurations, @@ -99,7 +99,7 @@ export interface CoveredRunOptions { readonly isolate?: boolean; readonly env?: Readonly<Record<string, string | undefined>>; readonly widthReason?: string; - readonly lease?: (want: number) => Promise<{ readonly count: number; release(): void }>; + readonly lease?: MachineLease; /** Test files per process. Absent is `COVERAGE_SLICE_FILES`; a test cuts smaller. */ readonly sliceFiles?: number; /** The machine's pace (`verify-pace.ts`): each test's default and declared deadline stretch by it. */ @@ -144,14 +144,14 @@ export const coveredArgs = ( export async function runCovered(options: CoveredRunOptions): Promise<CoveredRun> { const slices = coverageSlices(options.files, options.sliceFiles); const want = Math.max(1, Math.min(Math.trunc(options.workers), slices.length)); - const lease = want > 1 ? await options.lease?.(want) : undefined; + const lease = await options.lease?.(want); const width = Math.min(want, lease?.count ?? want); const base = join(options.root, COVERAGE_DIR, `unit-${String(process.pid)}`); const isolate = options.isolate === true; const overrides = { ...testEnvOverrides(options.root, options.env ?? Bun.env), ...(isolate ? { [ISOLATED_ENV]: '1' } : {}), - ...(lease === undefined ? {} : { [SLOT_HELD_ENV]: '1' }), + ...lease?.env, ...paceEnv(options.pace), }; const results = new Array<ExecResult>(slices.length); @@ -174,6 +174,7 @@ export async function runCovered(options: CoveredRunOptions): Promise<CoveredRun ), { cwd: options.root, + ...(lease?.fds === undefined || lease.fds.length === 0 ? {} : { inheritFds: lease.fds }), env: { ...overrides, ULTIMATE_TEST_WORKER: String(slot), diff --git a/packages/cli/src/verify-deadline.ts b/packages/cli/src/verify-deadline.ts index a5d87e840..b6dad9c14 100644 --- a/packages/cli/src/verify-deadline.ts +++ b/packages/cli/src/verify-deadline.ts @@ -5,6 +5,7 @@ import { TEST_TYPES } from '@ultimat3/testing'; import type { ExecResult, Runner } from './exec'; import { execOutput } from './exec'; +import { heavyWaitClock } from './heavy'; import type { VerifyStepName } from './verify-step'; /** The `x.verify.json` key: `{ "stepTimeoutMs": { "unit": 900000 } }`. */ @@ -462,14 +463,29 @@ export async function killTagged( return [...killed.values()]; } -/** `work`, or `{ timedOut: true }` once `ms` passes. The abandoned promise is never unhandled. */ +/** + * `work`, or `{ timedOut: true }` once `ms` of its own time passes. Time queued for the machine + * (`waited`, a monotonic clock) is not the step's: a gate behind another gate is not a hung one. The + * abandoned promise is never unhandled. + */ export async function raceDeadline<T>( work: Promise<T>, ms: number, + waited: () => number = heavyWaitClock, ): Promise<{ readonly timedOut: false; readonly value: T } | { readonly timedOut: true }> { let timer: ReturnType<typeof setTimeout> | undefined; + const started = performance.now(); + const waitedAtStart = waited(); const expired = new Promise<{ readonly timedOut: true }>((resolve) => { - timer = setTimeout(() => resolve({ timedOut: true }), ms); + const arm = (delay: number): void => { + timer = setTimeout(() => { + const due = started + ms + (waited() - waitedAtStart); + const left = due - performance.now(); + if (left > 0) arm(left); + else resolve({ timedOut: true }); + }, delay); + }; + arm(ms); }); try { const raced = await Promise.race([ diff --git a/packages/cli/src/verify-heavy-steps.ts b/packages/cli/src/verify-heavy-steps.ts new file mode 100644 index 000000000..8459d6078 --- /dev/null +++ b/packages/cli/src/verify-heavy-steps.ts @@ -0,0 +1,26 @@ +// The gate's two whole-tree tools, each run inside its measured reservation in the machine ledger. +// Split from `verify-checks.ts`, which owns what the steps conclude from these results. + +import type { ExecResult } from './exec'; +import { withCheckoutLock, withChild, withHeavy } from './heavy'; +import { LINT_PEAK_MB, typecheckPeakMb } from './heavy-weights'; +import { LINT_ARGV } from './lint-findings'; +import type { VerifyContext } from './verify-step'; + +/** The compiler's peak in the ledger, then this checkout's one `tsc` at a time. */ +export function runTypecheck(ctx: VerifyContext, argv: readonly string[]): Promise<ExecResult> { + const env = ctx.env ?? Bun.env; + return withHeavy( + { label: 'typecheck', cwd: ctx.root, env, unitMb: typecheckPeakMb(argv) }, + (held) => + withCheckoutLock({ name: 'typecheck', root: ctx.root, env }, held, (child) => + ctx.runner(['bunx', ...argv], withChild({ cwd: ctx.root }, child)), + ), + ); +} + +export const runLint = (ctx: VerifyContext): Promise<ExecResult> => + withHeavy( + { label: 'lint', cwd: ctx.root, env: ctx.env ?? Bun.env, unitMb: LINT_PEAK_MB }, + (child) => ctx.runner(LINT_ARGV, withChild({ cwd: ctx.root }, child)), + ); diff --git a/packages/cli/src/verify-test-run.ts b/packages/cli/src/verify-test-run.ts index 8d69ae914..91eed158b 100644 --- a/packages/cli/src/verify-test-run.ts +++ b/packages/cli/src/verify-test-run.ts @@ -13,6 +13,7 @@ import { testEnvOverrides } from './test-dotenv'; import { tidyBeforeTestRun } from './test-housekeeping'; import type { TestFile } from './test-select'; import { failureOf, TEST_TIMINGS_FILE, testArgs } from './test-shards'; +import type { MachineLease } from './test-slots'; import { loadTimings, slowestOf } from './test-timings'; import { BATCH_FILES_PER_WORKER, SHARED_BATCH_FILES_PER_WORKER } from './test-workers'; import type { MachinePace } from './verify-pace'; @@ -36,7 +37,7 @@ export interface ParallelRunOptions { /** Why the width is what it is — printed on the step line. */ readonly widthReason?: string; /** The machine-pool lease (`test-slots.ts`), for a default-width run. */ - readonly lease?: (want: number) => Promise<{ readonly count: number; release(): void }>; + readonly lease?: MachineLease; /** The machine's pace (`verify-pace.ts`): each test's default and declared deadline stretch by it. */ readonly pace?: MachinePace; } @@ -76,7 +77,7 @@ export async function runParallel(options: ParallelRunOptions): Promise<StepOutc runner: options.runner, batches, workers, - ...(options.lease === undefined || workers === 1 ? {} : { lease: options.lease }), + ...(options.lease === undefined ? {} : { lease: options.lease }), argsFor: (batch, width) => testArgs({ files: batch, diff --git a/packages/cli/src/verify-tests.ts b/packages/cli/src/verify-tests.ts index 68904416b..12994add6 100644 --- a/packages/cli/src/verify-tests.ts +++ b/packages/cli/src/verify-tests.ts @@ -16,13 +16,14 @@ import { TEST_TYPES } from '@ultimat3/testing'; import { narrowToApp } from './affected-tests'; import { checkEvalBaselines, checkEvalCoverage, checkEvalRecording } from './app-evals'; import { judgeCoverage } from './coverage-floor'; +import { withHeavy } from './heavy'; import { msg } from './messages'; import { countsOf } from './test-counts'; import { testEnvOverrides } from './test-dotenv'; import type { TestFile } from './test-select'; import { discoverTests } from './test-select'; import { TEST_TIMINGS_FILE } from './test-shards'; -import { machineLease } from './test-slots'; +import { machineLease, suiteWeight } from './test-slots'; import type { WorkerPlan } from './test-workers'; import { availableCpus, SERIAL_TYPES, totalMemory, workerPlan } from './test-workers'; import { runCovered } from './verify-coverage-run'; @@ -126,25 +127,26 @@ const runSerial = async ( // the slice: `bun test` reads a positional as a path filter either way. [...whole.filter((arg, at) => at < 2 || arg.startsWith('--')), ...[...files].sort()]; const envOverrides = { ...testEnvOverrides(ctx.root, ctx.env ?? Bun.env), ...paceEnv(ctx.pace) }; - const exec = (e2e: { command: readonly string[]; env: typeof envOverrides }) => - ctx.runner(e2e.command, { - cwd: ctx.root, - ...(Object.keys(e2e.env).length === 0 ? {} : { env: e2e.env }), - }); + // One serial suite is one reservation for its whole length, the app it boots included. + const execWith = + (fds: readonly number[]) => (e2e: { command: readonly string[]; env: typeof envOverrides }) => + ctx.runner(e2e.command, { + cwd: ctx.root, + ...(Object.keys(e2e.env).length === 0 ? {} : { env: e2e.env }), + ...(fds.length === 0 ? {} : { inheritFds: fds }), + }); + const hostEnv = ctx.env ?? Bun.env; // The e2e step drives a real browser against the app it spawns, when this machine has one. - const result = - type === 'e2e' - ? await withE2eApp( - { - root: ctx.root, - isApp: isApp(ctx.root), - command, - env: envOverrides, - hostEnv: ctx.env ?? Bun.env, - }, - exec, - ) - : await exec({ command, env: envOverrides }); + const result = await withHeavy( + { label: `verify ${type}`, cwd: ctx.root, env: hostEnv, ...suiteWeight(type) }, + (child) => { + const env = { ...envOverrides, ...child.env }; + const exec = execWith(child.fds); + return type === 'e2e' + ? withE2eApp({ root: ctx.root, isApp: isApp(ctx.root), command, env, hostEnv }, exec) + : exec({ command, env }); + }, + ); const outcome: StepOutcome = { ...fromExec(result, { code: 'X_TEST_FAILED', @@ -220,7 +222,12 @@ const runType = async (ctx: VerifyContext, type: TestType): Promise<StepOutcome> const plan = ctx.workers === undefined ? workerPlan(availableCpus(), totalMemory(), env) : undefined; const workers = ctx.workers ?? (plan as WorkerPlan).workers; - const lease = plan === undefined ? undefined : machineLease(plan.workers, env); + const lease = machineLease(env, { + cwd: ctx.root, + label: `verify ${type}`, + type, + ...(plan === undefined ? { exact: true } : {}), + }); const widthReason = plan === undefined ? `${String(Math.min(workers, files.length))} workers (--workers)` diff --git a/packages/cli/src/write-generated.test.ts b/packages/cli/src/write-generated.test.ts index 3ef986e96..103075f2c 100644 --- a/packages/cli/src/write-generated.test.ts +++ b/packages/cli/src/write-generated.test.ts @@ -71,7 +71,7 @@ describe('unit · x new on disk', () => { expect(lstatSync(agents).isSymbolicLink()).toBe(true); expect(readlinkSync(agents)).toBe('CLAUDE.md'); expect(lstatSync(join(root, 'CLAUDE.md')).isFile()).toBe(true); - expect(await Bun.file(agents).text()).toContain('`AGENTS.md` being a symlink to it'); + expect(await Bun.file(agents).text()).toContain('`AGENTS.md` is a symlink to it'); }, ); }); diff --git a/packages/core/src/framework-env.ts b/packages/core/src/framework-env.ts index 2a76fea29..0ebfd4803 100644 --- a/packages/core/src/framework-env.ts +++ b/packages/core/src/framework-env.ts @@ -60,6 +60,7 @@ export const FRAMEWORK_ENV: readonly FrameworkEnvName[] = [ ...internal('cli', 'ULTIMATE_DEV_CHILD', 'ULTIMATE_DEV_SUPERVISOR_PID', 'X_ENV'), ...internal('cli', 'ULTIMATE_KEEP_GLOBAL_CLI', 'ULTIMATE_VERIFY_STEP', 'CI_RUNNER'), ...host('cli', 'CI', 'GITHUB_SHA', 'GITHUB_BASE_REF', 'EDITOR', 'VISUAL', 'USERPROFILE'), + ...host('cli', 'PATH', 'GOMAXPROCS', 'RAYON_NUM_THREADS'), ...contract('core', 'ULTIMATE_ENV', 'ROLE', 'BUILD_ID', 'APP_VERSION', 'LOG_LEVEL'), ...contract('core', 'ULTIMATE_CURSOR_SECRET', 'ULTIMATE_SECRETS_KEY'), ...contract('core', 'ULTIMATE_SECRETS_RETIRED_KEYS', 'METRICS_INTERVAL_MS'), @@ -81,7 +82,7 @@ export const FRAMEWORK_ENV: readonly FrameworkEnvName[] = [ ...contract('db', 'DATABASE_REPLICA_URL', 'DB_RETRY'), ...internal('db', 'ROLLBACK_ALLOW_DESTRUCTIVE'), ...host('jobs', 'HOSTNAME'), - ...contract('mail', 'MAIL_FROM', 'MAIL_POOL_SIZE'), + ...contract('mail', 'MAIL_FROM', 'MAIL_POOL_SIZE', 'MAIL_TIMEOUT_MS'), ...contract('mail', 'RESEND_API_KEY', 'RESEND_WEBHOOK_SECRET', 'SMTP_URL'), ...contract('mail', 'SES_ACCESS_KEY_ID', 'SES_SECRET_ACCESS_KEY', 'SES_SESSION_TOKEN'), ...contract('mail', 'SES_REGION', 'SES_ENDPOINT', 'SES_CONFIGURATION_SET'), diff --git a/packages/http/README.md b/packages/http/README.md index 8c4881461..52e183f4c 100644 --- a/packages/http/README.md +++ b/packages/http/README.md @@ -550,8 +550,9 @@ export const choosePreferences = action({ | | | |---|---| | defaults | `Path=/; HttpOnly; Secure; SameSite=Lax` — core's; every option is `SetCookieOptions` | +| a line already built | `setCookie(line)`, `As of 2026-11`: **the one way to send what `@ultimat3/auth` answers** — `setCookie(sessionCookie(issued.token, policy))`, `setCookie(signedIn.cookie)`. Sent unchanged, and only if `serializeSetCookie` writes exactly those bytes back: CR/LF, an attribute it never writes or a hand-edited line is `X_COOKIE_INVALID`, and the refusal never repeats the line, which can hold a session token | | two calls | two `Set-Cookie` lines, in call order. The same name on two paths is two cookies; nothing overwrites | -| clearing | **`deleteCookie(name, { path, domain })` is the one way**: empty value, `Max-Age=0`, the `Path`/`Domain` (and `Partitioned`) the cookie was set with. `setCookie` with `maxAge: 0`, or an `expires` at or before `ctx.now()`, is `X_COOKIE_INVALID` naming `deleteCookie` | +| clearing | **`deleteCookie(name, { path, domain })` is the one way**: empty value, `Max-Age=0`, the `Path`/`Domain` (and `Partitioned`) the cookie was set with. `setCookie` with `maxAge: 0`, or an `expires` at or before `ctx.now()`, is `X_COOKIE_INVALID` naming `deleteCookie`. So is a line with an empty value and `Max-Age=0`; a line that still carries a value keeps its `Max-Age=0` (a rotated session in its last second) | | an invalid cookie | `X_COOKIE_INVALID` (core) before anything is appended: a non-token name, `__Host-` without `Secure`, a pair over 4096 octets | | off HTTP | `X_NO_REQUEST` in a job, a task or an impersonated child context; `X_NO_CONTEXT` outside any | | is there a request | `hasRequest()`: `true` in a request and in any context derived inside one (where readers still refuse), `false` in a job, a task or no context. For a reader whose honest answer off HTTP is `null` (`useActionAnswer`) | diff --git a/packages/http/src/set-cookie-line.test.ts b/packages/http/src/set-cookie-line.test.ts new file mode 100644 index 000000000..1a5d1372a --- /dev/null +++ b/packages/http/src/set-cookie-line.test.ts @@ -0,0 +1,151 @@ +// `setCookie(line)`: a line `serializeSetCookie` already built — what `@ultimat3/auth` hands back — +// goes on the response as it is, and anything else is refused before it reaches the header bag. +import { describe, expect, test } from 'bun:test'; +import { + ctxOf, + runWithContext, + type SetCookieOptions, + serializeSetCookie, + UltimateError, +} from '@ultimat3/core'; +import { defineHttpConfig } from './config'; +import { asCtx, requestContext } from './context'; +import { httpPipeline } from './pipeline'; +import { textResponse } from './response'; +import { httpRouter } from './router'; +import { setCookie } from './set-cookie'; + +const config = defineHttpConfig({ rateLimit: { scope: 'process' }, dev: false, buildId: null }); +const TOKEN = 'sess_1.c2VjcmV0LXZhbHVl'; +const SESSION = serializeSetCookie('__Host-x_session', TOKEN, { maxAge: 3600 }); + +const context = () => + requestContext({ url: new URL('https://example.com/'), method: 'GET', role: 'web', config }); + +const inRequest = (fn: () => void): readonly string[] => { + const ctx = context(); + runWithContext(asCtx(ctx), fn); + return ctx.headers.getSetCookie(); +}; + +/** The refusal `setCookie(line)` raises, and what reached the header bag beside it. */ +const refusalOf = (line: string) => { + const ctx = context(); + try { + runWithContext(asCtx(ctx), () => setCookie(line)); + } catch (error) { + if (error instanceof UltimateError) { + return { code: error.code, text: `${error.cause} ${error.fix}`, sent: ctx.headers }; + } + throw error; + } + return { code: 'did not throw', text: '', sent: ctx.headers }; +}; + +describe('setCookie(line)', () => { + test('a session line reaches the wire byte for byte, beside a cookie set by name', async () => { + const pipeline = httpPipeline({ + config, + table: httpRouter([ + { + method: 'GET', + path: '/both', + meta: { name: 'both', auth: 'public' }, + handler: () => { + setCookie(SESSION); + setCookie('theme', 'dark'); + return textResponse('ok'); + }, + }, + ]), + }); + const response = await pipeline.handle(new Request('http://localhost/both'), { role: 'web' }); + expect(response.headers.getSetCookie()).toEqual([ + SESSION, + 'theme=dark; Path=/; HttpOnly; Secure; SameSite=Lax', + ]); + }); + + const SERIALIZED: readonly (readonly [string, string, SetCookieOptions])[] = [ + ['plain', 'v', {}], + ['encoded', 'a b;c%', {}], + ['empty', '', {}], + ['loose', 'v', { httpOnly: false, secure: false, sameSite: 'Strict', path: '/app' }], + ['aged', 'v', { maxAge: 60, expires: new Date('2999-01-02T03:04:05Z') }], + ['scoped', 'v', { domain: 'example.com', partitioned: true, priority: 'High' }], + ['none', 'v', { sameSite: 'None' }], + ]; + + test.each(SERIALIZED)('every line the serializer writes is accepted: %s', (name, value, opts) => { + const line = serializeSetCookie(name, value, opts); + expect(inRequest(() => setCookie(line))).toEqual([line]); + }); + + test('a rotated session at its ceiling keeps its Max-Age=0 line', () => { + const line = serializeSetCookie('__Host-x_session', TOKEN, { maxAge: 0 }); + expect(inRequest(() => setCookie(line))).toEqual([line]); + }); + + const NOT_SERIALIZED: readonly (readonly [string, string])[] = [ + ['no attributes', 'sid=abc'], + ['no name', `=${TOKEN}; Path=/; HttpOnly; Secure; SameSite=Lax`], + ['no pair at all', TOKEN], + ['the empty string', ''], + ['a name that is not a token', 'bad name=v; Path=/; HttpOnly; Secure; SameSite=Lax'], + ['a value left unencoded', 'sid=a b; Path=/; HttpOnly; Secure; SameSite=Lax'], + ['a stray percent', 'sid=%; Path=/; HttpOnly; Secure; SameSite=Lax'], + ['a header smuggled after the value', `${SESSION}\r\nX-Evil: 1`], + ['a line feed inside the pair', 'sid=a\n; Path=/; HttpOnly; Secure; SameSite=Lax'], + ['an attribute it never writes', `${SESSION}; Evil=1`], + ['a repeated attribute', `${SESSION}; Path=/`], + ['attributes out of order', 'sid=v; HttpOnly; Path=/; Secure; SameSite=Lax'], + ['an attribute in another case', 'sid=v; path=/; HttpOnly; Secure; SameSite=Lax'], + ['a Max-Age that is not its digits', 'sid=v; Max-Age=1e3; Path=/; Secure; SameSite=Lax'], + ['an unknown SameSite', 'sid=v; Path=/; Secure; SameSite=Sometimes'], + ['an unreadable Expires', 'sid=v; Expires=soon; Path=/; Secure; SameSite=Lax'], + ['a __Host- cookie with a Domain', '__Host-s=v; Domain=a.io; Path=/; Secure; SameSite=Lax'], + ]; + + test.each(NOT_SERIALIZED)('refused, with nothing appended: %s', (_label, line) => { + const { code, sent } = refusalOf(line); + expect(code).toBe('X_COOKIE_INVALID'); + expect(sent.getSetCookie()).toEqual([]); + }); + + test('the refusal never repeats the line: it may hold a live session token', () => { + for (const line of [TOKEN, `${SESSION}; Evil=1`, `bad ${TOKEN}=v; Path=/`]) { + const { code, text } = refusalOf(line); + expect(code).toBe('X_COOKIE_INVALID'); + expect(text).not.toContain(TOKEN); + } + }); + + test('a missing line is refused: `updatePrivileges` answers no cookie when nothing rotated', () => { + const { code } = refusalOf(undefined as unknown as string); + expect(code).toBe('X_COOKIE_INVALID'); + }); + + test('a clearing line is refused: deleteCookie stays the one way to clear', () => { + const { code, text, sent } = refusalOf(serializeSetCookie('sid', '', { maxAge: 0 })); + expect(code).toBe('X_COOKIE_INVALID'); + expect(text).toContain('deleteCookie('); + expect(sent.getSetCookie()).toEqual([]); + }); + + test('an Expires already past is refused for the same reason', () => { + const { code, text } = refusalOf(serializeSetCookie('sid', 'v', { expires: new Date(0) })); + expect(code).toBe('X_COOKIE_INVALID'); + expect(text).toContain('deleteCookie('); + }); + + test('outside a request it refuses, as the named form does', () => { + const job = ctxOf({ role: 'worker' }); + let code = 'did not throw'; + try { + runWithContext(job, () => setCookie(SESSION)); + } catch (error) { + if (error instanceof UltimateError) code = error.code; + } + expect(code).toBe('X_NO_REQUEST'); + }); +}); diff --git a/packages/http/src/set-cookie.ts b/packages/http/src/set-cookie.ts index ef37fd4f5..b68ad682b 100644 --- a/packages/http/src/set-cookie.ts +++ b/packages/http/src/set-cookie.ts @@ -2,7 +2,12 @@ // one serialized line, from core's one serializer, appended to the context's response-header bag — // the bag the `response` stage APPENDS `set-cookie` from, so two cookies are two lines, never one. -import { CookieInvalidError, type SetCookieOptions, serializeSetCookie } from '@ultimat3/core'; +import { + CookieInvalidError, + readCookie, + type SetCookieOptions, + serializeSetCookie, +} from '@ultimat3/core'; import { assertInRequest } from './context'; /** @@ -38,17 +43,109 @@ function refuseClearing(name: string, options: SetCookieOptions, now: Date): voi } } +// A whole line is read back rather than parsed by a second grammar: it is accepted only if core's +// serializer writes exactly those bytes from what was read, so CR/LF, a stray attribute or a broken +// `__Host-` rule cannot reach the header bag. + +interface CookieLine { + readonly name: string; + readonly value: string; + readonly options: SetCookieOptions; +} + +type ReadOptions = { -readonly [K in keyof SetCookieOptions]: SetCookieOptions[K] }; + +const SAME_SITE = ['Strict', 'Lax', 'None'] as const; +const PRIORITY = ['Low', 'Medium', 'High'] as const; + +/** + * Never names the line or anything cut from it: it may hold a live session token, and this message + * reaches the log. + */ +const notSerialized = (): CookieInvalidError => + new CookieInvalidError( + 'value', + 'serialized line', + 'is not a line serializeSetCookie wrote', + 'setCookie(sessionCookie(token, policy)) # pass the line unchanged, or setCookie(name, value, options)', + ); + +/** Attribute names are compared lowercased; the byte comparison is what holds their spelling. */ +function readAttribute(options: ReadOptions, attribute: string): void { + const equals = attribute.indexOf('='); + const key = (equals === -1 ? attribute : attribute.slice(0, equals)).toLowerCase(); + const text = equals === -1 ? '' : attribute.slice(equals + 1); + if (key === 'max-age') options.maxAge = Number(text); + else if (key === 'expires') options.expires = new Date(text); + else if (key === 'domain') options.domain = text; + else if (key === 'path') options.path = text; + else if (key === 'httponly') options.httpOnly = true; + else if (key === 'secure') options.secure = true; + else if (key === 'partitioned') options.partitioned = true; + else if (key === 'samesite') { + const sameSite = SAME_SITE.find((known) => known === text); + if (sameSite !== undefined) options.sameSite = sameSite; + } else if (key === 'priority') { + const priority = PRIORITY.find((known) => known === text); + if (priority !== undefined) options.priority = priority; + } +} + +/** The cookie `line` declares, or `X_COOKIE_INVALID` when the serializer would not write it so. */ +function readCookieLine(line: string): CookieLine { + if (typeof line !== 'string') throw notSerialized(); + const [pair = '', ...attributes] = line.split('; '); + const equals = pair.indexOf('='); + if (equals < 1) throw notSerialized(); + const name = pair.slice(0, equals); + const value = readCookie(pair, name) ?? ''; + // An absent flag is false here, where the serializer's default is true: the line said so. + const options: ReadOptions = { secure: false, httpOnly: false }; + for (const attribute of attributes) readAttribute(options, attribute); + let written: string; + try { + written = serializeSetCookie(name, value, options); + } catch (error) { + // Its own message would echo the name, which on a malformed line can be anything. + if (error instanceof CookieInvalidError) throw notSerialized(); + throw error; + } + if (written !== line) throw notSerialized(); + return { name, value, options }; +} + /** - * Set cookie `name` on the response to the request in scope. Defaults are core's: + * A line the serializer already built, unchanged: what `@ultimat3/auth` answers (`sessionCookie()`, + * `login().cookie`) and cannot set itself, being http's tier-mate. A clearing line is refused like + * any other clear, but a session line keeps its `Max-Age=0`: `updatePrivileges` writes one when the + * rotated session has under a second left, and refusing it would fail the grant that just landed. + */ +function appendLine(ctx: ReturnType<typeof assertInRequest>, line: string): void { + const cookie = readCookieLine(line); + if (cookie.value === '' || cookie.options.maxAge !== 0) { + refuseClearing(cookie.name, cookie.options, ctx.now()); + } + ctx.headers.append('set-cookie', line); +} + +/** + * Put a cookie on the response to the request in scope, from a line `serializeSetCookie` built + * (`setCookie(sessionCookie(token, policy))`), or by name, on core's defaults: * `Path=/; HttpOnly; Secure; SameSite=Lax`. Every call APPENDS a line — the same name on two * paths is two cookies — and an invalid cookie is `X_COOKIE_INVALID` before anything is appended. * Outside a request it throws `X_NO_REQUEST`: a cookie nobody will receive is a silent failure. */ -export const setCookie = (name: string, value: string, options: SetCookieOptions = {}): void => { +export function setCookie(line: string): void; +export function setCookie(name: string, value: string, options?: SetCookieOptions): void; +export function setCookie(nameOrLine: string, value?: string, options: SetCookieOptions = {}) { const ctx = assertInRequest('setCookie()'); - refuseClearing(name, options, ctx.now()); - ctx.headers.append('set-cookie', serializeSetCookie(name, value, options)); -}; + if (value === undefined) { + appendLine(ctx, nameOrLine); + return; + } + refuseClearing(nameOrLine, options, ctx.now()); + ctx.headers.append('set-cookie', serializeSetCookie(nameOrLine, value, options)); +} /** * Clear cookie `name`: an empty value and `Max-Age=0`, with the `Path` and `Domain` it was set diff --git a/packages/jobs/src/queue-wake.live.test.ts b/packages/jobs/src/queue-wake.live.test.ts index b66201a7f..ae2359a60 100644 --- a/packages/jobs/src/queue-wake.live.test.ts +++ b/packages/jobs/src/queue-wake.live.test.ts @@ -10,7 +10,8 @@ import type { PostgresClient } from '@ultimat3/db'; import { dbExecutor, postgresClient, raw, sql } from '@ultimat3/db'; import { postgresJobDriver } from './driver-pg'; import { SQL_JOBS_TABLE } from './driver-pg-sql'; -import { setWakeLive, wakeIsLive } from './enqueue-signal'; +import { JOBS_WAKE_CHANNEL } from './driver-pg-wake-sql'; +import { onEnqueued, setWakeLive, wakeIsLive } from './enqueue-signal'; import { resetJobs } from './job'; import { itemJob } from './operator-surface-fixture'; import { startQueueWake } from './queue-wake'; @@ -20,6 +21,9 @@ const url = Bun.env['TEST_DATABASE_URL']; const hasPostgres = typeof url === 'string' && url.length > 0; const PROBE_DB = probeDatabaseName('x_jobs_wake_live'); const LISTENER_APP = 'x-wake-live-worker'; +/** How long a job gets to start: a small part of the rest its worker is first seen taking. */ +const START_BUDGET_MS = 3_000; +const IDLE_REST_MS = 60_000; const probeUrl = (): string => { const parsed = new URL(url ?? 'postgres://localhost/postgres'); @@ -91,21 +95,30 @@ describe.skipIf(!hasPostgres)('live · postgres · the cross-process wake', () = }); test('a web pod enqueues, an idle worker pod starts the job; the listener is killed, and it does so again', async () => { + // Issue #1167's two vantage points, read only on a miss: a second session, which says whether + // Postgres delivered the notification at all, and the signal the worker's loop subscribes to. + const witnessed: string[] = []; + const witness = await webPod.listen(JOBS_WAKE_CHANNEL, (queue) => witnessed.push(queue)); + cleanups.push(() => witness.unlisten()); + const heard: string[] = []; + const stopHearing = onEnqueued((queue) => heard.push(queue ?? '<any queue>')); + cleanups.push(() => Promise.resolve(stopHearing())); + const wake = startQueueWake({ listener: workerPod, executor: executorFor(workerPod) }); cleanups.push(() => wake.stop()); await until(() => wakeIsLive()); - const started: { item: string; at: number }[] = []; + const started: string[] = []; const handle = itemJob({ run({ input }) { - started.push({ item: input.item, at: performance.now() }); + started.push(input.item); return Promise.resolve(); }, }); const worker = jobWorker({ driver: postgresJobDriver({ executor: executorFor(workerPod) }), pollIntervalMs: 25, - idlePollMaxMs: 60_000, + idlePollMaxMs: IDLE_REST_MS, heartbeatIntervalMs: 3_600_000, context: () => ctxOf({ role: 'worker', buildId: 'test' }), drainOnShutdown: false, @@ -113,9 +126,10 @@ describe.skipIf(!hasPostgres)('live · postgres · the cross-process wake', () = worker.start(); cleanups.push(() => worker.stop()); const web = postgresJobDriver({ executor: executorFor(webPod) }); - const enqueueFromWeb = async (item: string): Promise<number> => { - await until(async () => (await worker.stats()).pollDelayMs >= 800); - const before = performance.now(); + // Resting: the next poll is a minute off, so a start inside the budget can only be the wake's. + // No bound on how fast — a loaded machine broke that one (3 of 60 runs past 200 ms at load 85). + const enqueueFromWeb = async (item: string): Promise<void> => { + await until(async () => (await worker.stats()).pollDelayMs === IDLE_REST_MS); await web.enqueue({ name: handle.name, queue: 'default', @@ -126,30 +140,32 @@ describe.skipIf(!hasPostgres)('live · postgres · the cross-process wake', () = runAt: 1, }); await until( - () => started.some((entry) => entry.item === item), - 3_000, - // Issue #1167: this wait missed once in CI and never locally. What a miss needs to tell — - // did the row commit, is it still waiting, was the wake live, is a LISTEN session held. + () => started.includes(item), + START_BUDGET_MS, + // Issue #1167: this wait missed once in CI and never locally. What a miss needs to tell: + // `witnessed` without the queue, Postgres sent nothing; `witnessed` but not `heard`, the + // worker's session lost it; `heard` with the row `ready`, the loop did not claim it; the + // row `running` or `done`, the claim happened and the run is what to read. async () => JSON.stringify({ rows: await webPod.query( sql`select state, attempt, run_at, created_at from x_jobs where idempotency_key = ${`live:${item}`}`, ), + witnessed, + heard, wakeIsLive: wakeIsLive(), wakeLive: wake.live(), - pollDelayMs: (await worker.stats()).pollDelayMs, + worker: await worker.stats(), listeners: await admin.query( sql`select pid, state, query from pg_stat_activity where application_name = ${LISTENER_APP} and query ilike 'listen%'`, ), }), ); - return (started.find((entry) => entry.item === item)?.at ?? 0) - before; }; - // The wait in hand is 800 ms or more; the job starts in a round trip or two. - expect(await enqueueFromWeb('first')).toBeLessThan(200); + await enqueueFromWeb('first'); // Kill the session the LISTEN is held on. const sessions = await admin.query<{ pid: number }>( @@ -167,6 +183,6 @@ describe.skipIf(!hasPostgres)('live · postgres · the cross-process wake', () = return now.length === 1 && now[0]?.pid !== sessions[0]?.pid && wakeIsLive(); }); - expect(await enqueueFromWeb('after-the-kill')).toBeLessThan(200); + await enqueueFromWeb('after-the-kill'); }); }); diff --git a/packages/mail/README.md b/packages/mail/README.md index 1484dbfbd..b4ced1703 100644 --- a/packages/mail/README.md +++ b/packages/mail/README.md @@ -101,7 +101,7 @@ the app changes between environments; the credential does. |---|---| | *(nothing set)*, `development` / `test` | `memoryMailDriver()` — caught, never sent | | *(nothing set)*, `staging` / `production` | `unconfiguredMailDriver(...)` — every send is `X_MAIL_CREDENTIAL_MISSING` | -| `SMTP_URL` + `MAIL_FROM` | `smtpMailDriver(...)`, `MAIL_POOL_SIZE` optional | +| `SMTP_URL` + `MAIL_FROM` | `smtpMailDriver(...)`, `MAIL_POOL_SIZE` and `MAIL_TIMEOUT_MS` (read/write deadline, default `30000`) optional | | `RESEND_API_KEY` + `MAIL_FROM` | `resendMailDriver(...)` | | `SES_REGION` + `SES_ACCESS_KEY_ID` + `SES_SECRET_ACCESS_KEY` + `MAIL_FROM` | `sesMailDriver(...)`; `SES_SESSION_TOKEN`, `SES_ENDPOINT`, `SES_CONFIGURATION_SET` optional | diff --git a/packages/mail/src/driver-env.test.ts b/packages/mail/src/driver-env.test.ts index 0fbbeae73..93d921b91 100644 --- a/packages/mail/src/driver-env.test.ts +++ b/packages/mail/src/driver-env.test.ts @@ -43,6 +43,40 @@ function memoryLike(members: MemoryMembers): MailDriver { return driver; } +/** + * What a send answers against a server that accepts the socket and never greets: the read deadline + * is the only thing that can end it, so its text proves which `timeoutMs` the driver was built with. + */ +async function silentServerRefusal(env: Record<string, string>): Promise<UltimateError> { + const server = Bun.listen({ + hostname: '127.0.0.1', + port: 0, + socket: { data: () => undefined }, + }); + try { + const { driver } = selectMailDriver({ + SMTP_URL: `smtp://127.0.0.1:${server.port}`, + MAIL_FROM: FROM, + ...env, + }); + const refusal: unknown = await driver + .send({ + to: ['ada@example.test'], + subject: 'Hello', + html: '<p>Hello</p>', + text: 'Hello', + mailId: 'welcome', + locale: 'en', + tz: 'UTC', + }) + .catch((error: unknown) => error); + if (refusal instanceof UltimateError) return refusal; + return expect.unreachable('expected the send to be refused with an UltimateError'); + } finally { + server.stop(true); + } +} + describe('selectMailDriver', () => { test('SMTP_URL selects the smtp transport', () => { const selection = selectMailDriver({ SMTP_URL, MAIL_FROM: FROM }); @@ -112,6 +146,44 @@ describe('selectMailDriver', () => { }, ); + test('MAIL_TIMEOUT_MS is the deadline the smtp driver reads under', async () => { + const error = await silentServerRefusal({ MAIL_TIMEOUT_MS: '25' }); + expect(error.code).toBe('X_MAIL_SEND_FAILED'); + expect(error.cause).toContain('sent nothing for 25ms'); + }); + + // The refusal an operator reads when the deadline is too short has to name the key that raises it. + test('an expired deadline names MAIL_TIMEOUT_MS as the fix', async () => { + const error = await silentServerRefusal({ MAIL_TIMEOUT_MS: '25' }); + expect(error.fix).toStartWith('MAIL_TIMEOUT_MS=60000'); + }); + + // `Number(' ')` is 0: read raw, an empty line in an env file would be a refused zero deadline. + test('a blank MAIL_TIMEOUT_MS is unset, not a zero deadline', () => { + const env = { SMTP_URL, MAIL_FROM: FROM, MAIL_TIMEOUT_MS: ' ' }; + expect(selectMailDriver(env).driver.name).toBe('smtp'); + }); + + // 2147483648 is a whole number `setTimeout` cannot hold: it overflows to 1ms, so the value meant + // as "wait longer" fails every read at once. + test.each([['nope'], ['0'], ['2.5'], ['-1'], ['2147483648']])( + 'MAIL_TIMEOUT_MS=%p is refused at the env boundary', + (raw) => { + const error = thrown(() => + selectMailDriver({ SMTP_URL, MAIL_FROM: FROM, MAIL_TIMEOUT_MS: raw }), + ); + expect(error.code).toBe('X_CONFIG_INVALID'); + expect(error.cause).toContain('MAIL_TIMEOUT_MS'); + expect(error.cause).not.toContain('NaN'); + expect(error.fix).toStartWith('MAIL_TIMEOUT_MS=60000'); + }, + ); + + test('the largest deadline a timer can hold is accepted', () => { + const env = { SMTP_URL, MAIL_FROM: FROM, MAIL_TIMEOUT_MS: '2147483647' }; + expect(selectMailDriver(env).driver.name).toBe('smtp'); + }); + // The keys are documented in the wiki and read by the CLI's service resolution. A key added // here without being added there is a key nothing tells an operator about. test('the read keys are exactly the declared ones', () => { @@ -120,6 +192,7 @@ describe('selectMailDriver', () => { 'RESEND_API_KEY', 'MAIL_FROM', 'MAIL_POOL_SIZE', + 'MAIL_TIMEOUT_MS', 'SES_REGION', 'SES_ACCESS_KEY_ID', 'SES_SECRET_ACCESS_KEY', diff --git a/packages/mail/src/driver-env.ts b/packages/mail/src/driver-env.ts index 99e5e422f..02fa2c314 100644 --- a/packages/mail/src/driver-env.ts +++ b/packages/mail/src/driver-env.ts @@ -22,6 +22,7 @@ export const MAIL_ENV_KEYS = [ 'RESEND_API_KEY', 'MAIL_FROM', 'MAIL_POOL_SIZE', + 'MAIL_TIMEOUT_MS', 'SES_REGION', 'SES_ACCESS_KEY_ID', 'SES_SECRET_ACCESS_KEY', @@ -97,6 +98,27 @@ function poolSizeFrom(env: MailEnvironment): number | undefined { return parsed; } +/** The longest delay `setTimeout` holds: one past it overflows to 1 ms and fires at once. */ +const MAX_TIMER_MS = 2_147_483_647; + +/** + * `MAIL_POOL_SIZE`'s rule, plus a ceiling: the deadline reaches `setTimeout`, so a value past + * `MAX_TIMER_MS` — written to wait LONGER — would expire every read on the next tick. + */ +function timeoutMsFrom(env: MailEnvironment): number | undefined { + const raw = nonEmpty(env['MAIL_TIMEOUT_MS']); + if (raw === undefined) return undefined; + const parsed = Number(raw); + if (!Number.isInteger(parsed) || parsed < 1 || parsed > MAX_TIMER_MS) { + throw new ConfigInvalidError({ + cause: `MAIL_TIMEOUT_MS is "${raw}", which is not a whole number of milliseconds from 1 to ${MAX_TIMER_MS}`, + fix: 'MAIL_TIMEOUT_MS=60000 — in .env.production, or unset it to keep the 30000 default', + meta: { MAIL_TIMEOUT_MS: raw }, + }); + } + return parsed; +} + /** * A credential selects its transport. Two credentials is the one case that cannot be answered by * picking a winner — whichever this chose would be the one an operator did not mean half the time, @@ -130,11 +152,13 @@ export function selectMailDriver( const smtpUrl = nonEmpty(env['SMTP_URL']); if (smtpUrl !== undefined) { const poolSize = poolSizeFrom(env); + const timeoutMs = timeoutMsFrom(env); return { driver: smtpMailDriver({ url: smtpUrl, from: requireFrom(env, 'SMTP_URL'), ...(poolSize === undefined ? {} : { poolSize }), + ...(timeoutMs === undefined ? {} : { timeoutMs }), retainMime: options.retainMime, }), detail: 'SMTP_URL', diff --git a/packages/mail/src/smtp-client.ts b/packages/mail/src/smtp-client.ts index 469da60cc..0861d18b9 100644 --- a/packages/mail/src/smtp-client.ts +++ b/packages/mail/src/smtp-client.ts @@ -164,7 +164,7 @@ class Conversation { stage, detail: `the server sent nothing for ${timeoutMs}ms, so the read deadline expired`, retryable: true, - fix: 'smtpMailDriver({ url, from, timeoutMs: 60_000 }) // where the driver is built', + fix: 'MAIL_TIMEOUT_MS=60000 # in .env.production — or smtpMailDriver({ url, from, timeoutMs: 60_000 })', }), ); }, timeoutMs); diff --git a/packages/mail/src/smtp-socket.ts b/packages/mail/src/smtp-socket.ts index d613d168d..1da4d546d 100644 --- a/packages/mail/src/smtp-socket.ts +++ b/packages/mail/src/smtp-socket.ts @@ -233,7 +233,7 @@ export function smtpStreamOver(runtime: BunConnect, target: SmtpTarget): Promise stage: 'data', detail: `the socket stopped accepting bytes for ${target.timeoutMs}ms`, retryable: true, - fix: 'smtpMailDriver({ url, from, timeoutMs: 60_000 }) // where the driver is built', + fix: 'MAIL_TIMEOUT_MS=60000 # in .env.production — or smtpMailDriver({ url, from, timeoutMs: 60_000 })', }), ); }, target.timeoutMs); diff --git a/packages/testing/README.md b/packages/testing/README.md index 526a5dd97..585e34841 100644 --- a/packages/testing/README.md +++ b/packages/testing/README.md @@ -60,7 +60,7 @@ unitTest('the clock moves only when the test moves it', ({ clock }) => { | `fault-proxy.ts` | `faultProxy({ target })` — a Bun TCP proxy whose mode switches mid-run: `pass`, `delay(ms)`, `blackhole` (bytes held, never dropped), `reset`, `refuse` | | `drills.ts` · `drill-verdict.ts` | the nine fault drills (`x drill`) and the verdict over evidence alone; a property with no evidence is red, and an answer may reach the load `DRILL_ANSWER_SLACK_MS` (500) past the deadline it was given by | | `drill-fleet.ts` · `drill-run.ts` | `startDrillFleet` — `web`, `worker` (and `sync` for `nats-down`) as processes on a scratch database, every service behind a proxy; `runDrill` — baseline, fault, recovery, judged | -| `drill-rollout.ts` · `drill-rollout-verdict.ts` · `drill-output.ts` | `roll-before-migrate`: `startDrillFleet({ beforeMigrate: { rewind, watch } })` migrates, runs the caller's `rewind`, starts `web`, `worker` and `scheduler` unready and counts each output line naming `watch`; `runRollout` waits for every role to meet the schema, runs `fleet.rollout.migrate()`, asks each `/readyz` — `roles-wait`, `serves-after-migrate`, `one-wait-line` | +| `drill-rollout.ts` · `drill-rollout-verdict.ts` · `drill-output.ts` | `roll-before-migrate`: `startDrillFleet({ beforeMigrate: { rewind, watch } })` migrates, runs the caller's `rewind`, starts the roles `drillRoles(drill, found)` names (`web`, `worker`, `scheduler`, then `sync` on a bus and `replicator` on a logical server; `drillRolesNotPlayed` states the rest) unready and counts each output line naming `watch`; `runRollout` waits for every role to meet the schema, runs `fleet.rollout.migrate()`, asks each `/readyz` — `roles-wait`, `serves-after-migrate`, `one-wait-line` | | `drill-load.ts` · `drill-writes.ts` | the load's lanes; `drillWriteTargets` — every action an anonymous POST reaches, each request a fresh seeded draw with the app's `exercise.ts` override, so a job enqueued over HTTP is on the table for `worker-sigkill`/`worker-sigterm` | | `drill-sync.ts` | `startDrillSync` — the sync-client lane: `hello`, resubscribe, redial on a drop, every `replay-gap` counted; `sync-resumes` reads it | | `tenant-proof-report.ts` | `tenantProofFindings` — one coded finding per crossing (`X_TENANT_LEAK`, `X_TENANT_WRITE_CROSSED`), `tenantFindingLines` the human lines. Paths, tables and seeded ids, never a bound value | diff --git a/packages/testing/src/drill-fleet.ts b/packages/testing/src/drill-fleet.ts index 9e06ef0e1..02ff27b60 100644 --- a/packages/testing/src/drill-fleet.ts +++ b/packages/testing/src/drill-fleet.ts @@ -22,6 +22,8 @@ import { testSealKeyEnv } from './test-seal-key'; export interface DrillServiceUrls { /** A Postgres URL with rights to CREATE DATABASE: the drill's scratch database is made there. */ readonly postgres: string; + /** That server runs `wal_level=logical`, so a replicator can stream from the scratch database. */ + readonly logicalPostgres?: boolean | undefined; readonly redis?: string | undefined; /** NATS, and the variable the app's `realtime.urlEnv` reads it from. */ readonly nats?: { readonly url: string; readonly env: string } | undefined; diff --git a/packages/testing/src/drill-services.test.ts b/packages/testing/src/drill-services.test.ts index f1a91c309..737630abc 100644 --- a/packages/testing/src/drill-services.test.ts +++ b/packages/testing/src/drill-services.test.ts @@ -56,6 +56,49 @@ describe('drillServices', () => { }); }); +describe('drillServices · the rollout order', () => { + const logical = 'postgres://u:p@127.0.0.1:5440/postgres?sslmode=disable'; + + test('the logical server and the bus when the run has them, whatever the app declares', () => { + expect( + drillServices(drill('roll-before-migrate'), { ...env, TEST_REPLICATION_URL: logical }, none), + ).toEqual({ + postgres: logical, + logicalPostgres: true, + redis: undefined, + nats: { url: env.TEST_NATS_URL, env: 'NATS_URL' }, + }); + }); + + test('Postgres alone still plays it', () => { + const { TEST_DATABASE_URL } = env; + expect(drillServices(drill('roll-before-migrate'), { TEST_DATABASE_URL }, none)).toEqual({ + postgres: TEST_DATABASE_URL, + logicalPostgres: false, + redis: undefined, + nats: undefined, + }); + }); + + test('the variable an app declares for its bus is the one its roles are handed', () => { + const declared = { redis: false, nats: { env: 'BUS_URL' } }; + expect(drillServices(drill('roll-before-migrate'), env, declared).nats).toEqual({ + url: env.TEST_NATS_URL, + env: 'BUS_URL', + }); + }); + + test('no other drill moves to the logical server or picks up an undeclared bus', () => { + const services = drillServices( + drill('web-sigterm'), + { ...env, TEST_REPLICATION_URL: logical }, + none, + ); + expect(services.postgres).toBe(env.TEST_DATABASE_URL); + expect(services.nats).toBeUndefined(); + }); +}); + describe('DrillFailedError', () => { test('names the property, its evidence and the command that replays the drill', () => { const error = new DrillFailedError({ diff --git a/packages/testing/src/drill-services.ts b/packages/testing/src/drill-services.ts index dcde9f9ad..47b7fcc7f 100644 --- a/packages/testing/src/drill-services.ts +++ b/packages/testing/src/drill-services.ts @@ -22,8 +22,18 @@ export interface DrillDeclarations { readonly nats: { readonly env: string } | undefined; } +/** The `wal_level=logical` server `docker/test-services.env` publishes: the replicator's. */ +const LOGICAL_POSTGRES_ENV = 'TEST_REPLICATION_URL'; +/** Where the rollout order hands a bus the app itself does not declare. */ +const DEFAULT_BUS_ENV = 'NATS_URL'; + type Env = Readonly<Record<string, string | undefined>>; +const present = (env: Env, name: string): string | undefined => { + const value = env[name]?.trim(); + return value === undefined || value === '' ? undefined : value; +}; + const read = (env: Env, name: string, drill: Drill): string => { const value = env[name]?.trim(); if (value === undefined || value === '') { @@ -60,8 +70,24 @@ export function drillServices( }, }); } - const postgres = read(env, DRILL_SERVICE_ENV.postgres, drill); const redis = declared.redis ? read(env, DRILL_SERVICE_ENV.redis, drill) : undefined; + if (drill.fault.kind === 'order') { + // Every role that can start is played, so the bus and the logical server are taken when the + // run has them; without them the drill still plays what Postgres alone serves, and says so. + const logical = present(env, LOGICAL_POSTGRES_ENV); + const bus = + declared.nats === undefined + ? present(env, DRILL_SERVICE_ENV.nats) + : read(env, DRILL_SERVICE_ENV.nats, drill); + return { + postgres: logical ?? read(env, DRILL_SERVICE_ENV.postgres, drill), + logicalPostgres: logical !== undefined, + redis, + nats: + bus === undefined ? undefined : { url: bus, env: declared.nats?.env ?? DEFAULT_BUS_ENV }, + }; + } + const postgres = read(env, DRILL_SERVICE_ENV.postgres, drill); const nats = declared.nats === undefined ? undefined diff --git a/packages/testing/src/drills.test.ts b/packages/testing/src/drills.test.ts index 210491738..24c89a364 100644 --- a/packages/testing/src/drills.test.ts +++ b/packages/testing/src/drills.test.ts @@ -4,7 +4,14 @@ import { describe, expect, test } from 'bun:test'; import { DRILL_DEFAULT_DEADLINE_MS, DRILL_PROMPT_MS, drillBudgets } from './drill-budgets'; import { DRILL_PROPERTIES } from './drill-verdict'; -import { DRILL_NAMES, DRILLS, drillNamed, drillRoles, drillService } from './drills'; +import { + DRILL_NAMES, + DRILLS, + drillNamed, + drillRoles, + drillRolesNotPlayed, + drillService, +} from './drills'; import { screenFaultMode } from './fault-proxy'; const config = { @@ -64,6 +71,35 @@ describe('DRILLS', () => { expect(drill.fault.waitCode).toBe('X_FRAMEWORK_SCHEMA_UNAPPLIED'); }); + test('roll-before-migrate adds the sync node on a bus and the replicator on a logical server', () => { + const drill = drillNamed('roll-before-migrate') ?? expect.unreachable(); + const found = { bus: true, logicalPostgres: true, publishable: true }; + expect(drillRoles(drill, found)).toEqual(['web', 'worker', 'scheduler', 'sync', 'replicator']); + expect(drillRolesNotPlayed(drill, found)).toEqual([]); + }); + + test('a role the run cannot start is stated with what it needs, never dropped silently', () => { + const drill = drillNamed('roll-before-migrate') ?? expect.unreachable(); + const bare = { bus: false, logicalPostgres: false, publishable: true }; + expect(drillRoles(drill, bare)).toEqual(['web', 'worker', 'scheduler']); + expect(drillRolesNotPlayed(drill, bare)).toEqual([ + { role: 'sync', needs: 'TEST_NATS_URL' }, + { role: 'replicator', needs: 'TEST_REPLICATION_URL' }, + ]); + const noTables = { bus: true, logicalPostgres: true, publishable: false }; + expect(drillRoles(drill, noTables)).toEqual(['web', 'worker', 'scheduler', 'sync']); + expect(drillRolesNotPlayed(drill, noTables)).toEqual([ + { role: 'replicator', needs: 'an entity on a table' }, + ]); + }); + + test('every other drill keeps its fleet, and owes no role', () => { + const drill = drillNamed('web-sigterm') ?? expect.unreachable(); + const found = { bus: true, logicalPostgres: true, publishable: true }; + expect(drillRoles(drill, found)).toEqual(['web', 'worker']); + expect(drillRolesNotPlayed(drill, found)).toEqual([]); + }); + test('drillService names the faulted service, or nothing for a signal', () => { expect(drillService(drillNamed('redis-down') ?? expect.unreachable())).toBe('redis'); expect(drillService(drillNamed('web-sigterm') ?? expect.unreachable())).toBeUndefined(); diff --git a/packages/testing/src/drills.ts b/packages/testing/src/drills.ts index 377c35959..8016bd60a 100644 --- a/packages/testing/src/drills.ts +++ b/packages/testing/src/drills.ts @@ -9,10 +9,11 @@ import type { FaultMode } from './fault-proxy'; export type DrillService = 'postgres' | 'redis' | 'nats'; /** - * A role the fleet runs as its own process (`ROLE`, `apps/web/server.ts`); `sync` only for a drill - * whose sync client needs a node to dial, `scheduler` only for the rollout-order drill. + * A role the fleet runs as its own process (`ROLE`, `apps/web/server.ts`); `sync` for a drill whose + * sync client needs a node to dial and for the rollout order, `scheduler` and `replicator` for the + * rollout order alone. */ -export type DrillRole = 'web' | 'worker' | 'sync' | 'scheduler'; +export type DrillRole = 'web' | 'worker' | 'sync' | 'scheduler' | 'replicator'; export type DrillName = | 'pg-blackhole' @@ -111,15 +112,60 @@ export const DRILL_NAMES: readonly DrillName[] = DRILLS.map((drill) => drill.nam export const drillNamed = (name: string): Drill | undefined => DRILLS.find((drill) => drill.name === name); +/** What this run can start beyond the roles Postgres alone serves. */ +export interface DrillRoleFacts { + /** A NATS server: `sync` needs a bus. */ + readonly bus: boolean; + /** The scratch database sits on a `wal_level=logical` server: the replicator streams from it. */ + readonly logicalPostgres: boolean; + /** The app declares an entity on a table: with none, the replicator refuses to boot. */ + readonly publishable: boolean; +} + +/** A role the rollout order did not start, and what would have let it. */ +export interface DrillRoleNotPlayed { + readonly role: DrillRole; + readonly needs: string; +} + +const POSTGRES_ONLY: DrillRoleFacts = { bus: false, logicalPostgres: false, publishable: false }; + +/** The rollout order's roles beyond `web`, `worker` and `scheduler`, each with its gap if any. */ +const rolloutExtras = ( + found: DrillRoleFacts, +): readonly (DrillRoleNotPlayed & { gap: boolean })[] => [ + { role: 'sync', needs: 'TEST_NATS_URL', gap: !found.bus }, + found.publishable + ? { role: 'replicator', needs: 'TEST_REPLICATION_URL', gap: !found.logicalPostgres } + : { role: 'replicator', needs: 'an entity on a table', gap: true }, +]; + /** * The roles a drill's fleet runs: a sync client needs the sync node it dials; the rollout order is - * played on every role Postgres alone serves (`replicator` needs a logical server, `sync` a bus). + * played on every role this run can start, since each one meets the same behind schema. */ -export const drillRoles = (drill: Drill): readonly DrillRole[] => { - if (drill.fault.kind === 'order') return ['web', 'worker', 'scheduler']; +export const drillRoles = ( + drill: Drill, + found: DrillRoleFacts = POSTGRES_ONLY, +): readonly DrillRole[] => { + if (drill.fault.kind === 'order') { + const extras = rolloutExtras(found).filter((extra) => !extra.gap); + return ['web', 'worker', 'scheduler', ...extras.map((extra) => extra.role)]; + } return drill.holds.includes('sync-resumes') ? ['web', 'worker', 'sync'] : ['web', 'worker']; }; +/** The rollout-order roles this run leaves out: stated in the report, never dropped silently. */ +export const drillRolesNotPlayed = ( + drill: Drill, + found: DrillRoleFacts, +): readonly DrillRoleNotPlayed[] => + drill.fault.kind === 'order' + ? rolloutExtras(found) + .filter((extra) => extra.gap) + .map(({ role, needs }) => ({ role, needs })) + : []; + /** The service a drill needs running, when it needs one beyond Postgres. */ export const drillService = (drill: Drill): DrillService | undefined => drill.fault.kind === 'service' ? drill.fault.service : undefined; diff --git a/packages/testing/src/index.ts b/packages/testing/src/index.ts index e61af4bae..35f227b05 100644 --- a/packages/testing/src/index.ts +++ b/packages/testing/src/index.ts @@ -89,8 +89,15 @@ export type { DrillBudgets, DrillEvidence, DrillFailure, DrillProperty } from '. export { judgeDrill } from './drill-verdict'; export type { DrillWrites } from './drill-writes'; export { drillWriteTargets } from './drill-writes'; -export type { Drill, DrillName, DrillRole, DrillService } from './drills'; -export { DRILL_NAMES, DRILLS, drillNamed, drillRoles } from './drills'; +export type { + Drill, + DrillName, + DrillRole, + DrillRoleFacts, + DrillRoleNotPlayed, + DrillService, +} from './drills'; +export { DRILL_NAMES, DRILLS, drillNamed, drillRoles, drillRolesNotPlayed } from './drills'; export type { A11yFinding, A11yRule } from './e2e-a11y'; export { auditTree, NAMED_ROLES } from './e2e-a11y'; export type { A11yAllow, A11yOptions } from './e2e-a11y-run'; diff --git a/scripts/base-recipes/pin.test.ts b/scripts/base-recipes/pin.test.ts index 1204378c4..0fb3a4291 100644 --- a/scripts/base-recipes/pin.test.ts +++ b/scripts/base-recipes/pin.test.ts @@ -9,7 +9,10 @@ import { defineAuth, type IssuedSession, memoryAuthAdapter, + sessionCookie, } from '@ultimat3/auth'; +import { type Ctx, runWithContext } from '@ultimat3/core'; +import { asCtx, defineHttpConfig, requestContext } from '@ultimat3/http'; import { exportsOf, type LoadedRecipe, loadRecipe } from '../lib/base-recipes'; import { REPO_SCAN_TIMEOUT_MS, repoRoot } from '../lib/run'; @@ -21,17 +24,20 @@ interface OperatorRow { readonly pinHash: string; } +interface OperatorSwitch { + operatorId: string; + pin: string; + operator: OperatorRow | null; + device: Pick<AuthSession, 'userId' | 'absoluteExpiresAt'>; +} + interface PinRecipe { enrolPin(pin: string): Promise<string>; switchOperator( auth: Auth, - input: { - operatorId: string; - pin: string; - operator: OperatorRow | null; - device: Pick<AuthSession, 'userId' | 'absoluteExpiresAt'>; - }, + input: OperatorSwitch, ): Promise<{ readonly issued: IssuedSession; readonly cookie: string }>; + switchOperatorOnResponse(auth: Auth, input: OperatorSwitch): Promise<IssuedSession>; } const MAX_ATTEMPTS = 3; @@ -43,7 +49,7 @@ let operator: OperatorRow; beforeAll(async () => { loaded = await loadRecipe(repoRoot(), 'apps/web/shared/base/operator-pin.ts'); - recipe = exportsOf<PinRecipe>(loaded, ['enrolPin', 'switchOperator']); + recipe = exportsOf<PinRecipe>(loaded, ['enrolPin', 'switchOperator', 'switchOperatorOnResponse']); auth = defineAuth({ adapter: memoryAuthAdapter(), rateLimit: { maxAttempts: MAX_ATTEMPTS } }); operator = { userId: 'user_op_7', pinHash: await recipe.enrolPin('4821') }; }); @@ -98,4 +104,27 @@ describe('recipe · a PIN for a shared device', () => { // Another device is another key: one locked till does not lock the operator everywhere. expect((await attempt('4821', 'till_4')).issued.session.userId).toBe('user_op_7'); }); + + test('the action’s half puts that session’s cookie on the response to the request', async () => { + const config = defineHttpConfig({ rateLimit: { scope: 'process' }, dev: false, buildId: null }); + const request = requestContext({ + url: new URL('https://example.com/'), + method: 'POST', + role: 'web', + config, + }); + const ctx: Ctx = asCtx(request); + const issued = await runWithContext(ctx, () => + recipe.switchOperatorOnResponse(auth, { + operatorId: 'op_7', + pin: '4821', + operator, + device: { userId: 'till_5', absoluteExpiresAt: deviceExpiry() }, + }), + ); + expect(issued.session.userId).toBe('user_op_7'); + expect(request.headers.getSetCookie()).toEqual([ + sessionCookie(issued.token, auth.sessions.policy), + ]); + }); }); diff --git a/scripts/bench/render-html.bench.test.ts b/scripts/bench/render-html.bench.test.ts index c350d9cc8..8b9a3afec 100644 --- a/scripts/bench/render-html.bench.test.ts +++ b/scripts/bench/render-html.bench.test.ts @@ -1,7 +1,7 @@ // The SSR walker's big-page cost as a regression test (plan 101 slice 14 P1). One `async` frame and // one `Promise.all` per node made a 50k-row table ~30× a template string and allocate ~150–210 MiB. -// Held as a RATIO to a template-string baseline timed in the same loop, never as wall-clock ms: a -// shared CI runner slows both sides alike, and an absolute ceiling there is a flake, not a test. +// Held as a RATIO of CPU time to a template-string baseline run in the same loop, never as wall-clock +// ms: time spent waiting for a core on a shared machine is no cost of the walker's. import { describe, expect, test } from 'bun:test'; import { renderToHtml } from '../../packages/render/src/render-html'; diff --git a/scripts/bench/render-html.bench.ts b/scripts/bench/render-html.bench.ts index ad42e1c7a..6399764ab 100644 --- a/scripts/bench/render-html.bench.ts +++ b/scripts/bench/render-html.bench.ts @@ -79,10 +79,20 @@ export interface RenderBenchResult { readonly ms: number; /** Best template-string run, timed in the same loop. */ readonly baselineMs: number; - /** `ms / baselineMs`: what the walker costs over writing the strings by hand. */ + /** + * Best CPU time of the walker over best CPU time of the baseline: what it costs over writing the + * strings by hand. CPU, not wall clock: on a loaded machine the long side rarely gets a quiet run + * while the short one does, which moved a wall-clock ratio 4.6-6.9 where this one read 5.3-5.7. + */ readonly ratio: number; } +/** Microseconds of CPU this process spent since `from`. */ +const cpuSince = (from: NodeJS.CpuUsage): number => { + const spent = process.cpuUsage(from); + return spent.user + spent.system; +}; + const round = (value: number, places: number): number => { const scale = 10 ** places; return Math.round(value * scale) / scale; @@ -94,12 +104,18 @@ export async function benchRender(rows: number, runs = 5): Promise<RenderBenchRe templateBaseline(rows); let best = Number.POSITIVE_INFINITY; let baseline = Number.POSITIVE_INFINITY; + let bestCpu = Number.POSITIVE_INFINITY; + let baselineCpu = Number.POSITIVE_INFINITY; for (let i = 0; i < runs; i++) { let started = performance.now(); + let cpu = process.cpuUsage(); (await renderToHtml(page)).length; + bestCpu = Math.min(bestCpu, cpuSince(cpu)); best = Math.min(best, performance.now() - started); started = performance.now(); + cpu = process.cpuUsage(); templateBaseline(rows).length; // `.length` flattens the rope, as a response body would + baselineCpu = Math.min(baselineCpu, cpuSince(cpu)); baseline = Math.min(baseline, performance.now() - started); } const html = await renderToHtml(page); @@ -108,7 +124,7 @@ export async function benchRender(rows: number, runs = 5): Promise<RenderBenchRe bytes: html.length, ms: round(best, 2), baselineMs: round(baseline, 2), - ratio: round(best / baseline, 1), + ratio: round(bestCpu / baselineCpu, 1), }; } diff --git a/scripts/coverage-gate.ts b/scripts/coverage-gate.ts index 2da13eead..a38c208d3 100644 --- a/scripts/coverage-gate.ts +++ b/scripts/coverage-gate.ts @@ -25,6 +25,8 @@ import { mkdirSync, rmSync } from 'node:fs'; import { join, resolve } from 'node:path'; import { finiteOption, renderFixShellArg } from '@ultimat3/core'; import { BadFlagError } from '../packages/cli/src/errors'; +import { withHeavy } from '../packages/cli/src/heavy'; +import { TEST_RUN_BASE_MB, TEST_WORKER_PEAK_MB } from '../packages/cli/src/heavy-weights'; import { quoteArg } from '../packages/cli/src/shell-quote'; import { E2E_PATH_GLOBS } from '../packages/cli/src/verify-tests'; import { flagBool, flagString, parseScriptArgs } from './lib/args'; @@ -271,11 +273,24 @@ async function measure( // Both pipes are read: under GitHub Actions Bun reports a failing test as an `::error` // annotation on stdout, so stderr alone named three failures as "3 tests failed:" and nothing // else. `run()` owns the deadline: a suite that hangs is killed and reported, never waited on. - const ran = await run(testCommand(unit, dir, workers, item.part?.files), { - cwd: root, - env: unitEnv(Bun.env, serviceKeys(root)), - timeoutMs, - }); + // One reservation per unit suite, its whole width, and the deadline starts once it is admitted. + const ran = await withHeavy( + { + label: `coverage ${item.name}`, + cwd: root, + env: Bun.env, + unitMb: TEST_WORKER_PEAK_MB, + baseMb: TEST_RUN_BASE_MB, + units: workers, + }, + (child) => + run(testCommand(unit, dir, workers, item.part?.files), { + cwd: root, + env: { ...unitEnv(Bun.env, serviceKeys(root)), ...child.env }, + inheritFds: child.fds, + timeoutMs, + }), + ); const files = item.part?.files ?? testFilesOf(root, unit); const failure = unitFailure(item.unit, ran, timeoutMs, files); if (failure !== undefined) throw new ScriptError(failure); diff --git a/scripts/heavy-peak.test.ts b/scripts/heavy-peak.test.ts new file mode 100644 index 000000000..abe17ee4b --- /dev/null +++ b/scripts/heavy-peak.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, test } from 'bun:test'; +import { parseProc, treeRssBytes } from './heavy-peak'; + +describe('parseProc', () => { + test('reads pid and ppid past a command name holding spaces and parentheses', () => { + expect(parseProc('42 (a b) c) S 7 1 1', '100 3 0')).toEqual({ + pid: 42, + ppid: 7, + rssBytes: 12_288, + }); + }); + test('refuses text with no command name', () => { + expect(parseProc('garbage', '1 2')).toBeUndefined(); + }); +}); + +describe('treeRssBytes', () => { + test('sums the root and its descendants and nothing else', () => { + const samples = [ + { pid: 3, ppid: 2, rssBytes: 4 }, + { pid: 2, ppid: 1, rssBytes: 2 }, + { pid: 1, ppid: 0, rssBytes: 1 }, + { pid: 9, ppid: 0, rssBytes: 100 }, + ]; + expect(treeRssBytes(samples, 1)).toBe(7); + }); +}); diff --git a/scripts/heavy-peak.ts b/scripts/heavy-peak.ts new file mode 100644 index 000000000..b98574e57 --- /dev/null +++ b/scripts/heavy-peak.ts @@ -0,0 +1,117 @@ +#!/usr/bin/env bun +// The recipe behind every figure in `packages/cli/src/heavy-weights.ts`: run one command and report +// the peak RSS of its whole process tree, sampled from `/proc`. `getrusage` answers for the largest +// single process only, which under-reads a test run or a browser lane by the rest of the tree. +// +// bun run scripts/heavy-peak.ts [--json] -- <command…> + +// why: Bun ships no synchronous directory listing or file read for `/proc`. +import { readdirSync, readFileSync } from 'node:fs'; +import { report } from './lib/log'; + +const SAMPLE_MS = 200; +const PAGE_BYTES = 4096; +/** Past any heavy command this measures, on the slowest box it has run on. */ +const MEASURE_TIMEOUT_MS = 2 * 60 * 60_000; + +export interface ProcSample { + readonly pid: number; + readonly ppid: number; + readonly rssBytes: number; +} + +/** `pid (comm) state ppid …` plus `/proc/<pid>/statm`; `comm` may hold spaces and parentheses. */ +export function parseProc(stat: string, statm: string): ProcSample | undefined { + const close = stat.lastIndexOf(')'); + const pid = Number.parseInt(stat, 10); + const ppid = Number(stat.slice(close + 2).split(' ')[1]); + const pages = Number(statm.split(' ')[1]); + if (close < 0 || !Number.isInteger(pid) || !Number.isInteger(ppid)) return undefined; + return Number.isFinite(pages) ? { pid, ppid, rssBytes: pages * PAGE_BYTES } : undefined; +} + +/** The summed RSS of `root` and every descendant in one snapshot of the process table. */ +export function treeRssBytes(samples: readonly ProcSample[], root: number): number { + const inTree = new Set([root]); + for (let grew = true; grew; ) { + grew = false; + for (const sample of samples) { + if (inTree.has(sample.ppid) && !inTree.has(sample.pid)) { + inTree.add(sample.pid); + grew = true; + } + } + } + return samples.reduce((sum, one) => sum + (inTree.has(one.pid) ? one.rssBytes : 0), 0); +} + +function readProcTable(): ProcSample[] { + const samples: ProcSample[] = []; + for (const name of readdirSync('/proc')) { + if (!/^\d+$/.test(name)) continue; + try { + const sample = parseProc( + readFileSync(`/proc/${name}/stat`, 'utf8'), + readFileSync(`/proc/${name}/statm`, 'utf8'), + ); + if (sample !== undefined) samples.push(sample); + } catch { + // The process exited between the listing and the read: it is no longer in the tree. + } + } + return samples; +} + +if (import.meta.main) { + // Read by position, never parsed: every word after these belongs to the measured command, and + // `bun run` has already consumed the `--` when one was typed. + const argv = Bun.argv.slice(2); + const json = argv[0] === '--json'; + const rest = json ? argv.slice(1) : argv; + const command = rest[0] === '--' ? rest.slice(1) : rest; + if (command.length === 0) { + report( + { + ok: false, + script: 'heavy-peak', + summary: 'no command to measure', + findings: [ + { + code: 'X_CLI_BAD_FLAG', + cause: 'heavy-peak measures one command and was given none after its flags', + fix: 'bun run scripts/heavy-peak.ts -- bunx tsc -b --pretty false', + }, + ], + }, + json, + ); + } + const started = performance.now(); + const child = Bun.spawn(command, { + stdio: [null, 'ignore', 'inherit'], + timeout: MEASURE_TIMEOUT_MS, + }); + let peak = 0; + let done = false; + const exited = child.exited.then((code) => { + done = true; + return code; + }); + while (!done) { + peak = Math.max(peak, treeRssBytes(readProcTable(), child.pid)); + await Promise.race([exited, Bun.sleep(SAMPLE_MS)]); + } + const code = await exited; + const peakMb = Math.ceil(peak / 2 ** 20); + const seconds = Math.round((performance.now() - started) / 1000); + report( + { + ok: code === 0, + script: 'heavy-peak', + summary: `peak tree RSS ${String(peakMb)} MB, ${String(seconds)} s, exit ${String(code)}`, + findings: [], + data: { peakMb, seconds, exitCode: code, command }, + }, + json, + ); +} diff --git a/scripts/lib/ci/test-weights.json b/scripts/lib/ci/test-weights.json index dade73b09..c0ea072b3 100644 --- a/scripts/lib/ci/test-weights.json +++ b/scripts/lib/ci/test-weights.json @@ -4011,7 +4011,6 @@ "scripts/lib/doc-link-targets.test.ts": 339, "scripts/lib/error-code-pages.test.ts": 373, "scripts/lib/export-all.test.ts": 233, - "scripts/lib/file-lock.test.ts": 282, "scripts/lib/fix-shape.test.ts": 318, "scripts/lib/fix-shell-arg-sinks.test.ts": 302, "scripts/lib/framework-manifest.test.ts": 234, diff --git a/scripts/lib/file-lock.test.ts b/scripts/lib/file-lock.test.ts deleted file mode 100644 index c3c149700..000000000 --- a/scripts/lib/file-lock.test.ts +++ /dev/null @@ -1,99 +0,0 @@ -// The lock `bun run typecheck:locked` serialises `tsc -b` with: one holder at a time, a dead -// holder's file taken over, and a release that never removes a lock someone else now holds. - -import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; -// why: a throwaway directory per test; Bun ships no temp-dir primitive. -import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import { join } from 'node:path'; -import { acquireLock, pidAlive } from './file-lock'; -import { ScriptError } from './script-error'; - -let dir = ''; -let lock = ''; -beforeEach(() => { - dir = mkdtempSync(join(tmpdir(), 'file-lock-')); - lock = join(dir, 'tsc.lock'); -}); -afterEach(() => rmSync(dir, { recursive: true, force: true })); - -const noSleep = () => Promise.resolve(); - -describe('unit · an exclusive lock file held by a pid', () => { - test('a free lock is taken at once, written with our pid, and released', async () => { - const release = await acquireLock(lock, { pid: 111, isAlive: () => true, sleep: noSleep }); - expect(readFileSync(lock, 'utf8')).toBe('111'); - release(); - expect(existsSync(lock)).toBe(false); - }); - - test('a lock a LIVE process holds is waited on, and taken once it is released', async () => { - writeFileSync(lock, '222'); - const waitedOn: number[] = []; - let polls = 0; - const release = await acquireLock(lock, { - pid: 111, - isAlive: () => true, - onWait: (holder) => waitedOn.push(holder), - sleep: () => { - polls += 1; - if (polls === 3) rmSync(lock); // the holder finishes - return Promise.resolve(); - }, - }); - expect(polls).toBe(3); - expect(waitedOn).toEqual([222]); // told once, not once per poll - expect(readFileSync(lock, 'utf8')).toBe('111'); - release(); - }); - - test('a DEAD holder is taken over without waiting', async () => { - writeFileSync(lock, '333'); - const release = await acquireLock(lock, { - pid: 111, - isAlive: (pid) => pid !== 333, - sleep: () => expect.unreachable('a dead holder is never waited on'), - }); - expect(readFileSync(lock, 'utf8')).toBe('111'); - release(); - }); - - test('a file with no pid in it — a writer killed mid-create — is stale too', async () => { - writeFileSync(lock, ''); - const release = await acquireLock(lock, { - pid: 111, - isAlive: () => true, - sleep: () => expect.unreachable('an empty lock is never waited on'), - }); - expect(readFileSync(lock, 'utf8')).toBe('111'); - release(); - }); - - test('a release never removes a lock another process now holds', async () => { - const release = await acquireLock(lock, { pid: 111, isAlive: () => true, sleep: noSleep }); - writeFileSync(lock, '444'); // taken over after we were presumed dead - release(); - expect(readFileSync(lock, 'utf8')).toBe('444'); - }); - - // `??` lets NaN through and `sleep(NaN)` is no wait: the poll would spin (`bun run finite-bounds`). - test('a pollMs that is not a finite, non-negative number is refused before any wait', async () => { - for (const pollMs of [Number.NaN, Number.POSITIVE_INFINITY, -1]) { - const refused = await acquireLock(lock, { - pid: 111, - isAlive: () => true, - sleep: () => expect.unreachable('a refused poll never sleeps'), - pollMs, - }).then( - () => 'taken', - (error: unknown) => (error instanceof ScriptError ? error.code : 'not a ScriptError'), - ); - expect(refused).toBe('X_CLI_UNEXPECTED'); - } - }); - - test('pidAlive answers this process live and an unused pid dead', () => { - expect(pidAlive(process.pid)).toBe(true); - expect(pidAlive(2 ** 22 + 12_345)).toBe(false); - }); -}); diff --git a/scripts/lib/file-lock.ts b/scripts/lib/file-lock.ts deleted file mode 100644 index 4055de476..000000000 --- a/scripts/lib/file-lock.ts +++ /dev/null @@ -1,96 +0,0 @@ -// One exclusive lock file per resource, held by a pid, portable to every host the repo runs on: -// created with `O_EXCL` (`'wx'`), so two processes can never both create it, and taken over only -// when the pid written in it is dead. `flock(1)` did this on Linux alone; macOS and Windows have none. - -// why: `'wx'` is the exclusive-create flag; Bun ships no primitive that fails when the file exists. -import { closeSync, openSync, readFileSync, unlinkSync, writeSync } from 'node:fs'; -import { ScriptError } from './script-error'; - -export interface LockDeps { - readonly pid: number; - /** Whether `pid` is a live process. `process.kill(pid, 0)` in production. */ - readonly isAlive: (pid: number) => boolean; - readonly sleep: (ms: number) => Promise<void>; - /** Told once, when the first attempt finds the lock held by a live process. */ - readonly onWait?: (holder: number) => void; - readonly pollMs?: number; -} - -/** True when the process exists; EPERM means it exists under another user, ESRCH that it does not. */ -export const pidAlive = (pid: number): boolean => { - try { - process.kill(pid, 0); - return true; - } catch (error: unknown) { - return typeof error === 'object' && error !== null && 'code' in error && error.code === 'EPERM'; - } -}; - -const isExists = (error: unknown): boolean => - typeof error === 'object' && error !== null && 'code' in error && error.code === 'EEXIST'; - -/** The pid in the file, or undefined for an empty, half-written or vanished file. */ -function holderOf(path: string): number | undefined { - try { - const pid = Number.parseInt(readFileSync(path, 'utf8').trim(), 10); - return Number.isSafeInteger(pid) && pid > 0 ? pid : undefined; - } catch { - return undefined; - } -} - -function tryCreate(path: string, pid: number): boolean { - try { - const fd = openSync(path, 'wx'); - try { - writeSync(fd, String(pid)); - } finally { - closeSync(fd); - } - return true; - } catch (error: unknown) { - if (isExists(error)) return false; - throw error; - } -} - -/** - * Wait until this process holds `path`, then answer the release. A file whose holder is dead — - * or holds no pid at all, a writer killed between create and write — is removed and retried, so a - * crashed run never wedges the next one. The release removes the file only while it is still ours. - */ -export async function acquireLock(path: string, deps: LockDeps): Promise<() => void> { - // `??` lets NaN through, and `sleep(NaN)` is a zero wait: the poll would spin on the lock file. - const pollMs = deps.pollMs ?? 250; - if (!Number.isFinite(pollMs) || pollMs < 0) { - throw new ScriptError({ - code: 'X_CLI_UNEXPECTED', - cause: `acquireLock was given pollMs ${String(pollMs)}, so its wait between tries would be no wait at all`, - fix: 'acquireLock(path, { ...deps, pollMs: 250 }) # a finite, non-negative number of milliseconds, or omit it', - }); - } - let told = false; - for (;;) { - if (tryCreate(path, deps.pid)) break; - const holder = holderOf(path); - if (holder === undefined || !deps.isAlive(holder)) { - try { - unlinkSync(path); - } catch { - // Another waiter removed it first; the next create decides between us. - } - continue; - } - if (!told) deps.onWait?.(holder); - told = true; - await deps.sleep(pollMs); - } - return () => { - if (holderOf(path) !== deps.pid) return; - try { - unlinkSync(path); - } catch { - // Already gone: nothing of ours to remove. - } - }; -} diff --git a/scripts/lib/public-api-compat.test.ts b/scripts/lib/public-api-compat.test.ts index 49b77f885..6f6d80b34 100644 --- a/scripts/lib/public-api-compat.test.ts +++ b/scripts/lib/public-api-compat.test.ts @@ -320,6 +320,21 @@ describe('public-api-compat · overloads', () => { ); }); + // `export const f = (…) => …` growing an overload has to become `export function f`: same calls. + test('an arrow const that became overloaded functions, its call shape kept, is compatible', () => { + const arrow = 'const set : ( name : string , value : string ) => void'; + const overloaded = + 'function set ( line : string ) : void\nfunction set ( name : string , value : string ) : void'; + expect(verdict(arrow, overloaded)).toBe( + 'compatible: every recorded call shape still matches an overload with the same return', + ); + expect( + verdict(arrow, 'function set ( line : string ) : void\nfunction set ( n : 1 ) : void'), + ).toBe( + 'break: the recorded call shape matches no overload with the same return — closest: no overload accepts it', + ); + }); + test('an old call shape matched only by an overload with a widened return is a break', () => { const widened = MAP.replace('Promise < T [ ] >', 'Promise < T [ ] | S [ ] >'); expect(verdict(MAP, `${ALL}\n${widened}`)).toBe( diff --git a/scripts/lib/public-api-compat.ts b/scripts/lib/public-api-compat.ts index acdc3006b..0538270a5 100644 --- a/scripts/lib/public-api-compat.ts +++ b/scripts/lib/public-api-compat.ts @@ -93,6 +93,9 @@ function declarationChange( type Signature = Extract<ApiShape, { kind: 'function' }>; +const sameHeading = (one: Signature, next: Signature): boolean => + next.head === one.head || next.constHead === one.head; + const signaturesOf = (shape: ApiShape | undefined): readonly Signature[] | undefined => shape?.kind === 'function' ? [shape] : shape?.kind === 'overloads' ? shape.signatures : undefined; @@ -109,7 +112,7 @@ function callShapes(was: ApiShape, now: ApiShape | undefined): ApiChange | undef // The FIRST overload that accepts the call is the one TypeScript resolves it against: a wider // one put in front answers every existing call with its own return, whatever follows it. const first = current.find( - (next) => next.head === one.head && params(one.params, next.params).kind !== 'break', + (next) => sameHeading(one, next) && params(one.params, next.params).kind !== 'break', ); if (first === undefined) { return breaks( @@ -120,7 +123,7 @@ function callShapes(was: ApiShape, now: ApiShape | undefined): ApiChange | undef const later = current.slice(current.indexOf(first) + 1); const shadowed = later.some( (next) => - next.head === one.head && + sameHeading(one, next) && next.returns === one.returns && params(one.params, next.params).kind !== 'break', ); diff --git a/scripts/lib/public-api-shape-kinds.ts b/scripts/lib/public-api-shape-kinds.ts index f2f75e508..b0813eb0f 100644 --- a/scripts/lib/public-api-shape-kinds.ts +++ b/scripts/lib/public-api-shape-kinds.ts @@ -44,6 +44,11 @@ export type ApiShape = readonly kind: 'function'; /** The name, the type parameters and every local type the signature names. */ readonly head: string; + /** + * An overload's heading as `const name :` would spell it: an arrow const that gains an + * overload must become a `function`, and the calls it accepted are the same calls. + */ + readonly constHead?: string; readonly params: readonly ParamCode[]; readonly returns: string; /** diff --git a/scripts/lib/public-api-shape.ts b/scripts/lib/public-api-shape.ts index 1ba27fcf9..f8d5d9a68 100644 --- a/scripts/lib/public-api-shape.ts +++ b/scripts/lib/public-api-shape.ts @@ -104,6 +104,14 @@ function localShapes(lines: readonly string[]): Record<string, LocalShape> { ); } +/** One line of an overload set, carrying the heading the same name has as an arrow `const`. */ +function overloadOf(words: readonly string[], localLines: readonly string[]) { + const shape = structural(words, localLines); + if (shape?.kind !== 'function' || words[0] !== 'function') return shape; + const heading = ['const', words[1], ':', ...words.slice(2, topLevel(words, '(', 2))].join(' '); + return { ...shape, constHead: shortDigest([heading, localLines.join('\n')].join('\n')) }; +} + /** * The shape of one symbol's text as `symbolReader` builds it: its own declaration(s), then any * `local …` lines. Overloads, merged declarations, enums, namespaces and every type this cannot @@ -121,7 +129,7 @@ export function shapeOf(text: string): ApiShape { } if (shaped !== undefined) return shaped; } - const signatures = own.map((line) => structural(tokenize(line), localLines)); + const signatures = own.map((line) => overloadOf(tokenize(line), localLines)); if (own.length > 1 && signatures.every((one) => one?.kind === 'function')) { return { kind: 'overloads', signatures: signatures.filter((one) => one?.kind === 'function') }; } diff --git a/scripts/lib/public-api.lock.json b/scripts/lib/public-api.lock.json index a262e0154..8d7835795 100644 --- a/scripts/lib/public-api.lock.json +++ b/scripts/lib/public-api.lock.json @@ -1482,7 +1482,8 @@ "tagKeys": {"hash":"b889bf0932880b48","shape":{"kind":"function","head":"c616525b","params":["b022e537"],"returns":"21c248b4"}}, "tagMatches": {"hash":"7c6c92ae6b11fb81","shape":{"kind":"function","head":"44876b1b","params":["253677c8","253677c8"],"returns":"bf84d3d6"}}, "tagsIntersect": {"hash":"b42e401e35d9c407","shape":{"kind":"function","head":"5ad0f6b8","params":["b022e537","b022e537"],"returns":"bf84d3d6"}}, - "unregisterDependent": {"hash":"dfd66cde7c9b3154","shape":{"kind":"function","head":"a6f8d5d5","params":["868ec2f6"],"returns":"445be54d"}} + "unregisterDependent": {"hash":"dfd66cde7c9b3154","shape":{"kind":"function","head":"a6f8d5d5","params":["868ec2f6"],"returns":"445be54d"}}, + "usesDefaultRedisClient": {"hash":"28f38cb133058fd7","shape":{"kind":"function","head":"d2a7cb00","params":["aef20e29"],"returns":"bf84d3d6"}} }, "@ultimat3/cache/adapter": { "CacheDriverUnavailableError": {"hash":"d72014d5f72569b1","shape":{"kind":"class","head":"bf07adf1","members":{},"ctor":["{cause=119ef915,driver=119ef915,fix=119ef915}"]}}, @@ -1521,7 +1522,7 @@ "DEV_ROLES": {"hash":"07002d90bbf3eb91","shape":{"kind":"declaration","keyword":"const","head":"07002d90"}}, "DoctorProbe": {"hash":"be0df56aa3713cde","shape":{"kind":"members","head":"f89af633","members":{"appUrl":"6c665e22","authStorage":"1e94bd6a","bunVersion":"1fb0a448","database":"5451f576","drift":"fdc77de0","embeddedDatabase":"367bc00b","envNames":"d863606e","exists":"5b86e268","frameworkFindings":"a4c2a735","offlineFallback":"82e1ee67","pins":"c98810d0","pooler":"1e7e13de","port":"4e57c805","portFree":"b6d6532d","root":"6c665e22","rowSecurity":"?fdc77de0","rowSecurityOff":"4c6cda06","sealedKeys":"9187134c","snapshots":"fdc77de0","typescript":"f667253c","unflooredSteps":"6d71d972"}}}, "ErrorCodePage": {"hash":"aeb3d867808268d7","shape":{"kind":"members","head":"89b36b24","members":{"path":"1fb0a448","text":"1fb0a448"}}}, - "ExecOptions": {"hash":"fdee07f07c562213","shape":{"kind":"members","head":"c3190773","members":{"cwd":"1fb0a448","env":"?fa15e968","onSpawn":"?b0744bb9","stdin":"?1fb0a448"}}}, + "ExecOptions": {"hash":"8924e6bef7d6bc02","shape":{"kind":"members","head":"c3190773","members":{"cwd":"1fb0a448","env":"?fa15e968","inheritFds":"?6a01f5ed","onSpawn":"?b0744bb9","stdin":"?1fb0a448"}}}, "ExecResult": {"hash":"c238441be9cbf483","shape":{"kind":"members","head":"1f747144","members":{"code":"4e57c805","command":"f8639a23","durationMs":"4e57c805","ok":"a001763a","stderr":"1fb0a448","stdout":"1fb0a448"}}}, "FRAMEWORK_INLINE_SCRIPTS": {"hash":"57fbbb56aec54c29","shape":{"kind":"declaration","keyword":"const","head":"57fbbb56"}}, "FRAMEWORK_SCHEMA": {"hash":"79d7be1350b5a3c8","shape":{"kind":"declaration","keyword":"const","head":"79d7be13"}}, @@ -3985,7 +3986,7 @@ "routeNotFound": {"hash":"e24fcb363468f2f9","shape":{"kind":"function","head":"b7053452","params":["473287f8","473287f8"],"returns":"60b9b55c"}}, "securityHeaders": {"hash":"4964d61106c5272b","shape":{"kind":"function","head":"f0a66dc8","params":["0b9d2f7d","?{https=?b9f83688}"],"returns":"fcbe173a"}}, "selfOrigin": {"hash":"9e2d7679493c4c58","shape":{"kind":"function","head":"5fe1d9f9","params":["e7a241de","bf84d3d6"],"returns":"473287f8"}}, - "setCookie": {"hash":"b8460b4bf3f93029","shape":{"kind":"function","head":"96398d1b","params":["473287f8","473287f8","?e2187b48"],"returns":"445be54d"}}, + "setCookie": {"hash":"18aad9567ed68a2b","shape":{"kind":"overloads","signatures":[{"kind":"function","head":"9c702492","params":["473287f8"],"returns":"445be54d"},{"kind":"function","head":"9c702492","params":["473287f8","473287f8","?e2187b48"],"returns":"445be54d"}]}}, "setRedirect": {"hash":"368348bae0298c72","shape":{"kind":"function","head":"35823967","params":["473287f8","?c6422a13"],"returns":"445be54d"}}, "signInRedirect": {"hash":"eab0837d76503813","shape":{"kind":"function","head":"c4379d39","params":["{code=1fb0a448,ctx=37703631,request=4eed49ab,signInPath=c380b02d}"],"returns":"e4672166"}}, "socketIdleTimeoutSeconds": {"hash":"dcfbcb553f1a2308","shape":{"kind":"function","head":"2bae3ba6","params":["12886f9d"],"returns":"12886f9d"}}, @@ -7440,7 +7441,9 @@ "DrillName": {"hash":"6c28e62086e184a2","shape":{"kind":"literals","head":"36c09922","values":["453963c3","4573dc75","5146cb51","559531be","6206afb8","6ae5d649","7a45edd8","ce83ab56","e1324f02"]}}, "DrillProperty": {"hash":"c04c1847cccc5eda","shape":{"kind":"literals","head":"39ad8d1d","values":["25cca24c","27e69181","287b151a","37c306ad","68fa4c8f","74942053","7d04c3f4","9c889996","bf72375e","cdb26bbd","d82c3d3c","e88ec7f3","fd51f466"]}}, "DrillReport": {"hash":"ca80936b70916952","shape":{"kind":"members","head":"964884e0","members":{"baseline":"93b656bc","budgets":"c5acdd3b","evidence":"c427ef07","name":"1fb0a448","verdict":"4ab77250"}}}, - "DrillRole": {"hash":"d47cde92f8487f47","shape":{"kind":"literals","head":"28893621","values":["1636f9ec","5cfa1a73","877fc376","bde14896"]}}, + "DrillRole": {"hash":"f517b04d1c2206ca","shape":{"kind":"literals","head":"28893621","values":["150feb7a","1636f9ec","5cfa1a73","877fc376","bde14896"]}}, + "DrillRoleFacts": {"hash":"2d568898a3ec6537","shape":{"kind":"members","head":"6a971a6d","members":{"bus":"a001763a","logicalPostgres":"a001763a","publishable":"a001763a"}}}, + "DrillRoleNotPlayed": {"hash":"7bec48d7ce6ed514","shape":{"kind":"members","head":"6ee9c4af","members":{"needs":"1fb0a448","role":"a354f1a7"}}}, "DrillRoleProcess": {"hash":"a4de27dca1446f0a","shape":{"kind":"members","head":"f68d5332","members":{"exited":"f7bf3eb7","log":"4025f8b6","origin":"1fb0a448","readyz":"1fb0a448","respawn":"6c20bfe0","role":"a354f1a7","signal":"050d9e15"}}}, "DrillRollout": {"hash":"dfb6fc20460b8018","shape":{"kind":"members","head":"556c7eb8","members":{"exitCode":"8a524555","migrate":"6c20bfe0","watched":"e8e0051f"}}}, "DrillRolloutEvidence": {"hash":"1cb402635cf73f57","shape":{"kind":"members","head":"cb00327c","members":{"roles":"efac4e8b"}}}, @@ -7448,7 +7451,7 @@ "DrillRolloutRole": {"hash":"8834e1eb9a1840ee","shape":{"kind":"members","head":"7f9a1b7f","members":{"exit":"4a7123ef","logTail":"1fb0a448","ready":"a001763a","role":"1fb0a448","waitLines":"4e57c805"}}}, "DrillService": {"hash":"07f52834ef264f91","shape":{"kind":"literals","head":"d1f4dbb4","values":["264c0424","54d2e07e","840b960b"]}}, "DrillServiceMissingError": {"hash":"8b6e1fe6e8bda47f","shape":{"kind":"class","head":"4a51d7a7","members":{},"ctor":["{drill=1fb0a448,gap=37307166}"]}}, - "DrillServiceUrls": {"hash":"b8d4d3946d63d6ce","shape":{"kind":"members","head":"86eb70e4","members":{"nats":"?f3dc49ba","postgres":"1fb0a448","redis":"?6c665e22"}}}, + "DrillServiceUrls": {"hash":"62aaeb3c69f13ad3","shape":{"kind":"members","head":"86eb70e4","members":{"logicalPostgres":"?f4af6dab","nats":"?f3dc49ba","postgres":"1fb0a448","redis":"?6c665e22"}}}, "DrillSyncEvidence": {"hash":"e31ea87ff0cae5a8","shape":{"kind":"declaration","keyword":"type","head":"e31ea87f"}}, "DrillSyncLane": {"hash":"e60c3f6967590292","shape":{"kind":"members","head":"9e514524","members":{"stop":"4ec0b7ab"}}}, "DrillSyncOptions": {"hash":"352b8d18f117fee3","shape":{"kind":"members","head":"f37c45f8","members":{"connect":"?ae009dd9","protocolVersion":"4e57c805","redialMs":"?5d7469a7","sleep":"?8f142133","subscriptions":"?627c5b5e","url":"1fb0a448"}}}, @@ -7764,7 +7767,8 @@ "describeIslandState": {"hash":"d506080b9b74ebd4","shape":{"kind":"function","head":"8f672ef7","params":["c62b7b86","473287f8","33a3ea97","d7bb3f3a"],"returns":"445be54d"}}, "drillBudgets": {"hash":"0eb3d88093296d21","shape":{"kind":"function","head":"7a80c55d","params":["1055edba","?12886f9d"],"returns":"01eeb6b2"}}, "drillNamed": {"hash":"449f52c03f43f5cd","shape":{"kind":"function","head":"1bbd8c01","params":["473287f8"],"returns":"9832f246"}}, - "drillRoles": {"hash":"4e7ab68cf908dbd5","shape":{"kind":"function","head":"fd6b980e","params":["0e211378"],"returns":"91349f1a"}}, + "drillRoles": {"hash":"6507dcf242c1161d","shape":{"kind":"function","head":"fd6b980e","params":["0e211378","?d77cc8cf"],"returns":"91349f1a"}}, + "drillRolesNotPlayed": {"hash":"8c9332b562c87e55","shape":{"kind":"function","head":"5f4e738d","params":["0e211378","d77cc8cf"],"returns":"a154072b"}}, "drillServices": {"hash":"98549176c3dc0926","shape":{"kind":"function","head":"92be92e6","params":["0e211378","494d9aa0","6c8d38b7"],"returns":"182376ac"}}, "drillWriteTargets": {"hash":"b2ec0907a13556e0","shape":{"kind":"function","head":"610f79cb","params":["f98643d0"],"returns":"50e4cb41"}}, "dropSql": {"hash":"b7e9b1a54429898e","shape":{"kind":"function","head":"41fe336d","params":["473287f8"],"returns":"473287f8"}}, diff --git a/scripts/lib/release-due-next.test.ts b/scripts/lib/release-due-next.test.ts index 2b1791896..264e05cb1 100644 --- a/scripts/lib/release-due-next.test.ts +++ b/scripts/lib/release-due-next.test.ts @@ -154,15 +154,20 @@ describe('unit · release-due-next · an open window proposes only a ready next' }); }); -describe('unit · release-due-next · the brief keys one row per reason', () => { - test('the `data.nextLane.reason` table is the closed list, plus the proposal row', async () => { +describe('unit · release-due-next · the brief states the major path', () => { + test('it acts on the proposal, and every reason it names is a real one', async () => { const brief = await Bun.file(join(repoRoot(), '.dz/briefs/release-due.md')).text(); - const start = brief.indexOf('| `data.nextLane.reason` |'); - expect(start, 'the next-lane table vanished from the brief').toBeGreaterThan(-1); - const rows = brief.slice(start).split('\n\n')[0]?.split('\n').slice(2) ?? []; - const keys = rows.map((row) => /^\| `([a-z_]+)` \|/.exec(row)?.[1] ?? row.slice(0, 40)); - expect(keys.slice(0, -1).sort()).toEqual([...NEXT_LANE_REASONS].sort()); - expect(rows.at(-1)).toContain('`data.nextLane.steps`'); - expect(rows.at(-1)).toContain('`data.nextLane.prTitle`'); + const start = brief.indexOf('## The major path: `data.nextLane`'); + expect(start, 'the major path vanished from the brief').toBeGreaterThan(-1); + const section = brief.slice(start).split('\n## ')[0] ?? ''; + for (const field of ['propose: true', 'prTitle', 'steps', 'reason']) { + expect(section).toContain(`\`data.nextLane.${field}`); + } + // Every other reason is "do nothing", so only a reason it singles out can go stale. + const named = [...section.matchAll(/`([a-z]+_[a-z_]+)`/g)].map((match) => match[1] ?? ''); + expect(named.length).toBeGreaterThan(0); + expect( + named.filter((name) => !(NEXT_LANE_REASONS as readonly string[]).includes(name)), + ).toEqual([]); }); }); diff --git a/scripts/lib/release-preflight.test.ts b/scripts/lib/release-preflight.test.ts index c117deb5a..876223c8a 100644 --- a/scripts/lib/release-preflight.test.ts +++ b/scripts/lib/release-preflight.test.ts @@ -40,14 +40,24 @@ describe('unit · a release refuses a working tree with changes in it', () => { }); test('a dirty tree is refused with the paths in the cause and the rerun in the fix', () => { - const [finding, ...rest] = treeDirtyFindings(' M CHANGELOG.md\n?? scratch.ts', RERUN); + const [finding, ...rest] = treeDirtyFindings(' M README.md\n?? scratch.ts', RERUN); expect(rest).toEqual([]); expect(finding?.code).toBe('X_RELEASE_TREE_DIRTY'); - expect(finding?.cause).toContain('CHANGELOG.md'); + expect(finding?.cause).toContain('README.md'); expect(finding?.cause).toContain('scratch.ts'); expect(finding?.fix).toBe(`git stash -u && ${RERUN}`); }); + test('the release notes alone are not dirt: the bump promotes them', () => { + expect(treeDirtyFindings(' M CHANGELOG.md', RERUN)).toEqual([]); + }); + + test('the release notes do not excuse anything beside them', () => { + const [finding] = treeDirtyFindings(' M CHANGELOG.md\n M README.md', RERUN); + expect(finding?.cause).toContain('1 uncommitted change(s)'); + expect(finding?.cause).not.toContain('CHANGELOG.md'); + }); + test('a long list is capped, and says how many it left out', () => { const porcelain = Array.from({ length: DIRTY_PATHS_SHOWN + 3 }, (_, i) => `?? f${i}.ts`).join( '\n', diff --git a/scripts/lib/release-preflight.ts b/scripts/lib/release-preflight.ts index f264e3541..5798d8543 100644 --- a/scripts/lib/release-preflight.ts +++ b/scripts/lib/release-preflight.ts @@ -21,6 +21,13 @@ export const dirtyPaths = (porcelain: string): readonly string[] => .filter((line) => line.trim().length > 0) .map((line) => line.replace(/^[ MTADRCU?!]{1,2}\s+/, '')); +/** + * The one file a release may find edited: its `## [Unreleased]` notes are what the bump promotes, + * and a session whose runner makes the only commit (`.dz/briefs/release-due.md`) cannot commit them + * first. + */ +export const RELEASE_NOTES_PATH = 'CHANGELOG.md'; + /** * A release rewrites 47 files and commits them as one. Over a dirty tree that commit carries * whatever else was in flight, and the edit it mixes with is the one nobody reviewed as a release. @@ -42,7 +49,7 @@ export function treeDirtyFindings( }, ]; } - const paths = dirtyPaths(porcelain); + const paths = dirtyPaths(porcelain).filter((path) => path !== RELEASE_NOTES_PATH); if (paths.length === 0) return []; const shown = paths.slice(0, DIRTY_PATHS_SHOWN).join(', '); const more = paths.length - DIRTY_PATHS_SHOWN; diff --git a/scripts/lib/run.ts b/scripts/lib/run.ts index 522118d3e..936679862 100644 --- a/scripts/lib/run.ts +++ b/scripts/lib/run.ts @@ -24,6 +24,8 @@ export interface RunOptions { readonly timeoutMs?: number; /** The child's environment. Default: this process's. */ readonly env?: Readonly<Record<string, string | undefined>>; + /** Descriptors the child inherits from fd 3 up: the machine ledger's locks. */ + readonly inheritFds?: readonly number[]; } export interface RunResult { @@ -103,9 +105,7 @@ export async function run( const proc = Bun.spawn([head, ...rest], { cwd: options.cwd, env: { ...(options.env ?? Bun.env) }, - stdin: 'ignore', - stdout: 'pipe', - stderr: 'pipe', + stdio: ['ignore', 'pipe', 'pipe', ...(options.inheritFds ?? [])], signal: deadline.signal, killSignal: 'SIGKILL', }); diff --git a/scripts/lib/spawn-timeout-pins.ts b/scripts/lib/spawn-timeout-pins.ts index 0be236b74..496b5edbe 100644 --- a/scripts/lib/spawn-timeout-pins.ts +++ b/scripts/lib/spawn-timeout-pins.ts @@ -22,9 +22,9 @@ export interface SpawnTimeoutPin { export const SPAWN_TIMEOUT_PINS: Readonly<Record<string, SpawnTimeoutPin>> = { // why: four children whose lifetime is a human's or a caller's, never a number this file knows. cli: { - count: 4, + count: 5, reason: - "why: `exec.ts` is the CLI subprocess boundary — its deadline is the caller's: `x verify` ends a step's whole tree in `verify-deadline.ts` (`X_VERIFY_STEP_TIMEOUT`). `local-cli-handoff.ts` spawns the app's own CLI with the user's argv, forwards its stop signals and answers its exit code, and `bin.ts` exits with that code, so its lifetime is the command the user typed. `dev-supervisor.ts` runs `x dev`'s server until Ctrl-C. `cmd-secrets.ts` opens `$EDITOR` on a terminal and waits for the human to close it.", + "why: `cmd-heavy.ts` runs the command its caller named, for as long as the caller wants it, forwarding stop signals through `local-cli-handoff.ts`. `exec.ts` is the CLI subprocess boundary — its deadline is the caller's: `x verify` ends a step's whole tree in `verify-deadline.ts` (`X_VERIFY_STEP_TIMEOUT`). `local-cli-handoff.ts` spawns the app's own CLI with the user's argv, forwards its stop signals and answers its exit code, and `bin.ts` exits with that code, so its lifetime is the command the user typed. `dev-supervisor.ts` runs `x dev`'s server until Ctrl-C. `cmd-secrets.ts` opens `$EDITOR` on a terminal and waits for the human to close it.", }, // why: two long-lived children each with its own stop path. Chrome's launch left this row in // 31.0.0: it moved to `@ultimat3/scraping/cdp`, and its spawn carries `timeout: lifetimeMs`. diff --git a/scripts/preflight.ts b/scripts/preflight.ts index 79c98e14f..6897fc43f 100644 --- a/scripts/preflight.ts +++ b/scripts/preflight.ts @@ -9,6 +9,9 @@ // Paths narrow the diff to what they prefix: a worker in a shared checkout names its own file set, // and the units its teammates touched are neither run nor counted. +import { withHeavy } from '../packages/cli/src/heavy'; +import { LOCAL_CORE_CAP } from '../packages/cli/src/heavy-cores'; +import { TEST_RUN_BASE_MB, TEST_WORKER_PEAK_MB } from '../packages/cli/src/heavy-weights'; import { parseScriptArgs } from './lib/args'; import { changedSinceBase } from './lib/changed-files'; import { unreleasedCheck } from './lib/changelog-unreleased-git'; @@ -75,7 +78,29 @@ export async function runStep( runner: StepRunner = run, ): Promise<StepResult> { const timeoutMs = step.heavy ? LONG_RUN_TIMEOUT_MS : DEFAULT_RUN_TIMEOUT_MS; - const result = await runner(step.command, { cwd: root, timeoutMs }); + // A heavy step is one reservation; the coverage gate it starts inherits it instead of queueing. + const result = step.heavy + ? await withHeavy( + { + label: `preflight ${step.name}`, + cwd: root, + env: Bun.env, + unitMb: TEST_WORKER_PEAK_MB, + baseMb: TEST_RUN_BASE_MB, + units: LOCAL_CORE_CAP, + minUnits: 1, + }, + (child) => + runner(step.command, { + cwd: root, + timeoutMs, + // `run` takes a whole environment, so the hold is laid over this process's. + ...(Object.keys(child.env).length === 0 && child.fds.length === 0 + ? {} + : { env: { ...Bun.env, ...child.env }, inheritFds: child.fds }), + }), + ) + : await runner(step.command, { cwd: root, timeoutMs }); if (step.kind === 'test') { return { name: step.name, diff --git a/scripts/release-due.test.ts b/scripts/release-due.test.ts index f5e1fd1b2..a572ce8f3 100644 --- a/scripts/release-due.test.ts +++ b/scripts/release-due.test.ts @@ -16,23 +16,26 @@ // `release-due-flags.test.ts`, which owns the parser, the flag reader and the JSON envelope. import { afterAll, describe, expect, setDefaultTimeout, test } from 'bun:test'; -import { mkdtemp, rm } from 'node:fs/promises'; // why: Bun has no mkdtemp. -// why: Bun exposes no tmpdir(), so only node:os answers the platform temp root. -import { tmpdir } from 'node:os'; // why: Bun exposes no path-join primitive. import { join } from 'node:path'; -import { FETCH_MAIN } from './lib/base-ref'; +import { type TempTree, tempTree } from '@ultimat3/testing'; +import { FETCH_MAIN, type GitRunner } from './lib/base-ref'; import { git as gitIn } from './lib/release-due-git'; import { REPO_SCAN_TIMEOUT_MS } from './lib/run'; import type { TagProbe } from './registry-audit'; -import { decideAt, lastReleaseTag, readWindow } from './release-due'; +import { decideAt, lastReleaseTag, readWindow, refreshTags } from './release-due'; // Each test spawns a dozen git processes. setDefaultTimeout(REPO_SCAN_TIMEOUT_MS); -const madeDirs: string[] = []; +const trees: TempTree[] = []; +const ownedDir = (prefix: string): string => { + const tree = tempTree(prefix); + trees.push(tree); + return tree.path; +}; afterAll(async () => { - for (const dir of madeDirs.splice(0)) await rm(dir, { recursive: true, force: true }); + await Promise.all(trees.splice(0).map((tree) => tree.dispose())); }); const HOUR = 3_600_000; @@ -52,12 +55,10 @@ const CHANGELOG = /** A repository with one publishable package, released and tagged at 1.0.0 — or untagged. */ async function released(options: { readonly tag: boolean } = { tag: true }) { - const dir = await mkdtemp(join(tmpdir(), 'release-due-')); - madeDirs.push(dir); + const dir = ownedDir('release-due-'); // A bare upstream reachable as `origin`, so `baseRef()` resolves and `decideAt` can read - // both HEAD and the base SHA from the same fixture. Tracks in `madeDirs` for cleanup. - const upstream = await mkdtemp(join(tmpdir(), 'release-due-upstream-')); - madeDirs.push(upstream); + // both HEAD and the base SHA from the same fixture. Owned like `dir`. + const upstream = ownedDir('release-due-upstream-'); // Through the one bounded, non-interactive git the script uses: no call here can outlive // GIT_TIMEOUT_MS or wait on stdin, whatever git or the machine is doing. const git = async (argv: readonly string[], hoursAgo = 72): Promise<string> => { @@ -443,3 +444,28 @@ function decideAtCalls(source: string): readonly string[] { } return found; } + +describe('unit · release-due · refreshTags', () => { + const answering = + (ok: boolean, calls: string[][]): GitRunner => + async (command) => { + calls.push([...command]); + const result = { command, durationMs: 1, timedOut: false }; + if (command.includes('--is-shallow-repository')) { + return { ...result, code: 0, ok: true, output: 'false' }; + } + return { ...result, code: ok ? 0 : 1, ok, output: ok ? '' : 'could not read from remote' }; + }; + + test('fetches every tag from origin and has nothing to report', async () => { + const calls: string[][] = []; + expect(await refreshTags('/repo', answering(true, calls))).toBeUndefined(); + expect(calls.at(-1)).toEqual(['git', 'fetch', '--quiet', '--tags', 'origin']); + }); + + test('a fetch that failed refuses: stale tags would read as awaiting_tag', async () => { + const finding = await refreshTags('/repo', answering(false, [])); + expect(finding?.code).toBe('X_RELEASE_WINDOW_UNREADABLE'); + expect(finding?.cause).toContain('could not read from remote'); + }); +}); diff --git a/scripts/release-due.ts b/scripts/release-due.ts index c5b894f0d..b4f3e6a43 100644 --- a/scripts/release-due.ts +++ b/scripts/release-due.ts @@ -11,6 +11,7 @@ import { flagString, parseScriptArgs } from './lib/args'; import { BASE_REF, baseRef, FETCH_MAIN, type GitRunner } from './lib/base-ref'; import { loadChangelog } from './lib/changelog-store'; +import { fetchFromOrigin } from './lib/git-fetch'; import type { Finding } from './lib/log'; import { report } from './lib/log'; import { nextMajorWindow } from './lib/major-window'; @@ -28,7 +29,7 @@ import { } from './lib/release-due-rule'; import { packageFiles, shippingPackage } from './lib/release-due-ships'; import { askIssueArgs } from './lib/release-subject'; -import { repoRoot } from './lib/run'; +import { repoRoot, run } from './lib/run'; import { ScriptError } from './lib/script-error'; import { listWorkspaces, publishFloorFindings, publishOrder } from './lib/workspaces'; import type { TagProbe } from './registry-audit'; @@ -399,12 +400,29 @@ export function nextLaneLines(lane: NextLaneDecision): readonly string[] { ]; } +/** + * Bring the release tags up to date before anything reads them. A maintainer session's checkout + * fetches branches with `--no-tags` and may not run `git fetch` itself, and a stale tag list reads + * the tree as stamped past its last tag: `awaiting_tag`, a release that is not coming. + */ +export async function refreshTags( + root: string, + runner: GitRunner = run, +): Promise<Finding | undefined> { + const fetched = await fetchFromOrigin(root, ['--quiet', '--tags', 'origin'], runner); + return fetched.ok ? undefined : gitUnreadable('git fetch --tags origin', fetched.output); +} + if (import.meta.main) { const args = parseScriptArgs(Bun.argv.slice(2)); const settle = settleHoursOf(flagString(args, 'settle-hours')); if (typeof settle !== 'number') { report({ ok: false, script: SCRIPT, summary: 'refused', findings: [settle] }, args.json); } + const stale = await refreshTags(repoRoot()); + if (stale !== undefined) { + report({ ok: false, script: SCRIPT, summary: 'cannot tell', findings: [stale] }, args.json); + } const read = await decideAt(repoRoot(), settle, Date.now()); if (!('decision' in read)) { report({ ok: false, script: SCRIPT, summary: 'cannot tell', findings: [read] }, args.json); diff --git a/scripts/template-codemods-rows.ts b/scripts/template-codemods-rows.ts index b0cdac143..74d442bd1 100644 --- a/scripts/template-codemods-rows.ts +++ b/scripts/template-codemods-rows.ts @@ -9,8 +9,48 @@ export type TemplateCodemodRow = | { readonly template: string; readonly since: string; readonly codemod: string } | { readonly template: string; readonly since: string; readonly none: string }; +const BRIEF_TYPECHECK_WHY = + "an agent brief now names bunx x verify --only typecheck, which queues in the machine ledger; an existing app's brief still names bun run typecheck, which keeps working, and its .claude files are the app's to edit, so x upgrade leaves them alone"; + export const TEMPLATE_CODEMOD_ROWS: readonly TemplateCodemodRow[] = [ ...TEMPLATE_CODEMOD_HISTORY, + { + template: 'packages/cli/src/templates/scaffold-docs.ts', + since: '31.48.0', + codemod: 'docs-many-agents', + }, + { template: 'CLAUDE.md', since: '31.48.0', codemod: 'docs-many-agents' }, + { template: 'AGENTS.md', since: '31.48.0', codemod: 'docs-many-agents' }, + { + template: 'packages/cli/src/templates/scaffold-claude-agents.ts', + since: '31.48.0', + none: BRIEF_TYPECHECK_WHY, + }, + { + template: 'packages/cli/src/templates/scaffold-claude-commands.ts', + since: '31.48.0', + none: BRIEF_TYPECHECK_WHY, + }, + { + template: '.claude/agents/data.md', + since: '31.48.0', + none: BRIEF_TYPECHECK_WHY, + }, + { + template: '.claude/agents/server.md', + since: '31.48.0', + none: BRIEF_TYPECHECK_WHY, + }, + { + template: '.claude/agents/web.md', + since: '31.48.0', + none: BRIEF_TYPECHECK_WHY, + }, + { + template: '.claude/commands/feature.md', + since: '31.48.0', + none: BRIEF_TYPECHECK_WHY, + }, { template: 'packages/cli/src/templates/github/ci.yml.ts', since: '31.47.0', diff --git a/scripts/template-codemods.lock.json b/scripts/template-codemods.lock.json index 35dbc976f..9c156b78a 100644 --- a/scripts/template-codemods.lock.json +++ b/scripts/template-codemods.lock.json @@ -1,10 +1,10 @@ { "agent:.claude/README.md": "ee3a8fa6f1cabf9b", - "agent:.claude/agents/data.md": "c6d2d6e93ab57798", - "agent:.claude/agents/server.md": "afc575c3ba844823", + "agent:.claude/agents/data.md": "2d5edbd8b1fcf023", + "agent:.claude/agents/server.md": "6218350ef9d2a8e6", "agent:.claude/agents/shape.md": "bfcbc2af27675e76", - "agent:.claude/agents/web.md": "62b87156442398bb", - "agent:.claude/commands/feature.md": "a5390ffaef9df048", + "agent:.claude/agents/web.md": "2ea661febf79b8f2", + "agent:.claude/commands/feature.md": "f00026bc5bfb9095", "agent:.claude/commands/planx.md": "04060cc861439344", "agent:.claude/commands/verify.md": "8a09370eb3f4a6da", "agent:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -17,8 +17,8 @@ "agent:.github/workflows/image.yml": "09bfb04a5defec73", "agent:.gitignore": "8334474f1b1de99f", "agent:.mcp.json": "9767907974bc0333", - "agent:AGENTS.md": "1e1f008c73003db2", - "agent:CLAUDE.md": "80745b55a504094f", + "agent:AGENTS.md": "0eeb991509caa2b0", + "agent:CLAUDE.md": "a0baef9d76f1fbbc", "agent:README.md": "0a7edbe57ed90dac", "agent:app.config.ts": "67a281212900080d", "agent:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -211,11 +211,11 @@ "agent:types/scss.d.ts": "5121f2468ef324b3", "agent:x.verify.json": "c83d8442f43d3831", "api:.claude/README.md": "ee3a8fa6f1cabf9b", - "api:.claude/agents/data.md": "c6d2d6e93ab57798", - "api:.claude/agents/server.md": "afc575c3ba844823", + "api:.claude/agents/data.md": "2d5edbd8b1fcf023", + "api:.claude/agents/server.md": "6218350ef9d2a8e6", "api:.claude/agents/shape.md": "bfcbc2af27675e76", - "api:.claude/agents/web.md": "62b87156442398bb", - "api:.claude/commands/feature.md": "a5390ffaef9df048", + "api:.claude/agents/web.md": "2ea661febf79b8f2", + "api:.claude/commands/feature.md": "f00026bc5bfb9095", "api:.claude/commands/planx.md": "04060cc861439344", "api:.claude/commands/verify.md": "8a09370eb3f4a6da", "api:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -228,8 +228,8 @@ "api:.github/workflows/image.yml": "09bfb04a5defec73", "api:.gitignore": "8334474f1b1de99f", "api:.mcp.json": "9767907974bc0333", - "api:AGENTS.md": "1e1f008c73003db2", - "api:CLAUDE.md": "03b7cb947e4b5858", + "api:AGENTS.md": "0eeb991509caa2b0", + "api:CLAUDE.md": "c1207f84ee8c20a8", "api:README.md": "fbd552a917a3a2f4", "api:app.config.ts": "dd327b93afdb9a34", "api:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -378,11 +378,11 @@ "api:types/scss.d.ts": "5121f2468ef324b3", "api:x.verify.json": "7f5302deada4608a", "bare:.claude/README.md": "ee3a8fa6f1cabf9b", - "bare:.claude/agents/data.md": "c6d2d6e93ab57798", - "bare:.claude/agents/server.md": "afc575c3ba844823", + "bare:.claude/agents/data.md": "2d5edbd8b1fcf023", + "bare:.claude/agents/server.md": "6218350ef9d2a8e6", "bare:.claude/agents/shape.md": "bfcbc2af27675e76", - "bare:.claude/agents/web.md": "62b87156442398bb", - "bare:.claude/commands/feature.md": "a5390ffaef9df048", + "bare:.claude/agents/web.md": "2ea661febf79b8f2", + "bare:.claude/commands/feature.md": "f00026bc5bfb9095", "bare:.claude/commands/planx.md": "04060cc861439344", "bare:.claude/commands/verify.md": "8a09370eb3f4a6da", "bare:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -395,8 +395,8 @@ "bare:.github/workflows/image.yml": "09bfb04a5defec73", "bare:.gitignore": "8334474f1b1de99f", "bare:.mcp.json": "9767907974bc0333", - "bare:AGENTS.md": "1e1f008c73003db2", - "bare:CLAUDE.md": "1e1f008c73003db2", + "bare:AGENTS.md": "0eeb991509caa2b0", + "bare:CLAUDE.md": "0eeb991509caa2b0", "bare:README.md": "da7cc4b66b09c502", "bare:app.config.ts": "dd327b93afdb9a34", "bare:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -536,11 +536,11 @@ "bare:types/scss.d.ts": "5121f2468ef324b3", "bare:x.verify.json": "4a3d5c621844ea46", "blog:.claude/README.md": "ee3a8fa6f1cabf9b", - "blog:.claude/agents/data.md": "c6d2d6e93ab57798", - "blog:.claude/agents/server.md": "afc575c3ba844823", + "blog:.claude/agents/data.md": "2d5edbd8b1fcf023", + "blog:.claude/agents/server.md": "6218350ef9d2a8e6", "blog:.claude/agents/shape.md": "bfcbc2af27675e76", - "blog:.claude/agents/web.md": "62b87156442398bb", - "blog:.claude/commands/feature.md": "a5390ffaef9df048", + "blog:.claude/agents/web.md": "2ea661febf79b8f2", + "blog:.claude/commands/feature.md": "f00026bc5bfb9095", "blog:.claude/commands/planx.md": "04060cc861439344", "blog:.claude/commands/verify.md": "8a09370eb3f4a6da", "blog:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -553,8 +553,8 @@ "blog:.github/workflows/image.yml": "09bfb04a5defec73", "blog:.gitignore": "8334474f1b1de99f", "blog:.mcp.json": "9767907974bc0333", - "blog:AGENTS.md": "1e1f008c73003db2", - "blog:CLAUDE.md": "b9a12d911da7e724", + "blog:AGENTS.md": "0eeb991509caa2b0", + "blog:CLAUDE.md": "8a714913cb91bc13", "blog:README.md": "97f9d8c9b0fa380d", "blog:app.config.ts": "dd327b93afdb9a34", "blog:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -696,11 +696,11 @@ "blog:types/scss.d.ts": "5121f2468ef324b3", "blog:x.verify.json": "4a3d5c621844ea46", "content:.claude/README.md": "ee3a8fa6f1cabf9b", - "content:.claude/agents/data.md": "c6d2d6e93ab57798", - "content:.claude/agents/server.md": "afc575c3ba844823", + "content:.claude/agents/data.md": "2d5edbd8b1fcf023", + "content:.claude/agents/server.md": "6218350ef9d2a8e6", "content:.claude/agents/shape.md": "bfcbc2af27675e76", - "content:.claude/agents/web.md": "62b87156442398bb", - "content:.claude/commands/feature.md": "a5390ffaef9df048", + "content:.claude/agents/web.md": "2ea661febf79b8f2", + "content:.claude/commands/feature.md": "f00026bc5bfb9095", "content:.claude/commands/planx.md": "04060cc861439344", "content:.claude/commands/verify.md": "8a09370eb3f4a6da", "content:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -713,8 +713,8 @@ "content:.github/workflows/image.yml": "09bfb04a5defec73", "content:.gitignore": "8334474f1b1de99f", "content:.mcp.json": "9767907974bc0333", - "content:AGENTS.md": "1e1f008c73003db2", - "content:CLAUDE.md": "d33e06fcc8cc1948", + "content:AGENTS.md": "0eeb991509caa2b0", + "content:CLAUDE.md": "2f72cabbc534e89d", "content:README.md": "96c5c7483f34e07c", "content:app.config.ts": "dd327b93afdb9a34", "content:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -854,11 +854,11 @@ "content:types/scss.d.ts": "5121f2468ef324b3", "content:x.verify.json": "4a3d5c621844ea46", "dashboard:.claude/README.md": "ee3a8fa6f1cabf9b", - "dashboard:.claude/agents/data.md": "c6d2d6e93ab57798", - "dashboard:.claude/agents/server.md": "afc575c3ba844823", + "dashboard:.claude/agents/data.md": "2d5edbd8b1fcf023", + "dashboard:.claude/agents/server.md": "6218350ef9d2a8e6", "dashboard:.claude/agents/shape.md": "bfcbc2af27675e76", - "dashboard:.claude/agents/web.md": "62b87156442398bb", - "dashboard:.claude/commands/feature.md": "a5390ffaef9df048", + "dashboard:.claude/agents/web.md": "2ea661febf79b8f2", + "dashboard:.claude/commands/feature.md": "f00026bc5bfb9095", "dashboard:.claude/commands/planx.md": "04060cc861439344", "dashboard:.claude/commands/verify.md": "8a09370eb3f4a6da", "dashboard:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -871,8 +871,8 @@ "dashboard:.github/workflows/image.yml": "09bfb04a5defec73", "dashboard:.gitignore": "8334474f1b1de99f", "dashboard:.mcp.json": "9767907974bc0333", - "dashboard:AGENTS.md": "1e1f008c73003db2", - "dashboard:CLAUDE.md": "2e9ad4f17a776f69", + "dashboard:AGENTS.md": "0eeb991509caa2b0", + "dashboard:CLAUDE.md": "f1c3b3941e5100d0", "dashboard:README.md": "95d149363f07393c", "dashboard:app.config.ts": "dd327b93afdb9a34", "dashboard:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -1050,11 +1050,11 @@ "dashboard:types/scss.d.ts": "5121f2468ef324b3", "dashboard:x.verify.json": "c83d8442f43d3831", "example:.claude/README.md": "ee3a8fa6f1cabf9b", - "example:.claude/agents/data.md": "c6d2d6e93ab57798", - "example:.claude/agents/server.md": "afc575c3ba844823", + "example:.claude/agents/data.md": "2d5edbd8b1fcf023", + "example:.claude/agents/server.md": "6218350ef9d2a8e6", "example:.claude/agents/shape.md": "bfcbc2af27675e76", - "example:.claude/agents/web.md": "62b87156442398bb", - "example:.claude/commands/feature.md": "a5390ffaef9df048", + "example:.claude/agents/web.md": "2ea661febf79b8f2", + "example:.claude/commands/feature.md": "f00026bc5bfb9095", "example:.claude/commands/planx.md": "04060cc861439344", "example:.claude/commands/verify.md": "8a09370eb3f4a6da", "example:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -1067,8 +1067,8 @@ "example:.github/workflows/image.yml": "09bfb04a5defec73", "example:.gitignore": "8334474f1b1de99f", "example:.mcp.json": "9767907974bc0333", - "example:AGENTS.md": "1e1f008c73003db2", - "example:CLAUDE.md": "1e1f008c73003db2", + "example:AGENTS.md": "0eeb991509caa2b0", + "example:CLAUDE.md": "0eeb991509caa2b0", "example:README.md": "da7cc4b66b09c502", "example:app.config.ts": "dd327b93afdb9a34", "example:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -1241,11 +1241,11 @@ "example:types/scss.d.ts": "5121f2468ef324b3", "example:x.verify.json": "c83d8442f43d3831", "offline:.claude/README.md": "ee3a8fa6f1cabf9b", - "offline:.claude/agents/data.md": "c6d2d6e93ab57798", - "offline:.claude/agents/server.md": "afc575c3ba844823", + "offline:.claude/agents/data.md": "2d5edbd8b1fcf023", + "offline:.claude/agents/server.md": "6218350ef9d2a8e6", "offline:.claude/agents/shape.md": "bfcbc2af27675e76", - "offline:.claude/agents/web.md": "62b87156442398bb", - "offline:.claude/commands/feature.md": "a5390ffaef9df048", + "offline:.claude/agents/web.md": "2ea661febf79b8f2", + "offline:.claude/commands/feature.md": "f00026bc5bfb9095", "offline:.claude/commands/planx.md": "04060cc861439344", "offline:.claude/commands/verify.md": "8a09370eb3f4a6da", "offline:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -1258,8 +1258,8 @@ "offline:.github/workflows/image.yml": "09bfb04a5defec73", "offline:.gitignore": "8334474f1b1de99f", "offline:.mcp.json": "9767907974bc0333", - "offline:AGENTS.md": "1e1f008c73003db2", - "offline:CLAUDE.md": "115f69e459664190", + "offline:AGENTS.md": "0eeb991509caa2b0", + "offline:CLAUDE.md": "d804542360ece3f6", "offline:README.md": "a009d6acd81a6f78", "offline:app.config.ts": "dd327b93afdb9a34", "offline:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -1434,11 +1434,11 @@ "offline:types/scss.d.ts": "5121f2468ef324b3", "offline:x.verify.json": "c83d8442f43d3831", "realtime:.claude/README.md": "ee3a8fa6f1cabf9b", - "realtime:.claude/agents/data.md": "c6d2d6e93ab57798", - "realtime:.claude/agents/server.md": "afc575c3ba844823", + "realtime:.claude/agents/data.md": "2d5edbd8b1fcf023", + "realtime:.claude/agents/server.md": "6218350ef9d2a8e6", "realtime:.claude/agents/shape.md": "bfcbc2af27675e76", - "realtime:.claude/agents/web.md": "62b87156442398bb", - "realtime:.claude/commands/feature.md": "a5390ffaef9df048", + "realtime:.claude/agents/web.md": "2ea661febf79b8f2", + "realtime:.claude/commands/feature.md": "f00026bc5bfb9095", "realtime:.claude/commands/planx.md": "04060cc861439344", "realtime:.claude/commands/verify.md": "8a09370eb3f4a6da", "realtime:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -1451,8 +1451,8 @@ "realtime:.github/workflows/image.yml": "09bfb04a5defec73", "realtime:.gitignore": "8334474f1b1de99f", "realtime:.mcp.json": "9767907974bc0333", - "realtime:AGENTS.md": "1e1f008c73003db2", - "realtime:CLAUDE.md": "67b92d3f56ea4c9a", + "realtime:AGENTS.md": "0eeb991509caa2b0", + "realtime:CLAUDE.md": "2c2bd929d45f2ba1", "realtime:README.md": "df96f5e65ee1a4c1", "realtime:app.config.ts": "dd327b93afdb9a34", "realtime:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", @@ -1635,11 +1635,11 @@ "realtime:types/scss.d.ts": "5121f2468ef324b3", "realtime:x.verify.json": "c83d8442f43d3831", "saas:.claude/README.md": "ee3a8fa6f1cabf9b", - "saas:.claude/agents/data.md": "c6d2d6e93ab57798", - "saas:.claude/agents/server.md": "afc575c3ba844823", + "saas:.claude/agents/data.md": "2d5edbd8b1fcf023", + "saas:.claude/agents/server.md": "6218350ef9d2a8e6", "saas:.claude/agents/shape.md": "bfcbc2af27675e76", - "saas:.claude/agents/web.md": "62b87156442398bb", - "saas:.claude/commands/feature.md": "a5390ffaef9df048", + "saas:.claude/agents/web.md": "2ea661febf79b8f2", + "saas:.claude/commands/feature.md": "f00026bc5bfb9095", "saas:.claude/commands/planx.md": "04060cc861439344", "saas:.claude/commands/verify.md": "8a09370eb3f4a6da", "saas:.claude/hooks/format-edited.ts": "95565acd928432ab", @@ -1652,8 +1652,8 @@ "saas:.github/workflows/image.yml": "09bfb04a5defec73", "saas:.gitignore": "8334474f1b1de99f", "saas:.mcp.json": "9767907974bc0333", - "saas:AGENTS.md": "1e1f008c73003db2", - "saas:CLAUDE.md": "3729705310fe53fb", + "saas:AGENTS.md": "0eeb991509caa2b0", + "saas:CLAUDE.md": "eb0a633ec8c64d12", "saas:README.md": "dc8163d58d553a53", "saas:app.config.ts": "dd327b93afdb9a34", "saas:apps/admin/app/admin/admin.test.ts": "673a0113401e5f4f", diff --git a/scripts/typecheck-locked.test.ts b/scripts/typecheck-locked.test.ts new file mode 100644 index 000000000..39fea3a66 --- /dev/null +++ b/scripts/typecheck-locked.test.ts @@ -0,0 +1,26 @@ +// The root `typecheck` script is the ledgered one: a raw `tsc -b` there would let an agent's compiler +// race the gate's over one `dist/`, with no entry in the machine ledger to explain the memory. +import { describe, expect, setDefaultTimeout, test } from 'bun:test'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { REPO_SCAN_TIMEOUT_MS, repoRoot } from './lib/run'; +import { typecheckArgv } from './typecheck-locked'; + +setDefaultTimeout(REPO_SCAN_TIMEOUT_MS); + +describe('unit · bun run typecheck', () => { + test('the root script goes through the ledger, and no other script builds', async () => { + const manifest: { scripts: Record<string, string> } = await Bun.file( + join(repoRoot(), 'package.json'), + ).json(); + expect(manifest.scripts['typecheck']).toBe('bun run scripts/typecheck-locked.ts'); + const rawBuilds = Object.entries(manifest.scripts).filter( + ([, command]) => /\btsc -b\b/.test(command) && !command.includes('--clean'), + ); + expect(rawBuilds).toEqual([]); + }); + + test('flags reach the compiler after its own', () => { + expect(typecheckArgv('tsc', ['--force'])).toEqual(['tsc', '-b', '--pretty', '--force']); + }); +}); diff --git a/scripts/typecheck-locked.ts b/scripts/typecheck-locked.ts index 5e45eca84..d63a45cea 100644 --- a/scripts/typecheck-locked.ts +++ b/scripts/typecheck-locked.ts @@ -1,47 +1,47 @@ #!/usr/bin/env bun -// `bun run typecheck:locked`: `tsc -b` under one repo-wide lock. Agents share one checkout, and two -// unserialised `tsc -b` runs write the same `.tsbuildinfo` and `dist/` and report errors that -// belong to neither. Portable where `flock(1)` was Linux-only (`scripts/lib/file-lock.ts`). +// `bun run typecheck`: `tsc -b` with its measured memory reserved in the machine ledger and this +// checkout's one-compiler lock held. Agents share one checkout, and two unserialised `tsc -b` runs +// write the same `.tsbuildinfo` and `dist/` and report errors that belong to neither. // -// bun run typecheck:locked +// bun run typecheck [<tsc -b flag>…] -// why: one lock-file path under the repo root; Bun ships no path API. -import { join } from 'node:path'; -import { acquireLock, pidAlive } from './lib/file-lock'; +import { withCheckoutLock, withHeavy } from '../packages/cli/src/heavy'; +import { TYPECHECK_BUILD_PEAK_MB } from '../packages/cli/src/heavy-weights'; +import { handOff } from '../packages/cli/src/local-cli-handoff'; import { localBin } from './lib/local-bin'; import { repoRoot } from './lib/run'; -/** Under `node_modules/.cache`: ignored by git, and gone with the install it guards. */ -export const TSC_LOCK = join('node_modules', '.cache', 'tsc.lock'); - -/** A full `tsc -b` from cold is minutes; a run past this has hung, and its lock must not wedge. */ +/** A full `tsc -b` from cold is minutes; a run past this has hung. */ const TSC_TIMEOUT_MS = 30 * 60_000; +/** The compiler's command line; what the caller passed goes last, as `tsc -b` once took it. */ +export const typecheckArgv = (tsc: string, passed: readonly string[]): readonly string[] => [ + tsc, + '-b', + '--pretty', + ...passed, +]; + if (import.meta.main) { const root = repoRoot(); - const path = join(root, TSC_LOCK); - await Bun.write(join(root, 'node_modules', '.cache', '.keep'), ''); - const release = await acquireLock(path, { - pid: process.pid, - isAlive: pidAlive, - sleep: (ms) => Bun.sleep(ms), - onWait: (holder) => console.error(`typecheck:locked: waiting on pid ${holder} (${TSC_LOCK})`), - }); - // An interrupted run gives the lock back rather than leaving the next one to prove it dead. - for (const signal of ['SIGINT', 'SIGTERM'] as const) { - process.on(signal, () => { - release(); - process.exit(130); - }); - } - try { - const child = Bun.spawn([localBin(root, 'tsc'), '-b', '--pretty'], { - cwd: root, - stdio: ['inherit', 'inherit', 'inherit'], - timeout: TSC_TIMEOUT_MS, - }); - process.exitCode = await child.exited; - } finally { - release(); - } + const env = Bun.env; + process.exitCode = await withHeavy( + { label: 'typecheck', cwd: root, env, unitMb: TYPECHECK_BUILD_PEAK_MB }, + (held) => + withCheckoutLock({ name: 'typecheck', root, env }, held, (child) => + handOff( + typecheckArgv(localBin(root, 'tsc'), Bun.argv.slice(2)), + { ...env, ...child.env }, + { + spawn: (command, childProcessEnv) => + Bun.spawn([...command], { + cwd: root, + env: childProcessEnv, + stdio: ['inherit', 'inherit', 'inherit', ...child.fds], + timeout: TSC_TIMEOUT_MS, + }), + }, + ), + ), + ); } diff --git a/wiki/Building-Your-Own-Base.md b/wiki/Building-Your-Own-Base.md index d81de8226..9a522a825 100644 --- a/wiki/Building-Your-Own-Base.md +++ b/wiki/Building-Your-Own-Base.md @@ -191,14 +191,16 @@ an operator then proves who they are with a short PIN, and that browser's sessio | the operator's PIN | the app's own entity (`operators`: `userId`, `pinHash`), never `x_users.password_hash` — a PIN is weaker than the password policy allows (`X_PASSWORD_WEAK`) | `hashPassword(pin)` on enrolment | | the lockout | **before** the KDF, keyed per operator and per device: a 4-digit PIN is 10 000 guesses, so the budget is the whole defence | `appAuth().limiter.reserve('pin:<operatorId>:<deviceId>')` — `X_ACCOUNT_LOCKED` refuses; `recordSuccess` on a match | | the check | constant cost whether or not the operator exists | `verifyPassword({ hash: row?.pinHash ?? null, password: pin })` | -| the switch | a session for the operator, capped at the device session's ceiling, set as that browser's cookie; signing the operator out lands on the device sign-in again | `createSession(appAuth().sessions, { userId: operator.userId, absoluteExpiresAt: device.absoluteExpiresAt })`, then `sessionCookie(issued.token, appAuth().sessions.policy)` on the response | +| the switch | a session for the operator, capped at the device session's ceiling, set as that browser's cookie; signing the operator out lands on the device sign-in again | `createSession(appAuth().sessions, { userId: operator.userId, absoluteExpiresAt: device.absoluteExpiresAt })`, then `setCookie(sessionCookie(issued.token, appAuth().sessions.policy))` from the action | | who may switch | the app's policy on its own `switchOperator` action: the caller holds the `device` role | `can('operator:switch', …)` | The action is the app's (`apps/web/app/operators/actions/switch-operator.ts`), so it projects like any other — route, typed client, MCP tool, contract test. Nothing in the framework learns that a PIN exists, which is the point: the next app's PIN rules (length, rotation, which devices) are its own. -The check itself, one file the action calls with the operator row it loaded: +The check itself, one file the action calls with the operator row it loaded. `switchOperator` knows +nothing of HTTP, so a job or a test can run it; the action's `handle` calls +`switchOperatorOnResponse`, and no route carries the cookie: ```ts // apps/web/shared/base/operator-pin.ts @@ -212,6 +214,7 @@ import { sessionCookie, verifyPassword, } from '@ultimat3/auth'; +import { setCookie } from '@ultimat3/http'; /** The app's `operators` row: whose PIN, and its hash. Never `x_users.password_hash`. */ export interface OperatorRow { @@ -253,6 +256,16 @@ export async function switchOperator( }); return { issued, cookie: sessionCookie(issued.token, auth.sessions.policy) }; } + +/** The action's half: the same switch, its cookie set on the response to this request. */ +export async function switchOperatorOnResponse( + auth: Auth, + input: OperatorSwitch, +): Promise<IssuedSession> { + const { issued, cookie } = await switchOperator(auth, input); + setCookie(cookie); + return issued; +} ``` The comparison is the KDF's, constant-time inside `Bun.password.verify`; a PIN is never hashed diff --git a/wiki/CI-Parallel-Gate.md b/wiki/CI-Parallel-Gate.md index 5153221e6..3973a4521 100644 --- a/wiki/CI-Parallel-Gate.md +++ b/wiki/CI-Parallel-Gate.md @@ -254,9 +254,8 @@ Each job sizes its own test width the same way the laptop does (`x verify` print line, e.g. `3 workers (budget 4.0 GB)`): a budget of `max(2.75 GiB, 25% of the runner's RAM)`, 1.25 GiB planned per worker, never more workers than cores. On a 16 GB `ubuntu-latest` that is 3 workers; a 62 GB, 16-core workstation plans 12 (`As of 2026-11-07`, when the fixed 4 GiB cap went — -the machine-wide slot pool is what keeps two runs on one box inside one budget). `ULTIMATE_TEST_MEMORY_BUDGET=6g` or `ULTIMATE_TEST_MAX_WORKERS=4` override it; `--workers` -wins over both. The machine-wide slot pool (see [Testing](Testing#memory-width-and-isolation)) -has nothing to share on a one-job VM; `ULTIMATE_TEST_SLOTS=0` turns it off. +the [machine ledger](Many-Agents) is what keeps two runs on one box inside one budget). `ULTIMATE_TEST_MEMORY_BUDGET=6g` or `ULTIMATE_TEST_MAX_WORKERS=4` override it; `--workers` +wins over both. The [machine ledger](Many-Agents) has nothing to share on a one-job VM and is off under `CI`. ## Skipping the build's duplicate preflight diff --git a/wiki/CLI-Reference.md b/wiki/CLI-Reference.md index def7e81fc..8da802d61 100644 --- a/wiki/CLI-Reference.md +++ b/wiki/CLI-Reference.md @@ -72,6 +72,7 @@ x version # CLI version | `x cache graph\|bust <tag>\|clear` | the tag graph off the manifest (what each write evicts), a bust of one tag or every declared tag through the app's own tiers and bus · `stats` was removed `As of 2026-11-15` ([Known gaps](Known-Gaps#open-by-decision)) | shipped | | `x explain <query> [--input <json>] [--org <id>]` | one declared query run once on the dev database, every statement it sent planned by the `exercise` step's oracle | shipped | | `x profile <route> [--runs 20] [--top 10]` | a CPU profile of rendering one route in this process: the hottest functions, file and line | shipped | +| `x heavy [status]` · `x heavy run [--label s] [--mb n] -- <command…>` | the machine's shared memory ledger: who runs, who is queued, load and memory; any command run inside one reservation, its exit code passed through | shipped | | `x status [--url <origin>] [--samples n]` | the build each live worker runs (`x_job_workers`), and the build, status and role each sampled origin's `/readyz` answers | shipped | ## x new @@ -1288,16 +1289,16 @@ x doctor [--port 3000] [--json] Checks Bun version, env completeness, source drift, **the newest migration's `.snapshot.json`**, port availability, PWA prerequisites and **embedded-Postgres readiness** — each failing check carries its own fix command. -It also lists the shipped guards the app holds that differ from the current template — a copy from an older release, or the app's own edit — and warns `X_SHIPPED_GUARD_MISSING` for each one the app's `guards/` does not hold, its fix `x g guard <name> && x verify --only boundaries --json` (`As of 31.22`: a warning beside the verdict in `data.warnings[]`, so the exit status is unchanged; it was a listing). `As of 2026-10-07` (owner decision 8: shipped guards stay copies the app owns). +It also lists the copies of shipped guards the app's `guards/` still holds, each with `X_GUARD_VENDORED` beside the verdict in `data.warnings[]`: shipped guards run from the installed framework, so a copy runs in place of the shipped one and never receives a fix. `As of 31.28`. No shipped guard is ever missing, so `data.guards.missing` is always `[]`. | Listing | One line each | `--json` | |---|---|---| -| missing | its `X_SHIPPED_GUARD_MISSING` warning line (`As of 31.22`) | `data.guards.missing[]`: `{ name, add }`, and the warning in `data.warnings[]` | -| differs from the template (line endings ignored) | `x g guard <name> --force` — rewrites the guard and its test; review the diff, and restore your edit if it was deliberate | `data.guards.differs[]`: `{ name, refresh }` | +| a copy, identical or edited | the edit that resolves it: `rm -f` for an identical copy; for an edited one, `mv` to `guards/app-<name>.ts` and `"<name>": "off"` in `guards/shipped.json` | `data.guards.vendored[]`: every copy's name | +| an edited copy | the same line | `data.guards.differs[]`: `{ name, refresh, diff }` — `refresh` is that edit | -A listing, never a finding: deleting or editing a guard is how an app changes a rule, so the verdict does not move. +A listing, never a finding: the verdict does not move. `x upgrade` applies the same migration (`vendored-guards` codemod); how a shipped guard reports is a row in `guards/shipped.json`. -Four warnings about how the app is held, `As of 31.16.0` (a fifth, `X_SHIPPED_GUARD_MISSING`, `As of 31.22`) — beside the verdict, never in it: the exit status, `findings` and `data.codes` stay as they were, and `--json` carries each as `data.warnings[]` (`{ code, cause, fix, docs, at }`). None is a gate step; on `next` they join the verdict, red: +Warnings about how the app is held — beside the verdict, never in it: the exit status, `findings` and `data.codes` stay as they were, and `--json` carries each as `data.warnings[]` (`{ code, cause, fix, docs, at }`). None is a gate step; on `next` they join the verdict, red: | Code | Asked | Fix | |---|---|---| @@ -1305,7 +1306,7 @@ Four warnings about how the app is held, `As of 31.16.0` (a fifth, `X_SHIPPED_GU | `X_VERIFY_STEP_UNFLOORED` | a gate step applies here — its suite files exist, its input is committed — and `x.verify.json` `"steps"` does not hold it, so a deleted suite reads as skipped. `e2e` is never proposed (the scaffold's placeholder skips until a browser driver is registered) | `x verify --only <steps> --json`, then add each step that passed to `"steps"` | | `X_FRAMEWORK_PIN_RANGE` | an `@ultimat3/*` dependency in any of the app's manifests is anything but an exact version or a `workspace:`/`file:`/`link:` path (`^31.0.0`, `*`, `latest`, `>=31 <32`; an `npm:` alias by its own spec), one finding per manifest | the exact version `bun install` resolved, at the line named; then `bun install` | | `X_ENV_NAME_RESERVED` | a name the app declares in `envSchema` or seals in `secrets.enc.json` is one `FRAMEWORK_ENV` (`@ultimat3/core`) lists as a `switch` or `internal` name — setting it for the app also switches that package's behaviour. Deploy-contract names (`DATABASE_URL`, `PORT`) and host names are not collisions | rename it `APP_<name>` (or delete it where the value is meant for the framework); then `x env example` | -| `X_SHIPPED_GUARD_MISSING` (`As of 31.22`) | a guard `x new` ships is not in the app's `guards/`, so the gate does not hold the app to that rule — one per guard | `x g guard <name> && x verify --only boundaries --json` | +| `X_GUARD_VENDORED` (`As of 31.28`) | the app's `guards/` holds a copy of a shipped guard, which runs in place of the built-in one and never receives a fix — one per copy. `boundaries` lists it too | the resolving edit above | **Redundant indexes**, the same way (`As of 2026-11-14`): each implicit index the entities describe that another index, the primary key or a unique invariant already leads with, still in the newest migration's snapshot, is one line — `<table>.<index> (<columns>) — <covering index>` — under a header naming `x db gen "drop redundant indexes" --drop-redundant`; `--json`: `data.redundantIndexes[]` = `{ table, name, columns, coveredBy }`, `[]` when none, absent when the app would not load or has no snapshot. A listing, never a finding: every write pays for each twice, and dropping it is a migration the app opts into. @@ -1614,9 +1615,11 @@ sampled throughout. | `roll-before-migrate` | the roles started before `ROLE=migrate`, on a schema stamped by an older build | `roles-wait`, `serves-after-migrate`, `one-wait-line` | `roll-before-migrate`, `As of 2026-11-15`, is the rollout order a deploy cannot always hold: no -proxy and no load. The scratch database is migrated, its framework stamp (`x_jobs`' table comment) +fault and no load. The scratch database is migrated, its framework stamp (`x_jobs`' table comment) is rewound to name only an older build, then `web`, `worker` and `scheduler` start on it without -waiting for readiness. Once each has logged its wait (or exited), they are held 3 s and +waiting for readiness, with `sync` when `TEST_NATS_URL` is set and `replicator` when +`TEST_REPLICATION_URL` names a logical server (the scratch database then lives there) and the app +has an entity on a table. A role left out is in `data.rolesNotPlayed[]`, `{ drill, role, needs }`. Once each has logged its wait (or exited), they are held 3 s and `ROLE=migrate` runs again. Each must stay up (`roles-wait`: an exit is a crash-loop), log exactly one line naming `X_FRAMEWORK_SCHEMA_UNAPPLIED` (`one-wait-line`), and answer `/readyz` 200 once migrate has run (`serves-after-migrate`). `ULTIMATE_SCHEMA_WAIT_MS` is inherited: `0` — refuse at @@ -1635,7 +1638,7 @@ property is red when no job ran during the drill, `sync-resumes` when no sync cl A service the app uses must be up for every drill; an unset one is refused, never skipped. A named drill that faults a service the app does not declare is refused; a whole run lists it under `data.notRun`. `--list` prints the table and runs nothing. `--json`: `{ budgets, targets, -skippedWrites, drills[], notRun }`, each drill `{ name, verdict, evidence, budgets, baseline }`. +skippedWrites, drills[], notRun, rolesNotPlayed }`, each drill `{ name, verdict, evidence, budgets, baseline }`. Errors: `X_DRILL_FAILED` (one per property that did not hold), `X_DRILL_SERVICE_MISSING`, `X_CLI_UNKNOWN_COMMAND`, `X_CLI_BAD_FLAG`. @@ -1760,8 +1763,8 @@ One major per run, so each hop's gate attributes its red to that hop. `As of 30. | 2 | read every workspace's `@ultimat3/*` pins, `overrides` and `resolutions` included | `X_UPGRADE_PIN_UNREADABLE`, `X_UPGRADE_MIXED_PINS` | | 3 | `bun pm view @ultimat3/cli versions`: the next major's newest; the last major's newest when no later major exists | `X_UPGRADE_REGISTRY_UNREACHABLE`, `X_UPGRADE_HOP_TOO_WIDE`, `X_UPGRADE_VERSION_UNPUBLISHED`, `X_BUN_VERSION` | | 4 | write exact pins, raise the Bun floor | — | -| 5 | `bun install` | `X_UPGRADE_INSTALL_FAILED`, `fix:` carries the undo | -| 6 | the installed CLI: `x upgrade codemods --from <from> --to <to>` (below), then `x manifest`, `x doctor --json`, each missing guard (`x g guard <name>`, kept only when `x verify --only boundaries` finds nothing more), a Bun-floor hop's `x verify --only unit` coverage rebaseline, `x exercise --json` and its `exercise` floor figures, `x verify --json` | `X_UPGRADE_GATE_UNREADABLE`, as a finding on a red hop | +| 5 | `bun install`, then `bun update <names>` for each transitive package `bun.lock` holds under an advisory floor (`X_ADVISORY_FLOOR_UNMET`) | `X_UPGRADE_INSTALL_FAILED`, `fix:` carries the undo | +| 6 | the installed CLI: `x upgrade codemods --from <from> --to <to>` (below), then `x manifest`, `x doctor --json`, each guard that doctor reports missing (only a hop landing before 31.28: `x g guard <name>`, kept when `x verify --only boundaries` finds nothing more), a Bun-floor hop's `x verify --only unit` coverage rebaseline, `x exercise --json` and its `exercise` floor figures, `x verify --json` | `X_UPGRADE_GATE_UNREADABLE`, as a finding on a red hop | A dry run refuses exactly what the real run would, then stops before step 4 — after previewing this CLI's codemods for the span, written nowhere. It never commits; `git diff` is the hop. Under `--json` fd 1 is one document, fd 2 a `{"upgrade":"<stage>"}` line per stage and a heartbeat a minute. @@ -1779,7 +1782,7 @@ A dry run refuses exactly what the real run would, then stops before step 4 — | `pins[]` | `{ file, name, from, to }` | | `bun` | `{ from, to, edited[], manual[] }`. `manual[]`: `{ path, line, text, why }`, a site only you can move, listed on every run | | `agentsMd` | `link`: `AGENTS.md` was byte-identical to `CLAUDE.md` and the hop makes it a relative symlink to it · `linked`: already one · `differs`: two different files, left alone; merge what only `AGENTS.md` says into `CLAUDE.md`, then `ln -sf CLAUDE.md AGENTS.md` · `none`: one of the two is missing · `null`: no hop planned | -| `guards` | `{ reported, missing[], differs[], installed[], withheld[] }` from the hop's own `x doctor`. `As of 31.16.0` each missing shipped guard is written from the installed release's template and kept when `boundaries` finds nothing more with it on disk: `installed[]` names it. One the tree already breaks is taken back out: `withheld[]` is `{ name, add, findings }` (`findings: null` when the count could not be measured) and `missing[]` still lists it. `differs[]` — the app's edit or an older copy — is never overwritten: `x g guard <name> --force` is yours. `reported: false` is a CLI too old to list them, which is not the same as none | +| `guards` | `{ reported, missing[], differs[], installed[], withheld[], baselined[]? }` from the hop's own `x doctor`. A hop landing on 31.28 or later has `missing[]`, `installed[]` and `withheld[]` empty: shipped guards run built in. Landing earlier (`As of 31.16.0`), each missing shipped guard is written from the installed release's template and kept when `boundaries` finds nothing more with it on disk: `installed[]` names it; one the tree already breaks is taken back out: `withheld[]` is `{ name, add, findings }` (`findings: null` when the count could not be measured) and `missing[]` still lists it. `differs[]` is `{ name, refresh }`: an edited copy the hop's `vendored-guards` codemod left, never overwritten, `refresh` the edit that resolves it. `reported: false` is a CLI too old to list them, which is not the same as none | | `guards.baselined` | `[{ name, findings }]`, absent when empty, `As of 2026-11-15`: a guard the tree breaks, kept anyway — its measured count merged into `x.verify.json` as `"guardBaseline": { "<name>": n }` (beside the app's keys; a baseline the app already wrote for it, or a floor that is not a JSON object, is never touched) and kept only when the installed CLI's `boundaries` then finds no more than before. Also in `installed[]`. `findings: null` stays withheld | | `coverage` | a hop that moved the Bun floor re-measures unit coverage (`x verify --only unit`); only when that measurement is `X_COVERAGE_BELOW_FLOOR` is the `x.verify.json` `coverage` floor moved to it, the number that fell alone, with a dated `why`: `{ file, measured, from, to, why }`. `null` when nothing moved | | `exerciseFloor` | `{ file, written }`: the `exercise` keys (`unprovenMax`, `statementsMax` ≥ 100) the hop's own `x exercise` measured, written into an `x.verify.json` that stated none, so the hop stays green and the floor only falls. A key the app already wrote is never touched; `null` when nothing was written | @@ -2164,6 +2167,15 @@ For each violation involving the file it reports the offending edge, the full ch Every flag is read before the database opens: a refusal costs no boot. +## x heavy + +`As of 2026-11-18`. `x heavy [status] [--json]` · `x heavy run [--label s] [--mb n] [--json] -- <command…>` — the machine ledger every heavy run on this machine reserves its measured memory in, from any directory, in any repository. The rules, the weights and what stays outside: [Working with many agents](Many-Agents). + +| Form | Answers | +|---|---| +| `x heavy` / `x heavy status` | `data.holds` and `data.queue` (label, directory, MB, pid, age), `data.capacityMb`, `data.usedMb`, `data.coreCap`, `data.machine` (load, runnable tasks, available and usable memory); `data.enabled: false` with `data.off` naming why (`ci`, `test-run`, `disabled`, `unsupported`) | +| `x heavy run -- <command…>` | runs it inside one reservation of `--mb` (default: a monorepo typecheck). Exit code passed through, `128 + n` for a signal, 127 and `X_HEAVY_COMMAND_NOT_FOUND` for no such program. SIGTERM and SIGHUP are forwarded. Its summary goes to stderr, so stdout stays the command's; with `--json` the one document is on stdout and the command's stdout moves to stderr | + ## x status `As of 2026-11-15`. `x status [--url <origin>[,<origin>…]] [--samples n] [--json]` — what the running deployment is, from what it already records. A read, exit 0 whatever it finds: whether one build is fully out is [`x deploy verify`](#x-deploy-verify)'s question. diff --git a/wiki/Caching-And-Invalidation.md b/wiki/Caching-And-Invalidation.md index 60cb232b4..f1a0b3128 100644 --- a/wiki/Caching-And-Invalidation.md +++ b/wiki/Caching-And-Invalidation.md @@ -302,7 +302,7 @@ From code: `invalidateTags([...])` from `@ultimat3/cache`, the call `x cache bus |---|---|---| | `X_CACHE_UNTAGGED_QUERY` | **reserved, nothing raises it** `As of 2026-08` — a query's tables are covered by no tag, so it could never be invalidated ([Error codes → Reserved codes](Error-Codes#reserved-codes)) | declare the entity tag, then `x manifest` | | `X_CACHE_TAG_UNKNOWN` | `tag "<name>" is not declared by any entity` | `declareTags(['<name>'])` beside the app's other declared tags, plus the name in its `CacheTagRegistry`; for a typo, correct the tag | -| `X_CACHE_TOO_LARGE` | `entry "<key>" is <n>B, over the <tier> budget of <m>B` | `lruTier({ maxBytes: <n> })` / `redisTier({ maxValueBytes: <n> })` where the tier is built, or cache a projection instead of the row | +| `X_CACHE_TOO_LARGE` | `entry "<key>" is <n>B, over the <tier> budget of <m>B` | `cacheTiers: [lruTier({ maxBytes: <n> })]` / `cacheTiers: [redisTier({ maxValueBytes: <n> })]` in `apps/web/runtime.ts` ([limits](Configuration#cache)), or cache a projection instead of the row | | `X_CACHE_DRIVER_UNAVAILABLE` | `cache tier "<driver>" is unavailable` — a Redis command missed its deadline, or the tier is parked after one; or a purge driver built without its token | the error carries the exact config or command to fix | | `X_CACHE_PURGE_FAILED` | `<driver> refused the purge (HTTP <status>)` — a wrong token, a zone without tag purge, a throttle, or a key a CDN would split | `meta.retryable === true` → the identical purge can land again; otherwise set the env key the `fix` names | diff --git a/wiki/Configuration.md b/wiki/Configuration.md index 9ebafff01..3c608ae2e 100644 --- a/wiki/Configuration.md +++ b/wiki/Configuration.md @@ -231,14 +231,31 @@ its config into a variable first. | ~~`cache.driver`~~ | — | — | **Deleted in 9.0.0.** Refused at boot since 25.0.0 (`X_CONFIG_INVALID`, naming the replacement). It was the second way to ask for Redis and the losing one: the ladder is built from `cache.tiers`, so `driver: 'redis'` beside `tiers: ['request-memo', 'lru']` asked for a rung nothing built. Name `redis` in `tiers` | | ~~`cache.urlEnv`~~ | — | — | **Deleted in 9.0.0** and refused at boot since 25.0.0, `database.urlEnv`'s defect verbatim: the Redis tier reads the literal `REDIS_URL`, so `urlEnv: 'MY_REDIS'` made nothing read `MY_REDIS` | -**The per-tier byte caps and TTLs are constructor options, not config** — the same shape as the realtime caps above. `cache.memo.maxBytes`, `cache.lru.maxBytes`, `cache.redis.*` and `cache.ttl.*` are not fields and never were; writing one is `TS2353`, excess property on `Input<CacheConfig>`, and `X_CONFIG_INVALID` at boot. +**A rung's limits are constructor options, set in `runtime.ts`, never config** `As of 2026-11`. The boot builds `lru` and `redis` with their defaults; `cacheTiers` replaces the rung of the same name, and that is the one way to change a limit: + +```ts +// apps/web/runtime.ts — `cache.tiers` still decides which rungs exist +import { lruTier, redisTier } from '@ultimat3/cache'; + +export const runtime = { + cacheTiers: [ + lruTier({ maxBytes: 128 * 1024 * 1024 }), + redisTier({ maxValueBytes: 2 * 1024 * 1024, jitterFraction: 0.1 }), + ], +}; +``` + +`cache.memo.maxBytes`, `cache.lru.maxBytes`, `cache.redis.*` and `cache.ttl.*` are not fields and never were: writing one is `TS2353`, excess property on `Input<CacheConfig>`, and `X_CONFIG_INVALID` at boot. A bad value is refused where the tier is built (`X_CACHE_LIMIT_INVALID`, `X_CACHE_JITTER_INVALID`). There is no entry-count limit: both rungs are bounded in bytes. + +A supplied `redisTier` with no `client` uses core's client: `REDIS_URL`, then `VALKEY_URL`, then `redis://localhost:6379`. The refusal the boot's own `redis` rung gets for an unset URL does not apply to it; the boot logs `cache.tier.default-redis` instead. | Option | Where | Default | Effect | |---|---|---|---| | `maxBytes` | `lruTier({ … })` | 64 MiB | byte budget for the whole tier; a single value over it is `X_CACHE_TOO_LARGE` rather than a silent drop | -| `defaultTtlMs` | same | `60_000` | applied when a `set` omits `ttlMs` | -| `jitterFraction` | same | `DEFAULT_TTL_JITTER_FRACTION` | TTL spread in `[0, 1)`; `0` disables it, which is how a stampede is reproduced in a test | -| `clock` / `rng` | same | system | injected so a jittered expiry is deterministic | +| `maxValueBytes` | `redisTier({ … })` | 1 MiB | the largest encoded value one `set` may write; over it is `X_CACHE_TOO_LARGE` | +| `defaultTtlMs` | `lruTier({ … })` | `60_000` | applied when a `set` omits `ttlMs` | +| `jitterFraction` | `lruTier({ … })`, `redisTier({ … })` | `DEFAULT_TTL_JITTER_FRACTION` | TTL spread in `[0, 1)`; `0` disables it, which is how a stampede is reproduced in a test | +| `clock` / `rng` | `lruTier({ … })`, `redisTier({ … })` | system | injected so a jittered expiry is deterministic | | `loadDeadlineMs` | `cacheStack(tiers, { … })` | `30_000` | how long one `load()` may hold its key before a later reader starts its own. Frees the KEY, never the load — the wedged call runs on. Anchored to `http`'s own request timeout: a load still running at 30 s has no reader left to serve | | `schedule` | same | `setTimeout` | injected so the deadline above is provable without a test waiting one out | @@ -526,6 +543,7 @@ mail: { retainMime: { maxBytes: 262_144 } }, | `SES_CONFIGURATION_SET` | — | SES only, optional: the configuration set that publishes delivery events | | `MAIL_FROM` | — | required by SMTP, Resend and SES; the memory and unconfigured drivers do not read it. `Name <addr>`; also the envelope sender and the `Message-ID` domain | | `MAIL_POOL_SIZE` | — | SMTP connections open at once. Default `4`, whole number ≥ 1 | +| `MAIL_TIMEOUT_MS` | — | SMTP only: the deadline for any single read or write. Default `30000`, whole number from 1 to 2147483647 (the most a timer holds); anything else is `X_CONFIG_INVALID` at boot | The list is `MAIL_ENV_KEYS` in `packages/mail/src/driver-env.ts`, `As of 2026-10`. More than one of `SMTP_URL`, `RESEND_API_KEY` and `SES_REGION` is `X_CONFIG_INVALID`: a process delivers through diff --git a/wiki/Error-Codes-cache.md b/wiki/Error-Codes-cache.md index f5592cc8c..d0d5865c1 100644 --- a/wiki/Error-Codes-cache.md +++ b/wiki/Error-Codes-cache.md @@ -5,10 +5,10 @@ Part of [Error codes](Error-Codes): every code `@ultimat3/cache` registers. `x e | Code | Means | Typical cause | Fix | |---|---|---|---| | `X_CACHE_DRIVER_UNAVAILABLE` | a tier's backing store is missing | no Redis binding, or a purge driver built without `FASTLY_API_TOKEN` / `CLOUDFLARE_API_TOKEN` | provision the tier, or drop it from `app.config.ts` | -| `X_CACHE_JITTER_INVALID` | a TTL jitter fraction outside `[0, 1)` | `cache.<tier>.jitterFraction` at `1` or above, negative, or not finite. Jitter exists because 40,000 keys warmed by one rolling restart share one TTL and therefore one expiry instant, and spreading them is the only thing that stops the herd — a fraction of `1` shaves a whole lease away, a negative one **extends** it past what the caller asked for. Both are miswiring rather than preference, refused where the TTL rule already lives | `{ jitterFraction: 0.05 }` in `lruTier({ … })` / `redisTier({ … })` — a value in `[0, 1)`; `0.05` is the default, `0` disables jitter. A constructor option, never an `app.config.ts` key | -| `X_CACHE_LIMIT_INVALID` | a tier is given a numeric limit outside the domain its option accepts, refused at **construction**. Three domains, and the refusal names which: a **capacity** (`maxEntries`, `maxBytes`) is a whole number greater than zero; a **duration** (`defaultTtlMs`, `loadDeadlineMs`, a driver's request budget) is a whole number of milliseconds greater than zero; a **similarity floor** is a number from 0 to 1. So finite zero, a negative, a fraction and an unsafe integer are all refused too — not only `NaN` and `Infinity` | `maxEntries`, `maxBytes`, `defaultTtlMs`, `loadDeadlineMs` or a similarity floor arriving as `NaN` or `Infinity` — most often `Number(process.env.CACHE_LIMIT)` on an unset variable, which is `NaN` and is **not** nullish, so `??` does not catch it and `Math.max`/`Math.min` propagate it. The failure is the dangerous direction: a comparison against `NaN` is false for **every** entry, so the limit stops being enforced rather than being enforced wrongly — an unbounded cache that reports healthy. The cause names that mechanism **only for `NaN`**: the same refusal also catches `Infinity`, `0`, a fraction and an unsafe integer, where the bound is applied as given rather than disabled — a ceiling of `Infinity` is never exceeded and a similarity floor of `Infinity` is never met, which are opposite outcomes one sentence cannot claim at once. Raised at construction rather than on the first write, where `X_CACHE_TTL_INVALID` would have named the write instead of the config | `{ maxBytes: 10_000 }` — the fix pastes the corrected option, in the factory that read it: `lruTier()`, `redisTier()`, `memorySemanticCache()`, `fastlyPurgeDriver()`, `cloudflarePurgeDriver()`, `cacheStack()`, or the `lookup(embedding, override)` argument. None is an `app.config.ts` key. Parse an environment value first: `Number(process.env.CACHE_LIMIT)` is `NaN` when the variable is unset. `bun run finite-bounds` is the repo-wide rule | +| `X_CACHE_JITTER_INVALID` | a TTL jitter fraction outside `[0, 1)` | `jitterFraction` at `1` or above, negative, or not finite, refused when the tier is built. Jitter exists because 40,000 keys warmed by one rolling restart share one TTL and therefore one expiry instant, and spreading them is the only thing that stops the herd — a fraction of `1` shaves a whole lease away, a negative one **extends** it past what the caller asked for. Both are miswiring rather than preference, refused where the TTL rule already lives | `{ jitterFraction: 0.05 }` in `lruTier({ … })` / `redisTier({ … })` — a value in `[0, 1)`; `0.05` is the default, `0` disables jitter. For the rung a boot builds: `cacheTiers: [redisTier({ jitterFraction: 0.05 })]` in `apps/web/runtime.ts`. A constructor option, never an `app.config.ts` key | +| `X_CACHE_LIMIT_INVALID` | a tier is given a numeric limit outside the domain its option accepts, refused at **construction**. Three domains, and the refusal names which: a **capacity** (`maxEntries`, `maxBytes`) is a whole number greater than zero; a **duration** (`defaultTtlMs`, `loadDeadlineMs`, a driver's request budget) is a whole number of milliseconds greater than zero; a **similarity floor** is a number from 0 to 1. So finite zero, a negative, a fraction and an unsafe integer are all refused too — not only `NaN` and `Infinity` | `maxEntries`, `maxBytes`, `defaultTtlMs`, `loadDeadlineMs` or a similarity floor arriving as `NaN` or `Infinity` — most often `Number(process.env.CACHE_LIMIT)` on an unset variable, which is `NaN` and is **not** nullish, so `??` does not catch it and `Math.max`/`Math.min` propagate it. The failure is the dangerous direction: a comparison against `NaN` is false for **every** entry, so the limit stops being enforced rather than being enforced wrongly — an unbounded cache that reports healthy. The cause names that mechanism **only for `NaN`**: the same refusal also catches `Infinity`, `0`, a fraction and an unsafe integer, where the bound is applied as given rather than disabled — a ceiling of `Infinity` is never exceeded and a similarity floor of `Infinity` is never met, which are opposite outcomes one sentence cannot claim at once. Raised at construction rather than on the first write, where `X_CACHE_TTL_INVALID` would have named the write instead of the config | `{ maxBytes: 10_000 }` — the fix pastes the corrected option, in the factory that read it: `lruTier()`, `redisTier()`, `memorySemanticCache()`, `fastlyPurgeDriver()`, `cloudflarePurgeDriver()`, `cacheStack()`, or the `lookup(embedding, override)` argument. For the rung a boot builds, the factory call goes in `cacheTiers` in `apps/web/runtime.ts`. None is an `app.config.ts` key. Parse an environment value first: `Number(process.env.CACHE_LIMIT)` is `NaN` when the variable is unset. `bun run finite-bounds` is the repo-wide rule | | `X_CACHE_PURGE_FAILED` | the CDN refused a purge | a wrong or unscoped API token, a zone without tag purge, a throttle, a key carrying whitespace or a comma | `meta.retryable === true` → the identical purge can land again; `false` → set the env key the `fix` names, then `x dev` | | `X_CACHE_TAG_UNKNOWN` | a tag no entity declared | typo in `invalidates: [tag.pots]`, or a new tag the app never declared. Raised only once the app has declared at least one tag | `declareTags(['pots'])` in the module that already calls it (`packages/db/src/tags.ts` in the reference app), plus `pots: true` in its `CacheTagRegistry`; for a typo, change the tag at its use to a name the cause lists | -| `X_CACHE_TOO_LARGE` | one entry exceeds the tier's byte budget | caching a whole row set | `lruTier({ maxBytes: <n> })` or `redisTier({ maxValueBytes: <n> })` where the tier is built, or cache a projection | +| `X_CACHE_TOO_LARGE` | one entry exceeds the tier's byte budget | caching a whole row set | `cacheTiers: [lruTier({ maxBytes: <n> })]` or `cacheTiers: [redisTier({ maxValueBytes: <n> })]` in `apps/web/runtime.ts`, or cache a projection | | `X_CACHE_TTL_INVALID` | a cache TTL that is not a positive number of milliseconds | `ttlMs: 0`, a negative value, `NaN` or `Infinity`. `0` used to mean "never expires" in the memory tier and "one second" in the Redis tier, so a stack holding both answered differently depending on which tier hit | pass a positive `ttlMs`, or omit it and inherit the tier's default — "do not cache" is expressed by not declaring a `cache` block | | `X_CACHE_VALUE_UNENCODABLE` | a cached value the cache codec cannot encode | a value holding a cycle, a toJSON that throws, nesting deeper than 512 levels or a bigint longer than 4096 characters, written to any tier — refused at the write, so a value is either stored and readable on every tier or not stored at all | `cache a projection of the value: plain JSON plus Date, bigint, Map and Set, with no cycles` | diff --git a/wiki/Error-Codes-cli.md b/wiki/Error-Codes-cli.md index bd4322384..c2d8c8af8 100644 --- a/wiki/Error-Codes-cli.md +++ b/wiki/Error-Codes-cli.md @@ -115,6 +115,9 @@ Part of [Error codes](Error-Codes): every code `@ultimat3/cli` registers. `x err | `X_GUARD_INVALID` | a file in `guards/` exports no usable guard | the module has no `guard` export, or one missing `summary`/`check`. Discovery is the directory — nothing registers a guard, so a file that looks like one and is not must say so loudly rather than being skipped | `x verify --json` — then export a `guard` from the file the finding names | | `X_GUARD_OVER_BASELINE` | a guard reports more findings than its x.verify.json baseline allows | guards/<g>.ts reports more findings than x.verify.json's guardBaseline allows for it; the baseline never rises | `x verify --only boundaries --json` | | `X_GUARD_VENDORED` | a guards/ copy of a shipped guard | the copy runs in place of the guard the installed framework ships, and never receives its fixes | `x upgrade --json` | +| `X_HEAVY_COMMAND_NOT_FOUND` | x heavy run was given a command that is not on PATH | the first word after -- resolves to no executable from the working directory | `x heavy run --json -- bun test` | +| `X_HEAVY_LEDGER_UNAVAILABLE` | the machine memory ledger cannot be used here | the ledger directory is unsafe or unwritable, or its filesystem refuses flock(2); the run continues without admission control | `x heavy status --json` | +| `X_HEAVY_MEMORY_INVALID` | ULTIMATE_HEAVY_MEMORY is not a size | the variable holds something other than a byte size such as 8g, or 0 to turn the ledger off | `export ULTIMATE_HEAVY_MEMORY=8g` | | `X_IMAGE_NOT_PREBUILT` | a container role built at boot what the image build should have: island chunks or stylesheets | an image whose Dockerfile predates the prebuilt step, or one built from a context that carried a store for another Bun or framework version | `edit docker/Dockerfile: add RUN bun node_modules/@ultimat3/cli/src/bin.ts build --target prebuilt after COPY . . in the runtime stage, then rebuild the image` | | `X_IMAGE_SOURCE_UNREADABLE` | a responsiveImage src is not a file the static export holds | responsiveImage({ src }) on a static page named a site path the export does not contain, so no variant of it can be written | `x build --target static --json` — lists the export; put the file under apps/web/site/assets/ and pass asset('assets/<file>') as src, or give an absolute https:// URL | | `X_ISLAND_DISPOSER_MISSING` | an island's mount returns no disposer | an .island.tsx mount() is typed void or returns nothing, so the client router cannot dispose it on navigation and its listeners, hotkeys and live reads stay registered | `x verify --only boundaries --json` — names the file; end its `mount()` with `return mountOver(el, (root) => render(() => <Island {...props} />, root))` (`mountOver` from `@ultimat3/render/client`), the disposer the client router calls on navigation | diff --git a/wiki/Error-Codes.md b/wiki/Error-Codes.md index 6b56451cf..e7e89241e 100644 --- a/wiki/Error-Codes.md +++ b/wiki/Error-Codes.md @@ -39,7 +39,7 @@ Generated from the owner pages by `bun run scripts/error-code-pages.ts --write`; | `@ultimat3/ai` | [Error-Codes-ai](Error-Codes-ai) | 36 | | `@ultimat3/auth` | [Error-Codes-auth](Error-Codes-auth) | 30 | | `@ultimat3/cache` | [Error-Codes-cache](Error-Codes-cache) | 8 | -| `@ultimat3/cli` | [Error-Codes-cli](Error-Codes-cli) | 220 | +| `@ultimat3/cli` | [Error-Codes-cli](Error-Codes-cli) | 223 | | `@ultimat3/core` | [Error-Codes-core](Error-Codes-core) | 76 | | `@ultimat3/db` | [Error-Codes-db](Error-Codes-db) | 41 | | `@ultimat3/entity` | [Error-Codes-entity](Error-Codes-entity) | 44 | @@ -567,6 +567,9 @@ Generated from the owner pages by `bun run scripts/error-code-pages.ts --write`; | X_GUARD_OVER_BASELINE | [cli](Error-Codes-cli) | | X_GUARD_VENDORED | [cli](Error-Codes-cli) | | X_HEALTH_ROW_UNREADABLE | [scripts](Error-Codes-scripts) | +| X_HEAVY_COMMAND_NOT_FOUND | [cli](Error-Codes-cli) | +| X_HEAVY_LEDGER_UNAVAILABLE | [cli](Error-Codes-cli) | +| X_HEAVY_MEMORY_INVALID | [cli](Error-Codes-cli) | | X_HELPER_COPY | [scripts](Error-Codes-scripts) | | X_HIVE_EMPTY | [ai](Error-Codes-ai) | | X_HTTP_PATH_INPUT_CONFLICT | [core](Error-Codes-core) | diff --git a/wiki/Home.md b/wiki/Home.md index ac226fe97..1033ad19d 100644 --- a/wiki/Home.md +++ b/wiki/Home.md @@ -91,6 +91,7 @@ Follow in order. Each page states what it was executed against in its own first | [Client navigation](Client-Navigation) | soft navigation over server-rendered pages: one config line per surface, swap, prefetch, history, focus | | [MCP and AI](MCP-And-AI) | the dev MCP server, every action as a tool, the `llm()` gateway, evals | | [Agents](Agents) | `agent()` as an action factory, tools as real actions, `hive()`, `agentJob()`, and the at-least-once trap | +| [Working with many agents](Many-Agents) | a worker runs its own files, the coordinator runs the gate once; the machine's memory ledger and core cap | | [Admin dashboard](Admin-Dashboard) | `defineAdmin()` serving list, detail and form per entity with no host code; filters, scopes, sections, related rows, actions with `when` and `batch`, the audit log, the jobs dashboard; zero JavaScript; its MCP surface | | [Scraping](Scraping) | `scrape()` as a job factory, the driver-blind page vocabulary, robots and host gates, per-run egress, a rented CDP browser, usage, sealed sessions, mid-run prompts | | [Auth](Auth) | sessions, OAuth, MFA, API keys and service tokens — the output is an `Actor` | diff --git a/wiki/Interface-Rules.md b/wiki/Interface-Rules.md index 74532d1e9..2a97430b6 100644 --- a/wiki/Interface-Rules.md +++ b/wiki/Interface-Rules.md @@ -346,7 +346,7 @@ Read as a review checklist. Each one is a defect on sight. | live regions and the focus trap | `packages/ui/src/a11y.ts` | | the toast queue's rules | `packages/ui/src/toast/toast-state.ts` | | a list's URL state (sort, cursor, limit, `q`) | `packages/ui/src/list-url.ts` | -| your app's guards | `guards/` in the app root — `x new` writes every name in `SHIPPED_GUARD_NAMES`, `x g guard <name>` adds one, `x doctor` lists the shipped ones missing | +| your app's guards | `guards/` in the app root — `x new` writes only `guards/shipped.json`, `x g guard <name>` adds one of your own, `x doctor` lists any copy of a shipped guard still held | Components, props and token vocabulary: [UI components](UI-Components). Colour roles and the contrast table: [Theming](Theming). Seeing a component in a state you cannot click to: diff --git a/wiki/Known-Gaps.md b/wiki/Known-Gaps.md index d99f1b5fd..d8d78185d 100644 --- a/wiki/Known-Gaps.md +++ b/wiki/Known-Gaps.md @@ -83,7 +83,7 @@ Each of these is a defect, or a target, somebody has already argued about, and t | Retry-from-step on the jobs dashboard is a free-text field | `As of 2026-10`. `job.retry-from-step` takes `{ step: t.string.min(1) }` (`packages/admin/src/jobs/job-actions.ts`); the run's detail page shows its steps, but the form does not offer them as a choice. **Why it stays** (`As of 2026-11-18`): An action's `input` is one static schema shared by the form, the MCP tool and the audit entry, so it has no per-row list of choices to offer. **Do instead:** copy the step name from the run's `steps` section on the same page. | | The admin MCP batch tool takes `ids` only | `As of 2026-10`. A `batch` action is one MCP tool, `admin.action.<name>`, and it takes the rows by id; "all matching" — the list's own filter — is a screen control only. **Why it stays** (`As of 2026-11-18`): An agent names every row it changes, so the per-row gate and the `<entity>:<n> rows` confirmation state a set it chose. "All matching" stays a control a person reads the list beside. **Do instead:** list the rows with `admin.<entity>.list` and its `where`, then pass their ids. | | No counts on the jobs dashboard's state tabs | `As of 2026-10`. `stats()` counts per queue across every tenant, so a count on a tab would show an org-scoped operator other orgs' numbers; the tabs carry none and the overview's tiles carry the counts, for a platform operator only. A per-tenant count is a second query nothing issues yet. **Why it stays** (`As of 2026-11-18`): A per-tenant count per tab is one more read per state on every page view, over a column with no leading index (the org-scoped runs-list row). **Do instead:** a platform operator reads the overview tiles; an org-scoped operator reads a tab's own page. | -| `x drill` proves one node per role on one machine, and drills no object store and no CDN | `As of 2026-11-06`. The fleet is one `web`, one `worker` (and one `sync` for `nats-down`) on one host, every dependency a local container behind a Bun TCP proxy: a multi-node NATS cluster, a Postgres failover to a real standby, a real S3 bucket and a real CDN in front of the web role are never faulted. `nats-down`'s sync lane is proven on a stand-in node (`packages/testing/src/drill-sync.test.ts`), not on `examples/dummy`, whose transport is in-memory; its client subscribes to nothing by default, so a gap counts only what the node volunteers. CI runs four of the nine (`fault-drills`, `roll-before-migrate` among them) [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/testing/src/drills.test.ts#the%20nine%20drills%2C%20once%20each) **Why it stays** (`As of 2026-11-18`): A drill runs where a CI runner or one machine can hold the whole fleet. A real standby, bucket or CDN is a staging environment, which is the app's. **Do instead:** the rest by hand: `x drill --list --json`, then `x drill <name> --json` with `docker/test-services.env` loaded. A multi-node or object-store fault is a staging drill of your own, against [`docs/ops/06-runbooks.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/ops/06-runbooks.md) | +| `x drill` proves one node per role on one machine, and drills no object store and no CDN | `As of 2026-11-06`. The fleet is one `web`, one `worker` (one `sync` for `nats-down`, and one of every role for `roll-before-migrate`) on one host, every dependency a local container behind a Bun TCP proxy: a multi-node NATS cluster, a Postgres failover to a real standby, a real S3 bucket and a real CDN in front of the web role are never faulted. `nats-down`'s sync lane is proven on a stand-in node (`packages/testing/src/drill-sync.test.ts`), not on `examples/dummy`, whose transport is in-memory; its client subscribes to nothing by default, so a gap counts only what the node volunteers. CI runs four of the nine (`fault-drills`, `roll-before-migrate` among them) [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/testing/src/drills.test.ts#the%20nine%20drills%2C%20once%20each) **Why it stays** (`As of 2026-11-18`): A drill runs where a CI runner or one machine can hold the whole fleet. A real standby, bucket or CDN is a staging environment, which is the app's. **Do instead:** the rest by hand: `x drill --list --json`, then `x drill <name> --json` with `docker/test-services.env` loaded. A multi-node or object-store fault is a staging drill of your own, against [`docs/ops/06-runbooks.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/ops/06-runbooks.md) | | `x cache`'s `stats` | removed from the registry `As of 2026-11-15`, where it was planned and exited `X_NOT_IMPLEMENTED`; now `X_CLI_UNKNOWN_COMMAND`, exit 1 as before. A hit rate is `cache.hit`/`cache.miss` on each serving process's instrumentation bus (`packages/cache/src/tiers.ts`), and no process aggregates them: the `/_x` cache panel shows the tag graph and the invalidation log, not hits. The route is a counter on each role's `/metrics`, read by whatever scrapes it; a CLI process has served nothing to count | | `x status` for sync, scheduler and replicator | `As of 2026-11-15`, `x status` reads every live worker's build from `x_job_workers` and samples each `--url` origin's `/readyz` for `x-ultimate-build`. No fleet-wide record exists for the other roles: a sync node keeps its connected clients' builds in its own memory (the `/_x` live panel in `x dev`), and the scheduler and replicator write no row. Each answers `/readyz` on its health port, so `x status --url http://<pod>:9091` reads one pod; the build-ID distribution of connected clients needs an aggregate the realtime package does not keep (`data.unrecorded` names these roles) | | `api/**/route.ts` | not a file kind — owner decision 1 ([#648](https://github.com/developerz-ai/ultimate/issues/648)), `As of 2026-10`. `api/` holds the action and query projections `defineApi()` collects; a route file there is refused at registration (`X_ROUTE_FILE_INVALID`), and 26.0.0 deleted `ROUTE_FILENAME`'s `api` row ([#709](https://github.com/developerz-ai/ultimate/issues/709)). A wire format an action cannot speak — a webhook receiver, an OAuth endpoint — is a plain HTTP route in the `routes` runtime override ([Configuration](Configuration)) | diff --git a/wiki/Mail.md b/wiki/Mail.md index d3a98af8f..c7b64953e 100644 --- a/wiki/Mail.md +++ b/wiki/Mail.md @@ -55,7 +55,7 @@ between environments — the credential does. |---|---| | nothing set, `development` / `test` | `memoryMailDriver()` — caught in the `/_x` mail panel, never sent | | nothing set, `staging` / `production` | `unconfiguredMailDriver(...)` — every send is `X_MAIL_CREDENTIAL_MISSING`; the boot still succeeds, so an app that sends no mail deploys | -| `SMTP_URL` + `MAIL_FROM` | `smtpMailDriver(...)` — ESMTP over `Bun.connect`, STARTTLS required unless `allowInsecure` | +| `SMTP_URL` + `MAIL_FROM` | `smtpMailDriver(...)` — ESMTP over `Bun.connect`, STARTTLS required unless `allowInsecure`. `MAIL_POOL_SIZE`, `MAIL_TIMEOUT_MS` optional | | `RESEND_API_KEY` + `MAIL_FROM` | `resendMailDriver(...)` — one `POST /emails` with an `Idempotency-Key` | | `SES_REGION` + `SES_ACCESS_KEY_ID` + `SES_SECRET_ACCESS_KEY` + `MAIL_FROM` | `sesMailDriver(...)` — one SES v2 `SendEmail` with the raw MIME, SigV4-signed, no SDK. `SES_SESSION_TOKEN`, `SES_ENDPOINT`, `SES_CONFIGURATION_SET` optional | diff --git a/wiki/Many-Agents.md b/wiki/Many-Agents.md new file mode 100644 index 000000000..5f0a29c51 --- /dev/null +++ b/wiki/Many-Agents.md @@ -0,0 +1,60 @@ +# Working with many agents + +**A parallel worker runs its own files. The coordinator runs the whole gate, once, at the end.** Every other rule on this page exists so that this one holds when several agents, in several checkouts and repositories, share one machine. `As of 2026-11-18`. + +| Who | Runs | Never runs | +|---|---|---| +| a worker with a file set | `bun test ./its/file.test.ts`, `x test --filter <path>` over up to 3 files, `biome check <its paths>` | `x verify`, `bun run verify`, a whole `x test`, `bun run coverage` | +| the coordinator | `bun run verify` (or `x verify`) once, after the workers land | a second gate beside the first | +| a read-only agent (research, explore) | one command at a time, output bounded with `head` | any build, test, typecheck or index run | + +**Read-only fan-out has its own limit.** Those agents write nothing, so they take no file set, but each is a whole session no ledger can see: its memory is outside the machine's accounting. Start a handful, not one per question. + +## The machine ledger + +One ledger per machine and user, at `/tmp/ultimate-heavy-<uid>`, shared by every checkout and every Ultimate app on the machine. It admits **half of the memory this process may use** (the smallest of RAM and every cgroup `memory.max` above it). Each heavy run reserves its measured peak first, in arrival order, and waits when it does not fit. + +| Enters the ledger | Reserves | +|---|---| +| `tsc -b` of a monorepo (`x verify`'s `typecheck`, `bun run typecheck` at this repository's root) | 6,016 MB, plus this checkout's one-compiler lock | +| `tsc -p .` of an app (`x verify --only typecheck`) | 768 MB, plus the same lock | +| `biome check .` (`x verify --only lint`) | 2,112 MB | +| every batch of `bun test` workers (`x test`, `x verify`'s suites) | 704 MB + 1,280 MB per worker | +| an e2e lane (`x test e2e`, `x verify --only e2e`) | 3,072 MB | +| one coverage unit (`bun run coverage`), one heavy `bun run preflight` step | 704 MB + 1,280 MB per worker | +| anything under `x heavy run` | `--mb`, default a monorepo typecheck | + +**What does not enter it, stated:** a bare `bun test ./path`; `x test` selecting 3 files or fewer; an app's own `bun run typecheck` and `bun run lint` scripts (raw `tsc` and `biome`: a scaffolded agent brief says `bunx x verify --only typecheck` instead); `x build`, including the one `budgets` runs; the `drift`, `exercise` and `contract-diff` steps; `x dev`; the test services in Docker; the agent sessions themselves. Those are why only half the memory is offered. + +Every entry is an `flock(2)`, and a holder hands the locked descriptor to the command it starts, so the kernel frees a reservation only when the holder **and** that command are gone. Kill a wrapper with `kill -9` and its reservation lasts until the command it started exits; a killed waiter leaves the queue at once. Descendants keep it only by inheritance: a shell's background job does, and a process started through `Bun.spawn` or `node:child_process` does not, because both close the descriptor. So a daemon a shell left running keeps the reservation until it stops (`x heavy status` names the run), and workers a killed `bun` command leaves behind are unaccounted. `As of 2026-11-18`, Bun 1.4.4. + +```sh +x heavy status # who runs, who waits, load, memory +x heavy status --json +x heavy run --label suite --mb 4096 -- bun run test # any command, in any repository +``` + +A waiter prints one line on stderr saying its position and who holds the memory, and again only when that changes, so `--json` output stays one document. `x heavy run` passes the exit code through and forwards stop signals. A process started inside a reservation inherits it (`ULTIMATE_HEAVY_HOLD`) and never queues behind its own parent. Time spent queued does not count against an `x verify` step's deadline. + +## Cores + +Locally a run takes **at most 4 cores**, fewer when other sessions' runnable tasks already fill the machine, and never below 2. Test workers are capped per batch, so a run narrows and widens as the machine changes; `tsc` and `biome` get the cap as `GOMAXPROCS` and `RAYON_NUM_THREADS`. `--workers N` is taken as written. CI takes every core. + +## Switches + +| Variable | Effect | +|---|---| +| `ULTIMATE_HEAVY_MEMORY=8g` | what the ledger admits; `0` turns it off | +| `ULTIMATE_HEAVY_CORES=6` | the local core ceiling | +| `ULTIMATE_HEAVY_DIR=<path>` | another ledger; also opts a test run in | +| `ULTIMATE_TEST_SLOTS=0` | test workers skip the ledger | + +Admission is off under `CI`, inside `bun test` (a fixture's `x test` must not queue behind real work), and on **Windows**, which has no `flock`: the core cap still applies there and `x heavy status` says why admission is off. A ledger that cannot work is reported as `X_HEAVY_LEDGER_UNAVAILABLE` and the command goes ahead; it never fails a run. + +## Re-measuring a weight + +Each figure sits in [`packages/cli/src/heavy-weights.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/src/heavy-weights.ts) beside its command. Re-read one on a clean snapshot: + +```sh +GOMAXPROCS=4 bun run scripts/heavy-peak.ts -- node_modules/.bin/tsc -b --pretty false +``` diff --git a/wiki/Testing.md b/wiki/Testing.md index deb566ff1..98d5b139b 100644 --- a/wiki/Testing.md +++ b/wiki/Testing.md @@ -1387,10 +1387,10 @@ gates at once — and it is held to a **memory budget**, not to whatever looks f what it chose and why — `3 workers (budget 4.0 GB)`. `ULTIMATE_TEST_MEMORY_BUDGET=3g` replaces the budget, `ULTIMATE_TEST_MAX_WORKERS=2` caps the width, `--workers N` wins over both. Total RAM, never "free": page cache counts as free, and two gates each planning on it took a 45 GB box down. -- **One budget per machine.** Each batch of workers leases that many slots from a pool of lock - files under the OS temp dir (one per worker the budget allows; a dead holder's slot is taken - over). A second `x test` or `x verify` on the same box gets what is left and runs narrower, or - waits for one slot, instead of doubling the memory. `ULTIMATE_TEST_SLOTS=0` turns it off. +- **One budget per machine.** Each batch of workers reserves its measured memory in the + [machine ledger](Many-Agents), at most 4 cores locally. A second `x test` or `x verify` on the + same box runs narrower or queues instead of doubling the memory; a selection of 3 files or fewer + runs free. `ULTIMATE_TEST_SLOTS=0` turns it off. - **No per-file isolation by default.** A worker keeps one global and one module registry across the files it runs (`bun test --parallel=N --no-isolate`) — measured 2-5x faster than `--isolate` at the same width. Between two files the testing preload hands the next file the process it would @@ -1521,7 +1521,7 @@ This table is a hand-synced copy of it ([Contributing](Contributing)). |---|---| | `typecheck` | any error; `any` is banned by lint, not tolerated by a cast | | `lint` | biome only: formatting, `any`, default exports, unused imports. **Not** raw colours, bare `Error`, untranslated strings or unzoned dates — biome ignores `.scss` entirely and those are guards, one row down | -| `boundaries` | `site/` → `app/`, routes → DB, services → HTTP, tier violations in framework packages, **and every file in the app's own `guards/`** — every name in `SHIPPED_GUARD_NAMES` (`packages/cli/src/templates/scaffold-guards.ts`) that `x new` writes, or the ones you kept → [Interface rules](Interface-Rules) | +| `boundaries` | `site/` → `app/`, routes → DB, services → HTTP, tier violations in framework packages, **and every guard**: every name in `SHIPPED_GUARD_NAMES` (`packages/cli/src/templates/scaffold-guards.ts`), run from the installed framework at the severity `guards/shipped.json` gives it, and every file of your own in `guards/` → [Interface rules](Interface-Rules) | | `filesize` | a source file over 500 lines | | `package-shape` | a workspace package missing `README.md`, `CLAUDE.md`, `tsconfig.json`, or `src/index.ts` | | `errors` | an `X_*` code with no runnable fix or no docs page | diff --git a/wiki/Tutorial-01-First-App.md b/wiki/Tutorial-01-First-App.md index f067b673b..223a64cd6 100644 --- a/wiki/Tutorial-01-First-App.md +++ b/wiki/Tutorial-01-First-App.md @@ -223,8 +223,8 @@ myapp/ │ └── ui/ app components on semantic tokens ├── docker/ Dockerfile, .dockerignore, dev + prod compose, helm/ ├── guards/ one rule per file, run by `x verify`'s boundaries step. -│ Every name in SHIPPED_GUARD_NAMES ships; `x doctor` -│ lists any missing. Delete one by deleting its file +│ Shipped ones run from the framework; shipped.json +│ sets each one's severity (`off` turns it off) ├── bin/ setup.ts — `bun run setup`; `dev` and `verify` are x's own └── CLAUDE.md AGENTS.md what an agent reads first: one file, AGENTS.md a symlink ``` diff --git a/wiki/_Sidebar.md b/wiki/_Sidebar.md index 78924f0c9..e643d845e 100644 --- a/wiki/_Sidebar.md +++ b/wiki/_Sidebar.md @@ -42,6 +42,7 @@ - [Client navigation](Client-Navigation) - [MCP and AI](MCP-And-AI) - [Agents](Agents) +- [Working with many agents](Many-Agents) - [Admin dashboard](Admin-Dashboard) - [Scraping](Scraping) - [Auth](Auth)