diff --git a/.dz/briefs/security-release.md b/.dz/briefs/security-release.md index f664d381a..d52406602 100644 --- a/.dz/briefs/security-release.md +++ b/.dz/briefs/security-release.md @@ -1,15 +1,18 @@ # Brief: does a confirmed security fix need a release today? -Daily checkup on the days `release-due` does not run, fired by `schedules: security-release` in +Devops session on Thursdays and Sundays, so a confirmed fix waits at most three days with the Tuesday `release-due`, fired by `schedules: security-release` in `.dz/maintainer/maintainer.yml`. This is the one off-cycle release. Everything else waits for Tuesday's `.dz/briefs/release-due.md`, and a person can still cut by hand. ## Run ```bash -git fetch --tags origin && bun run scripts/release-due.ts --json +bun run scripts/release-due.ts --json ``` +You have a shell, no `git` writes and no `gh`; the runner ships what you leave in the tree. A refusal +(`ok:false`): report its `cause` and stop. + Read `data.shipping` for what merged since the last tag, then look for a commit that fixes a security advisory. @@ -27,10 +30,6 @@ such advisory does not qualify: it waits for Tuesday unless a person cuts it. ## When it qualifies -File one ask, **Release ``**, opening with the advisory's id and source URL, then -following `.dz/briefs/release-due.md` § The ask. Its dedupes and refusals apply, and a major stays a -maintainer's. Otherwise file nothing and end your run summary with `no release: `. - -## Rows - -- `security-fix-unreleased` — A commit in `data.shipping` fixes an advisory a first-tier source confirms, naming something these packages ship, and no release carries it yet. +Follow `.dz/briefs/release-due.md` § Cut the release, opening `PR.md` and the `### Highlights` with the +advisory's id and source URL. A major stays a maintainer's. Otherwise change no file and end +`REPORT.md` with `no release: `. diff --git a/.dz/maintainer/maintainer.yml b/.dz/maintainer/maintainer.yml index 39df3f9a8..42aa2a822 100644 --- a/.dz/maintainer/maintainer.yml +++ b/.dz/maintainer/maintainer.yml @@ -139,8 +139,9 @@ schedules: brief_file: .dz/briefs/release-due.md # The one exception, so a confirmed security fix does not wait up to a week for Tuesday. - name: security-release - days: [mon, wed, thu, fri, sat, sun] + days: [thu, sun] hours_utc: [4] + mode: devops brief_file: .dz/briefs/security-release.md # The failures nothing announces: a red `main`, a publish that did not land, npm drift, a stuck # Dependabot PR. Bot-authored alarm issues never pass intake, so this is how they reach the bot. diff --git a/CHANGELOG.md b/CHANGELOG.md index 99b5f8544..d1c8dc635 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,34 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major ## [Unreleased] -Nothing yet. +### Added + +- **Merging two features:** `x merge-driver install`, run by `bun run setup`, lets two branches that each ran `x g resource` merge with no hand edit: registries keep what both sides added, catalogs merge by key, `x.manifest.json` and `openapi.json` are rebuilt by `x manifest`. New apps write one registry entry per line, and `x upgrade` applies it (the `merge-drivers` codemod). `x doctor` warns `X_MERGE_DRIVER_UNSET` on a clone without the drivers ([Many agents → Merging two features](wiki/Many-Agents.md#merging-two-features)). +- `X_MIGRATION_FORKED`: `x db gen`, `x db migrate` and the gate's `drift` step refuse a migration chain two merged branches forked, before suggesting a `record …` migration that could never replay. `x db gen --merge-fork` corrects the newest snapshot, changes no SQL and deletes nothing. +- `X_MERGE_UNRESOLVED` names the files a merge left unresolved, and `X_CATALOG_UNPARSEABLE` names an i18n catalog that does not parse, in place of a parse error blamed on every module that imports it. +- `x heavy run --max-wait ` gives up with `X_HEAVY_WAIT_EXPIRED` (exit 75); `x heavy status` names each run's wrapper and command pids (`wrapperAlive`, `childPids` in `--json`), and a heavy run also waits while the CPU is saturated and another run is admitted. +- `x clean --services` stops this app's test service stack, and the gate prints one line saying whether it started, reused or shared one. +- `x dev`'s port refusal names the directory of the process holding the port. + +### Changed + +- A fix line that ends `then: x …` prints `bunx x …`, as does the gate's stderr pointer and `x errors explain`; a shipped guard's fix re-runs its own step (`x verify --only boundaries`), never the whole gate. +- Fixes no longer send a worker to a whole run: `X_TEST_NO_FILES` suggests `x test --affected --dirty`, `X_TYPECHECK_FAILED` names the file to edit and `x verify --only typecheck`, and an unknown command whose nearest match runs every suite suggests its `--help`. +- A ledger wait says to leave the command running (`bunx x heavy status`), and reads `admission in progress` while no holder is known. +- A new app's `CLAUDE.md` lists a worker's commands, its `/feature` and `/verify` skip the gate for a worker given a file set, and `.claude/settings.json` allows `bunx x heavy` and `bunx x stop` in place of the raw typecheck and lint scripts (the `docs-worker-commands` codemod updates `CLAUDE.md` on `x upgrade`). +- `x doctor`: a port another checkout's dev server holds is a warning, not a failure; the suggested port pair is free when asked, and so is the sync role's. +- `X_JOB_UNREGISTERED` names the job's file and prints the whole import line. +- `x db gen` runs one at a time per checkout; its success line names the tables, and "no migration needed" names the migration that already records them. +- The test step line reads `ran 2 at once (peak), planned 3`, or `1 worker, one per file (plan: …)`, in place of `2 of 4 of 6 workers`. +- A doctor probe that throws is an `X_DOCTOR_PROBE_FAILED` finding, never a crash whose fix is the doctor itself. + +### Fixed + +- `bun run preflight` and `bun run coverage` stopped by a signal kill the suites they started, so no orphan keeps its ledger reservation. +- In-process test fixtures no longer join the real ledger queue and leave frozen tickets at its head. +- `x upgrade`'s `docs-many-agents` codemod no longer reports `changed: [], skipped: []` when the app reworded its conventions table: it finds the table in `AGENTS.md` too, and otherwise reports the file as skipped with the row to add. It also checks the brief's byte budget (`agentsMdMaxBytes`, 12,000 by default) before writing, and skips with the byte count instead of pushing `CLAUDE.md` over it into `X_AGENTS_MD_TOO_LARGE`. `docs-worker-commands` does the same. +- `x upgrade` installs the merge drivers a hop's `.gitattributes` names, as `bun run setup` does, so an upgraded clone has no `X_MERGE_DRIVER_UNSET` left to fix by hand. +- A live suite no longer joins another app's test stack that shares its compose project name: it is refused with `X_TEST_SERVICES_UNAVAILABLE`. ## Releases diff --git a/PUBLISHING.md b/PUBLISHING.md index cb2e2192f..4bb21a169 100644 --- a/PUBLISHING.md +++ b/PUBLISHING.md @@ -500,7 +500,7 @@ a first-tier source confirms is the one off-cycle release (`.dz/briefs/security- tip when the checkout lacks it. The window is every commit since the highest `v*` tag HEAD contains, release commits excluded. A commit **ships** when it changes a path a published tarball packs — each package's `files` field, plus its `package.json` — minus `*.md`, tests and fixtures. Root -tooling, `bun.lock`, `.github/`, docs, the wiki and both tracked apps ship nothing. +tooling, `bun.lock`, `.github/`, docs, the wiki and the tracked apps ship nothing. | `.data.reason` | Not due because | |---|---| diff --git a/README.md b/README.md index 4097c5274..59173b86e 100644 --- a/README.md +++ b/README.md @@ -104,7 +104,7 @@ Something narrower than the default dashboard? Start from a shape: `--shape api` Or look first: **[the live demo](https://social-media.ultimate.demo.developerz.ai)**, a social network built on Ultimate, deployed from `main` on every green push. Fabricated data, every row decided by a policy. Source: [`dummy/social-media-clone/`](dummy/social-media-clone/). -No Docker, no env hunt: embedded Postgres, in-process NATS, storage in a local directory, a seeded database and a dev dashboard at `/_x`. `bun run setup` is required: `x new` installs nothing, and the app's own README names the six steps. → [Getting started](wiki/Getting-Started.md) +No Docker, no env hunt: embedded Postgres, in-process NATS, storage in a local directory, a seeded database and a dev dashboard at `/_x`. `bun run setup` is required: `x new` installs nothing, and the app's own README names the seven steps. → [Getting started](wiki/Getting-Started.md) ## Why pick it diff --git a/docs/architecture/12-generated-app.md b/docs/architecture/12-generated-app.md index dff8bbd28..fc6d6180e 100644 --- a/docs/architecture/12-generated-app.md +++ b/docs/architecture/12-generated-app.md @@ -21,7 +21,7 @@ myapp/ i18n/ # app catalogs (en, es, ...) ui/ # app-specific Solid components on top of @ultimat3/ui mcp/ # the app's own MCP tools (its dashboards are AI-first too) - bin/ # setup.ts (six steps); the gate is `bun run verify` (x verify), no file + bin/ # setup.ts (seven steps); the gate is `bun run verify` (x verify), no file .github/workflows/ # ci.yml — bun run setup then bun run verify, on push and pull request docker/ # compose (dev) + per-role compose (prod) + Dockerfile app.config.ts # the one config file diff --git a/docs/architecture/guards.md b/docs/architecture/guards.md index 2c1afc947..d7b753f25 100644 --- a/docs/architecture/guards.md +++ b/docs/architecture/guards.md @@ -60,6 +60,7 @@ The codes are in | `bun run finite-bounds` | Enforce, as a ratchet, that a NUMERIC option defaulted with `??` is checked for finiteness. | `X_FINITE_BOUND_PIN_STALE` `X_FINITE_BOUND_UNCHECKED` `X_FINITE_BOUND_UNSCANNED` | | `bun run scripts/fix-prose.ts` | Enforce, as a ratchet, that a `fix:` is a runnable command or a pasteable code shape, never prose. | `X_CLI_BAD_FLAG` `X_FIX_PROSE` `X_FIX_PROSE_PIN_STALE` `X_FIX_PROSE_PIN_UNEXPLAINED` `X_FIX_PROSE_UNSCANNED` `X_PIN_EDIT_UNPLACED` `X_PIN_RAISE_UNSTATED` | | `bun run fix-shell-arg` | Enforce, as a ratchet, that a value spliced into a `fix:` line never lands in a SHELL COMMAND POSITION unscreened. | `X_CLI_BAD_FLAG` `X_FIX_SHELL_ARG_PIN_STALE` `X_FIX_SHELL_ARG_PIN_UNEXPLAINED` `X_FIX_SHELL_ARG_UNSCANNED` `X_FIX_SHELL_ARG_UNSCREENED` `X_PIN_EDIT_UNPLACED` `X_PIN_RAISE_UNSTATED` | +| `bun run scripts/fix-tails.ts` | Enforce, as a gate rule, that the command a `fix:` line ends on runs as printed and stays narrow. | `X_FIX_TAIL_BARE` `X_FIX_TAIL_WHOLE_RUN` | | `bun run scripts/flake-hunt.ts` | The nightly flake hunt's two commands: `run` one shuffled round of a suite under a recorded seed, and `file` the rounds' failures as one GitHub issue per test. | `X_CLI_BAD_FLAG` `X_FLAKE_HUNT_TEST_FAILED` `X_FLAKE_HUNT_UNREADABLE` | | `bun run flight-copies` | One home per helper: a retry delay is computed in ONE place, no die rolls that a test cannot control, and each helper in `lib/helper-homes.ts` (escaper, cookie reader, hashes, `PgExecutor`, `storeMode`) has one implementation. | `X_FLIGHT_RANDOM_UNINJECTED` `X_FLIGHT_SECOND_CURVE` `X_HELPER_COPY` | | `bun run framework-env` | Enforce, as a build error, that every environment name shipped framework source reads by literal is in `FRAMEWORK_ENV` (`packages/core/src/framework-env.ts`), and that the list names nothing the source no longer mentions. | `X_FRAMEWORK_ENV_STALE` `X_FRAMEWORK_ENV_UNREGISTERED` | diff --git a/docs/idea/00-thesis.md b/docs/idea/00-thesis.md index acf7c3ca7..cfdb0e337 100644 --- a/docs/idea/00-thesis.md +++ b/docs/idea/00-thesis.md @@ -106,7 +106,7 @@ Each one is a permanent no, not a "later". bunx create-ultimate myapp && cd myapp && bun run setup && bun run dev ``` -`bun run setup` is six steps — `bun install`, an `.env.development.local` touch, `x db gen "initial"`, +`bun run setup` is seven steps — `bun install`, `x merge-driver install`, an `.env.development.local` touch, `x db gen "initial"`, `x db migrate`, `x db seed`, `x manifest` — the scaffold's own script, and not optional ([`13-dx.md`](./13-dx.md)). No Docker install. No `.env` scavenger hunt. Embedded Postgres, in-process NATS, S3 → local dir. A landing page in `site/` at 0kb JS, an authed dashboard in `app/` streaming, an admin app that already speaks MCP, and `x verify` green — before the first line of user code. **`x verify` alone**, `As of 30.0`: the gate's `budgets` step weighs `.x/build-stats.json`, and runs the static build that writes it whenever the file is missing or older than the source. diff --git a/docs/idea/13-dx.md b/docs/idea/13-dx.md index 4a88d3be1..a659da55d 100644 --- a/docs/idea/13-dx.md +++ b/docs/idea/13-dx.md @@ -6,11 +6,11 @@ bunx create-ultimate myapp && cd myapp && bun run setup && bun run dev ``` -Four commands. `bun run setup` is the scaffold's own script, six steps — `bun install`, an -`.env.development.local` touch, `x db gen "initial"` when `packages/db/migrations` holds no `.sql`, +Four commands. `bun run setup` is the scaffold's own script, seven steps — `bun install`, +`x merge-driver install`, an `.env.development.local` touch, `x db gen "initial"` when `packages/db/migrations` holds no `.sql`, `x db migrate`, `x db seed`, `x manifest` — and it is **not optional**: `x new` installs nothing, so `x dev` straight after `cd` stops on `X_BUILD_FAILED` naming `bun install`. Measured -`As of 2026-09-11` on a warm Bun cache: 6,802ms for all six steps, and 12.0s when the install is +`As of 2026-09-11` on a warm Bun cache: 6,802ms for the six steps it then had, and 12.0s when the install is cold. This page wrote the three-command form until 2026-08-23. ``` diff --git a/docs/idea/21-the-range.md b/docs/idea/21-the-range.md index e5e6499fb..6b60b4c3b 100644 --- a/docs/idea/21-the-range.md +++ b/docs/idea/21-the-range.md @@ -30,7 +30,7 @@ Measured on a scaffold written by this checkout, `As of 2026-08-23`. Re-derive: | files you write before the first page | **0 of 151** — re-derived `As of 2026-09-11`; `scripts/generator-counts.ts` turns a stale count red on the commit that moves it | | commands from nothing to a running app | **4** | | packages installed | **104**, in the one `bun install` `bun run setup` runs | -| `bun run setup` wall time, warm cache | **6,802ms** for all six steps, `As of 2026-09-11` — install, the env file, generate the first migration, apply it, seed, write the manifest. A **cold** first install takes it to 12.0s ([Bare VM](../../wiki/Bare-VM.md)) | +| `bun run setup` wall time, warm cache | **6,802ms** for the script as it stood `As of 2026-09-11` — install, the env file, generate the first migration, apply it, seed, write the manifest. A **cold** first install takes it to 12.0s ([Bare VM](../../wiki/Bare-VM.md)) | | what `x dev` starts | 4 roles in one process (`web` `sync` `worker` `scheduler`), 11 `/_x` panels, `db=embedded events=embedded storage=embedded mail=embedded` | The four commands, and they are the whole of it: diff --git a/docs/plans/2026/11/14/101-finished-and-self-maintained/status.yml b/docs/plans/2026/11/14/101-finished-and-self-maintained/status.yml index 27fe9fdd9..3ebf8906b 100644 --- a/docs/plans/2026/11/14/101-finished-and-self-maintained/status.yml +++ b/docs/plans/2026/11/14/101-finished-and-self-maintained/status.yml @@ -4,8 +4,8 @@ status: in_progress # not_started | in_progress | blocked | complete | su created_by: sebi worked_by: "sebi" # executor fills with their git user.name owner: sebi -percent: 88 -current_focus: "31.47.0 released and on every adopter. Two finished rows remain, both waiting on time or the platform: seven green flake-hunt nights, and the maintainer app opening a release PR." +percent: 90 +current_focus: "31.48.0 released (many-agent ledger); the multi-agent DX round (12 audit findings) is on its way to main, released by the maintainer app on its own. Two finished rows remain, both waiting on time or the platform: seven green flake-hunt nights, and the maintainer app opening a release PR." slices: - file: 01-production-hotfixes.md tier: 3 @@ -68,13 +68,13 @@ slices: sweep: "8" status: in_progress percent: 95 - note: "Pin burn-down done: every pin table row is paid off or reasoned. Closes when the maintainer app opens a release PR itself (slice 31, P8)." + note: "Pin burn-down done: every pin table row is paid off or reasoned. The platform side of P8 merged (#9507 there); closes when the maintainer app opens a release PR itself, the first due after 31.48.0." - file: 13-shapes-and-declaration.md tier: 5 sweep: "9" status: in_progress percent: 90 - note: "scripts/finished.ts: 8 of 10 rows green on main at 31.47.0. Left: tests-deterministic (seven nights) and loop-closed (maintainer app); then state maintenance mode in README and wiki/Stability.md." + note: "scripts/finished.ts: 8 of 10 rows green on main at 31.48.0. Left: tests-deterministic (seven nights) and loop-closed (maintainer app); then state maintenance mode in README and wiki/Stability.md." - file: 20-audit-rounds.md tier: 0 sweep: "A" @@ -83,13 +83,15 @@ slices: - file: 30-infra.md tier: 5 sweep: "R" - status: not_started - percent: 0 + status: in_progress + percent: 50 + note: "7 of 13 rows done in the infrastructure repository: I5 (#2466), I7 (#2457), I8 (#2456), I9 (#2455), the alert split (#2464); I6, I11, I13 merged there with their live check owed. I1-I4, I10 and I12 wait on the owner (sign-off, hCaptcha keys or an app decision)." - file: 31-maintainer-platform.md tier: 5 sweep: "R" status: in_progress - percent: 30 + percent: 65 + note: "Every row has a PR in the platform repository. Merged: P1 #9511, P2 #9520, P4 #9519, P7 #9499, P8 #9507, P9 #9500, P10 #9512. Open: P3 #9528, P5 #9522, P6 #9529. Left: those three, a re-measure of each row once deployed, and the staged drill per loop." - file: 40-adopter-a.md tier: 5 sweep: "R" @@ -141,6 +143,7 @@ slices: status: complete percent: 100 evidence: + - "multi-agent DX round (2026-11-19, at 31.48.0): one read-only auditor used the framework as parallel agents would (two clones, merged branches, a shared machine) and found 12 problems: 1 critical (a merged fork's fix wrote a migration that cannot replay), 6 high, 3 medium, 2 low. Four workers fixed all 12, each with a test that failed first; integrated in one PR (feat(cli): multi-agent developer experience)." - "finish sweep (2026-11-18): #1238 merged, released as 31.47.0 (#1255), registry-audit 31/31 attested. scripts/finished.ts went from 3 to 8 of 10 rows green: no-stub, gate-whole, no-exemption-debt, one-ledger, range-tested joined the green rows. Also: replayed offline writes bound to their principal, x exercise reproducible by default, weekly CI in the scaffold. Every adopter is on 31.47.0, CI green on each main, production apps deployed healthy after a verified backup." - "audit round 1 (2026-11-16, at 31.38.0): 4 read-only auditors (bug-hunter tiers 0–2, tiers 3–5 + scripts; concurrency; security authz + tenancy) + production evidence (demo pods/logs via ../infrastructure verbs) + open issues. Confirmed: P0 1 — live-query cross-tenant read on a tenant column not named orgId (realtime; #1188, reviewed CLOSED twice); P1 2 — parseAmount refused formatMoney output (de-AT, fr-CH, mr-IN, en-ZA code); release-due brief over the 10,000-char cap (#1170); P2 6 — robots non-ASCII Disallow; outbox staged row 60 s; sub-250 ms worker floor 60 s; repeatable-read stage (found in fix review); demo storage on emptyDir (→ 30 I13); CodeQL 7 real of 85 (78 dismissed with reasons). Not converged (P0+P1 > 0)." - "audit round 1 read ledger — READ: money, flags, time, i18n (part), cache (lru, codec, tags), storage (range, path, list, s3 listUploads), http (byte-range, forwarded, router, cookie, object-response, cache-policy), core (cookie, cursor, ids), schema (json-schema, url), entity (memory match, like, pg-sql predicates, arrays), db (statement-split, sql-scan, url), seo (xml, robots, feeds), auth (cbor, base64url, member-invite); jobs (wake, outbox, worker loop, webhook, cron, scheduler); realtime (tenancy, matcher-bridge, pg-replication row, live-replicator, record-publisher, offline-queue, client channels); action (meters, idempotency memory); query (matcher, download-token, read/cache); admin (gates, jobs resources, screen-system); mcp (exposed, tenant, server head); cli (shapes saas/realtime/api-keys/invite, sign-in, serve-retire/drain, runtime-jobs, schema wait, schema-diff); mail mime; ai cost; scraping robots/cookies; pwa route-rules; notify digest. UNREAD for round 2: policy; db migrate/pool/drift/generate/keyset/replica/row-security; entity pg-driver/write/upsert/history/partition; http pipeline/server/rate-limit/compress/csrf/cors; storage s3 body/upload/image/signed-url; cache tiers/redis-pubsub/invalidate/memo/semantic/cdn; core (rest); schema validators/coerce; auth sessions/OIDC/api keys/passkeys; cli dev/verify/deploy/upgrade/role-*/prerender/islands; testing; ui; manifest; scripts (most); jobs drivers/steps; realtime server/sync node/presence; admin (rest); mcp app-tool/transport-http; shapes content/blog/agent." @@ -158,4 +161,4 @@ evidence: - "sweep 0: #1138 merged; 31.23.0 released (#1139) — registry-audit: 31/31 publishable packages are on npm at 31.23.0, every one attested; demo /readyz 200, /llms.txt 200 (2026-11-14)" - "sweep 0 corrections: H3 premise false (sweep used the partial index; cost was dead index entries); H7 failures already fixed on main (#1036, #1060); bench release trigger never fired (GITHUB_TOKEN); released as a minor (additive API)" notes: "Written 2026-11-14 at 31.22.3 from four read-only passes: residual work in this tree, ten adopter repositories, 7–14 days of production evidence from two deployed apps, and a trace of the maintenance loop. Carries every open row of the 2026/11/02 and 2026/11/11 plans (12 step 1 marks them superseded). 90-owner-decisions.md is not a slice. Audit rounds (20) record per-round P0/P1/P2 counts and the read ledger here. Adopter letter mapping is off-tree with the owner." -last_updated: 2026-11-18 +last_updated: 2026-11-19 diff --git a/framework.manifest.json b/framework.manifest.json index 6d522400a..1445beac1 100644 --- a/framework.manifest.json +++ b/framework.manifest.json @@ -1,6 +1,6 @@ { "version": 1, - "buildId": "adb973a579d8f916a3a2da2dac2d7b01007bcc3e79b8a9dc57b20597b91fc8f4", + "buildId": "93058c71c6f55c6e741a0ae67486ab19331d2b5c440e61062f9694a10d07fc75", "tiers": { "0": [ "core", @@ -1004,6 +1004,11 @@ "owner": "cli", "at": "packages/cli/src/error-codes.ts" }, + { + "code": "X_CATALOG_UNPARSEABLE", + "owner": "cli", + "at": "packages/cli/src/error-codes-agents.ts" + }, { "code": "X_CATALOG_UNREGISTERED", "owner": "i18n", @@ -1647,7 +1652,7 @@ { "code": "X_DEV_ALREADY_RUNNING", "owner": "cli", - "at": "packages/cli/src/error-codes.ts" + "at": "packages/cli/src/error-codes-agents.ts" }, { "code": "X_DEV_DASHBOARD_IN_PROD", @@ -1667,7 +1672,7 @@ { "code": "X_DEV_LOCK_UNREADABLE", "owner": "cli", - "at": "packages/cli/src/error-codes.ts" + "at": "packages/cli/src/error-codes-agents.ts" }, { "code": "X_DEV_LOG_ABSENT", @@ -1929,6 +1934,11 @@ "owner": "scripts", "at": "scripts/dockerfile-template.ts" }, + { + "code": "X_DOCTOR_PROBE_FAILED", + "owner": "cli", + "at": "packages/cli/src/error-codes-agents.ts" + }, { "code": "X_DRAINING", "owner": "core", @@ -2484,6 +2494,16 @@ "owner": "scripts", "at": "scripts/fix-shell-arg.ts" }, + { + "code": "X_FIX_TAIL_BARE", + "owner": "scripts", + "at": "scripts/fix-tails.ts" + }, + { + "code": "X_FIX_TAIL_WHOLE_RUN", + "owner": "scripts", + "at": "scripts/fix-tails.ts" + }, { "code": "X_FIX_TARGET_UNKNOWN", "owner": "cli", @@ -2734,6 +2754,11 @@ "owner": "cli", "at": "packages/cli/src/error-codes-tooling.ts" }, + { + "code": "X_HEAVY_WAIT_EXPIRED", + "owner": "cli", + "at": "packages/cli/src/error-codes-agents.ts" + }, { "code": "X_HELPER_COPY", "owner": "scripts", @@ -3754,6 +3779,16 @@ "owner": "mcp", "at": "packages/mcp/src/errors.ts" }, + { + "code": "X_MERGE_DRIVER_UNSET", + "owner": "cli", + "at": "packages/cli/src/error-codes-agents.ts" + }, + { + "code": "X_MERGE_UNRESOLVED", + "owner": "cli", + "at": "packages/cli/src/error-codes-agents.ts" + }, { "code": "X_METER_EXCEEDED", "owner": "action", @@ -3829,6 +3864,11 @@ "owner": "db", "at": "packages/db/src/errors.ts" }, + { + "code": "X_MIGRATION_FORKED", + "owner": "db", + "at": "packages/db/src/errors.ts" + }, { "code": "X_MIGRATION_IRREVERSIBLE", "owner": "db", @@ -4357,7 +4397,7 @@ { "code": "X_PORT_IN_USE", "owner": "cli", - "at": "packages/cli/src/error-codes.ts" + "at": "packages/cli/src/error-codes-agents.ts" }, { "code": "X_PORT_INVALID", @@ -6847,7 +6887,7 @@ { "code": "X_UPGRADE_LOCKED", "owner": "cli", - "at": "packages/cli/src/error-codes.ts" + "at": "packages/cli/src/error-codes-agents.ts" }, { "code": "X_UPGRADE_MIXED_PINS", diff --git a/packages/ai/src/eval-errors.ts b/packages/ai/src/eval-errors.ts index 25d85b170..c9f544df8 100644 --- a/packages/ai/src/eval-errors.ts +++ b/packages/ai/src/eval-errors.ts @@ -119,7 +119,7 @@ export class EvalRecordingError extends UltimateError { super({ code: 'X_EVAL_RECORDING', cause: `${input.env} is set, so every eval would re-record its baseline instead of gating on it`, - fix: `env -u ${input.env} x verify`, + fix: `env -u ${input.env} x verify --only eval`, }); } } diff --git a/packages/cli/src/api-registration.ts b/packages/cli/src/api-registration.ts index 7f6bdaa56..10ff9328d 100644 --- a/packages/cli/src/api-registration.ts +++ b/packages/cli/src/api-registration.ts @@ -66,7 +66,9 @@ export function apiEntriesFor(written: readonly string[]): readonly ApiEntry[] { * comment between two entries ends nothing. (The list was closed at the first `]` in the raw text * and split at every comma — a commented list was rewritten with its comment as an entry.) */ -const listsOf = (source: string): ((key: string) => SourceList | undefined) | undefined => { +export const apiListsOf = ( + source: string, +): ((key: string) => SourceList | undefined) | undefined => { const masked = maskOf(source); const call = /\bdefineApi\(\s*\{/.exec(masked); if (call === null) return undefined; @@ -79,6 +81,28 @@ const listsOf = (source: string): ((key: string) => SourceList | undefined) | un }; }; +/** + * `source` with `key: value,` added as a property, on its own row: in the order `x new` writes — + * actions, queries, jobs, tasks — so after the nearest list before it the call already holds, which + * is `actions` at the least. `undefined` when the row would not land inside the call: an earlier + * list sharing its row with the call's own `});` puts "the row after it" outside. + */ +export function insertApiList( + source: string, + key: ApiEntry['key'], + value: string, +): string | undefined { + const listOf = apiListsOf(source); + const actions = listOf?.('actions'); + if (listOf === undefined || actions === undefined) return undefined; + const earlier = LIST_ORDER.slice(0, LIST_ORDER.indexOf(key)) + .map((name) => listOf(name)) + .findLast((found) => found !== undefined); + const after = source.indexOf('\n', (earlier ?? actions).close); + const candidate = `${source.slice(0, after + 1)} ${key}: ${value},\n${source.slice(after + 1)}`; + return apiListsOf(candidate)?.(key) === undefined ? undefined : candidate; +} + /** * `source` with each entry imported and listed. The import joins the `import * as` block in sorted * position; the binding joins its list, which is created after `actions: [...]` when the call has @@ -94,7 +118,7 @@ export function insertApiEntries( const skipped: ApiEntry[] = []; for (const entry of entries) { const importLine = `import * as ${entry.binding} from '${entry.specifier}';`; - const listOf = listsOf(next); + const listOf = apiListsOf(next); const actions = listOf?.('actions'); if (listOf === undefined || actions === undefined) { skipped.push(entry); @@ -103,18 +127,8 @@ export function insertApiEntries( const list = listOf(entry.key); if (list?.entries.some((one) => one.text === entry.binding) === true) continue; if (list === undefined) { - // In the order `x new` writes — actions, queries, jobs, tasks: after the nearest list - // before it that the call already holds, which is `actions` at the least. - const earlier = LIST_ORDER.slice(0, LIST_ORDER.indexOf(entry.key)) - .map((key) => listOf(key)) - .findLast((found) => found !== undefined); - const after = next.indexOf('\n', (earlier ?? actions).close); - const line = ` ${entry.key}: [${entry.binding}],`; - const candidate = `${next.slice(0, after + 1)}${line}\n${next.slice(after + 1)}`; - // Read back, as every other edit here is: when the earlier list shares its row with the - // call's own `});`, "the row after it" is OUTSIDE the call — a property of nothing. - const landed = listsOf(candidate)?.(entry.key); - if (landed?.entries.some((one) => one.text === entry.binding) !== true) { + const candidate = insertApiList(next, entry.key, `[${entry.binding}]`); + if (candidate === undefined) { skipped.push(entry); continue; } diff --git a/packages/cli/src/app-evals.test.ts b/packages/cli/src/app-evals.test.ts index 3cc8c2983..e2cc61827 100644 --- a/packages/cli/src/app-evals.test.ts +++ b/packages/cli/src/app-evals.test.ts @@ -17,9 +17,19 @@ import { TEST_STEPS } from './verify-tests'; // Under `packages/cli/` so the fixture's `@ultimat3/*` imports resolve through the same tsconfig // paths the framework's own sources use; a dot-prefixed name keeps it out of every workspace glob. -const COVERED = processRoot(join(import.meta.dir, '..', '.eval-covered-fixture')); -const BARE = processRoot(join(import.meta.dir, '..', '.eval-bare-fixture')); -const AGENT = processRoot(join(import.meta.dir, '..', '.eval-agent-fixture')); +const BASES = { + covered: processRoot(join(import.meta.dir, '..', '.eval-covered-fixture')), + bare: processRoot(join(import.meta.dir, '..', '.eval-bare-fixture')), + agent: processRoot(join(import.meta.dir, '..', '.eval-agent-fixture')), +}; +// A fresh directory per pass: `bun test --rerun-each` re-runs this file but keeps the module cache, +// so a second pass over the same paths imported nothing, registered no prompt, and every coverage +// case read an empty app. Random, because a counter here restarts with the file. +let COVERED = ''; +let BARE = ''; +let AGENT = ''; +/** The baseline the COVERED fixture's eval declares — written and removed by the tests below. */ +let COVERED_BASELINE = ''; /** An `agent()` on an unevaluated prompt: its fix must name the agent arm, not the prompt arm. */ const agentModule = `import { t } from '@ultimat3/action'; @@ -62,7 +72,7 @@ export const summarizeEval = defineEval({ const APP_CONFIG = "export const config = { name: 'fixture' };\n"; -const FIXTURES: Readonly>>> = { +const fixtures = (): Readonly>>> => ({ [COVERED]: { 'app.config.ts': APP_CONFIG, 'apps/web/app/posts/prompts/summarize.ts': promptModule('posts.summarize'), @@ -76,10 +86,7 @@ const FIXTURES: Readonly>>> = { 'app.config.ts': APP_CONFIG, 'apps/web/app/support/agent.ts': agentModule, }, -}; - -/** The baseline the COVERED fixture's eval declares — written and removed by the tests below. */ -const COVERED_BASELINE = join(COVERED, 'apps/web/app/posts/prompts/summarize.baseline.json'); +}); const RECORDED = `${JSON.stringify( { eval: 'summarize', prompt: 'posts.summarize@3', promptHash: 'abc123', score: 1, cases: {} }, @@ -94,7 +101,12 @@ const reset = (): void => { }; beforeAll(async () => { - for (const [root, files] of Object.entries(FIXTURES)) { + const pass = `pass-${crypto.randomUUID().slice(0, 8)}`; + COVERED = join(BASES.covered, pass); + BARE = join(BASES.bare, pass); + AGENT = join(BASES.agent, pass); + COVERED_BASELINE = join(COVERED, 'apps/web/app/posts/prompts/summarize.baseline.json'); + for (const [root, files] of Object.entries(fixtures())) { await rm(root, { recursive: true, force: true }); for (const [path, contents] of Object.entries(files)) { await Bun.write(join(root, path), contents); @@ -104,7 +116,7 @@ beforeAll(async () => { }); afterAll(async () => { - for (const root of Object.keys(FIXTURES)) await rm(root, { recursive: true, force: true }); + for (const root of Object.values(BASES)) await rm(root, { recursive: true, force: true }); reset(); }); @@ -252,7 +264,7 @@ describe('unit · the gate refuses to run while recording', () => { expect(findings).toHaveLength(1); expect(findings[0]?.code).toBe('X_EVAL_RECORDING'); expect(findings[0]?.cause).toContain(RECORD_ENV); - expect(findings[0]?.fix).toBe(`env -u ${RECORD_ENV} x verify`); + expect(findings[0]?.fix).toBe(`env -u ${RECORD_ENV} x verify --only eval`); }); test('the step fails and never runs the suite, so no baseline is rewritten', async () => { diff --git a/packages/cli/src/app-load.ts b/packages/cli/src/app-load.ts index 5fcc19c4a..78b15dc55 100644 --- a/packages/cli/src/app-load.ts +++ b/packages/cli/src/app-load.ts @@ -332,7 +332,12 @@ export async function loadApp(root: string): Promise { // reach is a finding here, owned by `manifest` like every other load finding. Imported here, not // at the top: a container boots through `scanAppModules` and never asks (`serve-graph.test.ts`). const { unscannedAdminFindings } = await import('./unscanned-admin'); - const findings = [...scan.findings, ...(await unscannedAdminFindings(root))]; + const scanned = [...scan.findings, ...(await unscannedAdminFindings(root))]; + // Only a load that failed asks why: the merge's leftovers explain it better than each importer. + const findings = + scanned.length === 0 + ? scanned + : [...(await (await import('./merge-state')).withMergeCauses(root, scanned))]; await trackStylesheets(root); // An app that imported NOTHING and reported nothing is a registry every later step reads as // empty-and-fine. With an `app.config.ts` beside it, that is never what the author meant. diff --git a/packages/cli/src/clean-services.ts b/packages/cli/src/clean-services.ts new file mode 100644 index 000000000..b82ce206e --- /dev/null +++ b/packages/cli/src/clean-services.ts @@ -0,0 +1,142 @@ +// `x clean --services`: stop this checkout's test stack, and list every other one on the machine. +// Others are only ever listed: on a machine shared by agents, another stack may be in use right now, +// and each line carries the command that stops it. + +// why: Bun ships no synchronous existence check. +import { existsSync } from 'node:fs'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { renderThrowable } from '@ultimat3/core'; +import type { Runner } from './exec'; +import { execOutput } from './exec'; +import { msg } from './messages'; +import type { Finding, JsonValue } from './output'; +import { TEST_COMPOSE_PATH } from './templates/scaffold-test-services'; +import type { TestStack } from './verify-services-stack'; +import { + jsonRows, + otherRoots, + readStack, + rootOfCompose, + SERVICES_LIST, + SERVICES_STOP, + STACKS_LIST, +} from './verify-services-stack'; + +export interface ServicesCleanup { + readonly data: JsonValue; + readonly lines: readonly string[]; + readonly finding?: Finding; +} + +interface OtherStack { + readonly project: string; + readonly status: string; + readonly root: string; + readonly gone: boolean; +} + +const refused = (command: readonly string[], why: string): Finding => ({ + code: 'X_TEST_SERVICES_UNAVAILABLE', + cause: `\`${command.join(' ')}\` could not run — ${why}`, + fix: `${command.join(' ')} # needs Docker running`, + at: TEST_COMPOSE_PATH, +}); + +/** Test stacks of other checkouts. Best effort: failing to list them stops nothing of ours. */ +async function otherStacks(root: string, own: string | undefined, runner: Runner) { + try { + const result = await runner(STACKS_LIST, { cwd: root }); + if (!result.ok) return []; + return jsonRows(result.stdout).flatMap((row): OtherStack[] => { + const [project, status, files] = [row['Name'], row['Status'], row['ConfigFiles']]; + if (typeof project !== 'string' || typeof files !== 'string' || project === own) return []; + const other = rootOfCompose(files); + if (other === undefined || other === root) return []; + const state = typeof status === 'string' ? status : ''; + return [{ project, status: state, root: other, gone: !existsSync(other) }]; + }); + } catch { + return []; + } +} + +/** `undefined` when the stack stopped; otherwise the refusal naming why. */ +async function stop(root: string, runner: Runner): Promise { + try { + const result = await runner(SERVICES_STOP, { cwd: root }); + return result.ok + ? undefined + : refused( + SERVICES_STOP, + `it exited ${String(result.code)}: ${execOutput(result).slice(-400)}`, + ); + } catch (error) { + return refused(SERVICES_STOP, renderThrowable(error)); + } +} + +type Action = 'none' | 'stopped' | 'would-stop' | 'kept' | 'failed'; + +async function settle( + stack: TestStack | undefined, + input: { readonly root: string; readonly runner: Runner; readonly dryRun: boolean }, +): Promise<{ readonly action: Action; readonly owner?: string; readonly finding?: Finding }> { + if (stack === undefined) return { action: 'none' }; + const [owner] = otherRoots(stack, input.root); + if (owner !== undefined) return { action: 'kept', owner }; + if (input.dryRun) return { action: 'would-stop' }; + const finding = await stop(input.root, input.runner); + return finding === undefined ? { action: 'stopped' } : { action: 'failed', finding }; +} + +export async function cleanServices(input: { + readonly root: string; + readonly runner: Runner; + readonly dryRun: boolean; +}): Promise { + const { root, runner } = input; + let stack: TestStack | undefined; + if (existsSync(join(root, TEST_COMPOSE_PATH))) { + const read = await readStack(root, runner); + if (!read.answered) { + return { + data: { action: 'failed' }, + lines: [], + finding: refused(SERVICES_LIST, 'Docker gave no answer about this app’s stack'), + }; + } + stack = read.stack; + } + const outcome = await settle(stack, input); + const others = await otherStacks(root, stack?.project, runner); + const project = stack?.project ?? ''; + const services = stack?.services.join(', ') ?? ''; + return { + data: { + action: outcome.action, + ...(stack === undefined ? {} : { project, services: [...stack.services] }), + ...(outcome.owner === undefined ? {} : { owner: outcome.owner }), + others: others.map((other) => ({ ...other })), + }, + lines: [ + ...(outcome.action === 'failed' + ? [] + : [ + msg(`cli.clean.services.${outcome.action}`, { + project, + services, + owner: outcome.owner ?? '', + }), + ]), + ...others.map((other) => + msg(other.gone ? 'cli.clean.services.otherGone' : 'cli.clean.services.other', { + project: other.project, + status: other.status, + root: other.root, + }), + ), + ], + ...(outcome.finding === undefined ? {} : { finding: outcome.finding }), + }; +} diff --git a/packages/cli/src/cmd-clean-spec.ts b/packages/cli/src/cmd-clean-spec.ts index 6820fc386..b56a91714 100644 --- a/packages/cli/src/cmd-clean-spec.ts +++ b/packages/cli/src/cmd-clean-spec.ts @@ -6,8 +6,15 @@ import type { CommandSpec } from './parse'; export const cleanSpec: CommandSpec = { name: 'clean', summary: - 'remove what test runs leave behind: migrated test templates, .x/cache, stray .x folders, stale probe databases', - usage: 'x clean [--dry-run] [--json]', + 'remove what test runs leave behind: migrated test templates, .x/cache, stray .x folders, stale probe databases; --services stops the test stack too', + usage: 'x clean [--dry-run] [--services] [--json]', requiresApp: true, - flags: [{ name: 'dry-run', type: 'boolean', summary: 'name what would go, remove nothing' }], + flags: [ + { name: 'dry-run', type: 'boolean', summary: 'name what would go, remove nothing' }, + { + name: 'services', + type: 'boolean', + summary: "also stop this checkout's test stack, and list the machine's others", + }, + ], }; diff --git a/packages/cli/src/cmd-clean.test.ts b/packages/cli/src/cmd-clean.test.ts index 78ead1898..6bc36bddf 100644 --- a/packages/cli/src/cmd-clean.test.ts +++ b/packages/cli/src/cmd-clean.test.ts @@ -14,6 +14,9 @@ import type { CommandContext } from './command'; import type { ExecResult } from './exec'; import type { JsonValue } from './output'; import { parseArgs } from './parse'; +import { TEST_COMPOSE_PATH } from './templates/scaffold-test-services'; +import { psLine } from './verify-services-fixture'; +import { SERVICES_LIST, SERVICES_STOP, STACKS_LIST } from './verify-services-stack'; const scratch = mkdtempSync(join(tmpdir(), 'x-cmd-clean-')); afterAll(() => rmSync(scratch, { recursive: true, force: true })); @@ -89,3 +92,112 @@ describe('x clean', () => { expect(data(result.data)['paths']).toEqual([]); }); }); + +describe('x clean --services', () => { + /** An app with a test stack, over a fake docker that records what it was told. */ + function stacked( + name: string, + answers: { ps?: string; ls?: string; stop?: number; psCode?: number }, + ) { + const root = app(name); + put(join(root, TEST_COMPOSE_PATH)); + const calls: string[] = []; + const docker: CommandContext['runner'] = async (command) => { + const line = command.join(' '); + calls.push(line); + const reply = (stdout: string, code = 0): ExecResult => ({ + command, + ok: code === 0, + code, + stdout, + stderr: code === 0 ? '' : 'docker said no', + durationMs: 0, + }); + if (line === SERVICES_LIST.join(' ')) return reply(answers.ps ?? '', answers.psCode ?? 0); + if (line === STACKS_LIST.join(' ')) return reply(answers.ls ?? '[]'); + if (line === SERVICES_STOP.join(' ')) return reply('', answers.stop ?? 0); + return reply(''); + }; + return { + root, + calls, + ctx: (argv: readonly string[]) => ({ ...context(argv, root), runner: docker }), + }; + } + + const services = (value: JsonValue | undefined) => data(data(value)['services']); + + test('without the flag docker is never asked', async () => { + const { calls, ctx } = stacked('untouched', {}); + await cleanCommand.run(ctx(['clean'])); + expect(calls).toEqual([]); + }); + + test("stops this checkout's stack and says so", async () => { + const root = join(scratch, 'stops'); + const { calls, ctx } = stacked('stops', { ps: psLine('shop-test', 'postgres', root) }); + const result = await cleanCommand.run(ctx(['clean', '--services'])); + expect(result.ok).toBe(true); + expect(services(result.data)).toMatchObject({ project: 'shop-test', action: 'stopped' }); + expect(calls).toContain(SERVICES_STOP.join(' ')); + }); + + test('--dry-run names the stack and stops nothing', async () => { + const root = join(scratch, 'dry-stack'); + const { calls, ctx } = stacked('dry-stack', { ps: psLine('shop-test', 'redis', root) }); + const result = await cleanCommand.run(ctx(['clean', '--services', '--dry-run'])); + expect(services(result.data)).toMatchObject({ project: 'shop-test', action: 'would-stop' }); + expect(calls).not.toContain(SERVICES_STOP.join(' ')); + }); + + test('a stack a live sibling started is kept, with the line that would stop it', async () => { + const sibling = app('sibling'); + const { calls, ctx } = stacked('kept', { ps: psLine('shop-test', 'redis', sibling) }); + const result = await cleanCommand.run(ctx(['clean', '--services'])); + expect(result.ok).toBe(true); + expect(services(result.data)).toMatchObject({ action: 'kept', owner: sibling }); + expect(result.lines?.join('\n')).toContain('docker compose -p shop-test down'); + expect(calls).not.toContain(SERVICES_STOP.join(' ')); + }); + + test("other apps' stacks are listed, flagged when their checkout is gone, and left alone", async () => { + const other = app('other-app'); + const ls = JSON.stringify([ + { Name: 'a-test', Status: 'running(4)', ConfigFiles: join(other, TEST_COMPOSE_PATH) }, + { Name: 'b-test', Status: 'running(1)', ConfigFiles: `/gone/${TEST_COMPOSE_PATH}` }, + { Name: 'prod', Status: 'running(9)', ConfigFiles: '/srv/prod/compose.yml' }, + // This checkout's own, stopped: it is not an "other". + { Name: 'own', Status: 'exited(1)', ConfigFiles: join(scratch, 'lister', TEST_COMPOSE_PATH) }, + ]); + const { calls, ctx } = stacked('lister', { ls }); + const result = await cleanCommand.run(ctx(['clean', '--services', '--dry-run'])); + expect(services(result.data)['others']).toEqual([ + { project: 'a-test', status: 'running(4)', root: other, gone: false }, + { project: 'b-test', status: 'running(1)', root: '/gone', gone: true }, + ]); + expect(result.lines?.join('\n')).toContain('docker compose -p b-test down'); + expect(calls.some((line) => line.includes('-p a-test'))).toBe(false); + }); + + test('docker failing is a coded refusal, not a clean bill of health', async () => { + const { ctx } = stacked('no-docker', { psCode: 1 }); + const result = await cleanCommand.run(ctx(['clean', '--services'])); + expect(result.ok).toBe(false); + expect(result.findings?.[0]?.code).toBe('X_TEST_SERVICES_UNAVAILABLE'); + }); + + test('a stop docker refuses is a coded refusal', async () => { + const root = join(scratch, 'stuck'); + const { ctx } = stacked('stuck', { ps: psLine('shop-test', 'redis', root), stop: 1 }); + const result = await cleanCommand.run(ctx(['clean', '--services'])); + expect(result.ok).toBe(false); + expect(result.findings?.[0]?.cause).toContain('docker said no'); + }); + + test('an app with no test stack has nothing to stop', async () => { + const root = app('stackless'); + const result = await cleanCommand.run(context(['clean', '--services'], root)); + expect(result.ok).toBe(true); + expect(services(result.data)).toMatchObject({ action: 'none' }); + }); +}); diff --git a/packages/cli/src/cmd-clean.ts b/packages/cli/src/cmd-clean.ts index a5f0109a1..15004ad2c 100644 --- a/packages/cli/src/cmd-clean.ts +++ b/packages/cli/src/cmd-clean.ts @@ -6,9 +6,10 @@ // `TEST_DATABASE_URL` set — the probe databases whose run is over. The test runner sweeps the same // set before every run, keeping the newest two of each cache (`test-housekeeping.ts`); this is the // same sweep, keeping none. Never touched: `.x/pgdata` (the dev database), `.x/storage`, the build -// output. +// output. `--services` adds this checkout's test stack (`clean-services.ts`). import { requireAppRoot } from './app-root'; +import { cleanServices } from './clean-services'; import { cleanSpec } from './cmd-clean-spec'; import type { CliCommand, CommandContext } from './command'; import { msg } from './messages'; @@ -23,17 +24,30 @@ export const cleanCommand: CliCommand = { const dryRun = flagBool(ctx.args, 'dry-run'); const report = await tidyTestState({ root, mode: 'all', env: ctx.env, dryRun }); const count = report.paths.length + report.databases.length; + const services = flagBool(ctx.args, 'services') + ? await cleanServices({ root, runner: ctx.runner, dryRun }) + : undefined; + const finding = services?.finding; return { - ok: true, + ok: finding === undefined, command: 'clean', summary: - count === 0 - ? msg('cli.clean.nothing') - : msg(dryRun ? 'cli.clean.would' : 'cli.clean.removed', { - paths: report.paths.length, - databases: report.databases.length, - }), - data: { dryRun, paths: [...report.paths], databases: [...report.databases] }, + finding !== undefined + ? finding.cause + : count === 0 + ? msg('cli.clean.nothing') + : msg(dryRun ? 'cli.clean.would' : 'cli.clean.removed', { + paths: report.paths.length, + databases: report.databases.length, + }), + ...(finding === undefined ? {} : { findings: [finding] }), + ...(services === undefined ? {} : { lines: [...services.lines] }), + data: { + dryRun, + paths: [...report.paths], + databases: [...report.databases], + ...(services === undefined ? {} : { services: services.data }), + }, }; }, }; diff --git a/packages/cli/src/cmd-db-spec.ts b/packages/cli/src/cmd-db-spec.ts index a8b2f55d4..73173100a 100644 --- a/packages/cli/src/cmd-db-spec.ts +++ b/packages/cli/src/cmd-db-spec.ts @@ -20,7 +20,7 @@ export const dbSpec: CommandSpec = { name: 'db', summary: 'gen, migrate, reset, seed, branch, backfill, diagnostics, recount, status', usage: - 'x db gen "add publish_at" [--drop-redundant] | migrate | reset | seed [] [--tier reference|dev] [--dry-run] | seed --generate [--rows n] [--seed n] [--dry-run] | branch list | branch create | branch delete | backfill [|--all] [--write] [--force] | backfill --pending | backfill --list [--name n] [--status s] [--limit n] | diagnostics [--limit n] | recount . [--write] | status', + 'x db gen "add publish_at" [--drop-redundant] [--merge-fork] | migrate | reset | seed [] [--tier reference|dev] [--dry-run] | seed --generate [--rows n] [--seed n] [--dry-run] | branch list | branch create | branch delete | backfill [|--all] [--write] [--force] | backfill --pending | backfill --list [--name n] [--status s] [--limit n] | diagnostics [--limit n] | recount . [--write] | status', requiresApp: true, subcommands: DB_SUBCOMMANDS, // Declared from the constant `runBranchCommand` validates against, never a second literal: it @@ -126,6 +126,14 @@ export const dbSpec: CommandSpec = { summary: 'gen: drop each implicit index another index already leads with, concurrently', subcommands: ['gen'], }, + // Declared because `X_MIGRATION_FORKED`'s own fix line names it. Never implied: it rewrites a + // sidecar, and a chain that is not forked has no use for it. + { + name: 'merge-fork', + type: 'boolean', + summary: 'gen: repair a forked chain in the newest snapshot; no SQL changes, nothing deleted', + subcommands: ['gen'], + }, // Declared because `X_MIGRATION_ROLLING_UNSAFE`'s own fix line names it. { name: 'rolling-unsafe', diff --git a/packages/cli/src/cmd-db.ts b/packages/cli/src/cmd-db.ts index bd4ee83c8..5481b48a8 100644 --- a/packages/cli/src/cmd-db.ts +++ b/packages/cli/src/cmd-db.ts @@ -13,7 +13,8 @@ import { rm } from 'node:fs/promises'; import { join } from 'node:path'; import { NotImplementedError, resolveEnvironment } from '@ultimat3/core'; -import { type DriftReport, driftError, withTransaction } from '@ultimat3/db'; +import { type DriftReport, driftError, migrationForked, withTransaction } from '@ultimat3/db'; +import { migrationFork } from '@ultimat3/db/schema-dump'; import { postgresDriver } from '@ultimat3/entity'; import { requireAppRoot } from './app-root'; import { runBackfillCommand } from './cmd-db-backfill'; @@ -42,8 +43,10 @@ import { import { runGeneratedSeed } from './db-seed-generate'; import { DevAlreadyRunningError, liveDevLock } from './dev-lock'; import { MissingSubcommandError, UnknownCommandError } from './errors'; +import { type HeavyChild, withCheckoutLock } from './heavy'; import { withJobDriver } from './jobs-driver'; import { msg } from './messages'; +import { readMigrations } from './migrations'; import type { CommandResult, Finding } from './output'; import { findingFrom } from './output'; import { flagBool, flagString } from './parse'; @@ -99,6 +102,22 @@ export const dbCommand: CliCommand = { * exists when none does, or that nothing was written when the sidecar was. */ async function runGen(ctx: CommandContext, root: string, name: string): Promise { + // One at a time per checkout, from reading the newest sidecar to writing the dump. Two overlapping + // runs each diffed the same sidecar and each wrote `create table` for every new entity: a chain + // that cannot replay. The kernel drops the lock when its holder dies, so no stale file can wedge it. + return withCheckoutLock({ name: 'db-gen', root, env: ctx.env }, NO_CHILD, () => + generateLocked(ctx, root, name), + ); +} + +/** `x db gen` starts no process under the lock, so there is nothing to hand one. */ +const NO_CHILD: HeavyChild = { env: {}, fds: [] }; + +async function generateLocked( + ctx: CommandContext, + root: string, + name: string, +): Promise { let generated: Awaited>; try { generated = await generateAppMigration(root, { @@ -106,6 +125,7 @@ async function runGen(ctx: CommandContext, root: string, name: string): Promise< allowDestructive: flagBool(ctx.args, 'allow-destructive'), rollingUnsafe: flagBool(ctx.args, 'rolling-unsafe'), dropRedundant: flagBool(ctx.args, 'drop-redundant'), + mergeFork: flagBool(ctx.args, 'merge-fork'), }); } catch (error) { return { @@ -127,7 +147,12 @@ async function runGen(ctx: CommandContext, root: string, name: string): Promise< const dump = await refreshSchemaDump(root, ctx.env); const dumpFindings = dump.finding === undefined ? [] : [dump.finding]; const migration = generated.migration; + const newest = generated.newest; if (migration === undefined) { + const newestLine = + newest === undefined || generated.outcome === 'blocked' + ? [] + : [msg('cli.db.gen.newest', { id: newest })]; return { ok: generated.findings.length === 0 && dumpFindings.length === 0, command: 'db', @@ -139,17 +164,32 @@ async function runGen(ctx: CommandContext, root: string, name: string): Promise< summary: generated.outcome === 'blocked' ? msg('cli.db.gen.blocked', { count: generated.findings.length }) - : generated.outcome === 'hash-recorded' - ? msg('cli.db.gen.recorded', { file: generated.files[0] ?? '' }) - : dump.status === 'written' - ? msg('cli.db.gen.dumped', { count: dump.written.length + dump.removed.length }) - : msg('cli.db.gen.unchanged'), + : generated.outcome === 'fork-merged' + ? msg('cli.db.gen.forkMerged', { + count: generated.forkRepairs ?? 0, + id: newest ?? '', + }) + : generated.outcome === 'hash-recorded' + ? msg('cli.db.gen.recorded', { file: generated.files[0] ?? '' }) + : dump.status === 'written' + ? msg('cli.db.gen.dumped', { + count: dump.written.length + dump.removed.length, + newest: newest ?? msg('cli.db.status.none'), + }) + : msg('cli.db.gen.unchanged'), findings: [...generated.findings, ...dumpFindings], // `files` is what this command WROTE, so the empty array here was a false claim. - lines: [...generated.files.map((file) => ` ${file}`), ...schemaDumpLines(dump), ...lost], + lines: [ + ...generated.files.map((file) => ` ${file}`), + ...schemaDumpLines(dump), + ...newestLine, + ...lost, + ], data: { outcome: generated.outcome, migration: null, + newest: newest ?? null, + forkRepairs: generated.forkRepairs ?? 0, files: [...generated.files], schemaHash: generated.schemaHash ?? null, unrendered: unrenderedJson(generated.unrendered), @@ -157,13 +197,16 @@ async function runGen(ctx: CommandContext, root: string, name: string): Promise< }, }; } + const tables = generated.tables ?? []; return { ok: dumpFindings.length === 0, command: 'db', summary: - dumpFindings.length === 0 - ? msg('cli.db.gen.written', { id: migration.id }) - : msg('cli.db.gen.writtenDumpFailed', { id: migration.id }), + dumpFindings.length > 0 + ? msg('cli.db.gen.writtenDumpFailed', { id: migration.id }) + : tables.length === 0 + ? msg('cli.db.gen.written', { id: migration.id }) + : msg('cli.db.gen.writtenTables', { id: migration.id, tables: tables.join(', ') }), // Exit 1 with a migration on disk: the finding has to say so, or the next move is a duplicate. findings: dumpFindings.map((finding) => afterWrittenMigration(finding, migration.id)), lines: [...generated.files.map((file) => ` ${file}`), ...schemaDumpLines(dump), ...lost], @@ -173,6 +216,9 @@ async function runGen(ctx: CommandContext, root: string, name: string): Promise< // The `-- transaction: false` step written beside it, when the diff needed one. followUp: migration.followUp?.id ?? null, name: migration.name, + tables: [...tables], + newest: newest ?? migration.id, + forkRepairs: generated.forkRepairs ?? 0, files: [...generated.files], schemaHash: generated.schemaHash ?? null, unrendered: unrenderedJson(generated.unrendered), @@ -206,6 +252,10 @@ async function runMigrate( summary: string, ): Promise { try { + // Before anything is applied: a forked chain either stops halfway through a replay or finishes + // and reports its own table as drift, and neither result says what happened. + const fork = migrationFork(await readMigrations(root)); + if (fork !== undefined) throw migrationForked(fork); const migrated = await runMigrations({ root, env: ctx.env }); const report = migrated.report; // A hand-written or pulled migration reaches the dump here, with no `x db gen` in between. diff --git a/packages/cli/src/cmd-doctor-fixture.ts b/packages/cli/src/cmd-doctor-fixture.ts index 2fee423c0..361c56f52 100644 --- a/packages/cli/src/cmd-doctor-fixture.ts +++ b/packages/cli/src/cmd-doctor-fixture.ts @@ -18,6 +18,8 @@ export const probe = (over: Partial = {}): DoctorReportProbe frameworkFindings: [], exists: () => true, portFree: async () => true, + // Nobody the OS will name: a taken port stays this environment's finding. + portOwner: async () => ({}), // Reachable, or embedded — the probe's own `null`. A test that opened a pool would be asking // about the box it runs on rather than about `runDoctor`. database: async () => null, @@ -46,6 +48,9 @@ export const probe = (over: Partial = {}): DoctorReportProbe pins: async () => ({ pins: [], installed: new Map() }), // Every step that applies is floored. unflooredSteps: async () => [], + // No merge in progress, and the drivers `bun run setup` installs are there. + mergeState: async () => [], + mergeDriverUnset: async () => false, // Nothing to list: this machine's Chrome is never started under a unit test's clock. browser: async () => undefined, ...over, diff --git a/packages/cli/src/cmd-doctor-probe.test.ts b/packages/cli/src/cmd-doctor-probe.test.ts index 4727e8065..a02485926 100644 --- a/packages/cli/src/cmd-doctor-probe.test.ts +++ b/packages/cli/src/cmd-doctor-probe.test.ts @@ -162,6 +162,9 @@ describe('unit · x doctor · the command', () => { bunVersion: REQUIRED_BUN, }); + // The server is this test process, whose directory is not the app's: another checkout's port. The + // real owner lookup files it beside the verdict where the OS names the holder's directory, and in + // it where the OS will not — reported once either way, never dropped. test('a port that is taken is reported as X_PORT_IN_USE, and --json lists the codes', async () => { const server = Bun.serve({ port: 0, fetch: () => new Response('') }); const root = doctorAppRoot(); @@ -169,10 +172,17 @@ describe('unit · x doctor · the command', () => { const result = await doctorCommand.run( doctorContext(['doctor', '--port', String(server.port), '--json'], root), ); - expect(result.ok).toBe(false); expect(result.command).toBe('doctor'); - const data = result.data as { count: number; codes: readonly string[] }; - expect(data.codes).toContain('X_PORT_IN_USE'); + const data = result.data as { + count: number; + codes: readonly string[]; + warnings?: readonly { code: string; cause: string }[]; + }; + const warned = (data.warnings ?? []).filter((warning) => warning.code === 'X_PORT_IN_USE'); + const failed = data.codes.filter((code) => code === 'X_PORT_IN_USE'); + expect(warned.length + failed.length).toBe(1); + if (warned[0] !== undefined) + expect(warned[0].cause).toContain(`port ${server.port} is held by`); expect(data.count).toBe((result.findings ?? []).length); // Every finding the report counted is a finding the report carries. expect(data.codes).toEqual((result.findings ?? []).map((finding) => finding.code)); diff --git a/packages/cli/src/cmd-doctor.ts b/packages/cli/src/cmd-doctor.ts index e0a250406..e016160fa 100644 --- a/packages/cli/src/cmd-doctor.ts +++ b/packages/cli/src/cmd-doctor.ts @@ -15,6 +15,7 @@ import { browserProbe, browserWarnings, withBrowserReport } from './doctor-brows import { chartAlertsProbe, NOTHING_MISSING, withChartAlertsListing } from './doctor-chart-alerts'; import { coverageExcludesProbe, withCoverageExcludes } from './doctor-coverage'; import { embeddedDatabaseFinding } from './doctor-database'; +import { guardedCheck, guardedListing } from './doctor-guarded'; import { shippedGuardsProbe, vendoredGuardWarnings, withGuardListing } from './doctor-guards'; import { offlineFallbackFinding } from './doctor-offline'; import { NO_POOLER_FACT, poolerFindings, withPoolerReport } from './doctor-pooler'; @@ -70,9 +71,18 @@ export async function runDoctor(probe: DoctorProbe): Promise ); return findings; } + // Every check below through `check`: one that throws is that check's finding, not the report's end. + const check = (name: string, run: () => Promise): Promise => + guardedCheck(name, findings, run); + // First of the app's checks: an unfinished merge explains most of what any later one reports. + await check('merge state', async () => { + findings.push(...((await probe.mergeState?.()) ?? [])); + }); // Right after the runtime and for its reason: a compiler a major behind reds typecheck and build // on the framework's own files, and every finding below is read through a gate that cannot pass. - findings.push(...typescriptFindings(await probe.typescript())); + await check('typescript', async () => { + findings.push(...typescriptFindings(await probe.typescript())); + }); if (!probe.exists(ENV_DEVELOPMENT)) { findings.push( finding( @@ -93,7 +103,9 @@ export async function runDoctor(probe: DoctorProbe): Promise // Silent in development and on a shell naming no environment; reported where this machine names // a deploy. One list, one wording (`framework-env.ts`, #679). findings.push(...probe.frameworkFindings); - findings.push(...(await portFindings(probe))); + await check('port', async () => { + findings.push(...(await portFindings(probe))); + }); findings.push(...appUrlFindings(probe.appUrl, probe.port)); // `@ultimat3/pwa`'s own codes, not CLI twins of them. `X_PWA_NO_ICON_SOURCE` and // `X_PWA_NO_FALLBACK` used to be declared here for the same two conditions the package already @@ -116,29 +128,41 @@ export async function runDoctor(probe: DoctorProbe): Promise // `apps/web/app/offline.tsx`, which is not a route file at all (`assertRouteFilename` refuses // it), so no app could clear the finding and its own `fix:` did not either. `doctor-offline.ts` // holds the rule and the reasons. - const offline = offlineFallbackFinding(await probe.offlineFallback()); - if (offline !== undefined) findings.push(offline); - const pooler = await probe.pooler(); - findings.push(...poolerFindings(pooler)); - // A pooler's refusal is every other database probe's answer too, and its fix is the true one: this - // one's says "set DATABASE_URL" to an operator whose URL is right. - const database = pooler.refused === undefined ? await probe.database() : null; - if (database !== null) findings.push(database); - // The other half of the same question, and the half a bare VM lands on: `database()` is silent - // where there is no `DATABASE_URL`, and that silence IS the bare-VM configuration. - const embedded = embeddedDatabaseFinding(await probe.embeddedDatabase()); - if (embedded !== undefined) findings.push(embedded); - // Asked only of a database that answered: an unreachable one is already the finding above. - const reachable = pooler.refused === undefined && database === null; - if (reachable) findings.push(...((await probe.rowSecurity?.()) ?? [])); - findings.push(...(await probe.drift())); + await check('offline fallback', async () => { + const offline = offlineFallbackFinding(await probe.offlineFallback()); + if (offline !== undefined) findings.push(offline); + }); + await check('database', async () => { + const pooler = await probe.pooler(); + findings.push(...poolerFindings(pooler)); + // A pooler's refusal is every other database probe's answer too, and its fix is the true one: + // this one's says "set DATABASE_URL" to an operator whose URL is right. + const database = pooler.refused === undefined ? await probe.database() : null; + if (database !== null) findings.push(database); + // The other half of the same question, and the half a bare VM lands on: `database()` is silent + // where there is no `DATABASE_URL`, and that silence IS the bare-VM configuration. + const embedded = embeddedDatabaseFinding(await probe.embeddedDatabase()); + if (embedded !== undefined) findings.push(embedded); + // Asked only of a database that answered: an unreachable one is already the finding above. + const reachable = pooler.refused === undefined && database === null; + if (reachable) findings.push(...((await probe.rowSecurity?.()) ?? [])); + }); + await check('migration drift', async () => { + findings.push(...(await probe.drift())); + }); // Last, and it is why `X_CLI_UNEXPECTED`'s `fix: x doctor --json` is not a dead end on the path an // author reaches it from: `x db gen` throwing `X_MIGRATION_SNAPSHOT_MISSING` used to be a // condition this diagnostic could not see at all, so the fix line ran clean over a broken app. - findings.push(...(await probe.snapshots())); + await check('migration snapshots', async () => { + findings.push(...(await probe.snapshots())); + }); // Sealed columns against the key ring: no key at all, or a rotation whose re-seal is pending. - findings.push(...sealedKeyFindings(await probe.sealedKeys())); - findings.push(...authStorageFindings(await probe.authStorage())); + await check('sealed keys', async () => { + findings.push(...sealedKeyFindings(await probe.sealedKeys())); + }); + await check('auth storage', async () => { + findings.push(...authStorageFindings(await probe.authStorage())); + }); // `X_FRAMEWORK_PIN_RANGE`, `X_ENV_NAME_RESERVED`, `X_DB_ROW_SECURITY_OFF` and // `X_VERIFY_STEP_UNFLOORED` are not here: on `main` they warn beside the verdict // (`doctor-warnings.ts`), so an upgrade with no change keeps this green. @@ -156,10 +180,18 @@ export interface DoctorReportProbe extends DoctorProbe { /** The whole report over one probe: the verdict, then the listings and warnings beside it. */ export async function reportDoctor(probe: DoctorReportProbe): Promise { + // A listing that throws costs that listing, reported beside the rest (`doctor-guarded.ts`). + const failed: Finding[] = []; + const inApp = (name: string, read: (root: string) => Promise, none: T): Promise => { + const { root } = probe; + return root === undefined + ? Promise.resolve(none) + : guardedListing(name, failed, () => read(root), none); + }; // Asked once: `runDoctor` reads the verdict and the report beside it reads the numbers. - const pooler = probe.root === undefined ? NO_POOLER_FACT : await probe.pooler(); + const pooler = await inApp('pooler', () => probe.pooler(), NO_POOLER_FACT); const findings = await runDoctor({ ...probe, pooler: async () => pooler }); - const guardEntries = probe.root === undefined ? [] : await shippedGuardsProbe(probe.root); + const guardEntries = await inApp('shipped guards', shippedGuardsProbe, []); const listed = withGuardListing( { ok: findings.length === 0, @@ -177,37 +209,43 @@ export async function reportDoctor(probe: DoctorReportProbe): Promise chartAlertsProbe(root), NOTHING_MISSING), ); // The fourth: implicit indexes another already serves (`doctor-redundant-indexes.ts`). const indexed = withRedundantIndexes( charted, - probe.root === undefined ? undefined : await redundantIndexesProbe(probe.root), + await inApp('redundant indexes', redundantIndexesProbe, undefined), ); // A reminder, not a finding: the rolling rail is off until this is set (`doctor-rolling.ts`). const reminded = withPoolerReport( - withRollingReminder( - indexed, - probe.root === undefined ? undefined : await rollingSinceProbe(probe.root), - ), + withRollingReminder(indexed, await inApp('rolling deploys', rollingSinceProbe, undefined)), pooler, ); // Last, and beside the verdict on `main`: the 31.16.0 checks (`doctor-warnings.ts`). // A copy of a shipped guard among them (`X_GUARD_VENDORED`): a warning, so exit stays. - const browser = await probe.browser(); + const browser = await guardedListing('browser', failed, () => probe.browser(), undefined); + const warnings = await guardedListing('warnings', failed, () => doctorWarnings(probe), []); + const superseded = await inApp('superseded workarounds', supersededFindings, []); + const floors = await inApp('advisory floors', advisoryFloorFindings, []); + // An `x upgrade` driven by a CLI older than the codemod engine left files it owed. + const codemods = await inApp( + 'pending codemods', + (root) => pendingCodemodFindings(root, cliVersion()), + [], + ); return withDoctorWarnings(withBrowserReport(reminded, browser), [ - ...(await doctorWarnings(probe)), + ...warnings, ...vendoredGuardWarnings(guardEntries), ...browserWarnings(browser), - ...(probe.root === undefined ? [] : await supersededFindings(probe.root)), - ...(probe.root === undefined ? [] : await advisoryFloorFindings(probe.root)), - // An `x upgrade` driven by a CLI older than the codemod engine left files it owed. - ...(probe.root === undefined ? [] : await pendingCodemodFindings(probe.root, cliVersion())), + ...superseded, + ...floors, + ...codemods, + ...failed, ]); } diff --git a/packages/cli/src/cmd-errors.test.ts b/packages/cli/src/cmd-errors.test.ts index 624cf9b59..81f36e6e4 100644 --- a/packages/cli/src/cmd-errors.test.ts +++ b/packages/cli/src/cmd-errors.test.ts @@ -69,6 +69,15 @@ describe('unit · x errors explain', () => { ]); }); + test('explain prints the fix its reader can run, in the lines and the data', async () => { + const result = await errorsCommand.run({ + ...ctxFor(['errors', 'explain', 'X_DB_DRIFT']), + reader: { bunxRunner: true }, + }); + expect(String(record(result.data)['fix'])).toStartWith('bunx x db gen '); + expect(result.lines?.[1]).toStartWith(' fix: bunx x db gen '); + }); + test("a CLI code answers with the CLI's own fix, not the generic gate", async () => { const result = await run(['errors', 'explain', 'X_BUN_VERSION']); expect(record(result.data)['fix']).toBe('bun upgrade'); diff --git a/packages/cli/src/cmd-errors.ts b/packages/cli/src/cmd-errors.ts index e4bcdde00..6d9fc4fbb 100644 --- a/packages/cli/src/cmd-errors.ts +++ b/packages/cli/src/cmd-errors.ts @@ -15,6 +15,8 @@ import { ErrorCodeUnknownError, MissingPositionalError } from './errors'; import { explainErrorCode, explainEveryErrorCode } from './mcp-errors'; import { msg } from './messages'; import type { CommandResult, JsonValue } from './output'; +import type { ReaderOptions } from './output-findings'; +import { fixForReader } from './output-findings'; import { flagBool } from './parse'; export { ERRORS_SUBCOMMANDS } from './cmd-errors-spec'; @@ -25,21 +27,21 @@ export { ERRORS_SUBCOMMANDS } from './cmd-errors-spec'; * caller that has to regex `packages/x/src/y.ts:80` back out of an English sentence is a caller the * machine-readable surface failed. `null` when the installed framework raises the code nowhere. */ -const asJson = (explanation: ErrorExplanation): JsonValue => { +const asJson = (explanation: ErrorExplanation, reader?: ReaderOptions): JsonValue => { const site = codeFixes().get(explanation.code)?.[0]; return { code: explanation.code, cause: explanation.cause, - fix: explanation.fix, + fix: fixForReader(explanation.fix, reader), docs: explanation.docs, site: site === undefined ? null : { at: site.at, line: site.line }, }; }; /** The 3-line contract format, minus the leading blank code line `renderFinding` would add. */ -const detailLines = (explanation: ErrorExplanation): readonly string[] => [ +const detailLines = (explanation: ErrorExplanation, reader?: ReaderOptions): readonly string[] => [ ` cause: ${singleLine(explanation.cause)}`, - ` fix: ${singleLine(explanation.fix)}`, + ` fix: ${singleLine(fixForReader(explanation.fix, reader))}`, ` docs: ${singleLine(explanation.docs)}`, ]; @@ -47,7 +49,11 @@ const detailLines = (explanation: ErrorExplanation): readonly string[] => [ * A registered code, else a guard's — a shipped guard's or this app's own (`cmd-errors-guards.ts`): * the gate prints both, and the app's `CLAUDE.md` sends every printed code here. */ -async function explainOne(code: string, cwd: string): Promise { +async function explainOne( + code: string, + cwd: string, + reader?: ReaderOptions, +): Promise { const explanation = explainErrorCode(code) ?? (await explainGuardCode(code, cwd)); if (explanation === undefined) { const suggestion = nearestName( @@ -60,8 +66,8 @@ async function explainOne(code: string, cwd: string): Promise { ok: true, command: 'errors', summary: msg('cli.errors.explained', { code: explanation.code, title: explanation.cause }), - lines: detailLines(explanation), - data: asJson(explanation), + lines: detailLines(explanation, reader), + data: asJson(explanation, reader), }; } @@ -70,7 +76,7 @@ async function explainOne(code: string, cwd: string): Promise { * this answer and something is broken that a `fix:` can address. Reported as findings — and `ok` * goes false — so the incomplete catalog cannot read as a complete one. */ -function listAll(catalog: ErrorCatalog): CommandResult { +function listAll(catalog: ErrorCatalog, reader?: ReaderOptions): CommandResult { const all = explainEveryErrorCode(); return { ok: catalog.failed.length === 0, @@ -79,7 +85,7 @@ function listAll(catalog: ErrorCatalog): CommandResult { lines: all.map((entry) => ` ${entry.code.padEnd(30)} ${singleLine(entry.cause)}`), findings: catalog.failed, data: { - codes: all.map(asJson), + codes: all.map((entry) => asJson(entry, reader)), unavailable: [...catalog.unavailable], failed: catalog.failed.map((finding) => ({ code: finding.code, @@ -100,7 +106,7 @@ export const errorsCommand: CliCommand = { // and the fix index is what each one instructs. `x errors list` answering 397 codes with the // fallback line would be a complete-looking table of shrugs. const [catalog] = await Promise.all([loadErrorCatalog(), loadCodeFixes()]); - if (ctx.args.subcommand === 'list') return listAll(catalog); + if (ctx.args.subcommand === 'list') return listAll(catalog, ctx.reader); const code = ctx.args.positionals[0]; if (code === undefined) { // Not a `BadFlagError`: naming `--code` invented a flag that does not exist, so an agent @@ -114,8 +120,8 @@ export const errorsCommand: CliCommand = { } // `--recent`: what THIS app has seen of the code — groups and the bundles `x repro` replays. if (flagBool(ctx.args, 'recent')) { - return withRecent(ctx, code, await explainOne(code, ctx.cwd)); + return withRecent(ctx, code, await explainOne(code, ctx.cwd, ctx.reader)); } - return explainOne(code, ctx.cwd); + return explainOne(code, ctx.cwd, ctx.reader); }, }; diff --git a/packages/cli/src/cmd-generate-entity.test.ts b/packages/cli/src/cmd-generate-entity.test.ts index c808c02f9..cea441971 100644 --- a/packages/cli/src/cmd-generate-entity.test.ts +++ b/packages/cli/src/cmd-generate-entity.test.ts @@ -143,7 +143,9 @@ describe('unit · x g entity writes a repo over the typed handle', () => { ]); const handle = await read(HANDLE_FILE); expect(handle).toContain("import { widget } from '@gen/web/app/widget/entity';"); - expect(handle).toContain('const entities = {\n widgets: widget,\n};'); + expect(handle).toContain( + 'const entities = {\n // One entry per line: branches that each add one merge cleanly (x merge-driver).\n widgets: widget,\n};', + ); // The edges the two new imports are: db reads the entity, the slice reads the handle. expect(await read('packages/db/package.json')).toContain('"@gen/web": "0.0.0"'); expect(await read('apps/web/package.json')).toContain('"@gen/db": "0.0.0"'); @@ -198,7 +200,7 @@ describe('unit · x g entity writes a repo over the typed handle', () => { const result = await generateCommand.run(contextFor('gadget')); expect(result.ok).toBe(true); expect(await read(HANDLE_FILE)).toContain( - 'const entities = {\n gadgets: gadget,\n widgets: widget,\n};', + 'const entities = {\n // One entry per line: branches that each add one merge cleanly (x merge-driver).\n gadgets: gadget,\n widgets: widget,\n};', ); expect(nextOf(result.data)).toEqual(['bunx x db gen "create gadgets"', 'bunx x db migrate']); }); diff --git a/packages/cli/src/cmd-heavy-spec.ts b/packages/cli/src/cmd-heavy-spec.ts index 67f003bb6..ec12323ad 100644 --- a/packages/cli/src/cmd-heavy-spec.ts +++ b/packages/cli/src/cmd-heavy-spec.ts @@ -7,7 +7,8 @@ export const heavySpec: CommandSpec = { name: 'heavy', summary: "the machine's shared memory ledger: who is running, who is queued, and any command run inside it", - usage: 'x heavy [status] [--json] · x heavy run [--label s] [--mb n] [--json] -- ', + usage: + 'x heavy [status] [--json] · x heavy run [--label s] [--mb n] [--max-wait s] [--json] -- ', subcommands: ['status', 'run'], defaultSubcommand: 'status', passthrough: true, @@ -24,5 +25,12 @@ export const heavySpec: CommandSpec = { summary: 'the memory to reserve, its measured peak (default: a whole-repo typecheck)', subcommands: ['run'], }, + { + name: 'max-wait', + type: 'string', + summary: + 'seconds to stay queued before giving up with X_HEAVY_WAIT_EXPIRED, exit 75 (default: wait)', + subcommands: ['run'], + }, ], }; diff --git a/packages/cli/src/cmd-heavy.test.ts b/packages/cli/src/cmd-heavy.test.ts index c1fc691f5..1cde348e3 100644 --- a/packages/cli/src/cmd-heavy.test.ts +++ b/packages/cli/src/cmd-heavy.test.ts @@ -51,7 +51,7 @@ async function statusOf(env: Record) { return JSON.parse(out).data as { enabled: boolean; capacityMb: number; - holds: { label: string }[]; + holds: { label: string; pid: number; wrapperAlive: boolean; childPids: number[] }[]; queue: { label: string }[]; }; } @@ -129,12 +129,69 @@ describe.skipIf(NO_FLOCK)('x heavy run', () => { const workload = Number(new TextDecoder().decode(value).trim()); holder.kill('SIGKILL'); await holder.exited; - expect((await statusOf(env)).holds.map((hold) => hold.label)).toEqual(['doomed']); + const [hold] = (await statusOf(env)).holds; + expect(hold?.label).toBe('doomed'); + // The pid that can still free the memory, beside the one that can no longer. + expect(hold?.pid).toBe(holder.pid); + expect(hold?.wrapperAlive).toBe(false); + expect(hold?.childPids).toEqual([workload]); process.kill(workload, 'SIGKILL'); // Each read is a whole process, so this loop paces itself. while ((await statusOf(env)).holds.length > 0); }, 30_000); + test('status names a live wrapper and the command it started', async () => { + const env = ledgerEnv(); + const holder = x( + ['heavy', 'run', '--mb', '100', '--', 'sh', '-c', 'echo $$; exec sleep 60'], + env, + ); + const { value } = await holder.stdout.getReader().read(); + const workload = Number(new TextDecoder().decode(value).trim()); + const [hold] = (await statusOf(env)).holds; + expect(hold?.wrapperAlive).toBe(true); + expect(hold?.childPids).toEqual([workload]); + const { out } = await finished(x(['heavy', 'status'], env)); + expect(out).toContain(`command pid ${String(workload)}`); + process.kill(workload, 'SIGKILL'); + }, 30_000); + + test('--max-wait gives up with its own code and exit, and the command never starts', async () => { + const env = ledgerEnv(); + const holder = await holding(env, 'first'); + const run = await finished( + x( + // biome-ignore format: one argv + ['heavy', 'run', '--json', '--label', 'late', '--max-wait', '0', '--mb', '100', '--', 'sh', '-c', 'echo workload-output'], + env, + ), + ); + expect(run.code).toBe(75); + const document = JSON.parse(run.out); + expect(document.findings[0].code).toBe('X_HEAVY_WAIT_EXPIRED'); + expect(document.findings[0].cause).toContain('first'); + expect(document.data).toMatchObject({ exitCode: 75, started: false }); + expect(run.err).not.toContain('workload-output'); + expect((await statusOf(env)).queue).toEqual([]); + holder.kill('SIGKILL'); + await holder.exited; + }, 30_000); + + test('--max-wait with room to run changes nothing', async () => { + const run = await finished( + x(['heavy', 'run', '--max-wait', '30', '--mb', '1', '--', 'sh', '-c', 'exit 7'], ledgerEnv()), + ); + expect(run.code).toBe(7); + }, 30_000); + + test('a --max-wait that is no number is refused', async () => { + const run = await finished( + x(['heavy', 'run', '--json', '--max-wait', 'soon', '--', 'true'], ledgerEnv()), + ); + expect(run.code).toBe(1); + expect(JSON.parse(run.out).findings[0].code).toBe('X_CLI_BAD_FLAG'); + }, 30_000); + test('a command that does not exist is 127 with a coded finding', async () => { const run = await finished( x(['heavy', 'run', '--json', '--', 'no-such-tool-here'], ledgerEnv()), @@ -157,6 +214,20 @@ describe.skipIf(NO_FLOCK)('x heavy status', () => { expect(data.capacityMb).toBe(2048); }, 30_000); + test('outside a test run it reports the CPU reading the queue is judged by', async () => { + const env = { ...ledgerEnv('2g'), NODE_ENV: 'production' }; + const { out } = await finished(x(['heavy', 'status', '--json'], env)); + const document = JSON.parse(out); + expect(document.data.cpuGate).toBe(true); + expect(['psi', 'runnable']).toContain(document.data.machine.cpu.source); + const text = await finished(x(['heavy', 'status'], env)); + expect(text.out).toContain('cpu: '); + }, 30_000); + + test('inside a test run the CPU gate is off, and says so in --json', async () => { + expect(((await statusOf(ledgerEnv())) as unknown as { cpuGate: boolean }).cpuGate).toBe(false); + }, 30_000); + test('says why admission is off under CI', async () => { const { out } = await finished(x(['heavy', '--json'], { ...ledgerEnv(), CI: 'true' })); expect(JSON.parse(out).data).toMatchObject({ enabled: false, off: 'ci' }); diff --git a/packages/cli/src/cmd-heavy.ts b/packages/cli/src/cmd-heavy.ts index 16f86a5c7..7aab8814a 100644 --- a/packages/cli/src/cmd-heavy.ts +++ b/packages/cli/src/cmd-heavy.ts @@ -10,9 +10,12 @@ import { MissingPositionalError } from './errors'; import { readIntFlag } from './flag-number'; import { describeEntry, heavyState, withHeavy } from './heavy'; import { coreCap, runnableTasks } from './heavy-cores'; -import { HeavyCommandNotFoundError } from './heavy-errors'; +import { HeavyCommandNotFoundError, HeavyWaitExpiredError } from './heavy-errors'; +import { holderPids, isProcessRunning } from './heavy-holders'; +import type { LedgerEntry, LedgerHold } from './heavy-ledger'; import { readLedger } from './heavy-ledger'; import { availableMemoryBytes, usableMemoryBytes } from './heavy-memory'; +import { cpuGateOn, cpuText, readCpuPressure } from './heavy-pressure'; import { DEFAULT_RUN_MB } from './heavy-weights'; import type { HandOffChild } from './local-cli-handoff'; import { handOff } from './local-cli-handoff'; @@ -21,8 +24,35 @@ import type { CommandResult, JsonValue } from './output'; import { findingFrom } from './output'; import { flagString } from './parse'; +/** Read through `readIntFlag`, which refuses `5m`, `NaN` and a fraction with the flag's name. */ +const MAX_WAIT_FLAG = 'max-wait'; + const gb = (bytes: number): string => (bytes / 2 ** 30).toFixed(1); const MAX_RUN_MB = 1024 * 1024; +const MAX_WAIT_SECONDS = 7 * 24 * 60 * 60; +/** `EX_TEMPFAIL`: never the command's own answer, because the command never ran. */ +const WAIT_EXPIRED_EXIT = 75; + +/** A reservation by the processes keeping it: its wrapper may be gone while its command runs on. */ +async function holdProcesses(hold: LedgerHold, ctx: CommandContext) { + const pids = await holderPids(hold.file, ctx.runner, ctx.cwd); + return { + wrapperAlive: pids === undefined ? isProcessRunning(hold.pid) : pids.includes(hold.pid), + childPids: (pids ?? []).filter((pid) => pid !== hold.pid), + }; +} + +type HoldProcesses = Awaited>; + +const pidsText = (hold: LedgerHold, processes: HoldProcesses): string => + [ + msg(processes.wrapperAlive ? 'cli.heavy.status.wrapper' : 'cli.heavy.status.wrapperGone', { + pid: hold.pid, + }), + ...(processes.childPids.length === 0 + ? [] + : [msg('cli.heavy.status.command', { pids: processes.childPids.join(', ') })]), + ].join(', '); async function status(ctx: CommandContext): Promise { const state = await heavyState({ ...ctx.env, ULTIMATE_HEAVY_HOLD: undefined }); @@ -32,14 +62,21 @@ async function status(ctx: CommandContext): Promise { const coreText = cores === undefined ? String(availableParallelism()) : String(cores); const usedMb = view.holds.reduce((sum, entry) => sum + entry.mb, 0); const [one = 0, five = 0, fifteen = 0] = loadavg(); + const cpu = readCpuPressure(); + // The gate applies where admission does; the reading is shown either way. + const cpuGate = state.on && cpuGateOn(ctx.env); const machine = { cores: availableParallelism(), runnable: runnableTasks(), load: { one, five, fifteen }, totalBytes: usableMemoryBytes(totalmem()), availableBytes: availableMemoryBytes(), + cpu: { ...cpu }, }; - const entryJson = (entry: (typeof view.holds)[number]): JsonValue => ({ + const processes = await Promise.all(view.holds.map((hold) => holdProcesses(hold, ctx))); + const processesOf = (at: number): HoldProcesses => + processes[at] ?? { wrapperAlive: false, childPids: [] }; + const entryJson = (entry: LedgerEntry): JsonValue => ({ pid: entry.pid, mb: entry.mb, label: entry.label, @@ -66,8 +103,17 @@ async function status(ctx: CommandContext): Promise { free: gb(machine.availableBytes), total: gb(machine.totalBytes), }), - ...view.holds.map((entry) => - msg('cli.heavy.status.hold', { entry: describeEntry(entry, now) }), + cpuGate + ? msg('cli.heavy.status.cpu', { cpu: cpuText(cpu) }) + : msg('cli.heavy.status.cpuOff', { + cpu: cpuText(cpu), + reason: state.on ? 'a test run, or ULTIMATE_HEAVY_CPU=0' : state.off, + }), + ...view.holds.map((entry, at) => + msg('cli.heavy.status.hold', { + entry: describeEntry(entry, now), + pids: pidsText(entry, processesOf(at)), + }), ), ...view.queue.map((entry, at) => msg('cli.heavy.status.queued', { position: at + 1, entry: describeEntry(entry, now) }), @@ -79,8 +125,16 @@ async function status(ctx: CommandContext): Promise { ? { dir: state.dir, capacityMb: state.capacityMb, usedMb } : { off: state.off, ...(state.detail === undefined ? {} : { detail: state.detail }) }), coreCap: cores ?? null, + cpuGate, machine, - holds: view.holds.map(entryJson), + holds: view.holds.map((entry, at): JsonValue => { + const { wrapperAlive, childPids } = processesOf(at); + return { + ...(entryJson(entry) as Record), + wrapperAlive, + childPids: [...childPids], + }; + }), queue: view.queue.map(entryJson), }, }; @@ -113,31 +167,61 @@ async function run(ctx: CommandContext): Promise { max: MAX_RUN_MB, example: 'x heavy run --mb 4096 -- bun test', }); + const maxWait = readIntFlag(ctx.args, { + name: MAX_WAIT_FLAG, + command: 'heavy', + min: 0, + max: MAX_WAIT_SECONDS, + example: 'x heavy run --max-wait 300 -- bun test', + }); const label = flagString(ctx.args, 'label') ?? command.slice(0, 3).join(' '); const queued = performance.now(); let startedAt = queued; - const code = await withHeavy( - { label, cwd: ctx.cwd, env: ctx.env, unitMb: mb ?? DEFAULT_RUN_MB }, - (child) => { - startedAt = performance.now(); - return handOff( - command, - { ...ctx.env, ...child.env }, - { - spawn: (argv, env): HandOffChild => - // Untimed: the child is the command the caller asked to run, and its stop signals are - // forwarded by `handOff`. Counted under `cli`'s spawn-timeout pin with that hand-off. - Bun.spawn([...argv], { - cwd: ctx.cwd, - env, - // The reservation's lock goes with it: killing this wrapper frees nothing the - // command is still using. - stdio: ['inherit', ctx.args.json ? 2 : 'inherit', 'inherit', ...child.fds], - }), - }, - ); - }, - ); + const request = { + label, + cwd: ctx.cwd, + env: ctx.env, + unitMb: mb ?? DEFAULT_RUN_MB, + ...(maxWait === undefined ? {} : { maxWaitMs: maxWait * 1000 }), + }; + const admitted = withHeavy(request, (child) => { + startedAt = performance.now(); + return handOff( + command, + { ...ctx.env, ...child.env }, + { + spawn: (argv, env): HandOffChild => + // Untimed: the child is the command the caller asked to run, and its stop signals are + // forwarded by `handOff`. Counted under `cli`'s spawn-timeout pin with that hand-off. + Bun.spawn([...argv], { + cwd: ctx.cwd, + env, + // The reservation's lock goes with it: killing this wrapper frees nothing the + // command is still using. + stdio: ['inherit', ctx.args.json ? 2 : 'inherit', 'inherit', ...child.fds], + }), + }, + ); + }); + let code: number; + try { + code = await admitted; + } catch (error) { + if (!(error instanceof HeavyWaitExpiredError)) throw error; + const finding = findingFrom(error); + return { + ...envelope, + ok: false, + summary: finding.cause, + findings: [finding], + exitCode: WAIT_EXPIRED_EXIT, + data: { + exitCode: WAIT_EXPIRED_EXIT, + started: false, + waitedMs: Math.round(performance.now() - queued), + }, + }; + } const ranMs = performance.now() - startedAt; const waitedMs = startedAt - queued; return { diff --git a/packages/cli/src/cmd-merge-driver-spec.ts b/packages/cli/src/cmd-merge-driver-spec.ts new file mode 100644 index 000000000..710b71e90 --- /dev/null +++ b/packages/cli/src/cmd-merge-driver-spec.ts @@ -0,0 +1,12 @@ +// `x merge-driver`'s declaration, apart from its body: the parser, `x help` and the `errors` step +// read it without loading `cmd-merge-driver.ts`, which `registry.ts` imports only when it runs. + +import type { CommandSpec } from './parse'; + +export const mergeDriverSpec: CommandSpec = { + name: 'merge-driver', + summary: 'merges what parallel branches added to the registries; git runs it, setup installs it', + usage: 'x merge-driver install [--json] · x merge-driver run ', + subcommands: ['install', 'run'], + flags: [], +}; diff --git a/packages/cli/src/cmd-merge-driver.ts b/packages/cli/src/cmd-merge-driver.ts new file mode 100644 index 000000000..bd9610fb9 --- /dev/null +++ b/packages/cli/src/cmd-merge-driver.ts @@ -0,0 +1,92 @@ +// `x merge-driver`: `install` writes the two git drivers into this clone, `run` is the one git +// calls for a registry both sides changed. Exit 0 is "merged"; anything else leaves the file holding +// its remaining conflict markers and git reporting it as conflicted, as for any text merge. + +import { ERROR_DOCS_URL } from '@ultimat3/core'; +import { mergeDriverSpec } from './cmd-merge-driver-spec'; +import type { CliCommand, CommandContext } from './command'; +import { failed, ok } from './command'; +import { MissingPositionalError } from './errors'; +import { mergeCatalogText } from './merge-driver-catalog'; +import { driverConfig, gitPrefix } from './merge-driver-config'; +import { resolveAdditions } from './merge-driver-lines'; +import { UNMERGED_FIX } from './merge-state'; +import type { CommandResult, Finding } from './output'; + +const NAME = mergeDriverSpec.name; + +async function install(ctx: CommandContext): Promise { + const prefix = await gitPrefix(ctx.cwd, ctx.runner); + if (prefix === undefined) { + return ok(NAME, 'not a git checkout: no merge drivers to install', { + data: { installed: false }, + }); + } + for (const [key, value] of driverConfig(prefix)) { + const set = await ctx.runner(['git', 'config', '--local', key, value], { cwd: ctx.cwd }); + if (!set.ok) { + return failed(NAME, `could not set ${key}`, { + findings: [ + { + code: 'X_MERGE_DRIVER_UNSET', + cause: `git config refused ${key}: ${set.stderr.trim()}`, + fix: 'bunx x merge-driver install', + docs: ERROR_DOCS_URL, + at: '.git/config', + }, + ], + }); + } + } + return ok(NAME, 'merge drivers installed for this clone', { data: { installed: true } }); +} + +const conflictFinding = (path: string, cause: string): Finding => ({ + code: 'X_MERGE_UNRESOLVED', + cause, + fix: UNMERGED_FIX, + docs: ERROR_DOCS_URL, + at: path, +}); + +async function run(ctx: CommandContext): Promise { + const [base, ours, theirs, path] = ctx.args.positionals; + if (base === undefined || ours === undefined || theirs === undefined || path === undefined) { + throw new MissingPositionalError({ + command: 'merge-driver run', + positional: 'base> => Bun.file(file).text(); + if (path.endsWith('.json')) { + const merged = mergeCatalogText(await text(base), await text(ours), await text(theirs)); + if (merged !== undefined) { + await Bun.write(ours, merged); + return ok(NAME, `${path}: merged both sides' keys`, { data: { path, conflicts: 0 } }); + } + } + const labels = ['-L', 'ours', '-L', 'base', '-L', 'theirs']; + const argv = ['git', 'merge-file', '-p', '--diff3', ...labels, ours, base, theirs]; + const merge = await ctx.runner(argv, { cwd: ctx.cwd }); + // A positive exit is how many conflicts it left; git's own failures are negative, so 255 here. + if (merge.code < 0 || merge.code > 127) { + return failed(NAME, `${path}: git merge-file failed`, { + findings: [conflictFinding(path, `git merge-file refused ${path}: ${merge.stderr.trim()}`)], + }); + } + const { text: result, conflicts } = path.endsWith('.json') + ? { text: merge.stdout, conflicts: merge.code } + : resolveAdditions(merge.stdout); + await Bun.write(ours, result); + if (conflicts === 0) { + return ok(NAME, `${path}: merged both sides' additions`, { data: { path, conflicts } }); + } + const cause = `${path}: both branches changed the same existing lines (${conflicts} conflict(s))`; + return failed(NAME, cause, { findings: [conflictFinding(path, cause)] }); +} + +export const mergeDriverCommand: CliCommand = { + spec: mergeDriverSpec, + run: (ctx) => (ctx.args.subcommand === 'install' ? install(ctx) : run(ctx)), +}; diff --git a/packages/cli/src/cmd-new.test.ts b/packages/cli/src/cmd-new.test.ts index 22eb4b864..e5f4444ab 100644 --- a/packages/cli/src/cmd-new.test.ts +++ b/packages/cli/src/cmd-new.test.ts @@ -10,7 +10,7 @@ import { existsSync, mkdtempSync, readdirSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; // why: Bun exposes no path-join primitive; Bun.file and import() take one already joined. import { join } from 'node:path'; -import { decodeImage, probeImage } from '@ultimat3/core'; +import { decodeImage, probeImage, stripComments } from '@ultimat3/core'; import type { NestedCatalog } from '@ultimat3/i18n'; import { catalogKeys, defineCatalogs, loadCatalog } from '@ultimat3/i18n'; import { BuiltinImagePipeline } from '@ultimat3/pwa'; @@ -344,7 +344,8 @@ describe('unit · x new · the API surface registers the app own primitives by n for (const job of jobs) expect([job, imported.includes(job)]).toEqual([job, true]); // Handed over as JOBS: `actions: [reindexPost]` would register nothing and read as correct. const list = /jobs: \[(?[^\]]*)\]/.exec(apiIndex(true))?.groups?.['names'] ?? ''; - expect(list.split(',').map((entry) => entry.trim())).toContain('reindexPost'); + const entries = stripComments(list).split(','); + expect(entries.map((entry) => entry.trim())).toContain('reindexPost'); }); }); diff --git a/packages/cli/src/cmd-new.ts b/packages/cli/src/cmd-new.ts index 6eb4f40c4..c08867019 100644 --- a/packages/cli/src/cmd-new.ts +++ b/packages/cli/src/cmd-new.ts @@ -23,6 +23,7 @@ import { runShapeGenerators } from './new-shapes-generate'; import type { CommandResult } from './output'; import { flagBool, flagString } from './parse'; import { posixRelative } from './posix-path'; +import { withRegistryLines } from './registry-lines'; import { quoteArg } from './shell-quote'; import type { GeneratedFile } from './templates'; import { appFiles, names, repoFiles, resourceFiles } from './templates'; @@ -141,11 +142,23 @@ const cdTarget = (cwd: string, target: string): string => quoteArg(displayPath(c const shapeGenerateLines = (shape: NewShape): readonly string[] => (SHAPE_OVERLAYS[shape].generate ?? []).map((argv) => ['x', 'g', ...argv].join(' ')); -/** Pure: the complete file list for a new app, so `--dry-run` and the test see the same thing. */ +/** + * Pure: the complete file list for a new app, so `--dry-run` and the test see the same thing. The + * registries take their one-entry-per-line shape last: a shape's overlay edits the lists as the + * templates print them. + */ export function planNewApp(options: NewAppOptions): readonly GeneratedFile[] { + return planFiles(options).map((file) => + typeof file.contents === 'string' && file.merge === undefined + ? { ...file, contents: withRegistryLines(file.path, file.contents) } + : file, + ); +} + +function planFiles(options: NewAppOptions): readonly GeneratedFile[] { if (options.shape !== undefined) { const overlay = SHAPE_OVERLAYS[options.shape]; - const base = planNewApp({ name: options.name, example: overlay.example }); + const base = planFiles({ name: options.name, example: overlay.example }); // Merge-aware, as below: a shape's catalog keys land in the one catalog the base wrote. return withDeclaredExercises(dedupe(applyOverlay(base, overlay, names(options.name)))); } diff --git a/packages/cli/src/cmd-test.ts b/packages/cli/src/cmd-test.ts index a5ceaa5f3..b83adf7c3 100644 --- a/packages/cli/src/cmd-test.ts +++ b/packages/cli/src/cmd-test.ts @@ -16,7 +16,8 @@ import { appTestScope, narrowToApp } from './affected-tests'; import { testSpec } from './cmd-test-spec'; import type { CliCommand, CommandContext } from './command'; import { ok } from './command'; -import { BadFlagError, NoTestFilesError } from './errors'; +import { BadFlagError } from './errors'; +import { NoTestFilesError } from './errors-suite'; import type { Runner } from './exec'; import { readIntFlag } from './flag-number'; import { FREE_RUN_FILES } from './heavy-weights'; diff --git a/packages/cli/src/cmd-verify-typecheck.test.ts b/packages/cli/src/cmd-verify-typecheck.test.ts index f12c870e5..1fd9ab4e1 100644 --- a/packages/cli/src/cmd-verify-typecheck.test.ts +++ b/packages/cli/src/cmd-verify-typecheck.test.ts @@ -28,6 +28,30 @@ const ctx: VerifyContext = { }; describe('unit · x verify · typecheckBin', () => { + // The errors are printed under the finding: the fix is to edit them and re-run the ledgered step, + // never a raw `tsc` that repeats them outside the machine's memory accounting. + test('a red typecheck names the edit and the step, not raw tsc', async () => { + const step = VERIFY_STEPS.find((candidate) => candidate.name === 'typecheck'); + const root = await mkdtemp(join(tmpdir(), 'x-typecheck-fix-')); + try { + const runner: Runner = async (command) => ({ + command, + code: 2, + ok: false, + stdout: 'a.ts(1,1): error TS1: no', + stderr: '', + durationMs: 0, + }); + const outcome = await step?.run({ ...ctx, root, runner }); + const fix = outcome?.findings[0]?.fix ?? ''; + expect(fix).toStartWith('edit a.ts:1 — '); + expect(fix).toEndWith('then: x verify --only typecheck'); + expect(fix).not.toContain('tsc -'); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + // `x.verify.json`'s `typecheckBin` swaps the binary `bunx` invokes and nothing else — same // mode, same step, same finding either way. Absent means `tsc`, unchanged. test("the typecheck step runs the floor's typecheckBin, and tsc without one", async () => { diff --git a/packages/cli/src/cmd-verify.ts b/packages/cli/src/cmd-verify.ts index 7b4da26cf..983f60044 100644 --- a/packages/cli/src/cmd-verify.ts +++ b/packages/cli/src/cmd-verify.ts @@ -82,7 +82,10 @@ export const verifyCommand: CliCommand = { ...progress, }), // A line per step a person reads, JSON only for the `--json` reader that parses stderr. - { render: ctx.args.json ? 'json' : 'human' }, + { + render: ctx.args.json ? 'json' : 'human', + ...(ctx.reader === undefined ? {} : { reader: ctx.reader }), + }, ); const sharded = shard === undefined ? result : withShardData(result, shard); return changed === undefined ? sharded : withChangedData(sharded, changed); diff --git a/packages/cli/src/codemods/agents-doc-row.ts b/packages/cli/src/codemods/agents-doc-row.ts new file mode 100644 index 000000000..0449cadd9 --- /dev/null +++ b/packages/cli/src/codemods/agents-doc-row.ts @@ -0,0 +1,73 @@ +// One row of an app's agent brief, changed by a codemod: found in CLAUDE.md or AGENTS.md, held to +// the byte budget the manifest step enforces, and reported as a skip with the row itself whenever +// it cannot be placed. A silent `changed: []` left the row owed with nobody told. + +import { AGENTS_MD_MAX_BYTES } from '@ultimat3/manifest'; +import { parseVerifyFloor, VERIFY_FLOOR_FILE } from '../verify-floor'; +import type { CodemodApp, CodemodResult } from './codemod'; +import { readText } from './engine-edits'; + +/** Where an app keeps its brief: the scaffold writes CLAUDE.md, and some apps moved it. */ +const BRIEFS = ['CLAUDE.md', 'AGENTS.md'] as const; + +export interface AgentsDocRow { + /** The text the row is placed against, and what it becomes: an insert or a swap. */ + readonly from: string; + readonly to: string; + /** The row a person adds when it cannot be placed. */ + readonly row: string; + /** Already said, in the app's own words or the template's: nothing to do. */ + readonly said: (text: string) => boolean; + /** Why `from` is missing, in the skip a person reads. */ + readonly missing: string; +} + +/** `x.verify.json`'s `agentsMdMaxBytes`, else the manifest's default: the one the gate holds. */ +async function budgetOf(app: CodemodApp): Promise { + const floor = await app.read(VERIFY_FLOOR_FILE); + if (floor === undefined) return AGENTS_MD_MAX_BYTES; + try { + return parseVerifyFloor(floor).agentsMdMaxBytes ?? AGENTS_MD_MAX_BYTES; + } catch { + // An unreadable floor is the gate's own finding (`floorProblemFindings`); the default stands. + return AGENTS_MD_MAX_BYTES; + } +} + +const skip = (path: string, reason: string): CodemodResult => ({ + changed: [], + skipped: [{ path, reason }], +}); + +export async function editAgentsDocRow( + app: CodemodApp, + edit: AgentsDocRow, +): Promise { + const briefs: { path: string; text: string; restore: (text: string) => string }[] = []; + for (const path of BRIEFS) { + const file = await readText(app, path); + if (file !== undefined) briefs.push({ path, ...file }); + } + if (briefs.length === 0 || briefs.some((brief) => edit.said(brief.text))) { + return { changed: [], skipped: [] }; + } + const target = briefs.find((brief) => brief.text.split(edit.from).length === 2); + if (target === undefined) { + const path = briefs[0]?.path ?? BRIEFS[0]; + return skip( + path, + `${edit.missing}; add this row to the conventions table by hand: ${edit.row}`, + ); + } + const next = target.text.replace(edit.from, () => edit.to); + const bytes = Buffer.byteLength(target.restore(next), 'utf8'); + const budget = await budgetOf(app); + if (bytes > budget) { + return skip( + target.path, + `the row would take ${target.path} to ${String(bytes)} bytes, over its ${String(budget)}-byte budget (${VERIFY_FLOOR_FILE} agentsMdMaxBytes); make room, then add this row by hand: ${edit.row}`, + ); + } + await app.write(target.path, target.restore(next)); + return { changed: [target.path], skipped: [] }; +} diff --git a/packages/cli/src/codemods/docs-many-agents.test.ts b/packages/cli/src/codemods/docs-many-agents.test.ts index c1b4bc076..fec1bf7a7 100644 --- a/packages/cli/src/codemods/docs-many-agents.test.ts +++ b/packages/cli/src/codemods/docs-many-agents.test.ts @@ -1,6 +1,5 @@ import { describe, expect, test } from 'bun:test'; -import { names } from '../templates/naming'; -import { docsFiles } from '../templates/scaffold-docs'; +import { AGENTS_MD_MAX_BYTES } from '@ultimat3/manifest'; import { memoryCodemodApp } from './codemod'; import { codemod, MANY_AGENTS_ROW } from './docs-many-agents'; @@ -15,13 +14,38 @@ describe('docs-many-agents', () => { expect((await codemod.run(app)).changed).toEqual([]); }); + // 4 of 5 real apps reported `changed: [], skipped: []` here: the row was owed and nobody was told. test('leaves an app that reworded the table alone', async () => { const app = memoryCodemodApp({ 'CLAUDE.md': '| Rules | mine |\n' }); - expect(await codemod.run(app)).toEqual({ changed: [], skipped: [] }); + const result = await codemod.run(app); + expect(result.changed).toEqual([]); + expect(result.skipped.map((skip) => skip.path)).toEqual(['CLAUDE.md']); + expect(result.skipped[0]?.reason).toContain(MANY_AGENTS_ROW); + expect(app.files['CLAUDE.md']).toBe('| Rules | mine |\n'); }); - test('says what a new app is scaffolded with', () => { - const written = docsFiles(names('shop')).find((file) => file.path === 'CLAUDE.md'); - expect(written?.contents).toContain(MANY_AGENTS_ROW); + test('finds the table in AGENTS.md when CLAUDE.md does not hold it', async () => { + const app = memoryCodemodApp({ 'CLAUDE.md': 'See AGENTS.md.\n', 'AGENTS.md': TABLE }); + expect(await codemod.run(app)).toEqual({ changed: ['AGENTS.md'], skipped: [] }); + expect(app.files['AGENTS.md']).toContain(`${MANY_AGENTS_ROW}\n| One way |`); + expect((await codemod.run(app)).changed).toEqual([]); + }); + + // One real app went to 12,071 bytes and the manifest step refused it (`X_AGENTS_MD_TOO_LARGE`). + test('a row that would take the file over its byte budget is skipped with the count', async () => { + const full = `${'x'.repeat(AGENTS_MD_MAX_BYTES - TABLE.length - 10)}\n${TABLE}`; + const app = memoryCodemodApp({ 'CLAUDE.md': full }); + const result = await codemod.run(app); + expect(result.changed).toEqual([]); + expect(result.skipped[0]?.reason).toContain(String(AGENTS_MD_MAX_BYTES)); + expect(result.skipped[0]?.reason).toContain(MANY_AGENTS_ROW); + expect(app.files['CLAUDE.md']).toBe(full); + }); + + test('a budget raised in x.verify.json is the one the row is held to', async () => { + const full = `${'x'.repeat(AGENTS_MD_MAX_BYTES - TABLE.length - 10)}\n${TABLE}`; + const floor = JSON.stringify({ steps: [], agentsMdMaxBytes: AGENTS_MD_MAX_BYTES + 1000 }); + const app = memoryCodemodApp({ 'CLAUDE.md': full, 'x.verify.json': floor }); + expect((await codemod.run(app)).changed).toEqual(['CLAUDE.md']); }); }); diff --git a/packages/cli/src/codemods/docs-many-agents.ts b/packages/cli/src/codemods/docs-many-agents.ts index 243fafd27..689419f48 100644 --- a/packages/cli/src/codemods/docs-many-agents.ts +++ b/packages/cli/src/codemods/docs-many-agents.ts @@ -1,30 +1,26 @@ // 31.48.0: CLAUDE.md states who runs what when several agents share a machine, the rule the machine -// ledger exists to back. Prose: it lands where the table stands as `x new` wrote it, and is skipped -// silently where the app reworded that row or already says it. +// ledger exists to back. Prose: it lands above the One way row, in CLAUDE.md or AGENTS.md, within +// the brief's byte budget; otherwise it is reported as skipped with the row to add by hand. +import { editAgentsDocRow } from './agents-doc-row'; import type { Codemod } from './codemod'; -import type { SnippetEdit } from './engine-edits'; -import { editSnippets } from './engine-edits'; -const CLAUDE = 'CLAUDE.md'; const NEXT_ROW = '| One way | generators, not hand-rolled files:'; /** Terse on purpose: a shaped app's CLAUDE.md sits within bytes of `AGENTS_MD_MAX_BYTES`. */ export const MANY_AGENTS_ROW = '| Many agents | a worker runs its own files; the coordinator runs the gate once | `bunx x heavy status` |'; -export const DOCS_MANY_AGENTS_EDITS: readonly SnippetEdit[] = [ - { from: NEXT_ROW, to: `${MANY_AGENTS_ROW}\n${NEXT_ROW}`, optional: true }, -]; - export const codemod: Codemod = { id: 'docs-many-agents', since: '31.48.0', summary: "CLAUDE.md's table says a worker runs its own files and the coordinator runs the gate", - async run(app) { - if (((await app.read(CLAUDE)) ?? '').includes('x heavy status')) { - return { changed: [], skipped: [] }; - } - return editSnippets(app, CLAUDE, DOCS_MANY_AGENTS_EDITS); - }, + run: (app) => + editAgentsDocRow(app, { + from: NEXT_ROW, + to: `${MANY_AGENTS_ROW}\n${NEXT_ROW}`, + row: MANY_AGENTS_ROW, + said: (text) => text.includes('| Many agents |'), + missing: `no "${NEXT_ROW}" row to place it above in CLAUDE.md or AGENTS.md`, + }), }; diff --git a/packages/cli/src/codemods/docs-worker-commands.test.ts b/packages/cli/src/codemods/docs-worker-commands.test.ts new file mode 100644 index 000000000..8f1d751b6 --- /dev/null +++ b/packages/cli/src/codemods/docs-worker-commands.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, test } from 'bun:test'; +import { AGENTS_MD_MAX_BYTES } from '@ultimat3/manifest'; +import { names } from '../templates/naming'; +import { docsFiles } from '../templates/scaffold-docs'; +import { memoryCodemodApp } from './codemod'; +import { MANY_AGENTS_ROW } from './docs-many-agents'; +import { codemod, WORKER_COMMANDS_ROW } from './docs-worker-commands'; + +describe('docs-worker-commands', () => { + test('moves the row x new wrote, once', async () => { + const app = memoryCodemodApp({ 'CLAUDE.md': `| A | b | c |\n${MANY_AGENTS_ROW}\n` }); + expect((await codemod.run(app)).changed).toEqual(['CLAUDE.md']); + expect(app.files['CLAUDE.md']).toBe(`| A | b | c |\n${WORKER_COMMANDS_ROW}\n`); + expect((await codemod.run(app)).changed).toEqual([]); + }); + + test('a reworded row is left as written and reported, with the row to use', async () => { + const app = memoryCodemodApp({ 'CLAUDE.md': '| Many agents | ours | review |\n' }); + const result = await codemod.run(app); + expect(result.changed).toEqual([]); + expect(result.skipped[0]?.reason).toContain(WORKER_COMMANDS_ROW); + expect(app.files['CLAUDE.md']).toBe('| Many agents | ours | review |\n'); + }); + + test('a row that would take the file over its byte budget is skipped with the count', async () => { + const tail = `\n${MANY_AGENTS_ROW}\n`; + const full = `${'x'.repeat(AGENTS_MD_MAX_BYTES - tail.length - 5)}${tail}`; + const app = memoryCodemodApp({ 'CLAUDE.md': full }); + const result = await codemod.run(app); + expect(result.changed).toEqual([]); + expect(result.skipped[0]?.reason).toContain(String(AGENTS_MD_MAX_BYTES)); + expect(app.files['CLAUDE.md']).toBe(full); + }); + + test('says what a new app is scaffolded with', () => { + const written = docsFiles(names('shop')).find((file) => file.path === 'CLAUDE.md'); + expect(written?.contents).toContain(WORKER_COMMANDS_ROW); + expect(written?.contents).not.toContain(MANY_AGENTS_ROW); + }); +}); diff --git a/packages/cli/src/codemods/docs-worker-commands.ts b/packages/cli/src/codemods/docs-worker-commands.ts new file mode 100644 index 000000000..cf9b4c973 --- /dev/null +++ b/packages/cli/src/codemods/docs-worker-commands.ts @@ -0,0 +1,26 @@ +// 31.49.0: CLAUDE.md's many-agents row names the commands a worker runs. The row it replaces gave a +// worker none and listed `x heavy status` under "Refused by", which refuses nothing. Prose: it moves +// the row as written, within the brief's byte budget; a reworded or absent row is a reported skip. + +import { editAgentsDocRow } from './agents-doc-row'; +import type { Codemod } from './codemod'; +import { MANY_AGENTS_ROW } from './docs-many-agents'; + +/** Terse on purpose: a shaped app's CLAUDE.md sits within bytes of `AGENTS_MD_MAX_BYTES`. */ +export const WORKER_COMMANDS_ROW = + '| Many agents | a worker: `bun test ./`, `bunx x test --filter `; the coordinator: the gate, once | review |'; + +export const codemod: Codemod = { + id: 'docs-worker-commands', + since: '31.49.0', + summary: "CLAUDE.md's many-agents row names the commands a worker runs", + run: (app) => + editAgentsDocRow(app, { + from: MANY_AGENTS_ROW, + to: WORKER_COMMANDS_ROW, + row: WORKER_COMMANDS_ROW, + said: (text) => text.includes(WORKER_COMMANDS_ROW), + missing: + 'the many-agents row is not the one x new or docs-many-agents wrote (reworded, or never added), so it is left as written', + }), +}; diff --git a/packages/cli/src/codemods/index.ts b/packages/cli/src/codemods/index.ts index 6c83a45ca..49e4997da 100644 --- a/packages/cli/src/codemods/index.ts +++ b/packages/cli/src/codemods/index.ts @@ -21,10 +21,12 @@ import { codemod as docsIslandRows } from './docs-island-rows'; import { codemod as docsManyAgents } from './docs-many-agents'; import { codemod as docsShippedGuards } from './docs-shipped-guards'; import { codemod as docsTestServices } from './docs-test-services'; +import { codemod as docsWorkerCommands } from './docs-worker-commands'; import { codemod as e2eSuiteFloor } from './e2e-suite-floor'; import { codemod as formatEditedHook } from './format-edited-hook'; import { codemod as helmAlertsDeploy } from './helm-alerts-deploy'; import { codemod as helmAlertsObjective } from './helm-alerts-objective'; +import { codemod as mergeDrivers } from './merge-drivers'; import { codemod as metricsProbeHealthz } from './metrics-probe-healthz'; import { codemod as migrateJobOrdered } from './migrate-job-ordered'; import { codemod as reproGitignore } from './repro-gitignore'; @@ -73,4 +75,6 @@ export const CODEMODS: readonly Codemod[] = [ ciWeeklySchedule, e2eSuiteFloor, docsManyAgents, + docsWorkerCommands, + mergeDrivers, ]; diff --git a/packages/cli/src/codemods/merge-drivers.test.ts b/packages/cli/src/codemods/merge-drivers.test.ts new file mode 100644 index 000000000..7ded33669 --- /dev/null +++ b/packages/cli/src/codemods/merge-drivers.test.ts @@ -0,0 +1,53 @@ +// The codemod that carries an app to mergeable registries: it lands what `x new` now writes, is +// idempotent, and leaves what it cannot restate. + +import { describe, expect, test } from 'bun:test'; +import { REGISTRY_LINE_COMMENT } from '../registry-lines'; +import { MERGE_ATTRIBUTES } from '../templates/scaffold-gitattributes'; +import { memoryCodemodApp } from './codemod'; +import { codemod, INSTALL_STEP } from './merge-drivers'; + +const SETUP = "await bun('install');\nawait x('db', 'seed');\n"; +const API = `export const api = defineApi({\n actions: [health],\n});\n`; + +const app = (over: Record = {}) => + memoryCodemodApp({ + '.gitattributes': '* text=auto eol=lf\n', + 'bin/setup.ts': SETUP, + 'apps/web/api/index.ts': API, + ...over, + }); + +describe('unit · merge-drivers codemod', () => { + test('adds the rules, the setup step and the one-per-line lists', async () => { + const target = app(); + const result = await codemod.run(target); + expect(result.skipped).toEqual([]); + expect([...result.changed].sort()).toEqual([ + '.gitattributes', + 'apps/web/api/index.ts', + 'bin/setup.ts', + ]); + expect(target.files['.gitattributes']).toBe(`* text=auto eol=lf\n\n${MERGE_ATTRIBUTES}`); + expect(target.files['bin/setup.ts']).toBe( + `await bun('install');\n${INSTALL_STEP}await x('db', 'seed');\n`, + ); + expect(target.files['apps/web/api/index.ts']).toContain( + `actions: [\n ${REGISTRY_LINE_COMMENT}\n health,\n ],`, + ); + }); + + test('a second run changes nothing', async () => { + const target = app(); + await codemod.run(target); + expect(await codemod.run(target)).toEqual({ changed: [], skipped: [] }); + }); + + test('a rewritten setup script is reported, never guessed at', async () => { + const target = app({ 'bin/setup.ts': 'await myInstall();\n' }); + const result = await codemod.run(target); + expect(target.files['bin/setup.ts']).toBe('await myInstall();\n'); + expect(result.skipped.map((one) => one.path)).toEqual(['bin/setup.ts']); + expect(result.skipped[0]?.reason).toContain(INSTALL_STEP.trim()); + }); +}); diff --git a/packages/cli/src/codemods/merge-drivers.ts b/packages/cli/src/codemods/merge-drivers.ts new file mode 100644 index 000000000..19c2e6724 --- /dev/null +++ b/packages/cli/src/codemods/merge-drivers.ts @@ -0,0 +1,66 @@ +// 31.49.0: two branches that each generate a feature merge without a hand edit. An existing app +// gains the `.gitattributes` rules, the `bun run setup` step that installs the drivers, and its +// registries one entry per line. Every step is additive: a list this cannot restate is left alone, +// and it still works; it only keeps merging by hand. + +import { REGISTRY_FILES, withRegistryLines } from '../registry-lines'; +import { BIN_SCRIPTS } from '../templates/scaffold-bin'; +import { MERGE_ATTRIBUTES, ROOT_GITATTRIBUTES_PATH } from '../templates/scaffold-gitattributes'; +import type { Codemod, CodemodApp, CodemodResult } from './codemod'; + +const INSTALL_ANCHOR = "await bun('install');\n"; +export const INSTALL_STEP = "await x('merge-driver', 'install');\n"; + +const withAttributes = (held: string | undefined): string | undefined => { + if (held?.includes('merge=x-registry') === true) return undefined; + if (held === undefined) return MERGE_ATTRIBUTES; + return `${held.endsWith('\n') ? held : `${held}\n`}\n${MERGE_ATTRIBUTES}`; +}; + +async function edit( + app: CodemodApp, + path: string, + change: (held: string) => string | undefined, + changed: string[], +): Promise { + const held = await app.read(path); + if (held === undefined) return; + const next = change(held); + if (next === undefined || next === held) return; + await app.write(path, next); + changed.push(path); +} + +export const codemod: Codemod = { + id: 'merge-drivers', + since: '31.49.0', + summary: 'parallel branches merge generated features: drivers, setup step, one entry per line', + async run(app): Promise { + const changed: string[] = []; + const skipped: CodemodResult['skipped'] = []; + const attributes = withAttributes(await app.read(ROOT_GITATTRIBUTES_PATH)); + if (attributes !== undefined) { + await app.write(ROOT_GITATTRIBUTES_PATH, attributes); + changed.push(ROOT_GITATTRIBUTES_PATH); + } + const setup = await app.read(BIN_SCRIPTS.setup); + if (setup !== undefined && !setup.includes(INSTALL_STEP)) { + if (setup.includes(INSTALL_ANCHOR)) { + await app.write( + BIN_SCRIPTS.setup, + setup.replace(INSTALL_ANCHOR, `${INSTALL_ANCHOR}${INSTALL_STEP}`), + ); + changed.push(BIN_SCRIPTS.setup); + } else { + skipped.push({ + path: BIN_SCRIPTS.setup, + reason: `no "${INSTALL_ANCHOR.trim()}" line to follow; add ${INSTALL_STEP.trim()} after your install step`, + }); + } + } + for (const path of REGISTRY_FILES) { + await edit(app, path, (held) => withRegistryLines(path, held), changed); + } + return { changed, skipped }; + }, +}; diff --git a/packages/cli/src/command.ts b/packages/cli/src/command.ts index 250d2061b..59730656e 100644 --- a/packages/cli/src/command.ts +++ b/packages/cli/src/command.ts @@ -4,6 +4,7 @@ import type { Runner } from './exec'; import type { CommandResult } from './output'; +import type { ReaderOptions } from './output-findings'; import type { CommandSpec, ParsedArgs } from './parse'; export interface CommandContext { @@ -21,6 +22,8 @@ export interface CommandContext { * follow. Absent means the default below — a caller that builds a context by hand owes nothing. */ readonly invocation?: string; + /** Who reads this run's fix lines, for one printed outside a finding. `dispatch` decides. */ + readonly reader?: ReaderOptions; } /** What `ctx.invocation` means when nobody said: the binary, then the subcommand they typed. */ diff --git a/packages/cli/src/db-fork.test.ts b/packages/cli/src/db-fork.test.ts new file mode 100644 index 000000000..db1c5779c --- /dev/null +++ b/packages/cli/src/db-fork.test.ts @@ -0,0 +1,254 @@ +// The whole forked-chain path at command altitude: two branches' migrations on one disk, the +// refusal from every reader, its printed fix followed as printed, then a real replay and `drift`. + +import { afterAll, describe, expect, test } from 'bun:test'; +// why: `node:fs`/`node:os` — Bun has no temp-directory API; `node:path` — no Bun path joiner. +import { mkdtempSync, readdirSync, renameSync, rmSync } from 'node:fs'; +// why: Bun exposes no tmpdir(), so only node:os answers the platform temp root. +import { tmpdir } from 'node:os'; +// why: Bun exposes no path-join primitive; Bun.file and import() take one already joined. +import { join } from 'node:path'; +import { clearRegistry, entity, uuid } from '@ultimat3/entity'; +import { isolateEntityRegistry } from '@ultimat3/testing/registry-isolation'; +import { REQUIRED_BUN } from './app-root'; +import { dbCommand } from './cmd-db'; +import type { CommandContext } from './command'; +import { exec } from './exec'; +import { withCheckoutLock } from './heavy'; +import { HEAVY_DIR_ENV } from './heavy-env'; +import { msg } from './messages'; +import { MIGRATIONS_DIR, readMigrations } from './migrations'; +import { flagBool, parseArgs } from './parse'; +import { SPECS } from './registry'; +import { checkMigrationDrift } from './schema-drift'; +import { checkSchemaDump } from './schema-dump-drift'; + +/** Each `x db gen` here replays the chain on a scratch embedded database. A hang detector. */ +const REPLAY_MS = 180_000; + +const ctxFor = ( + argv: readonly string[], + cwd: string, + env: CommandContext['env'] = {}, +): CommandContext => ({ + args: parseArgs(argv, SPECS), + cwd, + runner: exec, + env, + bunVersion: REQUIRED_BUN, +}); + +async function appRoot(): Promise { + const dir = mkdtempSync(join(tmpdir(), 'x-db-fork-')); + await Bun.write(join(dir, 'app.config.ts'), "export const config = { name: 'fixture' };\n"); + await Bun.write(join(dir, 'apps/web/shared/app-name.ts'), "export const APP_NAME = 'fork';\n"); + await Bun.write(join(dir, 'packages/db/src/schema.ts'), 'export const schema = 1;\n'); + return dir; +} + +// A skipped suite still evaluates this module: the reset runs at file scope either way. +afterAll(() => { + clearRegistry(); +}); + +const declare = (...tables: readonly string[]): void => { + clearRegistry(); + for (const table of tables) entity(table, { columns: { id: uuid().primaryKey() } }); +}; + +const generate = async (root: string, name: string, stamp: string): Promise => { + const result = await dbCommand.run(ctxFor(['db', 'gen', name], root)); + expect(result.findings ?? []).toEqual([]); + const id = (result.data as { readonly migration: string }).migration; + const dir = join(root, MIGRATIONS_DIR); + for (const suffix of ['.sql', '.snapshot.json', '.hash']) { + renameSync(join(dir, `${id}${suffix}`), join(dir, `${stamp}${suffix}`)); + } +}; + +/** Everything under the migrations directory, as bytes: "nothing moved" is this being equal. */ +async function filesOf(root: string): Promise>> { + const dir = join(root, MIGRATIONS_DIR); + const entries = await Promise.all( + readdirSync(dir) + .sort() + .map(async (name) => [name, await Bun.file(join(dir, name)).text()] as const), + ); + return Object.fromEntries(entries); +} + +/** One parent, then each branch generated from it alone: what `git merge` leaves on disk. */ +async function mergedBranches(root: string): Promise { + declare('fork_base'); + await generate(root, 'initial', '20260101000001_initial'); + const parent = await filesOf(root); + declare('fork_base', 'fork_alphas'); + await generate(root, 'create alphas', '20260101000002_create_alphas'); + const left = await filesOf(root); + for (const name of Object.keys(left)) { + if (!(name in parent)) rmSync(join(root, MIGRATIONS_DIR, name)); + } + declare('fork_base', 'fork_betas'); + await generate(root, 'create betas', '20260101000003_create_betas'); + for (const [name, text] of Object.entries(left)) { + await Bun.write(join(root, MIGRATIONS_DIR, name), text); + } + declare('fork_alphas', 'fork_base', 'fork_betas'); +} + +describe('a forked migration chain', () => { + test( + 'is refused everywhere, its fix works as printed, and replay and drift end green', + async () => { + const restore = isolateEntityRegistry(); + const root = await appRoot(); + try { + await mergedBranches(root); + const before = await filesOf(root); + + const [finding, ...others] = await checkMigrationDrift(root); + expect(others).toEqual([]); + expect(finding?.code).toBe('X_MIGRATION_FORKED'); + expect(finding?.cause).toContain('table "fork_alphas"'); + const fix = finding?.fix ?? expect.unreachable('the refusal carries a fix'); + expect(fix).not.toContain("'record"); + expect(fix).not.toMatch(/\brm\b/); + expect( + await checkSchemaDump(root, {}, async () => + expect.unreachable('a forked chain answers once, with no replay behind it'), + ), + ).toEqual([]); + + for (const argv of [ + ['db', 'gen', 'record fork_alphas'], + ['db', 'migrate'], + ]) { + const refused = await dbCommand.run(ctxFor(argv, root)); + expect(refused.ok).toBe(false); + expect(refused.findings?.map((entry) => entry.code)).toEqual(['X_MIGRATION_FORKED']); + } + expect(await filesOf(root)).toEqual(before); + + const command = fix.split('#')[0]?.trim() ?? ''; + const argv = command.split(/\s+/).slice(1); + expect(argv).toEqual(['db', 'gen', '--merge-fork']); + const merged = await dbCommand.run(ctxFor(argv, root)); + expect(merged.findings ?? []).toEqual([]); + expect(merged.ok).toBe(true); + const id = '20260101000003_create_betas'; + expect(merged.data).toMatchObject({ outcome: 'fork-merged', migration: null, newest: id }); + expect(merged.summary).toBe(msg('cli.db.gen.forkMerged', { count: 1, id })); + // The dump is what a replay of the chain produces, so `written` is the replay succeeding. + expect((merged.data as { schemaDump: { status: string } }).schemaDump.status).toBe( + 'written', + ); + + const after = await filesOf(root); + expect(Object.keys(after)).toEqual(Object.keys(before)); + for (const name of Object.keys(before).filter((file) => file.endsWith('.sql'))) { + expect(after[name]).toBe(before[name] ?? ''); + } + expect(await checkMigrationDrift(root)).toEqual([]); + const again = await dbCommand.run(ctxFor(['db', 'gen'], root)); + expect(again.data).toMatchObject({ outcome: 'unchanged', migration: null, newest: id }); + expect(again.lines).toContain(msg('cli.db.gen.newest', { id })); + expect(await readMigrations(root)).toHaveLength(3); + } finally { + rmSync(root, { recursive: true, force: true }); + restore(); + } + }, + REPLAY_MS, + ); + + test('--merge-fork parses for gen and is refused elsewhere', () => { + expect(flagBool(parseArgs(['db', 'gen', '--merge-fork'], SPECS), 'merge-fork')).toBe(true); + expect(() => parseArgs(['db', 'migrate', '--merge-fork'], SPECS)).toThrow(); + }); +}); + +/** Reads `stream` until `wanted` appears, or to its end: a run that never waits has ended by then. */ +async function sawLine(stream: ReadableStream, wanted: string): Promise { + const decoder = new TextDecoder(); + let text = ''; + for await (const chunk of stream) { + text += decoder.decode(chunk, { stream: true }); + if (text.includes(wanted)) return true; + } + return false; +} + +/** A second process, because the lock is per descriptor and the race it closes is between CLIs. */ +const genInChild = (root: string, env: Readonly>) => + Bun.spawn( + [ + process.execPath, + '-e', + `import { entity, uuid } from '@ultimat3/entity'; + import { dbCommand } from ${JSON.stringify(join(import.meta.dir, 'cmd-db.ts'))}; + import { REQUIRED_BUN } from ${JSON.stringify(join(import.meta.dir, 'app-root.ts'))}; + import { exec } from ${JSON.stringify(join(import.meta.dir, 'exec.ts'))}; + import { parseArgs } from ${JSON.stringify(join(import.meta.dir, 'parse.ts'))}; + import { SPECS } from ${JSON.stringify(join(import.meta.dir, 'registry.ts'))}; + for (const table of ['fork_one', 'fork_two']) entity(table, { columns: { id: uuid().primaryKey() } }); + const result = await dbCommand.run({ + args: parseArgs(['db', 'gen', 'create one'], SPECS), + cwd: ${JSON.stringify(root)}, + runner: exec, + env: Bun.env, + bunVersion: REQUIRED_BUN, + }); + console.log(JSON.stringify({ ok: result.ok, summary: result.summary }));`, + ], + { cwd: import.meta.dir, env, stdout: 'pipe', stderr: 'pipe', timeout: REPLAY_MS }, + ); + +// flock(2) does not exist there, so the ledger and this lock are off (`heavy-flock.ts`). +describe.skipIf(process.platform === 'win32')('x db gen in one checkout', () => { + test( + 'a run waits for the one in progress, then sees its migration and writes none', + async () => { + const restore = isolateEntityRegistry(); + const root = await appRoot(); + const ledger = mkdtempSync(join(tmpdir(), 'x-db-fork-ledger-')); + // `CI` cleared and a ledger of its own: this lock is the machine ledger's, off under CI. + const env = { CI: '', [HEAVY_DIR_ENV]: ledger }; + const childEnv = { ...Bun.env, ...env } as Record; + try { + declare('fork_one', 'fork_two'); + const child = await withCheckoutLock( + { name: 'db-gen', root, env }, + { env: {}, fds: [] }, + async () => { + const started = genInChild(root, childEnv); + // Without the lock the child finishes here and the stream ends with no such line. + expect(await sawLine(started.stderr, 'db-gen lock')).toBe(true); + expect(await readMigrations(root)).toEqual([]); + return started; + }, + ); + expect(await child.exited).toBe(0); + const body = JSON.parse(await new Response(child.stdout).text()) as { + readonly summary: string; + }; + const migrations = await readMigrations(root); + expect(migrations).toHaveLength(1); + const id = migrations[0]?.id ?? expect.unreachable('one migration was written'); + expect(body.summary).toBe( + msg('cli.db.gen.writtenTables', { id, tables: 'fork_one, fork_two' }), + ); + + const later = await dbCommand.run(ctxFor(['db', 'gen', 'create two'], root, env)); + expect(later.ok).toBe(true); + expect(later.data).toMatchObject({ migration: null, newest: id }); + expect(later.lines).toContain(msg('cli.db.gen.newest', { id })); + expect(await readMigrations(root)).toHaveLength(1); + } finally { + rmSync(root, { recursive: true, force: true }); + rmSync(ledger, { recursive: true, force: true }); + restore(); + } + }, + REPLAY_MS, + ); +}); diff --git a/packages/cli/src/db-generate.ts b/packages/cli/src/db-generate.ts index 405b0fc2b..d31836d11 100644 --- a/packages/cli/src/db-generate.ts +++ b/packages/cli/src/db-generate.ts @@ -4,15 +4,22 @@ // why: `node:path` — Bun ships no path joiner of its own, and these paths are built, not opened. import { join } from 'node:path'; -import type { GeneratedMigration, UnrenderedDeclaration } from '@ultimat3/db'; +import type { + GeneratedMigration, + Migration, + SchemaDescription, + UnrenderedDeclaration, +} from '@ultimat3/db'; import { DESTRUCTIVE_MARKER, declaredSchema, generateMigration, + migrationForked, migrationSnapshotMissing, ROLLING_UNSAFE_MARKER, snapshotJson, } from '@ultimat3/db'; +import { migrationFork } from '@ultimat3/db/schema-dump'; import { registeredEntities } from '@ultimat3/entity'; import { describeQueries } from '@ultimat3/query'; import { loadApp } from './app-load'; @@ -35,16 +42,27 @@ export interface GenerateMigrationOptions { * concurrently. Never a default; once dropped, a plain run does not rebuild it. */ readonly dropRedundant?: boolean | undefined; + /** + * `x db gen --merge-fork`, the fix `X_MIGRATION_FORKED` prints. Rewrites the newest sidecar to what + * the merged chain builds, and goes on as any run does from there. No `.sql` is touched. + */ + readonly mergeFork?: boolean | undefined; /** `database.rowSecurity` and the tenant columns. The app's own unless a test injects one. */ readonly rowSecurity?: RowSecurityWanted | undefined; } /** - * What this run actually did — four different things, and `--json` has to tell them apart. A + * What this run actually did — five different things, and `--json` has to tell them apart. A * `hash-recorded` run wrote a file and generated no migration; reporting it as `generated` would * claim a migration nobody can apply, and reporting it as `unchanged` would hide the one write. + * `fork-merged` is the same honesty for `--merge-fork`: a sidecar moved and no migration exists. */ -export type GenerateOutcome = 'generated' | 'hash-recorded' | 'unchanged' | 'blocked'; +export type GenerateOutcome = + | 'generated' + | 'hash-recorded' + | 'fork-merged' + | 'unchanged' + | 'blocked'; export interface GeneratedFiles { readonly outcome: GenerateOutcome; @@ -52,6 +70,12 @@ export interface GeneratedFiles { readonly migration?: GeneratedMigration | undefined; /** App-root-relative paths written, in write order. Empty when there was nothing to write. */ readonly files: readonly string[]; + /** The tables `migration` creates, changes or drops, so a shared checkout can see whose they are. */ + readonly tables?: readonly string[] | undefined; + /** The newest migration on disk once this run ended: what an empty diff was measured against. */ + readonly newest?: string | undefined; + /** How many objects `--merge-fork` put right in the newest sidecar. Absent when it merged none. */ + readonly forkRepairs?: number | undefined; /** What the source hashes to, whenever this run was in a position to record it. */ readonly schemaHash?: string | undefined; /** @@ -91,6 +115,47 @@ export function migrationSql(migration: GeneratedMigration): string { ].join('\n'); } +/** Every table whose recorded shape differs between two snapshots, in name order. */ +function tablesBetween(before: SchemaDescription, after: SchemaDescription): readonly string[] { + const shape = (schema: SchemaDescription): Map => + new Map(schema.tables.map((table) => [table.name, JSON.stringify(table)])); + const was = shape(before); + const now = shape(after); + return [...new Set([...was.keys(), ...now.keys()])] + .filter((name) => was.get(name) !== now.get(name)) + .sort(); +} + +interface SettledChain { + readonly migrations: readonly Migration[]; + readonly files: readonly string[]; + readonly repairs: number; +} + +/** + * Before any diff: against a forked chain the diff is wrong in both directions, asking for a second + * `create table` the replay refuses. Merging only rewrites a sidecar, which no checksum covers; a + * collision has no such repair, so it is refused even under `--merge-fork`. + */ +async function settleFork( + root: string, + migrations: readonly Migration[], + mergeFork: boolean, +): Promise { + const fork = migrationFork(migrations); + if (fork === undefined) return { migrations, files: [], repairs: 0 }; + if (!mergeFork || fork.collisions.length > 0) throw migrationForked(fork); + const file = `${MIGRATIONS_DIR}/${snapshotFileName(fork.newest)}`; + await Bun.write(join(root, file), snapshotJson(fork.merged)); + return { + migrations: migrations.map((migration) => + migration.id === fork.newest ? { ...migration, snapshot: fork.merged } : migration, + ), + files: [file], + repairs: fork.lost.length + fork.stale.length, + }; +} + /** * Generation reads source and writes files; it never opens a database. The previous migration's * snapshot is what it diffs against (`declaredSchema`), so `x db gen` answers the same in CI, on a @@ -109,7 +174,10 @@ export async function generateAppMigration( return { outcome: 'blocked', files: [], findings: app.findings, unrendered: [] }; } - const migrations = await readMigrations(root); + const onDisk = await readMigrations(root); + const settled = await settleFork(root, onDisk, options.mergeFork === true); + const migrations = settled.migrations; + const forkFields = settled.repairs === 0 ? {} : { forkRepairs: settled.repairs }; const current = declaredSchema(migrations); // Refused, never defaulted to the empty schema: with nothing to diff against, every table the // database already holds looks new and the generated `up` is `create table` for all of them. @@ -166,10 +234,14 @@ export async function generateAppMigration( return { outcome: 'unchanged', files: [], findings: [], unrendered: migration.unrendered }; } const reconciled = await reconcileSchemaHash(root, newest.id); + const hashFiles = reconciled.written ? [`${MIGRATIONS_DIR}/${hashFileName(newest.id)}`] : []; + const quietOutcome = reconciled.written ? 'hash-recorded' : 'unchanged'; return { - outcome: reconciled.written ? 'hash-recorded' : 'unchanged', + outcome: settled.repairs > 0 ? 'fork-merged' : quietOutcome, schemaHash: reconciled.hash, - files: reconciled.written ? [`${MIGRATIONS_DIR}/${hashFileName(newest.id)}`] : [], + files: [...settled.files, ...hashFiles], + newest: newest.id, + ...forkFields, findings: [], unrendered: migration.unrendered, }; @@ -180,7 +252,7 @@ export async function generateAppMigration( // after this one. Each file is a ledger row of its own; see `@ultimat3/db`'s `generate-steps.ts`. const steps = [migration, ...(migration.followUp === undefined ? [] : [migration.followUp])]; const dir = join(root, MIGRATIONS_DIR); - const files: string[] = []; + const files: string[] = [...settled.files]; let schemaHash = ''; for (const step of steps) { const sql = `${step.id}.sql`; @@ -200,6 +272,9 @@ export async function generateAppMigration( migration, schemaHash, files, + tables: tablesBetween(current, steps[steps.length - 1]?.snapshot ?? migration.snapshot), + newest: steps[steps.length - 1]?.id ?? migration.id, + ...forkFields, findings: [], unrendered: migration.unrendered, }; diff --git a/packages/cli/src/dev-lock-holder-dir.test.ts b/packages/cli/src/dev-lock-holder-dir.test.ts new file mode 100644 index 000000000..32467a527 --- /dev/null +++ b/packages/cli/src/dev-lock-holder-dir.test.ts @@ -0,0 +1,55 @@ +// The directory a port's holder runs in: what lets a reader tell "my server" from a neighbour's. + +import { describe, expect, test } from 'bun:test'; +import { realpathSync } from 'node:fs'; // why: Bun ships no realpath. +import { holderDir, portOwner } from './dev-lock-holder-dir'; +import type { ExecResult, Runner } from './exec'; + +const answer = (stdout: string): Runner => { + return async (command): Promise => ({ + command, + code: 0, + ok: true, + stdout, + stderr: '', + durationMs: 0, + }); +}; + +describe('holderDir', () => { + test.skipIf(process.platform !== 'linux')('reads a live process from /proc', async () => { + expect(await holderDir(process.pid)).toBe(realpathSync(process.cwd())); + }); + + test('without /proc it asks lsof, whose cwd line starts with n', async () => { + const dir = await holderDir(9, answer('p9\nfcwd\nn/work/app\n'), '/no-such-proc'); + expect(dir).toBe('/work/app'); + }); + + test('a process it cannot read has no directory', async () => { + const refused: Runner = async () => expect.unreachable('lsof is not asked with /proc present'); + if (process.platform === 'linux') { + expect(await holderDir(2 ** 22 + 1, refused)).toBeUndefined(); + } + expect(await holderDir(9, answer(''), '/no-such-proc')).toBeUndefined(); + }); +}); + +describe('portOwner', () => { + test('no pid: nothing more to ask', async () => { + expect(await portOwner(3000, answer(''), async () => ({}))).toEqual({}); + }); + + test('a pid gains its directory', async () => { + const owner = await portOwner( + 3000, + answer('p9\nfcwd\nn/work/app\n'), + async () => ({ + pid: 9, + command: 'bun', + }), + '/no-such-proc', + ); + expect(owner).toEqual({ pid: 9, command: 'bun', cwd: '/work/app' }); + }); +}); diff --git a/packages/cli/src/dev-lock-holder-dir.ts b/packages/cli/src/dev-lock-holder-dir.ts new file mode 100644 index 000000000..5c60187a4 --- /dev/null +++ b/packages/cli/src/dev-lock-holder-dir.ts @@ -0,0 +1,64 @@ +// Where a port's holder runs. A pid alone cannot tell a reader whether the server is theirs; the +// directory can, and it is what lets `x doctor` separate this checkout from a neighbour. + +// why: Bun ships no readlink primitive and no existence check for a directory. +import { existsSync, readlinkSync } from 'node:fs'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { type PortHolder, portHolder } from './dev-lock-port-holder'; +import { exec, type Runner } from './exec'; + +export interface PortOwner extends PortHolder { + /** The holder's working directory, when the OS will say. */ + readonly cwd?: string; +} + +const PROC = '/proc'; + +/** `/proc` where it exists; `lsof`'s cwd record (`n`) elsewhere. `undefined` when neither says. */ +export async function holderDir( + pid: number, + runner: Runner = exec, + procRoot: string = PROC, +): Promise { + if (existsSync(join(procRoot, 'self'))) { + try { + return readlinkSync(join(procRoot, String(pid), 'cwd')); + } catch { + return undefined; // Gone, or another user's. + } + } + try { + const { stdout } = await runner(['lsof', '-a', '-p', String(pid), '-d', 'cwd', '-Fn'], { + cwd: process.cwd(), + }); + return stdout + .split('\n') + .find((line) => line.startsWith('n')) + ?.slice(1); + } catch { + return undefined; // No `lsof`: the refusal still stands, without a directory in it. + } +} + +/** Who holds `port`: the OS's answer, plus the directory of the process it names. */ +export async function portOwner( + port: number, + runner: Runner = exec, + find: (port: number, runner: Runner) => Promise = portHolder, + procRoot: string = PROC, +): Promise { + const holder = await find(port, runner); + if (holder.pid === undefined) return holder; + const cwd = await holderDir(holder.pid, runner, procRoot); + return cwd === undefined ? holder : { ...holder, cwd }; +} + +/** `pid 41 (bun) in /work/app`, as far as it is known; `undefined` with no pid to name. */ +export function ownerText(owner: PortOwner): string | undefined { + if (owner.pid === undefined) return undefined; + const command = owner.command === undefined ? '' : ` (${owner.command})`; + // Printed into one finding line: a directory name may hold anything. + const dir = owner.cwd === undefined ? '' : ` in ${owner.cwd.replace(/\p{Cc}/gu, ' ')}`; + return `pid ${String(owner.pid)}${command}${dir}`; +} diff --git a/packages/cli/src/dev-lock.test.ts b/packages/cli/src/dev-lock.test.ts index 3e69e107b..0cb8803b4 100644 --- a/packages/cli/src/dev-lock.test.ts +++ b/packages/cli/src/dev-lock.test.ts @@ -301,6 +301,24 @@ describe('preflight', () => { } }); + test('the suggested port is really free, and the holder is named with its directory', async () => { + const dir = scratch(); + try { + const thrown = (await preflight({ + stateDir: dir, + port: 3000, + hostname: 'localhost', + // A neighbour already sits on the pair the plain suggestion would name. + portBound: (port) => port < 3004, + holder: async () => ({ pid: 99, command: 'bun', cwd: '/work/other-app' }), + }).catch((error: unknown) => error)) as DevPortInUseError; + expect(thrown.fix).toContain('x dev --port 3004'); + expect(thrown.cause).toContain('pid 99 (bun) in /work/other-app'); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + test('an unidentifiable holder still gets a refusal with a remedy — root-owned ports are common', async () => { const dir = scratch(); try { diff --git a/packages/cli/src/dev-lock.ts b/packages/cli/src/dev-lock.ts index 30b38c659..99af159be 100644 --- a/packages/cli/src/dev-lock.ts +++ b/packages/cli/src/dev-lock.ts @@ -20,10 +20,11 @@ import { linkSync, mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { stringField, UltimateError } from '@ultimat3/core'; +import { ownerText, type PortOwner, portOwner } from './dev-lock-holder-dir'; import { reusedPid, startedOf } from './dev-lock-identity'; -import { type PortHolder, portHolder } from './dev-lock-port-holder'; +import type { PortHolder } from './dev-lock-port-holder'; import { stealStale } from './dev-lock-steal'; -import { portPairAfter } from './flag-number'; +import { freeDevPortPair } from './port-pair-free'; import { quoteArg } from './shell-quote'; /** Where the running dev server records itself, inside the state directory it already owns. */ @@ -191,13 +192,10 @@ export class DevPortInUseError extends UltimateError { constructor(input: { readonly port: number; readonly suggestion: number; - readonly holder?: PortHolder; + readonly holder?: PortOwner; }) { const holder = input.holder ?? {}; - const named = - holder.pid === undefined - ? 'another process' - : `pid ${holder.pid}${holder.command === undefined ? '' : ` (${holder.command})`}`; + const named = ownerText(holder) ?? 'another process'; super({ code: 'X_PORT_IN_USE', cause: `:${input.port} is already bound by ${named}, so the web role cannot listen — on a machine running several projects this is usually another one's dev server, not a stale copy of this one`, @@ -320,7 +318,7 @@ export interface PreflightInput { /** Injected by the test; `isPortBound` in production. */ readonly portBound?: (port: number, hostname: string) => boolean; readonly alive?: (pid: number) => boolean; - readonly holder?: (port: number) => Promise; + readonly holder?: (port: number) => Promise; /** Injected by the test; `node:fs` in production. */ readonly files?: ClaimFiles; } @@ -435,10 +433,11 @@ export const preflight = async (input: PreflightInput): Promise release(); throw new DevPortInUseError({ port: input.port, - // `portPairAfter`, never `port + 1`: `x dev` binds a PAIR, so the holder is most often - // another `x dev` whose sync role already sits on `port + 1`. - suggestion: portPairAfter(input.port), - holder: await (input.holder ?? portHolder)(input.port), + // A pair, never `port + 1`: `x dev` binds two ports, so the holder is most often another + // `x dev` whose sync role already sits on `port + 1`. Asked, because a third may sit on the + // next pair, and a fix naming it fails again. + suggestion: await freeDevPortPair(input.port, (port) => !bound(port, input.hostname)), + holder: await (input.holder ?? portOwner)(input.port), }); } return { clearedStale, release }; diff --git a/packages/cli/src/dispatch.test.ts b/packages/cli/src/dispatch.test.ts index d24c2aeba..83133553c 100644 --- a/packages/cli/src/dispatch.test.ts +++ b/packages/cli/src/dispatch.test.ts @@ -2,13 +2,15 @@ // because that is the one an agent hits by accident — a typo'd flag, a typo'd command — and it is // the one branch that has no `ParsedArgs` to read `--json` off. -import { afterAll, describe, expect, test } from 'bun:test'; +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; import { mkdtempSync, rmSync } from 'node:fs'; // why: Bun has no mkdtemp. // why: Bun exposes no tmpdir(), so only node:os answers the platform temp root. import { tmpdir } from 'node:os'; import { currentLogStream, setLogStream } from '@ultimat3/core'; import { REQUIRED_BUN } from './app-root'; import { dispatch, sinkFor } from './dispatch'; +import { loadErrorCatalog } from './error-catalog'; +import { loadCodeFixes } from './error-fixes'; import type { CommandResult } from './output'; import { SPECS } from './registry'; @@ -221,3 +223,25 @@ describe('unit · the dispatcher owns fd 1', () => { } }); }); + +// `x errors explain` prints its fix outside any finding, so the reader has to reach the command. +describe('unit · a command learns who reads its fix lines', () => { + // What `x errors explain` loads first, loaded once here: both are cached for the process. + beforeAll(() => Promise.all([loadErrorCatalog(), loadCodeFixes()]), 60_000); + + test('where the app has its own x, explain names bunx x', async () => { + const root = made(mkdtempSync(`${tmpdir()}/x-reader-`)); + await Bun.write(`${root}/node_modules/.bin/x`, ''); + const { out } = await run(['errors', 'explain', 'X_MANIFEST_STALE'], root); + expect(out).toContain('bunx x manifest'); + }); + + test('with no local x the line stays as thrown', async () => { + const { out } = await run( + ['errors', 'explain', 'X_MANIFEST_STALE'], + made(mkdtempSync(`${tmpdir()}/x-reader-`)), + ); + expect(out).not.toContain('bunx x manifest'); + expect(out).toContain('x manifest'); + }); +}); diff --git a/packages/cli/src/dispatch.ts b/packages/cli/src/dispatch.ts index 1962449b0..d5e19919a 100644 --- a/packages/cli/src/dispatch.ts +++ b/packages/cli/src/dispatch.ts @@ -155,6 +155,7 @@ export async function dispatch(options: DispatchOptions): Promise { env, bunVersion: options.bunVersion, ...(options.invocation === undefined ? {} : { invocation: options.invocation }), + reader, }; // Put back in the `finally`: the stream is process-wide, so a dispatch that left `stderr` behind diff --git a/packages/cli/src/doctor-guarded.test.ts b/packages/cli/src/doctor-guarded.test.ts new file mode 100644 index 000000000..1fc593de9 --- /dev/null +++ b/packages/cli/src/doctor-guarded.test.ts @@ -0,0 +1,52 @@ +// A doctor check that throws is reported and the rest still run: the loop this replaces had the +// doctor's own crash telling the reader to run the doctor. + +import { describe, expect, test } from 'bun:test'; +import { reportDoctor, runDoctor } from './cmd-doctor'; +import { codes, probe } from './cmd-doctor-fixture'; + +const CATALOG_ERROR = new SyntaxError("JSON Parse error: Unrecognized token '<'"); +const throws = async (): Promise => { + throw CATALOG_ERROR; +}; + +describe('unit · runDoctor survives a check that throws', () => { + test('the failure is a finding that never prescribes x doctor', async () => { + const findings = await runDoctor(probe({ offlineFallback: throws })); + const failed = findings.find((one) => one.code === 'X_DOCTOR_PROBE_FAILED'); + expect(failed?.cause).toContain('offline fallback'); + expect(failed?.cause).toContain('Unrecognized token'); + expect(failed?.fix).toBe('bunx x manifest --json'); + }); + + test('the checks after it still report', async () => { + const drift = { code: 'X_DB_DRIFT', cause: 'c', fix: 'f' }; + expect(await codes(probe({ offlineFallback: throws, drift: async () => [drift] }))).toEqual([ + 'X_DOCTOR_PROBE_FAILED', + 'X_DB_DRIFT', + ]); + }); + + test('what a merge left is reported first', async () => { + const merge = { code: 'X_MERGE_UNRESOLVED', cause: 'c', fix: 'f' }; + const drift = { code: 'X_DB_DRIFT', cause: 'c', fix: 'f' }; + expect( + await codes(probe({ mergeState: async () => [merge], drift: async () => [drift] })), + ).toEqual(['X_MERGE_UNRESOLVED', 'X_DB_DRIFT']); + }); +}); + +describe('unit · reportDoctor', () => { + test('a listing that throws is a warning and the verdict stands', async () => { + const result = await reportDoctor(probe({ browser: throws })); + expect(result.ok).toBe(true); + expect(JSON.stringify(result.data)).toContain('X_DOCTOR_PROBE_FAILED'); + }); + + test('a clone without the merge drivers is warned, not failed', async () => { + const result = await reportDoctor(probe({ mergeDriverUnset: async () => true })); + expect(result.ok).toBe(true); + expect(JSON.stringify(result.data)).toContain('X_MERGE_DRIVER_UNSET'); + expect(JSON.stringify(result.data)).toContain('bunx x merge-driver install'); + }); +}); diff --git a/packages/cli/src/doctor-guarded.ts b/packages/cli/src/doctor-guarded.ts new file mode 100644 index 000000000..528ff1436 --- /dev/null +++ b/packages/cli/src/doctor-guarded.ts @@ -0,0 +1,41 @@ +// One `x doctor` check that throws costs that check, never the report. The whole command used to +// die as `X_CLI_UNEXPECTED`, whose fix is `x doctor --json`: the diagnostic told you to run itself. + +import { ERROR_DOCS_URL, renderThrowable } from '@ultimat3/core'; +import type { Finding } from './output'; + +export const probeFailedFinding = (check: string, error: unknown): Finding => ({ + code: 'X_DOCTOR_PROBE_FAILED', + cause: `the ${check} check could not run: ${renderThrowable(error)}`, + // `x manifest` loads the app the same way and names the module that fails. + fix: 'bunx x manifest --json', + docs: ERROR_DOCS_URL, +}); + +/** Runs `check`, recording a throw on `findings` as that check's own finding. */ +export async function guardedCheck( + name: string, + findings: Finding[], + check: () => Promise, +): Promise { + try { + await check(); + } catch (error) { + findings.push(probeFailedFinding(name, error)); + } +} + +/** A listing beside the verdict: its value, or `otherwise` with the failure on `warnings`. */ +export async function guardedListing( + name: string, + warnings: Finding[], + read: () => Promise, + otherwise: T, +): Promise { + try { + return await read(); + } catch (error) { + warnings.push(probeFailedFinding(name, error)); + return otherwise; + } +} diff --git a/packages/cli/src/doctor-port.test.ts b/packages/cli/src/doctor-port.test.ts new file mode 100644 index 000000000..f33a49f13 --- /dev/null +++ b/packages/cli/src/doctor-port.test.ts @@ -0,0 +1,82 @@ +// A taken port by who has it: this checkout's own problem is a finding; another checkout's server +// is a warning, because nothing is wrong with this environment. + +import { describe, expect, test } from 'bun:test'; +// why: Bun ships no mkdtemp or mkdir. +import { mkdirSync, mkdtempSync, rmSync } from 'node:fs'; +// why: the test's scratch root. +import { tmpdir } from 'node:os'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import type { PortOwner } from './dev-lock-holder-dir'; +import { portFindings, portWarnings } from './doctor-port'; + +const taken = + (...ports: readonly number[]) => + async (port: number): Promise => + !ports.includes(port); + +function withApp(run: (app: string, other: string) => Promise): Promise { + const base = mkdtempSync(join(tmpdir(), 'doctor-port-')); + const app = join(base, 'app'); + const other = join(base, 'other'); + mkdirSync(join(app, 'apps'), { recursive: true }); + mkdirSync(other); + return run(app, other).finally(() => rmSync(base, { recursive: true, force: true })); +} + +const owned = (owner: PortOwner) => async (): Promise => owner; + +describe('a taken port', () => { + test("another checkout's server is a warning naming its directory, never a finding", () => + withApp(async (root, other) => { + const probe = { + root, + port: 3000, + portFree: taken(3000, 3001, 3002, 3003), + portOwner: owned({ pid: 41, command: 'bun', cwd: other }), + }; + expect(await portFindings(probe)).toEqual([]); + const warnings = await portWarnings(probe); + expect(warnings.map((one) => one.code)).toEqual(['X_PORT_IN_USE', 'X_PORT_IN_USE']); + expect(warnings[0]?.cause).toContain(other); + expect(warnings[0]?.cause).toContain('pid 41'); + // 3002 and 3003 are taken too: the fix names a pair that is really free. + expect(warnings[0]?.fix).toBe('x dev --port 3004'); + })); + + test('a server running in this checkout, or below it, stays a finding', () => + withApp(async (root) => { + for (const cwd of [root, join(root, 'apps')]) { + const probe = { + root, + port: 3000, + portFree: taken(3000), + portOwner: owned({ pid: 41, cwd }), + }; + expect((await portFindings(probe)).map((one) => one.code)).toEqual(['X_PORT_IN_USE']); + expect(await portWarnings(probe)).toEqual([]); + } + })); + + test('an owner the OS will not name stays a finding: unknown is never "not yours"', () => + withApp(async (root) => { + const probe = { root, port: 3000, portFree: taken(3000), portOwner: owned({}) }; + expect(await portFindings(probe)).toHaveLength(1); + expect(await portWarnings(probe)).toEqual([]); + })); + + test('a sibling sharing the path prefix is another checkout', () => + withApp(async (root) => { + const sibling = `${root}-2`; + mkdirSync(sibling); + const probe = { root, port: 3000, portFree: taken(3000), portOwner: owned({ cwd: sibling }) }; + expect(await portFindings(probe)).toEqual([]); + expect(await portWarnings(probe)).toHaveLength(1); + })); + + test('an injected portFree never asks the real machine who owns the port', async () => { + const findings = await portFindings({ root: '/app', port: 3000, portFree: taken(3000) }); + expect(findings.map((one) => one.code)).toEqual(['X_PORT_IN_USE']); + }); +}); diff --git a/packages/cli/src/doctor-port.ts b/packages/cli/src/doctor-port.ts index 58b67b603..8d6c97962 100644 --- a/packages/cli/src/doctor-port.ts +++ b/packages/cli/src/doctor-port.ts @@ -1,12 +1,20 @@ // `x doctor`'s port checks: both ports `x dev` binds, probed by role, and the one reading of which // port that is (`--port`, then `PORT`, then 3000). +// why: Bun ships no realpath primitive. +import { realpathSync } from 'node:fs'; +// why: Bun ships no path-separator primitive. +import { sep } from 'node:path'; import { ERROR_DOCS_URL } from '@ultimat3/core'; +import type { PortOwner } from './dev-lock-holder-dir'; +import { ownerText, portOwner } from './dev-lock-holder-dir'; import type { DoctorProbe } from './doctor-probe'; -import { PORT_RANGE, portPairAfter, readIntFlag } from './flag-number'; +import { PORT_RANGE, readIntFlag } from './flag-number'; import type { Finding } from './output'; import { findingFrom } from './output'; import type { ParsedArgs } from './parse'; +import { freeDevPortPair } from './port-pair-free'; +import { portFree } from './port-probe'; import { syncPortFor } from './role-sync'; import { portFromEnv } from './serve-env'; import { quoteArg } from './shell-quote'; @@ -18,6 +26,35 @@ const finding = (code: string, cause: string, fix: string): Finding => ({ docs: ERROR_DOCS_URL, }); +export type PortProbe = Pick & { + /** Who holds a port. Absent: the OS is asked, unless `portFree` is a test's stand-in. */ + readonly portOwner?: (port: number) => Promise; +}; + +const real = (path: string): string => { + try { + return realpathSync(path); + } catch { + return path; + } +}; + +/** Only a directory the OS named and that lies outside this app: unknown is never "not yours". */ +function inAnotherCheckout(owner: PortOwner, root: string | undefined): boolean { + if (owner.cwd === undefined || root === undefined) return false; + const [dir, app] = [real(owner.cwd), real(root)]; + return dir !== app && !dir.startsWith(`${app}${sep}`); +} + +/** A stand-in `portFree` describes an imagined machine; the real one has no say about it. */ +const ownerLookup = (probe: PortProbe): ((port: number) => Promise) => + probe.portOwner ?? (probe.portFree === portFree ? (port) => portOwner(port) : async () => ({})); + +interface PortReport { + readonly findings: readonly Finding[]; + readonly warnings: readonly Finding[]; +} + /** * Both ports `x dev` binds, each labelled with the role that wants it. `x dev --port 3999` printed * `web listening on 3999`, then died on 4000 as `X_CLI_UNEXPECTED` with a caught `Error` rendered @@ -29,8 +66,11 @@ const finding = (code: string, cause: string, fix: string): Finding => ({ * The suggested port moves BOTH: `x dev --port N` occupies N and N+1, so a free N beside a taken * N+1 is still not a runnable command. It did not move both until 2026-09 — the line was * `neighbouringPort(probe.port)`, which for the sync finding IS the port the finding is about, and - * a docblock claiming otherwise is how it survived. `portPairAfter` is the one reader of that rule - * and `role-sync.ts`'s own refusal shares it. + * a docblock claiming otherwise is how it survived. `freeDevPortPair` also asks that the pair is free: + * beside two dev servers the next pair was often the second one's. + * + * A port held from ANOTHER checkout is a warning: a neighbour's dev server says nothing about this + * environment, and it turned `x doctor` red in every other app on the machine. * * The sync port is `syncPortFor`, never `neighbouringPort` again: that helper answers 65534 for a * web port of 65535 — BELOW the web port, and a port `x dev` never binds — where the boot refuses @@ -39,35 +79,50 @@ const finding = (code: string, cause: string, fix: string): Finding => ({ * reported instead of a probe. Alone, and ahead of the web port: there is no runnable `x dev` at * this port whatever the other one answers, and a second finding about it is noise over the cause. */ -export async function portFindings( - probe: Pick, -): Promise { +async function portReport(probe: PortProbe): Promise { let syncPort: number; try { syncPort = syncPortFor(probe.port); } catch (error) { // The boot's own error, carried whole — `findingFrom` reads its code, cause and fix off the // value rather than rendering it, which is what `scripts/catch-render.ts` requires. - return [findingFrom(error)]; + return { findings: [findingFrom(error)], warnings: [] }; } const wanted = [ { port: probe.port, role: 'web' }, { port: syncPort, role: 'sync' }, ] as const; + const taken = []; + for (const entry of wanted) if (!(await probe.portFree(entry.port))) taken.push(entry); + if (taken.length === 0) return { findings: [], warnings: [] }; + const fix = `x dev --port ${quoteArg(String(await freeDevPortPair(probe.port, probe.portFree)))}`; + const lookup = ownerLookup(probe); const findings: Finding[] = []; - for (const entry of wanted) { - if (await probe.portFree(entry.port)) continue; - findings.push( - finding( - 'X_PORT_IN_USE', - `port ${entry.port} is already listening, and \`x dev --port ${probe.port}\` binds it for the ${entry.role} role`, - `x dev --port ${quoteArg(String(portPairAfter(probe.port)))}`, - ), - ); + const warnings: Finding[] = []; + for (const entry of taken) { + const owner = await lookup(entry.port); + const named = ownerText(owner); + const need = `\`x dev --port ${probe.port}\` binds it for the ${entry.role} role`; + if (inAnotherCheckout(owner, probe.root)) { + const cause = `port ${entry.port} is held by ${named ?? 'another process'}, another checkout, and ${need} — nothing is wrong with this environment`; + warnings.push(finding('X_PORT_IN_USE', cause, fix)); + } else { + const held = named === undefined ? '' : ` (${named})`; + const cause = `port ${entry.port} is already listening${held}, and ${need}`; + findings.push(finding('X_PORT_IN_USE', cause, fix)); + } } - return findings; + return { findings, warnings }; } +/** The taken ports that are this environment's problem: `x doctor`'s verdict. */ +export const portFindings = async (probe: PortProbe): Promise => + (await portReport(probe)).findings; + +/** The taken ports another checkout holds: reported beside the verdict. */ +export const portWarnings = async (probe: PortProbe): Promise => + (await portReport(probe)).warnings; + /** * The port to TEST, and the one place `x doctor` reads it: `devPortFor`'s order — `--port`, then * `PORT` (`portFromEnv`, the one reader of it), then 3000 — so the probe answers about the port diff --git a/packages/cli/src/doctor-probe.ts b/packages/cli/src/doctor-probe.ts index 1778d929f..459ae379d 100644 --- a/packages/cli/src/doctor-probe.ts +++ b/packages/cli/src/doctor-probe.ts @@ -9,6 +9,8 @@ import { loadApp } from './app-load'; import { IMPORT_UNRESOLVED } from './app-load-unresolved'; import { findAppRoot } from './app-root'; import { checkMigrationSnapshots } from './db-snapshot'; +import type { PortOwner } from './dev-lock-holder-dir'; +import { portOwner } from './dev-lock-holder-dir'; import { type AuthStorageFact, authStorageProbe } from './doctor-auth'; import type { EmbeddedDatabase } from './doctor-database'; import { externalUrl, installedAbove, probeDatabase } from './doctor-database'; @@ -26,7 +28,10 @@ import { rowSecurityOffProbe, rowSecurityProbe } from './doctor-row-security'; import type { SealedKeysFact } from './doctor-sealed'; import { sealedKeysProbe } from './doctor-sealed'; import { typescriptProbe } from './doctor-typescript'; +import { exec } from './exec'; import { machineSecretFindings } from './framework-env'; +import { driverUnset } from './merge-driver-config'; +import { catalogFileFindings, unmergedFindings } from './merge-state'; import type { Finding } from './output'; import { portFree } from './port-probe'; import { checkMigrationDrift } from './schema-drift'; @@ -55,6 +60,12 @@ export interface DoctorProbe { readonly frameworkFindings: readonly Finding[]; exists(relativePath: string): boolean; portFree(port: number): Promise; + /** + * Who holds a taken port: another checkout's server is a warning, this one's a finding. This and + * the two merge facts below are optional against the rule above: the type is exported, and the + * public-API lock reads a required member added on `main` as a break. Absent asks the OS. + */ + readonly portOwner?: (port: number) => Promise; /** * Is the configured database reachable, and what refused? `null` is "reachable, or there is * nothing external to reach" — an unset `DATABASE_URL` is embedded PGlite, which `x dev` owns @@ -113,6 +124,10 @@ export interface DoctorProbe { pins(): Promise; /** The gate steps that apply here and `x.verify.json` does not floor (`doctor-floor.ts`). */ unflooredSteps(): Promise; + /** What a merge left: unmerged paths, catalogs that do not parse (`merge-state.ts`). Absent: none. */ + readonly mergeState?: () => Promise; + /** Whether `.gitattributes` names a merge driver this clone has not installed. Absent: no. */ + readonly mergeDriverUnset?: () => Promise; } export function probeFor( @@ -131,6 +146,7 @@ export function probeFor( frameworkFindings: machineSecretFindings(env), exists: (relativePath) => (root === undefined ? false : existsSync(join(root, relativePath))), portFree, + portOwner, database: () => probeDatabase(process.env['DATABASE_URL'], env), // `root ?? cwd` because the walk needs a directory that exists; outside an app `runDoctor` // returns on `X_NOT_IN_APP` before this is ever asked, so the value only has to be honest. @@ -160,6 +176,11 @@ export function probeFor( envNames: async () => (root === undefined ? NO_APP_ENV_NAMES : appEnvNamesProbe(root, env)), pins: async () => (root === undefined ? { pins: [], installed: new Map() } : pinProbe(root)), unflooredSteps: async () => (root === undefined ? [] : unflooredStepsProbe(root)), + mergeState: async () => + root === undefined + ? [] + : [...(await unmergedFindings(root)), ...(await catalogFileFindings(root))], + mergeDriverUnset: async () => (root === undefined ? false : driverUnset(root, exec)), }; } diff --git a/packages/cli/src/doctor-warnings.test.ts b/packages/cli/src/doctor-warnings.test.ts index 602b00ca1..0606f1a92 100644 --- a/packages/cli/src/doctor-warnings.test.ts +++ b/packages/cli/src/doctor-warnings.test.ts @@ -83,6 +83,21 @@ describe('unit · x doctor · warnings beside the verdict', () => { expect(result.data).not.toHaveProperty('warnings'); }); + // A neighbour's dev server is a warning (`doctor-port.ts`); it was computed and never reported, + // so another checkout's port vanished from `x doctor` instead of moving beside the verdict. + test("another checkout's port is a warning beside a green verdict", async () => { + const root = appRoot(); + const result = await reportDoctor( + probe({ + root, + portFree: async (port) => port !== 3000, + portOwner: async () => ({ pid: 41, cwd: `${root}-other` }), + }), + ); + expect(result.ok).toBe(true); + expect(result.data).toMatchObject({ warnings: [{ code: 'X_PORT_IN_USE' }] }); + }); + test('a real finding still reds the doctor, with the warnings beside it', async () => { const result = await reportDoctor({ ...ranged(), exists: () => false }); expect(result.ok).toBe(false); diff --git a/packages/cli/src/doctor-warnings.ts b/packages/cli/src/doctor-warnings.ts index 24a68056e..d1bce7536 100644 --- a/packages/cli/src/doctor-warnings.ts +++ b/packages/cli/src/doctor-warnings.ts @@ -2,14 +2,25 @@ // never in it. On `main` an app that upgrades with no change keeps a green doctor // (`X_BREAKING_ON_MAIN`); on `next` these move into `runDoctor`'s findings and the verdict. +import { ERROR_DOCS_URL } from '@ultimat3/core'; import { reservedEnvFindings } from './doctor-env-reserved'; import { unflooredStepFindings } from './doctor-floor'; import { pinRangeFindings } from './doctor-pins'; +import { portWarnings } from './doctor-port'; import type { DoctorProbe } from './doctor-probe'; import { rowSecurityOffFindings } from './doctor-row-security'; +import { REGISTRY_DRIVER } from './merge-driver-config'; import { msg } from './messages'; import type { CommandResult, Finding, JsonValue } from './output'; +const MERGE_DRIVER_UNSET: Finding = { + code: 'X_MERGE_DRIVER_UNSET', + cause: `.gitattributes names merge=${REGISTRY_DRIVER}, and this clone has no such driver, so two branches' generated features conflict`, + fix: 'bunx x merge-driver install', + docs: ERROR_DOCS_URL, + at: '.gitattributes', +}; + /** The probes that only warn on `main`. Asked inside an app only; outside one there is no app. */ export async function doctorWarnings(probe: DoctorProbe): Promise { if (probe.root === undefined) return []; @@ -22,6 +33,10 @@ export async function doctorWarnings(probe: DoctorProbe): Promise` — unrunnable, and a cause that was simply untrue. + 'X_DEV_LOCK_UNREADABLE', + 'X_PORT_IN_USE', + 'X_UPGRADE_LOCKED', + 'X_HEAVY_WAIT_EXPIRED', + 'X_MERGE_UNRESOLVED', + 'X_MERGE_DRIVER_UNSET', + 'X_CATALOG_UNPARSEABLE', + 'X_DOCTOR_PROBE_FAILED', +] as const; + +export type CliAgentsErrorCode = (typeof CLI_AGENTS_ERROR_CODES)[number]; + +/** Their titles, typed over the list above so a code without one is a build error. */ +export const CLI_AGENTS_ERROR_TITLES: Readonly> = { + X_DEV_ALREADY_RUNNING: 'another x dev already owns this checkout', + X_DEV_LOCK_UNREADABLE: 'the dev lock file cannot be read or removed', + X_PORT_IN_USE: 'the dev port is taken', + X_UPGRADE_LOCKED: 'another x upgrade holds this app', + X_HEAVY_WAIT_EXPIRED: 'x heavy run stopped waiting for the machine ledger', + X_MERGE_UNRESOLVED: 'a merge left conflicts in the app', + X_MERGE_DRIVER_UNSET: 'the merge drivers .gitattributes names are not configured', + X_CATALOG_UNPARSEABLE: 'an i18n catalog is not valid JSON', + X_DOCTOR_PROBE_FAILED: 'one x doctor check could not run', +}; diff --git a/packages/cli/src/error-codes.ts b/packages/cli/src/error-codes.ts index 503216cdd..95ca3b486 100644 --- a/packages/cli/src/error-codes.ts +++ b/packages/cli/src/error-codes.ts @@ -3,6 +3,8 @@ // the CLI is the surface an agent reads first — a failure here has to be actionable without a doc // lookup or a second round-trip. The classes that throw these codes live in `./errors`. import { registerErrorCodes } from '@ultimat3/core'; +import type { CliAgentsErrorCode } from './error-codes-agents'; +import { CLI_AGENTS_ERROR_CODES, CLI_AGENTS_ERROR_TITLES } from './error-codes-agents'; import type { CliBoundaryErrorCode } from './error-codes-boundary'; import { CLI_BOUNDARY_ERROR_CODES, CLI_BOUNDARY_ERROR_TITLES } from './error-codes-boundary'; import type { CliDoctorErrorCode } from './error-codes-doctor'; @@ -82,11 +84,6 @@ export const CLI_OWNED_ERROR_CODES = [ // its health check with nothing in the log that names the cause. 'X_ROLE_UNKNOWN', 'X_PORT_INVALID', - 'X_DEV_ALREADY_RUNNING', - // The OTHER thing the preflight can find, and it was reported as the one above: an unreadable - // lock this process could not remove made `DevAlreadyRunningError` name THIS pid as the holder, - // so the remedy printed was `kill ` — unrunnable, and a cause that was simply untrue. - 'X_DEV_LOCK_UNREADABLE', // The boot's own consistency check. `startServices` captures the drivers it built, and // `loadApp` runs AFTER it — so an app module calling `installJobDriver(theirs)` moved the ambient // slot and left the captured object alone: every `handle.enqueue()` went to their queue while @@ -94,7 +91,6 @@ export const CLI_OWNED_ERROR_CODES = [ // the enqueue side. Nothing failed. Refused here rather than documented, per axiom 3. 'X_RUNTIME_DRIVER_SPLIT', 'X_GENERATE_CONFLICT', - 'X_PORT_IN_USE', 'X_DB_GEN_FAILED', // The two directions of the drift a hash cannot see, and they are two repairs. The `drift` step // compared a schema-source HASH to a `.hash` sidecar and never read what the migration recorded, @@ -239,7 +235,6 @@ export const CLI_OWNED_ERROR_CODES = [ 'X_ROWS_SLICE_ENTITY', 'X_GUARD_BUILT_IN', 'X_REPRO_BUNDLE_INVALID', - 'X_UPGRADE_LOCKED', 'X_APP_MIGRATIONS_UNAPPLIED', ] as const; @@ -276,6 +271,7 @@ export const CLI_BORROWED_ERROR_CODES = [ /** Every code the CLI can throw: the ones it owns plus the one it borrows. */ export const CLI_ERROR_CODES = [ ...CLI_OWNED_ERROR_CODES, + ...CLI_AGENTS_ERROR_CODES, ...CLI_BOUNDARY_ERROR_CODES, ...CLI_DOCTOR_ERROR_CODES, ...CLI_PACKAGE_ERROR_CODES, @@ -289,6 +285,7 @@ export const CLI_ERROR_CODES = [ type CliListedErrorCode = (typeof CLI_OWNED_ERROR_CODES)[number]; export type CliOwnedErrorCode = | CliListedErrorCode + | CliAgentsErrorCode | CliBoundaryErrorCode | CliDoctorErrorCode | CliPackageErrorCode @@ -340,11 +337,8 @@ export const CLI_ERROR_TITLES: Readonly> = { X_DEPLOY_FAILED: 'a deploy step failed', X_ROLE_UNKNOWN: 'ROLE names something that is not a role', X_PORT_INVALID: 'PORT is not a TCP port number', - X_DEV_ALREADY_RUNNING: 'another x dev already owns this checkout', - X_DEV_LOCK_UNREADABLE: 'the dev lock file cannot be read or removed', X_RUNTIME_DRIVER_SPLIT: 'the ambient driver is not the one this process serves', X_GENERATE_CONFLICT: 'a generator would overwrite a file', - X_PORT_IN_USE: 'the dev port is taken', X_DB_GEN_FAILED: 'x db gen failed', X_DB_SCHEMA_UNMIGRATED: 'an entity declaration no migration recorded', X_DB_SCHEMA_UNDECLARED: 'a migration records schema no entity declares', @@ -468,7 +462,6 @@ export const CLI_ERROR_TITLES: Readonly> = { X_ROWS_SLICE_ENTITY: "the slice's entity is not the scaffolded table", X_GUARD_BUILT_IN: 'x g guard named a shipped guard', X_REPRO_BUNDLE_INVALID: 'the file is not a repro bundle this version can open', - X_UPGRADE_LOCKED: 'another x upgrade holds this app', X_APP_MIGRATIONS_UNAPPLIED: 'a serving role booted over a database missing migrations this build ships', }; @@ -479,6 +472,7 @@ registerErrorCodes( Object.fromEntries( Object.entries({ ...CLI_ERROR_TITLES, + ...CLI_AGENTS_ERROR_TITLES, ...CLI_BOUNDARY_ERROR_TITLES, ...CLI_DOCTOR_ERROR_TITLES, ...CLI_PACKAGE_ERROR_TITLES, diff --git a/packages/cli/src/errors-suite.ts b/packages/cli/src/errors-suite.ts new file mode 100644 index 000000000..2ce1b5ff0 --- /dev/null +++ b/packages/cli/src/errors-suite.ts @@ -0,0 +1,27 @@ +// `x test` over a selection that matched nothing. Its fix never prints the bare command: pasted as +// printed, that starts every suite, which a worker on a file set must not (`wiki/Many-Agents.md`). + +import { UltimateError } from '@ultimat3/core'; + +/** + * `x test` discovered nothing. A green run over zero files is the most expensive false pass, so + * the selection that found nothing is named in full — a caller that cannot see whether the type + * or the filter emptied the set has to guess which one to drop. + */ +export class NoTestFilesError extends UltimateError { + constructor(input: { root: string; type?: string; filter?: string }) { + const parts = [ + input.type === undefined ? undefined : `of type ${input.type}`, + input.filter === undefined ? undefined : `matching "${input.filter}"`, + ].filter((part): part is string => part !== undefined); + const where = parts.length === 0 ? '' : ` ${parts.join(' ')}`; + super({ + code: 'X_TEST_NO_FILES', + cause: `no *.test.ts files${where} under ${input.root}`, + fix: + parts.length === 0 + ? 'x test --json # run it from the repo root' + : 'x test --affected --dirty # the tests your edits reach; or a path that exists: x test --filter ', + }); + } +} diff --git a/packages/cli/src/errors.test.ts b/packages/cli/src/errors.test.ts index a1d149511..441eb7de2 100644 --- a/packages/cli/src/errors.test.ts +++ b/packages/cli/src/errors.test.ts @@ -14,8 +14,27 @@ import { UnknownCommandError, VerifyFailedError, } from './errors'; +import { NoTestFilesError } from './errors-suite'; import { thrownBy } from './thrown-by-fixture'; +describe('unit · a fix never starts a whole suite by accident', () => { + test.each(['test', 'verify'])('a typo nearest x %s gets its help, not a run', (name) => { + const { fix } = new UnknownCommandError({ path: 'tset', known: [name], suggestion: name }); + expect(fix).toStartWith(`x ${name} --help`); + }); + + test('any other nearest command is still the command', () => { + expect(new UnknownCommandError({ path: 'rotes', known: [], suggestion: 'routes' }).fix).toBe( + 'x routes', + ); + }); + + test('a selection that matched nothing points at the tests the edits reach', () => { + const { fix } = new NoTestFilesError({ root: '/app', filter: 'nosuchfile' }); + expect(fix).toStartWith('x test --affected --dirty'); + }); +}); + describe('VerifyFailedError', () => { test('counts the failed steps and names every one of them', () => { const error = new VerifyFailedError({ failed: ['typecheck', 'drift', 'unit'] }); diff --git a/packages/cli/src/errors.ts b/packages/cli/src/errors.ts index 8711e9048..c37c62786 100644 --- a/packages/cli/src/errors.ts +++ b/packages/cli/src/errors.ts @@ -5,6 +5,15 @@ import { UltimateError } from '@ultimat3/core'; import type { Finding } from './output'; import { quoteArg } from './shell-quote'; +/** Commands whose bare form runs everything. */ +const WHOLE_RUN: ReadonlySet = new Set(['test', 'verify']); + +/** The nearest real command as a fix: its help when running it bare would start a whole suite. */ +const suggestedCommandFix = (suggestion: string): string => + WHOLE_RUN.has(suggestion) + ? `x ${suggestion} --help # the nearest command; bare, it runs every suite` + : `x ${quoteArg(suggestion.split(' '))}`; + /** An unknown command or subcommand. Carries a suggestion so the retry is one keystroke away. */ export class UnknownCommandError extends UltimateError { /** `why`: the decision behind a name that is deliberately not a command, said after the list. */ @@ -17,7 +26,7 @@ export class UnknownCommandError extends UltimateError { super({ code: 'X_CLI_UNKNOWN_COMMAND', cause: `"x ${input.path}" is not a command (known: ${input.known.join(', ')})${input.why === undefined ? '' : ` — ${input.why}`}`, - fix: input.suggestion === undefined ? 'x help' : `x ${quoteArg(input.suggestion.split(' '))}`, + fix: input.suggestion === undefined ? 'x help' : suggestedCommandFix(input.suggestion), }); } } @@ -178,26 +187,6 @@ export class BunVersionError extends UltimateError { } } -/** - * `x test` discovered nothing. A green run over zero files is the most expensive false pass, so - * the selection that found nothing is named in full — a caller that cannot see whether the type - * or the filter emptied the set has to guess which one to drop. - */ -export class NoTestFilesError extends UltimateError { - constructor(input: { root: string; type?: string; filter?: string }) { - const parts = [ - input.type === undefined ? undefined : `of type ${input.type}`, - input.filter === undefined ? undefined : `matching "${input.filter}"`, - ].filter((part): part is string => part !== undefined); - const where = parts.length === 0 ? '' : ` ${parts.join(' ')}`; - super({ - code: 'X_TEST_NO_FILES', - cause: `no *.test.ts files${where} under ${input.root}`, - fix: parts.length === 0 ? 'x test --json # run it from the repo root' : 'x test', - }); - } -} - /** * A generated path resolves outside the directory it is being written into — the scaffold gate's * sandbox, the app root `x g` writes into, or the catalog root a `--locales` segment names. `..` diff --git a/packages/cli/src/generate-grants.ts b/packages/cli/src/generate-grants.ts index 18cb1a960..e07875343 100644 --- a/packages/cli/src/generate-grants.ts +++ b/packages/cli/src/generate-grants.ts @@ -155,7 +155,7 @@ export function insertGrants( } /** Every `definePermissions([` in the masked text: the index of each call's opening `[`. */ -const declarationLists = (masked: string): readonly number[] => +export const declarationLists = (masked: string): readonly number[] => [...masked.matchAll(/\bdefinePermissions\(\s*\[/g)].map( (found) => found.index + found[0].length - 1, ); diff --git a/packages/cli/src/guards.ts b/packages/cli/src/guards.ts index 862b32b6f..038bc8cf8 100644 --- a/packages/cli/src/guards.ts +++ b/packages/cli/src/guards.ts @@ -124,7 +124,7 @@ const failed = (path: string, cause: string): Finding => ({ const invalid = (path: string, cause: string): Finding => ({ code: 'X_GUARD_INVALID', cause, - fix: `export a \`guard\` object — { summary, check } — from ${path}, then: x verify`, + fix: `export a \`guard\` object — { summary, check } — from ${path}, then: x verify --only boundaries --json`, docs: ERROR_DOCS_URL, at: path, }); @@ -157,7 +157,7 @@ const notADirectory: Finding = { const findingInvalid = (path: string, cause: string): Finding => ({ code: 'X_GUARD_FINDING_INVALID', cause: `${path} returned a finding that is not one: ${cause}`, - fix: `{ code: 'X_', cause: '', fix: '' } // what ${path} returns, then: x verify`, + fix: `{ code: 'X_', cause: '', fix: '' } // what ${path} returns, then: x verify --only boundaries --json`, docs: ERROR_DOCS_URL, at: path, }); diff --git a/packages/cli/src/heavy-env.ts b/packages/cli/src/heavy-env.ts index 356480cf8..14e8d9c62 100644 --- a/packages/cli/src/heavy-env.ts +++ b/packages/cli/src/heavy-env.ts @@ -9,6 +9,8 @@ export const HEAVY_DIR_ENV = 'ULTIMATE_HEAVY_DIR'; export const HEAVY_MEMORY_ENV = 'ULTIMATE_HEAVY_MEMORY'; /** The local core ceiling, replacing `LOCAL_CORE_CAP`. */ export const HEAVY_CORES_ENV = 'ULTIMATE_HEAVY_CORES'; +/** `0` turns the CPU half of admission off (`heavy-pressure.ts`); memory still applies. */ +export const HEAVY_CPU_ENV = 'ULTIMATE_HEAVY_CPU'; /** Set on a holder's children: the reservation file they run inside. A protocol, not a knob. */ export const HEAVY_HOLD_ENV = 'ULTIMATE_HEAVY_HOLD'; diff --git a/packages/cli/src/heavy-errors.ts b/packages/cli/src/heavy-errors.ts index fb0668430..3ea4fd67b 100644 --- a/packages/cli/src/heavy-errors.ts +++ b/packages/cli/src/heavy-errors.ts @@ -3,6 +3,7 @@ import { UltimateError } from '@ultimat3/core'; import { HEAVY_MEMORY_ENV } from './heavy-env'; +import type { LedgerWait } from './heavy-ledger'; import { quoteArg } from './shell-quote'; export class HeavyLedgerUnavailableError extends UltimateError { @@ -34,3 +35,20 @@ export class HeavyCommandNotFoundError extends UltimateError { }); } } + +/** `--max-wait` ran out in the queue. The one refusal admission makes, and only when asked to. */ +export class HeavyWaitExpiredError extends UltimateError { + constructor(input: { + readonly label: string; + readonly maxWaitMs: number; + readonly wait: LedgerWait; + }) { + const { wait } = input; + const holders = wait.holds.map((entry) => entry.label).join(', ') || 'an admission in progress'; + super({ + code: 'X_HEAVY_WAIT_EXPIRED', + cause: `${JSON.stringify(input.label)} was not admitted within --max-wait ${String(input.maxWaitMs / 1000)}s and did not start: it needs ${String(wait.needMb)} MB, was ${String(wait.position)} of ${String(wait.queued)} in the queue, and ${String(wait.usedMb)} of ${String(wait.capacityMb)} MB are held by ${holders}${wait.cpu === undefined ? '' : ', and the CPU was saturated'}; run it again later or with a longer --max-wait`, + fix: 'x heavy status --json', + }); + } +} diff --git a/packages/cli/src/heavy-holders.test.ts b/packages/cli/src/heavy-holders.test.ts new file mode 100644 index 000000000..4a923b21e --- /dev/null +++ b/packages/cli/src/heavy-holders.test.ts @@ -0,0 +1,77 @@ +// Who really keeps a reservation alive: the processes with its file open, read from the OS. + +import { afterEach, describe, expect, test } from 'bun:test'; +// why: Bun ships no descriptor-level file API, and the fixture must hold one open. +import { closeSync, mkdirSync, mkdtempSync, openSync, rmSync, symlinkSync } from 'node:fs'; +// why: the test's scratch root. +import { tmpdir } from 'node:os'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import type { ExecResult, Runner } from './exec'; +import { holderPids } from './heavy-holders'; + +const dirs: string[] = []; +afterEach(() => { + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); +const scratch = (): string => { + const dir = mkdtempSync(join(tmpdir(), 'heavy-holders-')); + dirs.push(dir); + return dir; +}; + +const lsof = + (stdout: string): Runner => + async (command): Promise => ({ + command, + code: stdout === '' ? 1 : 0, + ok: stdout !== '', + stdout, + stderr: '', + durationMs: 0, + }); + +const NO_PROC = '/no-such-proc'; + +describe('holderPids', () => { + test('reads a fake /proc: the processes with the file open, and only those', async () => { + const root = scratch(); + const file = join(root, 'hold'); + await Bun.write(file, '{}'); + const proc = join(root, 'proc'); + for (const [pid, target] of [ + ['41', file], + ['42', join(root, 'other')], + ['43', file], + ] as const) { + mkdirSync(join(proc, pid, 'fd'), { recursive: true }); + symlinkSync(target, join(proc, pid, 'fd', '3')); + } + mkdirSync(join(proc, 'self', 'fd'), { recursive: true }); + mkdirSync(join(proc, 'sys')); + expect(await holderPids(file, lsof(''), '/', proc)).toEqual([41, 43]); + }); + + test.skipIf(process.platform !== 'linux')('never counts the asking process', async () => { + const file = join(scratch(), 'hold'); + await Bun.write(file, '{}'); + const fd = openSync(file, 'r'); + try { + expect(await holderPids(file, lsof(''), '/')).toEqual([]); + } finally { + closeSync(fd); + } + }); + + test('without /proc it asks lsof', async () => { + expect(await holderPids('/l/hold', lsof('41\n43\n'), '/', NO_PROC)).toEqual([41, 43]); + }); + + test('lsof finding nobody is an answer; lsof missing is not', async () => { + expect(await holderPids('/l/hold', lsof(''), '/', NO_PROC)).toEqual([]); + const missing: Runner = async () => { + throw Object.assign(new TypeError('no lsof'), { code: 'ENOENT' }); + }; + expect(await holderPids('/l/hold', missing, '/', NO_PROC)).toBeUndefined(); + }); +}); diff --git a/packages/cli/src/heavy-holders.ts b/packages/cli/src/heavy-holders.ts new file mode 100644 index 000000000..c3d3b1978 --- /dev/null +++ b/packages/cli/src/heavy-holders.ts @@ -0,0 +1,78 @@ +// Which processes keep a reservation alive: every process with its lock file open, as the OS sees +// it. The ledger records only the wrapper's pid, and after `kill -9` of the wrapper the memory is +// held by the command it started, so a recorded pid cannot say what would free it. + +// why: Bun ships no directory listing, readlink or realpath primitive. +import { existsSync, readdirSync, readlinkSync, realpathSync } from 'node:fs'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import type { Runner } from './exec'; + +const PROC = '/proc'; + +function realOrSame(path: string): string { + try { + return realpathSync(path); + } catch { + return path; // Released while being asked about: nothing will match it, which is the answer. + } +} + +function hasOpen(procRoot: string, pid: string, target: string): boolean { + const fds = join(procRoot, pid, 'fd'); + try { + return readdirSync(fds).some((fd) => { + try { + return readlinkSync(join(fds, fd)) === target; + } catch { + return false; // The descriptor closed between the listing and the read. + } + }); + } catch { + return false; // The process exited, or belongs to another user. + } +} + +const parsePids = (text: string): readonly number[] => + text + .split('\n') + .map((line) => line.trim()) + .filter((line) => /^\d+$/.test(line)) + .map(Number); + +/** + * The pids with `file` open, this process excepted, or `undefined` when the OS will not say: `/proc` + * where it exists, `lsof` elsewhere (macOS; Windows has no ledger). + */ +export async function holderPids( + file: string, + runner: Runner, + cwd: string, + procRoot: string = PROC, +): Promise { + if (existsSync(join(procRoot, 'self', 'fd'))) { + const target = realOrSame(file); + return readdirSync(procRoot) + .filter((name) => /^\d+$/.test(name) && Number(name) !== process.pid) + .filter((pid) => hasOpen(procRoot, pid, target)) + .map(Number) + .sort((a, b) => a - b); + } + try { + // `lsof` exits 1 for "nobody has it open", so its stdout is read whatever the status. + const result = await runner(['lsof', '-t', file], { cwd }); + return parsePids(result.stdout).filter((pid) => pid !== process.pid); + } catch { + return undefined; // No `lsof` on this machine. + } +} + +/** The fallback when no holder list exists. Signal 0 delivers nothing; it only asks. */ +export function isProcessRunning(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} diff --git a/packages/cli/src/heavy-ledger.test.ts b/packages/cli/src/heavy-ledger.test.ts index e4b3caf32..edc9f7cfd 100644 --- a/packages/cli/src/heavy-ledger.test.ts +++ b/packages/cli/src/heavy-ledger.test.ts @@ -3,7 +3,7 @@ import { afterEach, describe, expect, test } from 'bun:test'; // why: temp directories and an append-only event log; Bun ships neither primitive. -import { mkdtempSync, readdirSync, readFileSync, rmSync } from 'node:fs'; +import { mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; // why: the test's scratch root. import { tmpdir } from 'node:os'; // why: Bun ships no path-join primitive. @@ -41,19 +41,24 @@ interface ChildPlan { readonly label: string; /** Runs once admitted, as source text; the child then prints `held` and awaits it. */ readonly body?: string; + /** The CPU reads busy for as long as this file exists. */ + readonly busyWhile?: string; } function reserver(plan: ChildPlan) { const source = ` import { flockBinding } from ${JSON.stringify(FLOCK)}; import { reserve } from ${JSON.stringify(LEDGER)}; - import { appendFileSync } from 'node:fs'; + import { appendFileSync, existsSync } from 'node:fs'; + const busyWhile = ${JSON.stringify(plan.busyWhile ?? null)}; + const cpu = busyWhile === null ? {} : { cpu: () => ({ source: 'psi', value: 99, limit: 80, + busy: existsSync(busyWhile) }) }; const { tryLock } = await flockBinding(); const log = (line) => appendFileSync(${JSON.stringify(join(plan.dir, 'events'))}, line + '\\n'); console.log('joining'); const held = await reserve({ dir: ${JSON.stringify(plan.dir)}, capacityMb: ${plan.capacityMb}, unitMb: ${plan.mb}, units: 1, label: ${JSON.stringify(plan.label)}, cwd: '/w', tryLock, - pollMs: 5, onWait: () => console.log('waiting') }); + pollMs: 5, ...cpu, onWait: (w) => console.log(w.cpu === undefined ? 'waiting' : 'waiting cpu') }); log('+ ${plan.mb} ${plan.label}'); console.log('held'); ${plan.body ?? 'await new Promise(() => {});'} @@ -205,6 +210,26 @@ describe.skipIf(NO_FLOCK)('reserve', () => { while (readLedger(dir, lock).holds.length > 0) await new Promise(setImmediate); }, 30_000); + test('a busy CPU holds back a run that fits, until it calms; alone, a run goes anyway', async () => { + const dir = scratch(); + const busyWhile = join(dir, 'busy'); + writeFileSync(busyWhile, ''); + // Nothing admitted: pressure from outside the ledger must not stop the first run. + const first = reserver({ dir, capacityMb: 100, mb: 10, label: 'first', busyWhile }); + await first.until('held'); + const second = reserver({ dir, capacityMb: 100, mb: 10, label: 'second', busyWhile }); + await second.until('waiting cpu'); + // Behind the head, memory to spare: the line names the CPU, not a holder that is not the cause. + const third = reserver({ dir, capacityMb: 100, mb: 10, label: 'third', busyWhile }); + await third.until('waiting cpu'); + expect(events(dir)).toEqual(['+ 10 first']); + rmSync(busyWhile); + await third.until('held'); + expect(events(dir)).toEqual(['+ 10 first', '+ 10 second', '+ 10 third']); + for (const one of [first, second, third]) one.child.kill('SIGKILL'); + await Promise.all([first.child.exited, second.child.exited, third.child.exited]); + }, 30_000); + test('a request over the capacity runs alone instead of being refused', async () => { const dir = scratch(); const huge = reserver({ dir, capacityMb: 10, mb: 50, label: 'huge', body: '' }); diff --git a/packages/cli/src/heavy-ledger.ts b/packages/cli/src/heavy-ledger.ts index c02dd53b8..0da2cc5f0 100644 --- a/packages/cli/src/heavy-ledger.ts +++ b/packages/cli/src/heavy-ledger.ts @@ -24,8 +24,10 @@ import { } from 'node:fs'; // why: Bun ships no path-join primitive. import { join } from 'node:path'; -import { HeavyLedgerUnavailableError } from './heavy-errors'; +import { HeavyLedgerUnavailableError, HeavyWaitExpiredError } from './heavy-errors'; import type { TryLock } from './heavy-flock'; +import type { CpuReading } from './heavy-pressure'; +import { cpuAdmission } from './heavy-pressure'; export const LEDGER_POLL_MS = 100; /** A temp file this old lost its creator: creating one takes microseconds. */ @@ -39,8 +41,13 @@ export interface LedgerEntry { readonly at: number; } +/** An admitted reservation, and the file whose lock it is. */ +export interface LedgerHold extends LedgerEntry { + readonly file: string; +} + export interface LedgerView { - readonly holds: readonly LedgerEntry[]; + readonly holds: readonly LedgerHold[]; /** Oldest first: index 0 is the next admission. */ readonly queue: readonly LedgerEntry[]; } @@ -72,6 +79,10 @@ export interface ReserveInput { readonly cwd: string; readonly tryLock: TryLock; readonly pollMs?: number; + /** Give up after this long in the queue. `0` is one attempt; unset waits for as long as it takes. */ + readonly maxWaitMs?: number; + /** The CPU reading the head is judged by (`heavy-pressure.ts`); absent, CPU never holds it. */ + readonly cpu?: () => CpuReading; readonly onWait?: (wait: LedgerWait) => void; } @@ -82,6 +93,8 @@ export interface LedgerWait { readonly position: number; readonly queued: number; readonly holds: readonly LedgerEntry[]; + /** Set when memory fits and the CPU is what holds the head back. */ + readonly cpu?: CpuReading; } const codeOf = (error: unknown): unknown => @@ -164,7 +177,10 @@ function liveEntries(dir: string, tryLock: TryLock, sweep: boolean): Live[] { } export const readLedger = (dir: string, tryLock: TryLock): LedgerView => ({ - holds: liveEntries(join(dir, 'holds'), tryLock, false).map((one) => one.entry), + holds: liveEntries(join(dir, 'holds'), tryLock, false).map((one) => ({ + ...one.entry, + file: join(dir, 'holds', one.name), + })), queue: liveEntries(join(dir, 'queue'), tryLock, false).map((one) => one.entry), }); @@ -298,8 +314,11 @@ export async function reserve(input: ReserveInput): Promise { if (ticket === undefined) throw new HeavyLedgerUnavailableError('a new queue file was locked'); const minUnits = input.minUnits ?? 1; let said = ''; + /** When the head started waiting on CPU alone: `CPU_WAIT_LIMIT_MS` is counted from here. */ + let cpuBlockedAt: number | undefined; try { for (;;) { + let cpu: CpuReading | undefined; const queue = liveEntries(join(dir, 'queue'), tryLock, true); const position = queue.findIndex((one) => one.name === name); if (position === -1) throw new HeavyLedgerUnavailableError(`${dir} was removed mid-wait`); @@ -313,7 +332,18 @@ export async function reserve(input: ReserveInput): Promise { holds = liveEntries(join(dir, 'holds'), tryLock, true); const usedMb = holds.reduce((sum, one) => sum + one.entry.mb, 0); const granted = unitsThatFit({ capacityMb, usedMb, unitMb, units, minUnits, baseMb }); - if (granted > 0) { + const reading = granted > 0 ? input.cpu?.() : undefined; + const now = performance.now(); + const verdict = cpuAdmission({ + reading, + running: holds.length, + blockedMs: cpuBlockedAt === undefined ? 0 : now - cpuBlockedAt, + }); + if (granted > 0 && verdict === 'wait') { + cpuBlockedAt ??= now; + cpu = reading; + } else cpuBlockedAt = undefined; + if (granted > 0 && verdict === 'admit') { const mb = baseMb + granted * unitMb; const hold = createEntry( dir, @@ -338,19 +368,31 @@ export async function reserve(input: ReserveInput): Promise { } } const usedMb = holds.reduce((sum, one) => sum + one.entry.mb, 0); - const key = `${String(position)}|${holds.map((one) => one.name).join(',')}`; + // Behind the head, the queue is held by whatever holds the head: say CPU when that is it. + if (cpu === undefined && position > 0 && holds.length > 0) { + const reading = input.cpu?.(); + if (reading?.busy === true) cpu = reading; + } + const wait: LedgerWait = { + needMb: Math.min(baseMb + units * unitMb, capacityMb), + usedMb, + capacityMb, + position: position + 1, + queued: queue.length, + holds: holds.map((one) => one.entry), + ...(cpu === undefined ? {} : { cpu }), + }; + const key = `${String(position)}|${cpu === undefined ? '' : 'cpu'}|${holds.map((one) => one.name).join(',')}`; if (key !== said) { said = key; - input.onWait?.({ - needMb: Math.min(baseMb + units * unitMb, capacityMb), - usedMb, - capacityMb, - position: position + 1, - queued: queue.length, - holds: holds.map((one) => one.entry), - }); + input.onWait?.(wait); + } + const { maxWaitMs } = input; + const left = (maxWaitMs ?? Number.POSITIVE_INFINITY) - (performance.now() - started); + if (maxWaitMs !== undefined && left <= 0) { + throw new HeavyWaitExpiredError({ label: input.label, maxWaitMs, wait }); } - await Bun.sleep(pollMs); + await Bun.sleep(Math.min(pollMs, left)); } } finally { ticket.release(); diff --git a/packages/cli/src/heavy-pressure.test.ts b/packages/cli/src/heavy-pressure.test.ts new file mode 100644 index 000000000..4ac1c7b9c --- /dev/null +++ b/packages/cli/src/heavy-pressure.test.ts @@ -0,0 +1,97 @@ +// The CPU half of admission as a pure decision: when a run that fits in memory still waits, and the +// two rules that keep the queue moving whatever the machine is doing. + +import { describe, expect, test } from 'bun:test'; +// why: the same synchronous read `heavy-pressure.ts` makes of `/proc/pressure/cpu`. +import { readFileSync } from 'node:fs'; +// why: Bun exposes no path-join primitive. +import { join } from 'node:path'; +import { + CPU_SOME_BUSY_PERCENT, + CPU_WAIT_LIMIT_MS, + cpuAdmission, + cpuGateOn, + cpuText, + parseCpuSome, + RUNNABLE_PER_CORE_BUSY, + readCpuPressure, +} from './heavy-pressure'; + +const PSI = (avg10: number) => + `some avg10=${avg10.toFixed(2)} avg60=1.00 avg300=1.00 total=5\nfull avg10=0.00 avg60=0.00 avg300=0.00 total=0\n`; +/** A kernel with no PSI: the real read of a file that is not there, as `/proc/pressure/cpu` is. */ +const noPsi = (): string => readFileSync(join(import.meta.dir, 'no-such-pressure', 'cpu'), 'utf8'); + +describe('reading the pressure', () => { + test('PSI `cpu some` avg10 is the signal where the kernel has it', () => { + expect(parseCpuSome(PSI(95.83))).toBe(95.83); + expect(parseCpuSome('nonsense')).toBeUndefined(); + expect(readCpuPressure({ read: () => PSI(95.83), cores: 8, runnable: 0 })).toEqual({ + source: 'psi', + value: 95.83, + limit: CPU_SOME_BUSY_PERCENT, + busy: true, + }); + }); + + test('the limit itself is busy, just below it is not', () => { + const at = (value: number) => + readCpuPressure({ read: () => PSI(value), cores: 8, runnable: 0 }); + expect(at(CPU_SOME_BUSY_PERCENT).busy).toBe(true); + expect(at(CPU_SOME_BUSY_PERCENT - 0.01).busy).toBe(false); + }); + + test('without PSI it falls back to runnable tasks per core', () => { + const at = (runnable: number) => readCpuPressure({ read: noPsi, cores: 8, runnable }); + expect(at(8 * RUNNABLE_PER_CORE_BUSY)).toMatchObject({ + source: 'runnable', + value: 2, + busy: true, + }); + expect(at(8 * RUNNABLE_PER_CORE_BUSY - 1).busy).toBe(false); + }); + + test('the reason names the figure and the limit', () => { + const psi = readCpuPressure({ read: () => PSI(95.83), cores: 8, runnable: 0 }); + expect(cpuText(psi)).toBe('CPU pressure 96% (new runs wait at 80%)'); + const queue = readCpuPressure({ read: noPsi, cores: 8, runnable: 69 }); + expect(cpuText(queue)).toBe('8.6 runnable tasks per core (new runs wait at 2)'); + }); +}); + +describe('the decision', () => { + const busy = readCpuPressure({ read: () => PSI(99), cores: 8, runnable: 0 }); + const calm = readCpuPressure({ read: () => PSI(5), cores: 8, runnable: 0 }); + + test('a calm machine admits beside running work', () => { + expect(cpuAdmission({ reading: calm, running: 3, blockedMs: 0 })).toBe('admit'); + }); + + test('a busy machine holds a run back while another is running', () => { + expect(cpuAdmission({ reading: busy, running: 1, blockedMs: 0 })).toBe('wait'); + }); + + test('with nothing admitted one run always goes, even under outside pressure', () => { + expect(cpuAdmission({ reading: busy, running: 0, blockedMs: 0 })).toBe('admit'); + }); + + test('a long holder cannot starve the head: the wait on CPU is bounded', () => { + const near = { reading: busy, running: 1, blockedMs: CPU_WAIT_LIMIT_MS - 1 }; + expect(cpuAdmission(near)).toBe('wait'); + expect(cpuAdmission({ ...near, blockedMs: CPU_WAIT_LIMIT_MS })).toBe('admit'); + }); + + test('no reading is no reason to wait', () => { + expect(cpuAdmission({ reading: undefined, running: 4, blockedMs: 0 })).toBe('admit'); + }); +}); + +describe('where the gate applies', () => { + test('on for a local run; off in CI, inside a test run and by the switch', () => { + expect(cpuGateOn({}, {})).toBe(true); + expect(cpuGateOn({ CI: 'true' }, {})).toBe(false); + expect(cpuGateOn({ ULTIMATE_HEAVY_CPU: '0' }, {})).toBe(false); + expect(cpuGateOn({ NODE_ENV: 'test' }, {})).toBe(false); + expect(cpuGateOn({}, { NODE_ENV: 'test' })).toBe(false); + }); +}); diff --git a/packages/cli/src/heavy-pressure.ts b/packages/cli/src/heavy-pressure.ts new file mode 100644 index 000000000..35b5c9598 --- /dev/null +++ b/packages/cli/src/heavy-pressure.ts @@ -0,0 +1,104 @@ +// The CPU half of admission: a run that fits in memory still waits while the machine's CPU is +// saturated and another heavy run is going. The thresholds, their measurement and the decision live +// here; `heavy-ledger.ts` asks it only for the queue's head, so first-come order is kept. + +// why: Bun ships no synchronous file read, and `/proc/pressure/cpu` is read on every poll. +import { readFileSync } from 'node:fs'; +// why: Bun exposes no available-parallelism primitive. +import { availableParallelism } from 'node:os'; +import { runnableTasks } from './heavy-cores'; +import type { HeavyEnv } from './heavy-env'; +import { HEAVY_CPU_ENV, isCi } from './heavy-env'; +import { msg } from './messages'; + +/** + * Measured on the 8-core shared box, 2026-11-18, `As of` Linux 7.0: four admitted runs held 14.3 of + * 15.9 GB while load was 74-94 with 69-84 runnable tasks, and `cpu some avg10` read 95.8. Memory was + * not the bound; CPU was, and every run on it slowed. `some` is the share of time at least one task + * waited for a CPU: an 8-core box running one 4-core typecheck stays far below 80, and at 80 most of + * every second already has a queue. avg10 is the shortest window the kernel keeps. ONE reading; a + * re-measure that moves it changes only these numbers. + */ +export const CPU_SOME_BUSY_PERCENT = 80; +/** No PSI (macOS, older kernels): 69 runnable on 8 cores was 8.6 a core; 2 a core is a run queue. */ +export const RUNNABLE_PER_CORE_BUSY = 2; +/** + * How long the queue's head waits on CPU alone before it goes anyway. A holder can run for hours + * while outside work keeps the CPU busy; this bounds the wait to one admission per window. + */ +export const CPU_WAIT_LIMIT_MS = 5 * 60_000; + +export interface CpuReading { + readonly source: 'psi' | 'runnable'; + /** `cpu some avg10` in percent, or runnable tasks per core. */ + readonly value: number; + readonly limit: number; + readonly busy: boolean; +} + +/** `some avg10=95.83 …` → 95.83. */ +export function parseCpuSome(text: string): number | undefined { + const match = /^some avg10=(\d+(?:\.\d+)?)/m.exec(text); + return match === null ? undefined : Number(match[1]); +} + +const readPsi = (): string => readFileSync('/proc/pressure/cpu', 'utf8'); + +/** PSI where the kernel has it, else runnable tasks per core. Never throws. */ +export function readCpuPressure( + input: { readonly read?: () => string; readonly cores?: number; readonly runnable?: number } = {}, +): CpuReading { + let some: number | undefined; + try { + some = parseCpuSome((input.read ?? readPsi)()); + } catch { + some = undefined; // No PSI here. + } + if (some !== undefined) { + return { + source: 'psi', + value: some, + limit: CPU_SOME_BUSY_PERCENT, + busy: some >= CPU_SOME_BUSY_PERCENT, + }; + } + const cores = Math.max(1, input.cores ?? availableParallelism()); + const perCore = Math.round(((input.runnable ?? runnableTasks()) / cores) * 10) / 10; + return { + source: 'runnable', + value: perCore, + limit: RUNNABLE_PER_CORE_BUSY, + busy: perCore >= RUNNABLE_PER_CORE_BUSY, + }; +} + +/** + * Whether the head may start on CPU grounds. Never `wait` with nothing admitted: pressure from work + * outside the ledger must not stop the only run, and a ledger with no holder cannot deadlock. + */ +export function cpuAdmission(input: { + readonly reading: CpuReading | undefined; + /** Reservations admitted now. */ + readonly running: number; + /** How long the head has waited on CPU alone. */ + readonly blockedMs: number; +}): 'admit' | 'wait' { + if (input.reading?.busy !== true || input.running === 0) return 'admit'; + return input.blockedMs >= CPU_WAIT_LIMIT_MS ? 'admit' : 'wait'; +} + +/** + * Off in CI (single-tenant), off inside a test run (a fixture must not wait on the machine it runs + * on; `ambient` is this process's own env), and off with `ULTIMATE_HEAVY_CPU=0`. + */ +export function cpuGateOn(env: HeavyEnv, ambient: HeavyEnv = Bun.env): boolean { + if (isCi(env) || env[HEAVY_CPU_ENV] === '0') return false; + return env['NODE_ENV'] !== 'test' && ambient['NODE_ENV'] !== 'test'; +} + +/** `CPU pressure 96% (new runs wait at 80%)`. */ +export const cpuText = (reading: CpuReading): string => + msg(reading.source === 'psi' ? 'cli.heavy.cpu.psi' : 'cli.heavy.cpu.runnable', { + value: reading.source === 'psi' ? Math.round(reading.value) : reading.value, + limit: reading.limit, + }); diff --git a/packages/cli/src/heavy-wait.test.ts b/packages/cli/src/heavy-wait.test.ts new file mode 100644 index 000000000..6cea43501 --- /dev/null +++ b/packages/cli/src/heavy-wait.test.ts @@ -0,0 +1,109 @@ +// What a waiter is told and how long it stays: the line that says what to do, and `maxWaitMs`, the +// one bound on a queue that otherwise waits for as long as the holders run. + +import { afterEach, describe, expect, test } from 'bun:test'; +// why: a temp ledger per test; Bun ships no mkdtemp. +import { mkdtempSync, readdirSync, rmSync } from 'node:fs'; +// why: the test's scratch root. +import { tmpdir } from 'node:os'; +// why: Bun ships no path-join primitive. +import { join } from 'node:path'; +import { waitLine } from './heavy'; +import { flockBinding } from './heavy-flock'; +import type { LedgerWait } from './heavy-ledger'; +import { reserve } from './heavy-ledger'; + +// flock(2) does not exist on Windows, where admission is off (`heavy-flock.ts`). +const NO_FLOCK = process.platform === 'win32'; +const dirs: string[] = []; +afterEach(() => { + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); +const scratch = (): string => { + const dir = mkdtempSync(join(tmpdir(), 'heavy-wait-')); + dirs.push(dir); + return dir; +}; + +const tryLock = async () => { + const binding = await flockBinding(); + if (!binding.ok) expect.unreachable(binding.reason); + return binding.tryLock; +}; + +const WAIT: LedgerWait = { + needMb: 100, + usedMb: 90, + capacityMb: 100, + position: 1, + queued: 2, + holds: [{ pid: 7, mb: 90, label: 'gate', cwd: '/w/app', at: 1000 }], +}; + +describe('the wait line', () => { + test('names the holder and says the wait needs nothing from the reader', () => { + const line = waitLine('x test unit', WAIT, 4000); + expect(line).toContain('position 1 of 2'); + expect(line).toContain('gate (/w/app, 90 MB, 3s)'); + expect(line).toContain('leave it running; it starts by itself'); + expect(line).toContain('bunx x heavy status'); + }); + + test('with nobody holding yet it says an admission is under way, never "held by -"', () => { + const line = waitLine('x test unit', { ...WAIT, usedMb: 0, holds: [] }, 4000); + expect(line).toContain('admission in progress'); + expect(line).not.toContain('held by'); + expect(line).toContain('leave it running'); + }); +}); + +describe('the wait line when the CPU is the reason', () => { + test('says so, with the figure, and still says to leave it running', () => { + const cpu = { source: 'psi', value: 95.8, limit: 80, busy: true } as const; + const line = waitLine('x test unit', { ...WAIT, cpu }, 4000); + expect(line).toContain('waits for CPU'); + expect(line).toContain('CPU pressure 96% (new runs wait at 80%)'); + expect(line).toContain('gate (/w/app'); + expect(line).toContain('leave it running'); + }); +}); + +describe.skipIf(NO_FLOCK)('reserve with maxWaitMs', () => { + const input = async (dir: string, label: string, maxWaitMs?: number) => ({ + dir, + capacityMb: 10, + unitMb: 10, + units: 1, + label, + cwd: '/w', + tryLock: await tryLock(), + pollMs: 5, + ...(maxWaitMs === undefined ? {} : { maxWaitMs }), + }); + + test('gives up with the coded error naming the holder, and leaves the queue', async () => { + const dir = scratch(); + const holder = await reserve(await input(dir, 'holder')); + const refused = await reserve(await input(dir, 'late', 20)).catch((error: unknown) => error); + expect(refused).toMatchObject({ code: 'X_HEAVY_WAIT_EXPIRED' }); + expect((refused as { cause: string }).cause).toContain('holder'); + expect(readdirSync(join(dir, 'queue'))).toEqual([]); + expect(readdirSync(join(dir, 'holds'))).toHaveLength(1); + holder.release(); + }); + + test('0 still admits a request that fits now', async () => { + const held = await reserve(await input(scratch(), 'alone', 0)); + expect(held.units).toBe(1); + held.release(); + }); + + test('a holder that leaves in time admits the waiter', async () => { + const dir = scratch(); + const holder = await reserve(await input(dir, 'holder')); + const waiter = reserve({ ...(await input(dir, 'next', 30_000)), onWait: holder.release }); + const held = await waiter; + expect(held.units).toBe(1); + held.release(); + }); +}); diff --git a/packages/cli/src/heavy.test.ts b/packages/cli/src/heavy.test.ts index 11ff1bbad..8b153d777 100644 --- a/packages/cli/src/heavy.test.ts +++ b/packages/cli/src/heavy.test.ts @@ -42,6 +42,13 @@ describe.skipIf(NO_FLOCK)('heavyState', () => { expect(on.on).toBe(true); }); + // A fixture hands the code under test its own small env. Without this, every such test joined the + // real machine queue, and one that timed out left a ticket ahead of every agent on the machine. + test('inside a test process a narrowed env is off too: fixtures never join the real queue', async () => { + expect(await heavyState({})).toMatchObject({ on: false, off: 'test-run' }); + expect((await heavyState(envFor(scratch()))).on).toBe(true); + }); + test('ULTIMATE_HEAVY_MEMORY=0 is off', async () => { const state = await heavyState({ ...envFor(scratch()), ULTIMATE_HEAVY_MEMORY: '0' }); expect(state).toMatchObject({ on: false, off: 'disabled' }); diff --git a/packages/cli/src/heavy.ts b/packages/cli/src/heavy.ts index 65f420fc7..c19b1a1a1 100644 --- a/packages/cli/src/heavy.ts +++ b/packages/cli/src/heavy.ts @@ -10,12 +10,13 @@ import { isUltimateError } from '@ultimat3/core'; import { coreCap, nativeToolEnv } from './heavy-cores'; import type { HeavyEnv } from './heavy-env'; import { HEAVY_DIR_ENV, HEAVY_HOLD_ENV, isCi } from './heavy-env'; -import { HeavyLedgerUnavailableError } from './heavy-errors'; +import { HeavyLedgerUnavailableError, HeavyWaitExpiredError } from './heavy-errors'; import type { TryLock } from './heavy-flock'; import { flockBinding } from './heavy-flock'; import type { Held, LedgerEntry, LedgerWait, Reservation } from './heavy-ledger'; import { isHeld, lockPath, prepareLedger, reserve } from './heavy-ledger'; import { ledgerCapacityMb } from './heavy-memory'; +import { cpuGateOn, cpuText, readCpuPressure } from './heavy-pressure'; import { msg } from './messages'; import { writeErrorLine } from './write-line'; @@ -61,9 +62,11 @@ export async function heavyState(env: HeavyEnv): Promise { if (isCi(env)) return { on: false, off: 'ci' }; const explicitDir = env[HEAVY_DIR_ENV]; // A fixture's `x test` runs inside `bun test`, whose NODE_ENV every child inherits: queueing it - // behind real work would turn a test's timeout into a machine-load verdict. - if (env['NODE_ENV'] === 'test' && explicitDir === undefined) - return { on: false, off: 'test-run' }; + // behind real work would turn a test's timeout into a machine-load verdict. `Bun.env` as well: a + // fixture calling in-process with its own small env is just as much inside the test run, and a + // timed-out one left its ticket at the head of the machine's queue until the worker exited. + const inTestRun = env['NODE_ENV'] === 'test' || Bun.env['NODE_ENV'] === 'test'; + if (inTestRun && explicitDir === undefined) return { on: false, off: 'test-run' }; const capacityMb = ledgerCapacityMb(env); if (capacityMb === 0) return { on: false, off: 'disabled' }; const binding = await flockBinding(); @@ -105,15 +108,24 @@ export const describeEntry = (entry: LedgerEntry, now: number): string => /** The waiter's one line. Stderr always: stdout may be a `--json` document. */ export const waitLine = (label: string, wait: LedgerWait, now: number = Date.now()): string => - msg('cli.heavy.waiting', { - label: printable(label), - need: wait.needMb, - position: wait.position, - queued: wait.queued, - used: wait.usedMb, - capacity: wait.capacityMb, - holders: wait.holds.map((entry) => describeEntry(entry, now)).join('; ') || '-', - }); + // No holder to name is the instant between two admissions, not a mystery owner. + msg( + wait.cpu !== undefined + ? 'cli.heavy.waitingCpu' + : wait.holds.length === 0 + ? 'cli.heavy.waitingAdmission' + : 'cli.heavy.waiting', + { + label: printable(label), + need: wait.needMb, + position: wait.position, + queued: wait.queued, + used: wait.usedMb, + capacity: wait.capacityMb, + holders: wait.holds.map((entry) => describeEntry(entry, now)).join('; '), + cpu: wait.cpu === undefined ? '' : cpuText(wait.cpu), + }, + ); // Time this process has spent queued, so a gate step's deadline can stop counting it. let finishedWaitMs = 0; @@ -145,6 +157,8 @@ export interface HeavyRequest { readonly baseMb?: number; readonly units?: number; readonly minUnits?: number; + /** Give up after this long in the queue (`x heavy run --max-wait`). */ + readonly maxWaitMs?: number; } /** A reservation, or nothing when admission does not apply or could not work. */ @@ -160,12 +174,16 @@ async function enter(request: HeavyRequest, state: HeavyState): Promise readCpuPressure() } : {}), label: request.label, cwd: request.cwd, onWait: (wait) => writeErrorLine(waitLine(request.label, wait)), }), ); } catch (error) { + // The caller set this bound: running anyway, unadmitted, is what it asked not to happen. + if (error instanceof HeavyWaitExpiredError) throw error; warnOnce(describe(error)); return undefined; } diff --git a/packages/cli/src/job-registration.test.ts b/packages/cli/src/job-registration.test.ts index 353b61367..43505c952 100644 --- a/packages/cli/src/job-registration.test.ts +++ b/packages/cli/src/job-registration.test.ts @@ -2,7 +2,12 @@ // `defineApi`, and the name it carries is the one the queue and every trace will show. import { afterEach, describe, expect, test } from 'bun:test'; -import { cronJob, partitions, resetJobs, resetTasks } from '@ultimat3/jobs'; +// why: Bun ships no temp-directory or recursive-remove API, and no path joiner. +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { cronJob, job, partitions, resetJobs, resetTasks } from '@ultimat3/jobs'; +import { t } from '@ultimat3/schema'; import { positionalNames, unmaintainedPartitionFindings, @@ -24,6 +29,66 @@ describe('unit · a job no defineApi call names is a finding', () => { expect(unregisteredJobFinding('anonymous-task-1').fix).toContain('added to tasks: [...]'); }); + test('a job whose file is known gets that file, and an import line that pastes', () => { + const finding = unregisteredJobFinding( + 'anonymous-job-4', + 'apps/web/app/post/jobs/nudge-post.ts', + ); + expect(finding.at).toBe('apps/web/app/post/jobs/nudge-post.ts'); + expect(finding.cause).toContain('apps/web/app/post/jobs/nudge-post.ts'); + expect(finding.fix).toStartWith("import * as nudgePost from '../app/post/jobs/nudge-post';"); + expect(finding.fix).toContain('nudgePost added to jobs: [...] there'); + expect(finding.fix).not.toContain('<'); + }); + + test('a module the index reaches through a package keeps a relative path that resolves', () => { + expect( + unregisteredJobFinding('anonymous-task-1', 'packages/ops/src/1-sweep.ts').fix, + ).toStartWith("import * as module1Sweep from '../../../packages/ops/src/1-sweep';"); + }); + + test('the step finds the file by the exported handle, in what the load imported', async () => { + const dir = await mkdtemp(join(tmpdir(), 'job-registration-')); + try { + await Bun.write( + join(dir, 'apps/web/app/post/jobs/nudge-post.ts'), + `import { job } from ${JSON.stringify(import.meta.resolve('@ultimat3/jobs'))}; +import { t } from ${JSON.stringify(import.meta.resolve('@ultimat3/schema'))}; +export const nudge = job({ input: t.object({}), tenant: 'none', idempotencyKey: () => 'k', retry: { attempts: 1, backoff: 'exponential' }, async run() {} }); +`, + ); + await Bun.write(join(dir, 'apps/web/app/post/plain.ts'), 'export const plain = 1;\n'); + const files = ['apps/web/app/post/jobs/nudge-post.ts', 'apps/web/app/post/plain.ts']; + for (const file of files) await import(join(dir, file)); + const [finding] = await unregisteredJobFindings(dir, async () => ({ findings: [], files })); + expect(finding?.code).toBe('X_JOB_UNREGISTERED'); + expect(finding?.at).toBe('apps/web/app/post/jobs/nudge-post.ts'); + } finally { + resetJobs(); + await rm(dir, { recursive: true, force: true }); + } + }); + + test('a job no module exports still says where the import goes', async () => { + job({ + input: t.object({}), + tenant: 'none', + idempotencyKey: () => 'k', + retry: { attempts: 1, backoff: 'exponential' }, + async run() {}, + }); + try { + const [finding] = await unregisteredJobFindings('/nowhere', async () => ({ + findings: [], + files: [], + })); + expect(finding?.at).toBe('apps/web/api/index.ts'); + expect(finding?.fix).toContain("from ''"); + } finally { + resetJobs(); + } + }); + test('a load that failed reports nothing here — the step that loaded it reports the load', async () => { const failed = async () => ({ findings: [{ code: 'X_APP_LOAD_FAILED', cause: 'broken', fix: 'x verify --json' }], @@ -34,7 +99,7 @@ describe('unit · a job no defineApi call names is a finding', () => { // The partition half: an entity declaring `partitionBy` with no `partitions()` job registered never // gets a partition created ahead — the first insert past the last migrated one fails in production. -const job = (name: string, schedule?: string) => ({ +const handle = (name: string, schedule?: string) => ({ name, ...(schedule === undefined ? {} : { schedule: { name: schedule } }), }); @@ -46,7 +111,7 @@ describe('unit · a partitioned entity needs its maintenance job', () => { }); test('a partitioned table and no partitions() job is X_PARTITIONS_UNMAINTAINED', () => { - const findings = unmaintainedPartitionFindings(['events'], [job('reindexPost')]); + const findings = unmaintainedPartitionFindings(['events'], [handle('reindexPost')]); expect(findings.map((finding) => finding.code)).toEqual(['X_PARTITIONS_UNMAINTAINED']); expect(findings[0]?.cause).toContain('events'); expect(findings[0]?.fix).toContain('partitions()'); @@ -75,7 +140,7 @@ describe('unit · a partitioned entity needs its maintenance job', () => { test('a look-alike shape is not one either: only what partitions() minted counts', () => { expect( - unmaintainedPartitionFindings(['events'], [job('partitions', 'partitions.schedule')]), + unmaintainedPartitionFindings(['events'], [handle('partitions', 'partitions.schedule')]), ).toHaveLength(1); }); diff --git a/packages/cli/src/job-registration.ts b/packages/cli/src/job-registration.ts index 211c81a3e..060b7e0b0 100644 --- a/packages/cli/src/job-registration.ts +++ b/packages/cli/src/job-registration.ts @@ -3,17 +3,27 @@ // and every dead-letter trace — moves with the import order (`X_JOB_UNREGISTERED`). And an entity // declaring `partitionBy` with no `partitions()` job registered (`X_PARTITIONS_UNMAINTAINED`). +// why: Bun ships no path API; `posix` because the fix prints an import specifier on every OS. +import { join, posix } from 'node:path'; import { ERROR_DOCS_URL } from '@ultimat3/core'; import { + isJobHandle, isPartitionsJob, + isTaskHandle, partitionedEntities, registeredJobs, registeredTasks, } from '@ultimat3/jobs'; import { loadApp } from './app-load'; import type { AppLoader } from './app-permissions'; +import { API_INDEX } from './app-root'; import type { Finding } from './output'; +/** The load, plus the modules it imported: where a positional job's file is looked up. */ +type JobAppLoader = ( + root: string, +) => Promise> & { readonly files?: readonly string[] }>; + /** What `job()` and `task()` name a declaration nobody named (`packages/jobs/src/job.ts`). */ const POSITIONAL = /^anonymous-(job|task)-\d+$/; @@ -24,16 +34,59 @@ export function positionalNames( return names.filter((name) => POSITIONAL.test(name)).sort(); } -export function unregisteredJobFinding(name: string): Finding { - const kind = name.startsWith('anonymous-task-') ? 'tasks' : 'jobs'; +/** `nudge-post` → `nudgePost`: the name the import line binds, a valid identifier either way. */ +const moduleAlias = (file: string): string => { + const base = posix.basename(file).replace(/\.tsx?$/, ''); + const [head = '', ...rest] = base.split(/[^A-Za-z0-9]+/).filter((word) => word !== ''); + const alias = [head, ...rest.map((word) => word.charAt(0).toUpperCase() + word.slice(1))].join( + '', + ); + return /^[A-Za-z_]/.test(alias) ? alias : `module${alias}`; +}; + +/** The import the API index needs, relative to it and without the extension. */ +const importLine = (file: string, alias: string): string => { + const path = posix.relative(posix.dirname(API_INDEX), file).replace(/\.tsx?$/, ''); + return `import * as ${alias} from '${path.startsWith('.') ? path : `./${path}`}';`; +}; + +/** + * `file` is the module exporting the job, when one does: the positional name moves between runs, so + * the path is the only thing that tells several agents in one checkout whose job this is. + */ +export function unregisteredJobFinding(name: string, file?: string): Finding { + const task = name.startsWith('anonymous-task-'); + const alias = file === undefined ? '' : moduleAlias(file); + const line = + file === undefined ? `import * as ${alias} from '';` : importLine(file, alias); + const where = file === undefined ? '' : ` declared in ${file}`; return { code: 'X_JOB_UNREGISTERED', - cause: `${name} is a ${kind === 'tasks' ? 'task' : 'job'} no defineApi() call names, so it carries the positional name ${kind === 'tasks' ? 'task()' : 'job()'} minted — on the queue row, in the manifest and in every dead-letter trace, and it moves when the import order does`, - fix: `import * as from ''; // in apps/web/api/index.ts, and added to ${kind}: [...] there — then x manifest`, - at: 'apps/web/api/index.ts', + cause: `${name}${where} is a ${task ? 'task' : 'job'} no defineApi() call names, so it carries the positional name ${task ? 'task()' : 'job()'} minted — on the queue row, in the manifest and in every dead-letter trace, and it moves when the import order does`, + fix: `${line} // in ${API_INDEX}, and ${alias} added to ${task ? 'tasks' : 'jobs'}: [...] there — then x manifest`, + at: file ?? API_INDEX, }; } +/** + * Which loaded module exports each positional handle. The imports answer from the module cache: the + * load just evaluated these files. A handle no module exports has no file to name. + */ +async function declaringFiles( + root: string, + files: readonly string[], +): Promise> { + const found = new Map(); + for (const file of files) { + const module = (await import(join(root, file))) as Record; + for (const value of Object.values(module)) { + if (!isJobHandle(value) && !isTaskHandle(value)) continue; + if (POSITIONAL.test(value.name) && !found.has(value.name)) found.set(value.name, file); + } + } + return found; +} + /** * The step's answer. Loads the app the way every other gate step does — the registries are only * whole after the scan — and says nothing when the load failed: a short registry is the load's @@ -41,11 +94,14 @@ export function unregisteredJobFinding(name: string): Finding { */ export async function unregisteredJobFindings( root: string, - load: AppLoader = loadApp, + load: JobAppLoader = loadApp, ): Promise { - if ((await load(root)).findings.length > 0) return []; + const app = await load(root); + if (app.findings.length > 0) return []; + const names = positionalNames(); + const files = names.length === 0 ? new Map() : await declaringFiles(root, app.files ?? []); return [ - ...positionalNames().map(unregisteredJobFinding), + ...names.map((name) => unregisteredJobFinding(name, files.get(name))), ...unmaintainedPartitionFindings( partitionedEntities().map((described) => described.table), registeredJobs(), diff --git a/packages/cli/src/mcp-errors-agents.ts b/packages/cli/src/mcp-errors-agents.ts new file mode 100644 index 000000000..237b614e1 --- /dev/null +++ b/packages/cli/src/mcp-errors-agents.ts @@ -0,0 +1,20 @@ +// `errors.explain`'s rows for the codes of agents sharing a machine and a repository (`error-codes- +// agents.ts`), split out of `mcp-errors.ts` under the 500-line ceiling. Spread into `CLI_FIXES` +// there, whose `Record` is still what makes a code without a row a build error. + +import type { CliErrorCode } from './error-codes'; + +export const AGENTS_FIXES = { + X_PORT_IN_USE: 'x dev --port 3001 --json', + X_DEV_ALREADY_RUNNING: + 'x dev --json # after stopping the x dev that already owns this checkout', + // The path, not `x dev`: the boot has already refused and rerunning it refuses again. The + // error's own `fix:` carries the resolved `.x/dev.lock`; this table cannot know the state dir. + X_DEV_LOCK_UNREADABLE: 'rm .x/dev.lock # then: x dev --json', + X_UPGRADE_LOCKED: 'cat .x/upgrade.lock', + X_HEAVY_WAIT_EXPIRED: 'x heavy status --json', + X_MERGE_UNRESOLVED: 'git diff --name-only --diff-filter=U', + X_MERGE_DRIVER_UNSET: 'bunx x merge-driver install', + X_CATALOG_UNPARSEABLE: 'bunx biome check packages/i18n/catalogs', + X_DOCTOR_PROBE_FAILED: 'bunx x manifest --json', +} as const satisfies Partial>; diff --git a/packages/cli/src/mcp-errors.ts b/packages/cli/src/mcp-errors.ts index 4c4188d99..f419c3754 100644 --- a/packages/cli/src/mcp-errors.ts +++ b/packages/cli/src/mcp-errors.ts @@ -7,6 +7,7 @@ import type { ErrorExplanation } from '@ultimat3/mcp'; import type { CliErrorCode } from './error-codes'; import { CLI_ERROR_CODES } from './error-codes'; import { codeFixes, codeFixScan } from './error-fixes'; +import { AGENTS_FIXES } from './mcp-errors-agents'; import { SHOT_FIXES } from './mcp-errors-shot'; import { VERIFY_FIXES } from './mcp-errors-verify'; @@ -61,6 +62,7 @@ const CLI_FIXES: Readonly> = { X_PACKAGE_DUPLICATED: 'x i18n check --json # the finding names both copies and the package.json to pin', // `x shot` and the `ui.*` tools: their own family, in `mcp-errors-shot.ts`. + ...AGENTS_FIXES, ...SHOT_FIXES, // The gate's floor, run and split: their own family, in `mcp-errors-verify.ts`. ...VERIFY_FIXES, @@ -77,8 +79,8 @@ const CLI_FIXES: Readonly> = { 'x verify --json # the finding names the file, the line and the name it could not resolve', X_ERROR_CODE_UNTHROWN: 'x errors explain X_ERROR_CODE_UNTHROWN --json', X_CLI_UNEXPECTED: 'x doctor --json', - // `-p .`: an app root has no `references`, so the gate's own step runs this form (#450). - X_TYPECHECK_FAILED: 'bunx tsc -p . --pretty false', + // The step and not raw `tsc`: it queues in the machine ledger and takes the compiler lock. + X_TYPECHECK_FAILED: 'x verify --only typecheck --json', X_LINT_FAILED: 'bunx @biomejs/biome check .', X_TEST_FAILED: 'x test --json # the finding carries the exact bun test invocation that failed', X_FILE_TOO_LONG: 'x verify --json # the finding names the file to split', @@ -137,12 +139,6 @@ const CLI_FIXES: Readonly> = { 'x new my-app --json # a name with letters or digits; it becomes the directory', X_APP_EMPTY: 'x doctor --json # then check the app root: no module under apps/ or packages/ loaded', - X_PORT_IN_USE: 'x dev --port 3001 --json', - X_DEV_ALREADY_RUNNING: - 'x dev --json # after stopping the x dev that already owns this checkout', - // The path, not `x dev`: the boot has already refused and rerunning it refuses again. The - // error's own `fix:` carries the resolved `.x/dev.lock`; this table cannot know the state dir. - X_DEV_LOCK_UNREADABLE: 'rm .x/dev.lock # then: x dev --json', // `x doctor`, not `x db status`: generation failing is a reachability question, and `x db status` // reads the migration ledger only. `x doctor` is already this table's answer for // X_DB_STUDIO_FAILED. @@ -327,7 +323,7 @@ const CLI_FIXES: Readonly> = { X_REPRO_CAPTURE_OFF: 'x jobs list --json # a job id replays with capture off; for requests set failureCapture: { enabled: true } in app.config.ts', X_ENTITY_RETAIN_UNSWEPT: - "add export const retention = purge({ entities: 'declared' }) to a jobs module named in jobs: [...] in apps/web/api/index.ts, then x verify", + "add export const retention = purge({ entities: 'declared' }) to a jobs module named in jobs: [...] in apps/web/api/index.ts, then x manifest", X_NEW_SHAPE_UNKNOWN: 'bunx @ultimat3/cli help new', X_NEW_SHAPE_CONFLICT: 'bunx @ultimat3/cli new myapp --shape blog', X_SEO_CARD_ROUTE_UNKNOWN: @@ -402,7 +398,6 @@ const CLI_FIXES: Readonly> = { 'x repro --json # lists the failing jobs and error groups; then run it again with one id and --bundle, and attach the file it writes', X_ADVISORY_FLOOR_UNMET: 'bun update source-map-js immutable', X_UPGRADE_CODEMODS_PENDING: 'x upgrade codemods --from 0.0.0 --json', - X_UPGRADE_LOCKED: 'cat .x/upgrade.lock', X_APP_MIGRATIONS_UNAPPLIED: 'x db migrate --json', X_HEAVY_LEDGER_UNAVAILABLE: 'x heavy status --json', X_HEAVY_MEMORY_INVALID: 'export ULTIMATE_HEAVY_MEMORY=8g', diff --git a/packages/cli/src/merge-driver-catalog.test.ts b/packages/cli/src/merge-driver-catalog.test.ts new file mode 100644 index 000000000..23a001178 --- /dev/null +++ b/packages/cli/src/merge-driver-catalog.test.ts @@ -0,0 +1,55 @@ +// The catalog merge: every key either side added survives, and two wordings of one key do not. + +import { describe, expect, test } from 'bun:test'; +import { mergeCatalogText } from './merge-driver-catalog'; + +const json = (value: unknown): string => `${JSON.stringify(value, null, 2)}\n`; + +describe('unit · mergeCatalogText', () => { + const base = json({ a: { x: 'X' } }); + + test('both sides add keys under one parent: all kept, sorted, either direction', () => { + const ours = json({ a: { beta: 'B', x: 'X' } }); + const theirs = json({ a: { alpha: 'A', x: 'X' } }); + const expected = json({ a: { alpha: 'A', beta: 'B', x: 'X' } }); + expect(mergeCatalogText(base, ours, theirs)).toBe(expected); + expect(mergeCatalogText(base, theirs, ours)).toBe(expected); + }); + + test('one side rewording and the other adding is no conflict', () => { + expect(mergeCatalogText(base, json({ a: { x: 'Y' } }), json({ a: { n: 'N', x: 'X' } }))).toBe( + json({ a: { n: 'N', x: 'Y' } }), + ); + }); + + test('a key one side deleted stays deleted', () => { + expect(mergeCatalogText(base, json({ a: {} }), json({ a: { x: 'X', y: 'Y' } }))).toBe( + json({ a: { y: 'Y' } }), + ); + }); + + test('two wordings of one key are a conflict', () => { + expect( + mergeCatalogText(base, json({ a: { x: 'Y' } }), json({ a: { x: 'Z' } })), + ).toBeUndefined(); + }); + + test('deleted on one side, reworded on the other is a conflict', () => { + expect(mergeCatalogText(base, json({ a: {} }), json({ a: { x: 'Z' } }))).toBeUndefined(); + }); + + test('a side that is not JSON falls through to the text merge', () => { + expect(mergeCatalogText(base, '<<<<<<< HEAD\n', base)).toBeUndefined(); + }); + + test('a hand-grouped catalog keeps its order, the new key after it', () => { + const grouped = json({ z: '1', a: '2' }); + expect(mergeCatalogText(grouped, json({ z: '1', a: '2', m: '3' }), grouped)).toBe( + json({ z: '1', a: '2', m: '3' }), + ); + }); + + test('two branches adding the first keys to an absent file merge from an empty base', () => { + expect(mergeCatalogText('', json({ b: '1' }), json({ a: '2' }))).toBe(json({ a: '2', b: '1' })); + }); +}); diff --git a/packages/cli/src/merge-driver-catalog.ts b/packages/cli/src/merge-driver-catalog.ts new file mode 100644 index 000000000..feb6361fc --- /dev/null +++ b/packages/cli/src/merge-driver-catalog.ts @@ -0,0 +1,58 @@ +// The catalog half of `x merge-driver`: a three-way merge of the JSON, key by key. Lines cannot do +// it — the entry before an addition gains a comma on both sides. Two values for one key stay a +// conflict: which wording ships is a decision, and this file does not make decisions. + +import { isJsonObject } from '@ultimat3/core'; +import type { JsonObject } from './json-merge'; +import { sortJsonDeep } from './json-merge'; + +const CONFLICT = Symbol('conflict'); +const ABSENT = Symbol('absent'); + +const same = (a: unknown, b: unknown): boolean => Bun.deepEquals(a, b, true); + +function mergeValue(base: unknown, ours: unknown, theirs: unknown): unknown { + if (same(ours, theirs) || same(theirs, base)) return ours; + if (same(ours, base)) return theirs; + if (!isJsonObject(ours) || !isJsonObject(theirs)) return CONFLICT; + return mergeObject(isJsonObject(base) ? base : {}, ours, theirs); +} + +function mergeObject(base: JsonObject, ours: JsonObject, theirs: JsonObject): unknown { + const merged: [string, unknown][] = []; + for (const key of new Set([...Object.keys(ours), ...Object.keys(theirs)])) { + const read = (side: JsonObject): unknown => (Object.hasOwn(side, key) ? side[key] : ABSENT); + const value = mergeValue(read(base), read(ours), read(theirs)); + if (value === CONFLICT) return CONFLICT; + if (value !== ABSENT) merged.push([key, value]); + } + return Object.fromEntries(merged); +} + +const parsed = (text: string): JsonObject | undefined => { + try { + const value: unknown = JSON.parse(text.trim() === '' ? '{}' : text); + return isJsonObject(value) ? value : undefined; + } catch { + // Not JSON is an answer here: the caller falls back to a text merge, with markers. + return undefined; + } +}; + +const isSortedDeep = (value: unknown): boolean => + JSON.stringify(value) === JSON.stringify(sortJsonDeep(value)); + +/** + * The merged catalog as `x g` prints one, or `undefined` when a side is not JSON or two sides + * disagree on a key. Sorted where both sides were, which is every catalog a generator wrote, so + * the merge order leaves no trace; a hand-grouped catalog keeps its order, the other side's new + * keys after it. + */ +export function mergeCatalogText(base: string, ours: string, theirs: string): string | undefined { + const [b, o, t] = [parsed(base), parsed(ours), parsed(theirs)]; + if (b === undefined || o === undefined || t === undefined) return undefined; + const merged = mergeObject(b, o, t); + if (merged === CONFLICT) return undefined; + const value = isSortedDeep(o) && isSortedDeep(t) ? sortJsonDeep(merged) : merged; + return `${JSON.stringify(value, null, 2)}\n`; +} diff --git a/packages/cli/src/merge-driver-config.ts b/packages/cli/src/merge-driver-config.ts new file mode 100644 index 000000000..71688e1f6 --- /dev/null +++ b/packages/cli/src/merge-driver-config.ts @@ -0,0 +1,43 @@ +// The two git merge drivers `.gitattributes` names, and whether this clone has them. Git keeps a +// driver's command in `.git/config`, which no clone inherits, so `x merge-driver install` (a step +// of `bun run setup`) writes it; an attribute naming an unknown driver is an ordinary text merge. + +import type { Runner } from './exec'; +import { quoteArg } from './shell-quote'; + +export const GENERATED_DRIVER = 'x-generated'; +export const REGISTRY_DRIVER = 'x-registry'; + +/** + * Through the installed binary, never `bunx x`: a clone with no install would fetch an unrelated + * npm package named `x` and run it on the merge. With none, this is git's own text merge. + */ +export const registryDriverCommand = (prefix: string): string => { + const bin = quoteArg(`${prefix}node_modules/.bin/x`); + return `if [ -x ${bin} ]; then ${bin} merge-driver run %O %A %B %P; else git merge-file -L ours -L base -L theirs %A %O %B; fi`; +}; + +/** Every `git config` key and value, in the order they are written. */ +export const driverConfig = (prefix: string): readonly (readonly [string, string])[] => [ + [`merge.${GENERATED_DRIVER}.name`, 'keep ours: x manifest rebuilds this file'], + // `true` exits 0 and leaves our side in place: the file is a projection, rebuilt after the merge. + [`merge.${GENERATED_DRIVER}.driver`, 'true'], + [`merge.${REGISTRY_DRIVER}.name`, 'keep what both sides added to an x registry'], + [`merge.${REGISTRY_DRIVER}.driver`, registryDriverCommand(prefix)], +]; + +/** `undefined` outside a git checkout. */ +export async function gitPrefix(root: string, runner: Runner): Promise { + const found = await runner(['git', 'rev-parse', '--show-prefix'], { cwd: root }); + return found.ok ? found.stdout.trim() : undefined; +} + +/** Whether `.gitattributes` names the driver and this clone does not define it. */ +export async function driverUnset(root: string, runner: Runner): Promise { + const attributes = Bun.file(`${root}/.gitattributes`); + if (!(await attributes.exists())) return false; + if (!(await attributes.text()).includes(`merge=${REGISTRY_DRIVER}`)) return false; + if ((await gitPrefix(root, runner)) === undefined) return false; + const key = `merge.${REGISTRY_DRIVER}.driver`; + return !(await runner(['git', 'config', '--get', key], { cwd: root })).ok; +} diff --git a/packages/cli/src/merge-driver-lines.test.ts b/packages/cli/src/merge-driver-lines.test.ts new file mode 100644 index 000000000..0c5833102 --- /dev/null +++ b/packages/cli/src/merge-driver-lines.test.ts @@ -0,0 +1,88 @@ +// Which conflicts `x merge-driver` may resolve: both sides adding whole entries, and nothing else. + +import { describe, expect, test } from 'bun:test'; +import { resolveAdditions } from './merge-driver-lines'; + +const conflict = (ours: string[], base: string[], theirs: string[]): string[] => [ + '<<<<<<< ours', + ...ours, + '||||||| base', + ...base, + '=======', + ...theirs, + '>>>>>>> theirs', +]; + +const merged = (ours: string[], theirs: string[], base: string[] = []) => + resolveAdditions(['before', ...conflict(ours, base, theirs), 'after', ''].join('\n')); + +describe('unit · both sides added', () => { + test('list entries are all kept, in one order whichever side is ours', () => { + const one = merged([' beta,'], [' alpha,']); + const other = merged([' alpha,'], [' beta,']); + expect(one).toEqual({ text: 'before\n alpha,\n beta,\nafter\n', conflicts: 0 }); + expect(other).toEqual(one); + }); + + test('imports land sorted by module, as the formatter requires', () => { + const { text } = merged( + ["import * as z from '../app/beta/z';"], + ["import * as a from '../app/alpha/a';"], + ); + expect(text.split('\n').slice(1, 3)).toEqual([ + "import * as a from '../app/alpha/a';", + "import * as z from '../app/beta/z';", + ]); + }); + + test('multi-line entries stay whole: their shared closing lines are not spliced', () => { + const entry = (name: string): string[] => [` ${name}: {`, " off: 'why',", ' },']; + const { text, conflicts } = merged(entry('zeta'), entry('alpha')); + expect(conflicts).toBe(0); + expect(text.split('\n').slice(1, 7)).toEqual([...entry('alpha'), ...entry('zeta')]); + }); + + test('an entry both sides added identically appears once', () => { + const line = "import { userActor } from '@ultimat3/core';"; + expect(merged([line], [line]).text).toBe(`before\n${line}\nafter\n`); + }); + + test('a comment travels with the entry below it', () => { + const { text, conflicts } = merged([' // why', ' zeta,'], [' alpha,']); + expect(conflicts).toBe(0); + expect(text).toBe('before\n // why\n zeta,\n alpha,\nafter\n'); + }); +}); + +describe('unit · a real conflict stays one, markers and all', () => { + const stays = (result: ReturnType): void => { + expect(result.conflicts).toBe(1); + expect(result.text).toContain('<<<<<<< ours'); + expect(result.text).toContain('>>>>>>> theirs'); + }; + + test('a line that existed before was changed: a revoked grant is not resurrected', () => { + stays(merged([" 'a:read',"], [" 'a:write',", " 'b:read',"], [" 'a:write',"])); + }); + + test('one key, two values', () => { + stays(merged([' createPost: { off: 1 },'], [' createPost: { off: 2 },'])); + }); + + test('one import binding from two modules', () => { + stays(merged(["import * as list from './a';"], ["import * as list from './b';"])); + }); + + test('two imports from one module, which only a person can combine', () => { + stays(merged(["import { a } from 'm';"], ["import { b } from 'm';"])); + }); + + test('an addition that is not a whole entry', () => { + stays(merged([' a: {'], [' b,'])); + stays(merged([' no-trailing-comma'], [' b,'])); + }); + + test('text with no conflict passes through byte for byte', () => { + expect(resolveAdditions('a\nb\n')).toEqual({ text: 'a\nb\n', conflicts: 0 }); + }); +}); diff --git a/packages/cli/src/merge-driver-lines.ts b/packages/cli/src/merge-driver-lines.ts new file mode 100644 index 000000000..4b8d45ee7 --- /dev/null +++ b/packages/cli/src/merge-driver-lines.ts @@ -0,0 +1,114 @@ +// The registry half of `x merge-driver`: conflicts that are only "both sides ADDED here" keep both. +// Not `merge=union`, which also keeps a line one side deleted (a revoked grant in `roles.ts` comes +// back) and splices multi-line entries together. A conflict touching an existing line stays one. + +import { maskOf } from './source-list-edit'; + +export const OURS_MARKER = '<<<<<<< ours'; +const BASE_MARKER = '||||||| base'; +const SPLIT_MARKER = '======='; +const THEIRS_MARKER = '>>>>>>> theirs'; + +export interface LineMerge { + readonly text: string; + /** The conflicts left in `text`, each still between its markers. */ + readonly conflicts: number; +} + +const indentOf = (line: string): number => /^[ \t]*/.exec(line)?.[0].length ?? 0; + +const depthOf = (masked: string): number => { + let depth = 0; + for (const ch of masked) { + if ('([{'.includes(ch)) depth += 1; + else if (')]}'.includes(ch)) depth -= 1; + } + return depth; +}; + +/** Brackets balanced and the last code a `,` or `;`: nothing a neighbour could be spliced into. */ +const isWhole = (lines: readonly string[]): boolean => { + const masked = maskOf(lines.join('\n')).trimEnd(); + return depthOf(masked) === 0 && /[,;]$/.test(masked); +}; + +/** + * The added lines as whole entries — an entry starts at the block's own indent and runs until it + * is complete, so `foo: {`…`},` is one and a comment joins what it precedes. `undefined` when they + * do not divide that way: such a hunk is not a list of additions. + */ +function entriesOf(lines: readonly string[]): readonly string[] | undefined { + if (lines.some((line) => line.trim() === '')) return undefined; + const indent = Math.min(...lines.map(indentOf)); + const entries: string[][] = []; + for (const line of lines) { + const last = entries.at(-1); + if (last === undefined || (indentOf(line) === indent && isWhole(last))) entries.push([line]); + else last.push(line); + } + return entries.every(isWhole) ? entries.map((entry) => entry.join('\n')) : undefined; +} + +const SPECIFIER = /\bfrom\s+'([^']+)';\s*$/; + +/** What may appear once: an import's binding or module, an object entry's key. */ +function identityOf(entry: string): string | undefined { + const code = entry.trimStart(); + if (code.startsWith('import ')) { + const namespace = /^import \* as ([\w$]+)/.exec(code)?.[1]; + return namespace === undefined ? `from:${SPECIFIER.exec(code)?.[1]}` : `as:${namespace}`; + } + const key = /^(?:([\w$]+)|'([^']+)'|"([^"]+)")\s*:/.exec(code); + return key === null ? undefined : `key:${key[1] ?? key[2] ?? key[3]}`; +} + +const sortKeyOf = (entry: string): string => SPECIFIER.exec(entry)?.[1] ?? entry.trimStart(); + +/** + * Both sides' entries, each once, in one order. Sorted so that merging A into B and B into A write + * the same bytes, and so an import block stays in the order Biome requires. `undefined` when the + * two sides declare one name differently, which is a real conflict. + */ +function bothSides(ours: readonly string[], theirs: readonly string[]): string[] | undefined { + const left = entriesOf(ours); + const right = entriesOf(theirs); + if (left === undefined || right === undefined) return undefined; + const entries = [...new Set([...left, ...right])]; + const identities = entries.flatMap((entry) => identityOf(entry) ?? []); + if (new Set(identities).size !== identities.length) return undefined; + return entries + .sort((a, b) => { + const [x, y] = [sortKeyOf(a), sortKeyOf(b)]; + return x < y ? -1 : x > y ? 1 : 0; + }) + .flatMap((entry) => entry.split('\n')); +} + +/** `text` — `git merge-file --diff3 -L ours -L base -L theirs` output — with additions resolved. */ +export function resolveAdditions(text: string): LineMerge { + const lines = text.split('\n'); + const out: string[] = []; + let conflicts = 0; + for (let index = 0; index < lines.length; index += 1) { + const line = lines[index] as string; + if (line !== OURS_MARKER) { + out.push(line); + continue; + } + const baseAt = lines.indexOf(BASE_MARKER, index); + const splitAt = lines.indexOf(SPLIT_MARKER, baseAt); + const endAt = lines.indexOf(THEIRS_MARKER, splitAt); + if (baseAt === -1 || splitAt === -1 || endAt === -1) { + out.push(line); + continue; + } + const merged = + splitAt === baseAt + 1 + ? bothSides(lines.slice(index + 1, baseAt), lines.slice(splitAt + 1, endAt)) + : undefined; + if (merged === undefined) conflicts += 1; + out.push(...(merged ?? lines.slice(index, endAt + 1))); + index = endAt; + } + return { text: out.join('\n'), conflicts }; +} diff --git a/packages/cli/src/merge-state.test.ts b/packages/cli/src/merge-state.test.ts new file mode 100644 index 000000000..ce2d6df4e --- /dev/null +++ b/packages/cli/src/merge-state.test.ts @@ -0,0 +1,124 @@ +// What a merge left, read off a real directory: the catalog is named, and its symptoms are not. + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +// why: Bun has no temp-directory or recursive-remove API. +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { loadApp } from './app-load'; +import type { Runner } from './exec'; +import { + CATALOG_ROOT, + catalogFileFindings, + unmergedFindings, + withMergeCauses, +} from './merge-state'; +import type { Finding } from './output'; +import { CATALOG_ROOT as TEMPLATE_CATALOG_ROOT } from './templates/locales'; + +const EN = 'packages/i18n/catalogs/en.json'; +let root = ''; + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'merge-state-')); + mkdirSync(join(root, 'packages/i18n/catalogs'), { recursive: true }); +}); +afterEach(() => rmSync(root, { recursive: true, force: true })); + +const catalog = (text: string): void => writeFileSync(join(root, EN), text); + +const answering = + (code: number, stdout: string): Runner => + async (command) => ({ + command, + code, + ok: code === 0, + stdout, + stderr: '', + durationMs: 0, + }); + +describe('unit · catalogFileFindings', () => { + test('reads the directory the templates write catalogs to', () => { + expect(CATALOG_ROOT).toBe(TEMPLATE_CATALOG_ROOT); + }); + + test('an app with no catalog directory has nothing to report', async () => { + rmSync(join(root, 'packages'), { recursive: true }); + expect(await catalogFileFindings(root)).toEqual([]); + }); + + test('a valid catalog is silent', async () => { + catalog('{"a":"b"}\n'); + expect(await catalogFileFindings(root)).toEqual([]); + }); + + test('a conflict marker is X_MERGE_UNRESOLVED, at the file, with a fix naming it', async () => { + catalog('<<<<<<< HEAD\n{}\n'); + const [found] = await catalogFileFindings(root); + expect(found?.code).toBe('X_MERGE_UNRESOLVED'); + expect(found?.at).toBe(EN); + expect(found?.fix).toContain(EN); + expect(found?.fix).not.toContain('x doctor'); + }); + + test('anything else malformed is X_CATALOG_UNPARSEABLE, naming the file', async () => { + catalog('{ "broken": \n'); + const [found] = await catalogFileFindings(root); + expect(found?.code).toBe('X_CATALOG_UNPARSEABLE'); + expect(found?.at).toBe(EN); + expect(found?.fix).toBe(`bunx biome check ${EN} # names the line to correct`); + }); +}); + +describe('unit · unmergedFindings', () => { + test('one finding per path git holds unmerged', async () => { + const found = await unmergedFindings(root, answering(0, 'a.ts\nb.ts\n')); + expect(found.map((one) => [one.code, one.at])).toEqual([ + ['X_MERGE_UNRESOLVED', 'a.ts'], + ['X_MERGE_UNRESOLVED', 'b.ts'], + ]); + }); + + test('outside a checkout there is nothing to report', async () => { + expect(await unmergedFindings(root, answering(128, ''))).toEqual([]); + }); +}); + +describe('unit · withMergeCauses', () => { + const symptom = (at: string): Finding => ({ + code: 'X_CLI_UNEXPECTED', + cause: "SyntaxError: JSON Parse error: Unrecognized token '<'", + fix: 'x doctor --json', + at, + }); + const other: Finding = { code: 'X_OTHER', cause: 'unrelated', fix: 'x help', at: 'z.ts' }; + + test('a broken catalog replaces the parse errors of its importers, and nothing else', async () => { + catalog('<<<<<<< HEAD\n'); + const found = await withMergeCauses(root, [symptom('a.tsx'), symptom('b.tsx'), other]); + expect(found.map((one) => one.code)).toEqual(['X_MERGE_UNRESOLVED', 'X_OTHER']); + expect(found[0]?.at).toBe(EN); + }); + + test('a healthy catalog leaves the load findings as they were', async () => { + catalog('{}\n'); + expect(await withMergeCauses(root, [symptom('a.tsx')])).toEqual([symptom('a.tsx')]); + }); +}); + +describe('unit · loadApp over a catalog a merge broke', () => { + test('names the catalog instead of the module that imported it', async () => { + catalog('<<<<<<< HEAD\n{}\n'); + writeFileSync(join(root, 'app.config.ts'), 'export default {};\n'); + mkdirSync(join(root, 'apps/web/api'), { recursive: true }); + writeFileSync( + join(root, 'apps/web/api/uses-catalog.ts'), + `import en from '../../../${EN}';\nexport const size = Object.keys(en).length;\n`, + ); + const { findings } = await loadApp(root); + expect(findings[0]?.code).toBe('X_MERGE_UNRESOLVED'); + expect(findings[0]?.at).toBe(EN); + expect(findings.some((one) => one.at === 'apps/web/api/uses-catalog.ts')).toBe(false); + }); +}); diff --git a/packages/cli/src/merge-state.ts b/packages/cli/src/merge-state.ts new file mode 100644 index 000000000..7bb3ec4cc --- /dev/null +++ b/packages/cli/src/merge-state.ts @@ -0,0 +1,105 @@ +// What a merge left behind, as findings: paths git still holds unmerged, and i18n catalogs that do +// not parse. A catalog is imported by nearly every page, so one stray marker used to surface as a +// SyntaxError blamed on each importer, with the catalog itself named nowhere. + +// why: `Bun.Glob.scan` throws on a missing directory, and an app need not have any catalog. +import { existsSync } from 'node:fs'; +import { ERROR_DOCS_URL, renderThrowable } from '@ultimat3/core'; +import type { Runner } from './exec'; +import { exec } from './exec'; +import type { Finding } from './output'; +import { quoteArg } from './shell-quote'; + +/** + * `templates/locales.ts`'s `CATALOG_ROOT`, restated: `loadApp` reaches this module, and the serve + * graph carries no template (`serve-graph.test.ts`). `merge-state.test.ts` holds the two equal. + */ +export const CATALOG_ROOT = 'packages/i18n/catalogs'; + +export const UNMERGED_FIX = 'git diff --name-only --diff-filter=U'; + +const MARKER = /^(?:<{7}|\|{7}|={7}|>{7})(?: |$)/m; + +const finding = (code: string, cause: string, fix: string, at: string): Finding => ({ + code, + cause, + fix, + docs: ERROR_DOCS_URL, + at, +}); + +/** Every path git holds unmerged, relative to `root`; none outside a checkout. */ +export async function unmergedPaths( + root: string, + runner: Runner = exec, +): Promise { + const argv = ['git', 'diff', '--name-only', '--relative', '--diff-filter=U']; + const found = await runner(argv, { cwd: root }).catch(() => undefined); + if (found?.ok !== true) return []; + return found.stdout.split('\n').filter((path) => path !== ''); +} + +export async function unmergedFindings(root: string, runner?: Runner): Promise { + return (await unmergedPaths(root, runner)).map((path) => + finding( + 'X_MERGE_UNRESOLVED', + `${path} is still unmerged: git holds conflicting versions of it`, + UNMERGED_FIX, + path, + ), + ); +} + +/** One finding per catalog that will not load, naming the file every importer failed on. */ +export async function catalogFileFindings(root: string): Promise { + const findings: Finding[] = []; + const directory = `${root}/${CATALOG_ROOT}`; + if (!existsSync(directory)) return findings; + for await (const name of new Bun.Glob('*.json').scan(directory)) { + const at = `${CATALOG_ROOT}/${name}`; + const text = await Bun.file(`${root}/${at}`).text(); + if (MARKER.test(text)) { + findings.push( + finding( + 'X_MERGE_UNRESOLVED', + `${at} holds conflict markers, so it is not JSON and no page importing it loads`, + `grep -nE '^(<{7}|={7}|>{7})' ${quoteArg(at)} # keep the right side of each, delete the markers`, + at, + ), + ); + continue; + } + try { + JSON.parse(text); + } catch (error) { + findings.push( + finding( + 'X_CATALOG_UNPARSEABLE', + `${at} is not valid JSON (${renderThrowable(error)}), so no page importing it loads`, + `bunx biome check ${quoteArg(at)} # names the line to correct`, + at, + ), + ); + } + } + return findings; +} + +const isCatalogSymptom = (one: Finding): boolean => + one.code === 'X_CLI_UNEXPECTED' && one.cause.includes('JSON Parse error'); + +/** + * A failed app load, its causes first: an unparseable catalog replaces the parse errors it caused + * in every importer, and an unmerged path leads, because it explains whatever else failed. Asked + * only once the load has already failed, so a healthy app pays nothing. + */ +export async function withMergeCauses( + root: string, + findings: readonly Finding[], +): Promise { + const catalogs = await catalogFileFindings(root); + const held = new Set(catalogs.map((one) => one.at)); + const unmerged = (await unmergedFindings(root)).filter((one) => !held.has(one.at)); + const rest = catalogs.length > 0 ? findings.filter((one) => !isCatalogSymptom(one)) : findings; + return [...unmerged, ...catalogs, ...rest]; +} diff --git a/packages/cli/src/merge-two-features.test.ts b/packages/cli/src/merge-two-features.test.ts new file mode 100644 index 000000000..e0fbddd34 --- /dev/null +++ b/packages/cli/src/merge-two-features.test.ts @@ -0,0 +1,144 @@ +// The finding this exists for, end to end with real git and the real CLI: two branches each run +// `x g resource`, and the merge needs no hand edit. Only a genuine disagreement stays a conflict. + +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +// why: Bun has no directory or symlink API. +import { mkdirSync, symlinkSync } from 'node:fs'; +// why: Bun exposes no path-join primitive. +import { join } from 'node:path'; +import type { TempTree } from '@ultimat3/testing'; +import { tempTree } from '@ultimat3/testing'; + +const BIN = join(import.meta.dir, 'bin.ts'); +const REGISTRIES = [ + 'apps/web/api/index.ts', + 'apps/web/exercise.ts', + 'apps/web/shared/roles.ts', + 'packages/db/src/client.ts', + 'packages/i18n/catalogs/en.json', +]; +const PROJECTIONS = ['x.manifest.json', 'openapi.json']; + +let tree: TempTree | undefined; +let app = ''; + +const run = (argv: readonly string[]): { code: number; out: string } => { + const done = Bun.spawnSync([...argv], { + cwd: app, + // git discovery stops at the tree, and its background maintenance stays off. + env: { ...Bun.env, ...tree?.gitEnv }, + stdout: 'pipe', + stderr: 'pipe', + timeout: 60_000, + }); + return { code: done.exitCode, out: `${done.stdout.toString()}${done.stderr.toString()}` }; +}; +const must = (argv: readonly string[]): string => { + const { code, out } = run(argv); + if (code !== 0) expect.unreachable(`${argv.join(' ')} exited ${code}: ${out}`); + return out; +}; +const git = (...args: string[]): string => must(['git', ...args]); +const x = (...args: string[]): string => must([process.execPath, BIN, ...args]); +const read = (path: string): Promise => Bun.file(join(app, path)).text(); +const unmerged = (): string => git('diff', '--name-only', '--diff-filter=U').trim(); + +const commitAll = (message: string): void => { + git('add', '-A'); + git('commit', '-qm', message); +}; + +/** + * A feature on its own branch, as one agent would produce it. The projections are written after + * `x g`, which would otherwise refresh them by loading an app this fixture never installed. + */ +const feature = async (name: string): Promise => { + git('checkout', '-q', '-b', name, 'main'); + x('g', 'resource', name); + for (const path of PROJECTIONS) await Bun.write(join(app, path), `{"feature":"${name}"}\n`); + commitAll(name); +}; + +beforeAll(async () => { + tree = tempTree('merge-features-'); + const parent = tree.path; + const created = Bun.spawnSync([process.execPath, BIN, 'new', 'demo', '--dir', parent], { + env: { ...Bun.env, ...tree.gitEnv }, + stdout: 'pipe', + stderr: 'pipe', + timeout: 60_000, + }); + if (created.exitCode !== 0) expect.unreachable(`x new: ${created.stderr.toString()}`); + app = join(parent, 'demo'); + mkdirSync(join(app, 'node_modules/.bin'), { recursive: true }); + symlinkSync(BIN, join(app, 'node_modules/.bin/x')); + git('config', 'user.email', 'test@example.com'); + git('config', 'user.name', 'test'); + git('config', 'commit.gpgsign', 'false'); + git('config', 'rerere.enabled', 'false'); + git('branch', '-M', 'main'); + await Bun.write(join(app, '.git/info/exclude'), 'node_modules\n'); + x('merge-driver', 'install'); + await feature('alpha'); + await feature('beta'); +}, 120_000); + +afterAll(async () => { + await tree?.dispose(); +}); + +describe('unit · two generated features, one merge', () => { + let first: readonly string[] = []; + + test('beta merges alpha with no conflict and keeps both features in every registry', async () => { + git('checkout', '-q', '-B', 'one', 'alpha'); + git('merge', '--no-edit', 'beta'); + expect(unmerged()).toBe(''); + first = await Promise.all(REGISTRIES.map(read)); + for (const [index, path] of REGISTRIES.entries()) { + expect(first[index], path).not.toContain('<<<<<<<'); + if (path === 'apps/web/exercise.ts') continue; + expect(first[index], path).toMatch(/alpha/); + expect(first[index], path).toMatch(/beta/); + } + expect(first[0]).toContain("import * as createAlpha from '../app/alpha/actions/create-alpha';"); + expect(first[0]).toContain("import * as createBeta from '../app/beta/actions/create-beta';"); + expect(JSON.parse(first[4] as string)).toMatchObject({ admin: { alphas: {}, betas: {} } }); + }, 30_000); + + test('the other direction writes the same bytes', async () => { + expect(first).toHaveLength(REGISTRIES.length); + git('checkout', '-q', '-B', 'other', 'beta'); + git('merge', '--no-edit', 'alpha'); + expect(unmerged()).toBe(''); + expect(await Promise.all(REGISTRIES.map(read))).toEqual([...first]); + }, 30_000); + + test('the projections keep our side, for x manifest to rebuild', async () => { + git('checkout', '-q', '-B', 'projected', 'alpha'); + git('merge', '--no-edit', 'beta'); + expect(await read('x.manifest.json')).toBe('{"feature":"alpha"}\n'); + }, 30_000); + + test('both branches rewriting one grant is still a conflict, and is told as one', async () => { + const ROLES = 'apps/web/shared/roles.ts'; + const edit = async (branch: string, grant: string): Promise => { + git('checkout', '-q', '-B', branch, 'main'); + const source = await read(ROLES); + await Bun.write(join(app, ROLES), source.replace("'post:read',", `'${grant}',`)); + commitAll(branch); + }; + await edit('left', 'post:left'); + await edit('right', 'post:right'); + git('checkout', '-q', 'left'); + const merge = run(['git', 'merge', '--no-edit', 'right']); + expect(merge.code).not.toBe(0); + expect(merge.out).toContain('X_MERGE_UNRESOLVED'); + expect(unmerged()).toBe(ROLES); + const held = await read(ROLES); + expect(held).toContain('<<<<<<< ours'); + expect(held).toContain("'post:left'"); + expect(held).toContain("'post:right'"); + git('merge', '--abort'); + }, 30_000); +}); diff --git a/packages/cli/src/messages-db.ts b/packages/cli/src/messages-db.ts index 07aaac9c1..7846457aa 100644 --- a/packages/cli/src/messages-db.ts +++ b/packages/cli/src/messages-db.ts @@ -38,8 +38,16 @@ export const DB_MESSAGES = { // The fourth: no DDL and no sidecar moved, but `packages/db/schema/` did — a hand edit repaired, // or a framework table that changed under an upgrade. `unchanged` would deny a write the lines // below it list, and this command is `X_SCHEMA_DUMP_DRIFT`'s whole `fix:`. - 'cli.db.gen.dumped': 'no migration needed — schema dump rewritten ({count} file(s))', + // Names the migration, because in a shared checkout another agent's concurrent `x db gen` may be + // what recorded this run's table: the entities are one registry, so its migration carries them. + 'cli.db.gen.dumped': + 'no migration needed — {newest} already records every entity (a concurrent x db gen may have written it); schema dump rewritten ({count} file(s))', + // A sidecar moved and no migration exists: `written` would name one nobody can apply. + 'cli.db.gen.forkMerged': + 'forked migrations merged — {count} object(s) corrected in {id}.snapshot.json; no SQL changed, no migration deleted', + 'cli.db.gen.newest': ' newest migration: {id}', 'cli.db.gen.written': 'migration {id} generated', + 'cli.db.gen.writtenTables': 'migration {id} generated — tables: {tables}', // Exit 1 and a migration on disk at once: the summary carries the half the exit code hides. 'cli.db.gen.writtenDumpFailed': 'migration {id} generated and written — the schema dump was not; do not generate it again', diff --git a/packages/cli/src/messages.ts b/packages/cli/src/messages.ts index f65b650c7..ff4e4d06f 100644 --- a/packages/cli/src/messages.ts +++ b/packages/cli/src/messages.ts @@ -318,16 +318,52 @@ const CATALOG = { ...FLOOR_MESSAGES, ...STATUS_MESSAGES, 'cli.heavy.waiting': - 'heavy: {label} waits for {need} MB, position {position} of {queued}; {used} of {capacity} MB held by {holders}', - 'cli.heavy.lockWait': 'heavy: waiting for this checkout’s {name} lock, held by {holder}', + 'heavy: {label} waits for {need} MB, position {position} of {queued}; {used} of {capacity} MB held by {holders} — leave it running; it starts by itself (bunx x heavy status)', + 'cli.heavy.waitingAdmission': + 'heavy: {label} waits for {need} MB, position {position} of {queued}; admission in progress — leave it running; it starts by itself (bunx x heavy status)', + 'cli.heavy.waitingCpu': + 'heavy: {label} waits for CPU, position {position} of {queued}: {cpu}; running: {holders} — leave it running; it starts by itself (bunx x heavy status)', + 'cli.heavy.cpu.psi': 'CPU pressure {value}% (new runs wait at {limit}%)', + 'cli.heavy.cpu.runnable': '{value} runnable tasks per core (new runs wait at {limit})', + 'cli.heavy.status.cpu': 'cpu: {cpu} — a new heavy run waits while another is running', + 'cli.heavy.status.cpuOff': 'cpu: {cpu} — not gating here ({reason})', + 'cli.heavy.lockWait': + 'heavy: waiting for this checkout’s {name} lock, held by {holder} — leave it running; it starts by itself', 'cli.heavy.status.summary': '{used} of {capacity} MB reserved, {holds} running, {queued} queued; {cores} core(s) per run', 'cli.heavy.status.off': 'admission is off here ({reason}); {cores} core(s) per run', 'cli.heavy.status.machine': 'machine: load {load} on {cpus} cores, {runnable} runnable, {free} GB of {total} GB available', - 'cli.heavy.status.hold': 'running: {entry}', + 'cli.heavy.status.hold': 'running: {entry}; {pids}', + 'cli.heavy.status.wrapper': 'wrapper pid {pid}', + 'cli.heavy.status.wrapperGone': 'wrapper pid {pid} exited', + 'cli.heavy.status.command': 'command pid {pids}', 'cli.heavy.status.queued': 'queued {position}: {entry}', 'cli.heavy.run.done': '{command} exited {code} after {ran}, {waited} in the queue', + // `test-workers.ts`: a step line for a run the machine held below its plan. + 'cli.test.narrowed': 'ran {peak} at once (peak), planned {planned}', + 'cli.test.narrowedWhy': + 'ran {peak} at once (peak), planned {planned} — the machine allowed no more (bunx x heavy status); plan: {reason}', + // `verify-live.ts`: which test stack the suites got, one line on stderr. + 'cli.verify.services.line.started': + 'services: started {project} ({services}) — bunx x clean --services stops it', + 'cli.verify.services.line.reused': + 'services: reused {project} — bunx x clean --services stops it', + 'cli.verify.services.line.shared': + 'services: shared {project} with {other} ({why}) — it stays up for that checkout too', + 'cli.verify.services.why.same-app': 'another checkout of this app', + 'cli.verify.services.why.declared': 'ULTIMATE_TEST_SERVICES_SHARED=1', + // `clean-services.ts`: what `x clean --services` did to the stack, and the ones it left alone. + 'cli.clean.services.stopped': 'stopped test stack {project} ({services})', + 'cli.clean.services.would-stop': + 'would stop test stack {project} ({services}) — --dry-run stopped nothing', + 'cli.clean.services.kept': + 'kept test stack {project}: {owner} started it and may be using it — stop it from there, or: docker compose -p {project} down --volumes', + 'cli.clean.services.none': 'no test stack is running for this app', + 'cli.clean.services.other': + 'other test stack {project} ({status}) from {root} — left alone; stop it with: docker compose -p {project} down --volumes', + 'cli.clean.services.otherGone': + 'other test stack {project} ({status}) from {root}, a checkout that no longer exists — left alone; stop it with: docker compose -p {project} down --volumes', } as const; export type MessageKey = keyof typeof CATALOG; diff --git a/packages/cli/src/new-shapes.test.ts b/packages/cli/src/new-shapes.test.ts index e159e2248..57eeb9509 100644 --- a/packages/cli/src/new-shapes.test.ts +++ b/packages/cli/src/new-shapes.test.ts @@ -95,7 +95,9 @@ describe('unit · x new --shape', () => { expect(files).toContain('apps/web/app/posts/[id]/page.tsx'); const api = await Bun.file(join(parent, 'ops', 'apps/web/api/index.ts')).text(); expect(api).toContain('createSession'); - expect(api).toContain('queries: [postList, postById]'); + expect(api).toContain( + ' queries: [\n // One entry per line: branches that each add one merge cleanly (x merge-driver).\n postList,\n postById,\n ],', + ); const dry = await newCommand.run( contextFor(['new', 'ops2', '--shape', 'dashboard', '--dry-run'], parent), ); diff --git a/packages/cli/src/output-findings.test.ts b/packages/cli/src/output-findings.test.ts index 1dbf6e189..aed610880 100644 --- a/packages/cli/src/output-findings.test.ts +++ b/packages/cli/src/output-findings.test.ts @@ -13,14 +13,48 @@ describe('runnableFix', () => { ['bun install && x db migrate', 'bun install && bunx x db migrate'], ['x db gen "a" && x db migrate', 'bunx x db gen "a" && bunx x db migrate'], ['METRICS_PORT=9091 x dev --json', 'METRICS_PORT=9091 bunx x dev --json'], + // The tail of a line is an instruction as much as its head: `x` is on nobody's PATH there either. + [ + 'replace #f00 in a.scss, then: x verify --only boundaries', + 'replace #f00 in a.scss, then: bunx x verify --only boundaries', + ], + ['edit a.ts, then x db migrate', 'edit a.ts, then bunx x db migrate'], + ['edit a.ts — re-run: x upgrade', 'edit a.ts — re-run: bunx x upgrade'], + [ + 'bunx biome check a.ts # then x verify --only lint', + 'bunx biome check a.ts # then bunx x verify --only lint', + ], + [ + 'grants: [] // in a.ts, then: x db gen ', + 'grants: [] // in a.ts, then: bunx x db gen ', + ], + ['env -u DATABASE_URL x db migrate', 'env -u DATABASE_URL bunx x db migrate'], + ['start it with `x dev`', 'start it with `bunx x dev`'], + // An apostrophe opens no quote, or everything up to the next one would pass as an argument. + [ + "add to its roles' grants, then x verify --only policy, in 'a.ts'", + "add to its roles' grants, then bunx x verify --only policy, in 'a.ts'", + ], + [ + "add to its roles' grants, then x verify --only policy", + "add to its roles' grants, then bunx x verify --only policy", + ], + [ + `in action "a"'s declaration, then x verify --only manifest`, + `in action "a"'s declaration, then bunx x verify --only manifest`, + ], + [ + "tokens.role('fg'), then: x verify --only boundaries", + "tokens.role('fg'), then: bunx x verify --only boundaries", + ], ])('%s', (fix, runnable) => { expect(runnableFix(fix)).toBe(runnable); }); test.each([ - // A comment is not run, and quoted text is an argument. - ['bunx biome check a.ts # then x verify'], - ['bun add mail # && x db migrate'], + // Quoted text is an argument, and the repo's own spelling already runs. + ['git commit -m "before x upgrade"'], + ['bun run x -- verify'], ['echo "x y && x z"'], ["echo 'a && x b'"], // Already runnable, and every other word. @@ -34,8 +68,8 @@ describe('runnableFix', () => { }); test('applied once it is a no-op: a report read back and rendered again does not grow', () => { - expect(runnableFix(runnableFix('x verify && x db migrate'))).toBe( - 'bunx x verify && bunx x db migrate', + expect(runnableFix(runnableFix('x verify && x db migrate, then: x doctor'))).toBe( + 'bunx x verify && bunx x db migrate, then: bunx x doctor', ); }); }); diff --git a/packages/cli/src/output-findings.ts b/packages/cli/src/output-findings.ts index 725210bc7..67a9b2653 100644 --- a/packages/cli/src/output-findings.ts +++ b/packages/cli/src/output-findings.ts @@ -21,14 +21,32 @@ const commandWordAt = (fix: string, from: number): number => { const isBareX = (fix: string, at: number): boolean => fix[at] === 'x' && (at + 1 === fix.length || fix[at + 1] === ' '); +/** What may stand before a quote that opens one: after a letter or a `"` it is an apostrophe. */ +const BEFORE_QUOTE = /[\s([{=,:]/; +/** What may stand before a cited `x`: `apps/web/x.ts` and `max y` cite nothing. */ +const BEFORE_CITATION = /[\s(`]/; +/** A runner already in front of this `x`. `then run x verify` has none: `run` alone is prose. */ +const RUNNER_BEFORE = /(?:^|\s)(?:bunx|npx|bun|bun run|--bun)\s+$/; + +const opensQuote = (fix: string, at: number): boolean => + (fix[at] === "'" || fix[at] === '"') && (at === 0 || BEFORE_QUOTE.test(fix[at - 1] ?? '')); + +/** `x ` anywhere in the line: a `then: x verify` tail is followed as literally as a head. */ +const citesX = (fix: string, at: number): boolean => + fix[at] === 'x' && + fix[at + 1] === ' ' && + /[a-z<]/.test(fix[at + 2] ?? '') && + (at === 0 || BEFORE_CITATION.test(fix[at - 1] ?? '')) && + !RUNNER_BEFORE.test(fix.slice(0, at)); + /** - * A leading `x ` becomes `bunx x `, and after `&&`, `||` and `;`. Quoted text is an argument and a - * `#` tail is never run, so both are copied as they are. `bunx x` itself does not match, so a report - * read back by `x verify merge` and rendered again does not grow. + * Every `x` the line tells its reader to run becomes `bunx x`: the head of each `&&`, `||` and `;` + * segment, and a citation anywhere after it, a `#` or `//` tail included, because a reader follows + * `# then: x verify` too. A quoted span is an argument and is copied. `bunx x` itself does not + * match, so a report read back by `x verify merge` and rendered again does not grow. */ export function runnableFix(fix: string): string { let out = ''; - let quote: string | undefined; let at = 0; let segmentStart = true; while (at < fix.length) { @@ -36,27 +54,32 @@ export function runnableFix(fix: string): string { const word = commandWordAt(fix, at); out += fix.slice(at, word); at = word; - if (isBareX(fix, at)) out += 'bunx '; segmentStart = false; + if (isBareX(fix, at)) { + out += 'bunx x'; + at += 1; + } continue; } - const char = fix[at] ?? ''; - if (quote !== undefined) { - if (char === quote) quote = undefined; - } else if (char === "'" || char === '"') { - quote = char; - } else if (char === '#' && (at === 0 || fix[at - 1] === ' ')) { - return out + fix.slice(at); - } else { - const end = SEGMENT_END.exec(fix.slice(at)); - if (end !== null) { - out += end[0]; - at += end[0].length; - segmentStart = true; - continue; - } + const close = opensQuote(fix, at) ? fix.indexOf(fix[at] ?? '', at + 1) : -1; + if (close !== -1) { + out += fix.slice(at, close + 1); + at = close + 1; + continue; } - out += char; + const end = SEGMENT_END.exec(fix.slice(at)); + if (end !== null) { + out += end[0]; + at += end[0].length; + segmentStart = true; + continue; + } + if (citesX(fix, at)) { + out += 'bunx x'; + at += 1; + continue; + } + out += fix[at] ?? ''; at += 1; } return out; @@ -106,6 +129,10 @@ export interface ReaderOptions { readonly bunxRunner?: boolean; } +/** One fix line outside a finding: a pointer on stderr, or `x errors explain`'s answer. */ +export const fixForReader = (fix: string, reader: ReaderOptions = {}): string => + reader.bunxRunner === true ? runnableFix(fix) : fix; + /** A finding read out of a CI log is fenced text and is never turned into an instruction. */ const withRunnableFix = (finding: Finding): Finding => { if (finding.source === 'ci-log') return finding; diff --git a/packages/cli/src/parse.test.ts b/packages/cli/src/parse.test.ts index 00bbad423..92e1f15ea 100644 --- a/packages/cli/src/parse.test.ts +++ b/packages/cli/src/parse.test.ts @@ -70,11 +70,12 @@ describe('unit · parseArgs', () => { // and it has no read-only sibling to default to — the count it would report is `x doctor`'s. // `x token` joined them in 31.x: `create` mints a credential, `revoke` ends one, and a bare // `x token` reaching either by default is a mistyped word with a side effect nobody asked for. + // `x merge-driver` too: `install` writes git config, and `run` is git's to call with its files. test('exactly the commands whose bare form is dangerous refuse it', () => { const refusing = SPECS_SHIPPED.filter( (spec) => (spec.subcommands ?? []).length > 0 && spec.defaultSubcommand === undefined, ).map((spec) => spec.name); - expect(refusing.sort()).toEqual(['auth', 'client', 'db', 'mcp', 'pr', 'token']); + expect(refusing.sort()).toEqual(['auth', 'client', 'db', 'mcp', 'merge-driver', 'pr', 'token']); }); // `--json` is axiom 4: every command and every error answers in machine-readable form, so @@ -177,7 +178,7 @@ describe('unit · parseArgs', () => { expect(() => parseArgs(['verfy'], SPECS)).toThrow(UnknownCommandError); const failure = thrownBy(() => parseArgs(['verfy'], SPECS)); expect(failure.code).toBe('X_CLI_UNKNOWN_COMMAND'); - expect(failure.fix).toBe('x verify'); + expect(failure.fix).toStartWith('x verify --help'); }); test('an unknown subcommand names the command path, not just the token', () => { @@ -209,7 +210,7 @@ describe('unit · parseArgs', () => { ); expect(failure.fix).toBe('x help'); // and the near-miss arm still suggests. - expect(thrownBy(() => parseArgs(['verifyy'], SPECS)).fix).toBe('x verify'); + expect(thrownBy(() => parseArgs(['verifyy'], SPECS)).fix).toStartWith('x verify --help'); }); test('a subcommand that resembles nothing is refused with no suggestion either', () => { diff --git a/packages/cli/src/port-pair-free.test.ts b/packages/cli/src/port-pair-free.test.ts new file mode 100644 index 000000000..a8473caa6 --- /dev/null +++ b/packages/cli/src/port-pair-free.test.ts @@ -0,0 +1,37 @@ +// The port a refusal suggests: free for both roles `x dev` binds, or the fix fails a second time. + +import { describe, expect, test } from 'bun:test'; +import { PORT_RANGE } from './flag-number'; +import { freeDevPortPair } from './port-pair-free'; + +const freeExcept = + (...taken: readonly number[]) => + (port: number): boolean => + !taken.includes(port); + +describe('freeDevPortPair', () => { + test('nothing else taken: the next pair', async () => { + expect(await freeDevPortPair(3000, freeExcept(3000, 3001))).toBe(3002); + }); + + test('walks past a pair whose web port is taken', async () => { + expect(await freeDevPortPair(3000, freeExcept(3000, 3001, 3002, 3003))).toBe(3004); + }); + + test('walks past a pair where only the sync port is taken', async () => { + expect(await freeDevPortPair(3000, freeExcept(3000, 3003))).toBe(3004); + }); + + test('accepts an async probe', async () => { + expect(await freeDevPortPair(3000, async (port) => port >= 3006)).toBe(3006); + }); + + test('at the top of the range it walks down, and every answer has a sync port', async () => { + const top = PORT_RANGE.max - 2; + expect(await freeDevPortPair(top, freeExcept(top - 2))).toBe(top - 4); + }); + + test('a machine with nothing free keeps the plain suggestion', async () => { + expect(await freeDevPortPair(3000, () => false)).toBe(3002); + }); +}); diff --git a/packages/cli/src/port-pair-free.ts b/packages/cli/src/port-pair-free.ts new file mode 100644 index 000000000..b2ec00779 --- /dev/null +++ b/packages/cli/src/port-pair-free.ts @@ -0,0 +1,28 @@ +// The port a "port in use" fix names. On a machine running several dev servers the next pair is +// often taken as well, and a fix that fails the same way is not a fix. Not `@ultimat3/testing`'s +// `freePortPair`, which binds; this one only asks. + +import { PORT_RANGE, portPairAfter } from './flag-number'; + +/** How far to look: past this the machine is unusual and the plain suggestion is as good. */ +const PAIRS_TRIED = 64; + +/** + * The first pair after `port` with both ports free, walking the way `portPairAfter` points (down at + * the top of the range). `x dev --port N` binds N and N + 1, so both are asked. The answer is true + * when asked: another process can still take it. + */ +export async function freeDevPortPair( + port: number, + isFree: (port: number) => boolean | Promise, +): Promise { + const first = portPairAfter(port); + const step = first > port ? 2 : -2; + let candidate = first; + for (let tried = 0; tried < PAIRS_TRIED; tried += 1) { + if (candidate < 1 || candidate + 1 >= PORT_RANGE.max) break; + if ((await isFree(candidate)) && (await isFree(candidate + 1))) return candidate; + candidate += step; + } + return first; +} diff --git a/packages/cli/src/registry-lines.test.ts b/packages/cli/src/registry-lines.test.ts new file mode 100644 index 000000000..691510c59 --- /dev/null +++ b/packages/cli/src/registry-lines.test.ts @@ -0,0 +1,120 @@ +// The registries' one-entry-per-line shape: what `x new` writes, what the codemod leaves, and that +// a generator's next edit lands as one added line — the property the merge driver depends on. + +import { describe, expect, test } from 'bun:test'; +import { insertApiEntries } from './api-registration'; +import { planNewApp } from './cmd-new'; +import { editExerciseFile } from './exercise-entry-edit'; +import { insertGrants } from './generate-grants'; +import { insertHandleEntries } from './handle-registration'; +import { NEW_SHAPES } from './new-shapes'; +import { REGISTRY_FILES, REGISTRY_LINE_COMMENT, withRegistryLines } from './registry-lines'; + +const API = 'apps/web/api/index.ts'; +const ROLES = 'apps/web/shared/roles.ts'; +const EXERCISE = 'apps/web/exercise.ts'; +const CLIENT = 'packages/db/src/client.ts'; + +const planned = (path: string, example = true): string => { + const file = planNewApp({ name: 'demo', example }).find((one) => one.path === path); + if (typeof file?.contents !== 'string') expect.unreachable(`x new writes no ${path}`); + return file.contents; +}; + +/** The lines `after` has that `before` lacks; fails when `before` lost or changed any line. */ +const addedLines = (before: string, after: string): readonly string[] => { + const rest = after.split('\n'); + for (const line of before.split('\n')) { + const at = rest.indexOf(line); + if (at === -1) expect.unreachable(`the edit changed an existing line: ${line}`); + rest.splice(at, 1); + } + return rest; +}; + +describe('unit · x new writes every registry one entry per line', () => { + test('each list of the API index is expanded, and tasks exists before anyone adds one', () => { + const source = planned(API); + expect(source).toContain(` actions: [\n ${REGISTRY_LINE_COMMENT}\n health,\n`); + expect(source).toContain(` tasks: [\n ${REGISTRY_LINE_COMMENT}\n ],`); + expect(source).not.toContain('actions: [health'); + }); + + test("a role's short grants list is expanded too", () => { + expect(planned(ROLES)).toContain( + `grants: [\n ${REGISTRY_LINE_COMMENT}\n 'dashboard:read',\n 'post:read',\n ],`, + ); + }); + + test('the bare app opens its two empty object registries', () => { + expect(planned(EXERCISE, false)).toContain(`defineExercise({\n ${REGISTRY_LINE_COMMENT}\n});`); + expect(planned(CLIENT, false)).toContain(`const entities = {\n ${REGISTRY_LINE_COMMENT}\n};`); + }); + + test('every shape still applies its edits and ends in the same shape', () => { + for (const shape of NEW_SHAPES) { + const file = planNewApp({ name: 'demo', example: true, shape }).find((f) => f.path === API); + expect(String(file?.contents)).toContain(`actions: [\n ${REGISTRY_LINE_COMMENT}`); + } + }); +}); + +describe('unit · a generator edit is one added line', () => { + test('defineApi: the import and the list entry, nothing rewritten', () => { + const before = planned(API); + const { source } = insertApiEntries(before, [ + { key: 'jobs', binding: 'reindexAlpha', specifier: '../app/alpha/jobs/reindex-alpha' }, + ]); + expect([...addedLines(before, source)].sort()).toEqual([ + ' reindexAlpha,', + "import * as reindexAlpha from '../app/alpha/jobs/reindex-alpha';", + ]); + }); + + test('roles: a grant into a list that used to fit on one line', () => { + const before = planned(ROLES); + const { source } = insertGrants(before, [{ role: 'member', permission: 'alpha:read' }]); + expect(addedLines(before, source)).toEqual([" 'alpha:read',"]); + }); + + test('the handle and the exercise file gain rows inside their opened empty object', () => { + const client = planned(CLIENT, false); + const entry = { key: 'alphas', binding: 'alpha', specifier: '@demo/web/app/alpha/entity' }; + expect([...addedLines(client, insertHandleEntries(client, [entry]).source)].sort()).toEqual([ + ' alphas: alpha,', + "import { alpha } from '@demo/web/app/alpha/entity';", + ]); + const exercise = planned(EXERCISE, false); + const edit = editExerciseFile(exercise, [{ name: 'createAlpha', value: "{ off: 'why' }" }]); + expect(edit.kind === 'edited' && addedLines(exercise, edit.source)).toEqual([ + " createAlpha: { off: 'why' },", + ]); + }); +}); + +describe('unit · withRegistryLines over an existing app', () => { + test('is idempotent on every registry', () => { + for (const path of REGISTRY_FILES) { + const once = planned(path); + expect(withRegistryLines(path, once)).toBe(once); + } + }); + + test('a list already over several lines only gains the comment', () => { + const source = `const r = defineRoles({\n admin: {\n grants: [\n 'a:b', // kept\n 'c:d',\n ],\n },\n});\n`; + expect(withRegistryLines(ROLES, source)).toBe( + `const r = defineRoles({\n admin: {\n grants: [\n ${REGISTRY_LINE_COMMENT}\n 'a:b', // kept\n 'c:d',\n ],\n },\n});\n`, + ); + }); + + test("a one-line list holding the author's comment is left alone", () => { + const source = `const r = defineRoles({\n admin: { grants: ['a:b' /* why */] },\n});\n`; + expect(withRegistryLines(ROLES, source)).toBe(source); + }); + + test('any other file is untouched', () => { + expect(withRegistryLines('apps/web/other.ts', 'const a = [1, 2];\n')).toBe( + 'const a = [1, 2];\n', + ); + }); +}); diff --git a/packages/cli/src/registry-lines.ts b/packages/cli/src/registry-lines.ts new file mode 100644 index 000000000..71edea0d1 --- /dev/null +++ b/packages/cli/src/registry-lines.ts @@ -0,0 +1,110 @@ +// The four registries every generated feature adds to, held one entry per line. Two branches that +// each add a feature then touch different lines, so `x merge-driver` can keep both; a list on one +// line is the same line changed twice, which no merge can tell from a real disagreement. `x new` +// writes this shape and the `registry-lines` codemod carries an existing app to it. + +import { stripComments } from '@ultimat3/core'; +import { apiListsOf, insertApiList } from './api-registration'; +import { API_INDEX } from './app-root'; +import { declarationLists } from './generate-grants'; +import { ROLES_FILE } from './permission-grants'; +import { maskOf, readList, readProperties } from './source-list-edit'; +import { HANDLE_FILE } from './templates/scaffold-db-client'; + +const EXERCISE_FILE = 'apps/web/exercise.ts'; + +/** Inside the bracket, it is also what keeps Biome from joining the list back onto one line. */ +export const REGISTRY_LINE_COMMENT = + '// One entry per line: branches that each add one merge cleanly (x merge-driver).'; + +/** Every file `withRegistryLines` holds to the shape. */ +export const REGISTRY_FILES: readonly string[] = [ + API_INDEX, + ROLES_FILE, + EXERCISE_FILE, + HANDLE_FILE, +]; + +const API_LISTS = ['actions', 'queries', 'jobs', 'tasks'] as const; + +const indentAt = (source: string, at: number): string => + /^[ \t]*/.exec(source.slice(source.lastIndexOf('\n', at) + 1))?.[0] ?? ''; + +const startsItsLine = (source: string, at: number): boolean => + source.slice(source.lastIndexOf('\n', at - 1) + 1, at).trim() === ''; + +/** + * The list opening at `open`, a row per entry under the comment. A list this cannot restate + * without moving the author's text — a comment beside entries on one line, two entries sharing a + * row — is left as it is: it still works, it just merges by hand. + */ +export function listWithLines(source: string, open: number): string { + const list = readList(source, maskOf(source), open); + if (list === undefined) return source; + const span = source.slice(list.open, list.close + 1); + if (span.includes(REGISTRY_LINE_COMMENT)) return source; + const indent = indentAt(source, open); + const comment = `${indent} ${REGISTRY_LINE_COMMENT}`; + if (span.includes('\n')) { + const lineEnd = source.indexOf('\n', open); + const opensAlone = source.slice(open + 1, lineEnd).trim() === ''; + if (!opensAlone || list.entries.some((entry) => !startsItsLine(source, entry.start))) { + return source; + } + return `${source.slice(0, lineEnd + 1)}${comment}\n${source.slice(lineEnd + 1)}`; + } + if (stripComments(source).slice(list.open, list.close + 1) !== span) return source; + const rows = list.entries.map((entry) => `${indent} ${entry.text},\n`).join(''); + const bracket = source[list.close] as string; + const body = `${source[open] as string}\n${comment}\n${rows}${indent}${bracket}`; + return `${source.slice(0, open)}${body}${source.slice(list.close + 1)}`; +} + +function apiWithLines(source: string): string { + let next = source; + for (const key of API_LISTS) { + // A list every branch can add to exists before either does: two branches each creating + // `tasks:` is one key written twice, the one shape no line merge can keep. + next = apiListsOf(next)?.(key) === undefined ? (insertApiList(next, key, '[]') ?? next) : next; + const list = apiListsOf(next)?.(key); + if (list !== undefined) next = listWithLines(next, list.open); + } + return next; +} + +/** Applied from the last list back, so an edit never moves an offset still to be used. */ +const eachFromLast = (source: string, opens: readonly number[]): string => + [...opens].sort((a, b) => b - a).reduce(listWithLines, source); + +function grantLists(source: string): readonly number[] { + const masked = maskOf(source); + const call = /\bdefineRoles\(\s*\{/.exec(masked); + if (call === null) return []; + return readProperties(source, masked, call.index + call[0].length - 1).flatMap((role) => { + if (masked[role.value] !== '{') return []; + const grants = readProperties(source, masked, role.value).find((p) => p.key === 'grants'); + return grants !== undefined && masked[grants.value] === '[' ? [grants.value] : []; + }); +} + +const rolesWithLines = (source: string): string => + eachFromLast(source, [...declarationLists(maskOf(source)), ...grantLists(source)]); + +/** An EMPTY object only: a filled one is already a row per entry, which Biome keeps. */ +function emptyObjectWithLines(source: string, opening: RegExp): string { + const masked = maskOf(source); + const found = opening.exec(masked); + if (found === null) return source; + const open = found.index + found[0].length - 1; + const list = readList(source, masked, open); + return list?.entries.length === 0 ? listWithLines(source, open) : source; +} + +/** `source`, the registry at `path`, in the one-entry-per-line shape. Any other path: unchanged. */ +export function withRegistryLines(path: string, source: string): string { + if (path === API_INDEX) return apiWithLines(source); + if (path === ROLES_FILE) return rolesWithLines(source); + if (path === EXERCISE_FILE) return emptyObjectWithLines(source, /\bdefineExercise\(\s*\{/); + if (path === HANDLE_FILE) return emptyObjectWithLines(source, /\bconst entities\s*=\s*\{/); + return source; +} diff --git a/packages/cli/src/registry.ts b/packages/cli/src/registry.ts index 360c2b81e..62e5f9517 100644 --- a/packages/cli/src/registry.ts +++ b/packages/cli/src/registry.ts @@ -40,6 +40,7 @@ import { logsSpec } from './cmd-logs-spec'; import { mailSpec } from './cmd-mail-spec'; import { manifestSpec } from './cmd-manifest-spec'; import { mcpSpec } from './cmd-mcp-spec'; +import { mergeDriverSpec } from './cmd-merge-driver-spec'; import { newSpec } from './cmd-new-spec'; import { policySpec } from './cmd-policy-spec'; import { prSpec } from './cmd-pr-spec'; @@ -133,6 +134,7 @@ export const LAZY_COMMANDS: readonly LazyCommand[] = [ lazy(graphSpec, async () => (await import('./cmd-graph')).graphCommand), lazy(impactSpec, async () => (await import('./cmd-impact')).impactCommand), lazy(tokenSpec, async () => (await import('./cmd-token')).tokenCommand), + lazy(mergeDriverSpec, async () => (await import('./cmd-merge-driver')).mergeDriverCommand), lazy(configSpec, async () => (await import('./cmd-config')).configCommand), lazy(logsSpec, async () => (await import('./cmd-logs')).logsCommand), lazy(cacheSpec, async () => (await import('./cmd-cache')).cacheCommand), diff --git a/packages/cli/src/retain-sweep.ts b/packages/cli/src/retain-sweep.ts index 891e047b7..493af4d24 100644 --- a/packages/cli/src/retain-sweep.ts +++ b/packages/cli/src/retain-sweep.ts @@ -11,7 +11,7 @@ export function retainUnsweptFinding(entity: string): Finding { return { code: 'X_ENTITY_RETAIN_UNSWEPT', cause: `entity "${entity}" declares retain, and no purge() registered in this app sweeps it — rows past its window are kept forever`, - fix: "export const retention = purge({ entities: 'declared' }) — in a jobs module named in jobs: [...] in apps/web/api/index.ts, then x verify", + fix: "export const retention = purge({ entities: 'declared' }) — in a jobs module named in jobs: [...] in apps/web/api/index.ts, then x manifest", at: 'apps/web/api/index.ts', }; } diff --git a/packages/cli/src/role-sync.test.ts b/packages/cli/src/role-sync.test.ts index db207a62c..8ea799c3f 100644 --- a/packages/cli/src/role-sync.test.ts +++ b/packages/cli/src/role-sync.test.ts @@ -76,6 +76,14 @@ describe('unit · the sync node cannot bind, and says which port', () => { expect(fixProblem(refusal?.fix ?? '')).toBeUndefined(); }); + // Beside two dev servers the next pair was often the second one's: the fix walks to a pair that + // is free when asked, as `x doctor`'s does (`freeDevPortPair`). + test('the suggested pair skips a pair another server holds', async () => { + const taken = new Set([4000, 4001, 4002]); + const refusal = await syncBindRefusal(3999, 4000, async (port) => !taken.has(port)); + expect(refusal?.fix).toContain('x dev --port 4003'); + }); + test('the suggested pair clears the taken port at the top of the range too', async () => { const webPort = PORT_RANGE.max - 2; const refusal = await syncBindRefusal(webPort, PORT_RANGE.max - 1, async () => false); diff --git a/packages/cli/src/role-sync.ts b/packages/cli/src/role-sync.ts index 87f95fbdc..a4fd03b1b 100644 --- a/packages/cli/src/role-sync.ts +++ b/packages/cli/src/role-sync.ts @@ -23,7 +23,8 @@ import { SocketRegistry, syncNode, } from '@ultimat3/realtime/server'; -import { portPairAfter, syncPortFor } from './flag-number'; +import { syncPortFor } from './flag-number'; +import { freeDevPortPair } from './port-pair-free'; import { portFree } from './port-probe'; import type { StartRolesOptions } from './role-start'; import { startOrWaitForBus } from './role-sync-bus-wait'; @@ -48,15 +49,15 @@ import { DEV_BINDING } from './web-binding'; * "no findings — environment is shippable" when run (#F5). */ class SyncPortInUseError extends UltimateError { - constructor(input: { port: number; webPort: number }) { + constructor(input: { port: number; webPort: number; suggested: number }) { super({ code: 'X_PORT_IN_USE', cause: `the sync role binds PORT + 1, so \`x dev --port ${input.webPort}\` needs port ${input.port} and something is already listening on it`, - // `portPairAfter`, never `neighbouringPort`: `x dev` binds a PAIR, so the neighbour of the + // `freeDevPortPair`, never `neighbouringPort`: `x dev` binds a PAIR, so the neighbour of the // web port IS the sync port this refusal is about — the fix said `x dev --port 4000` for a // run that had just died on 4000, and a test named "its fix is a command that ends the // failure" pinned it. - fix: `x dev --port ${quoteArg(String(portPairAfter(input.webPort)))} # or free port ${input.port}: lsof -nP -iTCP:${input.port} -sTCP:LISTEN`, + fix: `x dev --port ${quoteArg(String(input.suggested))} # or free port ${input.port}: lsof -nP -iTCP:${input.port} -sTCP:LISTEN`, meta: { port: input.port, webPort: input.webPort }, }); } @@ -77,7 +78,11 @@ export async function syncBindRefusal( probe: (value: number) => Promise = portFree, ): Promise { if (await probe(port)) return undefined; - return new SyncPortInUseError({ port, webPort }); + return new SyncPortInUseError({ + port, + webPort, + suggested: await freeDevPortPair(webPort, probe), + }); } // Its rule lives in `flag-number.ts`, where the scrape port's reservation reads it too (`rolePorts`). diff --git a/packages/cli/src/schema-drift.ts b/packages/cli/src/schema-drift.ts index 98f2a9443..92d96353e 100644 --- a/packages/cli/src/schema-drift.ts +++ b/packages/cli/src/schema-drift.ts @@ -15,7 +15,14 @@ import { existsSync } from 'node:fs'; import { join } from 'node:path'; import { ERROR_DOCS_URL } from '@ultimat3/core'; import type { EntityDescriptionLike, UnrenderedDeclaration } from '@ultimat3/db'; -import { declaredSchema, recordRetired, snapshotOf, unrenderedOf } from '@ultimat3/db'; +import { + declaredSchema, + migrationForked, + recordRetired, + snapshotOf, + unrenderedOf, +} from '@ultimat3/db'; +import { migrationFork } from '@ultimat3/db/schema-dump'; import { registeredEntities } from '@ultimat3/entity'; import { loadApp } from './app-load'; import { checkSourceDrift, DB_PACKAGE } from './drift'; @@ -26,6 +33,7 @@ import { } from './drift-replica-identity'; import { MIGRATIONS_DIR, readMigrations } from './migrations'; import type { Finding } from './output'; +import { findingFrom } from './output'; import { settledDescriptions } from './redundant-indexes'; import { appRowSecurity, type RowSecurityWanted, rowSecurityFindings } from './row-security-wanted'; import { diffDeclaredSchema, type SchemaDifference } from './schema-diff'; @@ -137,6 +145,8 @@ function findingFor(difference: SchemaDifference, repair: Repair): Finding { * One finding per difference, never one per table: each names one declaration and one edit, and a * grouped finding would hand a reader nine constraint names under one instruction. * + * A forked chain answers first and alone (`X_MIGRATION_FORKED`). + * * Three conditions are deliberately left to their own reporters, because each already has a code * and a remedy: an app with no `packages/db`, an app whose first migration has not been generated * (`checkSourceDrift`, `x db gen "initial"`), and a newest migration carrying no sidecar @@ -149,9 +159,16 @@ export async function checkSnapshotDrift( rowSecurity: RowSecurityWanted = appRowSecurity, ): Promise { if (!existsSync(join(root, DB_PACKAGE))) return []; + const migrations = await readMigrations(root); + // Ahead of the entities, and the only finding: the newest sidecar is the thing a fork makes wrong, + // so every difference below would be measured against it and `x db gen 'record …'` would write + // the second `create table` a replay refuses. It needs no entity, so a load failure hides nothing. + const fork = migrationFork(migrations); + if (fork !== undefined) { + return [{ ...findingFrom(migrationForked(fork)), docs: ERROR_DOCS_URL, at: MIGRATIONS_DIR }]; + } const entities = await declared(root); if (entities === undefined) return []; - const migrations = await readMigrations(root); if (migrations.length === 0) return []; const recorded = declaredSchema(migrations); if (recorded === undefined) return []; diff --git a/packages/cli/src/schema-dump-drift.ts b/packages/cli/src/schema-dump-drift.ts index 90e4acb95..97d215ef5 100644 --- a/packages/cli/src/schema-dump-drift.ts +++ b/packages/cli/src/schema-dump-drift.ts @@ -11,7 +11,7 @@ import { ERROR_DOCS_URL } from '@ultimat3/core'; import type { Migration } from '@ultimat3/db'; import { ungeneratableStatements } from '@ultimat3/db'; import type { SchemaDumpDifference, SchemaDumpFile } from '@ultimat3/db/schema-dump'; -import { compareSchemaDump, schemaDumpDrift } from '@ultimat3/db/schema-dump'; +import { compareSchemaDump, migrationFork, schemaDumpDrift } from '@ultimat3/db/schema-dump'; import { hasSchemaDump, type ReplayedSchema, @@ -128,6 +128,9 @@ export async function checkSchemaDump( ): Promise { if (!existsSync(join(root, DB_PACKAGE))) return []; const migrations = await readMigrations(root); + // `X_MIGRATION_FORKED` owns this chain, in the same step. A replay of it may fail, and that + // finding's `x db migrate` fix would only lead back to the refusal. + if (migrationFork(migrations) !== undefined) return []; if (!hasSchemaDump(root)) return migrations.length === 0 ? [] : [dumpNeverGenerated()]; const committed = await readSchemaDump(root); const columns = dumpColumnFindings(migrations, committed); diff --git a/packages/cli/src/serve-graph-serving-ceiling-fixture.ts b/packages/cli/src/serve-graph-serving-ceiling-fixture.ts index c90eeff2e..23d279388 100644 --- a/packages/cli/src/serve-graph-serving-ceiling-fixture.ts +++ b/packages/cli/src/serve-graph-serving-ceiling-fixture.ts @@ -336,5 +336,8 @@ * `time/src/plain-date.ts`, `db/src/not-null-contract.ts`. * raised 1545 → 1547, measured 1547 (2026-11-18, a queued write's binding): `core/src/client-scope-id.ts` * and `action/src/replay-scope.ts`, the scope id and the check that compares it. + * raised 1547 → 1548, measured 1548 (2026-11-19, multi-agent DX): `cli/src/port-pair-free.ts`, so + * the sync role's `X_PORT_IN_USE` names a pair that is free when asked, as `x doctor`'s fix does. + * The fork check that also arrived (three modules) went to `@ultimat3/db/schema-dump` instead. */ -export const SERVING_ROLE_CEILING = 1547; +export const SERVING_ROLE_CEILING = 1548; diff --git a/packages/cli/src/serve-graph-web-ceiling-fixture.ts b/packages/cli/src/serve-graph-web-ceiling-fixture.ts index da200cfb3..6b5ffbd32 100644 --- a/packages/cli/src/serve-graph-web-ceiling-fixture.ts +++ b/packages/cli/src/serve-graph-web-ceiling-fixture.ts @@ -297,5 +297,7 @@ * `SERVING_ROLE_CEILING`. Nothing web-only. * raised 1745 → 1747, measured 1747 (2026-11-18, a queued write's binding): `core/src/client-scope-id.ts` * and `action/src/replay-scope.ts`, the scope id and the check that compares it. + * raised 1747 → 1748, measured 1748 (2026-11-19, multi-agent DX): the one on + * `SERVING_ROLE_CEILING`, `cli/src/port-pair-free.ts`. */ -export const WEB_ROLE_CEILING = 1747; +export const WEB_ROLE_CEILING = 1748; diff --git a/packages/cli/src/serve-graph.test.ts b/packages/cli/src/serve-graph.test.ts index fbbef1741..ad216f29a 100644 --- a/packages/cli/src/serve-graph.test.ts +++ b/packages/cli/src/serve-graph.test.ts @@ -56,6 +56,9 @@ const NEVER_AT_BOOT: readonly (readonly [string, RegExp])[] = [ 'the schema-dump family', /packages\/db\/src\/(?:catalog(?:-fold|-objects|-relations)?|dump-drift|introspect-catalog|object-drift|schema-dump(?:-table|-entry)?|schema-load)\.ts$/, ], + // The fork check `x db gen`, `x db migrate` and `drift` run over the migrations folder: three + // modules that landed on the barrel and raised every role's count by three. + ['the migration-fork family', /packages\/db\/src\/migration-fork(?:-fold|-sql)?\.ts$/], // Five modules only `x db diagnostics` calls, on `@ultimat3/db/diagnostics` from the day they // landed: on the barrel every role evaluated them to serve nothing. [ diff --git a/packages/cli/src/templates/guard-animated-layout-property.ts b/packages/cli/src/templates/guard-animated-layout-property.ts index 0b63f50f4..61a82f982 100644 --- a/packages/cli/src/templates/guard-animated-layout-property.ts +++ b/packages/cli/src/templates/guard-animated-layout-property.ts @@ -162,7 +162,7 @@ export function animatedLayoutProperties(files: readonly StyleFile[]): readonly findings.push({ code: CODE, cause: \`\${at} transitions every property of this element — \${animated === undefined ? \`\\\`\${property}: \${value}\\\` names none, and an omitted property is \\\`all\\\`\` : 'written out as \`all\`'} — so it animates properties nobody chose, the ones that cost a layout pass included, and no part of it can be composited\`, - fix: \`name the properties in \${at} — \\\`transition: transform tokens.duration('fast') tokens.easing('out')\\\` — then: x verify\`, + fix: \`name the properties in \${at} — \\\`transition: transform tokens.duration('fast') tokens.easing('out')\\\` — then: x verify --only boundaries\`, at: file.path, }); continue; @@ -180,7 +180,7 @@ function layoutFinding(path: string, at: string, property: string, where: string return { code: CODE, cause: \`\${at} animates \${property} in \${where} — only transform and opacity are compositor-only, so every frame of this costs a full layout or paint pass on the same main thread the app runs on, and a moving box feeds the layout-shift metric while it does it\`, - fix: \`animate \${instead} instead of \${property} at \${at}, then: x verify\`, + fix: \`animate \${instead} instead of \${property} at \${at}, then: x verify --only boundaries\`, at: path, }; } diff --git a/packages/cli/src/templates/guard-bare-error.ts b/packages/cli/src/templates/guard-bare-error.ts index 5bd866b57..a145f3721 100644 --- a/packages/cli/src/templates/guard-bare-error.ts +++ b/packages/cli/src/templates/guard-bare-error.ts @@ -59,7 +59,7 @@ export function bareThrows(files: readonly SourceFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${file.path}:\${line} throws a bare \${thrown} — it reaches its reader as a stack trace with no code, no cause and nothing to run\`, - fix: \`declare the failure in \${file.path} with defineError({ code, title, status, cause, fix }) from @ultimat3/core/page — an X_SCREAMING_SNAKE code and a fix naming a command — and throw its class, then: x verify\`, + fix: \`declare the failure in \${file.path} with defineError({ code, title, status, cause, fix }) from @ultimat3/core/page — an X_SCREAMING_SNAKE code and a fix naming a command — and throw its class, then: x verify --only boundaries\`, at: file.path, }); } diff --git a/packages/cli/src/templates/guard-error-declared.ts b/packages/cli/src/templates/guard-error-declared.ts index 156e48ada..e9a95c8e3 100644 --- a/packages/cli/src/templates/guard-error-declared.ts +++ b/packages/cli/src/templates/guard-error-declared.ts @@ -146,7 +146,7 @@ const finding = (file: SourceFile, line: number, what: string, declared: Declare return { code: CODE, cause: \`\${file.path}:\${line} \${what} — the title, the HTTP status and the class are then declared apart, and a code with no status answers 500 and hides its cause\`, - fix: \`rewrite it in \${file.path} as one call per class, from @ultimat3/core/page: export const \${name} = defineError({ code: \${code}, title, status, cause, fix, problemMeta: [''] }) — it replaces the class and its registerErrorStatus / registerProblemMeta rows (offSocket: true instead of status for a code no response carries; publicCause: true for a 5xx whose cause a caller may read; drop problemMeta when its meta stays operator-only), then: x verify\`, + fix: \`rewrite it in \${file.path} as one call per class, from @ultimat3/core/page: export const \${name} = defineError({ code: \${code}, title, status, cause, fix, problemMeta: [''] }) — it replaces the class and its registerErrorStatus / registerProblemMeta rows (offSocket: true instead of status for a code no response carries; publicCause: true for a 5xx whose cause a caller may read; drop problemMeta when its meta stays operator-only), then: x verify --only boundaries\`, at: file.path, }; }; @@ -154,7 +154,7 @@ const finding = (file: SourceFile, line: number, what: string, declared: Declare const barrelFinding = (file: SourceFile, line: number): Finding => ({ code: CODE, cause: \`\${file.path}:\${line} takes defineError from the @ultimat3/core barrel — every island that throws the code then carries the framework's titles table, ~5.5 kB it never reads\`, - fix: \`import { defineError } from '@ultimat3/core/page' in \${file.path} — the same function, without the table — then: x verify\`, + fix: \`import { defineError } from '@ultimat3/core/page' in \${file.path} — the same function, without the table — then: x verify --only boundaries\`, at: file.path, }); diff --git a/packages/cli/src/templates/guard-focus-visible.ts b/packages/cli/src/templates/guard-focus-visible.ts index 01085b77d..3756ac5ab 100644 --- a/packages/cli/src/templates/guard-focus-visible.ts +++ b/packages/cli/src/templates/guard-focus-visible.ts @@ -153,7 +153,7 @@ export function unreplacedFocusRings(files: readonly StyleFile[]): readonly Find findings.push({ code: CODE, cause: \`\${file.path}:\${lineOf(scss, match.index)} removes the focus ring with \${property}: \${value} and nothing in the rule or beside it paints one back — a keyboard user loses every trace of where they are, and WCAG 2.2 asks for an indicator at least 2px around the control at 3:1 against what is behind it\`, - fix: \`add \\\`@include tokens.focus-ring;\\\` to the rule at \${file.path}:\${lineOf(scss, match.index)}, or a sibling \\\`:focus-visible\\\` rule with a box-shadow — which follows the border radius where an outline does not — then: x verify\`, + fix: \`add \\\`@include tokens.focus-ring;\\\` to the rule at \${file.path}:\${lineOf(scss, match.index)}, or a sibling \\\`:focus-visible\\\` rule with a box-shadow — which follows the border radius where an outline does not — then: x verify --only boundaries\`, at: file.path, }); } diff --git a/packages/cli/src/templates/guard-image-dimensions.ts b/packages/cli/src/templates/guard-image-dimensions.ts index 747760907..79f7efdb8 100644 --- a/packages/cli/src/templates/guard-image-dimensions.ts +++ b/packages/cli/src/templates/guard-image-dimensions.ts @@ -124,7 +124,7 @@ export function unsizedImages(files: readonly SourceFile[]): readonly Finding[] findings.push({ code: CODE, cause: \`\${at} renders <\${tag.name}> with no width and height pair and no aspect-ratio — the browser reserves no box for it, so every element under it jumps the moment the bytes land\`, - fix: \`add width and height to the <\${tag.name}> at \${at} — the intrinsic pixel size, since CSS still decides what it is drawn at — or an aspect-ratio, then: x verify\`, + fix: \`add width and height to the <\${tag.name}> at \${at} — the intrinsic pixel size, since CSS still decides what it is drawn at — or an aspect-ratio, then: x verify --only boundaries\`, at: file.path, }); } @@ -134,7 +134,7 @@ export function unsizedImages(files: readonly SourceFile[]): readonly Finding[] findings.push({ code: CODE, cause: \`\${at} marks <\${tag.name}> as the priority image AND loading="lazy" — a lazy image is fetched after layout, and this is the one the page's largest paint is measured on, so the attribute delays the metric it is the subject of\`, - fix: \`delete loading="lazy" from the <\${tag.name}> at \${at}, then: x verify\`, + fix: \`delete loading="lazy" from the <\${tag.name}> at \${at}, then: x verify --only boundaries\`, at: file.path, }); } diff --git a/packages/cli/src/templates/guard-island-mount-clears.ts b/packages/cli/src/templates/guard-island-mount-clears.ts index 3445b6676..86471dd8d 100644 --- a/packages/cli/src/templates/guard-island-mount-clears.ts +++ b/packages/cli/src/templates/guard-island-mount-clears.ts @@ -75,7 +75,7 @@ export function islandMountClears(files: readonly SourceFile[]): readonly Findin findings.push({ code: CODE, cause: \`\${at} empties the island's server markup outside mountOver, so a render that throws leaves a blank box where the server's HTML was, and the visitor sees nothing to retry\`, - fix: \`edit \${at} — delete the clear and end mount() with return mountOver(el, (root) => render(() => , root)), adding mountOver to the island's imports from '@ultimat3/render/client' — then: x verify\`, + fix: \`edit \${at} — delete the clear and end mount() with return mountOver(el, (root) => render(() => , root)), adding mountOver to the island's imports from '@ultimat3/render/client' — then: x verify --only boundaries\`, at, }); } diff --git a/packages/cli/src/templates/guard-native-confirm.ts b/packages/cli/src/templates/guard-native-confirm.ts index 713fccaa4..818ef4d1b 100644 --- a/packages/cli/src/templates/guard-native-confirm.ts +++ b/packages/cli/src/templates/guard-native-confirm.ts @@ -64,7 +64,7 @@ export function nativeConfirms(files: readonly SourceFile[]): readonly Finding[] findings.push({ code: CODE, cause: \`\${at} asks with confirm(), whose OK holds focus — a stray Enter confirms — and which cannot be translated or refused\`, - fix: \`replace the confirm() at \${at} with from '@ultimat3/ui' (action= for a native post, onConfirm= in an island) — or add a // why: comment on that line naming why it is not a destructive question — then: x verify\`, + fix: \`replace the confirm() at \${at} with from '@ultimat3/ui' (action= for a native post, onConfirm= in an island) — or add a // why: comment on that line naming why it is not a destructive question — then: x verify --only boundaries\`, at, }); } diff --git a/packages/cli/src/templates/guard-raw-breakpoint.ts b/packages/cli/src/templates/guard-raw-breakpoint.ts index c85124074..d3c2d3f8a 100644 --- a/packages/cli/src/templates/guard-raw-breakpoint.ts +++ b/packages/cli/src/templates/guard-raw-breakpoint.ts @@ -97,7 +97,7 @@ export function rawBreakpoints(files: readonly StyleFile[]): readonly Finding[] findings.push({ code: CODE, cause: \`\${file.path}:\${line} defines its own \${match[1]} — @ultimat3/ui/tokens ships one, and a local definition silently shadows it, so the app has two breakpoint ladders\`, - fix: \`delete @mixin \${match[1]} from \${file.path}:\${line} and write @include \${ns}.\${match[1]}(…), then: x verify\`, + fix: \`delete @mixin \${match[1]} from \${file.path}:\${line} and write @include \${ns}.\${match[1]}(…), then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } @@ -111,7 +111,7 @@ export function rawBreakpoints(files: readonly StyleFile[]): readonly Finding[] findings.push({ code: CODE, cause: \`\${file.path}:\${line} writes \\\`@media\${prelude.trimEnd()}\\\` — a viewport width of its own, off the breakpoint ladder every other sheet shares\`, - fix: \`\${include} — at \${file.path}:\${line}, then: x verify\`, + fix: \`\${include} — at \${file.path}:\${line}, then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } diff --git a/packages/cli/src/templates/guard-raw-colour.ts b/packages/cli/src/templates/guard-raw-colour.ts index 917067e50..4a6b4bcd4 100644 --- a/packages/cli/src/templates/guard-raw-colour.ts +++ b/packages/cli/src/templates/guard-raw-colour.ts @@ -156,7 +156,7 @@ export function rawColours(files: readonly StyleFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${file.path}:\${lineOf(scss, index)} sets \${property} to the raw colour \${literal} — a value no theme can restate, so dark theme renders it unchanged\`, - fix: \`replace \${literal} in \${file.path} with tokens.role('fg'), tokens.role('bg') or the role this element means, then: x verify\`, + fix: \`replace \${literal} in \${file.path} with tokens.role('fg'), tokens.role('bg') or the role this element means, then: x verify --only boundaries\`, at: file.path, }); } @@ -197,7 +197,7 @@ export function inlineColours( findings.push({ code: CODE, cause: \`\${file.path}:\${lineOf(text, attribute.index)} sets \${property} to the raw colour \${literal} in an inline style — a value no theme can restate, so dark theme renders it unchanged\`, - fix: \`replace \${literal} in \${file.path} with rgb(var(--color-fg)), rgb(var(--color-bg)) or the role this element means — or a class whose stylesheet sets tokens.role('fg') — then: x verify\`, + fix: \`replace \${literal} in \${file.path} with rgb(var(--color-fg)), rgb(var(--color-bg)) or the role this element means — or a class whose stylesheet sets tokens.role('fg') — then: x verify --only boundaries\`, at: file.path, }); } diff --git a/packages/cli/src/templates/guard-raw-fetch.ts b/packages/cli/src/templates/guard-raw-fetch.ts index 1e430ffbd..18e4e2fe4 100644 --- a/packages/cli/src/templates/guard-raw-fetch.ts +++ b/packages/cli/src/templates/guard-raw-fetch.ts @@ -92,7 +92,7 @@ export function rawFetches(files: readonly SourceFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${at} calls fetch with no signal in its own options, so the call outlives the request that made it: a stalled vendor holds the handler, the drain and a socket\`, - fix: \`call outbound({ url, vendor, dial: 'public', deadlineMs: 10_000 }) from @ultimat3/core/outbound in place of the fetch at \${at}, or pass { signal: ctx.signal } in its options — then: x verify\`, + fix: \`call outbound({ url, vendor, dial: 'public', deadlineMs: 10_000 }) from @ultimat3/core/outbound in place of the fetch at \${at}, or pass { signal: ctx.signal } in its options — then: x verify --only boundaries\`, at, }); } diff --git a/packages/cli/src/templates/guard-raw-interval.ts b/packages/cli/src/templates/guard-raw-interval.ts index 81569f4aa..89fa0f457 100644 --- a/packages/cli/src/templates/guard-raw-interval.ts +++ b/packages/cli/src/templates/guard-raw-interval.ts @@ -65,7 +65,7 @@ export function rawIntervals(files: readonly SourceFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${at} polls with setInterval, which keeps asking from a hidden tab and asks for a 401 forever once the session is gone\`, - fix: \`re-read a query with useQuery(ref, input, { poll: pollEvery(ms) }) (pollEvery is exported by @ultimat3/realtime) at \${at}; any other read: import { visibleInterval } from '@ultimat3/core/page' and call visibleInterval(run, ms) there, disposing it on cleanup — or add a // why: comment on that line naming why this interval is not a read — then: x verify\`, + fix: \`re-read a query with useQuery(ref, input, { poll: pollEvery(ms) }) (pollEvery is exported by @ultimat3/realtime) at \${at}; any other read: import { visibleInterval } from '@ultimat3/core/page' and call visibleInterval(run, ms) there, disposing it on cleanup — or add a // why: comment on that line naming why this interval is not a read — then: x verify --only boundaries\`, at, }); } diff --git a/packages/cli/src/templates/guard-raw-length.ts b/packages/cli/src/templates/guard-raw-length.ts index 22f931655..b6b667b3f 100644 --- a/packages/cli/src/templates/guard-raw-length.ts +++ b/packages/cli/src/templates/guard-raw-length.ts @@ -164,7 +164,7 @@ export function rawLengths(files: readonly StyleFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${file.path}:\${line} defines its own \${match[1]}() — @ultimat3/ui/tokens ships one, and a local definition silently shadows it, so two sheets can disagree about what a px is\`, - fix: \`delete @function \${match[1]} from \${file.path}:\${line} and call \${ns}.\${match[1]}(), then: x verify\`, + fix: \`delete @function \${match[1]} from \${file.path}:\${line} and call \${ns}.\${match[1]}(), then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } @@ -196,7 +196,7 @@ export function rawLengths(files: readonly StyleFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${file.path}:\${line} writes \\\`\${statement}\\\` — a px length off the scale: it does not grow with the reader's text size, and no token names it\`, - fix: \`\${rewritten} — at \${file.path}:\${line}, then: x verify\`, + fix: \`\${rewritten} — at \${file.path}:\${line}, then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } @@ -223,7 +223,7 @@ export function rawInlineLengths(files: readonly MarkupFile[]): readonly Finding findings.push({ code: CODE, cause: \`\${file.path}:\${line} sets \${px}px in an inline style — a length off the scale, in the one stylesheet nothing else reads\`, - fix: \`\${written} in place of \${px}px — at \${file.path}:\${line}; better, move the declaration into the element's .module.scss, then: x verify\`, + fix: \`\${written} in place of \${px}px — at \${file.path}:\${line}; better, move the declaration into the element's .module.scss, then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } diff --git a/packages/cli/src/templates/guard-raw-motion.ts b/packages/cli/src/templates/guard-raw-motion.ts index e0cff11bc..3bafc1096 100644 --- a/packages/cli/src/templates/guard-raw-motion.ts +++ b/packages/cli/src/templates/guard-raw-motion.ts @@ -127,7 +127,7 @@ export function rawMotion(files: readonly StyleFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${file.path}:\${line} writes \\\`\${property}: \${value}\\\` — \${literals.join(' and ')} off the motion scale, a tempo of this file's own\`, - fix: \`\${property}: \${written} — at \${file.path}:\${line} (durations: \${DURATIONS.map(([name]) => name).join(' ')}; a longer one is calc(#{\${ns}.duration(slower)} * 2)), then: x verify\`, + fix: \`\${property}: \${written} — at \${file.path}:\${line} (durations: \${DURATIONS.map(([name]) => name).join(' ')}; a longer one is calc(#{\${ns}.duration(slower)} * 2)), then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } diff --git a/packages/cli/src/templates/guard-raw-shadow.ts b/packages/cli/src/templates/guard-raw-shadow.ts index 1365735e8..b0fe4606b 100644 --- a/packages/cli/src/templates/guard-raw-shadow.ts +++ b/packages/cli/src/templates/guard-raw-shadow.ts @@ -111,7 +111,7 @@ export function rawShadows(files: readonly StyleFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${file.path}:\${line} writes \\\`\${property}: \${value}\\\` — a shadow with the hand-written length \${literal}, which no theme restates: dark mode keeps the light theme's elevation\`, - fix: \`\${written} — at \${file.path}:\${line}, then: x verify\`, + fix: \`\${written} — at \${file.path}:\${line}, then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } diff --git a/packages/cli/src/templates/guard-raw-z-index.ts b/packages/cli/src/templates/guard-raw-z-index.ts index 18ebbca1e..3487c6e74 100644 --- a/packages/cli/src/templates/guard-raw-z-index.ts +++ b/packages/cli/src/templates/guard-raw-z-index.ts @@ -89,7 +89,7 @@ export function rawZIndexes(files: readonly StyleFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${file.path}:\${line} writes \\\`z-index: \${value}\\\` — a number only this file knows, on no ladder another layer can be placed against\`, - fix: \`z-index: \${written} — at \${file.path}:\${line} (layers: \${LAYERS.map(([name]) => name).join(' ')}), then: x verify\`, + fix: \`z-index: \${written} — at \${file.path}:\${line} (layers: \${LAYERS.map(([name]) => name).join(' ')}), then: x verify --only boundaries\`, at: \`\${file.path}:\${line}\`, }); } diff --git a/packages/cli/src/templates/guard-released-forms.ts b/packages/cli/src/templates/guard-released-forms.ts index 8cc18c656..fe9d2d8ee 100644 --- a/packages/cli/src/templates/guard-released-forms.ts +++ b/packages/cli/src/templates/guard-released-forms.ts @@ -6,6 +6,7 @@ export const RELEASED_GUARD_FORMS: Readonly< Record > = { 'animated-layout-property': [ + ['21sfece05i0e3', 'error'], ['2hqr04589t23q', 'error'], ['2z9j3sd8e7lgl', 'error'], ], @@ -15,22 +16,29 @@ export const RELEASED_GUARD_FORMS: Readonly< ['1ot9kwydyzd0j', 'error'], ['2aaavkemg4qee', 'error'], ['2s43pcjqyhmfi', 'error'], + ['n9w3ww2famyi', 'error'], ], 'error-declared': [ ['1rlzex7nalpiw', 'error'], ['2tfw2hq80msju', 'error'], + ['3drdb0cke61iw', 'error'], ['3kqd7d8pzpcnn', 'error'], ], 'focus-visible': [ ['3i4t0gznxyytf', 'error'], ['4c2nf0mpzn0z', 'error'], + ['wndpmcuvl3o6', 'error'], ], 'image-dimensions': [ ['1cfzth883qzt5', 'error'], ['1upoweaclyd8', 'error'], + ['1yk40qifcror', 'error'], ['33uv2ua4a9zcf', 'error'], ], - 'island-mount-clears': [['1v09h5ptzqvs2', 'error']], + 'island-mount-clears': [ + ['1v09h5ptzqvs2', 'error'], + ['2buinh3zw0lqs', 'error'], + ], 'island-without-states': [ ['2os1f5fb6aass', 'error'], ['31gjhz3nyf0gz', 'error'], @@ -39,53 +47,84 @@ export const RELEASED_GUARD_FORMS: Readonly< ['1f7eh34hj085a', 'error'], ['2b134wpkoaqso', 'error'], ['2eb7ld3hg0tpb', 'error'], + ['3sabhehzp5i6g', 'error'], + ], + 'raw-breakpoint': [ + ['21j2j5t89cmax', 'error'], + ['wdo3lsqmvsya', 'error'], ], - 'raw-breakpoint': [['wdo3lsqmvsya', 'error']], 'raw-colour': [ ['17jw6r17ah3dq', 'error'], ['1e2pkqnmrjt44', 'error'], + ['2ml4gr1ou95vp', 'error'], ['3onnbx54g9zmd', 'error'], ['mdo7f20dnyof', 'error'], ], 'raw-fetch': [ ['17xdeprvzx5v2', 'error'], ['1hgdwthi7j353', 'error'], + ['21v2jbczhqz8k', 'error'], ['mazudnz7t6iy', 'error'], ], 'raw-interval': [ ['1o6p0fmjhzdhw', 'error'], ['2xrqszz066coc', 'error'], ['3qipy7ijjnz7r', 'error'], + ['3qu74px1wei5s', 'error'], ['ll7h7zpr075u', 'error'], ], - 'raw-length': [['2vfl0ssvkqvua', 'error']], + 'raw-length': [ + ['19by09okc9bse', 'error'], + ['2vfl0ssvkqvua', 'error'], + ], 'raw-motion': [ ['14c1bs20ysc1u', 'error'], + ['u6julhh0et2k', 'error'], ['w8aw0q20tfsz', 'error'], ], - 'raw-shadow': [['1vhy6xulc5cs5', 'error']], - 'raw-z-index': [['2nnuj62o3qvvr', 'error']], - 'repo-await-in-loop': [['3a3v6vvmbf8q2', 'warning']], + 'raw-shadow': [ + ['13t0n8kesciv6', 'error'], + ['1vhy6xulc5cs5', 'error'], + ], + 'raw-z-index': [ + ['2fae6nsqzvylo', 'error'], + ['2nnuj62o3qvvr', 'error'], + ], + 'repo-await-in-loop': [ + ['2453q2b7xpeu3', 'warning'], + ['3a3v6vvmbf8q2', 'warning'], + ], 'repo-raw-sql': [ ['12qj02j416sib', 'error'], ['2kr7gbv7j2m0v', 'error'], ['2m5sh5923zcdx', 'error'], + ['2od9gruczoiim', 'error'], ['3nmlelym16o78', 'error'], ], 'repo-unbounded': [ + ['1p97kt8y158hu', 'error'], ['2yjuyaj4n8dqq', 'error'], + ['3btoad27s4eai', 'warning'], ['ae4czielms3c', 'warning'], ], - 'runtime-pin': [['28ffbpwy4q2ji', 'error']], + 'runtime-pin': [ + ['28ffbpwy4q2ji', 'error'], + ['2xr2x2hjhyo1n', 'error'], + ], 'semantic-interactive': [ + ['24ry4tuzqa82q', 'error'], ['29pfa4ea628cv', 'error'], ['2m280ivq01zs2', 'error'], ['d4ncwu39wtrk', 'error'], ], - 'unbounded-fanout': [['xk4dv9vsq8gg', 'error']], + 'unbounded-fanout': [ + ['27m870avd47et', 'error'], + ['xk4dv9vsq8gg', 'error'], + ], 'undeclared-custom-property': [ ['2xnl4zuy9xv6f', 'error'], ['3b0u9i8hze00x', 'error'], + ['5q7i306j0imi', 'error'], ], 'undeclared-env': [ ['16pof11n1trhg', 'error'], @@ -96,6 +135,7 @@ export const RELEASED_GUARD_FORMS: Readonly< ], 'undefined-style-class': [ ['3food9t4vrzv8', 'error'], + ['ps08370p9qlg', 'error'], ['w7as1ib1ido8', 'error'], ], 'untranslated-string': [ @@ -108,6 +148,7 @@ export const RELEASED_GUARD_FORMS: Readonly< ], 'unzoned-date': [ ['14sc1shft1opi', 'error'], + ['1ljvva96m9b9z', 'error'], ['38jqi2md5adsi', 'error'], ['3caz4m0vn66dz', 'error'], ['8apjndc5ybvm', 'error'], diff --git a/packages/cli/src/templates/guard-repo-await-in-loop.ts b/packages/cli/src/templates/guard-repo-await-in-loop.ts index 9a8ff9e98..5a7528230 100644 --- a/packages/cli/src/templates/guard-repo-await-in-loop.ts +++ b/packages/cli/src/templates/guard-repo-await-in-loop.ts @@ -186,7 +186,7 @@ export function awaitsInLoops(files: readonly RepoFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${where} awaits inside the \${inner.kind} at \${loopLine} — one round trip per element, so the statements a call sends grow with the rows it is handed\`, - fix: \`db..andWhere('id', 'in', ids).limit(ids.length).all() (or insertAll / updateWhere / deleteWhere) — one statement for the set in place of the \${inner.kind} at \${loopLine}, or expectedQueryLoop('', fn) from @ultimat3/db around it — then: x verify\`, + fix: \`db.
.andWhere('id', 'in', ids).limit(ids.length).all() (or insertAll / updateWhere / deleteWhere) — one statement for the set in place of the \${inner.kind} at \${loopLine}, or expectedQueryLoop('', fn) from @ultimat3/db around it — then: x verify --only boundaries\`, at: where, }); } diff --git a/packages/cli/src/templates/guard-repo-raw-sql.ts b/packages/cli/src/templates/guard-repo-raw-sql.ts index 646c7989c..e62aa06fc 100644 --- a/packages/cli/src/templates/guard-repo-raw-sql.ts +++ b/packages/cli/src/templates/guard-repo-raw-sql.ts @@ -294,7 +294,7 @@ const KEYSET_CALL = const truncatedFinding = (at: string): Finding => ({ code: CODE, cause: \`\${at} orders by date_trunc('milliseconds', …), the workaround for a Date losing microseconds — and the index cannot serve that expression: measured on 300,000 rows, an Index Only Scan (0.2 ms) became a Parallel Seq Scan and a sort (100 ms)\`, - fix: \`\${KEYSET_CALL} — from @ultimat3/db, its select, where, orderBy and limit spliced in place of the date_trunc order at \${at}, then: x verify\`, + fix: \`\${KEYSET_CALL} — from @ultimat3/db, its select, where, orderBy and limit spliced in place of the date_trunc order at \${at}, then: x verify --only boundaries\`, at, }); @@ -328,7 +328,7 @@ const ANY_STATEMENT: readonly { finding: (at) => ({ code: CODE, cause: \`\${at} pages with a hand row comparison, (a, b) < (?, ?) — the seek keyset() writes with its tie-break, its direction and its cursor codec, each one a detail a hand copy gets wrong when a key column changes\`, - fix: \`\${KEYSET_CALL} — from @ultimat3/db, its keys the columns of the row comparison at \${at}, then: x verify\`, + fix: \`\${KEYSET_CALL} — from @ultimat3/db, its keys the columns of the row comparison at \${at}, then: x verify --only boundaries\`, at, }), }, @@ -337,7 +337,7 @@ const ANY_STATEMENT: readonly { finding: (at) => ({ code: CODE, cause: \`\${at} takes pg_advisory_xact_lock by hand — a key hashed where nothing else can see it, and two keys taken in whatever order two callers reach them, which is a deadlock\`, - fix: \`db.
.lockKeys([key], async (tx) => …) — the typed handle of the table the lock guards, sorted keys in one transaction, in place of the sql literal at \${at}, then: x verify\`, + fix: \`db.
.lockKeys([key], async (tx) => …) — the typed handle of the table the lock guards, sorted keys in one transaction, in place of the sql literal at \${at}, then: x verify --only boundaries\`, at, }), }, @@ -359,7 +359,7 @@ export function rawRepoSql(files: readonly RepoFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${at} sends a statement the typed handle expresses as hand-written SQL — no tenant guard on it, no codec for money or a sealed column, and no compile error when a column is renamed\`, - fix: \`\${call} — in place of the sql literal at \${at}, with db imported from this app's db package, then: x verify\`, + fix: \`\${call} — in place of the sql literal at \${at}, with db imported from this app's db package, then: x verify --only boundaries\`, at, }); } diff --git a/packages/cli/src/templates/guard-repo-unbounded.ts b/packages/cli/src/templates/guard-repo-unbounded.ts index 3d0bfead7..ca2836731 100644 --- a/packages/cli/src/templates/guard-repo-unbounded.ts +++ b/packages/cli/src/templates/guard-repo-unbounded.ts @@ -177,7 +177,7 @@ export function unboundedReads(files: readonly RepoFile[]): readonly Finding[] { findings.push({ code: CODE, cause: \`\${where} reads \${read} with .all() and no .limit(n) — .all() answers its first page, 50 rows, and drops the cursor to the rest, so the caller holds neither every row nor a page it chose\`, - fix: \`\${read}….limit().all() at \${where} — or .inBatches(size) to walk every row, .page() to hand a cursor on, or a // why: comment on that line naming the bound — then: x verify\`, + fix: \`\${read}….limit().all() at \${where} — or .inBatches(size) to walk every row, .page() to hand a cursor on, or a // why: comment on that line naming the bound — then: x verify --only boundaries\`, at: where, }); } diff --git a/packages/cli/src/templates/guard-runtime-pin.ts b/packages/cli/src/templates/guard-runtime-pin.ts index 1eb7e3fda..902385ad6 100644 --- a/packages/cli/src/templates/guard-runtime-pin.ts +++ b/packages/cli/src/templates/guard-runtime-pin.ts @@ -69,7 +69,7 @@ const imageFor = (version: string, variant: string, rest: string): string => const finding = (at: string, cause: string, write: string): Finding => ({ code: CODE, cause, - fix: \`edit \${at} — \${write} — then: x verify\`, + fix: \`edit \${at} — \${write} — then: x verify --only boundaries\`, at, }); diff --git a/packages/cli/src/templates/guard-semantic-interactive.ts b/packages/cli/src/templates/guard-semantic-interactive.ts index d11e52491..cd79b3da5 100644 --- a/packages/cli/src/templates/guard-semantic-interactive.ts +++ b/packages/cli/src/templates/guard-semantic-interactive.ts @@ -153,7 +153,7 @@ export function semanticInteractive(files: readonly SourceFile[]): readonly Find findings.push({ code: CODE, cause: \`\${at} puts role="\${role}" on <\${tag.name}> — a role is a promise, and this one obliges the element to answer Space, Enter, focus and disabled, which the native element already does\`, - fix: \`replace <\${tag.name} role="\${role}"> in \${file.path} with \${native.write}, then: x verify\`, + fix: \`replace <\${tag.name} role="\${role}"> in \${file.path} with \${native.write}, then: x verify --only boundaries\`, at: file.path, }); continue; @@ -163,7 +163,7 @@ export function semanticInteractive(files: readonly SourceFile[]): readonly Find findings.push({ code: CODE, cause: \`\${at} hangs \${handler} on <\${tag.name}>, which takes no focus and answers no key — a keyboard, a screen reader and a switch reach a mouse handler on an inert element in exactly one way, which is not at all\`, - fix: \`write
"` → it refuses with `X_MIGRATION_IRREVERSIBLE` → re-run as `x db gen "drop
" --allow-destructive`, or put the entity back | +| `X_MIGRATION_FORKED` after merging two branches | each ran `x db gen` from one parent, so the newest `.snapshot.json` omits the other's work | `x db gen --merge-fork`: no SQL changes, nothing is deleted. Do not generate a `record …` migration. If the cause says a migration creates something *again*, follow its `x db status --json` check, and never delete a migration a database applied | | `X_MIGRATION_SNAPSHOT_MISSING`, and `x db gen` refuses before writing anything | the newest migration on disk carries no `.snapshot.json` — hand-written, or the sidecar was deleted | `git checkout -- packages/db/migrations/.snapshot.json`; if it was never written, `rm packages/db/migrations/.* && x db gen ""` — the delete **first**, or the generate refuses again | | `ROLE=migrate` sits there doing nothing | another version's `ROLE=migrate` holds the session-pinned migration lock. This one **waits up to 60s**, polling once per 500ms, then exits non-zero with `X_MIGRATE_CONCURRENT` (`As of 2026-08`; on 1.2.0 it waits forever) | let the first exit 0 — the second then applies. If nothing else is deploying, the holder is a wedged backend: `psql "$DATABASE_URL" -c "select pid, application_name, state from pg_stat_activity join pg_locks using (pid) where locktype = 'advisory'"`, then `pg_terminate_backend(pid)` it | | `ROLE=migrate` exits non-zero, deploy blocked | migration failure — correct, the roll is supposed to stop | read the SQL error, fix the migration, re-run. Do not start `web` on the old schema | diff --git a/wiki/Tutorial-01-First-App.md b/wiki/Tutorial-01-First-App.md index 223a64cd6..b1297fb8d 100644 --- a/wiki/Tutorial-01-First-App.md +++ b/wiki/Tutorial-01-First-App.md @@ -40,7 +40,7 @@ cd myapp && bun run setup setup complete — next: bun run dev ``` -`bun run setup` is `bun install`, an `.env.development.local` touch, `x db gen "initial"`, `x db migrate`, `x db seed`, then `x manifest` — so the database is migrated and `x.manifest.json` exists before the gate asks for either. `x new` installs nothing, so this is not optional. +`bun run setup` is `bun install`, `x merge-driver install`, an `.env.development.local` touch, `x db gen "initial"`, `x db migrate`, `x db seed`, then `x manifest` — so the database is migrated and `x.manifest.json` exists before the gate asks for either. `x new` installs nothing, so this is not optional. `x` is a workspace dependency, not a global install. `bunx x ` resolves `node_modules/.bin/x`; the `package.json` scripts (`bun run verify`, `bun run dev`) resolve the same binary. A globally installed `x` (`bun link`, `bun add -g`) run inside the app **hands over** to that same file, `As of 2026-09-05`, and says so on stderr: a global copy carries its own `@ultimat3/entity` with an empty registry, and it wrote a zero-entity manifest under a green exit code before it did. `ULTIMATE_KEEP_GLOBAL_CLI=1` is the opt-out — [`@ultimat3/cli`'s README](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/README.md) has the measurement. @@ -79,7 +79,7 @@ The default scaffold, after `bun run setup`, with no `.x/build-stats.json` befor ✓ unit 2790ms 3 workers (budget 4.0 GB) ✓ contract 1233ms 3 workers (budget 4.0 GB) ✓ live 840ms serial - ✓ job 345ms 1 of 3 workers (budget 4.0 GB) + ✓ job 345ms 1 worker, one per file (plan: 3 workers (budget 4.0 GB)) ✓ exercise 1952ms ✓ e2e 12124ms serial ✓ eval 91ms @@ -240,7 +240,7 @@ bunx x db gen "initial" # entities → .sql, .snapshot.json, .has bunx x db migrate # applies them, then diffs the live schema against the ledger it wrote ``` -`bun run setup` runs both — `bun install`, an `.env.development.local` touch, `x db gen "initial"` when the directory holds no `.sql`, `x db migrate`, `x db seed`, then `x manifest`. `x db migrate` is `@ultimat3/db`'s own migrator, the one `ROLE=migrate` runs, so nothing extra has to be installed. +`bun run setup` runs both — `bun install`, `x merge-driver install`, an `.env.development.local` touch, `x db gen "initial"` when the directory holds no `.sql`, `x db migrate`, `x db seed`, then `x manifest`. `x db migrate` is `@ultimat3/db`'s own migrator, the one `ROLE=migrate` runs, so nothing extra has to be installed. Until the generate has run, `x verify` is **red on its `drift` step** — correct behaviour with a runnable fix, not a defect: diff --git a/wiki/Upgrading.md b/wiki/Upgrading.md index c0adbe8fb..bd47f19b1 100644 --- a/wiki/Upgrading.md +++ b/wiki/Upgrading.md @@ -319,6 +319,7 @@ A setter still on a package's `.` barrel is `X_ADAPTER_PORT_INCOMPLETE` on the ` | an offline replay sends `x-ultimate-replay-scope`, and the action route answers a session other than the queue's owner `409 X_REPLAY_PRINCIPAL_MISMATCH`, the entry kept `pending` (**31.47.0**, [Realtime](Realtime#tier-3-shipped-in-2100)); `DEFAULT_CORS.allowHeaders` lists the header | an app setting `cors.allowHeaders` replaces that list (`packages/http/src/config.ts`): add `x-ultimate-replay-scope` to it. A cross-origin caller sending the header needs this; the page outbox replays to a same-origin path, which has no preflight. An app on several processes sets `SESSION_SECRET` (32+ characters), or a replay reaching another process is refused | | `x exercise` with no `--seed`, and so `x verify`'s `exercise` step, runs one constant seed (**31.47.0**) where each run drew a random one, and `seed` in `.x/exercise.json` is always the number used, never `null`. One tree now gets one verdict, but an `exercise.unprovenMax` measured under random seeds can sit just below the constant seed's figure, so the first gate after the hop can be `X_EXERCISE_UNPROVEN_ABOVE_FLOOR`. `x upgrade` writes a figure only where the app states none, and never raises one | `x exercise --json`, then fix the helper or override each `unproven` record names: pick rows by what they are, never by id order, and assume no order between sized calls ([Testing](Testing)). If the figure that remains is the honest one, raising `unprovenMax` to it is your own reviewed edit of `x.verify.json`. `x exercise --seed random` looks for seed-dependent helpers | | `x upgrade` stops requiring an `e2e` suite no codemod wrote (**31.47.1**): an app with pages but no `apps/web/site/page.tsx` was left on `X_VERIFY_SUITE_VANISHED`. Rerunning `x upgrade` at the target no longer replays every codemod | nothing: `x upgrade` applies codemod `e2e-suite-floor`, which writes a `page.e2e.test.ts` beside the shortest parameter-free page, or moves `e2e` to `skip` with its why when no page can be visited. To apply codemods an older CLI left behind: `x upgrade codemods --from 0.0.0 --json` | +| 24 shipped guards' fixes change (**31.49.0**), built in; their findings do not: `animated-layout-property`, `bare-error`, `error-declared`, `focus-visible`, `image-dimensions`, `island-mount-clears`, `native-confirm`, `raw-breakpoint`, `raw-colour`, `raw-fetch`, `raw-interval`, `raw-length`, `raw-motion`, `raw-shadow`, `raw-z-index`, `repo-await-in-loop`, `repo-raw-sql`, `repo-unbounded`, `runtime-pin`, `semantic-interactive`, `unbounded-fanout`, `undeclared-custom-property`, `undefined-style-class` and `unzoned-date` end their fix on the one step that re-checks it (`then: x verify --only boundaries`) in place of `then: x verify`, which started the whole gate. A copy still in `guards/` keeps the old text until `x upgrade` migrates it to the release after 31.48.0 | nothing | ## 29.x → 30.0.0, entry by entry