Repository navigation
Closed
Labels
flake-hunta test flake-hunt.yml saw fail under a shuffled ordera test flake-hunt.yml saw fail under a shuffled order
Description
flake-hunt saw this fail in https://github.com/developerz-ai/ultimate/actions/runs/44443484119.
| suite | round | seed |
|---|---|---|
| unit | 1 | 4213420217 |
| unit | 2 | 2879767910 |
| unit | 3 | 3435136434 |
Replay each round in its order:
bun run x -- test unit --workers 4 --isolate -- --randomize --seed=4213420217
bun run x -- test unit --workers 4 --isolate -- --randomize --seed=2879767910
bun run x -- test unit --workers 4 --isolate -- --randomize --seed=3435136434The file alone, same seed — order across files is lost, so green here is not a fix:
bun test ./packages/cli/src/write-generated.test.ts --randomize --seed=4213420217What Bun printed:
ts it itself when nothing answers at `TEST_DATABASE_URL` and Docker is\nthere; by hand it is `docker compose -f docker/docker-compose.test.yml up -d --wait` then\n`set -a; . docker/test-services.env; set +a`. A resource a test opens is held with\n`await using` — `await using db = await acquireWorkerDatabase()` drops its database however\nthe test ends:\n\n| What | How its test runs it | Copy from |\n|---|---|---|\n| a page | `renderRoute(page, { url, actor })` — `load`, `meta` and the page on one resolution; assert on `view.text`, `view.html`, `view.meta`, `view.islands` | `apps/web/app/dashboard/page.test.ts` |\n| a component | `renderView(Component, props)` | `apps/web/shared/shell.test.ts` |\n| an island | `mountIsland`, once per state its `*.island.states.ts` declares | `apps/web/shared/theme-toggle.island.test.ts` |\n| an action or a query | `target.as(actor, input)` — the refusal first, then what it answers | `apps/web/api/health.test.ts` |\n| a job, a task, a backfill | `test('…', async ({ runJobs }) => …)` — a worker in this process; assert on `trace.steps` and each run's `outcome` | what `bunx x g job` writes |\n| a repo, a seed | as an actor, `runWithContext(ctxOf({ actor }), …)`, with `driver.reset?.()` after each | `packages/db/src/seed.test.ts` |\n\nMoney is the one row with no guard, deliberately: a float has no static signature a text rule can\nsee, and the type already fires — measured, `price: 19.99` in a seed is\n`TS2322: Type 'number' is not assignable to type 'MoneyInput'`. A guard that pretended to check\nit would be worse than the type that really does.\n\nCommands: `bun run setup`, `bun run dev`, `bun run verify`, `bunx x g <primitive>`,\n`bunx x g guard <name>`, `bunx x db branch create <name>`, `bunx x doctor`. `setup` is\n`bin/setup.ts`, a file Bun runs, so all three are the same command in PowerShell, cmd and bash.\n\n- `bunx x g --help` lists every generator kind, and is the only place that list is stated.\n- Destructive database work goes in a branch, `bunx x db branch create <name>`, never the shared\n dev database.\n- `bunx x db gen` also writes `packages/db/schema/`, the whole schema as SQL, one file per\n table: read it, never edit it. `bunx x verify` fails with `X_SCHEMA_DUMP_DRIFT` when it is not\n what the migrations produce, and `bunx x db gen` rewrites it.\n- `bunx x doctor` explains a broken environment and prints the fix for every finding.\n- `.claude/` holds a harness that reads this file: `/feature`, `/planx`, `/verify` and four\n boundary-scoped subagents. `.claude/README.md` says what each costs; every file is deletable.\n\n## Agent tools\n\n| Tool | What it answers | How to reach it |\n|---|---|---|\n| `x.manifest.json` | every route, action, query, job and policy, generated — read it before searching the tree | `bunx x manifest` rewrites it |\n| `bunx x impact <file|primitive> --json` · `bunx x graph --json` | what an edit reaches, and the app as one graph (declared, import and observed edges) | rebuilt on every call |\n| `.mcp.json` | the dev tools over MCP: routes, schema, policies, the database, tests, the gate — `bunx x mcp tools` lists them | a client that reads `.mcp.json` starts it; from any other, run `bunx @ultimat3/cli mcp serve --transport stdio` here |\n| `POST /mcp` | this app's own tools — every action declaring `mcp: { expose: true }` — for a remote agent | `apps/web/mcp.ts`; under `bun run dev`, `Authorization: Bearer dev-agent.admin`, else an API key |\n| `.claude/commands/feature.md` | build or fix one thing end to end, done when `bun run verify` is green | plain markdown any agent can follow |\n| `.claude/commands/planx.md` | write a plan to `docs/plans/` | the same |\n| `.claude/commands/verify.md` | run the gate and fix what it reports | the same |\n\nProject notes for link-demo: replace this line with the conventions a newcomer could not guess.\n"%0A%0A at <anonymous> (/home/runner/work/ultimate/ultimate/packages/cli/src/write-generated.test.ts:74:45)
Metadata
Metadata
Assignees
Labels
flake-hunta test flake-hunt.yml saw fail under a shuffled ordera test flake-hunt.yml saw fail under a shuffled order
Activity
sebyx07 commentedon Nov 19, 2026
Fixed on main by a22539b (#1261), which carried later commits than the hunted a5733be.
Cause: not order dependence: the test pinned a CLAUDE.md template sentence (
AGENTS.mdbeing a symlink to it) that the template had reworded. The fix asserts the current wording.Evidence
packages/cli/src/write-generated.test.tson main makes this test fail again under--randomize --seed=4213420217. With main's version it passes.CI=true.