Skip to content

test(cli): derive the vitest tier population instead of freezing it in a list #14717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
os-trump merged 3 commits into from
Sep 3, 2026
Merged

test(cli): derive the vitest tier population instead of freezing it in a list #14717

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7c07428
test(cli): derive the vitest tier population instead of freezing it i…
claude Sep 2, 2026
192e497
docs(cli): rewrite the tiers header for a derived population
claude Sep 2, 2026
d29fbf2
Merge remote-tracking branch 'origin/main' into claude/issue-14554-vi…
claude Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 111 additions & 143 deletions packages/cli/test/vitest-tiers-partition.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,150 +1,92 @@
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.

/**
* The two tiers of this package's suite stay a PARTITION, and the integration
* list stays equal to what the files DO (#13504).
* The two tiers of this package's suite stay a PARTITION, and the tier of
* every file stays what the file DOES (#13504, #14554).
*
* `vitest.config.ts` splits the suite into two named projects — `unit` (the
* local default) and `integration` (spawns the real CLI or boots a real
* kernel/driver; CI-mandatory, local on demand). Two things can rot under a
* split like that, and both rot silently, which is why this pin exists:
* kernel/driver; CI-mandatory, local on demand). The membership is DERIVED at
* config load by `../vitest-tiers.ts`, not written down, and that module's
* header carries the merge-queue ejections the old frozen list caused.
*
* 1. A test file that matches NO project is not run by `vitest run` at all —
* not by the fast tier AND not by `pnpm test` in CI, because with
* `projects` configured the root run IS the union of the projects. A file
* matching BOTH runs twice and reports twice. So the first two cases hold
* `unit ⊎ integration = every test file on disk`, read from vitest's own
* resolution (`vitest list --filesOnly`, with and without `--project`)
* against a filesystem walk — the config's spelling is judged by what
* vitest actually collects, never by re-reading the config.
* ## What is left for a pin once the list is derived (#14554)
*
* 2. `INTEGRATION_FILES` is an explicit list, and the `*.e2e.test.ts` NAME is
* not the predicate (the ACCEPT on #13504 measured 18 of 220 files where
* name and behaviour disagree). So the third case re-derives the tier of
* every file from its comment-masked SOURCE and fails when the list and
* the derivation disagree — a new spawner cannot land in the fast tier
* unnoticed, and a stale entry cannot linger. The predicate, in code
* position (comments masked by `scripts/js-comment-mask.mjs`):
* A derived list cannot go stale, so the assertion this pin used to lead with
* — declared list == predicate — is gone with the list. Three jobs remain, and
* they are the ones that were always doing the real work:
*
* SPAWN = calls `runServe(` (the helper in `test/helpers/serve-process.ts`
* whose body spawns the source entry), OR value-imports
* `node:child_process` AND (names an entry basename — the
* `run-dev` / `run` scripts under `bin/` — OR imports `CLI` /
* `TSX` from that helper OR names the `tsx` binary under
* `node_modules/.bin`);
* KERNEL = value-imports `bootSchemaStack` from `schema-migrate`, OR
* value-imports `better-sqlite3`, OR value-imports any
* `@objectstack/driver-*` package, OR constructs `new ObjectQL(`.
* INTEGRATION = SPAWN ∨ KERNEL.
* 1. **COVERAGE.** A test file that matches NO project is not run by
* `vitest run` at all — not by the fast tier AND not by `pnpm test` in CI,
* because with `projects` configured the root run IS the union of the
* projects. vitest 4.1.10 reports that whole-suite run GREEN while never
* executing the file. A file matching BOTH runs twice and reports twice.
* So the first two cases hold `unit ⊎ integration = every test file on
* disk`, read from vitest's own resolution (`vitest list --filesOnly`,
* with and without `--project`) against a filesystem walk — the config's
* spelling is judged by what vitest actually COLLECTS, never by re-reading
* the config. This is the defect the tier split can cause and the reason
* the split can never simply be deleted.
*
* Value imports only: `import type { … } from '@objectstack/driver-sql'`
* loads nothing, a spelling list that SAYS `'better-sqlite3'` opens no
* database, and `expect(deps).toContain('better-sqlite3')` boots nothing —
* every one of those was a false positive of the text-match census this
* predicate replaced. An import statement is one `import … from '<spec>'`
* span containing neither `;` nor another `from` (every import in this
* package's tests ends in `;`, measured on 00ff228fe0).
* 2. **THE DERIVATION REACHES VITEST.** Computing the right set in the config
* and having vitest collect it are two different facts: the entries are
* handed to `include` / `exclude` as GLOBS, so a path the walk spells one
* way and tinyglobby reads another lands in the wrong tier while every
* count still looks right. The third case therefore compares the
* `integration` population vitest REPORTS against an independent
* re-derivation over the tree — the end-to-end reading the old list-vs-
* predicate comparison could not make. ⚠️ Unlike that comparison, it
* cannot fire merely because a qualifying file arrived on `main`: such a
* file is classified by the same config that collects it.
*
* The fourth case classifies THIS file: it imports `node:child_process` (to
* ask vitest for its file lists) and must still read as `unit`, which is the
* predicate's own regression test against matching its own source.
* 3. **THE PREDICATE ITSELF.** The config and this pin now share one
* predicate, so nothing that compares them can see a predicate that is
* WRONG — where the frozen list, an independent human record, would have
* disagreed with it. `../vitest-tiers.fixtures.ts` replaces that
* independence in kind: whole tiny sources whose tier is known by
* construction, one per signal and one per false positive the predicate
* was tuned against, plus a union check so a newly declared signal without
* a fixture is red. A predicate that matched nothing would otherwise empty
* the integration tier, serialise the suite back into `unit`, and leave
* every population assertion above green.
*
* The predicate is stated once, in `../vitest-tiers.ts`, and is NOT restated
* here: SPAWN or KERNEL, in code position, comments masked. ⛔ #14554 changed
* how membership is maintained and nothing about what a tier MEANS.
*
* The last case classifies THIS file: it imports child_process (to ask vitest
* for its file lists) and must still read as `unit`, which is the predicate's
* own regression test against matching its own source.
*
* Runs in the `unit` tier and needs no built `dist/`: it spawns `vitest list`,
* which only globs, and reads sources.
*/

import { execFileSync } from 'node:child_process';
import { readdirSync, readFileSync } from 'node:fs';
import { readFileSync } from 'node:fs';
import { createRequire } from 'node:module';
import { dirname, join, relative, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { describe, expect, it } from 'vitest';
import { maskComments } from '../../../scripts/js-comment-mask.mjs';
import { childEnv } from './helpers/serve-process.js';
import {
firedSignals,
integrationTestFiles,
isIntegration,
testFilesOnDisk,
tierOfFile,
tierSignals,
type TierSignals,
} from '../vitest-tiers.js';
import { PREDICATE_CASES } from '../vitest-tiers.fixtures.js';

const HERE = dirname(fileURLToPath(import.meta.url));
const PKG = resolve(HERE, '..');
const THIS_FILE = relative(PKG, fileURLToPath(import.meta.url));
const require = createRequire(import.meta.url);
const VITEST_ENTRY = resolve(dirname(require.resolve('vitest/package.json')), 'vitest.mjs');

// ---------------------------------------------------------------------------
// The predicate
// ---------------------------------------------------------------------------

interface ValueImport {
clause: string;
spec: string;
}

/** One `import … from '<spec>'` statement; `import type` is skipped. */
const IMPORT_RE = /\bimport\s+(?!type\b)((?:(?!\bfrom\b)[^;])*?)\bfrom\s+['"]([^'"]+)['"]/g;

function valueImports(code: string): ValueImport[] {
const out: ValueImport[] = [];
for (const m of code.matchAll(IMPORT_RE)) out.push({ clause: m[1], spec: m[2] });
return out;
}

/** Inline `type X` specifiers do not make a value import of `X`. */
function importsValue(imports: ValueImport[], spec: RegExp, name?: RegExp): boolean {
return imports.some((i) => spec.test(i.spec) && (!name || name.test(i.clause.replace(/\btype\s+\w+/g, ''))));
}

export interface TierSignals {
runServe: boolean;
childProcess: boolean;
entryBasename: boolean;
helperCliOrTsx: boolean;
tsxBin: boolean;
bootSchemaStack: boolean;
betterSqlite3: boolean;
driverPackage: boolean;
objectQLCtor: boolean;
}

export function tierSignals(maskedCode: string): TierSignals {
const imports = valueImports(maskedCode);
return {
runServe: /\brunServe\s*[(]/.test(maskedCode),
childProcess: importsValue(imports, /^(?:node:)?child_process$/),
entryBasename: /\brun(?:-dev)?[.]js\b/.test(maskedCode),
helperCliOrTsx: importsValue(imports, /helpers\/serve-process(?:\.js)?$/, /\b(?:CLI|TSX)\b/),
tsxBin: /[.]bin[/]tsx\b/.test(maskedCode),
bootSchemaStack: importsValue(imports, /schema-migrate(?:\.js)?$/, /\bbootSchemaStack\b/),
betterSqlite3: importsValue(imports, /^better-sqlite3$/) || /require[(]\s*['"]better-sqlite3['"]\s*[)]/.test(maskedCode),
driverPackage: importsValue(imports, /^@objectstack\/driver-/),
objectQLCtor: /new\s+ObjectQL\s*[(]/.test(maskedCode),
};
}

export function isIntegration(s: TierSignals): boolean {
const spawn = s.runServe || (s.childProcess && (s.entryBasename || s.helperCliOrTsx || s.tsxBin));
const kernel = s.bootSchemaStack || s.betterSqlite3 || s.driverPackage || s.objectQLCtor;
return spawn || kernel;
}

function firedSignals(s: TierSignals): string {
return (Object.keys(s) as Array<keyof TierSignals>).filter((k) => s[k]).join(', ') || 'none';
}

// ---------------------------------------------------------------------------
// The two readings: the filesystem, and vitest's own resolution
// ---------------------------------------------------------------------------

const TEST_FILE_RE = /\.(?:test|spec)\.[cm]?[jt]sx?$/;
const SKIP_DIRS = new Set(['node_modules', 'dist', '.git', '.turbo', 'coverage']);

function walk(dir: string, out: string[] = []): string[] {
for (const entry of readdirSync(dir, { withFileTypes: true })) {
if (SKIP_DIRS.has(entry.name)) continue;
const abs = join(dir, entry.name);
if (entry.isDirectory()) walk(abs, out);
else if (TEST_FILE_RE.test(entry.name)) out.push(relative(PKG, abs));
}
return out;
}

/** `vitest list --filesOnly [--project NAME]`, one relative path per line. */
function vitestFiles(project?: string): string[] {
const args = [VITEST_ENTRY, 'list', '--filesOnly', ...(project ? ['--project', project] : [])];
Expand All @@ -161,18 +103,10 @@ function vitestFiles(project?: string): string[] {
.map((line) => line.replace(/^\[[^\]]+\]\s+/, ''));
}

/** The explicit list as `vitest.config.ts` declares it — read as text, not imported. */
function declaredIntegrationFiles(): string[] {
const masked = maskComments(readFileSync(join(PKG, 'vitest.config.ts'), 'utf8'));
const block = /INTEGRATION_FILES\s*=\s*\[([^\]]*)\]/.exec(masked);
if (!block) throw new Error('vitest.config.ts no longer declares `INTEGRATION_FILES = [ … ]`');
return Array.from(block[1].matchAll(/['"]([^'"]+)['"]/g), (m) => m[1]);
}

const sorted = (xs: Iterable<string>): string[] => [...xs].sort();

describe('the two tiers of packages/cli (#13504)', () => {
const onDisk = sorted(walk(PKG));
describe('the two tiers of packages/cli (#13504, #14554)', () => {
const onDisk = testFilesOnDisk(PKG);
const all = sorted(vitestFiles());
const unit = sorted(vitestFiles('unit'));
const integration = sorted(vitestFiles('integration'));
Expand All @@ -189,31 +123,65 @@ describe('the two tiers of packages/cli (#13504)', () => {
expect(union, 'files matched by NEITHER project fall out of every tier, including CI').toEqual(all);
});

it('INTEGRATION_FILES equals the behavioural predicate over every file on disk', () => {
const declared = declaredIntegrationFiles();
expect(sorted(new Set(declared)), 'INTEGRATION_FILES carries a duplicate').toEqual(sorted(declared));
expect(sorted(declared), 'an INTEGRATION_FILES entry names no file vitest can find').toEqual(integration);
it('the integration tier vitest collects is the behavioural predicate, re-derived over the tree', () => {
const predicted = integrationTestFiles(PKG);

const missing: string[] = [];
const stale: string[] = [];
for (const file of onDisk) {
const signals = tierSignals(maskComments(readFileSync(join(PKG, file), 'utf8')));
const predicted = isIntegration(signals);
const listed = integration.includes(file);
if (predicted && !listed) missing.push(`${file} [${firedSignals(signals)}]`);
if (!predicted && listed) stale.push(file);
}
const missing = predicted
.filter((f) => !integration.includes(f))
.map((f) => `${f} [${firedSignals(tierOfFile(PKG, f))}]`);
expect(
missing,
'files that spawn the CLI or boot a kernel/driver but are NOT in INTEGRATION_FILES (add them)',
'files the predicate calls integration that vitest did NOT collect into that project — ' +
'the derivation did not reach vitest (a path spelled one way by the walk and another by the glob)',
).toEqual([]);
expect(stale, 'INTEGRATION_FILES entries that neither spawn nor boot (remove them)').toEqual([]);

const extra = integration.filter((f) => !predicted.includes(f));
expect(extra, 'files vitest collected as integration that the predicate does not call integration').toEqual([]);
});

it('the config DERIVES that population rather than freezing a copy of it', () => {
const masked = maskComments(readFileSync(join(PKG, 'vitest.config.ts'), 'utf8'));
expect(
/integrationTestFiles\s*[(]/.test(masked),
'vitest.config.ts no longer derives its integration tier from `integrationTestFiles(`',
).toBe(true);
expect(
/INTEGRATION_FILES\s*=\s*\[/.test(masked),
'vitest.config.ts froze the integration tier back into a literal list. That list goes stale when ' +
'ANOTHER PR lands a qualifying test file, and the pin then reds in the merge queue against a tree ' +
'the failing PR never touched — five ejected PRs in 24 hours, four of them bystanders (#14554).',
).toBe(false);
});

it('this pin is itself unit-tier: importing child_process to ask vitest is not spawning the CLI', () => {
const signals = tierSignals(maskComments(readFileSync(join(PKG, THIS_FILE), 'utf8')));
const signals = tierOfFile(PKG, THIS_FILE);
expect(signals.childProcess).toBe(true);
expect(isIntegration(signals), `fired: ${firedSignals(signals)}`).toBe(false);
expect(unit).toContain(THIS_FILE);
});
});

describe('the predicate itself, against sources whose tier is known by construction (#14554)', () => {
it('every signal the predicate declares is fired by some fixture', () => {
const declared = sorted(Object.keys(tierSignals('')));
const covered = sorted(new Set(PREDICATE_CASES.flatMap((c) => c.fires)));
expect(
covered,
'a signal with no fixture is a signal whose regex can be deleted with every assertion still green',
).toEqual(declared);
});

for (const testCase of PREDICATE_CASES) {
const tier = testCase.integration ? 'integration' : 'unit';
it(`${tier}: ${testCase.name}`, () => {
const signals: TierSignals = tierSignals(maskComments(testCase.source));
expect(isIntegration(signals), `${testCase.why} — fired: ${firedSignals(signals)}`).toBe(testCase.integration);
for (const signal of testCase.fires) {
expect(signals[signal], `${signal} must fire here — fired: ${firedSignals(signals)}`).toBe(true);
}
for (const signal of testCase.silent ?? []) {
expect(signals[signal], `${signal} must NOT fire here — fired: ${firedSignals(signals)}`).toBe(false);
}
});
}
});
Loading
Loading