diff --git a/.changeset/29961-stored-expression-backfill.md b/.changeset/29961-stored-expression-backfill.md new file mode 100644 index 00000000000..259f34c5fe0 --- /dev/null +++ b/.changeset/29961-stored-expression-backfill.md @@ -0,0 +1,15 @@ +--- +'@objectstack/objectql': patch +'@objectstack/cli': patch +--- + +fix(objectql,cli): `os migrate stored-expressions` fills a stored computed field on rows stored before it was declared, and the boot names the command when a sample of rows needs it + +Clause-②: yes (widening) +Metadata-protocol: none + +A stored computed field (a non-`formula` field declaring `expression`) is computed by the engine on every insert and update and on nothing else, so declaring one on an object that already holds rows left the new column NULL on every existing row until each was next written; a filter, sort, roll-up or analytics anchor over it read NULL where the expression gives a value, with no error. A fresh database was right and an upgraded one was not. + +- **`@objectstack/cli`: `os migrate stored-expressions [--object ]… [--field ] [--changed] [--apply] [--yes] [--json] [--force] [--max-records] [--database-url]`.** A dry run (the default) reports, per field, the rows holding a NULL the expression gives a value; `--apply` writes them. Each value is the write's own computation over the stored row (option labels, the one reference hop, the `can()` map), and the apply writes through `engine.update` with `updated_at` kept, so hooks, roll-ups over the column and audit capture see an ordinary write. A NULL the expression itself answers (an empty reference, a null input) is left alone. A value that differs from the expression — the expression changed after the row was written — is counted on every run and rewritten only with `--changed`; an expression reading the clock always differs there. A row on which an expression does not evaluate while it holds a value is listed and not written, because the write would store NULL over it. The command exits 1 while a row is skipped or could not be written, or when `--max-records` cut the walk short. Idempotent: a re-run finds nothing to fill. +- **Boot advisory.** At `kernel:bootstrapped` the engine plugin reads the first 100 rows of each object declaring a stored computed field (nothing for an object declaring none) and, when that sample holds such a NULL, warns once naming each `object.field` and `os migrate stored-expressions`. It writes nothing and never fails a boot. +- **`@objectstack/objectql`:** `backfillStoredExpressions`, `formatStoredExpressionBackfillReport`, `storedExpressionBackfillComplete`, `announceStoredExpressionHoles`, `sampleStoredExpressionHoles`, and `ObjectQL.computeStoredExpressionsOver(object, rows, context?, onFault?)` — a stored computed field's values over rows already stored, computed as a write computes them. diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index 3ab16296211..1cffc977e5a 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -1303,6 +1303,7 @@ where the data lives. | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag | | `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean | | `os migrate summary-nulls` | Backfill roll-up `count` / `sum` columns still stored as `NULL` on parent rows created before the insert-time seed. Repairs values; no flag, nothing depends on it having run | +| `os migrate stored-expressions` | Fill a stored computed field (a non-formula field declaring `expression`) on the rows stored before the field was declared, which hold `NULL` until each is next written; the boot warns, naming the field, when a sample of rows holds such a `NULL`. `--changed` also recomputes values an edited expression no longer gives. Rows an expression cannot compute while they hold a value are listed, never written. No flag; idempotent | | `os migrate blank-single-values` | Rewrite the empty strings (`''`) still stored on optional single-value option, reference and file columns to `null`, the form a cleared value is stored in today, so `$empty` finds those rows. Repairs values; no flag, nothing depends on it having run. See [below](#os-migrate-blank-single-values) | | `os migrate option-codes` | Rewrite option values still stored as a number, a boolean or SQLite `"N.0"` text (rows written before option fields stored the code string) to that code string, so filters, reads and unique keys see one value. Values it cannot decide are listed, never rewritten. No flag; idempotent. See [below](#os-migrate-option-codes) | | `os migrate text-scalars` | Rewrite numbers and booleans still stored raw on text fields (rows written before text fields stored the string) to that string, so filters, reads and unique keys see one value. SQLite `"N.0"` text it cannot decide is listed, never rewritten. No flag; idempotent. See [below](#os-migrate-text-scalars) | diff --git a/packages/cli/src/commands/migrate/stored-expressions.test.ts b/packages/cli/src/commands/migrate/stored-expressions.test.ts new file mode 100644 index 00000000000..324479b410c --- /dev/null +++ b/packages/cli/src/commands/migrate/stored-expressions.test.ts @@ -0,0 +1,161 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `os migrate stored-expressions` command shape (#29961). + * + * The backfill itself is proven in `@objectstack/objectql`'s + * `stored-expression-backfill.test.ts`. What is pinned here is what a unit + * test of the walk cannot see: the command is a dry run by default, every + * flag reaches `backfillStoredExpressions` as its option, `--field` needs + * `--object`, a run that leaves a row skipped or unwritten exits 1, and the + * human face closes a dry run with the report's own line. The seams that + * would boot a database are replaced; the command's own parse and control + * flow, and the report's formatter, run for real. + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { dirname, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { stripVTControlCharacters } from 'node:util'; +import MigrateStoredExpressions from './stored-expressions.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; +import { isExitSignal } from '../../utils/format.js'; +import { backfillStoredExpressions } from '@objectstack/objectql'; + +vi.mock('../../utils/schema-migrate.js', () => ({ bootSchemaStack: vi.fn() })); +vi.mock('../../utils/migrate-occupancy-gate.js', () => ({ + OCCUPANCY_HINT: 'occupancy hint', + probeMigrationTarget: vi.fn(), +})); +vi.mock('../../utils/data-migration-plugins.js', () => ({ buildDataMigrationPlugins: vi.fn(async () => []) })); +// The walk is replaced; the formatter is the real one. +vi.mock('@objectstack/objectql', async (importOriginal) => { + const real = await importOriginal(); + return { + backfillStoredExpressions: vi.fn(), + formatStoredExpressionBackfillReport: real.formatStoredExpressionBackfillReport, + }; +}); + +const HERE = dirname(fileURLToPath(import.meta.url)); +const CLI_ROOT = resolve(HERE, '..', '..', '..'); +/** oclif builds its whole command table on the first `run()` in a process. */ +const RUN_TIMEOUT = 60_000; + +/** The engine surface the command checks before it runs, and one loaded app object. */ +const engine = { + computeStoredExpressionsOver: async () => [], + getConfigs: () => ({ contract: {}, sys_user: {} }), +}; + +const EMPTY_REPORT = { + scannedObjects: [], scannedRecords: 0, fields: [], nullRows: 0, staleRows: 0, faultedRows: 0, + skippedRows: 0, writtenRows: 0, changed: false, applied: false, truncated: false, + unreadableObjects: [], faults: [], faultsTruncated: false, failures: [], +}; + +/** A dry run that found the hotclm shape: one field NULL on every row. */ +const DRY_REPORT_WITH_HOLES = { + ...EMPTY_REPORT, + scannedObjects: ['contract'], + scannedRecords: 2, + fields: [{ object: 'contract', field: 'review_sla_days_offset', nullRows: 2, staleRows: 0, faultedRows: 0, written: 0, sampleRecordIds: ['c1', 'c2'] }], + nullRows: 2, +}; + +let stdout: ReturnType; +let log: ReturnType; +beforeEach(() => { + vi.mocked(probeMigrationTarget).mockResolvedValue({ status: 'free' } as any); + vi.mocked(bootSchemaStack).mockClear(); + vi.mocked(bootSchemaStack).mockResolvedValue({ + kernel: { getService: () => engine }, + dbLabel: 'postgres://db.example.com/app', + driver: { config: { client: 'pg' } }, + shutdown: vi.fn(async () => {}), + } as any); + vi.mocked(backfillStoredExpressions).mockReset(); + vi.mocked(backfillStoredExpressions).mockResolvedValue(EMPTY_REPORT as any); + // `emitJson` awaits the write's DRAIN callback, so the double has to invoke it. + stdout = vi.spyOn(process.stdout, 'write').mockImplementation(((chunk: unknown, enc?: unknown, cb?: unknown) => { + const done = typeof enc === 'function' ? enc : cb; + if (typeof done === 'function') done(); + return true; + }) as typeof process.stdout.write); + log = vi.spyOn(console, 'log').mockImplementation(() => {}); +}); +afterEach(() => { + stdout.mockRestore(); + log.mockRestore(); +}); + +const optionsHandedToBackfill = () => { + expect(vi.mocked(backfillStoredExpressions)).toHaveBeenCalledTimes(1); + return vi.mocked(backfillStoredExpressions).mock.calls[0][2]!; +}; +const exitCodeOf = (err: unknown) => (isExitSignal(err) ? (err as { oclif?: { exit?: number } }).oclif?.exit : undefined); + +describe('os migrate stored-expressions', () => { + it('is a dry run by default, and writing needs --yes', () => { + expect(MigrateStoredExpressions.flags.apply.default).toBe(false); + expect(MigrateStoredExpressions.flags.yes.default).toBe(false); + expect(MigrateStoredExpressions.flags.changed.default).toBe(false); + }); + + it('a bare run walks every object, NULL holes only, without writing', async () => { + await MigrateStoredExpressions.run(['--json'], { root: CLI_ROOT }); + expect(optionsHandedToBackfill()).toEqual({ + apply: false, objects: undefined, field: undefined, changed: false, maxRecordsPerObject: undefined, + }); + }, RUN_TIMEOUT); + + it('hands --object, --field, --changed, --max-records and --apply --yes through', async () => { + await MigrateStoredExpressions.run( + ['--json', '--object', 'contract', '--field', 'review_sla_days_offset', '--changed', '--max-records', '50', '--apply', '--yes'], + { root: CLI_ROOT }, + ); + expect(optionsHandedToBackfill()).toEqual({ + apply: true, objects: ['contract'], field: 'review_sla_days_offset', changed: true, maxRecordsPerObject: 50, + }); + }, RUN_TIMEOUT); + + it('refuses --field without --object at parse time', async () => { + const err = await MigrateStoredExpressions.run(['--json', '--field', 'x'], { root: CLI_ROOT }).catch((e: unknown) => e); + expect(String((err as Error)?.message)).toMatch(/--object/); + expect(vi.mocked(backfillStoredExpressions)).not.toHaveBeenCalled(); + }, RUN_TIMEOUT); + + it('--apply without --yes in a non-interactive run writes nothing and exits 1', async () => { + const err = await MigrateStoredExpressions.run(['--json', '--apply'], { root: CLI_ROOT }).catch((e: unknown) => e); + expect(exitCodeOf(err)).toBe(1); + expect(vi.mocked(bootSchemaStack)).not.toHaveBeenCalled(); + expect(vi.mocked(backfillStoredExpressions)).not.toHaveBeenCalled(); + }, RUN_TIMEOUT); + + it('an object the deployment does not declare is refused before the walk', async () => { + const err = await MigrateStoredExpressions.run(['--json', '--object', 'nope'], { root: CLI_ROOT }).catch((e: unknown) => e); + expect(exitCodeOf(err)).toBe(1); + expect(vi.mocked(backfillStoredExpressions)).not.toHaveBeenCalled(); + const payload = JSON.parse(stdout.mock.calls.map((c: unknown[]) => String(c[0])).join('')); + expect(payload).toMatchObject({ code: 'OBJECT_NOT_FOUND' }); + }, RUN_TIMEOUT); + + it.each([ + ['a row skipped because a faulting field holds a value', { skippedRows: 1 }], + ['a row the write refused', { failures: [{ object: 'contract', recordId: 'c1', error: 'refused' }] }], + ['a walk --max-records cut short', { truncated: true }], + ])('exits 1 on %s', async (_label, extra) => { + vi.mocked(backfillStoredExpressions).mockResolvedValue({ ...EMPTY_REPORT, ...extra } as any); + const err = await MigrateStoredExpressions.run(['--json'], { root: CLI_ROOT }).catch((e: unknown) => e); + expect(exitCodeOf(err)).toBe(1); + }, RUN_TIMEOUT); + + it('a dry run that finds holes exits 0 and closes with the report\'s dry-run line', async () => { + vi.mocked(backfillStoredExpressions).mockResolvedValue(DRY_REPORT_WITH_HOLES as any); + await MigrateStoredExpressions.run([], { root: CLI_ROOT }); + const printed = log.mock.calls.map((c: unknown[]) => stripVTControlCharacters(String(c[0]))).join('\n'); + expect(printed).toContain('contract.review_sla_days_offset — 2 NULL row(s)'); + expect(printed).toContain('Dry run — no row or table was written. Re-run with --apply to write.'); + }, RUN_TIMEOUT); +}); diff --git a/packages/cli/src/commands/migrate/stored-expressions.ts b/packages/cli/src/commands/migrate/stored-expressions.ts new file mode 100644 index 00000000000..538afdec87f --- /dev/null +++ b/packages/cli/src/commands/migrate/stored-expressions.ts @@ -0,0 +1,266 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { Command, Flags } from '@oclif/core'; +import chalk from 'chalk'; +import { createInterface } from 'node:readline'; +import { + printHeader, + printSuccess, + printWarning, + printError, + printInfo, + printStep, + createTimer, + emitJson, + isExitSignal, + errorCodeFields, +} from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; +import { describeOccupancy, JOURNAL_MODE_SWITCH_NOTE } from '../../utils/sqlite-occupancy.js'; +import { describeDriverSqliteFile } from '../../utils/connection-display.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; +import { refuseUndeclaredObjects } from '../../utils/migrate-object-scope.js'; +// Type-only: the engine package is imported dynamically inside run(), never at +// module load, so `os --help` and every other command pay nothing for it. +import type { StoredExpressionBackfillEngine } from '@objectstack/objectql'; + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((resolve) => rl.question(question, resolve)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} + +/** + * `os migrate stored-expressions` — fill the stored computed fields (a + * non-`formula` field declaring `expression`) of rows no write has computed + * since the field was declared. + * + * The engine computes such a field on every insert and update, and on nothing + * else, so a field declared on an object that already holds rows is NULL on + * every one of them until each is next written. This command computes each + * through the write's own computation and writes it; the boot names it when + * a sample of rows holds such a NULL. + * + * The command is the shell; the walk, its refusals and its report are + * `@objectstack/objectql`'s `backfillStoredExpressions` — the split + * `os migrate summary-nulls` has with `summary-backfill.ts`. + * + * Dry run by default (writes no row), `--apply` to write. `--changed` also + * rewrites values that differ from the expression — the run to make after an + * expression changed; every run counts them. Rows on which an expression does + * not evaluate are listed; one that holds a value there is skipped, because + * the write would store NULL over it. Idempotent: a re-run with nothing + * listed is the verification. No `sys_migration` flag is recorded: nothing + * gates on the run. + */ +export default class MigrateStoredExpressions extends Command { + static override description = + 'Fill stored computed fields (fields declaring an expression, other than formula fields) on rows no write has ' + + 'computed since the field was declared — they hold NULL until each row is next written. Dry run by default; ' + + '--apply writes, --changed also recomputes values that differ from a changed expression.'; + + static override examples = [ + '$ os migrate stored-expressions', + '$ os migrate stored-expressions --object contract', + '$ os migrate stored-expressions --object contract --field review_sla_days_offset --apply', + '$ os migrate stored-expressions --changed --apply --yes --json', + ]; + + static override flags = { + 'database-url': Flags.string({ + description: 'Database URL to migrate (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + object: Flags.string({ + description: + 'Restrict to this object (repeatable; default: every object declaring a stored computed field). A name the ' + + 'deployment does not declare is refused', + multiple: true, + }), + field: Flags.string({ + description: 'One stored computed field of the one --object (default: every one it declares)', + dependsOn: ['object'], + }), + changed: Flags.boolean({ + description: + 'Also recompute rows whose stored value differs from the expression — after an expression changed. An ' + + 'expression reading the clock (now(), today()) recomputes to the current time', + default: false, + }), + apply: Flags.boolean({ description: 'Write the computed values (default is a read-only dry run)', default: false }), + yes: Flags.boolean({ char: 'y', description: 'Skip the --apply confirmation prompt', default: false }), + force: Flags.boolean({ + description: 'Apply even when another process is using the database (SQLite occupancy check)', + default: false, + }), + 'max-records': Flags.integer({ description: 'Safety bound on rows read per object — exceeding it truncates the walk' }), + json: Flags.boolean({ description: 'Output as JSON (implies non-interactive; requires --yes to apply)' }), + }; + + async run(): Promise { + const { flags } = await this.parse(MigrateStoredExpressions); + const timer = createTimer(); + const apply = flags.apply; + + if (!flags.json) printHeader('Migrate · stored-expressions'); + + // Occupancy gate, as `summary-nulls`: an apply rewrites ROWS, so a second writer on the same + // SQLite file is a real hazard. Probed before boot and before the prompt. + const occupancy = await probeMigrationTarget(flags['database-url']); + if (occupancy.status === 'busy' && apply && !flags.force) { + if (flags.json) { + await emitJson({ + error: 'database_busy', + database: occupancy.filename, + signal: occupancy.signal, + detail: occupancy.detail, + hint: OCCUPANCY_HINT, + }, 0, { compact: true }); + this.exit(1); + return; + } + printError(describeOccupancy(occupancy)); + printWarning(OCCUPANCY_HINT); + this.exit(1); + return; + } + if (occupancy.status === 'busy' && !flags.json) { + printWarning(apply + ? `--force: ${describeOccupancy(occupancy)} Filling anyway — the live process may write rows mid-walk.` + : `${describeOccupancy(occupancy)} The dry run below writes no row or table, but its counts may shift while that process is running. ${JOURNAL_MODE_SWITCH_NOTE}`); + } + if (occupancy.status === 'unknown' && !flags.json) { + printWarning(`Could not check whether the database is in use — ${occupancy.detail}`); + } + + if (apply && !flags.yes) { + if (flags.json || !process.stdin.isTTY) { + if (flags.json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); + this.exit(1); + return; + } + printWarning('Apply mode rewrites record data. Re-run with --yes to confirm, or run without --apply to preview.'); + this.exit(1); + return; + } + const ok = await confirm( + chalk.bold( + `\nCompute and write the stored computed fields ${flags.changed ? 'that are NULL or differ from their expression' : 'still NULL'}` + + `${flags.object ? ` of ${flags.object.join(', ')}` : ''} on this database? [y/N] `, + ), + ); + if (!ok) { + printInfo('Aborted — no changes made.'); + return; + } + } + + if (!flags.json) printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (dry run)…'); + + let stack; + try { + // The dry run boots read-only, as `summary-nulls` does; `--apply` keeps + // the plain boot, because the tables and columns must exist before it writes. + stack = await bootSchemaStack({ + jsonOutput: flags.json, + databaseUrl: flags['database-url'], + extraPlugins: await buildDataMigrationPlugins(), + ...(apply ? {} : { deferSchemaDdl: true, readOnlyProbe: true }), + }); + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + return; + } + + try { + const engine: StoredExpressionBackfillEngine = stack.kernel.getService('objectql'); + if (typeof engine?.computeStoredExpressionsOver !== 'function') { + throw new Error('No ObjectQL engine on this stack — cannot compute stored expressions.'); + } + // An empty walk is indistinguishable from a clean one: refuse when no app metadata is loaded. + const loadedObjects: string[] = + typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []; + if (!loadedObjects.some((name) => !name.startsWith('sys_'))) { + throw new Error( + 'No app objects are loaded, so the walk would examine nothing. ' + + 'Run "os build" in your project root first (the migration reads dist/objectstack.json), then re-run.', + ); + } + refuseUndeclaredObjects(flags.object, loadedObjects); + + const { backfillStoredExpressions, formatStoredExpressionBackfillReport } = await import('@objectstack/objectql'); + + // In JSON mode keep stdout parseable — route warnings to stderr. + const logger = flags.json + ? { info: (m: string) => console.error(m), warn: (m: string) => console.error(m) } + : { info: (m: string) => printInfo(m), warn: (m: string) => printWarning(m) }; + + const report = await backfillStoredExpressions(engine, logger, { + apply, + objects: flags.object, + field: flags.field, + changed: flags.changed, + maxRecordsPerObject: flags['max-records'], + }); + // Not done while a row could not be written, a row was skipped, or the walk was cut short. + const gap = report.failures.length > 0 || report.skippedRows > 0 || report.truncated; + const pending = report.nullRows + (report.changed ? report.staleRows : 0); + + if (flags.json) { + await emitJson({ database: stack.dbLabel, apply, report, duration: timer.elapsed() }); + if (gap) this.exit(1); + return; + } + + printInfo(`Database: ${chalk.white(stack.dbLabel)}`); + console.log(''); + // The report's dry-run line claims no row or table; on a SQLite file the read-only boot's + // connect may still have switched the journal mode (its header, no row), so the note follows. + const journalNote = !report.applied && pending > 0 + && describeDriverSqliteFile(stack.driver?.config) !== undefined + ? ` ${JOURNAL_MODE_SWITCH_NOTE}` + : ''; + console.log(`${formatStoredExpressionBackfillReport(report).join('\n')}${journalNote}`); + console.log(''); + + if (report.failures.length > 0) { + printError(`${report.failures.length} row(s) could not be written — re-run to finish them.`); + } else if (report.skippedRows > 0) { + printError(`${report.skippedRows} row(s) were listed and not written — fix each one listed above, then re-run.`); + } else if (report.truncated) { + printError('The walk stopped at --max-records: rows past it were not looked at. Re-run with a higher --max-records.'); + } else if (apply && report.writtenRows > 0) { + printSuccess(`Wrote ${report.writtenRows} row(s). Re-run any time — it only rewrites rows that still differ.`); + } else if (!apply && pending > 0) { + printInfo(`Dry run only — ${pending} value(s) would be written. Re-run with --apply.`); + } else { + printSuccess('Nothing to fill — every stored computed field already holds its expression\'s value.'); + } + console.log(chalk.dim(` ${timer.display()}`)); + console.log(''); + if (gap) this.exit(1); + } catch (error: any) { + if (isExitSignal(error)) throw error; + if (flags.json) { await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); this.exit(1); } + // `INVALID_FIELD` is the run refusing an object or field before its walk ("Refused before + // the walk read or wrote any row."). The boot ran first, so on a SQLite file the note follows. + const journalNote = error?.code === 'INVALID_FIELD' + && describeDriverSqliteFile(stack.driver?.config) !== undefined + ? ` ${JOURNAL_MODE_SWITCH_NOTE}` + : ''; + printError(`${error.message || String(error)}${journalNote}`); + this.exit(1); + } finally { + await stack.shutdown(); + } + } +} diff --git a/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts b/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts index 9e82ff51aa5..9ce0f4d71f5 100644 --- a/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts +++ b/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts @@ -81,6 +81,7 @@ import MigrateRecordCurrencyCodes from '../commands/migrate/record-currency-code import MigrateRecordedBy from '../commands/migrate/recorded-by.js'; import MigrateResume from '../commands/migrate/resume.js'; import MigrateSecurityCatalogOverlays from '../commands/migrate/security-catalog-overlays.js'; +import MigrateStoredExpressions from '../commands/migrate/stored-expressions.js'; import MigrateSummaryNulls from '../commands/migrate/summary-nulls.js'; import MigrateUnmappedColumns from '../commands/migrate/unmapped-columns.js'; import MigrateValueShapes from '../commands/migrate/value-shapes.js'; @@ -344,6 +345,13 @@ const CALLERS: Record = { argv: ['--apply', '--yes', '--database-url', '@DB@', '--json'], }], }, + 'commands/migrate/stored-expressions.ts': { + run: invoke(MigrateStoredExpressions), + // The fixture's app declares no stored computed field, so the walk finds + // nothing and `--apply` writes nothing — the boot is what these cases are about. + noWrite: [{ label: 'migrate stored-expressions', argv: ['--database-url', '@DB@', '--json'] }], + write: [{ label: 'migrate stored-expressions --apply', argv: ['--apply', '--yes', '--database-url', '@DB@', '--json'] }], + }, 'commands/migrate/summary-nulls.ts': { run: invoke(MigrateSummaryNulls), noWrite: [{ label: 'migrate summary-nulls', argv: ['--database-url', '@DB@', '--json'] }], diff --git a/packages/cli/test/json-stdout-purity.e2e.test.ts b/packages/cli/test/json-stdout-purity.e2e.test.ts index ce875583760..8c2123af960 100644 --- a/packages/cli/test/json-stdout-purity.e2e.test.ts +++ b/packages/cli/test/json-stdout-purity.e2e.test.ts @@ -168,6 +168,8 @@ const FAMILY: Record = { // `sys_metadata` hydration, so it deletes nothing. No table exists in the // fixture, so the face driven here is its empty listing. 'migrate security-catalog-overlays': [], + // Its dry run takes the `summary-nulls` shape: a read-only boot, no row written. + 'migrate stored-expressions': [], 'migrate summary-nulls': [], // One object per run, so it needs a name: the fixture's own `jp_ticket`, // which the composed host config registers. No database file exists, so the diff --git a/packages/cli/test/migrate-confirmation-exit.pin.test.ts b/packages/cli/test/migrate-confirmation-exit.pin.test.ts index fa9abfd7049..dc17b1433d4 100644 --- a/packages/cli/test/migrate-confirmation-exit.pin.test.ts +++ b/packages/cli/test/migrate-confirmation-exit.pin.test.ts @@ -290,6 +290,7 @@ const ROSTER: Record = { writes: BOOT, }, 'security-catalog-overlays': { argv: ['--apply'], confirm: '--yes', writes: BOOT }, + 'stored-expressions': { argv: ['--apply'], confirm: '--yes', writes: BOOT }, 'summary-nulls': { argv: ['--apply'], confirm: '--yes', writes: BOOT }, 'value-shapes': { argv: ['--apply'], confirm: '--yes', writes: BOOT }, }; diff --git a/packages/objectql/src/engine-security-resolvers.ts b/packages/objectql/src/engine-security-resolvers.ts index 9935ee3151c..54f6644ea8d 100644 --- a/packages/objectql/src/engine-security-resolvers.ts +++ b/packages/objectql/src/engine-security-resolvers.ts @@ -42,8 +42,8 @@ import type { ComposableReadableIdsSite } from './filter-comparand-shape.js'; import { isStoredMetadataBodyObject } from '@objectstack/spec/kernel'; import type { ExecutionContext } from '@objectstack/spec/kernel'; import type { EvalOptionLabels } from '@objectstack/formula'; -import { readerOptionLabels } from './formula-option-labels.js'; -import { applyFormulaPlan, planFormulaProjection, undeclaredRelatedFields } from './engine-top-level.js'; +import { declaredOptionLabels, readerOptionLabels } from './formula-option-labels.js'; +import { applyFormulaPlan, computeStoredExpressions, planFormulaProjection, planStoredExpressions, undeclaredRelatedFields } from './engine-top-level.js'; import type { FieldOwnedFileReadAuthorizer, FormulaFaultSink, FormulaPlanEntry, FormulaRelated, HeldFileResolver, ReadableIdsChain, WriteWallVerdict, @@ -343,6 +343,18 @@ export class SecurityResolversFacet { return scratch[field]; } + /** The body of `ObjectQL#computeStoredExpressionsOver` (#29961); the method's docblock there is its contract. */ + static async computeStoredExpressionsOver(this: Pick, object: string, rows: ReadonlyArray>, context?: ExecutionContext, onFault?: (field: string, message: string) => void): Promise>> { + const schema = this._registry.getObject(object); + const resolution = this.permissionResolution(context); + return computeStoredExpressions(planStoredExpressions(schema), rows, context, { + permissionsFor: (plan, records) => this.resolveFormulaPermissions(object, plan, records, context, resolution), + onFault: onFault ? (field, error) => onFault(field, error.message) : this.formulaFaultSink(object), + relatedFor: (plan, records) => this.resolveFormulaRelated(schema, plan, records, context), + optionLabelsFor: (plan) => declaredOptionLabels(schema, plan), + }); + } + /** * [#26608] The option labels `label(record.)` reads on a read or a * write response of `schema`'s object: in the reader's locale diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index e7f68d2a7da..67f26ec1f4c 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -2101,6 +2101,28 @@ export class ObjectQL implements IObjectQLEngine { return SecurityResolversFacet.serveFormulaField.call(securityResolversHost.of(this), objectName, record, field, context); } + /** + * [#29961] The values `object`'s stored computed fields ({@link planStoredExpressions}) + * take over rows ALREADY stored, computed and served exactly as a write + * computes them: the declared option labels, the one hop, and the `can()` + * map, under `context`. Index-aligned with `rows`, keyed by every stored + * computed field the object declares; evaluated over copies, writes nothing. + * + * For a row that never went through a write since its field was declared — + * `os migrate stored-expressions` (`stored-expression-backfill.ts`) and the + * boot advisory it shares. A field that does not evaluate computes `null`, + * the write's answer, and is told to `onFault` when given (else the engine's + * once-per-field report). + */ + computeStoredExpressionsOver( + object: string, + rows: ReadonlyArray>, + context?: ExecutionContext, + onFault?: (field: string, message: string) => void, + ): Promise>> { + return SecurityResolversFacet.computeStoredExpressionsOver.call(securityResolversHost.of(this), object, rows, context, onFault); + } + private formulaOptionLabels( schema: unknown, plan: FormulaPlanEntry[], diff --git a/packages/objectql/src/federated-injected-column-readers.test.ts b/packages/objectql/src/federated-injected-column-readers.test.ts index 337d869d99b..b3075db55a3 100644 --- a/packages/objectql/src/federated-injected-column-readers.test.ts +++ b/packages/objectql/src/federated-injected-column-readers.test.ts @@ -377,6 +377,14 @@ const READERS: Record = { disposition: 'skips', why: 'the recompute reads the stamp its apply keeps, and leaves the column out of its read (and its write) on a federated object', }, + 'stored-expression-backfill.ts#backfillStoredExpressions :: isFederatedUnprovisionedInjectedColumn()': { + disposition: 'skips', + why: 'the backfill never sends an injected column a federated object does not provision', + }, + 'stored-expression-backfill.ts#backfillStoredExpressions :: updated_at': { + disposition: 'skips', + why: 'the backfill keeps the stamp of the row it writes, and sends none on a federated object', + }, 'engine-migration-state.ts#recordObservedDeviation :: updated_at': { disposition: 'not-a-read', why: PLATFORM_ROW }, 'engine-migration-state.ts#retractCreationAttestation :: updated_at': { disposition: 'not-a-read', why: PLATFORM_ROW }, 'plugin.ts#reconcileFederatedBindings :: isFederatedObject()': { disposition: 'not-a-read', why: SYNC_ROUTE }, @@ -703,7 +711,7 @@ describe('[#21918] every engine reader of an injected column has a disposition t } }, SCAN_BUDGET_MS); - it('the sites that skip are exactly the cascade scan, its plan, the organization-delete count, the lifecycle tenant partition, and the home-amount recompute', () => { + it('the sites that skip are exactly the cascade scan, its plan, the organization-delete count, the lifecycle tenant partition, the home-amount recompute and the stored-expression backfill', () => { const skipping = new Set( Object.entries(READERS) .filter(([, row]) => row.disposition === 'skips') @@ -715,6 +723,7 @@ describe('[#21918] every engine reader of an injected column has a disposition t 'engine-cascade-planning.ts#planCascadeAtomicity', 'home-amount-recompute.ts#recomputeHomeAmounts', 'lifecycle/lifecycle-service.ts#tenantWindowsFor', + 'stored-expression-backfill.ts#backfillStoredExpressions', ]); }); diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts index 237f6a00e84..526c646892f 100644 --- a/packages/objectql/src/index.ts +++ b/packages/objectql/src/index.ts @@ -531,6 +531,26 @@ export type { SummaryBackfillLogger, SummaryBackfillOptions, } from './summary-backfill.js'; +// [#29961] The backfill behind `os migrate stored-expressions`: stored computed +// fields over rows no write has computed since the field was declared, and the +// boot advisory naming it. Same split as the roll-up backfill above. +export { + backfillStoredExpressions, + formatStoredExpressionBackfillReport, + storedExpressionBackfillComplete, + announceStoredExpressionHoles, + sampleStoredExpressionHoles, +} from './stored-expression-backfill.js'; +export type { + StoredExpressionBackfillReport, + StoredExpressionBackfillFieldOutcome, + StoredExpressionBackfillFailure, + StoredExpressionFault, + StoredExpressionBackfillEngine, + StoredExpressionBackfillLogger, + StoredExpressionBackfillOptions, + StoredExpressionHoleSample, +} from './stored-expression-backfill.js'; // [ADR-0144, ruling 6492000994] The explicit code backfill behind // `os migrate record-currency-codes`: rows stored before an amount switched to // `currencyMode: 'record'`. Same split as the roll-up backfill above. diff --git a/packages/objectql/src/plugin.ts b/packages/objectql/src/plugin.ts index 0f1a979df2f..57b67264c79 100644 --- a/packages/objectql/src/plugin.ts +++ b/packages/objectql/src/plugin.ts @@ -34,6 +34,7 @@ import { applyProtection } from '@objectstack/spec/shared'; import { isFederatedObject } from './federated-object.js'; // [#22070] The builtin audit stamps are wrapped like a bound hook but registered in code. import { wrapDeclarativeHook } from './hook-wrappers.js'; +import { announceStoredExpressionHoles } from './stored-expression-backfill.js'; import { LifecycleService } from './lifecycle/lifecycle-service.js'; import { lifecycleSettingsManifest } from './lifecycle/lifecycle-settings.js'; import type { DanglingReferenceAuditOptions } from './integrity/dangling-reference-audit.js'; @@ -1196,6 +1197,13 @@ export class ObjectQLPlugin implements Plugin { ); if (unservable) throw unservable; await this.ql?.announceOpenMigrationGates(); + // [#29961] A stored computed field declared on an object that already + // held rows is NULL on them until each is next written: name the + // fields and `os migrate stored-expressions`. Here, beside the gate + // advisory, for its reason: schema sync has run and every object is + // registered. A bounded sample read that writes nothing and never fails + // the boot (`stored-expression-backfill.ts`). + if (this.ql) await announceStoredExpressionHoles(this.ql, ctx.logger); }); ctx.hook('metadata:reloaded', async (payload?: unknown) => { await this.resyncAuthoredHooks(ctx); diff --git a/packages/objectql/src/stored-expression-backfill.test.ts b/packages/objectql/src/stored-expression-backfill.test.ts new file mode 100644 index 00000000000..d3117d6847d --- /dev/null +++ b/packages/objectql/src/stored-expression-backfill.test.ts @@ -0,0 +1,293 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// #29961 — `os migrate stored-expressions`: a stored computed field declared +// on an object that already holds rows is NULL on every one of them, because +// the engine computes it on writes only. +// +// Every fixture builds the state the run exists for: rows written STRAIGHT +// INTO THE DRIVER STORE, as an upgraded database holds them — stored before +// the field was declared, never written since. Inserting them through the +// engine would compute the field and leave nothing to fill. +// +// The engine is the real one, over an in-file driver; the values compared are +// the ones a write of the same row stores. + +import { describe, it, expect, beforeEach } from 'vitest'; +import { ObjectQL } from './engine.js'; +import { + announceStoredExpressionHoles, + backfillStoredExpressions, + formatStoredExpressionBackfillReport, + sameStoredValue, + storedExpressionBackfillComplete, +} from './stored-expression-backfill.js'; + +function makeDriver() { + const stores = new Map>(); + const writes: Array<{ object: string; id: string; data: Record }> = []; + const storeFor = (o: string) => { + let s = stores.get(o); + if (!s) { s = new Map(); stores.set(o, s); } + return s; + }; + // Implicit equality, the comparison operators the keyset walk emits (`$gt` on `id`), and `$in`. + const checkOp = (value: any, cond: any): boolean => { + if (cond === null || typeof cond !== 'object' || Array.isArray(cond) || cond instanceof Date) return value === cond; + return Object.entries(cond).every(([op, target]: [string, any]) => { + switch (op) { + case '$eq': return value === target; + case '$ne': return value !== target; + case '$gt': return value > target; + case '$in': return Array.isArray(target) && target.includes(value); + default: return true; + } + }); + }; + const matches = (row: any, where: any): boolean => { + if (!where || typeof where !== 'object') return true; + return Object.entries(where).every(([k, v]: [string, any]) => { + if (k === '$and') return (v as any[]).every((w) => matches(row, w)); + if (k === '$or') return (v as any[]).some((w) => matches(row, w)); + return checkOp(row?.[k], v); + }); + }; + let failFind: string | null = null; + let failUpdateFor: string | null = null; + const driver: any = { + name: 'memory', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; }, + async find(object: string, ast: any) { + if (failFind === object) throw new Error('driver cannot read this table'); + const rows = Array.from(storeFor(object).values()).filter((r) => matches(r, ast?.where)); + rows.sort((a, b) => String(a.id).localeCompare(String(b.id))); + return typeof ast?.limit === 'number' ? rows.slice(0, ast.limit) : rows; + }, + async findOne(object: string, ast: any) { + for (const r of storeFor(object).values()) if (matches(r, ast?.where)) return r; + return null; + }, + async create(object: string, data: Record) { + const row = { ...data, id: data.id as string }; + storeFor(object).set(row.id, row); + return row; + }, + async update(object: string, id: string, data: Record) { + if (failUpdateFor === id) throw new Error('driver refused this row'); + writes.push({ object, id, data }); + const s = storeFor(object); + const row = { ...s.get(id), ...data, id }; + s.set(id, row); + return row; + }, + async delete(object: string, id: string) { return storeFor(object).delete(id); }, + async count() { return 0; }, + async bulkCreate() { return []; }, async bulkUpdate() { return []; }, async bulkDelete() {}, + async beginTransaction() { return { __trx: true, commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { + driver, + storeFor, + writes, + failFind: (object: string | null) => { failFind = object; }, + failUpdateFor: (id: string | null) => { failUpdateFor = id; }, + }; +} + +const quietLogger = { info: () => {}, warn: () => {} }; +const cel = (source: string) => ({ dialect: 'cel', source }); + +describe('backfillStoredExpressions — stored computed fields on rows stored before the field (#29961)', () => { + let engine: ObjectQL; + let d: ReturnType; + const contract = (id: string) => d.storeFor('contract').get(id); + + beforeEach(async () => { + engine = new ObjectQL(); + d = makeDriver(); + engine.registerDriver(d.driver, true); + await engine.init(); + engine.registry.registerObject({ + name: 'contract_type', + fields: { name: { type: 'text' }, review_sla_days: { type: 'number' } }, + } as any); + engine.registry.registerObject({ + name: 'contract', + fields: { + name: { type: 'text' }, + qty: { type: 'number' }, + contract_type: { type: 'lookup', reference: 'contract_type' }, + updated_at: { type: 'datetime' }, + // The hotclm shape: an offset read through a lookup. + review_sla_days_offset: { type: 'number', expression: cel('0 - record.contract_type.review_sla_days') }, + qty_double: { type: 'number', expression: cel('record.qty * 2') }, + // Read on every read; it has no column and is never a backfill's. + qty_label: { type: 'formula', expression: cel("string(record.qty) + ' pcs'") }, + }, + } as any); + engine.registry.registerObject({ name: 'note', fields: { body: { type: 'text' } } } as any); + d.storeFor('contract_type').set('t1', { id: 't1', name: 'NDA', review_sla_days: 5 }); + // Stored before the two fields were declared: neither column holds a value. + d.storeFor('contract').set('c1', { id: 'c1', name: 'c1', qty: 1, contract_type: 't1', updated_at: '2026-01-01T00:00:00.000Z' }); + d.storeFor('contract').set('c2', { id: 'c2', name: 'c2', qty: 2, contract_type: 't1', updated_at: '2026-01-02T00:00:00.000Z' }); + // No contract type: the offset's own value is NULL here (the hop has nothing to read). + d.storeFor('contract').set('c3', { id: 'c3', name: 'c3', qty: 3, contract_type: null, updated_at: '2026-01-03T00:00:00.000Z' }); + }); + + it('a dry run counts each NULL the expression fills and writes nothing', async () => { + const report = await backfillStoredExpressions(engine, quietLogger); + + expect(report.applied).toBe(false); + expect(report.scannedObjects).toEqual(['contract']); + expect(report.scannedRecords).toBe(3); + const byField = Object.fromEntries(report.fields.map((f) => [f.field, f])); + expect(byField.qty_double).toMatchObject({ nullRows: 3, written: 0 }); + // c3's NULL is the expression's own value (an empty reference), not a hole. + expect(byField.review_sla_days_offset).toMatchObject({ nullRows: 2, faultedRows: 1, written: 0 }); + expect(byField.review_sla_days_offset.sampleRecordIds).toEqual(['c1', 'c2']); + expect(report.fields.map((f) => f.field)).not.toContain('qty_label'); + expect(report.nullRows).toBe(5); + expect(d.writes).toEqual([]); + expect(contract('c1').qty_double).toBeUndefined(); + expect(formatStoredExpressionBackfillReport(report).at(-1)).toMatch(/^Dry run — no row or table was written\./); + expect(storedExpressionBackfillComplete(report)).toBe(false); + }); + + it('--apply fills each hole with what a write stores, keeps updated_at, and a re-run finds nothing', async () => { + const report = await backfillStoredExpressions(engine, quietLogger, { apply: true }); + + expect(report.writtenRows).toBe(3); + expect(report.failures).toEqual([]); + expect(contract('c1')).toMatchObject({ qty_double: 2, review_sla_days_offset: -5, updated_at: '2026-01-01T00:00:00.000Z' }); + expect(contract('c2')).toMatchObject({ qty_double: 4, review_sla_days_offset: -5, updated_at: '2026-01-02T00:00:00.000Z' }); + expect(contract('c3').qty_double).toBe(6); + expect(contract('c3').review_sla_days_offset ?? null).toBeNull(); + expect(storedExpressionBackfillComplete(report)).toBe(true); + + const again = await backfillStoredExpressions(engine, quietLogger); + expect(again.nullRows).toBe(0); + expect(again.staleRows).toBe(0); + expect(storedExpressionBackfillComplete(again)).toBe(true); + }); + + it('a value the expression no longer gives is counted on every run and rewritten only under changed', async () => { + await backfillStoredExpressions(engine, quietLogger, { apply: true }); + engine.registry.registerObject({ + name: 'contract', + fields: { ...(engine.getObject('contract') as any).fields, qty_double: { type: 'number', expression: cel('record.qty * 3') } }, + } as any); + d.writes.length = 0; + + const plain = await backfillStoredExpressions(engine, quietLogger, { apply: true }); + expect(plain.staleRows).toBe(3); + expect(plain.writtenRows).toBe(0); + expect(contract('c1').qty_double).toBe(2); + expect(formatStoredExpressionBackfillReport(plain).join('\n')).toContain('re-run with --changed'); + + const changed = await backfillStoredExpressions(engine, quietLogger, { apply: true, changed: true }); + expect(changed.writtenRows).toBe(3); + expect([contract('c1').qty_double, contract('c2').qty_double, contract('c3').qty_double]).toEqual([3, 6, 9]); + expect((await backfillStoredExpressions(engine, quietLogger, { changed: true })).staleRows).toBe(0); + }); + + it('a row holding a value an expression can no longer compute is listed and never written over', async () => { + // c3 holds an offset, but has no contract type: the write would store NULL over it. + d.storeFor('contract').set('c3', { ...contract('c3'), review_sla_days_offset: -9 }); + + const report = await backfillStoredExpressions(engine, quietLogger, { apply: true }); + + expect(report.skippedRows).toBe(1); + expect(report.faults).toEqual([ + expect.objectContaining({ object: 'contract', field: 'review_sla_days_offset', recordId: 'c3', blocksWrite: true }), + ]); + expect(contract('c3')).toMatchObject({ review_sla_days_offset: -9 }); + expect(contract('c3').qty_double).toBeUndefined(); + expect(contract('c1').qty_double).toBe(2); + expect(storedExpressionBackfillComplete(report)).toBe(false); + }); + + it('one row failing to write does not stop the run, and is listed', async () => { + d.failUpdateFor('c2'); + const report = await backfillStoredExpressions(engine, quietLogger, { apply: true, retry: { maxRetries: 1 } }); + + expect(report.failures).toEqual([expect.objectContaining({ object: 'contract', recordId: 'c2' })]); + expect(report.writtenRows).toBe(2); + expect(contract('c1').qty_double).toBe(2); + expect(contract('c3').qty_double).toBe(6); + expect(storedExpressionBackfillComplete(report)).toBe(false); + }); + + it('field narrows the walk to one stored computed field', async () => { + const report = await backfillStoredExpressions(engine, quietLogger, { objects: ['contract'], field: 'qty_double' }); + expect(report.fields.map((f) => f.field)).toEqual(['qty_double']); + expect(report.nullRows).toBe(3); + }); + + it('refuses, before any row is read, a scope it cannot act on', async () => { + const refused = (options: Parameters[2]) => + backfillStoredExpressions(engine, quietLogger, options).then(() => null, (e) => e); + + expect(await refused({ objects: ['nope'] })).toMatchObject({ code: 'OBJECT_NOT_FOUND' }); + expect(await refused({ objects: ['note'] })).toMatchObject({ code: 'INVALID_FIELD', status: 400, httpStatus: 400 }); + expect(await refused({ objects: ['contract'], field: 'qty_label' })) + .toMatchObject({ code: 'INVALID_FIELD', status: 400, field: 'qty_label' }); + expect(await refused({ field: 'qty_double' })).toMatchObject({ code: 'INVALID_FIELD', status: 400 }); + expect(d.writes).toEqual([]); + }); + + it('an object it cannot read is reported unreadable and truncates the run', async () => { + d.failFind('contract'); + const report = await backfillStoredExpressions(engine, quietLogger); + expect(report.unreadableObjects).toEqual(['contract']); + expect(report.truncated).toBe(true); + expect(storedExpressionBackfillComplete(report)).toBe(false); + }); + + describe('the boot advisory', () => { + const capture = () => { + const warned: string[] = []; + return { warned, logger: { info: () => {}, warn: (m: string) => warned.push(m), debug: () => {} } }; + }; + + it('names each field holding a hole and the command, once', async () => { + const { warned, logger } = capture(); + const samples = await announceStoredExpressionHoles(engine, logger); + + expect(samples).toEqual([{ object: 'contract', fields: { review_sla_days_offset: 2, qty_double: 3 }, sampled: 3 }]); + expect(warned).toHaveLength(1); + expect(warned[0]).toContain('contract.qty_double (3 of the first 3 rows)'); + expect(warned[0]).toContain('contract.review_sla_days_offset (2 of the first 3 rows)'); + expect(warned[0]).toContain('os migrate stored-expressions'); + expect(d.writes).toEqual([]); + }); + + it('is silent once the holes are filled', async () => { + await backfillStoredExpressions(engine, quietLogger, { apply: true }); + const { warned, logger } = capture(); + expect(await announceStoredExpressionHoles(engine, logger)).toEqual([]); + expect(warned).toEqual([]); + }); + + it('never fails the boot over a table it cannot read', async () => { + d.failFind('contract'); + const { warned, logger } = capture(); + await expect(announceStoredExpressionHoles(engine, logger)).resolves.toEqual([]); + expect(warned).toEqual([]); + }); + }); +}); + +describe('sameStoredValue — a driver\'s spelling of the value a write stored', () => { + it.each([ + [4, 4, true], ['4', 4, true], ['4.50', 4.5, true], [5, 4, false], + [1, true, true], [0, false, true], ['1', true, true], [1, false, false], + ['c1-x', 'c1-x', true], ['c1', 'c1-x', false], + ['2026-01-02T00:00:00.000Z', new Date('2026-01-02T00:00:00.000Z'), true], + ['2026-01-02', new Date('2026-01-02T00:00:00.000Z'), true], + ['2026-01-03', new Date('2026-01-02T00:00:00.000Z'), false], + ['["a","b"]', ['a', 'b'], true], + [null, null, true], [undefined, null, true], [null, 0, false], [0, null, false], + ])('%j vs %j → %s', (stored, computed, same) => { + expect(sameStoredValue(stored, computed)).toBe(same); + }); +}); diff --git a/packages/objectql/src/stored-expression-backfill.ts b/packages/objectql/src/stored-expression-backfill.ts new file mode 100644 index 00000000000..f2d980dccbb --- /dev/null +++ b/packages/objectql/src/stored-expression-backfill.ts @@ -0,0 +1,590 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The backfill of stored computed fields over rows that never went through a + * write since the field was declared — `os migrate stored-expressions` — and + * the boot advisory that names it. + * + * ## The hole this fills + * + * A stored computed field (a non-`formula` field declaring `expression`) is + * computed by the engine on every insert and update, and on nothing else + * (`computeStoredExpressions`, the write path). Declare one on an object that + * already holds rows and schema sync adds the column, NULL on every row: no + * write has computed it, and none will until each row is next written. Every + * reader of the column — a filter, a sort, a roll-up, an analytics anchor + * reading a NULL offset as 0 — answers from a value the declaration never + * produced, with no error. A fresh database is right, an upgraded one is not; + * this run removes that asymmetry, the way `os migrate summary-nulls` does for + * roll-ups. + * + * ## The write's own computation, never a copy + * + * Each value comes from {@link StoredExpressionBackfillEngine.computeStoredExpressionsOver}: + * the engine's plan and serving for a write (option labels, the one hop, the + * `can()` map), over the row as stored. The apply writes through + * `engine.update` with the computed values in the payload; the write computes + * every stored field of the row again over the stored row, so a filled row + * holds exactly what a fresh write of it would store. + * + * ## Which rows it writes + * + * By default, the rows holding a NULL the expression gives a value — the hole. + * A NULL the expression itself answers (an empty reference, a null input) is + * already the write's value and is left alone. With `changed`, also every row + * whose stored value differs from the expression over it — the run to make + * after an expression is CHANGED, which no boot can detect; the report counts + * such rows on every run (`staleRows`), which is how an operator learns to ask + * for it. ⚠️ An expression reading the clock (`now()`, `today()`) recomputes + * to the current time, so under `changed` every row of it differs. + * + * Because the apply IS a write, it stores every stored field the write + * computes for that row, so a stale value on a row this run fills is + * rewritten with it (counted under `written`). A row on which a stored field + * does not evaluate while it holds a value is listed and NOT written: the + * write would replace that value with the fault's `null`. + * + * ## Writes + * + * Dry run unless `apply`; a dry run writes nothing. An apply writes under + * `preserveAudit` with the row's stored `updated_at`, as the other `os migrate` + * data runs that rewrite a computed column do: `updated_at` does not move. + * Hooks, roll-ups over the column and the audit writers see each write as any + * write. One row's failure does not stop the run; it is listed. Idempotent: a + * second run finds nothing to fill. No `sys_migration` flag is recorded: + * nothing gates on the run (the `summary-nulls` reasoning). + * + * ## The boot advisory + * + * {@link announceStoredExpressionHoles} reads the first page of each object + * declaring a stored computed field and, when that sample holds a hole, warns + * once naming the fields and this command. Bounded on purpose — one read of + * at most {@link ADVISORY_SAMPLE_SIZE} rows per such object, and nothing for + * an object declaring none — so it never scans a large table at boot; a + * column added to an object with rows is NULL on every row, so the first page + * finds it. It writes nothing: rewriting data at boot fires hooks and audit + * rows nobody ordered, which is why the backfill is a command. + */ + +import { keysetWalk, type KeysetPageQuery } from '@objectstack/types'; +import { objectNotFoundError, withTransientRetry, type RetryOptions } from '@objectstack/core'; + +import { planStoredExpressions } from './engine-top-level.js'; +import { isFederatedUnprovisionedInjectedColumn } from './federated-object.js'; + +/** The engine surface the run needs: the declarations, a read, the stored-expression computation and the update. */ +export interface StoredExpressionBackfillEngine { + getObject(name: string): { fields?: Record } | undefined; + getConfigs?(): Record; + find(object: string, options: Record): Promise>>; + update(object: string, data: Record, options?: any): Promise; + computeStoredExpressionsOver( + object: string, + rows: ReadonlyArray>, + context?: any, + onFault?: (field: string, message: string) => void, + ): Promise>>; +} + +export interface StoredExpressionBackfillLogger { + info(msg: string, meta?: unknown): void; + warn(msg: string, meta?: unknown): void; +} + +export interface StoredExpressionBackfillOptions { + /** Restrict to these objects (default: every object declaring a stored computed field). */ + objects?: string[]; + /** One stored computed field of the ONE object named in `objects` (default: every one it declares). */ + field?: string; + /** Also rewrite rows whose stored (non-NULL) value differs from the expression — after an expression changed. */ + changed?: boolean; + /** Write the computed values. Omit for a read-only dry run. */ + apply?: boolean; + /** Safety bound on rows read per object. */ + maxRecordsPerObject?: number; + /** How many faulted rows and failures the report lists (each); the counts are always whole. */ + maxListed?: number; + /** Retry policy for one row's update; defaults to the transient defaults. */ + retry?: RetryOptions; +} + +/** One stored computed field the run examined. */ +export interface StoredExpressionBackfillFieldOutcome { + object: string; + field: string; + /** Rows storing NULL where the expression gives a value — the hole. */ + nullRows: number; + /** Rows storing a value that differs from the expression (rewritten only under `changed`, or on a filled row). */ + staleRows: number; + /** Rows on which the expression does not evaluate (it computes `null` there, as on a write). */ + faultedRows: number; + /** Values written (an apply run). */ + written: number; + /** A few record ids with a hole, so an operator can go and look. */ + sampleRecordIds: string[]; +} + +/** A row on which a stored computed field does not evaluate. */ +export interface StoredExpressionFault { + object: string; + field: string; + recordId: string; + error: string; + /** True when the row holds a value there, so the row is not written (the write would store `null`). */ + blocksWrite: boolean; +} + +/** A row whose write failed after retries; the run continues past it. */ +export interface StoredExpressionBackfillFailure { + object: string; + recordId: string; + error: string; +} + +export interface StoredExpressionBackfillReport { + /** Objects walked — those declaring a stored computed field in scope. */ + scannedObjects: string[]; + scannedRecords: number; + /** Every field walked, in declaration order per object. */ + fields: StoredExpressionBackfillFieldOutcome[]; + nullRows: number; + staleRows: number; + faultedRows: number; + /** Rows a write would change but which hold a value a faulting field would erase: listed, never written. */ + skippedRows: number; + /** Rows written (an apply run). */ + writtenRows: number; + /** The scope: NULL holes only (false), or every stale value too (true). */ + changed: boolean; + applied: boolean; + /** A per-object cap or an unreadable object cut the walk short. */ + truncated: boolean; + unreadableObjects: string[]; + faults: StoredExpressionFault[]; + faultsTruncated: boolean; + failures: StoredExpressionBackfillFailure[]; +} + +const SCAN_PAGE_SIZE = 200; +const DEFAULT_MAX_LISTED = 50; +const MAX_SAMPLE_IDS = 5; +/** How many rows of each object the boot advisory reads. */ +export const ADVISORY_SAMPLE_SIZE = 100; +/** Unscoped by design, the context every `os migrate` data run reads with: every organization's rows. */ +const SYSTEM_CTX = { isSystem: true } as const; +/** The write keeps the row's own `updated_at`: the run changes only computed columns. */ +const SYSTEM_PRESERVING_CTX = { isSystem: true, preserveAudit: true } as const; + +type RefusalError = Error & { code: string; status: number; httpStatus: number; field?: string }; + +function invalidField(message: string, field?: string): RefusalError { + const err = new Error(message) as RefusalError; + err.code = 'INVALID_FIELD'; + err.status = 400; + // …and `httpStatus`, the same number under ADR-0112 D5's spelling (the CLI `--json` envelope reads it). + err.httpStatus = 400; + if (field !== undefined) err.field = field; + return err; +} + +/** The stored computed fields `object` declares, by the write's own predicate. */ +function storedExpressionFields(engine: Pick, object: string): string[] { + return planStoredExpressions(engine.getObject(object)).map((entry) => entry.name); +} + +/** + * The fields to walk per object, judged before any row is read: an unknown + * object (`OBJECT_NOT_FOUND`), a named object declaring no stored computed + * field, a `field` without exactly one object, and a `field` that is not one + * of its stored computed fields (`INVALID_FIELD`) are refused as a whole. + */ +function resolveScope(engine: StoredExpressionBackfillEngine, options: StoredExpressionBackfillOptions): Map { + const scope = new Map(); + const named = options.objects; + if (options.field !== undefined && named?.length !== 1) { + throw invalidField( + `[stored-expression-backfill] field '${options.field}' names a field of ONE object: give exactly one object with it. ` + + 'Refused before the walk read or wrote any row.', + options.field, + ); + } + if (!named) { + const all = typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []; + for (const object of all) { + const fields = storedExpressionFields(engine, object); + if (fields.length > 0) scope.set(object, fields); + } + return scope; + } + for (const object of named) { + if (!engine.getObject(object)) throw objectNotFoundError(object); + const fields = storedExpressionFields(engine, object); + if (fields.length === 0) { + throw invalidField( + `[stored-expression-backfill] '${object}' declares no stored computed field: none of its fields other than ` + + '`formula` ones declares `expression`, so it stores no computed value to fill. ' + + 'Refused before the walk read or wrote any row.', + ); + } + if (options.field !== undefined && !fields.includes(options.field)) { + throw invalidField( + `[stored-expression-backfill] '${object}.${options.field}' is not a stored computed field. The ones '${object}' ` + + `declares: ${fields.map((f) => `'${f}'`).join(', ')}. Refused before the walk read or wrote any row.`, + options.field, + ); + } + scope.set(object, options.field !== undefined ? [options.field] : fields); + } + return scope; +} + +const isBlank = (value: unknown): boolean => value === null || value === undefined; + +/** + * Does a stored value spell what the expression computes? A driver hands back + * its own spelling of the value the write stored — a number as a decimal + * string, a boolean as 0/1, a date as text — so a value is compared as the + * type the computation produced, never by identity alone. + */ +export function sameStoredValue(stored: unknown, computed: unknown): boolean { + if (isBlank(stored) || isBlank(computed)) return isBlank(stored) && isBlank(computed); + if (Object.is(stored, computed)) return true; + if (typeof computed === 'number' || typeof computed === 'bigint') { + const s = typeof stored === 'string' && stored.trim() !== '' ? Number(stored) : stored; + return typeof s === 'number' && Number(computed) === s; + } + if (typeof computed === 'boolean') { + if (stored === 1 || stored === '1' || stored === 'true') return computed; + if (stored === 0 || stored === '0' || stored === 'false') return !computed; + return false; + } + if (computed instanceof Date) { + const ms = stored instanceof Date ? stored.getTime() : Date.parse(String(stored)); + if (ms === computed.getTime()) return true; + // A `date` column stores the calendar day alone. + return typeof stored === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(stored) && computed.toISOString().slice(0, 10) === stored; + } + if (typeof computed === 'object' || typeof stored === 'object') { + const s = typeof stored === 'string' ? safeParse(stored) : stored; + return JSON.stringify(s) === JSON.stringify(computed); + } + return String(stored) === String(computed); +} + +function safeParse(text: string): unknown { + try { + return JSON.parse(text); + } catch { + return text; + } +} + +/** + * Compute one page, and the faults of each row. One batched computation per + * page; only when it reported a fault is each row whose value came back + * `null` computed again on its own, which is what ties a fault to its row. + */ +async function computePage( + engine: StoredExpressionBackfillEngine, + object: string, + page: ReadonlyArray>, +): Promise<{ values: Array>; faults: Array> }> { + const faulted = new Set(); + const values = await engine.computeStoredExpressionsOver(object, page, { ...SYSTEM_CTX }, (field) => { + faulted.add(field); + }); + const faults = page.map(() => new Map()); + if (faulted.size === 0) return { values, faults }; + for (let i = 0; i < page.length; i++) { + if (![...faulted].some((field) => values[i]?.[field] === null)) continue; + await engine.computeStoredExpressionsOver(object, [page[i]], { ...SYSTEM_CTX }, (field, message) => { + if (!faults[i].has(field)) faults[i].set(field, message); + }); + } + return { values, faults }; +} + +/** + * Fill the stored computed fields of stored rows — the NULL holes by default, + * every stale value too under `options.changed`. + * + * Dry run unless `options.apply`. Refusals before the walk: an unknown object + * (`OBJECT_NOT_FOUND`), and a named object or `field` declaring no stored + * computed field (`INVALID_FIELD`). + */ +export async function backfillStoredExpressions( + engine: StoredExpressionBackfillEngine, + logger: StoredExpressionBackfillLogger, + options: StoredExpressionBackfillOptions = {}, +): Promise { + const apply = options.apply === true; + const changed = options.changed === true; + const maxListed = options.maxListed ?? DEFAULT_MAX_LISTED; + const scope = resolveScope(engine, options); + + const report: StoredExpressionBackfillReport = { + scannedObjects: [], + scannedRecords: 0, + fields: [], + nullRows: 0, + staleRows: 0, + faultedRows: 0, + skippedRows: 0, + writtenRows: 0, + changed, + applied: apply, + truncated: false, + unreadableObjects: [], + faults: [], + faultsTruncated: false, + failures: [], + }; + + for (const [object, fields] of scope) { + report.scannedObjects.push(object); + const outcomes = new Map(); + for (const field of fields) { + const outcome = { object, field, nullRows: 0, staleRows: 0, faultedRows: 0, written: 0, sampleRecordIds: [] }; + outcomes.set(field, outcome); + report.fields.push(outcome); + } + // The stamp the apply keeps; a federated object's injected columns are not provisioned, so its write sends none. + const schema = engine.getObject(object); + const hasUpdatedAt = 'updated_at' in (schema?.fields ?? {}) && !isFederatedUnprovisionedInjectedColumn(schema, 'updated_at'); + + // Seek by `id`: this run's claim is about EVERY stored row, which an offset walk cannot promise. + const walk = keysetWalk>( + (q: KeysetPageQuery) => engine.find(object, { ...q, context: { ...SYSTEM_CTX } }), + { pageSize: SCAN_PAGE_SIZE, max: options.maxRecordsPerObject ?? 100_000 }, + ); + + try { + for await (const page of walk.pages()) { + const { values, faults } = await computePage(engine, object, page); + for (let i = 0; i < page.length; i++) { + const row = page[i]; + report.scannedRecords++; + if (row.id == null || row.id === '') continue; + const recordId = String(row.id); + const patch: Record = {}; + let hole = false; + let stale = false; + let blocked = false; + for (const field of fields) { + const outcome = outcomes.get(field)!; + const stored = row[field]; + const fault = faults[i].get(field); + if (fault !== undefined) { + outcome.faultedRows++; + report.faultedRows++; + const blocksWrite = !isBlank(stored); + if (blocksWrite) blocked = true; + if (report.faults.length < maxListed) report.faults.push({ object, field, recordId, error: fault, blocksWrite }); + else report.faultsTruncated = true; + continue; + } + const value = values[i]?.[field] ?? null; + if (isBlank(stored)) { + if (isBlank(value)) continue; + outcome.nullRows++; + report.nullRows++; + if (outcome.sampleRecordIds.length < MAX_SAMPLE_IDS) outcome.sampleRecordIds.push(recordId); + hole = true; + patch[field] = value; + } else if (!sameStoredValue(stored, value)) { + outcome.staleRows++; + report.staleRows++; + stale = true; + patch[field] = value; + } + } + if (!(hole || (changed && stale))) continue; + // The write computes EVERY stored field of the row, the ones this run was not asked about too. + for (const [field] of faults[i]) if (!fields.includes(field) && !isBlank(row[field])) blocked = true; + if (blocked) { + report.skippedRows++; + continue; + } + if (!apply) continue; + try { + await withTransientRetry( + () => engine.update( + object, + { id: row.id, ...patch, ...(hasUpdatedAt && !isBlank(row.updated_at) ? { updated_at: row.updated_at } : {}) }, + { context: { ...SYSTEM_PRESERVING_CTX } }, + ), + options.retry ?? {}, + ); + // Bumped AFTER the retry settles, never inside the retried closure. + report.writtenRows++; + for (const field of Object.keys(patch)) outcomes.get(field)!.written++; + } catch (err) { + // One row's failure must not abort the rest; a re-run revisits it, because it still holds the hole. + const message = (err as Error)?.message ?? String(err); + logger.warn(`[stored-expression-backfill] ${object} record ${recordId}: ${message}`); + report.failures.push({ object, recordId, error: message }); + } + } + } + } catch (err) { + // An object this run cannot read is one whose rows it cannot vouch for: reported, and it truncates. + logger.warn( + `[stored-expression-backfill] cannot read ${object} (${(err as Error)?.message ?? err}) — ` + + 'reported as unreadable; this run does not cover it', + ); + report.unreadableObjects.push(object); + report.truncated = true; + continue; + } + if (walk.truncated) report.truncated = true; + } + return report; +} + +/** Did this run leave no hole it was asked to fill (and, under `changed`, no stale value)? */ +export function storedExpressionBackfillComplete(report: StoredExpressionBackfillReport): boolean { + const pending = report.nullRows + (report.changed ? report.staleRows : 0); + return report.failures.length === 0 + && report.skippedRows === 0 + && !report.truncated + && (report.applied || pending === 0); +} + +/** Human-readable report body, shared by the CLI's dry-run and apply output. */ +export function formatStoredExpressionBackfillReport(report: StoredExpressionBackfillReport): string[] { + const lines: string[] = []; + lines.push( + `Scanned ${report.scannedRecords} row(s) across ${report.scannedObjects.length} object(s) for stored computed ` + + `fields ${report.changed ? 'whose stored value differs from their expression' : 'still stored as NULL'}.`, + ); + const pending = report.fields.filter((f) => f.nullRows > 0 || (report.changed && f.staleRows > 0)); + if (pending.length === 0) { + lines.push(report.changed + ? '✓ Every stored computed value already equals its expression — nothing to recompute.' + : '✓ No NULL a stored computed field\'s expression would fill — nothing to backfill.'); + } else { + lines.push(report.applied + ? `Wrote ${report.fields.reduce((n, f) => n + f.written, 0)} value(s) on ${report.writtenRows} row(s) in ${pending.length} field(s):` + : `Would write ${report.nullRows + (report.changed ? report.staleRows : 0)} value(s) in ${pending.length} field(s):`); + for (const f of pending) { + const parts = [`${f.nullRows} NULL row(s)`]; + if (report.changed) parts.push(`${f.staleRows} stale row(s)`); + if (report.applied) parts.push(`${f.written} written`); + lines.push(` • ${f.object}.${f.field} — ${parts.join(', ')}${f.sampleRecordIds.length > 0 ? `\n e.g. ${f.sampleRecordIds.join(', ')}` : ''}`); + } + } + const stale = report.fields.filter((f) => f.staleRows > 0); + if (!report.changed && stale.length > 0) { + lines.push( + `· ${report.staleRows} stored value(s) differ from their expression — the expression changed after they were written ` + + `(${stale.map((f) => `${f.object}.${f.field} ${f.staleRows}`).join(', ')}). Not rewritten by this run, except on a row ` + + 'it fills; re-run with --changed to recompute them. An expression reading the clock always differs.', + ); + } + const faulted = report.fields.filter((f) => f.faultedRows > 0); + if (faulted.length > 0) { + lines.push( + `· ${report.faultedRows} row(s) where an expression does not evaluate — it computes NULL there, as on any write ` + + `(${faulted.map((f) => `${f.object}.${f.field} ${f.faultedRows}`).join(', ')}):`, + ); + for (const fault of report.faults.slice(0, MAX_SAMPLE_IDS)) { + lines.push(` • ${fault.object}.${fault.field} ${fault.recordId}${fault.blocksWrite ? ' (holds a value: row not written)' : ''}: ${fault.error.split('\n')[0]}`); + } + if (report.faultedRows > MAX_SAMPLE_IDS) lines.push(` … and ${report.faultedRows - MAX_SAMPLE_IDS} more`); + } + if (report.skippedRows > 0) { + lines.push( + `✗ ${report.skippedRows} row(s) listed and NOT written: a field that does not evaluate there holds a value the write ` + + 'would replace with NULL. Fix the expression or the row, then re-run.', + ); + } + if (report.failures.length > 0) { + lines.push(`✗ ${report.failures.length} row(s) could not be written:`); + for (const f of report.failures.slice(0, MAX_SAMPLE_IDS)) lines.push(` • ${f.object} ${f.recordId}: ${f.error}`); + if (report.failures.length > MAX_SAMPLE_IDS) lines.push(` … and ${report.failures.length - MAX_SAMPLE_IDS} more`); + lines.push('Re-running is safe: the backfill only revisits rows that still differ.'); + } + if (report.unreadableObjects.length > 0) lines.push(`⚠ Unreadable object(s): ${report.unreadableObjects.join(', ')}`); + if (report.truncated) lines.push('⚠ Walk incomplete — some rows were not examined; re-run with a higher --max-records.'); + // Claims no row or table, never that nothing was written: the command boots through the SQL driver's connect, + // which can switch a SQLite file's journal mode (its header, no row). The command appends that note on SQLite. + if (!report.applied && pending.length > 0) lines.push('Dry run — no row or table was written. Re-run with --apply to write.'); + return lines; +} + +/** One object's holes in the advisory's sample. */ +export interface StoredExpressionHoleSample { + object: string; + /** `field` → rows of the sample holding a NULL its expression fills. */ + fields: Record; + sampled: number; +} + +/** + * The boot advisory: read the first {@link ADVISORY_SAMPLE_SIZE} rows of each + * object declaring a stored computed field and answer the samples holding a + * hole. Reads only; an object whose sample cannot be read or computed is left + * out (the command, which walks every row, is the authority). + */ +export async function sampleStoredExpressionHoles( + engine: Pick, + onSkip?: (object: string, error: unknown) => void, +): Promise { + const out: StoredExpressionHoleSample[] = []; + const objects = typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []; + for (const object of objects) { + const fields = storedExpressionFields(engine, object); + if (fields.length === 0) continue; + try { + const rows = await engine.find(object, { + orderBy: [{ field: 'id', order: 'asc' }], + limit: ADVISORY_SAMPLE_SIZE, + context: { ...SYSTEM_CTX }, + }); + if (rows.length === 0) continue; + // A fault computes `null`, never a hole; it is the write's to report, not a boot's. + const values = await engine.computeStoredExpressionsOver(object, rows, { ...SYSTEM_CTX }, () => {}); + const holes: Record = {}; + rows.forEach((row, i) => { + for (const field of fields) { + if (isBlank(row[field]) && !isBlank(values[i]?.[field])) holes[field] = (holes[field] ?? 0) + 1; + } + }); + if (Object.keys(holes).length > 0) out.push({ object, fields: holes, sampled: rows.length }); + } catch (err) { + onSkip?.(object, err); + } + } + return out; +} + +/** + * Warn once when a stored computed field holds NULLs its expression fills, + * naming each and `os migrate stored-expressions`. An advisory: it never fails + * a boot and writes nothing. + */ +export async function announceStoredExpressionHoles( + engine: Pick, + logger: StoredExpressionBackfillLogger & { debug?(msg: string, meta?: unknown): void }, +): Promise { + let samples: StoredExpressionHoleSample[] = []; + try { + samples = await sampleStoredExpressionHoles(engine, (object, err) => { + logger.debug?.('[stored-expression] boot sample skipped', { object, error: (err as Error)?.message ?? String(err) }); + }); + } catch (err) { + // An advisory must never be the reason a boot fails. + logger.debug?.('[stored-expression] boot sample failed', { error: (err as Error)?.message ?? String(err) }); + return []; + } + if (samples.length === 0) return samples; + const named = samples + .flatMap((s) => Object.entries(s.fields).map(([field, n]) => `${s.object}.${field} (${n} of the first ${s.sampled} rows)`)) + .join(', '); + logger.warn( + `[stored-expression] stored computed field(s) hold NULL on rows no write has computed since the field was declared: ${named}. ` + + 'Every reader of these columns (filters, sorts, roll-ups, reports) reads NULL where the expression gives a value, until ' + + 'each row is next written. Run `os migrate stored-expressions` (dry run) to see every such row, then `--apply` to fill them.', + ); + return samples; +}