Skip to content

Commit 51dbc7b

Browse files
committed
fix(migrate): a view recreated only around a table change is not drift (D3)
An adopter on 1.1.0-rc.2 rebuilt a SQLite database from the committed migration chain. Every report and projection view's stored SQL was byte-identical to the metadata's, yet `meta verify --db` and `meta migrate --from-db` listed each view as `- view` / `+ view`, and `verify` exited 1. Neither the text nor `--format json` said what differed. The view comparison was not at fault. When a migration alters a table (a column on every dialect; an FK or CHECK on SQLite/D1, which rebuild the table, #243), Pass 2c drops and recreates every view that reads it. That pair carried nothing that set it apart from a view whose definition changed. - migrate-ts: every view change carries a `reason` (ViewChangeReason: missing | undeclared | definition | unfingerprinted | unchanged). A recreate is `unchanged` only when equality is proven (the fingerprint on Postgres, the normalized text on SQLite/D1). `isViewRecreateOnly` and `withoutViewRecreates` are exported. computeDriftFromActual and classifyDrift leave recreate-only views out. The emitted SQL is unchanged. - A text difference carries the first differing excerpt (firstViewSqlDifference). - Pass 2c keeps `unmanagedActual` from a superseded adoption, so an unstamped Postgres view over an altered table needs --allow adopt-view again (it bypassed the #239 gate before). - cli: describeChange says why each view change was planned. The committed-snapshot gate ignores recreate-only views. `verify --format json` gains `schemaDrift` (changes[] with kind/object/detail, plus findings[]). `migrate` (online, offline-snapshot and D1 paths) adds a note naming the views that match, in text and in a `notes` array in structured output. - Tests: migrate-ts unit and drift tests; the CLI's verify-db-view-recreate integration test; integration-tests lanes on a real SQLite (plus the D1 diff) and a real Postgres, both of which fail with the filter disabled. - CHANGELOG (1.1.0 Fixed), docs/features/cli.md, docs/features/migrations-and-drift.md.
1 parent 0c6d362 commit 51dbc7b

20 files changed

Lines changed: 1090 additions & 34 deletions

‎CHANGELOG.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -470,6 +470,30 @@ until you regenerate.
470470

471471
### Fixed
472472

473+
- **`meta verify --db` no longer reports a matching view as drift, and says what differs when a
474+
view does not.** An adopter on 1.1.0-rc.2 rebuilt a SQLite database from the committed chain:
475+
every report and projection view's stored SQL was byte-identical to the metadata's, yet
476+
`meta verify --db` and `meta migrate --from-db` listed each one as `- view` / `+ view` and
477+
`verify` exited 1, with nothing in the text or `--format json` saying what differed. The cause
478+
was not the view comparison. When a migration alters a table (a column on every dialect, and an
479+
FK or CHECK on SQLite and D1, which rebuild the table, #243), the diff drops and recreates every
480+
view that reads it, and that pair looked the same as a view whose definition changed. Every
481+
view change now carries a `reason` (`ViewChangeReason`, exported from
482+
`@metaobjectsdev/migrate-ts` with `isViewRecreateOnly` and `withoutViewRecreates`), and a
483+
recreate whose definition matches is `unchanged`. `verify --db`, the D1 and committed-snapshot
484+
drift gates, `computeDriftFromActual` and `classifyDrift` leave those out, so the table change
485+
is reported alone. `meta migrate` still emits the pair, because the SQL needs it, and adds a
486+
note naming the views that match ("recreated only because the migration alters a table they
487+
read") to its text output and to a new `notes` array in `--format json`. A view that does
488+
differ is reported with why: SQLite and D1 show the first differing excerpt of the definition
489+
text (`definition text differs: metadata «…» vs database «…»`), Postgres says the fingerprint
490+
does not match, and an unstamped Postgres view says it cannot be compared. `verify --format
491+
json` gains a `schemaDrift` section listing each schema difference and its explanation. The
492+
emitted migration SQL does not change. One gate is tightened on the way: an unstamped Postgres
493+
view over an altered table used to be recreated without `--allow adopt-view`; it now needs the
494+
flag, as it does when no table changes. Gated in `migrate-ts` unit and drift tests, the CLI's
495+
`verify-db-view-recreate` test, and new `integration-tests` lanes on a real SQLite (with the D1
496+
diff) and a real Postgres.
473497
- **TypeScript: a report's decimal fields reach the wire as strings on SQLite, as on Postgres
474498
(FR-044).** A ratio is typed `decimal`, the TypeScript read schema types a decimal as
475499
`string`, and SQLite has no decimal: the view computes a `REAL`, which the driver hands the

‎docs/features/cli.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -550,14 +550,19 @@ meta gen --format json → { gen[], summary, help[], antiPatterns: { status
550550
meta verify --format json → { verify[], exitCode, summary, help[],
551551
antiPatterns: { status, total, rows[] },
552552
requirements: { status, total, rows[] },
553-
requirementCounts?, notRepresented[] }
553+
requirementCounts?, schemaDrift?, notRepresented[] }
554554
```
555555

556556
A pass that did **not** run says so (`status: "skipped"` with a `note` giving the
557557
reason) rather than reporting an empty list — "found nothing" and "never looked"
558558
are different answers. `meta verify`'s payload carries each gate's pass/fail
559-
verdict; the per-gate drift **detail** stays on stderr as text, and the payload's
560-
own `notRepresented[]` says so.
559+
verdict. The schema gate's differences are in `schemaDrift` (`changes[]`, one
560+
`{ kind, object, detail }` per difference with the same explanation the text prints,
561+
plus the ledger `findings[]`); every other gate's drift **detail** stays on stderr as
562+
text, and the payload's own `notRepresented[]` says so. A view that matches the
563+
metadata and is recreated only because the migration alters a table it reads is not
564+
drift, so it appears in neither. `meta migrate` still emits that drop/create pair and
565+
names the views in a note (`notes[]` in its structured output).
561566

562567
In a structured run every narration line moves to **stderr**, so stdout is one
563568
parseable document. `--format` is honored by `gen`, `verify` and `migrate`; any

‎docs/features/migrations-and-drift.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ There are **7 drift sources**, and the toolchain has a guard for each.
1313
|---|---|---|
1414
| **Code-vs-DB** | Codegen — the generated SQL DDL is emitted from the same metadata as the entity / table code. | Build time |
1515
| **Code-vs-API-doc** | Cross-port codegen from the same metadata. | Build time |
16-
| **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is `replace-view` drift); a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 <binding>] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR |
16+
| **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is `replace-view` drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, a fingerprint mismatch on Postgres). A view `meta migrate` drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 <binding>] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR |
1717
| **Migration-vs-metadata** | The Node `meta migrate` emits migrations FROM metadata diffs — they cannot drift from metadata by construction. Schema migrations for **every** port are owned by this Node toolchain (`@metaobjectsdev/cli migrate`, ADR-0015); the C# and Python migrate surfaces were removed. | Build time |
1818
| **Generated-edited** | `@generated` headers in emitted code + three-way merge that preserves hand-edits inside non-generated regions. | Code review |
1919
| **Prompt-vs-payload** | FR-004 `Renderer.verify` parses `{{...}}` references in templates and checks each one exists on the payload VO. | Build time + runtime |

‎server/typescript/packages/cli/src/commands/migrate.ts‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ import {
6161
type WranglerRunner,
6262
} from "../lib/wrangler.js";
6363
import { buildProjectionViews } from "@metaobjectsdev/codegen-ts";
64-
import { tokensToAllowOptions, blockedEntriesFor, blockedHintLines } from "../lib/allow.js";
64+
import { tokensToAllowOptions, blockedEntriesFor, blockedHintLines, viewRecreateNote } from "../lib/allow.js";
6565
import { reportLoadError } from "../lib/load-error.js";
6666
import { scanForReferentialActionConflicts } from "../lib/referential-action-advisory.js";
6767
import type { MetaData } from "@metaobjectsdev/metadata";
@@ -253,6 +253,17 @@ function migrateResultDefaults(dryRun: boolean): Pick<MigrateResultShape, "block
253253
return { blocked: [], ambiguous: [], writtenPaths: [], dryRun };
254254
}
255255

256+
/** `viewRecreateNote` as the `notes` list a result carries: empty when there is nothing to say. */
257+
function viewRecreateNotes(changes: Change[]): string[] {
258+
const note = viewRecreateNote(changes);
259+
return note === undefined ? [] : [note];
260+
}
261+
262+
/** A result's notes as prose lines, for the paths that report through `finishMigrate`. */
263+
function narratedNotes(notes: readonly string[]): string[] {
264+
return notes.map((n) => `migrate: ${n}`);
265+
}
266+
256267
/** The `-- UP -- / -- DOWN --` preview a dry run prints in text format. */
257268
function sqlPreview(sql: { up: string; down: string }): string {
258269
return `-- UP --\n${sql.up}\n\n-- DOWN --\n${sql.down}`;
@@ -739,6 +750,7 @@ export async function migrateCommand(
739750
let writtenPaths: string[] = [];
740751
/** A dry run's SQL: printed in text format, carried in the document otherwise. */
741752
let dryRunSql: { up: string; down: string } | undefined;
753+
const notes: string[] = [];
742754
let appliedNames: string[] = [];
743755
let applyFailed = false;
744756
let blocked: BlockedEntry[] = [];
@@ -896,6 +908,7 @@ export async function migrateCommand(
896908
if (diffResult.changes.length === 0) {
897909
// no-op — output will say "No schema changes"
898910
} else {
911+
notes.push(...viewRecreateNotes(diffResult.changes));
899912
let emitted: EmitResult | undefined;
900913
try {
901914
emitted = emit(diffResult.changes, {
@@ -1051,6 +1064,7 @@ export async function migrateCommand(
10511064
applied: appliedNames,
10521065
applyFailed,
10531066
warnings: hazardWarnings,
1067+
notes,
10541068
...(dryRunSql !== undefined ? { sql: dryRunSql } : {}),
10551069
};
10561070
const output =
@@ -1425,12 +1439,14 @@ export async function runOfflineGenerate(
14251439
const { diff: diffResult, nextSnapshot, expected: governedExpected } = plan;
14261440
logOutOfScope(plan.outOfScope, plan.importedOutOfScope ?? [], fmt);
14271441

1442+
const offlineNotes = viewRecreateNotes(diffResult.changes);
14281443
const offlineResult = (extra: Partial<MigrateResultShape>): MigrateResultShape => ({
14291444
dialect: offlineDialect,
14301445
displayUrl: "",
14311446
changeCounts: summarizeChanges(diffResult.changes),
14321447
...migrateResultDefaults(config.dryRun),
14331448
format: config.format,
1449+
notes: offlineNotes,
14341450
...extra,
14351451
});
14361452
if (diffResult.blocked.length > 0) {
@@ -1459,7 +1475,11 @@ export async function runOfflineGenerate(
14591475

14601476
if (config.dryRun) {
14611477
const sql = { up: emitResult.up, down: emitResult.down };
1462-
finishMigrate(fmt, [sqlPreview(sql)], migrateResultToData(offlineResult({ sql, warnings: offlineWarnings })));
1478+
finishMigrate(
1479+
fmt,
1480+
[...narratedNotes(offlineNotes), sqlPreview(sql)],
1481+
migrateResultToData(offlineResult({ sql, warnings: offlineWarnings })),
1482+
);
14631483
return 0;
14641484
}
14651485

@@ -1482,7 +1502,7 @@ export async function runOfflineGenerate(
14821502
await writeSnapshot(path, nextSnapshot);
14831503
finishMigrate(
14841504
fmt,
1485-
[`migrate: wrote ${res.upPath}`, `migrate: wrote ${res.downPath}`],
1505+
[`migrate: wrote ${res.upPath}`, `migrate: wrote ${res.downPath}`, ...narratedNotes(offlineNotes)],
14861506
migrateResultToData(offlineResult({ writtenPaths: [res.upPath, res.downPath], warnings: offlineWarnings })),
14871507
);
14881508
return 0;
@@ -1700,6 +1720,7 @@ async function runD1Migrate(
17001720
}
17011721

17021722
const changeCounts = summarizeChanges(diffResult.changes);
1723+
const d1Notes = viewRecreateNotes(diffResult.changes);
17031724
warnDataHazards(diffResult.hazards);
17041725

17051726
// Views are emitted by the one schema-diff path: renderD1 = renderSqlite (which
@@ -1748,7 +1769,7 @@ async function runD1Migrate(
17481769

17491770
if (config.dryRun) {
17501771
const sql = { up: combinedUp, down: combinedDown };
1751-
finishMigrate(fmt, [sqlPreview(sql)], migrateResultToData(d1Result({ sql })));
1772+
finishMigrate(fmt, [...narratedNotes(d1Notes), sqlPreview(sql)], migrateResultToData(d1Result({ sql, notes: d1Notes })));
17521773
return 0;
17531774
}
17541775

@@ -1762,8 +1783,9 @@ async function runD1Migrate(
17621783
`migrate: wrote ${writeResult.upPath}`,
17631784
`migrate: wrote ${writeResult.downPath}`,
17641785
...Object.entries(changeCounts).map(([kind, count]) => ` ${kind}: ${count}`),
1786+
...narratedNotes(d1Notes),
17651787
],
1766-
migrateResultToData(d1Result({ writtenPaths: [writeResult.upPath, writeResult.downPath] })),
1788+
migrateResultToData(d1Result({ writtenPaths: [writeResult.upPath, writeResult.downPath], notes: d1Notes })),
17671789
);
17681790

17691791
// 7. Optional --apply: run `wrangler d1 migrations apply`.

‎server/typescript/packages/cli/src/commands/verify.ts‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ import {
7070
verifyReplay,
7171
introspect,
7272
diff,
73+
withoutViewRecreates,
7374
readSnapshot,
7475
snapshotPath,
7576
type SchemaSnapshot,
@@ -382,6 +383,9 @@ export async function verifyCommand(
382383
// Set when the schema gate could not reach or read the database: a failure, but not
383384
// drift, and the payload must not report it as drift.
384385
let schemaRunError: string | undefined;
386+
// The schema gate's findings, for the structured payload: what differs, not only that
387+
// something does. Left undefined when the gate did not compare anything.
388+
let schemaDrift: SchemaDriftSection | undefined;
385389
const schemaExit = await runSchemaVerify();
386390
const codegenExit = runCodegen ? await runCodegenVerify() : 0;
387391
const docsExit = runDocs ? await runDocsVerify() : 0;
@@ -467,6 +471,7 @@ export async function verifyCommand(
467471
names: nameSection,
468472
deprecations: deprecationSection,
469473
fields: fieldSection,
474+
...(schemaDrift !== undefined ? { schemaDrift } : {}),
470475
errors: schemaRunError !== undefined ? [{ gate: "schema", error: schemaRunError }] : [],
471476
}),
472477
fmt,
@@ -1478,7 +1483,9 @@ export async function verifyCommand(
14781483
// MERGED with the out-of-scope set, and a second key would silently drop that half.
14791484
dialect,
14801485
});
1481-
if (result.changes.length === 0) return [];
1486+
// A view recreated only around a table change matches the snapshot (D3) — not a difference.
1487+
const changes = withoutViewRecreates(result.changes);
1488+
if (changes.length === 0) return [];
14821489

14831490
return [
14841491
// `meta migrate --from-db` is NOT the repair: it writes a snapshot only when it has
@@ -1488,10 +1495,10 @@ export async function verifyCommand(
14881495
// been told everything is in sync. `baseline --from-db` rewrites it unconditionally,
14891496
// which is the whole point of the subcommand.
14901497
`the committed schema snapshot disagrees with ${displayUrl} ` +
1491-
`(${result.changes.length} difference(s)) — the next 'meta migrate' would emit DDL from it ` +
1498+
`(${changes.length} difference(s)) — the next 'meta migrate' would emit DDL from it ` +
14921499
`and fail at apply. Re-derive it with ` +
14931500
`'meta migrate baseline --from-db --db <url> --dialect ${dialect}'.`,
1494-
...summarizeDrift(result.changes),
1501+
...summarizeDrift(changes),
14951502
];
14961503
}
14971504

@@ -1515,6 +1522,7 @@ export async function verifyCommand(
15151522
}
15161523

15171524
const changes = driftResult.changes;
1525+
schemaDrift = { changes: changes.map(toSchemaDriftRow), findings: ledgerDrift };
15181526
if (changes.length === 0 && ledgerDrift.length === 0) {
15191527
say(`meta verify — schema in sync with ${displayUrl}.`);
15201528
return 0;
@@ -1822,6 +1830,28 @@ function summarizeDrift(changes: Change[]): string[] {
18221830
});
18231831
}
18241832

1833+
/** One schema difference in the structured payload — the same text the stderr summary prints. */
1834+
interface SchemaDriftRow {
1835+
kind: Change["kind"];
1836+
object: string;
1837+
detail: string;
1838+
}
1839+
1840+
/**
1841+
* The schema gate's findings in the structured payload. An adopter whose `verify --db` failed
1842+
* on views could not tell from `--format json` what differed (D3): the payload carried the
1843+
* verdict only. `changes` is the metadata↔database comparison; `findings` are the
1844+
* migration-ledger and committed-snapshot lines, which are text by construction.
1845+
*/
1846+
interface SchemaDriftSection {
1847+
changes: SchemaDriftRow[];
1848+
findings: string[];
1849+
}
1850+
1851+
function toSchemaDriftRow(c: Change): SchemaDriftRow {
1852+
return { kind: c.kind, object: DRIFT_PRESENTATION[c.kind].noun, detail: describeChange(c) };
1853+
}
1854+
18251855
// ---------------------------------------------------------------------------
18261856
// structured output (--format toon|json)
18271857
// ---------------------------------------------------------------------------
@@ -1948,6 +1978,8 @@ function buildVerifyPayload(input: {
19481978
names: AdvisorySection<AdvisoryDiagnosticRow>;
19491979
deprecations: AdvisorySection<AdvisoryDiagnosticRow>;
19501980
fields: AdvisorySection<AdvisoryDiagnosticRow>;
1981+
/** What the schema gate found, when it compared anything. */
1982+
schemaDrift?: SchemaDriftSection;
19511983
/** Gates that could not run at all (an unreachable database) — failures, not drift. */
19521984
errors?: readonly { gate: string; error: string }[];
19531985
}): Record<string, unknown> {
@@ -1988,7 +2020,12 @@ function buildVerifyPayload(input: {
19882020

19892021
const help: string[] = errors.map((e) =>
19902022
`the ${e.gate} gate could not run, which is not drift: ${e.error} — fix the connection and re-run`);
1991-
if (drifted.length > 0) {
2023+
if (drifted.some((g) => g.gate === "schema") && input.schemaDrift !== undefined) {
2024+
help.push(
2025+
`the schema gate's differences are in schemaDrift — changes[] (metadata vs database, one row per change) and findings[] (migration ledger and committed snapshot)`,
2026+
);
2027+
}
2028+
if (drifted.some((g) => g.gate !== "schema")) {
19922029
help.push(
19932030
`the failing gate's drift DETAIL is printed as text on stderr — this payload carries the verdict only`,
19942031
);
@@ -2041,11 +2078,12 @@ function buildVerifyPayload(input: {
20412078
deprecations: input.deprecations,
20422079
fields: input.fields,
20432080
...(input.requirementCounts !== undefined ? { requirementCounts: input.requirementCounts } : {}),
2081+
...(input.schemaDrift !== undefined ? { schemaDrift: input.schemaDrift } : {}),
20442082
// The honest boundary. Everything named here is REACHABLE — it is printed as
20452083
// text on stderr — but it is not in this document, and a reader must not have
20462084
// to discover that by its absence.
20472085
notRepresented: [
2048-
"per-gate drift detail (which template variable drifted, which schema change, which generated file differs, which migration failed to replay) — printed as text on stderr; this payload carries each gate's pass/fail verdict",
2086+
"per-gate drift detail for every gate but schema (which template variable drifted, which generated file differs, which migration failed to replay) — printed as text on stderr; this payload carries each gate's pass/fail verdict, and the schema gate's differences in schemaDrift",
20492087
"the loader's own warnings and the agent-context/manifest advisories — printed as text on stderr",
20502088
],
20512089
};

0 commit comments

Comments
 (0)