You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8a727b1
Browse filesBrowse the repository at this point in the historyBrowse files
feat(migrate-ts): decide Postgres drift against a shadow database (#510)
* fix(migrate-ts): let Postgres canonicalise expected column defaults and the column-rename CHECK carry
* feat(migrate-ts,cli): decide Postgres expression drift against a scratch (shadow) database first
* feat(migrate-ts): make the shadow path a schema-wide diff of introspect(shadow) against introspect(live)
* no-mistakes(review): Make shadow path catalog-wide, strict, shared and reported
* no-mistakes(review): Gate comment clears, version snapshots, fix quoted-expression defaults
* no-mistakes(test): Make plain migrate say it skips the database and refuse --shadow-db
* no-mistakes(document): Docs updated for shadow-db drift path: CLI README, CHANGELOG 1.2.0 heading, drift table row.
* no-mistakes(document): Restored [Unreleased] heading; added drop-unmanaged to migrate --allow help and snapshot
Copy file name to clipboardExpand all lines: docs/features/migrations-and-drift.md
+61-6Lines changed: 61 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,7 +13,7 @@ There are **7 drift sources**, and the toolchain has a guard for each.
13
13
|---|---|---|
14
14
|**Code-vs-DB**| Codegen — the generated SQL DDL is emitted from the same metadata as the entity / table code. | Build time |
15
15
|**Code-vs-API-doc**| Cross-port codegen from the same metadata. | Build time |
16
-
| **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, a fingerprint mismatch on Postgres). A view `meta migrate` drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 <binding>] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR |
16
+
| **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, the deparsed body on Postgres's scratch-database path, a fingerprint mismatch otherwise). A view `meta migrate` drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 <binding>] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR |
17
17
|**Migration-vs-metadata**| The Node `meta migrate` emits migrations FROM metadata diffs — they cannot drift from metadata by construction. Schema migrations for **every** port are owned by this Node toolchain (`@metaobjectsdev/cli migrate`, ADR-0015); the C# and Python migrate surfaces were removed. | Build time |
18
18
|**Generated-edited**|`@generated` headers in emitted code + a committed hash manifest that refuses an edited file by name; in TypeScript only, a three-way merge also preserves hand-edits on the machine that generated the file. | Code review |
19
19
|**Prompt-vs-payload**| FR-004 `Renderer.verify` parses `{{...}}` references in templates and checks each one exists on the payload VO. | Build time + runtime |
@@ -377,17 +377,65 @@ Postgres rewrites an expression before it stores it (`"k" IN ('x')` becomes `k =
377
377
`= ANY (ARRAY[…])`, and casts and brackets are added), so the text `meta migrate` generates never
378
378
comes back from the database as written. On Postgres, a plan against the database
379
379
(`meta verify --db`, `meta migrate --from-db` and `--apply`) therefore **asks Postgres** instead of
380
-
guessing. For each table it creates a **temporary table with the live table's columns** inside a
381
-
transaction it rolls back (a savepoint, when the connection is already in one), adds the expected
380
+
guessing. There are three ways it asks, tried in this order, and every Postgres run of
381
+
`meta verify --db`, `meta migrate` and `meta migrate baseline --from-db` says which one ran.
382
+
383
+
**1. A scratch (shadow) database, the primary path.** The practice Atlas, Prisma and migra share:
384
+
the model's full expected schema (the DDL `meta migrate` emits for an empty database) is applied to
385
+
a throwaway Postgres database, in one transaction that is always rolled back, and read with the same
386
+
introspector that reads your live database. Drift is then the **same diff run on two catalogs**,
387
+
`introspect(shadow)` as the expected side against `introspect(live)`, for every object in the run's
388
+
scope: tables, columns and types, nullability, defaults, identity, CHECKs, primary/foreign/unique
389
+
constraints and their actions, indexes, view bodies and comments. Both sides are Postgres's own
390
+
spelling, so a shape nobody wrote a rule for (`int4` against `integer`, `ON DELETE SET NULL`, a
391
+
view body written differently from Postgres's deparse) needs none. Model-only knowledge stays an
392
+
input to that diff, not a second comparison: renames, `@unmanaged` and scope are applied as always.
393
+
Identity is read from `pg_attribute.attidentity` on both sides (a legacy `serial` column also reads
394
+
as an `increment` key, so it is not drift against one); a GENERATED identity added or dropped is a
395
+
`change-column-identity`, and dropping one is gated behind `--allow drop-identity-default`. Table
396
+
and column comments are read from `pg_description` on both sides, and a changed one is a
397
+
`change-comment` (`COMMENT ON … IS …`). Clearing a live comment the metadata does not declare
398
+
(somebody else may have written it) needs `--allow drop-comment`: without it `meta migrate`
399
+
leaves the comment in place, plans the rest and says so in a note, and `meta verify --db` still
400
+
reports it as drift. A committed snapshot written before `formatVersion` 4 never recorded comments
401
+
or a GENERATED identity, so the committed-snapshot check treats both as unknown rather than absent
402
+
and does not compare them; re-run `meta migrate baseline --from-db` to record them. A view's body is compared as Postgres's deparse
403
+
(`pg_get_viewdef`) of the shadow's copy against the live one; each view is created under its own
404
+
savepoint, so a view whose body reads something only the live database has is left out of the
405
+
shadow and compared by its fingerprint instead, without costing the run its table comparison.
406
+
Off the shadow path identity and comments are not compared, and views are compared by fingerprint.
407
+
No per-shape rule is involved, so `NOW()`/`now()`, `current_date`, a date
408
+
literal on a `timestamptz` (read in the session `TimeZone`), `'1e3'` on a double, `'007'` on an
409
+
integer, `now() + interval '1 day'`, a regrouped `NOT` and a precedence change are all decided by
410
+
the database. A real change of any of them is still reported. The scratch database comes from:
411
+
412
+
| Source | How |
413
+
|---|---|
414
+
| An explicit URL |`--shadow-db <url>` on `verify` and on `migrate --from-db` / `--apply` (a plain `meta migrate` reads no database, says so, and refuses the flag), else `SHADOW_DATABASE_URL`, else `migrate.shadowDatabaseUrl` in `.metaobjects/config.json`. Used as given and never dropped; nothing is left in it because the schema is applied inside a rolled-back transaction. |
415
+
|`CREATE DATABASE`| With none of the above, a database named `mo_shadow_<random>` is created on the `--db` server when the role may, and **always dropped** afterwards, including on error. |
416
+
417
+
A run opens its scratch database once, and every comparison it makes uses it: `meta verify --db`
418
+
compares the committed snapshot with the database on the same scratch database as the drift check.
419
+
420
+
An explicitly configured scratch database that cannot be opened, or that the expected schema does
421
+
not apply to (an FK to a table outside the run's scope, a missing extension), is an **error** naming
422
+
the cause: you asked for a scratch database so the target would not be written to, so the run does
423
+
not fall back to the probe on the target. A `CREATE DATABASE` that the role may not run, or that the
424
+
schema does not apply to, falls back to the next path, and the output says why.
425
+
426
+
**2. The temp-object probe, the first fallback.** For each table it creates a **temporary table with
427
+
the live table's columns** on the target itself, inside a transaction it rolls back (a savepoint, when the connection is already in one), adds the expected
382
428
CHECK to it, creates each expected expression or partial-index predicate as a temporary index,
383
429
and reads back `pg_get_constraintdef` / `pg_get_indexdef` / `pg_get_expr`. Those are compared with
384
-
the live definitions as plain strings. Nothing persists, and nothing is written to your table.
430
+
the live definitions as plain strings. Nothing persists, and nothing is written to your table, but
431
+
the probe **needs write privileges on the target** (it creates temporary objects) and **takes brief
432
+
locks** on it, which is why a scratch database is preferred.
385
433
Two expressions that mean different things never compare equal: a precedence change
386
434
(`(a OR b) AND c` against `a OR (b AND c)`), a regrouped `NOT`, and a changed enum-member or regex
387
435
literal are all drift.
388
436
389
-
Where the question cannot be asked, the comparison falls back to a **string normaliser**
390
-
(`check-expr-compare.ts`), which rewrites both texts toward one shape. The fallback applies to:
437
+
**3. The string normaliser, the last fallback.** Where neither question can be asked, the comparison uses it
438
+
(`check-expr-compare.ts`), which rewrites both texts toward one shape. It always applies to:
391
439
392
440
- SQLite and D1, which store the text as written;
393
441
- the default **offline**`meta migrate`, which diffs the committed snapshot and has no database;
@@ -403,6 +451,13 @@ The normaliser cannot tell a precedence change from a bracket-only change (it de
403
451
bracket), so on the fallback path `(a OR b) AND c` and `a OR (b AND c)` compare as equal. A fallback
404
452
never reports drift of its own: a probe that fails changes nothing about the answer.
405
453
454
+
The probe answers column defaults the same way. Postgres re-spells a default (`NOW()` becomes
455
+
`now()`, `current_date` becomes `CURRENT_DATE`, a date literal on a `timestamptz` becomes a full
456
+
timestamp read in the session's `TimeZone`, `'1e3'` on a double becomes `'1000'`), so for a column
457
+
with a default on both sides the expected one is set on the scratch table and `pg_get_expr` is
458
+
read back. A column rename that carries a CHECK is answered on a scratch copy whose columns are
459
+
renamed first.
460
+
406
461
`meta migrate baseline --from-db` on Postgres closes the offline gap. Where the live database
407
462
proves a CHECK, or an expression index's key or predicate, is what the metadata would generate, the
408
463
snapshot records the **generated** text rather than Postgres's spelling, so the offline plan
-`--dialect sqlite|postgres|d1` — auto-detected from URL scheme; use `d1` for Cloudflare D1
176
+
-`--shadow-db <url>` — Postgres scratch database the expected schema is applied to for `--from-db` / `--apply` drift (also `SHADOW_DATABASE_URL` or `migrate.shadowDatabaseUrl`). Read [How CHECK constraints and expression indexes are compared](../../../../docs/features/migrations-and-drift.md#how-check-constraints-and-expression-indexes-are-compared) for the three paths.
-`--rename-table [schema.]old=new` / `--rename-column [schema.]table.old=new` — declare a rename (repeatable), so the migration is `RENAME` instead of drop+add whatever the rename heuristic makes of the names. Refused if nothing matches. See [Renames and populated tables](../../../../docs/features/migrations-and-drift.md#renames-and-populated-tables).
181
182
-`--dry-run` — print the SQL pair, write nothing (under `--format json|toon` the pair is the document's `sql` field, so stdout stays one parseable document)
0 commit comments