Skip to content

Commit 8a727b1

Browse files
authored
feat(migrate-ts): decide Postgres drift against a shadow database (#510)
* fix(migrate-ts): let Postgres canonicalise expected column defaults and the column-rename CHECK carry * feat(migrate-ts,cli): decide Postgres expression drift against a scratch (shadow) database first * feat(migrate-ts): make the shadow path a schema-wide diff of introspect(shadow) against introspect(live) * no-mistakes(review): Make shadow path catalog-wide, strict, shared and reported * no-mistakes(review): Gate comment clears, version snapshots, fix quoted-expression defaults * no-mistakes(test): Make plain migrate say it skips the database and refuse --shadow-db * no-mistakes(document): Docs updated for shadow-db drift path: CLI README, CHANGELOG 1.2.0 heading, drift table row. * no-mistakes(document): Restored [Unreleased] heading; added drop-unmanaged to migrate --allow help and snapshot
1 parent 198d75d commit 8a727b1

41 files changed

Lines changed: 2065 additions & 80 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,54 @@ here.**
299299
so. The three flags it never read (`--server`, `--client`, `--out`) are still accepted but
300300
ignored and hidden from help, so the old invocation still reaches the pointer.
301301

302+
- **Postgres drift is decided by a scratch (shadow) database first.** `meta verify --db` and
303+
`meta migrate` (`--from-db`, `--apply`, and `baseline --from-db`) now apply the model's expected
304+
schema to a throwaway database, as Atlas, Prisma and migra do, read it with the introspector that
305+
reads the live database, and diff the two catalogs, so tables, columns and types, defaults,
306+
identity, CHECKs, constraints and their actions, indexes, view bodies and comments are all
307+
compared in Postgres's own spelling with no per-shape rule (renames, `@unmanaged` and scope stay
308+
inputs to the diff). On that path the introspector reads identity (`attidentity`) and comments
309+
(`pg_description`) from both catalogs, a GENERATED identity added or dropped is planned as
310+
`change-column-identity` (dropping one is gated behind `--allow drop-identity-default`), a
311+
changed comment as `change-comment`, and a view's deparsed body is compared with the live one; a
312+
view that does not apply to the scratch database is compared by its fingerprint, without costing
313+
the run its table comparison. The scratch database is `--shadow-db <url>`, else
314+
`SHADOW_DATABASE_URL`, else `migrate.shadowDatabaseUrl`; with none, a database is `CREATE`d on the
315+
`--db` server when the role may and always dropped afterwards. A run opens it once, so
316+
`meta verify --db` compares the committed snapshot on the same scratch database. The schema is
317+
applied in a rolled-back transaction, so an explicit URL is left empty, and an explicit one that
318+
cannot be opened or applied is an error naming the cause. When a `CREATE`d one is not available,
319+
the rolled-back temp-object probe on the target answers (it needs write privileges and takes
320+
brief locks), and SQLite and offline plans keep the string normaliser. Every Postgres run says
321+
which path ran and, on a fallback, why: a plain `meta migrate` (no `--from-db` or `--apply`)
322+
says it compared the committed snapshot and read no database, and refuses `--shadow-db`, which
323+
it would otherwise have ignored. Clearing a live comment the metadata does not declare
324+
needs the new `--allow drop-comment`; without it `meta migrate` keeps the comment, plans the
325+
rest and says so, and `meta verify --db` still reports it. **Upgrade note:** the committed
326+
snapshot moves to `formatVersion` 4, which records comments and GENERATED identity read off the
327+
database; an older toolchain refuses a v4 snapshot. A v3 snapshot still loads, and the
328+
committed-snapshot check treats its comments and identity as unknown rather than absent, so it
329+
does not go red after the upgrade; re-run `meta migrate baseline --from-db` to record them. The Postgres emitter no longer writes a `DEFAULT` beside
330+
`GENERATED BY DEFAULT AS IDENTITY` (a snapshot of a legacy `serial` column carries both, and
331+
Postgres refuses the pair). See `docs/features/migrations-and-drift.md`.
332+
333+
- **Postgres column `DEFAULT`s that Postgres re-spells no longer read as drift.** `NOW()`,
334+
`current_date`, a date literal on a `timestamptz`, `'1e3'` on a double, `'007'` on an integer
335+
and a composite expression such as `now() + interval '1 day'` were reported by `meta verify
336+
--db` as `change-column-default` on every run, and `meta migrate --from-db` proposed a
337+
`SET DEFAULT` for each, because the comparison was the classifier's text against the text
338+
Postgres hands back. The same expression-canon probe that answers CHECKs now answers defaults:
339+
inside its rolled-back transaction it sets each expected default on the scratch copy and reads
340+
`pg_get_expr`, and a literal on a `timestamptz` is compared in the reading session's
341+
`TimeZone`. A real change (another value, another instant, another function) is still reported.
342+
The classifier is still the comparison for SQLite and offline runs. `baseline --from-db` records the
343+
authored default where the probe proved it equal, so the offline plan starts clean too.
344+
A column rename that carries a CHECK is now decided the same way: the scratch copy renames the
345+
column first, so a rename plus a literal-case change (`'OPEN'` to `'open'`) is not folded into a
346+
plain `RENAME CONSTRAINT`. The anonymous-CHECK match on SQLite and D1 shares the comparison.
347+
Separately, the rename rewrite no longer touches a cast type, a function name or the inside of
348+
an `E'..'` literal when a column of that word is renamed.
349+
302350
- **`@default: "now"` is the current time on Postgres, not the table's creation time.** The
303351
expected schema stored it as the literal `'now'`, and Postgres converts that literal to a
304352
timestamp once, when it creates the column's `DEFAULT`, so every row got the same value. It

‎docs/features/migrations-and-drift.md‎

Lines changed: 61 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ There are **7 drift sources**, and the toolchain has a guard for each.
1313
|---|---|---|
1414
| **Code-vs-DB** | Codegen — the generated SQL DDL is emitted from the same metadata as the entity / table code. | Build time |
1515
| **Code-vs-API-doc** | Cross-port codegen from the same metadata. | Build time |
16-
| **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, a fingerprint mismatch on Postgres). A view `meta migrate` drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 <binding>] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR |
16+
| **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, the deparsed body on Postgres's scratch-database path, a fingerprint mismatch otherwise). A view `meta migrate` drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 <binding>] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR |
1717
| **Migration-vs-metadata** | The Node `meta migrate` emits migrations FROM metadata diffs — they cannot drift from metadata by construction. Schema migrations for **every** port are owned by this Node toolchain (`@metaobjectsdev/cli migrate`, ADR-0015); the C# and Python migrate surfaces were removed. | Build time |
1818
| **Generated-edited** | `@generated` headers in emitted code + a committed hash manifest that refuses an edited file by name; in TypeScript only, a three-way merge also preserves hand-edits on the machine that generated the file. | Code review |
1919
| **Prompt-vs-payload** | FR-004 `Renderer.verify` parses `{{...}}` references in templates and checks each one exists on the payload VO. | Build time + runtime |
@@ -377,17 +377,65 @@ Postgres rewrites an expression before it stores it (`"k" IN ('x')` becomes `k =
377377
`= ANY (ARRAY[…])`, and casts and brackets are added), so the text `meta migrate` generates never
378378
comes back from the database as written. On Postgres, a plan against the database
379379
(`meta verify --db`, `meta migrate --from-db` and `--apply`) therefore **asks Postgres** instead of
380-
guessing. For each table it creates a **temporary table with the live table's columns** inside a
381-
transaction it rolls back (a savepoint, when the connection is already in one), adds the expected
380+
guessing. There are three ways it asks, tried in this order, and every Postgres run of
381+
`meta verify --db`, `meta migrate` and `meta migrate baseline --from-db` says which one ran.
382+
383+
**1. A scratch (shadow) database, the primary path.** The practice Atlas, Prisma and migra share:
384+
the model's full expected schema (the DDL `meta migrate` emits for an empty database) is applied to
385+
a throwaway Postgres database, in one transaction that is always rolled back, and read with the same
386+
introspector that reads your live database. Drift is then the **same diff run on two catalogs**,
387+
`introspect(shadow)` as the expected side against `introspect(live)`, for every object in the run's
388+
scope: tables, columns and types, nullability, defaults, identity, CHECKs, primary/foreign/unique
389+
constraints and their actions, indexes, view bodies and comments. Both sides are Postgres's own
390+
spelling, so a shape nobody wrote a rule for (`int4` against `integer`, `ON DELETE SET NULL`, a
391+
view body written differently from Postgres's deparse) needs none. Model-only knowledge stays an
392+
input to that diff, not a second comparison: renames, `@unmanaged` and scope are applied as always.
393+
Identity is read from `pg_attribute.attidentity` on both sides (a legacy `serial` column also reads
394+
as an `increment` key, so it is not drift against one); a GENERATED identity added or dropped is a
395+
`change-column-identity`, and dropping one is gated behind `--allow drop-identity-default`. Table
396+
and column comments are read from `pg_description` on both sides, and a changed one is a
397+
`change-comment` (`COMMENT ON … IS …`). Clearing a live comment the metadata does not declare
398+
(somebody else may have written it) needs `--allow drop-comment`: without it `meta migrate`
399+
leaves the comment in place, plans the rest and says so in a note, and `meta verify --db` still
400+
reports it as drift. A committed snapshot written before `formatVersion` 4 never recorded comments
401+
or a GENERATED identity, so the committed-snapshot check treats both as unknown rather than absent
402+
and does not compare them; re-run `meta migrate baseline --from-db` to record them. A view's body is compared as Postgres's deparse
403+
(`pg_get_viewdef`) of the shadow's copy against the live one; each view is created under its own
404+
savepoint, so a view whose body reads something only the live database has is left out of the
405+
shadow and compared by its fingerprint instead, without costing the run its table comparison.
406+
Off the shadow path identity and comments are not compared, and views are compared by fingerprint.
407+
No per-shape rule is involved, so `NOW()`/`now()`, `current_date`, a date
408+
literal on a `timestamptz` (read in the session `TimeZone`), `'1e3'` on a double, `'007'` on an
409+
integer, `now() + interval '1 day'`, a regrouped `NOT` and a precedence change are all decided by
410+
the database. A real change of any of them is still reported. The scratch database comes from:
411+
412+
| Source | How |
413+
|---|---|
414+
| An explicit URL | `--shadow-db <url>` on `verify` and on `migrate --from-db` / `--apply` (a plain `meta migrate` reads no database, says so, and refuses the flag), else `SHADOW_DATABASE_URL`, else `migrate.shadowDatabaseUrl` in `.metaobjects/config.json`. Used as given and never dropped; nothing is left in it because the schema is applied inside a rolled-back transaction. |
415+
| `CREATE DATABASE` | With none of the above, a database named `mo_shadow_<random>` is created on the `--db` server when the role may, and **always dropped** afterwards, including on error. |
416+
417+
A run opens its scratch database once, and every comparison it makes uses it: `meta verify --db`
418+
compares the committed snapshot with the database on the same scratch database as the drift check.
419+
420+
An explicitly configured scratch database that cannot be opened, or that the expected schema does
421+
not apply to (an FK to a table outside the run's scope, a missing extension), is an **error** naming
422+
the cause: you asked for a scratch database so the target would not be written to, so the run does
423+
not fall back to the probe on the target. A `CREATE DATABASE` that the role may not run, or that the
424+
schema does not apply to, falls back to the next path, and the output says why.
425+
426+
**2. The temp-object probe, the first fallback.** For each table it creates a **temporary table with
427+
the live table's columns** on the target itself, inside a transaction it rolls back (a savepoint, when the connection is already in one), adds the expected
382428
CHECK to it, creates each expected expression or partial-index predicate as a temporary index,
383429
and reads back `pg_get_constraintdef` / `pg_get_indexdef` / `pg_get_expr`. Those are compared with
384-
the live definitions as plain strings. Nothing persists, and nothing is written to your table.
430+
the live definitions as plain strings. Nothing persists, and nothing is written to your table, but
431+
the probe **needs write privileges on the target** (it creates temporary objects) and **takes brief
432+
locks** on it, which is why a scratch database is preferred.
385433
Two expressions that mean different things never compare equal: a precedence change
386434
(`(a OR b) AND c` against `a OR (b AND c)`), a regrouped `NOT`, and a changed enum-member or regex
387435
literal are all drift.
388436

389-
Where the question cannot be asked, the comparison falls back to a **string normaliser**
390-
(`check-expr-compare.ts`), which rewrites both texts toward one shape. The fallback applies to:
437+
**3. The string normaliser, the last fallback.** Where neither question can be asked, the comparison uses it
438+
(`check-expr-compare.ts`), which rewrites both texts toward one shape. It always applies to:
391439

392440
- SQLite and D1, which store the text as written;
393441
- the default **offline** `meta migrate`, which diffs the committed snapshot and has no database;
@@ -403,6 +451,13 @@ The normaliser cannot tell a precedence change from a bracket-only change (it de
403451
bracket), so on the fallback path `(a OR b) AND c` and `a OR (b AND c)` compare as equal. A fallback
404452
never reports drift of its own: a probe that fails changes nothing about the answer.
405453

454+
The probe answers column defaults the same way. Postgres re-spells a default (`NOW()` becomes
455+
`now()`, `current_date` becomes `CURRENT_DATE`, a date literal on a `timestamptz` becomes a full
456+
timestamp read in the session's `TimeZone`, `'1e3'` on a double becomes `'1000'`), so for a column
457+
with a default on both sides the expected one is set on the scratch table and `pg_get_expr` is
458+
read back. A column rename that carries a CHECK is answered on a scratch copy whose columns are
459+
renamed first.
460+
406461
`meta migrate baseline --from-db` on Postgres closes the offline gap. Where the live database
407462
proves a CHECK, or an expression index's key or predicate, is what the metadata would generate, the
408463
snapshot records the **generated** text rather than Postgres's spelling, so the offline plan

‎server/typescript/packages/cli/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,9 +173,10 @@ Flags:
173173
- `--db <url>` (required or via `$DATABASE_URL` or `.metaobjects/config.json`)
174174
- Supported schemes: `file:`, `libsql:`, `postgres:`, `postgresql:`
175175
- `--dialect sqlite|postgres|d1` — auto-detected from URL scheme; use `d1` for Cloudflare D1
176+
- `--shadow-db <url>` — Postgres scratch database the expected schema is applied to for `--from-db` / `--apply` drift (also `SHADOW_DATABASE_URL` or `migrate.shadowDatabaseUrl`). Read [How CHECK constraints and expression indexes are compared](../../../../docs/features/migrations-and-drift.md#how-check-constraints-and-expression-indexes-are-compared) for the three paths.
176177
- `--out-dir <path>` (default `./.metaobjects/migrations`)
177178
- `--slug <name>` — required when changes are pending (e.g., `add-user-shipping`)
178-
- `--allow <csv>` — destructive-change permissions: `drop-column,drop-table,type-change,drop-index,drop-fk,drop-check,drop-view,drop-view-cascade,adopt-view,nullable-to-not-null,drop-identity-default`
179+
- `--allow <csv>` — destructive-change permissions: `drop-column,drop-table,type-change,drop-index,drop-fk,drop-check,drop-view,drop-view-cascade,adopt-view,nullable-to-not-null,drop-identity-default,drop-unmanaged,drop-comment`
179180
- `--on-ambiguous abort|rename|drop-add` (default `abort`) — non-interactive
180181
- `--rename-table [schema.]old=new` / `--rename-column [schema.]table.old=new` — declare a rename (repeatable), so the migration is `RENAME` instead of drop+add whatever the rename heuristic makes of the names. Refused if nothing matches. See [Renames and populated tables](../../../../docs/features/migrations-and-drift.md#renames-and-populated-tables).
181182
- `--dry-run` — print the SQL pair, write nothing (under `--format json|toon` the pair is the document's `sql` field, so stdout stays one parseable document)

0 commit comments

Comments
 (0)