Skip to content

migrations-and-drift.md

Latest commit

 

History

History
670 lines (555 loc) · 45 KB

migrations-and-drift.md

File metadata and controls

670 lines (555 loc) · 45 KB
Raw

Migrations and drift detection

MetaObjects treats metadata as the source of truth and generated code + database schema as derived. Migration is the build-time pipeline that emits SQL DDL from metadata diffs; drift detection is the cross-cutting discipline that catches divergence between metadata, code, DB, and prompts.

There are 7 drift sources, and the toolchain has a guard for each.

The 7 drift sources

Drift source Caught by When
Code-vs-DB Codegen — the generated SQL DDL is emitted from the same metadata as the entity / table code. Build time
Code-vs-API-doc Cross-port codegen from the same metadata. Build time
DB-vs-metadata meta verify --db (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection view bodies (a changed CREATE VIEW is drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, the deparsed body on Postgres's scratch-database path, a fingerprint mismatch otherwise). A view meta migrate drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored unmodeled view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the metaobjects:verify Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through --db's Kysely-driver introspection — use meta verify --dialect d1 [--d1 <binding>] [--remote] instead (the same wrangler-shelled-out path meta migrate --dialect d1 uses); --remote is required to check the deployed database, not the local wrangler dev shadow copy. Pointing --db file: at wrangler's local D1 state directory (.wrangler/state/**/d1/**) still runs, but only verifies that local copy — verify warns when it detects this. CI on every PR
Migration-vs-metadata The Node meta migrate emits migrations FROM metadata diffs — they cannot drift from metadata by construction. Schema migrations for every port are owned by this Node toolchain (@metaobjectsdev/cli migrate, ADR-0015); the C# and Python migrate surfaces were removed. Build time
Generated-edited @generated headers in emitted code + a committed hash manifest that refuses an edited file by name; in TypeScript only, a three-way merge also preserves hand-edits on the machine that generated the file. Code review
Prompt-vs-payload FR-004 Renderer.verify parses {{...}} references in templates and checks each one exists on the payload VO. Build time + runtime
Generated-vs-runtime Kotlin / Java MetadataStartupValidator (from Spring ApplicationReadyEvent) re-loads metadata at startup and asserts the generated table objects match. App startup

Authoring (no metadata change required)

Migration is invisible to the metadata author — the same Author declaration drives a CREATE TABLE on the first run and an ALTER TABLE on later runs.

{
  "object.entity": {
    "name": "Author",
    "children": [
      { "source.rdb": { "@table": "authors" } },
      { "field.long":   { "name": "id" } },
      { "field.string": { "name": "name", "@required": true, "@maxLength": 200 } },
      { "field.string": { "name": "bio",  "@maxLength": 2000 } },
      { "identity.primary": { "@fields": "id", "@generation": "increment" } }
    ]
  }
}

Add a new email field tomorrow — the next meta migrate emits the ALTER TABLE for the new column.

Migration commands per port

TypeScript

meta migrate                  # diff metadata vs DB → emit migration SQL
meta migrate --dry-run        # preview without writing
meta migrate --dialect d1     # Cloudflare D1 dialect (TS-only)

Dialects: postgres (default), sqlite, d1. MySQL is a code generation and runtime dialect only; meta migrate --dialect mysql is refused with that explanation, and the DDL is yours (docs/recipes/mysql.md). Output lands under the path configured in metaobjects.config.ts (typically ./migrations/<timestamp>__<slug>.sql).

meta migrate apply-pending replays the committed migration files against --db in order, ledger-tracked (_metaobjects_migrations) and transactional — with no diff and no metadata load. For a project whose chain builds the schema, it is the way to provision a fresh or CI database from the committed migrations; run meta verify --replay to know you are one of those. A project adopted with migrate baseline --from-db is not: its chain starts after the schema already existed, so there is nothing for apply-pending to build. meta migrate --apply, by contrast, is diff-first — it authors a new migration from the metadata-vs-DB diff before applying it. apply-pending just runs the pending already-committed files, making it idempotent; --dry-run lists what would run. postgres/sqlite only — on D1 use wrangler d1 migrations apply.

Renames and populated tables

A replay against a fresh database proves a migration applies to no rows. Four kinds of change behave differently once the table has data, and meta migrate handles each one explicitly.

A rename the heuristic does not see loses the data. The diff pairs a dropped column with an added one as a possible rename only when their names are close (first_name / firstname). A pair like origin_city → origin_port is too far apart to be offered at all, so the migration is DROP COLUMN + ADD COLUMN and every value is gone. Declare it:

meta migrate --slug rename-origin --rename-column shipment.origin_city=origin_port
meta migrate --slug rename-stop   --rename-table stop=port_call

Both flags are repeatable and take an optional schema prefix (ops.shipment.origin_city=origin_port). A declared rename is resolved before the heuristic, without asking --on-ambiguous, and needs no --allow drop-column. For a column, table is the table's name in the metadata, so it is the new name when the table is renamed in the same run. A declared rename that finds nothing to rename is refused (exit 1) rather than falling back to drop+add: the old name is not in the database, the new name is not in the metadata, or the rename was already applied. A column's type, nullability and default must not change in the same run. Migrate the rename first, then the shape.

A renamed column carries the constraints named after it. CHECK and FK names derive from the physical column (reviews_rating_numeric_chk, reviews_book_id_fk), so the new name is a new constraint name, but it is the same rule over the same data and needs no --allow drop-check or --allow drop-fk. On Postgres the migration is RENAME COLUMN followed by RENAME CONSTRAINT (Postgres rewrites the constraint body itself); on SQLite and D1 the table is rebuilt with the constraint under its new name while the column's values are copied across. This applies to any resolved rename, declared or accepted from the heuristic. A constraint whose rule changed beyond the rename (a validator.numeric whose max moved in the same run) is not carried: its drop stays gated, so migrate the rename first, then the rule.

The flags are deliberately not metadata. A rename is an event in the migration history, not a lasting fact about the model.

Three changes apply to an empty table and fail on a populated one. The diff cannot see rows, so it warns instead of refusing. On Postgres the migration file also carries the preparation step as a comment above the statement:

Change Fails when Preparation the comment names
New required field with no @default (ADD COLUMN … NOT NULL) the table has any row give the field a @default, or add it optional, backfill, then make it required
Optional field made required (SET NOT NULL, needs --allow nullable-to-not-null) any row holds NULL UPDATE … SET <col> = <value> WHERE <col> IS NULL
New CHECK on an existing table, e.g. an enum narrowed to fewer @values any row violates it SELECT * FROM … WHERE NOT (<check>), then fix those rows

Postgres runs each migration in a transaction, so a failure rolls the migration back and loses nothing. The warning moves the discovery from deploy time to authoring time.

meta verify --replay — the chain applies from empty

meta verify --replay              # the committed chain applies to an empty database
meta verify --replay-snapshot     # ...and reproduces the committed schema snapshot

A committed chain can be broken without anything saying so. meta migrate writes a DROP TABLE "x" whenever x is in the live database and absent from the metadata — including when no migration in the chain ever created x, which is what happens when another tool owns that table. Every incremental migrate keeps succeeding against the database that already has x; the chain only fails the day somebody provisions a fresh one (#313).

--replay replays the whole committed chain into an empty throwaway database and asserts it applies. Nothing is compared. This is the gate for the promise apply-pending makes.

--replay-snapshot additionally asserts the replayed schema equals the committed snapshot. That catches a different thing: hand-edited structural DDL — a committed up.sql someone changed so the chain still applies but no longer produces the schema the snapshot records.

Neither needs a database. The engine is local and disposable — real Postgres in-process via PGlite, a throwaway temp file for sqlite — so there is nothing to provision, no credentials, and no scratch database to collide with or accidentally drop. PGlite is an optional peer dependency of @metaobjectsdev/migrate-ts: install it (npm i -D @electric-sql/pglite) to replay a postgres chain. Without it the gate exits 2 and says so.

With no --db there is no URL to infer a dialect from, so: --dialect wins, else migrate.dialect from .metaobjects/config.json, else the run refuses naming --dialect. --migration-format flyway and --dialect d1 are refused — replay those with flyway migrate and wrangler d1 migrations apply against a scratch database.

Exit codes follow verify's convention: a chain that fails to apply, or a snapshot mismatch, is drift → 1; an engine that cannot start is operational → 2. An empty chain, or a missing snapshot, passes → 0, and says which, because a gate that is silent when it checked nothing cannot be told apart from one that passed.

--replay-snapshot does not apply to a project adopted with migrate baseline --from-db. Such a project's snapshot is the whole introspected database while its chain is empty, so the comparison cannot pass by construction. Use --replay there. This is documented rather than auto-detected: the only signal that could distinguish the two lives in the target database's ledger, and this gate runs against a fresh engine that has no ledger at all.

If the gate goes red, applied migrations are checksum-immutable — hand-editing a committed up.sql is rejected on any database that already applied it. The supported fix is a compensating migration: author a new one that creates the missing object, or that supersedes the bad drop.

--allow drop-unmanaged — the same problem, caught earlier

meta migrate --from-db refuses to author a drop for a table or view the committed snapshot never contained, because that is precisely the statement that cannot replay:

migrate: refusing to drop public.other_owned_table — absent from the committed schema
snapshot, so this toolchain never managed it and the migration could not replay against
a database where it never existed. Re-run with '--allow drop-unmanaged' if the drop is
intended.

Pass --allow drop-unmanaged when the drop is genuinely what you want. The refusal does not fire for brownfield projects: baseline --from-db puts the foreign table in the snapshot, and a project declaring migrate.scope carries its out-of-scope entries forward, so both read as managed. It fires precisely when nothing ever claimed the object. A project with no snapshot yet — every greenfield first run — is unaffected.

Emitted forward drops carry IF EXISTS

Every forward DROP TABLE / DROP VIEW / DROP INDEX / DROP CONSTRAINT a migration emits is now IF EXISTS, in both dialects, so an already-absent object cannot break a replay. Down statements stay bare deliberately: rollbackTo runs down.sql and the ledger delete in one transaction, so a silently-no-op down would still record the rollback as done — rollback is the one place a loud failure is load-bearing.

A chain that creates a table or view in a non-default schema now also emits CREATE SCHEMA IF NOT EXISTS "<schema>"; ahead of it. Without that, an @schema project's chain could never apply to a virgin database, because nothing ever created the schema. The down does not drop the schema: it may hold objects this tool does not own and cannot restore.

D1: rebuilding a foreign-key-referenced table

Cloudflare D1 applies each migration inside its own implicit transaction, and SQLite ignores PRAGMA foreign_keys while a transaction is open. The SQLite table-rebuild recipe (used for a CHECK, column type/nullability/default, foreign-key, or field.enum values change) relies on PRAGMA foreign_keys = OFF taking effect before it drops and recreates the table — which does not happen on remote D1.

meta migrate --dialect d1 handles this by auto-generating a cascade (#241, closing #226's residual under-refuse gap below) instead of refusing outright. When a change would rebuild a table that another table's foreign key references, the emitter rebuilds that table together with every table that transitively references it, in one pass: the affected tables are dropped referrers-first and recreated parents-first, under PRAGMA defer_foreign_keys = ON so every foreign-key check defers to the end of D1's implicit transaction instead of firing mid-rebuild. The result applies cleanly against a populated production database and re-converges — a follow-up meta verify/meta migrate sees no drift. The cascade is built over the union of the actual (live) and expected (target) schemas' foreign-key graphs, so it also covers the case a target-schema-only check would miss: a single migration that both rebuilds a referenced table and drops the referencing foreign key in the same run. A projection/view that reads a rebuilt table is dropped before the rebuild and recreated after — for the CHECK/FK/enum-values rebuild class as well as column changes — so a dependent view is never stranded mid-migration.

The one case still hand-written: a multi-table foreign-key cycle (table A references B references … references A, two or more tables). A cycle has no parents-first rebuild order, so meta migrate --dialect d1 still refuses at generation time — hand-write the migration (drop the foreign key on one side of the cycle, rebuild the tables, then restore it) or break the cycle in your metadata. A self-referencing table (a table whose own foreign key targets itself) is not a cycle in this sense and is handled by the cascade like any other rebuild.

A moved primary key (adoption-time refusal)

The diff/emit has no add-primary-key / drop-primary-key change kind, so an existing table whose live PRIMARY KEY differs from the metadata identity cannot be expressed as a migration. When adopting such a database (--from-db), meta migrate now refuses at generation time instead of emitting un-appliable SQL — detect-and-refuse, the same arc as #226→#241 for the D1 foreign-key rebuilds above. It throws a PrimaryKeyChangeError (naming the table and both PKs), the CLI catches it and exits 1 (#258).

Previously the move degraded silently into an add-column + drop-column: the old PK column and its constraint were dropped while the new column was never made primary key, leaving the table with no primary key, so every foreign key referencing it failed at apply (there is no unique constraint matching given keys). This surfaces only when adopting an existing database whose PK disagrees with the metadata — a greenfield create-table carries its primary key inline.

The check is engine-wide (postgres / sqlite / d1 — the diff is shared) and runs after rename detection, mapping live PK column names through any detected rename-column change, so a primary-key column that was merely renamed (the engine preserves the PK through RENAME COLUMN) is not mistaken for a move. The read-only meta verify / drift path does not set the refusal flag, so verify keeps reporting primary-key drift rather than throwing. Auto-migrating the move (adding the add-primary-key / drop-primary-key change kinds) is a documented future follow-up.

A legacy serial primary key (adoption-time refusal)

When migrating against a live Postgres database whose primary key is a legacy serial / bigserial column — one carrying a live nextval(...) default — and the metadata declares that identity.primary without @generation, the diff would otherwise emit ALTER COLUMN … DROP DEFAULT. That is destructive: every insert that omits the id starts failing. The missing @generation is genuinely ambiguous — it reads identically whether the author simply never declared it (and wants to keep auto-increment) or deliberately dropped it (to move the column onto app-assigned ids) — so meta migrate refuses rather than guessing, the same detect-and-refuse arc as the primary-key move above. Declare @generation: increment on the identity to keep the sequence, or pass --allow drop-identity-default if removing auto-increment is intentional. An identity that does declare @generation: increment never reaches this gate (its default diff is skipped), so only the undeclared case fires.

Java

Schema migrations for Java projects are owned by the TypeScript toolchain (@metaobjectsdev/cli migrate). The Java Maven plugin's meta:migrate goal was removed, and per ADR-0015 the OMDB runtime auto-create path was removed too — OMDB is pure data-access (CRUD/query/codec/transactions). Provision the schema by applying the TS-produced DDL/migrations to the database the Java service connects to.

Use the TS CLI against the same database the Java service connects to:

meta migrate --db postgresql://... --slug initial   # emit migration SQL
meta migrate --db postgresql://... --apply          # apply pending migrations

Kotlin

Kotlin schema migrations follow the same story as Java: the Java Maven plugin's meta:migrate mojo was removed and there is no Kotlin-specific migrate command. Schema migrations are owned by the TS toolchain:

meta migrate --db postgresql://... --slug initial
meta migrate --db postgresql://... --apply

C#

Schema migrations for C# projects are owned by the TypeScript toolchain (@metaobjectsdev/cli migrate). Per ADR-0015 the C# migrate engine and the --from-db introspection surface were removed — the C# CLI (dotnet meta) is gen / verify only. Use the TS CLI against the same database the C# service connects to:

meta migrate --db postgresql://... --slug initial   # emit migration SQL
meta migrate --db postgresql://... --apply          # apply pending migrations

Python

Schema migrations for Python projects are owned by the TypeScript toolchain (@metaobjectsdev/cli migrate). Per ADR-0015 the Python migrate module was removed — the metaobjects console script is gen / verify only (the runtime is pure data-access via ObjectManager). Use the TS CLI against the same database the Python service connects to:

meta migrate --db postgresql://... --slug initial   # emit migration SQL
meta migrate --db postgresql://... --apply          # apply pending migrations

SQLite version floor

SQLite parses every CREATE TABLE / CREATE VIEW in sqlite_master when a database is OPENED, so one statement an engine cannot parse makes the whole file unopenable there (malformed database schema (<name>) - near "…") — not just that table. What meta migrate emits for sqlite / d1 is therefore held to a floor:

Construct Needs Emitted when
ALTER TABLE … DROP COLUMN 3.35 a column is dropped (--allow drop-column)
ALTER TABLE … RENAME COLUMN 3.25 a declared column rename
aggregate FILTER (WHERE …), NULLS LAST 3.30 a projection view with origin.* scoping / ordering
json_each, json_group_array (JSON1) 3.35 in practice (JSON1 built in since 3.38; earlier builds usually enable it) a projection view using origin.collect

The floor is SQLite 3.35, for tables and views alike (Ubuntu 22.04 ships 3.37.2; D1's baseline is 3.44). Up to 1.0.9 a view using origin.collect ordered its elements with an in-aggregate ORDER BY (json_group_array(x ORDER BY …)), which needs SQLite 3.44. A collect column is now built in two steps: the grouped query gathers each element together with its sort keys, and an outer query sorts them, which every SQLite from 3.35 runs. The next meta migrate against a database made before that change replaces such views once; the re-diff is empty afterwards.

A field.timestamp whose default is "now" (@autoSet, or @default: "now") defaults to strftime('%Y-%m-%dT%H:%M:%fZ','now'), the same ISO UTC form generated code writes, rather than CURRENT_TIMESTAMP's YYYY-MM-DD HH:MM:SS. A column that already defaults to CURRENT_TIMESTAMP is not rebuilt for the spelling: the two are treated as the same default, and the ISO one arrives the next time something else rebuilds that table.

Null-safe (in)equality in a derived CHECK (validator.requiredWhen, validator.presentIff) is spelled with SQLite's own IS NOT / IS, which every version parses. Up to 1.0.8 it was the standard IS [NOT] DISTINCT FROM, which needs SQLite 3.39: such a database cannot be opened by an older SQLite (Python 3.10's stdlib, Ubuntu 22.04's sqlite3). The next meta migrate against it re-spells those CHECKs ONCE — a table rebuild that is not gated behind --allow drop-check and raises no data hazard, because the rule is the same — and the re-diff is empty afterwards. Postgres keeps IS [NOT] DISTINCT FROM.

How CHECK constraints and expression indexes are compared

Postgres rewrites an expression before it stores it ("k" IN ('x') becomes k = 'x'::text, a IS NOT DISTINCT FROM b becomes NOT (a IS DISTINCT FROM b), an IN list becomes = ANY (ARRAY[…]), and casts and brackets are added), so the text meta migrate generates never comes back from the database as written. On Postgres, a plan against the database (meta verify --db, meta migrate --from-db and --apply) therefore asks Postgres instead of guessing. There are three ways it asks, tried in this order, and every Postgres run of meta verify --db, meta migrate and meta migrate baseline --from-db says which one ran.

1. A scratch (shadow) database, the primary path. The practice Atlas, Prisma and migra share: the model's full expected schema (the DDL meta migrate emits for an empty database) is applied to a throwaway Postgres database, in one transaction that is always rolled back, and read with the same introspector that reads your live database. Drift is then the same diff run on two catalogs, introspect(shadow) as the expected side against introspect(live), for every object in the run's scope: tables, columns and types, nullability, defaults, identity, CHECKs, primary/foreign/unique constraints and their actions, indexes, view bodies and comments. Both sides are Postgres's own spelling, so a shape nobody wrote a rule for (int4 against integer, ON DELETE SET NULL, a view body written differently from Postgres's deparse) needs none. Model-only knowledge stays an input to that diff, not a second comparison: renames, @unmanaged and scope are applied as always. Identity is read from pg_attribute.attidentity on both sides (a legacy serial column also reads as an increment key, so it is not drift against one); a GENERATED identity added or dropped is a change-column-identity, and dropping one is gated behind --allow drop-identity-default. Table and column comments are read from pg_description on both sides, and a changed one is a change-comment (COMMENT ON … IS …). Clearing a live comment the metadata does not declare (somebody else may have written it) needs --allow drop-comment: without it meta migrate leaves the comment in place, plans the rest and says so in a note, and meta verify --db still reports it as drift. A committed snapshot written before formatVersion 4 never recorded comments or a GENERATED identity, so the committed-snapshot check treats both as unknown rather than absent and does not compare them; re-run meta migrate baseline --from-db to record them. A view's body is compared as Postgres's deparse (pg_get_viewdef) of the shadow's copy against the live one; each view is created under its own savepoint, so a view whose body reads something only the live database has is left out of the shadow and compared by its fingerprint instead, without costing the run its table comparison. Off the shadow path identity and comments are not compared, and views are compared by fingerprint. No per-shape rule is involved, so NOW()/now(), current_date, a date literal on a timestamptz (read in the session TimeZone), '1e3' on a double, '007' on an integer, now() + interval '1 day', a regrouped NOT and a precedence change are all decided by the database. A real change of any of them is still reported. The scratch database comes from:

Source How
An explicit URL --shadow-db <url> on verify and on migrate --from-db / --apply (a plain meta migrate reads no database, says so, and refuses the flag), else SHADOW_DATABASE_URL, else migrate.shadowDatabaseUrl in .metaobjects/config.json. Used as given and never dropped; nothing is left in it because the schema is applied inside a rolled-back transaction.
CREATE DATABASE With none of the above, a database named mo_shadow_<random> is created on the --db server when the role may, and always dropped afterwards, including on error.

A run opens its scratch database once, and every comparison it makes uses it: meta verify --db compares the committed snapshot with the database on the same scratch database as the drift check.

An explicitly configured scratch database that cannot be opened, or that the expected schema does not apply to (an FK to a table outside the run's scope, a missing extension), is an error naming the cause: you asked for a scratch database so the target would not be written to, so the run does not fall back to the probe on the target. A CREATE DATABASE that the role may not run, or that the schema does not apply to, falls back to the next path, and the output says why.

2. The temp-object probe, the first fallback. For each table it creates a temporary table with the live table's columns on the target itself, inside a transaction it rolls back (a savepoint, when the connection is already in one), adds the expected CHECK to it, creates each expected expression or partial-index predicate as a temporary index, and reads back pg_get_constraintdef / pg_get_indexdef / pg_get_expr. Those are compared with the live definitions as plain strings. Nothing persists, and nothing is written to your table, but the probe needs write privileges on the target (it creates temporary objects) and takes brief locks on it, which is why a scratch database is preferred. Two expressions that mean different things never compare equal: a precedence change ((a OR b) AND c against a OR (b AND c)), a regrouped NOT, and a changed enum-member or regex literal are all drift.

3. The string normaliser, the last fallback. Where neither question can be asked, the comparison uses it (check-expr-compare.ts), which rewrites both texts toward one shape. It always applies to:

  • SQLite and D1, which store the text as written;
  • the default offline meta migrate, which diffs the committed snapshot and has no database;
  • a read-only connection or a hot standby, and a role without the TEMP privilege;
  • a table the role cannot read (no SELECT on it, or no USAGE on its schema): that table only, the other tables are still asked;
  • an expected expression that does not compile against the live columns (the column diff reports that case);
  • an expression containing a ; anywhere, quoted or not, or an index @using that is not a plain access-method name: neither is ever sent to the database.

The normaliser cannot tell a precedence change from a bracket-only change (it deletes every bracket), so on the fallback path (a OR b) AND c and a OR (b AND c) compare as equal. A fallback never reports drift of its own: a probe that fails changes nothing about the answer.

The probe answers column defaults the same way. Postgres re-spells a default (NOW() becomes now(), current_date becomes CURRENT_DATE, a date literal on a timestamptz becomes a full timestamp read in the session's TimeZone, '1e3' on a double becomes '1000'), so for a column with a default on both sides the expected one is set on the scratch table and pg_get_expr is read back. A column rename that carries a CHECK is answered on a scratch copy whose columns are renamed first.

meta migrate baseline --from-db on Postgres closes the offline gap. Where the live database proves a CHECK, or an expression index's key or predicate, is what the metadata would generate, the snapshot records the generated text rather than Postgres's spelling, so the offline plan compares like with like and a baselined project starts clean. Anything it cannot prove keeps Postgres's text. If the metadata does not load, the baseline is written as before.

Drift verify commands per port

A bare verify runs the same default gate set in every port: codegen drift, prompt-template drift and the requirement ledger. What a check does when the run has no configuration for it differs by port. TypeScript, Java and Python skip it with a one-line note and do not fail: TypeScript skips the codegen gate when there is no metaobjects.config.ts (and its template gate passes with a note when the model has no template.* nodes), Java skips the template gate when templateRoot is unset, and Python skips the template gate when there is no metaobjects.config.yaml and no --prompts. C# does not skip: a bare dotnet meta verify needs a prompts/ (else templates/) directory, or --templates <root>, and exits 2 with templates directory not found when neither exists (full contract: cli.md, "verify is one verb with explicit subverbs").

Port Bare command Config each check needs One check alone
TypeScript meta verify Codegen: metaobjects.config.ts. Templates: the template.* nodes in the model. meta verify --codegen, meta verify --templates
Java mvn metaobjects:verify Codegen: the <generators> in the plugin configuration. Templates: -Dmeta.verify.templateRoot. -Dmeta.verify.mode=codegen, -Dmeta.verify.mode=templates
Kotlin mvn metaobjects:verify (the same goal as Java) As Java. MetadataStartupValidator adds a startup check. As Java
C# dotnet meta verify Codegen: an output directory (csharp.outDir in .metaobjects/config.json, or --out <dir>) and codegen/Generators.cs. Templates: --templates <root>, or a prompts/ (else templates/) directory in the working directory; with neither, a bare run exits 2. dotnet meta verify --codegen, dotnet meta verify --templates <root>
Python metaobjects verify Codegen: metaobjects.config.yaml. Templates: metaobjects.config.yaml or --prompts <dir>. metaobjects verify --codegen, metaobjects verify --templates

Schema drift is a separate, opt-in check and Node-only (ADR-0015): meta verify --db <url> introspects the live database and reports database-versus-metadata drift. The other four ports reject --db (C#, Python and the Maven goal exit 2 or fail with a pointer to the TypeScript CLI); the live-DB-schema metaobjects:verify goal was removed along with the runtime auto-create validator.

Generated-vs-runtime: Kotlin startup validator

KotlinSpringConfigGenerator emits a @Configuration class that re-loads metadata at Spring ApplicationReadyEvent and asserts that the generated Table objects still match the metadata. If a developer hand-edited a generated table and the regen didn't catch it (or a CI race shipped a stale build), the app fails-fast at startup instead of silently serving wrong data.

// GENERATED — MetadataExposedConfig.kt
@Configuration
class MetadataExposedConfig(private val dataSource: DataSource) {
    init { Database.connect(dataSource) }

    @EventListener(ApplicationReadyEvent::class)
    fun validateMetadata() {
        val loader = loadResources("app", listOf("meta.entities.json"))
        MetadataStartupValidator.validate(loader)
    }
}

Migrating views (projections)

A projection's CREATE VIEW is generated from its origin.* children, so meta migrate owns the view. Three things are worth knowing.

Append projection fields; don't insert them. Postgres can update a view in place (CREATE OR REPLACE VIEW) only when every existing output column keeps its name, type and position and any new ones are added at the end. A view's columns come out in projection declaration order, so:

  • adding a field at the end of a projection, or changing a body without changing a column (a different join path, an origin.aggregate @filter) → the view can be replaced in place. Dependent views, grants, and the view's identity all survive.
  • inserting a field in the middle, reordering, renaming, removing or retyping one → the view must be dropped and recreated, which is destructive to anything that depends on it.

Postgres decides which, and only a plan against the database can ask it. The types a query produces are Postgres's to say: max of a numeric(9,2) column is numeric, COALESCE of a varchar(40) column and a literal is a bare character varying, and json is not jsonb. So meta migrate does not predict them. When a live view exists and its definition differs, a plan against the database (meta migrate --from-db, --apply, and the drift report of meta verify --db) creates the new definition as a temporary view inside a transaction it rolls back, reads the column names and types Postgres gave it, and compares them with the live view's. Planning on this path therefore writes inside a rolled-back transaction (a savepoint, when the connection is already in one); nothing persists. A replace is planned only on that proof. When the question cannot be asked, the plan is a DROP VIEW and a CREATE VIEW:

  • the default offline meta migrate, which diffs the committed snapshot and has no database;
  • a role that may not create a temporary view (no TEMPORARY privilege on the database);
  • a new definition that does not compile against the schema as it stands, because it reads a column the same migration adds;
  • an @sql body, in a migration that also changes any table's columns (adds, drops, renames or retypes a column, changes its nullability or default, or drops a table). meta migrate does not parse the body, so it cannot tell which tables the body joins. The probe sees the schema before those changes, the replace runs after them, and Postgres refuses to retype or drop what a standing view reads. A derived projection or report view lists every table its new definition reads, so a change to some other table does not do this to it;
  • a body that reads columns it does not name (SELECT *, NATURAL JOIN), in a migration that also alters any table or any other view. Its column list comes from what it reads, so an answer taken before the change would not hold;
  • any view, derived ones included, whose definition in the database reads a table the new definition no longer reads, in a migration that also retypes, drops or renames a column of that table, or drops the table. The probe proves the new definition only. The old view stands while the table changes, and Postgres refuses to retype or drop what it reads. meta migrate finds the table by name in the old view's stored definition, so a view that merely names something spelled like that table is dropped and created too. Renaming the table alone does not do this: a view follows a renamed table.

A drop and a create of the same view needs no --allow. It has two costs a replace does not:

  • Privileges granted on the view are discarded. DROP VIEW removes the view's grants with it, and meta migrate writes no GRANT. A role that had SELECT on the view loses it, and nothing reports that. In an offline plan every changed Postgres view takes this path.
  • If something depends on the view, Postgres refuses the DROP VIEW at apply and names the dependent.

Plan with --from-db to get the in-place replace when it is legal, which keeps both. Otherwise grant the privileges again after the migration is applied.

A view whose definition does not change is dropped and created too, when the migration retypes, drops or renames a column the view reads: Postgres refuses to retype or drop it while the view stands. The table may be one the model does not list for the view, which is the case of an @sql body that joins it. meta migrate finds it by name in the view's stored definition, and recreates the view only when that definition also names the changed column. The same two costs apply.

SQLite and D1 have no CREATE OR REPLACE VIEW: every replaced view is written as DROP VIEW and CREATE VIEW there, so there is no refusal to avoid and no probe. They refuse something else: a DROP COLUMN, or the rebuild of a table, while a view that reads the table stands. So a view is dropped before such a change and created after it, for every table its definition in the database names, including one the new definition no longer reads and one an @sql body joins. The D1 foreign-key cascade does the same for every table it rebuilds.

A cascading drop is blocked, loudly. If dropping a view would destroy dependent objects — another application's view, a materialized view — meta migrate blocks and names every one of them. Proceeding requires --allow drop-view,drop-view-cascade, and the emitted migration carries a WARNING: CASCADE DROP banner listing what it destroys. MetaObjects does not manage those objects and cannot restore them. --allow drop-view alone never cascades.

One-time upgrade step (--allow adopt-view). Managed views carry a MetaObjects fingerprint in their COMMENT ON VIEW; that fingerprint — not the view's SQL text — is how migrate knows whether a view is current. (It cannot use the text: Postgres does not store view SQL, it stores a parse tree and regenerates the SQL in its own style, so what you wrote never comes back.) A view with no fingerprint is either hand-written or was created before fingerprinting existed, and those are indistinguishable — so migrate fails closed rather than overwrite somebody's hand-written SQL:

meta migrate --allow adopt-view      # once per environment, after upgrading

That stamps the existing views. Afterwards they converge silently. An adopted view follows the rule above, whether the tool derives its body or it is an @sql one: replaced in place when the database shows its columns are kept, dropped and recreated otherwise, including whenever the probe cannot run. This is also what closes the loop on the doctrine in downstream-metadata-decisions.md: a hand-written view sitting where a projection expects one is now visible to meta verify --db as drift, instead of being silently invisible to it.

Genuinely-irreducible views and externally-owned objects (@sql / @unmanaged, #208). Some read models cannot be expressed as origin.* (a recursive CTE, a window function, a set operation). Rather than hand-write such a view outside the tool — where it degrades to accidentally-unmanaged — carry its body in a source.rdb @sql attribute. The tool registers, fingerprints, and drift-checks that verbatim body (never parsing it) exactly like a synthesized view: it emits CREATE VIEW … AS <your body> with a fingerprint stamp, a second migrate is a no-op, and a pre-existing hand-written view at that name is adopted via the same --allow adopt-view step above. An @sql view may carry an extends-bound identity.primary / fields (for row identity and read-model shape) without the tool mis-synthesizing a body. For a DB object whose DDL is owned entirely elsewhere (Flyway, a hand-migration) — a view or a table — mark its source @unmanaged: true: meta migrate never creates, drops, or drift-checks it, and meta verify --db reports it as external (declared). The two are mutually exclusive. See ADR-0043.

Verified by

The following conformance fixtures gate this feature's behavior across ports:

Schema migration (fixtures/persistence-conformance/migrations/)

  • bootstrap-canonical-from-empty.yaml — full-CREATE bootstrap from an empty database
  • add-nullable-column.yaml — incremental ALTER TABLE … ADD COLUMN for a new nullable field
  • drop-table-blocked-without-allow.yaml — destructive operations require an explicit allow-flag
  • noop-converged-canonical.yaml — the idempotence gate: apply the whole canonical schema, then diff the same metadata against the database it just produced. A converged schema must emit zero SQL. This is what catches any asymmetry between what emit writes, what introspect reads back, and what the expected schema models — the class of bug that makes meta migrate re-propose (and on SQLite, destructively rebuild) unchanged tables forever.

Template drift (fixtures/verify-conformance/) — the Renderer.verify engine asserts that every variable, section, partial, and required-tag in a template resolves against its declared payload. 37 fixtures, grouped:

  • Variables: verify-var-known-clean, verify-var-unknown, verify-var-implicit-iterator-clean, verify-var-unescaped-and-triple-unknown
  • Dotted paths: verify-dotted-path-clean, verify-dotted-path-head-nonfield, verify-dotted-path-tail-nonfield
  • Sections: verify-section-over-nonfield, verify-section-pushes-element-clean, verify-section-element-nonfield, verify-nested-sections-clean, verify-nested-section-nonfield, verify-parent-context-in-section-clean, verify-scalar-section-conditional-clean, verify-scalar-section-conditional-nonfield
  • Inverted sections: verify-inverted-section-clean, verify-inverted-section-over-nonfield
  • Positional accessor (-last): verify-positional-accessor-clean, verify-positional-accessor-transparent-sections-clean, verify-positional-accessor-outside-element, verify-positional-accessor-object-section, verify-positional-accessor-scalar-list, verify-positional-accessor-nested-scalar-list
  • Partials: verify-partial-no-provider, verify-partial-unresolved, verify-partial-resolved-clean, verify-partial-resolved-drift, verify-partial-in-section-context
  • Required slots / tags: verify-required-slot-used-clean, verify-required-slot-unused, verify-required-slot-via-section, verify-required-tags-present, verify-required-tag-in-partial, verify-required-tag-self-closing, verify-required-tag-missing-open, verify-required-tag-missing-close, verify-required-tag-prefix-no-overmatch

Browse the full set under fixtures/verify-conformance/.

Cross-port runner coverage: TS / Java / Kotlin / C# / Python all execute these via their respective conformance runners. See docs/CONFORMANCE.md for the per-port pass/skip ledger.

See also