diff --git a/CHANGELOG.md b/CHANGELOG.md index 122f58d7b..e6fdb9615 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,27 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi one over measures as a `number` member (one reading a dimension is refused). Loader rule M11, report rule R13 and a conformance fixture per refusal, in every loader port. +- **`verify.handEdits`: a hand-edited generated file can fail `meta verify`.** `meta verify + --codegen` already listed each generated file carrying a hand edit, but always exited 0, so a + rule changed by hand in generated code (a `max(32)` made `max(64)`) passed the gate. Set + `"verify": { "handEdits": "fail" }` in `.metaobjects/config.json` and it exits 1; `"notice"` + keeps the report and exit 0. `--allow-hand-edits` relaxes `fail` for one run, and the existing + `--forbid-hand-edits` still forces a failure for one run (passing both is refused). The report + names each file, how to see the edit, how to drop it (or move a changed rule into the + metadata), and how to keep it. A project without the key behaves exactly as before. The C#, + Java, Kotlin and Python gates compare generated files byte for byte and already fail on any + edit; the docs now say so. +- **`migrate.foreignKeyIndexes`: an index on every foreign key, without the `efcore` renames.** + With it on, `meta migrate` gives each enforced foreign key that the primary key, a unique key or + an index does not already lead with a `__idx` index. The Drizzle and Prisma + generators, the C# `AppDbContext` and the Kotlin Exposed tables declare the same indexes under + the same names; Java and Python map no indexes. It adds nothing under the `efcore` conventions + (which already index every foreign key) or on MySQL. A project without the key is unchanged and + keeps the unindexed-foreign-key advisory, which now names the option. Gated by two new + persistence-corpus migration scenarios (adding the indexes to an existing schema, and the + converged no-op). Java's config reader now carries the `migrate` block's + `foreignKeyIndexes`, `conventions` and `dialect` on the loaded root (`MigrateSettings`). + - **`meta init --skills-dir ` and `meta agent-docs --skills-dir `: install the skills for any coding assistant.** The `metaobjects-*` skills went only to `.claude/skills/`, which Claude Code reads. `--skills-dir .agents/skills` (or any project-relative directory) puts them @@ -89,6 +110,10 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi a barrel) now fail with the collision error; set `outputLayout: "package"` (and drop the barrel, or rename one), and update imports of the old `AcmeAlphaNote` names. An owned generator copy is refreshed with `meta eject --list`. +- **`meta init` writes stricter defaults for a new project.** The scaffolded + `.metaobjects/config.json` now carries `"verify": { "handEdits": "fail" }` and (except on + MySQL) `"migrate": { "foreignKeyIndexes": true }`, each as its own visible setting. An existing + config is left as it is. - **Agent context routes report work to the reporting vocabulary.** The `meta init` agent guide and the `metaobjects-authoring` skill now send grouped totals, dashboard numbers and read-only report views to `object.report` (`references/declaring-reports.md`) before the `@sql` escape, diff --git a/agent-context/skills/metaobjects-verify/SKILL.md b/agent-context/skills/metaobjects-verify/SKILL.md index fcdb55024..14b1f94cc 100644 --- a/agent-context/skills/metaobjects-verify/SKILL.md +++ b/agent-context/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index da030620d..bf3f4b202 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -35,7 +35,7 @@ regenerate with `ls -d fixtures//*/ | wc -l` for directory-shaped corpor | [`fixtures/jinja-verify-conformance/`](../fixtures/jinja-verify-conformance/) (ADR-0061 — the payload check over a Jinja body) | 31 | ✓ | refuses Jinja | refuses, via Java | refuses Jinja | ✓ | | [`fixtures/extract-conformance/`](../fixtures/extract-conformance/) | 58 | ✓ | ✓ | inherits the shared JVM engine | ✓ | ✓ | | [`fixtures/output-prompt-conformance/`](../fixtures/output-prompt-conformance/) | 17 | ✓ | ✓ | ✓ | ✓ | ✓ | -| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 62 (55 query + 7 migration) | all 62; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 55 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) | +| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 64 (55 query + 9 migration) | all 64; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 55 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) | | [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 123 (31 core + 10 tph + 9 m2m + 5 jsonb + 2 write-through + 2 secret-fields + 2 enum-default + 12 projection + 37 report + 11 decimal + 2 duration) | ✓ (Fastify reference + generated lane); `report/`, `decimal/` and `duration/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/`, `decimal/` and `duration/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (Kestrel + Npgsql); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | | [`fixtures/decimal-wire/`](../fixtures/decimal-wire/) | 1 vector file (format, parse, filter operand, 15-digit read) | ✓ (`runtime-ts` and the generated helper) | ✓ (`CanonicalDecimal`) | ✓ (the generated wire file) | ✓ (`DecimalWire`) | ✓ (`decimal_wire`) | | [`fixtures/duration-wire/`](../fixtures/duration-wire/) | 1 vector file: the `wire` vectors (parse, precision, refused) every port passes, and the `codecs` groups (Postgres text read and write, stored microseconds) each port passes only for the codecs it has | ✓ (`metadata` / `runtime-ts` and the generated helper) | ✓ | ✓ | ✓ | ✓ | @@ -423,9 +423,9 @@ All 37 fixtures → [features/migrations-and-drift.md](features/migrations-and-d may carry `"isArray": true`; with `fields` it is a list of objects, without it a list of scalars. The six `verify-positional-accessor-*` cases pin where `-last` is available. -### `fixtures/persistence-conformance/` (62 — 55 query + 7 migration) +### `fixtures/persistence-conformance/` (64 — 55 query + 9 migration) -- `migrations/*` (7) → [features/migrations-and-drift.md](features/migrations-and-drift.md) (schema migration section) +- `migrations/*` (9) → [features/migrations-and-drift.md](features/migrations-and-drift.md) (schema migration section) - `queries/*` (55) → [features/source-kinds.md](features/source-kinds.md) (query semantics against `source.rdb`) **Engines beyond Postgres (FR-044 reports, decimals).** The twenty-four `queries/report-*.yaml` @@ -725,7 +725,7 @@ JSON and verify payload feeds it unchanged. ## Orphaned fixtures (tested but not yet documented) The fixtures in the nine corpora mapped above (metamodel 510 + yaml 16 + verify 45 -+ render 18 + persistence 62 + api-contract 123 + source-resolution 25 + scope 10 + ++ render 18 + persistence 64 + api-contract 123 + source-resolution 25 + scope 10 + dependency 47) each map to a feature doc, and so do the two requirement corpora, `shared-model-conformance`, the `cube-model` corpus and `explorer-conformance`, whose case lists live in their own READMEs. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider diff --git a/docs/features/cli.md b/docs/features/cli.md index 0eece216b..d8aa76528 100644 --- a/docs/features/cli.md +++ b/docs/features/cli.md @@ -53,7 +53,7 @@ vocabulary everywhere, each port implementing the modes it supports. | Subverb | What it checks | Touches a DB? | |---|---|---| | `verify --db` | **Schema drift** — does the live database (or snapshot) match the metadata? (migrate engine, ADR-0015) | yes | -| `verify --codegen` | **Codegen drift** — regenerate from metadata into a temp dir and fail if it differs from the committed generated output. Catches "metadata changed but `meta gen` wasn't re-run". A hand-edited generated file is NOT drift — `meta gen` three-way-merges hand edits by design, so the gate compares the *generated contribution* against the committed `.gen-state/.hashes.json` rather than the file byte-for-byte. It still LISTS every generated file carrying a hand edit (count + paths, and how to see the edit) as a notice that does not change the exit code; `--forbid-hand-edits` makes any hand edit fail the build, for teams that want generated code untouchable (Node `meta` only). | no | +| `verify --codegen` | **Codegen drift** — regenerate from metadata into a temp dir and fail if it differs from the committed generated output. Catches "metadata changed but `meta gen` wasn't re-run". **A hand-edited generated file** (Node `meta`): `meta gen` three-way-merges hand edits, so the gate compares the *generated contribution* against the committed `.gen-state/.hashes.json` and reports each edited file separately, by path, with how to see the edit and how to drop it. What that report does to the exit code is the `verify.handEdits` key in `.metaobjects/config.json`: `"fail"` exits 1, `"notice"` (and a project without the key, as before 1.3) prints it and exits 0. `meta init` writes `"fail"` for a new project. `--allow-hand-edits` relaxes `fail` to a notice for one run; `--forbid-hand-edits` makes a hand edit fail for one run whatever the key says. The other ports compare the generated files byte for byte, so any edit is drift and fails: `dotnet meta verify --codegen`, `mvn metaobjects:verify` (Java and Kotlin) and `metaobjects verify --codegen` each exit non-zero and name the file. | no | | `verify --templates` | **Template/prompt drift** — `Renderer.verify` checks each `template.*` node's `{{field}}` references against its payload VO (FR-004). A Jinja body (Python and TypeScript, ADR-0061) is checked the same way. | no | | `verify --docs` | **Removed in 1.1.0** (ADR-0034 Amendment 9). It prints where the check went and exits 2: documentation is written by entries in `generators`, so `verify --codegen` checks it with the rest of the declared output. | — | | `verify --deps` | **Dependency drift** (FR-023) — re-resolves each declared dependency (its `path`, its installed `npm` / `python` package, or its `maven` / `nuget` package in the local cache) and compares its installed artifact's hash against `.metaobjects/deps.lock.json`, same comparison as `meta deps check`. Needs the publisher reachable, so it is never part of the bare-`verify` default. A `drifted` or `unresolved` dependency fails with `ERR_DEPENDENCY_UPSTREAM_DRIFT`. **Node `meta`; Python (`metaobjects verify --deps`, plus `metaobjects deps check`), C# (`dotnet meta verify --deps`), and Java and Kotlin (`-Dmeta.verify.mode=deps`) have the `path`-only counterpart.** See [`metadata-dependencies.md`](metadata-dependencies.md). | no | diff --git a/docs/features/cube-export.md b/docs/features/cube-export.md index be9542dcb..1ed2c54e9 100644 --- a/docs/features/cube-export.md +++ b/docs/features/cube-export.md @@ -582,8 +582,9 @@ there. **`meta verify --codegen`** regenerates the configured output into a temporary directory and compares it with the committed files. The Cube project is a target, so its files are checked like any other generated output: a changed model that was not regenerated, a missing cube file -and a stale one are drift. A hand edit to a generated file is not drift, but the gate lists it, and -`meta verify --codegen --forbid-hand-edits` makes it fail. +and a stale one are drift. A hand edit to a generated file is reported by path, and fails the gate +when `.metaobjects/config.json` sets `"verify": { "handEdits": "fail" }` (what `meta init` writes) +or the run passes `--forbid-hand-edits`. **The mapping corpus** is [`fixtures/cube-model/`](../../fixtures/cube-model/): 46 cases, each the smallest model for one rule, 36 with the exact tree the generator writes and 10 with the diff --git a/docs/features/language-differences.md b/docs/features/language-differences.md index 27f74d8a5..a0391d5ef 100644 --- a/docs/features/language-differences.md +++ b/docs/features/language-differences.md @@ -46,6 +46,8 @@ project. | **Field names on the wire** | The model's names, in every port | | | | | | **Decimals and null members on the wire** | Each port writes its serializer's default; a `wire` setting makes them identical. The per-port table is under [The wire setting](api-contract.md#the-wire-setting) | | | | | | **Hand edits to a generated file** | Merged (three-way) on the machine that generated it; refused by name elsewhere | Refused by name, or overwritten with `WithHandEdits(HandEdits.Overwrite)` | Refused by name, or overwritten with `overwrite` | Same as Java | Refused by name, or overwritten with `handEdits: overwrite` | +| **`verify --codegen` on a hand-edited generated file** | Set by `verify.handEdits`: `"fail"` (what `meta init` writes) exits 1, `"notice"` or no key reports the file and exits 0; `--allow-hand-edits` / `--forbid-hand-edits` change it for one run | Byte comparison: any edit is drift and fails | Same as C# | Same as C# | Same as C# | +| **Foreign-key indexes (`migrate.foreignKeyIndexes`)** | `meta migrate` creates them; the Drizzle and Prisma mappings declare them | The `AppDbContext` declares them | No index mapping to declare | The Exposed tables declare them | No index mapping to declare | | **Own a generator** | `npx meta eject` | `dotnet meta eject` | `mvn metaobjects:eject` | `mvn metaobjects:eject` | `metaobjects eject` | | **Requirement authoring lint** | Yes (advisory) | No | No | No | No | diff --git a/docs/features/metadata-sources.md b/docs/features/metadata-sources.md index 690ce0155..1155ce259 100644 --- a/docs/features/metadata-sources.md +++ b/docs/features/metadata-sources.md @@ -355,8 +355,9 @@ out-of-scope table that already exists would become a proposed `DROP TABLE`. ### `migrate.conventions` — names for objects no code addresses Beside `scope`, the `migrate` block takes `conventions`: the name of a schema-conventions preset -that decides primary-key and foreign-key constraint names, whether a foreign key gets an index, -and whether a unique key is an index or a constraint. `default` (absent) is what migrate has +that decides primary-key and foreign-key constraint names, whether a foreign key gets an index +(without a preset, [`migrate.foreignKeyIndexes`](migrations-and-drift.md#foreign-key-indexes-migrateforeignkeyindexes) +indexes foreign keys), and whether a unique key is an index or a constraint. `default` (absent) is what migrate has always emitted; `efcore` is a reference preset that reproduces EF Core's naming. A project can supply its own conventions from `metaobjects.config.ts` instead. Table, view and column names are not conventions: they come from `naming`. See diff --git a/docs/features/migrations-and-drift.md b/docs/features/migrations-and-drift.md index 552010f8c..85ab3b758 100644 --- a/docs/features/migrations-and-drift.md +++ b/docs/features/migrations-and-drift.md @@ -15,7 +15,7 @@ There are **7 drift sources**, and the toolchain has a guard for each. | **Code-vs-API-doc** | Cross-port codegen from the same metadata. | Build time | | **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, the deparsed body on Postgres's scratch-database path, a fingerprint mismatch otherwise). A view `meta migrate` drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 ] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR | | **Migration-vs-metadata** | The Node `meta migrate` emits migrations FROM metadata diffs — they cannot drift from metadata by construction. Schema migrations for **every** port are owned by this Node toolchain (`@metaobjectsdev/cli migrate`, ADR-0015); the C# and Python migrate surfaces were removed. | Build time | -| **Generated-edited** | `@generated` headers in emitted code + a committed hash manifest that refuses an edited file by name; in TypeScript only, a three-way merge also preserves hand-edits on the machine that generated the file. | Code review | +| **Generated-edited** | `@generated` headers in emitted code + a committed hash manifest that refuses an edited file by name; in TypeScript only, a three-way merge also preserves hand-edits on the machine that generated the file. `verify --codegen` names each edited file in every port. In TypeScript it fails on one when `.metaobjects/config.json` sets `"verify": { "handEdits": "fail" }` (what `meta init` writes) and reports it as a notice otherwise; the C#, Java, Kotlin and Python gates compare bytes, so an edit always fails there ([cli.md](cli.md)). | Build time (CI) | | **Prompt-vs-payload** | FR-004 `Renderer.verify` parses `{{...}}` references in templates and checks each one exists on the payload VO. | Build time + runtime | | **Generated-vs-runtime** | Kotlin / Java `MetadataStartupValidator` (from Spring `ApplicationReadyEvent`) re-loads metadata at startup and asserts the generated table objects match. | App startup | @@ -496,7 +496,7 @@ come from the project's **schema conventions**, selected in `.metaobjects/config |---|---|---| | Primary-key constraint name (Postgres; SQLite and D1 emit an unnamed primary key) | `
_pkey` | `PK_
` | | Foreign-key constraint name | `
__fk` | `FK_
__` | -| Index on a foreign key | none | `IX_
_`, unless the primary key or another index (a partial one included) already starts with the FK's columns; an FK index another FK index covers is dropped, so FK order does not matter | +| Index on a foreign key | none (unless [`migrate.foreignKeyIndexes`](#foreign-key-indexes-migrateforeignkeyindexes) is on, which gives each uncovered one `
__idx`) | `IX_
_`, unless the primary key or another index (a partial one included) already starts with the FK's columns; an FK index another FK index covers is dropped, so FK order does not matter | | Unique key (`identity.secondary`, a `@unique` field) | a unique index; a live UNIQUE constraint is also accepted | a UNIQUE constraint when a foreign key references it, else a unique index | `efcore` is a reference preset. It reproduces one ORM's documented naming to show what the @@ -535,6 +535,11 @@ export default defineConfig({ }); ``` +Such an object reaches only the TypeScript toolchain: `meta migrate`, `meta verify --db`, and +the TypeScript generators resolve it. The C# and Kotlin generators read only the +`migrate.conventions` preset name from `.metaobjects/config.json` and apply the default rule for +every derived name, foreign-key indexes included. + **Changing the conventions of an existing database.** On Postgres, a primary key, FK, CHECK or index whose definition is unchanged and whose name differs is renamed in place (`RENAME CONSTRAINT` / `ALTER INDEX … RENAME`), never dropped and re-created; new FK indexes are created. @@ -546,6 +551,55 @@ onto that key is dropped first and added back after, so the plan is gated behind drop-index` and `--allow drop-fk`. `meta verify --db` reports each of these as drift. SQLite cannot rename a constraint and rebuilds the table instead. +## Foreign-key indexes: `migrate.foreignKeyIndexes` + +Postgres and SQLite index the referenced side of a foreign key and not the referencing side, so +without an index every join from the parent, and every `ON DELETE` check it runs, scans the child +table. `migrate.foreignKeyIndexes` asks `meta migrate` to index each foreign key, with no other +change to the schema's names: + +```json +{ "schema_version": 1, "migrate": { "foreignKeyIndexes": true } } +``` + +With it on, each enforced foreign key that nothing already leads with gets a plain index named +`
_[_…]_idx`. The primary key, a `@unique` field, an `identity.secondary` +and an `index.lookup` without `@expr` (a partial one included) cover a foreign key whose columns +they start with, in any order; a junction's first foreign key is usually covered by its composite +primary key. Two foreign keys over the same columns get one index, and a foreign key that leads a +longer one shares the longer one's index. A name that collides with another index is refused with +`ERR_DUPLICATE_SQL_NAME`. + +It is separate from the `efcore` conventions, which index every foreign key too but also rename +the primary-key and foreign-key constraints: a team can have the indexes without the renames. +Under `efcore` the option adds nothing, since the preset's `IX_` indexes already cover every +foreign key. On MySQL it adds nothing either: MetaObjects emits no DDL there, and InnoDB indexes +each foreign key itself. + +`meta init` writes it on for a new project. A project without the key is unchanged: migrate adds +no foreign-key index, and `meta verify` keeps its advisory naming each unindexed foreign key +(which now also names the option). Turning it on for an existing database is an additive +migration (`CREATE INDEX` per key), with nothing blocked. + +The generators that map indexes declare the same indexes under the same names, from the same +setting, so the mapping and the database agree: + +| Generator | With the option on | +|---|---| +| Drizzle (`entityFile()`, TypeScript) | `index("").on(table.)` in the table's index callback | +| Prisma (`prismaSchemaFile()`, TypeScript) | `@@index([...], map: "")`, read from the same expected schema | +| EF Core `AppDbContext` (C#) | `HasIndex(x => x., "").HasDatabaseName("")` | +| Exposed table (Kotlin) | `index("", false, )` in the table's `init {}` | + +Java (OMDB is data access only and maps no indexes) and Python (Pydantic models and routes, no +ORM mapping) generate nothing that declares an index, so the option changes no output there. + +A project that pairs its own conventions object with `migrate.foreignKeyIndexes` and a C# or +Kotlin mapping should make the object's `foreignKeyIndexes` match the default rule (index each +foreign key nothing leads with, named `
__idx`) or leave the option off; +otherwise the C# DbContext or Kotlin Exposed tables may declare an index `meta migrate` did not +create. + ## How CHECK constraints and expression indexes are compared Postgres rewrites an expression before it stores it (`"k" IN ('x')` becomes `k = 'x'::text`, @@ -845,6 +899,8 @@ The following conformance fixtures gate this feature's behavior across ports: - [`bootstrap-canonical-from-empty.yaml`](../../fixtures/persistence-conformance/migrations/bootstrap-canonical-from-empty.yaml) — full-CREATE bootstrap from an empty database - [`add-nullable-column.yaml`](../../fixtures/persistence-conformance/migrations/add-nullable-column.yaml) — incremental `ALTER TABLE … ADD COLUMN` for a new nullable field - [`drop-table-blocked-without-allow.yaml`](../../fixtures/persistence-conformance/migrations/drop-table-blocked-without-allow.yaml) — destructive operations require an explicit allow-flag +- [`foreign-key-indexes-added.yaml`](../../fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml) — `migrate.foreignKeyIndexes` turned on for an existing schema adds one `
__idx` per uncovered foreign key and nothing else +- [`noop-converged-foreign-key-indexes.yaml`](../../fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml) — with the option on, a converged schema emits zero SQL (the indexes introspect back exactly as emitted) - [`noop-converged-canonical.yaml`](../../fixtures/persistence-conformance/migrations/noop-converged-canonical.yaml) — **the idempotence gate**: apply the whole canonical schema, then diff the same metadata against the database it just produced. A converged schema must emit **zero** SQL. This is what catches any asymmetry between what `emit` writes, what `introspect` reads back, and what the expected schema models — the class of bug that makes `meta migrate` re-propose (and on SQLite, destructively rebuild) unchanged tables forever. **Template drift (`fixtures/verify-conformance/`)** — the `Renderer.verify` engine diff --git a/docs/recipes/prove-your-gate-fails.md b/docs/recipes/prove-your-gate-fails.md index 68b1706bf..ffbf68a4b 100644 --- a/docs/recipes/prove-your-gate-fails.md +++ b/docs/recipes/prove-your-gate-fails.md @@ -15,14 +15,17 @@ It works in a throwaway project and asserts, in this order: | 1 | Clean tree | **0** | | 2 | A field added to the metadata, `meta gen` not re-run | **1** (codegen gate) | | 3 | Metadata restored | **0** (the failure was the break, not the harness) | -| 4 | A hand edit inside a generated file | **0** by default, **1** with `meta verify --codegen --forbid-hand-edits` | +| 4 | A hand edit inside a generated file | **1** (the project `meta init` wrote sets `verify.handEdits: "fail"`); **0** with `meta verify --codegen --allow-hand-edits` | | 5 | A generated file deleted | **1** | | 6 | `meta gen` re-run | **0** | -Step 4 is deliberate: `meta gen` three-way-merges hand edits, so a hand-edited generated file -is a notice, not drift. If your policy is "generated code is never touched", gate on -`--forbid-hand-edits`. A second test in the same file pins why step 0 exists: a project with no -generators passes `meta verify` with exit 0 having checked nothing. +Step 4 follows the project's `verify.handEdits` setting. `meta gen` three-way-merges hand edits, so +a hand-edited generated file is reported apart from drift, by path; a project with +`"verify": { "handEdits": "fail" }` in `.metaobjects/config.json` (what `meta init` writes) fails +on it, and one without the key (a project from before 1.3) gets the report as a notice and exit 0. +On such a project, set the key, or gate on `meta verify --codegen --forbid-hand-edits`. A second +test in the same file pins why step 0 exists: a project with no generators passes `meta verify` +with exit 0 having checked nothing. ## The same check in a shell, for your own CI diff --git a/examples/advanced-modeling/codegen/templates/drizzle-schema.ts b/examples/advanced-modeling/codegen/templates/drizzle-schema.ts index f943b82ef..e35ed5146 100644 --- a/examples/advanced-modeling/codegen/templates/drizzle-schema.ts +++ b/examples/advanced-modeling/codegen/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "@metaobjectsdev/codegen-ts/engine"; import { flattenedMembersOf } from "@metaobjectsdev/codegen-ts/engine"; import { effectivePackage } from "@metaobjectsdev/codegen-ts/engine"; +import { foreignKeyIndexesOf } from "@metaobjectsdev/codegen-ts/engine"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) { diff --git a/examples/showcase/codegen/templates/drizzle-schema.ts b/examples/showcase/codegen/templates/drizzle-schema.ts index f943b82ef..e35ed5146 100644 --- a/examples/showcase/codegen/templates/drizzle-schema.ts +++ b/examples/showcase/codegen/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "@metaobjectsdev/codegen-ts/engine"; import { flattenedMembersOf } from "@metaobjectsdev/codegen-ts/engine"; import { effectivePackage } from "@metaobjectsdev/codegen-ts/engine"; +import { foreignKeyIndexesOf } from "@metaobjectsdev/codegen-ts/engine"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) { diff --git a/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb55024..14b1f94cc 100644 --- a/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb55024..14b1f94cc 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb55024..14b1f94cc 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb55024..14b1f94cc 100644 --- a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb55024..14b1f94cc 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/generator-registry-conformance/registry.json b/fixtures/generator-registry-conformance/registry.json index f01caa68e..2f1d47b07 100644 --- a/fixtures/generator-registry-conformance/registry.json +++ b/fixtures/generator-registry-conformance/registry.json @@ -364,6 +364,7 @@ "codegen-ts/src/enum-meta.ts", "codegen-ts/src/enum-shared.ts", "codegen-ts/src/flattened.ts", + "codegen-ts/src/foreign-key-indexes.ts", "codegen-ts/src/format.ts", "codegen-ts/src/generator.ts", "codegen-ts/src/generators/agent-schema-input.ts", diff --git a/fixtures/persistence-conformance/README.md b/fixtures/persistence-conformance/README.md index 978409826..724734bac 100644 --- a/fixtures/persistence-conformance/README.md +++ b/fixtures/persistence-conformance/README.md @@ -118,6 +118,14 @@ seed-data: | # Target schema — what we want the DB to look like. target-metadata: ./states/program-v2/ +# Optional: the project schema settings (`migrate.*` in .metaobjects/config.json) the target, +# and separately the seed, are built with. Only `foreignKeyIndexes` is applied; any other key +# is refused, so a typo cannot silently run the default. +# schema-settings: +# foreignKeyIndexes: true +# seed-schema-settings: +# foreignKeyIndexes: true + # Assertions on the generated up migration (any of these may be omitted). expect: blocked: [] # no blocked changes diff --git a/fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml b/fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml new file mode 100644 index 000000000..5c49408cd --- /dev/null +++ b/fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml @@ -0,0 +1,36 @@ +name: foreign-key-indexes-added +description: | + `migrate.foreignKeyIndexes` turned on for a project whose schema has none. The canonical + schema is applied as it always is (no foreign key has an index), then migrated to the SAME + metadata with the option on: the up migration only adds a `
__idx` index for + each foreign key nothing already leads with. A junction's foreign key that leads its composite + primary key (post_tags.postId, follows.followerId, ...) is covered by the key and gets none. + No table, column or constraint changes, and nothing is blocked. + +seed-metadata: ../canonical/ +target-metadata: ../canonical/ +schema-settings: + foreignKeyIndexes: true + +expect: + blocked: [] + up-contains: + - 'CREATE INDEX "weeks_programId_idx" ON "weeks" ("programId");' + - 'CREATE INDEX "nodes_parentId_idx" ON "nodes" ("parentId");' + - 'CREATE INDEX "sessions_weekId_idx" ON "sessions" ("weekId");' + - 'CREATE INDEX "post_tags_tagId_idx" ON "post_tags" ("tagId");' + - 'CREATE INDEX "post_referrals_referralId_idx" ON "post_referrals" ("referralId");' + apply-up-then-query: + sql: | + SELECT tablename, indexname FROM pg_indexes + WHERE schemaname = 'public' AND indexname LIKE '%\_idx' + ORDER BY indexname + rows: + - { tablename: "follows", indexname: "follows_followeeId_idx" } + - { tablename: "friendships", indexname: "friendships_personBId_idx" } + - { tablename: "gadget_events", indexname: "gadget_events_gadgetId_idx" } + - { tablename: "nodes", indexname: "nodes_parentId_idx" } + - { tablename: "post_referrals", indexname: "post_referrals_referralId_idx" } + - { tablename: "post_tags", indexname: "post_tags_tagId_idx" } + - { tablename: "sessions", indexname: "sessions_weekId_idx" } + - { tablename: "weeks", indexname: "weeks_programId_idx" } diff --git a/fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml b/fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml new file mode 100644 index 000000000..6ad2a2c0b --- /dev/null +++ b/fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml @@ -0,0 +1,17 @@ +name: noop-converged-foreign-key-indexes +description: | + IDEMPOTENCE GATE for `migrate.foreignKeyIndexes`: the canonical schema built WITH the option, + then diffed against the same metadata with the option still on, must produce zero SQL. Catches + a foreign-key index whose emitted form introspection reads back differently (a name, a column + list), which would otherwise re-propose the index on every migrate. + +seed-metadata: ../canonical/ +seed-schema-settings: + foreignKeyIndexes: true +target-metadata: ../canonical/ +schema-settings: + foreignKeyIndexes: true + +expect: + blocked: [] + up-empty: true diff --git a/server/csharp/MetaObjects.Cli/VerifyCommand.cs b/server/csharp/MetaObjects.Cli/VerifyCommand.cs index a64702984..a12afa7af 100644 --- a/server/csharp/MetaObjects.Cli/VerifyCommand.cs +++ b/server/csharp/MetaObjects.Cli/VerifyCommand.cs @@ -457,6 +457,9 @@ private static Codegen.CodegenDrift.Result RunCodegenDrift(Options opts, out boo // services read as drift. Dialect = neutralConfig?.MigrateDialect, SkipRedundantChecks = neutralConfig?.MigrateSkipRedundantChecks ?? false, + // The foreign-key indexes gen declared, so the regen indexes what gen indexed. + ForeignKeyIndexes = neutralConfig?.MigrateForeignKeyIndexes ?? false, + SchemaConventions = neutralConfig?.MigrateConventions, }.WithCSharpLayout(neutralConfig); return Codegen.CodegenDrift.Compute(config, load.Root, generators); } diff --git a/server/csharp/MetaObjects.Codegen.Tests/ForeignKeyIndexTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ForeignKeyIndexTests.cs new file mode 100644 index 000000000..fbce6aac9 --- /dev/null +++ b/server/csharp/MetaObjects.Codegen.Tests/ForeignKeyIndexTests.cs @@ -0,0 +1,115 @@ +// `migrate.foreignKeyIndexes` (.metaobjects/config.json; GenConfig.ForeignKeyIndexes): `meta migrate` +// gives every enforced foreign key that nothing already leads with a `
__idx` +// index, so the EF model declares the same index under the same name. The model is the one +// migrate-ts's expected-schema-foreign-key-indexes test builds, so the two sides are held to one +// answer. + +using MetaObjects.Codegen.Generators; +using MetaObjects.Config; +using MetaObjects.Loader; +using Xunit; + +namespace MetaObjects.Codegen.Tests; + +public sealed class ForeignKeyIndexTests +{ + private const string Model = """ + { "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Customer", "children": [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.secondary": { "name": "customers_code_key", "@fields": ["code"] } } + ] } }, + { "object.entity": { "name": "Order", "children": [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { "name": "id" } }, + { "field.long": { "name": "customerId", "@column": "customer_id" } }, + { "field.long": { "name": "billToId", "@column": "bill_to_id" } }, + { "field.string": { "name": "customerCode", "@column": "customer_code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.reference": { "name": "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { "name": "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "identity.reference": { "name": "byCode", "@fields": ["customerCode"], "@references": "Customer.code" } }, + { "index.lookup": { "name": "orders_bill_to", "@fields": ["billToId", "id"] } } + ] } }, + { "object.entity": { "name": "OrderTag", "children": [ + { "source.rdb": { "@table": "order_tags" } }, + { "field.long": { "name": "orderId", "@column": "order_id" } }, + { "field.long": { "name": "tagId", "@column": "tag_id" } }, + { "identity.primary": { "name": "pk", "@fields": ["orderId", "tagId"] } }, + { "identity.reference": { "name": "order", "@fields": ["orderId"], "@references": "Order" } }, + { "identity.reference": { "name": "tag", "@fields": ["tagId"], "@references": "Tag" } } + ] } }, + { "object.entity": { "name": "Tag", "children": [ + { "source.rdb": { "@table": "tags" } }, + { "field.long": { "name": "id" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } } + ] } } + ] } } + """; + + private static string Context(bool on, string? conventions = null, string? dialect = null) + { + var load = new MetaDataLoader().Load([new InMemoryStringSource(Model, id: "m.json")]); + Assert.Empty(load.Errors); + var ctx = new GenContext + { + Entities = load.Root.Objects(), Root = load.Root, Warn = _ => { }, + Config = new GenConfig + { + OutDir = "/unused", Namespace = "Acme", + ForeignKeyIndexes = on, SchemaConventions = conventions, Dialect = dialect, + }, + }; + return string.Join("\n", new DbContextGenerator().Generate(ctx).Select(f => f.Content)); + } + + private static string Index(string cls, string prop, string name) => + $"modelBuilder.Entity<{cls}>().HasIndex(x => x.{prop}, \"{name}\").HasDatabaseName(\"{name}\");"; + + [Fact] + public void Off_the_model_declares_no_foreign_key_index() + { + Assert.DoesNotContain("_idx\"", Context(on: false)); + } + + [Fact] + public void On_each_foreign_key_nothing_leads_with_gets_migrates_index() + { + var src = Context(on: true); + Assert.Contains(Index("Order", "CustomerId", "orders_customer_id_idx"), src); + Assert.Contains(Index("Order", "CustomerCode", "orders_customer_code_idx"), src); + Assert.Contains(Index("OrderTag", "TagId", "order_tags_tag_id_idx"), src); + // Led by a declared lookup index, and by the junction's primary key. + Assert.DoesNotContain("orders_bill_to_id_idx", src); + Assert.DoesNotContain("order_tags_order_id_idx", src); + } + + [Fact] + public void The_efcore_conventions_and_mysql_add_nothing() + { + Assert.DoesNotContain("_idx\"", Context(on: true, conventions: "efcore")); + Assert.DoesNotContain("_idx\"", Context(on: true, dialect: "mysql")); + } + + [Fact] + public void The_config_reader_takes_the_key_from_the_migrate_block() + { + var dir = Directory.CreateTempSubdirectory("mo-fk-idx-"); + try + { + Directory.CreateDirectory(Path.Combine(dir.FullName, ".metaobjects")); + File.WriteAllText(Path.Combine(dir.FullName, ".metaobjects", "config.json"), + """{ "schema_version": 1, "migrate": { "foreignKeyIndexes": true, "conventions": "efcore" } }"""); + var cfg = NeutralConfig.Read(dir.FullName, validateNaming: false); + Assert.NotNull(cfg); + Assert.True(cfg!.MigrateForeignKeyIndexes); + Assert.Equal("efcore", cfg.MigrateConventions); + File.WriteAllText(Path.Combine(dir.FullName, ".metaobjects", "config.json"), """{ "schema_version": 1 }"""); + Assert.False(NeutralConfig.Read(dir.FullName, validateNaming: false)!.MigrateForeignKeyIndexes); + } + finally { dir.Delete(recursive: true); } + } +} diff --git a/server/csharp/MetaObjects.Codegen/CodegenCli.cs b/server/csharp/MetaObjects.Codegen/CodegenCli.cs index 5bab41876..0196264aa 100644 --- a/server/csharp/MetaObjects.Codegen/CodegenCli.cs +++ b/server/csharp/MetaObjects.Codegen/CodegenCli.cs @@ -260,6 +260,8 @@ public static GenOutcome RunGen( SweepOrphans = sweepOrphans, Dialect = neutralConfig?.MigrateDialect, SkipRedundantChecks = neutralConfig?.MigrateSkipRedundantChecks ?? false, + ForeignKeyIndexes = neutralConfig?.MigrateForeignKeyIndexes ?? false, + SchemaConventions = neutralConfig?.MigrateConventions, }.WithCSharpLayout(neutralConfig); CodegenRunner.RunResult result; @@ -963,6 +965,8 @@ private static int RunVerifyArgs(string[] rest, IReadOnlyList declar // services read as drift. Dialect = neutralConfig?.MigrateDialect, SkipRedundantChecks = neutralConfig?.MigrateSkipRedundantChecks ?? false, + ForeignKeyIndexes = neutralConfig?.MigrateForeignKeyIndexes ?? false, + SchemaConventions = neutralConfig?.MigrateConventions, }.WithCSharpLayout(neutralConfig); var result = CodegenDrift.Compute(config, load.Root, suite, narrowed: a.Selection is { Count: > 0 }); if (result.Error is not null) diff --git a/server/csharp/MetaObjects.Codegen/Generator.cs b/server/csharp/MetaObjects.Codegen/Generator.cs index a8e6a6aff..43124727b 100644 --- a/server/csharp/MetaObjects.Codegen/Generator.cs +++ b/server/csharp/MetaObjects.Codegen/Generator.cs @@ -232,6 +232,22 @@ public string? ContextAssemblyName } private readonly string? _contextAssemblyName; + /// + /// migrate.foreignKeyIndexes in .metaobjects/config.json: the model declares the + /// index meta migrate gives each foreign key nothing else indexes + /// (<table>_<column>[_<column>…]_idx). False, the default, leaves the + /// model as it was. + /// + public bool ForeignKeyIndexes { get; init; } + + /// + /// migrate.conventions: the schema-conventions preset meta migrate builds with, or + /// null for its default. Under efcore migrate already indexes every foreign key + /// with EF Core's own names, which EF's convention reproduces, so + /// adds nothing. + /// + public string? SchemaConventions { get; init; } + /// /// The C# property name of a field, for every generator that names one: the entity and /// value-object classes, the composite [PrimaryKey], the DbContext's model diff --git a/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs b/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs index aa08e1693..6f0556357 100644 --- a/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs +++ b/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs @@ -270,6 +270,7 @@ public virtual IEnumerable Generate(GenContext ctx) junctionOwnedFks.TryGetValue(e, out var ownedByUsingEntity); if (EmitsReferenceForeignKeys) EmitReferenceConfig(e, tph, ctx, modelLines, ownedByUsingEntity); + EmitForeignKeyIndexConfig(e, tph, ctx, modelLines); // The adopter's own per-entity lines (indexes, CHECKs, query filters, computed // columns), last so they see everything the entity was configured with. @@ -1368,6 +1369,97 @@ private void EmitIndexConfig(string owner, MetaObject entity, GenContext ctx, Li } } + /// + /// migrate.foreignKeyIndexes — the index meta migrate gives each enforced foreign + /// key that nothing already leads with, so the model declares the index the database has, + /// under migrate's name (<table>_<column>[_<column>…]_idx). The + /// rule is migrate's own (uncoveredForeignKeyColumns in migrate-ts): the primary key, + /// a field's @unique, an identity.secondary and an index.lookup without + /// @expr cover a foreign key whose columns they lead with, in any order; two foreign + /// keys over one column set get one index; and a foreign key that leads a longer one shares + /// the longer one's index. Nothing under the efcore conventions, which already index + /// every foreign key with EF Core's own names, and nothing on MySQL, where InnoDB indexes each + /// foreign key itself. A TPH table is read whole: the base and every concrete subtype. + /// + private void EmitForeignKeyIndexConfig(MetaObject entity, TphPlan? tph, GenContext ctx, List modelLines) + { + var config = ctx.Config; + if (!config.ForeignKeyIndexes + || string.Equals(config.SchemaConventions, "efcore", StringComparison.Ordinal) + || string.Equals(config.Dialect, "mysql", StringComparison.OrdinalIgnoreCase)) + return; + var strategy = config.ColumnNamingStrategy; + var members = new List { entity }; + if (tph is not null) members.AddRange(tph.Subtypes.Select(st => st.Entity)); + + List? Columns(MetaObject owner, IEnumerable fieldNames) + { + var cols = new List(); + foreach (var name in fieldNames) + { + if (owner.FindField(name) is not { } f) return null; + cols.Add(CSharpNaming.Column(f, strategy)); + } + return cols; + } + + var existing = new List>(); + if (entity.PrimaryIdentity()?.Fields is { Count: > 0 } pk && Columns(entity, pk) is { } pkCols) existing.Add(pkCols); + foreach (var m in members) + { + // ADR-0039: resolving — @unique and index nodes may be inherited via extends. + foreach (var f in m.Fields().Where(f => f.Attr(FIELD_ATTR_UNIQUE) is true)) + existing.Add([CSharpNaming.Column(f, strategy)]); + foreach (var (node, fields) in m.SecondaryIdentities().Select(i => ((MetaData)i, i.Fields)) + .Concat(m.LookupIndexes().Select(i => ((MetaData)i, i.Fields)))) + { + if (node.Attr(DbConstants.IDENTITY_ATTR_EXPR) is string { Length: > 0 } || fields.Count == 0) continue; + if (Columns(m, fields) is { } cols) existing.Add(cols); + } + } + + // Each physical foreign key once, on the first class of the table that declares it. + var seen = new HashSet(StringComparer.Ordinal); + var candidates = new List<(MetaObject Owner, List Fields, List Columns)>(); + foreach (var m in members) + { + foreach (var reference in m.ReferenceIdentities().Where(r => r.Enforce)) + { + if (!seen.Add(FkKey(reference))) continue; + var fields = new List(); + foreach (var name in ReferentialActions.ReadIdentityFields(reference)) + { + if (m.FindField(name) is not { } f) { fields.Clear(); break; } + fields.Add(f); + } + if (fields.Count == 0) continue; + var cols = fields.Select(f => CSharpNaming.Column(f, strategy)).ToList(); + if (existing.Any(e => LeadsWith(e, cols))) continue; + if (candidates.Any(c => c.Columns.Count == cols.Count && LeadsWith(c.Columns, cols))) continue; + candidates.Add((m, fields, cols)); + } + } + + var table = CSharpNaming.Table(entity); + foreach (var (owner, fields, cols) in candidates) + { + if (candidates.Any(o => o.Columns.Count > cols.Count && LeadsWith(o.Columns, cols))) continue; + var name = $"\"{CSharpNaming.CSharpStringLiteral($"{table}_{string.Join("_", cols)}_idx")}\""; + var props = fields.Select(PropertyName).ToList(); + var selector = props.Count == 1 ? $"x => x.{props[0]}" : $"x => new {{ {string.Join(", ", props.Select(p => "x." + p))} }}"; + modelLines.Add($" modelBuilder.Entity<{CSharpNaming.Pascal(owner.Name)}>().HasIndex({selector}, {name}).HasDatabaseName({name});"); + } + } + + /// True when leads with exactly , + /// in any order (migrate's leadsWith). + private static bool LeadsWith(IReadOnlyList prefix, IReadOnlyList columns) + { + if (columns.Count == 0 || prefix.Count < columns.Count) return false; + var want = new HashSet(columns, StringComparer.Ordinal); + return prefix.Take(columns.Count).All(want.Contains); + } + // The SQL expressions migrate treats as an expression default (expected-schema.ts // EXPR_DEFAULT_PATTERNS): the rest of a string default is a literal. private static readonly System.Text.RegularExpressions.Regex ExprDefault = new( diff --git a/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs b/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs index d50a4a599..eb882e272 100644 --- a/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs +++ b/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs @@ -1400,4 +1400,46 @@ public async Task The_checks_the_model_declares_are_the_ones_meta_migrate_create "INSERT INTO gadgets (kind, tags, email, score) VALUES ('plain', '{}', 'a@b.c', 9)")); Assert.True(ex.ConstraintName == "gadgets_score_numeric_chk", ex.MessageText); } + + // migrate.foreignKeyIndexes: the model declares the
__idx index `meta migrate` + // gives each foreign key nothing leads with, on the base and on a TPH subtype's own reference, + // and EF's convention IX_ index on that key gives way to it. + private const string ForeignKeyIndexModel = """ + { "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Owner", "children": [ + { "source.rdb": { "@table": "owners" } }, + { "field.long": { "name": "id" } }, + { "identity.primary": { "@fields": "id", "@generation": "increment" } } + ]}}, + { "object.entity": { "name": "Vehicle", "@discriminator": "kind", "children": [ + { "source.rdb": { "@table": "vehicles" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "kind", "@maxLength": 10 } }, + { "field.long": { "name": "ownerId" } }, + { "field.long": { "name": "keeperId" } }, + { "identity.primary": { "@fields": "id", "@generation": "increment" } }, + { "identity.reference": { "name": "owner", "@fields": ["ownerId"], "@references": "Owner" } }, + { "identity.reference": { "name": "keeper", "@fields": ["keeperId"], "@references": "Owner" } }, + { "index.lookup": { "name": "ix_vehicles_keeper", "@fields": ["keeperId", "kind"] } } + ]}}, + { "object.entity": { "name": "Car", "extends": "Vehicle", "@discriminatorValue": "Car", "children": [ + { "field.long": { "name": "dealerId" } }, + { "identity.reference": { "name": "dealer", "@fields": ["dealerId"], "@references": "Owner" } } + ]}} + ]}} + """; + + [Fact] + public void Foreign_key_indexes_reach_the_model_under_migrates_names() + { + var gen = GeneratedEfModel.Compile(ForeignKeyIndexModel, "Acme.FkIndex", c => c with { ForeignKeyIndexes = true }); + using var db = gen.NewContext(o => o.UseNpgsql("Host=unused")); + var vehicle = DesignEntityType(db, "Vehicle").GetDeclaredIndexes().Select(i => i.GetDatabaseName()).ToList(); + var car = DesignEntityType(db, "Car").GetDeclaredIndexes().Select(i => i.GetDatabaseName()).ToList(); + Assert.Contains("vehicles_ownerId_idx", vehicle); + Assert.Contains("vehicles_dealerId_idx", car); + // keeperId leads a declared lookup index, so it gets no second one. + Assert.DoesNotContain("vehicles_keeperId_idx", vehicle); + Assert.DoesNotContain(vehicle.Concat(car), n => n!.StartsWith("IX_", StringComparison.Ordinal)); + } } diff --git a/server/csharp/MetaObjects/Config/NeutralConfig.cs b/server/csharp/MetaObjects/Config/NeutralConfig.cs index eecab3b07..0ed6bff9c 100644 --- a/server/csharp/MetaObjects/Config/NeutralConfig.cs +++ b/server/csharp/MetaObjects/Config/NeutralConfig.cs @@ -88,6 +88,14 @@ public sealed class NeutralConfig /// migrate.checks.skipRedundant: leave out the CHECKs a column's own type already enforces. public bool MigrateSkipRedundantChecks { get; private init; } + /// `migrate.foreignKeyIndexes`: index every foreign key the schema conventions leave + /// without one, as `meta migrate` does. False when absent or not a boolean. + public bool MigrateForeignKeyIndexes { get; private init; } + + /// `migrate.conventions`: the schema-conventions preset `meta migrate` builds with + /// (`default`, `efcore`), or null when absent. + public string? MigrateConventions { get; private init; } + /// /// FR-023 — requirements.inherit: the dependencies whose requirements this project takes on /// as its own ledger (). Empty when absent. @@ -164,12 +172,21 @@ private static bool ReadMigrateSkipRedundant(JsonElement root) => && checks.TryGetProperty("skipRedundant", out var skip) && skip.ValueKind == JsonValueKind.True; /// `migrate.dialect`, as the TypeScript `meta migrate` reads it; null when absent or not a string. - private static string? ReadMigrateDialect(JsonElement root) => + private static string? ReadMigrateDialect(JsonElement root) => ReadMigrateString(root, "dialect"); + + /// One string in the TypeScript-owned `migrate` block; null when absent or not a string. + private static string? ReadMigrateString(JsonElement root, string key) => root.TryGetProperty("migrate", out var migrate) && migrate.ValueKind == JsonValueKind.Object - && migrate.TryGetProperty("dialect", out var d) && d.ValueKind == JsonValueKind.String + && migrate.TryGetProperty(key, out var d) && d.ValueKind == JsonValueKind.String ? d.GetString() : null; + /// `migrate.foreignKeyIndexes`; false when absent or not a boolean (the TypeScript reader + /// refuses a non-boolean, and `meta migrate` is where the file is validated). + private static bool ReadMigrateForeignKeyIndexes(JsonElement root) => + root.TryGetProperty("migrate", out var migrate) && migrate.ValueKind == JsonValueKind.Object + && migrate.TryGetProperty("foreignKeyIndexes", out var v) && v.ValueKind == JsonValueKind.True; + /// /// As ; with false a /// naming value no strategy registered in THIS process answers is kept by name @@ -370,6 +387,8 @@ private static bool ReadMigrateSkipRedundant(JsonElement root) => PackageNamespaces = ReadPackageNamespaces(root, path), MigrateDialect = ReadMigrateDialect(root), MigrateSkipRedundantChecks = ReadMigrateSkipRedundant(root), + MigrateForeignKeyIndexes = ReadMigrateForeignKeyIndexes(root), + MigrateConventions = ReadMigrateString(root, "conventions"), }; } } diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt index ab328f2e5..795e1690b 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt @@ -5,6 +5,7 @@ package com.metaobjects.generator.kotlin // `mvn metaobjects:eject` and its own package is renamed — an explicit import, not // same-package bare-name resolution, is what keeps the ejected copy compiling. import com.metaobjects.identity.PrimaryIdentity +import com.metaobjects.config.MigrateSettings import com.metaobjects.config.WireFormat import com.metaobjects.generator.util.GeneratorUtil import com.metaobjects.generator.kotlin.ExposedApi @@ -1207,7 +1208,8 @@ open class KotlinExposedTableGenerator : MultiFileDirectGeneratorBase_[_…]_idx`). The rule is + * migrate's own (`uncoveredForeignKeyColumns` in migrate-ts): the primary key, a field's + * `@unique`, an `identity.secondary` and an `index.lookup` without `@expr` cover a foreign + * key whose columns they lead with, in any order; two foreign keys over one column set get one + * index; and one that leads a longer one shares the longer one's index. Empty unless the + * project sets the option, and under the `efcore` conventions or on MySQL + * ([MigrateSettings.indexesForeignKeys]). A TPH table is read whole: base and subtypes. + */ + protected fun foreignKeyIndexes(entity: MetaObject, loader: MetaDataLoader, tableName: String): List>> { + if (!MigrateSettings.of(entity).indexesForeignKeys()) return emptyList() + val members = listOf(entity) + KotlinTphPlan.planFor(entity, loader)?.subtypes.orEmpty().map { it.entity } + // ADR-0039: resolving — fields, identities, index nodes and @unique may be inherited. + fun field(owner: MetaObject, name: String): MetaField<*>? = owner.metaFields.firstOrNull { it.name == name } + fun columns(owner: MetaObject, names: List): List? = + names.map { n -> field(owner, n)?.let { KotlinGenUtil.resolveColumnName(it) } ?: return null } + fun attr(node: MetaData, name: String): String? = + (if (node.hasMetaAttr(name)) node.getMetaAttr(name).value?.toString() else null)?.trim()?.takeIf { it.isNotEmpty() } + fun leadsWith(prefix: List, cols: List): Boolean = + cols.isNotEmpty() && prefix.size >= cols.size && prefix.take(cols.size).all { it in cols.toSet() } + + val existing = mutableListOf>() + entity.getIdentities(true).firstOrNull { it.isPrimary } + ?.let { pk -> columns(entity, KotlinGenUtil.keyFields(entity, pk)) } + ?.takeIf { it.isNotEmpty() }?.let(existing::add) + for (m in members) { + for (f in m.metaFields) { + if (attr(f, MetaField.ATTR_UNIQUE)?.toBoolean() == true) existing.add(listOf(KotlinGenUtil.resolveColumnName(f))) + } + val nodes: List>> = + m.getIdentities(true).filter { it.isSecondary }.map { it to it.fields } + + m.getChildren(LookupIndex::class.java, true).map { it to it.fields } + for ((node, names) in nodes) { + if (attr(node, com.metaobjects.identity.SecondaryIdentity.ATTR_EXPR) != null || names.isEmpty()) continue + columns(m, names)?.let(existing::add) + } + } + + val seen = hashSetOf() + val candidates = mutableListOf, List>>() // (columns, field names) + for (m in members) { + for (ref in m.getIdentities(true).filterIsInstance().filter { it.isEnforced }) { + if (!seen.add(ref.fields.joinToString(","))) continue + val cols = columns(m, ref.fields)?.takeIf { it.isNotEmpty() } ?: continue + if (existing.any { leadsWith(it, cols) }) continue + if (candidates.any { it.first.size == cols.size && leadsWith(it.first, cols) }) continue + candidates.add(cols to ref.fields) + } + } + return candidates + .filter { (cols, _) -> candidates.none { (o, _) -> o.size > cols.size && leadsWith(o, cols) } } + .map { (cols, fields) -> "${tableName}_${cols.joinToString("_")}_idx" to fields } + } + protected fun indexNameExpr(node: MetaData, shortName: String): String { val name = IndexNaming.resolve(node) if (!useNames()) return "\"$name\"" diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedForeignKeyIndexTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedForeignKeyIndexTest.kt new file mode 100644 index 000000000..d89288baa --- /dev/null +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedForeignKeyIndexTest.kt @@ -0,0 +1,88 @@ +package com.metaobjects.generator.kotlin + +import com.metaobjects.config.MigrateSettings +import com.metaobjects.metadata.ktx.loadString +import java.nio.file.Files +import kotlin.test.Test +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** + * `migrate.foreignKeyIndexes` (.metaobjects/config.json, carried on the loaded root as + * [MigrateSettings]): `meta migrate` gives every enforced foreign key that nothing already leads + * with a `
__idx` index, so the Exposed table declares the same index under the + * same name. The model is the one migrate-ts's expected-schema-foreign-key-indexes test builds. + */ +class KotlinExposedForeignKeyIndexTest { + + private val fixture = """{ + "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Customer", "children": [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.secondary": { "name": "customers_code_key", "@fields": ["code"] } } + ] } }, + { "object.entity": { "name": "Order", "children": [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { "name": "id" } }, + { "field.long": { "name": "customerId", "@column": "customer_id" } }, + { "field.long": { "name": "billToId", "@column": "bill_to_id" } }, + { "field.string": { "name": "customerCode", "@column": "customer_code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.reference": { "name": "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { "name": "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "identity.reference": { "name": "byCode", "@fields": ["customerCode"], "@references": "Customer.code" } }, + { "index.lookup": { "name": "orders_bill_to", "@fields": ["billToId", "id"] } } + ] } }, + { "object.entity": { "name": "OrderTag", "children": [ + { "source.rdb": { "@table": "order_tags" } }, + { "field.long": { "name": "orderId", "@column": "order_id" } }, + { "field.long": { "name": "tagId", "@column": "tag_id" } }, + { "identity.primary": { "name": "pk", "@fields": ["orderId", "tagId"] } }, + { "identity.reference": { "name": "order", "@fields": ["orderId"], "@references": "Order" } }, + { "identity.reference": { "name": "tag", "@fields": ["tagId"], "@references": "Tag" } } + ] } }, + { "object.entity": { "name": "Tag", "children": [ + { "source.rdb": { "@table": "tags" } }, + { "field.long": { "name": "id" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } } + ] } } + ] } + }""".trimIndent() + + private fun tables(settings: MigrateSettings): String { + val outDir = Files.createTempDirectory("ktbl-fkidx-") + try { + val loader = loadString("fk-idx", fixture) + loader.root.setMigrateSettings(settings) + val gen = KotlinExposedTableGenerator() + gen.setArgs(mapOf("outputDir" to outDir.toString())) + gen.execute(loader) + return listOf("OrderTable.kt", "OrderTagTable.kt") + .joinToString("\n") { Files.readString(outDir.resolve("acme/$it")) } + } finally { + outDir.toFile().deleteRecursively() + } + } + + @Test fun `off, the table declares no foreign key index`() { + assertFalse("_idx\"" in tables(MigrateSettings.DEFAULT)) + } + + @Test fun `on, each foreign key nothing leads with gets migrate's index`() { + val src = tables(MigrateSettings(true, null, null)) + assertTrue("index(\"orders_customer_id_idx\", false, customerId)" in src, src) + assertTrue("index(\"orders_customer_code_idx\", false, customerCode)" in src, src) + assertTrue("index(\"order_tags_tag_id_idx\", false, tagId)" in src, src) + // Led by a declared lookup index, and by the junction's primary key. + assertFalse("orders_bill_to_id_idx" in src, src) + assertFalse("order_tags_order_id_idx" in src, src) + } + + @Test fun `the efcore conventions and mysql add nothing`() { + assertFalse("_idx\"" in tables(MigrateSettings(true, "efcore", null))) + assertFalse("_idx\"" in tables(MigrateSettings(true, null, "mysql"))) + } +} diff --git a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java index dd6c41826..0033ddcb3 100644 --- a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java +++ b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java @@ -803,6 +803,11 @@ protected MetaDataLoader createLoader(ClassLoader projectClassLoader) { loader.getRoot().setEnumDefaults(resolveProjectConfig(srcDir) .map(com.metaobjects.config.NeutralConfig::getEnumDefaults) .orElse(com.metaobjects.config.EnumDefaults.DEFAULT)); + // The `migrate` schema settings (foreignKeyIndexes, conventions, dialect), read by the + // table generators so a mapping declares the indexes `meta migrate` creates. + loader.getRoot().setMigrateSettings(resolveProjectConfig(srcDir) + .map(com.metaobjects.config.NeutralConfig::getMigrateSettings) + .orElse(com.metaobjects.config.MigrateSettings.DEFAULT)); getLog().info("MetaData Mojo > Create Loader: " + loader.toString()); diff --git a/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java b/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java index de98496cd..13d00e618 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java +++ b/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java @@ -229,6 +229,21 @@ public void setEnumDefaults(com.metaobjects.config.EnumDefaults enumDefaults) { this.enumDefaults = (enumDefaults != null) ? enumDefaults : com.metaobjects.config.EnumDefaults.DEFAULT; } + /** + * The project's {@code migrate} schema settings, set by the same project-aware load as the + * naming rules and read through {@link com.metaobjects.config.MigrateSettings#of(MetaData)}. + */ + private transient com.metaobjects.config.MigrateSettings migrateSettings = + com.metaobjects.config.MigrateSettings.DEFAULT; + + public com.metaobjects.config.MigrateSettings getMigrateSettings() { + return migrateSettings; + } + + public void setMigrateSettings(com.metaobjects.config.MigrateSettings migrateSettings) { + this.migrateSettings = (migrateSettings != null) ? migrateSettings : com.metaobjects.config.MigrateSettings.DEFAULT; + } + /** The project's wire-format setting for this tree. Never {@code null}. */ public com.metaobjects.config.WireFormat getWireFormat() { return wireFormat; diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/MigrateSettings.java b/server/java/metadata/src/main/java/com/metaobjects/config/MigrateSettings.java new file mode 100644 index 000000000..0d6d93bd4 --- /dev/null +++ b/server/java/metadata/src/main/java/com/metaobjects/config/MigrateSettings.java @@ -0,0 +1,61 @@ +package com.metaobjects.config; + +import com.metaobjects.MetaData; +import com.metaobjects.MetaRoot; + +/** + * The schema settings in the TypeScript-owned {@code migrate} block of + * {@code .metaobjects/config.json} that a generator mapping the schema has to agree with: + * {@code foreignKeyIndexes} (index every foreign key nothing else leads with, under + * {@code
__idx}), {@code conventions} (the schema-conventions preset, or + * {@code null} for the default) and {@code dialect}. Read leniently: {@code meta migrate} + * validates the block, and a value of the wrong type here reads as absent. Carried on the loaded + * root beside the naming rules. + */ +public final class MigrateSettings { + + /** No {@code migrate} settings: no foreign-key indexes, default conventions, no dialect. */ + public static final MigrateSettings DEFAULT = new MigrateSettings(false, null, null); + + private final boolean foreignKeyIndexes; + private final String conventions; + private final String dialect; + + public MigrateSettings(boolean foreignKeyIndexes, String conventions, String dialect) { + this.foreignKeyIndexes = foreignKeyIndexes; + this.conventions = conventions; + this.dialect = dialect; + } + + /** {@code migrate.foreignKeyIndexes}; false when absent. */ + public boolean isForeignKeyIndexes() { + return foreignKeyIndexes; + } + + /** {@code migrate.conventions}, or {@code null} when absent. */ + public String getConventions() { + return conventions; + } + + /** {@code migrate.dialect}, or {@code null} when absent. */ + public String getDialect() { + return dialect; + } + + /** + * Whether a schema mapping declares migrate's foreign-key indexes: the option is on, and + * neither the {@code efcore} conventions (which index every foreign key with their own names) + * nor MySQL (where InnoDB indexes each foreign key itself) already account for them. + */ + public boolean indexesForeignKeys() { + return foreignKeyIndexes && !"efcore".equals(conventions) && !"mysql".equalsIgnoreCase(dialect); + } + + /** The settings for the tree {@code node} belongs to: what its root carries, else none. */ + public static MigrateSettings of(MetaData node) { + for (MetaData n = node; n != null; n = n.getParent()) { + if (n instanceof MetaRoot root) return root.getMigrateSettings(); + } + return DEFAULT; + } +} diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java index 3ee082e13..ea5760550 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java @@ -70,6 +70,7 @@ public final class NeutralConfig { private final PhysicalNaming physicalNaming; private final WireFormat wireFormat; private EnumDefaults enumDefaults = EnumDefaults.DEFAULT; + private MigrateSettings migrateSettings = MigrateSettings.DEFAULT; private List requirementsInherit = List.of(); private NeutralConfig(List> sources, List> dependencies, @@ -151,6 +152,11 @@ public EnumDefaults getEnumDefaults() { return enumDefaults; } + /** The schema settings a mapping generator agrees with ({@code migrate.foreignKeyIndexes} and friends). */ + public MigrateSettings getMigrateSettings() { + return migrateSettings; + } + /** * Returns empty when {@code /.metaobjects/config.json} does not exist. * A file that EXISTS but is malformed throws — swallowing it would make a typo'd @@ -360,6 +366,7 @@ public static NeutralConfig parse(String content, String origin) { NeutralConfig cfg = new NeutralConfig(specs, dependencies, scopeInclude, migrateScope, physicalNaming, wireFormat); cfg.enumDefaults = parseEnums(root.get("enums"), origin); + cfg.migrateSettings = parseMigrateSettings(root.get("migrate")); cfg.requirementsInherit = parseRequirementsInherit(root.get("requirements"), origin); return cfg; } @@ -509,6 +516,23 @@ private static List parseRequirementsInherit(JsonElement requirements, S return stringArray(inherit, "requirements.inherit", origin); } + /** + * {@code migrate.foreignKeyIndexes}, {@code migrate.conventions} and {@code migrate.dialect}, + * read leniently (a wrong type reads as absent): {@code meta migrate} owns and validates them. + */ + private static MigrateSettings parseMigrateSettings(JsonElement migrate) { + if (migrate == null || !migrate.isJsonObject()) return MigrateSettings.DEFAULT; + JsonObject m = migrate.getAsJsonObject(); + JsonElement fk = m.get("foreignKeyIndexes"); + boolean foreignKeyIndexes = fk != null && fk.isJsonPrimitive() && fk.getAsJsonPrimitive().isBoolean() + && fk.getAsBoolean(); + return new MigrateSettings(foreignKeyIndexes, stringOrNull(m.get("conventions")), stringOrNull(m.get("dialect"))); + } + + private static String stringOrNull(JsonElement e) { + return e != null && e.isJsonPrimitive() && e.getAsJsonPrimitive().isString() ? e.getAsString() : null; + } + /** {@code migrate.scope}, or {@code null} when absent; every other {@code migrate} key is ignored. */ private static List parseMigrateScope(JsonElement migrate, String origin) { if (migrate == null || migrate.isJsonNull()) return null; diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java index fac3312f0..fe2fab8c6 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -230,6 +230,11 @@ public static java.util.Optional discoverEnumDefaults(List discoverMigrateSettings(List uris) { + return discoverConfig(uris).map(NeutralConfig::getMigrateSettings); + } + /** * ADR-0058 — the physical-naming rules of the {@code .metaobjects/config.json} found by * walking up from the first {@code file:} URI (stopping after a directory holding diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java b/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java index 1423f8d3b..f4cfa15a3 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java @@ -751,6 +751,9 @@ public static MetaDataLoader fromDirectory(String name, Path directory, loader.getRoot().setEnumDefaults(com.metaobjects.config.SourceResolver .discoverEnumDefaults(List.of(directory.toAbsolutePath().toUri())) .orElse(com.metaobjects.config.EnumDefaults.DEFAULT)); + loader.getRoot().setMigrateSettings(com.metaobjects.config.SourceResolver + .discoverMigrateSettings(List.of(directory.toAbsolutePath().toUri())) + .orElse(com.metaobjects.config.MigrateSettings.DEFAULT)); } catch (MetaDataLoadingException e) { throw e; } catch (Exception e) { @@ -858,6 +861,8 @@ public static MetaDataLoader fromUris(String name, List uris, LoaderOptions .discoverWireFormat(uris).orElse(com.metaobjects.config.WireFormat.DEFAULT)); loader.getRoot().setEnumDefaults(com.metaobjects.config.SourceResolver .discoverEnumDefaults(uris).orElse(com.metaobjects.config.EnumDefaults.DEFAULT)); + loader.getRoot().setMigrateSettings(com.metaobjects.config.SourceResolver + .discoverMigrateSettings(uris).orElse(com.metaobjects.config.MigrateSettings.DEFAULT)); } catch (MetaDataLoadingException e) { throw e; } catch (MetaDataException e) { diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java index 3094c57cc..3aa7ccc51 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java @@ -25,6 +25,9 @@ import java.nio.file.Path; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertThrows; /** @@ -53,6 +56,25 @@ public void whitespaceOnlyPathRaises() throws IOException { assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); } + @Test + public void migrateSchemaSettingsAreReadLeniently() throws IOException { + // migrate.foreignKeyIndexes / conventions / dialect: the settings a table generator + // must agree with `meta migrate` on. TS-owned and validated there, so a wrong type + // reads as absent rather than failing this port's load. + Path dir = writeConfig("{ \"schema_version\": 1, \"migrate\": { \"foreignKeyIndexes\": true, " + + "\"conventions\": \"efcore\", \"dialect\": \"postgres\" } }"); + MigrateSettings m = NeutralConfig.read(dir).orElseThrow().getMigrateSettings(); + assertTrue(m.isForeignKeyIndexes()); + assertEquals("efcore", m.getConventions()); + assertEquals("postgres", m.getDialect()); + assertFalse("the efcore conventions already index every foreign key", m.indexesForeignKeys()); + + Path loose = writeConfig("{ \"schema_version\": 1, \"migrate\": { \"foreignKeyIndexes\": \"yes\" } }"); + assertFalse(NeutralConfig.read(loose).orElseThrow().getMigrateSettings().isForeignKeyIndexes()); + Path none = writeConfig("{ \"schema_version\": 1 }"); + assertSame(MigrateSettings.DEFAULT, NeutralConfig.read(none).orElseThrow().getMigrateSettings()); + } + @Test public void nonStringSourceValueRaises() throws IOException { Path dir = writeConfig("{ \"schema_version\": 1, \"sources\": [ { \"path\": 123 } ] }"); diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index 941a91778..970d037ec 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -570,6 +570,11 @@ async function writeCodegenTsconfig(opts: InitOptions, result: InitResult): Prom result.created.push(CODEGEN_TSCONFIG_REL); } +/** The `migrate` settings a new project starts with (absent in an existing one: no FK indexes). */ +export const NEW_PROJECT_MIGRATE_DEFAULTS = { foreignKeyIndexes: true } as const; +/** The `verify` settings a new project starts with (absent in an existing one: hand edits are a notice). */ +export const NEW_PROJECT_VERIFY_DEFAULTS = { handEdits: "fail" } as const; + /** * .metaobjects/config.json — write fresh defaults, or preserve+merge an existing * valid config. Shared by the full scaffold and `--config-only` so the two paths @@ -579,11 +584,19 @@ async function writeConfigFile(opts: InitOptions, result: InitResult, agentDir: // An explicit sqlite/postgres dialect is written so `meta migrate` needs no --dialect; an // unflagged init keeps the defaults it always wrote. MySQL schema is not migrate's. const dialect = resolveInitDialect(opts); - const freshConfig = dialect === "d1" - ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: buildD1MigrateBlock(opts.cwd) }) - : opts.dialect !== undefined && dialect !== "mysql" - ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: { dialect } }) - : DEFAULT_CONFIG; + const migrate = dialect === "d1" + ? buildD1MigrateBlock(opts.cwd) + : opts.dialect !== undefined ? { dialect } : {}; + // New-project defaults, each written as its own visible setting (a project that predates + // a setting keeps the behaviour it had, because the key is absent there): every foreign key + // gets an index, and a hand edit to a generated file fails `meta verify --codegen`. + // MySQL writes no `migrate` block at all: MetaObjects does not own MySQL schema, and InnoDB + // indexes every foreign key itself. + const freshConfig = ConfigSchema.parse({ + ...DEFAULT_CONFIG, + ...(dialect === "mysql" ? {} : { migrate: { ...migrate, ...NEW_PROJECT_MIGRATE_DEFAULTS } }), + verify: NEW_PROJECT_VERIFY_DEFAULTS, + }); const writeFresh = (): Promise => writeFile(join(agentDir, "config.json"), JSON.stringify(freshConfig, null, 2) + "\n", "utf8"); diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 60987efa2..b3f1cce36 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -10,7 +10,7 @@ import { ENUM_LENGTH_HINT, isEnumLengthRefusal } from "../lib/enum-length-refusal.js"; import { existsSync } from "node:fs"; import { join, relative, resolve as resolvePath } from "node:path"; -import { parseVerifyArgs, type MigrateFlags } from "../lib/args.js"; +import { parseVerifyArgs, type MigrateFlags, type VerifyFlags } from "../lib/args.js"; import { log } from "../lib/log.js"; import { requirementLedgerRows, type RequirementLedgerRow } from "../lib/requirement-ledger.js"; import { warnUnenforcedFkTargets } from "../lib/fk-target-warning.js"; @@ -108,6 +108,7 @@ import { loadMemory, resolveCollection, type DependencySpec, + type HandEditsPolicy, } from "@metaobjectsdev/sdk"; import { checkDependencies, ERR_DEPENDENCY_UPSTREAM_DRIFT, formatCheckLine, readLockOrThrow } from "../lib/dependency-sync.js"; import { exclusionNotes, importedOption, migrateScopeMismatch } from "../lib/migrate-scope.js"; @@ -1088,10 +1089,10 @@ export async function verifyCommand( // Same discipline as its siblings: an advisory scan never breaks verify. } - // Foreign keys migrate emits with no index on the referencing columns. Advisory: an - // automatic index would propose a migration to every existing adopter, so this names - // each one and says how to declare it (`index.lookup`) instead. Read off the SAME - // expected schema `meta migrate` builds, so "covered" means covered in its DDL. + // Foreign keys migrate emits with no index on the referencing columns. Advisory: the + // index is opt-in (`migrate.foreignKeyIndexes`), since turning it on proposes a migration + // to an existing project, so this names each one and says how to index it. Read off the + // SAME expected schema `meta migrate` builds, so with the option on there is none. let unindexedFks: UnindexedFkFinding[] = []; // Skipped on MySQL: MetaObjects emits no DDL there, and InnoDB indexes every foreign // key itself. @@ -1822,8 +1823,8 @@ export async function verifyCommand( // Hand edits are not drift, but they must never be INVISIBLE: a validator weakened // inside a generated file used to pass this gate with "every file's generated - // contribution is current" and nothing else. Reported on every run; fails the build - // only under --forbid-hand-edits. + // contribution is current" and nothing else. Reported on every run; fails the build when + // the run's policy is `fail` (verify.handEdits, or --forbid-hand-edits). const handEditExit = reportHandEdits(result.handEdited); // Say how much documentation was NOT checked, on the pass and the fail line alike, so a // gate that compared two pages of 589 cannot read as one that compared all of them. @@ -1877,13 +1878,19 @@ export async function verifyCommand( return 1; } - /** Print the hand-edit notice (or failure, under --forbid-hand-edits); return its exit code. */ + /** + * Print the hand-edit notice, or the failure when the run's policy is `fail`; return its exit + * code. The policy is the run's flag when one is passed, else `verify.handEdits` in + * `.metaobjects/config.json`, else `notice` (what a project that predates the key has always had). + */ function reportHandEdits(handEdited: readonly string[]): number { if (handEdited.length === 0) return 0; - const [header, ...rest] = handEditNotice(handEdited, flags.forbidHandEdits); + const policy = handEditPolicy(flags, collection.handEdits); + const fail = policy.policy === "fail"; + const [header, ...rest] = handEditNotice(handEdited, policy); const paths = rest.slice(0, handEdited.length); const footer = rest.slice(handEdited.length); - if (flags.forbidHandEdits) { + if (fail) { // A failure lists every path: truncating the list a build failed on hides the cause. for (const line of [header!, ...paths, ...footer]) log.error(line); return 1; @@ -2070,18 +2077,60 @@ export function wireDiffersMessage(committed: WireFormat, used: WireFormat): str ); } -export function handEditNotice(paths: readonly string[], forbidden: boolean): string[] { - const header = forbidden - ? `meta verify — ${paths.length} generated file(s) carry hand edits, and --forbid-hand-edits is set:` - : `meta verify — ${paths.length} generated file(s) carry hand edits ` + - `(notice — 'meta gen' preserves them by design; does not fail the build — ` + - `pass --forbid-hand-edits to make it):`; +/** Where a run's hand-edit policy came from: a flag on this run, the project's config, or neither. */ +export type HandEditPolicySource = "--forbid-hand-edits" | "--allow-hand-edits" | "config" | "default"; + +/** The hand-edit policy one `meta verify --codegen` run applies, and what decided it. */ +export function handEditPolicy( + flags: Pick, + configured: HandEditsPolicy | undefined, +): { policy: HandEditsPolicy; source: HandEditPolicySource } { + if (flags.forbidHandEdits) return { policy: "fail", source: "--forbid-hand-edits" }; + if (flags.allowHandEdits) return { policy: "notice", source: "--allow-hand-edits" }; + if (configured !== undefined) return { policy: configured, source: "config" }; + return { policy: "notice", source: "default" }; +} + +const HAND_EDITS_KEY = `"verify": { "handEdits": … } in .metaobjects/config.json`; + +/** + * The hand-edit report: a header, one line per file, then how to see an edit and the two ways + * forward — drop it (the change belongs in the metadata) or keep it on purpose. + */ +export function handEditNotice( + paths: readonly string[], + run: { policy: HandEditsPolicy; source: HandEditPolicySource }, +): string[] { + const n = `${paths.length} generated file(s) carry hand edits`; + let header: string; + switch (run.source) { + case "--forbid-hand-edits": + header = `meta verify — ${n}, and --forbid-hand-edits is set:`; + break; + case "--allow-hand-edits": + header = `meta verify — ${n} (notice — --allow-hand-edits is set, so this run does not fail on them):`; + break; + case "config": + header = run.policy === "fail" + ? `meta verify — ${n}, and verify.handEdits is "fail" in .metaobjects/config.json:` + : `meta verify — ${n} (notice — verify.handEdits is "notice" in .metaobjects/config.json, so this does not fail the build):`; + break; + default: + header = `meta verify — ${n} (notice — 'meta gen' preserves them by design; does not fail the build — ` + + `set "verify": { "handEdits": "fail" } in .metaobjects/config.json, or pass --forbid-hand-edits, to make it):`; + } + const keep = run.policy === "fail" + ? `To keep an edit on purpose: pass --allow-hand-edits for one run, or set "handEdits": "notice" (${HAND_EDITS_KEY}).` + : `To keep an edit on purpose: nothing to do while hand edits are a notice; a project that wants them to fail sets "handEdits": "fail" (${HAND_EDITS_KEY}).`; return [ header, ...paths.map((p) => ` ${p}`), "To see an edit: delete the file and run 'meta gen' (it re-emits the pristine generation), " + - "then 'git diff -R -- ' shows the edit and 'git checkout -- ' restores it" + - (forbidden ? "; to drop the edit, commit the regenerated file instead." : "."), + "then 'git diff -R -- ' shows the edit and 'git checkout -- ' restores it.", + "To drop an edit: delete the file, run 'meta gen' and commit the regenerated file; if the edit " + + "changed a rule the model declares (a length, a range, a required field), make that change in " + + "the metadata instead, so the database and every port follow it.", + keep, ]; } diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index 7588c84b9..76c9dae03 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -336,10 +336,15 @@ FLAGS: An .names.ts artifact is the exception: it must match a fresh regen exactly — a hand edit in it is itself the defect. Needs metaobjects.config.ts; exit 2 if absent. - Every generated file that carries a hand edit is still LISTED - (a notice, exit code unchanged), so an edit is never invisible. + Every generated file that carries a hand edit is LISTED, so an + edit is never invisible. Whether that fails the run is the + project's verify.handEdits in .metaobjects/config.json: "fail" + exits 1 ('meta init' writes it for a new project); "notice", or + no key, lists them and leaves the exit code alone. --forbid-hand-edits With --codegen: fail (exit 1) on ANY hand edit inside generated - code, for teams that want generated files untouchable. + code for this run, whatever verify.handEdits says. + --allow-hand-edits With --codegen: list hand edits as a notice and exit 0 for this + run, relaxing verify.handEdits: "fail". --docs Removed in 1.1.0: says so and exits 2. --codegen checks documentation with everything else \`generators\` declares. --deps Dependency drift — re-resolve each declared dependency exactly as @@ -413,7 +418,8 @@ A bare 'meta verify' also runs an ADVISORY anti-pattern pass: it scans your auth source for hand-rolled aggregates, money-as-float, and CHECK-IN enums and points you at the construct that models them (origin.aggregate / field.currency / field.enum). The same pass names every foreign key with no index on its referencing column(s) -and the index.lookup that would cover it (migrate never adds one on its own). +and the index.lookup that would cover it (migrate adds one only when +.metaobjects/config.json sets "migrate": { "foreignKeyIndexes": true }). Warnings only — it never fails the build. Opt out with --no-antipatterns or META_NO_ANTIPATTERNS=1. diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index ea8965a75..787c51d6f 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -398,13 +398,17 @@ export interface VerifyFlags { /** Run the codegen-drift gate: is the GENERATED contribution of each file current? */ codegen: boolean; /** - * With `--codegen`: fail the build on any hand edit inside a generated file, for teams - * that want generated code untouchable. Off by default, because hand edits are the - * documented workflow (`meta gen` three-way-merges them) — without it they are - * reported as a notice and never change the exit code. Refused without `--codegen`: - * it modifies that gate and would otherwise parse cleanly and do nothing. + * With `--codegen`: fail the build on any hand edit inside a generated file for THIS run, + * whatever `verify.handEdits` in `.metaobjects/config.json` says. Refused without + * `--codegen`: it modifies that gate and would otherwise parse cleanly and do nothing. */ forbidHandEdits: boolean; + /** + * With `--codegen`: list hand edits as a notice and exit 0 for THIS run, relaxing a + * project's `verify.handEdits: "fail"`. Refused without `--codegen`, and refused beside + * `--forbid-hand-edits`, which asks the opposite. + */ + allowHandEdits: boolean; /** * `--docs`, removed in 1.1.0 (ADR-0034 Amendment 9): `--codegen` checks documentation with * everything else the generator list declares. Still parsed, so the command can say where @@ -528,6 +532,7 @@ export const VERIFY_OPTIONS = { templates: { type: "boolean", default: false }, codegen: { type: "boolean", default: false }, "forbid-hand-edits": { type: "boolean", default: false }, + "allow-hand-edits": { type: "boolean", default: false }, // Removed in 1.1.0 (ADR-0034 Amendment 9). Still parsed, so `meta verify --docs` prints // where the check went instead of an unknown-flag error. docs: { type: "boolean", default: false }, @@ -595,6 +600,15 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { "--forbid-hand-edits only applies to the codegen gate. Run: meta verify --codegen --forbid-hand-edits", ); } + const allowHandEdits = !!values["allow-hand-edits"]; + if (allowHandEdits && !codegen) { + throw new Error( + "--allow-hand-edits only applies to the codegen gate. Run: meta verify --codegen --allow-hand-edits", + ); + } + if (allowHandEdits && forbidHandEdits) { + throw new Error("--allow-hand-edits and --forbid-hand-edits ask for opposite things; pass one of them."); + } const docs = !!values.docs; const deps = !!values.deps; const replay = !!values.replay; @@ -619,6 +633,7 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { templates, codegen, forbidHandEdits, + allowHandEdits, docs, deps, replay, diff --git a/server/typescript/packages/cli/src/lib/fk-index-advisory.ts b/server/typescript/packages/cli/src/lib/fk-index-advisory.ts index 4edfc4444..2668a580f 100644 --- a/server/typescript/packages/cli/src/lib/fk-index-advisory.ts +++ b/server/typescript/packages/cli/src/lib/fk-index-advisory.ts @@ -1,11 +1,13 @@ // Foreign keys with no index covering their columns — an advisory authoring lint. // // `meta migrate` emits a FOREIGN KEY constraint for every enforced `identity.reference` -// and no index on its columns: neither Postgres nor SQLite indexes the referencing side -// of a foreign key. Every join from the parent, and every ON DELETE cascade / set-null / -// restrict check from it, then scans the child table. Generating the index automatically -// would propose a migration to every existing adopter the release it shipped, so this is -// advice, never a gate: it names each such key and says how to declare the index. +// and, unless the project sets `migrate.foreignKeyIndexes`, no index on its columns: +// neither Postgres nor SQLite indexes the referencing side of a foreign key. Every join +// from the parent, and every ON DELETE cascade / set-null / restrict check from it, then +// scans the child table. The option is opt-in (`meta init` writes it for a new project), +// because turning it on proposes a migration to an existing one, so this stays advice, +// never a gate: it names each such key and says how to index it. With the option on, the +// expected schema already carries the index and there is nothing to name. // // Coverage is read off the EXPECTED SCHEMA migrate itself builds (primary key, every // `identity.secondary`, every `index.lookup`, every `@unique` field), not re-derived from @@ -141,8 +143,9 @@ export function summarizeUnindexedFks( construct: findings.map((f) => f.construct).join(", "), message: `${findings.length} foreign key(s) have no index covering their columns, so joins from the ` + - `referenced table and its ON DELETE checks scan the referencing one. To index them, declare ` + - `an ${INDEX_LOOKUP} on each: ` + + `referenced table and its ON DELETE checks scan the referencing one. To index every one, set ` + + `"migrate": { "foreignKeyIndexes": true } in .metaobjects/config.json and run 'meta migrate'; ` + + `to index them one at a time, declare an ${INDEX_LOOKUP} on each: ` + findings.map((f) => `${f.label} ${f.declaration}`).join("; "), }; } diff --git a/server/typescript/packages/cli/src/lib/schema-build-options.ts b/server/typescript/packages/cli/src/lib/schema-build-options.ts index 8067c6e57..5784235a6 100644 --- a/server/typescript/packages/cli/src/lib/schema-build-options.ts +++ b/server/typescript/packages/cli/src/lib/schema-build-options.ts @@ -17,6 +17,8 @@ import type { Collection } from "@metaobjectsdev/sdk"; export interface SchemaBuildOptions { conventions: SchemaConventions; skipRedundantChecks: boolean; + /** `migrate.foreignKeyIndexes`: index every foreign key nothing else indexes. */ + foreignKeyIndexes: boolean; } /** @@ -26,12 +28,13 @@ export interface SchemaBuildOptions { * member (`isUnknownSchemaConventionsError` recognises both). */ export function schemaBuildOptions( - collection: Pick, + collection: Pick, tsConfig: Pick | undefined, ): SchemaBuildOptions { return { conventions: resolveSchemaConventions(tsConfig?.schemaConventions ?? collection.schemaConventions), skipRedundantChecks: collection.skipRedundantChecks, + foreignKeyIndexes: collection.foreignKeyIndexes, }; } @@ -42,7 +45,7 @@ export function schemaBuildOptions( * refuse the same config with exit 2. */ export function schemaBuildOptionsOrWarn( - collection: Pick, + collection: Pick, tsConfig: Pick | undefined, warn: (message: string) => void, ): SchemaBuildOptions | undefined { diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index f909ea2b1..2202d16c3 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -182,6 +182,19 @@ describe("init() — happy path", () => { expect(config.pending_in_git).toBe(true); }); + test("writes the new-project defaults as visible settings: FK indexes on, hand edits fail verify", async () => { + await init({ cwd }); + const config = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(config.migrate).toEqual({ foreignKeyIndexes: true }); + expect(config.verify).toEqual({ handEdits: "fail" }); + }); + + test("a dialect flag keeps the new-project defaults beside migrate.dialect", async () => { + await init({ cwd, dialect: "postgres" }); + const config = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(config.migrate).toEqual({ dialect: "postgres", foreignKeyIndexes: true }); + }); + test("writes a .gitignore under .metaobjects/ that includes .gen-state", async () => { await init({ cwd }); const ignore = readFileSync(join(cwd, ".metaobjects", ".gitignore"), "utf8"); @@ -508,6 +521,9 @@ describe("init() --config-only", () => { expect(result.preserved).toContain(".metaobjects/config.json"); const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); expect(cfg.sources).toEqual([{ path: "model" }]); + // An existing project keeps the behaviour it had: no new-project default is added to it. + expect(cfg.migrate).toBeUndefined(); + expect(cfg.verify).toBeUndefined(); }); test("refuses to overwrite an existing config it cannot parse, without --force", async () => { diff --git a/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts b/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts index cef8b17a9..eb752aff5 100644 --- a/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts +++ b/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts @@ -373,6 +373,10 @@ describe("meta verify --codegen — hand edits are visible", () => { expect(report).toContain("--forbid-hand-edits"); expect(report).toMatch(/\s generated\/User\.ts$/m); expect(report).toContain("git diff -R -- "); + // The no-key notice names the config key that makes it a failure, and both remedies. + expect(report).toContain('"verify": { "handEdits": "fail" }'); + expect(report).toContain("To drop an edit:"); + expect(report).toContain("To keep an edit on purpose:"); } finally { rmSync(root, { recursive: true, force: true }); } @@ -454,4 +458,104 @@ describe("meta verify --codegen — hand edits are visible", () => { rmSync(root, { recursive: true, force: true }); } }); + + /** A project's own `verify.handEdits`, written into its .metaobjects/config.json. */ + function setHandEdits(root: string, handEdits: string): void { + mkdirSync(join(root, ".metaobjects"), { recursive: true }); + const path = join(root, ".metaobjects", "config.json"); + const prior = existsSync(path) ? JSON.parse(readFileSync(path, "utf8")) as Record : { schema_version: 1 }; + writeFileSync(path, JSON.stringify({ ...prior, verify: { handEdits } }, null, 2)); + } + + test('verify.handEdits: "fail" fails the run, naming the file and both remedies', async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "fail"); + weakenValidator(root); + + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(1); + const report = all(); + expect(report).toContain('verify.handEdits is "fail"'); + expect(report).toMatch(/\s generated\/User\.ts$/m); + expect(report).toContain("To drop an edit: delete the file, run 'meta gen'"); + expect(report).toContain("make that change in the metadata instead"); + expect(report).toContain("pass --allow-hand-edits for one run"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("--allow-hand-edits relaxes verify.handEdits: \"fail\" for one run", async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "fail"); + weakenValidator(root); + + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen", "--allow-hand-edits"])).toBe(0); + expect(all()).toContain("--allow-hand-edits is set"); + expect(all()).toContain("1 generated file(s) carry hand edits"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test('verify.handEdits: "fail" passes a project with no hand edits', async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "fail"); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(0); + expect(all()).not.toContain("carry hand edits"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test('verify.handEdits: "notice" keeps the notice, and --forbid-hand-edits still tightens it', async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "notice"); + weakenValidator(root); + + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(0); + expect(all()).toContain('verify.handEdits is "notice"'); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen", "--forbid-hand-edits"])).toBe(1); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("a verify.handEdits value this version does not know is refused", async () => { + const root = setupRepo(); + try { + setHandEdits(root, "warn"); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(2); + expect(all()).toContain('"verify.handEdits" must be one of "fail", "notice"'); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("--allow-hand-edits needs --codegen, and is refused beside --forbid-hand-edits", async () => { + const root = setupRepo(); + try { + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--allow-hand-edits"])).toBe(2); + expect(all()).toContain("--codegen"); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen", "--allow-hand-edits", "--forbid-hand-edits"])).toBe(2); + expect(all()).toContain("opposite"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); }); diff --git a/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts b/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts index 0f0a8adad..6142db596 100644 --- a/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts +++ b/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts @@ -24,7 +24,7 @@ afterAll(() => { }); describe("meta verify is not a vacuous gate", () => { - test("clean tree exits 0; metadata drift, a forbidden hand edit and a deleted file each exit 1", async () => { + test("clean tree exits 0; metadata drift, a hand edit and a deleted file each exit 1", async () => { root = await newShopProject("verify-gate-"); const configPath = join(root, "metaobjects.config.ts"); writeFileSync( @@ -61,15 +61,17 @@ describe("meta verify is not a vacuous gate", () => { writeModel(root, SHOP_MODEL); expect((await meta(root, "verify")).exit).toBe(0); - // 4. A hand edit inside a generated file is NOT drift by default: `meta gen` three-way-merges - // it, so plain `verify` stays green. A team that wants generated code untouchable opts in - // with --forbid-hand-edits; that is the gate to assert if that is your policy. + // 4. Break B: a hand edit inside a generated file. `meta init` writes + // `"verify": { "handEdits": "fail" }`, so plain `verify` fails on it; --allow-hand-edits + // relaxes that for one run (and still names the file). const edit = pristine.replace(/email: (\w+)\("email"\)/, 'emailAddress: $1("email")'); expect(edit).not.toBe(pristine); writeFileSync(customer, edit); - expect((await meta(root, "verify")).exit).toBe(0); - const edited = await meta(root, "verify", "--codegen", "--forbid-hand-edits"); + const edited = await meta(root, "verify"); expect({ step: "edited generated part", exit: edited.exit }).toEqual({ step: "edited generated part", exit: 1 }); + expect(edited.output).toContain("Customer.ts"); + const allowed = await meta(root, "verify", "--codegen", "--allow-hand-edits"); + expect({ step: "edit allowed for one run", exit: allowed.exit }).toEqual({ step: "edit allowed for one run", exit: 0 }); writeFileSync(customer, pristine); // 5. Break C: a generated file removed. diff --git a/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts b/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts index fb36b12dd..fc86c6d83 100644 --- a/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts +++ b/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts @@ -36,13 +36,17 @@ const MODEL = JSON.stringify({ }, }); -async function project(conventions: string | undefined): Promise { +async function project(conventions: string | undefined, foreignKeyIndexes?: boolean): Promise { const root = await mkdtemp(join(tmpdir(), "schema-conventions-")); dirs.push(root); await mkdir(join(root, "metaobjects"), { recursive: true }); await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); await mkdir(join(root, ".metaobjects", "migrations"), { recursive: true }); - const migrate = conventions === undefined ? { dialect: "sqlite" } : { dialect: "sqlite", conventions }; + const migrate = { + dialect: "sqlite", + ...(conventions !== undefined && { conventions }), + ...(foreignKeyIndexes !== undefined && { foreignKeyIndexes }), + }; await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, migrate }), "utf8"); return root; } @@ -77,6 +81,25 @@ describe("meta migrate with migrate.conventions", () => { expect(out.join("\n")).not.toContain("IX_OrderLines_OrderId"); }); + test("migrate.foreignKeyIndexes: the foreign key gets
__idx, with no renamed constraint", async () => { + const root = await project(undefined, true); + + expect(await dryRun(root)).toBe(0); + const text = out.join("\n"); + expect(text).toContain('CREATE INDEX "OrderLines_OrderId_idx" ON "OrderLines" ("OrderId");'); + expect(text).not.toContain("IX_OrderLines_OrderId"); + expect(text).not.toContain("PK_"); + }); + + test("migrate.foreignKeyIndexes under efcore adds nothing: the preset already indexes it", async () => { + const root = await project("efcore", true); + + expect(await dryRun(root)).toBe(0); + const text = out.join("\n"); + expect(text).toContain('CREATE INDEX "IX_OrderLines_OrderId"'); + expect(text).not.toContain("OrderLines_OrderId_idx"); + }); + test("an unknown preset is refused by name, not as an unexpected error", async () => { const root = await project("hibernate"); diff --git a/server/typescript/packages/cli/test/unit/args-verify.test.ts b/server/typescript/packages/cli/test/unit/args-verify.test.ts index fe758ff27..ff8597293 100644 --- a/server/typescript/packages/cli/test/unit/args-verify.test.ts +++ b/server/typescript/packages/cli/test/unit/args-verify.test.ts @@ -6,7 +6,7 @@ describe("parseVerifyArgs", () => { test("defaults: prompts/db/dialect undefined, allow empty, skipSchema false, no explicit subverb", () => { expect(parseVerifyArgs([])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: [], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -14,7 +14,7 @@ describe("parseVerifyArgs", () => { test("--prompts is captured", () => { expect(parseVerifyArgs(["--prompts", "templates"])).toEqual({ prompts: "templates", db: undefined, dialect: undefined, allow: [], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -22,7 +22,7 @@ describe("parseVerifyArgs", () => { test("--db / --dialect / --skip-schema are captured", () => { expect(parseVerifyArgs(["--db", "file:x.db", "--dialect", "sqlite", "--skip-schema"])).toEqual({ prompts: undefined, db: "file:x.db", dialect: "sqlite", allow: [], skipSchema: true, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -32,7 +32,7 @@ describe("parseVerifyArgs", () => { test("--dialect d1 --d1 --remote are captured; --dialect d1 alone is an explicit subverb", () => { expect(parseVerifyArgs(["--dialect", "d1", "--d1", "DB", "--remote"])).toEqual({ prompts: undefined, db: undefined, dialect: "d1", allow: [], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: "DB", remote: true, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -77,7 +77,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--allow", "drop-column,drop-table"])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: ["drop-column", "drop-table"], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -157,6 +157,13 @@ describe("parseVerifyArgs", () => { // so it must not switch the bare-verify template default off. expect(parseVerifyArgs(["--require-implementers"]).anyExplicit).toBe(false); }); + test("--allow-hand-edits rides on --codegen, and is refused without it or beside --forbid-hand-edits", () => { + expect(parseVerifyArgs(["--codegen"]).allowHandEdits).toBe(false); + expect(parseVerifyArgs(["--codegen", "--allow-hand-edits"]).allowHandEdits).toBe(true); + expect(() => parseVerifyArgs(["--allow-hand-edits"])).toThrow(/only applies to the codegen gate/); + expect(() => parseVerifyArgs(["--codegen", "--allow-hand-edits", "--forbid-hand-edits"])).toThrow(/opposite/); + }); + test("--forbid-hand-edits rides on --codegen, and is refused without it", () => { expect(parseVerifyArgs(["--codegen"]).forbidHandEdits).toBe(false); expect(parseVerifyArgs(["--codegen", "--forbid-hand-edits"]).forbidHandEdits).toBe(true); diff --git a/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts b/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts index 375b4be8a..c2d73eceb 100644 --- a/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts +++ b/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts @@ -49,7 +49,7 @@ function input(over: Partial & Pick { +async function project(migrate?: Record): Promise { const root = await mkdtemp(join(tmpdir(), "verify-fk-index-")); dirs.push(root); await mkdir(join(root, "metaobjects"), { recursive: true }); await writeFile(join(root, "metaobjects", "meta.tracker.json"), MODEL, "utf8"); + if (migrate !== undefined) { + await mkdir(join(root, ".metaobjects"), { recursive: true }); + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, migrate }), "utf8"); + } return root; } @@ -86,6 +90,20 @@ describe("meta verify — unindexed foreign keys (advisory)", () => { expect(out).not.toContain("Issue.fkOwner"); }); + test("names migrate.foreignKeyIndexes as the way to index every one", async () => { + const root = await project(); + const { out } = await capture(() => verifyCommand([], root)); + expect(out).toContain('"migrate": { "foreignKeyIndexes": true }'); + }); + + test("with migrate.foreignKeyIndexes on, migrate indexes the key, so there is nothing to name", async () => { + const root = await project({ foreignKeyIndexes: true }); + const { code, out } = await capture(() => verifyCommand([], root)); + expect(code).toBe(0); + expect(out).not.toContain("Issue.fkProject"); + expect(out).not.toMatch(/foreign key\(s\) with no index/); + }); + test("--no-antipatterns suppresses it with the rest of the advisory tier", async () => { const root = await project(); const { out } = await capture(() => verifyCommand(["--no-antipatterns"], root)); diff --git a/server/typescript/packages/codegen-ts/src/engine-surface.ts b/server/typescript/packages/codegen-ts/src/engine-surface.ts index b5df86a8a..7df2cc777 100644 --- a/server/typescript/packages/codegen-ts/src/engine-surface.ts +++ b/server/typescript/packages/codegen-ts/src/engine-surface.ts @@ -36,6 +36,7 @@ export const ENGINE_MODULES: readonly string[] = [ "codegen-ts/src/enum-meta.ts", "codegen-ts/src/enum-shared.ts", "codegen-ts/src/flattened.ts", + "codegen-ts/src/foreign-key-indexes.ts", "codegen-ts/src/format.ts", "codegen-ts/src/generator.ts", "codegen-ts/src/generators/agent-schema-input.ts", diff --git a/server/typescript/packages/codegen-ts/src/engine.ts b/server/typescript/packages/codegen-ts/src/engine.ts index 2ce675ed6..d0ff8bd03 100644 --- a/server/typescript/packages/codegen-ts/src/engine.ts +++ b/server/typescript/packages/codegen-ts/src/engine.ts @@ -23,6 +23,7 @@ export * from "./enum-import.js"; export * from "./enum-meta.js"; export * from "./enum-shared.js"; export * from "./flattened.js"; +export * from "./foreign-key-indexes.js"; export * from "./format.js"; export * from "./generator.js"; export * from "./generators/agent-schema-input.js"; diff --git a/server/typescript/packages/codegen-ts/src/foreign-key-indexes.ts b/server/typescript/packages/codegen-ts/src/foreign-key-indexes.ts new file mode 100644 index 000000000..f9bf6e035 --- /dev/null +++ b/server/typescript/packages/codegen-ts/src/foreign-key-indexes.ts @@ -0,0 +1,61 @@ +// The indexes `meta migrate` creates because a table has foreign keys, for a generator that maps +// the schema (the Drizzle table's index callback). +// +// Read off the SAME expected schema migrate builds, with the project's schema settings, so the +// mapping declares exactly the indexes the database gets, under the names it gets them. Only +// when `migrate.foreignKeyIndexes` is on: a project without it keeps the mapping it had, and +// `meta verify` advises about each unindexed foreign key instead. + +import type { MetaObject, MetaRoot } from "@metaobjectsdev/metadata"; +import { + buildExpectedSchemaWithProvenance, + type SchemaBuildSettings, +} from "@metaobjectsdev/migrate-ts/expected"; +import type { Dialect } from "./column-mapper.js"; + +export interface ForeignKeyIndex { + name: string; + columns: readonly string[]; +} + +type ByFqn = ReadonlyMap; + +// One schema build per (root, settings, dialect) — a run renders every entity against one root. +const cache = new WeakMap>>(); + +function build(root: MetaRoot, settings: SchemaBuildSettings, dialect: "sqlite" | "postgres"): ByFqn { + const out = new Map(); + let built: ReturnType; + try { + built = buildExpectedSchemaWithProvenance(root, { ...settings, dialect }); + } catch { + // A model the schema builder refuses is `meta migrate`'s to report, with its own message. + return out; + } + for (const [table, indexes] of built.foreignKeyIndexes ?? []) { + const fqn = built.provenance.get(table); + if (fqn !== undefined) out.set(fqn, indexes.map((ix) => ({ name: ix.name, columns: [...ix.columns] }))); + } + return out; +} + +/** + * The foreign-key indexes of `obj`'s table, in the order migrate adds them; none unless the + * project sets `migrate.foreignKeyIndexes`. None on MySQL either: MetaObjects writes no DDL + * there, and InnoDB indexes every foreign key itself. + */ +export function foreignKeyIndexesOf( + obj: MetaObject, + root: MetaRoot, + settings: SchemaBuildSettings | undefined, + dialect: Dialect, +): readonly ForeignKeyIndex[] { + if (settings?.foreignKeyIndexes !== true || dialect === "mysql") return []; + let bySettings = cache.get(root); + if (bySettings === undefined) cache.set(root, (bySettings = new WeakMap())); + let byDialect = bySettings.get(settings); + if (byDialect === undefined) bySettings.set(settings, (byDialect = new Map())); + let byFqn = byDialect.get(dialect); + if (byFqn === undefined) byDialect.set(dialect, (byFqn = build(root, settings, dialect))); + return byFqn.get(obj.resolutionKey()) ?? []; +} diff --git a/server/typescript/packages/codegen-ts/src/render-context.ts b/server/typescript/packages/codegen-ts/src/render-context.ts index ac4a111aa..45aef1f77 100644 --- a/server/typescript/packages/codegen-ts/src/render-context.ts +++ b/server/typescript/packages/codegen-ts/src/render-context.ts @@ -3,6 +3,7 @@ import type { MetaRoot, MetaData } from "@metaobjectsdev/metadata"; import { FIELD_ATTR_OBJECT_REF, PACKAGE_SEPARATOR, TYPE_OBJECT, collectionNameOf, declaredCollectionName, physicalNamingOf, refTarget, stripPackage } from "@metaobjectsdev/metadata"; import type { Dialect } from "./column-mapper.js"; +import type { SchemaBuildSettings } from "@metaobjectsdev/migrate-ts"; import { normalizeTimestampMode } from "./dialect-module.js"; import type { PkInfo } from "./pk-resolver.js"; import type { RelationMap } from "./relation-resolver.js"; @@ -99,6 +100,11 @@ export interface RenderContext { * at the `makeRenderContext` / `normalizeConfig` choke points; see the fuller * doc comment on `MetaobjectsGenConfig.timestampMode` in metaobjects-config.ts. */ + /** + * The project's schema settings (`migrate.conventions` / `checks` / `foreignKeyIndexes`), for + * a template that maps the schema `meta migrate` builds. Absent in a bare-context render. + */ + schemaSettings?: SchemaBuildSettings; timestampMode: "date" | "string"; /** * Prepend `"use client";` to generated CLIENT artifacts (forms, hooks, grid @@ -180,6 +186,7 @@ export type RenderContextInput = Omit { selfTarget, entityModuleTarget, ...(config.providedEnumModule !== undefined && { providedEnumModule: config.providedEnumModule }), + ...(opts.schemaSettings !== undefined && { schemaSettings: opts.schemaSettings }), }); const ctx: GenContext = { entities: safeEntities, diff --git a/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts b/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts index 517f7c871..813142ef0 100644 --- a/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts +++ b/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "../tph.js"; import { flattenedMembersOf } from "../flattened.js"; import { effectivePackage } from "../docs-paths.js"; +import { foreignKeyIndexesOf } from "../foreign-key-indexes.js"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) { diff --git a/server/typescript/packages/codegen-ts/test/drizzle-foreign-key-indexes.test.ts b/server/typescript/packages/codegen-ts/test/drizzle-foreign-key-indexes.test.ts new file mode 100644 index 000000000..7912b6a70 --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/drizzle-foreign-key-indexes.test.ts @@ -0,0 +1,70 @@ +// `migrate.foreignKeyIndexes`: the Drizzle table declares the index `meta migrate` gives each +// foreign key nothing else indexes, under migrate's name. Without the setting the table is +// what it was. +import { describe, test, expect } from "bun:test"; +import { MetaDataLoader, InMemoryStringSource } from "@metaobjectsdev/metadata"; +import { defaultConventions, efcoreConventions } from "@metaobjectsdev/migrate-ts"; +import { renderDrizzleSchema } from "../src/templates/drizzle-schema.js"; +import { makeRenderContext } from "../src/render-context.js"; +import { buildPkMap } from "../src/pk-resolver.js"; +import { buildRelationMap } from "../src/relation-resolver.js"; +import type { Dialect } from "../src/metaobjects-config.js"; +import type { SchemaBuildSettings } from "@metaobjectsdev/migrate-ts"; + +const DOC = JSON.stringify({ "metadata.root": { package: "acme", children: [ + { "object.entity": { name: "Customer", children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ] } }, + { "object.entity": { name: "Order", children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "field.long": { name: "customerId" } }, + { "field.long": { name: "billToId" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + { "identity.reference": { name: "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { name: "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "index.lookup": { name: "orders_bill_to", "@fields": ["billToId"] } }, + ] } }, +] } }); + +async function emit(dialect: Dialect, schemaSettings?: SchemaBuildSettings): Promise { + const res = await new MetaDataLoader().load([new InMemoryStringSource(DOC)]); + expect(res.errors).toEqual([]); + const ctx = makeRenderContext({ + dialect, loadedRoot: res.root, outDir: "/x", dbImport: "~/db", + pkMap: buildPkMap(res.root), relationMap: buildRelationMap(res.root), + ...(schemaSettings !== undefined ? { schemaSettings } : {}), + }); + return renderDrizzleSchema(res.root.findObject("Order")!, ctx).toString(); +} + +const ON: SchemaBuildSettings = { conventions: defaultConventions, skipRedundantChecks: false, foreignKeyIndexes: true }; + +describe("Drizzle: foreign-key indexes", () => { + test("off: no index beyond the ones the model declares", async () => { + const off = await emit("postgres", { ...ON, foreignKeyIndexes: false }); + expect(off).not.toContain("customer_id_idx"); + expect(off).toBe(await emit("postgres")); + }); + + test("on: migrate's index for each uncovered foreign key, on the column's key", async () => { + for (const dialect of ["postgres", "sqlite"] as const) { + const on = await emit(dialect, ON); + expect(on).toContain('index("orders_customer_id_idx").on(table.customerId)'); + // billToId is already led by the declared orders_bill_to index. + expect(on).not.toContain("orders_bill_to_id_idx"); + } + }); + + test("on, efcore preset: the preset's IX_ names, which are the indexes migrate creates", async () => { + const on = await emit("postgres", { ...ON, conventions: efcoreConventions }); + expect(on).toContain('index("IX_orders_customer_id").on(table.customerId)'); + expect(on).not.toContain("orders_customer_id_idx"); + }); + + test("MySQL: none, since InnoDB indexes every foreign key itself", async () => { + expect(await emit("mysql", ON)).not.toContain("customer_id_idx"); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts b/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts index d125a4356..b1d08db54 100644 --- a/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts +++ b/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts @@ -172,6 +172,14 @@ describe("prisma-schema — what the schema says", () => { .toStartWith('@relation("FK_employees_employees_manager_id", '); }); + test("migrate.foreignKeyIndexes declares the index meta migrate gives each foreign key", async () => { + const root = await loadShop(); + const blockOf = (spec: ReturnType) => spec.models.find((m) => m.name === "Employee")!.blockAttributes; + expect(blockOf(buildPrismaSchema(root, { provider: "postgresql" })).join("\n")).not.toContain("employees_manager_id_idx"); + expect(blockOf(buildPrismaSchema(root, { provider: "postgresql", schemaSettings: { foreignKeyIndexes: true } }))) + .toContain('@@index([managerId], map: "employees_manager_id_idx")'); + }); + test("the generator reads the settings from the run (GenContext.schemaSettings)", async () => { await runGen({ config: defineConfig({ outDir: tmp, dialect: "postgres", generators: [prismaSchemaFile({ provider: "postgresql" })] }), diff --git a/server/typescript/packages/integration-tests/src/migration-scenario.ts b/server/typescript/packages/integration-tests/src/migration-scenario.ts index 9422f7779..9e2f9745b 100644 --- a/server/typescript/packages/integration-tests/src/migration-scenario.ts +++ b/server/typescript/packages/integration-tests/src/migration-scenario.ts @@ -13,12 +13,12 @@ import { Pool } from "pg"; import { applyNearestNaming, loadMetadataDir } from "./load-metadata.ts"; import { canonicalJson, normalizeRow } from "./normalization.ts"; import { executeSql, queryRows } from "./postgres-sql.ts"; -import type { MigrationScenario } from "./scenario.ts"; +import type { MigrationScenario, MigrationSchemaSettings } from "./scenario.ts"; export async function runMigrationScenario(scenario: MigrationScenario, connectionUri: string): Promise { // 1. Bootstrap the starting schema, if any. if (scenario.seedMetadataDir) { - const seedSql = await buildFullCreate(scenario.seedMetadataDir); + const seedSql = await buildFullCreate(scenario.seedMetadataDir, scenario.seedSchemaSettings); await executeSql(connectionUri, seedSql); } // 2. Seed data. @@ -33,6 +33,7 @@ export async function runMigrationScenario(scenario: MigrationScenario, connecti // without it `programId` would become `program_id` and the bootstrap up-SQL assertion // for `weeks_programId_fk` would never match. const expected = buildExpectedSchema(targetRoot, { + ...scenario.schemaSettings, views: buildProjectionViews(targetRoot, { dialect: "postgres" }), }); @@ -79,9 +80,10 @@ export async function runMigrationScenario(scenario: MigrationScenario, connecti // Helpers // --------------------------------------------------------------------------- -async function buildFullCreate(metadataDir: string): Promise { +async function buildFullCreate(metadataDir: string, settings: MigrationSchemaSettings): Promise { const root = await loadMetadataDir(metadataDir); const expected = buildExpectedSchema(root, { + ...settings, views: buildProjectionViews(root, { dialect: "postgres" }), }); const result = await diff({ expected, actual: { tables: [], views: [] } }); diff --git a/server/typescript/packages/integration-tests/src/scenario.ts b/server/typescript/packages/integration-tests/src/scenario.ts index 7254ce221..3cbd32a84 100644 --- a/server/typescript/packages/integration-tests/src/scenario.ts +++ b/server/typescript/packages/integration-tests/src/scenario.ts @@ -14,9 +14,18 @@ export interface MigrationScenario { readonly seedData: string | null; readonly targetMetadataDir: string | null; readonly targetMetadataInline: string | null; + /** `seed-schema-settings`: the project schema settings the seed schema is built with. */ + readonly seedSchemaSettings: MigrationSchemaSettings; + /** `schema-settings`: the project schema settings the target is built with (`migrate.*`). */ + readonly schemaSettings: MigrationSchemaSettings; readonly expect: MigrationExpect; } +/** The `migrate` settings in `.metaobjects/config.json` a scenario can set, by their config names. */ +export interface MigrationSchemaSettings { + readonly foreignKeyIndexes?: boolean; +} + export interface MigrationExpect { readonly blocked: ReadonlyArray<{ kind: string; reasonContains: string }>; readonly upContains: ReadonlyArray; @@ -128,6 +137,8 @@ export function loadMigration(yamlPath: string): MigrationScenario { seedData: raw["seed-data"] ?? null, targetMetadataDir: resolveRel(base, raw["target-metadata"]), targetMetadataInline: raw["target-metadata-inline"] ?? null, + seedSchemaSettings: schemaSettings(raw["seed-schema-settings"], yamlPath, "seed-schema-settings"), + schemaSettings: schemaSettings(raw["schema-settings"], yamlPath, "schema-settings"), expect: { blocked: (raw.expect?.blocked ?? []).map((b) => ({ kind: b.kind ?? "", @@ -193,6 +204,17 @@ function resolveRel(base: string, rel: string | undefined | null): string | null return rel ? resolve(base, rel) : null; } +// A key this runner does not know is refused, so a typo cannot silently run the default. +function schemaSettings(raw: Record | undefined, path: string, key: string): MigrationSchemaSettings { + if (raw === undefined || raw === null) return {}; + const out: { foreignKeyIndexes?: boolean } = {}; + for (const [k, v] of Object.entries(raw)) { + if (k === "foreignKeyIndexes" && typeof v === "boolean") out.foreignKeyIndexes = v; + else throw new Error(`${path}: ${key}.${k} is not a schema setting this runner applies (known: foreignKeyIndexes, a boolean)`); + } + return out; +} + // ---------- YAML wire shapes (hyphenated keys per the fixture format) ---------- interface MigrationYaml { @@ -202,6 +224,8 @@ interface MigrationYaml { "seed-data"?: string; "target-metadata"?: string; "target-metadata-inline"?: string; + "seed-schema-settings"?: Record; + "schema-settings"?: Record; expect?: { blocked?: { kind?: string; "reason-contains"?: string }[]; "up-contains"?: string[]; diff --git a/server/typescript/packages/migrate-ts/src/drift/drift.ts b/server/typescript/packages/migrate-ts/src/drift/drift.ts index 246247301..00116243f 100644 --- a/server/typescript/packages/migrate-ts/src/drift/drift.ts +++ b/server/typescript/packages/migrate-ts/src/drift/drift.ts @@ -147,6 +147,7 @@ export async function computeDriftFromActual( ...(opts?.views !== undefined ? { views: opts.views } : {}), conventions: opts?.conventions, skipRedundantChecks: opts?.skipRedundantChecks, + foreignKeyIndexes: opts?.foreignKeyIndexes, }), opts?.inScope, opts?.imported !== undefined ? { imported: opts.imported } : undefined, diff --git a/server/typescript/packages/migrate-ts/src/expected-schema.ts b/server/typescript/packages/migrate-ts/src/expected-schema.ts index 362648727..a711be5bb 100644 --- a/server/typescript/packages/migrate-ts/src/expected-schema.ts +++ b/server/typescript/packages/migrate-ts/src/expected-schema.ts @@ -136,13 +136,22 @@ export interface BuildExpectedSchemaOptions { * absent = false, every CHECK emitted. */ skipRedundantChecks?: boolean | undefined; + /** + * Index every foreign key the conventions left without one: each enforced foreign key whose + * columns no primary key, index or earlier foreign-key index already leads with gets a plain + * index named `
_[_…]_idx`. `migrate.foreignKeyIndexes` in + * `.metaobjects/config.json`; absent = false, the schema migrate has always built. Separate + * from the conventions, so a project can have the indexes without renamed constraints; under + * the `efcore` preset every foreign key is already indexed and this adds nothing. + */ + foreignKeyIndexes?: boolean | undefined; } /** * The project settings an expected-schema build takes, which every path that builds one * (migrate, drift, the offline plan, a baseline) passes through unchanged so they agree. */ -export type SchemaBuildSettings = Pick; +export type SchemaBuildSettings = Pick; /** * A view as the caller (codegen-ts `buildProjectionViews`) produces it — structurally @@ -193,6 +202,12 @@ export interface ExpectedSchemaWithProvenance { * a write with enforcement on fails. The violations are then reported, never refused. */ fkTargetsUnenforced?: true; + /** + * The indexes the schema carries because a table has foreign keys — added by the conventions + * (`efcore`) or by `foreignKeyIndexes` — keyed by the table's qualified name. A generator that + * maps the schema (Drizzle) declares these beside the indexes the model names. Absent when none. + */ + foreignKeyIndexes?: ReadonlyMap; } /** @@ -329,7 +344,7 @@ export function buildExpectedSchemaWithProvenance( // Checked against the BUILT tables, before the conventions add FK indexes: a key the model // declares but never emits (on a TPH subtype) is no key a database can reference. const fkTargetViolations = collectFkTargetViolations(tables, fkCtx.origins, provenance); - applySchemaConventions(tables, conventions, dialect); + const foreignKeyIndexes = applySchemaConventions(tables, conventions, dialect, opts?.foreignKeyIndexes === true); checkDeclaredCheckNames(tables, dialect); // Pass 3: dialect-specific SqlType normalization. @@ -491,6 +506,7 @@ export function buildExpectedSchemaWithProvenance( return { snapshot: { tables, views }, provenance, + ...(foreignKeyIndexes.size > 0 ? { foreignKeyIndexes } : {}), ...(fkTargetViolations.length > 0 ? { fkTargetViolations } : {}), // D1 always enforces foreign keys; plain SQLite only with the pragma on. ...(fkTargetViolations.length > 0 && opts?.dialect === DIALECT_SQLITE ? { fkTargetsUnenforced: true as const } : {}), @@ -1368,7 +1384,10 @@ function collectFkTargetViolations( * foreign key gets. FK names were decided while the FKs were built, where a declared * `@constraintName` is known. The default conventions change nothing here. */ -function applySchemaConventions(tables: TableDescriptor[], conventions: SchemaConventions, dialect: Dialect | undefined): void { +function applySchemaConventions( + tables: TableDescriptor[], conventions: SchemaConventions, dialect: Dialect | undefined, indexForeignKeys: boolean, +): Map { + const added = new Map(); // An FK references a table in its own schema (the emitter's assumption), so the key is schema-qualified. const keyOf = (schema: string | undefined, table: string, columns: readonly string[]) => `${schema ?? ""}.${table}|${[...columns].sort().join(",")}`; @@ -1393,20 +1412,64 @@ function applySchemaConventions(tables: TableDescriptor[], conventions: SchemaCo indexes: t.indexes, foreignKeys: t.foreignKeys.map((fk) => ({ table: t.name, columns: fk.columns, refTable: fk.refTable, refColumns: fk.refColumns, dialect })), }); - for (const index of fkIndexes) { + const owner = (index: { columns: readonly string[] }) => + `the ${conventions.id} conventions' foreign-key index on [${index.columns.join(", ")}] of table "${t.name}"`; + const tableAdded: IndexDescriptor[] = []; + const push = (index: { name: string; columns: readonly string[] }, label: string) => { if (t.indexes.some((ix) => ix.name === index.name)) { throw new DuplicateSqlNameError([{ name: index.name, - owners: [ - `the ${conventions.id} conventions' foreign-key index on [${index.columns.join(", ")}] of table "${t.name}"`, - `an index already named "${index.name}" on table "${t.name}"`, - ], + owners: [label, `an index already named "${index.name}" on table "${t.name}"`], involvesIndex: true, }]); } - t.indexes.push({ name: index.name, columns: [...index.columns], unique: false }); + const descriptor: IndexDescriptor = { name: index.name, columns: [...index.columns], unique: false }; + t.indexes.push(descriptor); + tableAdded.push(descriptor); + }; + for (const index of fkIndexes) push(index, owner(index)); + // `foreignKeyIndexes`: after the conventions, so a preset that indexes foreign keys + // (efcore) leaves this nothing to add and its names stand. + if (indexForeignKeys) { + for (const columns of uncoveredForeignKeyColumns(t)) { + push( + { name: foreignKeyIndexName(t.name, columns), columns }, + `the foreign-key index on [${columns.join(", ")}] of table "${t.name}" (migrate.foreignKeyIndexes)`, + ); + } } + if (tableAdded.length > 0) added.set(qualifiedDbName(t), tableAdded); + } + return added; +} + +/** The name `foreignKeyIndexes` gives a foreign key's index: `
_[_…]_idx`. */ +export function foreignKeyIndexName(table: string, columns: readonly string[]): string { + return `${table}_${columns.join("_")}_idx`; +} + +/** True when `prefix` leads with exactly `columns`, in any order (the advisory's rule, too). */ +function leadsWith(prefix: readonly string[], columns: readonly string[]): boolean { + if (columns.length === 0 || prefix.length < columns.length) return false; + const want = new Set(columns); + return prefix.slice(0, columns.length).every((c) => want.has(c)); +} + +/** + * The column lists `foreignKeyIndexes` indexes on one table, in foreign-key order: each foreign + * key whose columns the primary key or a column index (a partial one counts; an expression one + * does not) does not lead with, once per column set, and none that a longer one here leads with. + * Every port's mapping generator applies this same rule. + */ +export function uncoveredForeignKeyColumns(t: Pick): string[][] { + const existing = [t.primaryKey, ...t.indexes.filter((ix) => ix.expr === undefined).map((ix) => ix.columns)]; + const candidates: string[][] = []; + for (const fk of t.foreignKeys) { + if (existing.some((cols) => leadsWith(cols, fk.columns))) continue; + if (candidates.some((c) => leadsWith(c, fk.columns) && c.length === fk.columns.length)) continue; + candidates.push([...fk.columns]); } + return candidates.filter((c) => !candidates.some((o) => o.length > c.length && leadsWith(o, c))); } diff --git a/server/typescript/packages/migrate-ts/src/expected.ts b/server/typescript/packages/migrate-ts/src/expected.ts index 076fdc427..7c446ae33 100644 --- a/server/typescript/packages/migrate-ts/src/expected.ts +++ b/server/typescript/packages/migrate-ts/src/expected.ts @@ -6,7 +6,7 @@ // not need kysely installed. Its import graph is the model walk and nothing that opens a // connection; keep it that way. -export { buildExpectedSchema, buildExpectedSchemaWithProvenance, fieldSqlType } from "./expected-schema.js"; +export { buildExpectedSchema, buildExpectedSchemaWithProvenance, fieldSqlType, foreignKeyIndexName, uncoveredForeignKeyColumns } from "./expected-schema.js"; export { pgType } from "./emit/postgres.js"; export type { BuildExpectedSchemaOptions, SchemaBuildSettings, ExpectedSchemaWithProvenance, SchemaProvenance } from "./expected-schema.js"; export { qualifiedDbName } from "./qualified-name.js"; diff --git a/server/typescript/packages/migrate-ts/src/index.ts b/server/typescript/packages/migrate-ts/src/index.ts index 5a4c573b4..797114879 100644 --- a/server/typescript/packages/migrate-ts/src/index.ts +++ b/server/typescript/packages/migrate-ts/src/index.ts @@ -8,7 +8,7 @@ // See docs/specs/2026-05-11-v0.2-sp4-migrate-ts-design.md. // Pipeline functions -export { buildExpectedSchema, buildExpectedSchemaWithProvenance } from "./expected-schema.js"; +export { buildExpectedSchema, buildExpectedSchemaWithProvenance, foreignKeyIndexName, uncoveredForeignKeyColumns } from "./expected-schema.js"; export { findReferentialActionConflict, describeReferentialActionConflict, diff --git a/server/typescript/packages/migrate-ts/src/snapshot/plan.ts b/server/typescript/packages/migrate-ts/src/snapshot/plan.ts index a96e61318..576a4caae 100644 --- a/server/typescript/packages/migrate-ts/src/snapshot/plan.ts +++ b/server/typescript/packages/migrate-ts/src/snapshot/plan.ts @@ -70,6 +70,7 @@ export async function planOffline(args: PlanOfflineArgs): Promise { test("by default: unchanged — the length check is emitted whole", async () => { const r = await root({ "@min": 0, "@max": 3 }, 3); expect(checksOf(buildExpectedSchema(r))).toEqual(['length("code") >= 0 AND length("code") <= 3']); - expect(checksOf(buildExpectedSchema(r, { skipRedundantChecks: false }))).toEqual(['length("code") >= 0 AND length("code") <= 3']); + expect(checksOf(buildExpectedSchema(r, { skipRedundantChecks: false, foreignKeyIndexes: false }))).toEqual(['length("code") >= 0 AND length("code") <= 3']); }); test("the efcore conventions alone skip nothing", async () => { diff --git a/server/typescript/packages/migrate-ts/test/unit/expected-schema-foreign-key-indexes.test.ts b/server/typescript/packages/migrate-ts/test/unit/expected-schema-foreign-key-indexes.test.ts new file mode 100644 index 000000000..6137ac65b --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/unit/expected-schema-foreign-key-indexes.test.ts @@ -0,0 +1,126 @@ +/** + * `foreignKeyIndexes` (`migrate.foreignKeyIndexes` in .metaobjects/config.json): every foreign key + * nothing already indexes gets a plain `
__idx` index. Off, the schema is exactly + * what migrate has always built; under the `efcore` preset every foreign key is already indexed, + * so it adds nothing and the preset's names stand. + */ +import { describe, test, expect } from "bun:test"; +import { MetaDataLoader, InMemoryStringSource } from "@metaobjectsdev/metadata"; +import { buildExpectedSchema, buildExpectedSchemaWithProvenance, uncoveredForeignKeyColumns } from "../../src/expected-schema.js"; +import { efcoreConventions } from "../../src/schema-conventions.js"; +import { emit } from "../../src/emit/index.js"; +import { diff } from "../../src/diff/index.js"; + +const DOC = { + "metadata.root": { + package: "acme", + children: [ + { "object.entity": { name: "Customer", children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "code" } }, + { "identity.primary": { name: "pk", "@fields": "id" } }, + { "identity.secondary": { name: "customers_code_key", "@fields": ["code"] } }, + ] } }, + { "object.entity": { name: "Order", children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "field.long": { name: "customerId", "@column": "customer_id" } }, + { "field.long": { name: "billToId", "@column": "bill_to_id" } }, + { "field.string": { name: "customerCode", "@column": "customer_code" } }, + { "identity.primary": { name: "pk", "@fields": "id" } }, + { "identity.reference": { name: "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { name: "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "identity.reference": { name: "byCode", "@fields": ["customerCode"], "@references": "Customer.code" } }, + // Already indexed by a declared lookup: no second index. + { "index.lookup": { name: "orders_bill_to", "@fields": ["billToId", "id"] } }, + ] } }, + // A junction: its primary key leads with orderId, so only tagId needs an index. + { "object.entity": { name: "OrderTag", children: [ + { "source.rdb": { "@table": "order_tags" } }, + { "field.long": { name: "orderId", "@column": "order_id" } }, + { "field.long": { name: "tagId", "@column": "tag_id" } }, + { "identity.primary": { name: "pk", "@fields": ["orderId", "tagId"] } }, + { "identity.reference": { name: "order", "@fields": ["orderId"], "@references": "Order" } }, + { "identity.reference": { name: "tag", "@fields": ["tagId"], "@references": "Tag" } }, + ] } }, + { "object.entity": { name: "Tag", children: [ + { "source.rdb": { "@table": "tags" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": "id" } }, + ] } }, + ], + }, +}; + +async function load() { + const { root, errors } = await new MetaDataLoader().load([new InMemoryStringSource(JSON.stringify(DOC))]); + expect(errors).toHaveLength(0); + return root; +} + +const indexes = (s: ReturnType, name: string) => + s.tables.find((t) => t.name === name)!.indexes.map((i) => [i.name, i.columns, i.unique]); + +describe("buildExpectedSchema with foreignKeyIndexes", () => { + test("off (and absent): no foreign key gets an index, as before", async () => { + const root = await load(); + expect(buildExpectedSchema(root, { foreignKeyIndexes: false })).toEqual(buildExpectedSchema(root)); + expect(indexes(buildExpectedSchema(root), "orders")).toEqual([["orders_bill_to", ["bill_to_id", "id"], false]]); + }); + + test("on: each foreign key nothing leads with gets
__idx", async () => { + const s = buildExpectedSchema(await load(), { foreignKeyIndexes: true }); + expect(indexes(s, "orders")).toEqual([ + ["orders_bill_to", ["bill_to_id", "id"], false], + ["orders_customer_id_idx", ["customer_id"], false], + ["orders_customer_code_idx", ["customer_code"], false], + ]); + expect(indexes(s, "order_tags")).toEqual([["order_tags_tag_id_idx", ["tag_id"], false]]); + }); + + test("the build reports which indexes it added, by table", async () => { + const built = buildExpectedSchemaWithProvenance(await load(), { foreignKeyIndexes: true }); + const added = [...(built.foreignKeyIndexes ?? [])].map(([t, ix]) => [t, ix.map((i) => i.name)]); + expect(added).toEqual([ + ["public.orders", ["orders_customer_id_idx", "orders_customer_code_idx"]], + ["public.order_tags", ["order_tags_tag_id_idx"]], + ]); + expect(buildExpectedSchemaWithProvenance(await load()).foreignKeyIndexes).toBeUndefined(); + }); + + test("under the efcore preset it adds nothing: the preset already indexes every foreign key", async () => { + const root = await load(); + const preset = buildExpectedSchema(root, { conventions: efcoreConventions }); + expect(buildExpectedSchema(root, { conventions: efcoreConventions, foreignKeyIndexes: true })).toEqual(preset); + }); + + test("the Postgres emitter creates the indexes", async () => { + const s = buildExpectedSchema(await load(), { foreignKeyIndexes: true }); + const { changes } = await diff(s, { tables: [], views: [] }, { dialect: "postgres" }); + const { up } = emit(changes, { dialect: "postgres" }); + expect(up).toContain('CREATE INDEX "orders_customer_id_idx" ON "orders" ("customer_id");'); + expect(up).toContain('CREATE INDEX "order_tags_tag_id_idx" ON "order_tags" ("tag_id");'); + expect(up).not.toContain('"orders_bill_to_id_idx"'); + }); +}); + +describe("uncoveredForeignKeyColumns", () => { + const fk = (...columns: string[]) => ({ name: "", columns, refTable: "t", refColumns: columns.map(() => "id") }); + + test("a key, or an index, that leads with the columns in any order covers them; an expression index does not", () => { + expect(uncoveredForeignKeyColumns({ + primaryKey: ["b", "a"], + indexes: [{ name: "x", columns: [], unique: false, expr: "lower(c)" }], + foreignKeys: [fk("a", "b"), fk("c")], + })).toEqual([["c"]]); + }); + + test("one index per column set, and none that a longer one leads with", () => { + expect(uncoveredForeignKeyColumns({ + primaryKey: ["id"], + indexes: [], + foreignKeys: [fk("a"), fk("a"), fk("a", "b"), fk("c")], + })).toEqual([["a", "b"], ["c"]]); + }); +}); diff --git a/server/typescript/packages/sdk/src/collection.ts b/server/typescript/packages/sdk/src/collection.ts index 75de1013f..e2ae67d15 100644 --- a/server/typescript/packages/sdk/src/collection.ts +++ b/server/typescript/packages/sdk/src/collection.ts @@ -25,7 +25,7 @@ import { type EnumDefaults, DEFAULT_ENUM_DEFAULTS, } from "@metaobjectsdev/metadata"; -import { CONFIG_FILE, enumDefaultsFromConfig, loadConfig, physicalNamingFromConfig, wireFormatFromConfig, type Config } from "./config.js"; +import { CONFIG_FILE, enumDefaultsFromConfig, loadConfig, physicalNamingFromConfig, wireFormatFromConfig, type Config, type HandEditsPolicy } from "./config.js"; import { discoverCollectionRoot, exists, isDir } from "./discovery.js"; import { compileScope, matchesScope, type Scope } from "./scope.js"; import { DEFAULT_METADATA_DIR, DEFAULT_METAOBJECTS_DIR, isMetadataFile } from "./metadata-files.js"; @@ -95,6 +95,14 @@ export interface Collection { /** `migrate.checks.skipRedundant` of the project config: leave out the validator CHECKs a * column type already enforces. False when absent. */ readonly skipRedundantChecks: boolean; + /** `migrate.foreignKeyIndexes` in `.metaobjects/config.json`: index every foreign key the + * conventions leave without one. Absent = false. Read by every expected-schema build and by + * the mapping generators. */ + readonly foreignKeyIndexes: boolean; + /** `verify.handEdits` in `.metaobjects/config.json`: whether `meta verify --codegen` FAILS on a + * hand-edited generated file (`fail`) or lists it as a notice (`notice`). Undefined when the + * project does not say, which behaves as `notice`. */ + readonly handEdits: HandEditsPolicy | undefined; /** FR-023 — the `FileSource` id each file loads under, for the dependency * ARTIFACTS only (`dep:/`); own files are absent and keep the * default `basename(path)`. Threaded to `loadMemory` so every node an artifact @@ -293,6 +301,8 @@ export async function resolveCollection( let enumDefaults: EnumDefaults = DEFAULT_ENUM_DEFAULTS; let schemaConventions: string | undefined; let skipRedundantChecks = false; + let foreignKeyIndexes = false; + let handEdits: HandEditsPolicy | undefined; if (hasConfig) { // No try/catch here: a config.json that EXISTS but fails to load @@ -313,6 +323,8 @@ export async function resolveCollection( enumDefaults = enumDefaultsFromConfig(cfg); schemaConventions = cfg.migrate?.conventions; skipRedundantChecks = cfg.migrate?.checks?.skipRedundant === true; + foreignKeyIndexes = cfg.migrate?.foreignKeyIndexes === true; + handEdits = cfg.verify?.handEdits; // An unknown token is a hard config error naming the valid ones, while // `librarySources` keeps skipping one silently for a programmatic caller. The two // are deliberately different: an API caller asking for a library this version does @@ -430,6 +442,8 @@ export async function resolveCollection( enumDefaults, schemaConventions, skipRedundantChecks, + foreignKeyIndexes, + handEdits, fileIds: new Map(dependencies.map((d) => [d.artifactPath, d.sourceId])), importedPackages: [...importedPackages].sort(), importedNodes, diff --git a/server/typescript/packages/sdk/src/config.ts b/server/typescript/packages/sdk/src/config.ts index 0139c831b..78de4d66b 100644 --- a/server/typescript/packages/sdk/src/config.ts +++ b/server/typescript/packages/sdk/src/config.ts @@ -68,6 +68,10 @@ const D1Block = z.object({ wranglerConfigPath: z.string(), }).partial().strict(); +/** `verify.handEdits`: what `meta verify --codegen` does with a hand-edited generated file. */ +export const HAND_EDITS_POLICIES = ["fail", "notice"] as const; +export type HandEditsPolicy = (typeof HAND_EDITS_POLICIES)[number]; + /** #192 — migration output-format adapters; orthogonal to dialect. */ const MigrateFormatEnum = z.enum(["default", "flyway"]); @@ -94,6 +98,13 @@ const MigrateBlock = z.object({ * out a CHECK the column type already enforces, for a database whose CHECKs were written by * hand only where the type does not; a CHECK with a declared `@constraintName` is kept. */ checks: z.object({ skipRedundant: z.boolean() }).partial().strict(), + /** Index every foreign key the conventions leave without one, named + * `
_[_…]_idx`. Separate from `conventions`, so a project can have the + * indexes without the `efcore` preset's constraint names. Absent = false: the schema migrate + * has always built, and `meta verify` names each unindexed foreign key instead. Read by + * `meta migrate` and by every generator that maps indexes (Drizzle, Prisma, EF Core, + * Exposed); `meta init` writes it on for a new project. */ + foreignKeyIndexes: z.boolean(), // .strict() for the same reason as the top level and the source arms below, and // with sharper teeth here: `{ migrate: { scopee: [...], dialect: "postgres" } }` // used to parse to `{ dialect: "postgres" }`, so a typo'd `scope` key meant @@ -207,6 +218,25 @@ export const ConfigSchema = z.object({ }) .default({}), migrate: MigrateBlock.optional(), + /** + * `meta verify` settings. `handEdits`: `fail` makes `meta verify --codegen` exit 1 when a + * generated file carries a hand edit (`--allow-hand-edits` relaxes that for one run); + * `notice` lists the file and exits 0 (`--forbid-hand-edits` tightens that for one run). + * Absent means `notice`, what verify has always done; `meta init` writes `fail` for a new + * project. Only the TypeScript `meta verify` reads it: the other ports' verify already fails + * on any edit to a generated file. + */ + verify: z + .strictObject({ + handEdits: z.enum(HAND_EDITS_POLICIES, { + message: `"verify.handEdits" must be one of ${HAND_EDITS_POLICIES.map((v) => `"${v}"`).join(", ")}`, + }).optional(), + }, { + error: (iss) => iss.code === "unrecognized_keys" + ? `"verify" has an unknown key "${iss.keys[0]}" (expected "handEdits")` + : iss.code === "invalid_type" ? `"verify" must be an object` : undefined, + }) + .optional(), /** * FR-023 — which dependencies' requirements this project takes on as its own ledger. * `inherit` lists scope patterns, and a pattern counts only when it NAMES a dependency's diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index fdd0ea82c..a613c09aa 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -16,7 +16,7 @@ // shows `resolveCollection`, the documented single authority for locating a project. // Config -export { ConfigSchema, DEFAULT_CONFIG, loadConfig, setConfigWarningSink, splitUnknownTopLevelKeys, CONFIG_TOP_LEVEL_KEYS, saveConfig, AllowTokenEnum, physicalNamingFromConfig, wireFormatFromConfig, enumDefaultsFromConfig } from "./config.js"; +export { ConfigSchema, DEFAULT_CONFIG, loadConfig, setConfigWarningSink, splitUnknownTopLevelKeys, CONFIG_TOP_LEVEL_KEYS, saveConfig, AllowTokenEnum, physicalNamingFromConfig, wireFormatFromConfig, enumDefaultsFromConfig, HAND_EDITS_POLICIES, type HandEditsPolicy } from "./config.js"; export type { Config } from "./config.js"; // Metadata dependencies (FR-023) — the `dependencies` config key's schema, diff --git a/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts b/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts index f943b82ef..e35ed5146 100644 --- a/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts +++ b/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "@metaobjectsdev/codegen-ts/engine"; import { flattenedMembersOf } from "@metaobjectsdev/codegen-ts/engine"; import { effectivePackage } from "@metaobjectsdev/codegen-ts/engine"; +import { foreignKeyIndexesOf } from "@metaobjectsdev/codegen-ts/engine"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) {