From 26a841f9044fe99d7516b4ce571fa7fb62161877 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 13 Nov 2026 11:02:32 -0500 Subject: [PATCH 1/4] feat: fail verify on hand edits and index foreign keys on request (1.3.0 PRs 6-7) verify.handEdits in .metaobjects/config.json: "fail" makes `meta verify --codegen` exit 1 on a hand-edited generated file, naming the file and both remedies; --allow-hand-edits relaxes it for one run, --forbid-hand-edits still forces it. A project without the key keeps the notice and exit 0; `meta init` writes "fail". The C#, Java, Kotlin and Python gates already byte-compare and fail; the docs now say so. migrate.foreignKeyIndexes: every foreign key nothing already leads with gets a plain __idx index, separate from the efcore preset (which indexes them already and so adds nothing). Honoured by meta migrate / verify --db, the Drizzle reference template, the C# DbContext generator and the Kotlin Exposed table generator; Java OMDB and Python map no indexes. `meta init` writes it on; projects without it are unchanged and keep the advisory, which now names the option. Persistence corpus gains two migration scenarios. C# gen and both verify --codegen paths pass foreignKeyIndexes and conventions. --- CHANGELOG.md | 25 ++++ .../skills/metaobjects-verify/SKILL.md | 24 ++-- docs/CONFORMANCE.md | 8 +- docs/features/cli.md | 2 +- docs/features/cube-export.md | 5 +- docs/features/migrations-and-drift.md | 47 ++++++- docs/recipes/prove-your-gate-fails.md | 13 +- .../codegen/templates/drizzle-schema.ts | 18 +++ .../codegen/templates/drizzle-schema.ts | 18 +++ .../skills/metaobjects-verify/SKILL.md | 24 ++-- .../skills/metaobjects-verify/SKILL.md | 24 ++-- .../skills/metaobjects-verify/SKILL.md | 24 ++-- .../skills/metaobjects-verify/SKILL.md | 24 ++-- .../skills/metaobjects-verify/SKILL.md | 24 ++-- .../registry.json | 1 + fixtures/persistence-conformance/README.md | 8 ++ .../migrations/foreign-key-indexes-added.yaml | 36 +++++ .../noop-converged-foreign-key-indexes.yaml | 17 +++ .../csharp/MetaObjects.Cli/VerifyCommand.cs | 3 + .../ForeignKeyIndexTests.cs | 115 ++++++++++++++++ .../csharp/MetaObjects.Codegen/CodegenCli.cs | 4 + .../csharp/MetaObjects.Codegen/Generator.cs | 16 +++ .../Generators/DbContextGenerator.cs | 92 +++++++++++++ .../EfModelGapsTests.cs | 42 ++++++ .../MetaObjects/Config/NeutralConfig.cs | 23 +++- .../kotlin/KotlinExposedTableGenerator.kt | 64 ++++++++- .../KotlinExposedForeignKeyIndexTest.kt | 88 ++++++++++++ .../mojo/AbstractMetaDataMojo.java | 5 + .../main/java/com/metaobjects/MetaRoot.java | 15 +++ .../metaobjects/config/MigrateSettings.java | 61 +++++++++ .../com/metaobjects/config/NeutralConfig.java | 24 ++++ .../metaobjects/config/SourceResolver.java | 5 + .../metaobjects/loader/MetaDataLoader.java | 5 + .../metaobjects/config/NeutralConfigTest.java | 22 +++ .../packages/cli/src/commands/init.ts | 23 +++- .../packages/cli/src/commands/verify.ts | 85 +++++++++--- server/typescript/packages/cli/src/index.ts | 14 +- .../typescript/packages/cli/src/lib/args.ts | 25 +++- .../packages/cli/src/lib/fk-index-advisory.ts | 17 ++- .../cli/src/lib/schema-build-options.ts | 7 +- .../typescript/packages/cli/test/init.test.ts | 16 +++ .../verify-codegen-hand-edits.test.ts | 104 +++++++++++++++ .../verify-gate-not-vacuous.test.ts | 14 +- .../test/migrate-schema-conventions.test.ts | 27 +++- .../cli/test/unit/args-verify.test.ts | 17 ++- .../cli/test/unit/gen-migrate-advice.test.ts | 2 +- .../cli/test/verify-fk-index-lint.test.ts | 20 ++- .../packages/codegen-ts/src/engine-surface.ts | 1 + .../packages/codegen-ts/src/engine.ts | 1 + .../codegen-ts/src/foreign-key-indexes.ts | 61 +++++++++ .../packages/codegen-ts/src/render-context.ts | 7 + .../packages/codegen-ts/src/runner.ts | 1 + .../src/templates/drizzle-schema.ts | 18 +++ .../test/drizzle-foreign-key-indexes.test.ts | 70 ++++++++++ .../test/prisma/prisma-schema.test.ts | 8 ++ .../src/migration-scenario.ts | 8 +- .../integration-tests/src/scenario.ts | 24 ++++ .../packages/migrate-ts/src/drift/drift.ts | 1 + .../migrate-ts/src/expected-schema.ts | 81 +++++++++-- .../packages/migrate-ts/src/expected.ts | 2 +- .../packages/migrate-ts/src/index.ts | 2 +- .../packages/migrate-ts/src/snapshot/plan.ts | 2 + .../test/unit/check-redundant.test.ts | 2 +- ...xpected-schema-foreign-key-indexes.test.ts | 126 ++++++++++++++++++ .../typescript/packages/sdk/src/collection.ts | 16 ++- server/typescript/packages/sdk/src/config.ts | 30 +++++ server/typescript/packages/sdk/src/index.ts | 2 +- .../src/templates/drizzle-schema.ts | 18 +++ 68 files changed, 1629 insertions(+), 149 deletions(-) create mode 100644 fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml create mode 100644 fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml create mode 100644 server/csharp/MetaObjects.Codegen.Tests/ForeignKeyIndexTests.cs create mode 100644 server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedForeignKeyIndexTest.kt create mode 100644 server/java/metadata/src/main/java/com/metaobjects/config/MigrateSettings.java create mode 100644 server/typescript/packages/codegen-ts/src/foreign-key-indexes.ts create mode 100644 server/typescript/packages/codegen-ts/test/drizzle-foreign-key-indexes.test.ts create mode 100644 server/typescript/packages/migrate-ts/test/unit/expected-schema-foreign-key-indexes.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f3a4d4e40b..77d0ffe865 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,27 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi one over measures as a `number` member (one reading a dimension is refused). Loader rule M11, report rule R13 and a conformance fixture per refusal, in every loader port. +- **`verify.handEdits`: a hand-edited generated file can fail `meta verify`.** `meta verify + --codegen` already listed each generated file carrying a hand edit, but always exited 0, so a + rule changed by hand in generated code (a `max(32)` made `max(64)`) passed the gate. Set + `"verify": { "handEdits": "fail" }` in `.metaobjects/config.json` and it exits 1; `"notice"` + keeps the report and exit 0. `--allow-hand-edits` relaxes `fail` for one run, and the existing + `--forbid-hand-edits` still forces a failure for one run (passing both is refused). The report + names each file, how to see the edit, how to drop it (or move a changed rule into the + metadata), and how to keep it. A project without the key behaves exactly as before. The C#, + Java, Kotlin and Python gates compare generated files byte for byte and already fail on any + edit; the docs now say so. +- **`migrate.foreignKeyIndexes`: an index on every foreign key, without the `efcore` renames.** + With it on, `meta migrate` gives each enforced foreign key that the primary key, a unique key or + an index does not already lead with a `
__idx` index. The Drizzle and Prisma + generators, the C# `AppDbContext` and the Kotlin Exposed tables declare the same indexes under + the same names; Java and Python map no indexes. It adds nothing under the `efcore` conventions + (which already index every foreign key) or on MySQL. A project without the key is unchanged and + keeps the unindexed-foreign-key advisory, which now names the option. Gated by two new + persistence-corpus migration scenarios (adding the indexes to an existing schema, and the + converged no-op). Java's config reader now carries the `migrate` block's + `foreignKeyIndexes`, `conventions` and `dialect` on the loaded root (`MigrateSettings`). + ### Changed - **Breaking, TypeScript: a value object is never renamed; one claim registry refuses real @@ -56,6 +77,10 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi a barrel) now fail with the collision error; set `outputLayout: "package"` (and drop the barrel, or rename one), and update imports of the old `AcmeAlphaNote` names. An owned generator copy is refreshed with `meta eject --list`. +- **`meta init` writes stricter defaults for a new project.** The scaffolded + `.metaobjects/config.json` now carries `"verify": { "handEdits": "fail" }` and (except on + MySQL) `"migrate": { "foreignKeyIndexes": true }`, each as its own visible setting. An existing + config is left as it is. - **Agent context routes report work to the reporting vocabulary.** The `meta init` agent guide and the `metaobjects-authoring` skill now send grouped totals, dashboard numbers and read-only report views to `object.report` (`references/declaring-reports.md`) before the `@sql` escape, diff --git a/agent-context/skills/metaobjects-verify/SKILL.md b/agent-context/skills/metaobjects-verify/SKILL.md index fcdb550249..14b1f94cca 100644 --- a/agent-context/skills/metaobjects-verify/SKILL.md +++ b/agent-context/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index da030620d9..bf3f4b2029 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -35,7 +35,7 @@ regenerate with `ls -d fixtures//*/ | wc -l` for directory-shaped corpor | [`fixtures/jinja-verify-conformance/`](../fixtures/jinja-verify-conformance/) (ADR-0061 — the payload check over a Jinja body) | 31 | ✓ | refuses Jinja | refuses, via Java | refuses Jinja | ✓ | | [`fixtures/extract-conformance/`](../fixtures/extract-conformance/) | 58 | ✓ | ✓ | inherits the shared JVM engine | ✓ | ✓ | | [`fixtures/output-prompt-conformance/`](../fixtures/output-prompt-conformance/) | 17 | ✓ | ✓ | ✓ | ✓ | ✓ | -| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 62 (55 query + 7 migration) | all 62; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 55 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) | +| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 64 (55 query + 9 migration) | all 64; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 55 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) | | [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 123 (31 core + 10 tph + 9 m2m + 5 jsonb + 2 write-through + 2 secret-fields + 2 enum-default + 12 projection + 37 report + 11 decimal + 2 duration) | ✓ (Fastify reference + generated lane); `report/`, `decimal/` and `duration/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/`, `decimal/` and `duration/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (Kestrel + Npgsql); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | | [`fixtures/decimal-wire/`](../fixtures/decimal-wire/) | 1 vector file (format, parse, filter operand, 15-digit read) | ✓ (`runtime-ts` and the generated helper) | ✓ (`CanonicalDecimal`) | ✓ (the generated wire file) | ✓ (`DecimalWire`) | ✓ (`decimal_wire`) | | [`fixtures/duration-wire/`](../fixtures/duration-wire/) | 1 vector file: the `wire` vectors (parse, precision, refused) every port passes, and the `codecs` groups (Postgres text read and write, stored microseconds) each port passes only for the codecs it has | ✓ (`metadata` / `runtime-ts` and the generated helper) | ✓ | ✓ | ✓ | ✓ | @@ -423,9 +423,9 @@ All 37 fixtures → [features/migrations-and-drift.md](features/migrations-and-d may carry `"isArray": true`; with `fields` it is a list of objects, without it a list of scalars. The six `verify-positional-accessor-*` cases pin where `-last` is available. -### `fixtures/persistence-conformance/` (62 — 55 query + 7 migration) +### `fixtures/persistence-conformance/` (64 — 55 query + 9 migration) -- `migrations/*` (7) → [features/migrations-and-drift.md](features/migrations-and-drift.md) (schema migration section) +- `migrations/*` (9) → [features/migrations-and-drift.md](features/migrations-and-drift.md) (schema migration section) - `queries/*` (55) → [features/source-kinds.md](features/source-kinds.md) (query semantics against `source.rdb`) **Engines beyond Postgres (FR-044 reports, decimals).** The twenty-four `queries/report-*.yaml` @@ -725,7 +725,7 @@ JSON and verify payload feeds it unchanged. ## Orphaned fixtures (tested but not yet documented) The fixtures in the nine corpora mapped above (metamodel 510 + yaml 16 + verify 45 -+ render 18 + persistence 62 + api-contract 123 + source-resolution 25 + scope 10 + ++ render 18 + persistence 64 + api-contract 123 + source-resolution 25 + scope 10 + dependency 47) each map to a feature doc, and so do the two requirement corpora, `shared-model-conformance`, the `cube-model` corpus and `explorer-conformance`, whose case lists live in their own READMEs. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider diff --git a/docs/features/cli.md b/docs/features/cli.md index a2c84a40ee..09d458ca55 100644 --- a/docs/features/cli.md +++ b/docs/features/cli.md @@ -53,7 +53,7 @@ vocabulary everywhere, each port implementing the modes it supports. | Subverb | What it checks | Touches a DB? | |---|---|---| | `verify --db` | **Schema drift** — does the live database (or snapshot) match the metadata? (migrate engine, ADR-0015) | yes | -| `verify --codegen` | **Codegen drift** — regenerate from metadata into a temp dir and fail if it differs from the committed generated output. Catches "metadata changed but `meta gen` wasn't re-run". A hand-edited generated file is NOT drift — `meta gen` three-way-merges hand edits by design, so the gate compares the *generated contribution* against the committed `.gen-state/.hashes.json` rather than the file byte-for-byte. It still LISTS every generated file carrying a hand edit (count + paths, and how to see the edit) as a notice that does not change the exit code; `--forbid-hand-edits` makes any hand edit fail the build, for teams that want generated code untouchable (Node `meta` only). | no | +| `verify --codegen` | **Codegen drift** — regenerate from metadata into a temp dir and fail if it differs from the committed generated output. Catches "metadata changed but `meta gen` wasn't re-run". **A hand-edited generated file** (Node `meta`): `meta gen` three-way-merges hand edits, so the gate compares the *generated contribution* against the committed `.gen-state/.hashes.json` and reports each edited file separately, by path, with how to see the edit and how to drop it. What that report does to the exit code is the `verify.handEdits` key in `.metaobjects/config.json`: `"fail"` exits 1, `"notice"` (and a project without the key, as before 1.3) prints it and exits 0. `meta init` writes `"fail"` for a new project. `--allow-hand-edits` relaxes `fail` to a notice for one run; `--forbid-hand-edits` makes a hand edit fail for one run whatever the key says. The other ports compare the generated files byte for byte, so any edit is drift and fails: `dotnet meta verify --codegen`, `mvn metaobjects:verify` (Java and Kotlin) and `metaobjects verify --codegen` each exit non-zero and name the file. | no | | `verify --templates` | **Template/prompt drift** — `Renderer.verify` checks each `template.*` node's `{{field}}` references against its payload VO (FR-004). A Jinja body (Python and TypeScript, ADR-0061) is checked the same way. | no | | `verify --docs` | **Removed in 1.1.0** (ADR-0034 Amendment 9). It prints where the check went and exits 2: documentation is written by entries in `generators`, so `verify --codegen` checks it with the rest of the declared output. | — | | `verify --deps` | **Dependency drift** (FR-023) — re-resolves each declared dependency (its `path`, its installed `npm` / `python` package, or its `maven` / `nuget` package in the local cache) and compares its installed artifact's hash against `.metaobjects/deps.lock.json`, same comparison as `meta deps check`. Needs the publisher reachable, so it is never part of the bare-`verify` default. A `drifted` or `unresolved` dependency fails with `ERR_DEPENDENCY_UPSTREAM_DRIFT`. **Node `meta`; Python (`metaobjects verify --deps`, plus `metaobjects deps check`), C# (`dotnet meta verify --deps`), and Java and Kotlin (`-Dmeta.verify.mode=deps`) have the `path`-only counterpart.** See [`metadata-dependencies.md`](metadata-dependencies.md). | no | diff --git a/docs/features/cube-export.md b/docs/features/cube-export.md index be9542dcb8..1ed2c54e98 100644 --- a/docs/features/cube-export.md +++ b/docs/features/cube-export.md @@ -582,8 +582,9 @@ there. **`meta verify --codegen`** regenerates the configured output into a temporary directory and compares it with the committed files. The Cube project is a target, so its files are checked like any other generated output: a changed model that was not regenerated, a missing cube file -and a stale one are drift. A hand edit to a generated file is not drift, but the gate lists it, and -`meta verify --codegen --forbid-hand-edits` makes it fail. +and a stale one are drift. A hand edit to a generated file is reported by path, and fails the gate +when `.metaobjects/config.json` sets `"verify": { "handEdits": "fail" }` (what `meta init` writes) +or the run passes `--forbid-hand-edits`. **The mapping corpus** is [`fixtures/cube-model/`](../../fixtures/cube-model/): 46 cases, each the smallest model for one rule, 36 with the exact tree the generator writes and 10 with the diff --git a/docs/features/migrations-and-drift.md b/docs/features/migrations-and-drift.md index 552010f8c8..14963b2267 100644 --- a/docs/features/migrations-and-drift.md +++ b/docs/features/migrations-and-drift.md @@ -15,7 +15,7 @@ There are **7 drift sources**, and the toolchain has a guard for each. | **Code-vs-API-doc** | Cross-port codegen from the same metadata. | Build time | | **DB-vs-metadata** | `meta verify --db` (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection **view bodies** (a changed `CREATE VIEW` is drift, reported with what differs: the first differing excerpt of the text on SQLite/D1, the deparsed body on Postgres's scratch-database path, a fingerprint mismatch otherwise). A view `meta migrate` drops and recreates only because the migration alters a table it reads, with its definition unchanged, is not drift and is not reported; a hand-authored *unmodeled* view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the `metaobjects:verify` Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through `--db`'s Kysely-driver introspection — use `meta verify --dialect d1 [--d1 ] [--remote]` instead (the same wrangler-shelled-out path `meta migrate --dialect d1` uses); `--remote` is required to check the *deployed* database, not the local `wrangler dev` shadow copy. Pointing `--db file:` at wrangler's local D1 state directory (`.wrangler/state/**/d1/**`) still runs, but only verifies that local copy — `verify` warns when it detects this. | CI on every PR | | **Migration-vs-metadata** | The Node `meta migrate` emits migrations FROM metadata diffs — they cannot drift from metadata by construction. Schema migrations for **every** port are owned by this Node toolchain (`@metaobjectsdev/cli migrate`, ADR-0015); the C# and Python migrate surfaces were removed. | Build time | -| **Generated-edited** | `@generated` headers in emitted code + a committed hash manifest that refuses an edited file by name; in TypeScript only, a three-way merge also preserves hand-edits on the machine that generated the file. | Code review | +| **Generated-edited** | `@generated` headers in emitted code + a committed hash manifest that refuses an edited file by name; in TypeScript only, a three-way merge also preserves hand-edits on the machine that generated the file. `verify --codegen` names each edited file in every port. In TypeScript it fails on one when `.metaobjects/config.json` sets `"verify": { "handEdits": "fail" }` (what `meta init` writes) and reports it as a notice otherwise; the C#, Java, Kotlin and Python gates compare bytes, so an edit always fails there ([cli.md](cli.md)). | Build time (CI) | | **Prompt-vs-payload** | FR-004 `Renderer.verify` parses `{{...}}` references in templates and checks each one exists on the payload VO. | Build time + runtime | | **Generated-vs-runtime** | Kotlin / Java `MetadataStartupValidator` (from Spring `ApplicationReadyEvent`) re-loads metadata at startup and asserts the generated table objects match. | App startup | @@ -546,6 +546,49 @@ onto that key is dropped first and added back after, so the plan is gated behind drop-index` and `--allow drop-fk`. `meta verify --db` reports each of these as drift. SQLite cannot rename a constraint and rebuilds the table instead. +## Foreign-key indexes: `migrate.foreignKeyIndexes` + +Postgres and SQLite index the referenced side of a foreign key and not the referencing side, so +without an index every join from the parent, and every `ON DELETE` check it runs, scans the child +table. `migrate.foreignKeyIndexes` asks `meta migrate` to index each foreign key, with no other +change to the schema's names: + +```json +{ "schema_version": 1, "migrate": { "foreignKeyIndexes": true } } +``` + +With it on, each enforced foreign key that nothing already leads with gets a plain index named +`
_[_…]_idx`. The primary key, a `@unique` field, an `identity.secondary` +and an `index.lookup` without `@expr` (a partial one included) cover a foreign key whose columns +they start with, in any order; a junction's first foreign key is usually covered by its composite +primary key. Two foreign keys over the same columns get one index, and a foreign key that leads a +longer one shares the longer one's index. A name that collides with another index is refused with +`ERR_DUPLICATE_SQL_NAME`. + +It is separate from the `efcore` conventions, which index every foreign key too but also rename +the primary-key and foreign-key constraints: a team can have the indexes without the renames. +Under `efcore` the option adds nothing, since the preset's `IX_` indexes already cover every +foreign key. On MySQL it adds nothing either: MetaObjects emits no DDL there, and InnoDB indexes +each foreign key itself. + +`meta init` writes it on for a new project. A project without the key is unchanged: migrate adds +no foreign-key index, and `meta verify` keeps its advisory naming each unindexed foreign key +(which now also names the option). Turning it on for an existing database is an additive +migration (`CREATE INDEX` per key), with nothing blocked. + +The generators that map indexes declare the same indexes under the same names, from the same +setting, so the mapping and the database agree: + +| Generator | With the option on | +|---|---| +| Drizzle (`entityFile()`, TypeScript) | `index("").on(table.)` in the table's index callback | +| Prisma (`prismaSchemaFile()`, TypeScript) | `@@index([...], map: "")`, read from the same expected schema | +| EF Core `AppDbContext` (C#) | `HasIndex(x => x., "").HasDatabaseName("")` | +| Exposed table (Kotlin) | `index("", false, )` in the table's `init {}` | + +Java (OMDB is data access only and maps no indexes) and Python (Pydantic models and routes, no +ORM mapping) generate nothing that declares an index, so the option changes no output there. + ## How CHECK constraints and expression indexes are compared Postgres rewrites an expression before it stores it (`"k" IN ('x')` becomes `k = 'x'::text`, @@ -845,6 +888,8 @@ The following conformance fixtures gate this feature's behavior across ports: - [`bootstrap-canonical-from-empty.yaml`](../../fixtures/persistence-conformance/migrations/bootstrap-canonical-from-empty.yaml) — full-CREATE bootstrap from an empty database - [`add-nullable-column.yaml`](../../fixtures/persistence-conformance/migrations/add-nullable-column.yaml) — incremental `ALTER TABLE … ADD COLUMN` for a new nullable field - [`drop-table-blocked-without-allow.yaml`](../../fixtures/persistence-conformance/migrations/drop-table-blocked-without-allow.yaml) — destructive operations require an explicit allow-flag +- [`foreign-key-indexes-added.yaml`](../../fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml) — `migrate.foreignKeyIndexes` turned on for an existing schema adds one `
__idx` per uncovered foreign key and nothing else +- [`noop-converged-foreign-key-indexes.yaml`](../../fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml) — with the option on, a converged schema emits zero SQL (the indexes introspect back exactly as emitted) - [`noop-converged-canonical.yaml`](../../fixtures/persistence-conformance/migrations/noop-converged-canonical.yaml) — **the idempotence gate**: apply the whole canonical schema, then diff the same metadata against the database it just produced. A converged schema must emit **zero** SQL. This is what catches any asymmetry between what `emit` writes, what `introspect` reads back, and what the expected schema models — the class of bug that makes `meta migrate` re-propose (and on SQLite, destructively rebuild) unchanged tables forever. **Template drift (`fixtures/verify-conformance/`)** — the `Renderer.verify` engine diff --git a/docs/recipes/prove-your-gate-fails.md b/docs/recipes/prove-your-gate-fails.md index 68b1706bf9..ffbf68a4b0 100644 --- a/docs/recipes/prove-your-gate-fails.md +++ b/docs/recipes/prove-your-gate-fails.md @@ -15,14 +15,17 @@ It works in a throwaway project and asserts, in this order: | 1 | Clean tree | **0** | | 2 | A field added to the metadata, `meta gen` not re-run | **1** (codegen gate) | | 3 | Metadata restored | **0** (the failure was the break, not the harness) | -| 4 | A hand edit inside a generated file | **0** by default, **1** with `meta verify --codegen --forbid-hand-edits` | +| 4 | A hand edit inside a generated file | **1** (the project `meta init` wrote sets `verify.handEdits: "fail"`); **0** with `meta verify --codegen --allow-hand-edits` | | 5 | A generated file deleted | **1** | | 6 | `meta gen` re-run | **0** | -Step 4 is deliberate: `meta gen` three-way-merges hand edits, so a hand-edited generated file -is a notice, not drift. If your policy is "generated code is never touched", gate on -`--forbid-hand-edits`. A second test in the same file pins why step 0 exists: a project with no -generators passes `meta verify` with exit 0 having checked nothing. +Step 4 follows the project's `verify.handEdits` setting. `meta gen` three-way-merges hand edits, so +a hand-edited generated file is reported apart from drift, by path; a project with +`"verify": { "handEdits": "fail" }` in `.metaobjects/config.json` (what `meta init` writes) fails +on it, and one without the key (a project from before 1.3) gets the report as a notice and exit 0. +On such a project, set the key, or gate on `meta verify --codegen --forbid-hand-edits`. A second +test in the same file pins why step 0 exists: a project with no generators passes `meta verify` +with exit 0 having checked nothing. ## The same check in a shell, for your own CI diff --git a/examples/advanced-modeling/codegen/templates/drizzle-schema.ts b/examples/advanced-modeling/codegen/templates/drizzle-schema.ts index f943b82eff..e35ed5146a 100644 --- a/examples/advanced-modeling/codegen/templates/drizzle-schema.ts +++ b/examples/advanced-modeling/codegen/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "@metaobjectsdev/codegen-ts/engine"; import { flattenedMembersOf } from "@metaobjectsdev/codegen-ts/engine"; import { effectivePackage } from "@metaobjectsdev/codegen-ts/engine"; +import { foreignKeyIndexesOf } from "@metaobjectsdev/codegen-ts/engine"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) { diff --git a/examples/showcase/codegen/templates/drizzle-schema.ts b/examples/showcase/codegen/templates/drizzle-schema.ts index f943b82eff..e35ed5146a 100644 --- a/examples/showcase/codegen/templates/drizzle-schema.ts +++ b/examples/showcase/codegen/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "@metaobjectsdev/codegen-ts/engine"; import { flattenedMembersOf } from "@metaobjectsdev/codegen-ts/engine"; import { effectivePackage } from "@metaobjectsdev/codegen-ts/engine"; +import { foreignKeyIndexesOf } from "@metaobjectsdev/codegen-ts/engine"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) { diff --git a/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb550249..14b1f94cca 100644 --- a/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/csharp/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb550249..14b1f94cca 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb550249..14b1f94cca 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb550249..14b1f94cca 100644 --- a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md index fcdb550249..14b1f94cca 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-verify/SKILL.md @@ -150,16 +150,20 @@ with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. - **`--codegen`** — regeneration drift. Re-runs generation and diffs the result against the committed generated files; a stale generated part (someone skipped a regen) fails the build. Wire it into CI so a stale `@generated` file fails. - **A hand edit inside a generated file is NOT drift** — `meta gen` three-way-merges - hand edits by design, and the Node gate judges the *generated contribution* against - the committed `.metaobjects/.gen-state/.hashes.json`, not the bytes. So a weakened - validator passes this gate. The Node `meta verify --codegen` therefore LISTS every - generated file carrying a hand edit (count + paths, a notice that does not change - the exit code), and `--forbid-hand-edits` turns any hand edit into a failure for a - team that wants generated code untouchable. Read that list at review time. To see - one edit: delete the file, run `meta gen` (it re-emits the pristine generation), - then `git diff -R -- `; `git checkout -- ` restores it. The one - exception is an `.names.ts` artifact: a hand edit there always fails. + **A hand edit inside a generated file is reported apart from drift** — `meta gen` + three-way-merges hand edits by design, and the Node gate judges the *generated + contribution* against the committed `.metaobjects/.gen-state/.hashes.json`, not the + bytes, then lists every generated file carrying a hand edit (count + paths). Whether + that list fails the build is `verify.handEdits` in `.metaobjects/config.json`: + `"fail"` exits 1 (what `meta init` writes for a new project), `"notice"` (and a + project without the key) exits 0, so a weakened validator passes there. Check the + key; `--allow-hand-edits` relaxes `fail` for one run, `--forbid-hand-edits` forces it + for one run. If an edit changed a rule the model declares (a length, a range, a + required field), make the change in the metadata instead, so the database and every + port follow it. To see one edit: delete the file, run `meta gen` (it re-emits the + pristine generation), then `git diff -R -- `; `git checkout -- ` + restores it. An `.names.ts` artifact always fails on a hand edit. The C#, + Java, Kotlin and Python gates compare bytes, so any edit fails there. - **`--templates`** — prompt/payload drift. For every `template.prompt` / `template.output`, resolves the text, parses each `{{...}}` reference, and fails diff --git a/fixtures/generator-registry-conformance/registry.json b/fixtures/generator-registry-conformance/registry.json index f01caa68ef..2f1d47b071 100644 --- a/fixtures/generator-registry-conformance/registry.json +++ b/fixtures/generator-registry-conformance/registry.json @@ -364,6 +364,7 @@ "codegen-ts/src/enum-meta.ts", "codegen-ts/src/enum-shared.ts", "codegen-ts/src/flattened.ts", + "codegen-ts/src/foreign-key-indexes.ts", "codegen-ts/src/format.ts", "codegen-ts/src/generator.ts", "codegen-ts/src/generators/agent-schema-input.ts", diff --git a/fixtures/persistence-conformance/README.md b/fixtures/persistence-conformance/README.md index 9784098268..724734bac9 100644 --- a/fixtures/persistence-conformance/README.md +++ b/fixtures/persistence-conformance/README.md @@ -118,6 +118,14 @@ seed-data: | # Target schema — what we want the DB to look like. target-metadata: ./states/program-v2/ +# Optional: the project schema settings (`migrate.*` in .metaobjects/config.json) the target, +# and separately the seed, are built with. Only `foreignKeyIndexes` is applied; any other key +# is refused, so a typo cannot silently run the default. +# schema-settings: +# foreignKeyIndexes: true +# seed-schema-settings: +# foreignKeyIndexes: true + # Assertions on the generated up migration (any of these may be omitted). expect: blocked: [] # no blocked changes diff --git a/fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml b/fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml new file mode 100644 index 0000000000..5c49408cdc --- /dev/null +++ b/fixtures/persistence-conformance/migrations/foreign-key-indexes-added.yaml @@ -0,0 +1,36 @@ +name: foreign-key-indexes-added +description: | + `migrate.foreignKeyIndexes` turned on for a project whose schema has none. The canonical + schema is applied as it always is (no foreign key has an index), then migrated to the SAME + metadata with the option on: the up migration only adds a `
__idx` index for + each foreign key nothing already leads with. A junction's foreign key that leads its composite + primary key (post_tags.postId, follows.followerId, ...) is covered by the key and gets none. + No table, column or constraint changes, and nothing is blocked. + +seed-metadata: ../canonical/ +target-metadata: ../canonical/ +schema-settings: + foreignKeyIndexes: true + +expect: + blocked: [] + up-contains: + - 'CREATE INDEX "weeks_programId_idx" ON "weeks" ("programId");' + - 'CREATE INDEX "nodes_parentId_idx" ON "nodes" ("parentId");' + - 'CREATE INDEX "sessions_weekId_idx" ON "sessions" ("weekId");' + - 'CREATE INDEX "post_tags_tagId_idx" ON "post_tags" ("tagId");' + - 'CREATE INDEX "post_referrals_referralId_idx" ON "post_referrals" ("referralId");' + apply-up-then-query: + sql: | + SELECT tablename, indexname FROM pg_indexes + WHERE schemaname = 'public' AND indexname LIKE '%\_idx' + ORDER BY indexname + rows: + - { tablename: "follows", indexname: "follows_followeeId_idx" } + - { tablename: "friendships", indexname: "friendships_personBId_idx" } + - { tablename: "gadget_events", indexname: "gadget_events_gadgetId_idx" } + - { tablename: "nodes", indexname: "nodes_parentId_idx" } + - { tablename: "post_referrals", indexname: "post_referrals_referralId_idx" } + - { tablename: "post_tags", indexname: "post_tags_tagId_idx" } + - { tablename: "sessions", indexname: "sessions_weekId_idx" } + - { tablename: "weeks", indexname: "weeks_programId_idx" } diff --git a/fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml b/fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml new file mode 100644 index 0000000000..6ad2a2c0b5 --- /dev/null +++ b/fixtures/persistence-conformance/migrations/noop-converged-foreign-key-indexes.yaml @@ -0,0 +1,17 @@ +name: noop-converged-foreign-key-indexes +description: | + IDEMPOTENCE GATE for `migrate.foreignKeyIndexes`: the canonical schema built WITH the option, + then diffed against the same metadata with the option still on, must produce zero SQL. Catches + a foreign-key index whose emitted form introspection reads back differently (a name, a column + list), which would otherwise re-propose the index on every migrate. + +seed-metadata: ../canonical/ +seed-schema-settings: + foreignKeyIndexes: true +target-metadata: ../canonical/ +schema-settings: + foreignKeyIndexes: true + +expect: + blocked: [] + up-empty: true diff --git a/server/csharp/MetaObjects.Cli/VerifyCommand.cs b/server/csharp/MetaObjects.Cli/VerifyCommand.cs index a647029848..a12afa7af0 100644 --- a/server/csharp/MetaObjects.Cli/VerifyCommand.cs +++ b/server/csharp/MetaObjects.Cli/VerifyCommand.cs @@ -457,6 +457,9 @@ private static Codegen.CodegenDrift.Result RunCodegenDrift(Options opts, out boo // services read as drift. Dialect = neutralConfig?.MigrateDialect, SkipRedundantChecks = neutralConfig?.MigrateSkipRedundantChecks ?? false, + // The foreign-key indexes gen declared, so the regen indexes what gen indexed. + ForeignKeyIndexes = neutralConfig?.MigrateForeignKeyIndexes ?? false, + SchemaConventions = neutralConfig?.MigrateConventions, }.WithCSharpLayout(neutralConfig); return Codegen.CodegenDrift.Compute(config, load.Root, generators); } diff --git a/server/csharp/MetaObjects.Codegen.Tests/ForeignKeyIndexTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ForeignKeyIndexTests.cs new file mode 100644 index 0000000000..fbce6aac90 --- /dev/null +++ b/server/csharp/MetaObjects.Codegen.Tests/ForeignKeyIndexTests.cs @@ -0,0 +1,115 @@ +// `migrate.foreignKeyIndexes` (.metaobjects/config.json; GenConfig.ForeignKeyIndexes): `meta migrate` +// gives every enforced foreign key that nothing already leads with a `
__idx` +// index, so the EF model declares the same index under the same name. The model is the one +// migrate-ts's expected-schema-foreign-key-indexes test builds, so the two sides are held to one +// answer. + +using MetaObjects.Codegen.Generators; +using MetaObjects.Config; +using MetaObjects.Loader; +using Xunit; + +namespace MetaObjects.Codegen.Tests; + +public sealed class ForeignKeyIndexTests +{ + private const string Model = """ + { "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Customer", "children": [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.secondary": { "name": "customers_code_key", "@fields": ["code"] } } + ] } }, + { "object.entity": { "name": "Order", "children": [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { "name": "id" } }, + { "field.long": { "name": "customerId", "@column": "customer_id" } }, + { "field.long": { "name": "billToId", "@column": "bill_to_id" } }, + { "field.string": { "name": "customerCode", "@column": "customer_code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.reference": { "name": "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { "name": "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "identity.reference": { "name": "byCode", "@fields": ["customerCode"], "@references": "Customer.code" } }, + { "index.lookup": { "name": "orders_bill_to", "@fields": ["billToId", "id"] } } + ] } }, + { "object.entity": { "name": "OrderTag", "children": [ + { "source.rdb": { "@table": "order_tags" } }, + { "field.long": { "name": "orderId", "@column": "order_id" } }, + { "field.long": { "name": "tagId", "@column": "tag_id" } }, + { "identity.primary": { "name": "pk", "@fields": ["orderId", "tagId"] } }, + { "identity.reference": { "name": "order", "@fields": ["orderId"], "@references": "Order" } }, + { "identity.reference": { "name": "tag", "@fields": ["tagId"], "@references": "Tag" } } + ] } }, + { "object.entity": { "name": "Tag", "children": [ + { "source.rdb": { "@table": "tags" } }, + { "field.long": { "name": "id" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } } + ] } } + ] } } + """; + + private static string Context(bool on, string? conventions = null, string? dialect = null) + { + var load = new MetaDataLoader().Load([new InMemoryStringSource(Model, id: "m.json")]); + Assert.Empty(load.Errors); + var ctx = new GenContext + { + Entities = load.Root.Objects(), Root = load.Root, Warn = _ => { }, + Config = new GenConfig + { + OutDir = "/unused", Namespace = "Acme", + ForeignKeyIndexes = on, SchemaConventions = conventions, Dialect = dialect, + }, + }; + return string.Join("\n", new DbContextGenerator().Generate(ctx).Select(f => f.Content)); + } + + private static string Index(string cls, string prop, string name) => + $"modelBuilder.Entity<{cls}>().HasIndex(x => x.{prop}, \"{name}\").HasDatabaseName(\"{name}\");"; + + [Fact] + public void Off_the_model_declares_no_foreign_key_index() + { + Assert.DoesNotContain("_idx\"", Context(on: false)); + } + + [Fact] + public void On_each_foreign_key_nothing_leads_with_gets_migrates_index() + { + var src = Context(on: true); + Assert.Contains(Index("Order", "CustomerId", "orders_customer_id_idx"), src); + Assert.Contains(Index("Order", "CustomerCode", "orders_customer_code_idx"), src); + Assert.Contains(Index("OrderTag", "TagId", "order_tags_tag_id_idx"), src); + // Led by a declared lookup index, and by the junction's primary key. + Assert.DoesNotContain("orders_bill_to_id_idx", src); + Assert.DoesNotContain("order_tags_order_id_idx", src); + } + + [Fact] + public void The_efcore_conventions_and_mysql_add_nothing() + { + Assert.DoesNotContain("_idx\"", Context(on: true, conventions: "efcore")); + Assert.DoesNotContain("_idx\"", Context(on: true, dialect: "mysql")); + } + + [Fact] + public void The_config_reader_takes_the_key_from_the_migrate_block() + { + var dir = Directory.CreateTempSubdirectory("mo-fk-idx-"); + try + { + Directory.CreateDirectory(Path.Combine(dir.FullName, ".metaobjects")); + File.WriteAllText(Path.Combine(dir.FullName, ".metaobjects", "config.json"), + """{ "schema_version": 1, "migrate": { "foreignKeyIndexes": true, "conventions": "efcore" } }"""); + var cfg = NeutralConfig.Read(dir.FullName, validateNaming: false); + Assert.NotNull(cfg); + Assert.True(cfg!.MigrateForeignKeyIndexes); + Assert.Equal("efcore", cfg.MigrateConventions); + File.WriteAllText(Path.Combine(dir.FullName, ".metaobjects", "config.json"), """{ "schema_version": 1 }"""); + Assert.False(NeutralConfig.Read(dir.FullName, validateNaming: false)!.MigrateForeignKeyIndexes); + } + finally { dir.Delete(recursive: true); } + } +} diff --git a/server/csharp/MetaObjects.Codegen/CodegenCli.cs b/server/csharp/MetaObjects.Codegen/CodegenCli.cs index 5bab418763..0196264aae 100644 --- a/server/csharp/MetaObjects.Codegen/CodegenCli.cs +++ b/server/csharp/MetaObjects.Codegen/CodegenCli.cs @@ -260,6 +260,8 @@ public static GenOutcome RunGen( SweepOrphans = sweepOrphans, Dialect = neutralConfig?.MigrateDialect, SkipRedundantChecks = neutralConfig?.MigrateSkipRedundantChecks ?? false, + ForeignKeyIndexes = neutralConfig?.MigrateForeignKeyIndexes ?? false, + SchemaConventions = neutralConfig?.MigrateConventions, }.WithCSharpLayout(neutralConfig); CodegenRunner.RunResult result; @@ -963,6 +965,8 @@ private static int RunVerifyArgs(string[] rest, IReadOnlyList declar // services read as drift. Dialect = neutralConfig?.MigrateDialect, SkipRedundantChecks = neutralConfig?.MigrateSkipRedundantChecks ?? false, + ForeignKeyIndexes = neutralConfig?.MigrateForeignKeyIndexes ?? false, + SchemaConventions = neutralConfig?.MigrateConventions, }.WithCSharpLayout(neutralConfig); var result = CodegenDrift.Compute(config, load.Root, suite, narrowed: a.Selection is { Count: > 0 }); if (result.Error is not null) diff --git a/server/csharp/MetaObjects.Codegen/Generator.cs b/server/csharp/MetaObjects.Codegen/Generator.cs index a8e6a6aff5..43124727ba 100644 --- a/server/csharp/MetaObjects.Codegen/Generator.cs +++ b/server/csharp/MetaObjects.Codegen/Generator.cs @@ -232,6 +232,22 @@ public string? ContextAssemblyName } private readonly string? _contextAssemblyName; + /// + /// migrate.foreignKeyIndexes in .metaobjects/config.json: the model declares the + /// index meta migrate gives each foreign key nothing else indexes + /// (<table>_<column>[_<column>…]_idx). False, the default, leaves the + /// model as it was. + /// + public bool ForeignKeyIndexes { get; init; } + + /// + /// migrate.conventions: the schema-conventions preset meta migrate builds with, or + /// null for its default. Under efcore migrate already indexes every foreign key + /// with EF Core's own names, which EF's convention reproduces, so + /// adds nothing. + /// + public string? SchemaConventions { get; init; } + /// /// The C# property name of a field, for every generator that names one: the entity and /// value-object classes, the composite [PrimaryKey], the DbContext's model diff --git a/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs b/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs index aa08e1693a..6f05563578 100644 --- a/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs +++ b/server/csharp/MetaObjects.Codegen/Generators/DbContextGenerator.cs @@ -270,6 +270,7 @@ public virtual IEnumerable Generate(GenContext ctx) junctionOwnedFks.TryGetValue(e, out var ownedByUsingEntity); if (EmitsReferenceForeignKeys) EmitReferenceConfig(e, tph, ctx, modelLines, ownedByUsingEntity); + EmitForeignKeyIndexConfig(e, tph, ctx, modelLines); // The adopter's own per-entity lines (indexes, CHECKs, query filters, computed // columns), last so they see everything the entity was configured with. @@ -1368,6 +1369,97 @@ private void EmitIndexConfig(string owner, MetaObject entity, GenContext ctx, Li } } + /// + /// migrate.foreignKeyIndexes — the index meta migrate gives each enforced foreign + /// key that nothing already leads with, so the model declares the index the database has, + /// under migrate's name (<table>_<column>[_<column>…]_idx). The + /// rule is migrate's own (uncoveredForeignKeyColumns in migrate-ts): the primary key, + /// a field's @unique, an identity.secondary and an index.lookup without + /// @expr cover a foreign key whose columns they lead with, in any order; two foreign + /// keys over one column set get one index; and a foreign key that leads a longer one shares + /// the longer one's index. Nothing under the efcore conventions, which already index + /// every foreign key with EF Core's own names, and nothing on MySQL, where InnoDB indexes each + /// foreign key itself. A TPH table is read whole: the base and every concrete subtype. + /// + private void EmitForeignKeyIndexConfig(MetaObject entity, TphPlan? tph, GenContext ctx, List modelLines) + { + var config = ctx.Config; + if (!config.ForeignKeyIndexes + || string.Equals(config.SchemaConventions, "efcore", StringComparison.Ordinal) + || string.Equals(config.Dialect, "mysql", StringComparison.OrdinalIgnoreCase)) + return; + var strategy = config.ColumnNamingStrategy; + var members = new List { entity }; + if (tph is not null) members.AddRange(tph.Subtypes.Select(st => st.Entity)); + + List? Columns(MetaObject owner, IEnumerable fieldNames) + { + var cols = new List(); + foreach (var name in fieldNames) + { + if (owner.FindField(name) is not { } f) return null; + cols.Add(CSharpNaming.Column(f, strategy)); + } + return cols; + } + + var existing = new List>(); + if (entity.PrimaryIdentity()?.Fields is { Count: > 0 } pk && Columns(entity, pk) is { } pkCols) existing.Add(pkCols); + foreach (var m in members) + { + // ADR-0039: resolving — @unique and index nodes may be inherited via extends. + foreach (var f in m.Fields().Where(f => f.Attr(FIELD_ATTR_UNIQUE) is true)) + existing.Add([CSharpNaming.Column(f, strategy)]); + foreach (var (node, fields) in m.SecondaryIdentities().Select(i => ((MetaData)i, i.Fields)) + .Concat(m.LookupIndexes().Select(i => ((MetaData)i, i.Fields)))) + { + if (node.Attr(DbConstants.IDENTITY_ATTR_EXPR) is string { Length: > 0 } || fields.Count == 0) continue; + if (Columns(m, fields) is { } cols) existing.Add(cols); + } + } + + // Each physical foreign key once, on the first class of the table that declares it. + var seen = new HashSet(StringComparer.Ordinal); + var candidates = new List<(MetaObject Owner, List Fields, List Columns)>(); + foreach (var m in members) + { + foreach (var reference in m.ReferenceIdentities().Where(r => r.Enforce)) + { + if (!seen.Add(FkKey(reference))) continue; + var fields = new List(); + foreach (var name in ReferentialActions.ReadIdentityFields(reference)) + { + if (m.FindField(name) is not { } f) { fields.Clear(); break; } + fields.Add(f); + } + if (fields.Count == 0) continue; + var cols = fields.Select(f => CSharpNaming.Column(f, strategy)).ToList(); + if (existing.Any(e => LeadsWith(e, cols))) continue; + if (candidates.Any(c => c.Columns.Count == cols.Count && LeadsWith(c.Columns, cols))) continue; + candidates.Add((m, fields, cols)); + } + } + + var table = CSharpNaming.Table(entity); + foreach (var (owner, fields, cols) in candidates) + { + if (candidates.Any(o => o.Columns.Count > cols.Count && LeadsWith(o.Columns, cols))) continue; + var name = $"\"{CSharpNaming.CSharpStringLiteral($"{table}_{string.Join("_", cols)}_idx")}\""; + var props = fields.Select(PropertyName).ToList(); + var selector = props.Count == 1 ? $"x => x.{props[0]}" : $"x => new {{ {string.Join(", ", props.Select(p => "x." + p))} }}"; + modelLines.Add($" modelBuilder.Entity<{CSharpNaming.Pascal(owner.Name)}>().HasIndex({selector}, {name}).HasDatabaseName({name});"); + } + } + + /// True when leads with exactly , + /// in any order (migrate's leadsWith). + private static bool LeadsWith(IReadOnlyList prefix, IReadOnlyList columns) + { + if (columns.Count == 0 || prefix.Count < columns.Count) return false; + var want = new HashSet(columns, StringComparer.Ordinal); + return prefix.Take(columns.Count).All(want.Contains); + } + // The SQL expressions migrate treats as an expression default (expected-schema.ts // EXPR_DEFAULT_PATTERNS): the rest of a string default is a literal. private static readonly System.Text.RegularExpressions.Regex ExprDefault = new( diff --git a/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs b/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs index d50a4a599c..eb882e272b 100644 --- a/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs +++ b/server/csharp/MetaObjects.IntegrationTests/EfModelGapsTests.cs @@ -1400,4 +1400,46 @@ public async Task The_checks_the_model_declares_are_the_ones_meta_migrate_create "INSERT INTO gadgets (kind, tags, email, score) VALUES ('plain', '{}', 'a@b.c', 9)")); Assert.True(ex.ConstraintName == "gadgets_score_numeric_chk", ex.MessageText); } + + // migrate.foreignKeyIndexes: the model declares the
__idx index `meta migrate` + // gives each foreign key nothing leads with, on the base and on a TPH subtype's own reference, + // and EF's convention IX_ index on that key gives way to it. + private const string ForeignKeyIndexModel = """ + { "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Owner", "children": [ + { "source.rdb": { "@table": "owners" } }, + { "field.long": { "name": "id" } }, + { "identity.primary": { "@fields": "id", "@generation": "increment" } } + ]}}, + { "object.entity": { "name": "Vehicle", "@discriminator": "kind", "children": [ + { "source.rdb": { "@table": "vehicles" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "kind", "@maxLength": 10 } }, + { "field.long": { "name": "ownerId" } }, + { "field.long": { "name": "keeperId" } }, + { "identity.primary": { "@fields": "id", "@generation": "increment" } }, + { "identity.reference": { "name": "owner", "@fields": ["ownerId"], "@references": "Owner" } }, + { "identity.reference": { "name": "keeper", "@fields": ["keeperId"], "@references": "Owner" } }, + { "index.lookup": { "name": "ix_vehicles_keeper", "@fields": ["keeperId", "kind"] } } + ]}}, + { "object.entity": { "name": "Car", "extends": "Vehicle", "@discriminatorValue": "Car", "children": [ + { "field.long": { "name": "dealerId" } }, + { "identity.reference": { "name": "dealer", "@fields": ["dealerId"], "@references": "Owner" } } + ]}} + ]}} + """; + + [Fact] + public void Foreign_key_indexes_reach_the_model_under_migrates_names() + { + var gen = GeneratedEfModel.Compile(ForeignKeyIndexModel, "Acme.FkIndex", c => c with { ForeignKeyIndexes = true }); + using var db = gen.NewContext(o => o.UseNpgsql("Host=unused")); + var vehicle = DesignEntityType(db, "Vehicle").GetDeclaredIndexes().Select(i => i.GetDatabaseName()).ToList(); + var car = DesignEntityType(db, "Car").GetDeclaredIndexes().Select(i => i.GetDatabaseName()).ToList(); + Assert.Contains("vehicles_ownerId_idx", vehicle); + Assert.Contains("vehicles_dealerId_idx", car); + // keeperId leads a declared lookup index, so it gets no second one. + Assert.DoesNotContain("vehicles_keeperId_idx", vehicle); + Assert.DoesNotContain(vehicle.Concat(car), n => n!.StartsWith("IX_", StringComparison.Ordinal)); + } } diff --git a/server/csharp/MetaObjects/Config/NeutralConfig.cs b/server/csharp/MetaObjects/Config/NeutralConfig.cs index eecab3b072..0ed6bff9ca 100644 --- a/server/csharp/MetaObjects/Config/NeutralConfig.cs +++ b/server/csharp/MetaObjects/Config/NeutralConfig.cs @@ -88,6 +88,14 @@ public sealed class NeutralConfig /// migrate.checks.skipRedundant: leave out the CHECKs a column's own type already enforces. public bool MigrateSkipRedundantChecks { get; private init; } + /// `migrate.foreignKeyIndexes`: index every foreign key the schema conventions leave + /// without one, as `meta migrate` does. False when absent or not a boolean. + public bool MigrateForeignKeyIndexes { get; private init; } + + /// `migrate.conventions`: the schema-conventions preset `meta migrate` builds with + /// (`default`, `efcore`), or null when absent. + public string? MigrateConventions { get; private init; } + /// /// FR-023 — requirements.inherit: the dependencies whose requirements this project takes on /// as its own ledger (). Empty when absent. @@ -164,12 +172,21 @@ private static bool ReadMigrateSkipRedundant(JsonElement root) => && checks.TryGetProperty("skipRedundant", out var skip) && skip.ValueKind == JsonValueKind.True; /// `migrate.dialect`, as the TypeScript `meta migrate` reads it; null when absent or not a string. - private static string? ReadMigrateDialect(JsonElement root) => + private static string? ReadMigrateDialect(JsonElement root) => ReadMigrateString(root, "dialect"); + + /// One string in the TypeScript-owned `migrate` block; null when absent or not a string. + private static string? ReadMigrateString(JsonElement root, string key) => root.TryGetProperty("migrate", out var migrate) && migrate.ValueKind == JsonValueKind.Object - && migrate.TryGetProperty("dialect", out var d) && d.ValueKind == JsonValueKind.String + && migrate.TryGetProperty(key, out var d) && d.ValueKind == JsonValueKind.String ? d.GetString() : null; + /// `migrate.foreignKeyIndexes`; false when absent or not a boolean (the TypeScript reader + /// refuses a non-boolean, and `meta migrate` is where the file is validated). + private static bool ReadMigrateForeignKeyIndexes(JsonElement root) => + root.TryGetProperty("migrate", out var migrate) && migrate.ValueKind == JsonValueKind.Object + && migrate.TryGetProperty("foreignKeyIndexes", out var v) && v.ValueKind == JsonValueKind.True; + /// /// As ; with false a /// naming value no strategy registered in THIS process answers is kept by name @@ -370,6 +387,8 @@ private static bool ReadMigrateSkipRedundant(JsonElement root) => PackageNamespaces = ReadPackageNamespaces(root, path), MigrateDialect = ReadMigrateDialect(root), MigrateSkipRedundantChecks = ReadMigrateSkipRedundant(root), + MigrateForeignKeyIndexes = ReadMigrateForeignKeyIndexes(root), + MigrateConventions = ReadMigrateString(root, "conventions"), }; } } diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt index ab328f2e52..795e1690b6 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt @@ -5,6 +5,7 @@ package com.metaobjects.generator.kotlin // `mvn metaobjects:eject` and its own package is renamed — an explicit import, not // same-package bare-name resolution, is what keeps the ejected copy compiling. import com.metaobjects.identity.PrimaryIdentity +import com.metaobjects.config.MigrateSettings import com.metaobjects.config.WireFormat import com.metaobjects.generator.util.GeneratorUtil import com.metaobjects.generator.kotlin.ExposedApi @@ -1207,7 +1208,8 @@ open class KotlinExposedTableGenerator : MultiFileDirectGeneratorBase_[_…]_idx`). The rule is + * migrate's own (`uncoveredForeignKeyColumns` in migrate-ts): the primary key, a field's + * `@unique`, an `identity.secondary` and an `index.lookup` without `@expr` cover a foreign + * key whose columns they lead with, in any order; two foreign keys over one column set get one + * index; and one that leads a longer one shares the longer one's index. Empty unless the + * project sets the option, and under the `efcore` conventions or on MySQL + * ([MigrateSettings.indexesForeignKeys]). A TPH table is read whole: base and subtypes. + */ + protected fun foreignKeyIndexes(entity: MetaObject, loader: MetaDataLoader, tableName: String): List>> { + if (!MigrateSettings.of(entity).indexesForeignKeys()) return emptyList() + val members = listOf(entity) + KotlinTphPlan.planFor(entity, loader)?.subtypes.orEmpty().map { it.entity } + // ADR-0039: resolving — fields, identities, index nodes and @unique may be inherited. + fun field(owner: MetaObject, name: String): MetaField<*>? = owner.metaFields.firstOrNull { it.name == name } + fun columns(owner: MetaObject, names: List): List? = + names.map { n -> field(owner, n)?.let { KotlinGenUtil.resolveColumnName(it) } ?: return null } + fun attr(node: MetaData, name: String): String? = + (if (node.hasMetaAttr(name)) node.getMetaAttr(name).value?.toString() else null)?.trim()?.takeIf { it.isNotEmpty() } + fun leadsWith(prefix: List, cols: List): Boolean = + cols.isNotEmpty() && prefix.size >= cols.size && prefix.take(cols.size).all { it in cols.toSet() } + + val existing = mutableListOf>() + entity.getIdentities(true).firstOrNull { it.isPrimary } + ?.let { pk -> columns(entity, KotlinGenUtil.keyFields(entity, pk)) } + ?.takeIf { it.isNotEmpty() }?.let(existing::add) + for (m in members) { + for (f in m.metaFields) { + if (attr(f, MetaField.ATTR_UNIQUE)?.toBoolean() == true) existing.add(listOf(KotlinGenUtil.resolveColumnName(f))) + } + val nodes: List>> = + m.getIdentities(true).filter { it.isSecondary }.map { it to it.fields } + + m.getChildren(LookupIndex::class.java, true).map { it to it.fields } + for ((node, names) in nodes) { + if (attr(node, com.metaobjects.identity.SecondaryIdentity.ATTR_EXPR) != null || names.isEmpty()) continue + columns(m, names)?.let(existing::add) + } + } + + val seen = hashSetOf() + val candidates = mutableListOf, List>>() // (columns, field names) + for (m in members) { + for (ref in m.getIdentities(true).filterIsInstance().filter { it.isEnforced }) { + if (!seen.add(ref.fields.joinToString(","))) continue + val cols = columns(m, ref.fields)?.takeIf { it.isNotEmpty() } ?: continue + if (existing.any { leadsWith(it, cols) }) continue + if (candidates.any { it.first.size == cols.size && leadsWith(it.first, cols) }) continue + candidates.add(cols to ref.fields) + } + } + return candidates + .filter { (cols, _) -> candidates.none { (o, _) -> o.size > cols.size && leadsWith(o, cols) } } + .map { (cols, fields) -> "${tableName}_${cols.joinToString("_")}_idx" to fields } + } + protected fun indexNameExpr(node: MetaData, shortName: String): String { val name = IndexNaming.resolve(node) if (!useNames()) return "\"$name\"" diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedForeignKeyIndexTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedForeignKeyIndexTest.kt new file mode 100644 index 0000000000..d89288baa8 --- /dev/null +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedForeignKeyIndexTest.kt @@ -0,0 +1,88 @@ +package com.metaobjects.generator.kotlin + +import com.metaobjects.config.MigrateSettings +import com.metaobjects.metadata.ktx.loadString +import java.nio.file.Files +import kotlin.test.Test +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** + * `migrate.foreignKeyIndexes` (.metaobjects/config.json, carried on the loaded root as + * [MigrateSettings]): `meta migrate` gives every enforced foreign key that nothing already leads + * with a `
__idx` index, so the Exposed table declares the same index under the + * same name. The model is the one migrate-ts's expected-schema-foreign-key-indexes test builds. + */ +class KotlinExposedForeignKeyIndexTest { + + private val fixture = """{ + "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Customer", "children": [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.secondary": { "name": "customers_code_key", "@fields": ["code"] } } + ] } }, + { "object.entity": { "name": "Order", "children": [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { "name": "id" } }, + { "field.long": { "name": "customerId", "@column": "customer_id" } }, + { "field.long": { "name": "billToId", "@column": "bill_to_id" } }, + { "field.string": { "name": "customerCode", "@column": "customer_code" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } }, + { "identity.reference": { "name": "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { "name": "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "identity.reference": { "name": "byCode", "@fields": ["customerCode"], "@references": "Customer.code" } }, + { "index.lookup": { "name": "orders_bill_to", "@fields": ["billToId", "id"] } } + ] } }, + { "object.entity": { "name": "OrderTag", "children": [ + { "source.rdb": { "@table": "order_tags" } }, + { "field.long": { "name": "orderId", "@column": "order_id" } }, + { "field.long": { "name": "tagId", "@column": "tag_id" } }, + { "identity.primary": { "name": "pk", "@fields": ["orderId", "tagId"] } }, + { "identity.reference": { "name": "order", "@fields": ["orderId"], "@references": "Order" } }, + { "identity.reference": { "name": "tag", "@fields": ["tagId"], "@references": "Tag" } } + ] } }, + { "object.entity": { "name": "Tag", "children": [ + { "source.rdb": { "@table": "tags" } }, + { "field.long": { "name": "id" } }, + { "identity.primary": { "name": "pk", "@fields": "id" } } + ] } } + ] } + }""".trimIndent() + + private fun tables(settings: MigrateSettings): String { + val outDir = Files.createTempDirectory("ktbl-fkidx-") + try { + val loader = loadString("fk-idx", fixture) + loader.root.setMigrateSettings(settings) + val gen = KotlinExposedTableGenerator() + gen.setArgs(mapOf("outputDir" to outDir.toString())) + gen.execute(loader) + return listOf("OrderTable.kt", "OrderTagTable.kt") + .joinToString("\n") { Files.readString(outDir.resolve("acme/$it")) } + } finally { + outDir.toFile().deleteRecursively() + } + } + + @Test fun `off, the table declares no foreign key index`() { + assertFalse("_idx\"" in tables(MigrateSettings.DEFAULT)) + } + + @Test fun `on, each foreign key nothing leads with gets migrate's index`() { + val src = tables(MigrateSettings(true, null, null)) + assertTrue("index(\"orders_customer_id_idx\", false, customerId)" in src, src) + assertTrue("index(\"orders_customer_code_idx\", false, customerCode)" in src, src) + assertTrue("index(\"order_tags_tag_id_idx\", false, tagId)" in src, src) + // Led by a declared lookup index, and by the junction's primary key. + assertFalse("orders_bill_to_id_idx" in src, src) + assertFalse("order_tags_order_id_idx" in src, src) + } + + @Test fun `the efcore conventions and mysql add nothing`() { + assertFalse("_idx\"" in tables(MigrateSettings(true, "efcore", null))) + assertFalse("_idx\"" in tables(MigrateSettings(true, null, "mysql"))) + } +} diff --git a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java index dd6c418262..0033ddcb3a 100644 --- a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java +++ b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java @@ -803,6 +803,11 @@ protected MetaDataLoader createLoader(ClassLoader projectClassLoader) { loader.getRoot().setEnumDefaults(resolveProjectConfig(srcDir) .map(com.metaobjects.config.NeutralConfig::getEnumDefaults) .orElse(com.metaobjects.config.EnumDefaults.DEFAULT)); + // The `migrate` schema settings (foreignKeyIndexes, conventions, dialect), read by the + // table generators so a mapping declares the indexes `meta migrate` creates. + loader.getRoot().setMigrateSettings(resolveProjectConfig(srcDir) + .map(com.metaobjects.config.NeutralConfig::getMigrateSettings) + .orElse(com.metaobjects.config.MigrateSettings.DEFAULT)); getLog().info("MetaData Mojo > Create Loader: " + loader.toString()); diff --git a/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java b/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java index de98496cd6..13d00e6181 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java +++ b/server/java/metadata/src/main/java/com/metaobjects/MetaRoot.java @@ -229,6 +229,21 @@ public void setEnumDefaults(com.metaobjects.config.EnumDefaults enumDefaults) { this.enumDefaults = (enumDefaults != null) ? enumDefaults : com.metaobjects.config.EnumDefaults.DEFAULT; } + /** + * The project's {@code migrate} schema settings, set by the same project-aware load as the + * naming rules and read through {@link com.metaobjects.config.MigrateSettings#of(MetaData)}. + */ + private transient com.metaobjects.config.MigrateSettings migrateSettings = + com.metaobjects.config.MigrateSettings.DEFAULT; + + public com.metaobjects.config.MigrateSettings getMigrateSettings() { + return migrateSettings; + } + + public void setMigrateSettings(com.metaobjects.config.MigrateSettings migrateSettings) { + this.migrateSettings = (migrateSettings != null) ? migrateSettings : com.metaobjects.config.MigrateSettings.DEFAULT; + } + /** The project's wire-format setting for this tree. Never {@code null}. */ public com.metaobjects.config.WireFormat getWireFormat() { return wireFormat; diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/MigrateSettings.java b/server/java/metadata/src/main/java/com/metaobjects/config/MigrateSettings.java new file mode 100644 index 0000000000..0d6d93bd4d --- /dev/null +++ b/server/java/metadata/src/main/java/com/metaobjects/config/MigrateSettings.java @@ -0,0 +1,61 @@ +package com.metaobjects.config; + +import com.metaobjects.MetaData; +import com.metaobjects.MetaRoot; + +/** + * The schema settings in the TypeScript-owned {@code migrate} block of + * {@code .metaobjects/config.json} that a generator mapping the schema has to agree with: + * {@code foreignKeyIndexes} (index every foreign key nothing else leads with, under + * {@code
__idx}), {@code conventions} (the schema-conventions preset, or + * {@code null} for the default) and {@code dialect}. Read leniently: {@code meta migrate} + * validates the block, and a value of the wrong type here reads as absent. Carried on the loaded + * root beside the naming rules. + */ +public final class MigrateSettings { + + /** No {@code migrate} settings: no foreign-key indexes, default conventions, no dialect. */ + public static final MigrateSettings DEFAULT = new MigrateSettings(false, null, null); + + private final boolean foreignKeyIndexes; + private final String conventions; + private final String dialect; + + public MigrateSettings(boolean foreignKeyIndexes, String conventions, String dialect) { + this.foreignKeyIndexes = foreignKeyIndexes; + this.conventions = conventions; + this.dialect = dialect; + } + + /** {@code migrate.foreignKeyIndexes}; false when absent. */ + public boolean isForeignKeyIndexes() { + return foreignKeyIndexes; + } + + /** {@code migrate.conventions}, or {@code null} when absent. */ + public String getConventions() { + return conventions; + } + + /** {@code migrate.dialect}, or {@code null} when absent. */ + public String getDialect() { + return dialect; + } + + /** + * Whether a schema mapping declares migrate's foreign-key indexes: the option is on, and + * neither the {@code efcore} conventions (which index every foreign key with their own names) + * nor MySQL (where InnoDB indexes each foreign key itself) already account for them. + */ + public boolean indexesForeignKeys() { + return foreignKeyIndexes && !"efcore".equals(conventions) && !"mysql".equalsIgnoreCase(dialect); + } + + /** The settings for the tree {@code node} belongs to: what its root carries, else none. */ + public static MigrateSettings of(MetaData node) { + for (MetaData n = node; n != null; n = n.getParent()) { + if (n instanceof MetaRoot root) return root.getMigrateSettings(); + } + return DEFAULT; + } +} diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java index 3ee082e13d..ea5760550f 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java @@ -70,6 +70,7 @@ public final class NeutralConfig { private final PhysicalNaming physicalNaming; private final WireFormat wireFormat; private EnumDefaults enumDefaults = EnumDefaults.DEFAULT; + private MigrateSettings migrateSettings = MigrateSettings.DEFAULT; private List requirementsInherit = List.of(); private NeutralConfig(List> sources, List> dependencies, @@ -151,6 +152,11 @@ public EnumDefaults getEnumDefaults() { return enumDefaults; } + /** The schema settings a mapping generator agrees with ({@code migrate.foreignKeyIndexes} and friends). */ + public MigrateSettings getMigrateSettings() { + return migrateSettings; + } + /** * Returns empty when {@code /.metaobjects/config.json} does not exist. * A file that EXISTS but is malformed throws — swallowing it would make a typo'd @@ -360,6 +366,7 @@ public static NeutralConfig parse(String content, String origin) { NeutralConfig cfg = new NeutralConfig(specs, dependencies, scopeInclude, migrateScope, physicalNaming, wireFormat); cfg.enumDefaults = parseEnums(root.get("enums"), origin); + cfg.migrateSettings = parseMigrateSettings(root.get("migrate")); cfg.requirementsInherit = parseRequirementsInherit(root.get("requirements"), origin); return cfg; } @@ -509,6 +516,23 @@ private static List parseRequirementsInherit(JsonElement requirements, S return stringArray(inherit, "requirements.inherit", origin); } + /** + * {@code migrate.foreignKeyIndexes}, {@code migrate.conventions} and {@code migrate.dialect}, + * read leniently (a wrong type reads as absent): {@code meta migrate} owns and validates them. + */ + private static MigrateSettings parseMigrateSettings(JsonElement migrate) { + if (migrate == null || !migrate.isJsonObject()) return MigrateSettings.DEFAULT; + JsonObject m = migrate.getAsJsonObject(); + JsonElement fk = m.get("foreignKeyIndexes"); + boolean foreignKeyIndexes = fk != null && fk.isJsonPrimitive() && fk.getAsJsonPrimitive().isBoolean() + && fk.getAsBoolean(); + return new MigrateSettings(foreignKeyIndexes, stringOrNull(m.get("conventions")), stringOrNull(m.get("dialect"))); + } + + private static String stringOrNull(JsonElement e) { + return e != null && e.isJsonPrimitive() && e.getAsJsonPrimitive().isString() ? e.getAsString() : null; + } + /** {@code migrate.scope}, or {@code null} when absent; every other {@code migrate} key is ignored. */ private static List parseMigrateScope(JsonElement migrate, String origin) { if (migrate == null || migrate.isJsonNull()) return null; diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java index fac3312f07..fe2fab8c60 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -230,6 +230,11 @@ public static java.util.Optional discoverEnumDefaults(List discoverMigrateSettings(List uris) { + return discoverConfig(uris).map(NeutralConfig::getMigrateSettings); + } + /** * ADR-0058 — the physical-naming rules of the {@code .metaobjects/config.json} found by * walking up from the first {@code file:} URI (stopping after a directory holding diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java b/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java index 1423f8d3ba..f4cfa15a3b 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java @@ -751,6 +751,9 @@ public static MetaDataLoader fromDirectory(String name, Path directory, loader.getRoot().setEnumDefaults(com.metaobjects.config.SourceResolver .discoverEnumDefaults(List.of(directory.toAbsolutePath().toUri())) .orElse(com.metaobjects.config.EnumDefaults.DEFAULT)); + loader.getRoot().setMigrateSettings(com.metaobjects.config.SourceResolver + .discoverMigrateSettings(List.of(directory.toAbsolutePath().toUri())) + .orElse(com.metaobjects.config.MigrateSettings.DEFAULT)); } catch (MetaDataLoadingException e) { throw e; } catch (Exception e) { @@ -858,6 +861,8 @@ public static MetaDataLoader fromUris(String name, List uris, LoaderOptions .discoverWireFormat(uris).orElse(com.metaobjects.config.WireFormat.DEFAULT)); loader.getRoot().setEnumDefaults(com.metaobjects.config.SourceResolver .discoverEnumDefaults(uris).orElse(com.metaobjects.config.EnumDefaults.DEFAULT)); + loader.getRoot().setMigrateSettings(com.metaobjects.config.SourceResolver + .discoverMigrateSettings(uris).orElse(com.metaobjects.config.MigrateSettings.DEFAULT)); } catch (MetaDataLoadingException e) { throw e; } catch (MetaDataException e) { diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java index 3094c57ccc..3aa7ccc519 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java @@ -25,6 +25,9 @@ import java.nio.file.Path; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertThrows; /** @@ -53,6 +56,25 @@ public void whitespaceOnlyPathRaises() throws IOException { assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); } + @Test + public void migrateSchemaSettingsAreReadLeniently() throws IOException { + // migrate.foreignKeyIndexes / conventions / dialect: the settings a table generator + // must agree with `meta migrate` on. TS-owned and validated there, so a wrong type + // reads as absent rather than failing this port's load. + Path dir = writeConfig("{ \"schema_version\": 1, \"migrate\": { \"foreignKeyIndexes\": true, " + + "\"conventions\": \"efcore\", \"dialect\": \"postgres\" } }"); + MigrateSettings m = NeutralConfig.read(dir).orElseThrow().getMigrateSettings(); + assertTrue(m.isForeignKeyIndexes()); + assertEquals("efcore", m.getConventions()); + assertEquals("postgres", m.getDialect()); + assertFalse("the efcore conventions already index every foreign key", m.indexesForeignKeys()); + + Path loose = writeConfig("{ \"schema_version\": 1, \"migrate\": { \"foreignKeyIndexes\": \"yes\" } }"); + assertFalse(NeutralConfig.read(loose).orElseThrow().getMigrateSettings().isForeignKeyIndexes()); + Path none = writeConfig("{ \"schema_version\": 1 }"); + assertSame(MigrateSettings.DEFAULT, NeutralConfig.read(none).orElseThrow().getMigrateSettings()); + } + @Test public void nonStringSourceValueRaises() throws IOException { Path dir = writeConfig("{ \"schema_version\": 1, \"sources\": [ { \"path\": 123 } ] }"); diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index daf6720b01..a3ed1f6c85 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -530,6 +530,11 @@ async function writeCodegenTsconfig(opts: InitOptions, result: InitResult): Prom result.created.push(CODEGEN_TSCONFIG_REL); } +/** The `migrate` settings a new project starts with (absent in an existing one: no FK indexes). */ +export const NEW_PROJECT_MIGRATE_DEFAULTS = { foreignKeyIndexes: true } as const; +/** The `verify` settings a new project starts with (absent in an existing one: hand edits are a notice). */ +export const NEW_PROJECT_VERIFY_DEFAULTS = { handEdits: "fail" } as const; + /** * .metaobjects/config.json — write fresh defaults, or preserve+merge an existing * valid config. Shared by the full scaffold and `--config-only` so the two paths @@ -539,11 +544,19 @@ async function writeConfigFile(opts: InitOptions, result: InitResult, agentDir: // An explicit sqlite/postgres dialect is written so `meta migrate` needs no --dialect; an // unflagged init keeps the defaults it always wrote. MySQL schema is not migrate's. const dialect = resolveInitDialect(opts); - const freshConfig = dialect === "d1" - ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: buildD1MigrateBlock(opts.cwd) }) - : opts.dialect !== undefined && dialect !== "mysql" - ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: { dialect } }) - : DEFAULT_CONFIG; + const migrate = dialect === "d1" + ? buildD1MigrateBlock(opts.cwd) + : opts.dialect !== undefined ? { dialect } : {}; + // New-project defaults, each written as its own visible setting (a project that predates + // a setting keeps the behaviour it had, because the key is absent there): every foreign key + // gets an index, and a hand edit to a generated file fails `meta verify --codegen`. + // MySQL writes no `migrate` block at all: MetaObjects does not own MySQL schema, and InnoDB + // indexes every foreign key itself. + const freshConfig = ConfigSchema.parse({ + ...DEFAULT_CONFIG, + ...(dialect === "mysql" ? {} : { migrate: { ...migrate, ...NEW_PROJECT_MIGRATE_DEFAULTS } }), + verify: NEW_PROJECT_VERIFY_DEFAULTS, + }); const writeFresh = (): Promise => writeFile(join(agentDir, "config.json"), JSON.stringify(freshConfig, null, 2) + "\n", "utf8"); diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 3e03dd836d..08faede021 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -10,7 +10,7 @@ import { ENUM_LENGTH_HINT, isEnumLengthRefusal } from "../lib/enum-length-refusal.js"; import { existsSync } from "node:fs"; import { join, relative, resolve as resolvePath } from "node:path"; -import { parseVerifyArgs, type MigrateFlags } from "../lib/args.js"; +import { parseVerifyArgs, type MigrateFlags, type VerifyFlags } from "../lib/args.js"; import { log } from "../lib/log.js"; import { requirementLedgerRows, type RequirementLedgerRow } from "../lib/requirement-ledger.js"; import { warnUnenforcedFkTargets } from "../lib/fk-target-warning.js"; @@ -108,6 +108,7 @@ import { loadMemory, resolveCollection, type DependencySpec, + type HandEditsPolicy, } from "@metaobjectsdev/sdk"; import { checkDependencies, ERR_DEPENDENCY_UPSTREAM_DRIFT, formatCheckLine, readLockOrThrow } from "../lib/dependency-sync.js"; import { exclusionNotes, importedOption, migrateScopeMismatch } from "../lib/migrate-scope.js"; @@ -1088,10 +1089,10 @@ export async function verifyCommand( // Same discipline as its siblings: an advisory scan never breaks verify. } - // Foreign keys migrate emits with no index on the referencing columns. Advisory: an - // automatic index would propose a migration to every existing adopter, so this names - // each one and says how to declare it (`index.lookup`) instead. Read off the SAME - // expected schema `meta migrate` builds, so "covered" means covered in its DDL. + // Foreign keys migrate emits with no index on the referencing columns. Advisory: the + // index is opt-in (`migrate.foreignKeyIndexes`), since turning it on proposes a migration + // to an existing project, so this names each one and says how to index it. Read off the + // SAME expected schema `meta migrate` builds, so with the option on there is none. let unindexedFks: UnindexedFkFinding[] = []; // Skipped on MySQL: MetaObjects emits no DDL there, and InnoDB indexes every foreign // key itself. @@ -1822,8 +1823,8 @@ export async function verifyCommand( // Hand edits are not drift, but they must never be INVISIBLE: a validator weakened // inside a generated file used to pass this gate with "every file's generated - // contribution is current" and nothing else. Reported on every run; fails the build - // only under --forbid-hand-edits. + // contribution is current" and nothing else. Reported on every run; fails the build when + // the run's policy is `fail` (verify.handEdits, or --forbid-hand-edits). const handEditExit = reportHandEdits(result.handEdited); // Say how much documentation was NOT checked, on the pass and the fail line alike, so a // gate that compared two pages of 589 cannot read as one that compared all of them. @@ -1877,13 +1878,19 @@ export async function verifyCommand( return 1; } - /** Print the hand-edit notice (or failure, under --forbid-hand-edits); return its exit code. */ + /** + * Print the hand-edit notice, or the failure when the run's policy is `fail`; return its exit + * code. The policy is the run's flag when one is passed, else `verify.handEdits` in + * `.metaobjects/config.json`, else `notice` (what a project that predates the key has always had). + */ function reportHandEdits(handEdited: readonly string[]): number { if (handEdited.length === 0) return 0; - const [header, ...rest] = handEditNotice(handEdited, flags.forbidHandEdits); + const policy = handEditPolicy(flags, collection.handEdits); + const fail = policy.policy === "fail"; + const [header, ...rest] = handEditNotice(handEdited, policy); const paths = rest.slice(0, handEdited.length); const footer = rest.slice(handEdited.length); - if (flags.forbidHandEdits) { + if (fail) { // A failure lists every path: truncating the list a build failed on hides the cause. for (const line of [header!, ...paths, ...footer]) log.error(line); return 1; @@ -2070,18 +2077,60 @@ export function wireDiffersMessage(committed: WireFormat, used: WireFormat): str ); } -export function handEditNotice(paths: readonly string[], forbidden: boolean): string[] { - const header = forbidden - ? `meta verify — ${paths.length} generated file(s) carry hand edits, and --forbid-hand-edits is set:` - : `meta verify — ${paths.length} generated file(s) carry hand edits ` + - `(notice — 'meta gen' preserves them by design; does not fail the build — ` + - `pass --forbid-hand-edits to make it):`; +/** Where a run's hand-edit policy came from: a flag on this run, the project's config, or neither. */ +export type HandEditPolicySource = "--forbid-hand-edits" | "--allow-hand-edits" | "config" | "default"; + +/** The hand-edit policy one `meta verify --codegen` run applies, and what decided it. */ +export function handEditPolicy( + flags: Pick, + configured: HandEditsPolicy | undefined, +): { policy: HandEditsPolicy; source: HandEditPolicySource } { + if (flags.forbidHandEdits) return { policy: "fail", source: "--forbid-hand-edits" }; + if (flags.allowHandEdits) return { policy: "notice", source: "--allow-hand-edits" }; + if (configured !== undefined) return { policy: configured, source: "config" }; + return { policy: "notice", source: "default" }; +} + +const HAND_EDITS_KEY = `"verify": { "handEdits": … } in .metaobjects/config.json`; + +/** + * The hand-edit report: a header, one line per file, then how to see an edit and the two ways + * forward — drop it (the change belongs in the metadata) or keep it on purpose. + */ +export function handEditNotice( + paths: readonly string[], + run: { policy: HandEditsPolicy; source: HandEditPolicySource }, +): string[] { + const n = `${paths.length} generated file(s) carry hand edits`; + let header: string; + switch (run.source) { + case "--forbid-hand-edits": + header = `meta verify — ${n}, and --forbid-hand-edits is set:`; + break; + case "--allow-hand-edits": + header = `meta verify — ${n} (notice — --allow-hand-edits is set, so this run does not fail on them):`; + break; + case "config": + header = run.policy === "fail" + ? `meta verify — ${n}, and verify.handEdits is "fail" in .metaobjects/config.json:` + : `meta verify — ${n} (notice — verify.handEdits is "notice" in .metaobjects/config.json, so this does not fail the build):`; + break; + default: + header = `meta verify — ${n} (notice — 'meta gen' preserves them by design; does not fail the build — ` + + `set "verify": { "handEdits": "fail" } in .metaobjects/config.json, or pass --forbid-hand-edits, to make it):`; + } + const keep = run.policy === "fail" + ? `To keep an edit on purpose: pass --allow-hand-edits for one run, or set "handEdits": "notice" (${HAND_EDITS_KEY}).` + : `To keep an edit on purpose: nothing to do while hand edits are a notice; a project that wants them to fail sets "handEdits": "fail" (${HAND_EDITS_KEY}).`; return [ header, ...paths.map((p) => ` ${p}`), "To see an edit: delete the file and run 'meta gen' (it re-emits the pristine generation), " + - "then 'git diff -R -- ' shows the edit and 'git checkout -- ' restores it" + - (forbidden ? "; to drop the edit, commit the regenerated file instead." : "."), + "then 'git diff -R -- ' shows the edit and 'git checkout -- ' restores it.", + "To drop an edit: delete the file, run 'meta gen' and commit the regenerated file; if the edit " + + "changed a rule the model declares (a length, a range, a required field), make that change in " + + "the metadata instead, so the database and every port follow it.", + keep, ]; } diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index f60793bc2d..6aaf67b650 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -336,10 +336,15 @@ FLAGS: An .names.ts artifact is the exception: it must match a fresh regen exactly — a hand edit in it is itself the defect. Needs metaobjects.config.ts; exit 2 if absent. - Every generated file that carries a hand edit is still LISTED - (a notice, exit code unchanged), so an edit is never invisible. + Every generated file that carries a hand edit is LISTED, so an + edit is never invisible. Whether that fails the run is the + project's verify.handEdits in .metaobjects/config.json: "fail" + exits 1 ('meta init' writes it for a new project); "notice", or + no key, lists them and leaves the exit code alone. --forbid-hand-edits With --codegen: fail (exit 1) on ANY hand edit inside generated - code, for teams that want generated files untouchable. + code for this run, whatever verify.handEdits says. + --allow-hand-edits With --codegen: list hand edits as a notice and exit 0 for this + run, relaxing verify.handEdits: "fail". --docs Removed in 1.1.0: says so and exits 2. --codegen checks documentation with everything else \`generators\` declares. --deps Dependency drift — re-resolve each declared dependency exactly as @@ -413,7 +418,8 @@ A bare 'meta verify' also runs an ADVISORY anti-pattern pass: it scans your auth source for hand-rolled aggregates, money-as-float, and CHECK-IN enums and points you at the construct that models them (origin.aggregate / field.currency / field.enum). The same pass names every foreign key with no index on its referencing column(s) -and the index.lookup that would cover it (migrate never adds one on its own). +and the index.lookup that would cover it (migrate adds one only when +.metaobjects/config.json sets "migrate": { "foreignKeyIndexes": true }). Warnings only — it never fails the build. Opt out with --no-antipatterns or META_NO_ANTIPATTERNS=1. diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index fc00d6c938..47ca9b5e28 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -391,13 +391,17 @@ export interface VerifyFlags { /** Run the codegen-drift gate: is the GENERATED contribution of each file current? */ codegen: boolean; /** - * With `--codegen`: fail the build on any hand edit inside a generated file, for teams - * that want generated code untouchable. Off by default, because hand edits are the - * documented workflow (`meta gen` three-way-merges them) — without it they are - * reported as a notice and never change the exit code. Refused without `--codegen`: - * it modifies that gate and would otherwise parse cleanly and do nothing. + * With `--codegen`: fail the build on any hand edit inside a generated file for THIS run, + * whatever `verify.handEdits` in `.metaobjects/config.json` says. Refused without + * `--codegen`: it modifies that gate and would otherwise parse cleanly and do nothing. */ forbidHandEdits: boolean; + /** + * With `--codegen`: list hand edits as a notice and exit 0 for THIS run, relaxing a + * project's `verify.handEdits: "fail"`. Refused without `--codegen`, and refused beside + * `--forbid-hand-edits`, which asks the opposite. + */ + allowHandEdits: boolean; /** * `--docs`, removed in 1.1.0 (ADR-0034 Amendment 9): `--codegen` checks documentation with * everything else the generator list declares. Still parsed, so the command can say where @@ -521,6 +525,7 @@ export const VERIFY_OPTIONS = { templates: { type: "boolean", default: false }, codegen: { type: "boolean", default: false }, "forbid-hand-edits": { type: "boolean", default: false }, + "allow-hand-edits": { type: "boolean", default: false }, // Removed in 1.1.0 (ADR-0034 Amendment 9). Still parsed, so `meta verify --docs` prints // where the check went instead of an unknown-flag error. docs: { type: "boolean", default: false }, @@ -588,6 +593,15 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { "--forbid-hand-edits only applies to the codegen gate. Run: meta verify --codegen --forbid-hand-edits", ); } + const allowHandEdits = !!values["allow-hand-edits"]; + if (allowHandEdits && !codegen) { + throw new Error( + "--allow-hand-edits only applies to the codegen gate. Run: meta verify --codegen --allow-hand-edits", + ); + } + if (allowHandEdits && forbidHandEdits) { + throw new Error("--allow-hand-edits and --forbid-hand-edits ask for opposite things; pass one of them."); + } const docs = !!values.docs; const deps = !!values.deps; const replay = !!values.replay; @@ -612,6 +626,7 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { templates, codegen, forbidHandEdits, + allowHandEdits, docs, deps, replay, diff --git a/server/typescript/packages/cli/src/lib/fk-index-advisory.ts b/server/typescript/packages/cli/src/lib/fk-index-advisory.ts index 4edfc44447..2668a580f5 100644 --- a/server/typescript/packages/cli/src/lib/fk-index-advisory.ts +++ b/server/typescript/packages/cli/src/lib/fk-index-advisory.ts @@ -1,11 +1,13 @@ // Foreign keys with no index covering their columns — an advisory authoring lint. // // `meta migrate` emits a FOREIGN KEY constraint for every enforced `identity.reference` -// and no index on its columns: neither Postgres nor SQLite indexes the referencing side -// of a foreign key. Every join from the parent, and every ON DELETE cascade / set-null / -// restrict check from it, then scans the child table. Generating the index automatically -// would propose a migration to every existing adopter the release it shipped, so this is -// advice, never a gate: it names each such key and says how to declare the index. +// and, unless the project sets `migrate.foreignKeyIndexes`, no index on its columns: +// neither Postgres nor SQLite indexes the referencing side of a foreign key. Every join +// from the parent, and every ON DELETE cascade / set-null / restrict check from it, then +// scans the child table. The option is opt-in (`meta init` writes it for a new project), +// because turning it on proposes a migration to an existing one, so this stays advice, +// never a gate: it names each such key and says how to index it. With the option on, the +// expected schema already carries the index and there is nothing to name. // // Coverage is read off the EXPECTED SCHEMA migrate itself builds (primary key, every // `identity.secondary`, every `index.lookup`, every `@unique` field), not re-derived from @@ -141,8 +143,9 @@ export function summarizeUnindexedFks( construct: findings.map((f) => f.construct).join(", "), message: `${findings.length} foreign key(s) have no index covering their columns, so joins from the ` + - `referenced table and its ON DELETE checks scan the referencing one. To index them, declare ` + - `an ${INDEX_LOOKUP} on each: ` + + `referenced table and its ON DELETE checks scan the referencing one. To index every one, set ` + + `"migrate": { "foreignKeyIndexes": true } in .metaobjects/config.json and run 'meta migrate'; ` + + `to index them one at a time, declare an ${INDEX_LOOKUP} on each: ` + findings.map((f) => `${f.label} ${f.declaration}`).join("; "), }; } diff --git a/server/typescript/packages/cli/src/lib/schema-build-options.ts b/server/typescript/packages/cli/src/lib/schema-build-options.ts index 8067c6e57b..5784235a6f 100644 --- a/server/typescript/packages/cli/src/lib/schema-build-options.ts +++ b/server/typescript/packages/cli/src/lib/schema-build-options.ts @@ -17,6 +17,8 @@ import type { Collection } from "@metaobjectsdev/sdk"; export interface SchemaBuildOptions { conventions: SchemaConventions; skipRedundantChecks: boolean; + /** `migrate.foreignKeyIndexes`: index every foreign key nothing else indexes. */ + foreignKeyIndexes: boolean; } /** @@ -26,12 +28,13 @@ export interface SchemaBuildOptions { * member (`isUnknownSchemaConventionsError` recognises both). */ export function schemaBuildOptions( - collection: Pick, + collection: Pick, tsConfig: Pick | undefined, ): SchemaBuildOptions { return { conventions: resolveSchemaConventions(tsConfig?.schemaConventions ?? collection.schemaConventions), skipRedundantChecks: collection.skipRedundantChecks, + foreignKeyIndexes: collection.foreignKeyIndexes, }; } @@ -42,7 +45,7 @@ export function schemaBuildOptions( * refuse the same config with exit 2. */ export function schemaBuildOptionsOrWarn( - collection: Pick, + collection: Pick, tsConfig: Pick | undefined, warn: (message: string) => void, ): SchemaBuildOptions | undefined { diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index f909ea2b1f..2202d16c3b 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -182,6 +182,19 @@ describe("init() — happy path", () => { expect(config.pending_in_git).toBe(true); }); + test("writes the new-project defaults as visible settings: FK indexes on, hand edits fail verify", async () => { + await init({ cwd }); + const config = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(config.migrate).toEqual({ foreignKeyIndexes: true }); + expect(config.verify).toEqual({ handEdits: "fail" }); + }); + + test("a dialect flag keeps the new-project defaults beside migrate.dialect", async () => { + await init({ cwd, dialect: "postgres" }); + const config = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(config.migrate).toEqual({ dialect: "postgres", foreignKeyIndexes: true }); + }); + test("writes a .gitignore under .metaobjects/ that includes .gen-state", async () => { await init({ cwd }); const ignore = readFileSync(join(cwd, ".metaobjects", ".gitignore"), "utf8"); @@ -508,6 +521,9 @@ describe("init() --config-only", () => { expect(result.preserved).toContain(".metaobjects/config.json"); const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); expect(cfg.sources).toEqual([{ path: "model" }]); + // An existing project keeps the behaviour it had: no new-project default is added to it. + expect(cfg.migrate).toBeUndefined(); + expect(cfg.verify).toBeUndefined(); }); test("refuses to overwrite an existing config it cannot parse, without --force", async () => { diff --git a/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts b/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts index cef8b17a9f..eb752aff58 100644 --- a/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts +++ b/server/typescript/packages/cli/test/integration/verify-codegen-hand-edits.test.ts @@ -373,6 +373,10 @@ describe("meta verify --codegen — hand edits are visible", () => { expect(report).toContain("--forbid-hand-edits"); expect(report).toMatch(/\s generated\/User\.ts$/m); expect(report).toContain("git diff -R -- "); + // The no-key notice names the config key that makes it a failure, and both remedies. + expect(report).toContain('"verify": { "handEdits": "fail" }'); + expect(report).toContain("To drop an edit:"); + expect(report).toContain("To keep an edit on purpose:"); } finally { rmSync(root, { recursive: true, force: true }); } @@ -454,4 +458,104 @@ describe("meta verify --codegen — hand edits are visible", () => { rmSync(root, { recursive: true, force: true }); } }); + + /** A project's own `verify.handEdits`, written into its .metaobjects/config.json. */ + function setHandEdits(root: string, handEdits: string): void { + mkdirSync(join(root, ".metaobjects"), { recursive: true }); + const path = join(root, ".metaobjects", "config.json"); + const prior = existsSync(path) ? JSON.parse(readFileSync(path, "utf8")) as Record : { schema_version: 1 }; + writeFileSync(path, JSON.stringify({ ...prior, verify: { handEdits } }, null, 2)); + } + + test('verify.handEdits: "fail" fails the run, naming the file and both remedies', async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "fail"); + weakenValidator(root); + + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(1); + const report = all(); + expect(report).toContain('verify.handEdits is "fail"'); + expect(report).toMatch(/\s generated\/User\.ts$/m); + expect(report).toContain("To drop an edit: delete the file, run 'meta gen'"); + expect(report).toContain("make that change in the metadata instead"); + expect(report).toContain("pass --allow-hand-edits for one run"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("--allow-hand-edits relaxes verify.handEdits: \"fail\" for one run", async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "fail"); + weakenValidator(root); + + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen", "--allow-hand-edits"])).toBe(0); + expect(all()).toContain("--allow-hand-edits is set"); + expect(all()).toContain("1 generated file(s) carry hand edits"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test('verify.handEdits: "fail" passes a project with no hand edits', async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "fail"); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(0); + expect(all()).not.toContain("carry hand edits"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test('verify.handEdits: "notice" keeps the notice, and --forbid-hand-edits still tightens it', async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + setHandEdits(root, "notice"); + weakenValidator(root); + + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(0); + expect(all()).toContain('verify.handEdits is "notice"'); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen", "--forbid-hand-edits"])).toBe(1); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("a verify.handEdits value this version does not know is refused", async () => { + const root = setupRepo(); + try { + setHandEdits(root, "warn"); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen"])).toBe(2); + expect(all()).toContain('"verify.handEdits" must be one of "fail", "notice"'); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("--allow-hand-edits needs --codegen, and is refused beside --forbid-hand-edits", async () => { + const root = setupRepo(); + try { + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--allow-hand-edits"])).toBe(2); + expect(all()).toContain("--codegen"); + out = []; err = []; + expect(await run(["verify", "--cwd", root, "--codegen", "--allow-hand-edits", "--forbid-hand-edits"])).toBe(2); + expect(all()).toContain("opposite"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); }); diff --git a/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts b/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts index 0f0a8adadf..6142db596f 100644 --- a/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts +++ b/server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts @@ -24,7 +24,7 @@ afterAll(() => { }); describe("meta verify is not a vacuous gate", () => { - test("clean tree exits 0; metadata drift, a forbidden hand edit and a deleted file each exit 1", async () => { + test("clean tree exits 0; metadata drift, a hand edit and a deleted file each exit 1", async () => { root = await newShopProject("verify-gate-"); const configPath = join(root, "metaobjects.config.ts"); writeFileSync( @@ -61,15 +61,17 @@ describe("meta verify is not a vacuous gate", () => { writeModel(root, SHOP_MODEL); expect((await meta(root, "verify")).exit).toBe(0); - // 4. A hand edit inside a generated file is NOT drift by default: `meta gen` three-way-merges - // it, so plain `verify` stays green. A team that wants generated code untouchable opts in - // with --forbid-hand-edits; that is the gate to assert if that is your policy. + // 4. Break B: a hand edit inside a generated file. `meta init` writes + // `"verify": { "handEdits": "fail" }`, so plain `verify` fails on it; --allow-hand-edits + // relaxes that for one run (and still names the file). const edit = pristine.replace(/email: (\w+)\("email"\)/, 'emailAddress: $1("email")'); expect(edit).not.toBe(pristine); writeFileSync(customer, edit); - expect((await meta(root, "verify")).exit).toBe(0); - const edited = await meta(root, "verify", "--codegen", "--forbid-hand-edits"); + const edited = await meta(root, "verify"); expect({ step: "edited generated part", exit: edited.exit }).toEqual({ step: "edited generated part", exit: 1 }); + expect(edited.output).toContain("Customer.ts"); + const allowed = await meta(root, "verify", "--codegen", "--allow-hand-edits"); + expect({ step: "edit allowed for one run", exit: allowed.exit }).toEqual({ step: "edit allowed for one run", exit: 0 }); writeFileSync(customer, pristine); // 5. Break C: a generated file removed. diff --git a/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts b/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts index fb36b12ddb..fc86c6d835 100644 --- a/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts +++ b/server/typescript/packages/cli/test/migrate-schema-conventions.test.ts @@ -36,13 +36,17 @@ const MODEL = JSON.stringify({ }, }); -async function project(conventions: string | undefined): Promise { +async function project(conventions: string | undefined, foreignKeyIndexes?: boolean): Promise { const root = await mkdtemp(join(tmpdir(), "schema-conventions-")); dirs.push(root); await mkdir(join(root, "metaobjects"), { recursive: true }); await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); await mkdir(join(root, ".metaobjects", "migrations"), { recursive: true }); - const migrate = conventions === undefined ? { dialect: "sqlite" } : { dialect: "sqlite", conventions }; + const migrate = { + dialect: "sqlite", + ...(conventions !== undefined && { conventions }), + ...(foreignKeyIndexes !== undefined && { foreignKeyIndexes }), + }; await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, migrate }), "utf8"); return root; } @@ -77,6 +81,25 @@ describe("meta migrate with migrate.conventions", () => { expect(out.join("\n")).not.toContain("IX_OrderLines_OrderId"); }); + test("migrate.foreignKeyIndexes: the foreign key gets
__idx, with no renamed constraint", async () => { + const root = await project(undefined, true); + + expect(await dryRun(root)).toBe(0); + const text = out.join("\n"); + expect(text).toContain('CREATE INDEX "OrderLines_OrderId_idx" ON "OrderLines" ("OrderId");'); + expect(text).not.toContain("IX_OrderLines_OrderId"); + expect(text).not.toContain("PK_"); + }); + + test("migrate.foreignKeyIndexes under efcore adds nothing: the preset already indexes it", async () => { + const root = await project("efcore", true); + + expect(await dryRun(root)).toBe(0); + const text = out.join("\n"); + expect(text).toContain('CREATE INDEX "IX_OrderLines_OrderId"'); + expect(text).not.toContain("OrderLines_OrderId_idx"); + }); + test("an unknown preset is refused by name, not as an unexpected error", async () => { const root = await project("hibernate"); diff --git a/server/typescript/packages/cli/test/unit/args-verify.test.ts b/server/typescript/packages/cli/test/unit/args-verify.test.ts index fe758ff27a..ff85972930 100644 --- a/server/typescript/packages/cli/test/unit/args-verify.test.ts +++ b/server/typescript/packages/cli/test/unit/args-verify.test.ts @@ -6,7 +6,7 @@ describe("parseVerifyArgs", () => { test("defaults: prompts/db/dialect undefined, allow empty, skipSchema false, no explicit subverb", () => { expect(parseVerifyArgs([])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: [], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -14,7 +14,7 @@ describe("parseVerifyArgs", () => { test("--prompts is captured", () => { expect(parseVerifyArgs(["--prompts", "templates"])).toEqual({ prompts: "templates", db: undefined, dialect: undefined, allow: [], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -22,7 +22,7 @@ describe("parseVerifyArgs", () => { test("--db / --dialect / --skip-schema are captured", () => { expect(parseVerifyArgs(["--db", "file:x.db", "--dialect", "sqlite", "--skip-schema"])).toEqual({ prompts: undefined, db: "file:x.db", dialect: "sqlite", allow: [], skipSchema: true, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -32,7 +32,7 @@ describe("parseVerifyArgs", () => { test("--dialect d1 --d1 --remote are captured; --dialect d1 alone is an explicit subverb", () => { expect(parseVerifyArgs(["--dialect", "d1", "--d1", "DB", "--remote"])).toEqual({ prompts: undefined, db: undefined, dialect: "d1", allow: [], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: "DB", remote: true, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -77,7 +77,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--allow", "drop-column,drop-table"])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: ["drop-column", "drop-table"], skipSchema: false, migrationFormat: undefined, - templates: false, codegen: false, forbidHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, + templates: false, codegen: false, forbidHandEdits: false, allowHandEdits: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, requireImplementers: false, noOverlayLint: false, noNameLint: false, noDeprecationLint: false, noFieldLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -157,6 +157,13 @@ describe("parseVerifyArgs", () => { // so it must not switch the bare-verify template default off. expect(parseVerifyArgs(["--require-implementers"]).anyExplicit).toBe(false); }); + test("--allow-hand-edits rides on --codegen, and is refused without it or beside --forbid-hand-edits", () => { + expect(parseVerifyArgs(["--codegen"]).allowHandEdits).toBe(false); + expect(parseVerifyArgs(["--codegen", "--allow-hand-edits"]).allowHandEdits).toBe(true); + expect(() => parseVerifyArgs(["--allow-hand-edits"])).toThrow(/only applies to the codegen gate/); + expect(() => parseVerifyArgs(["--codegen", "--allow-hand-edits", "--forbid-hand-edits"])).toThrow(/opposite/); + }); + test("--forbid-hand-edits rides on --codegen, and is refused without it", () => { expect(parseVerifyArgs(["--codegen"]).forbidHandEdits).toBe(false); expect(parseVerifyArgs(["--codegen", "--forbid-hand-edits"]).forbidHandEdits).toBe(true); diff --git a/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts b/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts index 375b4be8af..c2d73eceb4 100644 --- a/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts +++ b/server/typescript/packages/cli/test/unit/gen-migrate-advice.test.ts @@ -49,7 +49,7 @@ function input(over: Partial & Pick { +async function project(migrate?: Record): Promise { const root = await mkdtemp(join(tmpdir(), "verify-fk-index-")); dirs.push(root); await mkdir(join(root, "metaobjects"), { recursive: true }); await writeFile(join(root, "metaobjects", "meta.tracker.json"), MODEL, "utf8"); + if (migrate !== undefined) { + await mkdir(join(root, ".metaobjects"), { recursive: true }); + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, migrate }), "utf8"); + } return root; } @@ -86,6 +90,20 @@ describe("meta verify — unindexed foreign keys (advisory)", () => { expect(out).not.toContain("Issue.fkOwner"); }); + test("names migrate.foreignKeyIndexes as the way to index every one", async () => { + const root = await project(); + const { out } = await capture(() => verifyCommand([], root)); + expect(out).toContain('"migrate": { "foreignKeyIndexes": true }'); + }); + + test("with migrate.foreignKeyIndexes on, migrate indexes the key, so there is nothing to name", async () => { + const root = await project({ foreignKeyIndexes: true }); + const { code, out } = await capture(() => verifyCommand([], root)); + expect(code).toBe(0); + expect(out).not.toContain("Issue.fkProject"); + expect(out).not.toMatch(/foreign key\(s\) with no index/); + }); + test("--no-antipatterns suppresses it with the rest of the advisory tier", async () => { const root = await project(); const { out } = await capture(() => verifyCommand(["--no-antipatterns"], root)); diff --git a/server/typescript/packages/codegen-ts/src/engine-surface.ts b/server/typescript/packages/codegen-ts/src/engine-surface.ts index b5df86a8af..7df2cc7771 100644 --- a/server/typescript/packages/codegen-ts/src/engine-surface.ts +++ b/server/typescript/packages/codegen-ts/src/engine-surface.ts @@ -36,6 +36,7 @@ export const ENGINE_MODULES: readonly string[] = [ "codegen-ts/src/enum-meta.ts", "codegen-ts/src/enum-shared.ts", "codegen-ts/src/flattened.ts", + "codegen-ts/src/foreign-key-indexes.ts", "codegen-ts/src/format.ts", "codegen-ts/src/generator.ts", "codegen-ts/src/generators/agent-schema-input.ts", diff --git a/server/typescript/packages/codegen-ts/src/engine.ts b/server/typescript/packages/codegen-ts/src/engine.ts index 2ce675ed6f..d0ff8bd03b 100644 --- a/server/typescript/packages/codegen-ts/src/engine.ts +++ b/server/typescript/packages/codegen-ts/src/engine.ts @@ -23,6 +23,7 @@ export * from "./enum-import.js"; export * from "./enum-meta.js"; export * from "./enum-shared.js"; export * from "./flattened.js"; +export * from "./foreign-key-indexes.js"; export * from "./format.js"; export * from "./generator.js"; export * from "./generators/agent-schema-input.js"; diff --git a/server/typescript/packages/codegen-ts/src/foreign-key-indexes.ts b/server/typescript/packages/codegen-ts/src/foreign-key-indexes.ts new file mode 100644 index 0000000000..f9bf6e0354 --- /dev/null +++ b/server/typescript/packages/codegen-ts/src/foreign-key-indexes.ts @@ -0,0 +1,61 @@ +// The indexes `meta migrate` creates because a table has foreign keys, for a generator that maps +// the schema (the Drizzle table's index callback). +// +// Read off the SAME expected schema migrate builds, with the project's schema settings, so the +// mapping declares exactly the indexes the database gets, under the names it gets them. Only +// when `migrate.foreignKeyIndexes` is on: a project without it keeps the mapping it had, and +// `meta verify` advises about each unindexed foreign key instead. + +import type { MetaObject, MetaRoot } from "@metaobjectsdev/metadata"; +import { + buildExpectedSchemaWithProvenance, + type SchemaBuildSettings, +} from "@metaobjectsdev/migrate-ts/expected"; +import type { Dialect } from "./column-mapper.js"; + +export interface ForeignKeyIndex { + name: string; + columns: readonly string[]; +} + +type ByFqn = ReadonlyMap; + +// One schema build per (root, settings, dialect) — a run renders every entity against one root. +const cache = new WeakMap>>(); + +function build(root: MetaRoot, settings: SchemaBuildSettings, dialect: "sqlite" | "postgres"): ByFqn { + const out = new Map(); + let built: ReturnType; + try { + built = buildExpectedSchemaWithProvenance(root, { ...settings, dialect }); + } catch { + // A model the schema builder refuses is `meta migrate`'s to report, with its own message. + return out; + } + for (const [table, indexes] of built.foreignKeyIndexes ?? []) { + const fqn = built.provenance.get(table); + if (fqn !== undefined) out.set(fqn, indexes.map((ix) => ({ name: ix.name, columns: [...ix.columns] }))); + } + return out; +} + +/** + * The foreign-key indexes of `obj`'s table, in the order migrate adds them; none unless the + * project sets `migrate.foreignKeyIndexes`. None on MySQL either: MetaObjects writes no DDL + * there, and InnoDB indexes every foreign key itself. + */ +export function foreignKeyIndexesOf( + obj: MetaObject, + root: MetaRoot, + settings: SchemaBuildSettings | undefined, + dialect: Dialect, +): readonly ForeignKeyIndex[] { + if (settings?.foreignKeyIndexes !== true || dialect === "mysql") return []; + let bySettings = cache.get(root); + if (bySettings === undefined) cache.set(root, (bySettings = new WeakMap())); + let byDialect = bySettings.get(settings); + if (byDialect === undefined) bySettings.set(settings, (byDialect = new Map())); + let byFqn = byDialect.get(dialect); + if (byFqn === undefined) byDialect.set(dialect, (byFqn = build(root, settings, dialect))); + return byFqn.get(obj.resolutionKey()) ?? []; +} diff --git a/server/typescript/packages/codegen-ts/src/render-context.ts b/server/typescript/packages/codegen-ts/src/render-context.ts index ac4a111aa8..45aef1f778 100644 --- a/server/typescript/packages/codegen-ts/src/render-context.ts +++ b/server/typescript/packages/codegen-ts/src/render-context.ts @@ -3,6 +3,7 @@ import type { MetaRoot, MetaData } from "@metaobjectsdev/metadata"; import { FIELD_ATTR_OBJECT_REF, PACKAGE_SEPARATOR, TYPE_OBJECT, collectionNameOf, declaredCollectionName, physicalNamingOf, refTarget, stripPackage } from "@metaobjectsdev/metadata"; import type { Dialect } from "./column-mapper.js"; +import type { SchemaBuildSettings } from "@metaobjectsdev/migrate-ts"; import { normalizeTimestampMode } from "./dialect-module.js"; import type { PkInfo } from "./pk-resolver.js"; import type { RelationMap } from "./relation-resolver.js"; @@ -99,6 +100,11 @@ export interface RenderContext { * at the `makeRenderContext` / `normalizeConfig` choke points; see the fuller * doc comment on `MetaobjectsGenConfig.timestampMode` in metaobjects-config.ts. */ + /** + * The project's schema settings (`migrate.conventions` / `checks` / `foreignKeyIndexes`), for + * a template that maps the schema `meta migrate` builds. Absent in a bare-context render. + */ + schemaSettings?: SchemaBuildSettings; timestampMode: "date" | "string"; /** * Prepend `"use client";` to generated CLIENT artifacts (forms, hooks, grid @@ -180,6 +186,7 @@ export type RenderContextInput = Omit { selfTarget, entityModuleTarget, ...(config.providedEnumModule !== undefined && { providedEnumModule: config.providedEnumModule }), + ...(opts.schemaSettings !== undefined && { schemaSettings: opts.schemaSettings }), }); const ctx: GenContext = { entities: safeEntities, diff --git a/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts b/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts index 517f7c8715..813142ef0c 100644 --- a/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts +++ b/server/typescript/packages/codegen-ts/src/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "../tph.js"; import { flattenedMembersOf } from "../flattened.js"; import { effectivePackage } from "../docs-paths.js"; +import { foreignKeyIndexesOf } from "../foreign-key-indexes.js"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) { diff --git a/server/typescript/packages/codegen-ts/test/drizzle-foreign-key-indexes.test.ts b/server/typescript/packages/codegen-ts/test/drizzle-foreign-key-indexes.test.ts new file mode 100644 index 0000000000..7912b6a70c --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/drizzle-foreign-key-indexes.test.ts @@ -0,0 +1,70 @@ +// `migrate.foreignKeyIndexes`: the Drizzle table declares the index `meta migrate` gives each +// foreign key nothing else indexes, under migrate's name. Without the setting the table is +// what it was. +import { describe, test, expect } from "bun:test"; +import { MetaDataLoader, InMemoryStringSource } from "@metaobjectsdev/metadata"; +import { defaultConventions, efcoreConventions } from "@metaobjectsdev/migrate-ts"; +import { renderDrizzleSchema } from "../src/templates/drizzle-schema.js"; +import { makeRenderContext } from "../src/render-context.js"; +import { buildPkMap } from "../src/pk-resolver.js"; +import { buildRelationMap } from "../src/relation-resolver.js"; +import type { Dialect } from "../src/metaobjects-config.js"; +import type { SchemaBuildSettings } from "@metaobjectsdev/migrate-ts"; + +const DOC = JSON.stringify({ "metadata.root": { package: "acme", children: [ + { "object.entity": { name: "Customer", children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ] } }, + { "object.entity": { name: "Order", children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "field.long": { name: "customerId" } }, + { "field.long": { name: "billToId" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + { "identity.reference": { name: "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { name: "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "index.lookup": { name: "orders_bill_to", "@fields": ["billToId"] } }, + ] } }, +] } }); + +async function emit(dialect: Dialect, schemaSettings?: SchemaBuildSettings): Promise { + const res = await new MetaDataLoader().load([new InMemoryStringSource(DOC)]); + expect(res.errors).toEqual([]); + const ctx = makeRenderContext({ + dialect, loadedRoot: res.root, outDir: "/x", dbImport: "~/db", + pkMap: buildPkMap(res.root), relationMap: buildRelationMap(res.root), + ...(schemaSettings !== undefined ? { schemaSettings } : {}), + }); + return renderDrizzleSchema(res.root.findObject("Order")!, ctx).toString(); +} + +const ON: SchemaBuildSettings = { conventions: defaultConventions, skipRedundantChecks: false, foreignKeyIndexes: true }; + +describe("Drizzle: foreign-key indexes", () => { + test("off: no index beyond the ones the model declares", async () => { + const off = await emit("postgres", { ...ON, foreignKeyIndexes: false }); + expect(off).not.toContain("customer_id_idx"); + expect(off).toBe(await emit("postgres")); + }); + + test("on: migrate's index for each uncovered foreign key, on the column's key", async () => { + for (const dialect of ["postgres", "sqlite"] as const) { + const on = await emit(dialect, ON); + expect(on).toContain('index("orders_customer_id_idx").on(table.customerId)'); + // billToId is already led by the declared orders_bill_to index. + expect(on).not.toContain("orders_bill_to_id_idx"); + } + }); + + test("on, efcore preset: the preset's IX_ names, which are the indexes migrate creates", async () => { + const on = await emit("postgres", { ...ON, conventions: efcoreConventions }); + expect(on).toContain('index("IX_orders_customer_id").on(table.customerId)'); + expect(on).not.toContain("orders_customer_id_idx"); + }); + + test("MySQL: none, since InnoDB indexes every foreign key itself", async () => { + expect(await emit("mysql", ON)).not.toContain("customer_id_idx"); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts b/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts index d125a43561..b1d08db54b 100644 --- a/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts +++ b/server/typescript/packages/codegen-ts/test/prisma/prisma-schema.test.ts @@ -172,6 +172,14 @@ describe("prisma-schema — what the schema says", () => { .toStartWith('@relation("FK_employees_employees_manager_id", '); }); + test("migrate.foreignKeyIndexes declares the index meta migrate gives each foreign key", async () => { + const root = await loadShop(); + const blockOf = (spec: ReturnType) => spec.models.find((m) => m.name === "Employee")!.blockAttributes; + expect(blockOf(buildPrismaSchema(root, { provider: "postgresql" })).join("\n")).not.toContain("employees_manager_id_idx"); + expect(blockOf(buildPrismaSchema(root, { provider: "postgresql", schemaSettings: { foreignKeyIndexes: true } }))) + .toContain('@@index([managerId], map: "employees_manager_id_idx")'); + }); + test("the generator reads the settings from the run (GenContext.schemaSettings)", async () => { await runGen({ config: defineConfig({ outDir: tmp, dialect: "postgres", generators: [prismaSchemaFile({ provider: "postgresql" })] }), diff --git a/server/typescript/packages/integration-tests/src/migration-scenario.ts b/server/typescript/packages/integration-tests/src/migration-scenario.ts index 9422f7779d..9e2f9745b7 100644 --- a/server/typescript/packages/integration-tests/src/migration-scenario.ts +++ b/server/typescript/packages/integration-tests/src/migration-scenario.ts @@ -13,12 +13,12 @@ import { Pool } from "pg"; import { applyNearestNaming, loadMetadataDir } from "./load-metadata.ts"; import { canonicalJson, normalizeRow } from "./normalization.ts"; import { executeSql, queryRows } from "./postgres-sql.ts"; -import type { MigrationScenario } from "./scenario.ts"; +import type { MigrationScenario, MigrationSchemaSettings } from "./scenario.ts"; export async function runMigrationScenario(scenario: MigrationScenario, connectionUri: string): Promise { // 1. Bootstrap the starting schema, if any. if (scenario.seedMetadataDir) { - const seedSql = await buildFullCreate(scenario.seedMetadataDir); + const seedSql = await buildFullCreate(scenario.seedMetadataDir, scenario.seedSchemaSettings); await executeSql(connectionUri, seedSql); } // 2. Seed data. @@ -33,6 +33,7 @@ export async function runMigrationScenario(scenario: MigrationScenario, connecti // without it `programId` would become `program_id` and the bootstrap up-SQL assertion // for `weeks_programId_fk` would never match. const expected = buildExpectedSchema(targetRoot, { + ...scenario.schemaSettings, views: buildProjectionViews(targetRoot, { dialect: "postgres" }), }); @@ -79,9 +80,10 @@ export async function runMigrationScenario(scenario: MigrationScenario, connecti // Helpers // --------------------------------------------------------------------------- -async function buildFullCreate(metadataDir: string): Promise { +async function buildFullCreate(metadataDir: string, settings: MigrationSchemaSettings): Promise { const root = await loadMetadataDir(metadataDir); const expected = buildExpectedSchema(root, { + ...settings, views: buildProjectionViews(root, { dialect: "postgres" }), }); const result = await diff({ expected, actual: { tables: [], views: [] } }); diff --git a/server/typescript/packages/integration-tests/src/scenario.ts b/server/typescript/packages/integration-tests/src/scenario.ts index 7254ce2214..3cbd32a84a 100644 --- a/server/typescript/packages/integration-tests/src/scenario.ts +++ b/server/typescript/packages/integration-tests/src/scenario.ts @@ -14,9 +14,18 @@ export interface MigrationScenario { readonly seedData: string | null; readonly targetMetadataDir: string | null; readonly targetMetadataInline: string | null; + /** `seed-schema-settings`: the project schema settings the seed schema is built with. */ + readonly seedSchemaSettings: MigrationSchemaSettings; + /** `schema-settings`: the project schema settings the target is built with (`migrate.*`). */ + readonly schemaSettings: MigrationSchemaSettings; readonly expect: MigrationExpect; } +/** The `migrate` settings in `.metaobjects/config.json` a scenario can set, by their config names. */ +export interface MigrationSchemaSettings { + readonly foreignKeyIndexes?: boolean; +} + export interface MigrationExpect { readonly blocked: ReadonlyArray<{ kind: string; reasonContains: string }>; readonly upContains: ReadonlyArray; @@ -128,6 +137,8 @@ export function loadMigration(yamlPath: string): MigrationScenario { seedData: raw["seed-data"] ?? null, targetMetadataDir: resolveRel(base, raw["target-metadata"]), targetMetadataInline: raw["target-metadata-inline"] ?? null, + seedSchemaSettings: schemaSettings(raw["seed-schema-settings"], yamlPath, "seed-schema-settings"), + schemaSettings: schemaSettings(raw["schema-settings"], yamlPath, "schema-settings"), expect: { blocked: (raw.expect?.blocked ?? []).map((b) => ({ kind: b.kind ?? "", @@ -193,6 +204,17 @@ function resolveRel(base: string, rel: string | undefined | null): string | null return rel ? resolve(base, rel) : null; } +// A key this runner does not know is refused, so a typo cannot silently run the default. +function schemaSettings(raw: Record | undefined, path: string, key: string): MigrationSchemaSettings { + if (raw === undefined || raw === null) return {}; + const out: { foreignKeyIndexes?: boolean } = {}; + for (const [k, v] of Object.entries(raw)) { + if (k === "foreignKeyIndexes" && typeof v === "boolean") out.foreignKeyIndexes = v; + else throw new Error(`${path}: ${key}.${k} is not a schema setting this runner applies (known: foreignKeyIndexes, a boolean)`); + } + return out; +} + // ---------- YAML wire shapes (hyphenated keys per the fixture format) ---------- interface MigrationYaml { @@ -202,6 +224,8 @@ interface MigrationYaml { "seed-data"?: string; "target-metadata"?: string; "target-metadata-inline"?: string; + "seed-schema-settings"?: Record; + "schema-settings"?: Record; expect?: { blocked?: { kind?: string; "reason-contains"?: string }[]; "up-contains"?: string[]; diff --git a/server/typescript/packages/migrate-ts/src/drift/drift.ts b/server/typescript/packages/migrate-ts/src/drift/drift.ts index 2462473011..00116243fa 100644 --- a/server/typescript/packages/migrate-ts/src/drift/drift.ts +++ b/server/typescript/packages/migrate-ts/src/drift/drift.ts @@ -147,6 +147,7 @@ export async function computeDriftFromActual( ...(opts?.views !== undefined ? { views: opts.views } : {}), conventions: opts?.conventions, skipRedundantChecks: opts?.skipRedundantChecks, + foreignKeyIndexes: opts?.foreignKeyIndexes, }), opts?.inScope, opts?.imported !== undefined ? { imported: opts.imported } : undefined, diff --git a/server/typescript/packages/migrate-ts/src/expected-schema.ts b/server/typescript/packages/migrate-ts/src/expected-schema.ts index 3626487271..a711be5bb9 100644 --- a/server/typescript/packages/migrate-ts/src/expected-schema.ts +++ b/server/typescript/packages/migrate-ts/src/expected-schema.ts @@ -136,13 +136,22 @@ export interface BuildExpectedSchemaOptions { * absent = false, every CHECK emitted. */ skipRedundantChecks?: boolean | undefined; + /** + * Index every foreign key the conventions left without one: each enforced foreign key whose + * columns no primary key, index or earlier foreign-key index already leads with gets a plain + * index named `
_[_…]_idx`. `migrate.foreignKeyIndexes` in + * `.metaobjects/config.json`; absent = false, the schema migrate has always built. Separate + * from the conventions, so a project can have the indexes without renamed constraints; under + * the `efcore` preset every foreign key is already indexed and this adds nothing. + */ + foreignKeyIndexes?: boolean | undefined; } /** * The project settings an expected-schema build takes, which every path that builds one * (migrate, drift, the offline plan, a baseline) passes through unchanged so they agree. */ -export type SchemaBuildSettings = Pick; +export type SchemaBuildSettings = Pick; /** * A view as the caller (codegen-ts `buildProjectionViews`) produces it — structurally @@ -193,6 +202,12 @@ export interface ExpectedSchemaWithProvenance { * a write with enforcement on fails. The violations are then reported, never refused. */ fkTargetsUnenforced?: true; + /** + * The indexes the schema carries because a table has foreign keys — added by the conventions + * (`efcore`) or by `foreignKeyIndexes` — keyed by the table's qualified name. A generator that + * maps the schema (Drizzle) declares these beside the indexes the model names. Absent when none. + */ + foreignKeyIndexes?: ReadonlyMap; } /** @@ -329,7 +344,7 @@ export function buildExpectedSchemaWithProvenance( // Checked against the BUILT tables, before the conventions add FK indexes: a key the model // declares but never emits (on a TPH subtype) is no key a database can reference. const fkTargetViolations = collectFkTargetViolations(tables, fkCtx.origins, provenance); - applySchemaConventions(tables, conventions, dialect); + const foreignKeyIndexes = applySchemaConventions(tables, conventions, dialect, opts?.foreignKeyIndexes === true); checkDeclaredCheckNames(tables, dialect); // Pass 3: dialect-specific SqlType normalization. @@ -491,6 +506,7 @@ export function buildExpectedSchemaWithProvenance( return { snapshot: { tables, views }, provenance, + ...(foreignKeyIndexes.size > 0 ? { foreignKeyIndexes } : {}), ...(fkTargetViolations.length > 0 ? { fkTargetViolations } : {}), // D1 always enforces foreign keys; plain SQLite only with the pragma on. ...(fkTargetViolations.length > 0 && opts?.dialect === DIALECT_SQLITE ? { fkTargetsUnenforced: true as const } : {}), @@ -1368,7 +1384,10 @@ function collectFkTargetViolations( * foreign key gets. FK names were decided while the FKs were built, where a declared * `@constraintName` is known. The default conventions change nothing here. */ -function applySchemaConventions(tables: TableDescriptor[], conventions: SchemaConventions, dialect: Dialect | undefined): void { +function applySchemaConventions( + tables: TableDescriptor[], conventions: SchemaConventions, dialect: Dialect | undefined, indexForeignKeys: boolean, +): Map { + const added = new Map(); // An FK references a table in its own schema (the emitter's assumption), so the key is schema-qualified. const keyOf = (schema: string | undefined, table: string, columns: readonly string[]) => `${schema ?? ""}.${table}|${[...columns].sort().join(",")}`; @@ -1393,20 +1412,64 @@ function applySchemaConventions(tables: TableDescriptor[], conventions: SchemaCo indexes: t.indexes, foreignKeys: t.foreignKeys.map((fk) => ({ table: t.name, columns: fk.columns, refTable: fk.refTable, refColumns: fk.refColumns, dialect })), }); - for (const index of fkIndexes) { + const owner = (index: { columns: readonly string[] }) => + `the ${conventions.id} conventions' foreign-key index on [${index.columns.join(", ")}] of table "${t.name}"`; + const tableAdded: IndexDescriptor[] = []; + const push = (index: { name: string; columns: readonly string[] }, label: string) => { if (t.indexes.some((ix) => ix.name === index.name)) { throw new DuplicateSqlNameError([{ name: index.name, - owners: [ - `the ${conventions.id} conventions' foreign-key index on [${index.columns.join(", ")}] of table "${t.name}"`, - `an index already named "${index.name}" on table "${t.name}"`, - ], + owners: [label, `an index already named "${index.name}" on table "${t.name}"`], involvesIndex: true, }]); } - t.indexes.push({ name: index.name, columns: [...index.columns], unique: false }); + const descriptor: IndexDescriptor = { name: index.name, columns: [...index.columns], unique: false }; + t.indexes.push(descriptor); + tableAdded.push(descriptor); + }; + for (const index of fkIndexes) push(index, owner(index)); + // `foreignKeyIndexes`: after the conventions, so a preset that indexes foreign keys + // (efcore) leaves this nothing to add and its names stand. + if (indexForeignKeys) { + for (const columns of uncoveredForeignKeyColumns(t)) { + push( + { name: foreignKeyIndexName(t.name, columns), columns }, + `the foreign-key index on [${columns.join(", ")}] of table "${t.name}" (migrate.foreignKeyIndexes)`, + ); + } } + if (tableAdded.length > 0) added.set(qualifiedDbName(t), tableAdded); + } + return added; +} + +/** The name `foreignKeyIndexes` gives a foreign key's index: `
_[_…]_idx`. */ +export function foreignKeyIndexName(table: string, columns: readonly string[]): string { + return `${table}_${columns.join("_")}_idx`; +} + +/** True when `prefix` leads with exactly `columns`, in any order (the advisory's rule, too). */ +function leadsWith(prefix: readonly string[], columns: readonly string[]): boolean { + if (columns.length === 0 || prefix.length < columns.length) return false; + const want = new Set(columns); + return prefix.slice(0, columns.length).every((c) => want.has(c)); +} + +/** + * The column lists `foreignKeyIndexes` indexes on one table, in foreign-key order: each foreign + * key whose columns the primary key or a column index (a partial one counts; an expression one + * does not) does not lead with, once per column set, and none that a longer one here leads with. + * Every port's mapping generator applies this same rule. + */ +export function uncoveredForeignKeyColumns(t: Pick): string[][] { + const existing = [t.primaryKey, ...t.indexes.filter((ix) => ix.expr === undefined).map((ix) => ix.columns)]; + const candidates: string[][] = []; + for (const fk of t.foreignKeys) { + if (existing.some((cols) => leadsWith(cols, fk.columns))) continue; + if (candidates.some((c) => leadsWith(c, fk.columns) && c.length === fk.columns.length)) continue; + candidates.push([...fk.columns]); } + return candidates.filter((c) => !candidates.some((o) => o.length > c.length && leadsWith(o, c))); } diff --git a/server/typescript/packages/migrate-ts/src/expected.ts b/server/typescript/packages/migrate-ts/src/expected.ts index 076fdc4277..7c446ae334 100644 --- a/server/typescript/packages/migrate-ts/src/expected.ts +++ b/server/typescript/packages/migrate-ts/src/expected.ts @@ -6,7 +6,7 @@ // not need kysely installed. Its import graph is the model walk and nothing that opens a // connection; keep it that way. -export { buildExpectedSchema, buildExpectedSchemaWithProvenance, fieldSqlType } from "./expected-schema.js"; +export { buildExpectedSchema, buildExpectedSchemaWithProvenance, fieldSqlType, foreignKeyIndexName, uncoveredForeignKeyColumns } from "./expected-schema.js"; export { pgType } from "./emit/postgres.js"; export type { BuildExpectedSchemaOptions, SchemaBuildSettings, ExpectedSchemaWithProvenance, SchemaProvenance } from "./expected-schema.js"; export { qualifiedDbName } from "./qualified-name.js"; diff --git a/server/typescript/packages/migrate-ts/src/index.ts b/server/typescript/packages/migrate-ts/src/index.ts index 5a4c573b4b..7971148795 100644 --- a/server/typescript/packages/migrate-ts/src/index.ts +++ b/server/typescript/packages/migrate-ts/src/index.ts @@ -8,7 +8,7 @@ // See docs/specs/2026-05-11-v0.2-sp4-migrate-ts-design.md. // Pipeline functions -export { buildExpectedSchema, buildExpectedSchemaWithProvenance } from "./expected-schema.js"; +export { buildExpectedSchema, buildExpectedSchemaWithProvenance, foreignKeyIndexName, uncoveredForeignKeyColumns } from "./expected-schema.js"; export { findReferentialActionConflict, describeReferentialActionConflict, diff --git a/server/typescript/packages/migrate-ts/src/snapshot/plan.ts b/server/typescript/packages/migrate-ts/src/snapshot/plan.ts index a96e613189..576a4caaef 100644 --- a/server/typescript/packages/migrate-ts/src/snapshot/plan.ts +++ b/server/typescript/packages/migrate-ts/src/snapshot/plan.ts @@ -70,6 +70,7 @@ export async function planOffline(args: PlanOfflineArgs): Promise { test("by default: unchanged — the length check is emitted whole", async () => { const r = await root({ "@min": 0, "@max": 3 }, 3); expect(checksOf(buildExpectedSchema(r))).toEqual(['length("code") >= 0 AND length("code") <= 3']); - expect(checksOf(buildExpectedSchema(r, { skipRedundantChecks: false }))).toEqual(['length("code") >= 0 AND length("code") <= 3']); + expect(checksOf(buildExpectedSchema(r, { skipRedundantChecks: false, foreignKeyIndexes: false }))).toEqual(['length("code") >= 0 AND length("code") <= 3']); }); test("the efcore conventions alone skip nothing", async () => { diff --git a/server/typescript/packages/migrate-ts/test/unit/expected-schema-foreign-key-indexes.test.ts b/server/typescript/packages/migrate-ts/test/unit/expected-schema-foreign-key-indexes.test.ts new file mode 100644 index 0000000000..6137ac65b7 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/unit/expected-schema-foreign-key-indexes.test.ts @@ -0,0 +1,126 @@ +/** + * `foreignKeyIndexes` (`migrate.foreignKeyIndexes` in .metaobjects/config.json): every foreign key + * nothing already indexes gets a plain `
__idx` index. Off, the schema is exactly + * what migrate has always built; under the `efcore` preset every foreign key is already indexed, + * so it adds nothing and the preset's names stand. + */ +import { describe, test, expect } from "bun:test"; +import { MetaDataLoader, InMemoryStringSource } from "@metaobjectsdev/metadata"; +import { buildExpectedSchema, buildExpectedSchemaWithProvenance, uncoveredForeignKeyColumns } from "../../src/expected-schema.js"; +import { efcoreConventions } from "../../src/schema-conventions.js"; +import { emit } from "../../src/emit/index.js"; +import { diff } from "../../src/diff/index.js"; + +const DOC = { + "metadata.root": { + package: "acme", + children: [ + { "object.entity": { name: "Customer", children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "code" } }, + { "identity.primary": { name: "pk", "@fields": "id" } }, + { "identity.secondary": { name: "customers_code_key", "@fields": ["code"] } }, + ] } }, + { "object.entity": { name: "Order", children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "field.long": { name: "customerId", "@column": "customer_id" } }, + { "field.long": { name: "billToId", "@column": "bill_to_id" } }, + { "field.string": { name: "customerCode", "@column": "customer_code" } }, + { "identity.primary": { name: "pk", "@fields": "id" } }, + { "identity.reference": { name: "customer", "@fields": ["customerId"], "@references": "Customer" } }, + { "identity.reference": { name: "billTo", "@fields": ["billToId"], "@references": "Customer" } }, + { "identity.reference": { name: "byCode", "@fields": ["customerCode"], "@references": "Customer.code" } }, + // Already indexed by a declared lookup: no second index. + { "index.lookup": { name: "orders_bill_to", "@fields": ["billToId", "id"] } }, + ] } }, + // A junction: its primary key leads with orderId, so only tagId needs an index. + { "object.entity": { name: "OrderTag", children: [ + { "source.rdb": { "@table": "order_tags" } }, + { "field.long": { name: "orderId", "@column": "order_id" } }, + { "field.long": { name: "tagId", "@column": "tag_id" } }, + { "identity.primary": { name: "pk", "@fields": ["orderId", "tagId"] } }, + { "identity.reference": { name: "order", "@fields": ["orderId"], "@references": "Order" } }, + { "identity.reference": { name: "tag", "@fields": ["tagId"], "@references": "Tag" } }, + ] } }, + { "object.entity": { name: "Tag", children: [ + { "source.rdb": { "@table": "tags" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": "id" } }, + ] } }, + ], + }, +}; + +async function load() { + const { root, errors } = await new MetaDataLoader().load([new InMemoryStringSource(JSON.stringify(DOC))]); + expect(errors).toHaveLength(0); + return root; +} + +const indexes = (s: ReturnType, name: string) => + s.tables.find((t) => t.name === name)!.indexes.map((i) => [i.name, i.columns, i.unique]); + +describe("buildExpectedSchema with foreignKeyIndexes", () => { + test("off (and absent): no foreign key gets an index, as before", async () => { + const root = await load(); + expect(buildExpectedSchema(root, { foreignKeyIndexes: false })).toEqual(buildExpectedSchema(root)); + expect(indexes(buildExpectedSchema(root), "orders")).toEqual([["orders_bill_to", ["bill_to_id", "id"], false]]); + }); + + test("on: each foreign key nothing leads with gets
__idx", async () => { + const s = buildExpectedSchema(await load(), { foreignKeyIndexes: true }); + expect(indexes(s, "orders")).toEqual([ + ["orders_bill_to", ["bill_to_id", "id"], false], + ["orders_customer_id_idx", ["customer_id"], false], + ["orders_customer_code_idx", ["customer_code"], false], + ]); + expect(indexes(s, "order_tags")).toEqual([["order_tags_tag_id_idx", ["tag_id"], false]]); + }); + + test("the build reports which indexes it added, by table", async () => { + const built = buildExpectedSchemaWithProvenance(await load(), { foreignKeyIndexes: true }); + const added = [...(built.foreignKeyIndexes ?? [])].map(([t, ix]) => [t, ix.map((i) => i.name)]); + expect(added).toEqual([ + ["public.orders", ["orders_customer_id_idx", "orders_customer_code_idx"]], + ["public.order_tags", ["order_tags_tag_id_idx"]], + ]); + expect(buildExpectedSchemaWithProvenance(await load()).foreignKeyIndexes).toBeUndefined(); + }); + + test("under the efcore preset it adds nothing: the preset already indexes every foreign key", async () => { + const root = await load(); + const preset = buildExpectedSchema(root, { conventions: efcoreConventions }); + expect(buildExpectedSchema(root, { conventions: efcoreConventions, foreignKeyIndexes: true })).toEqual(preset); + }); + + test("the Postgres emitter creates the indexes", async () => { + const s = buildExpectedSchema(await load(), { foreignKeyIndexes: true }); + const { changes } = await diff(s, { tables: [], views: [] }, { dialect: "postgres" }); + const { up } = emit(changes, { dialect: "postgres" }); + expect(up).toContain('CREATE INDEX "orders_customer_id_idx" ON "orders" ("customer_id");'); + expect(up).toContain('CREATE INDEX "order_tags_tag_id_idx" ON "order_tags" ("tag_id");'); + expect(up).not.toContain('"orders_bill_to_id_idx"'); + }); +}); + +describe("uncoveredForeignKeyColumns", () => { + const fk = (...columns: string[]) => ({ name: "", columns, refTable: "t", refColumns: columns.map(() => "id") }); + + test("a key, or an index, that leads with the columns in any order covers them; an expression index does not", () => { + expect(uncoveredForeignKeyColumns({ + primaryKey: ["b", "a"], + indexes: [{ name: "x", columns: [], unique: false, expr: "lower(c)" }], + foreignKeys: [fk("a", "b"), fk("c")], + })).toEqual([["c"]]); + }); + + test("one index per column set, and none that a longer one leads with", () => { + expect(uncoveredForeignKeyColumns({ + primaryKey: ["id"], + indexes: [], + foreignKeys: [fk("a"), fk("a"), fk("a", "b"), fk("c")], + })).toEqual([["a", "b"], ["c"]]); + }); +}); diff --git a/server/typescript/packages/sdk/src/collection.ts b/server/typescript/packages/sdk/src/collection.ts index 75de1013f5..e2ae67d15c 100644 --- a/server/typescript/packages/sdk/src/collection.ts +++ b/server/typescript/packages/sdk/src/collection.ts @@ -25,7 +25,7 @@ import { type EnumDefaults, DEFAULT_ENUM_DEFAULTS, } from "@metaobjectsdev/metadata"; -import { CONFIG_FILE, enumDefaultsFromConfig, loadConfig, physicalNamingFromConfig, wireFormatFromConfig, type Config } from "./config.js"; +import { CONFIG_FILE, enumDefaultsFromConfig, loadConfig, physicalNamingFromConfig, wireFormatFromConfig, type Config, type HandEditsPolicy } from "./config.js"; import { discoverCollectionRoot, exists, isDir } from "./discovery.js"; import { compileScope, matchesScope, type Scope } from "./scope.js"; import { DEFAULT_METADATA_DIR, DEFAULT_METAOBJECTS_DIR, isMetadataFile } from "./metadata-files.js"; @@ -95,6 +95,14 @@ export interface Collection { /** `migrate.checks.skipRedundant` of the project config: leave out the validator CHECKs a * column type already enforces. False when absent. */ readonly skipRedundantChecks: boolean; + /** `migrate.foreignKeyIndexes` in `.metaobjects/config.json`: index every foreign key the + * conventions leave without one. Absent = false. Read by every expected-schema build and by + * the mapping generators. */ + readonly foreignKeyIndexes: boolean; + /** `verify.handEdits` in `.metaobjects/config.json`: whether `meta verify --codegen` FAILS on a + * hand-edited generated file (`fail`) or lists it as a notice (`notice`). Undefined when the + * project does not say, which behaves as `notice`. */ + readonly handEdits: HandEditsPolicy | undefined; /** FR-023 — the `FileSource` id each file loads under, for the dependency * ARTIFACTS only (`dep:/`); own files are absent and keep the * default `basename(path)`. Threaded to `loadMemory` so every node an artifact @@ -293,6 +301,8 @@ export async function resolveCollection( let enumDefaults: EnumDefaults = DEFAULT_ENUM_DEFAULTS; let schemaConventions: string | undefined; let skipRedundantChecks = false; + let foreignKeyIndexes = false; + let handEdits: HandEditsPolicy | undefined; if (hasConfig) { // No try/catch here: a config.json that EXISTS but fails to load @@ -313,6 +323,8 @@ export async function resolveCollection( enumDefaults = enumDefaultsFromConfig(cfg); schemaConventions = cfg.migrate?.conventions; skipRedundantChecks = cfg.migrate?.checks?.skipRedundant === true; + foreignKeyIndexes = cfg.migrate?.foreignKeyIndexes === true; + handEdits = cfg.verify?.handEdits; // An unknown token is a hard config error naming the valid ones, while // `librarySources` keeps skipping one silently for a programmatic caller. The two // are deliberately different: an API caller asking for a library this version does @@ -430,6 +442,8 @@ export async function resolveCollection( enumDefaults, schemaConventions, skipRedundantChecks, + foreignKeyIndexes, + handEdits, fileIds: new Map(dependencies.map((d) => [d.artifactPath, d.sourceId])), importedPackages: [...importedPackages].sort(), importedNodes, diff --git a/server/typescript/packages/sdk/src/config.ts b/server/typescript/packages/sdk/src/config.ts index 0139c831b7..78de4d66b4 100644 --- a/server/typescript/packages/sdk/src/config.ts +++ b/server/typescript/packages/sdk/src/config.ts @@ -68,6 +68,10 @@ const D1Block = z.object({ wranglerConfigPath: z.string(), }).partial().strict(); +/** `verify.handEdits`: what `meta verify --codegen` does with a hand-edited generated file. */ +export const HAND_EDITS_POLICIES = ["fail", "notice"] as const; +export type HandEditsPolicy = (typeof HAND_EDITS_POLICIES)[number]; + /** #192 — migration output-format adapters; orthogonal to dialect. */ const MigrateFormatEnum = z.enum(["default", "flyway"]); @@ -94,6 +98,13 @@ const MigrateBlock = z.object({ * out a CHECK the column type already enforces, for a database whose CHECKs were written by * hand only where the type does not; a CHECK with a declared `@constraintName` is kept. */ checks: z.object({ skipRedundant: z.boolean() }).partial().strict(), + /** Index every foreign key the conventions leave without one, named + * `
_[_…]_idx`. Separate from `conventions`, so a project can have the + * indexes without the `efcore` preset's constraint names. Absent = false: the schema migrate + * has always built, and `meta verify` names each unindexed foreign key instead. Read by + * `meta migrate` and by every generator that maps indexes (Drizzle, Prisma, EF Core, + * Exposed); `meta init` writes it on for a new project. */ + foreignKeyIndexes: z.boolean(), // .strict() for the same reason as the top level and the source arms below, and // with sharper teeth here: `{ migrate: { scopee: [...], dialect: "postgres" } }` // used to parse to `{ dialect: "postgres" }`, so a typo'd `scope` key meant @@ -207,6 +218,25 @@ export const ConfigSchema = z.object({ }) .default({}), migrate: MigrateBlock.optional(), + /** + * `meta verify` settings. `handEdits`: `fail` makes `meta verify --codegen` exit 1 when a + * generated file carries a hand edit (`--allow-hand-edits` relaxes that for one run); + * `notice` lists the file and exits 0 (`--forbid-hand-edits` tightens that for one run). + * Absent means `notice`, what verify has always done; `meta init` writes `fail` for a new + * project. Only the TypeScript `meta verify` reads it: the other ports' verify already fails + * on any edit to a generated file. + */ + verify: z + .strictObject({ + handEdits: z.enum(HAND_EDITS_POLICIES, { + message: `"verify.handEdits" must be one of ${HAND_EDITS_POLICIES.map((v) => `"${v}"`).join(", ")}`, + }).optional(), + }, { + error: (iss) => iss.code === "unrecognized_keys" + ? `"verify" has an unknown key "${iss.keys[0]}" (expected "handEdits")` + : iss.code === "invalid_type" ? `"verify" must be an object` : undefined, + }) + .optional(), /** * FR-023 — which dependencies' requirements this project takes on as its own ledger. * `inherit` lists scope patterns, and a pattern counts only when it NAMES a dependency's diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index fdd0ea82cd..a613c09aad 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -16,7 +16,7 @@ // shows `resolveCollection`, the documented single authority for locating a project. // Config -export { ConfigSchema, DEFAULT_CONFIG, loadConfig, setConfigWarningSink, splitUnknownTopLevelKeys, CONFIG_TOP_LEVEL_KEYS, saveConfig, AllowTokenEnum, physicalNamingFromConfig, wireFormatFromConfig, enumDefaultsFromConfig } from "./config.js"; +export { ConfigSchema, DEFAULT_CONFIG, loadConfig, setConfigWarningSink, splitUnknownTopLevelKeys, CONFIG_TOP_LEVEL_KEYS, saveConfig, AllowTokenEnum, physicalNamingFromConfig, wireFormatFromConfig, enumDefaultsFromConfig, HAND_EDITS_POLICIES, type HandEditsPolicy } from "./config.js"; export type { Config } from "./config.js"; // Metadata dependencies (FR-023) — the `dependencies` config key's schema, diff --git a/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts b/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts index f943b82eff..e35ed5146a 100644 --- a/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts +++ b/server/typescript/packages/test-generators/src/templates/drizzle-schema.ts @@ -35,6 +35,7 @@ import { collectTphSubtypeFields, tphConcreteSubtypes } from "./tph-discriminato import { tphStorageObject } from "@metaobjectsdev/codegen-ts/engine"; import { flattenedMembersOf } from "@metaobjectsdev/codegen-ts/engine"; import { effectivePackage } from "@metaobjectsdev/codegen-ts/engine"; +import { foreignKeyIndexesOf } from "@metaobjectsdev/codegen-ts/engine"; /** * Render the Drizzle table definition for one entity, including: @@ -228,6 +229,9 @@ export function renderDrizzleSchema( // core naming (`resolveFlattenedColumnName`), the name `meta migrate` creates; the // `Names` constant when the artifact carries it. No CHECK constraint: migrate derives // CHECKs from the entity's own fields only. + // Database column → the Drizzle column key, for the foreign-key indexes below (they come off + // migrate's schema, in column terms). + const keyByColumn = new Map(); const pushFlattenedColumns = ( owner: MetaField, members: NonNullable>, @@ -235,6 +239,7 @@ export function renderDrizzleSchema( for (const m of members) { const base = mapColumnType(m.flat, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); const spec: ColumnSpec = { ...base, dbName: m.column }; + keyByColumn.set(m.column, m.key); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -266,6 +271,7 @@ export function renderDrizzleSchema( // Compute the column spec once per field and reuse it for both the column // line and the CHECK collection. const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -286,6 +292,7 @@ export function renderDrizzleSchema( // #213 — a TPH subtype's derived field is read-only too; never a table column. if (child.isDerived()) continue; const spec = mapColumnType(child, ctx.dialect, ctx.columnNamingStrategy, ctx.timestampMode); + keyByColumn.set(spec.dbName, child.name); if (spec.enumIntCustomType !== undefined) { enumIntTypes.set(spec.enumIntCustomType.fnConstName, spec.enumIntCustomType); } @@ -414,6 +421,17 @@ export function renderDrizzleSchema( if (entry !== undefined) callbackEntries.push(entry); } + // `migrate.foreignKeyIndexes`: the index migrate gives each foreign key nothing else indexes, + // under migrate's name, so the mapping declares the indexes the database has. Off (no entry) + // for a project without the setting. + for (const fkIndex of foreignKeyIndexesOf(obj, ctx.loadedRoot, ctx.schemaSettings, dialect)) { + const keys = fkIndex.columns.map((c) => keyByColumn.get(c)); + if (keys.some((k) => k === undefined)) continue; + callbackEntries.push( + code`${imp(`index@${importModule}`)}(${JSON.stringify(fkIndex.name)}).on(${keys.map((k) => `table.${k}`).join(", ")})`, + ); + } + // Emit table-level CHECK constraints for enum fields. Both halves are composed by // `checkEntry` above, which owns the constant-vs-literal decision for the pair. for (const { name, expr } of checkConstraints) { From 1cc17ac3df90f1b60e8ec17246adb5f5b7f15543 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 13 Nov 2026 14:59:43 -0500 Subject: [PATCH 2/4] no-mistakes(review): Document that custom schemaConventions reach only the TS toolchain --- docs/features/migrations-and-drift.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/features/migrations-and-drift.md b/docs/features/migrations-and-drift.md index 14963b2267..4d02469e15 100644 --- a/docs/features/migrations-and-drift.md +++ b/docs/features/migrations-and-drift.md @@ -535,6 +535,11 @@ export default defineConfig({ }); ``` +Such an object reaches only the TypeScript toolchain: `meta migrate`, `meta verify --db`, and +the TypeScript generators resolve it. The C# and Kotlin generators read only the +`migrate.conventions` preset name from `.metaobjects/config.json` and apply the default rule for +every derived name, foreign-key indexes included. + **Changing the conventions of an existing database.** On Postgres, a primary key, FK, CHECK or index whose definition is unchanged and whose name differs is renamed in place (`RENAME CONSTRAINT` / `ALTER INDEX … RENAME`), never dropped and re-created; new FK indexes are created. @@ -589,6 +594,12 @@ setting, so the mapping and the database agree: Java (OMDB is data access only and maps no indexes) and Python (Pydantic models and routes, no ORM mapping) generate nothing that declares an index, so the option changes no output there. +A project that pairs its own conventions object with `migrate.foreignKeyIndexes` and a C# or +Kotlin mapping should make the object's `foreignKeyIndexes` match the default rule (index each +foreign key nothing leads with, named `
__idx`) or leave the option off; +otherwise the C# DbContext or Kotlin Exposed tables may declare an index `meta migrate` did not +create. + ## How CHECK constraints and expression indexes are compared Postgres rewrites an expression before it stores it (`"k" IN ('x')` becomes `k = 'x'::text`, From f404e1cc8d8283b1d01e6ff26500780e6b401c69 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 13 Nov 2026 16:45:31 -0500 Subject: [PATCH 3/4] no-mistakes(document): docs: note FK-index option in conventions table --- docs/features/migrations-and-drift.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/migrations-and-drift.md b/docs/features/migrations-and-drift.md index 4d02469e15..85ab3b7588 100644 --- a/docs/features/migrations-and-drift.md +++ b/docs/features/migrations-and-drift.md @@ -496,7 +496,7 @@ come from the project's **schema conventions**, selected in `.metaobjects/config |---|---|---| | Primary-key constraint name (Postgres; SQLite and D1 emit an unnamed primary key) | `
_pkey` | `PK_
` | | Foreign-key constraint name | `
__fk` | `FK_
__` | -| Index on a foreign key | none | `IX_
_`, unless the primary key or another index (a partial one included) already starts with the FK's columns; an FK index another FK index covers is dropped, so FK order does not matter | +| Index on a foreign key | none (unless [`migrate.foreignKeyIndexes`](#foreign-key-indexes-migrateforeignkeyindexes) is on, which gives each uncovered one `
__idx`) | `IX_
_`, unless the primary key or another index (a partial one included) already starts with the FK's columns; an FK index another FK index covers is dropped, so FK order does not matter | | Unique key (`identity.secondary`, a `@unique` field) | a unique index; a live UNIQUE constraint is also accepted | a UNIQUE constraint when a foreign key references it, else a unique index | `efcore` is a reference preset. It reproduces one ORM's documented naming to show what the From 3f4d75cd2e650bbda5c86fb7beb1e718e734bee0 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 13 Nov 2026 16:48:05 -0500 Subject: [PATCH 4/4] no-mistakes(document): docs: link conventions note to FK-index option --- docs/features/metadata-sources.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/features/metadata-sources.md b/docs/features/metadata-sources.md index 690ce0155a..1155ce259b 100644 --- a/docs/features/metadata-sources.md +++ b/docs/features/metadata-sources.md @@ -355,8 +355,9 @@ out-of-scope table that already exists would become a proposed `DROP TABLE`. ### `migrate.conventions` — names for objects no code addresses Beside `scope`, the `migrate` block takes `conventions`: the name of a schema-conventions preset -that decides primary-key and foreign-key constraint names, whether a foreign key gets an index, -and whether a unique key is an index or a constraint. `default` (absent) is what migrate has +that decides primary-key and foreign-key constraint names, whether a foreign key gets an index +(without a preset, [`migrate.foreignKeyIndexes`](migrations-and-drift.md#foreign-key-indexes-migrateforeignkeyindexes) +indexes foreign keys), and whether a unique key is an index or a constraint. `default` (absent) is what migrate has always emitted; `efcore` is a reference preset that reproduces EF Core's naming. A project can supply its own conventions from `metaobjects.config.ts` instead. Table, view and column names are not conventions: they come from `naming`. See