|
| 1 | +# D1 FK-cascade auto-fix — rebuild referenced tables on remote D1 (#241) |
| 2 | + |
| 3 | +- **Date:** 2026-07-28 |
| 4 | +- **Status:** Design approved; ready for implementation plan |
| 5 | +- **Issue:** [#241](https://github.com/metaobjectsdev/metaobjects/issues/241) — auto-migrate referenced-table rebuilds via FK cascade (follow-up to [#226](https://github.com/metaobjectsdev/metaobjects/issues/226)) |
| 6 | +- **Surface:** `@metaobjectsdev/migrate-ts` (D1 emitter) + a small `@metaobjectsdev/cli` call-site change. TS-only; no other-port impact. |
| 7 | + |
| 8 | +## Problem |
| 9 | + |
| 10 | +On remote Cloudflare D1, the SQLite table-rebuild recipe cannot rebuild a table that |
| 11 | +another table's foreign key references: D1 runs the migration inside an implicit |
| 12 | +transaction where `PRAGMA foreign_keys = OFF` is a no-op, so `DROP TABLE <referenced>` |
| 13 | +fails with `FOREIGN KEY constraint failed`. [#226](https://github.com/metaobjectsdev/metaobjects/issues/226) |
| 14 | +made the D1 emitter **refuse** that case at generation time (safe, but the user must |
| 15 | +hand-write the migration). This issue replaces the refusal with a **cascade** that |
| 16 | +generates a correct, appliable migration — and closes the residual under-refuse gap |
| 17 | +#226 documented. |
| 18 | + |
| 19 | +## Prototype evidence |
| 20 | + |
| 21 | +Throwaway libSQL prototypes (single connection, the emitted batch wrapped in one |
| 22 | +explicit transaction with `foreign_keys = ON` — a faithful model of remote D1's |
| 23 | +implicit transaction) validated the cascade **10/10** across single-parent, |
| 24 | +transitive (`g→c→p`), multi-child, and self-referential graphs: each commits with row |
| 25 | +data intact and foreign keys re-enforced. The same prototypes established that |
| 26 | +`PRAGMA defer_foreign_keys = ON` alone cannot rescue a parent rebuild, and that |
| 27 | +multi-table cycles have no reliable atomic sequence (they must be refused). |
| 28 | + |
| 29 | +## The cascade algorithm |
| 30 | + |
| 31 | +When a migration would rebuild an FK-referenced table (and the FK graph over the |
| 32 | +affected set is acyclic — self-loops excepted), the D1 emitter emits one pass: |
| 33 | + |
| 34 | +```sql |
| 35 | +PRAGMA defer_foreign_keys = ON; -- belt for self-referential / lazy checks |
| 36 | +-- for each table t in the affected set A: |
| 37 | +CREATE TABLE __f_<t> ( <t's final schema> ); -- FK cols reference __f_<ref> when ref ∈ A, else <ref>'s real name |
| 38 | +-- for each t in A: |
| 39 | +INSERT INTO __f_<t> ( <carry cols> ) SELECT <mapped cols> FROM <t>; |
| 40 | +-- DROP old tables, REFERRERS-FIRST (reverse-topological): |
| 41 | +DROP TABLE <t>; |
| 42 | +-- RENAME temps, PARENTS-FIRST (topological) — RENAME rewrites __f_ FK refs to final names: |
| 43 | +ALTER TABLE __f_<t> RENAME TO <t>; |
| 44 | +``` |
| 45 | + |
| 46 | +The two ordering rules are load-bearing: referrers-first `DROP` avoids dropping a |
| 47 | +still-referenced table; parents-first `RENAME` lets SQLite rewrite each temp-name FK to |
| 48 | +its final target. Forward-references in the `CREATE` statements are fine — SQLite |
| 49 | +resolves FK *target* tables lazily. |
| 50 | + |
| 51 | +The D1 safety pass strips the (now unnecessary) explicit transaction control, relying |
| 52 | +on D1's implicit transaction for atomicity; the useless `PRAGMA foreign_keys = OFF/ON` |
| 53 | +bracket is not emitted on the cascade path. |
| 54 | + |
| 55 | +## Affected set, ordering, cycles |
| 56 | + |
| 57 | +- **Affected set `A` = (tables the diff rebuilds) ∪ (all tables that transitively |
| 58 | + reference a table in `A`)**, where "reference" is the **union of the actual and |
| 59 | + expected foreign-key graphs**. Using the union (not expected-only) is what closes |
| 60 | + #226's residual gap: a migration that rebuilds a parent *and* drops a child's FK to |
| 61 | + it in the same run — the child references the parent in the *actual* schema, so it |
| 62 | + must be rebuilt (and dropped) before the parent, which only the actual graph reveals. |
| 63 | +- A table pulled into `A` only as a referrer (not itself changed) is rebuilt to its |
| 64 | + **expected descriptor** (identical to its current schema, so a re-diff converges), |
| 65 | + including its columns, indexes, and foreign keys. |
| 66 | +- **Ordering** is a topological sort of the union FK graph restricted to `A`. |
| 67 | +- **Multi-table cycles** (`A→B→…→A`, length ≥ 2) have no reliable atomic FKs-on |
| 68 | + sequence and are **refused** (the topological sort detects them). Self-loops |
| 69 | + (self-referential FKs) are excluded from the cycle check — the temp-name mechanism |
| 70 | + handles them. The existing `D1ReferencedTableRebuildError` is reused/renamed for the |
| 71 | + cycle-refusal message (naming the cycle and the hand-write workaround). |
| 72 | + |
| 73 | +## Threading the actual schema |
| 74 | + |
| 75 | +`EmitOptions` gains `actualSchema?: SchemaSnapshot`. Both `meta migrate` emit call |
| 76 | +sites already hold it: the live-introspection path passes `actual` (the introspected |
| 77 | +snapshot given to `diff`), and the offline path passes the prior committed `snapshot`. |
| 78 | +`renderD1` uses it (unioned with `expectedSchema`) to build the FK graph. When absent |
| 79 | +(e.g. a caller that doesn't supply it), the emitter falls back to the expected graph |
| 80 | +and **refuses** any referenced rebuild it cannot prove safe — never emits an unproven |
| 81 | +cascade. |
| 82 | + |
| 83 | +## Where it lives |
| 84 | + |
| 85 | +- New `src/emit/d1-cascade.ts`: the reverse FK-graph builder, topological sort + cycle |
| 86 | + detection, and the cascade emitter. It reuses `src/emit/sqlite.ts` primitives |
| 87 | + (`renderCreateTable`, the column carry/remap logic from `renderRecreate`, |
| 88 | + `renderCreateIndex`), extracted read-only without changing SQLite behavior. |
| 89 | +- `src/emit/d1.ts` (`renderD1`): when `recreatedTables` contains a referenced table, |
| 90 | + dispatch to the cascade for an acyclic graph, or throw the cycle-refusal for a cycle. |
| 91 | + The no-referenced-rebuild common path stays exactly as today — **byte-identical**. |
| 92 | +- `src/emit/index.ts` (`EmitOptions` + `emit`) and the two `cli` emit call sites: add |
| 93 | + and pass `actualSchema`. |
| 94 | +- No change to `emit/postgres.ts` or the local-SQLite path. |
| 95 | + |
| 96 | +## Down migrations |
| 97 | + |
| 98 | +As today, the recreate/cascade down migration is best-effort — the emitter emits a |
| 99 | +`-- WARNING` block (data-destructive reversals cannot be auto-generated). Unchanged |
| 100 | +from the existing SQLite recreate behavior. |
| 101 | + |
| 102 | +## Testing |
| 103 | + |
| 104 | +Extend the libSQL real-engine gate (`test/integration/`, one-transaction model = remote |
| 105 | +D1), each scenario asserting: the cascade applies cleanly, row data is intact, foreign |
| 106 | +keys are re-enforced afterward, and a re-diff is **EMPTY** (the migrate-engine |
| 107 | +doctrine): |
| 108 | + |
| 109 | +- single parent + populated child; |
| 110 | +- transitive chain `g→c→p`; |
| 111 | +- multiple children of one parent; |
| 112 | +- self-referential table; |
| 113 | +- **the #226 residual gap**: one migration that rebuilds a parent AND drops a child's |
| 114 | + FK to it — applies cleanly (this is the case that regresses if the actual graph |
| 115 | + isn't used); |
| 116 | +- **multi-table cycle** `A↔B` → refused at generation time (unit-level); |
| 117 | +- a no-referenced-rebuild migration → D1 output byte-identical to today. |
| 118 | + |
| 119 | +Plus unit tests for the FK-graph builder, topological ordering, and cycle detection. |
| 120 | + |
| 121 | +## Non-goals |
| 122 | + |
| 123 | +- Multi-table foreign-key cycles on D1 (refused, documented). |
| 124 | +- Any change to the local SQLite or Postgres emitters. |
| 125 | +- Changing `applyPending` / replay semantics. |
| 126 | + |
| 127 | +## Docs |
| 128 | + |
| 129 | +Remove #226's "Known limitation of the current refusal" note from |
| 130 | +`docs/features/migrations-and-drift.md` (the gap is closed) and update the D1 |
| 131 | +limitation subsection to describe the cascade (with cycles as the remaining |
| 132 | +hand-write case). CHANGELOG entry. Close #241. |
0 commit comments