Skip to content

fix(codegen-ts): emit one Drizzle column per member for flattened value objects - #538

Merged
dmealing merged 12 commits into
mainfrom
fm/mo-issue-524-drizzle-flattened
Nov 6, 2026
Merged

dmealing merged 12 commits into
mainfrom
fm/mo-issue-524-drizzle-flattened

Conversation

@dmealing

@dmealing dmealing commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

What Changed

  • codegen-ts now binds a field.object @storage: flattened to its member columns, named by resolveFlattenedColumnName, instead of one jsonb column for the field. This matches what meta migrate creates, so generated queries no longer fail with undefined-column errors (codegen-ts: Drizzle binds a flattened value object as one jsonb column that migrate never creates #524).
  • Zod schemas, Row/Insert types, the filter and sort allowlists, the OpenAPI parameters and the React form and TanStack grid columns use flat keys (e.g. shipToStreet). A flattened field on a write-through entity is refused with a CodegenError. A flattened field on a TPH subtype keeps a single column, as migrate does.
  • Adds the shared flattened.ts helper, the matching template and example updates, and tests (flattened-storage-e2e.test.ts, form-flattened-storage.test.ts). Adds a CHANGELOG entry under Fixed.
Validation

Risk Assessment

✅ Low: After eight fix rounds, the flat surface is consistent across Drizzle, Zod, types, filter and sort, OpenAPI, the docs, the form and the grid, and it matches migrate's columns, with end-to-end coverage. The one remaining gap (a derived flattened parent) is a narrow edge case.

Testing

Ran the e2e and unit tests for flattened storage, then drove the real meta CLI on a throwaway sqlite project. meta gen emitted Drizzle columns shipToStreet, shipToCity and so on, in line with the ship_to_* and bill_to_* columns that meta migrate created. Queries and Hono routes then worked against that DB. Inserts, updates, reads and listing returned flat keys. A filter on shipToCity and a sort by shipToCity worked. A filter on shipTo or on the non-filterable shipToStreet was rejected with a 400. The generated code passed tsc --strict. The full scripts/ci-local.sh --quick --strict-toolchains passed; it skips the other-language ports, the Java reactor and the docker suite. The Postgres dialect check ran only through the e2e test's parity assertion, not live. Scratch files were removed and the worktree is clean.

  • Live validation: ✅ go - 6 of 6 scenarios driven live against the product
Scenario Result Live Evidence
meta gen emits one Drizzle column per member of a flattened value object, matching the columns meta migrate creates ✅ pass live Real meta gen and meta migrate --apply on a throwaway project: up.sql has ship_to_street/city/floor and bill_to_*; generated Shipment.ts has shipToStreet etc. bound to the same names, and no `ship…
Generated queries create, update, find and list rows with flat keys against the migrated DB with no undefined-column error ✅ pass live live-queries.txt; the raw sqlite row shows ship_to_* populated
Generated Hono routes accept flat POST and PATCH bodies and filter and sort on shipToCity ✅ pass live live-routes.txt: POST 201, filter returns the Zagreb row, sort ascending puts Athens first, PATCH billToCity 200
Adversarial: filter on nested name shipTo or non-filterable member shipToStreet is rejected, and a nested insert body does not write members ✅ pass live live-routes.txt: both filters return 400 invalid_filter_field with allowed [shipToCity, billToCity]; live-queries.txt: a nested shipTo insert is stripped without error, so the members stay null
Zod schemas, Row/Insert types and filter allowlist use flat keys and compile ✅ pass live Generated Shipment.ts shows flat Zod and allowlist keys; tsc --strict over src/generated gave no errors
Regression: flattened-storage e2e (including Postgres parity), names, form tests and ci-local --quick ✅ pass live ci-quick.log: LOCAL CI PASSED; targeted tests 31 pass, 0 fail
Evidence: Live queries against migrated sqlite DB
created {"id":1,"ref":"R1","shipToStreet":"1 Main","shipToCity":"Leeds","shipToFloor":3,"billToStreet":"9 Bill","billToCity":"York","billToFloor":null}
updated {"id":1,"ref":"R1","shipToStreet":"1 Main","shipToCity":"Bath","shipToFloor":3,"billToStreet":"9 Bill","billToCity":"York","billToFloor":null}
found {"id":1,"ref":"R1","shipToStreet":"1 Main","shipToCity":"Bath","shipToFloor":3,"billToStreet":"9 Bill","billToCity":"York","billToFloor":null}
list 1
raw [{"id":1,"ref":"R1","ship_to_street":"1 Main","ship_to_city":"Bath","ship_to_floor":3,"bill_to_street":"9 Bill","bill_to_city":"York","bill_to_floor":null}]
nested-insert {"id":2,"ref":"R2","shipToStreet":null,"shipToCity":null,"shipToFloor":null,"billToStreet":null,"billToCity":null,"billToFloor":null}
Evidence: Live Hono routes (POST/PATCH/filter/sort/rejected filters)
POST 201 {"id":1,"ref":"A","shipToStreet":"s1","shipToCity":"Zagreb","shipToFloor":null,"billToStreet":null,"billToCity":"Oslo","billToFloor":null}
POST 201 {"id":2,"ref":"B","shipToStreet":null,"shipToCity":"Athens","shipToFloor":null,"billToStreet":null,"billToCity":null,"billToFloor":null}
GET filter 200 [{"id":1,"ref":"A","shipToStreet":"s1","shipToCity":"Zagreb","shipToFloor":null,"billToStreet":null,"billToCity":"Oslo","billToFloor":null}]
GET sort 200 [{"id":2,"ref":"B","shipToStreet":null,"shipToCity":"Athens","shipToFloor":null,"billToStreet":null,"billToCity":null,"billToFloor":null},{"id":1,"ref":"A","shipToStreet":"s1","shipToCity":"Zagreb","shipToFloor":null,"billToStreet":null,"billToCity":"Oslo","billToFloor":null}]
GET bad filter (nested name) 400 {"error":"invalid_filter_field","field":"shipTo","allowed":["shipToCity","billToCity"]}
GET non-filterable member 400 {"error":"invalid_filter_field","field":"shipToStreet","allowed":["shipToCity","billToCity"]}
PATCH 200 {"id":1,"ref":"A","shipToStreet":"s1","shipToCity":"Zagreb","shipToFloor":null,"billToStreet":null,"billToCity":"Bergen","billToFloor":null}
Evidence: ci-local --quick full log
metaobjects local CI  (mode: quick — TS + drift gates only)

── ▶ leak-scan (security) ──────────────────────────────────────────────
leak-scan: clean
── ✓ leak-scan (security) (0s)

── ▶ pom-version parity ──────────────────────────────────────────────
check-pom-versions: ✓ 3 parent + 1 property pom(s) match reactor 8.2.0
── ✓ pom-version parity (0s)

── ▶ bun-version parity ──────────────────────────────────────────────
check-bun-version: bun 1.3.14 matches the workflow pin.
── ✓ bun-version parity (0s)

── ▶ uv.lock version parity ──────────────────────────────────────────────
check-uv-lock-version: uv.lock agrees with pyproject.toml (1.2.0)
── ✓ uv.lock version parity (0s)

── ▶ publish-intent parity ──────────────────────────────────────────────
publish-intent parity: OK (lockstep 1.2.0; 2 declared source-only)
── ✓ publish-intent parity (0s)

── ▶ publish-set parity ──────────────────────────────────────────────
   1. @metaobjectsdev/metadata  (server/typescript/packages/metadata)
   2. @metaobjectsdev/render  (server/typescript/packages/render)
   3. @metaobjectsdev/docs-site  (server/typescript/packages/docs-site)
   4. @metaobjectsdev/migrate-ts  (server/typescript/packages/migrate-ts)
   5. @metaobjectsdev/codegen-ts  (server/typescript/packages/codegen-ts)
   6. @metaobjectsdev/runtime-ts  (server/typescript/packages/runtime-ts)
   7. @metaobjectsdev/sdk  (server/typescript/packages/sdk)
   8. @metaobjectsdev/runtime-web  (client/web/packages/runtime-web)
   9. @metaobjectsdev/codegen-ts-react  (server/typescript/packages/codegen-ts-react)
  10. @metaobjectsdev/codegen-ts-tanstack  (server/typescript/packages/codegen-ts-tanstack)
  11. @metaobjectsdev/react  (client/web/packages/react)
  12. @metaobjectsdev/tanstack  (client/web/packages/tanstack)
  13. @metaobjectsdev/cli  (server/typescript/packages/cli)
  14. @metaobjectsdev/ai-runtime  (server/typescript/packages/ai-runtime)
publish set: OK (lockstep 1.2.0; 14 packages, tier-ordered, closed over sibling deps)
ok:   happy path: private + off-lockstep excluded, order [metadata, render, docs-site, cli]
ok:   untiered member: threw on "publish order undeclared"
ok:   unclosed set: threw on "not closed over its own runtime dependencies"
ok:   inverted tier order: threw on "publishes a dependency AFTER its dependent"
ok:   regression: docs-site (#2) publishes before cli (#12) in the real tree
ok:   regression: docs-site has a declared tier

publish-set tests: all passed
── ✓ publish-set parity (0s)

── ▶ no committed pre-release version ──────────────────────────────────────────────
check-no-prerelease-versions: ✓ no pre-release version in any version declaration
── ✓ no committed pre-release version (1s)

── ▶ script-name hook collisions ──────────────────────────────────────────────
script-name-hooks: ✓ 10 root scripts, no lifecycle-hook collisions
── ✓ script-name hook collisions (0s)

── ▶ ci lane selection ──────────────────────────────────────────────
ok:   [server/typescript/packages/cli/src/x.ts] -> 'ts'
ok:   [client/web/packages/react/src/y.tsx] -> 'ts'
ok:   [server/java/metadata/src/main/java/A.java] -> 'java'
ok:   [server/java/codegen-kotlin/src/main/kotlin/B.kt] -> 'java'
ok:   [server/python/src/metaobjects/loader.py] -> 'python'
ok:   [server/csharp/MetaObjects/Loader.cs] -> 'csharp'
ok:   [fixtures/conformance/foo/meta.json] -> 'ts java python csharp'
ok:   [spec/metamodel.md] -> 'ts java python csharp'
ok:   [scripts/ci-local.sh] -> 'ts java python csharp'
ok:   [.github/workflows/local-ci.yml] -> 'ts java python csharp'
ok:   [agent-context/skills/x.md] -> 'ts java python csharp'
ok:   [weird-new-toplevel/file.txt] -> 'ts java python csharp'
ok:   [docs/features/cli.md] -> ''
ok:   [README.md CHANGELOG.md] -> ''
ok:   [server/typescript/a.ts server/python/b.py] -> 'ts python'
ok:   [docs/x.md fixtures/y.json] -> 'ts java python csharp'
ALL PASS
ok:   union all -> 'ts java python csharp'
ok:   union reorders to canonical -> 'ts java python csharp'
ok:   union deduplicates -> 'ts java'
ok:   union single -> 'python'
ok:   union empty -> ''
ok:   union drops unknown tokens -> 'ts'
ok:   every lane green -> nothing to re-run -> ''
ok:   docs-only run skips a red java -> java still not green -> 'java'
ok:   one red ts lane selects the whole ts port -> 'ts'
ok:   skipped is walked past to the real verdict -> ''
ok:   cancelled is not green -> 'csharp'
ok:   a lane with no verdict in the window is not green -> 'python'
ok:   no history at all -> everything -> 'ts java python csharp'
ok:   every port-testing job in .github/workflows/local-ci.yml is mapped by lane_port -> ''
ok:   every lane in lane_port exists as a job in .github/workflows/local-ci.yml -> ''
all checks passed
── ✓ ci lane selection (0s)

── ▶ extract field kinds ──────────────────────────────────────────────
extract FieldKind: 4 ports agree on [STRING, INT, LONG, DOUBLE, BOOLEAN, ENUM, OBJECT] (1 sanctioned deviation(s): csharp)
── ✓ extract field kinds (0s)

── ▶ no-magic gate wired (5 ports) ──────────────────────────────────────────────
no-magic gate coverage: 5 ports wired; 13 shapes tracked, 0 known gaps
── ✓ no-magic gate wired (5 ports) (0s)

── ▶ test-file references resolve ──────────────────────────────────────────────
test-file references: 33 named, all resolve
── ✓ test-file references resolve (0s)

── ▶ metamodel-version bump ──────────────────────────────────────────────
  metamodel-version: additive change since v1.2.0; declared 1.3 (1.2 → 1.3) — ok.
ok:   no change classifies as nothing
ok:   a removed subtype is BREAKING
ok:   an added subtype is ADDITIVE
ok:   a removed attr is BREAKING
ok:   an added OPTIONAL attr is ADDITIVE
ok:   an added REQUIRED attr is BREAKING
ok:   optional → required is BREAKING
ok:   required → optional is ADDITIVE
ok:   a changed valueType is BREAKING
ok:   a changed isArray is BREAKING
ok:   an enum member REMOVED is BREAKING
ok:   an enum member ADDED is ADDITIVE
ok:   an OPEN attr becoming a closed enum is BREAKING
ok:   a closed enum OPENING is ADDITIVE
ok:   a removed child rule is BREAKING
ok:   an added OPTIONAL child rule is ADDITIVE
ok:   an added MANDATORY child rule is BREAKING
ok:   raising min is BREAKING
ok:   lowering min is ADDITIVE
ok:   capping an unbounded max is BREAKING
ok:   lifting a cap is ADDITIVE
ok:   lowering a finite max is BREAKING
ok:   a removed commonAttr is BREAKING
ok:   a changed default subtype is BREAKING (it changes what an unqualified declaration MEANS)
ok:   a removed default subtype is BREAKING
ok:   an added default subtype is ADDITIVE
ok:   a type description change is PROSE ONLY
ok:   a `rules` change is PROSE ONLY — this is exactly #210, and why prose only warns
ok:   an attr description change is PROSE ONLY
ok:   no change requires no bump
ok:   additive requires a minor

... [500761 bytes truncated] ...

   if (true) {

[Survived] ConditionalExpression
src/super-resolve.ts:295:11
-         if (parsed !== undefined) {
+         if (true) {

[Survived] ConditionalExpression
src/super-resolve.ts:297:13
-           if (ownerNode !== undefined && ownerNode !== "over-drop") resolveNode(ownerNode);
+           if (true) resolveNode(ownerNode);

[Survived] LogicalOperator
src/super-resolve.ts:297:13
-           if (ownerNode !== undefined && ownerNode !== "over-drop") resolveNode(ownerNode);
+           if (ownerNode !== undefined || ownerNode !== "over-drop") resolveNode(ownerNode);

[Survived] ConditionalExpression
src/super-resolve.ts:297:13
-           if (ownerNode !== undefined && ownerNode !== "over-drop") resolveNode(ownerNode);
+           if (true && ownerNode !== "over-drop") resolveNode(ownerNode);

[Survived] ConditionalExpression
src/super-resolve.ts:297:40
-           if (ownerNode !== undefined && ownerNode !== "over-drop") resolveNode(ownerNode);
+           if (ownerNode !== undefined && true) resolveNode(ownerNode);

[Survived] StringLiteral
src/super-resolve.ts:297:54
-           if (ownerNode !== undefined && ownerNode !== "over-drop") resolveNode(ownerNode);
+           if (ownerNode !== undefined && ownerNode !== "") resolveNode(ownerNode);

[Survived] ObjectLiteral
src/super-resolve.ts:333:21
-             referrer: { type: node.type, subType: node.subType },
+             referrer: {},

[Survived] ObjectLiteral
src/super-resolve.ts:332:19
-             target: { type: target.type, subType: target.subType },
+             target: {},

[Survived] StringLiteral
src/super-resolve.ts:351:15
-           kind: "unresolved",
+           kind: "",

Ran 0.98 tests per mutant on average.
--------------------|------------------|----------|-----------|------------|----------|----------|
                    | % Mutation score |          |           |            |          |          |
File                |  total | covered | # killed | # timeout | # survived | # no cov | # errors |
--------------------|--------|---------|----------|-----------|------------|----------|----------|
All files           |  55.52 |   55.52 |      944 |        37 |        786 |        0 |        0 |
 parser-core.ts     |  46.79 |   46.79 |      473 |        16 |        556 |        0 |        0 |
 provider.ts        |  62.12 |   62.12 |       40 |         1 |         25 |        0 |        0 |
 serializer-json.ts |  65.11 |   65.11 |      193 |        16 |        112 |        0 |        0 |
 subtype-rules.ts   |  69.23 |   69.23 |      117 |         0 |         52 |        0 |        0 |
 super-resolve.ts   |  75.30 |   75.30 |      121 |         4 |         41 |        0 |        0 |
--------------------|--------|---------|----------|-----------|------------|----------|----------|
^[[32m19:48:33 (3951637) INFO MutationTestReportHelper^[[39m Final mutation score of 55.52 is greater than or equal to break threshold 35
(path-bearing lines of the local CI log omitted; the mutation and doc-template drift steps passed.)
^[[32m19:48:33 (3951637) INFO MutationTestExecutor^[[39m Done in 12 minutes and 39 seconds.
── ✓ completeness-gate (mutation) (12m39s)

── ▶ doc-template drift ──────────────────────────────────────────────
── ✓ doc-template drift (1s)

── ▶ embedded-library drift ──────────────────────────────────────────────
wrote 6 embedded library file(s) + 2 manifest(s) to server/typescript/packages/metadata/src/library/embedded-library.generated.ts, server/java/metadata/src/main/java/com/metaobjects/library/EmbeddedLibrary.java and server/csharp/MetaObjects/Library/EmbeddedLibrary.cs
── ✓ embedded-library drift (0s)

── ⊘ --quick: SKIPPING the other-language ports (csharp/java/python/kotlin),
        the full Java reactor, the docker integration suite and the cube live
        check. Run the full `scripts/ci-local.sh` (no flag) before opening/merging
        a PR or tagging.

══════════════════════ SUMMARY ══════════════════════
  ✓ leak-scan (security)
  ✓ pom-version parity
  ✓ bun-version parity
  ✓ uv.lock version parity
  ✓ publish-intent parity
  ✓ publish-set parity
  ✓ no committed pre-release version  (1s)
  ✓ script-name hook collisions
  ✓ ci lane selection
  ✓ extract field kinds
  ✓ no-magic gate wired (5 ports)
  ✓ test-file references resolve
  ✓ metamodel-version bump
  ✓ positioning claims
  ✓ hosted assess prompt builds
  ✓ migration guides vs registry
  ✓ dependencies installed
  ✓ peer-range bounds
  ✓ owned template copies current
  ✓ reference templates lint  (1s)
  ✓ shipped doc examples load  (1s)
  ✓ no retired $apiPrefix
  ✓ site payload is true  (26s)
  ✓ site reference is fresh
  ✓ release tag gate  (11s)
  ✓ scripts/ typecheck  (5s)
  ✓ requirements ledger verifies  (2s)
  ✓ requirements cover vocabulary  (1s)
  ✓ metamodel scaffolder  (1s)
  ✓ requirement harness
  ✓ fixture-lint
  ✓ ts build + typecheck  (24s)
  ✓ conformance: typescript  (4m13s)
  ✓ ts unit suites  (19s)
  ✓ completeness-gate (mutation)  (12m39s)
  ✓ doc-template drift  (1s)
  ✓ embedded-library drift
  ⊘ csharp/java/python/kotlin conformance (--quick)
  ⊘ java-reactor (--quick)
  ⊘ integration-tests (--quick)
  ⊘ cube-model live check (--quick)
── slowest steps ──
    12m39s  completeness-gate (mutation)
     4m13s  conformance: typescript
       26s  site payload is true
       24s  ts build + typecheck
       19s  ts unit suites
       11s  release tag gate
        5s  scripts/ typecheck
        2s  requirements ledger verifies
        1s  doc-template drift
        1s  metamodel scaffolder
        1s  no committed pre-release version
        1s  reference templates lint
  total wall time: 18m25s
══════════════════════════════════════════════════════
LOCAL CI PASSED (with 4 skipped — see ⊘ above).

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ℹ️ server/typescript/packages/codegen-ts/src/flattened.ts:260 - flatSurfaceFields expands a flattened field.object even when the parent field is derived (carries an origin.*), and asFlatKey forwards isDerived() to the member, not the parent. The Drizzle table (drizzle-schema.ts:362) and meta migrate (expected-schema.ts:636) both skip a derived parent, so no member columns exist. The flat surface still offers shipToStreet etc. as non-derived keys. They appear in the read, insert and update schemas (zod-validators.ts:603, :611, :621), the filter and sort allowlists, the OpenAPI bodies and the form. Before this change the derived parent was excluded from the write schemas. Nothing errors: a write silently drops the keys, and the Row/read contract claims columns that do not exist. The case is narrow, but the loader allows origin.* on an entity field. Fix: in flatSurfaceFields, return [f] unchanged (or [], to match Drizzle) when f.isDerived(), or make the proxy report the parent's isDerived().
    This ships as a separate follow-up PR right after this one merges, not in this PR.
✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 6 of 6 scenarios driven live against the product
Scenario Result Live Evidence
meta gen emits one Drizzle column per member of a flattened value object, matching the columns meta migrate creates ✅ pass live Real meta gen and meta migrate --apply on a throwaway project: up.sql has ship_to_street/city/floor and bill_to_*; generated Shipment.ts has shipToStreet etc. bound to the same names, and no `ship…
Generated queries create, update, find and list rows with flat keys against the migrated DB with no undefined-column error ✅ pass live live-queries.txt; the raw sqlite row shows ship_to_* populated
Generated Hono routes accept flat POST and PATCH bodies and filter and sort on shipToCity ✅ pass live live-routes.txt: POST 201, filter returns the Zagreb row, sort ascending puts Athens first, PATCH billToCity 200
Adversarial: filter on nested name shipTo or non-filterable member shipToStreet is rejected, and a nested insert body does not write members ✅ pass live live-routes.txt: both filters return 400 invalid_filter_field with allowed [shipToCity, billToCity]; live-queries.txt: a nested shipTo insert is stripped without error, so the members stay null
Zod schemas, Row/Insert types and filter allowlist use flat keys and compile ✅ pass live Generated Shipment.ts shows flat Zod and allowlist keys; tsc --strict over src/generated gave no errors
Regression: flattened-storage e2e (including Postgres parity), names, form tests and ci-local --quick ✅ pass live ci-quick.log: LOCAL CI PASSED; targeted tests 31 pass, 0 fail
  • scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains
  • bun test test/flattened-storage-e2e.test.ts test/names-flattened.test.ts in codegen-ts (30 pass)
  • bun test test/form-flattened-storage.test.ts in codegen-ts-react (1 pass)
  • Live: meta init, meta gen, meta migrate --from-db --apply and meta migrate apply-pending on a throwaway sqlite project with two flattened field.object fields; generated entity, queries and routes inspected and tsc --strict run over them
  • Live: ran the generated createShipment, updateShipment, findShipmentById and listShipments against the migrated sqlite DB (bun drive.ts)
  • Live: mounted the generated Hono routes and sent POST, PATCH, GET with filter and sort, and rejected-filter requests against the migrated DB (bun drive2.ts)
  • scripts/ci-local.sh --quick --strict-toolchains (exit 0, LOCAL CI PASSED, 4 gates skipped by design: other-language ports, Java reactor, docker suite, cube check)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Fixes #524

Gate runs: 3. Run 1 failed on the gate daemon failing to fetch main (ssh permission denied), run 2 died in the test step on a model usage limit, and run 3 passed. Prisma already exposes flat keys (shipToStreet) and was not changed.

@dmealing
dmealing merged commit f53b8f2 into main Nov 6, 2026
9 checks passed
@dmealing
dmealing deleted the fm/mo-issue-524-drizzle-flattened branch November 6, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

codegen-ts: Drizzle binds a flattened value object as one jsonb column that migrate never creates

1 participant