Skip to content

feat: page size, unknown-field 400s and validation issues in every port - #580

Merged
dmealing merged 35 commits into
mainfrom
fm/mo-py-input-fix-main
Nov 14, 2026
Merged

dmealing merged 35 commits into
mainfrom
fm/mo-py-input-fix-main

Conversation

@dmealing

@dmealing dmealing commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

What Changed

  • Generated list routes in all five ports read api.pagination (defaultLimit, maxLimit) from .metaobjects/config.json. A list with no limit returns the default page. A limit above the cap answers 400 limit_too_large, and a negative limit answers 400 pagination.invalid_value. meta init writes 50/1000 for new projects. A project with no setting keeps today's behaviour, and meta verify advises (unbounded-list-pages) when routes are generated without one.
  • Generated create, PUT and PATCH routes refuse a body key the entity does not declare with 400 {"error":"unknown_fields","fields":[...]}, before validation and before any write. This is the new default, so existing projects get it on regenerate. "api": {"unknownFields": "drop"} restores silent dropping. Declared server-owned keys (generated id, readOnly, derived, @autoSet, the TPH discriminator) are still dropped silently. Python routes pass the repository only validated input, carrying forward the 1.2.1 fix, and Java, Kotlin, C# and TypeScript close the same create-path input holes in their own idioms.
  • A validation failure answers 400 {"error":"validation","issues":[{code,path,message}]} in every port, listing every failure instead of a bare {"error":"validation"}. The hand-rolled reference servers match, and shared api-contract scenarios gate both the generated and reference lanes.
Validation

Risk Assessment

⚠️ Medium: The change is a very large cross-port API behavior change (350 files); the incremental fixes made after the last review pass (C# limit and offset refusal, Python test removals) are correct, but I did not re-read the rest of the changed files in this pass.

Pipeline

Updates from git push no-mistakes

No-mistakes runs: 1. It passed with one recorded Test exception (see Test below). The triage bundle-size check was overridden by decision: this is one coherent API-trust bundle, and the captain's 2026-11-12 rule is never to split a coherent change by line count.

Go-live item A5: the render-raises cherry-pick c8425953f stays in this branch on purpose.

Review (10 rounds). Every warning was fixed in the run:

  • The Kotlin TPH per-subtype create now qualifies its column writes.
  • @JsonIgnoreProperties(ignoreUnknown = true) now sits only on entities that have a generated write route and the value objects nested in them. Output-parser targets, prompt payloads, stored-proc args and projections keep the host mapper's strictness (operator decision).
  • The Python TPH update drops the body discriminator, as create does.
  • A C# limit or offset that is not a whole number, or is out of range, now answers 400 pagination.invalid_value, so the client never gets a silent default page (operator decision).
  • The rest were vacuous assertions, stale comments, doc counts, and the eject runtime lists in own-your-codegen.md.

The remaining info-level notes were declined by decision:

  • Source-substring assertions in generator tests are left as they are. Executed api-contract lanes and compile-and-run tests cover the same behaviour.
  • The dotnet meta init / metaobjects init migrate/verify keys stay. Init parity is in scope, and meta migrate reads migrate.foreignKeyIndexes for these projects too.

Test:

  • The first test turn caught a stale owned copy (test-generators/src/templates/zod-validators.ts), and 62faf0204 re-synced it.
  • The configured command (scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains) then passed at the run head.
  • The live Python and TypeScript generated APIs were driven over HTTP: unknown and server-owned keys dropped or refused, page default, cap, limit=0 and negative limits, and validation issue lists.
  • Recorded exception: the test agent's own ci-local --quick run was cut off by its time budget. The same scripts/ci-local.sh --quick was rerun separately on that head (1dfe213cb) and passed (EXIT 0). The step was approved on that evidence.

Document: the C# and Python port pages are corrected to say that init writes .metaobjects/config.json. Consolidating the restated page-size, unknown-fields and issue-shape facts into one owner page is assigned to the follow-up docs PR.

Lint / Push: passed. CI is local for this repo, and the hosted checks run on the PR.

… routes (#566)

* fix(python): generated routes hand the repository only the validated input (1.2.1)

The generated FastAPI create, PATCH and PUT handlers validated the body against
<Entity>Create / <Entity>Patch and then passed the RAW body to the repository.
Pydantic ignores undeclared keys and the input models leave out server-owned
fields, so a client-sent id, a readOnly field and a made-up field such as
isAdmin all reached the write. Security fix.

Each handler (plain and TPH per-subtype) now rebinds dto to the keys the input
model validated, at every depth, before any repository call, as the TypeScript
routes do. Undeclared and non-settable keys are dropped silently, never 400'd,
so a client sending an extra harmless field keeps working; values stay the
body's own, so repositories see the same value types as in 1.2.0.

Python package version 1.2.1 only; CHANGELOG entry under Security.
Regression gate: tests/integration/test_generated_router_validated_input.py.

* no-mistakes(document): python routes pass only validated input

* no-mistakes(lint): regenerate showcase site payload for PyPI 1.2.1
… in every port

An audit after the Python 1.2.1 fix found the same request-input hole in the
other ports' generated create handlers. Each port now drops, silently and before
the write, a server-generated key, readOnly and derived fields, and on a TPH
per-subtype route the discriminator and sibling-subtype columns; undeclared keys
are dropped everywhere.

- Python: forward-port of the 1.2.1 fix; TPH create drops a body discriminator.
- Java: create nulls the dropped DTO components; DTOs ignore unknown properties.
- Kotlin: create writes only settable columns; entity classes ignore unknown keys.
- C#: create resets the generated key, discriminator and M:N navigations; every
  readOnly property gets a private setter.
- TypeScript: an assigned primary key leaves the update schema.

Shared api-contract scenarios input-server-owned-dropped (core) and
tph-input-server-owned-dropped, a readOnly column in secret-fields, and an
`absent` assertion in every runner gate it.
The same model answered a failed create or update differently by language:
TypeScript listed each problem, while C#, Java, Kotlin and Python returned a
bare {"error":"validation"}. Every port's generated routes and its hand-rolled
reference servers now answer 400 {"error":"validation","issues":[{code,path,
message}]} with every failure, codes in Zod's vocabulary and path as an array of
wire names.

- C#: ValidationIssues runtime file (ejected with routes); create reads the
  body itself so an unbindable body lists its issue; PATCH collects instead of
  returning on the first failure.
- Java: ValidationIssues runtime helper; a Patch binds every field before it
  throws. Kotlin: per-package MetaValidationIssues.kt; create lists every
  absent required value.
- Python: every Pydantic error mapped in constraint_errors; decimal failures
  carry their wire path.

Shared scenarios validation-lists-issues and tph-validation-lists-issues, with a
new `issues` assertion in every runner, gate it in both lanes.
…eScript)

List endpoints had no hard cap: a TypeScript or C# list with no `limit`
returned the whole table and `?limit=100000` was accepted everywhere. A project
can now set `"api": { "pagination": { "defaultLimit": N, "maxLimit": M } }` in
.metaobjects/config.json: a generated entity or projection list with no
`limit` answers the default page, and a `limit` above the cap answers
400 {"error":"limit_too_large","maxLimit":M}. Reports keep their own row cap
and opaque views their 1000.

- sdk reads and validates the block; the loaded root carries it
  (apiPaginationOf), as it carries the wire format.
- runtime-ts mounts take a `pagination` option (pageLimit); the packaged and
  reference Fastify and Hono routes emit it. Owned copies re-synced.
- meta init writes 50 / 1000 for a new project; `--force` over an existing
  config does not add it. A project without the setting behaves as before, and
  meta verify advises (rule unbounded-list-pages) when routes are wired.
- New api-contract `pagination/` sub-corpus, run in the TypeScript generated
  lane. The C#, Java, Kotlin and Python routes do not read it yet: their REST
  generators are under the non-TS freeze, pending that ruling.

A config key, not vocabulary: metamodelVersion does not move.
…t by default)

api.pagination (default page size and hard cap on generated list routes)
now reaches the C#, Java, Kotlin and Python generators as well as
TypeScript: no limit answers the default page, a limit over the cap is
400 limit_too_large. No setting keeps today's behaviour. verify advises
when routes are generated without it (TypeScript, Python, Java/Kotlin,
and C# under verify --codegen).

api.unknownFields (reject | drop, default reject): a generated create,
PUT or PATCH body carrying a key the entity does not declare answers
400 {"error":"unknown_fields","fields":[...]} in every port, before
validation, and writes nothing. drop keeps the old silent behaviour.
Declared server-owned keys (id, readOnly, @autoset, writeOnce on update,
the TPH discriminator, a sibling subtype's column) are still dropped, not
refused, so a client echoing back a row keeps working. Java and Kotlin
check the raw body, so the answer does not depend on the host's Jackson
mapper; a test proves it under both FAIL_ON_UNKNOWN_PROPERTIES values.

The pagination sub-corpus becomes api-settings/ and runs in every port's
generated lane; the core and tph corpora gain reject scenarios. meta init
writes both settings into a fresh config.
# Conflicts:
#	CHANGELOG.md
#	server/python/src/metaobjects/codegen/generators/router_generator.py
…ion issue, a negative limit refused under api.pagination

- unknown_fields lists keys in code-point order in every port (TypeScript and
  C# sorted by UTF-16 code unit, which put a key outside the BMP first).
- A validation 400 keeps two distinct failures of one rule kind on one field
  (Python and C# de-duplicated on path and code); a type error still lists once.
- Under api.pagination a negative limit answers 400 pagination.invalid_value in
  every port instead of reaching the store (SQLite reads LIMIT -1 as no limit).
  New api-settings/limit-negative scenario.
- Python validated-input tests follow api.unknownFields; advanced-modeling
  example regenerated; C# routes generator header names the current contract.
# Conflicts:
#	CHANGELOG.md
#	docs/features/api-contract.md
…2.x security) (#568)

* fix: generated create routes store only client-settable fields (1.2.x security)

Python 1.2.1 stopped its generated routes from handing raw request input to
the repository. The same audit found related create-path holes in 1.2.0 in
the other four ports; this closes each in its own idiom. Server-owned values
are dropped silently, never answered with 400.

- Java (codegen-spring): create strips a generated id, readOnly and derived
  fields; a TPH per-subtype create also strips the body discriminator and
  sibling-subtype fields.
- Kotlin (codegen-kotlin): create skips readOnly/derived columns; a TPH
  per-subtype create writes only the route subtype's own columns.
- C# (MetaObjects.Codegen): every readOnly property shape gets a private
  setter; create resets a generated id, the TPH discriminator and M:N
  navigations before Add.
- TypeScript (codegen-ts): the update schema omits primary-key fields, so
  PUT/PATCH can no longer re-key a row with an assigned key.

Proof: new shared api-contract scenarios (core + TPH input-server-owned,
TPH foreign discriminator) and a readOnly `role` field in secret-fields, run
in every port's generated lane and the reference lanes; a new `unset`
assertion key in all five harnesses. In-memory test repositories now store
what they are handed, so the scenarios fail on the 1.2.0 generators.

Proposed versions (not published): npm 1.2.2, NuGet 1.2.2, Maven 8.2.2;
PyPI unchanged at 1.2.1. metamodelVersion unchanged.

* fix: keep a @required readOnly field creatable in C# and Java generated create

Pre-gate review follow-ups to the 1.2.x create-input patch:

- C#: the required-key check skips readOnly fields (the private setter
  never binds them), and create validation ignores a violation naming
  only a @required readOnly member, as Java's SERVER_OWNED_ON_CREATE does.
- Java: the create-validation server-owned set covers every DTO component,
  so a nulled readOnly or derived value-object column no longer 400s.
- TypeScript: drop the now-dead PK guard in the update-schema nullable arm
  and list the PK rule in updateSchemaFields' doc comment.

* no-mistakes(review): apply reviewer inline fixes

* no-mistakes(review): Kotlin readOnly create fix + C# reset skip

* no-mistakes(document): docs: finish stale corpus counts from input fix
…install every module the Kotlin integration lane needs

meta init --refresh-docs --skills-dir <new> removed the untouched skill files at
the old path but left their directories behind, which read as the old copy still
being there. Walk up from each pruned file and remove the directories it left
empty, stopping at the first that still holds anything and never removing the
skills directory itself.

scripts/integration-test.sh kotlin installed only metadata and codegen-kotlin, so
on an .m2 without the other metaobjects artifacts integration-tests-kotlin failed
to resolve core-spring and om, and codegen-kotlin-exposed1x-check metadata-ktx.
Every generated render path in the five ports already raises
ERR_VAR_NOT_ON_PAYLOAD for a template variable not on the payload
(1.3.0, #520), and runtime-drift/ held the document and prompt helpers.
The @kind: email helper was not in the case. Add WelcomeEmail plus
email-subject/, email-html/ and email-text/ drift directories, one
drifted part each, and check every part raises in the TS, Python, C#,
Java and Kotlin runners.
….3.0 input work

Carries forward every server-owned-on-create path from the 1.2.x line: the C#
readOnly key/discriminator reset guard and RequiredReadOnlyProps (now through
ValidationIssues.ValidateObjectExcept), the Kotlin isDerivedField skip and
KotlinServerOwnedCreateBindTest, the Java dtoComponentFields create loop, and the
tph-input-foreign-discriminator-dropped scenario. Scenarios that sent an undeclared
key expecting a drop keep the 1.3.0 api.unknownFields reject answer instead.
… TypeScript meta init writes

A new project gets api.pagination 50/1000 and api.unknownFields reject in
.metaobjects/config.json from every port's init; an existing config is never edited.
# Conflicts:
#	docs/CONFORMANCE.md
#	server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java
#	server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java
#	server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java
#	server/typescript/packages/cli/src/commands/init.ts
#	server/typescript/packages/cli/test/init.test.ts
#	server/typescript/packages/sdk/src/collection.ts
#	server/typescript/packages/sdk/src/index.ts
…foreignKeyIndexes and verify.handEdits

The same new-project keys TypeScript meta init writes since #577.
…the api config cases

- New jsonb scenario jsonb-nested-undeclared-key-dropped: api.unknownFields judges the top
  level only, so a member a value object does not declare is dropped on create and PATCH and
  never stored, while the open jsonb bag keeps every key. The reference servers in every port
  drop such members; the Java generated value-object record now carries
  @JsonIgnoreProperties(ignoreUnknown = true) (it answered 400 invalid_type).
- New shared config corpus api-settings/config-blocks.json (valid and refused api blocks), run
  by the TypeScript, C#, Java and Python config readers. A page size is a positive whole number
  however JSON spells it, at most 2147483647: C# now reads 50.0, TypeScript and Python refuse
  anything above the int32 range.
- CHANGELOG: an Upgrading section (api.unknownFields reject as the one deliberate default change,
  the C# readOnly private setter), init parity across ports, the nested-member rule. Counts,
  READMEs, api-contract.md and the migration guide updated; showcase Java regenerated.
- C#: with api.pagination set, `limit=0` served the default page; it is now a page of zero rows,
  as in every other port. New shared scenario api-settings/limit-zero (all five ports green).
- C#: the unknown-field check matches a key the way the route binds the body (the app's
  System.Text.Json options; the ASP.NET web default ignores case), so `Title` is the declared
  `title`, not a 400. The verify advisory matches the routes generator by type or exact name,
  not by a "routes" substring.
- Python: a rejected decimal's text is no longer echoed into the 400 issue message; the
  unbounded-list-pages advisory fires only when the routes generator is wired; the dead
  _DECIMAL_COERCE constant is gone.
- Java: UnreadableBody's Javadoc describes the issues envelope it returns.
- Docs: M:N traversal routes take no limit and are not paged by api.pagination (stated as the
  boundary); the C# port doc no longer says dotnet meta init writes no config. Counts updated.
…p path coercion in Python issues

Simplify pass: AbstractMetaDataMojo maps its four project settings from one read of
.metaobjects/config.json, and pydantic_issues copies loc as-is (its entries are already int or str).
…ference change

The input-handling change edited the codegen-ts reference source but not the
owned copy under test-generators, so cli/test/owned-copies-current.test.ts
(the 'owned copies are what meta eject writes' gate) failed: the copy on
disk kept the pre-change update-shape text. bun scripts/sync-owned-template-copies.ts
re-synced 1 of 128 owned copies; no other copy drifted.
…t tests

The TPH per-subtype create now writes every column through the table object
(it[AuthTable.autoCreatedAt]), so a column named body, issues or unknownKeys
cannot be shadowed by the handler's locals. Two KotlinAutoSetStampingTest
cases still asserted the unqualified form and failed the java-slow lane.
@dmealing
dmealing merged commit 00b08dc into main Nov 14, 2026
9 checks passed
@dmealing
dmealing deleted the fm/mo-py-input-fix-main branch November 14, 2026 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant