Skip to content

Commit 00b08dc

Browse files
authored
feat: page size, unknown-field 400s and validation issues in every port (#580)
* fix(python): pass only validated input to the repository in generated routes (#566) * fix(python): generated routes hand the repository only the validated input (1.2.1) The generated FastAPI create, PATCH and PUT handlers validated the body against <Entity>Create / <Entity>Patch and then passed the RAW body to the repository. Pydantic ignores undeclared keys and the input models leave out server-owned fields, so a client-sent id, a readOnly field and a made-up field such as isAdmin all reached the write. Security fix. Each handler (plain and TPH per-subtype) now rebinds dto to the keys the input model validated, at every depth, before any repository call, as the TypeScript routes do. Undeclared and non-settable keys are dropped silently, never 400'd, so a client sending an extra harmless field keeps working; values stay the body's own, so repositories see the same value types as in 1.2.0. Python package version 1.2.1 only; CHANGELOG entry under Security. Regression gate: tests/integration/test_generated_router_validated_input.py. * no-mistakes(document): python routes pass only validated input * no-mistakes(lint): regenerate showcase site payload for PyPI 1.2.1 * fix(codegen): generated routes drop server-owned and undeclared input in every port An audit after the Python 1.2.1 fix found the same request-input hole in the other ports' generated create handlers. Each port now drops, silently and before the write, a server-generated key, readOnly and derived fields, and on a TPH per-subtype route the discriminator and sibling-subtype columns; undeclared keys are dropped everywhere. - Python: forward-port of the 1.2.1 fix; TPH create drops a body discriminator. - Java: create nulls the dropped DTO components; DTOs ignore unknown properties. - Kotlin: create writes only settable columns; entity classes ignore unknown keys. - C#: create resets the generated key, discriminator and M:N navigations; every readOnly property gets a private setter. - TypeScript: an assigned primary key leaves the update schema. Shared api-contract scenarios input-server-owned-dropped (core) and tph-input-server-owned-dropped, a readOnly column in secret-fields, and an `absent` assertion in every runner gate it. * feat(codegen): a validation 400 lists every issue in every port The same model answered a failed create or update differently by language: TypeScript listed each problem, while C#, Java, Kotlin and Python returned a bare {"error":"validation"}. Every port's generated routes and its hand-rolled reference servers now answer 400 {"error":"validation","issues":[{code,path, message}]} with every failure, codes in Zod's vocabulary and path as an array of wire names. - C#: ValidationIssues runtime file (ejected with routes); create reads the body itself so an unbindable body lists its issue; PATCH collects instead of returning on the first failure. - Java: ValidationIssues runtime helper; a Patch binds every field before it throws. Kotlin: per-package MetaValidationIssues.kt; create lists every absent required value. - Python: every Pydantic error mapped in constraint_errors; decimal failures carry their wire path. Shared scenarios validation-lists-issues and tph-validation-lists-issues, with a new `issues` assertion in every runner, gate it in both lanes. * feat(api): a page size for generated list routes, api.pagination (TypeScript) List endpoints had no hard cap: a TypeScript or C# list with no `limit` returned the whole table and `?limit=100000` was accepted everywhere. A project can now set `"api": { "pagination": { "defaultLimit": N, "maxLimit": M } }` in .metaobjects/config.json: a generated entity or projection list with no `limit` answers the default page, and a `limit` above the cap answers 400 {"error":"limit_too_large","maxLimit":M}. Reports keep their own row cap and opaque views their 1000. - sdk reads and validates the block; the loaded root carries it (apiPaginationOf), as it carries the wire format. - runtime-ts mounts take a `pagination` option (pageLimit); the packaged and reference Fastify and Hono routes emit it. Owned copies re-synced. - meta init writes 50 / 1000 for a new project; `--force` over an existing config does not add it. A project without the setting behaves as before, and meta verify advises (rule unbounded-list-pages) when routes are wired. - New api-contract `pagination/` sub-corpus, run in the TypeScript generated lane. The C#, Java, Kotlin and Python routes do not read it yet: their REST generators are under the non-TS freeze, pending that ruling. A config key, not vocabulary: metamodelVersion does not move. * feat(api): api.pagination in every port, and api.unknownFields (reject by default) api.pagination (default page size and hard cap on generated list routes) now reaches the C#, Java, Kotlin and Python generators as well as TypeScript: no limit answers the default page, a limit over the cap is 400 limit_too_large. No setting keeps today's behaviour. verify advises when routes are generated without it (TypeScript, Python, Java/Kotlin, and C# under verify --codegen). api.unknownFields (reject | drop, default reject): a generated create, PUT or PATCH body carrying a key the entity does not declare answers 400 {"error":"unknown_fields","fields":[...]} in every port, before validation, and writes nothing. drop keeps the old silent behaviour. Declared server-owned keys (id, readOnly, @autoset, writeOnce on update, the TPH discriminator, a sibling subtype's column) are still dropped, not refused, so a client echoing back a row keeps working. Java and Kotlin check the raw body, so the answer does not depend on the host's Jackson mapper; a test proves it under both FAIL_ON_UNKNOWN_PROPERTIES values. The pagination sub-corpus becomes api-settings/ and runs in every port's generated lane; the core and tph corpora gain reject scenarios. meta init writes both settings into a fresh config. * chore(showcase): regenerate for api.unknownFields * fix(api): code-point order for unknown_fields, every distinct validation issue, a negative limit refused under api.pagination - unknown_fields lists keys in code-point order in every port (TypeScript and C# sorted by UTF-16 code unit, which put a key outside the BMP first). - A validation 400 keeps two distinct failures of one rule kind on one field (Python and C# de-duplicated on path and code); a type error still lists once. - Under api.pagination a negative limit answers 400 pagination.invalid_value in every port instead of reaching the store (SQLite reads LIMIT -1 as no limit). New api-settings/limit-negative scenario. - Python validated-input tests follow api.unknownFields; advanced-modeling example regenerated; C# routes generator header names the current contract. * fix: generated create routes drop server-owned input across ports (1.2.x security) (#568) * fix: generated create routes store only client-settable fields (1.2.x security) Python 1.2.1 stopped its generated routes from handing raw request input to the repository. The same audit found related create-path holes in 1.2.0 in the other four ports; this closes each in its own idiom. Server-owned values are dropped silently, never answered with 400. - Java (codegen-spring): create strips a generated id, readOnly and derived fields; a TPH per-subtype create also strips the body discriminator and sibling-subtype fields. - Kotlin (codegen-kotlin): create skips readOnly/derived columns; a TPH per-subtype create writes only the route subtype's own columns. - C# (MetaObjects.Codegen): every readOnly property shape gets a private setter; create resets a generated id, the TPH discriminator and M:N navigations before Add. - TypeScript (codegen-ts): the update schema omits primary-key fields, so PUT/PATCH can no longer re-key a row with an assigned key. Proof: new shared api-contract scenarios (core + TPH input-server-owned, TPH foreign discriminator) and a readOnly `role` field in secret-fields, run in every port's generated lane and the reference lanes; a new `unset` assertion key in all five harnesses. In-memory test repositories now store what they are handed, so the scenarios fail on the 1.2.0 generators. Proposed versions (not published): npm 1.2.2, NuGet 1.2.2, Maven 8.2.2; PyPI unchanged at 1.2.1. metamodelVersion unchanged. * fix: keep a @required readOnly field creatable in C# and Java generated create Pre-gate review follow-ups to the 1.2.x create-input patch: - C#: the required-key check skips readOnly fields (the private setter never binds them), and create validation ignores a violation naming only a @required readOnly member, as Java's SERVER_OWNED_ON_CREATE does. - Java: the create-validation server-owned set covers every DTO component, so a nulled readOnly or derived value-object column no longer 400s. - TypeScript: drop the now-dead PK guard in the update-schema nullable arm and list the PK rule in updateSchemaFields' doc comment. * no-mistakes(review): apply reviewer inline fixes * no-mistakes(review): Kotlin readOnly create fix + C# reset skip * no-mistakes(document): docs: finish stale corpus counts from input fix * fix(cli): prune emptied skill directories after a --skills-dir move; install every module the Kotlin integration lane needs meta init --refresh-docs --skills-dir <new> removed the untouched skill files at the old path but left their directories behind, which read as the old copy still being there. Walk up from each pruned file and remove the directories it left empty, stopping at the first that still holds anything and never removing the skills directory itself. scripts/integration-test.sh kotlin installed only metadata and codegen-kotlin, so on an .m2 without the other metaobjects artifacts integration-tests-kotlin failed to resolve core-spring and om, and codegen-kotlin-exposed1x-check metadata-ktx. * test(render): hold the email helper in the runtime-drift corpus case Every generated render path in the five ports already raises ERR_VAR_NOT_ON_PAYLOAD for a template variable not on the payload (1.3.0, #520), and runtime-drift/ held the document and prompt helpers. The @kind: email helper was not in the case. Add WelcomeEmail plus email-subject/, email-html/ and email-text/ drift directories, one drifted part each, and check every part raises in the TS, Python, C#, Java and Kotlin runners. * feat(init): dotnet meta init and metaobjects init write the api block TypeScript meta init writes A new project gets api.pagination 50/1000 and api.unknownFields reject in .metaobjects/config.json from every port's init; an existing config is never edited. * feat(init): dotnet meta init and metaobjects init also write migrate.foreignKeyIndexes and verify.handEdits The same new-project keys TypeScript meta init writes since #577. * feat(api): drop undeclared value-object members everywhere and share the api config cases - New jsonb scenario jsonb-nested-undeclared-key-dropped: api.unknownFields judges the top level only, so a member a value object does not declare is dropped on create and PATCH and never stored, while the open jsonb bag keeps every key. The reference servers in every port drop such members; the Java generated value-object record now carries @JsonIgnoreProperties(ignoreUnknown = true) (it answered 400 invalid_type). - New shared config corpus api-settings/config-blocks.json (valid and refused api blocks), run by the TypeScript, C#, Java and Python config readers. A page size is a positive whole number however JSON spells it, at most 2147483647: C# now reads 50.0, TypeScript and Python refuse anything above the int32 range. - CHANGELOG: an Upgrading section (api.unknownFields reject as the one deliberate default change, the C# readOnly private setter), init parity across ports, the nested-member rule. Counts, READMEs, api-contract.md and the migration guide updated; showcase Java regenerated. * fix(api): close the code-review findings on the input and page-size work - C#: with api.pagination set, `limit=0` served the default page; it is now a page of zero rows, as in every other port. New shared scenario api-settings/limit-zero (all five ports green). - C#: the unknown-field check matches a key the way the route binds the body (the app's System.Text.Json options; the ASP.NET web default ignores case), so `Title` is the declared `title`, not a 400. The verify advisory matches the routes generator by type or exact name, not by a "routes" substring. - Python: a rejected decimal's text is no longer echoed into the 400 issue message; the unbounded-list-pages advisory fires only when the routes generator is wired; the dead _DECIMAL_COERCE constant is gone. - Java: UnreadableBody's Javadoc describes the issues envelope it returns. - Docs: M:N traversal routes take no limit and are not paged by api.pagination (stated as the boundary); the C# port doc no longer says dotnet meta init writes no config. Counts updated. * refactor: read the project config once in the Maven mojo; drop a no-op path coercion in Python issues Simplify pass: AbstractMetaDataMojo maps its four project settings from one read of .metaobjects/config.json, and pydantic_issues copies loc as-is (its entries are already int or str). * no-mistakes(review): Scope ignoreUnknown to request bodies; fix review items * no-mistakes(review): Drop discriminator on Python TPH update; fix scenario text * no-mistakes(review): apply reviewer inline fixes * no-mistakes(review): Fix doc counts, dead guard, stale comments, vacuous assertion * no-mistakes(review): apply reviewer inline fixes * no-mistakes(review): Fix stale routes comment in C# api-settings factory * no-mistakes(review): Fix stale eject docs, vacuous C# check, assertion messages * no-mistakes(review): Refuse non-numeric C# page bounds with 400 * no-mistakes(review): Refuse out-of-range or negative C# offset * no-mistakes(review): Drop source-grepping Python router tests * no-mistakes(review): apply reviewer inline fixes * test(ts): re-sync the owned zod-validators copy after the pkFields reference change The input-handling change edited the codegen-ts reference source but not the owned copy under test-generators, so cli/test/owned-copies-current.test.ts (the 'owned copies are what meta eject writes' gate) failed: the copy on disk kept the pre-change update-shape text. bun scripts/sync-owned-template-copies.ts re-synced 1 of 128 owned copies; no other copy drifted. * no-mistakes(test): Validate re-synced zod-validators owned copy * no-mistakes(test): apply changes made during test validation * no-mistakes(document): Correct init config docs for C# and Python * test(kotlin): expect table-qualified columns in the TPH create autoSet tests The TPH per-subtype create now writes every column through the table object (it[AuthTable.autoCreatedAt]), so a column named body, issues or unknownKeys cannot be shadowed by the handler's locals. Two KotlinAutoSetStampingTest cases still asserted the unqualified form and failed the java-slow lane.
1 parent dbd7e42 commit 00b08dc

350 files changed

Lines changed: 12100 additions & 1192 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 173 additions & 3 deletions
Large diffs are not rendered by default.

‎agent-context/templates/always-on.md.mustache‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,11 @@ When the project serves the generated routes (any port):
4747
(case-sensitive, `%` and `_`), `gt`, `gte`, `lt`, `lte`. Filters AND; only `@filterable`
4848
fields filter.
4949
- `?sort=<field>:asc|desc`, `?limit=N&offset=N`; `withCount=1` returns `{"rows", "total"}`.
50-
Always send `limit`: without it TypeScript and C# return every row, the other ports 50.
51-
- `PUT` is the same partial update as `PATCH`.
52-
- Errors are `{"error": "<code>"}`: 400 for validation (only TypeScript adds `issues`) and for
50+
Always send `limit`: without it the project's `api.pagination.defaultLimit` applies, or with
51+
none set TypeScript and C# return every row, the other ports 50; above `maxLimit` is 400.
52+
- `PUT` is the same partial update as `PATCH`. Send only declared fields: an undeclared key is
53+
400 `unknown_fields` (unless `api.unknownFields` is `drop`).
54+
- Errors are `{"error": "<code>"}`: 400 for validation (`issues` lists each problem) and for
5355
a bad filter or sort (`invalid_filter_field`, `_op`, `_value`, `invalid_sort`, each naming
5456
`field`), 404 `not_found`, 409 for a unique or foreign-key conflict.
5557

‎docs/CONFORMANCE.md‎

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ regenerate with `ls -d fixtures/<corpus>/*/ | wc -l` for directory-shaped corpor
3636
| [`fixtures/extract-conformance/`](../fixtures/extract-conformance/) | 58 | ✓ | ✓ | inherits the shared JVM engine | ✓ | ✓ |
3737
| [`fixtures/output-prompt-conformance/`](../fixtures/output-prompt-conformance/) | 17 | ✓ | ✓ | ✓ | ✓ | ✓ |
3838
| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 64 (55 query + 9 migration) | all 64; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 55 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 55 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) |
39-
| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 123 (31 core + 10 tph + 9 m2m + 5 jsonb + 2 write-through + 2 secret-fields + 2 enum-default + 12 projection + 37 report + 11 decimal + 2 duration) | ✓ (Fastify reference + generated lane); `report/`, `decimal/` and `duration/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/`, `decimal/` and `duration/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (Kestrel + Npgsql); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/`, `decimal/` and `duration/` also on SQLite and MySQL |
39+
| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 139 (34 core + 14 tph + 9 m2m + 6 jsonb + 2 write-through + 2 secret-fields + 2 enum-default + 12 projection + 37 report + 11 decimal + 2 duration + 8 api-settings) | ✓ (Fastify reference + generated lane); `report/`, `decimal/` and `duration/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/`, `decimal/` and `duration/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (Kestrel + Npgsql); `report/`, `decimal/` and `duration/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/`, `decimal/` and `duration/` also on SQLite and MySQL |
4040
| [`fixtures/decimal-wire/`](../fixtures/decimal-wire/) | 1 vector file (format, parse, filter operand, 15-digit read) | ✓ (`runtime-ts` and the generated helper) | ✓ (`CanonicalDecimal`) | ✓ (the generated wire file) | ✓ (`DecimalWire`) | ✓ (`decimal_wire`) |
4141
| [`fixtures/duration-wire/`](../fixtures/duration-wire/) | 1 vector file: the `wire` vectors (parse, precision, refused) every port passes, and the `codecs` groups (Postgres text read and write, stored microseconds) each port passes only for the codecs it has | ✓ (`metadata` / `runtime-ts` and the generated helper) | ✓ | ✓ | ✓ | ✓ |
4242
| [`fixtures/validation-conformance/`](../fixtures/validation-conformance/) | 42 cases + 43 cross-field cases (`cross-field-cases.json`, generated record model only) | ✓ (generated Zod + run-time `runValidators`) | ✓ | ✓ | ✓ | ✓ (generated Pydantic + run-time `run_validators`) |
@@ -442,9 +442,9 @@ REAL, a MySQL `DECIMAL` keeps its scale, a MySQL `DATETIME` is the UTC wall cloc
442442
`report-relative-date` carries a per-engine seed (`seed-data-engine`). `op: relate` and the
443443
write ops stay Postgres-only. See [features/reporting.md](features/reporting.md#what-the-corpus-gates).
444444

445-
### `fixtures/api-contract-conformance/` (119)
445+
### `fixtures/api-contract-conformance/` (139)
446446

447-
All 119 scenarios → [features/api-contract.md](features/api-contract.md) (cross-port
447+
All 139 scenarios → [features/api-contract.md](features/api-contract.md) (cross-port
448448
REST API URL grammar + JSON wire format). Verifies every backend's emitted CRUD
449449
routes answer identically over HTTP — list / get / create / patch+put / delete,
450450
plus pagination (`limit`/`offset`), sort (`sort=field:dir`), the `withCount=1`
@@ -454,18 +454,24 @@ status codes.
454454
The corpus also covers the 9 cross-port filter operators (`eq`, `ne`, `gt`,
455455
`gte`, `lt`, `lte`, `in`, `like`, `isNull`) plus the implicit-AND combinator
456456
and 2 error shapes (`invalid_filter_field` / `invalid_filter_op`) under the
457-
URL grammar `?filter[<field>][<op>]=<value>` (FR-009). On top of the 31 core
458-
scenarios the corpus carries ten sub-corpora — `tph/` (10, single-table
459-
inheritance; a subtype decimal filters and refuses extra digits), `m2m/` (9 — 3 plain, 5 gating TPH x M:N together, the combination
457+
URL grammar `?filter[<field>][<op>]=<value>` (FR-009). On top of the 34 core
458+
scenarios (among them the write-input rules: server-owned keys dropped, an undeclared key
459+
refused under the default `api.unknownFields: reject`, and every validation issue listed)
460+
the corpus carries eleven sub-corpora — `tph/` (14, single-table
461+
inheritance; a subtype decimal filters and refuses extra digits, and the input rules hold over
462+
the whole hierarchy), `m2m/` (9 — 3 plain, 5 gating TPH x M:N together, the combination
460463
each corpus alone could not reach, and 1 pinning the collection-URL spelling),
461-
`jsonb/` (5, typed value-object columns; numbers inside them: a double stays a JSON number, a decimal is the canonical string),
464+
`jsonb/` (6, typed value-object columns; numbers inside them: a double stays a JSON number, a decimal is the canonical string; an undeclared member inside a value object is dropped),
462465
`write-through/` (2, table-write + view-read entities), `secret-fields/` (2, #484: a `@formExclude` or credential `view.password` field is ignored on create, PATCH and PUT, and a credential is never returned), `enum-default/` (2, a create that omits a `field.enum` with `@default` stores the default, and a supplied value is kept), `projection/` (12, a
463466
read-only view answers reads and refuses writes with 405; a projection with no declared
464467
identity has no item route; a projection keyed on a field not named `id` is addressed by
465468
it, whether its identity names `@fields` or derives them; decimal and float fields filter), `decimal/` (11, ADR-0019 Amendment 1: a decimal, array element and undeclared-precision column included, is one
466469
canonical string, read exactly, filtered exactly and refused past its declared digits), `duration/` (2, a
467470
`field.duration` under `"wire": { "durations": "iso" }` is one canonical ISO-8601 string, refused
468-
in a calendar or clock spelling, and filtered and sorted as a span on every engine) and `report/` (37,
471+
in a calendar or clock spelling, and filtered and sorted as a span on every engine), `api-settings/` (8,
472+
the project's `api` block: `api.pagination`'s default page, `limit_too_large` cap and refused
473+
negative `limit`, an empty page for `limit=0`, and
474+
`api.unknownFields: drop`) and `report/` (37,
469475
FR-044: a view-backed `object.report` is listed, filtered, sorted and paged on
470476
its derived fields, answers `POST` with 405 and mounts no `/{id}`; a `@spine` report keeps
471477
a row with no facts, and a measure with `@default` filters and sorts as its default). All 5 ports — TS, Java,
@@ -725,7 +731,7 @@ JSON and verify payload feeds it unchanged.
725731
## Orphaned fixtures (tested but not yet documented)
726732

727733
The fixtures in the nine corpora mapped above (metamodel 510 + yaml 16 + verify 45
728-
+ render 18 + persistence 64 + api-contract 123 + source-resolution 25 + scope 10 +
734+
+ render 18 + persistence 64 + api-contract 139 + source-resolution 25 + scope 10 +
729735
dependency 47) each map to a feature doc, and so do the two requirement corpora, `shared-model-conformance`, the
730736
`cube-model` corpus and `explorer-conformance`, whose case lists live in their own READMEs. None are orphaned today. The remaining
731737
corpora in the totals table gate tooling contracts (registry manifests, provider

0 commit comments

Comments
 (0)