Skip to content

Commit 58e22b2

Browse files
authored
feat(metadata): resolve references once after the tree (#444)
* feat(codegen): orphan sweep in every port, JVM hash manifest, per-generator handEdits - gen deletes a recorded, untouched file no generator emits any more (TS, Python, C#, Java, Kotlin) - Java/Kotlin decide overwrites from the committed hash manifest (ADR-0059) - handEdits: keep|overwrite per generator in all five ports * no-mistakes(review): Add JVM manifest owners and fix nested-dir and verify sweeps * no-mistakes(review): Key JVM manifest owners on generator and outputDir * no-mistakes(review): Limit JVM shared-path error to differing owned emissions * no-mistakes(review): Count profile-declared generators as configured in JVM sweep * no-mistakes(review): Read profile generators via Maven; ports keep JVM entries * feat(metadata): references resolve once, after the tree, in every port (ADR-0059) A reference was interpreted in two places: the YAML desugar rewrote reference values while lowering, and canonical JSON took them as written, refused `::` and `..::`, and resolved them later in each port's loader. The two paths disagreed, so the same model loaded differently from YAML and from JSON, and depended on file order. Now every source keeps references exactly as written. One resolver per port binds them after the whole tree has loaded and validated, records the bound target and the authored spelling, and rewrites each value to its canonical spelling before freeze. - One grammar in every source: `::Name` (root-absolute), `..::Name` (parent-relative, over-drop is ERR_BAD_ATTR_VALUE), `pkg::Name`, and a bare name (own file's package, then root). ERR_RELATIVE_REF_IN_CANONICAL is retired. - The referrer package is the package of the top-level node that declares the attribute, so an inherited reference means what it meant where it was written. `@of` after `@via` uses the package the walk reaches, and `@spine` uses the report's `@from` package. - The default package is per file, nested nodes included. - The serializer writes canonical values plus a package rule that lets a multi-package export load back to itself. `meta fmt` keeps authored spellings. - Consumers in codegen and runtime read the bound target (TS `refTarget`), never a short-name match. `findObject` accepts canonical values. Human-facing text keeps quoting the authored spelling. - Two new gates in every port: the YAML twin (each fixture loads to the same model from JSON and from YAML) and the round trip (each expected.json serializes back to itself), with zero exclusions. - YAML-twin defects fixed: C# YAML overlays now defer, and Java accepts a non-string field `default:` and list-valued `aliases`/`seeAlso`. - Shared goldens regenerated from the TypeScript oracle. Ten new conformance fixtures; one retired. * no-mistakes(review): Align gen and verify sweeps; union JVM profile owners * no-mistakes(review): Memoize JVM configured owners and compute them lazily * no-mistakes(review): Qualify per-port manifest generator records in docs and ADR * no-mistakes(review): Sweep shared JVM generator pairs only in last execution * no-mistakes(review): Refuse one generator/outputDir in two JVM executions at build time * no-mistakes(review): Exclude phase-none executions from JVM duplicate-pair check * no-mistakes(review): Keep phase-none executions' generators configured for sweep and verify * no-mistakes(review): Resolve inherited refs by bound target; fix Python @from regression * no-mistakes(review): Apply JVM handEdits overwrite verdict over all configured directory writers * no-mistakes(review): Bind post-load references by bound target across ports * no-mistakes(review): Make public payload resolvers exact canonical lookups * chore(showcase): refresh site payload fixture count (395) * fix(java,kotlin): keep pre-1.1.0 helper signatures so ejected generators compile A review round renamed KotlinGenUtil.resolveObjectByShortOrFqn and changed RelationshipReferences' target parameter from a String to the bound node. Generators an adopter ejected before 1.1.0 call the old forms, so EjectedAtBaseGeneratorsCompatTest no longer compiled them. Restore the old signatures as deprecated delegates that do the exact canonical lookup (ADR-0059 R7), never a short-name match. * no-mistakes(review): apply reviewer inline fixes * no-mistakes(document): Fixed stale template-header overwrite claims in three docs and one stale audit-axis reference; ADRs, CHANGELOG and JVM docs already matched the change.
1 parent bc8167b commit 58e22b2

481 files changed

Lines changed: 16048 additions & 4598 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ PyPI has had no product change since `0.25.0` — nothing is broken.
7272
- **Kotlin** — `codegen-kotlin` (KotlinPoet on JVM): entity + Exposed table + Spring controller + payload + relations + filter allowlist + validator + stored-proc + output-parser generators. `integration-tests-kotlin` runs the persistence-conformance corpus through Exposed against Testcontainers Postgres.
7373

7474
**Cross-port conformance corpora** (every port runs the shared corpus):
75-
- Metamodel: `fixtures/conformance/` (384 fixtures; 29 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green.
75+
- Metamodel: `fixtures/conformance/` (395 fixtures; 29 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green.
7676
- Render: `fixtures/render-conformance/`. TS / C# / Java / Kotlin / Python byte-identical.
7777
- Persistence: `fixtures/persistence-conformance/`. **Query** scenarios run on every port (TS / C# / Java / Kotlin / Python), each provisioning its test DB by executing the committed, TS-produced `canonical/schema.postgres.sql` (Postgres only — Derby dropped for the cross-port query corpus, ADR-0015; the FR-044 `report-*` scenarios also run on SQLite, MySQL and D1 where a port has the engine, see [docs/CONFORMANCE.md](docs/CONFORMANCE.md)). The **migration** scenarios are exercised by **TS only** (TS owns schema migrations). **The corpus now gates WRITES, not just reads (SP-H):** an `op: roundtrip` scenario type INSERTs through each port's runtime/ORM write codec (NOT raw SQL), reads the row back, and asserts the wire-normalized value. The `AllTypes` entity (`roundtrip-all-types.yaml`) carries one field of **every** persistable `field.*` subtype — string/int/long/double/float/decimal/boolean/date/time/timestamp(+tz)/currency/enum/uuid/object — plus an **array-of-VO** `field.object @isArray @storage:jsonb` column (`labels`, written as 2-element / empty-`[]` / single-element arrays across the three rows) — so every subtype write+read (incl. the array-of-value-object jsonb codec) round-trips through every port against Testcontainers PG. (`field.byte`/`field.short`/`field.class` were cut as non-functional registration-only stubs — the matrix tracks only genuinely-supported subtypes; see `fixtures/registry-conformance/README.md` → "Per-subtype write-round-trip matrix".)
7878
- API-contract: `fixtures/api-contract-conformance/`. TS / C# / Java / Kotlin / Python all green — each port runs **two lanes**: a hand-rolled reference server AND its **generated** API artifact booted over HTTP (the deployed controller/routes; TS+C# full-stack vs Testcontainers PG, Java/Kotlin/Python generated controller + in-memory repo behind the consumer seam). The generated fan-out found 10 real deployment bugs golden snapshots missed. Three sub-corpora run the **generated lane only, on all five ports** — `write-through/`, `projection/` (F22: a view-only `object.projection` serves GET list + GET by id and answers every write verb with `405 {"error": "method_not_allowed"}`) and `report/` (FR-044: a view-backed `object.report` serves GET list, answers `POST` with that same `405`, and mounts no `/{id}` route). That is deliberate, not a gap: what is under test is whether a port's GENERATOR emits those routes, and a hand-rolled reference server would answer every scenario by construction. The `m2m/` sub-corpus also gates **TPH x M:N together** (base-declared, subtype-declared, abstract-mid-declared, a non-subtype source onto a subtype TARGET, and the cross-subtype source id answering `200 []`) — the two corpora were originally built disjoint (`tph/` had no relationships, `m2m/` no discriminators), which is precisely how that defect class survived.

‎CHANGELOG.md‎

Lines changed: 183 additions & 1 deletion
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -263,7 +263,7 @@ metaobjects/
263263
├── CLAUDE.md # project instructions for Claude
264264
├── spec/ # canonical metamodel docs, ADRs, roadmap
265265
├── fixtures/ # 25 cross-language conformance corpora — the oracle
266-
│ ├── conformance/ # metamodel (loader + serializer + navigation), 384 fixtures
266+
│ ├── conformance/ # metamodel (loader + serializer + navigation), 395 fixtures
267267
│ ├── yaml-conformance/ # YAML authoring desugar
268268
│ ├── render-conformance/ # FR-004 byte-identical render oracle
269269
│ ├── verify-conformance/ # FR-004 template-drift gate

‎agent-context/skills/metaobjects-audit/references/surface-review-axes.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -141,8 +141,7 @@ The full hunt for each axis of Phase 1b (Partial / Deep path). Each axis runs in
141141
taxonomy impurity (entity over read-only primary source; read model that should be
142142
`object.projection`; `value` carrying identity/source, ADR-0028); copy-pasted base-field
143143
blocks — or the same-named field re-declared across objects (drift signature 10) — instead of
144-
abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); relative refs
145-
in committed canonical JSON (ADR-0032); DB-type-as-logical-subtype (ADR-0013); per-port
144+
abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); DB-type-as-logical-subtype (ADR-0013); per-port
146145
migration engine where schema is Node-`meta`-owned (ADR-0015).
147146

148147
- [ ] **H2. Wrong native type — `field.<x>` + `@dbColumnType` that hides the real type

‎agent-context/skills/metaobjects-authoring/references/abstracts-and-overlay.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,11 @@ Resolution facts:
3535
extends Auditable`) — each `extends` is a super-*reference*, so resolution walks the
3636
full chain (it does not flatten inherited members onto the child; see ADR-0039).
3737
- **Cross-package** refs use the fully-qualified name (`extends: "shared::auditable"`);
38-
same-package refs use the bare name.
38+
same-package refs use the bare name. A bare name looks in the declaring file's own
39+
package, then at the root, and never binds an object in another package. `::Name`
40+
names a root-level object even when the package has one of the same name; `..::Name`
41+
names one in the parent package. The forms mean the same in YAML and JSON (ADR-0059),
42+
and a loaded or exported model carries every reference fully qualified.
3943
- An unresolved reference fails with `ERR_UNRESOLVED_SUPER`.
4044

4145
`abstract` and `extends` are **structural keys** (bare, no `@`).

‎agent-context/skills/metaobjects-codegen/SKILL.md‎

Lines changed: 35 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -160,38 +160,48 @@ You run a `gen` step. The runner:
160160
2. Resolves output targets and precomputes shared render state.
161161
3. Runs each configured **generator** — most emit one file per entity; some emit a
162162
single shared file (a barrel, a DB-context, an app-config).
163-
4. Decides whether it may overwrite a file — and **the rule differs by port**:
164-
- **TypeScript, C#, Python** — by a committed **hash manifest**
165-
(`.metaobjects/.gen-state/.hashes.json`). If a file still hashes to what the
166-
generator recorded writing, it is safe to overwrite; if it was edited, or there is
167-
no record of it, the write is **refused by name**. TypeScript additionally
168-
three-way-merges against a snapshot when one is present locally.
169-
- **Java, Kotlin** — by a bare **`GENERATED`** token in the file's header comment
170-
(`GeneratedFileWriter.GENERATED_MARKER`). The token is `GENERATED`, **not**
171-
`@generated`: the matcher allows only whitespace between the comment punctuation
172-
and the token, so an `@`-prefixed tag does not match it. Remove that token and
173-
regeneration never touches the file again.
174-
175-
**A refusal FAILS the run — exit 1 on every port** (TypeScript always did; Python and
176-
C# warned and exited 0 until 1.0, so a `gen` wired into CI was green while codegen
177-
refused to write). The one-time fix for a project that predates the committed manifest
163+
4. Decides whether it may overwrite a file, by a committed **hash manifest**
164+
(`.metaobjects/.gen-state/.hashes.json`) on every port. If a file still hashes to what
165+
the generator recorded writing, it is safe to overwrite; if it was edited, or there is
166+
no record of it, the write is **refused by name**. TypeScript additionally
167+
three-way-merges against a snapshot when one is present locally. **Commit the
168+
manifest.** Java and Kotlin have used it since 8.1.0: there, a file with no record that
169+
carries the bare **`GENERATED`** header token (not `@generated`) is adopted as generated
170+
output, which is how the first run after upgrading from 8.0.x records everything.
171+
172+
**A refusal FAILS the run — exit 1 on TypeScript, C# and Python** (Python and C# warned
173+
and exited 0 until 1.0, so a `gen` wired into CI was green while codegen refused to
174+
write). On Java and Kotlin a refusal is a warning and `mvn metaobjects:verify` is the
175+
gate that fails. The one-time fix for a project that predates the committed manifest
178176
is **`gen --baseline=adopt`**: it records the files you have as the baseline and writes
179177
nothing, so there is finally a `.hashes.json` to commit — then `gen` again, and the
180178
regeneration lands as its own diff. Adopting DECLARES those files to be generated
181179
output, so an edit already inside one is part of the baseline and that regeneration
182180
replaces it; commit before you run it. (TypeScript additionally has
183181
`--baseline=fresh`: write fresh output now and discard the edits.)
184182

185-
**The two rules protect a hand edit in opposite ways, so do not carry a habit across
186-
ports.** On TypeScript, C# and Python, *editing the content* is what takes ownership —
187-
that is what breaks the hash — and deleting the header changes nothing except your
188-
ability to tell what generated the file. On Java and Kotlin the reverse holds: editing
189-
a file protects it not at all, because the `GENERATED` token is still there and the
190-
next run overwrites the edit; only removing that token does.
191-
192-
(The header text differs per port and none of it is the write decision on the
193-
hash-manifest ports: TypeScript and Python emit `@generated by …`, C# emits
194-
`<auto-generated/>`, Java and Kotlin emit the `GENERATED` token above.)
183+
**Output nobody should edit: set `handEdits: overwrite` on its generator entry.** The
184+
default, `keep`, refuses an edited file. `overwrite` replaces it with fresh output, names
185+
it, and re-baselines it; `verify --codegen` then reports the edit as drift that `gen`
186+
fixes. A file the manifest has no record of is refused either way. TypeScript
187+
`{ ...entityFile(), handEdits: "overwrite" }`; Python `{ name: entity, handEdits: overwrite }`
188+
in `metaobjects.config.yaml`; C# `new EntityGenerator().WithHandEdits(HandEdits.Overwrite)`;
189+
Java and Kotlin `<handEdits>overwrite</handEdits>` on the `<generator>`.
190+
191+
**A full `gen` deletes generated files it no longer emits** (a renamed or deleted object,
192+
a page that moved package). Only files the manifest records, only when untouched, and
193+
never on a run narrowed by `--generators` or an entity filter. An edited one is refused by
194+
name, and `verify --codegen` says to delete it: regeneration cannot follow a rename, so
195+
move the edits into the new file first if you want them. Details:
196+
`docs/features/own-your-codegen.md`, "Files that are no longer generated".
197+
198+
On every port *editing the content* is what takes ownership — that is what breaks the
199+
hash — and deleting the header is just another edit. (Through Maven 8.0.x, Java and
200+
Kotlin did the reverse: an edit was overwritten and only removing the `GENERATED` token
201+
protected a file. Do not carry that habit forward.)
202+
203+
(The header text differs per port and is not the write decision: TypeScript and Python
204+
emit `@generated by …`, C# emits `<auto-generated/>`, Java and Kotlin emit `GENERATED`.)
195205

196206
The output is normal idiomatic code in your language — you import it and use it
197207
like any hand-written module.

‎docs/CONFORMANCE.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ regenerate with `ls -d fixtures/<corpus>/*/ | wc -l` for directory-shaped corpor
2626

2727
| Corpus | Fixtures | TS | Java | Kotlin | C# | Python |
2828
|---|---|---|---|---|---|---|
29-
| [`fixtures/conformance/`](../fixtures/conformance/) (metamodel) | 384 | ✓ | ✓ | inherits via `metadata-ktx` | ✓ | ✓ |
29+
| [`fixtures/conformance/`](../fixtures/conformance/) (metamodel) | 395 | ✓ | ✓ | inherits via `metadata-ktx` | ✓ | ✓ |
3030
| [`fixtures/yaml-conformance/`](../fixtures/yaml-conformance/) | 16 | 16 / 16 | 15 / 16 (1 ledgered: `yaml-quoted-leading-zero` — Java pipeline strips quotes off `"007"`) | inherits via Java | 15 / 16 (1 ledgered: `error-yaml-coerced-hex-in-string` — YamlDotNet doesn't coerce `0xFF`) | 16 / 16 |
3131
| [`fixtures/verify-conformance/`](../fixtures/verify-conformance/) | 37 | ✓ | ✓ | inherits via Java | ✓ | ✓ |
3232
| [`fixtures/verify-strict-conformance/`](../fixtures/verify-strict-conformance/) | 1 | ✓ | — | — | — | ✓ |
@@ -38,7 +38,7 @@ regenerate with `ls -d fixtures/<corpus>/*/ | wc -l` for directory-shaped corpor
3838
| [`fixtures/validation-conformance/`](../fixtures/validation-conformance/) | 42 cases | ✓ (generated Zod + run-time `runValidators`) | ✓ | ✓ | ✓ | ✓ (generated Pydantic + run-time `run_validators`) |
3939
| [`fixtures/registry-conformance/`](../fixtures/registry-conformance/) | 1 canonical manifest | ✓ (reference emitter) | ✓ | ✓ | ✓ | ✓ |
4040
| [`fixtures/object-model-conformance/`](../fixtures/object-model-conformance/) | 1 shared metadata fixture (per-port scenarios) | ✓ | ✓ | ✓ | ✓ | ✓ |
41-
| [`fixtures/codegen-conformance/`](../fixtures/codegen-conformance/) | 5 shared inputs; `shared-provided-enum` also carries the enum-type decision table every port answers | ✓ | ✓ | ✓ | ✓ | ✓ |
41+
| [`fixtures/codegen-conformance/`](../fixtures/codegen-conformance/) | 6 shared inputs; `shared-provided-enum` carries the enum-type decision table and `cross-package-binding` the reference-binding decision table (ADR-0059) every port answers | ✓ | ✓ | ✓ | ✓ | ✓ |
4242
| [`fixtures/template-codegen-conformance/`](../fixtures/template-codegen-conformance/) | 3 | ✓ | ✓ | ✓ | ✓ | ✓ |
4343
| [`fixtures/template-output-render-conformance/`](../fixtures/template-output-render-conformance/) | 5 | ✓ | ✓ | ✓ | ✓ | ✓ |
4444
| [`fixtures/generator-registry-conformance/`](../fixtures/generator-registry-conformance/) | 1 canonical manifest | ✓ | ✓ | ✓ | ✓ | ✓ |
@@ -267,12 +267,13 @@ unit-test runners (`bun test`, `dotnet test`, `pytest`, `mvn test`) pull Docker.
267267

268268
## Fixture-to-doc mapping
269269

270-
### `fixtures/conformance/` — metamodel loader + canonical serializer (384)
270+
### `fixtures/conformance/` — metamodel loader + canonical serializer (395)
271271

272272
| Fixture prefix | Feature doc |
273273
|---|---|
274274
| `loader-basic-*`, `error-parse-*` | [features/loaders.md](features/loaders.md) |
275275
| `extends-*`, `error-extends-*` | [features/entities.md](features/entities.md) (inheritance) |
276+
| `ref-*`, `error-ref-*`, `package-default-per-file`, `serialize-multi-package-document` | [features/abstracts-and-inheritance.md](features/abstracts-and-inheritance.md) (how a reference resolves: bare, `::`, `..::`, per-file package; ADR-0059) |
276277
| `identity-*`, `subtype-entity-*`, `subtype-value-*` | [features/entities.md](features/entities.md) (identity / entity vs value) |
277278
| `attr-*`, `error-attr-*`, `error-reserved-word-as-attr` | [features/entities.md](features/entities.md) (attributes) |
278279
| `doc-common-attrs-*` | [features/entities.md](features/entities.md) (documentation common attrs) |
@@ -541,7 +542,7 @@ files into a real Cube.
541542

542543
## Orphaned fixtures (tested but not yet documented)
543544

544-
The fixtures in the nine corpora mapped above (metamodel 384 + yaml 16 + verify 37
545+
The fixtures in the nine corpora mapped above (metamodel 395 + yaml 16 + verify 37
545546
+ render 16 + persistence 54 + api-contract 91 + source-resolution 25 + scope 10 +
546547
dependency 23) each map to a feature doc, and so do the two requirement corpora and the
547548
`cube-model` corpus, whose case lists live in their own READMEs. None are orphaned today. The remaining

‎docs/features/abstracts-and-inheritance.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,12 @@ This means:
150150
attrs.
151151
- **Cross-package references** use fully-qualified names:
152152
`extends: "shared::auditable"`. Same-package references can use the bare
153-
name.
153+
name: a bare name binds the declaring file's own package, then the root, and
154+
never an object in another package. `::Name` reaches a root-level object even
155+
when the package declares the same name, and `..::Name` names a sibling in the
156+
parent package. These forms mean the same in YAML and canonical JSON
157+
([ADR-0059](../../spec/decisions/ADR-0059-references-resolve-once-after-the-tree.md)).
158+
Exported JSON writes every reference fully qualified.
154159

155160
If `extends:` references a name that doesn't resolve, the loader emits
156161
`ERR_UNRESOLVED_SUPER` with the unresolved reference and the source location.

‎docs/features/cli.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,24 @@ one-time fix.
234234
| `--baseline=adopt` | TS · Python · C# | record the files you HAVE as the merge base and **write nothing** |
235235
| `--baseline=fresh` | TS only | overwrite the refused files from fresh output and re-baseline (**discards** hand edits). A local snapshot body for a path the manifest has no record of is ignored, not merged against, so stale content from an earlier checkout is not kept as a hand edit |
236236

237+
`--baseline` answers for one run. To decide once, per generator, set `handEdits` on its
238+
entry instead:
239+
240+
| setting | ports | what it does |
241+
|---|---|---|
242+
| `handEdits: keep` | all | the default: an edited generated file is refused, and so is an edited file no generator emits any more |
243+
| `handEdits: overwrite` | all | replace that generator's edited files with fresh output and re-baseline them; the orphan sweep deletes an edited file no generator emits any more when every generator writing into its directory overwrites. Each discarded file is named. A file the manifest has no record of is still refused |
244+
245+
| | `--baseline=fresh` | `handEdits: overwrite` |
246+
|---|---|---|
247+
| scope | one run, every generator | every run, one generator |
248+
| a file with no manifest record | overwritten | refused |
249+
| an edited orphan | untouched | deleted (when the whole directory overwrites) |
250+
| `verify --codegen` | not affected | reports the edit as drift that `gen` fixes |
251+
252+
Syntax per port, and the orphan rule both follow:
253+
[own-your-codegen](own-your-codegen.md#handedits-keep-or-discard-edits-to-one-generators-files).
254+
237255
**`adopt` exists because the refusal's own remedy was unreachable.** It told you to commit
238256
`.hashes.json` — but nothing writes a manifest until a `gen` succeeds, and a run where
239257
every file refuses writes none, so there was nothing to commit and the next run refused

0 commit comments

Comments
 (0)