Skip to content

Commit 4bbd6cc

Browse files
authored
feat(python): eject owns the emitters its generators write through (#436)
* feat(python): eject hands over the emitters a generator writes through; api-docs is ejectable An ejected Python generator used to be one file over packaged emitters, so a defect in the shared-enum, M:N, prompt-tier or api-reference writers still waited for a release. Eject now copies those emitters with the generator into codegen/runtime/, through each registry entry's existing runtime list, and the copy reads them from there. The emitters moved into metaobjects.codegen.runtime; their old import paths stay as deprecated aliases so a copy ejected earlier keeps running. Naming and type-mapping tables, the model walks and the generator base stay in the package as the engine. api-docs is registered for Python and ejectable. The api reference is a generator (docs_surface = "api") that `metaobjects docs` runs; an owned copy is listed under the new docs.generators config block. `gen` and `verify --codegen` skip a docs generator and say so. The page builder reads the router gates from shared_predicates (new emits_writable_router, moved is_read_only_routed), the module an owned routes and filter-allowlist read, so the reference describes an owned copy. test_eject_alone derives its list from the registry: each ejectable generator is ejected alone into an empty project, checked against a written engine surface, run with every packaged emit module unimportable, and compared byte for byte with the packaged generator. Packaged output is unchanged. * feat(python): eject hands over the emitters a generator writes through; api-docs is ejectable An ejected Python generator used to be one file over packaged emitters, so a defect in the shared-enum, M:N, prompt-tier or api-reference writers still waited for a release. Eject now copies those emitters with the generator into codegen/runtime/, through each registry entry's existing runtime list, and the copy reads them from there. The emitters moved into metaobjects.codegen.runtime; their old import paths stay as deprecated aliases so a copy ejected earlier keeps running. Naming and type-mapping tables, the model walks and the generator base stay in the package as the engine. api-docs is registered for Python and ejectable. The api reference is a generator (docs_surface = "api") that `metaobjects docs` runs; an owned copy is listed under the new docs.generators config block. `gen` and `verify --codegen` skip a docs generator and say so. The page builder reads the router gates from shared_predicates (new emits_writable_router, moved is_read_only_routed), the module an owned routes and filter-allowlist read, so the reference describes an owned copy. test_eject_alone derives its list from the registry: each ejectable generator is ejected alone into an empty project, checked against a written engine surface, run with every packaged emit module unimportable, and compared byte for byte with the packaged generator. Packaged output is unchanged. * no-mistakes(review): test(python): defer engine surface to shared eject manifest * no-mistakes(review): fix(python): eject reports stale kept runtime copies * no-mistakes(review): fix(python): eject refresh hint prints diff first, cwd-valid paths * no-mistakes(review): fix(python): gen skip warning distinguishes packaged and owned docs generators * no-mistakes(review): docs(adr-0034): record Python eject gaps 4-6 as closed * no-mistakes(document): Drop Python from ADR-0034 Amendment 8 pending wording
1 parent ce9b01c commit 4bbd6cc

73 files changed

Lines changed: 3400 additions & 2295 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,69 @@ generator catalog and the Atlas evaluation are 1.2 work.
217217
calls one.
218218
The cross-port manifest lists `api-docs` for `csharp`.
219219

220+
- **Python: an ejected generator owns the emitters it writes through, and `api-docs` can be
221+
ejected.** `metaobjects eject` used to copy one file per generator, so an owned `entity`
222+
still called the shared-enum and M:N writers, and an owned `output-parser`, `extractor`,
223+
`output-prompt` or `trace-helper` still called the prompt tier's field mapping, extract
224+
delegate, extract schema and format-spec writers. A defect in any of them waited for a
225+
release. Eject now copies those emitters with the generator, into `codegen/runtime/`, and the
226+
copy reads them from there: `fr019_shared_enum` and `m2m_codegen` with `entity`;
227+
`m2m_codegen` with `filter-allowlist`; `fr010_field_mapping` and `m2m_codegen` with `routes`;
228+
`extract_delegate_emitter` and `fr010_field_mapping` with `output-parser` and `extractor`;
229+
`output_format_spec_emitter` and `fr010_field_mapping` with `output-prompt`;
230+
`extract_schema_emitter` and `fr010_field_mapping` with `trace-helper`. A file two generators
231+
share is copied once and never overwritten, and `metaobjects gen --list` marks each copy
232+
`identical` or `DIFFERS`. What stays in the package is the engine: the generator base and
233+
file writer, `metaobjects.apidocs.naming`, `type_map` and `value_objects` (the naming and
234+
type-mapping tables), and the model walks (`find_inbound`, `tph_plan`,
235+
`instance_artifacts`). `template` is not ejectable: it has no emit logic of its own. Packaged
236+
output is unchanged and gated to be.
237+
- **`api-docs` is a generator you can own.** `metaobjects eject api-docs` copies the page
238+
generator with its page model, builder, renderer and path math. The api reference is now a
239+
generator (`docs_surface = "api"`) that `metaobjects docs` runs: an owned copy is listed
240+
under the new `docs.generators` block of `metaobjects.config.yaml` and runs in place of the
241+
packaged one. `gen` and `verify --codegen` skip a docs generator they find under a target
242+
and say so: the packaged `api-docs` name is to be removed (`metaobjects docs` already runs
243+
it), an owned copy is to be listed under `docs.generators`. The reference asks `shared_predicates` which objects get a router (the new
244+
`emits_writable_router` and the moved `is_read_only_routed`, beside `emits_router`), the
245+
module your owned `routes` and `filter-allowlist` read, so one edit to the owned predicate
246+
moves the router, the allowlist and the pages together. The cross-port manifest lists
247+
`api-docs` for `python`.
248+
- **Where things moved.** The emitters now live in `metaobjects.codegen.runtime` (a
249+
module the helper-runtime copy already used): `extract_delegate_emitter`,
250+
`fr010_field_mapping`, `output_format_spec_emitter`, `extract_schema_emitter`,
251+
`fr019_shared_enum`, `m2m_codegen`, and the api reference as `api_docs_model`,
252+
`api_docs_builder`, `api_docs_renderer` and `api_docs_paths`. `metaobjects.apidocs` keeps
253+
`naming` and exports `PythonApiModelBuilder`, `PythonApiDocsRenderer` and the path
254+
functions as before. New public API: `DocsRunFacts`, `GenContext.docs`, `DEFAULT_API_SUBDIR`
255+
and `DOCS_SURFACE_API` in `metaobjects.codegen.generator`, `ProjectConfig.docs_generators`,
256+
`GeneratorEntry.docs_surface`, and `metaobjects.codegen.docs_run`.
257+
- **A copy you ejected before this release** keeps running: it imports the emitters from
258+
their old paths, which are deprecated aliases (see **Deprecated**), so it calls the
259+
packaged emitters as it always did. `metaobjects gen --list` marks it `DIFFERS: N behind`.
260+
To own them, eject the same name into a scratch directory and copy the generator's
261+
`runtime/` files over (see
262+
[own-your-codegen](docs/features/own-your-codegen.md#the-emitters-a-generator-writes-through-come-with-it)).
263+
- **A `codegen/runtime/shared_predicates.py` you already own needs two new functions.**
264+
`routes` now reads `is_read_only_routed` from it and `api-docs` reads that and
265+
`emits_writable_router`, and eject never overwrites a runtime copy. When eject keeps a
266+
runtime copy it now says how the copy compares with the installed module
267+
(`kept your copy — identical`, or `kept your copy — DIFFERS: N behind, M of your own`),
268+
and for a copy that is behind it prints two commands that run from the working
269+
directory, `--root` or not: the `diff -u` of your copy against the installed file, whose
270+
hunks you apply, and the `cp` of the installed file over yours, for a copy you have not
271+
changed.
272+
- **A gate for the rule.** The engine surface is written once, in the shared manifest
273+
(`eject.engineSurface.python`), and `tests/conformance/test_eject_contract.py` ejects every
274+
ejectable generator alone into an empty project and fails on any import outside it. A
275+
second test (`tests/codegen/test_eject_alone.py`) runs each owned copy with every packaged
276+
emit module made unimportable and compares its output byte for byte with the packaged
277+
generator's over a model that exercises each one (the router's own runtime imports are the
278+
one documented difference). Another fails on a helper module no registry entry hands over,
279+
or a generator module no registry entry names.
280+
The nine Python entries of the shared eject contract's `eject.pending` list are closed, and the
281+
contract test runs `api-docs` through `metaobjects docs`, the way it is run.
282+
220283
- **The stock `entity`, `queries`, `routes` and `barrel` generators are importable again
221284
(TypeScript).** `entityFile`, `queriesFile`, `routesFile` and `barrel` are exported from
222285
`@metaobjectsdev/codegen-ts/generators` once more, beside `namesFile`, `routesFileHono`
@@ -871,6 +934,16 @@ until you regenerate.
871934
- **C#: `Fr010FieldMapping.CSharpStringLiteral(string)` is `[Obsolete]`.** It forwards to
872935
`CSharpNaming.CSharpStringLiteral(string)`, which every generator now calls (so an ejected
873936
`names` needs no emitter). Same signature and result; switch the call when you next touch it.
937+
- **Python: the old import paths of the emitters that moved into `metaobjects.codegen.runtime`.**
938+
`metaobjects.codegen.extract_delegate_emitter`, `.fr010_field_mapping`,
939+
`.output_format_spec_emitter`, `.extract_schema_emitter`, `metaobjects.codegen.generators.fr019_shared_enum`,
940+
`.m2m_codegen`, and `metaobjects.apidocs.api_model`, `.builder`, `.renderer`, `.paths` are
941+
each one-line modules that become the new module and warn (`DeprecationWarning`) on import.
942+
Same objects, so nothing else changes; import from `metaobjects.codegen.runtime` (the api
943+
reference modules are `api_docs_model`, `api_docs_builder`, `api_docs_renderer` and
944+
`api_docs_paths` there). The old paths will be removed in a later release.
945+
`metaobjects.codegen.generators.router_generator.is_read_only_routed` moved to
946+
`metaobjects.codegen.runtime.shared_predicates`; the router module still re-exports it.
874947

875948
### Removed
876949

‎docs/features/cli.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,9 @@ requirementTests: # optional; options of the `requirement-tests`
376376
filter: codegen.requirement_hooks:include # "module:symbol", resolved config-relative
377377
renderer: codegen.requirement_hooks:render # "module:symbol", resolved config-relative
378378
warnUncovered: true # the default
379+
docs: # optional; options of `metaobjects docs`
380+
generators: # owned docs generators as "module:symbol", run in place of the packaged one
381+
- codegen.generators.api_docs:api_docs_generator
379382
targets:
380383
api:
381384
outDir: src/generated/api

‎docs/features/own-your-codegen.md‎

Lines changed: 114 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -695,9 +695,22 @@ targets:
695695
Eject prints the exact entry for each copy. It never overwrites an existing copy without
696696
`--force` and never edits your config. `metaobjects gen --list` marks each owned copy
697697
`identical` or `DIFFERS: N behind, M of your own` against the packaged reference, so you
698-
can see when an upgrade changed the generator you copied. A copy imports the same
699-
`metaobjects.codegen.*` modules the packaged one does, and those module paths are the
700-
surface an owned generator builds on.
698+
can see when an upgrade changed the generator you copied. Every generator `metaobjects
699+
gen --list` shows is ejectable except `template`, the Mustache primitive, which has no emit
700+
logic of its own to own.
701+
702+
An ejected generator owns every line that writes its output. Eject copies the generator
703+
with the emitters it writes through (below), and the copy imports only the engine from the
704+
package: the generator base and file writer (`metaobjects.codegen.generator`), the
705+
formatter, the naming and type-mapping tables (`metaobjects.apidocs.naming`,
706+
`metaobjects.codegen.type_map`, `metaobjects.codegen.value_objects`), the model walks
707+
(`find_inbound`, `tph_plan`, `instance_artifacts`) and the loader. Those module paths are
708+
the surface an owned generator builds on. It is written once, in the shared manifest
709+
(`fixtures/generator-registry-conformance/registry.json`, `eject.engineSurface.python`), and
710+
`tests/conformance/test_eject_contract.py` ejects each generator alone into an empty project
711+
and rejects any import outside it. `tests/codegen/test_eject_alone.py` then runs each owned
712+
copy with the packaged emitters made unimportable and compares its output byte for byte with
713+
the packaged generator's.
701714

702715
`requirement-tests` ejects the same way, to `codegen/generators/requirement_tests.py`,
703716
wired as `codegen.generators.requirement_tests:requirement_tests_generator`. The copy holds
@@ -742,9 +755,12 @@ like the rest of the package.
742755
--codegen` reports the stale `_runtime/` file until you run `gen`. That is the right
743756
answer, because the committed package is out of date.
744757
- **Eject never overwrites your runtime.** An existing `codegen/runtime/<module>.py` is
745-
kept, even with `--force`, and eject says so. `metaobjects gen --list` marks each
746-
runtime copy `identical` or `DIFFERS: N behind, M of your own` against the installed
747-
one.
758+
kept, even with `--force`, and eject says so, with how the copy compares with the
759+
installed one (`kept your copy — identical` or `kept your copy — DIFFERS: N behind, M of
760+
your own`). When the copy is behind, the next line names the generator that may need the
761+
newer one and two commands, both written to run from the directory you ran eject in: the
762+
`diff -u` of your copy against the installed file, and the `cp` to use only if you have
763+
not changed yours. `metaobjects gen --list` marks each runtime copy the same way.
748764
- **A project that has not ejected `routes` is unchanged.** The packaged generator still
749765
imports `metaobjects.codegen.runtime.*` and emits no `_runtime/`.
750766

@@ -764,34 +780,109 @@ codegen/runtime/`). The same recipe covers the generator itself: `metaobjects ej
764780
routes` into a scratch directory gives you the new reference to diff your
765781
`codegen/generators/routes.py` against.
766782

767-
### The rules two generators share come with them
783+
### The emitters a generator writes through come with it
768784

769-
Some emit rules are asked by more than one generator, and the answers have to match.
770-
`routes` and `filter-allowlist` both ask `emits_router` (which objects get a router; the
771-
generated router imports the allowlist module, so a mismatch is an import error at
772-
startup). `entity` and `extractor` both ask `has_literal_default`. These live in one
773-
module, `shared_predicates`, and ejecting any of those four generators copies it too:
785+
A generator that calls a packaged emitter is a shell around it: a defect in the emitter
786+
still waits for a release. So eject copies the emitters too, into `codegen/runtime/`, next
787+
to the generator. The set is each registry entry's `runtime` list:
788+
789+
| Ejecting | also copies |
790+
|---|---|
791+
| `entity` | `fr019_shared_enum`, `m2m_codegen`, `shared_predicates` |
792+
| `routes` | `fr010_field_mapping`, `m2m_codegen`, `shared_predicates`, and the router's own runtime (above) |
793+
| `filter-allowlist` | `m2m_codegen`, `shared_predicates` |
794+
| `output-parser` | `extract_delegate_emitter`, `fr010_field_mapping` |
795+
| `extractor` | `extract_delegate_emitter`, `fr010_field_mapping`, `shared_predicates` |
796+
| `output-prompt` | `output_format_spec_emitter`, `fr010_field_mapping` |
797+
| `trace-helper` | `extract_schema_emitter`, `fr010_field_mapping` |
798+
| `api-docs` | `api_docs_model`, `api_docs_builder`, `api_docs_renderer`, `api_docs_paths`, `m2m_codegen`, `shared_predicates` |
799+
800+
`names`, `render-helper` and `requirement-tests` call no packaged emitter. A file two
801+
generators share is copied once and an existing copy is never overwritten, even with
802+
`--force`; `metaobjects gen --list` marks each copy `identical` or `DIFFERS: N behind, M of
803+
your own`.
774804

775805
```text
776806
codegen/
777-
generators/routes.py
778-
generators/filter_allowlist.py
779-
runtime/shared_predicates.py # the shared rules, now your code
807+
generators/output_parser.py
808+
generators/extractor.py
809+
runtime/extract_delegate_emitter.py # the one copy both read
810+
runtime/fr010_field_mapping.py
780811
```
781812

782-
Each generator imports it relatively (`from ..runtime.shared_predicates import ...`), so
783-
the verbatim copy in `codegen/generators/` reads `codegen/runtime/shared_predicates.py`
784-
and never the installed module. Two owned generators read the one copy: change
785-
`emits_router` there and your `routes` and your `filter-allowlist` change together. No
786-
generated code imports this module. It runs at `gen` time only, so nothing is emitted into
787-
`_runtime/` for it. The rest works as for the runtime above: an existing copy is kept, and
788-
`--list` marks it `identical` or `DIFFERS`.
813+
Each generator imports its helpers relatively (`from ..runtime import
814+
extract_delegate_emitter as rde`), so the verbatim copy in `codegen/generators/` reads
815+
`codegen/runtime/` and never the installed package. Two owned generators read the one copy:
816+
change it and both change. No generated code imports these modules; they run at `gen` time
817+
only, so nothing is emitted into `_runtime/` for them.
818+
819+
**The rules two generators share** live in the same place. `routes` and `filter-allowlist`
820+
both ask `emits_router` (which objects get a router; the generated router imports the
821+
allowlist module, so a mismatch is an import error at startup). `entity` and `extractor`
822+
both ask `has_literal_default`. They are in `shared_predicates`, which each of those
823+
generators lists, so editing `emits_router` there moves your `routes` and your
824+
`filter-allowlist` together.
789825

790-
A packaged generator keeps reading the installed rules. If you own `routes` and change
826+
A packaged generator keeps reading the installed copy. If you own `routes` and change
791827
`emits_router` but still run the packaged `filter-allowlist`, the two disagree. Eject
792828
`filter-allowlist` too before you change that rule; the same goes for `entity` and
793829
`extractor` with `has_literal_default`.
794830

831+
**What stays in the package.** The engine: the generator base, writer and formatter, the
832+
naming and type-mapping tables, the model walks and the loader. A table that decides a name
833+
or a type is a question about the model, and the generators, the docs and the runtime must
834+
agree on it, so it stays one packaged, public copy.
835+
836+
**A copy you ejected before this release** keeps running. It imports the emitters from the
837+
paths they had (`metaobjects.codegen.fr010_field_mapping`,
838+
`metaobjects.codegen.generators.m2m_codegen` and the others in the CHANGELOG), and each of
839+
those is now a deprecated alias of the module in `metaobjects.codegen.runtime`, so your copy
840+
calls the packaged emitter exactly as before and a `DeprecationWarning` names the new path.
841+
`metaobjects gen --list` marks it `DIFFERS: N behind`. To own the emitters, eject the same name
842+
into a scratch directory (`metaobjects eject <name> --root <scratch>`), copy the `runtime/`
843+
files over into `codegen/runtime/`, and change your imports to the relative form the new copy
844+
uses (`from ..runtime import fr010_field_mapping as fm`).
845+
846+
**An existing `shared_predicates.py` needs two new functions.** `routes` now reads
847+
`is_read_only_routed` from it, and `api-docs` reads that and `emits_writable_router`. A
848+
`codegen/runtime/shared_predicates.py` you already own does not have them, and eject keeps
849+
it, so `metaobjects eject api-docs` or `metaobjects eject routes --force` leaves a generator
850+
that fails to import (`cannot import name 'emits_writable_router'`). Eject marks the kept
851+
copy `DIFFERS: N behind` and prints how to bring it up: first the `diff -u` of your copy
852+
against the installed file, whose hunks you apply, then the `cp` of the installed file over
853+
yours, for a copy you have not changed. Eject does not decide which applies (the recipe under
854+
[The runtime your generated code imports comes with it](#the-runtime-your-generated-code-imports-comes-with-it)).
855+
856+
### `api-docs`: the api reference as an owned generator
857+
858+
`metaobjects docs` writes the Python api reference (`api/python`). `metaobjects eject
859+
api-docs` copies the generator that writes it, with the page model (`api_docs_model.py`), the
860+
builder that decides what to document (`api_docs_builder.py`), the renderer
861+
(`api_docs_renderer.py`) and the page-path math (`api_docs_paths.py`). Change which nodes
862+
get a page, how a page reads, or where it lands, by editing them.
863+
864+
A docs generator is not a code generator, so it is wired in its own block, never in a
865+
target's `generators`:
866+
867+
```yaml
868+
docs:
869+
generators: [codegen.generators.api_docs:api_docs_generator]
870+
```
871+
872+
`metaobjects docs` runs the copy in place of the packaged generator. Without the block it
873+
runs the packaged one, and the positional `metaobjects docs <metadata_dir>` form reads no
874+
config, so it always does. An entry that is not a docs generator, or a second one for the
875+
`api` surface, is refused by name. `gen` and `verify --codegen` skip a docs generator
876+
they find under a target and say so (`'api-docs' writes the api reference, so gen skipped
877+
it.`), with what to do about the entry: the packaged `api-docs` name is removed, because
878+
`metaobjects docs` already runs it, and an owned copy moves to `docs.generators`.
879+
880+
**It describes the code your generators emit.** The page builder asks `shared_predicates`
881+
which objects get a router, the same module your owned `routes` and `filter-allowlist`
882+
read. Eject them together and edit the predicate once: the router, the allowlist and the
883+
reference change together. The naming seam (`metaobjects.apidocs.naming`) is the engine,
884+
the one set of names every generator and every page uses.
885+
795886
## Java and Kotlin: `mvn metaobjects:eject`
796887

797888
```bash

‎docs/ports/python.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -660,6 +660,16 @@ standard-library module name (`types`, `json`) is refused there with an error na
660660
rename it. See
661661
[Own your codegen → Python](../features/own-your-codegen.md#python-metaobjects-eject).
662662

663+
Every generator `metaobjects gen --list` shows is ejectable except `template`, the Mustache
664+
primitive, which has no emit logic of its own to own. An ejected generator comes with the
665+
emitters it writes its output through (the shared-enum and M:N writers, the prompt tier's
666+
field mapping, extract delegate, extract schema and format-spec writers), copied into
667+
`codegen/runtime/` and read from there, so the copy owns the code that writes the file and
668+
not a shell around the package's. The naming and type-mapping tables, the model walks and
669+
the generator base stay in the package as the engine. `metaobjects eject api-docs` does the
670+
same for the api reference `metaobjects docs` writes: the copy is wired under
671+
`docs.generators` in `metaobjects.config.yaml`, never in a target's `generators`.
672+
663673
Ejecting `routes` also hands over the helper runtime the generated routers call:
664674
`filter_parser` and `constraint_errors` are copied into `codegen/runtime/`, and the owned
665675
generator emits them into the generated package as `_runtime/` and imports them from

0 commit comments

Comments
 (0)