Repository navigation
fix(codegen-ts): keep whole-model output whole and write the manifest once per gen run - #584
Merged
Merged
Conversation
…load metadata once - TypeScript, Python and C# hold .gen-state/.hashes.json in memory for a gen run and write it once at the end (Java already did); an unchanged run writes nothing. Python gen no longer lists unchanged files as written. - A run narrowed to some entities (meta gen <Entity>, metaobjects gen --entities) writes only those entities' own files; every generator still sees the whole model (ctx.entities), with the narrowing as ctx.selected. Whole-model files are now byte-identical to a full run's. A Python target's declared entities still bound its model. - meta gen reuses its generation load for the unknown-attribute warning, re-parsing strictly only when a bare unknown key was reported. - meta gen no longer imports a database driver: CLI modules on the gen path read the new @metaobjectsdev/migrate-ts/offline subpath. Gated by an import-graph test. - Under Node the config file loads from its own path, so jiti's cache hits instead of filling with randomly named entries.
… ends Database test containers mount a size-capped tmpfs at the image's VOLUME path, Postgres runs with fsync, synchronous_commit and full_page_writes off, and every docker-CLI launcher also removes its containers on exit and SIGTERM (covering a disabled Ryuk). The CI sidecar and workflow services get the same treatment. The Stryker sandbox moves to a RAM scratch dir and its runners stop with it. Work in progress: AGENTS.md section, CHANGELOG and suite proof follow.
…ails too A run that throws still writes the manifest for the files it wrote, but a failed write no longer replaces the run's error (TypeScript, Python, C#). Also: Python's gen summary counts skip-existing files as left as they were, not unchanged; a Postgres image tag that names no major takes the Postgres 18 data directory; CHANGELOG notes what a narrowed run now checks and what an owned per-object generator must do.
…ion, remove containers in one call
… warnings; restore config rewrite
dmealing
added a commit
that referenced
this pull request
Nov 15, 2026
…anged (ADR-0064) (#590) * feat(codegen): skip gen when nothing changed, in every port (ADR-0064) Forward-port of #585 from release/1.2 (b9bc9c7) onto main, merged with the manifest written once per run (#584) and the uniform entity selection (#588). The skip ADR is numbered ADR-0064 here: ADR-0062 and ADR-0063 are taken on main. - gen builds a key from the loaded model (each root node in canonical effective form), the generators' identity (built engine code, config, owned generators, templates, declared extra inputs, project config) and the output-changing flags, before anything renders. A match with the last successful run, with every generated file still holding that run's bytes, renders and writes nothing. --force (Maven: -Dmetaobjects.force=true) turns it off; a generator that cannot declare its inputs opts out. - A run that names entities (positional, --entities, -Dmeta.entities) forces exactly those: it never skips, never stores a key and leaves the stored full-run key alone, in every port. - verify --codegen may answer "no drift" locally from the key; CI and --full always render, and every port compares in memory. - The hash manifest stays loaded once and written once, and only when an entry changed; identical bytes are never rewritten; TypeScript runs git merge-file only for a hand-edited file. - fixtures/codegen-skip-conformance/cases.json: the shared case table. Hand-resolved: TypeScript runner, overwrite-policy, gen/verify/args/index and codegen-drift; Python cli, runner, overwrite_policy; C# CodegenRunner, CodegenCli, CodegenDrift, Generator, Program; Java generate and verify mojos; docs, changelog and the ADR index. * no-mistakes(review): apply reviewer inline fixes * no-mistakes(review): apply reviewer inline fixes * no-mistakes(review): warn unknown attrs before gen skip; add regression test * no-mistakes(review): apply reviewer inline fixes * no-mistakes(review): Carry warnings in skipped gen json/toon document * no-mistakes(test): Align C# agent-context fixture wording; sdk suite green * no-mistakes(document): align codegen-skip docs with ADR-0064
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
gen <Entity>now narrows only the files it writes. Every generator sees the whole model inctx.entitiesand the selection asctx.selected(selectedEntities/perEntityapply it), so whole-model files such as the TypeScript barrel and Cube model, and Python's shared enums and object index, are byte-identical to a full run. This is implemented in the TypeScript runner and the Python runner..hashes.jsonis held in memory for the duration of a run and written once at its end, and not at all when nothing changed. This applies to the TypeScript, Python and C# runners (withManifestBatch,manifest_batch,HashManifest.Batch). Python'sgenoutput now lists only the files it wrote, and its summary counts unchanged or skipped files separately. A failed run still records the files it wrote before failing.meta genloads the metadata once, reusing the generation load for the unknown-attribute check, and no longer imports a database driver: migration advice points to the new@metaobjectsdev/migrate-ts/offlinesubpath. The Docker test databases in every port now run on a size-capped tmpfs with Postgres durability off and are removed when each run ends, and the Stryker sandbox lives under$DEV_SCRATCH(scripts/run-stryker.sh).Validation
Risk Assessment
ctx.entitieswithctx.selected, single metadata load and RAM-backed test databases all trace correctly across the TS, Python and C# runners, but the change is large and cross-port. The Python CLI output change (it lists only the files it wrote and counts the rest separately) was reviewed and kept by the task owner; the CHANGELOG records it.Testing
The baseline configured command (ci-local ts-fast + ts-unit) was already green on this tree. I then drove eleven live scenarios: in a disposable RAM-scratch project the real Node
metaCLI showed narrowed runs keeping whole-model files byte-identical (barrel md5-verified), jsonSchemaFile writing only the named entity's schemas, the manifest written exactly once on changed runs and zero times on unchanged runs (strace-counted opens), a failed run recording its 7 partial writes and surfacing the refusal with exit 1, zero database-driver loads under a refusing Node resolve hook (with live controls), and both unknown-attribute warning paths from the single load with correct strict behavior; the real PythonmetaobjectsCLI showed the same narrowing, write-once and verify --codegen in-sync behaviors; the C# lane ran the realdotnet meta gen+verify --codegenand all 5,428 of its tests green including OverwriteHashManifestTests; and both lanes' integration runs proved the throwaway test databases mount size-capped tmpfs with durability off and are removed after the run. Full regression beyond the baseline covered the python and csharp lanes (both LOCAL CI PASSED); Java/Kotlin lanes were not run within budget — the diff's only Java/Kotlin delta is test-container/reaper infrastructure exercised by those ports' integration lanes, with no Java/Kotlin product code in the change. One limitation: this sandbox could not simulate an unwritable manifest for the failed-manifest-write-preserves-the-run's-error path live, so that sub-behavior rests on its unit tests (green in ts-unit and the C# lane). No failures; no action needed.meta gen Postkeeps whole-model files whole: deleted barrel regenerated byte-identical with both entities' exports; unselected entity's files untouchedjsonSchemaFile()on a narrowed run writes only the named entity's schema files (deleted Author schemas are NOT recreated)meta gen Postrecreated 0 of themgenwrites .hashes.json once per run and not at all when nothing changedsrc/generated/blog/content/Post.ts,refused, exit 1, hand edit intact on disk, 1 manifest write, Author.website writtenmeta genloads no database driver: full gen completes under a Node resolve hook refusing kysely/pg/mysql2/better-sqlite3/@libsql, with live controlsmetaobjects gen --entities Postkeeps whole-model files whole: shared enums.py regenerated byte-identical (AuthorRank + PostState); Author.py untouched; unchanged files reported separatelyentitiesstill bounds that target's model: scoped out dir gets only Post's files and enums.py holds only PostStatemetaobjects verify --codegenin sync (10 file(s)) afterwardsgenwrites .hashes.json once per run and not at all when nothing changeddotnet meta genwrites the manifest once per run: real CLI drives gen (3 file(s) written) andverify --codegenreports in sync; OverwriteHashManifestTests greenEvidence: TS narrowed run: barrel byte-identical, Author schemas not recreated
meta gen Post after deleting index.ts + Author schemas: "1 written, 7 unchanged"; barrel md5 identical before/after, exports both Author.js and Post.js; Author schema files recreated: 0Evidence: strace: changed 19-file TS run writes .hashes.json once; unchanged run zero times
An
strace -f -e openatof the realmeta genon a 19-file project showed.gen-state/.hashes.jsonopened for writing exactly once on a run that changed files, and zero times on an unchanged re-run (its mtime was untouched). The raw trace is omitted here: it lists local scratch paths, including six three-way-merge temp files under the system temp directory.Evidence: TS hand-edit refusal: exit 1, partial writes recorded, one manifest write
summary: "7 written, 8 unchanged, 1 refused"; Refused to overwrite .../Post.ts ... NOT overwritten; manifest writes during failed run: 1Evidence: TS gen under no-driver hook: exit 0, zero FORBIDDEN refusals, real writes
Evidence: TS unknown-attr + bare-key warnings; strict fails
Evidence: TS bare-key typo warning path
Evidence: TS strict gen fails the load (exit 1)
Evidence: Python unchanged re-run output
metaobjects gen: wrote 0 file(s) across 1 target(s). 9 file(s) left as they were.Evidence: Python narrowed run: enums.py byte-identical, Author untouched
wrote 1 file(s) across 1 target(s). 5 file(s) left as they were. — regenerated enums.py holds class AuthorRank AND class PostState; cmp: BYTE-IDENTICAL to full runEvidence: CI lane results (python + csharp, both PASSED; C# test suites incl. OverwriteHashManifestTests; live dotnet meta gen)
Evidence: Throwaway test DBs on tmpfs with durability off; run's containers removed
musing_lalande: tmpfs {"/var/lib/postgresql":"size=1g"}, -c fsync=off -c synchronous_commit=off -c full_page_writes=off; jolly_neumann: size=2g, same flagsD5: integration suites and container cleanup (run after the push, no new commit)
Each port's integration suite was run on the pushed head
96f52076, one at a time, against Docker. After each one I compareddocker ps -awith a snapshot taken before the first.integration-tests(bun test, serial)pytest tests/integration -n 8MetaObjects.IntegrationTestsintegration-testsintegration-tests-kotlin+codegen-kotlin-exposed1x-checkThe four non-TS suites were run through
scripts/integration-test.sh <lane>with a private Maven repository andTMPDIRon RAM scratch. Both were deleted afterwards.docker ps -aafter each suite: clean. None of the Postgres, MySQL or Ryuk containers these suites started is still present. The TS reaper exited on its own about 15 seconds after its suite finished. While the Kotlin suite was running, another worktree started its own TypeScript suite. That run'smetaobjects-mysql-<id>container was running when I checked, and I left it alone: it carries the TS/Java name pattern, not Kotlin'smetaobjects-mysql-kt-*. This also covers the Java and Kotlin lanes, which the pipeline's Test step did not run.Two exited
metaobjects-mysql-py-*containers were left by Python runs that happened before this task started, so they predate the change. They were left untouched, as were the unrelated containers of other projects.Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
server/python/src/metaobjects/cli.py:1850- Pythonmetaobjects gennow prints only the files it changed and splits the summary line into 'wrote N file(s)' and 'M file(s) left as they were'. That is not needed to write the manifest once, which is the stated item 1. It changes the output of a command that scripts may parse (unchanged_out,STATUS_UNCHANGED/STATUS_SKIPPED, thechangedlist at cli.py:1850, the summary at cli.py:1861). TypeScript, C# and Java do not do this. This was raised in review round 1; the config-rewrite half was restored, but this half is still present and the CHANGELOG records it as deliberate. The narrower form that satisfies the intent is to keep printing every emitted path and the old count, and keep only the manifest batching and the_recordearly return. Removing it would also drop the test edit in test_cli_verify_requirements.py and thetest_an_unchanged_file_is_not_reported_as_writtentest. Keep it only with the author's explicit sign-off.Outcome: kept. The task owner confirmed the output change; the CHANGELOG
[Unreleased]entry names it for script authors.meta gen Postkeeps whole-model files whole: deleted barrel regenerated byte-identical with both entities' exports; unselected entity's files untouchedjsonSchemaFile()on a narrowed run writes only the named entity's schema files (deleted Author schemas are NOT recreated)meta gen Postrecreated 0 of themgenwrites .hashes.json once per run and not at all when nothing changedsrc/generated/blog/content/Post.ts,refused, exit 1, hand edit intact on disk, 1 manifest write, Author.website writtenmeta genloads no database driver: full gen completes under a Node resolve hook refusing kysely/pg/mysql2/better-sqlite3/@libsql, with live controlsmetaobjects gen --entities Postkeeps whole-model files whole: shared enums.py regenerated byte-identical (AuthorRank + PostState); Author.py untouched; unchanged files reported separatelyentitiesstill bounds that target's model: scoped out dir gets only Post's files and enums.py holds only PostStatemetaobjects verify --codegenin sync (10 file(s)) afterwardsgenwrites .hashes.json once per run and not at all when nothing changeddotnet meta genwrites the manifest once per run: real CLI drives gen (3 file(s) written) andverify --codegenreports in sync; OverwriteHashManifestTests greenscripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchainsbaseline (pre-run, green): scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchainsS1: fullmeta genthenmeta gen Postwith the barrel deleted — barrel regenerated byte-identical (md5 1ded10ec80314740d34ce58e3de878a6 both runs) exporting Author AND Post; only Post's 7 own files + barrel consideredS2:meta gen Postafter deleting all Author schema files — 0 recreated (jsonSchemaFile narrowed fix)S3: strace openat on the real meta CLI — changed 19-file run opens .hashes.json for write exactly once; unchanged re-run opens it zero times and its mtime is untouchedS4: hand-edit refusal run (snapshot removed, file edited, unrelated entity changed) — exit 1, Post.ts refused by name with the hand edit intact, 7 other files written and recorded, manifest written exactly onceS5: Node module resolve hook refusing kysely/pg/mysql2/better-sqlite3/@libsql — controls prove it live (migrate-ts root import refused on kysely; /offline subpath loads clean); realmeta genwith a real content change completes exit 0 with zero refusalsS6: unknown@maximalLengthand bare-keyisAbstrakttypos — lenient gen warns naming attribute, node, file+jsonPath and exits 0;--strictexits 1 before writing (both load paths of loadMemoryWithFindings)S7:metaobjects genfull → 9 files; re-run → "wrote 0 file(s)... 9 file(s) left as they were";--entities Postafter deleting enums.py + Post files — enums.py regenerated byte-identical (AuthorRank + PostState), Author.py byte-identicalS8: Python targetentities: [Post]— fresh out dir gets only Post files, scoped enums.py holds only PostState;metaobjects verify --codegenreports in sync (10 file(s))S9: strace on the real metaobjects CLI — changed run writes .hashes.json once, unchanged run zero timesS10: C# lane drove the realdotnet meta gen(3 file(s) written) +dotnet meta verify --codegenOK; 1590 MetaObjects.Codegen.Tests passed incl. OverwriteHashManifestTestsS11: drive-wear — integration lanes ran the throwaway containers; docker inspect shows tmpfs on /var/lib/postgresql (size=1g / 2g) with fsync/synchronous_commit/full_page_writes off; lane sidecars reaped, my containers removed at teardownregression: scripts/ci-local.sh --only python --strict-toolchains (PASSED) and --only csharp --strict-toolchains (PASSED)🔧 **Document** - 2 issues found → auto-fixed (2) ✅
server/typescript/packages/codegen-ts/src/index.ts:10:selectedEntitiesis now exported besideperEntity/perPackage/perModel, so owned generators outside the package can import the predicate the docs tell them to apply.AGENTS.mdgains the "Disk I/O: keep throwaway data off the drives" section, naming where this repository starts its test databases,CONTRIBUTING.mdgains a test-storage note, and the CHANGELOG[Unreleased]section has an entry for the tmpfs, durability-off, container-removal and Stryker-sandbox changes.ctx.selected,selectedEntities) is documented for generator authors.✅ Re-checked - no issues remain.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.