From 714311ef8c9840674d797754205ea2730717589c Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 11:40:47 -0500 Subject: [PATCH 1/9] fix(codegen): write the manifest once, keep whole-model files whole, load metadata once - TypeScript, Python and C# hold .gen-state/.hashes.json in memory for a gen run and write it once at the end (Java already did); an unchanged run writes nothing. Python gen no longer lists unchanged files as written. - A run narrowed to some entities (meta gen , metaobjects gen --entities) writes only those entities' own files; every generator still sees the whole model (ctx.entities), with the narrowing as ctx.selected. Whole-model files are now byte-identical to a full run's. A Python target's declared entities still bound its model. - meta gen reuses its generation load for the unknown-attribute warning, re-parsing strictly only when a bare unknown key was reported. - meta gen no longer imports a database driver: CLI modules on the gen path read the new @metaobjectsdev/migrate-ts/offline subpath. Gated by an import-graph test. - Under Node the config file loads from its own path, so jiti's cache hits instead of filling with randomly named entries. --- CHANGELOG.md | 24 +++ .../OverwriteHashManifestTests.cs | 46 ++++++ .../MetaObjects.Codegen/CodegenRunner.cs | 3 + .../MetaObjects.Codegen/HashManifest.cs | 95 ++++++++++- server/python/src/metaobjects/cli.py | 86 ++++++---- .../src/metaobjects/codegen/generator.py | 13 +- .../generators/report_export_generator.py | 2 + .../metaobjects/codegen/overwrite_policy.py | 71 +++++++- .../src/metaobjects/codegen/project_config.py | 9 + .../python/src/metaobjects/codegen/runner.py | 155 +++++++++++------- .../codegen/test_cli_verify_requirements.py | 3 + .../test_fr019_shared_provided_conformance.py | 8 +- .../python/tests/codegen/test_gen_run_core.py | 119 ++++++++++++++ .../packages/cli/src/commands/gen.ts | 12 +- .../packages/cli/src/lib/agent-schema.ts | 2 +- .../typescript/packages/cli/src/lib/allow.ts | 4 +- .../cli/src/lib/gen-migrate-advice.ts | 2 +- .../cli/src/lib/load-metaobjects-config.ts | 9 +- .../cli/src/lib/schema-build-options.ts | 2 +- .../cli/src/lib/unknown-attr-warnings.ts | 22 +-- .../cli/test/gen-import-graph.test.ts | 58 +++++++ .../packages/codegen-ts/src/generator.ts | 14 +- .../codegen-ts/src/generators/cube-model.ts | 2 +- .../codegen-ts/src/generators/names-file.ts | 3 +- .../src/generators/report-export-file.ts | 3 +- .../codegen-ts/src/overwrite-policy.ts | 78 ++++++++- .../codegen-ts/src/reference/cube-model.ts | 2 +- .../codegen-ts/src/reference/names.ts | 3 +- .../codegen-ts/src/reference/report-export.ts | 3 +- .../packages/codegen-ts/src/runner.ts | 124 ++++++++------ .../test/manifest-write-once.test.ts | 68 ++++++++ .../packages/codegen-ts/test/run-gen.test.ts | 40 +++-- .../packages/migrate-ts/package.json | 5 + .../packages/migrate-ts/src/offline.ts | 18 ++ server/typescript/packages/sdk/src/index.ts | 2 +- server/typescript/packages/sdk/src/memory.ts | 59 ++++++- .../test/load-memory-with-findings.test.ts | 91 ++++++++++ .../src/generators/report-export.ts | 3 +- 38 files changed, 1041 insertions(+), 222 deletions(-) create mode 100644 server/python/tests/codegen/test_gen_run_core.py create mode 100644 server/typescript/packages/cli/test/gen-import-graph.test.ts create mode 100644 server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts create mode 100644 server/typescript/packages/migrate-ts/src/offline.ts create mode 100644 server/typescript/packages/sdk/test/load-memory-with-findings.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0714a58b48..39642e5575 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -297,6 +297,30 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi ### Fixed +- **`gen ` no longer shrinks the files rendered from the whole model.** Narrowing a run + to some entities (`meta gen Post`, Python `metaobjects gen --entities Post`) handed every + generator only the named entities, so a whole-model file (the TypeScript barrel and Cube + model, Python's shared enums module, duration codec and report writers, and the object index + Python's entity and route generators resolve relations through) was rendered from just them. A narrowed run now writes only the named entities' own files, + and every whole-model file it writes is byte-identical to a full run's. A generator sees the + whole model in `ctx.entities` and the narrowing as `ctx.selected`; `perEntity` (`per_entity`) + applies it. A Python target's declared `entities` still bounds that target's model. C# and + Java take no entity filter and were not affected. +- **`gen` writes `.hashes.json` once per run, and not at all when nothing changed.** TypeScript, + Python and C# re-read and re-wrote the whole manifest after every generated file, so a run of + N files wrote it N times; they now hold it in memory for the run and write it once, as Java + already did. Python's `metaobjects gen` also stops listing unchanged files as written: it + prints the files it wrote and counts the unchanged ones separately. +- **`meta gen` loads the metadata once.** The unknown-attribute warning re-loaded the project + strictly on every run; it now reuses the generation load and parses strictly again only when + the lenient load reported an unknown bare key. `meta gen` also no longer imports a database + driver: the migration advice it prints reads `@metaobjectsdev/migrate-ts/offline`, a new + subpath with the schema builder and diff and none of the introspection or apply code. +- **`meta gen` under Node stops filling the jiti cache.** Loading `metaobjects.config.ts` + rewrote it to a randomly named temp file, which missed jiti's cache on every run and left a + new cache entry behind each time. Under Node the config now loads from its own path, with the + package aliases applied by jiti. + - **A projection view keeps the columns a reverse `@via` hop feeds.** An origin `@via` that walks an `identity.reference` backwards (`Project.timeEntries`, #560) loaded, but `meta migrate` left its `origin.aggregate` / `origin.first` columns out of `CREATE VIEW` without a word. The view diff --git a/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs b/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs index 541472b5f4..7918e3b07f 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs @@ -107,6 +107,52 @@ private static MetaRoot Load() return r.Root; } + /// Emits several files LAZILY, noting before each one whether the manifest is on + /// disk yet. The runner writes each file as it is yielded, so a manifest saved per + /// recorded file appears on disk part-way through the run. + private sealed class ObservingGenerator(string stateDir, string[] contents) : IGenerator + { + public readonly List ManifestOnDisk = new(); + public string Name => "observing"; + public IEnumerable Generate(GenContext ctx) + { + for (var i = 0; i < contents.Length; i++) + { + ManifestOnDisk.Add(File.Exists(Path.Combine(stateDir, ".hashes.json"))); + yield return new EmittedFile($"F{i}.cs", contents[i]); + } + } + } + + private (CodegenRunner.RunResult Result, ObservingGenerator Gen) RunObserving(params string[] contents) + { + var gen = new ObservingGenerator(StateDir, contents); + var config = new GenConfig { OutDir = OutDir, Namespace = "Probe", GenStateDir = StateDir }; + return (CodegenRunner.Run(config, Load(), new IGenerator[] { gen }), gen); + } + + [Fact] + public void ARun_WritesTheManifestOnce_AtTheEnd() + { + var bodies = Enumerable.Range(0, 4).Select(i => $"// \nclass F{i} {{ }}\n").ToArray(); + var (first, gen) = RunObserving(bodies); + Assert.All(first.Files, f => Assert.Equal("written", f.Status)); + // Not on disk while the run is writing: it is saved once, after the last file. + Assert.Equal(new[] { false, false, false, false }, gen.ManifestOnDisk); + var manifest = Path.Combine(StateDir, ".hashes.json"); + var recorded = File.ReadAllText(manifest); + using (var doc = JsonDocument.Parse(recorded)) + Assert.Equal(4, doc.RootElement.EnumerateObject().Count()); + + // An unchanged run records nothing new and leaves the file untouched. + var stamp = new DateTime(2001, 1, 1, 0, 0, 0, DateTimeKind.Utc); + File.SetLastWriteTimeUtc(manifest, stamp); + var (again, _) = RunObserving(bodies); + Assert.All(again.Files, f => Assert.Equal("unchanged", f.Status)); + Assert.Equal(stamp, File.GetLastWriteTimeUtc(manifest)); + Assert.Equal(recorded, File.ReadAllText(manifest)); + } + [Fact] public void HandEditedFile_IsRefused_AndSurvives() { diff --git a/server/csharp/MetaObjects.Codegen/CodegenRunner.cs b/server/csharp/MetaObjects.Codegen/CodegenRunner.cs index 1dcf4b1407..2e20cbc999 100644 --- a/server/csharp/MetaObjects.Codegen/CodegenRunner.cs +++ b/server/csharp/MetaObjects.Codegen/CodegenRunner.cs @@ -88,6 +88,9 @@ internal static RunResult Run( // Captured BEFORE any write, because the first write creates the manifest — // read it afterwards and every project looks migrated. var hadManifest = config.GenStateDir is null || HashManifest.Exists(config.GenStateDir); + // One manifest write for the whole run (the writes and the orphan sweep), not one + // per recorded file. + using var manifestBatch = HashManifest.Batch(config.GenStateDir); foreach (var generator in generators) { diff --git a/server/csharp/MetaObjects.Codegen/HashManifest.cs b/server/csharp/MetaObjects.Codegen/HashManifest.cs index 0024ffa961..a259cbaaf0 100644 --- a/server/csharp/MetaObjects.Codegen/HashManifest.cs +++ b/server/csharp/MetaObjects.Codegen/HashManifest.cs @@ -48,9 +48,72 @@ public static string ContentHash(string content) => /// committed, so every refusal in it shares one cause and deserves one instruction /// rather than one warning per file. /// - public static bool Exists(string genStateDir) => File.Exists(PathFor(genStateDir)); + public static bool Exists(string genStateDir) => + File.Exists(PathFor(genStateDir)) || (BatchFor(genStateDir)?.Dirty ?? false); - private static Dictionary ReadRaw(string genStateDir) + private sealed class ManifestBatch(Dictionary raw) + { + public Dictionary Raw = raw; + public bool Dirty; + } + + // One in-memory manifest per gen-state dir while a run is in progress. Without it every + // recorded file re-read and re-wrote the whole manifest, so a run of N files wrote it N + // times. Mirrors codegen-ts's withManifestBatch, Python's manifest_batch and Java's + // per-run save. + private static readonly Dictionary Batches = new(StringComparer.Ordinal); + + private static ManifestBatch? BatchFor(string genStateDir) + { + lock (Batches) return Batches.GetValueOrDefault(Path.GetFullPath(genStateDir)); + } + + /// + /// Hold the manifest in memory until the returned handle is disposed, then write it at + /// most once. A nested batch for the same directory joins the outer one. The write + /// happens even when the run throws: every file recorded so far IS on disk, and dropping + /// its record would make the next run refuse output it wrote itself. + /// + public static IDisposable Batch(string? genStateDir) + { + if (genStateDir is null) return NoBatch.Instance; + var key = Path.GetFullPath(genStateDir); + lock (Batches) + { + if (Batches.ContainsKey(key)) return NoBatch.Instance; + Batches[key] = new ManifestBatch(ReadFile(genStateDir)); + } + return new BatchHandle(genStateDir, key); + } + + private sealed class NoBatch : IDisposable + { + public static readonly NoBatch Instance = new(); + public void Dispose() { } + } + + private sealed class BatchHandle(string genStateDir, string key) : IDisposable + { + private bool _disposed; + + public void Dispose() + { + if (_disposed) return; + _disposed = true; + ManifestBatch batch; + lock (Batches) + { + batch = Batches[key]; + Batches.Remove(key); + } + if (batch.Dirty) WriteFile(genStateDir, batch.Raw); + } + } + + private static Dictionary ReadRaw(string genStateDir) => + BatchFor(genStateDir)?.Raw ?? ReadFile(genStateDir); + + private static Dictionary ReadFile(string genStateDir) { var path = PathFor(genStateDir); if (!File.Exists(path)) return new Dictionary(StringComparer.Ordinal); @@ -98,9 +161,8 @@ public static Dictionary Load(string genStateDir) /// public static void Save(string genStateDir, Dictionary hashes) { - Directory.CreateDirectory(genStateDir); var raw = ReadRaw(genStateDir); - var ordered = new SortedDictionary(StringComparer.Ordinal); + var next = new Dictionary(StringComparer.Ordinal); foreach (var (key, hash) in hashes) { if (raw.TryGetValue(key, out var existing) && existing.ValueKind == JsonValueKind.Object) @@ -108,10 +170,23 @@ public static void Save(string genStateDir, Dictionary hashes) var entry = new Dictionary(StringComparer.Ordinal); foreach (var prop in existing.EnumerateObject()) entry[prop.Name] = prop.Value; entry["hash"] = hash; - ordered[key] = entry; + next[key] = JsonSerializer.SerializeToElement(entry); } - else ordered[key] = hash; + else next[key] = JsonSerializer.SerializeToElement(hash); + } + if (BatchFor(genStateDir) is { } batch) + { + batch.Raw = next; + batch.Dirty = true; + return; } + WriteFile(genStateDir, next); + } + + private static void WriteFile(string genStateDir, Dictionary raw) + { + Directory.CreateDirectory(genStateDir); + var ordered = new SortedDictionary(raw, StringComparer.Ordinal); var json = JsonSerializer.Serialize(ordered, new JsonSerializerOptions { WriteIndented = true }); File.WriteAllText(PathFor(genStateDir), json + "\n"); } @@ -127,7 +202,13 @@ public static void Record( string genStateDir, string relPath, string content, string? legacyRelPath = null) { var hashes = Load(genStateDir); - hashes[relPath] = ContentHash(content); + var hash = ContentHash(content); + var staleLegacy = legacyRelPath is not null + && !string.Equals(legacyRelPath, relPath, StringComparison.Ordinal) + && hashes.ContainsKey(legacyRelPath); + // Already recorded exactly so: an unchanged file does not rewrite the manifest. + if (!staleLegacy && hashes.TryGetValue(relPath, out var recorded) && recorded == hash) return; + hashes[relPath] = hash; if (legacyRelPath is not null && !string.Equals(legacyRelPath, relPath, StringComparison.Ordinal)) hashes.Remove(legacyRelPath); Save(genStateDir, hashes); diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 7064094474..d313e7519f 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -115,6 +115,7 @@ from metaobjects.codegen.overwrite_policy import ( has_hash_manifest, is_pristine_generated, + manifest_batch, read_generated_hash, ) from metaobjects.codegen.project_config import ( @@ -929,6 +930,8 @@ def _run_suite( output_layout: str = "flat", overwrite_hand_edits: frozenset[str] = frozenset(), overwrite_paths_out: set[str] | None = None, + unchanged_out: set[str] | None = None, + entity_scope: list[str] | None = None, ) -> list[str]: """Run a generator suite against an ALREADY-LOADED ``root`` into ``out_dir``. @@ -988,7 +991,8 @@ def _run_suite( raise NoGeneratorsSelectedError suite = generators result = run_gen( - config, root, generators=suite, entity_filter=entity_filter, select=select, in_ledger=in_ledger + config, root, generators=suite, entity_filter=entity_filter, select=select, + in_ledger=in_ledger, entity_scope=entity_scope, ) if overwrite_paths_out is not None: overwrite_paths_out.update( @@ -1002,6 +1006,13 @@ def _run_suite( # and only the ones that own an exit code care about the refusals. if refused_out is not None: refused_out.extend(path for path, status in result.files if status == "refused") + # Emitted but not written: the file already held this output. Still in the return + # value (the orphan sweep and the cross-target guard need every emitted path), but a + # caller reporting what it WROTE must leave these out. + if unchanged_out is not None: + unchanged_out.update( + path for path, status in result.files if status in ("unchanged", "skipped") + ) return [path for path, status in result.files if status != "refused"] @@ -1528,13 +1539,15 @@ def _narrow_entities( targets: list[TargetConfig], args: argparse.Namespace ) -> list[TargetConfig]: """Apply ``--entities`` to each target: only the named entities are emitted, and a - target that already lists ``entities`` keeps only the names in both.""" + target that already lists ``entities`` keeps only the names in both. The target's + ``entities`` (its model) is left alone: the narrowing is a ``run_selection``.""" names = _parse_entities(getattr(args, "entities", None)) if names is None: return targets return [ dataclasses.replace( - t, entities=names if t.entities is None else [n for n in t.entities if n in names] + t, + run_selection=names if t.entities is None else [n for n in t.entities if n in names], ) for t in targets ] @@ -1569,6 +1582,7 @@ def _run_gen_targets( selection: list[str] | None = None, targeted_out: set[str] | None = None, overwrite_paths_out: set[str] | None = None, + unchanged_out: set[str] | None = None, ) -> tuple[list[str], list[str]]: """Run each target's declared generators into its ``outDir``. Returns (all_written, errors). @@ -1690,19 +1704,21 @@ def selected(gen: Generator, token: str) -> bool: try: if gens: written = _run_suite( - root, out_dir, gens, t.entities, gen_state_dir=gen_state_dir, + root, out_dir, gens, t.written_entities, gen_state_dir=gen_state_dir, column_naming=column_naming, project_root=project_root, baseline=baseline, refused_out=refused_out, select=select, in_ledger=in_ledger, api_prefix=config.api_prefix, output_layout=config.output_layout, overwrite_hand_edits=overwrite, overwrite_paths_out=overwrite_paths_out, + unchanged_out=unchanged_out, entity_scope=t.entities, ) if spec_gens: written = written + _run_suite( - root, out_dir, spec_gens, t.entities, emit_package_init=False, + root, out_dir, spec_gens, t.written_entities, emit_package_init=False, gen_state_dir=gen_state_dir, column_naming=column_naming, project_root=project_root, baseline=baseline, refused_out=refused_out, select=select, in_ledger=in_ledger, api_prefix=config.api_prefix, - output_layout=config.output_layout, + output_layout=config.output_layout, unchanged_out=unchanged_out, + entity_scope=t.entities, ) except ValueError as exc: # intra-target run_gen collision → clean error errors.append(f"target '{t.name}': {exc}") @@ -1798,43 +1814,49 @@ def _cmd_gen_config(args: argparse.Namespace) -> int: if _refuse_narrowing(args, selection, root, "metaobjects gen"): return 2 for t in targets: - imported_refusal = _refuse_imported_entities(t.entities, root, collection) + imported_refusal = _refuse_imported_entities(t.written_entities, root, collection) if imported_refusal is not None: print(imported_refusal, file=sys.stderr) return 2 refused: list[str] = [] targeted: set[str] = set() + unchanged: set[str] = set() baseline = getattr(args, "baseline", None) or "default" gen_state_dir = gen_state_dir_for(config.metadata_dir()) project_root = str(project_root_for(config.metadata_dir())) - written, errors = _run_gen_targets( - config, targets, root, - gen_state_dir=gen_state_dir, - project_root=project_root, - baseline=baseline, - refused_out=refused, - select=collection.in_scope, - in_ledger=collection.in_ledger, - build_ctx=GeneratorBuildContext(template_root), - column_naming=column_naming, - selection=selection, - targeted_out=targeted, - ) - if errors: - for msg in errors: - print(f"error: {msg}", file=sys.stderr) - return 1 - for path in written: - print(path) - removed = _sweep_orphans_after_gen( - args, config, targets, targeted, selection, baseline, gen_state_dir, project_root, - emitted={*written, *refused}, - ) + # Every target and the orphan sweep share one in-memory manifest: one write per run. + with manifest_batch(gen_state_dir): + written, errors = _run_gen_targets( + config, targets, root, + gen_state_dir=gen_state_dir, + project_root=project_root, + baseline=baseline, + refused_out=refused, + select=collection.in_scope, + in_ledger=collection.in_ledger, + build_ctx=GeneratorBuildContext(template_root), + column_naming=column_naming, + selection=selection, + targeted_out=targeted, + unchanged_out=unchanged, + ) + if errors: + for msg in errors: + print(f"error: {msg}", file=sys.stderr) + return 1 + changed = [path for path in written if path not in unchanged] + for path in changed: + print(path) + removed = _sweep_orphans_after_gen( + args, config, targets, targeted, selection, baseline, gen_state_dir, project_root, + emitted={*written, *refused}, + ) for path in removed: print(f"removed {path}") print( - f"metaobjects gen: wrote {len(written)} file(s) across {len(targets)} target(s)." + f"metaobjects gen: wrote {len(changed)} file(s) across {len(targets)} target(s)." + + (f" {len(unchanged)} file(s) unchanged." if unchanged else "") + (f" Removed {len(removed)} file(s) no generator emits any more." if removed else "") ) if refused: @@ -2131,7 +2153,7 @@ def _verify_codegen_config(args: argparse.Namespace) -> int: if _refuse_narrowing(args, selection, root, "metaobjects verify --codegen"): return 2 for t in closure: - imported_refusal = _refuse_imported_entities(t.entities, root, collection) + imported_refusal = _refuse_imported_entities(t.written_entities, root, collection) if imported_refusal is not None: print(imported_refusal, file=sys.stderr) return 2 diff --git a/server/python/src/metaobjects/codegen/generator.py b/server/python/src/metaobjects/codegen/generator.py index 460dd475d5..d6dfeca496 100644 --- a/server/python/src/metaobjects/codegen/generator.py +++ b/server/python/src/metaobjects/codegen/generator.py @@ -40,6 +40,14 @@ class GenContext: #: its generated package directory, and a second claimant is refused with #: ``ERR_NAME_COLLISION``. ``run_gen`` shares one across every generator of the run. claims: NameClaims = field(default_factory=NameClaims) + #: ``gen --entities``: whether an entity in ``entities`` was selected. ``entities`` is + #: always the WHOLE model in scope, so a whole-model file renders the same on a narrowed + #: run; a per-entity file honours this as well (``per_entity`` applies it). ``None``: + #: every entity is selected. + selected: Callable[[MetaObject], bool] | None = None + + def is_selected(self, entity: MetaObject) -> bool: + return self.selected is None or self.selected(entity) def claim(self, scope: str, name: str, claimant: str, remedy: str) -> None: """Claim *name* in *scope* for the metadata node *claimant* (its ``resolution_key()``).""" @@ -60,7 +68,7 @@ def per_entity( def run(ctx: GenContext) -> list[EmittedFile]: out: list[EmittedFile] = [] for e in ctx.entities: - if not ctx.matches(e): + if not ctx.matches(e) or not ctx.is_selected(e): continue r = fn(e, ctx) out.extend(r if isinstance(r, list) else [r]) @@ -72,7 +80,8 @@ def run(ctx: GenContext) -> list[EmittedFile]: def once_per_run( fn: Callable[[list[MetaObject], GenContext], "EmittedFile | list[EmittedFile]"], ) -> Callable[[GenContext], list[EmittedFile]]: - """Called once with all matching entities (barrels / cross-entity files).""" + """Called once with all matching entities (barrels / cross-entity files) — the whole + model, even on a ``gen --entities`` run.""" def run(ctx: GenContext) -> list[EmittedFile]: matched = [e for e in ctx.entities if ctx.matches(e)] diff --git a/server/python/src/metaobjects/codegen/generators/report_export_generator.py b/server/python/src/metaobjects/codegen/generators/report_export_generator.py index f7d57a3ad5..3d74154c6f 100644 --- a/server/python/src/metaobjects/codegen/generators/report_export_generator.py +++ b/server/python/src/metaobjects/codegen/generators/report_export_generator.py @@ -227,6 +227,8 @@ def generate(self, ctx: GenContext) -> list[EmittedFile]: ) ] for report in reports: + if not ctx.is_selected(report): + continue # the writers module is shared; a report's own module is per-entity files.append( EmittedFile( path=claimed_module_path( diff --git a/server/python/src/metaobjects/codegen/overwrite_policy.py b/server/python/src/metaobjects/codegen/overwrite_policy.py index c83a0e10d5..5191a30192 100644 --- a/server/python/src/metaobjects/codegen/overwrite_policy.py +++ b/server/python/src/metaobjects/codegen/overwrite_policy.py @@ -34,6 +34,8 @@ import hashlib import json import os +from collections.abc import Iterator +from contextlib import contextmanager from .constants import GENERATED_MARKER @@ -71,10 +73,58 @@ def has_hash_manifest(gen_state_dir: str) -> bool: manifest predates the manifest being committed, so all its refusals share one cause and deserve one instruction rather than one warning per file. """ - return os.path.exists(_manifest_path(gen_state_dir)) + batch = _batches.get(os.path.abspath(gen_state_dir)) + return os.path.exists(_manifest_path(gen_state_dir)) or bool(batch and batch.dirty) + + +class _ManifestBatch: + __slots__ = ("raw", "dirty") + + def __init__(self, raw: dict[str, object]) -> None: + self.raw = raw + self.dirty = False + + +# One in-memory manifest per gen-state dir while a run is in progress. Without it every +# recorded file re-read and re-wrote the whole manifest, so a run of N files wrote it N +# times. Mirrors codegen-ts's ``withManifestBatch`` and C#'s and Java's per-run save. +_batches: dict[str, _ManifestBatch] = {} + + +@contextmanager +def manifest_batch(gen_state_dir: str | None) -> Iterator[None]: + """Hold the manifest in memory for the block and write it at most once, at the end. + + A nested batch for the same directory joins the outer one, so a CLI run over several + targets plus the orphan sweep writes the manifest once. The write happens even when + the block raises: every file recorded so far IS on disk, and dropping its record + would make the next run refuse output it wrote itself. + """ + if gen_state_dir is None: + yield + return + key = os.path.abspath(gen_state_dir) + if key in _batches: + yield + return + batch = _ManifestBatch(_read_manifest_file(gen_state_dir)) + _batches[key] = batch + try: + yield + finally: + del _batches[key] + if batch.dirty: + _write_manifest_file(gen_state_dir, batch.raw) def _read_raw(gen_state_dir: str) -> dict[str, object]: + batch = _batches.get(os.path.abspath(gen_state_dir)) + if batch is not None: + return batch.raw + return _read_manifest_file(gen_state_dir) + + +def _read_manifest_file(gen_state_dir: str) -> dict[str, object]: path = _manifest_path(gen_state_dir) if not os.path.exists(path): return {} @@ -108,7 +158,6 @@ def _load_hashes(gen_state_dir: str) -> dict[str, str]: def _save_hashes(gen_state_dir: str, hashes: dict[str, str]) -> None: - os.makedirs(gen_state_dir, exist_ok=True) raw = _read_raw(gen_state_dir) # Keys SORTED, because this file is committed. Insertion order would make the # diff — and any merge conflict between two people who both regenerated — depend @@ -120,6 +169,16 @@ def _save_hashes(gen_state_dir: str, hashes: dict[str, str]) -> None: ordered[k] = {**existing, "hash": hashes[k]} else: ordered[k] = hashes[k] + batch = _batches.get(os.path.abspath(gen_state_dir)) + if batch is not None: + batch.raw = ordered + batch.dirty = True + return + _write_manifest_file(gen_state_dir, ordered) + + +def _write_manifest_file(gen_state_dir: str, ordered: dict[str, object]) -> None: + os.makedirs(gen_state_dir, exist_ok=True) with open(_manifest_path(gen_state_dir), "w", encoding="utf-8") as fh: json.dump(ordered, fh, indent=2) fh.write("\n") @@ -160,7 +219,13 @@ def _record( gen_state_dir: str, rel_path: str, content: str, legacy_rel_path: str | None = None ) -> None: hashes = _load_hashes(gen_state_dir) - hashes[rel_path] = content_hash(content) + digest = content_hash(content) + stale_legacy = ( + legacy_rel_path is not None and legacy_rel_path != rel_path and legacy_rel_path in hashes + ) + if hashes.get(rel_path) == digest and not stale_legacy: + return # already recorded exactly so: an unchanged file does not rewrite the manifest + hashes[rel_path] = digest # Drop the pre-re-keying spelling in the same write. Leaving it would keep a key # that means "this name, relative to whichever out dir ran last" alive forever — # which is the ambiguity being removed. diff --git a/server/python/src/metaobjects/codegen/project_config.py b/server/python/src/metaobjects/codegen/project_config.py index 7e2014d7f3..03e4b00477 100644 --- a/server/python/src/metaobjects/codegen/project_config.py +++ b/server/python/src/metaobjects/codegen/project_config.py @@ -148,6 +148,15 @@ class TargetConfig: #: The ``generators`` entries (as written in ``generators``) declared with #: ``handEdits: overwrite``. Every other entry keeps hand edits. overwrite_hand_edits: frozenset[str] = frozenset() + #: ``gen --entities``: the entities this RUN writes, within ``entities``. Never read + #: from the file. ``entities`` stays the target's model, so a whole-model file (shared + #: enums, the duration codec, the report writers) still renders from all of it. + run_selection: list[str] | None = None + + @property + def written_entities(self) -> list[str] | None: + """The entities whose output this run writes: the selection, else the target's.""" + return self.run_selection if self.run_selection is not None else self.entities @dataclass(frozen=True) diff --git a/server/python/src/metaobjects/codegen/runner.py b/server/python/src/metaobjects/codegen/runner.py index 54a832173f..84fcaab362 100644 --- a/server/python/src/metaobjects/codegen/runner.py +++ b/server/python/src/metaobjects/codegen/runner.py @@ -19,7 +19,12 @@ from .import_path import OUTPUT_LAYOUT_PACKAGE from .name_claims import NameClaims, assert_importable_package_tree from .instance_artifacts import is_served_report -from .overwrite_policy import STATUS_OVERWROTE_EDIT, decide_and_write, has_hash_manifest +from .overwrite_policy import ( + STATUS_OVERWROTE_EDIT, + decide_and_write, + has_hash_manifest, + manifest_batch, +) _VALID_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") @@ -72,9 +77,15 @@ def run_gen( select: Callable[[str], bool] | None = None, merge_strategy: str = "overwrite", in_ledger: Callable[[str], bool] | None = None, + entity_scope: list[str] | None = None, ) -> RunGenResult: """Mirrors ``codegen-ts``'s ``runner.ts``. + ``entity_filter`` (``gen --entities``) narrows what is WRITTEN: per-entity files for + the named entities only, while every generator still sees the whole model in + ``ctx.entities`` (``ctx.selected`` says which entities were named). ``entity_scope`` + narrows the model itself — a config target's declared ``entities``. + ``select`` (FR-023 §11.1 item 2) — applied AFTER ``entity_filter``, over each object's ``resolution_key()`` (never the bare name — two packages may declare the same short name). Normally a resolved ``Collection``'s ``in_scope``: the @@ -92,6 +103,19 @@ def run_gen( objs = _objects(metadata) model_has_objects = bool(objs) + # The WHOLE model in scope. `entity_filter` narrows what is written, never what a + # generator sees: a whole-model file (shared enums, the duration codec, the report + # writers, relation targets in the object index) rendered from the filtered set would + # shrink to it, and `gen ` would rewrite those files for one entity. + # + # ``entity_scope`` is the one thing that does shrink it: a config target's declared + # ``entities``, which IS that target's model (two targets may split one out dir by + # entity, and each must render its shared files from its own half). + whole = [ + o for o in objs + if (entity_scope is None or o.name in entity_scope) + and (select is None or select(o.resolution_key())) + ] if entity_filter is not None: objs = [o for o in objs if o.name in entity_filter] if select is not None: @@ -115,12 +139,21 @@ def run_gen( # read-only generators emit as a keyless object. Every other report generates nothing: # a sourceless report, or one over a kind the lowering skips, has no relation to read. # Decided here, at the entity set every generator reads, and not per generator. - objs = [ - report_read_model(o, metadata) if o.sub_type == OBJECT_SUBTYPE_REPORT else o - for o in objs - if o.sub_type != OBJECT_SUBTYPE_REPORT or is_served_report(o) - ] - if not objs and not result.warnings: + def _generatable(candidates: list[MetaObject]) -> list[MetaObject]: + return [ + report_read_model(o, metadata) if o.sub_type == OBJECT_SUBTYPE_REPORT else o + for o in candidates + if o.sub_type != OBJECT_SUBTYPE_REPORT or is_served_report(o) + ] + + narrowed = entity_filter is not None + selected_objs = _generatable(objs) + selected_keys = {o.resolution_key() for o in [*objs, *selected_objs]} + + def is_selected(o: MetaObject) -> bool: + return o.resolution_key() in selected_keys + + if not selected_objs and not result.warnings: result.warnings.append( "No entities to generate — every selected object is an object.report with no " "view source, which has no generated output (FR-044)." @@ -128,14 +161,15 @@ def run_gen( return result safe: list[MetaObject] = [] - for o in objs: + for o in _generatable(whole) if narrowed else selected_objs: if not _VALID_NAME.match(o.name): - result.warnings.append( - f"Skipping entity with unsafe name {o.name!r} — must match ^[A-Za-z_]\\w*$." - ) + if is_selected(o): + result.warnings.append( + f"Skipping entity with unsafe name {o.name!r} — must match ^[A-Za-z_]\\w*$." + ) continue safe.append(o) - if not safe and objs: + if selected_objs and not any(is_selected(o) for o in safe): return result # Refuse before any generator runs — the one choke point that sees the WHOLE @@ -154,6 +188,7 @@ def run_gen( generators=tuple(g.name for g in generators), in_ledger=in_ledger, claims=claims, + selected=is_selected if narrowed else None, ) for f in gen.generate(ctx): full = os.path.join(config.out_dir, f.path) @@ -200,54 +235,56 @@ def run_gen( refused: list[str] = [] adopted: list[str] = [] untracked = 0 - for full, (content, by) in emitted.items(): - result.generated_by[full] = by - # The out-dir-relative name: what the manifest used to be keyed by, and still - # what a refusal is REPORTED as (it is the name the reader sees in `--out`). - rel = os.path.relpath(full, config.out_dir) - # The manifest key. Project-root-relative when the caller knows the project, so - # two runs with different --out cannot collide on one key inside the single - # manifest that is anchored on that project. `rel` rides along as the legacy - # spelling so an existing manifest keeps working and converges on the new one. - # BOTH SIDES RESOLVED. `project_root` arrives resolved (project_root_for calls - # .resolve()) while `full` is built from the raw `out_dir`, and relpath between a - # resolved anchor and an unresolved path is not the path between the two - # directories — under a directory symlink it walks OUT of the project - # (`../link/gen/X.py`). Every key then points outside, `_is_ours_for` matches - # none of them, and `verify --codegen` prints `in sync (0 file(s))` over stale - # output. The pre-existing `relpath(full, out_dir)` was immune by accident: both - # sides came from the same unresolved string, so it was symmetric. Symmetry is - # the property, and resolving both is how it is kept deliberately. - key = ( - os.path.relpath(os.path.realpath(full), config.project_root) - if config.project_root - else rel - ) - # A file written OUTSIDE the project has no place in the project's manifest: its - # key would be `../`-relative and committed into `.hashes.json`. It gets the - # manifest-less rule a run with no project gets — written, never recorded. - outside = bool(config.project_root) and (key.startswith("..") or os.path.isabs(key)) - if outside: - untracked += 1 - status = decide_and_write( - full, - content, - merge_strategy, - gen_state_dir=None if outside else config.gen_state_dir, - rel_path=key, - legacy_rel_path=rel if key != rel else None, - baseline=config.baseline, - hand_edits="overwrite" if by in config.overwrite_hand_edits else "keep", - ) - result.files.append((full, status)) - if status == STATUS_OVERWROTE_EDIT: - result.warnings.append( - f"overwrote hand-edited {rel} with fresh output (handEdits: overwrite on {by})" + # One manifest write for the whole loop, not one per recorded file. + with manifest_batch(config.gen_state_dir): + for full, (content, by) in emitted.items(): + result.generated_by[full] = by + # The out-dir-relative name: what the manifest used to be keyed by, and still + # what a refusal is REPORTED as (it is the name the reader sees in `--out`). + rel = os.path.relpath(full, config.out_dir) + # The manifest key. Project-root-relative when the caller knows the project, so + # two runs with different --out cannot collide on one key inside the single + # manifest that is anchored on that project. `rel` rides along as the legacy + # spelling so an existing manifest keeps working and converges on the new one. + # BOTH SIDES RESOLVED. `project_root` arrives resolved (project_root_for calls + # .resolve()) while `full` is built from the raw `out_dir`, and relpath between a + # resolved anchor and an unresolved path is not the path between the two + # directories — under a directory symlink it walks OUT of the project + # (`../link/gen/X.py`). Every key then points outside, `_is_ours_for` matches + # none of them, and `verify --codegen` prints `in sync (0 file(s))` over stale + # output. The pre-existing `relpath(full, out_dir)` was immune by accident: both + # sides came from the same unresolved string, so it was symmetric. Symmetry is + # the property, and resolving both is how it is kept deliberately. + key = ( + os.path.relpath(os.path.realpath(full), config.project_root) + if config.project_root + else rel + ) + # A file written OUTSIDE the project has no place in the project's manifest: its + # key would be `../`-relative and committed into `.hashes.json`. It gets the + # manifest-less rule a run with no project gets — written, never recorded. + outside = bool(config.project_root) and (key.startswith("..") or os.path.isabs(key)) + if outside: + untracked += 1 + status = decide_and_write( + full, + content, + merge_strategy, + gen_state_dir=None if outside else config.gen_state_dir, + rel_path=key, + legacy_rel_path=rel if key != rel else None, + baseline=config.baseline, + hand_edits="overwrite" if by in config.overwrite_hand_edits else "keep", ) - if status == "refused": - refused.append(rel) - elif status == "adopted": - adopted.append(rel) + result.files.append((full, status)) + if status == STATUS_OVERWROTE_EDIT: + result.warnings.append( + f"overwrote hand-edited {rel} with fresh output (handEdits: overwrite on {by})" + ) + if status == "refused": + refused.append(rel) + elif status == "adopted": + adopted.append(rel) if untracked: result.warnings.append( diff --git a/server/python/tests/codegen/test_cli_verify_requirements.py b/server/python/tests/codegen/test_cli_verify_requirements.py index c8f3dec2a7..47bc05012e 100644 --- a/server/python/tests/codegen/test_cli_verify_requirements.py +++ b/server/python/tests/codegen/test_cli_verify_requirements.py @@ -73,6 +73,9 @@ def test_a_model_with_no_requirements_prints_nothing_and_exits_as_before( # BEFORE: the same run with the gate unplugged — what verify printed without it. monkeypatch.setattr(cli, "_verify_requirements", lambda args, loaded=None: 0) + # From an empty out dir again, so gen writes (and prints) the same files as above + # rather than reporting them unchanged. + shutil.rmtree(tmp_path / "out") before_code = _verify(tmp_path, meta_dir) before = capsys.readouterr() diff --git a/server/python/tests/codegen/test_fr019_shared_provided_conformance.py b/server/python/tests/codegen/test_fr019_shared_provided_conformance.py index 2605473e04..e70de7f15a 100644 --- a/server/python/tests/codegen/test_fr019_shared_provided_conformance.py +++ b/server/python/tests/codegen/test_fr019_shared_provided_conformance.py @@ -277,8 +277,9 @@ def test_narrowed_run_on_only_the_projection_still_materializes_the_shared_enum( ) -> None: """A run narrowed to ONLY ``TicketSummary`` (a projection, via ``entity_filter``) must still emit ``Priority`` — must-generate turns on - whether any SURVIVING consumer references the shared type, never on the - hosting object's subtype.""" + whether any consumer references the shared type, never on the hosting + object's subtype. ``enums.py`` is a whole-model file: the narrowing decides + which per-entity files are written, not what it holds.""" root = MetaDataLoader.from_directory(_FIXTURE_DIR).root out_dir = tmp_path / "generated_narrow" config = GenConfig(out_dir=str(out_dir), provided_enum_namespace="acme_ext.enums") @@ -287,7 +288,8 @@ def test_narrowed_run_on_only_the_projection_still_materializes_the_shared_enum( assert (out_dir / "enums.py").exists(), "narrowing to TicketSummary must still emit enums.py" enums_src = (out_dir / "enums.py").read_text() assert enums_src.count("class Priority(str, Enum):") == 1 - assert "class Money" not in enums_src, "Money has no consumer in the narrowed run" + assert "class Money(str, Enum):" in enums_src, "enums.py renders from the whole model" + assert not (out_dir / "Ticket.py").exists(), "only the named entity's own file is written" ticket_summary_src = (out_dir / "TicketSummary.py").read_text() assert "level: Priority" in ticket_summary_src diff --git a/server/python/tests/codegen/test_gen_run_core.py b/server/python/tests/codegen/test_gen_run_core.py new file mode 100644 index 0000000000..65655ede1b --- /dev/null +++ b/server/python/tests/codegen/test_gen_run_core.py @@ -0,0 +1,119 @@ +"""One `metaobjects gen` run: the hash manifest is written once, and `--entities` narrows +what is written without shrinking the files rendered from the whole model. + +Parity with codegen-ts (`manifest-write-once.test.ts`, `run-gen.test.ts`). +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from metaobjects.cli import main +from metaobjects.codegen import overwrite_policy +from tests.codegen.config_helpers import as_config +from tests.codegen.gen_suite import GEN_SUITE + +FIXTURE = ( + Path(__file__).parents[4] + / "fixtures" + / "persistence-conformance" + / "canonical" + / "meta.fitness.json" +) + + +def _meta_dir(tmp_path: Path) -> str: + d = tmp_path / "meta" + d.mkdir() + (d / "meta.fitness.json").write_text(FIXTURE.read_text()) + return str(d) + + +def _gen(meta_dir: str, out: Path, *extra: str) -> int: + return main(as_config(["gen", "--generators", GEN_SUITE, meta_dir, "--out", str(out), *extra])) + + +@pytest.fixture +def manifest_writes(monkeypatch: pytest.MonkeyPatch) -> list[str]: + writes: list[str] = [] + real = overwrite_policy._write_manifest_file + + def counting(gen_state_dir: str, ordered: dict[str, object]) -> None: + writes.append(gen_state_dir) + real(gen_state_dir, ordered) + + monkeypatch.setattr(overwrite_policy, "_write_manifest_file", counting) + return writes + + +def _manifest(tmp_path: Path) -> dict[str, object]: + [path] = list(tmp_path.rglob(".hashes.json")) + return json.loads(path.read_text()) + + +def test_a_run_writes_the_manifest_once(tmp_path: Path, manifest_writes: list[str]) -> None: + meta_dir = _meta_dir(tmp_path) + out = tmp_path / "out" + + assert _gen(meta_dir, out) == 0 + assert len(manifest_writes) == 1 + recorded = _manifest(tmp_path) + assert len(recorded) > 2 # many files, one write + + # Two files regenerate and are recorded again: still one write. (Deleting a file + # alone is not enough: it regenerates to the hash already recorded, which writes + # nothing.) + [path] = list(tmp_path.rglob(".hashes.json")) + trimmed = {k: v for k, v in recorded.items() if k not in ("out/Program.py", "out/Week.py")} + assert len(trimmed) == len(recorded) - 2 + path.write_text(json.dumps(trimmed)) + (out / "Program.py").unlink() + (out / "Week.py").unlink() + manifest_writes.clear() + assert _gen(meta_dir, out) == 0 + assert len(manifest_writes) == 1 + + # Nothing changes: no write at all, and the manifest is the same. + manifest_writes.clear() + assert _gen(meta_dir, out) == 0 + assert manifest_writes == [] + assert _manifest(tmp_path) == recorded + + +def test_an_unchanged_file_is_not_reported_as_written( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + meta_dir = _meta_dir(tmp_path) + out = tmp_path / "out" + assert _gen(meta_dir, out) == 0 + capsys.readouterr() + + (out / "Program.py").unlink() + assert _gen(meta_dir, out) == 0 + printed = capsys.readouterr().out.splitlines() + assert [line for line in printed if line.endswith(".py")] == [str(out / "Program.py")] + assert "metaobjects gen: wrote 1 file(s)" in printed[-1] + assert "unchanged" in printed[-1] + + +def test_entities_narrows_the_write_not_the_whole_model_files(tmp_path: Path) -> None: + """`--entities Week` writes Week's files and leaves out Program's, but every file it + does write is byte-identical to the full run's: the shared enums module still + carries Program's enum, which Week does not declare.""" + meta_dir = _meta_dir(tmp_path) + full = tmp_path / "full" + narrow = tmp_path / "narrow" + assert _gen(meta_dir, full) == 0 + assert _gen(meta_dir, narrow, "--entities", "Week,Node") == 0 + + assert (narrow / "Week.py").exists() + assert (narrow / "Node.py").exists() + assert not (narrow / "Program.py").exists() + assert (narrow / "enums.py").exists() + + written = sorted(p.relative_to(narrow) for p in narrow.rglob("*") if p.is_file()) + assert written + for rel in written: + assert (narrow / rel).read_bytes() == (full / rel).read_bytes(), str(rel) diff --git a/server/typescript/packages/cli/src/commands/gen.ts b/server/typescript/packages/cli/src/commands/gen.ts index 0519e45c06..f1a70f8ea1 100644 --- a/server/typescript/packages/cli/src/commands/gen.ts +++ b/server/typescript/packages/cli/src/commands/gen.ts @@ -16,7 +16,7 @@ import { antiPatternRows, ranSection, skippedSection, warnAdvisory, type AdvisoryFindingRow, type AdvisorySection, } from "../lib/advisory.js"; -import { loadMemory, resolveCollection } from "@metaobjectsdev/sdk"; +import { loadMemory, loadMemoryWithFindings, resolveCollection } from "@metaobjectsdev/sdk"; import { declaredGeneratorNames, runGen, undeclaredGeneratorsMessage, walkRequirements } from "@metaobjectsdev/codegen-ts"; import type { WriteStatus } from "@metaobjectsdev/codegen-ts"; import { packageOfResolutionKey } from "@metaobjectsdev/metadata"; @@ -36,7 +36,7 @@ import { findRuntimeBoundaryCrossings, runtimeBoundaryWarnings } from "../lib/ru import { hasLegacyDocsBlock, legacyDocsConfigMessage, NO_CONFIG_MESSAGE } from "../lib/docs-removed.js"; import { buildAgentSchemaInput } from "../lib/agent-schema.js"; import { schemaBuildOptions, schemaBuildOptionsOrWarn, type SchemaBuildOptions } from "../lib/schema-build-options.js"; -import { isUnknownSchemaConventionsError } from "@metaobjectsdev/migrate-ts"; +import { isUnknownSchemaConventionsError } from "@metaobjectsdev/migrate-ts/offline"; import { requirementPayload } from "../lib/requirement-payload.js"; import { existsSync } from "node:fs"; @@ -189,7 +189,7 @@ export async function genCommand(args: string[], cwd: string, fmt: OutputFormat } } - let metadata; + let loaded; const loadOptions = { ...collectionLoadOptions(genCollection), ...loadMemoryOptionsFrom(forgeConfig), @@ -197,7 +197,7 @@ export async function genCommand(args: string[], cwd: string, fmt: OutputFormat try { // `--strict` is the load `meta verify` makes (ADR-0023): an unknown attribute stops the // run before anything is written, instead of being warned about and generated anyway. - metadata = await loadMemory(genCollection.configDir, flags.strict ? { ...loadOptions, strict: true } : loadOptions); + loaded = await loadMemoryWithFindings(genCollection.configDir, flags.strict ? { ...loadOptions, strict: true } : loadOptions); } catch (err) { reportLoadError(log, "failed to load metadata", err); // The hint explains the strict-only rejection (an undeclared attribute); any other load @@ -212,7 +212,9 @@ export async function genCommand(args: string[], cwd: string, fmt: OutputFormat // misspelt `@required`) used to pass without a word while `meta verify` rejected the // same file — and the typo changed what was generated. Say so, naming attribute, node // and file. Advisory: the exit code is unchanged. - const unknownAttrFindings = await unknownAttrWarnings(genCollection.configDir, loadOptions, projectRoot); + // The findings come from the same load: the model is loaded once. + const metadata = loaded.root; + const unknownAttrFindings = unknownAttrWarnings(loaded.unknownAttributes, projectRoot); const unknownAttrs = unknownAttrFindings.length > 0 ? [...unknownAttrFindings, UNKNOWN_ATTR_GEN_ADVICE] : []; diff --git a/server/typescript/packages/cli/src/lib/agent-schema.ts b/server/typescript/packages/cli/src/lib/agent-schema.ts index 125ca53f57..fd5d942fc6 100644 --- a/server/typescript/packages/cli/src/lib/agent-schema.ts +++ b/server/typescript/packages/cli/src/lib/agent-schema.ts @@ -11,7 +11,7 @@ import { missingDialectMessage, } from "@metaobjectsdev/codegen-ts"; import type { AgentSchemaInput, Dialect, SchemaColumnLike } from "@metaobjectsdev/codegen-ts"; -import { buildExpectedSchemaWithProvenance, columnTypeSql, qualifiedDbName } from "@metaobjectsdev/migrate-ts"; +import { buildExpectedSchemaWithProvenance, columnTypeSql, qualifiedDbName } from "@metaobjectsdev/migrate-ts/offline"; import type { MetaRoot } from "@metaobjectsdev/metadata"; import { describeError } from "./error-text.js"; import type { SchemaBuildOptions } from "./schema-build-options.js"; diff --git a/server/typescript/packages/cli/src/lib/allow.ts b/server/typescript/packages/cli/src/lib/allow.ts index 1344659d39..ef94d006a2 100644 --- a/server/typescript/packages/cli/src/lib/allow.ts +++ b/server/typescript/packages/cli/src/lib/allow.ts @@ -2,8 +2,8 @@ // both `meta migrate` and `meta verify --db`. Keeping a single copy avoids the // two commands drifting on which `--allow` tokens exist or how a change reads. -import { allowOptionFor, describeSqlType, isViewRecreateOnly, suggestColumnRenames, viewsNamingRenamedColumns } from "@metaobjectsdev/migrate-ts"; -import type { AllowOptions, Change, ColumnRenameSuggestion, ViewChangeReason } from "@metaobjectsdev/migrate-ts"; +import { allowOptionFor, describeSqlType, isViewRecreateOnly, suggestColumnRenames, viewsNamingRenamedColumns } from "@metaobjectsdev/migrate-ts/offline"; +import type { AllowOptions, Change, ColumnRenameSuggestion, ViewChangeReason } from "@metaobjectsdev/migrate-ts/offline"; import type { BlockedEntry } from "./output.js"; // Map CLI allow tokens → migrate-ts AllowOptions field names. diff --git a/server/typescript/packages/cli/src/lib/gen-migrate-advice.ts b/server/typescript/packages/cli/src/lib/gen-migrate-advice.ts index 068e9cd1db..1a52273a41 100644 --- a/server/typescript/packages/cli/src/lib/gen-migrate-advice.ts +++ b/server/typescript/packages/cli/src/lib/gen-migrate-advice.ts @@ -29,7 +29,7 @@ import { withoutFlywayHistory, type Change, type D1Binding, -} from "@metaobjectsdev/migrate-ts"; +} from "@metaobjectsdev/migrate-ts/offline"; import type { SchemaBuildOptions } from "./schema-build-options.js"; /** The FQN separator; an object's short name is the segment after the last one. */ diff --git a/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts b/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts index 2c086258cf..c9b5946938 100644 --- a/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts +++ b/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts @@ -469,8 +469,15 @@ export async function loadMetaobjectsConfig(projectRoot: string): Promise { - let findings; - try { - findings = await unknownAttributeFindings(configDir, options); - } catch { - // The lenient load already succeeded; a strict re-load that cannot even complete has - // nothing more precise to say, and must never turn an advisory into a failure. - return []; - } +): string[] { return findings.map((f) => { const source = f.source as { files?: readonly string[]; jsonPath?: string }; const files = (source.files ?? []) diff --git a/server/typescript/packages/cli/test/gen-import-graph.test.ts b/server/typescript/packages/cli/test/gen-import-graph.test.ts new file mode 100644 index 0000000000..330acdfb1d --- /dev/null +++ b/server/typescript/packages/cli/test/gen-import-graph.test.ts @@ -0,0 +1,58 @@ +// `meta gen` touches no database, so loading it loads no query builder and no driver. +// It used to: the schema advice it prints imported the migrate-ts package root, whose +// introspect / apply / replay modules import kysely, and kysely was a seventh of gen's +// start-up. The gen path reads `@metaobjectsdev/migrate-ts/offline` instead; a driver is +// imported dynamically, by the commands that connect, when they connect. +// +// Walks the STATIC import graph from the gen command (a dynamic `import()` is a lazy load +// by definition, so it is not followed) and fails naming the chain that reaches one. +import { describe, test, expect } from "bun:test"; +import { readFileSync } from "node:fs"; +import { dirname, resolve } from "node:path"; + +const GEN = resolve(import.meta.dir, "..", "src", "commands", "gen.ts"); +const DATABASE_PACKAGES = [/^kysely(\/|$)/, /^pg(\/|$)/, /^@libsql\//, /^better-sqlite3$/, /^bun:sqlite$/]; + +const transpiler = new Bun.Transpiler({ loader: "ts" }); + +function staticImports(file: string): string[] { + return transpiler + .scanImports(readFileSync(file, "utf-8")) + .filter((i) => i.kind === "import-statement") + .map((i) => i.path); +} + +describe("the meta gen import graph", () => { + test("reaches no database package statically", () => { + const parent = new Map(); + const seen = new Set([GEN]); + const queue = [GEN]; + const hits: string[] = []; + while (queue.length > 0) { + const file = queue.shift()!; + for (const spec of staticImports(file)) { + if (DATABASE_PACKAGES.some((re) => re.test(spec))) { + const chain = [spec, file]; + for (let p = parent.get(file); p !== undefined; p = parent.get(p)) chain.push(p); + hits.push(chain.reverse().join("\n -> ")); + continue; + } + if (spec.startsWith("node:")) continue; + let target: string; + try { + target = Bun.resolveSync(spec, dirname(file)); + } catch { + continue; + } + // Follow this repository's own sources only; a third-party package is a leaf. + if (target.includes("/node_modules/") || !/\.(ts|tsx|js|mjs)$/.test(target)) continue; + if (seen.has(target)) continue; + seen.add(target); + parent.set(target, file); + queue.push(target); + } + } + expect(seen.size).toBeGreaterThan(50); + expect(hits).toEqual([]); + }); +}); diff --git a/server/typescript/packages/codegen-ts/src/generator.ts b/server/typescript/packages/codegen-ts/src/generator.ts index 330fa46600..d0c5be3c0a 100644 --- a/server/typescript/packages/codegen-ts/src/generator.ts +++ b/server/typescript/packages/codegen-ts/src/generator.ts @@ -27,7 +27,16 @@ export interface EmittedFile { } export interface GenContext { + /** Every object the run generates for: the WHOLE model (after the project's `scope`), + * even when `meta gen ` names some of them. A file rendered from the whole + * model — a barrel, an index, a shared schema — reads this and so never shrinks to + * the named entities. A per-entity file honours `selected` as well. */ entities: MetaObject[]; + /** The objects `meta gen ` named, as a predicate. Undefined ⇒ every object + * in `entities` is selected. `perEntity` applies it for you; a generator that emits + * one file per object by other means applies it itself, and one that renders a + * single file from the whole model ignores it. */ + selected?: (entity: MetaObject) => boolean; loadedRoot: MetaRoot; /** Pre-composed by the runner from generator.filter (returns true when no * filter is set). Always call this from helpers; do not call generator.filter @@ -289,7 +298,7 @@ export function perEntity( | Promise, ): (ctx: GenContext) => Promise { return async (ctx) => { - const matched = ctx.entities.filter(ctx.matches); + const matched = ctx.entities.filter((e) => ctx.matches(e) && (ctx.selected?.(e) ?? true)); const results = await Promise.all(matched.map(async (e) => { const r = await fn(e, ctx); // Each file names the object it was rendered from, unless the generator said @@ -302,7 +311,8 @@ export function perEntity( }; } -/** Called once with all matching entities. Use for barrels and cross-entity files. +/** Called once with all matching entities of the whole model, whatever `meta gen ` + * named. Use for barrels and cross-entity files. * @deprecated Use {@link perModel} — "run" is ambiguous under multi-target output * (it reads as "per target"); `perModel` names the data scope (the whole model). */ export function oncePerRun( diff --git a/server/typescript/packages/codegen-ts/src/generators/cube-model.ts b/server/typescript/packages/codegen-ts/src/generators/cube-model.ts index aa96930a28..ddec779ec3 100644 --- a/server/typescript/packages/codegen-ts/src/generators/cube-model.ts +++ b/server/typescript/packages/codegen-ts/src/generators/cube-model.ts @@ -128,7 +128,7 @@ function generateCubeFiles(ctx: GenContext, options: CubeModelGeneratorOptions): // The run's selection, narrowed by the filter, keeping the entities that are their own cube // (Table A); a cube is named after its entity. A report is never selected itself: its rollup // rides with its @from cube. - const selected = ctx.entities.filter((o) => ctx.matches(o) && hasReportingVocabulary(o)); + const selected = ctx.entities.filter((o) => ctx.matches(o) && (ctx.selected?.(o) ?? true) && hasReportingVocabulary(o)); if (selected.length === 0) return []; const dialect = options.dialect ?? ctx.config.dialect; diff --git a/server/typescript/packages/codegen-ts/src/generators/names-file.ts b/server/typescript/packages/codegen-ts/src/generators/names-file.ts index 00ca34ac65..1d1baf9632 100644 --- a/server/typescript/packages/codegen-ts/src/generators/names-file.ts +++ b/server/typescript/packages/codegen-ts/src/generators/names-file.ts @@ -77,7 +77,8 @@ export function namesFile(opts?: NamesFileOpts): Generator { // abstract base emits nothing here, and seeding it as already-emitted is what would // make pass 2 skip the very object it exists to produce. const emitted = new Set(); - const participants = ctx.entities.filter(ctx.matches); + // The objects `meta gen ` named (all of them on a full run): one file each. + const participants = ctx.entities.filter((e) => ctx.matches(e) && (ctx.selected?.(e) ?? true)); for (const entity of participants) { const content = renderNamesDecl(entity, { strategy, superSpecifier: superSpecifierFor(entity), diff --git a/server/typescript/packages/codegen-ts/src/generators/report-export-file.ts b/server/typescript/packages/codegen-ts/src/generators/report-export-file.ts index 85c67f024c..28e9426220 100644 --- a/server/typescript/packages/codegen-ts/src/generators/report-export-file.ts +++ b/server/typescript/packages/codegen-ts/src/generators/report-export-file.ts @@ -60,7 +60,8 @@ export function reportExport(opts?: ReportExportOpts): Generator { const out: EmittedFile[] = [ { path: `${EXPORT_WRITERS_BASENAME}.ts`, content: await formatTs(renderExportWriters(emit)) }, ]; - for (const report of reports) { + // One module per report `meta gen ` named; the writers above serve them all. + for (const report of reports.filter((r) => ctx.selected?.(r) ?? true)) { const pkg = effectivePackage(report); out.push({ path: entityOutputPath(layout, pkg, `${report.name}.export.ts`), diff --git a/server/typescript/packages/codegen-ts/src/overwrite-policy.ts b/server/typescript/packages/codegen-ts/src/overwrite-policy.ts index 74d9d5dd04..548e23e9ba 100644 --- a/server/typescript/packages/codegen-ts/src/overwrite-policy.ts +++ b/server/typescript/packages/codegen-ts/src/overwrite-policy.ts @@ -150,7 +150,55 @@ function sha256(content: string | Buffer): string { return createHash("sha256").update(content).digest("hex"); } +/** + * The manifest of a run in progress, held in memory and written once at its end. + * + * Every write updates `.hashes.json`, and the file is the whole project's record, so + * rewriting it per written file made bookkeeping quadratic in the project's size: at + * a thousand generated files the rewrites were most of a run's wall time, for the same + * bytes one write at the end produces. `withManifestBatch` holds the manifest for one + * gen-state directory in memory while it runs; every reader and writer in this module + * goes through it, so a read inside the batch sees the batch's own writes. + * + * Synchronous on purpose: nothing can interleave with a batch, so two runs in one + * process cannot see each other's half-finished manifest. + */ +interface ManifestBatch { + raw: Record; + dirty: boolean; +} +const manifestBatches = new Map(); + +function batchFor(genStateDir: string): ManifestBatch | undefined { + return manifestBatches.size === 0 ? undefined : manifestBatches.get(resolve(genStateDir)); +} + +/** + * Run `fn` with the manifest under `genStateDir` held in memory, and write it once when + * `fn` returns or throws, only if something changed. A throw still writes what was + * recorded before it: those files are on disk, and a manifest that forgot them would + * read them as unrecorded and refuse them next run. A nested call for the same + * directory joins the outer batch. + */ +export function withManifestBatch(genStateDir: string, fn: () => T): T { + const key = resolve(genStateDir); + if (manifestBatches.has(key)) return fn(); + manifestBatches.set(key, { raw: readManifestFile(genStateDir), dirty: false }); + try { + return fn(); + } finally { + const batch = manifestBatches.get(key)!; + manifestBatches.delete(key); + if (batch.dirty) writeManifestFile(genStateDir, batch.raw); + } +} + function readRawManifest(genStateDir: string): Record { + const batch = batchFor(genStateDir); + return batch !== undefined ? batch.raw : readManifestFile(genStateDir); +} + +function readManifestFile(genStateDir: string): Record { const f = join(genStateDir, HASHES_FILE); if (!existsSync(f)) return {}; try { @@ -193,7 +241,25 @@ function saveHashes(genStateDir: string, hashes: HashesFile): void { writeRawManifest(genStateDir, next); } +/** Record one path's hash, keeping any other fields on its entry. */ +function setHash(genStateDir: string, relPath: string, hash: string): void { + const raw = readRawManifest(genStateDir); + const existing = raw[relPath]; + raw[relPath] = isRecord(existing) ? { ...existing, hash } : hash; + writeRawManifest(genStateDir, raw); +} + function writeRawManifest(genStateDir: string, entries: Record): void { + const batch = batchFor(genStateDir); + if (batch !== undefined) { + batch.raw = entries; + batch.dirty = true; + return; + } + writeManifestFile(genStateDir, entries); +} + +function writeManifestFile(genStateDir: string, entries: Record): void { mkdirSync(genStateDir, { recursive: true }); // Keys SORTED, because this file is committed. Insertion order would make the // diff — and any merge conflict between two people who both regenerated — @@ -236,7 +302,7 @@ export function recordGeneratedProvenance( genStateDir: string, provenance: ReadonlyMap, ): void { - if (!existsSync(join(genStateDir, HASHES_FILE))) return; + if (!hasHashManifest(genStateDir)) return; const raw = readRawManifest(genStateDir); let changed = false; for (const [key, prov] of provenance) { @@ -327,7 +393,7 @@ export function readGeneratedHash( genStateDir: string, relPath: string, ): string | undefined { - return loadHashes(genStateDir)[relPath]; + return entryHash(readRawManifest(genStateDir)[relPath]); } /** @@ -444,9 +510,7 @@ function advanceSnapshot( const dest = snapshotPath(genStateDir, relPath); mkdirSync(dirname(dest), { recursive: true }); writeFileSync(dest, content); - const hashes = loadHashes(genStateDir); - hashes[relPath] = sha256(content); - saveHashes(genStateDir, hashes); + setHash(genStateDir, relPath, sha256(content)); } /** Return the snapshot text iff present AND the hash matches; else undefined. */ @@ -457,8 +521,7 @@ function readSnapshotChecked( const path = snapshotPath(genStateDir, relPath); if (!existsSync(path)) return undefined; const text = readFileSync(path, "utf-8"); - const hashes = loadHashes(genStateDir); - const expected = hashes[relPath]; + const expected = readGeneratedHash(genStateDir, relPath); if (expected !== undefined && expected !== sha256(text)) { return undefined; } @@ -578,6 +641,7 @@ function resolveGenStateDir(opts: DecideAndWriteOpts): string { * problem rather than N independent hand edits, and it deserves one instruction * instead of a wall of per-file warnings. */ export function hasHashManifest(genStateDir: string): boolean { + if (batchFor(genStateDir)?.dirty === true) return true; return existsSync(join(genStateDir, HASHES_FILE)); } diff --git a/server/typescript/packages/codegen-ts/src/reference/cube-model.ts b/server/typescript/packages/codegen-ts/src/reference/cube-model.ts index 44cd4c6984..cf5219dc2c 100644 --- a/server/typescript/packages/codegen-ts/src/reference/cube-model.ts +++ b/server/typescript/packages/codegen-ts/src/reference/cube-model.ts @@ -161,7 +161,7 @@ function generateCubeFiles(ctx: GenContext, options: CubeModelGeneratorOptions): // The run's selection, narrowed by the filter, keeping the entities that are their own cube; // a cube is named after its entity. A report is never selected itself: its rollup rides with // its @from cube. - const selected = ctx.entities.filter((o) => ctx.matches(o) && hasReportingVocabulary(o)); + const selected = ctx.entities.filter((o) => ctx.matches(o) && (ctx.selected?.(o) ?? true) && hasReportingVocabulary(o)); if (selected.length === 0) return []; const dialect = options.dialect ?? ctx.config.dialect; diff --git a/server/typescript/packages/codegen-ts/src/reference/names.ts b/server/typescript/packages/codegen-ts/src/reference/names.ts index e600259dbd..a5fada496d 100644 --- a/server/typescript/packages/codegen-ts/src/reference/names.ts +++ b/server/typescript/packages/codegen-ts/src/reference/names.ts @@ -97,7 +97,8 @@ export function namesFile(opts?: NamesFileOpts): Generator { // abstract base emits nothing here, and seeding it as already-emitted is what would // make pass 2 skip the very object it exists to produce. const emitted = new Set(); - const participants = ctx.entities.filter(ctx.matches); + // The objects `meta gen ` named (all of them on a full run): one file each. + const participants = ctx.entities.filter((e) => ctx.matches(e) && (ctx.selected?.(e) ?? true)); for (const entity of participants) { const content = renderNamesDecl(entity, { strategy, superSpecifier: superSpecifierFor(entity), diff --git a/server/typescript/packages/codegen-ts/src/reference/report-export.ts b/server/typescript/packages/codegen-ts/src/reference/report-export.ts index 63310482ac..a192742d34 100644 --- a/server/typescript/packages/codegen-ts/src/reference/report-export.ts +++ b/server/typescript/packages/codegen-ts/src/reference/report-export.ts @@ -248,7 +248,8 @@ export function reportExport(opts?: ReportExportOpts): Generator { const out: EmittedFile[] = [ { path: `${EXPORT_WRITERS_BASENAME}.ts`, content: await formatTs(renderExportWriters(emit)) }, ]; - for (const report of reports) { + // One module per report `meta gen ` named; the writers above serve them all. + for (const report of reports.filter((r) => ctx.selected?.(r) ?? true)) { const pkg = effectivePackage(report); out.push({ path: entityOutputPath(layout, pkg, `${report.name}.export.ts`), diff --git a/server/typescript/packages/codegen-ts/src/runner.ts b/server/typescript/packages/codegen-ts/src/runner.ts index d87ccf699c..6042cb95c5 100644 --- a/server/typescript/packages/codegen-ts/src/runner.ts +++ b/server/typescript/packages/codegen-ts/src/runner.ts @@ -34,6 +34,7 @@ import { loadEngineVersion, saveEngineVersion, recordGeneratedProvenance, + withManifestBatch, listGeneratedProvenance, type GeneratedProvenance, type WriteResult, @@ -456,6 +457,7 @@ export async function runGen(opts: RunGenOpts): Promise { // packages may declare the same short name). const allObjects = root.objects(); const entityFilter = opts.entityFilter; + const entityNarrowed = entityFilter !== undefined && entityFilter.length > 0; const afterEntityFilter = entityFilter ? allObjects.filter((o) => entityFilter.includes(o.name)) : allObjects; @@ -498,6 +500,15 @@ export async function runGen(opts: RunGenOpts): Promise { // generates nothing. A selection made only of unserved reports gets the same // "nothing to generate" warning as an empty one. const generatable = generatableObjects(filtered, root); + // `meta gen ` narrows what is WRITTEN per entity, never what a whole-model file + // sees: a barrel, an index or a shared schema rendered from the named entities alone + // dropped every other entity's export. So every generator gets the whole (scoped) model + // in `ctx.entities`, and the entity filter reaches it only as `ctx.selected`. + // Keyed by the generated object (a served report generates as its read model) and by + // the declared one, so either names the selection. + const selectedKeys = new Set([...filtered, ...generatable].map((o) => o.resolutionKey())); + const isSelected = (o: MetaObject): boolean => selectedKeys.has(o.resolutionKey()); + const wholeModel = scope ? allObjects.filter((o) => scope(o.resolutionKey())) : allObjects; if (generatable.length === 0) { warnings.push( "No entities to generate — every selected object is an object.report that is not " + @@ -508,16 +519,20 @@ export async function runGen(opts: RunGenOpts): Promise { } const safeEntities: MetaObject[] = []; - for (const entity of generatable) { + for (const entity of entityNarrowed ? generatableObjects(wholeModel, root) : generatable) { if (!VALID_ENTITY_NAME.test(entity.name)) { - warnings.push( - `Skipping entity with unsafe name "${entity.name}" — must match /^[A-Za-z_][A-Za-z0-9_]*$/.`, - ); + if (isSelected(entity)) { + warnings.push( + `Skipping entity with unsafe name "${entity.name}" — must match /^[A-Za-z_][A-Za-z0-9_]*$/.`, + ); + } continue; } safeEntities.push(entity); } - if (safeEntities.length === 0) { + // The entities this run writes per-entity output for: all of them, or the ones named. + const selectedEntities = entityNarrowed ? safeEntities.filter(isSelected) : safeEntities; + if (selectedEntities.length === 0) { return { files: [], warnings, conflicts: [] }; } @@ -770,7 +785,7 @@ export async function runGen(opts: RunGenOpts): Promise { // the run's own entity set (the same set every generator sees), so `meta gen ` // reports on what it just generated rather than on objects it was told to skip. // See retired-codegen-attrs.ts. Self-extinguishing; warning only. - warnRetiredCodegenAttrs(safeEntities, (m) => warnings.push(m)); + warnRetiredCodegenAttrs(selectedEntities, (m) => warnings.push(m)); // What the documentation generators in one target must agree on (docs-run.ts), built from // the generators the run declares for that target. The schema builder is shared by all @@ -833,7 +848,6 @@ export async function runGen(opts: RunGenOpts): Promise { // for the rest — the declared generators it did not run, and the objects it did not render — // so a narrowed run reads what a full run reads. A record for a generator no longer declared, // or for a file no longer on disk, is stale and stays out. - const entityNarrowed = opts.entityFilter !== undefined && opts.entityFilter.length > 0; const runOutputFiles = (): RunOutputFile[] => { const out = new Map(); const keyOf = (fullPath: string): string | undefined => { @@ -844,7 +858,7 @@ export async function runGen(opts: RunGenOpts): Promise { if (opts.generatorNames !== undefined || entityNarrowed) { const ranNames = new Set(selected.map((g) => g.name)); const declaredNames = new Set(config.generators.map((g) => g.name)); - const renderedNodes = new Set(safeEntities.map((e) => e.resolutionKey())); + const renderedNodes = new Set(selectedEntities.map((e) => e.resolutionKey())); for (const rec of listGeneratedProvenance(genStateDir)) { if (!declaredNames.has(rec.generatedBy)) continue; // A generator that ran states its own output, except for the objects an entity @@ -866,6 +880,7 @@ export async function runGen(opts: RunGenOpts): Promise { return [...out.values()].sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0)); }; + const wholeKeys = new Set(safeEntities.map((e) => e.resolutionKey())); for (const generator of selected) { const selfTarget = targetOf(generator); // Resolve the WRITE-time location against projectRoot when outDir is @@ -919,6 +934,7 @@ export async function runGen(opts: RunGenOpts): Promise { }); const ctx: GenContext = { entities: safeEntities, + ...(entityNarrowed && { selected: isSelected }), loadedRoot: root, matches: (e) => generator.filter?.(e) ?? true, // FR-023 §11.1 item 2 — the SAME `scope` predicate that already narrowed @@ -1007,6 +1023,10 @@ export async function runGen(opts: RunGenOpts): Promise { } for (const file of files) { + // A file rendered from one object the entity filter did not name is not this run's + // to write: `perEntity` already skips those, and this holds a generator that renders + // per object by other means to the same rule. + if (entityNarrowed && file.sourceNode !== undefined && wholeKeys.has(file.sourceNode) && !selectedKeys.has(file.sourceNode)) continue; const fullPath = join(writeOutDir, file.path); const collision = emitted.find((prev) => prev.fullPath === fullPath); if (collision) { @@ -1308,53 +1328,57 @@ export async function runGen(opts: RunGenOpts): Promise { return { files: writes, warnings, conflicts, handEditsOverwrite: handEditsOverwrite(), orphanSweepJobs: sweepJobs }; } - for (const file of emitted) { - // Key the snapshot by project-relative path so multi-target projects keep - // distinct entries (e.g. `database/Post.ts` vs `web/Post.queries.ts`). - // Without an explicit projectRoot we let decideAndWrite derive a stable - // hash-of-path key — fine for ephemeral test runs. - const result = decideAndWrite(file.fullPath, file.content, policyFor(file.fullPath, file.handEdits)); - writes.push(result); - if (result.discardedEdit === true) { - warnings.push( - `overwrote hand-edited ${relativeForDisplay(file.fullPath)} with fresh output ` + - `(handEdits: "overwrite" on ${file.generatedBy})`, - ); - } - if (result.status === "conflict") { - conflicts.push(result); - warnings.push( - `Merge conflict in ${file.fullPath}: resolve diff3 markers and re-run ` + - `'meta gen' to advance the canonical state.`, - ); + // The whole write phase shares one in-memory manifest, written once at its end — the + // writes, the provenance and the sweep all record into it. See `withManifestBatch`. + withManifestBatch(genStateDir, () => { + for (const file of emitted) { + // Key the snapshot by project-relative path so multi-target projects keep + // distinct entries (e.g. `database/Post.ts` vs `web/Post.queries.ts`). + // Without an explicit projectRoot we let decideAndWrite derive a stable + // hash-of-path key — fine for ephemeral test runs. + const result = decideAndWrite(file.fullPath, file.content, policyFor(file.fullPath, file.handEdits)); + writes.push(result); + if (result.discardedEdit === true) { + warnings.push( + `overwrote hand-edited ${relativeForDisplay(file.fullPath)} with fresh output ` + + `(handEdits: "overwrite" on ${file.generatedBy})`, + ); + } + if (result.status === "conflict") { + conflicts.push(result); + warnings.push( + `Merge conflict in ${file.fullPath}: resolve diff3 markers and re-run ` + + `'meta gen' to advance the canonical state.`, + ); + } + // Refusals are reported together after the loop (see reportRefusals) so a + // whole-project cause can be stated once instead of once per file. } - // Refusals are reported together after the loop (see reportRefusals) so a - // whole-project cause can be stated once instead of once per file. - } - reportRefusals(); - noteUntracked(); - reportAdoptions(); + reportRefusals(); + noteUntracked(); + reportAdoptions(); - // G3 — which node each recorded file came from, beside its hash. Only with a real - // projectRoot: without one the manifest is keyed by path hashes, not paths. - if (projectRoot !== undefined) { - const provenance = new Map(); - for (const file of emitted) { - if (outsideProject(file.fullPath)) continue; - // The same key policyFor gives the write, so the two always name one entry. - provenance.set(relative(projectRoot, file.fullPath), { - generatedBy: file.generatedBy, - ...(file.sourceNode !== undefined && { sourceNode: file.sourceNode }), - }); + // G3 — which node each recorded file came from, beside its hash. Only with a real + // projectRoot: without one the manifest is keyed by path hashes, not paths. + if (projectRoot !== undefined) { + const provenance = new Map(); + for (const file of emitted) { + if (outsideProject(file.fullPath)) continue; + // The same key policyFor gives the write, so the two always name one entry. + provenance.set(relative(projectRoot, file.fullPath), { + generatedBy: file.generatedBy, + ...(file.sourceNode !== undefined && { sourceNode: file.sourceNode }), + }); + } + recordGeneratedProvenance(genStateDir, provenance); } - recordGeneratedProvenance(genStateDir, provenance); - } - // Sweep AFTER the writes: writing is the primary job, and a deletion that runs - // first would be unrecoverable if a later write threw. Ordering cannot change - // the decision — a path this run emitted is never an orphan either way. - sweep(false); + // Sweep AFTER the writes: writing is the primary job, and a deletion that runs + // first would be unrecoverable if a later write threw. Ordering cannot change + // the decision — a path this run emitted is never an orphan either way. + sweep(false); + }); // #232 — stamp the engine version that produced this snapshot, so the NEXT gen can // detect an engine change. Written after a successful run only, and only by a run that diff --git a/server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts b/server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts new file mode 100644 index 0000000000..356f408afd --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts @@ -0,0 +1,68 @@ +// `.gen-state/.hashes.json` is written ONCE per `meta gen`, however many files the run +// writes. It used to be rewritten per written file — quadratic bookkeeping that was most +// of a large project's run time, for the same bytes one write produces. +import { describe, test, expect, beforeEach, afterEach, spyOn } from "bun:test"; +import * as fs from "node:fs"; +import { mkdtempSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { MetaDataLoader } from "@metaobjectsdev/metadata"; +import { FileSource } from "@metaobjectsdev/metadata/core"; +import { runGen, defineConfig } from "../src/index.js"; +import { entityFile } from "../src/generators/entity-file.js"; +import { queriesFile } from "../src/generators/queries-file.js"; +import { barrel } from "../src/generators/barrel.js"; + +const FIXTURE = resolve(import.meta.dir, "fixtures", "two-entities-fk.json"); +const MANIFEST = ".hashes.json"; + +let project: string; +beforeEach(() => { project = mkdtempSync(join(tmpdir(), "codegen-manifest-once-")); }); +afterEach(() => { rmSync(project, { recursive: true, force: true }); }); + +async function gen(): Promise { + const { root, errors } = await new MetaDataLoader().load([new FileSource(FIXTURE)]); + expect(errors).toEqual([]); + const spy = spyOn(fs, "writeFileSync"); + try { + const result = await runGen({ + config: defineConfig({ + outDir: "src/generated", extStyle: "none", dbImport: "~/server/db", dialect: "postgres", + generators: [entityFile(), queriesFile(), barrel()], + }), + metadata: root, + projectRoot: project, + }); + expect(result.files.length).toBeGreaterThan(2); + return spy.mock.calls.filter(([path]) => String(path).endsWith(MANIFEST)).length; + } finally { + spy.mockRestore(); + } +} + +const manifestPath = (): string => join(project, ".metaobjects", ".gen-state", MANIFEST); + +describe("the hash manifest is written once per run", () => { + test("a first run that writes every file writes the manifest once, recording each", async () => { + expect(await gen()).toBe(1); + const recorded = Object.keys(JSON.parse(readFileSync(manifestPath(), "utf-8"))); + expect(recorded).toContain("src/generated/Post.ts"); + expect(recorded).toContain("src/generated/User.queries.ts"); + expect(recorded).toContain("src/generated/index.ts"); + }); + + test("a run that changes files rewrites the manifest once", async () => { + await gen(); + // Two files to write again: both are recorded, in one manifest write. + rmSync(join(project, "src/generated/Post.ts")); + rmSync(join(project, "src/generated/User.ts")); + expect(await gen()).toBe(1); + }); + + test("a run that changes nothing leaves the manifest unwritten", async () => { + await gen(); + const before = readFileSync(manifestPath(), "utf-8"); + expect(await gen()).toBe(0); + expect(readFileSync(manifestPath(), "utf-8")).toBe(before); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/run-gen.test.ts b/server/typescript/packages/codegen-ts/test/run-gen.test.ts index b451c193c5..aa2388c350 100644 --- a/server/typescript/packages/codegen-ts/test/run-gen.test.ts +++ b/server/typescript/packages/codegen-ts/test/run-gen.test.ts @@ -284,36 +284,40 @@ describe("runGen — mergeStrategy: 'skip-existing'", () => { // Item 5: entityFilter test // --------------------------------------------------------------------------- describe("runGen — entityFilter", () => { - test("entityFilter: ['Post'] produces only Post files; barrel reflects filter", async () => { + // A whole-model file sees the whole model whatever `meta gen ` names: the barrel + // re-exports every entity, while the per-entity files are written for the named one only. + // It used to be rendered from the named entities alone, so `meta gen Post` dropped every + // other entity's export from `index.ts`. + test("entityFilter: ['Post'] writes only Post's per-entity files; the barrel keeps every entity", async () => { const loader = new MetaDataLoader(); const result = await loader.load([new FileSource(join(FIXTURE_DIR, "two-entities-fk.json"))]); expect(result.errors).toEqual([]); - - const out = await runGen({ - config: defineConfig({ - outDir: tmp, - extStyle: "none", - dbImport: "~/server/db", - dialect: "postgres", - generators: [entityFile(), queriesFile(), barrel()], - }), - metadata: result.root, - entityFilter: ["Post"], + const config = defineConfig({ + outDir: tmp, + extStyle: "none", + dbImport: "~/server/db", + dialect: "postgres", + generators: [entityFile(), queriesFile(), barrel()], }); + const full = await runGen({ config, metadata: result.root }); + expect(full.warnings).toEqual([]); + const fullBarrel = readFileSync(join(tmp, "index.ts"), "utf-8"); + expect(fullBarrel).toContain('export * from "./Post"'); + expect(fullBarrel).toContain('export * from "./User"'); + for (const f of readdirSync(tmp)) rmSync(join(tmp, f), { recursive: true, force: true }); + + const out = await runGen({ config, metadata: result.root, entityFilter: ["Post"] }); expect(out.warnings).toEqual([]); - // Only Post files should exist — no User files + // Only Post's per-entity files are written — no User files. expect(existsSync(join(tmp, "Post.ts"))).toBe(true); expect(existsSync(join(tmp, "Post.queries.ts"))).toBe(true); - expect(existsSync(join(tmp, "index.ts"))).toBe(true); expect(existsSync(join(tmp, "User.ts"))).toBe(false); expect(existsSync(join(tmp, "User.queries.ts"))).toBe(false); - // Barrel only re-exports Post - const barrelContent = readFileSync(join(tmp, "index.ts"), "utf-8"); - expect(barrelContent).toContain('export * from "./Post"'); - expect(barrelContent).not.toContain("User"); + // The barrel is the full run's barrel, byte for byte. + expect(readFileSync(join(tmp, "index.ts"), "utf-8")).toBe(fullBarrel); }); test("entityFilter: ['NonExistent'] produces 0 entity files and a warning", async () => { diff --git a/server/typescript/packages/migrate-ts/package.json b/server/typescript/packages/migrate-ts/package.json index ee2f9823cb..5fbe7443ac 100644 --- a/server/typescript/packages/migrate-ts/package.json +++ b/server/typescript/packages/migrate-ts/package.json @@ -15,6 +15,11 @@ "bun": "./src/expected.ts", "types": "./dist/expected.d.ts", "default": "./dist/expected.js" + }, + "./offline": { + "bun": "./src/offline.ts", + "types": "./dist/offline.d.ts", + "default": "./dist/offline.js" } }, "files": [ diff --git a/server/typescript/packages/migrate-ts/src/offline.ts b/server/typescript/packages/migrate-ts/src/offline.ts new file mode 100644 index 0000000000..2a593b4af5 --- /dev/null +++ b/server/typescript/packages/migrate-ts/src/offline.ts @@ -0,0 +1,18 @@ +// `@metaobjectsdev/migrate-ts/offline` — the part of the schema engine that never opens a +// database: build the expected schema, diff it against a committed snapshot, and name what +// a change would need. `meta gen` and the other commands that touch no database import +// from here, so loading them loads no query builder and no driver. The package root +// re-exports all of this beside the parts that do connect (introspect, apply, replay). +export { buildExpectedSchemaWithProvenance } from "./expected-schema.js"; +export { diff } from "./diff/index.js"; +export { resolveSchemaConventions, type SchemaConventions } from "./schema-conventions.js"; +export { qualifiedDbName } from "./qualified-name.js"; +export { columnTypeSql } from "./column-type-sql.js"; +export { withoutFlywayHistory } from "./write-migration-flyway.js"; +export { snapshotPath, readSnapshot } from "./snapshot/store.js"; +export { allowOptionFor, isUnknownSchemaConventionsError } from "./errors.js"; +export { suggestColumnRenames, type ColumnRenameSuggestion } from "./diff/rename-suggestions.js"; +export { describeSqlType } from "./sql-type.js"; +export { isViewRecreateOnly, viewsNamingRenamedColumns } from "./view-change-reason.js"; +export { findWranglerConfig, parseWranglerConfig, type D1Binding } from "./wrangler-config.js"; +export type { AllowOptions, Change, ViewChangeReason } from "./types.js"; diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index d08a184623..220b301311 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -99,7 +99,7 @@ export type { ForgeType, ForgeAttr } from "./forge-types.js"; // Memory loader — read a project's resolved metadata into a MetaData tree. // Where those files come from is `resolveCollection`'s decision (below), which // `loadMemory` calls when the caller supplies no explicit file set. -export { loadErrors, loadMemory, unknownAttributeFindings, defaultLoadMemoryProviders } from "./memory.js"; +export { loadErrors, loadMemory, loadMemoryWithFindings, unknownAttributeFindings, defaultLoadMemoryProviders } from "./memory.js"; export type { LoadMemoryOptions } from "./memory.js"; // Default project layout — the DEFAULT value of `sources` (applied by diff --git a/server/typescript/packages/sdk/src/memory.ts b/server/typescript/packages/sdk/src/memory.ts index 541b2acf65..bb605a2809 100644 --- a/server/typescript/packages/sdk/src/memory.ts +++ b/server/typescript/packages/sdk/src/memory.ts @@ -179,8 +179,45 @@ export async function loadMemory( repoRoot: string, options?: LoadMemoryOptions, ): Promise { - const { result, importedPackages, importedNodes, libraries, physicalNaming, wireFormat, enumDefaults, apiPagination, apiUnknownFields } = - await loadCollectionRaw(repoRoot, options, options?.strict === true); + return finishLoad(await loadCollectionRaw(repoRoot, options, options?.strict === true)); +} + +/** + * {@link loadMemory} and {@link unknownAttributeFindings} from ONE load: the root, and the + * ADR-0023 unknown-attribute findings a strict load of the same collection would raise. + * For a command that loads leniently and must name those findings (`meta gen`), without + * loading the model three times to do it. + * + * A strict load (`options.strict`) has no findings: it throws on the first one instead. + * Otherwise the attribute-schema check runs strictly over the lenient tree, which is all + * a strict load adds for an `@attr`. A bare key (`isAbstrakt`) is the one thing the lenient + * tree cannot locate (the parser drops it, warning without a file), so only when the load + * reported one is the collection parsed strictly again, for the file and path. + */ +export async function loadMemoryWithFindings( + repoRoot: string, + options?: LoadMemoryOptions, +): Promise<{ root: MetaRoot; unknownAttributes: readonly (Error & LoaderError)[] }> { + const strict = options?.strict === true; + const run = await loadCollectionRaw(repoRoot, options, strict); + const root = finishLoad(run); + if (strict) return { root, unknownAttributes: [] }; + let unknownAttributes: readonly (Error & LoaderError)[]; + try { + unknownAttributes = await findingsFrom(repoRoot, options, run); + } catch { + // The lenient load succeeded; a strict re-parse that cannot complete has nothing more + // precise to say, and must never turn an advisory into a failure. + unknownAttributes = []; + } + return { root, unknownAttributes }; +} + +type RawLoad = Awaited>; + +/** The checks {@link loadMemory} makes after the loader, and the root they leave. */ +function finishLoad(run: RawLoad): MetaRoot { + const { result, importedPackages, importedNodes, libraries, physicalNaming, wireFormat, enumDefaults, apiPagination, apiUnknownFields } = run; if (result.errors.length > 0) { const first = result.errors[0]!; @@ -245,10 +282,22 @@ export async function unknownAttributeFindings( // the attribute-schema pass — so the strict load alone would hide every misspelt // `@attr` behind the first bare-key typo. The second pass runs that attribute-schema // check, strictly, over the lenient tree, where both kinds of typo are still present. - const strictRun = await loadCollectionRaw(repoRoot, options, true); - const lenientRun = await loadCollectionRaw(repoRoot, options, false); + return findingsFrom(repoRoot, options, await loadCollectionRaw(repoRoot, options, false)); +} + +/** The prefix the parser gives an unknown bare key's message (`parser-core.ts`), which a + * lenient load records as a warning with no file or path. */ +const UNKNOWN_BARE_KEY = "Unknown key '"; + +async function findingsFrom( + repoRoot: string, + options: LoadMemoryOptions | undefined, + lenientRun: RawLoad, +): Promise { + const bareKey = lenientRun.result.warnings.some((w) => w.message.startsWith(UNKNOWN_BARE_KEY)); + const strictErrors = bareKey ? (await loadCollectionRaw(repoRoot, options, true)).result.errors : []; const found = [ - ...strictRun.result.errors, + ...strictErrors, ...validateAttrSchema(lenientRun.result.root, lenientRun.registry, true).errors, ].filter(isUnknownAttrError); const seen = new Set(); diff --git a/server/typescript/packages/sdk/test/load-memory-with-findings.test.ts b/server/typescript/packages/sdk/test/load-memory-with-findings.test.ts new file mode 100644 index 0000000000..a818846b0c --- /dev/null +++ b/server/typescript/packages/sdk/test/load-memory-with-findings.test.ts @@ -0,0 +1,91 @@ +// `meta gen` loads leniently and names the unknown attributes a strict load would refuse. +// It used to load the model three times to do that (the lenient load, then a strict and a +// lenient one for the findings); `loadMemoryWithFindings` answers both from one load, and +// parses strictly again only when the model holds an unknown bare key. +import { describe, test, expect, beforeEach, afterEach, spyOn } from "bun:test"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { MetaDataLoader } from "@metaobjectsdev/metadata"; +import { loadMemoryWithFindings } from "../src/memory.js"; + +function model(opts: { bareKey?: boolean; attr?: boolean }): string { + return JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "BaseThing", + ...(opts.bareKey === true ? { isAbstrakt: true } : { abstract: true }), + children: [{ "field.long": { name: "id" } }], + }, + }, + { + "object.entity": { + name: "Talk", + extends: "BaseThing", + children: [ + { "source.rdb": { "@table": "talks" } }, + { "field.string": { name: "title", ...(opts.attr === true ? { "@requird": true } : { "@required": true }) } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, + }); +} + +let root: string; +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "sdk-load-findings-")); + mkdirSync(join(root, "metaobjects"), { recursive: true }); + mkdirSync(join(root, ".metaobjects"), { recursive: true }); + writeFileSync(join(root, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, sources: [] })); +}); +afterEach(() => { rmSync(root, { recursive: true, force: true }); }); + +async function load(opts: { bareKey?: boolean; attr?: boolean }, strict = false) { + writeFileSync(join(root, "metaobjects", "meta.app.json"), model(opts)); + const spy = spyOn(MetaDataLoader.prototype, "load"); + try { + const result = await loadMemoryWithFindings(root, strict ? { strict: true } : undefined); + return { ...result, loads: spy.mock.calls.length }; + } finally { + spy.mockRestore(); + } +} + +describe("loadMemoryWithFindings", () => { + test("a clean model loads once and has no findings", async () => { + const r = await load({}); + expect(r.loads).toBe(1); + expect(r.unknownAttributes).toEqual([]); + expect(r.root.objects().map((o) => o.name).sort()).toEqual(["BaseThing", "Talk"]); + }); + + test("an unknown @attr is found from the same load, with its file", async () => { + const r = await load({ attr: true }); + expect(r.loads).toBe(1); + expect(r.unknownAttributes).toHaveLength(1); + expect(r.unknownAttributes[0]!.message).toContain("@requird"); + expect(JSON.stringify(r.unknownAttributes[0]!.source)).toContain("meta.app.json"); + }); + + test("an unknown bare key costs one strict parse more, which locates it", async () => { + const r = await load({ bareKey: true, attr: true }); + expect(r.loads).toBe(2); + const messages = r.unknownAttributes.map((f) => f.message); + expect(messages.some((m) => m.includes("isAbstrakt"))).toBe(true); + expect(messages.some((m) => m.includes("@requird"))).toBe(true); + for (const f of r.unknownAttributes) expect(JSON.stringify(f.source)).toContain("meta.app.json"); + }); + + test("a strict load reports nothing: it refuses the first finding instead", async () => { + await expect(load({ attr: true }, true)).rejects.toThrow(/@requird/); + const clean = await load({}, true); + expect(clean.loads).toBe(1); + expect(clean.unknownAttributes).toEqual([]); + }); +}); diff --git a/server/typescript/packages/test-generators/src/generators/report-export.ts b/server/typescript/packages/test-generators/src/generators/report-export.ts index 63310482ac..a192742d34 100644 --- a/server/typescript/packages/test-generators/src/generators/report-export.ts +++ b/server/typescript/packages/test-generators/src/generators/report-export.ts @@ -248,7 +248,8 @@ export function reportExport(opts?: ReportExportOpts): Generator { const out: EmittedFile[] = [ { path: `${EXPORT_WRITERS_BASENAME}.ts`, content: await formatTs(renderExportWriters(emit)) }, ]; - for (const report of reports) { + // One module per report `meta gen ` named; the writers above serve them all. + for (const report of reports.filter((r) => ctx.selected?.(r) ?? true)) { const pkg = effectivePackage(report); out.push({ path: entityOutputPath(layout, pkg, `${report.name}.export.ts`), From d4619149e40db3904845ae24a14c3becb8c9c82d Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 12:32:00 -0500 Subject: [PATCH 2/9] test: keep throwaway test databases in RAM and remove them when a run ends Database test containers mount a size-capped tmpfs at the image's VOLUME path, Postgres runs with fsync, synchronous_commit and full_page_writes off, and every docker-CLI launcher also removes its containers on exit and SIGTERM (covering a disabled Ryuk). The CI sidecar and workflow services get the same treatment. The Stryker sandbox moves to a RAM scratch dir and its runners stop with it. Work in progress: AGENTS.md section, CHANGELOG and suite proof follow. --- .github/workflows/integration-tests.yml | 13 +++ .github/workflows/local-ci.yml | 52 +++++++++++ scripts/ci-local.sh | 6 +- scripts/run-stryker.sh | 40 +++++++++ .../Runner/MySqlDatabase.cs | 3 +- .../Runner/PostgresContainer.cs | 5 +- .../Runner/RamStorage.cs | 32 +++++++ .../kotlin/exposed1x/PostgresContainer.kt | 7 ++ .../kotlin/exposed1x/ReaperLabels.kt | 42 +++++++++ .../integration/kotlin/MySqlContainer.kt | 5 ++ .../integration/kotlin/PostgresContainer.kt | 7 ++ .../integration/kotlin/ReaperLabels.kt | 42 +++++++++ .../integration/MySqlContainer.java | 5 ++ .../integration/PostgresContainer.java | 17 ++-- .../metaobjects/integration/ReaperLabels.java | 46 ++++++++++ .../tests/integration/mysql_container.py | 11 ++- .../tests/integration/postgres_container.py | 11 ++- server/python/tests/integration/throwaway.py | 90 +++++++++++++++++++ .../integration-tests/src/cube-container.ts | 10 ++- .../integration-tests/src/mysql-container.ts | 13 ++- .../src/postgres-container.ts | 13 ++- .../integration-tests/src/throwaway.ts | 71 +++++++++++++++ .../typescript/packages/metadata/package.json | 2 +- .../packages/metadata/stryker.conf.mjs | 13 ++- 24 files changed, 532 insertions(+), 24 deletions(-) create mode 100755 scripts/run-stryker.sh create mode 100644 server/csharp/MetaObjects.IntegrationTests/Runner/RamStorage.cs create mode 100644 server/python/tests/integration/throwaway.py create mode 100644 server/typescript/packages/integration-tests/src/throwaway.ts diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index c80646a310..48d92dfe15 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -66,7 +66,10 @@ jobs: POSTGRES_DB: metaobjects_test ports: - 5432:5432 + # The data directory (the image's VOLUME) on a size-capped tmpfs: a throwaway database + # never touches the runner's drives. options: >- + --tmpfs /var/lib/postgresql/data:rw,size=2g --health-cmd "pg_isready -U metaobjects -d metaobjects_test" --health-interval 10s --health-timeout 5s @@ -75,6 +78,16 @@ jobs: - uses: actions/checkout@v4 with: persist-credentials: false + - name: Postgres durability off (a throwaway database) + # A service container takes no server flags, so set them on the running server. Its data + # directory is already a tmpfs (options above); nothing in it needs to survive a crash. + run: >- + docker exec ${{ job.services.postgres.id }} + psql -U metaobjects -d metaobjects_test + -c "ALTER SYSTEM SET fsync = off" + -c "ALTER SYSTEM SET synchronous_commit = off" + -c "ALTER SYSTEM SET full_page_writes = off" + -c "SELECT pg_reload_conf()" - name: Set up Bun (TS lanes) if: matrix.lane == 'ts-slow' || matrix.lane == 'cube' diff --git a/.github/workflows/local-ci.yml b/.github/workflows/local-ci.yml index a7b82e8e11..29c62bcb2d 100644 --- a/.github/workflows/local-ci.yml +++ b/.github/workflows/local-ci.yml @@ -146,7 +146,10 @@ jobs: POSTGRES_DB: metaobjects_test ports: - 5432 # dynamic host port — no fixed-port collision + # The data directory (the image's VOLUME) on a size-capped tmpfs: a throwaway database + # never touches the runner's drives. options: >- + --tmpfs /var/lib/postgresql/data:rw,size=2g --health-cmd "pg_isready -U metaobjects -d metaobjects_test" --health-interval 10s --health-timeout 5s @@ -155,6 +158,16 @@ jobs: - uses: actions/checkout@v4 with: persist-credentials: false + - name: Postgres durability off (a throwaway database) + # A service container takes no server flags, so set them on the running server. Its data + # directory is already a tmpfs (options above); nothing in it needs to survive a crash. + run: >- + docker exec ${{ job.services.postgres.id }} + psql -U metaobjects -d metaobjects_test + -c "ALTER SYSTEM SET fsync = off" + -c "ALTER SYSTEM SET synchronous_commit = off" + -c "ALTER SYSTEM SET full_page_writes = off" + -c "SELECT pg_reload_conf()" - env: METAOBJECTS_TEST_PG_URL: postgres://metaobjects:metaobjects@localhost:${{ job.services.postgres.ports['5432'] }}/metaobjects_test # Arms migrate-ts's real-Postgres suites (apply / lifecycle / rollback / @@ -205,7 +218,10 @@ jobs: POSTGRES_DB: metaobjects_test ports: - 5432 + # The data directory (the image's VOLUME) on a size-capped tmpfs: a throwaway database + # never touches the runner's drives. options: >- + --tmpfs /var/lib/postgresql/data:rw,size=2g --health-cmd "pg_isready -U metaobjects -d metaobjects_test" --health-interval 10s --health-timeout 5s @@ -214,6 +230,16 @@ jobs: - uses: actions/checkout@v4 with: persist-credentials: false + - name: Postgres durability off (a throwaway database) + # A service container takes no server flags, so set them on the running server. Its data + # directory is already a tmpfs (options above); nothing in it needs to survive a crash. + run: >- + docker exec ${{ job.services.postgres.id }} + psql -U metaobjects -d metaobjects_test + -c "ALTER SYSTEM SET fsync = off" + -c "ALTER SYSTEM SET synchronous_commit = off" + -c "ALTER SYSTEM SET full_page_writes = off" + -c "SELECT pg_reload_conf()" - env: METAOBJECTS_TEST_PG_URL: postgres://metaobjects:metaobjects@localhost:${{ job.services.postgres.ports['5432'] }}/metaobjects_test run: scripts/ci-local.sh --only java-slow --strict-toolchains @@ -232,7 +258,10 @@ jobs: POSTGRES_DB: metaobjects_test ports: - 5432 + # The data directory (the image's VOLUME) on a size-capped tmpfs: a throwaway database + # never touches the runner's drives. options: >- + --tmpfs /var/lib/postgresql/data:rw,size=2g --health-cmd "pg_isready -U metaobjects -d metaobjects_test" --health-interval 10s --health-timeout 5s @@ -241,6 +270,16 @@ jobs: - uses: actions/checkout@v4 with: persist-credentials: false + - name: Postgres durability off (a throwaway database) + # A service container takes no server flags, so set them on the running server. Its data + # directory is already a tmpfs (options above); nothing in it needs to survive a crash. + run: >- + docker exec ${{ job.services.postgres.id }} + psql -U metaobjects -d metaobjects_test + -c "ALTER SYSTEM SET fsync = off" + -c "ALTER SYSTEM SET synchronous_commit = off" + -c "ALTER SYSTEM SET full_page_writes = off" + -c "SELECT pg_reload_conf()" - env: METAOBJECTS_TEST_PG_URL: postgres://metaobjects:metaobjects@localhost:${{ job.services.postgres.ports['5432'] }}/metaobjects_test run: scripts/ci-local.sh --only python --strict-toolchains @@ -259,7 +298,10 @@ jobs: POSTGRES_DB: metaobjects_test ports: - 5432 + # The data directory (the image's VOLUME) on a size-capped tmpfs: a throwaway database + # never touches the runner's drives. options: >- + --tmpfs /var/lib/postgresql/data:rw,size=2g --health-cmd "pg_isready -U metaobjects -d metaobjects_test" --health-interval 10s --health-timeout 5s @@ -268,6 +310,16 @@ jobs: - uses: actions/checkout@v4 with: persist-credentials: false + - name: Postgres durability off (a throwaway database) + # A service container takes no server flags, so set them on the running server. Its data + # directory is already a tmpfs (options above); nothing in it needs to survive a crash. + run: >- + docker exec ${{ job.services.postgres.id }} + psql -U metaobjects -d metaobjects_test + -c "ALTER SYSTEM SET fsync = off" + -c "ALTER SYSTEM SET synchronous_commit = off" + -c "ALTER SYSTEM SET full_page_writes = off" + -c "SELECT pg_reload_conf()" - env: METAOBJECTS_TEST_PG_URL: postgres://metaobjects:metaobjects@localhost:${{ job.services.postgres.ports['5432'] }}/metaobjects_test run: scripts/ci-local.sh --only csharp --strict-toolchains diff --git a/scripts/ci-local.sh b/scripts/ci-local.sh index 17428d5859..e1accb6fec 100755 --- a/scripts/ci-local.sh +++ b/scripts/ci-local.sh @@ -952,10 +952,14 @@ start_pg_sidecar() { fi # Clear a stopped leftover of an MO_PG_SIDECAR_NAME override, then start fresh. docker rm -f -v "$name" >/dev/null 2>&1 || true + # The data directory (the image's VOLUME) is a size-capped tmpfs and durability is off: + # a throwaway database never touches the drives, and leaves no anonymous volume behind. if ! docker run -d --name "$name" --label "$MO_CI_SESSION_LABEL" \ + --tmpfs /var/lib/postgresql/data:rw,size=2g \ -e POSTGRES_USER=metaobjects -e POSTGRES_PASSWORD=metaobjects \ -e POSTGRES_DB=metaobjects_test \ - -p 127.0.0.1::5432 postgres:16 >/dev/null 2>&1; then + -p 127.0.0.1::5432 postgres:16 \ + -c fsync=off -c synchronous_commit=off -c full_page_writes=off >/dev/null 2>&1; then echo " ⚠ could not start the Postgres sidecar — falling back to per-scenario" >&2 echo " containers (slow). Integration results are unaffected." >&2 return 0 diff --git a/scripts/run-stryker.sh b/scripts/run-stryker.sh new file mode 100755 index 0000000000..71dfa57341 --- /dev/null +++ b/scripts/run-stryker.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Run a package's Stryker mutation gate without wearing the drives or leaving anything behind. +# +# scripts/run-stryker.sh [stryker run options] (run from the package directory) +# +# - The sandbox (a full copy of the package, rewritten once per mutant) goes under +# $DEV_SCRATCH/stryker/-XXXXXX, a RAM-backed scratch area, falling back to the +# system temp directory when DEV_SCRATCH is unset. The package's stryker.conf.mjs reads the +# directory from METAOBJECTS_STRYKER_TEMP. +# - Stryker runs in its own process group, so its test-runner children go with it: on exit, on +# failure and on Ctrl-C or SIGTERM the whole group is stopped before the sandbox is removed. +set -uo pipefail + +label="${1:?usage: run-stryker.sh }" +scratch="${DEV_SCRATCH:-${TMPDIR:-/tmp}}" +mkdir -p "$scratch/stryker" +sandbox="$(mktemp -d "$scratch/stryker/${label}-XXXXXX")" +export METAOBJECTS_STRYKER_TEMP="$sandbox" + +pid="" +cleanup() { + if [[ -n "$pid" ]]; then + kill -TERM -- "-$pid" 2>/dev/null + for _ in 1 2 3 4 5 6 7 8 9 10; do kill -0 -- "-$pid" 2>/dev/null || break; sleep 0.5; done + kill -KILL -- "-$pid" 2>/dev/null + fi + rm -rf "$sandbox" +} +trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +# A background job of a non-interactive shell is not a group leader, so setsid execs in place: +# $pid is Stryker's own pid and the id of its new process group. +setsid stryker run "${@:2}" & +pid=$! +wait "$pid" +status=$? +# Stryker has exited; the EXIT trap stops any runner left in its group and removes the sandbox. +exit "$status" diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/MySqlDatabase.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/MySqlDatabase.cs index bc74f275c3..a15e46906e 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/MySqlDatabase.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/MySqlDatabase.cs @@ -32,7 +32,8 @@ private async Task StartAsync() if (!string.IsNullOrWhiteSpace(existing)) return FromUrl(existing); _container = new MySqlBuilder("mysql:8.4").WithUsername("root") - .WithCommand("--innodb-use-native-aio=0").Build(); + .WithCommand("--innodb-use-native-aio=0") + .WithCreateParameterModifier(RamStorage.Tmpfs(RamStorage.MySql8DataDir)).Build(); await _container.StartAsync(); return _container.GetConnectionString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/PostgresContainer.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/PostgresContainer.cs index c12ed3d904..d85f1f800f 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/PostgresContainer.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/PostgresContainer.cs @@ -47,7 +47,10 @@ public static async Task StartAsync() return await StartOnSharedAsync(sharedUrl); } - var container = new PostgreSqlBuilder("postgres:16-alpine").Build(); + // Data on a tmpfs, durability off (the builder's default), Ryuk-reaped (RamStorage). + var container = new PostgreSqlBuilder("postgres:16-alpine") + .WithCreateParameterModifier(RamStorage.Tmpfs(RamStorage.Postgres16DataDir)) + .Build(); await container.StartAsync(); return new PostgresContainer(container); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/RamStorage.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/RamStorage.cs new file mode 100644 index 0000000000..45a4352ee8 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/RamStorage.cs @@ -0,0 +1,32 @@ +// RamStorage — a test database keeps its data in RAM. +// +// Each database image declares a VOLUME for its data directory; a container started without a +// mount there gets an anonymous volume on disk. A size-capped tmpfs at that path keeps a +// throwaway database off the drives. Postgres durability is already off: the Testcontainers +// PostgreSql builder starts the server with fsync, synchronous_commit and full_page_writes off. +// The Testcontainers resource reaper (Ryuk) removes the containers of a run that dies before +// its dispose. + +using Docker.DotNet.Models; + +namespace MetaObjects.IntegrationTests.Runner; + +internal static class RamStorage +{ + /// Upper bound for one database's data directory; tmpfs takes RAM only for what is written. + public const string TmpfsSize = "2g"; + + /// The VOLUME postgres:16* declares (docker image inspect). + public const string Postgres16DataDir = "/var/lib/postgresql/data"; + + /// The VOLUME mysql:8* declares (docker image inspect). + public const string MySql8DataDir = "/var/lib/mysql"; + + /// A create-parameter modifier mounting a size-capped tmpfs at . + public static Action Tmpfs(string path) => parameters => + { + parameters.HostConfig ??= new HostConfig(); + parameters.HostConfig.Tmpfs ??= new Dictionary(); + parameters.HostConfig.Tmpfs[path] = $"rw,size={TmpfsSize}"; + }; +} diff --git a/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/PostgresContainer.kt b/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/PostgresContainer.kt index fb44d53b03..c28800953d 100644 --- a/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/PostgresContainer.kt +++ b/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/PostgresContainer.kt @@ -50,12 +50,15 @@ class PostgresContainer : AutoCloseable { // Docker assigns the host port at bind time (`-p 127.0.0.1::5432`, read back with // `docker port`): a port picked, released and handed to docker races parallel lanes. jdbcUrl = try { + ReaperLabels.removeAtExit(name) runDocker( *ReaperLabels.dockerRun( "--name", name, + *ReaperLabels.ramVolume(DATA_DIR), "-e", "POSTGRES_PASSWORD=$PG_PASSWORD", "-p", "127.0.0.1::5432", IMAGE, + *ReaperLabels.POSTGRES_DURABILITY_OFF, ), ) val url = "jdbc:postgresql://127.0.0.1:${publishedPort(name, 5432)}/postgres" @@ -82,6 +85,7 @@ class PostgresContainer : AutoCloseable { } catch (_: RuntimeException) { // Container may already be gone; ignore. } + ReaperLabels.forget(name!!) } private fun execAdmin(sql: String) { @@ -129,6 +133,7 @@ class PostgresContainer : AutoCloseable { private fun forceRemove() { try { runDocker("rm", "-f", "-v", name!!) } catch (_: RuntimeException) { /* already gone */ } + ReaperLabels.forget(name!!) } private fun inspectState(): String = try { @@ -146,6 +151,8 @@ class PostgresContainer : AutoCloseable { companion object { private const val SHARED_PG_URL_ENV = "METAOBJECTS_TEST_PG_URL" private const val IMAGE = "postgres:16-alpine" + /** The image's VOLUME (`docker image inspect`); a tmpfs there keeps the data in RAM. */ + private const val DATA_DIR = "/var/lib/postgresql/data" private const val PG_USER = "postgres" private const val PG_PASSWORD = "test" private val READY_TIMEOUT_S: Int = diff --git a/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/ReaperLabels.kt b/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/ReaperLabels.kt index 7abcef839c..a91dba779c 100644 --- a/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/ReaperLabels.kt +++ b/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/ReaperLabels.kt @@ -16,6 +16,10 @@ import org.testcontainers.utility.ResourceReaper * Each JVM gets its own session, so one fork's death reaps that fork's containers and no other * run's. `TESTCONTAINERS_RYUK_DISABLED=true` turns the reaper off. * + * [removeAtExit] covers what Ryuk does not: a run with Ryuk disabled, and a SIGTERM, which runs + * shutdown hooks. [ramVolume] keeps a database's data directory in RAM: an image's `VOLUME` with no + * mount becomes an anonymous volume on disk. + * * Duplicated from `integration-tests-kotlin` for the reason [PostgresContainer] gives. */ internal object ReaperLabels { @@ -31,4 +35,42 @@ internal object ReaperLabels { */ fun dockerRun(vararg rest: String): Array = (listOf("run", "-d") + labelArgs + rest).toTypedArray() + + /** Upper bound for one database's data directory; tmpfs takes RAM only for what is written. */ + const val TMPFS_SIZE = "2g" + + /** `postgres` server options for a database whose contents may be lost at any moment. */ + val POSTGRES_DURABILITY_OFF: Array = + arrayOf("-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off") + + private val started: MutableSet = java.util.concurrent.ConcurrentHashMap.newKeySet() + private val shutdownHook: Unit by lazy { + Runtime.getRuntime().addShutdownHook(Thread(::removeAll, "remove-test-containers")) + } + + /** `docker run` flags mounting a size-capped tmpfs at [path], the image's VOLUME. */ + fun ramVolume(path: String): Array = arrayOf("--tmpfs", "$path:rw,size=$TMPFS_SIZE") + + /** Remove container [name] when the JVM shuts down, unless [forget] says it is gone. */ + fun removeAtExit(name: String) { + started.add(name) + shutdownHook + } + + /** The launcher removed [name] itself. */ + fun forget(name: String) { + started.remove(name) + } + + private fun removeAll() { + for (name in started) { + try { + ProcessBuilder("docker", "rm", "-f", "-v", name) + .redirectErrorStream(true).redirectOutput(ProcessBuilder.Redirect.DISCARD) + .start().waitFor(60, java.util.concurrent.TimeUnit.SECONDS) + } catch (_: Exception) { + // Best effort: Ryuk still removes a labelled container. + } + } + } } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/MySqlContainer.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/MySqlContainer.kt index a57f917ffd..74872b5c3c 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/MySqlContainer.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/MySqlContainer.kt @@ -34,9 +34,11 @@ class MySqlContainer : AutoCloseable { password = PASSWORD // Docker assigns the host port at bind time; see the port note in PostgresContainer. jdbcUrl = try { + ReaperLabels.removeAtExit(name) runDocker( *ReaperLabels.dockerRun( "--name", name, + *ReaperLabels.ramVolume(DATA_DIR), "-e", "MYSQL_ROOT_PASSWORD=$PASSWORD", "-e", "MYSQL_DATABASE=$DB", "-p", "127.0.0.1::3306", @@ -92,6 +94,7 @@ class MySqlContainer : AutoCloseable { private fun forceRemove() { try { runDocker("rm", "-f", "-v", name!!) } catch (_: RuntimeException) { /* already gone */ } + ReaperLabels.forget(name!!) } private fun inspectState(): String = try { @@ -105,6 +108,8 @@ class MySqlContainer : AutoCloseable { companion object { private const val URL_ENV = "METAOBJECTS_TEST_MYSQL_URL" private const val IMAGE = "mysql:8.4" + /** The image's VOLUME (`docker image inspect`); a tmpfs there keeps the data in RAM. */ + private const val DATA_DIR = "/var/lib/mysql" private const val DB = "mo_test" private const val PASSWORD = "test" private val READY_TIMEOUT_S: Int = (System.getenv("MO_MYSQL_READY_TIMEOUT_S") ?: "180").toInt() diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/PostgresContainer.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/PostgresContainer.kt index f1edc2c403..d585a63670 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/PostgresContainer.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/PostgresContainer.kt @@ -66,12 +66,15 @@ class PostgresContainer : AutoCloseable { // needed. close() force-removes, and Ryuk removes it when a killed run never // reaches close() (see ReaperLabels). jdbcUrl = try { + ReaperLabels.removeAtExit(name) runDocker( *ReaperLabels.dockerRun( "--name", name, + *ReaperLabels.ramVolume(DATA_DIR), "-e", "POSTGRES_PASSWORD=$PG_PASSWORD", "-p", "127.0.0.1::5432", IMAGE, + *ReaperLabels.POSTGRES_DURABILITY_OFF, ), ) val url = "jdbc:postgresql://127.0.0.1:${publishedPort(name, 5432)}/postgres" @@ -98,6 +101,7 @@ class PostgresContainer : AutoCloseable { } catch (_: RuntimeException) { // Container may already be gone; ignore. } + ReaperLabels.forget(name!!) } // ----------------------------------------------------------------------- @@ -159,6 +163,7 @@ class PostgresContainer : AutoCloseable { /** Best-effort teardown of a container that failed to start. */ private fun forceRemove() { try { runDocker("rm", "-f", "-v", name!!) } catch (_: RuntimeException) { /* already gone */ } // non-null in per-container mode + ReaperLabels.forget(name!!) } /** The container's docker state, or "gone" when it cannot be determined. */ @@ -179,6 +184,8 @@ class PostgresContainer : AutoCloseable { /** Env var naming the shared CI Postgres sidecar (admin URL). Unset = per-container fallback. */ private const val SHARED_PG_URL_ENV = "METAOBJECTS_TEST_PG_URL" private const val IMAGE = "postgres:16-alpine" + /** The image's VOLUME (`docker image inspect`); a tmpfs there keeps the data in RAM. */ + private const val DATA_DIR = "/var/lib/postgresql/data" private const val PG_USER = "postgres" private const val PG_PASSWORD = "test" diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ReaperLabels.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ReaperLabels.kt index fb924e9640..160bd9ce90 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ReaperLabels.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ReaperLabels.kt @@ -15,6 +15,10 @@ import org.testcontainers.utility.ResourceReaper * * Each JVM gets its own session, so one fork's death reaps that fork's containers and no other * run's. `TESTCONTAINERS_RYUK_DISABLED=true` turns the reaper off. + * + * [removeAtExit] covers what Ryuk does not: a run with Ryuk disabled, and a SIGTERM, which runs + * shutdown hooks. [ramVolume] keeps a database's data directory in RAM: an image's `VOLUME` with no + * mount becomes an anonymous volume on disk. */ internal object ReaperLabels { private val labelArgs: List by lazy { @@ -29,4 +33,42 @@ internal object ReaperLabels { */ fun dockerRun(vararg rest: String): Array = (listOf("run", "-d") + labelArgs + rest).toTypedArray() + + /** Upper bound for one database's data directory; tmpfs takes RAM only for what is written. */ + const val TMPFS_SIZE = "2g" + + /** `postgres` server options for a database whose contents may be lost at any moment. */ + val POSTGRES_DURABILITY_OFF: Array = + arrayOf("-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off") + + private val started: MutableSet = java.util.concurrent.ConcurrentHashMap.newKeySet() + private val shutdownHook: Unit by lazy { + Runtime.getRuntime().addShutdownHook(Thread(::removeAll, "remove-test-containers")) + } + + /** `docker run` flags mounting a size-capped tmpfs at [path], the image's VOLUME. */ + fun ramVolume(path: String): Array = arrayOf("--tmpfs", "$path:rw,size=$TMPFS_SIZE") + + /** Remove container [name] when the JVM shuts down, unless [forget] says it is gone. */ + fun removeAtExit(name: String) { + started.add(name) + shutdownHook + } + + /** The launcher removed [name] itself. */ + fun forget(name: String) { + started.remove(name) + } + + private fun removeAll() { + for (name in started) { + try { + ProcessBuilder("docker", "rm", "-f", "-v", name) + .redirectErrorStream(true).redirectOutput(ProcessBuilder.Redirect.DISCARD) + .start().waitFor(60, java.util.concurrent.TimeUnit.SECONDS) + } catch (_: Exception) { + // Best effort: Ryuk still removes a labelled container. + } + } + } } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/MySqlContainer.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/MySqlContainer.java index bd28f3865d..d254b4eda9 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/MySqlContainer.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/MySqlContainer.java @@ -23,6 +23,8 @@ public final class MySqlContainer implements AutoCloseable, JdbcTarget { private static final String URL_ENV = "METAOBJECTS_TEST_MYSQL_URL"; private static final String IMAGE = "mysql:8.4"; + /** The image's VOLUME ({@code docker image inspect}); a tmpfs there keeps the data in RAM. */ + private static final String DATA_DIR = "/var/lib/mysql"; private static final String DB = "mo_test"; private static final String PASSWORD = "test"; private static final int READY_TIMEOUT_S = @@ -47,7 +49,9 @@ public MySqlContainer() { this.password = PASSWORD; // Docker assigns the host port at bind time; see the port note in PostgresContainer. try { + ReaperLabels.removeAtExit(name); runDocker(ReaperLabels.dockerRun("--name", name, + "--tmpfs", DATA_DIR + ":rw,size=" + ReaperLabels.TMPFS_SIZE, "-e", "MYSQL_ROOT_PASSWORD=" + PASSWORD, "-e", "MYSQL_DATABASE=" + DB, "-p", "127.0.0.1::3306", @@ -110,6 +114,7 @@ private boolean canConnect(String url) { private void forceRemove() { try { runDocker("rm", "-f", "-v", name); } catch (RuntimeException ignored) { /* already gone */ } + ReaperLabels.forget(name); } private String inspectState() { diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/PostgresContainer.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/PostgresContainer.java index e057ee826d..48f0201fc3 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/PostgresContainer.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/PostgresContainer.java @@ -9,6 +9,8 @@ import java.sql.DriverManager; import java.sql.SQLException; import java.sql.Statement; +import java.util.ArrayList; +import java.util.List; import java.util.UUID; /** @@ -37,6 +39,8 @@ public final class PostgresContainer implements AutoCloseable, JdbcTarget { private static final String SHARED_PG_URL_ENV = "METAOBJECTS_TEST_PG_URL"; private static final String IMAGE = "postgres:16-alpine"; + /** The image's VOLUME ({@code docker image inspect}); a tmpfs there keeps the data in RAM. */ + private static final String DATA_DIR = "/var/lib/postgresql/data"; private static final String PG_USER = "postgres"; private static final String PG_PASSWORD = "test"; @@ -82,11 +86,12 @@ public PostgresContainer() { // needed. close() force-removes, and Ryuk removes it when a killed run never // reaches close() (see ReaperLabels). try { - runDocker(ReaperLabels.dockerRun( - "--name", name, - "-e", "POSTGRES_PASSWORD=" + PG_PASSWORD, - "-p", "127.0.0.1::5432", - IMAGE)); + ReaperLabels.removeAtExit(name); + List run = new ArrayList<>(List.of("--name", name)); + run.addAll(ReaperLabels.ramVolume(DATA_DIR)); + run.addAll(List.of("-e", "POSTGRES_PASSWORD=" + PG_PASSWORD, "-p", "127.0.0.1::5432", IMAGE)); + run.addAll(ReaperLabels.POSTGRES_DURABILITY_OFF); + runDocker(ReaperLabels.dockerRun(run.toArray(String[]::new))); String url = "jdbc:postgresql://127.0.0.1:" + publishedPort(name, 5432) + "/postgres"; waitForReady(url); this.jdbcUrl = url; @@ -108,6 +113,7 @@ public PostgresContainer() { } try { runDocker("rm", "-f", "-v", name); } catch (RuntimeException ignored) { /* container may already be gone */ } + ReaperLabels.forget(name); } /** Run a statement against the shared sidecar's admin database (auto-commit). */ @@ -203,6 +209,7 @@ private void waitForReady(String url) { /** Best-effort teardown of a container that failed to start. */ private void forceRemove() { try { runDocker("rm", "-f", "-v", name); } catch (RuntimeException ignored) { /* already gone */ } + ReaperLabels.forget(name); } /** The container's docker state, or null when it cannot be determined. */ diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ReaperLabels.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ReaperLabels.java index 6fe1913dd2..2d5e4d16be 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ReaperLabels.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ReaperLabels.java @@ -6,7 +6,10 @@ import java.util.ArrayList; import java.util.List; import java.util.Map; +import java.util.Set; import java.util.TreeMap; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; /** * Puts a {@code docker} CLI test container under Ryuk, the Testcontainers reaper. @@ -20,9 +23,21 @@ * *

Each JVM gets its own session, so one fork's death reaps that fork's containers and no other * run's. {@code TESTCONTAINERS_RYUK_DISABLED=true} turns the reaper off. + * + *

{@link #removeAtExit} covers what Ryuk does not: a run with Ryuk disabled, and a SIGTERM, which + * runs shutdown hooks. {@link #ramVolume} keeps a database's data directory in RAM: an image's + * {@code VOLUME} with no mount becomes an anonymous volume on disk. */ final class ReaperLabels { + /** Upper bound for one database's data directory; tmpfs takes RAM only for what is written. */ + static final String TMPFS_SIZE = "2g"; + /** {@code postgres} server options for a database whose contents may be lost at any moment. */ + static final List POSTGRES_DURABILITY_OFF = List.of( + "-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off"); + private static List labelArgs; + private static final Set started = ConcurrentHashMap.newKeySet(); + private static boolean shutdownHookInstalled; private ReaperLabels() {} @@ -48,4 +63,35 @@ private static synchronized List labelArgs() { } return labelArgs; } + + /** {@code docker run} flags mounting a size-capped tmpfs at {@code path}, the image's VOLUME. */ + static List ramVolume(String path) { + return List.of("--tmpfs", path + ":rw,size=" + TMPFS_SIZE); + } + + /** Remove container {@code name} when the JVM shuts down, unless {@link #forget} says it is gone. */ + static synchronized void removeAtExit(String name) { + started.add(name); + if (!shutdownHookInstalled) { + shutdownHookInstalled = true; + Runtime.getRuntime().addShutdownHook(new Thread(ReaperLabels::removeAll, "remove-test-containers")); + } + } + + /** The launcher removed {@code name} itself. */ + static void forget(String name) { + started.remove(name); + } + + private static void removeAll() { + for (String name : started) { + try { + new ProcessBuilder("docker", "rm", "-f", "-v", name) + .redirectErrorStream(true).redirectOutput(ProcessBuilder.Redirect.DISCARD) + .start().waitFor(60, TimeUnit.SECONDS); + } catch (Exception ignored) { + // Best effort: Ryuk still removes a labelled container. + } + } + } } diff --git a/server/python/tests/integration/mysql_container.py b/server/python/tests/integration/mysql_container.py index b1444a5b94..1b1fd3def2 100644 --- a/server/python/tests/integration/mysql_container.py +++ b/server/python/tests/integration/mysql_container.py @@ -1,7 +1,8 @@ """A throwaway MySQL 8.4 server for the MySQL conformance lanes. -Shells out to the ``docker`` CLI exactly like :mod:`postgres_container`, with Ryuk -removing it when a killed run never reaches teardown (see :mod:`reaper`). One server is started for a whole pytest session (see ``conftest.py``); every scenario starts +Shells out to the ``docker`` CLI exactly like :mod:`postgres_container`, with its data on a +tmpfs and Ryuk removing it when a killed run never reaches teardown (see :mod:`reaper` and +:mod:`throwaway`). One server is started for a whole pytest session (see ``conftest.py``); every scenario starts from an EMPTY schema via :func:`reset_database`, which is what the Java and Kotlin lanes do. Set ``METAOBJECTS_TEST_MYSQL_URL`` (``mysql://user:password@host:port/database``) to run @@ -20,6 +21,7 @@ from .postgres_container import published_port from .reaper import reaper_label_args +from .throwaway import forget, ram_volume, remove_on_exit SHARED_MYSQL_URL_ENV = "METAOBJECTS_TEST_MYSQL_URL" @@ -63,6 +65,8 @@ def reset_database(info: MySqlInfo) -> None: class MySqlContainer: IMAGE = "mysql:8.4" + #: The image's ``VOLUME`` (``docker image inspect``); a tmpfs there keeps the data in RAM. + DATA_DIR = "/var/lib/mysql" DATABASE = "mo_test" USER = "root" PASSWORD = "test" @@ -81,8 +85,10 @@ def __init__(self) -> None: self._name = f"metaobjects-mysql-py-{uuid.uuid4().hex[:8]}" # Docker assigns the host port at bind time; see the port note in postgres_container. try: + remove_on_exit(self._name) _docker( "run", "-d", *reaper_label_args(), "--name", self._name, + *ram_volume(self.DATA_DIR), "-e", f"MYSQL_ROOT_PASSWORD={self.PASSWORD}", "-e", f"MYSQL_DATABASE={self.DATABASE}", # Native AIO reserves ~17k kernel aio-nr slots per server; with the default @@ -115,6 +121,7 @@ def _force_remove(self) -> None: _docker("rm", "-f", "-v", str(self._name)) except Exception: # noqa: BLE001 -- already gone pass + forget(str(self._name)) def _state(self) -> str: r = subprocess.run( diff --git a/server/python/tests/integration/postgres_container.py b/server/python/tests/integration/postgres_container.py index cfa3d2b6ae..0d6f262c02 100644 --- a/server/python/tests/integration/postgres_container.py +++ b/server/python/tests/integration/postgres_container.py @@ -22,6 +22,7 @@ from urllib.parse import urlsplit from .reaper import reaper_label_args +from .throwaway import POSTGRES_DURABILITY_OFF, forget, ram_volume, remove_on_exit # Env var naming the shared CI Postgres sidecar (admin URL). Unset = per-container. SHARED_PG_URL_ENV = "METAOBJECTS_TEST_PG_URL" @@ -41,6 +42,8 @@ class PostgresContainer: """Context manager around ``docker run`` for a per-scenario Postgres.""" IMAGE = "postgres:16-alpine" + #: The image's ``VOLUME`` (``docker image inspect``); a tmpfs there keeps the data in RAM. + DATA_DIR = "/var/lib/postgresql/data" USER = "postgres" PASSWORD = "test" @@ -64,14 +67,16 @@ def __init__(self) -> None: # Deliberately NOT `--rm`: an auto-removed container takes its logs with it, so # the one artefact explaining a failure vanishes exactly when it is needed. # close() force-removes, and Ryuk removes it when a killed run never reaches - # close() (see :mod:`reaper`). + # close() (see :mod:`reaper`); :mod:`throwaway` covers SIGTERM and a disabled Ryuk. try: + remove_on_exit(self._name) _docker( "run", "-d", *reaper_label_args(), "--name", self._name, + *ram_volume(self.DATA_DIR), "-e", f"POSTGRES_PASSWORD={self.PASSWORD}", "-p", "127.0.0.1::5432", - self.IMAGE, + self.IMAGE, *POSTGRES_DURABILITY_OFF, ) self._port = published_port(self._name, 5432) self._info = PostgresInfo( @@ -136,6 +141,7 @@ def stop(self) -> None: _docker("rm", "-f", "-v", self._name) except subprocess.CalledProcessError: pass # container already gone — fine + forget(self._name) def __enter__(self) -> "PostgresContainer": return self @@ -162,6 +168,7 @@ def _force_remove(self) -> None: _docker("rm", "-f", "-v", self._name) except Exception: # noqa: BLE001 — already gone pass + forget(self._name) def _inspect_state(self) -> str: """The container's docker state, or 'gone' when it cannot be determined.""" diff --git a/server/python/tests/integration/throwaway.py b/server/python/tests/integration/throwaway.py new file mode 100644 index 0000000000..c68040f595 --- /dev/null +++ b/server/python/tests/integration/throwaway.py @@ -0,0 +1,90 @@ +"""A test database lives in RAM and never outlives the run that started it. + +Storage. Each database image declares a ``VOLUME`` for its data directory. A container started +without a mount there gets an anonymous volume on disk, which stays behind after the container +is removed without ``-v``. Every launcher therefore mounts a size-capped tmpfs at that path +(:func:`ram_volume`), and Postgres runs with durability off (:data:`POSTGRES_DURABILITY_OFF`): +nothing written to a throwaway database needs to survive a crash. + +Removal. Teardown in a fixture runs only when the process gets that far. Ryuk (see +:mod:`reaper`) removes the containers of a killed process from outside it. This module covers +the cases Ryuk does not: a run with ``TESTCONTAINERS_RYUK_DISABLED=true``, and a plain SIGTERM, +which ends Python without running a fixture's teardown. Every container a launcher starts is +registered with :func:`remove_on_exit`; it is removed at interpreter exit and on SIGTERM unless +its launcher already removed it and called :func:`forget`. +""" +from __future__ import annotations + +import atexit +import os +import signal +import subprocess +import threading +from types import FrameType + +#: Upper bound for one database's data directory. A tmpfs takes RAM only for what is written. +TMPFS_SIZE = "2g" + +#: ``postgres`` server options for a database whose contents may be lost at any moment. +POSTGRES_DURABILITY_OFF: tuple[str, ...] = ( + "-c", "fsync=off", + "-c", "synchronous_commit=off", + "-c", "full_page_writes=off", +) + + +def ram_volume(path: str) -> tuple[str, str]: + """``docker run`` flags mounting a size-capped tmpfs at *path*, the image's ``VOLUME``.""" + return ("--tmpfs", f"{path}:rw,size={TMPFS_SIZE}") + + +_lock = threading.Lock() +_started: set[str] = set() +_hooks_installed = False + + +def remove_on_exit(name: str) -> None: + """Remove container *name* when this process exits or receives SIGTERM.""" + global _hooks_installed + with _lock: + _started.add(name) + if _hooks_installed: + return + _hooks_installed = True + atexit.register(_remove_all) + _chain_sigterm() + + +def forget(name: str) -> None: + """*name* was removed by its launcher; nothing is left to do for it at exit.""" + with _lock: + _started.discard(name) + + +def _remove_all() -> None: + with _lock: + names = sorted(_started) + _started.clear() + for name in names: + subprocess.run(["docker", "rm", "-f", "-v", name], capture_output=True, timeout=120) + + +def _chain_sigterm() -> None: + """Remove the containers on SIGTERM, then end the process as the signal would have. + Signal handlers can be installed only from the main thread; elsewhere the atexit hook and + Ryuk still apply.""" + if threading.current_thread() is not threading.main_thread(): + return + previous = signal.getsignal(signal.SIGTERM) + if previous is signal.SIG_IGN: + return + + def on_sigterm(signum: int, frame: FrameType | None) -> None: + _remove_all() + if callable(previous): + previous(signum, frame) + return + signal.signal(signal.SIGTERM, signal.SIG_DFL) + os.kill(os.getpid(), signal.SIGTERM) + + signal.signal(signal.SIGTERM, on_sigterm) diff --git a/server/typescript/packages/integration-tests/src/cube-container.ts b/server/typescript/packages/integration-tests/src/cube-container.ts index 7efb454dad..3bae94b503 100644 --- a/server/typescript/packages/integration-tests/src/cube-container.ts +++ b/server/typescript/packages/integration-tests/src/cube-container.ts @@ -28,6 +28,7 @@ import { spawn, spawnSync } from "node:child_process"; import { randomUUID } from "node:crypto"; import { reaperLabelArgs } from "./test-reaper.ts"; +import { MYSQL_8_DATA_DIR, POSTGRES_DURABILITY_OFF, postgresDataDir, ramVolume } from "./throwaway.ts"; export const CUBE_IMAGE = "cubejs/cube:v1.7.43"; export const CUBE_POSTGRES_IMAGE = "postgres:16-alpine"; @@ -131,8 +132,8 @@ export async function startCubeStack(modelDir: string, options: CubeStackOptions process.off("exit", stop); process.off("SIGINT", onSignal); process.off("SIGTERM", onSignal); - // Containers first: a network with an attached container cannot be removed. `-v` takes the - // postgres image's anonymous data volume with its container. + // Containers first: a network with an attached container cannot be removed. The database's + // data directory is a tmpfs; `-v` would also take any anonymous volume with its container. for (const name of [cubeContainer, pgContainer]) removeQuietly(["rm", "-f", "-v", name], `container '${name}'`, NO_SUCH_CONTAINER); removeQuietly(["network", "rm", network], `network '${network}'`, NO_SUCH_NETWORK); }; @@ -150,13 +151,14 @@ export async function startCubeStack(modelDir: string, options: CubeStackOptions const labels = await reaperLabelArgs(); docker(["network", "create", ...labels, network]); if (dataSource === "mysql") { - docker(["run", "-d", ...labels, "--name", pgContainer, "--network", network, + docker(["run", "-d", ...labels, "--name", pgContainer, "--network", network, ...ramVolume(MYSQL_8_DATA_DIR), "-e", `MYSQL_ROOT_PASSWORD=${PG_PASSWORD}`, "-e", `MYSQL_DATABASE=${MYSQL_DATABASE}`, CUBE_MYSQL_IMAGE]); await waitForMysql(pgContainer); for (const script of options.initSql ?? []) mysqlExec(pgContainer, script); } else { docker(["run", "-d", ...labels, "--name", pgContainer, "--network", network, - "-e", `POSTGRES_PASSWORD=${PG_PASSWORD}`, CUBE_POSTGRES_IMAGE]); + ...ramVolume(postgresDataDir(CUBE_POSTGRES_IMAGE)), + "-e", `POSTGRES_PASSWORD=${PG_PASSWORD}`, CUBE_POSTGRES_IMAGE, ...POSTGRES_DURABILITY_OFF]); await waitForPostgres(pgContainer); for (const script of options.initSql ?? []) psql(pgContainer, script); } diff --git a/server/typescript/packages/integration-tests/src/mysql-container.ts b/server/typescript/packages/integration-tests/src/mysql-container.ts index 6296f49420..aa5ef91353 100644 --- a/server/typescript/packages/integration-tests/src/mysql-container.ts +++ b/server/typescript/packages/integration-tests/src/mysql-container.ts @@ -18,6 +18,7 @@ import { execFile, spawnSync } from "node:child_process"; import { randomUUID } from "node:crypto"; import { reaperLabelArgs } from "./test-reaper.ts"; +import { MYSQL_8_DATA_DIR, ramVolume, removeOnExit } from "./throwaway.ts"; import mysql from "mysql2/promise"; const URL_ENV = "METAOBJECTS_TEST_MYSQL_URL"; @@ -39,12 +40,18 @@ export async function startMysql(): Promise { if (existing !== undefined && existing !== "") return { url: existing, stop: () => {} }; const name = `metaobjects-mysql-${randomUUID().slice(0, 8)}`; - // Teardown is one quick call after the tests, the same as postgres-container.ts. - const stop = () => { spawnSync("docker", ["rm", "-f", "-v", name], { stdio: "ignore", timeout: 60_000 }); }; + // Teardown is one quick call after the tests, the same as postgres-container.ts, and + // removeOnExit covers a run interrupted before it (throwaway.ts). + const forget = removeOnExit(name); + const stop = () => { + spawnSync("docker", ["rm", "-f", "-v", name], { stdio: "ignore", timeout: 60_000 }); + forget(); + }; // Docker assigns the host port at bind time; see the port note in postgres-container.ts. let url: string; try { - await docker(["run", "-d", ...(await reaperLabelArgs()), "--name", name, "-e", `MYSQL_ROOT_PASSWORD=${PASSWORD}`, + await docker(["run", "-d", ...(await reaperLabelArgs()), "--name", name, ...ramVolume(MYSQL_8_DATA_DIR), + "-e", `MYSQL_ROOT_PASSWORD=${PASSWORD}`, "-e", `MYSQL_DATABASE=${DATABASE}`, "-p", "127.0.0.1::3306", IMAGE, "--innodb-use-native-aio=0"]); url = `mysql://root:${PASSWORD}@127.0.0.1:${await publishedPort(name)}/${DATABASE}`; } catch (e) { diff --git a/server/typescript/packages/integration-tests/src/postgres-container.ts b/server/typescript/packages/integration-tests/src/postgres-container.ts index 0e502e6ab9..feaae74e40 100644 --- a/server/typescript/packages/integration-tests/src/postgres-container.ts +++ b/server/typescript/packages/integration-tests/src/postgres-container.ts @@ -26,6 +26,7 @@ import { spawnSync } from "node:child_process"; import { randomUUID } from "node:crypto"; import { Client } from "pg"; import { reaperLabelArgs } from "./test-reaper.ts"; +import { POSTGRES_DURABILITY_OFF, postgresDataDir, ramVolume, removeOnExit } from "./throwaway.ts"; /** Env var naming the shared CI Postgres sidecar (admin URL). Unset = per-container fallback. */ const SHARED_PG_URL_ENV = "METAOBJECTS_TEST_PG_URL"; @@ -53,16 +54,22 @@ export async function startPostgres(image = "postgres:16-alpine"): Promise { try { runDocker(["rm", "-f", "-v", name]); } catch { /* already gone */ } }; + const forget = removeOnExit(name); + const forceRemove = (): void => { + try { runDocker(["rm", "-f", "-v", name]); } catch { /* already gone */ } + forget(); + }; try { runDocker([ "run", "-d", ...(await reaperLabelArgs()), "--name", name, + ...ramVolume(postgresDataDir(image)), "-e", "POSTGRES_PASSWORD=test", "-p", "127.0.0.1::5432", - image, + image, ...POSTGRES_DURABILITY_OFF, ]); const connectionUri = `postgres://postgres:test@127.0.0.1:${publishedPort(name, 5432)}/postgres`; await waitForPgReady(name, connectionUri); diff --git a/server/typescript/packages/integration-tests/src/throwaway.ts b/server/typescript/packages/integration-tests/src/throwaway.ts new file mode 100644 index 0000000000..0827c999c2 --- /dev/null +++ b/server/typescript/packages/integration-tests/src/throwaway.ts @@ -0,0 +1,71 @@ +// A test database lives in RAM and never outlives the run that started it. +// +// Storage. Each database image declares a VOLUME for its data directory. A container started +// without a mount there gets an anonymous volume on disk, which stays behind after the container +// is removed without `-v`. Every launcher therefore mounts a size-capped tmpfs at that path +// (ramVolume), and Postgres runs with durability off (POSTGRES_DURABILITY_OFF): nothing written +// to a throwaway database needs to survive a crash. +// +// Removal. stop() and afterAll run only when the test gets that far. Ryuk (test-reaper.ts) +// removes the containers of a killed process from outside it, ten seconds after it dies. +// removeOnExit covers the exits Ryuk does not see at once: a normal exit with a stop() that never +// ran, and Ctrl-C or SIGTERM, after which the process exits as the signal would have. + +import { spawnSync } from "node:child_process"; + +/** Upper bound for one database's data directory. A tmpfs takes RAM only for what is written. */ +export const TMPFS_SIZE = "2g"; + +/** The data directory `mysql:8*` declares as its VOLUME (`docker image inspect`). */ +export const MYSQL_8_DATA_DIR = "/var/lib/mysql"; + +/** + * The data directory a `postgres` image declares as its VOLUME: `/var/lib/postgresql` from + * Postgres 18, `/var/lib/postgresql/data` before it (`docker image inspect`). + */ +export function postgresDataDir(image: string): string { + const major = /^(?:[^/]+\/)*postgres:(\d+)/.exec(image)?.[1]; + return major !== undefined && Number(major) >= 18 ? "/var/lib/postgresql" : "/var/lib/postgresql/data"; +} + +/** `postgres` server options for a database whose contents may be lost at any moment. */ +export const POSTGRES_DURABILITY_OFF: readonly string[] = [ + "-c", "fsync=off", + "-c", "synchronous_commit=off", + "-c", "full_page_writes=off", +]; + +/** `docker run` flags mounting a size-capped tmpfs at `path`, the image's VOLUME. */ +export function ramVolume(path: string): string[] { + return ["--tmpfs", `${path}:rw,size=${TMPFS_SIZE}`]; +} + +const started = new Set(); +let hooksInstalled = false; + +function removeAll(): void { + for (const name of started) { + spawnSync("docker", ["rm", "-f", "-v", name], { stdio: "ignore", timeout: 60_000 }); + } + started.clear(); +} + +function onSignal(signal: NodeJS.Signals): void { + removeAll(); + process.exit(signal === "SIGINT" ? 130 : 143); +} + +/** + * Remove container `name` when this process exits or is interrupted. Returns the function the + * launcher calls once it removed the container itself, so exit has nothing left to do for it. + */ +export function removeOnExit(name: string): () => void { + started.add(name); + if (!hooksInstalled) { + hooksInstalled = true; + process.on("exit", removeAll); + process.on("SIGINT", onSignal); + process.on("SIGTERM", onSignal); + } + return () => { started.delete(name); }; +} diff --git a/server/typescript/packages/metadata/package.json b/server/typescript/packages/metadata/package.json index 063db3475a..c5601a467c 100644 --- a/server/typescript/packages/metadata/package.json +++ b/server/typescript/packages/metadata/package.json @@ -46,7 +46,7 @@ "scripts": { "build": "tsc -p .", "typecheck": "tsc -p tsconfig.typecheck.json", - "conformance:mutation": "stryker run" + "conformance:mutation": "../../../../scripts/run-stryker.sh metadata" }, "license": "Apache-2.0", "author": "Doug Mealing ", diff --git a/server/typescript/packages/metadata/stryker.conf.mjs b/server/typescript/packages/metadata/stryker.conf.mjs index 6ba334aa10..af7c21c65c 100644 --- a/server/typescript/packages/metadata/stryker.conf.mjs +++ b/server/typescript/packages/metadata/stryker.conf.mjs @@ -1,5 +1,6 @@ import { fileURLToPath } from "node:url"; -import { dirname, resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; const here = dirname(fileURLToPath(import.meta.url)); // `here` is packages/metadata, so reach the repo-root corpus with FOUR `../` levels @@ -7,6 +8,13 @@ const here = dirname(fileURLToPath(import.meta.url)); // yields a non-existent server/fixtures/conformance (ENOENT under the mutation gate). const corpus = resolve(here, "../../../../fixtures/conformance"); +// The sandbox is a full copy of this package, rewritten for every mutant: keep it off the drives. +// `bun run conformance:mutation` (scripts/run-stryker.sh) picks a RAM-backed scratch directory and +// removes it afterwards; a bare `stryker run` falls back to the same place, keyed by its pid. +const sandbox = + process.env.METAOBJECTS_STRYKER_TEMP ?? + join(process.env.DEV_SCRATCH || tmpdir(), "stryker", `metadata-${process.pid}`); + /** @type {import("@stryker-mutator/api/core").PartialStrykerOptions} */ export default { testRunner: "command", @@ -26,6 +34,9 @@ export default { // CI lanes (java-slow, csharp, ...), starved the box into OOM. 6 leaves ~18 // threads for the sibling jobs while keeping the mutation gate reasonably fast. concurrency: 6, + tempDirName: sandbox, + // Remove the sandbox after a failing run too, not only a passing one. + cleanTempDir: "always", thresholds: { high: 60, low: 40, break: 35 }, reporters: ["clear-text", "html"], }; From b7b8580ffe1ebf779a6c053b0681fdf229e62dce Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 14:44:29 -0500 Subject: [PATCH 3/9] fix(codegen): keep a failed run's own error when the manifest write fails too A run that throws still writes the manifest for the files it wrote, but a failed write no longer replaces the run's error (TypeScript, Python, C#). Also: Python's gen summary counts skip-existing files as left as they were, not unchanged; a Postgres image tag that names no major takes the Postgres 18 data directory; CHANGELOG notes what a narrowed run now checks and what an owned per-object generator must do. --- CHANGELOG.md | 11 +++- .../OverwriteHashManifestTests.cs | 15 +++++ .../MetaObjects.Codegen/CodegenRunner.cs | 1 + .../MetaObjects.Codegen/HashManifest.cs | 55 ++++++++++++++----- server/python/src/metaobjects/cli.py | 8 +-- .../metaobjects/codegen/overwrite_policy.py | 12 +++- .../python/tests/codegen/test_gen_run_core.py | 16 +++++- .../codegen-ts/src/overwrite-policy.ts | 22 ++++++-- .../test/manifest-write-once.test.ts | 15 ++++- .../integration-tests/src/throwaway.ts | 12 ++-- 10 files changed, 131 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 39642e5575..97519ab837 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -305,12 +305,19 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi and every whole-model file it writes is byte-identical to a full run's. A generator sees the whole model in `ctx.entities` and the narrowing as `ctx.selected`; `perEntity` (`per_entity`) applies it. A Python target's declared `entities` still bounds that target's model. C# and - Java take no entity filter and were not affected. + Java take no entity filter and were not affected. Two consequences to expect: a narrowed run + checks the whole model as a full run does (a sourced object with no `dialect`, colliding + collection names), because the files it writes are rendered from it; and an owned generator + that writes one file per object by looping over `ctx.entities` itself, instead of through + `perEntity`, now sees every object, so it should set each file's `sourceNode` (the runner + then drops the unnamed objects' files) or apply `ctx.selected`. - **`gen` writes `.hashes.json` once per run, and not at all when nothing changed.** TypeScript, Python and C# re-read and re-wrote the whole manifest after every generated file, so a run of N files wrote it N times; they now hold it in memory for the run and write it once, as Java already did. Python's `metaobjects gen` also stops listing unchanged files as written: it - prints the files it wrote and counts the unchanged ones separately. + prints the files it wrote and counts the ones it left as they were separately. A run that + fails still records the files it wrote before failing, and a failed manifest write then no + longer hides the error that stopped the run. - **`meta gen` loads the metadata once.** The unknown-attribute warning re-loaded the project strictly on every run; it now reuses the generation load and parses strictly again only when the lenient load reported an unknown bare key. `meta gen` also no longer imports a database diff --git a/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs b/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs index 7918e3b07f..459e6e06d4 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/OverwriteHashManifestTests.cs @@ -153,6 +153,21 @@ public void ARun_WritesTheManifestOnce_AtTheEnd() Assert.Equal(recorded, File.ReadAllText(manifest)); } + [Fact] + public void AFailedRun_KeepsItsOwnError_WhenTheManifestWriteFailsToo() + { + void FailingRun() + { + using var batch = HashManifest.Batch(StateDir); + HashManifest.Record(StateDir, "A.cs", "class A { }\n"); + // A directory where the manifest goes: the end-of-batch write now fails. + Directory.CreateDirectory(Path.Combine(StateDir, ".hashes.json")); + throw new InvalidOperationException("the generator's own failure"); + } + var ex = Assert.Throws(FailingRun); + Assert.Equal("the generator's own failure", ex.Message); + } + [Fact] public void HandEditedFile_IsRefused_AndSurvives() { diff --git a/server/csharp/MetaObjects.Codegen/CodegenRunner.cs b/server/csharp/MetaObjects.Codegen/CodegenRunner.cs index 2e20cbc999..c55090c5f3 100644 --- a/server/csharp/MetaObjects.Codegen/CodegenRunner.cs +++ b/server/csharp/MetaObjects.Codegen/CodegenRunner.cs @@ -169,6 +169,7 @@ internal static RunResult Run( "does not apply to them beyond the marker rule."); ReportRefusals(refused, hadManifest, warnings); ReportAdoptions(adopted, warnings); + manifestBatch.Complete(); return new RunResult(results, warnings) { OverwritePaths = overwritePaths }; } diff --git a/server/csharp/MetaObjects.Codegen/HashManifest.cs b/server/csharp/MetaObjects.Codegen/HashManifest.cs index a259cbaaf0..bb584f18ec 100644 --- a/server/csharp/MetaObjects.Codegen/HashManifest.cs +++ b/server/csharp/MetaObjects.Codegen/HashManifest.cs @@ -72,41 +72,66 @@ private sealed class ManifestBatch(Dictionary raw) /// Hold the manifest in memory until the returned handle is disposed, then write it at /// most once. A nested batch for the same directory joins the outer one. The write /// happens even when the run throws: every file recorded so far IS on disk, and dropping - /// its record would make the next run refuse output it wrote itself. + /// its record would make the next run refuse output it wrote itself. Call + /// when the run succeeds: a scope disposed without it is + /// unwinding an exception, and a failed manifest write then must not replace that + /// exception, so it is swallowed. /// - public static IDisposable Batch(string? genStateDir) + public static BatchScope Batch(string? genStateDir) { - if (genStateDir is null) return NoBatch.Instance; + if (genStateDir is null) return BatchScope.None; var key = Path.GetFullPath(genStateDir); lock (Batches) { - if (Batches.ContainsKey(key)) return NoBatch.Instance; + if (Batches.ContainsKey(key)) return BatchScope.None; Batches[key] = new ManifestBatch(ReadFile(genStateDir)); } - return new BatchHandle(genStateDir, key); + return new BatchScope(genStateDir, key); } - private sealed class NoBatch : IDisposable + ///

The handle returns; disposing it writes the manifest. + public sealed class BatchScope : IDisposable { - public static readonly NoBatch Instance = new(); - public void Dispose() { } - } + internal static readonly BatchScope None = new(null, null); - private sealed class BatchHandle(string genStateDir, string key) : IDisposable - { + private readonly string? _genStateDir; + private readonly string? _key; + private bool _completed; private bool _disposed; + internal BatchScope(string? genStateDir, string? key) + { + _genStateDir = genStateDir; + _key = key; + } + + /// Mark the run as finished normally, so a failed write is reported. + public void Complete() => _completed = true; + public void Dispose() { - if (_disposed) return; + if (_disposed || _key is null || _genStateDir is null) return; _disposed = true; ManifestBatch batch; lock (Batches) { - batch = Batches[key]; - Batches.Remove(key); + batch = Batches[_key]; + Batches.Remove(_key); + } + if (!batch.Dirty) return; + if (_completed) + { + WriteFile(_genStateDir, batch.Raw); + return; + } + try + { + WriteFile(_genStateDir, batch.Raw); + } + catch (Exception) + { + // The run is already unwinding an exception; that one is the cause to report. } - if (batch.Dirty) WriteFile(genStateDir, batch.Raw); } } diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index d313e7519f..250b8ebdb7 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -1006,9 +1006,9 @@ def _run_suite( # and only the ones that own an exit code care about the refusals. if refused_out is not None: refused_out.extend(path for path, status in result.files if status == "refused") - # Emitted but not written: the file already held this output. Still in the return - # value (the orphan sweep and the cross-target guard need every emitted path), but a - # caller reporting what it WROTE must leave these out. + # Emitted but not written: the file already held this output, or skip-existing kept + # it. Still in the return value (the orphan sweep and the cross-target guard need every + # emitted path), but a caller reporting what it WROTE must leave these out. if unchanged_out is not None: unchanged_out.update( path for path, status in result.files if status in ("unchanged", "skipped") @@ -1856,7 +1856,7 @@ def _cmd_gen_config(args: argparse.Namespace) -> int: print(f"removed {path}") print( f"metaobjects gen: wrote {len(changed)} file(s) across {len(targets)} target(s)." - + (f" {len(unchanged)} file(s) unchanged." if unchanged else "") + + (f" {len(unchanged)} file(s) left as they were." if unchanged else "") + (f" Removed {len(removed)} file(s) no generator emits any more." if removed else "") ) if refused: diff --git a/server/python/src/metaobjects/codegen/overwrite_policy.py b/server/python/src/metaobjects/codegen/overwrite_policy.py index 5191a30192..451947571f 100644 --- a/server/python/src/metaobjects/codegen/overwrite_policy.py +++ b/server/python/src/metaobjects/codegen/overwrite_policy.py @@ -35,7 +35,7 @@ import json import os from collections.abc import Iterator -from contextlib import contextmanager +from contextlib import contextmanager, suppress from .constants import GENERATED_MARKER @@ -111,10 +111,16 @@ def manifest_batch(gen_state_dir: str | None) -> Iterator[None]: _batches[key] = batch try: yield - finally: + except BaseException: del _batches[key] if batch.dirty: - _write_manifest_file(gen_state_dir, batch.raw) + # The block's exception is the cause to report; a failed write must not replace it. + with suppress(Exception): + _write_manifest_file(gen_state_dir, batch.raw) + raise + del _batches[key] + if batch.dirty: + _write_manifest_file(gen_state_dir, batch.raw) def _read_raw(gen_state_dir: str) -> dict[str, object]: diff --git a/server/python/tests/codegen/test_gen_run_core.py b/server/python/tests/codegen/test_gen_run_core.py index 65655ede1b..5317a6a591 100644 --- a/server/python/tests/codegen/test_gen_run_core.py +++ b/server/python/tests/codegen/test_gen_run_core.py @@ -95,7 +95,7 @@ def test_an_unchanged_file_is_not_reported_as_written( printed = capsys.readouterr().out.splitlines() assert [line for line in printed if line.endswith(".py")] == [str(out / "Program.py")] assert "metaobjects gen: wrote 1 file(s)" in printed[-1] - assert "unchanged" in printed[-1] + assert "left as they were" in printed[-1] def test_entities_narrows_the_write_not_the_whole_model_files(tmp_path: Path) -> None: @@ -117,3 +117,17 @@ def test_entities_narrows_the_write_not_the_whole_model_files(tmp_path: Path) -> assert written for rel in written: assert (narrow / rel).read_bytes() == (full / rel).read_bytes(), str(rel) + + +def test_a_failed_run_keeps_its_own_error_when_the_manifest_write_fails_too( + tmp_path: Path, +) -> None: + gen_state = tmp_path / "gen-state" + with pytest.raises(RuntimeError, match="the generator's own failure"): + with overwrite_policy.manifest_batch(str(gen_state)): + overwrite_policy.decide_and_write( + str(tmp_path / "a.py"), "x = 1\n", gen_state_dir=str(gen_state), rel_path="a.py" + ) + # A directory where the manifest goes: the end-of-batch write now fails. + (gen_state / overwrite_policy.HASHES_FILE).mkdir(parents=True) + raise RuntimeError("the generator's own failure") diff --git a/server/typescript/packages/codegen-ts/src/overwrite-policy.ts b/server/typescript/packages/codegen-ts/src/overwrite-policy.ts index 548e23e9ba..6c667707e2 100644 --- a/server/typescript/packages/codegen-ts/src/overwrite-policy.ts +++ b/server/typescript/packages/codegen-ts/src/overwrite-policy.ts @@ -177,20 +177,30 @@ function batchFor(genStateDir: string): ManifestBatch | undefined { * Run `fn` with the manifest under `genStateDir` held in memory, and write it once when * `fn` returns or throws, only if something changed. A throw still writes what was * recorded before it: those files are on disk, and a manifest that forgot them would - * read them as unrecorded and refuse them next run. A nested call for the same + * read them as unrecorded and refuse them next run. When `fn` threw, a failed manifest + * write is swallowed so `fn`'s error stays the one reported. A nested call for the same * directory joins the outer batch. */ export function withManifestBatch(genStateDir: string, fn: () => T): T { const key = resolve(genStateDir); if (manifestBatches.has(key)) return fn(); - manifestBatches.set(key, { raw: readManifestFile(genStateDir), dirty: false }); + const batch: ManifestBatch = { raw: readManifestFile(genStateDir), dirty: false }; + manifestBatches.set(key, batch); + let result: T; try { - return fn(); - } finally { - const batch = manifestBatches.get(key)!; + result = fn(); + } catch (err) { manifestBatches.delete(key); - if (batch.dirty) writeManifestFile(genStateDir, batch.raw); + try { + if (batch.dirty) writeManifestFile(genStateDir, batch.raw); + } catch { + // `fn`'s error is the cause to report; a failed manifest write must not replace it. + } + throw err; } + manifestBatches.delete(key); + if (batch.dirty) writeManifestFile(genStateDir, batch.raw); + return result; } function readRawManifest(genStateDir: string): Record { diff --git a/server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts b/server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts index 356f408afd..f7953bb0fb 100644 --- a/server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts +++ b/server/typescript/packages/codegen-ts/test/manifest-write-once.test.ts @@ -3,12 +3,13 @@ // of a large project's run time, for the same bytes one write produces. import { describe, test, expect, beforeEach, afterEach, spyOn } from "bun:test"; import * as fs from "node:fs"; -import { mkdtempSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { mkdirSync, mkdtempSync, rmSync, readFileSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; import { MetaDataLoader } from "@metaobjectsdev/metadata"; import { FileSource } from "@metaobjectsdev/metadata/core"; import { runGen, defineConfig } from "../src/index.js"; +import { decideAndWrite, withManifestBatch } from "../src/overwrite-policy.js"; import { entityFile } from "../src/generators/entity-file.js"; import { queriesFile } from "../src/generators/queries-file.js"; import { barrel } from "../src/generators/barrel.js"; @@ -65,4 +66,16 @@ describe("the hash manifest is written once per run", () => { expect(await gen()).toBe(0); expect(readFileSync(manifestPath(), "utf-8")).toBe(before); }); + + test("a run that throws keeps its own error when the manifest write fails too", () => { + const genStateDir = join(project, ".gen-state"); + expect(() => + withManifestBatch(genStateDir, () => { + decideAndWrite(join(project, "a.ts"), "export {};\n", { genStateDir, outputRelPath: "a.ts" }); + // A directory where the manifest goes: the end-of-batch write now fails (EISDIR). + mkdirSync(join(genStateDir, MANIFEST), { recursive: true }); + throw new Error("the generator's own failure"); + }), + ).toThrow("the generator's own failure"); + }); }); diff --git a/server/typescript/packages/integration-tests/src/throwaway.ts b/server/typescript/packages/integration-tests/src/throwaway.ts index 0827c999c2..2c2aadd91a 100644 --- a/server/typescript/packages/integration-tests/src/throwaway.ts +++ b/server/typescript/packages/integration-tests/src/throwaway.ts @@ -20,12 +20,16 @@ export const TMPFS_SIZE = "2g"; export const MYSQL_8_DATA_DIR = "/var/lib/mysql"; /** - * The data directory a `postgres` image declares as its VOLUME: `/var/lib/postgresql` from - * Postgres 18, `/var/lib/postgresql/data` before it (`docker image inspect`). + * The data directory a Postgres image declares as its VOLUME: `/var/lib/postgresql` from + * Postgres 18, `/var/lib/postgresql/data` before it (`docker image inspect`). The major comes + * from a `postgres:` tag or a `pg` tag (pgvector). A tag that names no major + * (`latest`, none, a digest) is today's release, 18 or later, so it takes the newer path: + * guessing the older one there would leave the data on an anonymous on-disk volume. */ export function postgresDataDir(image: string): string { - const major = /^(?:[^/]+\/)*postgres:(\d+)/.exec(image)?.[1]; - return major !== undefined && Number(major) >= 18 ? "/var/lib/postgresql" : "/var/lib/postgresql/data"; + const tag = /:([^/:@]+)(?:@.*)?$/.exec(image)?.[1] ?? ""; + const major = /^(\d+)/.exec(tag)?.[1] ?? /(?:^|-)pg(\d+)/.exec(tag)?.[1]; + return major === undefined || Number(major) >= 18 ? "/var/lib/postgresql" : "/var/lib/postgresql/data"; } /** `postgres` server options for a database whose contents may be lost at any moment. */ From 788cfc0da11274e19c06707e5e15d73c2b84fbd8 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 14:59:16 -0500 Subject: [PATCH 4/9] refactor: name the Python write statuses, share the per-object selection, remove containers in one call --- server/python/src/metaobjects/cli.py | 4 +++- server/python/src/metaobjects/codegen/overwrite_policy.py | 8 ++++++-- server/python/tests/integration/throwaway.py | 4 ++-- server/typescript/packages/codegen-ts/src/generator.ts | 8 +++++++- .../packages/codegen-ts/src/generators/names-file.ts | 3 ++- .../packages/integration-tests/src/throwaway.ts | 4 ++-- 6 files changed, 22 insertions(+), 9 deletions(-) diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 250b8ebdb7..f86d8b662e 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -113,6 +113,8 @@ ) from metaobjects.codegen.config import DEFAULT_API_PREFIX, GenConfig from metaobjects.codegen.overwrite_policy import ( + STATUS_SKIPPED, + STATUS_UNCHANGED, has_hash_manifest, is_pristine_generated, manifest_batch, @@ -1011,7 +1013,7 @@ def _run_suite( # emitted path), but a caller reporting what it WROTE must leave these out. if unchanged_out is not None: unchanged_out.update( - path for path, status in result.files if status in ("unchanged", "skipped") + path for path, status in result.files if status in (STATUS_UNCHANGED, STATUS_SKIPPED) ) return [path for path, status in result.files if status != "refused"] diff --git a/server/python/src/metaobjects/codegen/overwrite_policy.py b/server/python/src/metaobjects/codegen/overwrite_policy.py index 451947571f..9cb403a520 100644 --- a/server/python/src/metaobjects/codegen/overwrite_policy.py +++ b/server/python/src/metaobjects/codegen/overwrite_policy.py @@ -248,6 +248,10 @@ def _write(path: str, content: str) -> None: #: The status of a hand-edited file replaced under ``handEdits: overwrite``. STATUS_OVERWROTE_EDIT = "overwrote-edit" +#: The file already held this output, so nothing was written. +STATUS_UNCHANGED = "unchanged" +#: skip-existing kept the file as it was. +STATUS_SKIPPED = "skipped" def decide_and_write( @@ -291,7 +295,7 @@ def decide_and_write( return "new" if strategy == "skip-existing": - return "skipped" + return STATUS_SKIPPED with open(path, encoding="utf-8") as fh: current = fh.read() @@ -310,7 +314,7 @@ def decide_and_write( # Nothing to do, but record it: a first run over already-correct output should # leave the file recognisable as ours next time. _record(gen_state_dir, key, content, legacy_rel_path) - return "unchanged" + return STATUS_UNCHANGED if is_pristine_generated(gen_state_dir, key, current, legacy_rel_path): _write(path, content) diff --git a/server/python/tests/integration/throwaway.py b/server/python/tests/integration/throwaway.py index c68040f595..4d786c0ab8 100644 --- a/server/python/tests/integration/throwaway.py +++ b/server/python/tests/integration/throwaway.py @@ -65,8 +65,8 @@ def _remove_all() -> None: with _lock: names = sorted(_started) _started.clear() - for name in names: - subprocess.run(["docker", "rm", "-f", "-v", name], capture_output=True, timeout=120) + if names: + subprocess.run(["docker", "rm", "-f", "-v", *names], capture_output=True, timeout=120) def _chain_sigterm() -> None: diff --git a/server/typescript/packages/codegen-ts/src/generator.ts b/server/typescript/packages/codegen-ts/src/generator.ts index d0c5be3c0a..c8f3a393d7 100644 --- a/server/typescript/packages/codegen-ts/src/generator.ts +++ b/server/typescript/packages/codegen-ts/src/generator.ts @@ -290,6 +290,12 @@ export type GeneratorFactory = TOpts extends void ? () => Generator : (opts?: TOpts) => Generator; +/** The objects this run generates for: `ctx.entities` narrowed by `ctx.matches` and the + * `meta gen ` selection. The one predicate every per-object emitter applies. */ +export function selectedEntities(ctx: GenContext): MetaObject[] { + return ctx.entities.filter((e) => ctx.matches(e) && (ctx.selected?.(e) ?? true)); +} + /** One-file-per-entity convenience. Async-safe. */ export function perEntity( fn: (entity: MetaObject, ctx: GenContext) => @@ -298,7 +304,7 @@ export function perEntity( | Promise, ): (ctx: GenContext) => Promise { return async (ctx) => { - const matched = ctx.entities.filter((e) => ctx.matches(e) && (ctx.selected?.(e) ?? true)); + const matched = selectedEntities(ctx); const results = await Promise.all(matched.map(async (e) => { const r = await fn(e, ctx); // Each file names the object it was rendered from, unless the generator said diff --git a/server/typescript/packages/codegen-ts/src/generators/names-file.ts b/server/typescript/packages/codegen-ts/src/generators/names-file.ts index 1d1baf9632..4829087d18 100644 --- a/server/typescript/packages/codegen-ts/src/generators/names-file.ts +++ b/server/typescript/packages/codegen-ts/src/generators/names-file.ts @@ -1,4 +1,5 @@ import type { Generator, GenContext, EmittedFile } from "../generator.js"; +import { selectedEntities } from "../generator.js"; import { entityOutputPath, crossEntitySpecifier } from "../import-path.js"; import { NAMES_FILE_SUFFIX } from "../constants.js"; import { renderNamesDecl } from "../templates/names-decl.js"; @@ -78,7 +79,7 @@ export function namesFile(opts?: NamesFileOpts): Generator { // make pass 2 skip the very object it exists to produce. const emitted = new Set(); // The objects `meta gen ` named (all of them on a full run): one file each. - const participants = ctx.entities.filter((e) => ctx.matches(e) && (ctx.selected?.(e) ?? true)); + const participants = selectedEntities(ctx); for (const entity of participants) { const content = renderNamesDecl(entity, { strategy, superSpecifier: superSpecifierFor(entity), diff --git a/server/typescript/packages/integration-tests/src/throwaway.ts b/server/typescript/packages/integration-tests/src/throwaway.ts index 2c2aadd91a..695efe3139 100644 --- a/server/typescript/packages/integration-tests/src/throwaway.ts +++ b/server/typescript/packages/integration-tests/src/throwaway.ts @@ -48,8 +48,8 @@ const started = new Set(); let hooksInstalled = false; function removeAll(): void { - for (const name of started) { - spawnSync("docker", ["rm", "-f", "-v", name], { stdio: "ignore", timeout: 60_000 }); + if (started.size > 0) { + spawnSync("docker", ["rm", "-f", "-v", ...started], { stdio: "ignore", timeout: 60_000 }); } started.clear(); } From 5215299e12513b3d1218145dcffdd4e308c291d6 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 15:07:01 -0500 Subject: [PATCH 5/9] no-mistakes(review): Behaviour-test gen import graph; scope Hono/grid warnings; restore config rewrite --- CHANGELOG.md | 9 +- .../cli/src/lib/load-metaobjects-config.ts | 9 +- .../cli/test/gen-import-graph.test.ts | 164 ++++++++++++------ .../codegen-ts-tanstack/src/data-grid-gate.ts | 4 +- .../src/generators/routes-file-hono.ts | 2 +- .../codegen-ts/src/reference/routes-hono.ts | 2 +- 6 files changed, 123 insertions(+), 67 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 97519ab837..55af589958 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -314,8 +314,9 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi - **`gen` writes `.hashes.json` once per run, and not at all when nothing changed.** TypeScript, Python and C# re-read and re-wrote the whole manifest after every generated file, so a run of N files wrote it N times; they now hold it in memory for the run and write it once, as Java - already did. Python's `metaobjects gen` also stops listing unchanged files as written: it - prints the files it wrote and counts the ones it left as they were separately. A run that + already did. Python's `metaobjects gen` output changes, which matters to a script that parses it: + it now lists only the files it wrote, and its summary line counts the files it left as they + were (unchanged or skipped) separately instead of reporting them as written. A run that fails still records the files it wrote before failing, and a failed manifest write then no longer hides the error that stopped the run. - **`meta gen` loads the metadata once.** The unknown-attribute warning re-loaded the project @@ -323,10 +324,6 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi the lenient load reported an unknown bare key. `meta gen` also no longer imports a database driver: the migration advice it prints reads `@metaobjectsdev/migrate-ts/offline`, a new subpath with the schema builder and diff and none of the introspection or apply code. -- **`meta gen` under Node stops filling the jiti cache.** Loading `metaobjects.config.ts` - rewrote it to a randomly named temp file, which missed jiti's cache on every run and left a - new cache entry behind each time. Under Node the config now loads from its own path, with the - package aliases applied by jiti. - **A projection view keeps the columns a reverse `@via` hop feeds.** An origin `@via` that walks an `identity.reference` backwards (`Project.timeEntries`, #560) loaded, but `meta migrate` left diff --git a/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts b/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts index c9b5946938..2c086258cf 100644 --- a/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts +++ b/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts @@ -469,15 +469,8 @@ export async function loadMetaobjectsConfig(projectRoot: string): Promise i.kind === "import-statement") - .map((i) => i.path); -} +// Behaviour, not source: a real `genCommand` run, in a child Bun process that refuses to +// load the database packages the migrate commands connect with (the oracle is that list of +// package names; `bun:sqlite` is a builtin Bun will not let a plugin override). The run must finish and write the entity. +// The controls prove the refusal is live: the same child fails when asked to import +// kysely, or the migrate-ts package root that used to drag it in. +import { describe, test, expect, beforeAll, afterAll } from "bun:test"; +import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; + +const WORKSPACE_TMP = resolve(import.meta.dirname, "fixtures", "__tmp__"); +const GEN = resolve(import.meta.dirname, "..", "src", "commands", "gen.ts"); + +const DATABASE_PACKAGES = ["kysely", "pg", "mysql2", "better-sqlite3", "@libsql/client", "@libsql/kysely-libsql"]; -describe("the meta gen import graph", () => { - test("reaches no database package statically", () => { - const parent = new Map(); - const seen = new Set([GEN]); - const queue = [GEN]; - const hits: string[] = []; - while (queue.length > 0) { - const file = queue.shift()!; - for (const spec of staticImports(file)) { - if (DATABASE_PACKAGES.some((re) => re.test(spec))) { - const chain = [spec, file]; - for (let p = parent.get(file); p !== undefined; p = parent.get(p)) chain.push(p); - hits.push(chain.reverse().join("\n -> ")); - continue; - } - if (spec.startsWith("node:")) continue; - let target: string; - try { - target = Bun.resolveSync(spec, dirname(file)); - } catch { - continue; - } - // Follow this repository's own sources only; a third-party package is a leaf. - if (target.includes("/node_modules/") || !/\.(ts|tsx|js|mjs)$/.test(target)) continue; - if (seen.has(target)) continue; - seen.add(target); - parent.set(target, file); - queue.push(target); - } +const PLUGIN = ` +Bun.plugin({ + name: "refuse-database-packages", + setup(build) { + for (const name of ${JSON.stringify(DATABASE_PACKAGES)}) { + build.module(name, () => { + throw new Error("DATABASE_PACKAGE_LOADED: " + name); + }); } - expect(seen.size).toBeGreaterThan(50); - expect(hits).toEqual([]); + }, +}); +`; + +const RUNNER = ` +const [mode, root] = process.argv.slice(2); +if (mode === "gen") { + const { genCommand } = await import(${JSON.stringify(GEN)}); + process.exit(await genCommand([], root)); +} +await import(mode); +`; + +let dir: string; +let project: string; + +beforeAll(() => { + mkdirSync(WORKSPACE_TMP, { recursive: true }); + dir = mkdtempSync(join(WORKSPACE_TMP, "gen-import-graph-")); + writeFileSync(join(dir, "plugin.ts"), PLUGIN); + writeFileSync(join(dir, "runner.ts"), RUNNER); + + project = join(dir, "project"); + mkdirSync(join(project, "metaobjects"), { recursive: true }); + mkdirSync(join(project, ".metaobjects"), { recursive: true }); + writeFileSync( + join(project, "metaobjects", "meta.app.json"), + JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Talk", + children: [ + { "source.rdb": { "@table": "talks" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "title", "@required": true } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, + }), + ); + writeFileSync(join(project, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, sources: [] })); + writeFileSync( + join(project, "metaobjects.config.ts"), + ` +import { defineConfig } from "@metaobjectsdev/codegen-ts"; +export default defineConfig({ + outDir: ${JSON.stringify(join(project, "generated"))}, + dialect: "sqlite", + dbImport: "~/db", + extStyle: "none", + generators: ["entity"], +}); +`, + ); +}); + +afterAll(() => { + rmSync(dir, { recursive: true, force: true }); +}); + +function child(mode: string) { + const proc = Bun.spawnSync(["bun", "--preload", join(dir, "plugin.ts"), join(dir, "runner.ts"), mode, project], { + cwd: project, + stdout: "pipe", + stderr: "pipe", + }); + return { code: proc.exitCode, output: `${proc.stdout}${proc.stderr}` }; +} + +describe("meta gen loads no database package", () => { + test("a gen run finishes with every database package refused", () => { + const result = child("gen"); + expect(result.output).not.toContain("DATABASE_PACKAGE_LOADED"); + expect(result.code).toBe(0); + expect(existsSync(join(project, "generated"))).toBe(true); + }); + + test("control: the refusal rejects kysely", () => { + const result = child("kysely"); + expect(result.code).not.toBe(0); + expect(result.output).toContain("DATABASE_PACKAGE_LOADED: kysely"); + }); + + test("control: the refusal rejects the migrate-ts package root", () => { + const result = child("@metaobjectsdev/migrate-ts"); + expect(result.code).not.toBe(0); + expect(result.output).toContain("DATABASE_PACKAGE_LOADED"); }); }); diff --git a/server/typescript/packages/codegen-ts-tanstack/src/data-grid-gate.ts b/server/typescript/packages/codegen-ts-tanstack/src/data-grid-gate.ts index dfe6c6a53c..4abefb9b6d 100644 --- a/server/typescript/packages/codegen-ts-tanstack/src/data-grid-gate.ts +++ b/server/typescript/packages/codegen-ts-tanstack/src/data-grid-gate.ts @@ -41,9 +41,9 @@ export function warnMissingDataGridLayout( passesOtherGates: (entity: MetaObject) => boolean, artifact: string, ): void { - const candidates = ctx.entities.filter(passesOtherGates); + const candidates = ctx.entities.filter((e) => (ctx.selected?.(e) ?? true) && passesOtherGates(e)); if (candidates.length === 0) return; - if (candidates.some(hasDataGridLayout)) return; // the model uses grids — say nothing + if (ctx.entities.some((e) => passesOtherGates(e) && hasDataGridLayout(e))) return; // the model uses grids — say nothing ctx.warn( `no ${artifact} emitted for ${candidates.map((e) => e.name).join(", ")} — grid ` + `artifacts are opt-in per entity via a \`layout.dataGrid\` child, and no object in ` + diff --git a/server/typescript/packages/codegen-ts/src/generators/routes-file-hono.ts b/server/typescript/packages/codegen-ts/src/generators/routes-file-hono.ts index 55cab5976a..d752742f32 100644 --- a/server/typescript/packages/codegen-ts/src/generators/routes-file-hono.ts +++ b/server/typescript/packages/codegen-ts/src/generators/routes-file-hono.ts @@ -90,7 +90,7 @@ export const routesFileHono = function routesFileHono(opts?: RoutesFileHonoOpts) generate: async (ctx) => { // One note per run naming every TPH subtype held back, so the gap is visible at // `meta gen` time rather than discovered as missing endpoints in production. - const skipped = ctx.entities.filter((e) => passesOtherGates(e) && isTphSubtype(e)); + const skipped = ctx.entities.filter((e) => (ctx.selected?.(e) ?? true) && passesOtherGates(e) && isTphSubtype(e)); if (skipped.length > 0) { ctx.warn( `no Hono routes emitted for the TPH subtype(s) ${skipped.map((e) => e.name).join(", ")} — ` + diff --git a/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts b/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts index 712bb82eeb..cede0f74a2 100644 --- a/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts +++ b/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts @@ -322,7 +322,7 @@ export const routesFileHono = function routesFileHono(opts?: RoutesFileHonoOpts) generate: async (ctx) => { // One note per run naming every TPH subtype held back, so the gap is visible at // `meta gen` time rather than discovered as missing endpoints in production. - const skipped = ctx.entities.filter((e) => passesOtherGates(e) && isTphSubtype(e)); + const skipped = ctx.entities.filter((e) => (ctx.selected?.(e) ?? true) && passesOtherGates(e) && isTphSubtype(e)); if (skipped.length > 0) { const why = "the Hono adapter has no discriminator scoping yet, so per-subtype CRUD would " + From d6cc721e21623dc5daf7f037366d7f17261169a5 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 15:11:05 -0500 Subject: [PATCH 6/9] no-mistakes(review): Narrow jsonSchemaFile to named entities on gen runs --- .../src/generators/contract-schema-file.ts | 4 ++-- .../codegen-ts/src/reference/contract-schema.ts | 2 +- .../test/contract/contract-behaviours.test.ts | 15 +++++++++++++++ 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/server/typescript/packages/codegen-ts/src/generators/contract-schema-file.ts b/server/typescript/packages/codegen-ts/src/generators/contract-schema-file.ts index a426669f05..e4c2ff9d0d 100644 --- a/server/typescript/packages/codegen-ts/src/generators/contract-schema-file.ts +++ b/server/typescript/packages/codegen-ts/src/generators/contract-schema-file.ts @@ -8,7 +8,7 @@ import { type MetaObject, type MetaRoot } from "@metaobjectsdev/metadata"; import { effectivePackage } from "../docs-paths.js"; -import type { EmittedFile, GenContext, Generator } from "../generator.js"; +import { type EmittedFile, type GenContext, type Generator, selectedEntities } from "../generator.js"; import { packageToPath } from "../import-path.js"; import type { ContractShape, JsonSchema } from "../contract/field-schema.js"; import { collectContractEntries, contractObjects } from "../contract/object-schema.js"; @@ -71,7 +71,7 @@ function strictFiles(objects: readonly MetaObject[], root: MetaRoot): EmittedFil function generateSchemaFiles(ctx: GenContext, options: JsonSchemaFileOptions): EmittedFile[] { const root = ctx.loadedRoot; - const objects = contractObjects(ctx.entities.filter(ctx.matches), root); + const objects = contractObjects(selectedEntities(ctx), root); return options.profile === "strict" ? strictFiles(objects, root) : canonicalFiles(objects, root); } diff --git a/server/typescript/packages/codegen-ts/src/reference/contract-schema.ts b/server/typescript/packages/codegen-ts/src/reference/contract-schema.ts index 3d2d9ecd96..9e92d5e652 100644 --- a/server/typescript/packages/codegen-ts/src/reference/contract-schema.ts +++ b/server/typescript/packages/codegen-ts/src/reference/contract-schema.ts @@ -96,7 +96,7 @@ function strictFiles(objects: readonly MetaObject[], root: MetaRoot): EmittedFil function generateSchemaFiles(ctx: GenContext, options: JsonSchemaFileOptions): EmittedFile[] { const root = ctx.loadedRoot; - const objects = contractObjects(ctx.entities.filter(ctx.matches), root); + const objects = contractObjects(ctx.entities.filter((o) => ctx.matches(o) && (ctx.selected?.(o) ?? true)), root); return options.profile === "strict" ? strictFiles(objects, root) : canonicalFiles(objects, root); } diff --git a/server/typescript/packages/codegen-ts/test/contract/contract-behaviours.test.ts b/server/typescript/packages/codegen-ts/test/contract/contract-behaviours.test.ts index 3fd6e9be60..f715b76907 100644 --- a/server/typescript/packages/codegen-ts/test/contract/contract-behaviours.test.ts +++ b/server/typescript/packages/codegen-ts/test/contract/contract-behaviours.test.ts @@ -303,6 +303,21 @@ describe("what a project's run changes", () => { expect(narrowed.get("openapi.json")).toBe(whole.get("openapi.json")!); }); + for (const [label, factory] of [["jsonSchemaFile", jsonSchemaFile], ["the reference jsonSchemaFile", refJsonSchemaFile]] as const) { + test(`${label} writes only the named entity's schema on a narrowed run`, async () => { + const c = corpusCase("m2m"); + const whole = await emit(c, [factory()]); + const named = (await loadCaseModel(c)).objects()[0]!.name; + const narrowed = await emit(c, [factory()], { entityFilter: [named] }); + const ownFile = [...narrowed.keys()].filter((p) => p.endsWith(`/${named}.schema.json`)); + expect(ownFile.length).toBe(1); + expect(narrowed.size).toBeLessThan(whole.size); + for (const [path, content] of narrowed) expect(whole.get(path), path).toBe(content); + const others = [...whole.keys()].filter((p) => p.endsWith(".schema.json") && !p.endsWith(`/${named}.schema.json`) && !narrowed.has(p)); + expect(others.length).toBeGreaterThan(0); + }); + } + test("a generator filter narrows the document, and keeps what a kept object references", async () => { const c = corpusCase("all-fields"); const onlyAllFields = (o: MetaObject): boolean => o.name === "AllFields"; From 30fe815797f917102c328d3f93fc0deb444bdd30 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 19:33:22 -0500 Subject: [PATCH 7/9] no-mistakes(document): narrowed-run and test-storage docs --- CONTRIBUTING.md | 8 ++++++++ docs/features/cli.md | 3 ++- docs/features/codegen-concepts.md | 9 +++++++++ docs/recipes/typed-client-for-the-reference-routes.md | 3 ++- server/typescript/packages/migrate-ts/README.md | 3 +++ 5 files changed, 24 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 709cb8c33e..5bfc80dcb3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -78,6 +78,14 @@ cd server/python && pytest # Python (in its .venv) Cross-language persistence / api-contract corpora (Docker + Testcontainers) run via `scripts/integration-test.sh`. +Throwaway test databases stay off the drives. The Docker-backed launchers (Postgres, MySQL, the +Cube stack and the `scripts/ci-local.sh` Postgres sidecar) keep each database's data directory on +a size-capped tmpfs with Postgres durability off, and remove the container when the run ends. +The Stryker mutation gate (`bun run conformance:mutation` in +`server/typescript/packages/metadata`, via `scripts/run-stryker.sh`) builds its sandbox under +`$DEV_SCRATCH` when that is set, else the system temp directory, and removes it on exit. Point +`DEV_SCRATCH` at a RAM-backed directory to keep that scratch off disk. + ### Local CI (the checks GitHub runs — run it before opening a PR) GitHub Actions is enabled on this repository. The check workflows in diff --git a/docs/features/cli.md b/docs/features/cli.md index 9d53081a2a..c5075a59c3 100644 --- a/docs/features/cli.md +++ b/docs/features/cli.md @@ -468,7 +468,8 @@ targets: - **`--config `** picks the config file explicitly on either command; without it both look for `./metaobjects.config.yaml` in the current directory. **No config is an error**: `gen` and `verify --codegen` print `no metaobjects.config.yaml`, exit 2 and write nothing. -- **Run options still apply to the list:** `--entities` (narrows each target's entities), +- **Run options still apply to the list:** `--entities` (narrows the entities each target writes; a + target's own `entities` still bounds its model), `--provider` (added to `providers`), `--templates` (`--prompts` on `verify`), `--baseline`. Column naming is not a run option: it is `naming` in `.metaobjects/config.json`, which every port reads, and `--column-naming` is refused diff --git a/docs/features/codegen-concepts.md b/docs/features/codegen-concepts.md index 73be43de51..5c98ca4fde 100644 --- a/docs/features/codegen-concepts.md +++ b/docs/features/codegen-concepts.md @@ -227,6 +227,15 @@ custom iteration: is kept as a soft-deprecated alias — "run" is ambiguous under multi-target output; `perModel` names the data scope.) +**`meta gen ` narrows what is written, never what a whole-model file sees.** `ctx.entities` +is the whole model in scope (after the project's `scope`), even when the run names some entities. +`ctx.selected` is the predicate for the named ones; it is absent when every object is selected. +`perEntity` applies it. A file rendered from the whole model (a barrel, an index, a shared schema) +reads `ctx.entities`, so it is the same on a narrowed run as on a full one. A generator that loops +over `ctx.entities` itself must apply `ctx.selected` (TypeScript: `ctx.selected?.(o) ?? true`; +Python: `ctx.is_selected(o)`), or set `sourceNode` on each per-object file, which lets the runner +drop the files of objects the run did not name. + A generator picks its scope by which helper it composes — the same way it picks object vs model today. Reference templates and the guidance call out the scope each starting point is for, so "I need a per-package service layer" maps to a clear diff --git a/docs/recipes/typed-client-for-the-reference-routes.md b/docs/recipes/typed-client-for-the-reference-routes.md index 844e84596b..3d2a973681 100644 --- a/docs/recipes/typed-client-for-the-reference-routes.md +++ b/docs/recipes/typed-client-for-the-reference-routes.md @@ -162,7 +162,8 @@ export function fetchClientGenerator(): Generator { const layout = ctx.config.outputLayout ?? "flat"; const prefix = ctx.renderContext?.apiPrefix ?? ""; // the same apiPrefix routesFile mounts under const files: EmittedFile[] = [{ path: "fetch-client.runtime.ts", content: RUNTIME }]; - for (const obj of ctx.entities.filter(ctx.matches)) { + // ctx.entities is the whole model; ctx.selected narrows a `meta gen ` run to the named ones. + for (const obj of ctx.entities.filter((o) => ctx.matches(o) && (ctx.selected?.(o) ?? true))) { const n = obj.name, path = servedPath(obj, prefix); // Types come from the entity module `entity` wrote next to this file (flat layout). files.push({ diff --git a/server/typescript/packages/migrate-ts/README.md b/server/typescript/packages/migrate-ts/README.md index c239de7b48..3759457452 100644 --- a/server/typescript/packages/migrate-ts/README.md +++ b/server/typescript/packages/migrate-ts/README.md @@ -58,6 +58,9 @@ await writeMigration(sql, { dir: ".metaobjects/migrations", slug: "add-customer- - Rename detection via heuristic (Levenshtein on column names, column-set overlap on tables) + `onAmbiguous` callback — library doesn't prompt; CLI in SP5 wires the prompt. - Per-change-kind allow flags for destructive opt-in. +- `@metaobjectsdev/migrate-ts/offline` is the part that never opens a database: the expected + schema, the diff, the snapshot store and the advice helpers. Import from it where a driver must + not load (`meta gen` does). The package root re-exports it beside the parts that connect. ## Dialects From 8b88d61dd204013d7c1ae66baf1c7628a4b85e9c Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 19:36:43 -0500 Subject: [PATCH 8/9] no-mistakes(document): Document drive-wear test changes and Disk I/O rules --- AGENTS.md | 24 ++++++++++++++++++++++++ CHANGELOG.md | 10 ++++++++++ 2 files changed, 34 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 7cbf2847cc..5d9f43db66 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -499,3 +499,27 @@ inverse of metadata-is-the-spine. *OMDB engine debt* — FR-003 Plan 4 closed th anti-patterns; the Spring Boot 3 starter and OMDB autoconfiguration shipped. *WARN envelope-shape on `expected-warnings.json`* — every runner asserts it; the legacy string-list path is retired. + +## Disk I/O: keep throwaway data off the drives + +Test databases, sandboxes and temp files must not wear out the developer's drives. + +- **Throwaway data goes in RAM.** Put scratch files, mutation-testing sandboxes (Stryker +`tempDirName`) and test-only temp directories under `$DEV_SCRATCH` (fall back to the system temp +directory when it is unset). Delete what you create when the run ends. +- **Test databases run on a capped tmpfs at the image's `VOLUME` path** (`postgres:18*`: +`/var/lib/postgresql`; `postgres:16/17` and `pgvector:*-pg17`: `/var/lib/postgresql/data`; +`mysql:8*`: `/var/lib/mysql`; check others with `docker image inspect -f '{{json .Config.Volumes}}'`), +`size=2g`, with Postgres durability off: `-c fsync=off -c synchronous_commit=off -c full_page_writes=off`. +- **Every container you start, you remove.** Use `docker run --rm`, Testcontainers with Ryuk enabled, +or explicit teardown in a `finally`/`trap`. Never leave a hand-started database running after your +task. A database you mean to keep gets the label `dev.keep=true`. +- **Generators and tools never rewrite a file whose bytes did not change**, write any manifest once per +run (not once per file), and verify drift in memory rather than by rendering everything to a temp +directory. +- **Run the narrowest tests that prove the change** while iterating (one module, the affected test +classes); run the full suite once, before the PR. +- **No background processes outlive the task.** Kill test watchers, dev servers and sandboxes you +started before you finish. + +Project specifics: test databases start in the TypeScript `integration-tests` package (`src/postgres-container.ts`, `src/mysql-container.ts`, `src/cube-container.ts`, sharing `src/throwaway.ts`), the Python `tests/integration/` launchers (`postgres_container.py`, `mysql_container.py`, `throwaway.py`), the C# `MetaObjects.IntegrationTests/Runner/PostgresContainer.cs`, the Java and Kotlin Testcontainers `PostgresContainer` classes under `server/java/integration-tests*`, and the services and sidecars in `.github/workflows/integration-tests.yml` and `local-ci.yml`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 55af589958..b857c4dbfe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -288,6 +288,16 @@ loaded before stops loading. This vocabulary ships in 1.3.0 with `metamodelVersi `Cannot find module './dprint-node.linux-x64-gnu.node'` on every command that loads the codegen engine, `meta migrate` and `meta verify --db` included. Those schema commands, the binary's documented scope, now run; `meta gen` is still not supported inside the binary. +- **Test databases stay off the drives.** Every Docker-backed database launcher in every port + (the TypeScript `integration-tests` Postgres, MySQL and Cube containers; the Python MySQL and + Postgres launchers; the C# MySQL and Postgres runners; the Java and Kotlin Testcontainers + classes; the services and sidecars in the integration-test and local-CI workflows and + `scripts/ci-local.sh`) mounts a 2 GiB tmpfs at the image's `VOLUME` path, and Postgres runs + with `fsync`, `synchronous_commit` and `full_page_writes` off. Each container is removed when + its run ends: by `--rm`, by Ryuk when a killed process never reaches its own cleanup, and by an + exit or SIGTERM handler that also removes anonymous volumes. The Stryker mutation gate for the + metadata package builds its sandbox under `$DEV_SCRATCH` (the system temp directory when unset) + and stops its test runners with it. Contributor steps are in [CONTRIBUTING.md](CONTRIBUTING.md). ### Deprecated From 96f520764917ee5dcc4a59510489cd7eddfe7b8e Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 14 Nov 2026 19:37:28 -0500 Subject: [PATCH 9/9] no-mistakes(document): Export selectedEntities for owned generators --- server/typescript/packages/codegen-ts/src/index.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server/typescript/packages/codegen-ts/src/index.ts b/server/typescript/packages/codegen-ts/src/index.ts index 1acacdf4e6..5c837892ef 100644 --- a/server/typescript/packages/codegen-ts/src/index.ts +++ b/server/typescript/packages/codegen-ts/src/index.ts @@ -7,7 +7,7 @@ export { runGen, declaredGeneratorNames, stableGeneratorName, undeclaredGenerato export type { RunGenOpts, RunGenResult } from "./runner.js"; export type { Generator, GenContext, EmittedFile, GeneratorFactory, RunOutputFile } from "./generator.js"; -export { perEntity, perPackage, perModel, oncePerRun } from "./generator.js"; +export { perEntity, perPackage, perModel, oncePerRun, selectedEntities } from "./generator.js"; // The one aggregation of the UI-tier markers, Hono routes and the routes' `expose` option: // the run's own answers to "does this run emit a UI tier?" and "which verbs are mounted?". export { runEmitsUiTier, runEmitsHonoRoutes, runRoutesExpose, UI_TIER_GENERATOR_NAMES } from "./ui-tier-gate.js";