Gate 2: establish heap-free aggregate layouts

README.md

@@ -5,12 +5,15 @@
 > TypeRB backend, runtime, or release target. Everything in this repository may
 > change incompatibly or be removed without notice.
 
-TypeRB Native explores whether a TypeRB-specific native compiler and runtime
-can improve end-to-end build time, generated-program performance, and deployed
+TypeRB Native develops a TypeRB-specific native compiler and runtime intended
+to improve end-to-end build time, generated-program performance, and deployed
 binary size relative to an optimized release executable produced by the
 reference compiler's Go backend. Its long-term objective is a self-hosted
 compiler whose repository-owned implementation is written in TypeRB and whose
 ordinary release/bootstrap path does not require Go or another host language.
+The repository remains experimental while that implementation is incomplete;
+the gates are engineering checkpoints that keep correctness and whole-toolchain
+performance visible as the implementation grows.
 
 The [TypeRB repository](https://github.com/type-rb/type-rb) remains the source
 of truth for the language specification, reference compiler, supported
@@ -27,8 +30,9 @@ semantics; it does not define a native-only TypeRB dialect.
 - Compare multiple machine-code strategies behind the same MIR and semantics.
 - Measure complete toolchains, including code generation, linking, runtime,
   sidecars, and distribution size.
-- Keep the experiment removable if it does not improve the practical tradeoff
-  offered by the Go backend.
+- Preserve a credible path to a native implementation that is at least as
+  practical as the Go backend, and use measured regressions to direct
+  optimization work rather than treating early gates as disposable demos.
 
 ## Current status
 
@@ -46,11 +50,13 @@ binary64 Float, static UTF-8 output, and deterministic arithmetic failure.
 On the recorded Apple M2 Pro run, native warm build time improved by 30.5% to
 36.3%, stripped executable size improved by 96.85%, and the worst runtime result
 was a 16.0% regression, within the pre-registered 25% bound. See the
-[Gate 1 result](results/2026-08-28-gate1-qbe-darwin-arm64/README.md). Development
-stops before Gate 2 for maintainer review. The result does not select QBE for
-production or measure the final self-hosted compiler. The current path provides
-no production runtime, stable ABI, stable artifact format, or compatibility
-guarantee. Records and tagged values remain deferred to Gate 2.
+[Gate 1 result](results/2026-08-28-gate1-qbe-darwin-arm64/README.md). Gate 2 is
+active. Its first checkpoint adds heap-free, static-layout records and tagged
+values, including the representation needed for payload enums and `Result`,
+before dynamic strings, arrays, closures, or a memory manager are introduced.
+The Gate 1 result does not select QBE for production or measure the final
+self-hosted compiler. The current path provides no production runtime, stable
+ABI, stable artifact format, or compatibility guarantee.
 
 ## Intended boundary
 
@@ -114,8 +120,8 @@ gate. More than one implementation may remain only when distinct development,
 release, or target use cases show a durable benefit that justifies the
 maintenance cost.
 
-See the [experiment plan](docs/experiment-plan.md) for correctness gates,
-measurement rules, and abandonment criteria.
+See the [development and validation plan](docs/experiment-plan.md) for
+correctness gates, measurement rules, and backend selection criteria.
 
 ## Non-goals
 
@@ -124,8 +130,8 @@ The initial gates do not attempt to:
 - port the compiler to Rust, Zig, or another host implementation language;
 - replace external code generators, assemblers, linkers, SDKs, or system
   libraries merely to claim self-hosting;
-- implement the full TypeRB frontend before native execution feasibility has
-  passed its earlier gates;
+- implement the full TypeRB frontend before the shared native value model and
+  runtime boundaries are concrete enough to support it;
 - commit TypeRB to a supported native mode;
 - expose mutable compiler internals or backend hooks as a package API;
 - support the full standard library, Web, ORM, Jobs, or native package
@@ -142,12 +148,14 @@ repository.
 ## Documentation
 
 - [Architecture](docs/architecture.md)
-- [Experiment plan](docs/experiment-plan.md)
+- [Development and validation plan](docs/experiment-plan.md)
 - [Gate 1 QBE vertical slice](docs/gate-1-qbe.md)
 - [Gate 1 QBE Darwin arm64 result](results/2026-08-28-gate1-qbe-darwin-arm64/README.md)
+- [Gate 2 heap-free aggregate value model](docs/gate-2-aggregates.md)
 - [Decision 0001: Experimental native toolchain boundary](docs/decisions/0001-experimental-native-toolchain.md)
 - [Decision 0002: TypeRB-owned self-hosting](docs/decisions/0002-typerb-owned-self-hosting.md)
 - [Decision 0003: Gate 1 QBE and Darwin arm64 profile](docs/decisions/0003-gate-1-qbe-target.md)
+- [Decision 0004: Sustained native implementation and staged Gate 2](docs/decisions/0004-sustained-native-development.md)
 - [Contributing](CONTRIBUTING.md)
 - [Security](SECURITY.md)
 

docs/architecture.md

@@ -2,11 +2,12 @@
 
 ## Purpose
 
-TypeRB Native investigates a TypeRB-specific native compiler and runtime while
-keeping the supported language and reference compiler independent. The initial
-question is whether a native pipeline can improve the practical tradeoff among
-build time, execution performance, and deployed binary size after all required
-tooling is counted.
+TypeRB Native develops a TypeRB-specific native compiler and runtime while
+keeping the supported language and reference compiler independent. Its
+engineering objective is a self-hosted implementation that removes Go from the
+ordinary bootstrap and application-build path while matching or improving the
+practical tradeoff among build time, execution performance, and deployed binary
+size after all required tooling is counted.
 
 The experiment is not a port to a different host language. Native execution
 and self-hosting are separate checkpoints, but both belong to the intended
@@ -184,8 +185,9 @@ full-language target would require accepted solutions for:
 
 The initial runtime remains deliberately smaller: static data, scalar values,
 simple aggregate layout, observable output, and deterministic process failure.
-Heap management and concurrency begin only after the first backend comparison
-passes its gates.
+Gate 2 completes the heap-free aggregate layer before heap ownership and memory
+management are added. This separation keeps record and tagged-value semantics
+independent of the later allocation strategy.
 
 Runtime semantics are shared across backend candidates. Target-specific ABI
 profiles and small shims may differ, but the runtime must not be independently
@@ -217,6 +219,8 @@ runtime and package boundaries, reproducible builds, primary-platform support,
 an end-to-end advantage after the complete toolchain is counted, and a
 reproducible self-hosted compiler build whose ordinary path does not use Go.
 
-If the experiment is abandoned, the native implementation and any bootstrap
-bridge should remain removable. Reusable benchmark and conformance findings may
-be retained without preserving a dormant compatibility surface.
+The bootstrap bridge remains removable because the independent frontend will
+eventually replace it, not because removal is the default project outcome.
+Gates expose correctness, performance, and maintenance problems early enough to
+improve the shared MIR, runtime, backend, or build pipeline before those choices
+become public contracts.

docs/decisions/0004-sustained-native-development.md

@@ -0,0 +1,65 @@
+# 0004: Sustained Native Development and a Staged Gate 2
+
+## Status
+
+Accepted.
+
+## Context
+
+The initial plan described the repository primarily as a sequence of bounded
+experiments whose native path could be abandoned after an early performance
+miss. Gate 1 instead established a useful executable baseline: the QBE path was
+correct on its scalar corpus, substantially reduced build time and executable
+size, and kept runtime within the registered bound.
+
+The project's intent is now stronger. It should make a sustained engineering
+effort toward a better native implementation, ultimately self-hosted in TypeRB,
+rather than treating each incomplete stage as a reason to stop. Measurements
+remain essential, but their main purpose is to locate work and guide backend or
+runtime choices.
+
+Gate 2 previously grouped static aggregates with dynamic strings, arrays,
+closures, allocation, and failure behavior. Those features answer different
+architectural questions. Implementing them together would obscure whether a
+problem belongs to value layout or memory ownership.
+
+## Decision
+
+TypeRB Native is a sustained implementation effort with experimental stability.
+Gates are correctness, architecture, and whole-toolchain performance
+checkpoints. Missing a target keeps the gate open for diagnosis and improvement
+unless the evidence reveals a fundamental conflict with TypeRB semantics,
+safety, or sustainable self-hosting.
+
+Gate 2 is restricted to heap-free aggregate values:
+
+- nominal, immutable records;
+- payloadless and payload-bearing enum variants;
+- the static tagged representation needed for `Result`;
+- aggregate construction, projection, dispatch, direct calls, block parameters,
+  and returns; and
+- monomorphized layouts composed only from Gate 1 scalars and other Gate 2
+  aggregates.
+
+Dynamic strings, collections, closures, escaping values, allocation, and memory
+management move to Gate 3. This is a staging decision only; those capabilities
+remain required for the eventual native compiler and runtime.
+
+The bootstrap snapshot, Native MIR, `darwin-arm64-v0` ABI profile, and QBE
+adapter remain internal and disposable. Their instability lets the
+implementation improve without creating a native-only TypeRB dialect or a
+public compiler API.
+
+## Consequences
+
+- Gate 2 can establish deterministic layout and tagged-value semantics without
+  prematurely selecting ownership or garbage collection.
+- Records, payload enums, and `Result` use the existing TypeRB language contract;
+  this decision adds no syntax or user-visible semantic variation.
+- Performance misses produce optimization work or a backend decision rather
+  than automatic project abandonment.
+- The Go reference compiler remains an early bootstrap producer and semantic
+  oracle until the later TypeRB-authored frontend and reproducible self-hosting
+  gates replace it in the ordinary path.
+- Experimental adapters can still be removed when they no longer have a
+  distinct measured role.

docs/experiment-plan.md

@@ -1,17 +1,18 @@
-# Experiment Plan
+# Development and Validation Plan
 
-## Research question
+## Engineering objective
 
-Can a TypeRB-specific native AOT pipeline improve at least one of these primary
-outcomes without unacceptable regressions in the others?
+Build a TypeRB-specific native AOT pipeline that ultimately matches or improves
+the optimized Go backend across these primary outcomes:
 
 1. End-to-end application build time.
 2. Generated-program execution time.
 3. Deployed executable size.
 
-If it can, can the compiler and runtime be implemented in TypeRB, reproduce
-themselves, and retain competitive build time and generated-code behavior once
-the complete self-hosted toolchain is measured?
+The compiler and runtime are implemented in TypeRB, reproduce themselves, and
+must retain competitive build time and generated-code behavior once the
+complete self-hosted toolchain is measured. Early gates establish this outcome
+incrementally; they are not a sequence of throwaway demonstrations.
 
 Secondary outcomes include compiler and runtime peak memory, startup latency,
 toolchain distribution size, portability, diagnostics, correctness risk, and
@@ -30,19 +31,22 @@ intentionally unstripped, cold, or otherwise disadvantaged Go configuration.
 - Unsupported behavior fails explicitly.
 - Measurements include serialization, lowering, optimization, code generation,
   assembly, linking, runtime, and required external components.
-- Thresholds and removal rules are recorded before reviewing a result.
+- Quality and performance targets are recorded before reviewing a result.
 - Microbenchmarks diagnose a phase; representative programs determine
   viability.
 
 Before a gate begins, its issue must record metric-specific non-inferiority
-bounds, a minimum meaningful primary-metric improvement, catastrophic-regression
-limits, and a time or engineering-effort budget. These values cannot be revised
-after results are reviewed merely to keep a candidate alive.
+bounds, a minimum meaningful primary-metric improvement where the gate is
+expected to provide one, and catastrophic-regression limits. A miss identifies
+required engineering work or an architectural decision; it does not by itself
+end the native implementation. Targets cannot be weakened after results are
+reviewed merely to label a gate complete.
 
 ## Candidate sequence
 
-The candidates are not implemented to production completeness in parallel.
-They advance through small shared gates and can be removed early.
+Backend candidates are not implemented to production completeness in parallel.
+They advance through small shared gates, and only implementations with a clear
+role continue to accumulate maintenance cost.
 
 1. Use hand-authored bootstrap and MIR fixtures to validate the boundary.
 2. Use QBE for the cheapest runtime and ABI feasibility check.
@@ -110,24 +114,42 @@ stronger applicable optimized Go baseline. A regression greater than 2x stops
 the gate for review. TinyGo is measured only if the unchanged corpus works and
 the calibration costs no more than half a working day; it is not a deliverable.
 
-### Gate 2: Portable value model
+### Gate 2: Heap-free aggregate value model
 
-Scope may expand to:
-
-- records and payload enums;
-- static-layout records and tagged values;
-- `Result` representation and propagation;
-- arrays and dynamic strings;
-- closures and captured environments; and
-- deterministic observable behavior, defined allocation failure, and
-  reproducible artifacts.
+Scope:
 
-Only candidates that pass Gate 1 correctness and comparability continue.
+- nominal records with immutable, statically laid-out fields;
+- payloadless and payload-bearing enum variants represented as tagged values;
+- aggregate construction, field and payload projection, direct calls, returns,
+  block parameters, and exhaustive variant dispatch;
+- monomorphized static layouts needed for records and `Result<T, E>` whose
+  fields and payloads are themselves heap-free Gate 2 values;
+- deterministic snapshots, MIR, QBE output, executables, diagnostics, and
+  layout computation; and
+- the existing disposable `darwin-arm64-v0` profile and QBE 1.3 path.
+
+Dynamic strings, arrays, hashes, closures, captured environments, escaping
+values, heap allocation, and a memory manager remain outside Gate 2. A static
+string literal may still be used only for the existing observable-output
+operation; it is not yet a first-class aggregate field or payload.
+
+Exit condition: the pinned reference compiler and native path produce identical
+observable results for the registered source corpus covering records, nested
+records, payload enums, exhaustive `case`, explicit `Result` handling, and
+`try` propagation. Invalid snapshot and MIR inputs fail deterministically,
+layout boundary tests pass, and repeated builds reproduce the same snapshot,
+MIR, QBE IL, and executable. On the registered aggregate workloads, stripped
+native executable size remains at least 30% below the stronger applicable Go
+baseline, while warm end-to-end build time and runtime each remain within 25%
+and no primary metric regresses by more than 2x. A target miss keeps Gate 2 open
+for diagnosis and improvement.
 
 ### Gate 3: Runtime viability
 
 Scope may expand to:
 
+- arrays, hashes, and dynamic strings;
+- closures and captured environments;
 - memory-management strategy and cycles;
 - classes, interfaces, unions, and nullable values;
 - source-mapped failures and unwind behavior;
@@ -236,15 +258,16 @@ Every published result should include:
 Store results under a date- and experiment-specific directory only after the
 first executable benchmark exists. Do not commit placeholder result files.
 
-## Selection policy
+## Backend selection policy
 
-A candidate remains only when it:
+A backend implementation remains active only when it:
 
 - passes the current correctness and reproducibility gates;
 - satisfies the pre-registered non-inferiority and catastrophic-regression
   limits;
 - achieves the pre-registered minimum improvement in at least one primary
-  outcome before product feasibility;
+  outcome before product feasibility, or has a concrete diagnostic role in
+  reaching that outcome;
 - has a credible path for the next required target and runtime feature;
 - does not impose disproportionate distribution, security, or maintenance
   costs.
@@ -258,25 +281,15 @@ A secondary improvement may justify a bounded diagnostic experiment, but it
 does not pass product feasibility when all three primary outcomes miss their
 registered gates.
 
-## Abandonment policy
-
-Archive or remove the native path if time-boxed milestones show that:
-
-- no candidate improves the practical tradeoff over an optimized release
-  executable produced by the reference compiler's Go backend;
-- a candidate exceeds its registered time or engineering-effort budget without
-  passing the current gate;
-- gains disappear after linking, runtime, sidecars, and distribution are
-  counted;
-- correctness requires a competing dialect or weaker semantics;
-- the boundary repeatedly duplicates or leaks the reference frontend;
-- source mapping, runtime safety, or package interoperability requires
-  backend-specific language APIs;
-- a second primary target requires divergent language semantics, a separate
-  frontend or runtime, or disproportionate target-specific maintenance; or
-- maintenance and security costs outweigh the demonstrated benefit.
-
-If abandoned, retain generally useful benchmark methodology, conformance tests,
-and architectural findings. Remove experimental bootstrap surfaces that have no
-remaining consumer rather than preserving compatibility for a failed
-experiment.
+## Reassessment policy
+
+A missed checkpoint triggers diagnosis of the MIR, runtime, backend, or build
+pipeline and a recorded plan to close the gap. Backend adapters may be replaced
+or removed when another implementation serves their role better. The native
+implementation itself is reconsidered only when evidence exposes a fundamental
+conflict with portable TypeRB semantics, safe implementation, or sustainable
+self-hosting—not merely because an early implementation needs optimization.
+
+Temporary bootstrap surfaces still have no compatibility guarantee. Remove
+them when the independent frontend replaces them, and retain generally useful
+benchmark methodology, conformance tests, and architectural findings.