Repository navigation
Graduate the native runner from experimental #77
Description
Make Cratis.Specifications.Runner a supported, non-experimental package so Cratis repositories can move off xUnit where it is faster. #65 evaluated the runner; this tracks what remains before graduating it.
Remaining
- IDE support (decided: no manual IDE verification): document the supported IDEs from Microsoft's and JetBrains' Microsoft.Testing.Platform documentation (Visual Studio 2022 17.14+, Visual Studio 2026, Rider with Testing Platform discovery enabled, VS Code C# Dev Kit), and rely on the runner's conformance to the platform's discovery and execution contract, covered by specs.
- Full AOT benchmark: representative suites published with NativeAOT, compared with JIT and the other packages (Benchmarks: measure xUnit v3 on Microsoft.Testing.Platform and the runner's AOT route #73).
- Lifecycle consistency: identical semantics to the other packages for setup/act/teardown failures, cancellation and Stop, shared contract specs across packages.
- Fundamentals evidence: the runner variants in the Fundamentals trial (Move specifications to Cratis.Specifications 5.0 on xUnit v3, run as executables Fundamentals#1208) pass with identical test identities and outcomes and win on performance.
- Documentation: supported IDE versions, migration from xUnit, limits (constructed generic base types use the reflection fallback; not AOT-safe).
- Remove the experimental marker and announce in the release.
Related: #65, #73, #78 (runner-native API).
- Readable output and reports: Readable specification output and reports in the native runner #79 (the main user-facing reason to choose the runner).
Status
State: Done — Closed by release v5.1.0.
Branch: studio-980-translation-connector (#93, #95, #96)
Scope: Graduate the native runner: lifecycle consistency with the other packages, docs (IDEs, xUnit migration, limits); AOT benchmark and Fundamentals evidence after.
Delivered: Nothing yet.
Remaining: To be confirmed.
Follow-ups: None.
Activity
woksin commentedon Oct 25, 2026
Decision on lifecycle consistency (teardown failures): the native runner follows xUnit, xUnit v3, NUnit 4 and NUnit 5. A failure in
Destroyor disposal is reported as its own failed result for the specification class and fails the run, and the facts that already completed keep their outcomes. Today the runner rewrites every non-skipped fact to failed; that changes. TUnit differs because its engine attaches after-class hook failures to the last test of the class; that is engine-owned and is documented as a known difference rather than changed. Setup and act failures already match: facts fail,Becauseand fact bodies do not run, and teardown still runs once.woksin commentedon Oct 25, 2026
Benchmark results for the graduation decision (run 40174768971 on main 1fd416b; GitHub-hosted ubuntu-24.04; 500 classes x 5 facts = 2,500 facts; 5 measured repetitions; default parallelism; Cratis suites). https://github.com/Cratis/Specifications/actions/runs/40174768971
dotnet testuse, by about 35% on wall time and 37% on CPU.Graduation still needs evidence from a real repository: the runner variants of a Fundamentals spec project with identical test identities and outcomes. The experimental marker stays until then.
woksin commentedon Oct 25, 2026
Second review of the teardown decision (independent GPT-6.1 Sol review against the code): agreed. xUnit (
Source/XUnit/SpecificationTestClassRunner.cs:68-75), xUnit v3 (Source/XUnit.v3/SpecificationTestClassRunner.cs:45-51) and NUnit/NUnit5 (Source/NUnit/Specification.cs:63-73, shared by NUnit5) all report a cleanup failure at the class or suite boundary and keep the outcomes of completed facts, and the runner now does the same (Source/Runner/Execution/SpecificationClassRunner.cs:76-88). Precision: the boundary matches, but the result shape differs by framework. xUnit emits cleanup messages, NUnit fails the suite, and the runner adds one test node. TUnit 1.76.0 attaches cleanup errors to the completing test (TestExecutor.cs:697-724). Deciding without a human was appropriate because the runner is experimental.woksin commentedon Oct 25, 2026
Same-host Fundamentals trial completed: https://github.com/Cratis/Fundamentals/actions/runs/40183739733 (baseline cadf8e452b, candidates Specifications 5.0.3). Independent review rechecked all 36 TRXs and all 30 raw timing files: 793 Fundamentals and 94 OpenTelemetry identities/outcomes match after theory-row normalization, all passed. Five interleaved repetitions on one host: median wall V0/V4/V6 = 4.13/1.84/1.38 s for Fundamentals and 2.50/1.39/1.13 s for OpenTelemetry. V0 is current mtp-off/VSTest, V4 xUnit v3 MTP, V6 native runner JIT. The runner is 25%/19% faster than xUnit MTP on these workloads, not universally.
Graduation remains blocked: generated/reflection lifecycle cancellation needs alignment and verification; the lifecycle matrix must exercise each adapter; actual MTP Stop needs platform-level coverage. Fundamentals NativeAOT execution also remains blocked by Fundamentals#1217. The experimental marker stays.
github-actions commentedon Oct 25, 2026
Closed by release v5.1.0.