Skip to content

Commit 492decb

Browse files
authored
fix: run full local CI lanes in parallel and speed up Python codegen (#450)
* ci: run the full local CI as parallel lanes and fix the slow steps at the root The flagless scripts/ci-local.sh took about 85 minutes, every lane one after another. Measured on the same 24-core machine it now takes 10m17s, green, with the same tests: every suite reports the same counts, less the duplicate below. - Parallel lanes. The tree-writing steps (install, drift gates, one workspace build) run first; the rest run as independent lanes, at most MO_CI_JOBS (8) at once, each into its own log printed as one block when it ends. The summary lists every step's time, the slowest steps and each lane's wall time. --quick and --only stay serial and select exactly what they did. - Each test runs once. tests/integration ran inside "conformance: python" and again as "integration-tests (python)"; the java and kotlin conformance classes ran inside the reactor and again on their own; the workspace was rebuilt and reinstalled by several lanes; mvn install ran three times. - Python codegen ran ruff as `python -m ruff`, a second interpreter per call, twice per emitted file: ~90% of codegen time. It now runs ruff's binary (find_ruff_bin), same version, same output. Python conformance 16m49s -> 1m20s alone (with xdist). - The shared Java test logback config was never wired, so every module but metadata logged at DEBUG (1.86M lines a run). Surefire now points every module at it. - Runners in parallel: pytest-xdist, bun test --parallel for the TS integration suite, surefire forkCount=3 for the JVM integration modules. TS integration 15m52s -> 2m49s, Java 6m57s -> 1m55s, Kotlin 4m52s -> 2m33s, Python integration 5m51s -> 56s, each measured alone. - The Postgres sidecar is named per checkout; each scenario still gets its own database. * no-mistakes(review): apply reviewer inline fixes * no-mistakes(review): Keep python integration tests in conformance when docker down * no-mistakes(test): Bind servers and containers to OS-assigned ports * no-mistakes(document): Fix stale cube lane ordering in cube-export docs
1 parent 899b774 commit 492decb

37 files changed

Lines changed: 1036 additions & 881 deletions

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,16 @@ here.**
1010

1111
## [Unreleased]
1212

13+
### Changed
14+
15+
- **Python: `metaobjects gen` and `verify --codegen` run ruff as its own binary.** Every emitted
16+
file is import-sorted and formatted by ruff twice, and each call used to go through
17+
`python -m ruff`, which starts a second interpreter only to exec the binary the ruff package
18+
installed. That start was about 90% of codegen time (59 ms against 6 ms a call, measured). The
19+
formatter now resolves that binary once with ruff's own `find_ruff_bin`, so the ruff version
20+
and the generated output are unchanged; it falls back to `python -m ruff` if the binary cannot
21+
be found.
22+
1323
## [1.1.0] — 2026-10-22
1424

1525
_npm `1.1.0` (all 14 `@metaobjectsdev/*` packages in lockstep), PyPI `1.1.0`, NuGet `1.1.0` and

‎CONTRIBUTING.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,14 @@ scripts/ci-local.sh --quick # hygiene.yml in full + the TypeScript half of
9191
MO_CI_LIST_ONLY=1 scripts/ci-local.sh --quick # print the steps, run nothing
9292
```
9393

94+
The full run does its tree-writing steps first (install, the drift gates, one workspace
95+
build), then runs the rest as parallel lanes: at most `MO_CI_JOBS` at once, default 8. Each
96+
lane's output prints as one block when the lane finishes, so lanes never interleave. The
97+
summary gives every step's time, the slowest steps and each lane's wall time. `--quick` and
98+
`--only` run serially. `MO_CI_PY_WORKERS` (default 8), `MO_CI_TS_WORKERS` (default 6) and
99+
`MO_CI_JVM_FORKS` (default 3) set the worker counts for pytest-xdist, the TypeScript
100+
integration suite and the JVM integration modules' surefire forks.
101+
94102
It SKIPS (loudly, not silently) any port whose toolchain (bun/dotnet/uv/mvn) isn't
95103
installed. The `.githooks/pre-push` hook additionally runs the TS build+typecheck
96104
gate and a Java pom-version drift guard on every push (activate hooks once per

‎docs/CONFORMANCE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ regenerate with `ls -d fixtures/<corpus>/*/ | wc -l` for directory-shaped corpor
3434
| [`fixtures/extract-conformance/`](../fixtures/extract-conformance/) | 48 | ✓ | ✓ | inherits the shared JVM engine | ✓ | ✓ |
3535
| [`fixtures/output-prompt-conformance/`](../fixtures/output-prompt-conformance/) | 17 | ✓ | ✓ | ✓ | ✓ | ✓ |
3636
| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 59 (53 query + 6 migration) | all 59; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 53 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) |
37-
| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 106 (31 core + 10 tph + 9 m2m + 4 jsonb + 2 write-through + 12 projection + 27 report + 11 decimal) | ✓ (Fastify reference + generated lane); `report/` and `decimal/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/` and `decimal/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/` and `decimal/` also on SQLite and MySQL | ✓ (HttpListener + Npgsql); `report/` and `decimal/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/` and `decimal/` also on SQLite and MySQL |
37+
| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 106 (31 core + 10 tph + 9 m2m + 4 jsonb + 2 write-through + 12 projection + 27 report + 11 decimal) | ✓ (Fastify reference + generated lane); `report/` and `decimal/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/` and `decimal/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/` and `decimal/` also on SQLite and MySQL | ✓ (Kestrel + Npgsql); `report/` and `decimal/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/` and `decimal/` also on SQLite and MySQL |
3838
| [`fixtures/decimal-wire/`](../fixtures/decimal-wire/) | 1 vector file (format, parse, filter operand, 15-digit read) | ✓ (`runtime-ts` and the generated helper) | ✓ (`CanonicalDecimal`) | ✓ (the generated wire file) | ✓ (`DecimalWire`) | ✓ (`decimal_wire`) |
3939
| [`fixtures/validation-conformance/`](../fixtures/validation-conformance/) | 42 cases | ✓ (generated Zod + run-time `runValidators`) | ✓ | ✓ | ✓ | ✓ (generated Pydantic + run-time `run_validators`) |
4040
| [`fixtures/registry-conformance/`](../fixtures/registry-conformance/) | 1 canonical manifest | ✓ (reference emitter) | ✓ | ✓ | ✓ | ✓ |

‎docs/features/cube-export.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -648,8 +648,8 @@ both together), the TPH subtype's `sql`, a one-to-one join from both ends, the i
648648
`CASE`, and a two-hop `@spine` report, whose rows, the spine rows with no facts included, must
649649
equal the lowered view's. Each query's rows are compared with plain SQL on the same database.
650650

651-
Run it with `scripts/ci-local.sh --only cube`. The full `scripts/ci-local.sh` runs it after the
652-
integration suite, and `--quick` and `--no-integration` drop it. The first run pulls the Cube
651+
Run it with `scripts/ci-local.sh --only cube`. The full `scripts/ci-local.sh` runs it as its own
652+
parallel lane, and `--quick` and `--no-integration` drop it. The first run pulls the Cube
653653
image, which is about 1 GB. With Docker down, the lane records a SKIP behind a banner, never a
654654
pass; under `--strict-toolchains` it is a failure. In CI it is a `cube` entry in
655655
`integration-tests.yml`, which runs on release tags and on manual dispatch, not on every push.

‎fixtures/api-contract-conformance/jsonb/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ booted over HTTP) are wired and green for:
132132
converter refuses polymorphic bodies → HTTP 415), so the Kotlin generated lane
133133
wires a small Jackson `JsonElement` codec module (the consumer's serialization
134134
wiring).
135-
- **C#** — reference (hand-rolled `HttpListener` over Postgres testcontainer) +
135+
- **C#** — reference (hand-rolled Kestrel server over Postgres testcontainer) +
136136
generated (the real `MetaObjects.Codegen` `DocumentRoutes` + `AppDbContext`
137137
Roslyn-compiled and booted full-stack on Kestrel over Postgres testcontainer).
138138
The generated lane locks the #105 codegen: the `payload` property is typed

0 commit comments

Comments
 (0)