ci: cut Linux test lanes from ~50 to ~11 minutes - #735
Conversation
roborev: Combined Review (
|
|
looking |
69aef08 to
3e097ae
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
Every push to main since #717 failed in test-postgres and test-pgvector. Small packages died in under a second with "sorry, too many clients already", the pgvector schema migration hit "out of shared memory", and the heavy packages then ran into the 60-minute timeout. Each PostgreSQL-backed test binary opens its own connections: two for the admin handle, four for a warm-pool schema refill, plus the store under test. `go test ./...` starts up to one binary per CPU, and nothing budgets connections across binaries. On the managed runner the twenty-four packages that use PostgreSQL started together and exceeded the service container's 100-connection limit in the first seconds. GitHub-hosted runners have four CPUs, so the same lanes passed there with -p effectively at 4. The Makefile's PostgreSQL targets and the inline pgvector lane now pass -p 4 (PG_TEST_PARALLEL), so the connection footprint no longer depends on the host's CPU count. The pgvector service also gets the max_locks_per_transaction=256 setting the test-postgres service already had; its migration ran out of exactly that. Slower per-package times on the managed runner are a separate matter: the cap reduces contention on the one container but does not change the runner's I/O. Generated with Claude Code (claude-fable-5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every test fixture used to replay InitSchema() into an empty database: a few hundred DDL statements, ~140ms on SQLite and ~1.5s on PostgreSQL per fixture on a laptop, and several times that on a contended CI runner. With about a thousand fixtures across the suite, that replay was most of the wall clock of the largest packages — internal/store spent 39 minutes on the managed runner for tests that do milliseconds of real work each. The initialized schema is a pure function of the test binary, so each binary now builds it once and hands every fixture a copy. On SQLite the template is a file written into the test's temp dir. On PostgreSQL it is a template database and each fixture is a CREATE DATABASE ... TEMPLATE clone, a file-level copy the server makes in tens of milliseconds. Locally internal/store drops from 241s to 120s on SQLite, and a PostgreSQL fixture from ~750ms to ~200ms. A template database has to be reclaimed when the binary that built it is gone. Ownership is a session advisory lock held on a connection pinned for the binary's life; the server releases it on any exit, so the next binary reclaims whatever it can lock. That replaces the warm pool's pid-namespace sweep, which could not run on hosts without /proc and so left macOS on the slow path. A role without CREATEDB falls back to the per-schema fixture. One thing a clone must not copy: InitSchema() mints a durable archive UID, and tests that hand one archive's cursor to another depend on fixtures being distinct archives. The fixture assigns each clone its own UID. Generated with Claude Code (claude-fable-5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cmd/msgvault/cmd, internal/store and internal/api hold 1,300–2,000 tests each, and a Go test binary runs them one at a time. Whichever of the three is slowest sets the wall clock of the whole Linux lane no matter how many cores the runner has — 39 minutes for internal/store on the managed runner before the template fixtures, and still the critical path after them. The Windows lane already solves this with scripts/test-package-shards.ps1: compile the package once, list its tests, deal them into shards, and run each shard as its own process of the same binary. This adds the Linux twin and uses it for those three packages in their own matrix jobs, on both the SQLite lane and the PostgreSQL lane, where each sharded job also gets its own service container so four shards' connections never compete with the rest of the suite. The existing lanes run every other package through the new *-unsharded targets; `make test` and `make test-pg-shipped` still run everything, so a local run is unchanged. Locally the three packages take 225s, 120s and 125s whole; in four shards each they take 173s in total. Generated with Claude Code (claude-fable-5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The new PostgreSQL shard matrix must follow the hosted-runner policy from #737. Running those service-container jobs on the managed fleet would retain the same slow Docker storage path that caused the original timeouts. Keep hosted Go caching enabled so each isolated shard job avoids a cold module and build cache. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
Pull request checks use the workflow from main until this change merges. That workflow calls the full Make targets, so internal/store still ran as one test binary and hit its 60-minute PostgreSQL timeout even though the new branch workflow passed the same package in shards. Compose the full targets from the unsharded remainder and the existing shard runner. Keep those parts sequential for local PostgreSQL servers so no more than the configured shard count competes for one database. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
0956d26 to
8fc9c31
Compare
roborev: Combined Review (
|
The shard script listed a package's tests through a process substitution, which throws away the exit status of the test binary. A package whose init or TestMain fails produced an empty list, and the script reported "No tests found" and exited 0. CI would have passed a package that never ran. The same line used mapfile, which the macOS system Bash 3.2 does not have. The macOS lane runs `make test`, which now reaches this script, and it only runs on pushes to main, so the first push after merge would have failed there without a pull-request run to warn about it. Listing now writes to a file, checks the binary's exit status, and reads the names back with a plain read loop. Generated with Claude Code Co-authored-by: Claude <noreply@anthropic.com>
PostgreSQL advisory locks are local to the database a session is connected to. The template sweep took its locks in the configured database but listed candidate templates from the whole server. Two runs on one server that were configured against different databases could not see each other's locks, so one run's sweep could force-drop the other's live template and clones. Template and clone names now carry a hash of the configured database, and a sweep only judges names in its own scope, where the lock it tries is the same lock a live owner holds. The template tests now skip when fixtures are on the per-schema path, such as a role without CREATEDB, instead of failing on template-only contracts. Generated with Claude Code Co-authored-by: Claude <noreply@anthropic.com>
wesm
changed the title
roborev: Combined Review (
|
The managed Linux runners are about ten times slower than hosted Ubuntu for the Go test suite. On main the test job has hit go test's one-hour timeout in cmd/msgvault/cmd and internal/store in three of the last six runs, and on this branch the unsharded packages alone took 33 minutes there, followed by 20 minutes for the cmd/msgvault/cmd shards, during which a test that allows five seconds for a command to finish timed out. On hosted Ubuntu the same shards finish in under three minutes. The test and test-sharded jobs now run on hosted Ubuntu with Go caching, as #737 already did for the PostgreSQL lanes. The other Linux jobs stay on the managed runners. Generated with Claude Code Co-authored-by: Claude <noreply@anthropic.com>
roborev: Combined Review (
|
2 participants
The Linux test lanes now finish in about 11 minutes. On main they take 22 to 55 minutes for SQLite and 40 to 46 minutes for PostgreSQL, even on hosted runners, because the three largest packages (
cmd/msgvault/cmd,internal/store,internal/api) each built a fresh database from about 330 DDL statements for every one of their 1,300 to 2,000 tests, and a test binary runs its tests one at a time.internal/storealone set the wall clock of the whole lane.Three changes stack:
CREATE DATABASE ... TEMPLATE, which replaces the warm pool and its/proc-based cleanup. Every fixture is still a private, unused database produced by the sameInitSchema()path, and each clone gets its own archive UID. A role withoutCREATEDBfalls back to the per-schema fixture;MSGVAULT_TEST_PG_TEMPLATE=0forces that path.testjob has hit Go's one-hour test timeout in three of its last six runs there, and this branch's unsharded packages alone took 33 minutes on one. Thetestandtest-shardedjobs now use hosted Ubuntu with Go caching, as ci: keep PostgreSQL lanes on hosted runners #737 already did for the PostgreSQL lanes. The other Linux jobs stay on the managed runners.scripts/test-package-shards.shcompiles the package once, lists its tests, and runs them as four processes of the same binary, the way the Windows lane already does. Each PostgreSQL shard job gets its own service container. Thetestandtest-postgreslanes cover every other package.A PostgreSQL template is owned through a session advisory lock that the server releases when the binary exits, so the next binary reclaims what a crashed one left behind. Advisory locks are local to one database, so template names carry a hash of the configured database and a sweep only judges templates in its own scope.
The PostgreSQL targets also cap concurrent test binaries at four (
PG_TEST_PARALLEL). Every PostgreSQL-backed binary opens its own connections, and a wide runner starting them all at once exceeded a stock server's connection and lock limits. The CI service containers get a larger lock table.Pull-request checks run
ci.ymlfrom main, so the new shard jobs only appear in a dispatch run of this branch: https://github.com/kenn-io/msgvault/actions/runs/33679438735.Job times on hosted runners, first main run after #737 versus that dispatch run:
Main's SQLite figure is from the managed runner, which this PR stops using for that job.
generated by a clanker