fix: test_go_race times out on internal/app in CI (Go's 10m per-package limit) - #346
Merged
danielhanold merged 20 commits intoSep 28, 2026
Conversation
… corpus never runs real git) Docket-Plan-Path: docs/superpowers/plans/2026-09-28-test-go-race-times-out-on-internal-app-in-ci-go-s-10m-per-pa.md
… tag (TestIntegrationGateCancel, change 0465)
…n tag (TestIntegrationGateVerdict, change 0465)
…e integration tag (TestIntegrationGateFence, change 0465)
…hind the integration tag (TestIntegrationGateCompletion, change 0465)
…tegration tag (TestIntegrationGateEpoch, change 0465)
…the integration tag (TestIntegrationGateArm, change 0465)
…ind the integration tag (TestIntegrationGateLifecycle, change 0465)
…tion tag (TestIntegrationFinalizeOps, TestIntegrationFinalizeRebaseOps, change 0465) The 38 normal finalize real-git tests measured 56s solo in one runner (row 65), over the 60s parallel-lane ceiling, so they are split across two runners with disjoint prefixes: TestIntegrationFinalizeRebaseOps (the 23 finalize_rebase continue/resolver-budget/gate-halt/receipt tests, 35s solo, row 40) and TestIntegrationFinalizeOps (resolver-reserve, block, publish and PR-publish, 21.5s solo, row 30). TestFinalizeResolverReserveConcurrent joins the race shard as TestRaceIntegrationAppConcurrencyFinalizeResolverReserveConcurrent (race shard 29s solo, row 35 unchanged).
…egration tag (TestIntegrationEvidence, change 0465)
…d the integration tag (TestIntegrationRecordOps, change 0465)
…l-git tests behind the integration tag (TestIntegrationContextProbe, change 0465)
…t_go_race (change 0465)
tests/test_go_race.sh now passes RACE_TIMEOUT="4m" to `go test -race` and adds
an assert that names the backstop ("no package ran past the 4m -timeout
backstop") plus the offending FAIL line and a PARTITION AND LANE remedy on
overrun, instead of Go's 10m goroutine-dump panic. Header PARTITION AND LANE
paragraph updated for the change-0465 no-real-git invariant; new BACKSTOP
TIMEOUT paragraph.
Measurement (Step 1, CI-equivalent cap):
`GOMAXPROCS=2 go test -race -count=1 -p 2 ./...` exit 0; slowest packages
internal/repository/transaction 48.654s, internal/repository/workspace 44.1s,
internal/gatedrive 39.4s; internal/app 13.8s. 4 x 48.654s = 194.6s -> 4m
(>= 3m minimum, < 8m escalation bound).
Behavioral tests (internal/repoguard/race_gate_timeout_test.go) run a copy of
the real wrapper against a fake `go`; mutation probes (drop -timeout; drop the
backstop assert) each turn the matching test red.
…hange 0465, review finding) Review finding (important): RACE_TIMEOUT="4m" in tests/test_go_race.sh was sized from an idle local run (internal/repository/transaction 48.7s at -p 2). The spec's own data shows CI ~2.4-3.8x slower than local (238s vs 581-908s), projecting that package to ~120-230s against a 240s limit, so the backstop could trip intermittently. Raise it to 8m (>= 2x the worst projection, below Go's 10m default), rewrite the header derivation, and pin the floor in TestRaceGatePassesTimeoutBackstopBelowGoDefault.
- revert tests/test_go_toolchain.sh budget 60 -> 55 (out of spec scope) - nogit guard header states its known bypass limits instead of overclaiming - repoint stale *_test.go file references in maintained comments
danielhanold
deleted the
fix/test-go-race-times-out-on-internal-app-in-ci-go-s-10m-per-pa
branch
September 28, 2026 10:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
↩ Change 0465 — test_go_race times out on internal/app in CI (Go's 10m per-package limit) (change file)
Summary
The CI race gate (
tests/test_go_race.sh) intermittently hit Go's 10m per-package timeout ininternal/app. The cause: about 337 default tests there spawn realgit, and they took about 225s of the package's ~238s under-race.internal/app/nogit_guard_test.go) puts a refusinggitshim first onPATHand fails the package on any attempt, including tests that swallow the error. Tagged builds (integration,e2e) get a no-op twin.TestMain's supervisor/guardian re-exec routing is unchanged. The guard is mutation-tested.//go:build integrationinto 12 new plain shard runners (tests/test_go_integration_app_*.sh). Five genuinely concurrent tests went to the existing race shard.tests/test_go_integration_contract.shis green.go test -race ./internal/app/went from ~238s to ~14s.test_go_race.shpasses-timeout 8mand prints a readable overrun line, pinned byinternal/repoguard/race_gate_timeout_test.go.budgetKeyPath), so overruns accumulate across.worktrees/*and serial confirmation can fire.test_go_race.shstill measures ~67s solo against its 60 row.internal/appno longer drives this;internal/repository/transactionandinternal/workspacenow do. Those packages are follow-up work; this PR does not grant a larger ceiling.Results:
docs/results/2026-09-28-test-go-race-times-out-on-internal-app-in-ci-go-s-10m-per-pa-results.md. Human action after merge: confirm several green CI source-gate runs.Review
Reviewed by
docket-review-deep. The rung is the highest build profile, premium, and the diff is over 1500 lines. The review found 0 blockers, 1 important and 3 minors.test_go_toolchain.shrow raised 55→60 out of scopeBuild notes:
Build evidence
command: go run ./cmd/docket development test
result: green
head_sha: 6bf5932
ran_at: 2026-09-28T10:00:42Z
Budget report (green run, screening only; nothing serially confirmed): BUDGET WATCH lines for: