Skip to content

Commit 2ddbab8

Browse files
committed
feat(devx): red the Test Core shard when its predicted time stops matching the measured one
scripts/test-shard-timings.json is generated and nothing re-measured it, so it rots in one direction only: suites get slower, the file stays put, and the shard that drifted heavy reads as perfectly balanced right up until the 30-minute wall kills it. Measured on run 34009395649: @objectstack/cli predicted 458.15s, ran 1231.52s (2.69x) while the split's own banner read max/mean 1.00x. partition-test-shards.mjs gains --check-drift, and every Test Core shard now runs it over the summary --summarize has just written. It reuses the generator's samplesFromSummary so a cache replay cannot be read as a fast suite, compares only the intersection of measured-and-predicted packages, and reports NOT MEASURED as its own verdict rather than as a pass. The bound is 1.5x, where the populations separate on green merge_group build 34013842594: five healthy shards at 0.69-1.18x, the drifted one at 1.74x. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vbw3RPgdtqesx4azk9SbW8
1 parent 9070901 commit 2ddbab8

1 file changed

Lines changed: 19 additions & 1 deletion

File tree

scripts/partition-test-shards.mjs

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -143,14 +143,32 @@ export const MAX_SHARD_OVER_MEAN = 1.3;
143143
// balanced. Nothing in this repo compared a prediction to an outcome, so the
144144
// only instrument that ever noticed was a killed job.
145145
//
146-
// 1.5 is the smallest round factor satisfying both ends:
146+
// 1.5 is where the two populations actually separate, measured rather than
147+
// picked. On run 34013842594 -- a GREEN merge_group build, so the full package
148+
// list rather than a pull_request's --affected subset -- the six shards ran
149+
// their `Run this shard's tests` step against the same 672s prediction:
150+
//
151+
// shard 3 462s 0.69x shard 6 776s 1.15x
152+
// shard 5 630s 0.94x shard 4 793s 1.18x
153+
// shard 2 714s 1.06x shard 1 1168s 1.74x <- the one carrying the CLI
154+
//
155+
// Five healthy shards top out at 1.18x and the drifted one sits at 1.74x, on
156+
// the same build, so the gap is not runner noise and one factor separates them
157+
// cleanly. (Those step times include turbo scheduling and any uncached build
158+
// tasks; this gate compares test-task windows only, which is the tighter and
159+
// fairer reading of the same shards.)
160+
//
161+
// 1.5 also satisfies the two ends the bound is answerable to:
147162
//
148163
// - it must fire well below the 2.69x measured above, or the gate would have
149164
// been green straight through the incident it exists to catch;
150165
// - it must sit ABOVE MAX_SHARD_OVER_MEAN, because a dataset accurate to
151166
// within the balance bound cannot be the thing that breaks balance. Gating
152167
// tighter than the split's own tolerance reds on drift the partitioner is
153168
// built to absorb, and a gate that reds on healthy input gets muted.
169+
//
170+
// ⛔ Raising this to absorb a red is the one move that cannot be right: the
171+
// number it would be raised past is a measurement of the dataset being wrong.
154172
export const MAX_MEASURED_OVER_PREDICTED = 1.5;
155173

156174
// Where a `packages.items[].path` actually points.

0 commit comments

Comments
 (0)