Skip to content

measure-test-shard-timings: fold a package's test:repo leg into its sample - #17448

Merged
baozhoutao merged 1 commit into
mainfrom
claude/issue-16550-shard-timings-fold-test-repo
Sep 10, 2026
Merged

baozhoutao merged 1 commit into
mainfrom
claude/issue-16550-shard-timings-fold-test-repo

Conversation

@baozhoutao

Copy link
Copy Markdown
Contributor

Closes #16550

What changed

samplesFromSummary() in scripts/measure-test-shard-timings.mjs kept only
task.task === 'test', so the six packages #16466 split into test +
test:repo (core, objectql, rest, runtime, spec, types) had their
test:repo execution window silently dropped from every future refresh.

samplesFromSummary() now folds a package's test and test:repo legs by
summing their execution windows into one whole-package reading. The
rejection rule composes across the fold rather than being re-derived per
leg: every leg present for a package must be cache.status === 'MISS'
and exitCode === 0, or the whole package is skipped — one cached or
failed leg is exactly as disqualifying as it was for a lone test task
before. The existing throw paths (no execution window, negative seconds, no
package name) apply per leg, so they still fire for either task name.

The partitioner (scripts/partition-test-shards.mjs) is untouched — it only
weighs packages, and both --check-drift and the shard's own
turbo run test test:repo already run/read both tasks for a package through
this same function.

--check-drift self-consistency (boundary check)

Fed a synthetic two-leg summary for @objectstack/spec
(test 349.75s + test:repo 53.9s) against the current, unrefreshed
scripts/test-shard-timings.json (weight 403.65, measured before the
split when both halves still lived inside one test task):

$ node scripts/partition-test-shards.mjs --check-drift <fixture> --label "issue-16550 two-leg fixture"
shard-timing-drift: OK -- issue-16550 two-leg fixture, 403.6s measured vs 403.6s predicted
across 1 package(s) = 1.00x (bound 1.5x).

Measured and predicted agree (1.00x) because --check-drift reads legs
through the same now-fixed samplesFromSummary() — before this change, only
the 349.75s test leg would have been read against the 403.65s prediction,
a 0.87x undershoot invisible to nobody watching for it today but exactly the
"measured and predicted shrink together" blind spot the card names once the
dataset is refreshed and predicted also starts under-counting.

Dataset

scripts/test-shard-timings.json is not touched — this PR fixes the
generator only; "until the refresh the existing rows are right" per the
card. Refreshing is #16464's scheduled workflow's job.

Tests

  • node scripts/measure-test-shard-timings.mjs --self-test — OK. Added a
    two-task-summary battery: the sum case, an un-split control riding in the
    same summary (proves the fold does not disturb ordinary packages —
    the discriminating leg, since a before/after-identical reading on that
    control alone would not catch a fold that leaked), either-leg-cached (both
    orderings), and either-leg-failed. Floor raised 50 -> 56 for the 6 new
    cases (155 of 158 scripts/** self-tests have no assertion floor: a battery that never ran is indistinguishable from one that passed #13799).
  • node scripts/partition-test-shards.mjs --self-test — OK (71 measured
    packages, unaffected — it imports samplesFromSummary but its own fixtures
    are single-task).
  • Scoped gate derivation: node scripts/pm/dispatch-gates.mjs --commands
    named 34 families for this diff (editing a gated tooling script with its
    own --self-test pulls in the self-test-wiring/declaration-mirror family).
    All 34 ran green under scripts/pm/os-verify-lock.sh; reconciled with
    --ran — 34 derived, 34 run, 0 NOT-MEASURED.
  • pnpm exec eslint scripts/measure-test-shard-timings.mjs --no-inline-config --format json — 1 file checked, 0 errors, 0 warnings.
  • grep -naP '[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]' scripts/measure-test-shard-timings.mjs
    — no control bytes.

Changeset

skip-changeset — measured, not assumed: scripts/measure-test-shard-timings.mjs
lives at the repo root, outside every package directory, and every
publishing package's "files" array is limited to dist / README.md /
CHANGELOG.md (checked across packages/*/package.json). No package
bundles or re-exports this script (grep -rl measure-test-shard-timings
outside scripts/ returns nothing). Nothing published moves. Applying the
skip-changeset label on this PR now.

Clause-②: no


🤖 Generated with Claude Code

https://claude.ai/code/session_012GKcPZbMoGq7WPzKLfRBTU


Generated by Claude Code

…ples

samplesFromSummary() kept only task.task === 'test', so the six packages
split into `test` + `test:repo` by #16466 had their repo-scan seconds
silently dropped -- under-weighing spec by roughly an eighth on the next
refresh, in the direction --check-drift cannot see since it reads samples
through the same function.

Fold both legs per package by summing their execution windows; either leg
being a cache HIT or a failure skips the whole package, same as today's
single-leg rule. Self-test gains a two-task-summary battery (sum, either-leg-
cached, either-leg-failed, and an un-split control proving the fold does not
disturb ordinary packages), and its floor is raised 50 -> 56 accordingly.

Fixes #16550

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GKcPZbMoGq7WPzKLfRBTU
@baozhoutao baozhoutao added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Sep 10, 2026 — with Claude
@baozhoutao
baozhoutao marked this pull request as ready for review September 10, 2026 14:57
@baozhoutao
baozhoutao added this pull request to the merge queue Sep 10, 2026
Merged via the queue into main with commit 06df327 Sep 10, 2026
36 of 37 checks passed
@baozhoutao
baozhoutao deleted the claude/issue-16550-shard-timings-fold-test-repo branch September 10, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

measure-test-shard-timings: fold a package's test:repo task into its sample, or the next refresh under-weighs the six split packages

2 participants