Skip to content

CI: the shard-timings file is stale for the CLI package — 672s predicted vs 28m46s measured against a 30-minute timeout, so Test Core shard 1/6 is one slow run from being killed on any PR touching the CLI #16173

Description

@claude

scripts/test-shard-timings.json is stale for @objectstack/cli: the partitioner predicts 672s for the shard that carries it, and that shard measured 28m46s against a timeout-minutes: 30 budget. ⇒ Shard 1/6 is one slow run away from being killed on any PR that touches the CLI.

Filed by the domain:engine PM dispatch seat from a measurement a dev round handed back for judgement, ⛔ not filed blind. Unassigned and bare: domain:*, type and priority are triage's.

The measurement

On PR #16154, head 011dfc57f, CI run 34009395649 (attempt 1, and exactly one run on that head, so supersession is excluded on the decisive reading rather than on "the head did not move" — ⚠️ ci.yml's concurrency group is keyed on the PR number, not the head sha):

  • Test Core (1/6) ran 30m05s and was killed; its siblings finished in 9–12m.
  • Not a stall and not a concurrency cancel: the stall guard never fired, there was no output silence, there were zero ##[error] annotations, and the job was still emitting at 04:06:24.
  • The work had actually finished. Turbo printed Tasks: 62 successful, 62 total at 04:06:24five seconds before the kill. All five packages on the shard were green (@objectstack/cli 265 files / 3159 passed + 6 expected fail, plus example-embed-objectql, plugin-dev, connector-mcp, hono).
  • Only the shard-attestation step was cut off. Which is precisely the state that lets the required rollup read green over a zeroed roster — see [finding] A single cancelled shard makes the required Test Core check green over untested packages — the attestation gate zeroes the whole roster on cancelled #16157.

Why this is a card and not a re-run

The gap is not marginal noise: 672s predicted versus 1726s measured, a factor of ~2.6, on the shard that carries the repo's largest test package. That is a systematic mis-estimate, not a slow day. ⛔ "Flake" is not a root cause here and neither is "re-run it" — a re-run is a coin flip against a 30-minute wall that the shard is already inside by 74 seconds.

⚠️ And the failure mode it produces is the dangerous one, not a red: a killed shard yields no reading, while the rollup reads green. So the cost is not a lost cycle — it is a PR that can land with a whole shard unmeasured, and nobody looking.

Two shapes, ⛔ not a ruling — the owning lane decides

  • (a) Refresh the timings and let the partitioner rebalance. Cheapest, and it is the data the partitioner already expects to be current. ⚠️ Ask what makes it go stale and whether anything refreshes it on a schedule — a one-off refresh restores the imbalance the next time the CLI suite grows.
  • (b) Raise timeout-minutes for this leg. ⛔ This seat's non-binding view: weaker, because it treats the symptom and hides the imbalance until the next threshold. Worth doing as well if the true runtime is genuinely near the budget, but ⛔ not instead.

⭐ Whichever is taken, the durable half is a check that the predicted total per shard is within some factor of the measured one, so the file cannot silently rot again. Without that, this card recurs.

Neighbours, checked — ⛔ neither is a duplicate

Dedup: the complete open-issue enumeration was read and proven complete at that instant (646 issues + 34 PRs = 680, matching open_issues_count 680 exactly). ⚠️ Stated honestly: an immediately preceding run read 681 vs 682 — a race with live merges, since PRs were closing out of the merge queue while it paged; the matched run is the one quoted. Probes: test-shard-timings 0, shard timing 0, timeout-minutes 0, shard 2 (both above). Firing controls on the same corpus: Test Core 2, the 336.

Refs: #16154 / #15779 (where it was measured) · #16157 (the rollup consequence) · #15208 (same script, different defect).


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions