Skip to content

Re-baseline the device gate on iOS 27.0.1 - #690

Merged
leonardoaraujosantos merged 1 commit into
mainfrom
device/rebaseline-ios-27.0.1
Oct 5, 2026
Merged

leonardoaraujosantos merged 1 commit into
mainfrom
device/rebaseline-ios-27.0.1

Conversation

@leonardoaraujosantos

Copy link
Copy Markdown
Contributor

Why

The reference iPad (iPad15,5) moved from iOS 27.0 (24A437) to 27.0.1 (24A446). check_device_bench.py compares osVersion as an exact string before any number and refuses a mismatched run rather than scoring it, so no release could be gated until the baseline moved. v0.126.0 is blocked behind this. A point release invalidates the baseline exactly as a major one does; the gate skill now says so.

Measured from v0.120.1's engine, not from the release being gated

The baseline is written from a run at 7f6cf38f, v0.120.1's gate commit, whose src/, include/, backends/, bindings/ and CMakeLists.txt are identical to tag v0.120.1. A baseline taken from the release's own tree is compared against itself and leaves that release with no REGRESSION coverage (v0.120.0 shipped that way after #625). This one keeps the before/after comparison: v0.126.0's gate will compare v0.120.1's engine to v0.126.0's on the same OS.

The run

iPad15,5, iOS 27.0.1 (24A446), claycore 7f6cf38f at ABI 0.120.1. Seven cold sessions, 1800 s cooldowns, thermal state nominal at both ends of every case, treeDirty: false, valid: true, 75 cases. Budgets, growth and coverage pass against the new baseline (check_device_bench.py OK, check_device_coverage.py OK). The record is kept at build/device/runs/gate-rebaseline-ios27.0.1-7f6cf38f.json.

  • CONDITIONS CHANGED: the canary drifted x1.44 across the run (98.51 → 141.69 ms) while thermalState read nominal throughout, as it did on v0.120.1's gate (x1.65). Every case was measured with the canary inside tolerance.
  • 14 cases needed repeats for an honest p95 (median of 3 passes), led by mask_extrude (spread 983.5 ms) and sdf_consolidate (38.0 ms).

Getting a clean run took five attempts over two days, none of them the engine's fault: Xcode had been signed out (every target failed with "No Accounts" and a missing devicehost profile), the iPad had 0 bytes free, and then four sessions were heat-killed 36–80 s in — signal kill, no JetsamEvent, no crash report — while the iPad ran post-update Apple Intelligence and photo-analysis inference in the background and wrote three JetsamEvents of its own while idle. The run that passed had Wi-Fi off and the device under an air conditioner. All of it is recorded in the gate skill in this PR, with the preflights that would have caught each one in seconds.

The old numbers, not thrown away silently

Against the 27.0 baseline (704f2d4a, v0.120.0), taking each case's median across its points: overall median ratio 0.992 over all 75 cases, no case above both the 1.4 tolerance and the 0.125 ms floor, no case changed bundle. Largest rise armature_edit 1.24x (0.515 → 0.640 ms); largest falls stroke_build 0.79x (0.488 → 0.385 ms, #634's old watch item, now flat) and sdf_stamp_detail_bricks 0.79x.

case 27.0 27.0.1 ratio reading
mask_extrude at 1000 stamps 3938 ms 5297 ms 1.34x under tolerance, spread 983 ms over 3 passes; the 10- and 100-stamp points read 0.92x and 1.00x

Because that point is the budgeted figure, docs/09-brush-latency-and-coverage.md now quotes 5436 ms for mask_extrude (was 4040; check_doc_latency.py fails the old row at 1.35x). Watch item for the next gate, which can attribute it with a same-OS baseline.

Verification

  • python3 tools/check_device_bench.py build/device/device-bench.json against the new baseline: OK, recorded in last-gate.json.
  • python3 tools/check_device_coverage.py build/device/device-bench.json: OK.
  • python3 tools/check_doc_latency.py: OK, 46 figures.
  • pytest bindings/python/tests/test_doc_latency_matches_baseline.py: 8 passed.

tests/ and docs/ are not device-relevant paths, so this PR does not stale the release's device row; the release's own gate re-stamps last-gate.json at its commit.

The reference iPad15,5 moved from iOS 27.0 (24A437) to 27.0.1 (24A446), and
check_device_bench.py matches osVersion exactly, so no run could be scored
until the baseline moved. Measured at 7f6cf38, v0.120.1's gate commit, whose
engine paths are identical to the tag, so the next release keeps a same-OS
before/after comparison instead of comparing a run against itself.

75 cases, seven cold sessions, 1800 s cooldowns, thermal state nominal at both
ends, treeDirty false. Against the 27.0 baseline the median ratio is 0.992 over
75 cases with nothing above both the 1.4 tolerance and the 0.125 ms floor.
mask_extrude's 1000-stamp point read 5297 ms against 3938 (1.34x, spread 983 ms
over 3 passes), so docs/09 quotes 5436 ms for it now.

The gate skill records what this run taught: a point release invalidates the
baseline and the two-run recipe that keeps REGRESSION coverage; the Xcode
account and iPad storage preflights; that the temperature key and the console
are unreachable on iOS 27 with both libimobiledevice and pymobiledevice3; the
harness-based thermal probe; and that post-update on-device inference
heat-kills sessions that start from a nominal reading.
@leonardoaraujosantos
leonardoaraujosantos force-pushed the device/rebaseline-ios-27.0.1 branch from 96f0343 to cb1fd7c Compare October 5, 2026 17:10
@leonardoaraujosantos
leonardoaraujosantos merged commit c3a7d45 into main Oct 5, 2026
16 checks passed
@leonardoaraujosantos
leonardoaraujosantos deleted the device/rebaseline-ios-27.0.1 branch October 5, 2026 17:12
leonardoaraujosantos pushed a commit that referenced this pull request Oct 6, 2026
The gate passed at 7049e19 against the iOS 27.0.1 baseline #690 recorded
from v0.120.1's engine, so REGRESSION compares v0.120.1 to v0.126.0 on the
same OS: iPad15,5, 75 cases, seven cold sessions with 1800 s cooldowns,
nominal throughout, canary steady at x1.18, median ratio 0.996, none above
both the 1.4 tolerance and the 0.125 ms floor, largest rise 1.04x.

The first run at 7406eb1, before #691, failed on mask_extrude (x4.97,
26992 ms at 1000 stamps against an 8154 ms budget) and mask_extract (x6.26);
with #691 they read 1.01x and 1.04x. The stroke cases read 0.81-0.89x,
named as a candidate for #688 and not a claim. The run needed the iPad's
Wi-Fi off and an air conditioner; the notes say why.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant