Repository navigation
Re-baseline the device gate on iOS 27.0.1 - #690
Merged
Merged
Conversation
The reference iPad15,5 moved from iOS 27.0 (24A437) to 27.0.1 (24A446), and check_device_bench.py matches osVersion exactly, so no run could be scored until the baseline moved. Measured at 7f6cf38, v0.120.1's gate commit, whose engine paths are identical to the tag, so the next release keeps a same-OS before/after comparison instead of comparing a run against itself. 75 cases, seven cold sessions, 1800 s cooldowns, thermal state nominal at both ends, treeDirty false. Against the 27.0 baseline the median ratio is 0.992 over 75 cases with nothing above both the 1.4 tolerance and the 0.125 ms floor. mask_extrude's 1000-stamp point read 5297 ms against 3938 (1.34x, spread 983 ms over 3 passes), so docs/09 quotes 5436 ms for it now. The gate skill records what this run taught: a point release invalidates the baseline and the two-run recipe that keeps REGRESSION coverage; the Xcode account and iPad storage preflights; that the temperature key and the console are unreachable on iOS 27 with both libimobiledevice and pymobiledevice3; the harness-based thermal probe; and that post-update on-device inference heat-kills sessions that start from a nominal reading.
leonardoaraujosantos
force-pushed
the
device/rebaseline-ios-27.0.1
branch
from
October 5, 2026 17:10
96f0343 to
cb1fd7c
Compare
leonardoaraujosantos
pushed a commit
that referenced
this pull request
Oct 6, 2026
The gate passed at 7049e19 against the iOS 27.0.1 baseline #690 recorded from v0.120.1's engine, so REGRESSION compares v0.120.1 to v0.126.0 on the same OS: iPad15,5, 75 cases, seven cold sessions with 1800 s cooldowns, nominal throughout, canary steady at x1.18, median ratio 0.996, none above both the 1.4 tolerance and the 0.125 ms floor, largest rise 1.04x. The first run at 7406eb1, before #691, failed on mask_extrude (x4.97, 26992 ms at 1000 stamps against an 8154 ms budget) and mask_extract (x6.26); with #691 they read 1.01x and 1.04x. The stroke cases read 0.81-0.89x, named as a candidate for #688 and not a claim. The run needed the iPad's Wi-Fi off and an air conditioner; the notes say why.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The reference iPad (iPad15,5) moved from iOS 27.0 (24A437) to 27.0.1 (24A446).
check_device_bench.pycomparesosVersionas an exact string before any number and refuses a mismatched run rather than scoring it, so no release could be gated until the baseline moved. v0.126.0 is blocked behind this. A point release invalidates the baseline exactly as a major one does; the gate skill now says so.Measured from v0.120.1's engine, not from the release being gated
The baseline is written from a run at
7f6cf38f, v0.120.1's gate commit, whosesrc/,include/,backends/,bindings/andCMakeLists.txtare identical to tagv0.120.1. A baseline taken from the release's own tree is compared against itself and leaves that release with noREGRESSIONcoverage (v0.120.0 shipped that way after #625). This one keeps the before/after comparison: v0.126.0's gate will compare v0.120.1's engine to v0.126.0's on the same OS.The run
iPad15,5, iOS 27.0.1 (24A446), claycore
7f6cf38fat ABI 0.120.1. Seven cold sessions, 1800 s cooldowns, thermal state nominal at both ends of every case,treeDirty: false,valid: true, 75 cases. Budgets, growth and coverage pass against the new baseline (check_device_bench.pyOK,check_device_coverage.pyOK). The record is kept atbuild/device/runs/gate-rebaseline-ios27.0.1-7f6cf38f.json.thermalStatereadnominalthroughout, as it did on v0.120.1's gate (x1.65). Every case was measured with the canary inside tolerance.mask_extrude(spread 983.5 ms) andsdf_consolidate(38.0 ms).Getting a clean run took five attempts over two days, none of them the engine's fault: Xcode had been signed out (every target failed with "No Accounts" and a missing devicehost profile), the iPad had 0 bytes free, and then four sessions were heat-killed 36–80 s in —
signal kill, no JetsamEvent, no crash report — while the iPad ran post-update Apple Intelligence and photo-analysis inference in the background and wrote three JetsamEvents of its own while idle. The run that passed had Wi-Fi off and the device under an air conditioner. All of it is recorded in the gate skill in this PR, with the preflights that would have caught each one in seconds.The old numbers, not thrown away silently
Against the 27.0 baseline (
704f2d4a, v0.120.0), taking each case's median across its points: overall median ratio 0.992 over all 75 cases, no case above both the 1.4 tolerance and the 0.125 ms floor, no case changed bundle. Largest risearmature_edit1.24x (0.515 → 0.640 ms); largest fallsstroke_build0.79x (0.488 → 0.385 ms, #634's old watch item, now flat) andsdf_stamp_detail_bricks0.79x.mask_extrudeat 1000 stampsBecause that point is the budgeted figure,
docs/09-brush-latency-and-coverage.mdnow quotes 5436 ms formask_extrude(was 4040;check_doc_latency.pyfails the old row at 1.35x). Watch item for the next gate, which can attribute it with a same-OS baseline.Verification
python3 tools/check_device_bench.py build/device/device-bench.jsonagainst the new baseline: OK, recorded inlast-gate.json.python3 tools/check_device_coverage.py build/device/device-bench.json: OK.python3 tools/check_doc_latency.py: OK, 46 figures.pytest bindings/python/tests/test_doc_latency_matches_baseline.py: 8 passed.tests/anddocs/are not device-relevant paths, so this PR does not stale the release'sdevicerow; the release's own gate re-stampslast-gate.jsonat its commit.