Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ jobs:
run: bash tools/ios27/fullroot/firmware-test.sh
- name: One iOS 27 root build at a time per work directory and per image (the lock, in a scratch directory)
run: bash tools/ios27/fullroot/build-lock-test.sh
- name: "Metal frontend: a synchronous pipeline call always compiles and returns its result (no timeout refusal), completion handlers run on at most 3 workers, each once (host test of CompileQueue.h)"
run: sh tools/ios27/metal2/tests/run.sh
- name: The backup tool only reads, copies exactly, and proves every copy
run: python3 tools/backup-partitions-test.py
- name: Saved var archives use the newest intact slot
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,4 @@ __pycache__/

# the loader S-Boot starts, as built (tools/ios27/s9/build-loader.sh)
/xnuloader/sboot/out/
tools/ios27/fullroot/warm-m2v/
39 changes: 39 additions & 0 deletions docs/ios27/METAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,42 @@ profile of the textured case the per-draw path is spread over the runtime (`draw
thirds of `end_pass` for scissors that differ every draw (it marks the bin grid one tile at a time; a draw whose scissor
covers the whole pass skips it, as most of CoreAnimation's main-pass draws should). The M1 runs this single-threaded code
several times faster than the S9's M3 at 1794 MHz.

## Shader compiles: off the UI, bounded, shared and warm (19 Nov 2026)

What an iOS 27 boot on the S9 showed (phone 2, `con-xnu-boot-L4-1118-2103.log`): every Settings open compiled the same 24 522-NIR compute
shader (250 224 bytes) for 24 to 28 s on one core with the GPU idle and the UI stalled; Preferences then began a fragment shader of 3 023
NIR instructions that never ended (Preferences at 130 %, launchd at 40 %, for more than three minutes). The causes and what changed:

| Cause | Change |
|---|---|
| PanVK compiled on the thread that asked for the pipeline, and the completion-handler `new...PipelineState` calls ran the compile on the caller too | `patches/frontend-async-compile.diff` + `metal2/CompileQueue.h` (second version, 20 Nov): the synchronous calls compile on the calling thread, every time, with no timeout, and return the pipeline or the compile's own error; the completion-handler calls return at once and compile on a worker (at most three at a time, FIFO, at the QoS of the caller), each handler called once; an asynchronous compile that failed after more than 10 s is not started again for 10 minutes. The rules and their host test: `tools/ios27/metal2/tests/run.sh` (also in CI) |
| A compile with no end: Bifrost's register allocator spills a node (or, past four rounds, up to 32) per round and rebuilds the interference graph each time | `mesa26-panvk-compile-budget.patch`: `STARBITE_PANVK_COMPILE_BUDGET_MS` (default 90 000, 0 = none) is a budget of one compile in the compiling thread's CPU time (not wall time: a compile that only waits for a core is not a runaway); the allocator's rounds, its LCRA solve and its liveness loop check it and leave the compile by `longjmp`; the pipeline fails (`PANVK: ... abandoned after N ms (compile budget)` on the console) |
| One cache directory per program (`/var/local/m2v-cache/<progname>`), so only backboardd's was ever filled, a sandboxed program fell back to its own caches directory, and Preferences or SpringBoard never saw what another had compiled | `patches/hotmetal2-shared-cache.diff`: one directory per user, `/var/local/m2v-cache/u<uid>` (the runtime's files are owner-only), for every process; `STARBITE_M2V_CACHE=/dir` (mtl2-env) names another |
| The directory was made by the first process to run, with nobody to ship anything in it | `steps.d/81-m2v-cache.sh` makes `/var/local/m2v-cache` (0777), removes the per-program directories, and installs a warm cache (below) |

**Why the first version was reverted (#397).** It timed the synchronous calls out after `STARBITE_M2V_COMPILE_TIMEOUT_MS` (5000) and
failed them like a refused pipeline while the compile went on. On phone 2 (19 Nov, `con-xnu-boot-L6-1119-0030.log`) a cold cache made
backboardd's first render pipelines take longer than that; QuartzCore treats any refused render pipeline as fatal ("Metal failed to
build render pipeline"), backboardd aborted three times and launchd rebooted, twice. A synchronous Metal call cannot be answered "ask
again later", so the second version has no timeout at all on that path (CompileQueue.h, rule 1). With the timeout set to 0 (boot
`-0039.log`) nothing crashed but the screen stayed on the splash after six framebuffer swaps: the kernel's samples of that boot show
backboardd idle (render server and IOMFB threads in `mach_msg`, no thread in a compile) and SpringBoard's main thread inside a
synchronous XPC message to chronod (`-[CHSChronoServicesConnection _subscribeToExtensions]`) at 00:41:02 and 00:42:02, so SpringBoard
committed nothing new; neither SpringBoard nor chronod had opened the Metal device. That stall is not explained by the frontend; the
phone test of the second version (in its PR) samples chronod too (`sweetsample=`) in case it comes back.

**A warm cache.** The runtime's entries are named by content: translated functions and pipelines under `v1/<build>/` (the build is the
bridge library's), PanVK's pipeline data in the content directory beside them (named by the SPIR-V, the pipeline state and PanVK's cache
UUID, `driver.rs`). PanVK itself cannot compile on the Mac for the Mali-G72 (it needs the kernel's GPU), so a warm cache is harvested from a
phone: boot the bundle that will ship, open Settings, Display, Appearance (Liquid Glass), the app switcher and Control Center until the
`PANVK: ... NIR instructions` lines stop, copy `/private/var/local/m2v-cache` off, run `tools/ios27/metal2/harvest-m2v-cache.sh PULLED`
(it keeps the newest build directory and the content directory of each `u<uid>/`, into `tools/ios27/fullroot/warm-m2v`, which is not in git),
and build the root: step 81 installs it, owner-only, owned by the uid. It holds for that bundle and that PanVK (about 33 MB); another
build finds none of it and compiles as before.

**Format 500 is `MTLPixelFormatYCBCR8_420_2P`** (metal2vulkan's `generated.rs`), not an unknown format: the runtime has it (`pixel_format_info.rs`,
`bind/ycbcr_convert`, converted in the shader), and the frontend's texture over an IOSurface passes the surface's YCbCr matrix on. The
`texture memory fmt 500` line is that path (`how` is "memory" for textures over an IOSurface or a buffer: they have no device upload, by
design, so no `texture device` line follows), and nothing was refused. Why the reference image did not draw is not in that log: the pipelines that sample it may simply not have been made yet (the compute and
fragment compiles above were still running), which the two changes above address; a phone run of the new bundle will say.
44 changes: 44 additions & 0 deletions tools/ios27/fullroot/steps.d/81-m2v-cache.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#!/bin/bash
# 81-m2v-cache.sh ROOT : the Metal shader cache directory, and a warm cache in it.
# The plug-in (StarbiteMetal.bundle, HotMetal2.mm + patches/hotmetal2-shared-cache.diff) keeps one cache per USER, shared by every
# process of that user: /private/var/local/m2v-cache/u<uid>/ (v1/<build>/ = translated shaders, v1/<content>/ = PanVK's pipeline data;
# all named by content). It makes the directory itself, but only where /private/var/local/m2v-cache exists and is writable: this
# step makes it (0777), and removes the per-program directories (backboardd/, PosterBoard/, ...) older plug-ins left.
# WARM_M2V_CACHE=<dir> (default tools/ios27/fullroot/warm-m2v when it exists) holds u<uid>/ directories as harvest-m2v-cache.sh
# makes them from a phone's cache after a warm-up (Settings, Display, Liquid Glass, the app switcher...): they are copied in, owned
# by that uid, owner-only (the runtime refuses files others can write), so the first Settings open compiles nothing.
# A warm cache is valid for the bundle and PanVK it was harvested with (the entries live under the bridge's build hash and
# PanVK's cache UUID); another build finds none of it and compiles as before, leaving the stale files to the runtime's 30-day sweep.
# WARM_M2V_CACHE=none: no warm cache (and the existing directories are left as they are).
set -euo pipefail
R=$1
H=$(cd "$(dirname "$0")" && pwd)
L=$R/private/var/local
C=$L/m2v-cache
install -d -m 0777 "$L" 2>/dev/null || mkdir -p "$L"
install -d -m 0777 "$C" 2>/dev/null || mkdir -p "$C"
chmod 0777 "$C"
# a plug-in before the shared cache: one directory per program
for d in "$C"/*/; do
[[ -d $d ]] || continue
n=$(basename "$d")
case $n in u[0-9]*) ;; *) rm -rf "$d"; echo "81-m2v-cache: removed the per-program cache $n";; esac
done
WARM=${WARM_M2V_CACHE:-$H/../warm-m2v}
if [[ $WARM == none || ! -d $WARM ]]; then
echo "81-m2v-cache: $C ready (0777); no warm cache (WARM_M2V_CACHE unset or $H/../warm-m2v absent)"
exit 0
fi
files=0
for u in "$WARM"/u[0-9]*/; do
[[ -d $u ]] || continue
uid=$(basename "$u"); uid=${uid#u}
rm -rf "$C/u$uid"
mkdir -p "$C/u$uid"
cp -R "$u." "$C/u$uid/"
find "$C/u$uid" -type d -exec chmod 0700 {} +
find "$C/u$uid" -type f -exec chmod 0600 {} +
chown -R "$uid:$uid" "$C/u$uid" 2>/dev/null || echo "81-m2v-cache: u$uid: chown to $uid failed (not root?): the files stay with the building user"
files=$((files + $(find "$C/u$uid" -type f | wc -l)))
done
echo "81-m2v-cache: warm cache from $WARM: $files files, $(du -sk "$C" | cut -f1) KiB in $C"
155 changes: 155 additions & 0 deletions tools/ios27/mesa-ios/mesa26-panvk-compile-budget.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
diff -ru a/src/panfrost/compiler/bifrost/bi_ra.c b/src/panfrost/compiler/bifrost/bi_ra.c
--- a/src/panfrost/compiler/bifrost/bi_ra.c
+++ b/src/panfrost/compiler/bifrost/bi_ra.c
@@ -7,6 +7,7 @@
#include "bi_debug.h"
#include "bi_builder.h"
#include "compiler.h"
+#include "pan_compiler.h"
#include "nodearray.h"
#include "valhall.h"

@@ -145,6 +146,8 @@
lcra_solve(struct lcra_state *l)
{
for (unsigned step = 0; step < l->node_count; ++step) {
+ if ((step & 1023) == 1023)
+ PAN_COMPILE_CHECK();
if (l->solutions[step] != ~0)
continue;
if (l->affinity[step] == 0)
@@ -262,7 +265,11 @@
bi_worklist_push_tail(&worklist, block);
}

+ unsigned pops = 0;
while (!u_worklist_is_empty(&worklist)) {
+ if ((++pops & 255) == 0)
+ PAN_COMPILE_CHECK();
+
/* Pop off in reverse order since liveness is backwards */
bi_block *blk = bi_worklist_pop_tail(&worklist);

@@ -1295,6 +1302,7 @@

/* Otherwise, use the register file and spill until we succeed */
while (!success && ((iter_count++) < max_iters)) {
+ PAN_COMPILE_CHECK();
l = bi_allocate_registers(ctx, &success, true);

if (success) {
diff -ru a/src/panfrost/compiler/pan_compiler.c b/src/panfrost/compiler/pan_compiler.c
--- a/src/panfrost/compiler/pan_compiler.c
+++ b/src/panfrost/compiler/pan_compiler.c
@@ -4,6 +4,31 @@
*/

#include "pan_compiler.h"
+#ifdef __APPLE__
+#include <time.h>
+
+__thread jmp_buf *pan_compile_abort_jmp = NULL;
+__thread uint64_t pan_compile_deadline_ns = 0;
+
+void
+pan_compile_check_deadline(void)
+{
+ if (!pan_compile_abort_jmp || !pan_compile_deadline_ns)
+ return;
+
+ /* CPU time of this thread, not wall time: a compile that is only waiting for a core (a
+ * phone at 400 % with daemons, the worker on an A55) is not a runaway one and must not be
+ * abandoned; the caller may be backboardd, for which a failed render pipeline is fatal. */
+ struct timespec t;
+ clock_gettime(CLOCK_THREAD_CPUTIME_ID, &t);
+ uint64_t now = (uint64_t)t.tv_sec * 1000000000ull + (uint64_t)t.tv_nsec;
+ if (now > pan_compile_deadline_ns) {
+ jmp_buf *target = pan_compile_abort_jmp;
+ pan_compile_abort_jmp = NULL;
+ longjmp(*target, 1);
+ }
+}
+#endif
#include "pan_nir.h"

#include "bifrost/bi_debug.h"
diff -ru a/src/panfrost/compiler/pan_compiler.h b/src/panfrost/compiler/pan_compiler.h
--- a/src/panfrost/compiler/pan_compiler.h
+++ b/src/panfrost/compiler/pan_compiler.h
@@ -81,6 +81,23 @@
struct util_dynarray *binary,
struct pan_shader_info *info);

+#ifdef __APPLE__
+#include <setjmp.h>
+#include <stdint.h>
+/* Starbite: a CPU-time budget for one compile on the calling thread. A caller that
+ * wants one (PanVK's panvk_compile_nir) sets pan_compile_abort_jmp and
+ * pan_compile_deadline_ns (CLOCK_THREAD_CPUTIME_ID, ns) around pan_shader_compile(); the register
+ * allocator, whose spill rounds are what a runaway compile is made of, calls
+ * PAN_COMPILE_CHECK() and past the deadline the compile is left by longjmp (what it
+ * allocated is lost: a few MB, once per abandoned shader). */
+extern __thread jmp_buf *pan_compile_abort_jmp;
+extern __thread uint64_t pan_compile_deadline_ns;
+void pan_compile_check_deadline(void);
+#define PAN_COMPILE_CHECK() pan_compile_check_deadline()
+#else
+#define PAN_COMPILE_CHECK() ((void)0)
+#endif
+
#define PAN_PRINTF_BUFFER_SIZE 16384

/* Any address with the top bit set is treated OOB by the hardware when
diff -ru a/src/panfrost/vulkan/panvk_vX_shader.c b/src/panfrost/vulkan/panvk_vX_shader.c
--- a/src/panfrost/vulkan/panvk_vX_shader.c
+++ b/src/panfrost/vulkan/panvk_vX_shader.c
@@ -1060,8 +1060,49 @@
clock_gettime(CLOCK_MONOTONIC, &t0);
#endif
struct util_dynarray binary = UTIL_DYNARRAY_INIT;
+#ifdef __APPLE__
+ /* Starbite: a compile that uses more than STARBITE_PANVK_COMPILE_BUDGET_MS (default 90 s,
+ * 0 for none) of its thread's CPU time is abandoned and its pipeline fails with a real
+ * compile error (an iOS 27 Preferences fragment shader of 3023 NIR instructions never
+ * ended, the process at 130% for minutes). CPU time, not wall time: the longest real
+ * compile seen (a 24522-NIR compute shader) took 25 s of CPU, and a compile that is only
+ * starved of a core keeps going. The Metal frontend runs synchronous creation calls on the
+ * caller's thread with no timeout (this error is what such a caller gets, after the
+ * budget) and completion-handler calls on a worker, where a pipeline that failed after
+ * 10 s is not retried for 10 minutes. */
+ jmp_buf budget_jmp;
+ {
+ const char *benv = getenv("STARBITE_PANVK_COMPILE_BUDGET_MS");
+ unsigned long budget_ms = benv ? strtoul(benv, NULL, 0) : 90000;
+ if (budget_ms) {
+ struct timespec c0;
+ clock_gettime(CLOCK_THREAD_CPUTIME_ID, &c0);
+ pan_compile_deadline_ns = (uint64_t)c0.tv_sec * 1000000000ull +
+ (uint64_t)c0.tv_nsec + budget_ms * 1000000ull;
+ if (setjmp(budget_jmp)) {
+ pan_compile_abort_jmp = NULL;
+ pan_compile_deadline_ns = 0;
+ util_dynarray_fini(&binary);
+ char msg[160];
+ int fd = open("/dev/console", O_WRONLY | O_NOCTTY);
+ int n = snprintf(msg, sizeof(msg),
+ "PANVK: %s shader %.60s abandoned after %lu ms of CPU (compile budget)\n",
+ _mesa_shader_stage_to_string(nir->info.stage),
+ nir->info.name ? nir->info.name : "?", budget_ms);
+ if (fd >= 0) {
+ write(fd, msg, n > 0 ? (size_t)n : 0);
+ close(fd);
+ }
+ return VK_ERROR_OUT_OF_HOST_MEMORY;
+ }
+ pan_compile_abort_jmp = &budget_jmp;
+ }
+ }
+#endif
pan_shader_compile(nir, &input, &binary, &shader->info);
#ifdef __APPLE__
+ pan_compile_abort_jmp = NULL;
+ pan_compile_deadline_ns = 0;
clock_gettime(CLOCK_MONOTONIC, &t1);
{
long ms = (t1.tv_sec - t0.tv_sec) * 1000 + (t1.tv_nsec - t0.tv_nsec) / 1000000;
2 changes: 1 addition & 1 deletion tools/ios27/metal/mk-bundle27.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ rm -rf "$B" && mkdir -p "$B" $W/sym $W/stage
# the frontend and HotMetal2 are built from a copy with patches/frontend-dexr.diff, hotmetal2-diag.diff, frontend-flush.diff and
# frontend-turned.diff, frontend-dexr-surfaces.diff, frontend-nocopy-diag.diff, frontend-passlog.diff and frontend-nocopy-readonly.diff applied, in that order (NOPATCH=1: as they are in tools/ios27/metal2)
rm -rf $W/src27 && mkdir -p $W/src27/tools/ios27 && cp -R tools/ios27/metal2 $W/src27/tools/ios27/
[ -n "${NOPATCH:-}" ] || for p in frontend-dexr hotmetal2-diag frontend-flush frontend-turned frontend-dexr-surfaces frontend-nocopy-diag frontend-passlog frontend-nocopy-readonly; do patch -s -p1 -d $W/src27 < tools/ios27/metal/patches/$p.diff || exit 1; done
[ -n "${NOPATCH:-}" ] || for p in frontend-dexr hotmetal2-diag frontend-flush frontend-turned frontend-dexr-surfaces frontend-nocopy-diag frontend-passlog frontend-nocopy-readonly frontend-async-compile hotmetal2-shared-cache; do patch -s -p1 -d $W/src27 < tools/ios27/metal/patches/$p.diff || exit 1; done
M2=$W/src27/tools/ios27/metal2
# frontend-turned.diff includes TurnShaders.inc: tools/ios27/metal2/TurnShaders.metal (the pass that writes a turned texture into its
# surface) compiled by the same Xcode's Metal compiler and held as bytes
Expand Down
3 changes: 2 additions & 1 deletion tools/ios27/metal/mk-panvk27.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ SB=${SB:-$(cd "$(dirname "$0")/../../.." && pwd)}
# The libvulkan_panfrost.dylib of phone 1's c2 bundle (cc241e55..., 14 November) was built from exactly these patches, with the
# shim of that day, except for the two braces mesa26-panvk-batch-stats.patch restores in panvk_vX_cmd_query.c.
PATCHES="mesa26-panvk-ios mesa26-panvk-null-vs mesa26-panvk-shader-size-cap mesa26-bifrost-ra-multispill mesa26-panvk-host-images
mesa26-panvk-tiler-heap-halves mesa26-panvk-batch-stats mesa26-panvk-cpu-desc-copy mesa26-panvk-cpu-minmax mesa26-panvk-cpu-direct"
mesa26-panvk-tiler-heap-halves mesa26-panvk-batch-stats mesa26-panvk-cpu-desc-copy mesa26-panvk-cpu-minmax mesa26-panvk-cpu-direct
mesa26-panvk-compile-budget"
# mesa26-panvk-cpu-desc-copy (18 November): the descriptor copies on the CPU at submission with STARBITE_PANVK_CPU_DESC=1 (off
# without it), and the draw and copy counters of the shim's statistics line (docs/ios27/PERFORMANCE.md section 7).
# mesa26-panvk-cpu-direct (18 November): with STARBITE_PANVK_CPU_DIRECT=1 an indexed draw whose range the CPU found is recorded
Expand Down
Loading
Loading