Skip to content

metal: synchronous pipeline calls always compile and return their result; completion handlers on workers (second version of #396) - #398

Open
claudiusthebot wants to merge 2 commits into
mainfrom
fix/metal-sync-compile-v2
Open

claudiusthebot wants to merge 2 commits into
mainfrom
fix/metal-sync-compile-v2

Conversation

@claudiusthebot

Copy link
Copy Markdown
Collaborator

Built, not phone-tested. No phone was reachable on 20 Nov; nothing here has run on a phone. Do not merge before the phone test below passes.

Second version of #396 (reverted by #397). It keeps what #396 got right (completion-handler compiles on worker threads, PanVK's compile budget for runaway compiles, one shader cache per user, the warm-cache step) and drops the part that broke the phone: a synchronous pipeline call now always gets its pipeline or the compile's own error, never a timeout refusal.

Why #396 failed on phone 2 (19 Nov)

  1. 5 s timeout → backboardd crash loop. On a cold cache, backboardd's first render pipelines took longer than STARBITE_M2V_COMPILE_TIMEOUT_MS (5000 ms), so the frontend refused them ("still running … ask again"). QuartzCore treats a refused render pipeline as fatal, so backboardd aborted three times and launchd rebooted, on two boots in a row. A synchronous Metal call can't be told "ask again later".

  2. Timeout 0 → frames stop after the boot logo. The kernel's samples of that boot (con-xnu-boot-L6-1119-0039.log, symbolized against the 24B156 cache) show:

    • backboardd was idle at 00:42:02. The render server and IOMFB threads were in mach_msg. The only StarbiteMetal frames were the telemetry sleep() loops. No thread was compiling and there were no PANVK lines.
    • SpringBoard's main thread was blocked at 00:41:02 and 00:42:02, inside a synchronous XPC message to chronod (-[CHSChronoServicesConnection _subscribeToExtensions] → xpc_connection_send_message_with_reply_sync). It made no new commits, so backboardd had nothing new to show.
    • Neither SpringBoard nor chronod had opened the native Metal device. Only backboardd had, and it got cache hits.

    With the timeout at 0, metal: compiles off the UI thread (timeout + budget), per-user shader cache, warm-cache step #396's synchronous path compiled on the caller as before. Inferred: the stall was not caused by the frontend. It is not proven, because chronod wasn't sampled. The test image below samples it.

What changes

  • tools/ios27/metal2/CompileQueue.h (new, plain C++17). It holds the rules, and its header comment states them:

    • Synchronous calls compile on the calling thread every time. There is no timeout, no worker slot and no negative cache.
    • Completion-handler calls return at once and compile on a worker: at most 3 at a time, in a FIFO list, with no threads parked on a semaphore. Each handler is called exactly once.
    • An asynchronous compile that failed after more than 10 s isn't retried for 10 minutes. Only the async path uses this.
    • No lock is held while calling out, so handlers may request the next pipeline.
  • patches/frontend-async-compile.diff

    • The 3 pipeline makers go through gCompiler.sync and the 6 completion-handler methods through gCompiler.async.
    • The handler gets the state, reflection and NSError that the synchronous method made.
    • The worker queue has no QoS class of its own, so a compile runs at the QoS of the thread that asked for it. On this kernel, utility/background work runs on the A55s.
    • STARBITE_M2V_COMPILE_TIMEOUT_MS is gone. A leftover mtl2-env line changes nothing.
  • mesa26-panvk-compile-budget.patch. The budget now counts the compiling thread's CPU time (CLOCK_THREAD_CPUTIME_ID) instead of wall time. A compile that is only waiting for a core is not a runaway, and since synchronous callers now see this error, a wall-clock budget could fail backboardd's pipelines on a busy phone. The longest real compile seen took 25 s of CPU; the default budget stays at 90 s.

  • Tests: tools/ios27/metal2/tests/compile-queue-test.cpp + tests/run.sh, plus a CI step. Each test names the phone failure it would have caught:

    • a slow synchronous compile is waited for, even with the old timeout variable set (failure 1);
    • a synchronous call never waits behind busy async slots (the night's hypothesis for failure 2);
    • 200 async requests on 3 slots: each handler is called once, never more than 3 run at a time, and all finish;
    • make()→sync() and done()→async() re-entry doesn't deadlock;
    • a handler that throws doesn't keep its slot;
    • the negative cache is async-only (tested with a fake clock).

    The tests also run under ThreadSanitizer where it can start (with setarch -R on Linux kernels with large mmap randomisation). run.sh also lints the frontend diff: no timed wait or timeout refusal, 3 gCompiler.sync and 6 gCompiler.async. It fails on metal: compiles off the UI thread (timeout + budget), per-user shader cache, warm-cache step #396's diff (checked) and passes on this one.

  • docs/ios27/METAL.md explains why metal: compiles off the UI thread (timeout + budget), per-user shader cache, warm-cache step #396 was reverted and what the timeout-0 samples show.

Unchanged from #396: hotmetal2-shared-cache.diff, 81-m2v-cache.sh and harvest-m2v-cache.sh.

The first commit re-applies #396 (reverting #397) so the second one reads as a change to it.

Trade-off, stated plainly

A synchronous caller on a UI thread still waits for its compile, as it did before #396. A cold first Settings open can take as long as its 24 522-NIR compute shader, about 25 s of one core. The fix for that is the warm cache (step 81), not a timer. Opens after the first should come from the per-user cache.

Built

Built on the Mac mini with mk-panvk27.sh and mk-bundle27.sh, using the same Xcode (iOS 18.5 SDK), host tools and bridge library (Nov 12) as #396's bundle:

  • StarbiteMetal c1ba995d0a19f6af… (dSYM UUID 537181AC-2370-3012-861C-691F97A6FA4F)
  • libvulkan_panfrost.dylib 72136f02b948be76…

The binaries contain no "still running after" string. These local checks pass: tests/run.sh, provenance-check, arch-check --report, builds-test, build-lock-test. The talon preflight lane doesn't apply to this repository.

Phone test (to do)

On phone 2, with BOOT L6 + sweetsample=SpringBoard,backboardd,chronod and the root step "bundle + Preferences in mtl2-clients + step 81, cold cache":

  1. Cold-cache boot. Pass if:

    • PANVK: lines appear (the cache really was cold);
    • backboardd shows no refused, abandoned after, SWEET EXIT or CRASH;
    • frames continue past swap 8;
    • the lock screen shows.

    If it stalls after swap 6, symbolize the main threads of SpringBoard, chronod and backboardd from the samples before rebooting.

  2. Settings ×3. The first open may take as long as its compiles. Opens 2 and 3 should show Settings within 3 s with no new PANVK: lines. No crash and no refusal on any open.

  3. Display & Brightness ×3. The page draws each time, with no freeze or crash; the 2nd and 3rd opens in under 2 s.

  4. Warm-cache repeat. Reboot without a root step. PANVK: lines should be 0 or very few, and Settings' first open within 3 s.

The exact commands, images and hashes are in the test notes on the mini (prep-1120/TEST-ON-PHONE.md).

🤖 Generated with Claude Code

)"

This reverts commit bd4b4b4 (#397), so the next commit can change #396 rather than start again.
Not usable as it stands: see the next commit.
…ult; completion handlers on workers (second version of #396)

#396 timed the synchronous new*PipelineState calls out after 5 s and refused the pipeline while the compile went on. On phone 2
(19 Nov) a cold cache made backboardd's first render pipelines take longer than that; QuartzCore treats a refused render
pipeline as fatal, backboardd aborted three times and launchd rebooted, twice. #397 reverted it.

- metal2/CompileQueue.h (new, plain C++17): the rules. A synchronous call compiles on the calling thread, every time, and
  returns the pipeline or the compile's own error: no timeout, no worker slot, no negative cache. A completion-handler call
  returns at once; its compile runs on a worker (at most 3 at a time, FIFO list, no thread parked on a semaphore), the handler
  is called exactly once; an asynchronous compile that failed after more than 10 s is not started again for 10 minutes.
  Nothing is called with the lock held (handlers may ask for the next pipeline).
- patches/frontend-async-compile.diff: the frontend on CompileQueue.h. The 3 pipeline makers go through gCompiler.sync, the 6
  completion-handler methods through gCompiler.async (the handler gets the state, reflection and NSError the synchronous method
  made). The worker queue has no QoS of its own: a compile runs at the QoS of the thread that asked for it.
  STARBITE_M2V_COMPILE_TIMEOUT_MS is gone (a leftover mtl2-env line changes nothing).
- mesa26-panvk-compile-budget.patch: the budget is the compiling thread's CPU time (CLOCK_THREAD_CPUTIME_ID), not wall time:
  a compile starved of a core is not a runaway, and a failed render pipeline is fatal to backboardd.
- metal2/tests/compile-queue-test.cpp + tests/run.sh (and a CI step): sync waits for a slow compile even with the old timeout
  variable set; sync never waits behind busy async slots; 200 async requests on 3 slots each answered once; make() calling
  sync() and done() calling async() do not deadlock; a throwing handler keeps no slot; the negative cache is async-only. Under
  ThreadSanitizer too where the compiler has it. run.sh also checks the frontend diff: it fails on #396's version.
- docs/ios27/METAL.md: why #396 was reverted, and what the timeout-0 boot's samples show (SpringBoard's main thread in a
  synchronous XPC to chronod; backboardd idle; no compile running).

Per-user cache (hotmetal2-shared-cache.diff), the warm-cache step (81-m2v-cache.sh, harvest-m2v-cache.sh) unchanged from #396.
Built, not phone-tested.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant