Run the e2e harness on iOS, Android and in a browser - #179
Merged
Conversation
ruccho
force-pushed
the
feature/device-tests
branch
from
August 25, 2026 05:59
7e0b44f to
aa3c205
Compare
Checking an encoder change on a phone meant building a Unity player, so in practice the mobile and web backends were only ever exercised by hand. The harness now runs standalone wherever the encoders do, off one pipeline definition and one set of assertions in unienc_testkit. Each platform needed a different way in, and the reasons are worth stating: - **iOS simulator.** A Rust test binary is a plain executable, so simctl runs it directly — no app bundle, no provisioning profile, no signing. A runner in .cargo/config.toml is the whole integration. VideoToolbox does encode H.264 in the simulator, so this covers the encoder and not just the build. - **Android.** MediaCodec and MediaMuxer need a JavaVM but neither an Activity nor a Context, so a JVM is the only thing an adb shell lacks and app_process supplies one. Hence a library loaded by a small Java shim rather than an instrumented test inside an APK: no Gradle project, and the same command works against an emulator and a real device. - **Web.** The only target where the harness cannot be a `cargo test`. The encoders are the browser's own and report through callbacks delivered as browser tasks, so the thread has to keep returning to the event loop while libtest wants a test function that runs to completion. unienc_harness_web drives the pipeline the way Unity does instead: a callback per animation frame. ASYNCIFY is not the easier answer it looks like — it unwinds the stack while suspended and the encoder callbacks re-enter wasm during that window. Two web-specific details are worth knowing about. Emscripten is built here without pthreads, so TestRuntime uses a LocalPool, matching the build unienc_c ships for the web. And the muxer downloads its output rather than writing a file, so `web::capture_muxed_output` intercepts the one JavaScript function it calls and writes the same bytes into the in-memory filesystem — the backend is unchanged. The run now reports which stage it finished, because a harness that hangs on a device otherwise gives nothing to go on. The video duration check allows for one frame interval: MediaMuxer gives the last sample a duration of zero where AVFoundation and FFmpeg give it a full one. Verified on an iOS 26 simulator and an Android emulator; see docs/device-testing.md, which records what none of this covers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three jobs alongside the desktop matrix: an iOS simulator on a macOS runner, an Android emulator through android-emulator-runner, and Chrome on a Linux runner for WebAssembly. None of them covers a vendor's hardware encoder — an emulator and a simulator use software or host encoders, and the frame drops and colour shifts that have needed fixing before are specific to real hardware. What these do cover is that the backends work at all on their own platform, which until now nothing did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The muxer no longer rejects audio that arrives before the first video frame, so the open issue the document described is gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The script fell over silently with no simulator running: BSD grep does not take \s in a pattern, so the search for an available iPhone matched nothing, and under `set -euo pipefail` the failing pipeline ended the script before the message explaining what was missing could run. That is the path CI takes every time, and the only path a machine with a simulator already open never exercises. Spelled out as [[:space:]], with the no-match case allowed through so the explanation is what the reader gets. Verified from a cold start: the script booted a simulator, waited out its first boot, and ran the tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The emulator job in CI sat for 45 minutes after printing "running" and produced nothing at all — not even the JNI_OnLoad line that appears immediately on a local emulator. There was no way to tell what it was doing, because a hang left no trace and the job's own timeout was the only thing that ended it. Three changes, each aimed at a question that could not be answered: - The harness now runs under the device's own `timeout`, not a host-side one, because what hangs is the process on the device. A hang now ends in five minutes and says so, rather than holding the job until its overall limit. - The device log is dumped on failure. A native crash, a missing library or an ART complaint appears there and nowhere else; the harness's own output stops at whatever it managed to print. The log is cleared first so the dump covers only this run. - The Java shim reports either side of `System.load`. Producing no output at all has three quite different causes — app_process never reaching main, the load hanging, or the native harness hanging — and these two lines separate them. The hang itself is unexplained. It does not reproduce on arm64 against API 36, or against API 34 with the google_apis image and the same emulator options CI uses; both pass. What is left is the x86_64 ABI and the CI environment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Android job's first diagnostic run answered one question and raised another. It got much further than the silence suggested — through System.load, JNI_OnLoad and the creation of the video encoder — and only then stopped. But every line of that arrived at once, when the process was killed: stdout is block-buffered behind a pipe, which is what `adb shell` gives it, so nothing appears until the process exits and a hung one never does. So progress is now flushed as it is written, and each stage reports its first item as well as its total. A run that stops before the encoder accepts a frame has a different cause from one that stops part-way, and stage totals alone cannot tell those apart. Two things also stopped the evidence reaching anyone: the CI artifact collected `*.mp4`, which is exactly the file a hang does not produce, and the log filter was broad enough that Play services drowned it. The artifact now takes the directory, and the filter names the codec and process machinery. Also reads the H.264 profile out of `avcC` and reports it. What an encoder produces is not always what it was asked for — on the web it is negotiated with the browser at run time — so the summary should describe the file rather than the request. All three desktop backends turn out to produce High 3.1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ruccho
force-pushed
the
feature/device-tests
branch
from
August 25, 2026 06:23
aa3c205 to
ff4ea33
Compare
The Android emulator job hangs in CI and passes on a development machine, on the same emulator image and the same code. `UNIENC_TEST_THREADS` narrows the pool that `TestRuntime` builds, which is what made that difference reproducible: with two workers the run hangs after the encoder accepts its first frame, exactly as it does in CI, and with three or more it passes. A CI runner has two cores and a development machine has many, and the pool is sized to the machine. The cause is not yet understood — see HANDOFF.md — but the harness can now produce the failure on demand, which it could not before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The job ran on Linux against whatever `browser-actions/setup-chrome`
installed by default, and neither of those choices worked.
`setup-chrome` defaults to `latest`, which is a Chromium snapshot rather
than a Google Chrome build, and Chromium ships without the proprietary
codecs. Every H.264 profile the harness offered was refused, so the job
could not encode anything at all:
No supported H.264 profile for 1280x720 at 1000000 bps;
tried avc1.640028, avc1.4d0028, avc1.42001f
Linux could not have covered the audio either. Chrome reaches WebCodecs'
AAC encoder through `MojoAudioEncoder`, whose `IsSupported` is gated on
the `media::kPlatformAudioEncoder` feature, and
`media/base/media_switches.cc` enables that by default only on Windows,
macOS and Android — the platforms with an OS-level AAC encoder behind
it. On Linux the harness's audio configuration is refused outright, so
the path #178 fixed would have gone untested.
Run the job on macOS against the Chrome the runner image already
carries, which is where `run-in-browser.sh` looks by default. That
covers video and audio and needs no browser installed.
TypeScript is installed explicitly because `unienc_webcodecs`'s build
script shells out to `tsc`: the Linux image ships it globally, so the
job worked there by accident, and the macOS image does not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ruccho
force-pushed
the
feature/device-tests
branch
from
August 25, 2026 07:46
a492966 to
c10b086
Compare
The workflow triggers on both `pull_request` and an unrestricted `push`, so a branch with an open pull request runs every job twice for the same commit. That cost nothing when the workflow was a single `cargo fmt --check`, and it is easy to miss because both runs report the same names. With the jobs this branch adds it duplicates a simulator boot, an emulator boot and a second macOS runner. Restrict `push` to the trunk. A branch is covered by `pull_request`, and `main` is not the head of a pull request, so nothing loses coverage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These jobs are long — a simulator boot, an emulator boot and two macOS runners — so letting a superseded run finish holds runners for a result nobody will read. Runs on the trunk are exempt. Each of those is the record for one merged commit, and cancelling one loses the only result that commit ever gets; a branch's superseded run has a replacement on the way by definition. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ruccho
marked this pull request as ready for review
August 26, 2026 02:12
hkmt-mmy
approved these changes
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Checking an encoder change on a phone meant building a Unity player, so in practice the mobile and web backends were only ever exercised by hand.
What this does
One pipeline definition and one set of assertions in
unienc_testkit; each platform supplies only a way in.iOS simulator — a Rust test binary is a plain executable, so
simctlruns it directly. No app bundle, no provisioning profile, no signing; a runner in.cargo/config.tomlis the whole integration. VideoToolbox does encode H.264 in the simulator, so this covers the encoder and not just the build.Android — MediaCodec and MediaMuxer need a JavaVM but neither an
Activitynor aContext, so a JVM is the only thing anadbshell lacks andapp_processsupplies one. Hence a library loaded by a small Java shim rather than an instrumented test inside an APK: no Gradle project, and the same command works against an emulator and a real device.Web — the only target where the harness cannot be a
cargo test. The encoders are the browser's own and report through callbacks delivered as browser tasks, so the thread has to keep returning to the event loop while libtest wants a test function that runs to completion.unienc_harness_webdrives the pipeline the way Unity does instead: a callback per animation frame. ASYNCIFY is not the easier answer it looks like — it unwinds the wasm stack while suspended and the encoder callbacks re-enter wasm during that window.Two web-specific details: Emscripten is built here without pthreads, so
TestRuntimeuses aLocalPool, matching the buildunienc_cships for the web; and the muxer downloads its output rather than writing a file, soweb::capture_muxed_outputintercepts the one JavaScript function it calls and writes the same bytes into the in-memory filesystem, leaving the backend unchanged.Verification
Passes on an iOS 26 simulator (from a cold start), an Android emulator and headless Chrome, alongside the desktop matrix. CI runs all three.
The browser job took two corrections to become meaningful. It needs a Chrome that can encode H.264:
browser-actions/setup-chromedefaults tolatest, which is a Chromium snapshot rather than a Google Chrome build, and Chromium ships without the proprietary codecs, so every profile the harness offered was refused. The job therefore asks for thestablechannel, which resolves to a Chrome for Testing build.It also runs on macOS rather than Linux, because of the audio. Chrome reaches WebCodecs' AAC encoder through
MojoAudioEncoder, whoseIsSupportedis gated on themedia::kPlatformAudioEncoderfeature, andmedia/base/media_switches.ccenables that by default only on Windows, macOS and Android — the platforms with an OS-level AAC encoder behind it. On Linux the harness's audio configuration is refused outright, so the job could not have covered the backend's audio path, which is the path #178 fixed.The workflow also stops triggering on both
pull_requestand an unrestrictedpush, which ran every job twice for the same commit. It was easy to miss because both runs report the same job names, and it cost little while the workflow was onecargo fmt --check; with the jobs added here it duplicated a simulator boot, an emulator boot and a second macOS runner.pushis now restricted tomain, which a pull request never heads, so nothing loses coverage. Aconcurrencygroup cancels a branch's run once a new push supersedes it, for the same reason: a superseded run holds a runner for a result nobody will read. Runs on the trunk are exempt, because each of those is the only result its merged commit ever gets.The Android job is red on this PR alone, in the same way #176's Linux job is: the job is added here and the defect it finds is fixed above it.
unienc_android_mcholds both of a two-worker executor's workers in its push loops, so the pull tasks never run and the pipeline deadlocks. #181 fixes that, and the top of the stack is green.What this does not cover
The GPU blit path needs a Unity graphics device; the iOS static library link is not exercised by a simulator binary; and an emulator or simulator uses software or host encoders, so the frame drops and colour shifts specific to real hardware still need a device.
docs/device-testing.mdrecords this and how to reach for a real device.Stack created with GitHub Stacks CLI • Give Feedback 💬
🤖 Generated with Claude Code