Skip to content

Run the e2e harness on iOS, Android and in a browser - #179

Merged
ruccho merged 10 commits into
feature/webcodecs-fixfrom
feature/device-tests
Aug 26, 2026
Merged

Run the e2e harness on iOS, Android and in a browser#179
ruccho merged 10 commits into
feature/webcodecs-fixfrom
feature/device-tests

Conversation

@ruccho

@ruccho ruccho commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Problem

Checking an encoder change on a phone meant building a Unity player, so in practice the mobile and web backends were only ever exercised by hand.

What this does

One pipeline definition and one set of assertions in unienc_testkit; each platform supplies only a way in.

iOS simulator — a Rust test binary is a plain executable, so simctl runs it directly. No app bundle, no provisioning profile, no signing; a runner in .cargo/config.toml is the whole integration. VideoToolbox does encode H.264 in the simulator, so this covers the encoder and not just the build.

Android — MediaCodec and MediaMuxer need a JavaVM but neither an Activity nor a Context, so a JVM is the only thing an adb shell lacks and app_process supplies one. Hence a library loaded by a small Java shim rather than an instrumented test inside an APK: no Gradle project, and the same command works against an emulator and a real device.

Web — the only target where the harness cannot be a cargo test. The encoders are the browser's own and report through callbacks delivered as browser tasks, so the thread has to keep returning to the event loop while libtest wants a test function that runs to completion. unienc_harness_web drives the pipeline the way Unity does instead: a callback per animation frame. ASYNCIFY is not the easier answer it looks like — it unwinds the wasm stack while suspended and the encoder callbacks re-enter wasm during that window.

Two web-specific details: Emscripten is built here without pthreads, so TestRuntime uses a LocalPool, matching the build unienc_c ships for the web; and the muxer downloads its output rather than writing a file, so web::capture_muxed_output intercepts the one JavaScript function it calls and writes the same bytes into the in-memory filesystem, leaving the backend unchanged.

Verification

Passes on an iOS 26 simulator (from a cold start), an Android emulator and headless Chrome, alongside the desktop matrix. CI runs all three.

The browser job took two corrections to become meaningful. It needs a Chrome that can encode H.264: browser-actions/setup-chrome defaults to latest, which is a Chromium snapshot rather than a Google Chrome build, and Chromium ships without the proprietary codecs, so every profile the harness offered was refused. The job therefore asks for the stable channel, which resolves to a Chrome for Testing build.

It also runs on macOS rather than Linux, because of the audio. Chrome reaches WebCodecs' AAC encoder through MojoAudioEncoder, whose IsSupported is gated on the media::kPlatformAudioEncoder feature, and media/base/media_switches.cc enables that by default only on Windows, macOS and Android — the platforms with an OS-level AAC encoder behind it. On Linux the harness's audio configuration is refused outright, so the job could not have covered the backend's audio path, which is the path #178 fixed.

The workflow also stops triggering on both pull_request and an unrestricted push, which ran every job twice for the same commit. It was easy to miss because both runs report the same job names, and it cost little while the workflow was one cargo fmt --check; with the jobs added here it duplicated a simulator boot, an emulator boot and a second macOS runner. push is now restricted to main, which a pull request never heads, so nothing loses coverage. A concurrency group cancels a branch's run once a new push supersedes it, for the same reason: a superseded run holds a runner for a result nobody will read. Runs on the trunk are exempt, because each of those is the only result its merged commit ever gets.

The Android job is red on this PR alone, in the same way #176's Linux job is: the job is added here and the defect it finds is fixed above it. unienc_android_mc holds both of a two-worker executor's workers in its push loops, so the pull tasks never run and the pipeline deadlocks. #181 fixes that, and the top of the stack is green.

What this does not cover

The GPU blit path needs a Unity graphics device; the iOS static library link is not exercised by a simulator binary; and an emulator or simulator uses software or host encoders, so the frame drops and colour shifts specific to real hardware still need a device. docs/device-testing.md records this and how to reach for a real device.

Stack created with GitHub Stacks CLIGive Feedback 💬

🤖 Generated with Claude Code

@ruccho ruccho changed the title feature/device tests Run the e2e harness on iOS, Android and in a browser Aug 25, 2026
@ruccho
ruccho force-pushed the feature/device-tests branch from 7e0b44f to aa3c205 Compare August 25, 2026 05:59
ruccho and others added 6 commits August 25, 2026 15:22
Checking an encoder change on a phone meant building a Unity player, so in
practice the mobile and web backends were only ever exercised by hand. The
harness now runs standalone wherever the encoders do, off one pipeline definition
and one set of assertions in unienc_testkit.

Each platform needed a different way in, and the reasons are worth stating:

- **iOS simulator.** A Rust test binary is a plain executable, so simctl runs it
  directly — no app bundle, no provisioning profile, no signing. A runner in
  .cargo/config.toml is the whole integration. VideoToolbox does encode H.264 in
  the simulator, so this covers the encoder and not just the build.
- **Android.** MediaCodec and MediaMuxer need a JavaVM but neither an Activity
  nor a Context, so a JVM is the only thing an adb shell lacks and app_process
  supplies one. Hence a library loaded by a small Java shim rather than an
  instrumented test inside an APK: no Gradle project, and the same command works
  against an emulator and a real device.
- **Web.** The only target where the harness cannot be a `cargo test`. The
  encoders are the browser's own and report through callbacks delivered as
  browser tasks, so the thread has to keep returning to the event loop while
  libtest wants a test function that runs to completion. unienc_harness_web
  drives the pipeline the way Unity does instead: a callback per animation frame.
  ASYNCIFY is not the easier answer it looks like — it unwinds the stack while
  suspended and the encoder callbacks re-enter wasm during that window.

Two web-specific details are worth knowing about. Emscripten is built here
without pthreads, so TestRuntime uses a LocalPool, matching the build unienc_c
ships for the web. And the muxer downloads its output rather than writing a file,
so `web::capture_muxed_output` intercepts the one JavaScript function it calls and
writes the same bytes into the in-memory filesystem — the backend is unchanged.

The run now reports which stage it finished, because a harness that hangs on a
device otherwise gives nothing to go on. The video duration check allows for one
frame interval: MediaMuxer gives the last sample a duration of zero where
AVFoundation and FFmpeg give it a full one.

Verified on an iOS 26 simulator and an Android emulator; see
docs/device-testing.md, which records what none of this covers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three jobs alongside the desktop matrix: an iOS simulator on a macOS runner, an
Android emulator through android-emulator-runner, and Chrome on a Linux runner
for WebAssembly.

None of them covers a vendor's hardware encoder — an emulator and a simulator use
software or host encoders, and the frame drops and colour shifts that have needed
fixing before are specific to real hardware. What these do cover is that the
backends work at all on their own platform, which until now nothing did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The muxer no longer rejects audio that arrives before the first video frame, so
the open issue the document described is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The script fell over silently with no simulator running: BSD grep does not take
\s in a pattern, so the search for an available iPhone matched nothing, and under
`set -euo pipefail` the failing pipeline ended the script before the message
explaining what was missing could run.

That is the path CI takes every time, and the only path a machine with a
simulator already open never exercises. Spelled out as [[:space:]], with the
no-match case allowed through so the explanation is what the reader gets.

Verified from a cold start: the script booted a simulator, waited out its first
boot, and ran the tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The emulator job in CI sat for 45 minutes after printing "running" and produced
nothing at all — not even the JNI_OnLoad line that appears immediately on a local
emulator. There was no way to tell what it was doing, because a hang left no
trace and the job's own timeout was the only thing that ended it.

Three changes, each aimed at a question that could not be answered:

- The harness now runs under the device's own `timeout`, not a host-side one,
  because what hangs is the process on the device. A hang now ends in five
  minutes and says so, rather than holding the job until its overall limit.
- The device log is dumped on failure. A native crash, a missing library or an
  ART complaint appears there and nowhere else; the harness's own output stops at
  whatever it managed to print. The log is cleared first so the dump covers only
  this run.
- The Java shim reports either side of `System.load`. Producing no output at all
  has three quite different causes — app_process never reaching main, the load
  hanging, or the native harness hanging — and these two lines separate them.

The hang itself is unexplained. It does not reproduce on arm64 against API 36, or
against API 34 with the google_apis image and the same emulator options CI uses;
both pass. What is left is the x86_64 ABI and the CI environment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Android job's first diagnostic run answered one question and raised another.
It got much further than the silence suggested — through System.load, JNI_OnLoad
and the creation of the video encoder — and only then stopped. But every line of
that arrived at once, when the process was killed: stdout is block-buffered
behind a pipe, which is what `adb shell` gives it, so nothing appears until the
process exits and a hung one never does.

So progress is now flushed as it is written, and each stage reports its first
item as well as its total. A run that stops before the encoder accepts a frame
has a different cause from one that stops part-way, and stage totals alone cannot
tell those apart.

Two things also stopped the evidence reaching anyone: the CI artifact collected
`*.mp4`, which is exactly the file a hang does not produce, and the log filter was
broad enough that Play services drowned it. The artifact now takes the directory,
and the filter names the codec and process machinery.

Also reads the H.264 profile out of `avcC` and reports it. What an encoder
produces is not always what it was asked for — on the web it is negotiated with
the browser at run time — so the summary should describe the file rather than the
request. All three desktop backends turn out to produce High 3.1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ruccho
ruccho force-pushed the feature/device-tests branch from aa3c205 to ff4ea33 Compare August 25, 2026 06:23
The Android emulator job hangs in CI and passes on a development machine, on the
same emulator image and the same code. `UNIENC_TEST_THREADS` narrows the pool
that `TestRuntime` builds, which is what made that difference reproducible: with
two workers the run hangs after the encoder accepts its first frame, exactly as
it does in CI, and with three or more it passes. A CI runner has two cores and a
development machine has many, and the pool is sized to the machine.

The cause is not yet understood — see HANDOFF.md — but the harness can now
produce the failure on demand, which it could not before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The job ran on Linux against whatever `browser-actions/setup-chrome`
installed by default, and neither of those choices worked.

`setup-chrome` defaults to `latest`, which is a Chromium snapshot rather
than a Google Chrome build, and Chromium ships without the proprietary
codecs. Every H.264 profile the harness offered was refused, so the job
could not encode anything at all:

    No supported H.264 profile for 1280x720 at 1000000 bps;
    tried avc1.640028, avc1.4d0028, avc1.42001f

Linux could not have covered the audio either. Chrome reaches WebCodecs'
AAC encoder through `MojoAudioEncoder`, whose `IsSupported` is gated on
the `media::kPlatformAudioEncoder` feature, and
`media/base/media_switches.cc` enables that by default only on Windows,
macOS and Android — the platforms with an OS-level AAC encoder behind
it. On Linux the harness's audio configuration is refused outright, so
the path #178 fixed would have gone untested.

Run the job on macOS against the Chrome the runner image already
carries, which is where `run-in-browser.sh` looks by default. That
covers video and audio and needs no browser installed.

TypeScript is installed explicitly because `unienc_webcodecs`'s build
script shells out to `tsc`: the Linux image ships it globally, so the
job worked there by accident, and the macOS image does not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ruccho
ruccho force-pushed the feature/device-tests branch from a492966 to c10b086 Compare August 25, 2026 07:46
ruccho and others added 2 commits August 25, 2026 17:05
The workflow triggers on both `pull_request` and an unrestricted `push`,
so a branch with an open pull request runs every job twice for the same
commit. That cost nothing when the workflow was a single
`cargo fmt --check`, and it is easy to miss because both runs report the
same names. With the jobs this branch adds it duplicates a simulator
boot, an emulator boot and a second macOS runner.

Restrict `push` to the trunk. A branch is covered by `pull_request`, and
`main` is not the head of a pull request, so nothing loses coverage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These jobs are long — a simulator boot, an emulator boot and two macOS
runners — so letting a superseded run finish holds runners for a result
nobody will read.

Runs on the trunk are exempt. Each of those is the record for one merged
commit, and cancelling one loses the only result that commit ever gets;
a branch's superseded run has a replacement on the way by definition.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ruccho
ruccho marked this pull request as ready for review August 26, 2026 02:12
@ruccho
ruccho requested a review from hkmt-mmy August 26, 2026 02:12
@ruccho
ruccho merged commit 0f41028 into main Aug 26, 2026
6 of 7 checks passed
@ruccho
ruccho deleted the feature/device-tests branch August 26, 2026 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants