Skip to content

test(vision mixer): run the letterbox wipe test where the runner keeps real time - #1027

Merged
srperens merged 1 commit into
mainfrom
test/wipe-runner-realtime
Oct 7, 2026
Merged

srperens merged 1 commit into
mainfrom
test/wipe-runner-realtime

Conversation

@srperens

@srperens srperens commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1013, #994

Problem

vision_mixer_fx_test::wipe_between_letterboxed_sources_animates fails on the macOS runner on every main push since the perf merges: classic wipe: animated=false end white=0.00 red=0.76 (and sometimes inverted wipe should animate). It passes locally. The Linux llvmpipe runner showed the same signature before #1007 (see #994).

I traced it on macos-latest from a diagnostic branch. The traces ruled out the shader, the uniforms and the passthrough switch:

  • fx_take_1 leaves passthrough on the wipe's first buffer, its uniforms hold the latched u_start and u_duration=2, and a readback of its output shows the mask, with alpha 0 and the edge entering from the right on time.
  • The mixer's sink_1 keeps one buffer as its current one for the whole wipe. That buffer has PTS 0.684 and is an unmasked frame from before the take, while PGM runs from 1.0 s to 3 s.
  • GST_DEBUG=aggregator:6 says why. Every masked frame arrives after the mixer has produced the output frame it belongs to, and is dropped: gst_aggregator_pad_skip_buffers Skipping pts 0:00:09.049087417, and so on for each frame.

On the runner GL renders at about 20 fps, below the flow's 30. The mixer produces each frame at the clock plus its 86 ms latency. Once the take slot renders instead of passing through, it falls behind that deadline, so all its frames are late. PGM keeps the last on-time frame, which is unmasked, and the wipe reads as a cut. Before #1007 every FX slot rendered all the time, so the whole mixer ran behind the clock and nothing counted as late. That is why the failure appeared with the perf merges.

The traces also showed the take sometimes starting on a stretched, unfitted picture (white=1.00). The vision mixer fits a source's rect to its aspect from the GLib main context once the source's caps arrive. This tokio test never ran a main loop, so the fit happened only by chance. Sometimes that was mid-take, which re-applied the layout under the wipe.

Change

Test only:

  • Render at 640x360 (multiview 320x180) instead of 1280x720. The sources are 640x266 (2.40:1) and 640x274 (2.34:1), the same shapes, so the classic and inverted orientations are chosen as before.
  • Run a GLib main loop for the test, as vision_mixer_underlay_upload_test does.
  • Wait for the letterboxed opening picture (white between 0.5 and 0.9 of the frame), not just a mostly white one.

Rejected:

  • Raising the block's latency in the test. The runner's GL is slower than real time, so the take slot falls further behind every frame and a larger budget only delays the drop.
  • Changing the mixer. Dropping late frames on a live aggregator is GStreamer's normal behaviour, and on a host that keeps real time the frames are not late.

Evidence

  • Fixed on the runner: 6 of 6 runs of vision_mixer_fx_test passed on macos-latest with this change (run 37604297865), each about 5 s. The 1280x720 runs on the diagnostic branch before it failed 17 of 20.
  • Still a guard: with program_wipe_at_first_buffer not called on the classic path (local edit, reverted), the test fails with the CI signature: classic wipe: animated=false end white=0.00 red=0.76, classic wipe should animate.
  • Main loop: without it, at 640x360 locally the source rect stayed (0,0,640,360) with caps 640x266 for 30 s (PGM never settled on the letterboxed white source).

Tests

Ran (macOS, GStreamer 1.28.6, GL): STROM_REQUIRE_GL=1 cargo test --test vision_mixer_fx_test, 3 runs, 2 passed each; once with the classic mask removed, failing as above. On the macOS runner: 6 runs from the diagnostic branch, as above.
Not run: Linux (left to CI).

For the reviewer

  • On a GL host slower than real time, a wipe or any rendering FX slot can show as frozen or cut in production too. The mixer drops a late branch's frames. Nothing here changes that. It is the same for a source with an expensive look effect.
  • fix(vision mixer): keep the FX slot's pool across passthrough so wipes don't stall #1013 keeps the FX slot's GL pool across passthrough, and its description blames a slow ALLOCATION query for this failure. The runner traces do not show that (the pool exists and no query stalls). Its code change can still stand on its own guard, but it does not fix this test.

🤖 Generated with Claude Code

…s real time

On the macOS runner the wipe test failed almost every run with
'classic wipe: animated=false'. A trace on the runner showed the take
slot rendering the mask correctly, but the mixer skipping each masked
frame as late (gst_aggregator_pad_skip_buffers): GL there renders at
about 20 fps, the slot falls behind the mixer's clock deadline, and the
pad keeps the last unmasked frame. Render at 640x360 instead of
1280x720, which the runner keeps up with.

The test also never ran a GLib main loop, so the source rects were
fitted to their aspect only by chance, sometimes mid-take. Run one, and
wait for the letterboxed opening picture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@srperens srperens added the ci:macos Run Build (macOS) on this pull request label Oct 7, 2026
@srperens
srperens merged commit 1aea9e5 into main Oct 7, 2026
9 of 18 checks passed
@srperens
srperens deleted the test/wipe-runner-realtime branch October 8, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run Build (macOS) on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant