fix(whip): give a reused slot a fresh video decoder

[wagenet]

Description

Opened by Claude (AI) on behalf of @wagenet.

Current behaviour. A WHIP Input slot keeps its decode chain while publishers come and go. When a second publisher reuses a slot, it decodes through the video decoder the first publisher used. On macOS with a Safari publisher, a reused slot shows no video while its audio plays. In a local repro, the second session decoded at about 7 fps with frames up to 908 ms late. VideoToolbox flagged every frame of the third session as an error, so the slot produced no video. On any platform, a publisher that negotiated a different codec (VP8 after H.264) stops the slot with not-negotiated.

Fix. When a session claims a slot whose video decodebin has already run, restart that decodebin, which gives the session a fresh decoder, as on a slot no session has used yet. Audio keeps its running chain. A stalled chain is not restarted; that case is filed as #906. Frames the previous session left queued in the slot appsrc still reach the new decoder, and can stop the new session's keyframe requests early; that case is filed as #914.

Root cause and evidence

The slot's video chain is appsrc → decodebin (rtph264depay → h264parse → decoder) → videoconvert → tee. On macOS the decoder is vtdechw. The chain is built once per slot at flow start, and activate_slot_decoders only re-synced the running decodebin on reuse.

Local repro with a Safari publisher on one WHIP Input slot: publish, WHIP DELETE, publish again, three sessions in all. Pad probes on the slot appsrc, videoconvert sink and output tee, plus GST_DEBUG=videodecoder:4, showed:

• Frames reached the appsrc on time in every session.
• Only session 1 configured the decoder (set_format with the stream's codec_data). The later sessions carry the same caps, so they reconfigure nothing.
• Session 1 ran at 30 fps. Session 2 came out at about 7 fps, up to 908 ms late. In session 3, vtdechw0 reported Got frame N with an error flag for every frame from the first keyframe on.
• Frames left in the decoder's reorder queue come out at the start of the next session. A check for "video is decoding" can read them as the new session's video.

Chrome (headless and headed) and whipclientsink publishers ran through the same reused decoder without a problem, so only Safari's stream trips it. I did not isolate which property of the Safari stream does it.

How the restart works. restart_decodebin installs an IDLE probe on the slot appsrc source pad, so no buffer is in flight. It then:

1. unlinks the pad;
2. sets the decodebin to NULL, which drops the plugged elements and its source pad;
3. relinks, so the sticky events are sent again with the next buffer (stream-start, caps and segment, which the sink pad lost in NULL);
4. syncs the decodebin with the pipeline.

The existing pad-added handler links the new source pad to videoconvert when data arrives. A decodebin still in NULL has not run yet (first claim since flow start) and is only unlocked and synced, as before.

The existing video_decoding flag also works on a reused slot now. Before, pad-added never fired again on reuse, so the keyframe requester always sent all 5 requests. It now stops after 1 or 2, as on a fresh slot, or earlier when the previous session left frames queued (#914).

Related draft PRs.

• fix(whip): stop asking a reconnecting publisher for needless keyframes #805 stops the keyframe requester on the slot's output stamp. Stale frames from the kept decoder move that stamp, so on the macOS rig it asked for no keyframe at all on a failing reuse. This PR removes the frames the decoder held; frames still queued in the slot appsrc are WHIP Input: a reused slot's new session receives frames queued by the previous one #914. It does not conflict with fix(whip): stop asking a reconnecting publisher for needless keyframes #805.
• feat(whip): track a slot's liveness per medium #792 is the detector that reported these failures ("produced no video … while its audio is still flowing"). It does not conflict either.

How to review

• backend/src/whip_session_manager.rs: SlotDecodebin, restart_decodebin, and the branch in activate_slot_decoders.
• backend/src/blocks/builtin/whip.rs: only marks the video decodebin restart_on_reuse.
• backend/tests/whip_slot_decoder_reuse_test.rs: builds the slot chain through WHIPInputBuilder, runs two x264 sessions through one slot with allocate_slot/release_slot, and asserts that the second session decodes frames through a different decoder instance. A second test reuses the slot with VP8 after H.264 and asserts it decodes.

Related Issue

Follow-ups, not fixed here: #906 (stalled chain), #914 (frames queued by the previous session).

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Checklist

• I have run cargo fmt --all
• I have run cargo clippy --all-targets -- -D warnings (backend, the root default member)
• I have run cargo test --workspace. Ran instead: cargo test --lib (724 passed) and --test whip_slot_decoder_reuse_test --test whip_idle_slot_test --test whip_slot_caps_reuse_test --test pipeline_lifecycle_test --test whip_whep_ice_transport_policy_test (all passed), on macOS.
• I have updated documentation as needed (none needed)
• I have added tests for new functionality

Tests actually run:

• Both new tests pass: 5 of 5 runs for the first, 4 of 4 for both together. They also pass with avdec_h264 forced (the decoder Linux CI uses).
• With the restart disabled, both fail:
  • the second session decoded through the first session's vtdechw1 instead of a fresh decoder
  • the VP8 session on a reused slot decoded 3 frames, with not-negotiated from appsrc_video_0

End to end:

• Build: this commit plus fix(whip): keep a slot's video PTS moving forward frame to frame #893 (so the first-publish freeze does not mask the result) and temporary pad probes, 3 publish/stop cycles each.
  • Headless Chrome: 135 frames per session in all three sessions.
  • whipclientsink: 225, 225 and 210 frames.
• Build: the same fix ported to a build that also carries the WHIP sub-module split (refactor(whip): split whip.rs and whip_session_manager.rs into sub-modules #854).
  • Safari: all 3 reuses got video at 30 fps. Frames reached the output 200 to 250 ms ahead of the clock, as in the first session, and the decoder flagged no errors.
• Not tested: the CPU vision mixer.

🤖 Generated with Claude Code

[srperens]

This is marked as a draft, so I am holding off on a full review until it is marked ready for review. If you want a review before then, say so here or request me as a reviewer.

[srperens]

Written by Claude (AI) on behalf of @srperens.

Both commits are now part of #957, which consolidates the WHIP Input fixes into one series against current main. You are the author, and the logic is unchanged apart from conflicts with #756's per-slot output stamps.

One addition: the reuse test now also asserts that a reused slot sets video_decoding again. Your restart brings pad-added back for every session, which is what #805 was after. With the restart disabled, the assertion fails. #906 and #914 stay open.

Suggest closing this once #957 merges.

[srperens]

Closing in favour of #957, which carries this work (see the comment above).