Skip to content

DO NOT MERGE (premise under revision): report codecs whepserversink drops from the offer - #794

Closed
wagenet wants to merge 7 commits into
Eyevinn:mainfrom
wagenet:wagenet/whep-codec-discovery-report
Closed

wagenet wants to merge 7 commits into
Eyevinn:mainfrom
wagenet:wagenet/whep-codec-discovery-report

Conversation

@wagenet

@wagenet wagenet commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Caution

Hold. The premise below is wrong and this must not merge as written.

The startup codec discovery does fail for H.264 on macOS, exactly as described. But its result (stream.out_caps) is read only by request_webrtcbin_pad's media: None branch — the branch taken when webrtcsink creates the offer. In WHEP the client offers: whep_signaller's send_counter_offer defaults to false and Strom never sets it, so select_codec runs instead. That builds its candidates from the client's media.formats() against the global codec table, never consulting stream.out_caps, and re-runs discovery with the client's SDP caps — where force_profile is false, the constrained-baseline demand disappears, and vtenc passes.

So H.264 is dropped from a list nothing downstream reads, and consumers that ask for H.264 get it. Verified two ways: a whepclientsrc restricted to video-codecs="<H264>" negotiated to stable and got rtph264pay, and a separate session put real Chrome on a WHEP feed with no rank override and saw rtph264pay decoding at 1280x720 while 6 discovery failures were logged.

This PR would therefore report "H.264 was dropped from the offer" on every macOS WHEP flow while viewers are receiving H.264 — a false alarm, which is worse than the silence it set out to fix. The reporting mechanism and BlockDiagnostic are sound; what they are pointed at is not. Being reworked to report a codec a consumer asked for and did not get.


Stacked on #786, which this needs for BlockHealth. The first four commits are that PR; review it first. Only feat(whep): report codecs whepserversink drops from the offer is new here.

Long body: it carries the verbatim upstream evidence for a bug that belongs in gst-plugins-rs, and the measurements behind deciding not to ship the obvious workaround.

The defect

whepserversink proves each codec in a throwaway pipeline before offering it — encoder, parser, a codec-parser-caps capsfilter, payloader, appsink — and lookup_caps swallows each failure with a gst::warning!, "as long as we end up with one potential codec for each input stream".

None of that escapes the element. The discovery pipeline's bus sync handler returns BusSyncReply::Drop, consumer-pipeline-created fires only for real sessions, no property lists the survivors, and the sink still reaches PLAYING on whatever is left. A WHEP output that asked for four codecs and got three is indistinguishable from one that got four.

Verified 2026-09-10 on the five-seat rig, GStreamer 1.28.6, Apple Silicon, changing one environment variable between two runs of the same flow:

server env discovery failures codecs actually offered
default 4 (2 per WHEP sink) AV1, H265, VP9
GST_PLUGIN_FEATURE_RANK=x264enc:300 0 AV1, H264, H265, VP9

Chrome will not decode H265 over WebRTC in most builds, so Chrome viewers fall back to VP9, encoded per consumer in software: 31% of one core at rest to 91% with five viewers.

What this changes

The block hands an identity to request-encoded-filter. PayloadChainBuilder::build links that element immediately downstream of codec-parser-caps — the capsfilter that does the rejecting — so an EVENT_DOWNSTREAM probe on its sink pad sees a CAPS event exactly when that codec got past the encoder, and nothing when it did not. encoder-setup names the encoder per codec, keyed by the media type the encoder factory's src pad template produces, since the discoveries run concurrently and call order proves nothing.

Worth a reviewer's eye: discovery passes None as the consumer id to request-encoded-filter but the string "discovery" to encoder-setup. The filter is returned only when that id is None, so nothing is spliced into a live viewer's pipeline.

The verdict reaches block health as a new BlockHealthStatus::Degraded, logged at warn!, broadcast as BlockHealthChanged, and shown in amber in the flow list and detail panel:

Block 'whep_out' in flow 'Meeting' is degraded: H.264 was dropped from the offer:
vtenc_h264 failed codec discovery. The pipeline still reports Playing

Degraded deliberately does not satisfy has_failed_block() — the output still serves every viewer that can decode a surviving codec.

BlockDiagnostic is the general hook. The stalled-pad-task scan reaches the flow pipeline's own elements and nothing else; a codec that fails discovery stalls no pad anywhere in it.

The report waits 15 s after the last discovery pipeline is built before calling a verdict, so a slow software AV1 encoder is not mistaken for a dropped codec.

Only the first round of discovery counts, and that is load-bearing. The first round runs when the input caps arrive, with output caps of ANY; its result becomes the codec list the sink will answer a viewer with. Every later round belongs to a viewer that has already connected, and asks a different question: its output caps come from that viewer's SDP, so force_profile is false, codec-parser-caps stops demanding constrained-baseline, and H.264 negotiates on the viewer path even though it failed at startup. Folding that in cleared the verdict. Live on a real backend, before the fix: degraded at 16 s after the flow started, then healthy again 16 s after a viewer connected, while the codec list the sink answers with still had no H.264 in it. The first round's verdict is now final and later rounds are not instrumented at all.

Root cause, for a human to file upstream

Isolated with no webrtcsink in the pipeline. With no profile requested, Codec::parser_caps(force_profile=true) builds video/x-h264, stream-format=avc, profile=constrained-baseline. vtenc_h264 has no profile property and cannot emit it:

gst-launch-1.0 videotestsrc num-buffers=30 ! video/x-raw,width=640,height=360,framerate=30/1 \
  ! videoconvert ! vtenc_h264 ! h264parse \
  ! video/x-h264,stream-format=avc,alignment=au,profile=constrained-baseline ! fakesink

Fails with flow-return=(int)-4. The same line with profile=baseline or profile=main passes. Inside webrtcsink:

vtenc  gst_vtenc_encode_frame:<vtenc_h264-0> Output loop stopped with error (not-negotiated)
GST_CAPS pre_eventfunc_check:<codec-parser-caps:sink> caps video/x-h264, ...,
  stream-format=(string)avc, ..., profile=(string)baseline, ..., lcevc=(boolean)false
  not accepted
webrtcsink lookup_caps: Codec discovery pipeline failed: Internal data stream error.

force_profile is output_caps.is_any() && needs_encoding, and the startup discovery call site passes a hardcoded gst::Caps::new_any(), so nothing outside the element can relax it. gst-plugins-rs does not accept AI-written issues or merge requests, so this is written out for a human to file rather than filed from here. Standalone repro:

gst-launch-1.0 videotestsrc is-live=true pattern=smpte \
  ! video/x-raw,width=1920,height=1080,framerate=30/1 ! whepserversink

Why no workaround ships

Ranking x264enc above vtenc_h264 restores H.264 to the offer but not the reason to want it: x264enc is software, and the hardware encode is the thing that is broken. Total CPU for 30 s of 1080p30, each encoder configured as configure_encoder configures it, two passes in both orders to control for order bias:

encoder CPU seconds (pass 1 / pass 2)
no encoder: source, videoconvert, fakesink 6.2 / 5.2
vp9enc — today's Chrome fallback 15.5 / 11.4
x264enc — what the rank override selects 16.2 / 15.0
vtenc_h264 — what is broken 5.1 / 4.8

x264enc was dearer than the vp9enc it would replace in both passes. vtenc sits at the no-encoder floor. So the rank change swaps one software encoder for another, process-globally across ingest and recording too, for no measurable server gain — consistent with the earlier rig finding that both arms landed at ~90% of a core for five viewers. Constraining vtenc's caps has no lever either: no profile property, and the demand comes from a hardcoded Caps::new_any() inside the element.

Setting GST_PLUGIN_FEATURE_RANK=x264enc:300 still makes sense for viewer-side decode and battery, but that is a deployment decision, not something to hardcode.

Tests

backend/tests/whep_codec_discovery_test.rs builds a real WHEP Output block through WHEPOutputBuilder, feeds raw video into a real whepserversink, and asserts H.264 is reported as dropped. It forces the failure by removing h264parse from the registry rather than relying on Apple's encoder, so it runs on Linux CI — the same shape, not a different path: request-encoded-filter has already fired for H.264 by the time build_parser errors, so the block sees a codec attempted and never negotiated, exactly as vtenc produces. CODECS is a LazyLock built once per process, so the registry surgery keeps this file to one test.

Ran, green on this branch:

  • the integration guard, and confirmed it guards: with attach() stubbed to a no-op it fails; with it, passes
  • six unit tests in whep/codec_report.rs (verdict text, missing-encoder branch, no-discovery-yet, hold-across-a-later-round)
  • full backend suite with STROM_REQUIRE_GST_PLUGINS=1: 684 passed, 0 failed
  • cargo test --test openapi_test after updating the snapshot; types suite; cargo clippy --all-targets -D warnings; frontend clippy for wasm32-unknown-unknown

Verified on this machine but not shipped as a test: the unmodified block against the real vtenc_h264 reports H.264 was dropped from the offer: vtenc_h264 failed codec discovery. CI runs tests on Linux only, so a vtenc assertion would guard nothing there and would start failing the day upstream fixes it.

Also verified end to end on a running headless backend, which is what caught the viewer-round bug above: a flow of videotestsrc into a WHEP Output reports degraded in flow.block_health and in a warn! line 16 s after start, stays running: true with gst_state: Playing, and now holds that verdict across a connecting viewer for 50 s and six discovery rounds with a single health transition logged.

Risk

BlockHealthStatus gains a degraded variant, so openapi.json is regenerated. Additive, but the contract check may flag a new enum value on a response type, and any client matching exhaustively on that enum needs the arm.

backend/src/blocks/builtin/whep.rs is now 2758 lines and wants splitting like pipeline.rs and app.rs. New code went into whep/codec_report.rs rather than growing it further; the split is not in this PR.

🤖 Generated with Claude Code

wagenet and others added 7 commits September 9, 2026 17:48
A pipeline branch can stop dead while the flow still reports Playing.
When a downstream push fails with a non-fatal flow return such as
not-negotiated, GstAggregator marks its sink pads flushing and pauses its
source pad task, and video encoders pause their output loop. Neither
posts to the bus, and pausing a pad task is not a state change, so every
element stays PLAYING and nothing surfaces through gst_state or the API.
GstBaseSrc is the exception - it posts a flow error before pausing - so
until now only source-side failures were visible.

Scan the running pipeline every two seconds for pads whose task is paused
on an element that is itself PLAYING, attribute each to its owning block
via the block_id:internal_id element namespace, and expose the result as
Flow::block_health. Log once per transition and emit BlockHealthChanged
so the failure is not limited to GST_DEBUG output. The flow list marks an
affected flow as Failed rather than Running.

Elements held below PLAYING are skipped: a WebRTC session bin paused
mid-setup has paused pad tasks legitimately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gst_base_src_loop pauses its pad task on the way out for every reason,
end of stream included, and leaves the element in PLAYING. Any flow with
a finite source would therefore report that block as failed forever once
it played out. Skip pads carrying the EOS flag: a pad that has seen EOS
is finished by definition, and the stall this detects pushes no EOS.

Require a block to look stalled on two consecutive scans before reporting
it. Pad tasks pause briefly for legitimate reasons - a flushing seek, a
dynamic bin being linked in - and a single sample turned those into a
logged error and a UI flip. Costs one poll interval of latency.

Drop blocks that disappear from the scan, rather than leaving them in the
failed set where no later iteration could clear them.

Handle BlockHealthChanged in the frontend. block_health rides on the Flow
payload, and nothing refetched the flow on this event, so the indicator
only appeared if the user happened to refresh.

The reported detail said "pad task stopped" while the condition detected
is paused; Stopped is explicitly not reportable here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scan walks the flow pipeline and the bins beneath it. Blocks that
build an isolated gst::Pipeline - WHIP ingest sessions, a Media Player's
decode chain - are siblings of that pipeline rather than children, so
iterate_recurse never enters them. Without that written down an Ok reads
as "this flow is passing data", which it does not mean.

The guard that skips elements below PLAYING cited a WHIP session bin as
its example. Those live in a separate pipeline and are never scanned;
a webrtcbin added for a new WHEP consumer is one that actually occurs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A queue whose source pad is left unconnected takes not-linked from its
loop and parks the task there for good, with the element still PLAYING.
Blocks build optional outputs this way - the vision mixer's multiview_out
is one - so any flow wiring only the program output reported its mixer as
permanently failed.

Found on a real WHIP -> vision mixer -> WHEP rig, where a healthy flow
reported "pad task paused on mixer:queue_mv_out:src" seconds after start.
An unlinked pad has no branch to report on, so require a peer.

The regression test reproduces it without a rig: a tee with one queued
branch left unconnected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
whepserversink proves each codec in a private pipeline before offering
it and drops the ones that fail. That pipeline's bus sync handler
returns Drop for every message, no property lists the survivors, and the
sink still reaches PLAYING on whatever is left, so a lost codec is
invisible from the flow pipeline. On macOS vtenc_h264 emits
profile=baseline where the discovery pipeline's codec-parser-caps filter
demands constrained-baseline, H.264 drops out of every offer, and Chrome
viewers fall back to VP9 encoded in software once per viewer.

The WHEP Output block now hands an identity to request-encoded-filter,
which webrtcsink links immediately below the capsfilter that does the
rejecting. A codec whose filter sees a CAPS event negotiated; one that
sees nothing did not. encoder-setup names the encoder each codec was
tried with, keyed by the media type the encoder's factory produces so
the concurrent discoveries cannot be mixed up.

The verdict reaches block health as a new Degraded status. It is logged
and broadcast but does not make the flow read as failed, because the
block still serves every viewer that can decode a surviving codec.

BlockDiagnostic is the general hook behind it: the stalled-pad-task scan
only reaches the flow pipeline's own elements, so a block whose failure
happens outside it has to report that itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lets a caller pass an empty literal, and matches how the health scan
already takes its liveness reporters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A viewer connecting makes whepserversink run discovery again, and that
round asks a different question: its output caps come from the viewer's
SDP rather than ANY, so force_profile is false, codec-parser-caps stops
demanding constrained-baseline, and H.264 negotiates where it failed at
startup. Folding that into the verdict cleared it.

Seen end to end on a real backend: the block reported degraded 16 s
after the flow started, then healthy again 16 s after a viewer
connected, while the codec list the sink answers with still had no
H.264 in it.

The first round is the one whose result becomes that codec list, so its
verdict is now final and later rounds are not instrumented at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wagenet wagenet changed the title feat(whep): report codecs whepserversink drops from the offer DO NOT MERGE (premise under revision): report codecs whepserversink drops from the offer Sep 11, 2026
@wagenet

wagenet commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the premise does not hold.

The startup codec discovery failure this was built to surface reaches no WHEP consumer. stream.out_caps is read only by request_webrtcbin_pad's media: None branch, taken when webrtcsink creates the offer; a WHEP client always offers, so select_codec runs instead and re-discovers from the client's SDP caps, where force_profile is false and vtenc passes. Verified with real Chrome (rtph264pay decoding at 1280x720 with no rank override, alongside six logged discovery failures) and with a whepclientsrc pinned to video-codecs="<H264>".

So this would report a lost codec on every macOS WHEP flow while viewers receive it. What survives is BlockDiagnostic and BlockHealthStatus::Degraded, which now have no caller; infrastructure without a consumer is not worth a review. The branch stays if a real trigger turns up.

The underlying gst-plugins-rs defect is real and still worth fixing upstream — the discovery round rejects a working encoder over a constrained-baseline profile the element itself chose, and on a signaller where webrtcsink does create the offer it would cost the codec.

@wagenet wagenet closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant