Repository navigation
Conversation
A pipeline branch can stop dead while the flow still reports Playing. When a downstream push fails with a non-fatal flow return such as not-negotiated, GstAggregator marks its sink pads flushing and pauses its source pad task, and video encoders pause their output loop. Neither posts to the bus, and pausing a pad task is not a state change, so every element stays PLAYING and nothing surfaces through gst_state or the API. GstBaseSrc is the exception - it posts a flow error before pausing - so until now only source-side failures were visible. Scan the running pipeline every two seconds for pads whose task is paused on an element that is itself PLAYING, attribute each to its owning block via the block_id:internal_id element namespace, and expose the result as Flow::block_health. Log once per transition and emit BlockHealthChanged so the failure is not limited to GST_DEBUG output. The flow list marks an affected flow as Failed rather than Running. Elements held below PLAYING are skipped: a WebRTC session bin paused mid-setup has paused pad tasks legitimately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gst_base_src_loop pauses its pad task on the way out for every reason, end of stream included, and leaves the element in PLAYING. Any flow with a finite source would therefore report that block as failed forever once it played out. Skip pads carrying the EOS flag: a pad that has seen EOS is finished by definition, and the stall this detects pushes no EOS. Require a block to look stalled on two consecutive scans before reporting it. Pad tasks pause briefly for legitimate reasons - a flushing seek, a dynamic bin being linked in - and a single sample turned those into a logged error and a UI flip. Costs one poll interval of latency. Drop blocks that disappear from the scan, rather than leaving them in the failed set where no later iteration could clear them. Handle BlockHealthChanged in the frontend. block_health rides on the Flow payload, and nothing refetched the flow on this event, so the indicator only appeared if the user happened to refresh. The reported detail said "pad task stopped" while the condition detected is paused; Stopped is explicitly not reportable here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scan walks the flow pipeline and the bins beneath it. Blocks that build an isolated gst::Pipeline - WHIP ingest sessions, a Media Player's decode chain - are siblings of that pipeline rather than children, so iterate_recurse never enters them. Without that written down an Ok reads as "this flow is passing data", which it does not mean. The guard that skips elements below PLAYING cited a WHIP session bin as its example. Those live in a separate pipeline and are never scanned; a webrtcbin added for a new WHEP consumer is one that actually occurs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A queue whose source pad is left unconnected takes not-linked from its loop and parks the task there for good, with the element still PLAYING. Blocks build optional outputs this way - the vision mixer's multiview_out is one - so any flow wiring only the program output reported its mixer as permanently failed. Found on a real WHIP -> vision mixer -> WHEP rig, where a healthy flow reported "pad task paused on mixer:queue_mv_out:src" seconds after start. An unlinked pad has no branch to report on, so require a peer. The regression test reproduces it without a rig: a tee with one queued branch left unconnected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
whepserversink proves each codec in a private pipeline before offering it and drops the ones that fail. That pipeline's bus sync handler returns Drop for every message, no property lists the survivors, and the sink still reaches PLAYING on whatever is left, so a lost codec is invisible from the flow pipeline. On macOS vtenc_h264 emits profile=baseline where the discovery pipeline's codec-parser-caps filter demands constrained-baseline, H.264 drops out of every offer, and Chrome viewers fall back to VP9 encoded in software once per viewer. The WHEP Output block now hands an identity to request-encoded-filter, which webrtcsink links immediately below the capsfilter that does the rejecting. A codec whose filter sees a CAPS event negotiated; one that sees nothing did not. encoder-setup names the encoder each codec was tried with, keyed by the media type the encoder's factory produces so the concurrent discoveries cannot be mixed up. The verdict reaches block health as a new Degraded status. It is logged and broadcast but does not make the flow read as failed, because the block still serves every viewer that can decode a surviving codec. BlockDiagnostic is the general hook behind it: the stalled-pad-task scan only reaches the flow pipeline's own elements, so a block whose failure happens outside it has to report that itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lets a caller pass an empty literal, and matches how the health scan already takes its liveness reporters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A viewer connecting makes whepserversink run discovery again, and that round asks a different question: its output caps come from the viewer's SDP rather than ANY, so force_profile is false, codec-parser-caps stops demanding constrained-baseline, and H.264 negotiates where it failed at startup. Folding that into the verdict cleared it. Seen end to end on a real backend: the block reported degraded 16 s after the flow started, then healthy again 16 s after a viewer connected, while the codec list the sink answers with still had no H.264 in it. The first round is the one whose result becomes that codec list, so its verdict is now final and later rounds are not instrumented at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Closing: the premise does not hold. The startup codec discovery failure this was built to surface reaches no WHEP consumer. So this would report a lost codec on every macOS WHEP flow while viewers receive it. What survives is The underlying gst-plugins-rs defect is real and still worth fixing upstream — the discovery round rejects a working encoder over a |
Caution
Hold. The premise below is wrong and this must not merge as written.
The startup codec discovery does fail for H.264 on macOS, exactly as described. But its result (
stream.out_caps) is read only byrequest_webrtcbin_pad'smedia: Nonebranch — the branch taken when webrtcsink creates the offer. In WHEP the client offers:whep_signaller'ssend_counter_offerdefaults to false and Strom never sets it, soselect_codecruns instead. That builds its candidates from the client'smedia.formats()against the global codec table, never consultingstream.out_caps, and re-runs discovery with the client's SDP caps — whereforce_profileis false, theconstrained-baselinedemand disappears, and vtenc passes.So H.264 is dropped from a list nothing downstream reads, and consumers that ask for H.264 get it. Verified two ways: a
whepclientsrcrestricted tovideo-codecs="<H264>"negotiated to stable and gotrtph264pay, and a separate session put real Chrome on a WHEP feed with no rank override and sawrtph264paydecoding at 1280x720 while 6 discovery failures were logged.This PR would therefore report "H.264 was dropped from the offer" on every macOS WHEP flow while viewers are receiving H.264 — a false alarm, which is worse than the silence it set out to fix. The reporting mechanism and
BlockDiagnosticare sound; what they are pointed at is not. Being reworked to report a codec a consumer asked for and did not get.Stacked on #786, which this needs for
BlockHealth. The first four commits are that PR; review it first. Onlyfeat(whep): report codecs whepserversink drops from the offeris new here.Long body: it carries the verbatim upstream evidence for a bug that belongs in gst-plugins-rs, and the measurements behind deciding not to ship the obvious workaround.
The defect
whepserversinkproves each codec in a throwaway pipeline before offering it — encoder, parser, acodec-parser-capscapsfilter, payloader, appsink — andlookup_capsswallows each failure with agst::warning!, "as long as we end up with one potential codec for each input stream".None of that escapes the element. The discovery pipeline's bus sync handler returns
BusSyncReply::Drop,consumer-pipeline-createdfires only for real sessions, no property lists the survivors, and the sink still reachesPLAYINGon whatever is left. A WHEP output that asked for four codecs and got three is indistinguishable from one that got four.Verified 2026-09-10 on the five-seat rig, GStreamer 1.28.6, Apple Silicon, changing one environment variable between two runs of the same flow:
GST_PLUGIN_FEATURE_RANK=x264enc:300Chrome will not decode H265 over WebRTC in most builds, so Chrome viewers fall back to VP9, encoded per consumer in software: 31% of one core at rest to 91% with five viewers.
What this changes
The block hands an
identitytorequest-encoded-filter.PayloadChainBuilder::buildlinks that element immediately downstream ofcodec-parser-caps— the capsfilter that does the rejecting — so anEVENT_DOWNSTREAMprobe on its sink pad sees a CAPS event exactly when that codec got past the encoder, and nothing when it did not.encoder-setupnames the encoder per codec, keyed by the media type the encoder factory's src pad template produces, since the discoveries run concurrently and call order proves nothing.Worth a reviewer's eye: discovery passes
Noneas the consumer id torequest-encoded-filterbut the string"discovery"toencoder-setup. The filter is returned only when that id isNone, so nothing is spliced into a live viewer's pipeline.The verdict reaches block health as a new
BlockHealthStatus::Degraded, logged atwarn!, broadcast asBlockHealthChanged, and shown in amber in the flow list and detail panel:Degraded deliberately does not satisfy
has_failed_block()— the output still serves every viewer that can decode a surviving codec.BlockDiagnosticis the general hook. The stalled-pad-task scan reaches the flow pipeline's own elements and nothing else; a codec that fails discovery stalls no pad anywhere in it.The report waits 15 s after the last discovery pipeline is built before calling a verdict, so a slow software AV1 encoder is not mistaken for a dropped codec.
Only the first round of discovery counts, and that is load-bearing. The first round runs when the input caps arrive, with output caps of
ANY; its result becomes the codec list the sink will answer a viewer with. Every later round belongs to a viewer that has already connected, and asks a different question: its output caps come from that viewer's SDP, soforce_profileis false,codec-parser-capsstops demandingconstrained-baseline, and H.264 negotiates on the viewer path even though it failed at startup. Folding that in cleared the verdict. Live on a real backend, before the fix: degraded at 16 s after the flow started, then healthy again 16 s after a viewer connected, while the codec list the sink answers with still had no H.264 in it. The first round's verdict is now final and later rounds are not instrumented at all.Root cause, for a human to file upstream
Isolated with no webrtcsink in the pipeline. With no profile requested,
Codec::parser_caps(force_profile=true)buildsvideo/x-h264, stream-format=avc, profile=constrained-baseline.vtenc_h264has no profile property and cannot emit it:Fails with
flow-return=(int)-4. The same line withprofile=baselineorprofile=mainpasses. Inside webrtcsink:force_profileisoutput_caps.is_any() && needs_encoding, and the startup discovery call site passes a hardcodedgst::Caps::new_any(), so nothing outside the element can relax it. gst-plugins-rs does not accept AI-written issues or merge requests, so this is written out for a human to file rather than filed from here. Standalone repro:Why no workaround ships
Ranking
x264encabovevtenc_h264restores H.264 to the offer but not the reason to want it: x264enc is software, and the hardware encode is the thing that is broken. Total CPU for 30 s of 1080p30, each encoder configured asconfigure_encoderconfigures it, two passes in both orders to control for order bias:vp9enc— today's Chrome fallbackx264enc— what the rank override selectsvtenc_h264— what is brokenx264enc was dearer than the vp9enc it would replace in both passes. vtenc sits at the no-encoder floor. So the rank change swaps one software encoder for another, process-globally across ingest and recording too, for no measurable server gain — consistent with the earlier rig finding that both arms landed at ~90% of a core for five viewers. Constraining vtenc's caps has no lever either: no profile property, and the demand comes from a hardcoded
Caps::new_any()inside the element.Setting
GST_PLUGIN_FEATURE_RANK=x264enc:300still makes sense for viewer-side decode and battery, but that is a deployment decision, not something to hardcode.Tests
backend/tests/whep_codec_discovery_test.rsbuilds a real WHEP Output block throughWHEPOutputBuilder, feeds raw video into a realwhepserversink, and asserts H.264 is reported as dropped. It forces the failure by removingh264parsefrom the registry rather than relying on Apple's encoder, so it runs on Linux CI — the same shape, not a different path:request-encoded-filterhas already fired for H.264 by the timebuild_parsererrors, so the block sees a codec attempted and never negotiated, exactly as vtenc produces.CODECSis aLazyLockbuilt once per process, so the registry surgery keeps this file to one test.Ran, green on this branch:
attach()stubbed to a no-op it fails; with it, passeswhep/codec_report.rs(verdict text, missing-encoder branch, no-discovery-yet, hold-across-a-later-round)STROM_REQUIRE_GST_PLUGINS=1: 684 passed, 0 failedcargo test --test openapi_testafter updating the snapshot; types suite;cargo clippy --all-targets -D warnings; frontend clippy forwasm32-unknown-unknownVerified on this machine but not shipped as a test: the unmodified block against the real
vtenc_h264reportsH.264 was dropped from the offer: vtenc_h264 failed codec discovery. CI runs tests on Linux only, so a vtenc assertion would guard nothing there and would start failing the day upstream fixes it.Also verified end to end on a running headless backend, which is what caught the viewer-round bug above: a flow of
videotestsrcinto a WHEP Output reportsdegradedinflow.block_healthand in awarn!line 16 s after start, staysrunning: truewithgst_state: Playing, and now holds that verdict across a connecting viewer for 50 s and six discovery rounds with a single health transition logged.Risk
BlockHealthStatusgains adegradedvariant, soopenapi.jsonis regenerated. Additive, but the contract check may flag a new enum value on a response type, and any client matching exhaustively on that enum needs the arm.backend/src/blocks/builtin/whep.rsis now 2758 lines and wants splitting likepipeline.rsandapp.rs. New code went intowhep/codec_report.rsrather than growing it further; the split is not in this PR.🤖 Generated with Claude Code