Skip to content

feat(gst): report blocks whose element chain has stopped - #786

Open
wagenet wants to merge 11 commits into
Eyevinn:mainfrom
wagenet:wagenet/whep-encoder-failure-visible
Open

wagenet wants to merge 11 commits into
Eyevinn:mainfrom
wagenet:wagenet/whep-encoder-failure-visible

Conversation

@wagenet

@wagenet wagenet commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A flow can report gst_state: Playing while a branch of it has stopped dead, with nothing in the API, the UI, or the default log. On a non-fatal flow return such as not-negotiated, GstAggregator flushes its sink pads and pauses its source pad task without posting to the bus, and encoders pause their output loop the same way. No element changes state, and the symptom shows far upstream of the failure. GstBaseSrc posts a flow error before pausing, so the gap is the mixer-to-output path.

Change

A two-second scan reports pads whose task is PAUSED on an element that is itself PLAYING, attributed to the owning block through the block_id:internal_id element names. The result is Flow::block_health ({ block_id, status, detail, causes }, binary ok/failed), StromEvent::BlockHealthChanged on a change of status or causes, an ERROR log on failure and INFO on recovery, and Failed in the flow list. running keeps its meaning ("reached Playing"); Flow::has_failed_block() gives the distinction.

causes is a list of structured reasons a client can match on. The scan here gives none; #1025 fills the first kind, a WHIP seat missing a medium. It lands here so the event does not change shape after it is on main. A kind the client does not know reads as unknown instead of failing the whole flow. An older client still cannot read a known kind that lacks a field or carries a new medium or fault value, so such a change must ship as a new kind; the type's docs say so.

A paused task is not reported when:

  • its element is below PLAYING (state transitions pause tasks);
  • the pad carries EOS, or its last flow return was EOS: a source that played out, or a pull-mode parser (filesrc ! flacparse) at the end of its file;
  • the pad is unlinked: a block output the flow leaves unwired. This relies on every Strom-built tee setting allow-not-linked; a raw user tee with every branch unwired is reported against the element feeding it;
  • the element has never been fed (no segment on any sink pad) and its own last push did not fail. vtenc parks its output task until the first frame, so every empty seat's Video Encoder read as failed in a live show;
  • the block has not looked stalled on two consecutive scans.

Evidence

Each claim below has a test that fails when its rule is removed. Revert checks were run on macOS at fb19da3e, and for the unknown kind at f2552a6d.

  • Aggregator stall reported while the pipeline says Playing: stalled_branch_is_reported_while_the_pipeline_still_says_playing ("stalled compositor was not reported").
  • Element below Playing: same test, last step.
  • EOS: a_source_that_reaches_eos_is_not_reported_as_failed, a_pull_mode_parser_at_end_of_file_is_not_reported_as_failed (fails only with both EOS checks removed for the first; with the flow-return check removed for the second).
  • Unlinked pad: an_unwired_output_branch_is_not_reported_as_failed.
  • Never fed: an_element_that_was_never_fed_is_not_reported_as_failed (portable stand-in for vtenc's pad state) and an_idle_video_encoder_is_not_reported_as_failed (real Video Encoder block on vtenc, macOS only). a_parked_task_on_an_element_that_was_fed_is_reported and the compositor test fail if "never fed" is dropped.
  • Causes re-announced: a_failed_block_is_announced_again_when_its_causes_change. Unknown kind: a_flow_with_a_cause_kind_this_build_does_not_know_still_reads.
  • aggregator_stall_is_silent_and_leaves_pipeline_playing pins the GStreamer premise (no bus error).
  • Unverified: the two-scan rule (removing it fails nothing), and the API and flow-list plumbing.

Tests

  • Rebased onto main 0216c1c1 on 2026-10-08 (by Claude). The only conflict was the event variant count. Re-run locally after the rebase: fmt, clippy, cargo test --workspace --lib, block_health_test, openapi_test. Revert checks below were not re-run; they cite the pre-rebase commits.
  • Linux CI (Check (Linux), STROM_REQUIRE_GST_PLUGINS=1): every health test except the macOS-only one ran and passed.
  • macOS CI on the fork (run 37577938619, fb19da3e): all health tests passed, including the vtenc one. Two Media Player bridge timing tests failed; test(mediaplayer): align bridge timing slack with the re-sync threshold #991, now in this branch's base, loosened their timing.
  • Locally on macOS at fb19da3e: health tests, block_health_test, openapi_test, clippy, fmt. At f2552a6d: the strom-types tests and openapi_test. cargo test --workspace passed (1246) at b1cba94c; not re-run since.
  • Not run: real GPU hardware, Linux hardware encoders.

For the reviewer

  • Every running flow is scanned, Flow and StromEvent gain fields, and the flow list's state text changes.
  • One ERROR line per failure, two when structured event logging is on (the event logger logs the failed event at error too).
  • Up to 4 s to report. The scan takes object locks only: no pad probes, nothing per buffer.
  • The two-scan rule can collapse after the server stalls for more than 2 s: the tokio interval fires missed ticks back to back, so a brief harmless pause at that moment can be reported.

Out of scope

  • Elements outside the start-time element list are not scanned: isolated pipelines (WHIP sessions, the Media Player's decode chain) and elements added while the flow runs (the recorder's per-track queues, dynamic auto-tees). An ok means "no stalled task among the scanned elements", not "this flow is passing data".
  • A Media Player with only video_out wired already posts a not-linked error on main, and now also shows as failed: Media Player with only video_out wired posts a not-linked error on its audio branch #1032.
  • Auto-inserted tees report under auto_tee_<src>, a block id absent from the flow graph.

Changes since review and this description were made by Claude on Peter's behalf.

🤖 Generated with Claude Code

@wagenet
wagenet marked this pull request as ready for review September 10, 2026 04:03

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Comment — the mechanism is right, the guard test is real and provably ran in CI, and the lifetime handling is correct. What holds it short of approval is reach: the scan walks only the flow pipeline, and the two block families that most often "stop dead while reporting Playing" run their own gst::Pipeline and are invisible to it.

Requested changes

  1. Say the bound in backend/src/gst/pipeline/health.rs's module doc: the scan sees only elements of the flow pipeline and their child bins, so a Media Player's decode chain and a WHIP ingest session are out of scope. Without that line an ok report reads as "this flow is passing data", which it does not mean.

Claims

Claim Verdict Evidence
"Both new tests ... neither skips" CONFIRMED neither backend/tests/block_health_test.rs nor the health.rs test module has a plugins_available guard or an early return, so a missing element fails rather than skips; .github/workflows/ci.yml:149 — STROM_REQUIRE_GST_PLUGINS: 1
The guard would fail if the detector were reverted CONFIRMED backend/src/gst/pipeline/health.rs:444-454 asserts comp reaches Failed through PipelineManager::start() and get_block_health() — production code, not a rebuilt topology, and nothing else in the tree sets that status
The task takes no strong pipeline reference CONFIRMED backend/src/gst/pipeline/health.rs:131 — let weak_elements: Vec<(String, gst::glib::WeakRef<gst::Element>)> = self, and it is stopped on both teardown paths: backend/src/gst/pipeline/lifecycle.rs:208 and backend/src/gst/pipeline/mod.rs:252 — self.stop_block_health_task();
"a branch of its pipeline has stopped dead" is now reported CONFIRMED, coverage BOUNDED the bound is separate pipelines: backend/src/blocks/builtin/mediaplayer/bridge.rs:28 — let pipeline = gst::Pipeline::builder().name(&pipeline_name).build(); and backend/src/blocks/builtin/whip.rs:641 — let session_pipeline = gst::Pipeline::builder(). A sibling pipeline is not a child bin, so iterate_recurse() never reaches either
No false positive on topologies other than the tested one UNVERIFIED the heuristic is validated against one graph (videotestsrc ! compositor ! capsfilter ! fakesink); it is applied to every element of every running flow, and a single false hit logs ERROR and turns the whole flow red — frontend/src/app/rendering.rs:933 — "Failed"

Diagnosis — Root cause is stated correctly, and both exclusions are defended by a test rather than asserted — backend/src/gst/pipeline/health.rs:52 — if element.current_state() != gst::State::Playing { for the first, and its own test for the EOS one, which is the exclusion that would otherwise have made this unusable. CLOSED loop: the healthy path costs one lock-only scan every 2s and emits nothing.

Worth putting the separate-pipeline bound in the PR body too, because it inverts the risk you flagged. Having no WHIP rig does not matter here — the scan never enters a WHIP session bin at all. What is missing is reach, not coverage.

Radius — GLOBAL: every running flow is scanned, Flow gains an API-visible field, a StromEvent variant is added, and the flow list's colour and state text change for all flows. Contract work is complete: BlockHealth and BlockHealthStatus both carry ToSchema, openapi.json is regenerated, API Contract Check passes, no existing variant touched.

Tests & CI — run 34436702775 at 4d7c2f17: Check (Linux), both Build (Linux) jobs, Check & Build (WASM) and API Contract Check all pass. macOS/Windows skipping; no platform-specific code, so not a gap.

Confidence: HIGH

@srperens srperens mentioned this pull request Sep 10, 2026
@wagenet

wagenet commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Bound documented in fcf70e6 — module doc plus a Scope section in the PR body, both stating that ok means "no stalled pad task among this flow pipeline's elements", not "this flow is passing data".

The catch ran deeper than the doc line. The guard that skips elements below PLAYING justified itself with "a WebRTC session bin mid-setup" — those live in a session pipeline and are never scanned, so the example could not occur. Replaced with a webrtcbin added for a new WHEP consumer, which is in-pipeline.

On the UNVERIFIED false-positive claim: you were right, and it was not hypothetical. I built the topology you were implicitly asking about — whip_input → vision_mixer → whep_output, one headless publisher — and a healthy flow reported this seconds after start:

mixer: failed — pad task paused on mixer:queue_mv_out:src

queue_mv_out:src is the mixer's multiview_out external pad. Leave it unwired and the queue takes not-linked from its loop and parks the task, element still PLAYING. Any flow using only the program output would have shown a permanently failed mixer, which is worse than the problem this PR set out to fix. Fixed in 7a01fd2 by requiring a linked peer, with a regression test that reproduces it without a rig (tee, one queued branch left unconnected) and fails when the check is reverted.

After the fix, on the same rig: 30/30 samples clean over five minutes with media confirmed flowing, and 24/24 across a publisher disconnect/reconnect cycle, which exercises WHIP session teardown and setup.

That makes three of the four exclusions this needed — sub-PLAYING, EOS, not-linked — things code reading alone did not surface. Worth stating plainly since it bears on how much the remaining UNVERIFIED residue is worth: a topology where a legitimately paused task persists on a PLAYING element, linked, having pushed no EOS. I have not found one, which is not the same as there being none.

Still not fired on a production failure. Six fresh-server trials in the known-bad encoder configuration all ran clean, so the intermittent WHIP startup failure did not reproduce on a 1-seat rig and remains unattributed.

Keeping sub-pipeline reach as follow-up. Attribution already works — whip.rs:633 names session elements {instance_id}:whipserversrc_{uuid}, so owning_block maps them unchanged and only the task's element sourcing would move from a start-time snapshot to a live query; nothing in the types, event, contract or UI shifts. The larger reason is that sessions bridge through appsink→appsrc, so a stalled session presents as absence of data rather than a flow return — scanning them would catch a stall internal to one, but not the data-absence case, which wants a different signal.

— Claude

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Comment — supersedes my review of 4d7c2f17. The token is unchanged but the reason is not: my one requested change is done, the new exclusion is correct and covers more than the PR body claims, and what holds this short of Approve is now radius alone. Everything settled at 4d7c2f17 — the separate-pipeline bound, the WeakRef lifetime, the EOS exclusion, the guard test — stands; do not re-raise it.

Requested changes

  1. In backend/src/gst/pipeline/health.rs, state the invariant the new exclusion rests on rather than only its example. An unwired output parks on a peerless pad only because every tee this repo builds sets allow-not-linked=true. A tee added without it would send not-linked upstream instead, pausing an aggregator source pad that does have a peer — a false positive this exclusion cannot catch, and one that would be reported against the wrong block.

Claims

Claim Verdict Evidence
The change requested at 4d7c2f17 is done CONFIRMED backend/src/gst/pipeline/health.rs:14 — //! Scope: the scan reaches the flow pipeline's own elements and the bins beneath
"the vision mixer's multiview_out does exactly this" — the parked pad really has no peer CONFIRMED backend/src/blocks/builtin/vision_mixer/elements.rs:133 — .property("allow-not-linked", true), so tee_mv absorbs the not-linked and the park stays on the unlinked queue_mv_out source pad instead of travelling up to mv_comp
The exclusion covers the class, not one block CONFIRMED all 20 ElementFactory::make("tee") sites in backend/src set the property, the auto-inserted one included — backend/src/gst/pipeline/construction.rs:409 — tee.set_property("allow-not-linked", true);
Runtime unlinking does not blind the detector CONFIRMED the only runtime unlink in a flow pipeline is backend/src/gst/thumbnail_tap.rs:472 — let _ = tee_src_pad.unlink(&peer); — on a tee source pad, which runs no task in push mode
The new guard executed at this head CONFIRMED Check (Linux) job 102953843607 at 7a01fd22: test gst::pipeline::health::tests::an_unwired_output_branch_is_not_reported_as_failed ... ok

Diagnosis — that a real false positive surfaced after the last review is exactly what the UNVERIFIED "no false positive on other topologies" row anticipated, and this commit closes the systematic instance of it rather than the one reported case: every optional block output in this tree dangles behind an allow-not-linked tee, so every one of them parks on a peerless pad. The residual is narrower than before — an element that legitimately parks a linked task — and I could not name one in this tree. Coverage of the original defect is unchanged and still BOUNDED to the flow pipeline.

Radius — GLOBAL, unchanged, and the only bar left to Approve: every running flow is scanned, Flow gains an API-visible field, a StromEvent variant is added, and the flow list's state text changes for all flows. Whether that reach is acceptable for a first cut is a human call, not a defect I found.

Tests & CI — run 34501704947 at 7a01fd22: Check (Linux), both Build (Linux) jobs, Check & Build (WASM) and API Contract Check all pass. All five health tests appear as ok in the Check (Linux) log, so none skipped silently. Build (macOS) and Build (Windows) skipping; no platform-specific code here, so not a gap.

Confidence: HIGH

wagenet added a commit to wagenet/strom that referenced this pull request Sep 12, 2026
Brings in the block health scan (Eyevinn#786) and the unformed-link detector
(Eyevinn#796). The per-medium slot liveness work that follows reports through
that scan's `health.rs`, so it cannot build without it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wagenet added a commit to wagenet/strom that referenced this pull request Sep 27, 2026
The per-medium report reaches operators through the block health scan that Eyevinn#786 adds and Eyevinn#796 extends.
@wagenet
wagenet force-pushed the wagenet/whep-encoder-failure-visible branch from 7a01fd2 to 83326c4 Compare September 28, 2026 15:43
@srperens

Copy link
Copy Markdown
Collaborator

Heads-up: #789 (structured event logging) is on main now, and it makes this PR stop compiling once main is merged in. It adds three StromEvent accessors in types/src/events.rs with no wildcard arm, so every new variant has to be listed in each.

I merged main into this branch locally to check what it takes. It builds and cargo test --workspace passes after these changes. I haven't pushed anything, so this is left to you.

1. The accessors. BlockHealthChanged fires only when a block's status changes, so it is not high-frequency. I put each arm next to PipelineWarning:

// event_type()
StromEvent::BlockHealthChanged { .. } => "BlockHealthChanged",
// flow_id(), in the `=> Some(*flow_id)` group
| StromEvent::BlockHealthChanged { flow_id, .. }
// is_high_frequency(), in the `=> false` group
| StromEvent::BlockHealthChanged { .. }

2. The wire-tag test fixture (event_accessor_tests::one_of_each_variant). This list isn't checked by the compiler, only by the hand-kept count, so bump variant_count from 44 to 45:

StromEvent::BlockHealthChanged {
    flow_id: id,
    block_id: "b0".to_string(),
    status: BlockHealthStatus::Failed,
    detail: Some("src".to_string()),
},

3. A conflict in types/src/flow.rs. Main added Flow::partially_unwired_block_inputs() in the same spot where this PR adds failed_blocks() / has_failed_block(). Keep both: put main's method inside impl Flow next to yours, and keep BlockHealthStatus / BlockHealth after the impl block.

#796, #792 and #805 all build on this branch and add no variant of their own. Once this one is updated, rebasing or merging the stack onto it clears the accessor errors for all of them. #805 also has its own conflicts with main, in blocks/builtin/whip.rs and blocks/mod.rs.

Optional: log_strom_event() in backend/src/event_logging.rs logs everything except pipeline and TAMS errors/warnings at info. If you want BlockHealthChanged { status: Failed } to come out as a warn! in the structured logs, that is the place to add it.

@wagenet
wagenet force-pushed the wagenet/whep-encoder-failure-visible branch from 83326c4 to 2502c68 Compare September 29, 2026 13:59
@wagenet

wagenet commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for checking the merge. Items 1–3 were already on the branch: the force-push on 2026-09-28 rebased onto a main that included #789, with the three accessor arms, the fixture entry and `variant_count = 45`, and both `Flow` methods kept. The comment probably went out against an older fetch.

Since then:

  • Rebased onto current main (`f1a46dd6`, 0.6.11). No conflicts.
  • Took the optional one, in `2502c68d`. `log_strom_event` now logs `BlockHealthChanged { status: Failed }` with its own arm, adding `strom.block.id` and the detail as `error.message`. It logs at `error`, not `warn`, because the health scan's plain log for the same transition is already `error!`, and one condition should not get two levels. `Ok` (resumed) still falls through to `info`.

Ran locally on macOS: `cargo fmt --check`, `cargo clippy --workspace --all-targets -D warnings`, `cargo test --workspace`, 1077 passed, 0 failed, including the new smoke test and `event_type_matches_serde_wire_tag`.

#796, #792 and #805 still need a rebase onto this.

— Claude

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Comment. This replaces my Comment at 7a01fd22 (review 5179570599). The token and the reasons both stand. git range-diff shows commits 1–4 carried over by a rebase onto f1a46dd: 2–4 are identical, and 1 differs only by the StromEvent accessor arms that #789 now requires. Commit 5 (2502c68) is the only new work. It is correct and small. My one requested change from 2026-09-11 is still open, and radius is still GLOBAL.

Requested changes

  1. Still open from my earlier review, and now with a concrete route. In backend/src/gst/pipeline/health.rs, state the invariant the unlinked-pad exclusion rests on. The doc still gives only the example: backend/src/gst/pipeline/health.rs:63 — /// An unlinked pad has no branch to report on. A queue feeding a block output. All 20 ElementFactory::make("tee") sites in backend/src still set allow-not-linked. A user flow built from raw elements does not: backend/src/gst/pipeline/construction.rs:317 — let element = gst::ElementFactory::make(&element_def.element_type) creates a tee with its defaults. When every branch of such a tee is unconnected, the not-linked travels up to a source pad that has a peer (backend/src/gst/pipeline/health.rs:77 — && pad.peer().is_some()). That source is then reported as a failed block, at error, twice. Documenting the bound is enough. Handling the case is optional.

Claims

Claim Verdict Evidence
Commits 1–4 are unchanged apart from the rebase CONFIRMED git range-diff 95d7ccb..7a01fd2 f1a46dd..2502c68d: 2–4 =. The changes to 1 are the event_type/flow_id/is_high_frequency arms, the fixture entry and variant_count = 45
A failed health event now logs at error in the structured log CONFIRMED backend/src/event_logging.rs:68 — status: BlockHealthStatus::Failed, in its own arm ahead of the general case. Ok still falls through to info. Guard: logs_block_health_failure_without_panicking, which only proves the arm does not panic
Levels agree between the two logs CONFIRMED backend/src/gst/pipeline/health.rs:214 — tracing::error!( for the same transition. So when event logging is enabled, one failure produces two error lines, by design

Overlaps. #796, #792 and #805 each add a StromEvent variant. Whichever lands after this one must also list its variant in the three accessor matches, which have no wildcard arm, and bump variant_count. The author already says so on this thread.

Radius. GLOBAL, for the same reasons as before: every running flow is scanned, Flow gains a field, a StromEvent variant is added, and the flow list's state text changes. Whether that reach is acceptable for a first cut is still a human call.

Tests & CI. At 2502c68d, API Contract Check, Check & Build (WASM) and Static JS Tests pass. Check (Linux) job 109442905033 passed. All four health::tests, logs_block_health_failure_without_panicking and block_health_test (no skip guard) show ok. Build (Linux ARM64) passed. Build (Linux x86_64) was still running when this was posted. macOS and Windows show skipping, and no platform-specific code is involved.

Confidence: HIGH

@wagenet

wagenet commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Documented in dc881f67. The stalled_pad_on doc now says the unlinked-pad exclusion depends on every Strom-built tee setting allow-not-linked. It also says what breaks without that: a raw tee element in a user flow, with all its branches unconnected, sends not-linked upstream, and the element feeding it is reported as failed.

I did not handle the case. Telling "every branch below this tee dangles" apart from a real stall needs a walk downstream from the paused pad. That is more than a doc-level bound calls for, and it can be a follow-up if raw-tee flows show up in practice.

Doc-only change. The pre-commit fmt and clippy checks passed. No tests re-run.

— Claude

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Comment. Supersedes my Comment at 2502c68d (review 5353862911); the verdict stands. The only new commit is dc881f6, seven doc lines, and it does what I asked twice. Everything else I settled at 2502c68d and 7a01fd22 stands at this head: compare 2502c68...dc881f6 is ahead=1, behind=0, one file. Radius is the one thing left, and that is a human call, not a defect.

Claims

Claim Verdict Evidence
The tee invariant is now stated, with its failure case CONFIRMED backend/src/gst/pipeline/health.rs:68 — /// That exclusion holds only because every tee Strom builds sets
That invariant is true at this head CONFIRMED all 20 ElementFactory::make("tee") sites in backend/src still set allow-not-linked, the auto-inserted one included: backend/src/gst/pipeline/construction.rs:411 — tee.set_property("allow-not-linked", true);
Main has not drifted under it CONFIRMED 11 commits behind main. In types/src/events.rs and backend/src/gst/pipeline/, the only change on main is test-only, in pipeline/mod.rs's test module. It does not touch the PipelineManager fields or the Drop arm this PR adds

Overlaps. #755 now joins #796, #792 and #805 in adding StromEvent variants (StingerStarted/Completed/Failed, variant_count = 47). Because the three accessor matches have no wildcard arm, whichever of these lands second must add the other's arms and re-bump variant_count. Neither diff shows that break. Nothing open changes health.rs, and nothing else decides how flow health is reported.

Radius. GLOBAL, as before: every running flow is scanned every 2s, Flow gains an API field, StromEvent gains a variant, and the flow list's state text changes for every flow. That stays the only bar between this PR and Approve. Accepting it for a first cut is the maintainer's choice.

Tests & CI. Run 36637376071 at dc881f6: Check (Linux), Build (Linux x86_64), Build (Linux ARM64), Check & Build (WASM), API Contract Check and Static JS Tests all pass. A comment-only commit cannot change what the health tests exercise, and they passed at 2502c68d. I did not re-read this job's log. macOS and Windows show skipping. No platform-specific code is involved.

Confidence: HIGH

A pipeline branch can stop dead while the flow still reports Playing.
When a downstream push fails with a non-fatal flow return such as
not-negotiated, GstAggregator marks its sink pads flushing and pauses its
source pad task, and video encoders pause their output loop. Neither
posts to the bus, and pausing a pad task is not a state change, so every
element stays PLAYING and nothing surfaces through gst_state or the API.
GstBaseSrc is the exception - it posts a flow error before pausing - so
until now only source-side failures were visible.

Scan the running pipeline every two seconds for pads whose task is paused
on an element that is itself PLAYING, attribute each to its owning block
via the block_id:internal_id element namespace, and expose the result as
Flow::block_health. Log once per transition and emit BlockHealthChanged
so the failure is not limited to GST_DEBUG output. The flow list marks an
affected flow as Failed rather than Running.

Elements held below PLAYING are skipped: a WebRTC session bin paused
mid-setup has paused pad tasks legitimately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wagenet and others added 10 commits October 8, 2026 15:49
gst_base_src_loop pauses its pad task on the way out for every reason,
end of stream included, and leaves the element in PLAYING. Any flow with
a finite source would therefore report that block as failed forever once
it played out. Skip pads carrying the EOS flag: a pad that has seen EOS
is finished by definition, and the stall this detects pushes no EOS.

Require a block to look stalled on two consecutive scans before reporting
it. Pad tasks pause briefly for legitimate reasons - a flushing seek, a
dynamic bin being linked in - and a single sample turned those into a
logged error and a UI flip. Costs one poll interval of latency.

Drop blocks that disappear from the scan, rather than leaving them in the
failed set where no later iteration could clear them.

Handle BlockHealthChanged in the frontend. block_health rides on the Flow
payload, and nothing refetched the flow on this event, so the indicator
only appeared if the user happened to refresh.

The reported detail said "pad task stopped" while the condition detected
is paused; Stopped is explicitly not reportable here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scan walks the flow pipeline and the bins beneath it. Blocks that
build an isolated gst::Pipeline - WHIP ingest sessions, a Media Player's
decode chain - are siblings of that pipeline rather than children, so
iterate_recurse never enters them. Without that written down an Ok reads
as "this flow is passing data", which it does not mean.

The guard that skips elements below PLAYING cited a WHIP session bin as
its example. Those live in a separate pipeline and are never scanned;
a webrtcbin added for a new WHEP consumer is one that actually occurs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A queue whose source pad is left unconnected takes not-linked from its
loop and parks the task there for good, with the element still PLAYING.
Blocks build optional outputs this way - the vision mixer's multiview_out
is one - so any flow wiring only the program output reported its mixer as
permanently failed.

Found on a real WHIP -> vision mixer -> WHEP rig, where a healthy flow
reported "pad task paused on mixer:queue_mv_out:src" seconds after start.
An unlinked pad has no branch to report on, so require a peer.

The regression test reproduces it without a rig: a tee with one queued
branch left unconnected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A block that stops passing data is logged at error by the health scan;
the structured event log now matches it instead of logging at info.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vtenc parks its output task as soon as it reaches PLAYING and starts it
on the first frame. A Video Encoder block on an empty seat therefore sat
with a paused src task, and the health scan reported the block failed
for as long as the seat stayed empty.

Skip a paused task when the element has never received data and its own
last push did not fail. A segment must precede the first buffer, so no
segment on any sink pad means no data has arrived. The last flow return
is not enough on its own: an aggregator that fails to renegotiate pauses
without pushing and keeps a last flow of Ok, and the existing compositor
stall test fails without the segment check.

No element available on Linux CI parks a task before data, so the
portable test gives an unfed identity a self-pausing task to produce the
same pad state, with a fed counterpart that must still be reported. A
macOS-only test runs the real Video Encoder block on vtenc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A client that wants to act on a failed block (show "Guest 2: no audio",
say) had only the free-text detail to go on. BlockHealth and
BlockHealthChanged now carry an optional list of causes, a tagged enum a
client can match on. The first kind is a WHIP Input seat missing one
medium, with where it stops: the publisher stopped sending it, never sent
it, or it arrives and none of it comes out of the slot's chain.

A list, because one WHIP block has a seat per slot and each can lose a
medium on its own. For the same reason the health task now announces a
failed block again when its causes change, not only when its status
does; the detail is not compared, since it may carry a duration.

The existing stalled-pad scan gives no causes, so its reports are
unchanged on the wire.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A parser reading a file in pull mode (flacparse, h264parse and the other
baseparse parsers from filesrc, or rawaudioparse) runs its pad task on
its sink pad and pauses it at the end of the file. The EOS goes out
through the source pad, so the paused pad never carries the EOS flag,
and a flow built from plain elements showed the parser as failed for as
long as it kept running.

Treat a paused pad whose last flow return is EOS as finished, as the EOS
flag already is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BlockHealthCause tells clients to ignore a kind they do not know, but the
type itself rejected one: a flow or BlockHealthChanged event carrying a
newer kind failed to deserialize as a whole. The frontend is built on
these types, so a browser tab left on an older build would lose the flow
list as soon as a newer server reported a new kind of cause.

Read an unknown kind as BlockHealthCause::Unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An older client reads an unknown kind as Unknown, but cannot read a known
kind that lacks a field or carries a medium or fault value it does not
know. State that such changes ship as a new kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet
wagenet force-pushed the wagenet/whep-encoder-failure-visible branch from 2c7458f to d5a1c5f Compare October 8, 2026 23:03
wagenet added a commit to wagenet/open-live that referenced this pull request Oct 9, 2026
Strom's block health scan (Eyevinn/strom#786) reports a block whose
element chain stops passing data while the pipeline stays PLAYING, as a
BlockHealthChanged event on /api/ws and a block_health list on the flow.
A guest whose browser stops sending audio shows up there and nowhere
else, so the studio had no way to notice it.

The meter relay, which already holds each live production's Strom
socket, now maps BlockHealthChanged for the production's guest input
blocks (b-input-<pad>-<suffix>) to the guest seat and broadcasts
GUEST_HEALTH { mixerInput, blockId, status, detail?, causes? }. Each
time that socket opens it re-reads the flow's block_health and
broadcasts every guest seat, skipping any seat that had a live event
while the read was in flight, so a change during a reconnect is not
lost. The connect snapshot and reactivation reinit also send one
GUEST_HEALTH per guest seat from block_health, so a reconnecting studio
sees a failure that is still going on and a stale one is cleared.
Strom's optional structured causes are passed through unchanged while
failed. A malformed block_health is skipped and a status other than
ok/failed is dropped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
wagenet added a commit to wagenet/strom that referenced this pull request Oct 9, 2026

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Comment. This replaces my review at dc881f6, and the verdict is still Comment. The six commits I reviewed before came through the rebase onto 0216c1c unchanged, apart from variant_count, which is now 48 → 49 to match main. The five new commits are correct and each has a test that ran in CI. Two decisions are left for the maintainer: the radius, as before, and a new design question below.

For the maintainer

  1. Ship WhipMedium here, or with its producer? types/src/flow.rs:603 — WhipMedium { has no producer in this PR. The only one is draft #1025, which is stacked on this branch. The body gives the reason: "so the event does not change shape after it is on main". The empty causes list and the Unknown fallback (types/src/flow.rs:614 — #[serde(other)]) already meet that aim. With those in place, adding a kind later is non-breaking: an older client reads it as unknown. Once WhipMedium is on main, the new doc comment (d5a1c5f) freezes its slot/medium/fault fields: any change has to ship as a new kind. Merging this PR therefore fixes #1025's wire shape before #1025 has been reviewed. You can (a) accept that shape now, or (b) move WhipMedium, HealthMedium and MediumFault into #1025 and keep only causes plus Unknown here.
  2. Radius GLOBAL, unchanged. Every running flow is scanned, and Flow and StromEvent gain fields.

Claims

Claim Verdict Evidence
An unfed encoder's parked task is no longer reported, and a fed stall still is CONFIRMED backend/src/gst/pipeline/health.rs:102 — && !(never_fed && pad.last_flow_result().is_ok()). never_fed needs at least one sink pad (backend/src/gst/pipeline/health.rs:94 — let never_fed = sink_pads().next().is_some()), so sources are unaffected. an_element_that_was_never_fed_is_not_reported_as_failed, a_parked_task_on_an_element_that_was_fed_is_reported and the compositor stall test all ok in job 113584275591
A pull-mode parser at EOF is not reported CONFIRMED a_pull_mode_parser_at_end_of_file_is_not_reported_as_failed ... ok, same job
An unknown cause kind no longer breaks reading a whole Flow CONFIRMED flow::tests::a_flow_with_a_cause_kind_this_build_does_not_know_still_reads ... ok in the shared-types step
A failed block is announced again when its causes change CONFIRMED backend/src/gst/pipeline/health.rs:282 — Some(Transition::CausesChanged) => {. detail is deliberately not compared. Unit test ok
A FLUSH_STOP clears the Segment sticky event, so an element stalled between a flush and its next segment reads as never fed EXTERNAL GStreamer pad semantics. SPECULATIVE (not verified): this needs a stall with no new data after a seek, so it is a narrow window. It would hide one, not invent one

Overlaps. #1025 (draft) is stacked on this PR and owns the WHIP cause design (point 1). #1057 edits types/src/events.rs but adds no variant, so the overlap is textual only. #755, #792 and #805 are closed, so the variant_count race in my last review is gone.

Tests & CI. Run 37857238109 at d5a1c5f: Check (Linux) is green, with backend, shared-types and frontend test steps all success. All eight non-macOS health::tests ran and passed, and block_health_test ran. Both Linux builds, WASM, API Contract Check (openapi.json updated) and Static JS Tests are green. an_idle_video_encoder_is_not_reported_as_failed is #[cfg(target_os = "macos")], so it did not run here. It is the only real-vtenc evidence, and that comes from the author's fork run. The portable stand-in covers the same rule on Linux, so merging on Linux-green is a reasonable call.

Confidence: HIGH

wagenet added a commit to wagenet/open-live that referenced this pull request Oct 19, 2026
Strom's block health scan (Eyevinn/strom#786) reports a block whose
element chain stops passing data while the pipeline stays PLAYING, as a
BlockHealthChanged event on /api/ws and a block_health list on the flow.
A guest whose browser stops sending audio shows up there and nowhere
else, so the studio had no way to notice it.

The meter relay, which already holds each live production's Strom
socket, now maps BlockHealthChanged for the production's guest input
blocks (b-input-<pad>-<suffix>) to the guest seat and broadcasts
GUEST_HEALTH { mixerInput, blockId, status, detail?, causes? }. Each
time that socket opens it re-reads the flow's block_health and
broadcasts every guest seat, skipping any seat that had a live event
while the read was in flight, so a change during a reconnect is not
lost. The connect snapshot and reactivation reinit also send one
GUEST_HEALTH per guest seat from block_health, so a reconnecting studio
sees a failure that is still going on and a stale one is cleared.
Strom's optional structured causes are passed through unchanged while
failed. A malformed block_health is skipped and a status other than
ok/failed is dropped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants