Repository navigation
fix(whip): judge a session by the media it produces, not what arrives - #756
Conversation
|
Note for whoever merges this alongside #752. #752 replaces the watchdog's sleep-then-check with Done that way on my fork's main; its two tests carry over with the signature. Happy to |
|
Noted as a draft, so I am holding off on a full review until it is marked ready for |
be15584 to
a81cd15
Compare
|
This branch is based on the pre-review #753, and I think it needs a rebase before it can be reviewed on its own merits. The base
So four things from #753 are missing above it:
It is not only a textual conflictA test merge of this branch onto This PR makes the incoming session clear the slot's stamp: pub fn new(epoch: Instant, output: Arc<ActivityStamp>) -> Self {
output.reset();with the reasoning that a newcomer must not be judged on its predecessor's frames. The commit message puts the residual window at "buffers left in flight ... only ever makes the new session look healthier than it is", which is true of a clean leave. Takeover is not a clean leave. On #753's current head the gate closes it: the displaced session's I have read this rather than run it, so treat the window's width as unmeasured — but the shared appsrc and the missing gate are both checkable in the source. What I would suggestRebase onto For the record, the rest of the review is not affected by any of this. The BUFFER probe added here stays inside the per-buffer budget — one One thing worth deciding separately: |
|
Correction to the rebase target in my comment above: it said "rebase onto #753 has since been squash-merged to #758 landed just before it ( |
a81cd15 to
5e4f51e
Compare
srperens
left a comment
There was a problem hiding this comment.
Verdict: Comment — right mechanism, real tests that genuinely executed. One residual: judging a seat by its slot's output makes a stuck consumer cost a healthy publisher its seat on a ~15 s cycle, filed in the body as "costs a reconnect, not media".
Requested changes
- Name the fault direction when a seat is reaped. Both reasons read "without usable media" whichever stamp froze —
backend/src/whip_session_manager.rs:770—"displaced by a new client after {} ms without usable media",. Whereingressis still moving andoutputis not, say so there and in the watchdog'sinfo!: it is the only signal an operator gets that the flow's consumer, not the publisher, is the suspect. - Restate the "Flow-wide stalls" trade-off — one reconnect per ~15 s while the consumer is stuck, not one reconnect.
Claims
| Claim | Verdict | Evidence |
|---|---|---|
| The tee probe respects CLAUDE.md's BUFFER-probe rules | CONFIRMED |
backend/src/blocks/builtin/whip.rs:311 — stamp.touch(); is the whole callback: one clock read, relaxed atomics, no lock or allocation. It captures an Arc<ActivityStamp>, no Element/Bin/Pipeline, so no reference cycle |
| The new tests run in CI on stock packages | CONFIRMED |
Check (Linux) job 105241140495 log: all seven named tests ran and passed, incl. the_slot_stamp_follows_media_out_of_the_decode_chain_not_into_it ... ok. No skip guard |
| A flow-wide stall costs "a reconnect, not media" | CONTRADICTED |
The slot appsrc leaks rather than back-pressuring the session — backend/src/blocks/builtin/whip.rs:444 — .leaky_type(gst_app::AppLeakyType::Downstream) — so ingress keeps stamping while output freezes, idle() takes the max at whip_session_manager.rs:399 — Some(ingress_idle.max(output_idle)) — and the reconnecting publisher lands on the same blocked chain and is reaped again after DECODE_GRACE + INACTIVITY_TIMEOUT |
| Rig trials | EXTERNAL |
Not settleable here; assumed as stated |
I did not run the reverts behind "each new test fails with its fix reverted". Traced statically, two hold: receiving_but_never_usable and receiving_but_stalled tick ingress throughout, so an ingress-only idle() returns ~0 and both displacement assertions fail.
Diagnosis — Correct layer: the appsink is upstream of decodebin in the isolated session pipeline and can only see bytes arriving; the tee sink pad is the first point past the whole slot chain. Coverage is BOUNDED — one stamp serves a slot's audio and video tees, so frozen video beside live audio still reads live, correctly deferred to #792.
The grace holds unconditionally rather than only when output is frozen — whip_session_manager.rs:389 — let past_grace = self.ingress.since_first()?.checked_sub(DECODE_GRACE)?; — which stops a predecessor's in-flight frames starting the clock on a newcomer. That is the subtle part and it is right. DECODE_GRACE (5 s) and INACTIVITY_TIMEOUT (10 s) sit in different modules, nothing enforcing the ordering.
Radius — SHARED: two files, but it changes session lifetime policy for every WHIP Input seat, in both decode modes — the probe sits outside the if decode branch, so a non-decoding endpoint stamps straight off its appsrc and behaves as before.
Overlaps — #792 and #805 both contain this PR's commits and are stacked on it, so this is the bottom of the stack and lands first; neither is left for this merge to settle. No open issue owns the stalled-consumer question.
Tests & CI — Green at cb10dc83. macOS/Windows skipping, but no platform code here. Canary pipeline_lifecycle_test.rs ran, 3 passed — it matters because this installs a long-lived pad probe.
Confidence: HIGH
cb10dc8 to
777a6bc
Compare
|
Rebased onto 1. Fault direction in the reap message. Writing the test for it turned up something I had not expected: the two stamps keep separate epochs and store whole milliseconds, so a publisher drop — which freezes both at once — reads either way round depending on sub-millisecond truncation. A bare 2. The flow-wide stall trade-off. Restated in the body, and it is worse than "per ~15 s": the slot appsrc is On Re-ran on this head: — Claude (via @wagenet) |
srperens
left a comment
There was a problem hiding this comment.
Verdict: Request changes — both requested changes from my review at cb10dc8 are in, and CI at this head is green with all nine named tests actually executed. One new claim is contradicted: STALL_SIDE_MARGIN does not label only, it changes the idle duration idle() returns. This supersedes that review; it is COMMENTED and cannot be dismissed, so this is the current verdict.
Requested changes
- Make the label label-only.
backend/src/whip_session_manager.rs:440—Some((ingress_idle, StallSide::Ingress))— returnsingress_idlewhere the pre-margin code returned the max. Returningress_idle.max(output_idle)there and the body's sentence becomes true; no existing test asserts a duration inside that window, the threeidle_detailassertions read only the side (backend/src/whip_session_manager.rs:1595). The alternative is to correct the sentence and state the bound.
Claims
| Claim | Verdict | Evidence |
|---|---|---|
STALL_SIDE_MARGIN "labels only; the idle duration and the reap decision are unchanged" |
CONTRADICTED |
backend/src/whip_session_manager.rs:437 — if output_idle > ingress_idle + STALL_SIDE_MARGIN { — so whenever ingress_idle <= output_idle <= ingress_idle + margin, line 440 returns ingress_idle and not the max. Bounded by backend/src/whip_session_manager.rs:532 — const STALL_SIDE_MARGIN: Duration = Duration::from_millis(100); — one whole backend/src/whip_session_manager.rs:553 — const TAKEOVER_POLL: Duration = Duration::from_millis(100); — so a takeover can be deferred a poll |
The margin test fails with STALL_SIDE_MARGIN dropped |
CONFIRMED |
Traced, not run: the near-tie case backdates output 1 ms staler than ingress, so at a zero margin line 437 takes the Output branch and the StallSide::Ingress assertion fails |
| The new tests run in CI on stock packages | CONFIRMED |
Check (Linux) job 106837320851 log: all nine named tests ... ok, no skip guard, including a_reaped_session_names_which_side_of_the_chain_stopped and the_slot_stamp_follows_media_out_of_the_decode_chain_not_into_it |
| Rig trials (pre-rebase) | EXTERNAL |
Not settleable here; assumed as stated, and the commits since only lengthen protection under 5 s and change reap wording |
Diagnosis — Unchanged from cb10dc8. The probe callback is byte-identical at this head (backend/src/blocks/builtin/whip.rs:311 — stamp.touch();), so the BUFFER-probe row stands CONFIRMED in that review rather than being rebuilt. My earlier CONTRADICTED row, "costs a reconnect, not media", is closed: the body now states the ~7 s / ~15 s repeat cycle.
One cleanup, not blocking: backend/src/whip_session_manager.rs:415 — pub fn idle(&self) -> Option<Duration> { — has no production caller left, every live path goes through idle_detail (backend/src/whip_session_manager.rs:862). It is pub, so nothing warns.
Radius — SHARED, as argued at cb10dc8: two files, but session lifetime policy for every WHIP Input seat. The margin narrows that to a bounded timing shift.
Overlaps — #792 and #805 both carry this PR's commits, so this is the bottom of the stack and lands first; neither is left for this merge to settle. #793 also edits whip_session_manager.rs, but adds an independent accessor and owns no design here. No open issue's Ask is answered by merging this.
Tests & CI — Green at 777a6bc; macOS/Windows skipping, no platform code here. Canary pipeline_lifecycle_test.rs ran, 3 passed. Both files are past CLAUDE.md's 1500-line threshold (1705 and 2713); the body commits to splitting them separately before #792, which is the right order.
Confidence: HIGH
|
Both points are addressed in Label only, now for real. Both branches return
Re-ran on this head: — Claude (via @wagenet) |
srperens
left a comment
There was a problem hiding this comment.
Verdict: Approve — the one CONTRADICTED row from my review at 777a6bc is closed. idle() now returns the staler stamp whichever side it names, so the body's "it labels only" is true and the reap decision is back to what it was before the margin landed. This supersedes that review; its Request changes verdict no longer stands and I have dismissed it.
Claims
| Claim | Verdict | Evidence |
|---|---|---|
STALL_SIDE_MARGIN moves the label only, never the idle duration |
CONFIRMED |
backend/src/whip_session_manager.rs:437 — Some((ingress_idle.max(output_idle), side)) — the side is chosen separately and the duration is unconditionally the max. Both consumers take the duration from that tuple: backend/src/whip_session_manager.rs:804 — ` .filter( |
| The new assertion fails with that fix reverted | CONFIRMED |
Traced, not run: backend/src/whip_session_manager.rs:1587 — RUNNING_FOR / 2 + STALL_SIDE_MARGIN / 2, puts output 50 ms staler, inside the 100 ms margin, so the side stays Ingress; backend/src/whip_session_manager.rs:1601 — idle >= output_idle, then fails by ~50 ms against the reverted ingress_idle |
| The nine named tests ran at this head | CONFIRMED |
Check (Linux) job 107188638644: all nine ... ok, no skip guard, including a_reaped_session_names_which_side_of_the_chain_stopped and the_slot_stamp_follows_media_out_of_the_decode_chain_not_into_it |
| Rig trials (pre-rebase) | EXTERNAL |
Not settleable here; assumed as stated, and this commit only widens the duration back to the max |
Diagnosis — Unchanged from cb10dc8 and 777a6bc; the probe callback and the grace logic are untouched by this commit, so those rows stand CONFIRMED in those reviews rather than being rebuilt. The commit also folds idle_detail back into idle (backend/src/whip_session_manager.rs:416 — pub fn idle(&self) -> Option<(Duration, StallSide)> {), which closes the dead pub fn I flagged as non-blocking; grep finds no remaining caller of the old name, and the only other .idle() in the tree is an unrelated method in backend/src/mcp/session.rs.
Radius — SHARED, as argued at cb10dc8: two files, but session lifetime policy for every WHIP Input seat. The bounded timing shift the margin introduced is now gone, so the takeover ordering key is the pre-margin one (backend/src/whip_session_manager.rs:865 — .max_by_key(|c| (c.dying, c.has_audio, c.idle.map(|(idle, _)| idle)))).
Overlaps — None left for this merge to settle. #854 is the promised pure-move split and is stacked directly on d322652, so it lands after. No open issue's Ask touches WHIP Input session liveness; #839 is WHIP Output.
Not blocking, for the stack owner: #792 and #805 carry this PR's base commit but not d322652, so both need a rebase onto this head before they are read against it.
Tests & CI — Green at d322652, nine jobs, macOS/Windows skipping and no platform code here. Canary pipeline_lifecycle_test.rs ran, 3 passed in 0.94s. Both files remain past CLAUDE.md's 1500-line threshold (1710 and 2713); #854 is the split, in the order the body committed to.
Confidence: HIGH
Superseded by my review at d322652 under the same protocol generation: the requested change landed and the verdict is now Approve.
d322652 to
449fd20
Compare
srperens
left a comment
There was a problem hiding this comment.
Verdict: Comment — supersedes my dismissed review at d322652; the verdict does not stand, because the head is a different six-commit series. The mechanism is right and the BUFFER probe is written as CLAUDE.md requires. Two things hold approval back: an unmeasured teardown window the body names itself, and a constant relationship nothing enforces.
Requested changes
- Bind the two timeouts that must not cross.
backend/src/whip_session_manager.rs:530—const DECODE_GRACE: Duration = Duration::from_secs(5);— must stay underbackend/src/blocks/builtin/whip.rs:713—const INACTIVITY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);, and the body says nothing enforces it. MakeDECODE_GRACEpub(crate)and put aconst _: () = assert!(...)besideINACTIVITY_TIMEOUT: one line, and a comment becomes a build error.
Claims
| Claim | Verdict | Evidence |
|---|---|---|
| The BUFFER probe callback is lock-, allocation- and format-free | CONFIRMED |
backend/src/blocks/builtin/whip.rs:309 — ` gst::PadProbeType::BUFFER |
| The grace is held unconditionally, so a predecessor's draining frames cannot expose a newcomer | CONFIRMED |
backend/src/whip_session_manager.rs:418 — let past_grace = self.ingress.since_first()?.checked_sub(DECODE_GRACE)?; — keyed on the per-session ingress stamp, and idle() returns None before it elapses whatever output shows |
The displaced session's drain tail is shorter than DECODE_GRACE |
UNVERIFIED |
Stated unmeasured in the body. It is this repo's own teardown window, so it is settleable here — the consequence if it is ever longer is bounded and mild (a newcomer that decodes nothing looks alive until the tail ends), but it is the assumption the takeover correctness rests on |
| Post-rig commits only lengthen protection and change wording | CONFIRMED |
backend/src/whip_session_manager.rs:437 — Some((ingress_idle.max(output_idle), side)) — the label is chosen separately from the duration, which is the staler stamp either way, so reap timing is unchanged by acf9f42 |
| The nine named tests executed rather than skipped | CONFIRMED |
Check (Linux) job 107891171430 at 449fd201: all nine ... ok, and the pipeline_lifecycle_test canary "3 passed" |
Diagnosis — Right layer: liveness moves from where bytes arrive to where frames leave the decode chain, which is what the 45-minute held seat needed. Coverage is BOUNDED per slot, not per medium, so frozen video beside live audio still reads as live; #792 owns that.
The new repeated eviction is worth stating plainly. It is CLOSED — it stops as soon as output moves — and costs nothing on the healthy path, because it only fires where output is genuinely frozen. But a blocked consumer now churns the seat every ~7 s instead of holding it for ever. That is the intended trade and the stall-side label makes it legible, so it wants a maintainer's nod, not a change.
Radius — SHARED: every WHIP Input flow's session lifetime decisions move, and takeover and the watchdog both read the new idle(). The collector.rs edit is test-fixture only.
Overlaps — #792 (draft) owns the per-medium refinement and is stacked on this; #854 (draft) is the pure-move split of these two files, sequenced after this. Neither leaves a design question for this merge to settle.
Tests & CI — Nine checks green at 449fd20; macOS/Windows skipped, nothing here is platform-specific.
Confidence: HIGH
|
Both addressed. Timeouts bound. Drain tail. Settled by reading the code, and the claim was wrong as stated: the tail is not guaranteed shorter than the grace. Takeover never flushes the slot appsrc, so whatever sits in its queue when Ran at — Claude (via @wagenet) |
srperens
left a comment
There was a problem hiding this comment.
This review supersedes my review of 2026-09-25 at 449fd20, and the verdict moves from Comment to Approve. Since then the branch gained two commits (ahead_by=2, 12 lines in total), and they settle both points that were holding the verdict.
Verdict: Approve. Every row is now CONFIRMED. Of the earlier concerns, the constant relationship is enforced at compile time, and the drain-tail assumption was resolved in the author's reply, in the direction the old row feared. The author then bounded it and documented it instead of leaving it assumed.
Claims
| Claim | Verdict | Evidence |
|---|---|---|
DECODE_GRACE can no longer reach the watchdog timeout |
CONFIRMED |
backend/src/blocks/builtin/whip.rs:718 — const _: () = assert!(DECODE_GRACE.as_millis() < INACTIVITY_TIMEOUT.as_millis());. This is exactly what I asked for, and the build breaks if it is violated |
| A predecessor's drain tail can outlast the grace, bounded by the slot appsrc's queue | CONFIRMED |
backend/src/whip_session_manager.rs:413 — ` /// `APPSRC_MAX_TIME` of the predecessor's media, and a newcomer that decodes`. The slot appsrc is built with backend/src/blocks/builtin/whip.rs:444 — .max_time(APPSRC_MAX_TIME), and that constant is backend/src/blocks/mod.rs:23 — pub const APPSRC_MAX_TIME: gst::ClockTime = gst::ClockTime::from_seconds(10);. Worst case: a newcomer that decodes nothing looks live for up to ~10 s past the grace, and then it is reaped. The effect is bounded and mild, and it only happens when the chain was already behind real time |
BUFFER probe, grace keying and stall-side label are unchanged from 449fd20 |
CONFIRMED |
Carried from my 2026-09-25 review. Neither new commit touches whip.rs outside the assert and its import, and the duration is still backend/src/whip_session_manager.rs:441 — Some((ingress_idle.max(output_idle), side)) |
Design record — What was rejected: flushing the slot appsrc on takeover. That would close the drain-tail gap, but it is a new teardown behaviour with its own risk, and the gap it closes is bounded and mild. I agree with leaving it out. If a held seat ever shows a ~10 s false-live window in practice, that flush is the follow-up. The accepted trade from my last review still applies: a blocked consumer now churns the seat roughly every 7 s instead of holding it for ever, and the stall-side label makes that legible.
Radius — SHARED, as argued in the 2026-09-25 review. Session lifetime decisions change for every WHIP Input flow, but nothing outside the WHIP session manager and its watchdog reads idle().
Overlaps — #792 (draft) owns the per-medium refinement and is stacked on this PR. #805 (draft) carries that same per-medium restructuring. #854 (draft) is a pure move of these files. #763 touches WhipSessionManager in other hunks. None of them leaves a design question for this merge to settle, and this PR lands before all of them.
Tests & CI — Run 36215508270 at 47c41fd is green: Check (Linux), both Linux builds, WASM and API Contract Check. In job 108330585910, the whip_session_manager::tests cases all pass, and the pipeline_lifecycle_test canary reports "3 passed". The canary's gst_mini_object_unref CRITICALs also appear in a 2026-09-03 main run, so this diff did not cause them. macOS and Windows were skipped, and nothing in this diff is platform-specific.
Confidence: HIGH
A seat can receive RTP for a whole session while producing nothing the flow can use: its decoder never gets a keyframe it can use, or a consumer below the slot's tee blocks and backs pressure up the chain until nothing comes out. Liveness was stamped in the session pipeline's appsink, upstream of all of that, so such a seat looked perfectly healthy. Slot takeover never displaced it and the inactivity watchdog never reaped it, and every rejoining client got 503 for as long as it sat there — 45 minutes on the live rig. The slot's chain now carries its own `ActivityStamp`, written by a pad probe on the slot's output tee, past `decodebin` and the converter. `SessionActivity` holds that alongside the appsink's and reports the staler of the two, so a session counts as usable only while media both arrives and comes out. The stamp belongs to the slot, which outlives the sessions passing through it, so claiming a slot resets it — otherwise a newcomer is judged on its predecessor's frames. Media arrives before it can be decoded, because H.264 carries its parameter sets with a keyframe, so a session is not judged at all until DECODE_GRACE after its first buffer. Measured 0.9-1.0 s from the video pad appearing to `decodebin` exposing its own, against a 5 s grace. `wait_for_inactivity` reads that signal through `SessionActivity::idle()` rather than a raw arriving-buffer counter, so the watchdog now also reaps a seat that receives without producing. Its poll structure is untouched. Whether a session carries audio, which is what decides how short a silence may be held against it, stays on the arriving side: one stamp serves a slot's audio and video tees alike and cannot separate them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Takeover releases a slot while the displaced session's pipeline is still coming down. The session_finished gate stops it pushing new samples, but frames it already pushed into the slot's chain still cross the output tee after the newcomer reset the stamp. Those made `output.since_last()` return a value, which skipped DECODE_GRACE entirely: the newcomer could be judged idle about two seconds after that tail stopped, before its own decoder had its first keyframe, and the next client could evict it. `idle()` now refuses to judge any session within DECODE_GRACE of its first arriving buffer, whatever the output stamp says. The watchdog is unaffected in production because INACTIVITY_TIMEOUT is longer than the grace; its one-second-timeout test now backdates its session past the grace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both reap paths logged "without usable media" whichever stamp froze, so an operator could not tell a participant who went away from a blocked consumer inside their own flow — the second keeps costing every publisher its seat on reconnect, and only the log says which one it is. `idle_detail` returns the idle time with the stamp that produced it, and the displacement warning and the watchdog's info line both carry it. The two stamps keep separate epochs and store whole milliseconds, so a publisher drop freezes both within a millisecond either way round. `STALL_SIDE_MARGIN` keeps that noise from reading as a stuck consumer; the real fault holds ingress live for seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inside STALL_SIDE_MARGIN the Ingress branch returned ingress_idle rather than the staler stamp, so the label could defer a takeover by up to the margin. Both branches now return the max; the margin picks the label only. idle() had no production caller left once idle_detail() existed, so the two are folded into idle(), which keeps the existing doc links pointing at the function the reap paths call. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…meout A session is not judged until DECODE_GRACE has passed, so an INACTIVITY_TIMEOUT at or under it would never reap a session that decodes nothing. The two live in different modules; a const assertion beside the timeout turns the ordering into a build error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing flushes the slot's appsrc on takeover, so its queue can drain up to APPSRC_MAX_TIME of the displaced session's media after the gate closes. The grace covers that only while the queue is short. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
47c41fd to
b26bc11
Compare
srperens
left a comment
There was a problem hiding this comment.
This supersedes my Approve at 47c41fd. The verdict stands at Approve. The new head b26bc11 is a rebase onto b28be77 with no content change.
Verdict: Approve. I checked this directly rather than trusting the compare view, which shows 29 commits because the merge-base moved:
git diff 47c41fd b26bc11over the PR's three files is empty.- The PR's own patch against each base is the same 1491 lines, and its +/- lines are identical.
- No
maincommit between the two bases toucheswhip.rs,whip_session_manager.rsorcollector.rs.
So every claim row carries over from my review of 2026-09-28. Two spot checks at this head:
backend/src/blocks/builtin/whip.rs:718—const _: () = assert!(DECODE_GRACE.as_millis() < INACTIVITY_TIMEOUT.as_millis());backend/src/whip_session_manager.rs:441—Some((ingress_idle.max(output_idle), side))
Tests & CI: I checked run 36445821036 at b26bc11. Check (Linux), both Linux builds, WASM, API Contract Check and Static JS Tests are green. In job 109007773818, all 17 whip_session_manager::tests cases pass, including a_predecessors_trailing_frames_do_not_cut_the_decode_grace_short and a_reaped_session_names_which_side_of_the_chain_stopped. The pipeline_lifecycle_test canary ran: 3 passed; 0 failed. macOS and Windows were skipped, and nothing in this diff is platform-specific.
Radius: SHARED, carried from my previous review.
Overlaps: #792, #805, #854 and #763 are as in my previous review. Two WHIP PRs are new since then:
- #893 edits
create_whipserversrc_for_sessionandforward_sample_to_slot(PTS ordering at the slot bridge). It is near this PR's hunks, but it is a different mechanism, so the overlap is textual. - #901 (draft) restarts a reused slot's video
decodebin. Its evidence names the drain tail this PR bounds: "Frames left in the decoder's reorder queue come out at the start of the next session". That is the takeover-flush follow-up my previous review set aside. #901 would shrink the ~10 s false-live window this PR documents, and it does not contradict the design. This PR should land first, and #901 should then revisit the comment nearbackend/src/whip_session_manager.rs:413.
Neither leaves a design question for this merge to settle.
Confidence: HIGH
Eyevinn#756, Eyevinn#792 and Eyevinn#805 drop AtomicU64 from whip.rs's imports once nothing else there uses it. Naming it by path keeps this struct compiling in either merge order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and #943 (#957) * fix(whip): stop an unstamped buffer at the session bridge before qtmux sees it A WHIP session's appsink -> appsrc bridge forwarded buffers with no PTS. A recorder downstream answers one with "Buffer has no PTS" from qtmux and GST_FLOW_ERROR, which takes the whole flow down: on a live rig, a second participant joining killed the first. The slot bridge now goes through gst::pipeline_bridge::SessionBridge: a buffer with no PTS is dropped and counted (warned on the first and every hundredth), the shared A/V offset is set once per session with compare_exchange and taken from the first stamped buffer, DTS is shifted with PTS, and a sample without caps no longer panics. From #759 (cfc4518, f62e777, 9ba9941), adapted to main: - The slot callback keeps the session_finished gate from #753 and the ingress liveness stamp from #756, wrapped in install_slot_bridge so the wiring test drives the real callback. - The Media Player half of #759 is left out. #945 replaced that bridge's timing with its own, so SessionBridge no longer fits there. Co-Authored-By: Per Enstedt <per.enstedt@eyevinn.se> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(whip): keep a slot's video PTS moving forward frame to frame A WHIP publisher joining a slot can take the whole flow down with a GPU vision mixer downstream: "Need timestamped buffers!" from videoaggregator. The session's jitterbuffer can release a run of packets from different frames on one PTS; h264parse takes a PTS only when it differs from the previous one, so the later frames of the run leave it with none, and a browser stream has no framerate to derive one from. A video packet that starts a new RTP frame and whose PTS is at most 1 ms behind the previous frame's now gets that PTS plus 1 ns. Packets of one frame keep sharing a PTS, and any other PTS (a B-frame, a step in the sender's clock, a frame far ahead) passes through. On the rig, 5 of 8 runs of publish / switch camera / leave / rejoin died before, 0 of 8 after. From #893 (745ce02, e90f52b, 2540522), adapted to the SessionBridge from #759: the ordering runs on the session PTS before the bridge rebases it, and copies the buffer only when the PTS moves. Co-Authored-By: Per Enstedt <per.enstedt@eyevinn.se> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(whip): stop a session's second stream stalling below PLAYING A WHIP session pipeline is already PLAYING when whipserversrc adds its pads, and each pad gets a `tee -> fakesink + appsink` branch synced into it. The appsink ran with `GstBaseSink`'s default `async=true`, so it answered that state change with ASYNC and waited for a preroll. The pads arrive one after the other within a millisecond, so the second branch lands while the first is still waiting, and the second is left below PLAYING: it takes a single buffer and then blocks its streaming thread for good. That seat's audio or video then never reaches the slot's appsrc, so its decodebin exposes no pad, and the flow's recorder and mixer wait on a track that will never arrive. Nothing in the log says so — `sync_state_with_parent` reports the state change as accepted either way. Turn `async` off, as the fakesink beside it already does. Measured on a five-seat meeting rig, 18 fresh-server trials per arm: 7/18 seats failed to start before, 0/18 after. The branch construction moves into `attach_session_branch` so the regression test builds the real thing rather than a copy of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(thumbnail): turn off async on the thumbnail tap appsink The thumbnail branch is attached to a flow pipeline that is already running, and its appsink ran with GstBaseSink's default async=true, so it answered that state change with ASYNC and waited for a preroll. The activation guard admits the pipeline in PAUSED with PLAYING pending, which is that wait, and a branch attached inside one is left below PLAYING: it takes a single buffer and blocks its streaming thread. Activation returns without waiting for a preroll, and a branch is created per block from an HTTP request, so a client asking for several blocks' thumbnails serializes on one mutex but not on the async cycles behind it. Same defect and same fix as the WHIP session branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(whip): give a reused slot a fresh video decoder A WHIP Input slot keeps its decode chain while publishers come and go. The video decoder that decodebin plugged for the first session kept that session's reference frames, reorder queue and configuration, and a later session with equal caps sends none to reconfigure it. On macOS, VideoToolbox then decoded a second Safari publisher at a few frames per second and flagged every frame of a third as an error, so the slot showed no video while its audio played. When a session claims a slot whose video decodebin has already run, restart that decodebin from an IDLE probe on the slot appsrc: unlink, set it to NULL, relink so the sticky events are sent again, and sync it with the pipeline. The existing pad-added handler links the new chain when data arrives. Audio keeps its running chain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(whip): guard a codec change on a reused slot The test reuses a slot with a VP8 publisher after an H.264 one and asserts the VP8 session decodes. With the restart disabled, the kept decoder refuses the VP8 caps and the slot's source stops with not-negotiated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(whip): ask for a keyframe when a running session's video breaks A WHIP session asked its publisher for a keyframe only until video first decoded. After that, nothing asked again. A browser sends a keyframe only when asked, so a loss the jitterbuffer could not repair left the slot's decoder broken for the rest of the session: smeared pictures, or every frame rejected with an error flag, until the watchdog tore the session down and the guest rejoined. The slot's decodebin now reports damage through a per-slot flag: - a gap: the depayloader marks the first access unit after a sequence gap DISCONT; damaged until a keyframe with no gap in front of it leaves the depayloader (a repair keyframe can lose a packet too). - a silent decoder: data lost with no gap (the depayloader drops a fragment it cannot place) makes the decoder reject every frame without a bus message. 30 access units in with no picture out counts as damage, cleared when pictures come out again. Both are buffer probes that touch atomics only. A per-session thread on the session side reads the flag and sends the keyframe request on the whipserversrc pad, where it reaches the publisher: at once, then at most once a second, 10 times per damage episode, and never on healthy video. Each episode logs its recovery time and request count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(whip): guard that a reused slot reports its video as decoding The keyframe requester stops once the slot's video_decoding flag is set, and that flag is set from decodebin's pad-added. Before the decoder restart on reuse, pad-added fired once per flow, so every reconnecting publisher was asked for the full request budget while its video decoded fine (#805). The restart brings pad-added back for each session. The reuse test now re-arms the flag as a new session does and asserts the second session sets it again. With the restart disabled it fails with "the second session decoded, but the slot never reported its video as decoding". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Peter Wagenet <peter.wagenet@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The bug
A browser seat on the live rig kept receiving RTP but stopped producing usable video, and held its slot for 45 minutes. Every rejoining client got a 503, and #753's takeover never fired. Liveness was stamped by the session pipeline's appsink, which sits upstream of
decodebin: it recorded bytes arriving, not frames decoding or reaching a consumer.What changed
SessionActivitytracks twoActivityStamps — ingress (the session appsink: the publisher is sending) and output (a BUFFER probe on the slot's output tee, pastdecodebinand the converter: frames are reaching the flow's consumers).idle()returns the staler of the two, and both the watchdog and slot takeover read it, so both now catch a seat that receives without producing.idle()does not judge a session untilDECODE_GRACE(5 s) after its first arriving buffer. H.264 cannot decode before a keyframe brings its parameter sets; measured delay 0.9-1.0 s.Takeover. Both sessions share the slot's appsrc, and the displaced session is still tearing down when the newcomer resets the stamp. #753's
session_finishedgate stops it pushing new samples, but frames already inside the chain still cross the tee and move the newcomer's output stamp.idle()therefore holds the grace regardless of output, so that tail cannot let the next POST evict a newcomer that has not decoded anything yet. The cost: a session that dies within 5 s of its first buffer cannot be displaced until those 5 s are up, where before it could be after 2 s. The grace covers that tail only while it is short. Nothing flushes the slot appsrc on takeover, so the tail is bounded by its queue,APPSRC_MAX_TIME(10 s) or 20 MB, not by the grace. The queue fills only when the chain has fallen behind real time. In that case a newcomer that decodes nothing reads as live until the predecessor's backlog has drained, up to ~10 s past the grace, and is reaped after that.Which side stopped.
idle()returns the idle time together with the stamp that produced it, and both reap messages carry it.ingressfrozen is a publisher that went away;ingressstill moving withoutputfrozen is a fault inside the flow that will do the same to whoever reconnects. Only the log distinguishes them. The stamps keep separate epochs and store whole milliseconds, so a publisher drop freezes both within a millisecond in either order —STALL_SIDE_MARGIN(100 ms) keeps that noise from reading as a stuck consumer. It labels only: the duration is the staler stamp whichever side is named, so the reap decision is unchanged.Tests
Each new test fails with its fix reverted: ingress-only
idle(), the tee probe deleted, the conditional grace restored, the stall side hardcoded,STALL_SIDE_MARGINdropped, or theIngresslabel returningingress_idleinstead of the staler stamp. The last three were run on this head; the first three when written.whip_session_manager.rs:a_session_receiving_media_it_never_decodes_is_displaced,a_session_whose_output_has_stalled_is_displaced,a_session_still_waiting_for_its_first_decoded_frame_is_not_displaced,a_new_session_does_not_inherit_the_slots_previous_liveness,a_predecessors_trailing_frames_do_not_cut_the_decode_grace_short, anda_reaped_session_names_which_side_of_the_chain_stopped(both directions plus the near-tie the margin exists for).whip.rs:the_slot_stamp_follows_media_out_of_the_decode_chain_not_into_itpushes real buffers through the block's ownappsrc -> decodebin -> videoconvert -> teechain, blocks the consumer, and asserts the stamp stops moving —plugins-baseelements only, which CI installs.watchdog_reaps_a_session_that_receives_media_it_never_decodesruns under a 5 s bound so a regression fails rather than hangs, and now also asserts the reap is reported against the slot's output.watchdog_detects_inactivity_within_one_poll_of_the_timeout(fix(whip): evaluate the inactivity watchdog per poll tick, not per timeout #752) backdates its session and asserts the publisher-side label.a_finished_session_stops_feeding_its_slotanda_dead_session_is_displaced_past_a_quieter_video_only_one.Ran locally on macOS at this head:
cargo test --features efp— 679 lib tests plus all integration suites passed, 0 failed, 2 pre-existing ignores — with clippy--all-targets -D warningsandfmt --checkclean. Not run: thenvidiafeature, the WASM clippy.Rig verification (pre-rebase)
Six trials on the pre-#753 base, not repeated since. The commits after them only lengthen protection for sessions younger than 5 s and change reap wording, and every trial was judged well past 5 s.
Headless,
max_sessions: 1, awhipclientsinkH.264 publisher feeding a flow that ends in an MP4 recorder. Stopping the publisher's audio makes the muxer back-pressure video through the slot's tee while RTP keeps arriving.Trade-offs
outputfreezes whileingresskeeps moving — the slot appsrc isleaky-type=downstream, so the stall never back-pressures the publisher. The seat is reaped, the publisher reconnects onto the same blocked chain, and is reaped again: every ~7 s (DECODE_GRACE+TAKEOVER_IDLE_THRESHOLD) with a client waiting, or ~15 s (DECODE_GRACE+INACTIVITY_TIMEOUT) from the watchdog alone. The stall-side label makes it visible in the log; fixing the stall is out of scope.DECODE_GRACEhas no hard bound behind it beyond the ~1 s measurement. It must stay belowINACTIVITY_TIMEOUT, and aconstassertion beside the timeout makes crossing them a build error.whip_session_manager.rsandwhip.rsare both past the split threshold. A pure-move split, stacked on this PR, is in progress and lands before feat(whip): track a slot's liveness per medium #792.🤖 Generated with Claude Code