feat(returns): build the low-latency fast return feed

[wagenet]

Opened by Claude (AI) on Peter Wagenet's behalf.

Depends on Eyevinn/strom#844 (audio bridge blocks) and Eyevinn/strom#930 (mixer direct outputs). Do not merge until both are merged and deployed. The program flow uses both, so on a Strom without them a production with a lowLatency guest fails to start. Productions without one are unaffected.

What

returnFeed.lowLatency: true now builds the fast feed the guest-calling spec describes under "Low-latency mode": an audio-only mix-minus that reaches the guest about 500 ms before the picture feed's audio.

• Program flow: the audio mixer is built with direct_outs: true, and each channel's direct_out_N feeds a builtin.audio_bridge_output. A direct out carries what the channel sends to Main: after the fader, the mute and audio-follow-video (ch{N}_to_main), before the Main sum. Crew changes therefore reach the fast feed inside Strom, with the same fades as program.

• Conversation flow (a second Strom flow): one builtin.audio_bridge_input per channel feeds a builtin.liveaudiorouter. The router is a fixed mix-minus set at activation: each guest's output sums every channel except their own. Each output goes to an audio-only builtin.whep_output (whep-fast-<N>-<suffix>). Router settings are 15/10/5 ms, the values measured with five live seats. The bridge target is Strom's default (40 ms), or values.fast_return_latency_ms if set.

• Limit: Strom's router takes at most 8 inputs. A production with more than 8 audio channels, or with an audio channel that has no source, gets no fast feed and a warning; its guests keep the picture feed's audio.

• API: returnFeed.lowLatency: true is accepted on POST /productions/:id/sources (main accepts only false). The fast feed is served wherever the picture feed is, through the same proxy helpers:

  • crew: POST/DELETE /api/v1/productions/:id/returns/:mixerInput/fast/whep
  • guest: POST/DELETE /api/v1/guests/:inviteId/returns/fast/whep, the alias the OSC ingress gate lets through

  A guest's fast session is bound to its own fastWhepSessionId, so neither feed's session id tears down the other. Join, the crew return view and GET /guests/:inviteId/session/return list the fast feed (video: false) and the low-latency-minus mode only when the guest has one. Join hands out the guest-alias URL. low-latency-minus is still refused by the mode PUTs: it is a feed choice for the client, not a mix.

• Lifecycle: the conversation flow is tagged conv:<programFlowId> and is removed with the program flow. Activation and startup also remove any conversation flow whose program flow is gone.

Why a second flow, and why direct outs

Strom runs every sink in a flow at one latency. On a TRL-shaped production that is 360 ms, set by the mixer's monitor bus (Eyevinn/strom#920 cuts it). It sits on top of the 400 ms WHIP jitterbuffer offset. A return inside the program flow cannot be released earlier than that. The bridge restamps each buffer on arrival, so neither cost reaches the conversation flow.

Tapping each source before the mixer would need every crew change copied into the router's gains, and that copy cannot be kept correct from outside the mixer: writes to the mixer and to the router can be answered out of order or refused separately, and reading the mixer back returns live element values, which are 0 for a muted fader and stale for a stalled source. Fed from the direct outs, the router never changes after activation.

A direct out is not an aggregator output. It taps the channel strip before the Main sum, so one guest's bad link is not passed to the other guests through the mixer.

A running program flow is never edited

Per strom#930, a direct-out consumer that ends (EOS) or is torn down (flushing) while program runs silently stops that channel on program. open-live builds every audio_bridge_output into the program flow at activation and never edits a running flow: flows.update is unused, and the program flow is only stopped and deleted whole. The bridge sink is unsynchronised and returns OK whether or not anyone reads, so stopping the conversation flow does not reach program.

Measured

Rig: a fresh Strom per trial (Strom main with #844, plus #930), activated through this branch's activateStromFlow. Five WHIP seats (Opus + H.264), loopback. Delay is from tone ticks into seats 1–4 to the same ticks out of seat 0's feeds, all on one clock, through a GStreamer WHEP consumer. Over 4 trials at load average 6–8, the fast return was 188 ms (trial means 175–200 ms) and the picture return 688–703 ms. Worst dropout was 0.17 %.

Crew changes, with open-live not running: a mute on one channel, to_main off on another, and a fader at 0.1 on a third, each sent straight to Strom's mixer for 3 s. Each took that seat's ticks in seat 0's fast feed from 0.40–0.55 down to 0.02–0.08, and each came back when undone. Seat 0's own voice stayed at the noise floor throughout. Two runs, same result.

One bad link: one seat had 0–200 ms jitter, 0.4–1.2 s stalls and 14 % burst loss. A tone from another seat lost 0.14–0.17 % in seat 0's fast feed, with no change in delay. Three alternated control runs lost 0.13–0.21 %. Leave and rejoin: a guest left, then published again on the same slot 4 s later. They were back in the fast feed and on program by the next tick. Two runs, and the other seats were undisturbed.

These runs predate the rebase onto current main. The fast-feed flow code is unchanged since; the program flow around it now has main's shared viewer encode (#413) and compact vision-mixer pads (#463). The fast feed is keyed by audio channel, and its endpoint ids use the stored video_in_N, as the picture feed's do, so #463's pad map does not reach it.

Not in this PR

• Clients. feat(guests): play the fast return feed on the guest page #393 (stacked on this) plays the fast feed on the guest page. open-live-studio does not play it yet. A client must keep the fast feed and the picture on separate PeerConnections, or Chrome will lip-sync the fast audio back to the picture.
• The fast feed is not synced to the picture by design.
• A conversation flow that stops on its own while the program runs is not detected; its feeds stay advertised until the production stops.
• A channel routed to a group but not to Main reaches program through the group, but is silent on its direct out and so in the fast feed (strom#930, "Where the tap sits").
• The fast feed carries the whole channel strip, so pan and a fader above unity reach it, and the router sums with no limiter (feat(liveaudiorouter): give an output bus headroom and a ceiling strom#795).
• fastFeedRouter is stored on the production but nothing reads it yet.

How to review

Two commits, each builds and passes on its own:

1. feat(returns): build the low-latency fast return feed: src/lib/fast-returns.ts (the plan) and src/lib/flow-generator.ts (wiring it into activation and teardown), plus storage in productions.ts and the orphan sweep in reconcile.ts. The API still accepts only lowLatency: false here.
2. feat(returns): serve the fast return feed and accept lowLatency: true: src/routes/returns.ts turns main's picture-only proxy helpers and routes into ones that take a feed; guests.ts adds the fast feed to join; server.ts and guest-scope.ts widen the guest-token path patterns from picture to picture|fast.

Tests

• flow-generator-returns.test.ts: the mixer gets direct_outs: true only when fast returns are built; each bridge links from direct_out_<ch+1> and nothing else; the router matrix is the fixed mix-minus; no fast feed past 8 channels; the program keeps running if the conversation flow fails; teardown removes only this program's conversation flow.
• returns-guest-auth.test.ts: crew and guest-alias fast routes, separate session binding for the two feeds, a 409 retry on the binding, and the feed and mode listings.
• activation.test.ts, production-lifecycle.test.ts, guest-slot-declaration.test.ts: storage, the startup sweep and lowLatency: true on the sources route. The activation test's controller mock now stubs reinitConnectedControllers, which main's activation calls once the flow plays.
• pnpm typecheck and pnpm test pass on each commit: 1056 tests after the first, 1065 after the second (105 files).

Not run since the rebase: the Strom rig, a real remote guest, a browser.

🤖 Generated with Claude Code

[birme]

This PR looks like the base of a multi-PR dependency stack for the low-latency fast return feed, together with #393 (play the fast return feed on the guest page, which depends on this PR). Holding off on automated review/merge here — this needs a human (or the relevant dev agent) to assess the whole stack together, not a per-PR pass. Nothing further is needed from the author; this comment exists so the dependency is recorded on the PR rather than in a run log.

[wagenet]

Claude (Claude Code, for @wagenet): rebased onto main (af7bca1) to clear the conflicts. All resolutions are unions with main's changes: mixerInputMap (#463/#475) sits next to fastWhepEntries/fastFeedRouter on the activation result, the join handler keeps main's tile label (#466) ahead of the fast feed URLs, and the guest-token path pattern keeps slot and accepts returns/(picture|fast)/whep. #463's compact pad map does not reach the fast feed: it routes by audio channel, and its endpoint ids use the stored video_in_N, as the picture feed's do. pnpm typecheck and pnpm test pass on both commits (1056 and 1065 tests). The rig was not run.

[birme]

This PR is part of the low-latency return-feed feature on open-live. Holding off on automated review/merge — needs a human (or the relevant dev agent) to assess together.

Members:

• feat(returns): build the low-latency fast return feed #386 — build the low-latency fast return feed (draft; depends on upstream feat(audio): adaptive audio bridge for the conversation return strom#844 and strom#930 being merged + deployed — do not merge before then)
• feat(guests): play the fast return feed on the guest page #393 — play the fast return feed on the guest page (merges independently of feat(returns): build the low-latency fast return feed #386; until feat(returns): build the low-latency fast return feed #386 and its Strom deps ship, the page plays the picture feed only)

Nothing further is needed from the author; this records the dependency on the PRs themselves.