Skip to content

fix(recording): wire PGM video and AAC audio into the recorder - #358

Merged
birme merged 30 commits into
Eyevinn:mainfrom
wagenet:wagenet/fix-recorder-video-pad
Oct 17, 2026
Merged

birme merged 30 commits into
Eyevinn:mainfrom
wagenet:wagenet/fix-recorder-video-pad

Conversation

@wagenet

@wagenet wagenet commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Opened by Claude (AI) on behalf of @wagenet.

A production with a recording output activates, but writes no recording. The recorder has two broken inputs:

  • Video. The generator links PGM to <recorder>:video_in, but builtin.recorder numbers its inputs from zero: video_in_{i} / audio_in_{i} (Strom upstream/main 8079d581, backend/src/blocks/builtin/recorder.rs:491, :501; static definition :1665, :1672). POST /api/flows drops the link as stale (backend/src/api/flows.rs:436), so the recorder is built with video track 0 unconnected. This PR links to video_in_0.
  • Audio. The recorder refuses raw audio (backend/src/blocks/builtin/refusal.rs:65: "Recorder needs AAC, MP3, AC-3, DTS or Opus audio, but its audio input is raw"), and the audio mixer's main_out is raw. This PR routes main_out through a builtin.audioenc (codec: aac) into the recorder's audio_in_0.

builtin.audioenc first shipped in Strom 0.6.9, and an older Strom refuses to start any flow that references it. Open Live doesn't pin a Strom version, so activation checks GET /api/blocks first. If the block is missing, it leaves the encoder out and the production goes on air recording picture only. It also saves a recording-no-audio warning on the production (activationWarnings) and logs it. Once the production is live, the warning goes out as a WS ERROR frame, and again to each controller that connects while it stays live. Studio shows that frame as a toast. docs/controller-websocket.md documents it. Eyevinn/open-live-studio#161 tracks a persistent display. If the block list can't be read, activation keeps the encoder.

With only the video fix, the recorder writes a full-length MP4 with no sound, and deactivate uploads it and lists it as a normal recording. Both fixes are needed for a usable file.

The existing test asserted the wrong video pad name, which is why CI passed. It now asserts video_in_0 and an AAC encoder between main_out and audio_in_0, or no encoder plus a warning when Strom lacks the block.

Property rename and comments

The recorder's property is filename_prefix (recorder.rs:546, definition :1605). Strom ignores the unknown prefix, so files were named recording_…. They are now <production id>_…. The output dir and upload key already carry the production id, so nothing downstream changes. The filename-template comments in config.ts, flow-generator.ts and recording-uploader.ts now match Strom's {filename_prefix}_{timestamp}_%05d.{ext} (recorder.rs:638).

Testing

  • pnpm typecheck and pnpm test (709 tests) pass, rebased onto current main.
  • Reverting any of the three recorder fixes makes recording-wiring.test.ts fail on the matching assertion. Removing any of these fails its new test: the block check, saving the warning, sending it when the production goes live and not after an aborted activation, the connect-time frame and its live-only gate, or clearing the warning on re-activation and on failed activation.
  • Live, against Strom upstream/main 8079d581 (dev build, headless). I posted the flow the generator produces, then ran the deactivate sequence (splitNow, then uploadRecordings with the S3 PUT stubbed) against the running recorder. A 10 s run gives:
Flow Uploaded file
before none (video link pruned, audio refused)
video fix only H.264 only, 8.03 s, no sound
this PR H.264 + AAC, 8.09 s, no pipeline error

Against Strom 0.6.7 (eyevinntechnology/strom:latest, no builtin.audioenc), activation succeeds, returns the recording-no-audio warning, and records H.264 only (6.03 s). Deactivate uploads that file. Against a Strom with the block, there's no warning and the file is H.264 + AAC.

A probe recorder on the encoded PGM pad received the same bytes (25,000,052 at 20 s) with this PR's recorder as with no recorder, so the added branches do not stall program output.

Not tested: a real MinIO upload, a production with real participants, or the studio toast end to end.

Known issue, fixed in #359

Deactivate uploads everything in recordings/<production id>, which is shared across activations, then deletes each uploaded file from Strom (#378). A file whose delete fails, or is skipped because the production changed during the upload, is uploaded again at the next deactivate and inserts a second recording document. Now that recordings are written, this is reachable. #359 fixes it and is built on this branch, so it should merge after this PR.

Possible follow-ups

  • No AAC encoder on a new Strom. On Strom 0.6.9 or later, a host with none of the AAC encoders Strom tries (fdkaacenc, faac, voaacenc) fails flow start with "No encoder available for Aac", so recording can still keep a production off air. The Strom Docker image and Homebrew GStreamer both have one. The fix would be to retry the start without the encoder, plus the same warning, when it fails for that reason.
  • Sound on Strom older than 0.6.9. Plain GStreamer elements in place of builtin.audioenc (audioconvert → audioresample → voaacenc → aacparse from main_out to audio_in_0) started on Strom 0.6.7 and recorded H.264 plus audible AAC. That was one 6 s run: the audio came out at 44.1 kHz with no caps filter, A/V sync wasn't checked, and it needs voaacenc on the host. It could replace the picture-only fallback after proper testing.

How to review

  1. src/lib/flow-generator.ts recording branch: the video_in_0 link, the audio encoder and its block check (stromHasAudioEncoder), and filename_prefix.
  2. src/routes/productions.ts and src/ws/controller.ts: saving, sending and clearing activationWarnings.
  3. src/__tests__/recording-wiring.test.ts: the assertions for each.
  4. Other builtin.* links in the generator were checked against Strom's definitions. They match, except the findPgmFeedPad fallback (flow-generator.ts:98), which returns <Enc PGM>:video_out. builtin.videoenc exposes encoded_out. That fallback only runs when the template has no builtin.mpegtssrt_output, and the default template has one. Not changed here.

🤖 Generated with Claude Code

@wagenet wagenet changed the title fix(recording): link PGM video to the recorder's video_in_0 pad fix(recording): wire PGM video and AAC audio into the recorder Sep 28, 2026
@birme

birme commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

This PR looks like the base of a two-PR dependency stack:

Holding off on automated review/merge here — this needs a human (or the relevant dev agent) to assess the whole stack together, not a per-PR pass, and both PRs are drafts. Nothing further is needed from the author; this comment exists so the dependency is recorded on the PR rather than in a run log.

@birme

birme commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Heads up: main has moved (several recording/ws fixes merged), and this branch now reports a real
merge conflict with main (mergeable: CONFLICTING) — not mere staleness, so an automated
branch refresh won't resolve it. Please merge current main into this branch (or rebase) and
resolve the conflicting files, then push. This PR is otherwise still held for holistic review as
part of its dependency stack (see the earlier stack note); this comment only records the new
conflict so it isn't lost.

@wagenet
wagenet force-pushed the wagenet/fix-recorder-video-pad branch from f372b34 to 5358d93 Compare September 29, 2026 13:55
@wagenet

wagenet commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

@birme I've rebased this. Also, to be clear, this PR stands on its own and can be merged without #359. However, #359 does depend on this one.

wagenet added a commit to wagenet/open-live that referenced this pull request Oct 1, 2026
wagenet added a commit to wagenet/open-live that referenced this pull request Oct 1, 2026
wagenet added a commit to wagenet/open-live that referenced this pull request Oct 2, 2026
Resolutions:
- productions.ts: keep Eyevinn#359's recording-doc helpers; main moved the
  guest invite sweep to services/guest-sweep.ts. Activation sets
  activationSucceeded before main's controller re-init.
- production-lifecycle.test.ts: keep both sets of tests.
- flow-generator-encoders.test.ts: Strom mock lists builtin.audioenc for
  Eyevinn#358's block check, and the recorder is fed on video_in_0 (Eyevinn#358).
- record-inputs-lifecycle.test.ts: mock main's reinitConnectedControllers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wagenet added a commit to wagenet/open-live that referenced this pull request Oct 2, 2026
…load

Brings in main through Eyevinn#358. productions.ts: keep the recording-doc
helpers; main moved the guest invite sweep to services/guest-sweep.ts.
production-lifecycle.test.ts: keep both sets of tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet
wagenet force-pushed the wagenet/fix-recorder-video-pad branch from b1437f6 to d2dd364 Compare October 6, 2026 20:46
wagenet and others added 10 commits October 7, 2026 06:15
Strom's builtin.recorder numbers its inputs from zero (video_in_0,
audio_in_0). The generator linked PGM to `video_in`, which the recorder
does not have. Strom drops that link as stale when the flow is created,
so the recorder was built with its video track unconnected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorder's property is `filename_prefix`. Strom ignores the unknown
`prefix`, so segments were named recording_00001.mp4 instead of
<production id>_00001.mp4. Output dir and upload key already carry the
production id, so nothing else changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorder refuses raw audio ("Recorder needs AAC, MP3, AC-3, DTS or
Opus audio, but its audio input is raw"), and the audio mixer's main_out
is raw, so the recording had no audio track. Route main_out through a
builtin.audioenc (AAC) into the recorder's audio_in_0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strom writes {filename_prefix}_{timestamp}_%05d.{ext}. Update the three
comments that describe the segment filename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
builtin.audioenc first shipped in Strom 0.6.9. An older Strom refuses to
start any flow that references it, so a production with a recording
output could not go on air. Activation now checks Strom's block list
and leaves the encoder out when the block is missing. The recording then
has picture but no sound, and activation saves a warning on the
production (activationWarnings) and logs it. If the block list can't be
read, activation keeps the encoder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Studio shows WS ERROR frames as a toast, so activation warnings (such as
a recording with no sound) now go out that way: broadcast to connected
controllers when the production activates, and sent again to each
controller that connects while the production is live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A production that never went on air was left inactive with the warning
from the failed attempt still set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The warning toast is broadcast when the production goes active, and
not at all if activation was aborted. The connect-time frame needs
status 'active' as well as a stromFlowId, so an activating production
or one left with a stale flow id sends nothing. controller-websocket.md
documents the ERROR frames that carry activation warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture repeated makeActivationResult's field list. A branch that adds
a field to makeActivationResult (as Eyevinn#386 does) then merged into whichever
copy git matched, and left the other without it. Spreading the base keeps
one list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
activation-failure-relay-stop's makeActivationResult now returns
warnings: [], which runActivationFlow reads. flow-generator-encoders
looks for the recorder feed on video_in_0. activation.test's
makeActivationResult keeps main's mixerInputMap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wagenet
wagenet force-pushed the wagenet/fix-recorder-video-pad branch from 73a9ba4 to 34f961d Compare October 7, 2026 13:17
@birme

birme commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

This PR is part of wagenet's multi-PR recording feature line on open-live. Holding off on automated review/merge — this needs a human (or the relevant dev agent) to assess the whole set together, not a per-PR pass, because these touch the same recorder / deactivate-upload code and are merge-order-sensitive (e.g. #358 and #359 conflict in one test hunk).

Members, in dependency order:

Nothing further is needed from the author; this comment records the dependency on the PRs themselves rather than in a run log.

@birme birme left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-reviewer: APPROVE

Prior findings

# Source Finding Status
1 birme stack-flag Part of wagenet recording stack (#358→#359, #411/#489 stacked); hold for holistic review DROPPED — automation hold note, not a code finding. Author confirms #358 stands alone; #359/#411 depend on it, not the reverse.
2 birme branch-blocked Branch reported CONFLICTING with main FIXED — author rebased; now MERGEABLE (BEHIND only, no conflict).

No formal reviews/comments exist on the PR. No OPEN findings.
No Closes #N in the body.

Verification against real source (fabrication check — Strom 8079d581)

Grepped Eyevinn/strom at the cited commit — every pad/property/element the generator links is REAL:

  • builtin.recorder numbers inputs from 0: recorder.rs:491 format!("video_in_{}", i), :501 format!("audio_in_{}", i). The PR's video_in_0 / audio_in_0 are correct; the old video_in dropped the link. (Note audio_in_0 was already present pre-PR — only video was wrong, an internal inconsistency that confirms the bug.)
  • filename_prefix property: recorder.rs:545-546, :1605. Old prefix was silently ignored. Filename template comment {filename_prefix}_{timestamp}_%05d.{ext} matches recorder.rs:648 (the code path, authoritative over the slightly older header comment on :33).
  • builtin.audioenc: block id audioenc.rs:336; codec enum property default aac :342-353; pads audio_in :421 and encoded_out :428. The PR's { codec: 'aac' } + audio_in/encoded_out wiring are correct.
  • refusal.rs:65 confirms the recorder refuses raw audio and names builtin.audioenc as the fix — the raw main_out→audio_in_0 link was genuinely refused, so routing through the encoder is required.
  • stromHasAudioEncoder uses blocks.list() → BlockListResponse.blocks: BlockDefinition[], and BlockDefinition.id (strom.ts) exists, so b.id === 'builtin.audioenc' is sound; catch → true degrades safely.

The other output branches (whep_output, rtmp_output, mpegtssrt_output) legitimately keep :video_in — different block types, not recorders — so the scoped change is correct and complete.

Tests

recording-wiring.test.ts asserts video_in_0, audio_in_0, the AAC encoder (codec: aac, audio_in→encoded_out), filename_prefix, and the picture-only + recording-no-audio warning path when Strom lacks the block. Warning plumbing (save on activationWarnings, WS ERROR frame on go-live gated by !signal.aborted, connect-time re-send gated by status==='active', clear on re-activation and on failed activation) is covered.

CI

build-and-test PASS (1m3s). No lint script in this repo (per code-quality.md).

No secrets. Co-author trailer present on commits.

Safe to merge standalone on current main: yes — MERGEABLE, only BEHIND (no conflict). #359 is built on top and should merge after.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants