feat!: attribute embedded data to the stream it arrived on - #1
Merged
Merged
Conversation
Embedded data could be addressed to a stream on the way in but not recovered on the way out, which made the channel write-only: a receiver got bytes with a data type and no way to say which media stream they described. For a single-stream feed that is workable. For anything multi-stream it removes most of the point. The stream ID was never missing from the protocol. In ElasticFrameProtocol.cpp the C-API embedded callback is invoked as (data, size, data_type, pts, ctx) while rPacket->mStreamID sits fourteen lines further down in the same function, for the same packet. The C API typedef simply has no parameter for it. So stop using that callback. The receiver now leaves the C library's embedded callback unregistered, which makes the frame arrive whole, and splits the embedded blocks off the front in Rust — where the carrying frame, and its stream ID, are still in hand. split_embedded_data mirrors extractEmbeddedData and is pinned against the C library's own writer by a round-trip test, since the block header's layout comes from a memcpy'd struct rather than a declared wire format. With the stream ID available: - efpdemux gives each stream its own `embedded_<stream-id>` src pad, replacing the single cached `embedded` pad it returned for every stream and data type. Pad caps now carry `stream-id` as well as `data-type`, and a stream that changes data type renegotiates rather than carrying the new type under the first one's caps. - efpmux rejects embed caps that omit `stream-id` or `data-type`. Both used to default to 0, and stream 0 is never allocated to a sink pad, so data addressed there was queued forever: no error, no output, and a map that grew for the life of the pipeline. Data for a stream that carries no media is dropped with a warning, and the per-stream queue is bounded. Two further fixes found while testing the send side, which had no coverage at all: - efpmux built multi-block embedded chains in the wrong order. add_embedded_data prepends, so the last-block flag belongs on the block written first. Flagging the final iteration put it on the block the receiver reads first, which ended the chain there and handed every earlier block to the media stream as payload. With three blocks queued, two were silently corrupting the media. - An embed pad requested without caps was named from an unset stream ID, so every such pad asked to be `embed_0` and the second request failed. BREAKING CHANGE: efpdemux's `embedded` pad is now `embedded_%u`, its caps carry `stream-id`, efp::EmbeddedData gains a `stream_id` field, and efpmux rejects embed caps without both addressing fields. Also adds CI, which this repository did not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first CI run failed on `no element "h264parse"` in h264_roundtrip and mux_sink_template_forces_h264_byte_stream. h264parse is in plugins-bad, which the workflow did not install; every other test passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
srperens
added a commit
to Eyevinn/strom
that referenced
this pull request
Sep 9, 2026
…uting (#778) Three things, all in the EFP embedded-data channel. **The channel carried nothing in a real flow.** `efpsrt_output`'s `build` linked `data_input_<i>` to the `embed_%u` pad it had just requested. `gst_bin_add` drops any link whose peer is outside the bin, and the pipeline builder adds every element a block returns before it links anything, so the link was gone by the time data flowed and the source feeding `data_in_0` stopped with not-linked. Plain GStreamer, not EFP-specific. The fix reports the link through `internal_links` and lets the pipeline builder make it after the elements are in the bin. Video and audio were never affected: they link from caps probes, by which time everything is already in the pipeline. Nothing caught it because every test in `efpsrt_embedded_data_test.rs` inspects `build`'s output, where the link genuinely was. That test now asserts the reported link and says in its doc comment why the old form passed while the channel was dead. **Repin to gst-plugin-efp v0.4.0**, which gives each EFP stream its own `embedded_<stream-id>` pad with `stream-id` on the caps. That retires the `num_data_tracks > 1` rejection #700 added as a stopgap against a demuxer that had one shared `embedded` pad. The output property description is corrected rather than extended: v0.3.0 defaulted missing caps fields to 0 and buffered the data forever, v0.4.0 rejects those caps and drops data addressed to a stream that carries no media. **`data_stream_ids`** pins each data track to a sender stream, so `data_out_0` means the same thing on every run instead of being whichever embedded pad arrived first. Empty is the default and keeps arrival-order filling, so no existing flow changes. A list that does not name exactly one stream per track is rejected at build, as are stream 0, duplicates and non-integers. Two new end-to-end tests join both block builders over a real SRT connection and read the far end; both ran in CI on Linux rather than skipping. Reverting only the link fix reproduces the not-linked failure, so they guard rather than demonstrate. Follow-up to #700 and Eyevinn/efp#1. Answers the wire round trip asked for in #691. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes EFP's embedded-data channel usable on a multi-stream feed, and fixes three
defects found along the way. Follow-up to Eyevinn/strom#700, which wired the
channel into Strom's EFP blocks and hit the ceiling this removes.
The problem
Embedded data could be addressed to a stream on the way in but not recovered on
the way out.
efpdemuxpublished oneembeddedpad whose caps carried onlydata-type, so a receiver got bytes it could not attribute to a media stream.The channel was write-only. For a single-stream feed that is workable; for a
provenance sidecar that has to say which stream a manifest describes, it
removes most of the value.
The stream ID was never missing from the protocol
vendor/efp/ElasticFrameProtocol.cpp:82invokes the C-API embedded callback as(data, size, data_type, rPacket->mPts, ctx).rPacket->mStreamIDis fourteenlines further down, at
:96, in the same function, for the same packet. The CAPI typedef (
efp_c_api/elastic_frame_protocol_c_api.h:57-58) just has noparameter for it.
So the fix is not to fork the vendored C++ — it is to stop using that callback.
The receiver now leaves the C library's embedded callback unregistered, which
makes
gotDatadeliver the frame whole, and splits the embedded blocks off thefront in Rust, where the carrying frame and its stream ID are still in hand.
efp::split_embedded_datamirrorsextractEmbeddedData. The one thing itcannot take on faith is the block header's layout: the C++ sender memcpy's
ElasticEmbeddedHeader, so the wire layout is the platform's struct layoutrather than a declared format.
embedded_header_roundtrips_through_the_c_librarypins it by building a frame with the C library's own
efp_add_embedded_dataandparsing it back.
Considered and rejected: correlating the existing embedded callback with the
frame callback that follows it. The ordering holds, but a frame that fails the
bufferOutOfBoundscheck at:87fires its embedded callbacks and then neverdelivers the frame, so orphaned blocks would attach to the next frame — silent
mis-attribution, which for provenance is worse than no data.
What that enables
efpdemuxgives each stream its ownembedded_<stream-id>src pad, inplace of one
embedded_pad: Mutex<Option<gst::Pad>>returned for every streamand data type. Caps carry
stream-idas well asdata-type, and a stream thatswitches data type now renegotiates instead of carrying the new type under the
first one's caps.
efpmuxrejects embed caps that omitstream-idordata-type.That last one closes a silent hole. Both fields were read with
unwrap_or(0),and
StreamIdAllocator::new()starts atnext: 1withreleaseasserting theid is non-zero — so stream 0 is never allocated to a sink pad. Data addressed
there, which is exactly what omitting the field gave you, went into
pending_embeds[0]and was never drained: no error, no output, and a map thatgrew for the life of the pipeline. Data for a stream that carries no media is
now dropped with a warning, and the per-stream queue is bounded.
Two more defects, found because the send side had no tests
efpmux'sembed_%upad had zero coverage. The only embedded test droveefpdemuxfrom hand-encoded frames, so nothing exercised the muxer or the twoelements together. Writing that coverage turned up:
Multi-block chains were built in the wrong order.
add_embedded_dataprepends, so the last-block flag belongs on the block written first, which
ends up last on the wire. The loop flagged the final iteration instead, putting
it on the block the receiver reads first. The receiver stops there, so every
earlier block was handed to the media stream as payload. With three blocks
queued, one arrived and two silently corrupted the media. Reverting just this
fix reproduces it exactly:
An embed pad requested without caps was named from an unset stream ID, so
every such pad asked for
embed_0and the second request failed. Strom workedaround this by naming pads explicitly; it is fixed here instead.
Tests
47 pass locally on macOS (GStreamer 1.26.6, cargo 1.97.1), fmt and
clippy --all-targets --all-features -D warningsclean.New, in
efp/tests/embedded.rs(7) andgst-plugin-efp/tests/embedded.rs(6),the headline one being a real
efpmux ! efpdemuxpipeline with two mediastreams each carrying their own embedded data, asserting each block arrives on a
pad named for its stream with matching caps — which is what was impossible
before.
The existing
embedded_data_roundtripandembedded_data_outputarestrengthened to assert attribution and that the preamble is stripped; the latter
also needed its pad-name match updated for the rename. All 12 pre-existing
efpcrate tests pass unchanged against the Rust-side extraction, which is the
evidence that the reimplementation is byte-compatible with the C++ path.
Not a guard:
embedded_data_for_a_stream_with_no_media_produces_nothingstates the boundary but does not guard the fix — queueing forever and dropping
both deliver nothing, and
pending_embedsgrowth is not observable fromoutside the element. The memory bound is covered by reading the code only. Said
so in the test's own doc comment.
Also green on Linux in this PR's own CI run — the first this repository has ever
had — with all 47 tests executed, not skipped. The first run failed on
no element "h264parse"because the workflow was missinggstreamer1.0-plugins-bad; every other test passed, and the second commit fixesthe package list.
Not covered: Windows and macOS in CI. The macOS numbers above are from a
local run.
CI
This repository had no CI. Strom pins it as a git dependency and builds it into
every release, so nothing has ever verified the plugin it links against. The
added workflow does fmt, clippy and tests on Linux, with the submodule checkout
and the GStreamer plugin packages the pipeline tests need.
Breaking changes, and the Strom follow-up
efpdemux'sembeddedpad is nowembedded_%u.stream-id.efp::EmbeddedDatagainsstream_id.efpmuxrejects embed caps without both addressing fields.Versions bumped to 0.4.0.
Strom matches the embedded pad by caps name rather than pad name, so the rename
does not reach it. Once this is tagged, Strom can repin and lift the
MAX_DATA_TRACKS = 1limit added in Eyevinn/strom#700 — the input side can nowgenuinely carry N data tracks, which is what that limit was placed to prevent
pretending.
🤖 Generated with Claude Code