Skip to content

feat!: attribute embedded data to the stream it arrived on - #1

Merged
srperens merged 2 commits into
mainfrom
feat/embedded-stream-id
Sep 9, 2026
Merged

srperens merged 2 commits into
mainfrom
feat/embedded-stream-id

Conversation

@srperens

@srperens srperens commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Makes EFP's embedded-data channel usable on a multi-stream feed, and fixes three
defects found along the way. Follow-up to Eyevinn/strom#700, which wired the
channel into Strom's EFP blocks and hit the ceiling this removes.

The problem

Embedded data could be addressed to a stream on the way in but not recovered on
the way out. efpdemux published one embedded pad whose caps carried only
data-type, so a receiver got bytes it could not attribute to a media stream.
The channel was write-only. For a single-stream feed that is workable; for a
provenance sidecar that has to say which stream a manifest describes, it
removes most of the value.

The stream ID was never missing from the protocol

vendor/efp/ElasticFrameProtocol.cpp:82 invokes the C-API embedded callback as
(data, size, data_type, rPacket->mPts, ctx). rPacket->mStreamID is fourteen
lines further down, at :96, in the same function, for the same packet. The C
API typedef (efp_c_api/elastic_frame_protocol_c_api.h:57-58) just has no
parameter for it.

So the fix is not to fork the vendored C++ — it is to stop using that callback.
The receiver now leaves the C library's embedded callback unregistered, which
makes gotData deliver the frame whole, and splits the embedded blocks off the
front in Rust, where the carrying frame and its stream ID are still in hand.

efp::split_embedded_data mirrors extractEmbeddedData. The one thing it
cannot take on faith is the block header's layout: the C++ sender memcpy's
ElasticEmbeddedHeader, so the wire layout is the platform's struct layout
rather than a declared format. embedded_header_roundtrips_through_the_c_library
pins it by building a frame with the C library's own efp_add_embedded_data and
parsing it back.

Considered and rejected: correlating the existing embedded callback with the
frame callback that follows it. The ordering holds, but a frame that fails the
bufferOutOfBounds check at :87 fires its embedded callbacks and then never
delivers the frame, so orphaned blocks would attach to the next frame — silent
mis-attribution, which for provenance is worse than no data.

What that enables

efpdemux gives each stream its own embedded_<stream-id> src pad, in
place of one embedded_pad: Mutex<Option<gst::Pad>> returned for every stream
and data type. Caps carry stream-id as well as data-type, and a stream that
switches data type now renegotiates instead of carrying the new type under the
first one's caps.

efpmux rejects embed caps that omit stream-id or data-type.

That last one closes a silent hole. Both fields were read with unwrap_or(0),
and StreamIdAllocator::new() starts at next: 1 with release asserting the
id is non-zero — so stream 0 is never allocated to a sink pad. Data addressed
there, which is exactly what omitting the field gave you, went into
pending_embeds[0] and was never drained: no error, no output, and a map that
grew for the life of the pipeline. Data for a stream that carries no media is
now dropped with a warning, and the per-stream queue is bounded.

Two more defects, found because the send side had no tests

efpmux's embed_%u pad had zero coverage. The only embedded test drove
efpdemux from hand-encoded frames, so nothing exercised the muxer or the two
elements together. Writing that coverage turned up:

Multi-block chains were built in the wrong order. add_embedded_data
prepends, so the last-block flag belongs on the block written first, which
ends up last on the wire. The loop flagged the final iteration instead, putting
it on the block the receiver reads first. The receiver stops there, so every
earlier block was handed to the media stream as payload. With three blocks
queued, one arrived and two silently corrupted the media. Reverting just this
fix reproduces it exactly:

---- several_blocks_queued_for_one_stream_all_arrive ----
  left:  [[116, 104, 105, 114, 100]]                                     // "third"
  right: [[102,105,114,115,116], [115,101,99,111,110,100], [116,104,105,114,100]]

An embed pad requested without caps was named from an unset stream ID, so
every such pad asked for embed_0 and the second request failed. Strom worked
around this by naming pads explicitly; it is fixed here instead.

Tests

47 pass locally on macOS (GStreamer 1.26.6, cargo 1.97.1), fmt and
clippy --all-targets --all-features -D warnings clean.

New, in efp/tests/embedded.rs (7) and gst-plugin-efp/tests/embedded.rs (6),
the headline one being a real efpmux ! efpdemux pipeline with two media
streams each carrying their own embedded data, asserting each block arrives on a
pad named for its stream with matching caps — which is what was impossible
before.

The existing embedded_data_roundtrip and embedded_data_output are
strengthened to assert attribution and that the preamble is stripped; the latter
also needed its pad-name match updated for the rename. All 12 pre-existing efp
crate tests pass unchanged against the Rust-side extraction, which is the
evidence that the reimplementation is byte-compatible with the C++ path.

Not a guard: embedded_data_for_a_stream_with_no_media_produces_nothing
states the boundary but does not guard the fix — queueing forever and dropping
both deliver nothing, and pending_embeds growth is not observable from
outside the element. The memory bound is covered by reading the code only. Said
so in the test's own doc comment.

Also green on Linux in this PR's own CI run — the first this repository has ever
had — with all 47 tests executed, not skipped. The first run failed on
no element "h264parse" because the workflow was missing
gstreamer1.0-plugins-bad; every other test passed, and the second commit fixes
the package list.

Not covered: Windows and macOS in CI. The macOS numbers above are from a
local run.

CI

This repository had no CI. Strom pins it as a git dependency and builds it into
every release, so nothing has ever verified the plugin it links against. The
added workflow does fmt, clippy and tests on Linux, with the submodule checkout
and the GStreamer plugin packages the pipeline tests need.

Breaking changes, and the Strom follow-up

  • efpdemux's embedded pad is now embedded_%u.
  • Its caps carry stream-id.
  • efp::EmbeddedData gains stream_id.
  • efpmux rejects embed caps without both addressing fields.

Versions bumped to 0.4.0.

Strom matches the embedded pad by caps name rather than pad name, so the rename
does not reach it. Once this is tagged, Strom can repin and lift the
MAX_DATA_TRACKS = 1 limit added in Eyevinn/strom#700 — the input side can now
genuinely carry N data tracks, which is what that limit was placed to prevent
pretending.

🤖 Generated with Claude Code

srperens and others added 2 commits September 9, 2026 11:17
Embedded data could be addressed to a stream on the way in but not
recovered on the way out, which made the channel write-only: a receiver
got bytes with a data type and no way to say which media stream they
described. For a single-stream feed that is workable. For anything
multi-stream it removes most of the point.

The stream ID was never missing from the protocol. In
ElasticFrameProtocol.cpp the C-API embedded callback is invoked as
(data, size, data_type, pts, ctx) while rPacket->mStreamID sits fourteen
lines further down in the same function, for the same packet. The C API
typedef simply has no parameter for it.

So stop using that callback. The receiver now leaves the C library's
embedded callback unregistered, which makes the frame arrive whole, and
splits the embedded blocks off the front in Rust — where the carrying
frame, and its stream ID, are still in hand. split_embedded_data mirrors
extractEmbeddedData and is pinned against the C library's own writer by
a round-trip test, since the block header's layout comes from a memcpy'd
struct rather than a declared wire format.

With the stream ID available:

- efpdemux gives each stream its own `embedded_<stream-id>` src pad,
  replacing the single cached `embedded` pad it returned for every
  stream and data type. Pad caps now carry `stream-id` as well as
  `data-type`, and a stream that changes data type renegotiates rather
  than carrying the new type under the first one's caps.

- efpmux rejects embed caps that omit `stream-id` or `data-type`. Both
  used to default to 0, and stream 0 is never allocated to a sink pad,
  so data addressed there was queued forever: no error, no output, and a
  map that grew for the life of the pipeline. Data for a stream that
  carries no media is dropped with a warning, and the per-stream queue
  is bounded.

Two further fixes found while testing the send side, which had no
coverage at all:

- efpmux built multi-block embedded chains in the wrong order.
  add_embedded_data prepends, so the last-block flag belongs on the
  block written first. Flagging the final iteration put it on the block
  the receiver reads first, which ended the chain there and handed every
  earlier block to the media stream as payload. With three blocks
  queued, two were silently corrupting the media.

- An embed pad requested without caps was named from an unset stream ID,
  so every such pad asked to be `embed_0` and the second request failed.

BREAKING CHANGE: efpdemux's `embedded` pad is now `embedded_%u`, its caps
carry `stream-id`, efp::EmbeddedData gains a `stream_id` field, and
efpmux rejects embed caps without both addressing fields.

Also adds CI, which this repository did not have.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first CI run failed on `no element "h264parse"` in h264_roundtrip and
mux_sink_template_forces_h264_byte_stream. h264parse is in plugins-bad,
which the workflow did not install; every other test passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@srperens
srperens merged commit d6fd3bd into main Sep 9, 2026
1 check passed
@srperens
srperens deleted the feat/embedded-stream-id branch September 9, 2026 09:38
srperens added a commit to Eyevinn/strom that referenced this pull request Sep 9, 2026
…uting (#778)

Three things, all in the EFP embedded-data channel.

**The channel carried nothing in a real flow.** `efpsrt_output`'s `build`
linked `data_input_<i>` to the `embed_%u` pad it had just requested.
`gst_bin_add` drops any link whose peer is outside the bin, and the
pipeline builder adds every element a block returns before it links
anything, so the link was gone by the time data flowed and the source
feeding `data_in_0` stopped with not-linked. Plain GStreamer, not
EFP-specific. The fix reports the link through `internal_links` and lets
the pipeline builder make it after the elements are in the bin. Video and
audio were never affected: they link from caps probes, by which time
everything is already in the pipeline.

Nothing caught it because every test in `efpsrt_embedded_data_test.rs`
inspects `build`'s output, where the link genuinely was. That test now
asserts the reported link and says in its doc comment why the old form
passed while the channel was dead.

**Repin to gst-plugin-efp v0.4.0**, which gives each EFP stream its own
`embedded_<stream-id>` pad with `stream-id` on the caps. That retires the
`num_data_tracks > 1` rejection #700 added as a stopgap against a demuxer
that had one shared `embedded` pad. The output property description is
corrected rather than extended: v0.3.0 defaulted missing caps fields to 0
and buffered the data forever, v0.4.0 rejects those caps and drops data
addressed to a stream that carries no media.

**`data_stream_ids`** pins each data track to a sender stream, so
`data_out_0` means the same thing on every run instead of being whichever
embedded pad arrived first. Empty is the default and keeps arrival-order
filling, so no existing flow changes. A list that does not name exactly
one stream per track is rejected at build, as are stream 0, duplicates and
non-integers.

Two new end-to-end tests join both block builders over a real SRT
connection and read the far end; both ran in CI on Linux rather than
skipping. Reverting only the link fix reproduces the not-linked failure,
so they guard rather than demonstrate.

Follow-up to #700 and Eyevinn/efp#1. Answers the wire round trip asked for
in #691.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant