Skip to content

Add raw NV12 passthrough streaming mode (opt-in, for weak/older receivers)#142

Open
firegranger wants to merge 1 commit into
swellweb:mainfrom
firegranger:feat/raw-nv12-passthrough
Open

Add raw NV12 passthrough streaming mode (opt-in, for weak/older receivers)#142
firegranger wants to merge 1 commit into
swellweb:mainfrom
firegranger:feat/raw-nv12-passthrough

Conversation

@firegranger

Copy link
Copy Markdown

Disclosure

Full transparency: I did not write this code. I had the idea (send frames
uncompressed so an old receiver doesn't have to decode) and tested it end-to-end
on my own hardware; the implementation was written with Claude (Anthropic's
AI)
. I'm opening this PR so the project and its users benefit — take, adapt or
rewrite it however you see fit. No credit needed.

Offered freely, no strings attached. My only hope is that a capability like this
stays available to everyone in the free/open build rather than being gated behind
a paywall — so anyone with an ageing Mac can give it a second life. 🙂

What

Adds an opt-in raw passthrough mode (RAW=1) that streams uncompressed
NV12 planes instead of HEVC. The receiver renders them directly via its
existing SDL NV12 path, with no decoding at all.

Why

ScreenCaptureKit already delivers frames as NV12 (kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange),
which is then HEVC-encoded on the sender and decoded on the receiver. On an
older Intel receiver the HEVC decode is the real bottleneck at high
resolution/refresh — e.g. a 2017 5K iMac can't sustain 5K@60 HEVC decode: the
decoder saturates, latency climbs and the fans spin up.

Since the sender is Apple Silicon and the link is a direct Thunderbolt Bridge,
it's cheaper to just send the pixels raw and let the receiver blit them:

  • Receiver decode cost → zero (GPU YUV→RGB blit only).
  • Latency drops to essentially capture → wire → present.
  • Bandwidth cost is high but fine on Thunderbolt: 5K@60 4:2:0 ≈ 10.6 Gb/s;
    measured link throughput between the two Macs was ~17 Gb/s (iperf3).

Real-world result on a MacBook Pro M5 → 2017 5K iMac (Ventura): smooth 5K@60,
near-zero latency, iMac decoder idle.

How

New wire packet type 0x22 (TB_PKT_RAW_FRAME):

[1 byte format: 1=NV12]
[4 BE uint32 width][4 BE uint32 height]
[4 BE uint32 yStride][4 BE uint32 uvStride]
[Y plane: yStride*height]
[CbCr plane: uvStride*(height/2)]
  • Sender (TBDisplaySenderService.swift, TBMonitorProtocol.swift):
    when RAW=1, encode() routes the captured buffer to sendRawFrame(),
    which packages the two NV12 planes and sends them. Same first-frame session
    ack and pendingVideoPackets backpressure as the encoded path. The encoder
    is never involved.
  • Receiver (proto.h, main.c): handle_raw_frame() validates and parses
    the planes, then calls the existing tb_disp_render_nv12() — the same
    function the HEVC decoder feeds. So the SDL side is unchanged; the decoder is
    simply skipped for 0x22 packets.

Opt-in / compatibility

Entirely gated behind the RAW=1 environment variable. With RAW unset the
behaviour is byte-for-byte the encoded path as before. A patched receiver
handles both 0x21 (HEVC) and 0x22 (raw) transparently; an unpatched
receiver simply logs unknown pkt type=0x22 and ignores raw frames.

Testing

  • Sender built with Xcode; receiver built with the repo's
    build_tbreceiver_c_app.sh on the Intel iMac.
  • Verified end-to-end over Thunderbolt Bridge at native5k and crisp2160p60.

Notes / possible follow-ups

  • Could expose the mode in the UI instead of an env var.
  • Could add 4:2:2 for higher chroma fidelity where bandwidth allows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant