Add raw NV12 passthrough streaming mode (opt-in, for weak/older receivers)#142
Open
firegranger wants to merge 1 commit into
Open
Add raw NV12 passthrough streaming mode (opt-in, for weak/older receivers)#142firegranger wants to merge 1 commit into
firegranger wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclosure
Full transparency: I did not write this code. I had the idea (send frames
uncompressed so an old receiver doesn't have to decode) and tested it end-to-end
on my own hardware; the implementation was written with Claude (Anthropic's
AI). I'm opening this PR so the project and its users benefit — take, adapt or
rewrite it however you see fit. No credit needed.
Offered freely, no strings attached. My only hope is that a capability like this
stays available to everyone in the free/open build rather than being gated behind
a paywall — so anyone with an ageing Mac can give it a second life. 🙂
What
Adds an opt-in raw passthrough mode (
RAW=1) that streams uncompressedNV12 planes instead of HEVC. The receiver renders them directly via its
existing SDL NV12 path, with no decoding at all.
Why
ScreenCaptureKit already delivers frames as NV12 (
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange),which is then HEVC-encoded on the sender and decoded on the receiver. On an
older Intel receiver the HEVC decode is the real bottleneck at high
resolution/refresh — e.g. a 2017 5K iMac can't sustain 5K@60 HEVC decode: the
decoder saturates, latency climbs and the fans spin up.
Since the sender is Apple Silicon and the link is a direct Thunderbolt Bridge,
it's cheaper to just send the pixels raw and let the receiver blit them:
measured link throughput between the two Macs was ~17 Gb/s (iperf3).
Real-world result on a MacBook Pro M5 → 2017 5K iMac (Ventura): smooth 5K@60,
near-zero latency, iMac decoder idle.
How
New wire packet type
0x22(TB_PKT_RAW_FRAME):TBDisplaySenderService.swift,TBMonitorProtocol.swift):when
RAW=1,encode()routes the captured buffer tosendRawFrame(),which packages the two NV12 planes and sends them. Same first-frame session
ack and
pendingVideoPacketsbackpressure as the encoded path. The encoderis never involved.
proto.h,main.c):handle_raw_frame()validates and parsesthe planes, then calls the existing
tb_disp_render_nv12()— the samefunction the HEVC decoder feeds. So the SDL side is unchanged; the decoder is
simply skipped for
0x22packets.Opt-in / compatibility
Entirely gated behind the
RAW=1environment variable. WithRAWunset thebehaviour is byte-for-byte the encoded path as before. A patched receiver
handles both
0x21(HEVC) and0x22(raw) transparently; an unpatchedreceiver simply logs
unknown pkt type=0x22and ignores raw frames.Testing
build_tbreceiver_c_app.shon the Intel iMac.native5kandcrisp2160p60.Notes / possible follow-ups