feat(io): add fd_read_into — zero-allocation incremental read from an fd - #1476
Merged
Conversation
Closes #1471. The ask was for a streaming read from the fd `os.run_pipe` returns, so a producer/consumer pattern (decode frames as ffmpeg emits them) is possible in pure Aether rather than requiring C to drain the whole pipe first. One correction to the issue's premise, verified before writing anything: `io.fd_read_n` ALREADY does incremental reads from a run_pipe fd. Tested with a child writing two chunks 200ms apart to fd 3 — the first read returns immediately, the second after the gap, then a clean EOF. The issue looked for the call under `std.net` (which has fd_write and fd_close but no read); it lives under `std.io`. So streaming was already possible; what was missing is the ZERO-ALLOCATION form, which is the half that actually matters for the video case the issue describes. io.fd_read_into(fd, buf, length) -> (n, err) fills that gap, mirroring fs.pread_into. It returns as soon as any bytes are available rather than looping to fill, unlike fd_read_n — waiting to fill would stall on a live pipe until the producer sent a whole buffer's worth, which defeats the purpose. n=0 is EOF; a short read is not an error; EINTR is retried. Also worth knowing for anyone reading the issue: run_pipe's fd is the AETHER_IPC_FD side channel on fd 3, NOT the child's stdout. A child has to write to fd 3 deliberately, so `ffmpeg ... -` (which writes stdout) will not feed it as-is. That is a separate gap from this one and not what the issue asked for, so it is left alone here. Tests: tests/regression/test_fd_read_into.ae pins the properties that matter — chunks arrive incrementally BEFORE the child exits (the thing that makes streaming possible), one buffer serves every read, EOF is n==0 so the loop terminates, and bad arguments are refused. Verified it fails (3 assertions) when the read is made drain-to-fill, so it cannot silently rot into the behaviour it exists to prevent. Valgrind-clean, which matters for an API whose whole point is not allocating. Documented the one usability trap I hit writing the test: writing through bytes.data() does not publish the buffer length, so bytes.to_string() returns "" with no error until bytes.set_length() is called. Verified under gcc -Werror and clang -Werror as well as the normal build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1471.
First, a correction to the issue's premise
The issue says streaming from a
run_pipefd is impossible in pure Aether. I tested that before writing anything, andio.fd_read_nalready does incremental reads. A child writing two chunks 200ms apart to fd 3:The issue looked for the call under
std.net— which hasfd_writeandfd_closebut no read — and it lives understd.io, wherestd.taralready uses it.So the streaming half was already there. What was genuinely missing is the zero-allocation form, which is the half that actually matters for the video case the issue describes.
What this adds
Mirrors
fs.pread_into, which was already this shape for files — so a caller streaming from a pipe now gets the same option as one reading a file.n == 0is EOF,0 < n < lengthis a normal short read, EINTR is retried.The key difference from
fd_read_n: it returns as soon as any bytes are available rather than looping to fill the buffer. On a live pipe, waiting to fill would stall until the producer happened to send a whole buffer's worth, which defeats the purpose.At 1080p30, one 8 MB frame per read is ~250 MB/s of allocator churn if every read mints a fresh string. One reused caller-owned buffer avoids it entirely.
A gap the issue didn't mention, left alone deliberately
run_pipe's fd is theAETHER_IPC_FDside channel on fd 3, not the child's stdout. A child must write to fd 3 deliberately, so the issue's motivating example —ffmpeg -f rawvideo -pix_fmt rgba -, which writes stdout — will not feed it as-is.That is a separate gap (no way to capture a child's stdout incrementally) and not what was asked for, so this PR does not touch it. Recorded here and in the commit because anyone following the issue's example will hit it.
Tests
tests/regression/test_fd_read_into.aepins the properties that matter: chunks arrive before the child exits, one buffer serves every read, EOF isn == 0so the loop terminates, and bad arguments are refused.Verified the test has teeth — it fails on 3 assertions when the read is made drain-to-fill, so it cannot silently rot into the behaviour it exists to prevent.
Verification
make ci— C suite 230/230;.ae977/978 with[PASS] regression_test_fd_read_into, count up 977 → 978. The one failure isintegration_http_server_h2, pre-existing and diagnosed inpesky_bug.md.gcc -Werrorandclang -Werrorchecked locally before pushing, per the lesson from fix(fs): name the path and the cause in fs.read / fs.read_binary errors #1469.One usability trap, documented
Writing through
bytes.data(buf)does not publish the buffer's length, sobytes.to_string(buf, n)returns""with no error untilbytes.set_length(buf, n)is called. I hit this writing the test; it is now called out in the wrapper's doc comment and the CHANGELOG. Callers passing the bytes to another extern never need it.🤖 Generated with Claude Code