Skip to content

feat(io): add fd_read_into — zero-allocation incremental read from an fd - #1476

Merged
paul-hammant merged 1 commit into
mainfrom
feat/fd-read-into
Aug 9, 2026
Merged

feat(io): add fd_read_into — zero-allocation incremental read from an fd#1476
paul-hammant merged 1 commit into
mainfrom
feat/fd-read-into

Conversation

@paul-hammant

Copy link
Copy Markdown
Collaborator

Closes #1471.

First, a correction to the issue's premise

The issue says streaming from a run_pipe fd is impossible in pure Aether. I tested that before writing anything, and io.fd_read_n already does incremental reads. A child writing two chunks 200ms apart to fd 3:

first  -> n=6 data='chunk1'     ← returns immediately, child still running
second -> n=6 data='chunk2'     ← after the gap
eof    -> n=0

The issue looked for the call under std.net — which has fd_write and fd_close but no read — and it lives under std.io, where std.tar already uses it.

So the streaming half was already there. What was genuinely missing is the zero-allocation form, which is the half that actually matters for the video case the issue describes.

What this adds

io.fd_read_into(fd, buf, length) -> (n, err)

Mirrors fs.pread_into, which was already this shape for files — so a caller streaming from a pipe now gets the same option as one reading a file. n == 0 is EOF, 0 < n < length is a normal short read, EINTR is retried.

The key difference from fd_read_n: it returns as soon as any bytes are available rather than looping to fill the buffer. On a live pipe, waiting to fill would stall until the producer happened to send a whole buffer's worth, which defeats the purpose.

At 1080p30, one 8 MB frame per read is ~250 MB/s of allocator churn if every read mints a fresh string. One reused caller-owned buffer avoids it entirely.

A gap the issue didn't mention, left alone deliberately

run_pipe's fd is the AETHER_IPC_FD side channel on fd 3, not the child's stdout. A child must write to fd 3 deliberately, so the issue's motivating example — ffmpeg -f rawvideo -pix_fmt rgba -, which writes stdout — will not feed it as-is.

That is a separate gap (no way to capture a child's stdout incrementally) and not what was asked for, so this PR does not touch it. Recorded here and in the commit because anyone following the issue's example will hit it.

Tests

tests/regression/test_fd_read_into.ae pins the properties that matter: chunks arrive before the child exits, one buffer serves every read, EOF is n == 0 so the loop terminates, and bad arguments are refused.

Verified the test has teeth — it fails on 3 assertions when the read is made drain-to-fill, so it cannot silently rot into the behaviour it exists to prevent.

Verification

  • make ci — C suite 230/230; .ae 977/978 with [PASS] regression_test_fd_read_into, count up 977 → 978. The one failure is integration_http_server_h2, pre-existing and diagnosed in pesky_bug.md.
  • Valgrind clean — worth stating explicitly for an API whose entire point is not allocating.
  • gcc -Werror and clang -Werror checked locally before pushing, per the lesson from fix(fs): name the path and the cause in fs.read / fs.read_binary errors #1469.
  • CHANGELOG entry written before committing, not after — the omission that cost four releases their entries.

One usability trap, documented

Writing through bytes.data(buf) does not publish the buffer's length, so bytes.to_string(buf, n) returns "" with no error until bytes.set_length(buf, n) is called. I hit this writing the test; it is now called out in the wrapper's doc comment and the CHANGELOG. Callers passing the bytes to another extern never need it.

🤖 Generated with Claude Code

Closes #1471.

The ask was for a streaming read from the fd `os.run_pipe` returns, so a
producer/consumer pattern (decode frames as ffmpeg emits them) is possible in
pure Aether rather than requiring C to drain the whole pipe first.

One correction to the issue's premise, verified before writing anything:
`io.fd_read_n` ALREADY does incremental reads from a run_pipe fd. Tested with a
child writing two chunks 200ms apart to fd 3 — the first read returns
immediately, the second after the gap, then a clean EOF. The issue looked for
the call under `std.net` (which has fd_write and fd_close but no read); it
lives under `std.io`. So streaming was already possible; what was missing is
the ZERO-ALLOCATION form, which is the half that actually matters for the
video case the issue describes.

io.fd_read_into(fd, buf, length) -> (n, err) fills that gap, mirroring
fs.pread_into. It returns as soon as any bytes are available rather than
looping to fill, unlike fd_read_n — waiting to fill would stall on a live pipe
until the producer sent a whole buffer's worth, which defeats the purpose. n=0
is EOF; a short read is not an error; EINTR is retried.

Also worth knowing for anyone reading the issue: run_pipe's fd is the
AETHER_IPC_FD side channel on fd 3, NOT the child's stdout. A child has to
write to fd 3 deliberately, so `ffmpeg ... -` (which writes stdout) will not
feed it as-is. That is a separate gap from this one and not what the issue
asked for, so it is left alone here.

Tests: tests/regression/test_fd_read_into.ae pins the properties that matter —
chunks arrive incrementally BEFORE the child exits (the thing that makes
streaming possible), one buffer serves every read, EOF is n==0 so the loop
terminates, and bad arguments are refused. Verified it fails (3 assertions)
when the read is made drain-to-fill, so it cannot silently rot into the
behaviour it exists to prevent. Valgrind-clean, which matters for an API whose
whole point is not allocating.

Documented the one usability trap I hit writing the test: writing through
bytes.data() does not publish the buffer length, so bytes.to_string() returns
"" with no error until bytes.set_length() is called.

Verified under gcc -Werror and clang -Werror as well as the normal build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

std: no incremental read from a file descriptor — run_pipe returns an fd Aether cannot read

1 participant