Skip to content

quic: a stream's writes in one engine cycle go out as one STREAM frame - #280

Merged
MDA2AV merged 3 commits into
mainfrom
perf/quic-packetize-at-cycle-end
Oct 4, 2026
Merged

MDA2AV merged 3 commits into
mainfrom
perf/quic-packetize-at-cycle-end

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Stacked on #278 (base fix/quic-bare-fin-ack-only) - retarget to main once that merges.

Inside an engine cycle nothing reaches the wire before the cycle ends - its sends are GSO-batched until EndEngineCycle - yet each SendStream was packetized on the spot, a packet per call. A streamed HTTP/3 response is three calls (headers, body, FIN), so it cost three packets where a buffered one costs one: 365K req/s against ~1.0M for 1 KiB.

  • SendStream inside a cycle only queues. The cycle's end packetizes every stream with bytes waiting, then the engine's own frames (FlushEgress, now run from EndEngineCycle). Wire timing is unchanged. The capacity signal FlushEgress used to carry is SignalSendCapacity, left where FlushEgress was.
  • PumpOut hands ngtcp2 up to 8 of a stream's unsent chunks in one write, through a new shim export iq_conn_writev, so one STREAM frame spans them (ngtcp2 puts one STREAM frame in a packet). ABI 5 → 6; libioxide_ngtcp2.so rebuilt with scripts/build-ngtcp2-native.sh, pinned refs unchanged.
  • Close and the send-retention backstop packetize what the cycle queued before writing CONNECTION_CLOSE, since it went out ahead of it before. A teardown over an engine error does not: the next write could hit the same error.

Tests

quic: writes to a stream in one engine cycle arrive as one STREAM frame (new) - head, body and a bare FIN written in one cycle, STREAM frames counted at the client:

  • main: FAIL - 3 frames
  • deferral alone (no iq_conn_writev): FAIL - 2 frames
  • this PR: pass - 1 frame

quic: the send-retention backstop tells the peer before it drops the connection failed on the first version of this change (the echo queued before the flood was dropped), which is how the backstop's flush got in.

All suites pass: E2E 233, Unit 60, Chaos 47, Http 44, Tls 151 (the 7 kTLS tests skip without sudo), File 4. With #276, #277 and #279 merged in as well: E2E 241, the rest as above.

Bench

h2load, 16 conns × 32 streams, 2 reactors, 4 interleaved rounds with a second build of main as control; median req/s (CPU per request), and the within-round median vs main:

sample main this PR vs main deferral alone control
Http3/ManagedStreamedBoth 1 KiB 365,162 (5.47 µs) 990,856 (2.00 µs) +171% +57% +0.3%
Http3/Nghttp3Response 1 KiB 391,492 (5.11 µs) 631,052 (3.16 µs) +60% +62% +1.9%
Http3/ManagedBuffered 1 KiB 982,870 (2.02 µs) 1,018,981 (1.88 µs) +3.1% +10.5% +0.8%
Http3/Nghttp3Buffered 744,284 (2.69 µs) 790,716 (2.53 µs) +5.5% +5.5% -2.9%

nghttp3 gets all of its gain from the deferral: its egress already copies a stream's frames into one buffer per SendStream. Buffered ioxide.http3 ran two of its four rounds below 95% server utilisation; in the two saturated ones its CPU per request was 1-3% above deferral alone - inside the noise, but on the same side both times.

MDA2AV added 3 commits October 4, 2026 17:37
… packet

With the congestion window full and an ACK due, ngtcp2 answers a stream write with a
packet holding only that ACK and reports the stream frame unwritten (pdatalen -1).
PumpOut marked a bare FIN sent on any packet, so that ACK swallowed it: the stream
was done as far as the replay list knew, nothing retried it, and the peer waited for an
end that never came. Under h3x, 22 of 1.5M streamed HTTP/3 responses hung until the
client gave up - 22 FINs counted this way, 22 streams never closed. A bare FIN is now
sent only when the write reports 0 bytes consumed, as ngtcp2 documents.
Inside an engine cycle nothing reaches the wire before the cycle ends - its sends are
GSO-batched until then - yet each SendStream was packetized on the spot, a packet per call.
A streamed HTTP/3 response is three calls (headers, body, FIN) and went out as three packets
where a buffered one is one. Sends made inside a cycle are now packetized at its end, every
queued chunk of a stream in one write (new shim export iq_conn_writev, ABI 6), so a single
STREAM frame spans them. Close and the send-retention backstop packetize what the cycle
queued before their CONNECTION_CLOSE, as it would have gone out ahead of it before.

Streamed 1 KiB on 2 reactors under h2load: ioxide.http3 365K -> 991K req/s, nghttp3 391K ->
631K; buffered +3% to +10%. libioxide_ngtcp2.so rebuilt with scripts/build-ngtcp2-native.sh.
# Conflicts:
#	tests/Ioxide.Tests.E2E/Protocols/QuicEngineTests.cs
@MDA2AV
MDA2AV changed the base branch from fix/quic-bare-fin-ack-only to main October 4, 2026 18:42
@MDA2AV
MDA2AV merged commit dcf1a62 into main Oct 4, 2026
1 check passed
@MDA2AV MDA2AV mentioned this pull request Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant