quic: a stream's writes in one engine cycle go out as one STREAM frame - #280
Merged
Merged
Conversation
… packet With the congestion window full and an ACK due, ngtcp2 answers a stream write with a packet holding only that ACK and reports the stream frame unwritten (pdatalen -1). PumpOut marked a bare FIN sent on any packet, so that ACK swallowed it: the stream was done as far as the replay list knew, nothing retried it, and the peer waited for an end that never came. Under h3x, 22 of 1.5M streamed HTTP/3 responses hung until the client gave up - 22 FINs counted this way, 22 streams never closed. A bare FIN is now sent only when the write reports 0 bytes consumed, as ngtcp2 documents.
Inside an engine cycle nothing reaches the wire before the cycle ends - its sends are GSO-batched until then - yet each SendStream was packetized on the spot, a packet per call. A streamed HTTP/3 response is three calls (headers, body, FIN) and went out as three packets where a buffered one is one. Sends made inside a cycle are now packetized at its end, every queued chunk of a stream in one write (new shim export iq_conn_writev, ABI 6), so a single STREAM frame spans them. Close and the send-retention backstop packetize what the cycle queued before their CONNECTION_CLOSE, as it would have gone out ahead of it before. Streamed 1 KiB on 2 reactors under h2load: ioxide.http3 365K -> 991K req/s, nghttp3 391K -> 631K; buffered +3% to +10%. libioxide_ngtcp2.so rebuilt with scripts/build-ngtcp2-native.sh.
# Conflicts: # tests/Ioxide.Tests.E2E/Protocols/QuicEngineTests.cs
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #278 (base
fix/quic-bare-fin-ack-only) - retarget to main once that merges.Inside an engine cycle nothing reaches the wire before the cycle ends - its sends are GSO-batched until
EndEngineCycle- yet eachSendStreamwas packetized on the spot, a packet per call. A streamed HTTP/3 response is three calls (headers, body, FIN), so it cost three packets where a buffered one costs one: 365K req/s against ~1.0M for 1 KiB.SendStreaminside a cycle only queues. The cycle's end packetizes every stream with bytes waiting, then the engine's own frames (FlushEgress, now run fromEndEngineCycle). Wire timing is unchanged. The capacity signalFlushEgressused to carry isSignalSendCapacity, left whereFlushEgresswas.PumpOuthands ngtcp2 up to 8 of a stream's unsent chunks in one write, through a new shim exportiq_conn_writev, so one STREAM frame spans them (ngtcp2 puts one STREAM frame in a packet). ABI 5 → 6;libioxide_ngtcp2.sorebuilt withscripts/build-ngtcp2-native.sh, pinned refs unchanged.Closeand the send-retention backstop packetize what the cycle queued before writing CONNECTION_CLOSE, since it went out ahead of it before. A teardown over an engine error does not: the next write could hit the same error.Tests
quic: writes to a stream in one engine cycle arrive as one STREAM frame(new) - head, body and a bare FIN written in one cycle, STREAM frames counted at the client:iq_conn_writev): FAIL - 2 framesquic: the send-retention backstop tells the peer before it drops the connectionfailed on the first version of this change (the echo queued before the flood was dropped), which is how the backstop's flush got in.All suites pass: E2E 233, Unit 60, Chaos 47, Http 44, Tls 151 (the 7 kTLS tests skip without sudo), File 4. With #276, #277 and #279 merged in as well: E2E 241, the rest as above.
Bench
h2load, 16 conns × 32 streams, 2 reactors, 4 interleaved rounds with a second build of main as control; median req/s (CPU per request), and the within-round median vs main:
nghttp3 gets all of its gain from the deferral: its egress already copies a stream's frames into one buffer per
SendStream. Buffered ioxide.http3 ran two of its four rounds below 95% server utilisation; in the two saturated ones its CPU per request was 1-3% above deferral alone - inside the noise, but on the same side both times.