Split HTTP/3 module out from QUIC

[pimterry]

This PR creates a new node:http3 module, extracting the HTTP/3 parts of node:quic there, and making a few surgical additional changes to the APIs around that to make that effective. This exposes an API very similar to the current one for HTTP/3, but makes pure QUIC much simpler, and offers more flexibility to separate the two.

It's a large change! But the substantial majority is just moving code around (5.6k additions, 4.7k deletions). This doesn't drastically change the internal implementation details, it's mostly about modifying the structure to significantly change the user-facing API & behaviour on top.

I've tried to break it down by commits relatively cleanly, so that reviewing commit-by-commit might be helpful, up to you. There's been a fair bit of Claude's hands in here to cover this much ground, but I've been through every line myself quite a few times as well.

Background

QUIC & HTTP/3 are obviously linked but separate protocols. The current API mixes them together:

• All QUIC clients & servers default to HTTP/3 ALPN.
• If an h3 ALPN is ever negotiated within QUIC, we automatically do the whole HTTP/3 dance for you and return an HTTP/3 session that behaves quite differently to pure QUIC, with no configuration or opt-out.
• Pure QUIC streams have various HTTP/3 behaviours, options & methods that are ignored/unusable/broken for pure QUIC, e.g. quicStream.sendHeaders() or setPriority() - both HTTP/3 only features.
• HTTP/3 streams have some QUIC behaviours, options & methods that are ignored/unusable/broken for HTTP/3, e.g. h3Stream.sendDatagram() sent QUIC datagrams, which aren't valid HTTP/3 datagrams, and you can't just read & write arbitrary data to an HTTP/3 stream - you need headers etc.

This doesn't match the rest of our API (TLSSockets do not automatically turn into HTTP requests if they match the right ALPN). It is a fairly convenient API for purely HTTP/3 use, but it's awkward for many edge cases and for low-level control, and it's going to be increasingly limiting, as we do future development and want to add more QUIC-only or HTTP/3-only options & features, since all QUIC or HTTP/3 changes will directly affect the API used by the other.

I think we can fix it fairly cleanly so I've taken a shot at it. Demo (bring your own key/cert):

import * as http3 from 'node:http3';
import { bytes } from 'stream/iter';

// Server:
const endpoint = await http3.listen((session) => {
  session.onstream = (stream) => {
    stream.onheaders = (headers) => {
      stream.sendHeaders({ ':status': '200', 'content-type': 'text/plain' });
      stream.writer.writeSync(`hello h3 — you asked for ${headers[':path']}`);
      stream.writer.endSync();
    };
  };
}, { sni: { '*': { keys: [key], certs: [cert] } } });

// Client:
const session = await http3.connect(endpoint.address, {
  servername: 'localhost',
  verifyPeer: 'manual',
});
await session.opened;

const stream = await session.request({
  ':method': 'GET', ':path': '/hello', ':scheme': 'https', ':authority': 'localhost',
}, { onheaders: (h) => console.log('status:', h[':status']) });

console.log('body:', new TextDecoder().decode(await bytes(stream)));

await session.close();
await endpoint.close();

Core Changes

To do this, I've made a few core changes:

• Created a brand new node:http3 module, behind the same --experimental-quic flag.
• Removed default ALPN from QUIC, and dropped the automatic ALPN-based behaviour. If you use QUIC directly, you handle ALPN yourself (either read session.alpnProtocol and act appropriately, or you just set a single ALPN value in your options and then you always know what you're getting).
• Dropped all HTTP/3 options & methods from QUIC servers, sessions & streams.
• The whole Application structure is shrunk significantly overall. The Application interface is now pure QUIC, not HTTP related, Applications are now optional, and DefaultApplication is gone. Logic has moved into the QUIC Session, Http3Application, a new Http3Binding (JS binding for streams), and the rest of the HTTP/3 API. Applications are now pure integration points for built-in implementations to integrate with QUIC internals.
• Created new Http3Session and Http3Stream classes, which wrap QUIC streams and expose HTTP/3 APIs, with no direct dependency on ngtcp2.
  • Relatively close to the QUIC equivalent, but some notable changes: for clients, createBidirectionalStream() is now request() (much clearer imo) and for servers you now set onheaders on the stream, not the entire session (to simplify the API and match HTTP/2).
  • HTTP/3 settings are now exposed here explicitly, via options.settings and onsettings (extracting onapplication from QUIC etc).
• Created two ways to use these HTTP/3 classes:
  • Use listen() and connect() from node:http3 directly - very similar API to today, but using sensible HTTP/3 defaults, and producing Http3Sessions (and Http3Streams) not QuicSessions with a pure HTTP API.
  • Use QUIC directly to get a session, and then call new Http3Session(quicSession). This is only allowed before the session is 'active' - on the server, that means synchronously in the tick when session is emitted, and on the client it means before opened resolves. This is intended to allow dynamic application configuration in any way you like - by ALPN, SNI, remote address, you name it. The client use case here is more limited than the server one, but that API only supports single-ALPN right now anyway, and I have a follow-up plan to expand that side later.
• Changed the server onsession callback to fire at the end of ClientHello parsing, not the start.
  • This is necessary to make the new Http3Session(quicSession) useful, since right now when the session event fires the handshake hasn't been processed at all, so you can't read alpnProtocol or servername which are clearly useful.
  • This is still 0-RTT, but by waiting until slightly later in OpenSSL's process we also avoid exposing invalid sessions (wrong SNI/unacceptable ALPN/completely corrupted entirely) and ensure that servers can always read alpnProtocol/servername and all other details immediately (you could read those before by waiting for opened, but that does mean waiting a full round-trip for the handshake to complete - in practice, adding this delay actually reduces time to access a usable session).
  • To ensure events emitted during ClientHello processing (0RTT streams) aren't missed, we defer emits automatically until immediately after the session event.

Bonus improvements

There's a few other nice changes that fall out of this en route:

• Custom app ticket data, for 0-RTT without HTTP/3. If you set the appTicketData option to a fixed value (HTTP/3 settings equivalent) it will be included in server tickets, validated in resumption (exact match for now) and matching sessions can do 0RTT just like HTTP/3, if enabled. Generic version of HTTP/3's settings tokens & validation. Mutually exclusive with Http3Session etc - only for other QUIC protocols. Previous 0RTT was only usable if you have no server settings to validate before resume.

• Dropping datagrams from the HTTP/3 API. The existing functionality was pure QUIC and would not work as expected. Now we have separate APIs it's easy to not expose this for HTTP/3 yet. We'll add this back with proper HTTP/3 support shortly, once some new changes land in nghttp3: quic: http/3 datagrams partially diverge from spec #63891.

• HTTP/3 priority didn't work correctly: request({ priority }) applied priority with nghttp3_conn_set_client_stream_priority before submitting the request. This was silently returning an NGHTTP3_ERR_STREAM_NOT_FOUND error, and the priority was ignored. This nghttp3 API is intended for sending PRIORITY_UPDATE frames, not setting the initial priority of a stream. We now do that correctly instead (sending it in the headers).

• SendPendingData in the nwrite == 0 branch (congestion or nothing to send) would never reach TryWritePendingDatagram, so on idle connections datagrams would be queued but never sent. We now check for these specifically.

• ondatagram set before the stream had an id was never successfully registered

• Header validation in createBidirectionalStream (now request()) ran after the stream was created, so invalid headers leaked the stream. We now validate up front.

• Assorted small performance improvements (caching to avoid repeated FindStream lookups, using a single timestamp for each batch) that popped up en route, such that request-per-second for HTTP/3 here is actually fractionally faster than before in my benchmarks.

Deferrals

There's plenty of other things I'd like to do here, but this is already quite large, I'm trying to pull other changes into separate PRs for easier management & review.

As noted above, HTTP/3 datagrams aren't yet in and late-attach for clients is a bit limited, also it'd be nice to have separate HTTP/3 server/client session & stream APIs (like HTTP/3, there's various other small details of the HTTP/3 API that definitely could be refined, appTicketData could be more flexible, there's assorted small doc nits and bugs I've found but I don't want to bundle into this, etc.

I'd really like to keep this focused and break as much of that out into separate future PRs. Once this big refactor is in, a lot of that work can happen in parallel! Definitely happy to debate the core change & shape of this (it is a bit bold, let's discuss) but for nits & feedback on smaller details or possible extras, I'd like to lean towards building out future PRs on top if that's possible.

[nodejs-github-bot]

Review requested:

• @nodejs/loaders
• @nodejs/quic
• @nodejs/startup

[jasnell]

I'm firmly -1 on this for the time being. When designig the API explicitly did NOT want Yet Another Top Level HTTP module to add to our already overly complicated node:http and node:http2 split.

What we've discussed in the past is a new, version-agnostic, unified API model that allows for all three in a single interface. Let's call it node:httpx for the sake of discussion.

This node:httpx would sit on top of the node:quic when http3 is used rather than replacing it or fully splitting it out. It really should not need to extend to the C++ layer at all and it should be possible to layer on the quic JS API.

[pimterry]

It really should not need to extend to the C++ layer at all and it should be possible to layer on the quic JS API.

That is entirely possible with the implementation here - it's just slower. We could drop the HTTP/3 module completely from this PR, and userspace could implement it instead.

It's not possible however to do anything like this with the implementation on main today. Because HTTP/3 is welded into the ALPN negotiation, you can't replace it with any other version from JS. That's one of the reasons I'd like to change this.

What we've discussed in the past is a new, version-agnostic, unified API model that allows for all three in a single interface. Let's call it node:httpx for the sake of discussion.

For the client side, this is Undici I think, we agreed at the Paris summit to aim for that to eventually become the primary Node HTTP client API. For the server side, I know @marco-ippolito has a rough outline of an http/web module that's exploring this direction.

I can see a world where we expose http/send (Undici) and http/serve (httpx generic server API) modules that do protocol-generic HTTP. That'd be great.

That said, we still need per-protocol APIs. There are many low-level cases where this is very useful, there are substantial differences and sometimes you care about them. And the QUIC module here does already expose HTTP/3, and I agree it should.

I'm firmly -1 on this for the time being.

Would you prefer this if it was inside the quic module, so there's no top-level http3 module? The PR could be easily changed to instead have node:quic expose listenQuic() and listenHttp3() directly (names to bikeshed, lots of ways to split this) and then everything else here would work as is.

Design details aside, dropping the top-level module part is fine by me if that's what people prefer. I had thought separating them would be better, but that's not the core goal: from my POV the key point is just that QuicSession & QuicStream should be fully separate from Http3Session & Http3Stream, for all the reasons in the description here. As long as that happens, I'm happy, I don't much care where they live.

[metcoder95]

This node:httpx would sit on top of the node:quic when http3 is used rather than replacing it or fully splitting it out. It really should not need to extend to the C++ layer at all and it should be possible to layer on the quic JS API.

Do we have drafts on this already? Would be interesting to see what's in mind for that? I like the idea, but the primitives exposed should allow the different versions to be used (or built) on top of it, and given the nature of the versions, it sounds interesting how to consolidate them into a single API.

(if I got the idea correctly)

[mcollina]

lgtm

[jasnell]

My primary objection is around not having a top level node:http3 module. We absolutely should not. I'm also hesitant to split out more on the c++ side.

[jasnell]

Even if we do split out http3, which I'm far from convinced about, I'd be -1 on this. Future quic application protocols may support headers. The API was very intentionally designed here to provide flexibility. The stream.headers are not there only because of http3

[pimterry]

Future quic application protocols may support headers.

Lots of QUIC protocols do support headers, but they're all very different concepts, and they don't map to the same API. E.g. just from cursory digging: most QUIC protocols with 'headers' like MQTT/AMQP/RTP have them per-message, not per-stream - so stream.headers can't work.

Even if they did match, I still don't think that API should live on all raw QUIC streams. If you're using the raw QUIC API imo you should get the raw data, you shouldn't automatically get a parsed version with interpretation attached unless you explicitly ask for that (i.e. compose it with an application protocol implementation). We should make it easy to ask for that, but not automatic & unavoidable.

In the TCP analogy: many TCP protocols have header-like structures, but we don't (and imo shouldn't) add a headers field or event to all raw sockets. Why is this different?

In future, when implementing a new QUIC app protocol inside Node, with the setup I'm proposing here we'd expose a new class (MqttStream) with its own events & API that match the protocol. If it's close to HTTP then wonderful, we can share code internally and/or make the APIs consistent to easily mix & match them, but we're not obliged to do so if not, and users can opt in & out of that.

OTOH, when a new QUIC app protocol is implemented in userspace JS on top of this API, having a headers API here is actively unhelpful - e.g. you have a QUIC protocol with headers, and a field on all your streams called 'headers', but they don't work together at all and calling sendHeaders() throws The negotiated QUIC application protocol does not support headers even though you have a working implementation of a protocol and it does support headers. That is the current state today.

I really think we should keep QUIC doing pure QUIC, and focus on exposing the core primitives of the protocol here, such that they can be used either raw as-is, or explicitly wrapped with application protocol APIs on top.

[jasnell]

I'll go through this in detail next week once I'm back from traveling. For now I'm still firmly -1 on splitting this. Once I've done a thorough review pass I'll have more thoughts.

[martenrichter]

Misses somewhere the equivalent to the old text :
" both peers must additionally set
[application.enableDatagrams][] to true, which exchanges the
SETTINGS_H3_DATAGRAM setting on the HTTP/3 control stream."

[martenrichter]

Ok, I see that should be in a separate PR. Though settings are independent from implementation. A quic datagram should be still possible to being sent. And until a http datagram is implemented, you can still send datagrams using the quic one, if the settings is present.

[pimterry]

For HTTP/3 - I just dropped HTTP datagram API for now rather than migrate it, since unless you really know what you're doing, it probably doesn't do what you wanted or expected.

Not opposed to migrating the session-level sendDatagram & setting instead if people would prefer that. In any case, this will be coming back (into whatever form of API we have) with proper HTTP/3 support once ngtcp2/nghttp3#529 lands here. Note that HTTP/3 datagrams are mildly fiddly to implement from scratch, as you do need to reimplement QUIC varint en/decoding to correctly set the stream id - but yes it's definitely possible.

I do think long-term for HTTP/3, we should aim to eventually drop the per-session datagram API and expose only the per-stream API (since HTTP/3 strictly requires linking datagrams to streams - the session API is a footgun) but happy to defer that until we have proper support for this.

[martenrichter]

Well beside http/3 there may be other protocols, which will use datagrams with H3, that would mean one should keep it. Otherwise for these protocols, it would be very hard to implement it.

I would see the settings separate from the implementation and keep them. E.g., the W3C webtransport group is very eager to have Node.js WT support for WPT tests, and I would imagine, they would love to set it independently to simulate edge cases. (Also I personally need it to signal WT support while working on this feature).

I wondering with the whole things on nghttp3 we are waiting. Would it make sense to use during the heavy work on the apis awaiting support, not a release of nghttp3? I mean the quic api is in constant flux, and besides the people working on it , I do not think anyone else wants/should use it.

[pimterry]

Well beside http/3 there may be other protocols, which will use datagrams with H3, that would mean one should keep it. Otherwise for these protocols, it would be very hard to implement it.

I'm not sure if you're responding to the long-term or short-term plan here.

In the long-term, I think just exposing H3 stream-level datagrams is correct for those protocols - I think they always build inside H3's stream-id datagram framing, so that's the useful primitive for them. AFAICT you're never ever allowed to send datagrams without the h3 stream-id encoding on an H3 session.

In the short-term though, sure, happy to put raw QUIC datagram support back here if that would help. I'll hold off for now while we're still debating the overall shape of the change generally, but I'll leave this unresolved and do this later once we have a plan.