Skip to content

perf(relay): larger download writes: 64 KiB reads, TLS records written together - #124

Merged
Nemu-x merged 1 commit into
devfrom
perf/download-batching
Oct 18, 2026
Merged

Nemu-x merged 1 commit into
devfrom
perf/download-batching

Conversation

@Nemu-x

@Nemu-x Nemu-x commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Download (remote → client) gets the same kind of gain as upload got in #119: fewer, larger writes to the client socket.

Why download was at mihomo's level

Profiles of the bulk download (pprof, Citrus server): ~65–75% of the server CPU is read(2)/write(2), ~23% AES-GCM. The copy read the remote socket 32 KiB at a time, and:

  • TLS (Trojan, VLESS/REALITY, …): tls.Conn.Write sent every 16 KiB record with a write of its own → 2 write(2) per 32 KiB, each carrying a 16 KiB segment through the network stack.
  • Shadowsocks 2022 (the 8–11% sing-box lead): the protocol allows chunks up to 65535 bytes, and sing-box's copy switches to 65535-byte buffers after 512 KB (IncreaseBuffer), so every write is a 64 KiB chunk. Our copy sealed 32 KiB chunks: twice the writes and seals. The protocol code (shadowaead, shadowaead_2022) is the same in both; the difference was only the buffer size.

Changes

  • common/net: the copy from a socket (copyWait) switches to 65535-byte buffers (the largest pool buffer) once a read fills a 32 KiB one. The socket read waiter takes a buffer only once data is there, so an idle connection still holds none; short transfers never switch.
  • getmikan/tls (crypto/tls: write the records of one write together tls#3) and getmikan/utls (crypto/tls: write the records of one write together utls#2): the records of one write go out in one write (up to eight full records), encrypted in place. Nothing is held back, so latency is unchanged. go.mod points at the PR commits (pseudo-versions); after those merge I'll switch to tags.

A read-ahead goroutine for download (like the upload one) was tried as well: no gain on top of this (the server is not CPU-bound on one stream, the client is), so it is not included.

Numbers

4-core Docker bench (test/bench/compare, server cpuset 2 vCPUs), medians of 3 reps, same session; MB/s, CPU s/GB in parentheses. Order: sing-box / mihomo / Citrus before / Citrus after.

sing-box mihomo Citrus before Citrus after
REALITY dl1 1188 1188 1160 1174
REALITY dl32 2707 (0.70) 2836 (0.67) 2802 (0.67) 4235 (0.46)
Trojan dl1 1098 1161 1183 1137
Trojan dl32 2854 (0.67) 2829 (0.67) 2737 (0.68) 3936 (0.49)
SS2022 dl1 2020 1697 1711 2029
SS2022 dl32 4309 (0.44) 3975 (0.48) 3895 (0.49) 4392 (0.44)
XHTTP dl1 – – 363 390
XHTTP dl32 – – 1213 (1.55) 1410 (1.11)
Hysteria2 dl1 – – 754 790
TUIC dl32 – – 420 545

Single-stream TLS downloads stay client-bound (the Xray client uses ~1.6 cores, the server under 0.5): Trojan dl1 1137 (1125–1202) vs 1183 (1155–1209) before, TUIC dl1 481 (464–557) vs 530 (449–560), within the spread. Upload unchanged.

Latency phase (echo RTT on an idle tunnel while an 8-stream download runs), p50 / p99 ms, before → after: REALITY 82.1 / 148 → 56.9 / 104, Trojan 81.7 / 143 → 63.2 / 116, SS2022 69.2 / 149 → 57.0 / 121, XHTTP 1.43 / 19.6 → 0.77 / 2.89, Hysteria2 1.00 / 2.59 → 0.89 / 2.00. New-connection TTFB under load: REALITY 275 → 189 ms, Trojan 157 → 122 ms.

Tests

  • TestCopyConnFromSocketGrowsReads: 16 MiB from a socket to a non-socket destination: data intact, writes above 32 KiB appear, none over 65535.
  • TestCopyConnFromSocketShortReads: short messages stay on pool buffers.
  • Fork tests in their PRs.

…n together

The download of a proxied connection reads the remote socket and writes
an encrypting connection. It read 32 KiB at a time, and every TLS record
(16 KiB) went to the client socket with a write of its own, so a loaded
server spent most of its download time in write(2).

- The copy from a socket switches to 65535-byte buffers (the largest pool
  buffer) once a read fills a 32 KiB one. Shadowsocks 2022 seals each read
  as one chunk (chunks go up to 65535 bytes), as sing-box does after its
  first 512 KB. The socket read waiter still takes a buffer only once data
  is there, so an idle connection holds none.
- getmikan/tls and getmikan/utls write the records of one write together
  (up to eight full records), so a 64 KiB read leaves as one write of four
  records, for Trojan, VLESS/REALITY and everything else on TLS.

4-core Docker bench, medians of 3 (MB/s; CPU s/GB): REALITY dl32 2802 ->
4235 (0.67 -> 0.46), Trojan dl32 2737 -> 3936 (0.68 -> 0.49), SS2022 dl1
1711 -> 2029, dl32 3895 -> 4392 (0.49 -> 0.44), XHTTP dl32 1213 -> 1410
(1.55 -> 1.11). Echo RTT under load is lower (REALITY p50 82 -> 57 ms).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant