throughput 4c: MTU-aware packetization to cut packet count (fewer per-packet syscalls / TUN / SipHash)

[vxfemboy]

Lever 4c of the single-core throughput campaign (4a send GSO ✓ #55, 4b TUN offload ✓ #56).

The post-4a/4b profiling showed the single-core ceiling is per-packet cost (kernel UDP send sendmsg ~23%, TUN write ~20%, SipHash ~9% — all charged per packet). Raising the tunnel MTU cuts the packet count for the same bytes, so it reduces every per-packet cost proportionally — the cheapest, most broadly-applicable remaining lever.

Scope

• Make the tunnel MTU path-aware rather than a fixed conservative value; probe (PMTUD / PLPMTUD-style) and adapt, falling back cleanly to 1500-safe where the path can't carry more.
• Interacts with FEC symbol sizing and the outer-UDP overhead budget — needs its own design (spec).

Note

• Jumbo frames only help where the whole path supports them; 1500 stays the internet-safe default. The win is largest on controlled/LAN paths.
• Related parked idea: inner-header compression (ROHC-style) — assessed low-priority for yip (loss-fragile + anti-DPI conflict); revisit only after 4c if a header-overhead bottleneck survives.

[vxfemboy]

Investigation complete (2026-07-22): levers measured and bounded

Ran a measurement-first spike (real Transport::encode path + netns) before committing to a mechanism. Full write-up: docs/research/2026-07-22-mtu-packetization-findings.md. Summary:

The send path, and where the cost is

inner → +16 AEAD → FEC symbols (size 1200, **last zero-padded to full symbol_size on the wire**) → +23 wire frame → +11 obf → +28/48 IP/UDP. Two cost tiers: per-inner-packet (AEAD, TUN-write — the two biggest) and per-symbol (sendmsg ~23%, SipHash ~9%, framing). yipd does not currently set the TUN MTU (kernel default 1500); the QUIC MtuDiscovery is only the mimicry transport, not the data plane.

Finding 1 — wire padding makes symbol_size double-edged

Every FEC symbol is zero-padded to full symbol_size on the wire (fec.rs "last zero-padded"; reassembler rejects data.len() != symbol_size; framer sends it untrimmed). So a blind symbol_size increase is a bandwidth regression for small packets (a 576 B packet at symbol_size 9000 → two 9000 B symbols, 3047% overhead).

Finding 2 — two MTU levers, very different reach

• Lever 1 (symbol_size → ~1500): collapses a 1448 B packet from 2 source symbols to 1 (−33% datagrams). Reduces only per-symbol costs, not AEAD/TUN-write. Internet-safe but pays Finding 1's padding penalty on small packets. (Note: 1452 doesn't fit 1448+16 — need 1500.)
• Lever 2 (jumbo inner MTU ~9000 + matching symbol_size): ~6× fewer of everything incl. AEAD + TUN-write. But only where the whole path carries jumbo, and only helps jumbo-sized traffic. This is the issue's "largest on controlled/LAN paths" win.

The issue's premise ("raising MTU reduces every per-packet cost proportionally") holds only for Lever 2.

Finding 3 — the FEC-rate lever is a MIRAGE (load-bearing, not waste)

The bench's "2.00 symbols/packet" turned out to be the permanent steady state of the Default class (DSCP 0 = ~all traffic): non-ARQ, min_ratio floors at 0.10, so it emits 1 source + 1 repair forever, even on a pristine link. Tempting to decay it to 0. Measured the experiment (floor→0):

• Clean-link: 2.000 → 1.000 datagrams/packet (real 2× send-path reduction). ✅
• 10% netem, low-rate ping: baseline 10/10 → experiment 7/10, 9/10, 8/10 (~20% lost, unrecovered). ❌ Sparse traffic can't keep the adaptive controller armed; the ratio decays to ~0 between packets and losses go unprotected (Default is non-ARQ — nothing catches them).
• The change breaks non_arq_class_keeps_floor, a deliberate tested invariant. The floor protects exactly the low-rate latency-sensitive traffic (SSH/DNS/VoIP/gaming) the class exists for.

Conclusion: the current FEC tuning is correct; do not pursue the FEC-rate change.

Net recommendation

The only defensible remaining work is static configurable MTU + last-symbol wire-trim (Finding 4: pad for the GF math, trim the partial last symbol on the wire — removes Lever 1's padding penalty), worthwhile for controlled/LAN/jumbo deployments, not worth the codec risk for internet-only paths. Suggest either closing this as investigated (revisit for a controlled-path deployment) or descoping to the static-MTU + wire-trim work filed for when it pays.