federation: reuse established tunnel instead of re-dialing peer:9191

[mastaab]

Problem

Federation calls intermittently end up with one-way media and DTLS handshake timedout log pairs. Root cause was proven end-to-end on staging with pcaps, conntrack event logs, and coturn logs:

The federation send path only looks up dest:9191 in children_ss and dials a new outbound DTLS connection on a miss. When our egress NAT (any k8s node's SNAT) rewrites the dial's source port, the peer keys our tunnel under the rewritten port, so the peer's own send path misses on dest:9191 and it dials back into us. That second dial is redundant — a healthy tunnel between the two coturns already exists — and it is exactly the handshake that fails: our dial's conntrack entry occupies the local:9191 → peer:9191 tuple, so the kernel's clash resolution rewrites the inbound ClientHello fragments to a fresh source port each, coturn sees two phantom clients holding half a ClientHello each, and both half-open handshakes time out at +10s. The peer's media direction never establishes.

Fix

Reuse any established federation tunnel to the destination peer instead of dialing:

• New peers-by-IP index (ur_addr_map keyed by peer IP with port zeroed → established federation socket). Registered at the single handshake-completion point in dtls_listener.c (covers dialed and accepted tunnels); unregistered in delete_socket_from_map, which every socket close funnels through (close_ioa_socket), so entries can never dangle. A flag on the socket keeps the close-path hook a no-op for non-federation sockets.
• federation_send_data_imp: on a dest:9191 miss, look up the peer IP in the index and send over that tunnel (typically the accepted, peer-initiated one). Only dial when no established tunnel exists at all.
• DTLS application data flows both ways regardless of client/server role — the server-side heartbeat PONG already ships over accepted sockets via the same send_data_from_ioa_socket_nbh/ssl_send path, and dest_addr is ignored for SSL sockets.

Thread model unchanged: the index is only touched on the federation listener thread, exactly like children_ss (sends are marshaled there via the bufferevent pair; federation socket timers and reads run on the listener engine).

One side deploying this heals a pairing: the redundant dial never happens, and its conntrack entry — the clash precondition on the peer's node — is never created. The unpatched peer just receives data on the tunnel it already dialed.

Known limitation

If neither side has any established tunnel and both dial in the same instant, the simultaneous-dial race remains (millisecond window; not the failure in any captured incident). A deterministic-initiator rule would close it and is left as a follow-up.

Verification

• The three modified translation units compile clean with the project's warning flags (full macOS build fails identically on the base branch for pre-existing reasons; needs the CI Linux build).
• Staging validation plan: with one side on this fix and the node-level SNAT pin disabled (to recreate the clash conditions), capture federation traffic (tcpdump + conntrack events) on the receiving pods during a test call — expect no inbound-dial shredding, media returning over the peer-initiated tunnel, and Federation: reusing established tunnel in the verbose log.
• The node-level static-port SNAT mitigation (validated 2026-07-10) is independent and complementary; this change removes the failure mode on any network without node access.