Per-Connection Handshake Freeze via Cross-`message_seq` Defragment Poison in `Handshake::defragment`

[X1AOxiang]

Summary

Handshake::defragment in dimpl's DTLS 1.2 engine stops its assembly loop on a handshake message-type mismatch but not a message_seq
mismatch. A malicious peer (or MITM) can inject a single epoch-0 Handshake record carrying two handshake fragments with the same
msg_type but different message_seq. The defragment loop assembles both into the buffer, fails the length check, and returns a Transient
error that handle_packet swallows. The poison Incoming is not removed from queue_rx, and because insert_incoming_handshake deduplicates by (message_seq, fragment_offset), any legitimate ClientHello sharing the same key is dropped as an exact duplicate. The server cannot make
progress on that handshake. The attacker can keep the handshake stalled indefinitely by re-injecting the poison (or, at this revision, even
without re-injection, since the poison is not purged).

Affected Versions

• dimpl 0.6.2 (revision dd5a20f, 0.6.2-7-gdd5a20f)
• DTLS 1.2 engine.

Severity

MEDIUM — single-connection denial of service (handshake freeze). No crash, no auth bypass, no memory corruption. Recoverable when the
attacker stops injecting. A MITM can stall a third-party handshake.

CWE

• CWE-835 (Reachable State Without Graceful Handling)
• CWE-440 (Expected Behavior Violation)

Root Cause

The vulnerability has two cooperating defects, both present at dd5a20f:

1. Handshake::defragment stops its assembly loop on a handshake message-type mismatch but not a message_seq mismatch, so a same-msg_type cross-message_seq decoy fragment is assembled into the output buffer and triggers a length-mismatch failure.
2. On that failure, the poison Incoming is not removed from queue_rx. The receive-queue dedup (insert_incoming_handshake) then drops any legitimate ClientHello sharing the same (message_seq, fragment_offset) key as an exact duplicate, freezing the handshake.

Defect 1: defragment loop ignores message_seq

has_complete_handshake_with_seq (the predicate that decides whether next_handshake should attempt defragmentation) correctly skips fragments whose message_seq != wanted_seq:

for h in skip_handled {
    if wanted_seq != h.header.message_seq {
        continue;                       // cross-seq fragments are skipped
    }
    // check fragment contiguity ...
    if last_fragment_end == wanted_length {
        return true;                    // complete based solely on seq-matching fragments
    }
}

So for [complete seq=0 ClientHello fragment] + [same-msg_type seq=1 decoy fragment], it returns true based solely on the seq=0 coverage —
the decoy is never examined.

defragment (the assembly routine, src/dtls12/message/handshake.rs line 150), however, breaks only when msg_type differs. It does
not check message_seq:

let (first_handshake, first_buffer) = iter.next().unwrap();
// ...
let mut handled = ArrayVec::<&Handshake, MAX_DEFRAGMENT_HANDSHAKES>::new();
handled.try_push(first_handshake)...;
buffer.extend_from_slice(&first_buffer[range.clone()]);

for (handshake, source_buf) in iter {
    if handshake.header.msg_type != first_handshake.header.msg_type {
        break;                          // stops on msg_type mismatch only
    }
    // ...
    handled.try_push(handshake)...;
    buffer.extend_from_slice(&source_buf[range.clone()]);
}

if buffer.len() != first_handshake.header.length as usize {
    debug!("Defragmentation failed. Fragment length mismatch");
    return Err(crate::InternalError::parse_incomplete());   // no mark_handled, no queue purge
}

It assembles the seq=0 fragment and the same-msg_type seq=1 decoy into the buffer → buffer.len() = L + M != L → Err(parse_incomplete).

Defect 2: dedup blocks legitimate retransmits by key

insert_incoming_handshake (src/dtls12/engine.rs line 252) deduplicates incoming handshakes by (message_seq, fragment_offset) via binary_search_by:

let key_current = (
    handshake.header.message_seq,
    handshake.header.fragment_offset,
);
// ...
let search_result = self.queue_rx.binary_search_by(|item| {
    let key_other = item
        .first()
        .first_handshake()
        .as_ref()
        .map(|h| (h.header.message_seq, h.header.fragment_offset))
        .unwrap_or((u16::MAX, u32::MAX));
    key_other.cmp(&key_current)
});

While the poison Incoming remains in queue_rx, any legitimate ClientHello with the same (message_seq=0, fragment_offset=0) key is dropped as an exact duplicate. The server cannot make progress on that handshake.

Attack Chain

1. The victim server is in AwaitClientHello with peer_handshake_seq_no = 0 (fresh server, or one that just finished a previous handshake). No prior handshake stage is required.
2. The attacker injects one epoch-0 Handshake record whose fragment stream contains two handshake headers:
   • Fragment 1: a complete ClientHello, message_seq=0,
     fragment_offset=0, fragment_length=L, length=L (body bytes
     are arbitrary — defragment fails before the body is parsed).
   • Fragment 2: a decoy ClientHello (same msg_type!),
     message_seq=1, fragment_offset=K (any non-zero value),
     fragment_length=M.
3. handle_packet → parse_packet inserts the record into queue_rx. make_progress → next_handshake(ClientHello) →
   has_complete_handshake_with_seq(ClientHello, 0) returns true (based on the complete seq=0 fragment).
4. defragment assembles both fragments, calls set_handled() on both, then fails the length check → Err(Transient).
5. handle_packet swallows the Transient error and returns Ok(()). Both fragments are now handled=true; peer_handshake_seq_no is
   still 0.
6. A legitimate ClientHello retransmit (message_seq=0, fragment_offset=0) arrives. insert_incoming_handshake finds an exact duplicate by key (0, 0) and drops it. The server cannot process it.
7. The attacker re-injects the poison before each poll_output, keeping the handshake frozen.

Proof of Concept

The PoC is supplied as a Git patch (poc.patch) that adds one new test file tests/dtls12/poc.rs and registers it in tests/dtls12/main.rs.
The test poc_v2_handshake_poison_freezes_server uses only the public Sans-IO API and crafted DTLS byte streams.

Applying the patch

From the repository root (dimpl/, at revision dd5a20f):

git apply poc.patch

Running the PoC

cargo test --features rcgen --test dtls12 poc::poc_v2_handshake_poison_freezes_server -- --nocapture

What the test does

1. A fresh DTLS 1.2 server is created (passive, AwaitClientHello).
2. A poison packet is constructed: one epoch-0 Handshake record containing two handshake fragments — a complete ClientHello
   (message_seq=0, frag_off=0, frag_len=16, length=16) and a decoy ClientHello (message_seq=1, frag_off=4, frag_len=8, length=32).
3. server.handle_packet(&poison) returns Ok(()) — the Transient defragment error is swallowed.
4. A real Dtls client completes its timeout-driven ClientHello emission. Its packets are injected into the poisoned server without an intervening poll_output (so the poison is not yet purged).
5. drain_outputs(&server) returns no packets — the legitimate ClientHello was dropped as a duplicate; the server did not emit a HelloVerifyRequest or ServerHello.
6. Control: the same real ClientHello packets injected into a fresh, unpoisoned server produce a response (proving the drop is caused by
   the poison, not by the packet construction).
7. The attacker re-injects the poison each cycle for 3 more iterations; the server remains frozen (no output) across all of them.

Verified output

test poc::poc_v2_handshake_poison_freezes_server ... ok

Impact

• Per-connection DoS: the handshake for the targeted (message_seq, fragment_offset) key cannot complete while the poison
  Incoming remains in queue_rx.
• Recoverable: if the attacker stops and the poison Incoming is eventually evicted (e.g. by a higher-sequence record replacing it), a legitimate retransmission can advance. At this revision the poison is not auto-purged by purge_handled_queue_rx, so recovery requires either attacker inaction plus a non-duplicate packet, or a server restart.
• MITM variant harms third parties: a direct malicious client only stalls its own connection; a MITM can stall a victim's handshake.

Suggested Fix

1. In Handshake::defragment, make the loop's stop condition also check message_seq == first_handshake.header.message_seq, so the assembly
   iterator and the completeness predicate agree on which fragments belong to the same handshake. With this fix, the seq=1 decoy is excluded from the buffer, buffer.len() == L, and defragment succeeds (or fails for a real reason) — the poison never reaches queue_rx dedup.
2. On any defragment failure, evict the offending Incoming from queue_rx (or skip dedup for fragments belonging to a failed defragmentation attempt), so a legitimate retransmit with the same (message_seq, fragment_offset) key is not dropped.
3. Keep the existing deferral of set_handled() until after Body::parse succeeds (commit 1a47ca7); do not regress it.
4. Add a regression test: inject [complete seq=0 ClientHello fragment] + [same-msg_type seq=1 decoy] into an AwaitClientHello server an assert the handshake is not poisoned — a legitimate retransmit advances normally.

pocs.zip

[algesten]

Hi, thanks for reporting this.

MEDIUM — single-connection denial of service (handshake freeze). No crash, no auth bypass, no memory corruption. Recoverable when the
attacker stops injecting. A MITM can stall a third-party handshake.

I would argue LOW.

1. A MITM can always screw up a DTLS connection establishment
2. The inelegance is that the MITM attacker can screw up the state machine in a way that can't recover.

Let's fix it!

[algesten]

Fixed by #151.