fix(gateway): harden the WaveKV sync path (extracted from #1031, no protocol change)

[kvinwang]

Extracts from #1031 everything that does not depend on wavekv 2.0, so it can land while the dual-stack upgrade is still being compatibility-tested. Builds against wavekv = "1.0.0" as on next — no dependency bump, no wire-format change, no proto change.

Why separately

#1031 is a protocol change: it bumps wavekv to the delta-state branch, adds /wavekv/sync2 + /wavekv/push, and negotiates per peer. That needs a mixed-version test matrix before it can merge. But six of its changes are ordinary robustness work on the existing v1 path, and they were only in that branch because that is where they were written.

Extracted	From	Kind
Bounded decompression + bounded response bodies	769f66b5	fix
Publish own records after the sync bootstrap	e128d727	fix
Rocket-level coverage of the sync route	335fab6f, 6d0d20b1	test
Coverage of the route's authentication gate (+ authorize_peer split)	8d04ff22	test + refactor
Client-side identity check, status checks, response bound over real TLS	37f94318, 9a6e2174, 8de5d5b1	test
Key-namespace properties, repeat_n clippy fix	37f94318, 8de509c4	test

Left in #1031 because they need wavekv 2.0: the /wavekv/sync2 and /wavekv/push routes, protocol probe-and-fallback, state_digest/PeerLinkStatus reporting in WaveKvStatus (new proto fields), and the key-schema admission policy (it is wired through wavekv::AdmissionPolicy, which 1.0 does not have).

The wavekv-side hardening that is likewise independent of the protocol — tolerant WAL tail recovery and length-bound checks — is going to wavekv as its own PR against main, so it can ship as a 1.x patch.

Problem → Fix

1. Unbounded decompression (#1029 P0.1). The sync wire is gzipped and data.open(16.mebibytes()) caps the compressed size. gzip expands ~1000:1 on attacker-chosen input, so that cap admits a payload expanding into the gigabytes; every gateway in a cluster shares one app_id, so the RA-TLS gate proves only that the sender is some gateway of this deployment. On the client side it was worse — peer responses were read with Body::collect, which has no limit at all, so the memory was already spent before any decoding bound could apply.

Both points now go through one bounded helper (128 MiB decompressed ceiling), and response bodies go through Limited at the same 16 MiB the route accepts on a request.

2. A rebuilt gateway wedged itself (#1029, adjacent to P0.5). A local write allocates a sequence number. After losing its store the node keeps its id but has no record of which numbers it spent — only its peers do — so SyncManager::bootstrap() rebuilds the counter from their coverage (this is wavekv 1.0 behavior, sync.rs:90). Anything written before that reuses numbers the peers already treat as seen, and it drops out of log dissemination with no error on either side.

The three records written at startup were exactly the ones that must not be dropped: node/info carries the uuid peers check us against, __peer_addr carries the address they route to. They could not simply be moved, because HttpSyncNetwork::new read this node's uuid back out of the store — our own uuid is local configuration, not replicated state, and routing it through the store created the ordering constraint. It is passed in now, and all three writes happen after the bootstrap.

Scope: this completes the recovery when the node's uuid survives the loss — a corrupt or discarded WAL with {data_dir}/node_uuid preserved, which is the state the P0.5 quarantine flow should produce. If the uuid file is lost too, it is regenerated, and wavekv's peer-side uuid check (sync.rs:188) then refuses every sync from this node before applying anything — bootstrap included — so the counter is never recovered and the fresh uuid has no channel to propagate through. That gate cannot be fixed from this side of the wire; recovery must preserve node_uuid (the quarantine flow in #1029 P0.5 should exclude it from the quarantined path), tracked with #1031.

3. The HTTP layer and the auth gate had no coverage. Mutation testing found verify_gateway_peer replaceable with Ok(()) without turning the suite red — every route test sets insecure_skip_attestation, which is that function's first statement, so no test had ever executed a line of the gate. AppIdValidator (the client-side mirror, running inside the TLS handshake) had no tests at all. Neither did the status checks on a sync response or a bootnode fetch, because https_only() means a plain HTTP stub will not reach them.

None of this needs a TEE or a simulator: rcgen is already a dependency, CertRequest writes PHALA_RATLS_APP_ID unconditionally, and the checks read two X.509 extensions and compare bytes. RocketCert now holds the extension list instead of the Certificate (which has no public constructor outside a real handshake), and the authorization rule is split into authorize_peer — same behavior, testable.

Compatibility

No wire, proto, storage-format or config change; an upgraded node and a next node interoperate on the v1 route exactly as before. The behavioral deltas, all one-directional:

• A peer's sync response is now capped at 16 MiB compressed. Previously unbounded. This mirrors the cap the route has always applied to a request, so no peer can answer with more than it would have been allowed to ask. A full-state response is bounded by the gateway's own key set — instances, nodes, certificates — and 16 MiB compressed is 1-2 orders of magnitude above a realistic cluster; the_sync_limits_admit_the_largest_message_the_protocol_can_produce pins the relationship so the ceiling cannot be tightened into ordinary traffic.
• A decompressed sync payload is capped at 128 MiB. Previously unbounded, and reached only by a payload no legitimate peer produces.
• This node's node/info, node/status and __peer_addr appear a few hundred milliseconds later at startup (after the bootstrap round instead of before it). Nothing reads them in that window — the one thing that did, HttpSyncNetwork::new, is exactly what this fixes.

Verification

cargo test -p dstack-gateway — 103 tests, all green (78 on next; +25 here). cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables and cargo fmt --check --all clean.

The new coverage, all against the real code path:

• a compression bomb (128 MiB of zeroes, well under the request cap once compressed) is refused at the route; a payload landing exactly on the limit still decodes, so the bound cannot tighten by a byte with only the bomb test still passing;
• an oversized response is refused before decompression, using stored-mode gzip so the fixture clears the compressed ceiling while decompressing well inside the other one — a malformed body would have passed with the bound removed entirely;
• a sync round trip over Rocket's local client returns the state this node holds, both stores answer, an unknown store is 404, sync-disabled is 503, sender_id == 0 is refused;
• the auth gate: an unauthenticated caller gets 401, a matching app id is accepted, a foreign one is 403, a certificate without an app id is 401, and a gateway with no app id of its own authorizes nobody;
• the client half over a real TLS listener: a peer from another app cannot complete the handshake, a 500 does not decode as a sync response even when its body would, a 403 from a bootnode does not parse as a peer list;
• the key namespace: prefixes match what they iterate and do not capture a neighbour (inst-a must not swallow inst-ab), builders and parsers round-trip, parsers refuse foreign namespaces.

Related: #1029. Stacks cleanly with #1035 (verified: git merge-tree reports no conflicts).

[Copilot]

Pull request overview

This PR hardens the existing WaveKV v1 HTTP sync path in dstack-gateway without changing protocol, wire format, or dependencies, focusing on memory-safety limits, startup correctness, and adding coverage for previously untested sync/auth/TLS paths.

Changes:

• Add bounded gzip decompression (128 MiB decompressed ceiling) and bound peer response bodies to 16 MiB compressed.
• Fix startup ordering so this node’s critical KV records are published after sync bootstrap rebuilds the sequence counter.
• Add extensive tests covering sync route behavior, auth gate behavior, TLS handshake identity checks, and key-namespace properties; plus a small clippy-driven test fix.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
dstack/gateway/src/web_routes/wavekv_sync.rs	Uses bounded decompression for inbound sync messages, splits authorization decision for testability, and adds route/auth coverage tests.
dstack/gateway/src/pp.rs	Updates test helper to repeat_n to satisfy clippy and keep intent clear.
dstack/gateway/src/main_service.rs	Moves publishing of this node’s KV records to after bootstrap; passes UUID explicitly into sync service creation.
dstack/gateway/src/kv/sync_service.rs	Stops reading “my uuid” from replicated state; takes it from local configuration to avoid ordering pitfalls.
dstack/gateway/src/kv/mod.rs	Introduces compressed/decompressed sync size ceilings and a bounded gzip helper with dedicated tests; adds key-namespace tests.
dstack/gateway/src/kv/https_client.rs	Adds bounded response-body reads and bounded decompression for peer sync responses; adds TLS transport/identity/status coverage tests.

Suppressed comments (2)

dstack/gateway/src/main_service.rs:278

• Log/error messages should start with lowercase text (per CLAUDE.md). This message starts with "Failed".

        if let Err(err) = kv_store.set_node_status(config.sync.node_id, NodeStatus::Up) {
            error!("Failed to set node status: {err:?}");
        }

dstack/gateway/src/main_service.rs:282

• Log/error messages should start with lowercase text (per CLAUDE.md). This message starts with "Failed".

        if let Err(err) = kv_store.register_peer_url(config.sync.node_id, &config.sync.my_url) {
            error!("Failed to register peer URL: {err:?}");
        }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.