fix(gateway): harden the KV→data-plane boundary against bad replicated state - #1035
fix(gateway): harden the KV→data-plane boundary against bad replicated state#1035kvinwang wants to merge 12 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR hardens dstack-gateway against malformed or hostile WaveKV-replicated state by introducing a stricter KV→data-plane import boundary, fail-closed decoding for global records, and defensive handling for timestamp skew and local persistence corruption.
Changes:
- Add
kv::importvalidation to filter/resolveinst/records before they reachProxyState/WireGuard rendering, and re-check peers just before renderingwg.conf. - Make several global KV reads fail closed (
decode_strict) to avoid silently falling back to unsafe defaults; propagate errors through certbot/admin call paths. - Ignore future-dated
handshake/andlast_seen/observations; quarantine unreadable WaveKV data dirs to allow startup recovery.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| dstack/gateway/src/kv/import.rs | New validation/import boundary for replicated inst/ records (per-record + cross-record invariants). |
| dstack/gateway/src/main_service.rs | Routes KV instance import through the boundary; renders WireGuard peers from an explicitly validated list; fail-closed certbot config read path; remote deletion handling. |
| dstack/gateway/src/models.rs | Replace map-values adapter with explicit WgPeer list for WireGuard template rendering. |
| dstack/gateway/src/kv/mod.rs | Add decode_strict; drop future-dated observations; quarantine corrupt persistence; enforce cert config key/value agreement; propagate errors for global reads. |
| dstack/gateway/src/distributed_certbot.rs | Propagate KV read errors (esp. certbot config) instead of silently defaulting. |
| dstack/gateway/src/admin_service.rs | Update admin RPC paths to handle new Result<Option<_>> KV getters. |
| dstack/gateway/src/config.rs | Centralize WireGuard client-IP validity logic in WgConfig::is_valid_client_ip. |
| dstack/gateway/src/main_service/tests.rs | Update fixtures to use real base64 WG keys; add tests for poisoned peer records, remote deletions, and local-write grace behavior. |
| dstack/gateway/src/main_service/snapshots/*.snap | Snapshot updates reflecting real-looking WG keys in fixtures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| fn reload_instances_from_kv_store(proxy: &Proxy, store: &KvStore) -> Result<()> { | ||
| let instances = store.load_all_instances(); | ||
| let accepted = import::accept_instances(&proxy.config.wg, store.load_all_instances()); | ||
| report_rejected_instances(accepted.rejected); | ||
| let instances = accepted.instances; | ||
| let mut state = proxy.lock(); |
There was a problem hiding this comment.
Both correct, and both fixed.
Removals vs rejections — addressed in 98fcd67cc. The filter tested !accepted.instances.contains_key(id) alone, and a rejected record is absent from accepted.instances, so it read as a deletion. load_all_instances folding decode failures into "key not present" made the same thing happen a second way, before accept_instances even saw the record.
Simply exempting everything in rejected would have been wrong, though: it mixes two opposite cases. A record that lost an IP or key conflict to an older registration does say the address belongs to someone else, and keeping the loser routable would put the same address in wg.conf twice — a config wg will not load. Two gateways allocate IPs from their own local allocated_addresses, so a concurrent duplicate assignment is a real accident, not a hypothetical.
So refusals are now typed. Rejection::Unusable (fails validation, or does not decode) says nothing about whether the instance still exists → the instance keeps whatever the data plane already holds. Rejection::LostConflict → the loser stops being routable. load_all_instances returns LoadedInstances { decoded, undecodable } so an unreadable record is distinguishable from a tombstone, and the removal pass exempts only the first kind. Three tests cover the split; the first two fail without the exemption.
Future-dated reg_time — addressed in 700182a1e. Fixed at the import boundary rather than at the elapsed() call sites, because recycle() has the identical unwrap_or_default() and the instance needs to be immune to both to be stuck. reg_time now gets the same MAX_CLOCK_DRIFT_SECS horizon that P0.4 put on handshake/ and last_seen/, which leaves neither call site anything to mishandle.
Worth spelling out why this is reachable without an adversary: reg_time is the registering node's clock at one instant, so the clock only has to be wrong once — the window before chrony converges, or a time jump — for the future timestamp to be written into the KV. It does not heal when the clock does.
81dfd79 to
66b861e
Compare
Review round: six commits added, two amendedRebased onto Scope note: items below are reasoned about as accidental faults only — torn writes, schema drift across a rolling upgrade, a clock wrong once before chrony converges, a full disk. Nothing here assumes a peer is trying to cause them. Two justifications from the first round were written against a hostile-peer model and have been rewritten to say what they actually defend against; one check that only had an adversarial rationale (refusing a record that claims the gateway's own WireGuard public key) was dropped rather than kept on a story nobody believes.
Two worth expanding on, because the first fix for each was wrong:
Quarantine. Moving the data dir aside is right when the contents are unreadable and wrong when the storage is at fault — a full disk or an unmounted volume says nothing about the contents, and since the condition survives a restart, each boot attempt quarantines again and buries the real data. wavekv reports both through Verification: The two eviction tests were written first and confirmed failing against the old filter; the storage-fault test asserts no Known and not addressed here: conflict resolution is "oldest registration wins", which converges deterministically but has no notion of which node wrote a record. P1.7 (quarantine list + per-prefix metrics) and P1.10 (schema-evolution policy — #1030 covers added fields, not renames or type changes) remain follow-ups. |
801b2e8 to
06e598d
Compare
Correction: this PR broke multi-node clusters — fixed in
|
…the data plane Instance records synced from a peer went into ProxyState — and from there into the rendered wg.conf — without re-running any of the checks the registration path applies. That made a single malformed record a node-wide failure: `wg syncconf` rejects the *entire* config file when one peer key is malformed, and a key containing a newline can inject `Endpoint=`/`AllowedIPs=` directives. Last-writer-wins replication also cannot enforce invariants that span keys, so a synced instance could carry the gateway's own wg IP or an IP/public key already claimed by another instance. All KV instance records now pass through `kv::import`, which re-runs the registration checks (public key is 32 base64-encoded bytes, IP and key unique, address not one of this gateway's own) and skips only the offending record, never the batch. Conflicts resolve by registration time so every node reaches the same decision from the same KV contents. The renderer re-checks each peer as a last line of defense before values reach `wg`. The address check deliberately says nothing about which pool an address came from. A CVM registers with one gateway but is handed *every* gateway as a WireGuard server, so each node carries peers for the CVMs registered on the other nodes, and each node allocates from its own `client_ip_range`. Nothing in a node's config describes the other nodes' pools, and the deployments do not agree on a shape that could be inferred: `deploy-to-vmm.sh` puts every pool inside one /16 that each interface covers, while `test-run/cluster.sh` and the e2e configs give each node a /24 that no other node's interface covers. So `is_valid_client_ip` keeps governing allocation, and the import boundary and the renderer use `is_routable_client_ip`, which asserts only what a node can know on its own: an ordinary unicast address that is not one of its own. What keeps the peer list coherent is the uniqueness pass, which runs over the whole KV contents and therefore holds cluster-wide. Refs #1029
`reload_instances_from_kv_store` only ever upserted. An instance recycled or deregistered on node A stayed routable on node B — in ProxyState, in the top-N selection and in B's WireGuard config — until B's own recycle timeout expired, which is 10h by default. The reload now also removes instances that are present locally but gone from the KV store. Records that merely failed validation are left alone: the last known-good state of an instance is better than no state. A registration newer than the grace window is also kept, so an instance whose KV write failed is not evicted before the CVM's next registration refresh. Refs #1029
`get_acme_credentials()` already distinguished missing from corrupt, but its siblings folded a corrupt record into `None`, which silently changes global behavior: `get_certbot_config()` fell back to the defaults, switching `acme_url` to Let's Encrypt production and resetting every renewal interval; a corrupt `dns_cred_default` or per-credential record made the certbot issue through the wrong DNS account or none at all; a corrupt ACME attestation reported an attested account as unattested. The three-state read (missing/tombstoned vs. decodable vs. corrupt) is now a `decode_strict` helper on the KV codec, applied to every global record whose corruption must not silently change behavior. The renewal loop skips its round and retries instead of proceeding with defaults. Refs #1029
`handshake/` and `last_seen/` records are wall-clock seconds written by whichever node made the observation, and the gateway aggregates them with `max`. One node with a fast clock — or a single corrupt record near `u64::MAX` — therefore kept a dead CVM "alive" on every node in the cluster: `recycle()` never fired and top-N routing kept steering traffic at it, with no way to correct the record until real time caught up. Observations dated more than 5 minutes ahead of local time are now dropped on read, logged with a count. The allowance is well above NTP-synced drift and well below the recycle timeout. Refs #1029
…g to start `Node::new_with_persistence` hard-fails on a checksum or deserialize error in the WAL, so a torn tail — the normal artifact of a crash — kept the gateway from starting at all, and the node served no traffic until an operator intervened. The persistent state is a cache: every record is replicated on the peers and re-fetched by the sync service. `KvStore::new` now moves the unreadable directory to `<data_dir>.corrupt.<unix_ts>` and starts empty, logging loudly. Nothing is deleted, so the original bytes stay available for post-mortem. Refs #1029
`list_zt_domain_configs` returned the decoded value without checking it
against the `cert/{domain}/config` key it was filed under. Everything
downstream — certificate issuance, the DNS-01 challenge, `cert/{domain}/data`
— is driven by the value, so one poisoned record could point the certbot at a
domain nobody configured.
Refs #1029
The reload pass claimed that "records that merely failed validation are left alone", but it filtered on `!accepted.instances.contains_key(id)` alone, and a rejected record is absent from `accepted.instances`. So a record that stopped validating — and, because `load_all_instances` folded decode failures into "key not present", one that stopped decoding too — evicted a healthy instance from ProxyState after the 60s grace, taking its wg peer with it. A single corrupt or hostile record was enough to black-hole a live CVM until it re-registered. `load_all_instances` now returns `LoadedInstances`, which keeps undecodable records separate from absent ones, and `import` labels every refusal with a `Rejection`: - `Unusable` (fails validation, or does not decode) says nothing about whether the instance still exists, so the instance keeps whatever the data plane already holds for it; - `LostConflict` (a well-formed record that lost an IP or key conflict to an older registration) does say the address belongs to someone else, so the loser stops being routable — keeping it would put the same address in `wg.conf` twice. The removal pass now exempts only the first kind. Three tests cover the split: a record that stops validating and one that stops decoding both keep their instance, while a conflict loser is still dropped.
P0.4 put a drift horizon on `handshake/` and `last_seen/`, but left `reg_time` unchecked, and `reg_time` feeds the same kind of arithmetic. It is the registering node's clock at one instant, so the clock only has to be wrong once — during the window before chrony converges, or across a time jump — for a future timestamp to be written into the KV, and it does not heal when the clock does. From then on that record is permanent: the reload's "gone from KV" pass and `recycle()` both age instances with `elapsed().unwrap_or_default()`, which reads a future timestamp as zero age. The instance is immune to remote deletion and to local recycling at the same time, and stays routable on every node that loaded it until that process restarts, even after an operator deletes the record. Import now holds `reg_time` to the same `MAX_CLOCK_DRIFT_SECS` horizon as the observations, which closes the path at the boundary and leaves the two `elapsed().unwrap_or_default()` call sites nothing to mishandle. Drift inside the horizon is still accepted, since nodes are not perfectly synchronized; the batch samples the clock once so every record in it is judged against the same instant. Also check a public key's base64 length before decoding it. `wg` writes the padded form and accepts nothing else, so 44 characters is part of the format rather than something to discover from the decode.
`start_certbot_task` refuses to run against an unreadable `global/certbot_config` and says to "wait for an operator to repair the record instead" — but there was no way to repair it. The key is a singleton with no delete RPC, so the only write path is the read-modify-write inside SetCertbotConfig, whose first statement is the fail-closed `get_certbot_config()`. One bad record therefore stopped renewal permanently, and since `do_rotate_acme_credentials` reads the same key, it took RotateAcmeCredentials down with it: unlike `global/acme_credentials`, this one had no way back. SetCertbotConfig is a partial update — a field the operator leaves unset keeps its stored value — so simply reading through to the defaults would not do either. `acme_url` defaults to empty, meaning Let's Encrypt production, so an operator who hit a corrupt record and then tuned `renew_interval` would silently move issuance off their staging or private ACME server and start burning real rate limits. That is the very switch the fail-closed reader exists to prevent. The merge now happens in `merge_certbot_config`, which keeps the stored values when the record is readable and, when it is not, requires the request to state every field before it will replace it. Nothing is ever inherited from a record we cannot read, so the repair path exists without any field being guessed. The error names the fields to resend.
Quarantining an unreadable data dir is right when the *contents* cannot be read: a torn WAL tail is the normal artifact of a crash, every record is replicated, and a gateway that will not start serves nothing. It is wrong when the *storage* is at fault. A full disk, an exhausted fd table, a data volume that has not finished mounting — these say nothing about the contents, so moving the directory aside discards intact state. Worse, the condition survives a restart, so each boot attempt quarantines again and buries the real data under a pile of `.corrupt.*` directories. For a single-node deployment holding the only copy of the ACME account and DNS credentials, that is the difference between a restart and a rebuild. wavekv reports both classes through `anyhow`, so they are told apart by what is in the error chain. Unreadable content arrives as a decode failure, a checksum or header `bail!`, or a read that ran off the end of a truncated file — the last of which is an `io::Error`, but only ever `UnexpectedEof` or `InvalidData`. Any other `io::Error` is the storage layer, and now fails the boot with that error as the cause, which is also what puts the actual fault in front of the operator instead of a misleading "started empty" line.
A record is refused for what it contains, so nothing about the next reload makes it acceptable — it stays refused until someone rewrites it. Logging the whole refused set every round turns one stuck record into an unbounded stream of identical `error!` lines, emitted at whatever rate peer syncs happen to wake the watch task, which buries the first occurrence exactly when it matters. The reload now keeps the reason last logged per instance and reports a record when it starts being refused or its reason changes, and again at `info!` when it becomes usable. The log carries transitions instead of a level, and a refusal is still never silent.
`now_secs()` was already copied into `admin_service` and `distributed_certbot`, and the future-dated-observation fix earlier in this series added a third to `kv`. Three copies of four lines is not itself a problem; three copies that quietly differ would be, and they already had started to — two saturate with `unwrap_or_default()`, the new one with `unwrap_or(0)`. They agree today by luck. `main_service` also held `encode_ts`/`decode_ts`, the same conversion for an arbitrary `SystemTime`, so `now_secs()` is `encode_ts(now)` and belongs beside them rather than in whichever module needed it next. All three now live in `crate::time`, and the call sites that had the expression inlined — the peer last_seen write, the two cert-lock acquisitions, the proxy cert load, the node last_seen refresh, the debug service's reg_time — use them. No behaviour changes: every one of those saturated the same way already. Left alone: the call sites that write `duration_since(UNIX_EPOCH)?` and propagate. A clock behind the epoch is a real fault, and whether to report it or carry on is the caller's decision, not something to bury in a shared helper.
06e598d to
98e1bef
Compare
Correction to the correction: the first fix was still too narrowMy first attempt checked a replicated address against the gateway's interface network instead of its # Gateway IP uses /16 so it can route to all client ranges across the cluster.
WG_IP="10.8.${WG_THIRD_OCTET}.1/16"
WG_CLIENT_RANGE="10.8.${WG_THIRD_OCTET}.0/18"That works for the production shape and fails for the one in tree.
Under the second shape node 1 routes only The general point, which is what makes this a class rather than an incident: nothing in a node's config describes the other nodes' pools. Local topology is the wrong instrument for judging replicated data, no matter how wide you make the net. So What actually keeps the peer list coherent is the uniqueness pass in The regression test now covers both shapes in a loop and fails against either wrong predicate. Re-folded into |
Fixes the robustness findings of #1029 that do not depend on the WaveKV protocol upgrade. The wavekv-side items (decompression bound, key-schema admission, state digest, WAL recovery) stay in #1031 / Phala-Network/wavekv#2; there is no overlap between the two diffs.
Eleven fixes plus one cleanup, one per commit — each is readable on its own:
validate replicated instance records before they reach the data planedrop instances deleted on another nodeunwrap_or_default()on corrupt global keysfail closed on corrupt global KV recordsignore future-dated handshake and last_seen observationsquarantine an unreadable KV data dir instead of refusing to startlist_zt_domain_configskey/value consistencyskip zt-domain configs whose key and value disagreekeep instances whose replicated record is unreadablereg_timeescaped the P0.4 drift horizonreject instance records dated into the futureglobal/certbot_confighad no repair pathlet an operator replace a corrupt certbot configfail the boot on a storage fault instead of quarantiningerror!on every reloadreport a stuck KV record once, not on every reloadnow_secs()keep one epoch-seconds helperProblem
dstack-gatewayreplicates instances, nodes, certificates and DNS credentials across nodes with last-writer-wins semantics, then feeds that state straight into ProxyState, the certbot and the renderedwg.conf. A corrupt or hostile record for one CVM could take down unrelated instances or the whole node:valid_ip, public-key uniqueness) were applied at registration only, never on the sync path.wg syncconfrejects the entire config file when a single peer key is malformed, so one badinst/record froze WireGuard updates for every instance on the node — the failure was only logged. A key containing a newline could injectEndpoint=/AllowedIPs=lines (the template renders withescape = "none"), and LWW cannot enforce cross-key invariants, so a synced instance could claim the gateway's own wg IP, a reserved-net address, or an IP/key already held by another instance.reload_instances_from_kv_storeonly upserted. An instance recycled on node A stayed routable on node B until B's own recycle timeout — 10h by default.get_certbot_config()fell back to the defaults, silently switchingacme_urlto Let's Encrypt production and resetting renewal intervals; the DNS-credential and ACME-attestation readers did the same. Onlyget_acme_credentials()failed closed.maxover wall-clock timestamps. One node with a fast clock, or one record nearu64::MAX, kept a dead CVM "alive" cluster-wide:recycle()never fired and top-N routing kept selecting it, unfixable until real time caught up.read_all_ops()hard-fails on a checksum error, so the normal artifact of a crash stopped the gateway from booting — even though every record is replicated on the peers.list_zt_domain_configstrusted the value over the key. Issuance, DNS-01 andcert/{domain}/dataall key off the value, so a record filed under one domain could drive a certificate for another.Three more surfaced in review of the first six commits, each a defect in the fix rather than in the original code:
!accepted.instances.contains_key(id), and a rejected record is absent fromaccepted.instances— so a record that stopped validating, or (becauseload_all_instancesfolded decode failures into "key not present") stopped decoding, dropped a healthy instance from ProxyState after the 60s grace and took its wg peer with it. That is a regression against the pre-fix behavior, which merely declined to update.reg_timewas left out of the P0.4 horizon. Both the removal pass andrecycle()age instances withelapsed().unwrap_or_default(), which reads a future timestamp as zero age, so an instance dated forward is immune to remote deletion and to local recycling until the process restarts — surviving even after an operator deletes the record and evicts the peer that wrote it..corrupt.*directories. For a single-node deployment holding the only copy of the ACME account and DNS credentials, that is the difference between a restart and a rebuild.error!lines at whatever rate peer syncs wake the watch task, burying the first occurrence.certbot_configguidance pointed at a repair path that did not exist.start_certbot_tasksays to "wait for an operator to repair the record", but the key is a singleton with no delete RPC, and the only write path — SetCertbotConfig — starts with the now fail-closedget_certbot_config(). Renewal stopped permanently, and becausedo_rotate_acme_credentialsreads the same key, RotateAcmeCredentials was blocked too.Fix
gateway/src/kv/import.rs) that everyinst/record passes through: decode → per-record checks (32 base64-encoded key bytes, no whitespace/control chars, IP inside the routable network, bounded identifiers) → cross-record invariants (unique IP, unique key). An offending record is skipped and logged aterror!; the batch is never aborted. Conflicts resolve by(reg_time, instance_id)so all nodes converge on the same winner, matching how the registration path resolves them (oldest keeps the claim). The renderer re-checks every peer before values reachwg, andWgConfnow takes an explicit peer list instead of borrowing the instance map.load_all_instancesreturnsLoadedInstances, keeping undecodable records distinct from absent ones, and every refusal carries aRejection:Unusable(fails validation, or does not decode) says nothing about whether the instance still exists, so the instance keeps whatever the data plane already holds;LostConflict(well-formed, but lost an IP or key conflict to an older registration) does say the address belongs to someone else, so the loser stops being routable. The removal pass exempts only the first kind.reg_timeis held to the same drift horizon as the handshake observations, which closes the future-dating path at the boundary and leaves the twoelapsed().unwrap_or_default()call sites nothing to mishandle. Two cheap neighbors: a record claiming the gateway's own WireGuard public key is refused (matching the existing refusal of its wg IP), and a public key's base64 length is checked before it is decoded.anyhow, so they are told apart by the error chain: a decode failure, a checksum/headerbail!, or a read off the end of a truncated file (anio::Error, but only everUnexpectedEof/InvalidData) is content; any otherio::Erroris the storage layer and now fails the boot with that error as the cause, which is also what puts the real fault in front of the operator.now_secs()next to the two already inadmin_serviceanddistributed_certbot, and they had already drifted in spelling (unwrap_or_default()vsunwrap_or(0)). All three, plusencode_ts/decode_ts— the same conversion for an arbitrarySystemTime— now live incrate::time. Call sites that propagate withduration_since(UNIX_EPOCH)?keep doing so: whether a pre-epoch clock is fatal is the caller's call.info!when it recovers.acme_urlto Let's Encrypt production for an operator who only meant to tunerenew_interval— the exact switch the fail-closed reader exists to prevent. Nothing is inherited from a record we cannot read; readers keep the strict accessor.decode_strictcodec helper encoding the three states (missing/tombstoned →Ok(None), decodable →Ok(Some), corrupt →Err), applied to every global record whose corruption must not silently change behavior. The renewal loop skips the round and retries instead of running with defaults.handshake/andlast_seen/reads, logged with a drop count. Well above NTP-synced drift, well below the recycle timeout.KvStore::newquarantines an unreadable data dir to<data_dir>.corrupt.<unix_ts>and starts empty, re-fetching state from peers. Nothing is deleted.cert/{domain}/config.The address check says nothing about which pool an address came from. A CVM registers with one gateway but is handed every gateway as a WireGuard server, so each node carries peers for the CVMs registered elsewhere — holding addresses from those nodes' pools. Nothing in a node's config describes the other nodes' pools, and the deployments do not agree on a shape that could be inferred:
deploy-to-vmm.shputs every pool inside one /16 that each interface covers, whiletest-run/cluster.shand the e2e configs give each node a /24 that no other node's interface covers.is_valid_client_ipkeeps governing allocation, whereclient_ip_rangeis the right question; the import boundary and the renderer useis_routable_client_ip, which asserts only that the address is ordinary unicast and not one of this gateway's own. Coherence of the peer list comes from the uniqueness pass instead, which runs over the whole KV contents and so holds cluster-wide.Verification
cargo test -p dstack-gateway --all-features(137 tests), each of the 12 commits verified independently withgit rebase --exec,cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables,cargo fmt --check --all— all clean.New tests cover each failure mode as a behavior, not as a unit of the helper:
Endpoint=line, plus an instance claiming the gateway's own wg IP) leaves the healthy instance routable, and the injected directive never appears in the rendered config;global/certbot_configerrors instead of reading as the default (which would move issuance to LE production); same for the ACME attestation and DNS-credential defaults;u64::MAXhandshake observation is ignored while a 30s-old one survives, and drift inside the allowance stays usable;KvStore::newsucceed with empty state and leaves exactly one.corrupt.*directory behind;wg.conf, under both cluster shapes in tree (a /18 pool inside a /16 interface, and a /24 pool equal to the interface), while the gateway's own address, itsreserved_netand non-unicast addresses stay refused;.corrupt.*directory behind, while a garbage WAL still quarantines and starts empty;global/certbot_configcan be replaced by a complete SetCertbotConfig request, while a partial one is refused with an error naming the fields to resend, and a partial update against a readable record still keeps the fields it does not mention.Test fixtures that used placeholder strings as WireGuard keys (
"pubkey-allow","top-key-0", …) now use real 32-byte base64 keys, since both the import boundary and the renderer reject whatwgwould reject; the two affected snapshots change only in those key values.Not in this PR
Rebased onto
nextfor #1030 (named MessagePack encoding) and #1036 (sync-path hardening). Named encoding removes the rolling-upgrade amplification of item 7 — a new field no longer makes older nodes fail to decode — but not item 7 itself, which reproduces with a well-formed record that merely fails validation.Items 7-11 are scoped to accidental faults: torn writes, schema drift across a rolling upgrade, a clock that is wrong once before chrony converges, a full disk. Nothing here assumes a peer is trying to cause them.
P0.1 (decompression bound) landed with #1036; P1.8 (key-schema admission) is in #1031, P0.6 needs wavekv-side changes, and P1.7 (quarantine list + per-prefix metrics) / P1.10 (schema-evolution policy) are follow-ups.