Repository navigation
Conversation
The drain wait counted a holder's lease as expired by comparing the holder's stamped expires_at against this node's wall clock. A peer stamps from its own clock and its HLC is never merged in, so a fast local clock could retire a live holder's lease early — the exact window the drain exists to close. lease_expired gives the stamped deadline the same allowance the HLC ingress enforces (MAX_CLOCK_SKEW_NS): a lease is retired only once even a clock that far ahead would agree it lapsed. A deadline whose addition saturates is never retired — the filter drops holds only where the lapse is certain.
The lease GC releases a holder's descriptor leases on topology absence, which lags a crash: the node stays a member until the removal path runs, and its leases block DDL drains for the whole lease duration. This adds the faster signal as a lease-specific input — routing, placement, and rebalancing never read it. Dead and Left mark a holder; Alive revives it, because a verdict is refutable by a higher incarnation and a refuted node is serving again. Suspect stays a no-op: it is transient and the holder may still hold its lease. The subscriber runs on the detector task and only touches a set.
The periodic lease GC released only holders absent from ClusterTopology, so a crashed holder blocked every DDL drain on its descriptors until the lease expired. The GC now also releases a holder SWIM marked Dead/Left: the liveness set is created with the node's cluster handle, registered as a SWIM subscriber at subsystem start (with the same topology-backed resolver the routing hook uses), and handed to the RaftLoop through with_lease_liveness. Suspect is not a release signal, and a refuted verdict clears the mark again.
…ation A SWIM Dead verdict releases a holder's leases, but a node that restarted (re-acquired at a higher incarnation) could still be fenced by a stale verdict. The fenced grant variant carries the holder's incarnation: DescriptorLeaseGrantFenced is appended last — zerompk numbers variants by position — and is proposed only once the cluster reports LEASE_FENCING_VERSION, so mixed-version clusters keep the unfenced grant and run without the fence. The cache records the stamp, the liveness hook stores the incarnation the verdict landed at, and the GC releases a lease only when its stamp is <= that incarnation. An unstamped lease (pre-fencing or mixed-version) releases as before.
Member
|
Closing. A SWIM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two defects in the descriptor-lease release path, both from the P2 consensus-safety review:
expires_atagainst this node's wall clock. The holder stamps from its own clock and its HLC is never merged in, so a fast local clock could retire a live holder's lease early — the window the drain exists to close.Part of #165 (P2 — cluster consensus safety).
What
lease_expiredgives the stamped deadline the same allowance the HLC ingress enforces (MAX_CLOCK_SKEW_NS); a saturated deadline is never retired — the filter drops holds only where the lapse is certain.LeaseHolderLiveness(+LeaseHolderLivenessHook): Dead/Left mark a holder, Alive revives (verdicts are refutable), Suspect is a no-op by decision. The GC collects a holder absent from topology or marked gone —collect_non_member_lease_releases(topology, cache, liveness). The set is created with the node's cluster handle, registered as a SWIM subscriber at subsystem start (the same topology-backed resolver the routing hook uses), and handed to theRaftLoopviawith_lease_liveness.DescriptorLeaseGrantFenced { lease, holder_incarnation }is appended last (zerompk numbers variants by position) and proposed only undercan_activate_feature(LEASE_FENCING_VERSION); mixed-version clusters keep the unfenced grant.MetadataCache.lease_incarnationsrecords the stamp; a verdict atNreleases leases stamped<= N, so a restarted node that re-acquired is unaffected; an unstamped lease releases as before.Validation
cargo test -p nodedb-cluster --lib lease— 17 passed (collector incl. dead-member, fenced-newer-lease, revive; liveness hooks; clamp boundary).cargo test -p nodedb-cluster --lib metadata_group::cache— 2 passed (fenced grant records; release clears).cargo check -p nodedb --all-targets— clean; repository preflight passes.Notes
state/fields.rsandstate/init.rsstay at their base line counts: two adjacent doc comments were compacted to make room for the new field.