Skip to content

[Bug] KubernetesLease stamps an acquire before its GET, so a slow GET shortens the first lease period — and one slower than the TTL yields a win checkAlive() denies at once #1661

Open

Description

@Nevs08

What happened

KubernetesLease.acquirePass takes the renewTime / acquireTime stamp before the GET, and the write follows the GET:

src/coordination/leases/KubernetesLease.ts:475-478 (branch of #1658)
    const sentAt = instantNow();
    const now = new Date(sentAt.wall).toISOString();

    const existing = await getLease(credentials, namespace, name, this.callOptions);

Since #1658 the local deadline counts from that stamp, which is correct and safe: the record's renewTime and the deadline share one instant, and it is taken before the write. But a slow GET — up to operationTimeoutMs, 10 s by default — is charged to the first lease period for everyone:

  • On the recommended 15 s TTL, a GET that takes several seconds leaves the new holder, and every pod reading the record, with a correspondingly shorter first period.
  • An acquire whose GET plus write outlasts the TTL "wins" with a deadline that has already passed. checkAlive() denies it at once, the first renewal tick gives it up, and LeaseMajority can never get a usable win out of it. fix(coordination): make Lease.checkAlive() honour the TTL, re-validate LeaseMajority wins (#937) #1658 documents this as safe but unproductive.

What did you expect

The stamp taken immediately before the CREATE/PUT that writes it. The record and the deadline still share one instant — the invariant the #937 tests pin — but the GET's latency is no longer charged to the lease.

Reproduction

With the frozen-clock harness from #1658 (tests/unit/coordination/KubernetesLease.test.ts, "what moves the deadline"): hold the GET's answer back, advance both clocks by 200 ms, let it go. The acquire succeeds, and its deadline is the GET's send time plus the TTL — 200 ms earlier than the write warrants.

Suggested fix

  • Move instantNow() to just before the CREATE/PUT in acquirePass (both branches).
  • Add a test pinning that the stamp is taken after the GET: hold the GET's answer back, and assert the deadline equals the write's send time plus the TTL.

Context

Found by the PR review of #1658 (fix for #937) as a follow-up candidate. Pre-existing — the stamp position is unchanged from develop. Liveness and latency only; not a safety issue. Related: #937, #598.

Activity

  1. added
    bugSomething isn't working
    priority: lowNice-to-have / niche / demand-driven
    severity: lowMinor / informational / mitigated-by-design
    on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: lowNice-to-have / niche / demand-drivenseverity: lowMinor / informational / mitigated-by-design

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions