Skip to content

Bound and hash the drain PDB's node label - #511

Open
boddumanohar wants to merge 1 commit into
mainfrom
fix/drain-pdb-label-hash
Open

Bound and hash the drain PDB's node label#511
boddumanohar wants to merge 1 commit into
mainfrom
fix/drain-pdb-label-hash

Conversation

@boddumanohar

Copy link
Copy Markdown
Member

What

ensurePDB built the per-node PDB's selector from the raw Node name, while labelStoragePod labeled the pods that selector must match through sanitizeLabelValue. A Node name may be 253 characters; a label value stops at 63. Three things went wrong at once:

Rejected For a name over 63 bytes the selector value is illegal, so the API server refuses the PDB — drain protection is never established
Protects nothing Even if accepted, it selects a value no pod carries, so eviction proceeds unblocked
Collides sanitizeLabelValue truncated without a digest, so two names sharing a 63-character prefix yield one label value and one PDB name — draining the second node retargets the first node's budget

Fix

sanitizeLabelValue returns an already-legal value unchanged; otherwise it cuts the name to leave room for a digest of the whole original and joins the two. Bounded, readable, deterministic, and it cannot end on a character a label value may not end on. The PDB selector now uses it, so selector and pod label are derived the same way.

node-a                                        -> node-a                     (unchanged)
ip-10-0-4-118.us-east-2.compute.internal      -> ip-10-0-...ute.internal    (unchanged)
ip-10-0-4-118.eu-central-1...example.com (75) -> ip-10-0-4-118.eu-central-1.compute.internal.sub.sub.su-0c4b78d5
<70 a's>-one                                  -> <54 a's>-1a839ba6
<70 a's>-two                                  -> <54 a's>-a23dc5e9

Upgrade

No effect on existing clusters: an already-legal value passes through untouched, so every node name in use today keeps the label value it has, and a drain in flight across the upgrade is unaffected.

Tests

TestDrainPDBLabelIsBoundedAndMatchesPod — red on the unchanged tree in all three cases (illegal value, selector matching no pod, two names colliding), green after. The pre-existing TestSanitizeLabelValue and TestEnsurePDB* cases still pass, which is what pins the unchanged short-name behavior.

Gates

Operator suite (15 packages) passes; make -C operator lint 0 issues; no generator drift.


Two notes, neither addressed here:

  • The drain coordinator has no test-plan section anywhere under operator/docs/tests/, so there was no matrix to add a row to. Enumerating one is a test-scenarios job, not a drive-by.
  • sanitiseDNSLabel (storage.simplyblock.io/worker) and nodeprobe.labelValue are two more hand-rolled variants of this same primitive, and sanitiseDNSLabel has the same unbounded defect on a different label. Consolidating the three into atlas-lib/kube is the obvious follow-up.

🤖 Generated with Claude Code

ensurePDB built the per-node PodDisruptionBudget's selector from the raw
Node name, while labelStoragePod labeled the pods that selector has to
match through sanitizeLabelValue. A Node name may be 253 characters and
a label value stops at 63, so the two diverged the moment a name needed
shortening, and three things went wrong at once:

  - The selector value was illegal for a name over 63 bytes, so the API
    server rejected the PDB and drain protection was never established.
  - Even had it been accepted, it selected a value no pod carried, so
    the budget protected nothing and eviction proceeded unblocked.
  - sanitizeLabelValue truncated without a digest, so two names sharing
    a 63-character prefix produced one label value and one PDB name.
    Draining the second node then retargeted the first node's budget.

sanitizeLabelValue now returns a name that is already a legal label
value unchanged, and otherwise cuts it to leave room for a digest of the
whole original and joins the two. That bounds the result, keeps a
readable prefix, keeps distinct names distinct, and cannot end on a
character a label value may not end on. It is deterministic, so the same
node yields the same value on every reconcile. The PDB selector now uses
it, so the selector and the pod label are derived the same way.

Because an already-legal value passes through untouched, every node name
in use on a cluster today keeps the label value it has, and a drain in
flight across the upgrade is unaffected.

Verified: TestDrainPDBLabelIsBoundedAndMatchesPod is red on the
unchanged tree in all three cases — the selector value is "not a legal
label value", the selector "protects nothing", and two names "both
derive the label value" — and green after. The operator suite (15
packages) passes, including the pre-existing TestSanitizeLabelValue and
TestEnsurePDB cases, which pin the unchanged short-name behavior.
make -C operator lint reports 0 issues.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant