fix(storagenode): self-heal storage node ServiceAccount token expiration - #535
Closed
boddumanohar wants to merge 2 commits into
Closed
boddumanohar wants to merge 2 commits into
boddumanohar wants to merge 2 commits into
Conversation
Storage node pods used Kubernetes' default auto-injected ServiceAccount token, which lives roughly an hour. If simplyblock-storage-node-sa was ever deleted and recreated (e.g. during a chart upgrade), every already-running pod kept its stale token until the hour expired, then started failing API calls with 401 — recoverable only by deleting every pod by hand. Pods now mount an explicit ten-minute projected token instead (AutomountServiceAccountToken: false, with a kube-api-access-short projected volume built to match the shape of Kubernetes' own auto-injected one). kubelet refreshes a projected token at ~80% of its expiration, so a ServiceAccount recreation now self-heals within minutes. Found while testing #437's lblk device-mode work on config-israel, but unrelated to lblk itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
boddumanohar
force-pushed
the
fix/storagenode-deletion-satoken-selfheal
branch
from
September 15, 2026 08:48
d22b209 to
c7610e4
Compare
boddumanohar
changed the title
…t to one StorageNodeSet reconcileRBAC gave "simplyblock-storage-node-sa" a controller owner reference to whichever StorageNodeSet last reconciled it, even though the ServiceAccount is shared by every StorageNodeSet in the namespace (e.g., one StorageNodeSet per storage cluster, a supported topology). Deleting any StorageNodeSet that currently owned it, regardless of whether that StorageNodeSet was itself the one being decommissioned, cascade-deleted the ServiceAccount via Kubernetes garbage collection and took every other StorageNodeSet's pods' tokens down with it. This is the actual root cause behind the SA token expiring and needing a manual pod-by-pod restart to recover, previously worked around by shortening the token TTL alone. The ClusterRole and ClusterRoleBinding built by the same function already carry no owner reference, for the same reason. The ServiceAccount now matches: reconcileRBAC clears any owner reference on apply instead of setting one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #437 (lblk device-mode support): found while testing that work on config-israel, but unrelated to lblk itself.
Storage node ServiceAccount token expiration
The storage node pods used Kubernetes' default auto-injected ServiceAccount token, which lives roughly an hour. If
simplyblock-storage-node-sawas ever deleted, every already-running pod kept its stale token until the hour expired, then started failing API calls with 401 — recoverable only by deleting every pod by hand.Root cause:
reconcileRBACgave the sharedsimplyblock-storage-node-saServiceAccount a controller owner reference to whicheverStorageNodeSetlast reconciled it, even though the ServiceAccount is shared by everyStorageNodeSetin the namespace (e.g. oneStorageNodeSetper storage cluster, a supported topology). Deleting anyStorageNodeSetthat currently owned it — not necessarily the one being decommissioned — cascade-deleted the ServiceAccount via Kubernetes garbage collection, taking every otherStorageNodeSet's pods' tokens down with it.reconcileRBACnow clears any owner reference on the ServiceAccount instead of setting one, matching theClusterRole/ClusterRoleBindingit builds alongside it, which never carried one.Defense in depth: pods now also mount an explicit ten-minute projected token instead of the default hour-long one (
AutomountServiceAccountToken: false, with akube-api-access-shortprojected volume built to match the shape of Kubernetes' own auto-injected one). kubelet refreshes a projected token at ~80% of its expiration, so if the ServiceAccount is ever recreated for any other reason, pods self-heal within minutes instead of up to an hour.🤖 Generated with Claude Code