Skip to content

fix: report rollout status against the Deployment's own spec.replicas - #444

Open
nominal-charles wants to merge 1 commit into
authzed:mainfrom
nominal-charles:fix-rollout-status-external-replicas
Open

fix: report rollout status against the Deployment's own spec.replicas#444
nominal-charles wants to merge 1 commit into
authzed:mainfrom
nominal-charles:fix-rollout-status-external-replicas

Conversation

@nominal-charles

@nominal-charles nominal-charles commented Aug 27, 2026

Copy link
Copy Markdown

Problem

The operator reports a perpetual RollingDeployment condition when a SpiceDBCluster's Deployment has its replica count managed by something other than the operator.

DeploymentHandler gates rollout completion on the Deployment's Available/Ready/UpdatedReplicas all equalling config.replicas. That holds while the operator owns spec.replicas, but the operator's own spec.patches feature lets a user remove /spec/replicas from the applied Deployment so an external autoscaler (HPA/KEDA) can own the count. Once that happens the operator no longer sets spec.replicas, the live count no longer tracks config.replicas, and the condition never clears — the cluster reports e.g. 6/8 available indefinitely while the handler requeues every 2s. The same mismatch can occur transiently when config.replicas is changed mid-rollout.

Change

Measure rollout completion against the live Deployment's own spec.replicas rather than config.replicas, falling back to config.replicas only when the live field is unset. Completion now uses the same signals as kubectl rollout status: the latest generation has been observed, every desired replica has been updated to the current pod template and is available, and no replicas from a previous revision remain. The waiting-message denominators report the live desired count.

The logic lives in two small package-private helpers, desiredDeploymentReplicas and deploymentRolloutComplete. No API or CRD change.

Tests

Added focused unit tests for both helpers and a TestEnsureDeploymentHandler case covering a fully-available Deployment whose external replica count differs from config.replicas (previously stuck rolling, now clears). go test -race ./... passes and go mod tidy is clean; e2e was not run locally.

Fixes #108

The DeploymentHandler rollout gate compared the Deployment's
Available/Ready/UpdatedReplicas to config.replicas. When replicas are
managed externally -- for example an autoscaler scaling the Deployment
after a spec.patches entry removes /spec/replicas -- the operator no
longer sets spec.replicas, so config.replicas never matches the live
count and the SpiceDBCluster reports a RollingDeployment condition that
never clears (and requeues every 2s).

Measure the rollout against the live Deployment's own spec.replicas
instead, falling back to config.replicas when that field is unset, using
the same completion signals as `kubectl rollout status`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@nominal-charles

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Changing replicas during rollout can make cluster get stuck rolling out

1 participant