Add transport URL secret rotation with consumer finalizer#663
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: lmiccini The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
lmiccini
force-pushed
the
finalize-secret-rotation
branch
from
5897bda to
4f0979f
Compare
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 7m 40s |
lmiccini
force-pushed
the
finalize-secret-rotation
branch
from
4f0979f to
2f9c5d8
Compare
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages. Design: 1. Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile. 2. Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs. 3. Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status. 4. Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness. 5. Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret. The same pattern applies to notification transport URL secrets and application credential secrets where applicable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lmiccini
force-pushed
the
finalize-secret-rotation
branch
from
2f9c5d8 to
fd6bdd2
Compare
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
1 participant
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages.
Design:
Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile.
Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs.
Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status.
Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness.
Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret.
Delete path: reconcileDelete now cross-references TransportURL CRs directly (by name) as a fallback when removing consumer finalizers, handling the edge case where status was not yet updated before a crash. This matches the pattern used by manila, heat, and barbican.
The same pattern applies to notification transport URL secrets and application credential secrets where applicable.