argocd-redis password split: redis never restarts on Secret rotation → all Applications ComparisonError (WRONGPASS)

[aweingarten]

Summary

argocd-redis and its clients (repo-server, application-controller) both derive the Redis password from the argocd-redis Secret, but each reads it once at container start. When that Secret's value is rewritten while the argocd-redis pod keeps running, the server and clients diverge and every Argo CD Application flips to Unknown/ComparisonError:

ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1:
rpc error: code = Unknown desc = failed to list refs:
WRONGPASS invalid username-password pair or user is disabled.

The repo-server caches git refs/manifests in argocd-redis, so a bad AUTH turns all manifest generation into a ComparisonError at once — the whole app-of-apps stalls.

Mechanism

• argocd-redis Deployment runs redis-server --requirepass $(REDIS_PASSWORD), with REDIS_PASSWORD sourced from Secret argocd-redis key auth. The value is substituted into --requirepass at pod start and never re-read.
• argocd-repo-server / argocd-application-controller read REDIS_PASSWORD from the same Secret key, also at their pod start.
• If the argocd-redis Secret is rewritten (e.g. the ExternalSecret backing auth rotates, or the secret-init regenerates it) without restarting the redis pod, then:
  • redis keeps requiring the old password;
  • any client pod that (re)starts after the rewrite reads the new password;
  • every client→redis AUTH fails with WRONGPASS → all Applications ComparisonError.

Nothing restarts argocd-redis when its password Secret changes, so server and clients can silently diverge.

Where we hit it

A reused (kept) e2e cluster. Observed state at failure:

Pod	Age	Restarts	Password source
argocd-redis	121m	0	argocd-redis/auth (baked into --requirepass at 17:16)
argocd-repo-server ×3	5–7m	0	argocd-redis/auth (read at 19:11–19:13)
argocd-application-controller-0	~7m	0	argocd-redis/auth (read ~19:11)

The redis pod outlived a rewrite of the argocd-redis Secret; the repo-servers rolled and picked up the new value; redis (never restarted) kept the old one → WRONGPASS across all 55 Applications. It also reproduces any time the clients roll (e.g. HPA scale-up, a chart re-sync) after the Secret changes under a long-lived redis pod.

Impact

• Every Argo CD Application → Unknown/ComparisonError; the app-of-apps cannot reconcile until redis is manually restarted.
• Cosmetically indistinguishable from a mass config failure, so it's easy to misdiagnose.
• Workaround: kubectl -n argocd rollout restart deploy/argocd-redis (redis re-reads the current Secret and realigns with the clients).

Requested fix (either is sufficient)

1. Restart-on-rotation — add a checksum/secret (of the argocd-redis auth) annotation to the argocd-redis pod template so a password change rolls redis in lockstep with the clients that consume it. This is the standard Helm pattern and closes the divergence window.
2. Static password — make the argocd-redis password stable (seed once, never rotate), so client rolls can never diverge from a never-restarted redis.

Either eliminates the split. (1) is the more general fix since it also covers deliberate rotations.

Environment

• Argo CD v3.4.3, argocd-redis image redis:8.2.3-alpine, redis auth via argocd-redis Secret (redis-username / auth), populated by an ExternalSecret.