Repository navigation
ocp4_workload_openshift_gitops: use a UBI9 image for the kustomize-envvar sidecar - #203
Merged
Merged
Conversation
…vvar sidecar
The cmp sidecar executes the argocd-cmp-server binary mounted from the
repo-server image. That image is argocd-rhel9, so the binary needs glibc
2.34, while the default sidecar image is UBI8 with glibc 2.28. The sidecar
crashlooped on every cluster that enabled the plugin:
/var/run/argocd/argocd-cmp-server: /lib64/libc.so.6:
version GLIBC_2.34 not found
version GLIBC_2.32 not found
With the repo-server down no Application can generate manifests, so every
sync fails with a ComparisonError.
Switching the default to the ubi9 tag of the same image fixes it; the tag
carries the same kustomize, helm and perl tooling. Verified on a live
cluster: the sidecar reaches 2/2 Running with 0 restarts and the Application
that had been stuck goes from Unknown to Synced.
wkulhanek
requested review from
fridim,
rut31337 and
stencell
as code owners
November 12, 2026 15:21
juliaaano
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
kustomize-envvarCMP sidecar crashloops on any cluster that setsocp4_workload_openshift_gitops_kustomize_plugin: true, taking the ArgoCD repo-server down with it.The sidecar does not ship its own
argocd-cmp-server. It executes the binary mounted from the repo-server through the shared/var/run/argocdvolume, so the two containers have to agree on glibc:argocd-repo-serveropenshift-gitops-1/argocd-rhel9cmpquay.io/devfile/universal-developer-image:ubi8-latestA RHEL9-built binary cannot run on a UBI8 userspace, so the sidecar exits immediately and never recovers. Observed state:
cmpinCrashLoopBackOffafter 8 restarts whileargocd-repo-serverstayed healthy.With the repo-server pod never reaching ready, no Application can generate manifests. Every sync fails with:
In the deployment that surfaced this, the playbook failed at
ocp4_workload_s4 : Wait for S4 Application to syncafter 60 attempts (10m44s).This went unnoticed because the plugin is off by default. It is broken for everyone who turns it on.
Change
Default the sidecar image to the
ubi9-latesttag of the same image. It carries the same kustomize, helm and perl tooling the plugin needs:The quay repository lists several active
ubi9-*tags and noubi8-*ones, so the previous default was stale in more than one way.Verification
Applied to a live cluster by patching the ArgoCD CR's sidecar image:
1/2 CrashLoopBackOff(8 restarts) to2/2 Runningwith 0 restartsUnknown/Healthy->OutOfSync/Missing->Synced/Progressingafter a hard refreshansible-lintclean on the role, profileproduction, 0 failuresRelated