Skip to content

ocp4_workload_openshift_gitops: use a UBI9 image for the kustomize-envvar sidecar - #203

Merged
wkulhanek merged 1 commit into
mainfrom
gitops_cmp_sidecar_ubi9
Nov 12, 2026
Merged

wkulhanek merged 1 commit into
mainfrom
gitops_cmp_sidecar_ubi9

Conversation

@wkulhanek

Copy link
Copy Markdown
Collaborator

Problem

The kustomize-envvar CMP sidecar crashloops on any cluster that sets ocp4_workload_openshift_gitops_kustomize_plugin: true, taking the ArgoCD repo-server down with it.

/var/run/argocd/argocd-cmp-server: /lib64/libc.so.6: version `GLIBC_2.34' not found
/var/run/argocd/argocd-cmp-server: /lib64/libc.so.6: version `GLIBC_2.32' not found

The sidecar does not ship its own argocd-cmp-server. It executes the binary mounted from the repo-server through the shared /var/run/argocd volume, so the two containers have to agree on glibc:

container image glibc
argocd-repo-server openshift-gitops-1/argocd-rhel9 UBI9, 2.34
cmp quay.io/devfile/universal-developer-image:ubi8-latest UBI8, 2.28

A RHEL9-built binary cannot run on a UBI8 userspace, so the sidecar exits immediately and never recovers. Observed state: cmp in CrashLoopBackOff after 8 restarts while argocd-repo-server stayed healthy.

With the repo-server pod never reaching ready, no Application can generate manifests. Every sync fails with:

ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1:
rpc error: code = Unavailable desc = connection error: ... connect: connection refused

In the deployment that surfaced this, the playbook failed at ocp4_workload_s4 : Wait for S4 Application to sync after 60 attempts (10m44s).

This went unnoticed because the plugin is off by default. It is broken for everyone who turns it on.

Change

Default the sidecar image to the ubi9-latest tag of the same image. It carries the same kustomize, helm and perl tooling the plugin needs:

generate:
  command: ["sh", "-c"]
  args: ["kustomize build --enable-helm . | perl -pe '...'"]

The quay repository lists several active ubi9-* tags and no ubi8-* ones, so the previous default was stale in more than one way.

Verification

Applied to a live cluster by patching the ArgoCD CR's sidecar image:

  • repo-server pod went from 1/2 CrashLoopBackOff (8 restarts) to 2/2 Running with 0 restarts
  • the stuck Application went Unknown/Healthy -> OutOfSync/Missing -> Synced/Progressing after a hard refresh
  • ansible-lint clean on the role, profile production, 0 failures

Related

  • rhpds/agnosticv#29040 enables this plugin for LB1066, which is how the bug was found. That PR is fine as written but will crashloop until this one merges.

…vvar sidecar

The cmp sidecar executes the argocd-cmp-server binary mounted from the
repo-server image. That image is argocd-rhel9, so the binary needs glibc
2.34, while the default sidecar image is UBI8 with glibc 2.28. The sidecar
crashlooped on every cluster that enabled the plugin:

  /var/run/argocd/argocd-cmp-server: /lib64/libc.so.6:
    version GLIBC_2.34 not found
    version GLIBC_2.32 not found

With the repo-server down no Application can generate manifests, so every
sync fails with a ComparisonError.

Switching the default to the ubi9 tag of the same image fixes it; the tag
carries the same kustomize, helm and perl tooling. Verified on a live
cluster: the sidecar reaches 2/2 Running with 0 restarts and the Application
that had been stuck goes from Unknown to Synced.
@wkulhanek
wkulhanek merged commit 1286a6b into main Nov 12, 2026
1 check passed
@wkulhanek
wkulhanek deleted the gitops_cmp_sidecar_ubi9 branch November 12, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants