Bootstrap wedges on the 256KB metadata.annotations limit — core apps client-side-apply oversized CRDs/ConfigMaps (needs ServerSideApply)

[aweingarten]

Summary

On some clusters (reliably on reused/long-lived ones), the apl-core (v6.0.0) bootstrap and day-2 sync wedge because several core ArgoCD Applications fail to apply objects that exceed Kubernetes' 262144-byte metadata.annotations limit. The owning Applications never reach Synced/Healthy, so platform-bootstrap and the convergence gate stall.

Symptom

The application-controller logs a tight retry loop (every 2–5s) across multiple apps:

application/kyverno-kyverno  Sync operation ... failed: ... error when patching "/dev/shm/...":
  CustomResourceDefinition.apiextensions.k8s.io "clusterpolicies.kyverno.io" is invalid:
  metadata.annotations: Too long: may not be more than 262144 bytes (retried 5 times)

Confirmed across at least these apps / objects:

Application	Object(s) hitting the limit
kyverno-kyverno	clusterpolicies.kyverno.io, policies.kyverno.io (CRDs)
external-secrets-external-secrets	secretstores.external-secrets.io, clustersecretstores.external-secrets.io (CRDs)
cnpg-system-cloudnative-pg	clusters.postgresql.cnpg.io, poolers.postgresql.cnpg.io (CRDs)
grafana-grafana-dashboards	grafana-dashboards-k8s-admin, grafana-dashboards-istio-admin (ConfigMaps)

Root cause

These Applications sync with client-side apply, which writes the entire object into the kubectl.kubernetes.io/last-applied-configuration annotation. For an object with a large embedded schema/payload — Kyverno's policy CRDs, the Gateway-API/cnpg/ESO CRDs, and the Grafana dashboard ConfigMaps — that annotation runs 150–250KB and tips past the 256KB cap, after which every subsequent patch to the object fails, regardless of the change.

A cluster-wide scan on a healthy cluster showed how close to the cliff these already are under client-side apply:

CRD	annotations (bytes)	headroom
httproutes.gateway.networking.k8s.io	246,356	~16 KB
mutatingpolicies.policies.kyverno.io	240,835	~21 KB
imagevalidatingpolicies.policies.kyverno.io	234,982	~27 KB
opentelemetrycollectors.opentelemetry.io	166,188	~96 KB

Notably, on a healthy cluster the kyverno-kyverno Application runs with ServerSideApply=true and its CRD carries 0 bytes of last-applied-configuration — i.e. where SSA is set, the annotation is never written and the wedge cannot occur. The wedged clusters are the ones whose apps still use client-side apply.

Requested fix

Set ServerSideApply=true as a sync option on the core Applications that manage large CRDs/ConfigMaps (kyverno, external-secrets, cloudnative-pg, grafana-dashboards, and any others near the cap — Gateway-API/httproutes, otel-collector), or app-of-apps-wide. SSA does not write last-applied-configuration, which removes the annotation-size failure entirely and buys headroom as these schemas grow.

Notes / workarounds

Downstream in lke-landing-zone we added a best-effort mitigation (strip oversized last-applied-configuration annotations from CRDs during bootstrap + a converge self-heal), but it can't win the race against continuous client-side re-apply and doesn't cover ConfigMaps — the durable fix is SSA at the source in apl-core.

apl-core version: v6.0.0.

[aweingarten]

Opened #3469.

Tracing this against main sharpened the diagnosis: ARGOCD_APP_DEFAULT_SYNC_POLICY has carried ServerSideApply=true since v6.0.0, so nothing writes the annotation today — which is exactly why fresh clusters are fine and reused ones wedge. An object that picked up a last-applied-configuration under an earlier client-side apply keeps it forever and keeps spending the 262144-byte budget on it, and no amount of resyncing clears it.

So the PR adds the cleanup rather than another sync-option change: a migration (specVersion 72) sweeping CRDs and ConfigMaps, removing the annotation wherever it exceeds 100KB. Under SSA the annotation is inert, so removing it is free. Paginated, and non-fatal per object.

Scope question for reviewers in the PR: it covers CRDs and ConfigMaps, which is where every confirmed case in the table above landed — easy to widen if you'd prefer.