fix(migrate): strip oversized last-applied-configuration annotations

[aweingarten]

📌 Summary

Fixes #3449 (and item 1 of #3421).

Client-side apply writes the entire object into kubectl.kubernetes.io/last-applied-configuration. For an object with a large embedded schema — kyverno's policy CRDs, the ESO/cnpg/Gateway-API CRDs, grafana's dashboard ConfigMaps — that annotation runs 150–250KB and tips past the apiserver's 262144-byte cap, after which every subsequent patch to the object fails:

CustomResourceDefinition.apiextensions.k8s.io "clusterpolicies.kyverno.io" is invalid:
  metadata.annotations: Too long: may not be more than 262144 bytes (retried 5 times)

The owning Applications never reach Synced and platform-bootstrap stalls.

Half of this is already fixed, which is why it only bites reused clusters. ARGOCD_APP_DEFAULT_SYNC_POLICY has carried ServerSideApply=true since v6.0.0, so nothing writes the annotation today. But an object that picked one up under an earlier client-side apply keeps it forever — and keeps spending the annotation budget on it. Fresh clusters are fine; long-lived ones stay wedged no matter how many times they resync.

So this adds the cleanup rather than another sync-option change: a migration (specVersion 72) that sweeps CRDs and ConfigMaps and removes the annotation wherever it exceeds 100KB. Under server-side apply the annotation is inert, so removing it costs nothing and unwedges the object immediately.

🔍 Reviewer Notes

• Threshold, not blanket removal. 100KB is well under the 262144 cap and well above any ordinary annotation, so the sweep only touches objects that are actually near the cliff. On the healthy cluster in Bootstrap wedges on the 256KB metadata.annotations limit — core apps client-side-apply oversized CRDs/ConfigMaps (needs ServerSideApply) #3449 the top offenders were 246KB / 240KB / 234KB — the objects this targets are unambiguous.
• Paginated (100 per page) rather than one cluster-wide list, so a cluster with many ConfigMaps doesn't pull everything into the operator's memory at once.
• Non-fatal per object: a failed patch is logged and skipped, and 404/422 (someone else already removed it) is ignored. One unpatchable object can't fail the migration.
• Follows the existing custom-migration shape — injectable deps, dryRun/local/DISABLE_SYNC guard — same as preservePvcStorageClassInRawValues.
• Bumps specVersion to 72 in values-changes.yaml, helmfile.d/snippets/defaults.yaml and the test fixture; ci/src/check-schema-version.mjs passes.
• Scope question for you: the sweep covers CRDs and ConfigMaps, which is where every confirmed case landed. If you'd rather it also cover other kinds (or be restricted to apl-owned objects via a label selector), say which and I'll adjust — I kept it to the kinds with evidence behind them.

5 new unit tests (threshold behaviour, pagination, per-object failure isolation, JSON-pointer escaping). jest (24 tests in migrate.test.ts), eslint, tsc --noEmit all clean locally; the helmfile-backed gates need helmfile/gucci, which I don't have locally.

🧹 Checklist

• Code is readable, maintainable, and robust.
• Unit tests added/updated

[Copilot]

Pull request overview

Adds a new migration step to unblock Argo CD sync/bootstraps on long-lived clusters where oversized kubectl.kubernetes.io/last-applied-configuration annotations (left over from historical client-side apply) push objects over the apiserver annotation-size limit.

Changes:

• Introduces migration specVersion 72 that sweeps CRDs + ConfigMaps and removes last-applied-configuration when it exceeds a threshold.
• Adds unit tests covering threshold behavior, pagination, failure isolation, and JSON-pointer escaping.
• Bumps specVersion references and wires the migration into the versioned migration list.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
values-changes.yaml	Registers migration function for specVersion 72.
tests/fixtures/env/settings/versions.yaml	Updates fixture specVersion to 72 for migration tests.
src/cmd/migrate.ts	Implements the new sweep + patch logic and registers it as a custom migration function.
src/cmd/migrate.test.ts	Adds unit tests for the new migration and helper escaping.
helmfile.d/snippets/defaults.yaml	Updates default specVersion to 72.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/cmd/migrate.ts:855

• isOversized compares annotationValue.length (UTF-16 code units) against a threshold described in bytes, but the Kubernetes annotations limit is byte-based. This can under-count non-ASCII content (e.g., CRD schemas/descriptions with Unicode), potentially skipping objects that are actually near/over the apiserver byte cap.

export const isOversized = (object: AnnotatedObject): boolean =>
  (object.metadata?.annotations?.[LAST_APPLIED_ANNOTATION]?.length ?? 0) > LAST_APPLIED_SIZE_THRESHOLD

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/cmd/migrate.ts:854

• isOversized compares the annotation value using .length, but the Kubernetes annotation limit (and your threshold comment) are in bytes. For non-ASCII content, .length (UTF-16 code units) can significantly undercount bytes, causing genuinely oversized last-applied-configuration values to be skipped and leaving the cluster wedged.

export const isOversized = (object: AnnotatedObject): boolean =>
  (object.metadata?.annotations?.[LAST_APPLIED_ANNOTATION]?.length ?? 0) > LAST_APPLIED_SIZE_THRESHOLD