Flux ignores schema error

[Cajga]

Describe the bug

Flux kustomization is used to deploy an application. We made a change in the deployment that makes the deployment manifest invalid by schema.
Flux reconciles the kustomization, but acts like there was not change and shows that everything is ready without pointing our the schema error.

Consider the following example change:
The deployment that was originally applied by flux without any issues:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
  labels:
    app.kubernetes.io/instance: rester
    app.kubernetes.io/name: rester
    app.kubernetes.io/part-of: rester
    app.kubernetes.io/version: 0.2.7
  name: rester
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: rester
      app.kubernetes.io/name: rester
  strategy: {}
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: rester
        app.kubernetes.io/name: rester
        app.kubernetes.io/part-of: rester
        app.kubernetes.io/version: 0.2.7
    spec:
      containers:
      - env:
        image: registry.example.com/rester/rester:0.2.7-21
        imagePullPolicy: Always
        name: rester
        resources:
          limits:
            cpu: 500m
            ephemeral-storage: 512Mi
            memory: 512Mi
          requests:
            cpu: 200m
            ephemeral-storage: 256Mi
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
      imagePullSecrets:
      - name: private-registry-auth
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

The change that introduces an invalid field by schema:

$ git diff
diff --git a/kubernetes/plain/flux-schema-issue/deplyoment.yaml b/kubernetes/plain/flux-schema-issue/deplyoment.yaml
index 50c6aec..98e91e3 100644
--- a/kubernetes/plain/flux-schema-issue/deplyoment.yaml
+++ b/kubernetes/plain/flux-schema-issue/deplyoment.yaml
@@ -18,6 +18,8 @@ spec:
       app.kubernetes.io/name: rester
   strategy: {}
   template:
+    labels:
+      flux-test: test
     metadata:
       labels:
         app.kubernetes.io/instance: rester

kubectl server side diff sees this invalid:

$ kubectl diff -f deployment.yaml --server-side
Error from server: failed to create typed patch object (default/rester; apps/v1, Kind=Deployment): .spec.template.labels: field not declared in schema

flux ignores the schema issue and says everything is fine:

$ flux diff kustomization --namespace=gitops-rester-kustomize-dev rester-on-gitlabx --path=./ --verbose
✓  Kustomization diffing...

NOTE: when we push this change kustomize-controller does the same like flux cli and shows everything in ready state (but obviously the change is not applied)

Steps to reproduce

• create a deployment and apply it with flux
• modify the deployment manifest and add .spec.template.labels to it
• commit the change and wait for reconcile. flux will not report error

Expected behavior

Flux should report an error about the invalid schema so users understand why their "change" is not synced.

Screenshots and recordings

No response

OS / Distro

Ubuntu 24.04 LTS

Flux version

v2.7.5

Flux check

$ flux check
► checking prerequisites
✗ flux 2.7.5 <2.8.3 (new CLI version is available, please upgrade)
✔ Kubernetes 1.34.3-eks-3c60543 >=1.32.0-0
► checking version in cluster
✔ distribution: flux-v2.7.5
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.4.5
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.7.3
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.7.5
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.7.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1
✔ externalartifacts.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

GitLab

Container Registry provider

Harbor

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Flux drops unknown fields from apps/v1 and v1

[Cajga]

Hi @stefanprodan , thanks for the information.

The current behavior (no error, neither a warning on a schema issue) took a lot of time to our devs to debug (all what they saw is that their change is not synced even if the commit hash was updated). In fact, they could not even find the problem (which resulted in a disappointment in flux that they raised as well) so they escalated it to the platform team. It would be really nice to have some feedback on such issues.

[stefanprodan]

There is no way we can surface these issues, the default behaviour of Kubernetes is to drop unknown fields at admission. What you should do as a platform admin, is to follow our best practices and configure static validation of all Kubernetes manifests in CI. Devs should not be able to merge PRs with invalid YAML. Please see: https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/scripts/validate.sh

[Cajga]

Maybe this could be something that could go together with some implementation of surfacing warnings from applies (although the scope of that request if about the warning given back by the API itself): #5703

The problem is the different behavior in the tooling that devs use. If you tryu to apply this with kubectl it drops an error:

$ kubectl apply -f deplyoment.yaml --server-side --namespace app-rester-kustomize-dev
Error from server: failed to create typed patch object (app-rester-kustomize-dev/rester; apps/v1, Kind=Deployment): .spec.template.labels: field not declared in schema

I guess, based on your comment above, this validation is done by kubectl itself. But flux does not do such validation.

While I understand you point about early feedback and validate at CI (and I fully agree with you), in our case, CI pipelines are not managed by platform admins (note: we use flux-multi-tenancy and each project have their own deployment repo). Some projects have devops teams who build these but some are built directly by devs.

Would it be possible/make sense to add a feature to flux so, the controller does this validation before apply?

[stefanprodan]

Flux uses server-side apply, pulling all the CRDs and the 1500 OpenAI Specs of the builtin resources to validate client-side would result in GB of extra RAM usage.

[Cajga]

Hmm... now that you wrote this, I realized that kubectl is in fact did not do this validation on the client side (I used --server-side option anyway).
So, the error of the apply command that I did above is indeed coming from the API server. Here is the audit log snippet:

  "responseObject": {
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Failure",
    "message": "failed to create typed patch object (app-rester-kustomize-dev/rester; apps/v1, Kind=Deployment): .spec.template.labels: field not declared in schema",
    "code": 500
  },

So, and it comes even with dry-run... So if flux would have try to apply this with server side dry-run then it should have received the same error. Why is this not happening in this case?

[stefanprodan]

Because native kinds go via the default converter which drops unknown fields https://github.com/fluxcd/pkg/tree/main/ssa/normalize

[Cajga]

Ok, so if I understand correctly, the "schema error hiding issue" is only there for native kinds due to this normalization. K8s API/and flux would report error for all other kinds (e.g. CRDs.) (I will test this now).
So, if flux would implement the validation, it would only need to "pull the/deal with" native kinds or? If yes, would that be possible/make sense to implement so the behavior is the same for both CRDs and native kinds?

[stefanprodan]

We can not send native kinds to the API without normalisation as this would result in continuous drift due to various inconsistencies in Kubernetes API. Like Secrets with stringData that always drift, ports in pod spec that fail SSA, HPA random ordering of rules, etc.

[Cajga]

Yes, the behavior of flux is different between CRDs and native kinds. I tried the same with a probe from prometheus-operator:

$ flux tree -n gitops-rester-kustomize-dev kustomization rester-on-gitlabx 
Kustomization/gitops-rester-kustomize-dev/rester-on-gitlabx
├── Deployment/app-rester-kustomize-dev/rester
└── Probe/app-rester-kustomize-dev/platform-demo

$ vim probe.yaml 
$ git diff
diff --git a/kubernetes/plain/flux-schema-issue/probe.yaml b/kubernetes/plain/flux-schema-issue/probe.yaml
index c3efa6d..f150ee9 100644
--- a/kubernetes/plain/flux-schema-issue/probe.yaml
+++ b/kubernetes/plain/flux-schema-issue/probe.yaml
@@ -5,6 +5,7 @@ metadata:
   labels:
     type: prometheus
 spec:
+  thisIsASchemaIssue: test
   jobName: platform-demo-probe
   interval: 60s
   prober:

$ flux diff kustomization --namespace=gitops-rester-kustomize-dev rester-on-gitlabx --path=./ --verbose
✓  Kustomization diffing...
✗ Probe/app-rester-kustomize-dev/platform-demo dry-run failed: failed to create typed patch object (app-rester-kustomize-dev/platform-demo; monitoring.coreos.com/v1, Kind=Probe): .spec.thisIsASchemaIssue: field not declared in schema
$

We can not send native kinds to the API without normalisation as this would result in continuous drift due to various inconsistencies in Kubernetes API.

Yes, I understand this. But can we somehow reach the same behavior? Wouldn't it be possible to try to validate the native kinds (or just do a dry-run?) right before we normalize it (or during) to catch these issues?

[stefanprodan]

Wouldn't it be possible to try to validate the native kinds (or just do a dry-run?) right before we normalize it (or during) to catch these issues?

No, as a dry-run will always result in drift.

[Cajga]

The extra dry-run before the normalization would not be used to check the drift but to validate the not-yet-normalized manifests that the user would want to apply. An extra step before the normalization and drift checking.
In case the manifests are wrong (like my example), the API would immediately return an error like above with kubectl -> no need to normalize and check the drift...

[Cajga]

With the current behavior, what is getting applied by flux is not what the user wants. This is due to the normalization. We silently fix the error in the manifest. IMO, in some cases (like above) this behavior breaks the GitOps principles.

[stefanprodan]

I'm not considering doing 2x API calls, this is how Flux worked since forever.

[Cajga]

All right @stefanprodan, I see the overhead that this might cause and possibly it would not worth it for the benefit. In your view, would client side validation (before normalization) only of the native kinds that gets normalized be a valid option? Or would that also be too large overhead?