Skip to content

chore(observability): vendor kubernetes-event-exporter and drop the Bitnami source - #2075

Merged
Smana merged 3 commits into
mainfrom
chore/vendor-event-exporter
Sep 21, 2026
Merged

Smana merged 3 commits into
mainfrom
chore/vendor-event-exporter

Conversation

@Smana

@Smana Smana commented Sep 21, 2026

Copy link
Copy Markdown
Owner

The last Bitnami dependency was the kubernetes-event-exporter chart; the image had already moved to ghcr.io/civitatis/kubernetes-event-exporter. No maintained upstream chart exists, so this vendors the chart's rendered output as manifests the repo owns, and deletes the bitnami HelmRepository. Closes #1089.

ADR-0040 records the choice, and why the itakurah fork, the ownkube rewrite and staying on Bitnami each lost.

What changed

  • observability/base/kubernetes-event-exporter/ now holds 8 plain manifests plus config.yaml, byte-identical to the Bitnami 3.6.3 render apart from the stripped helm.sh/chart, managed-by: Helm, app.kubernetes.io/version and checksum/config. The Deployment selector matches the live one exactly (app.kubernetes.io/instance + name), so the immutable field can't wedge the apply.
  • A config change still rolls the pod. The chart did this with a checksum annotation. A configMapGenerator does it now: the ConfigMap is kubernetes-event-exporter-<hash>, and the Deployment's volume follows the hash. It's the repo's first generator; a comment in kustomization.yaml says why. Flux postBuild substitution still applies, since it runs after kustomize build.
  • flux/sources/helmrepo-bitnami.yaml is deleted; nothing else referenced it.
  • Renovate now tracks the image for the first time. A HelmRelease's split image.* values were read by no manager; the kubernetes manager's .ya?ml glob covers the Deployment.

Rollout: expect a gap of about 3 minutes on each cluster

On merge, kustomize-controller applies the plain objects, then prunes the HelmRelease. helm-controller then uninstalls the release, which deletes the same-named Deployment, Service, RBAC and so on. The Kustomization recreates them on its next interval, 3 minutes later. The generated ConfigMap's hashed name keeps it out of that race.

Events raised during the gap aren't exported, because the exporter drops events older than 5s at startup. Nothing reads this stream in real time: its only alert needs 15 minutes of errors, and no dashboard or LogsQL rule uses it. Running flux reconcile ks observability right after the HelmRelease disappears closes the gap in seconds.

A zero-gap handover needs a prior PR: either a helm.sh/resource-policy: keep postRenderer, which leaves stale Helm metadata behind, or suspending the release, which orphans the HelmChart and the release Secrets. Neither is worth it for a stateless exporter.

After merge, on each cluster:

  • flux get ks observability
  • kubectl -n observability get deploy kubernetes-event-exporter --show-labels should show no Helm labels.
  • The sh.helm.release.v1.kubernetes-event-exporter.* Secrets should be gone.

Follow-ups, now that the repo owns these files

  • Replace the chart's plain NetworkPolicy with a default-deny CiliumNetworkPolicy: ingress 2112 from vmagent; egress to kube-apiserver, kube-dns and vlsingle 9428. This gap predates this PR.
  • The ClusterRole grants get/list/watch on */*, Secrets included. Narrow it, or set omitLookup.
  • Convert the ServiceMonitor to a VMServiceScrape, the repo's convention (it already works through the operator's converter).

Evidence

Run on this branch, rebased on main @ affdd50b:

  • task ci:validate exits 0: Summary: 2101 resources found in 272 files - Valid: 2101, Invalid: 0, Skipped: 0, and "All gates passed" (Polaris 88).
  • task ci:test: 22 passed, 1 skipped, 0 failed.
  • task ci:links, task ci:doc-paths, and task ci:doc-claims (30/30) exit 0.
  • kustomize build: the ConfigMap and the Deployment volume both name kubernetes-event-exporter-m4mhb72d8m, and ${cluster_name} survives for postBuild.

…itnami source

The image already ran a community fork (ghcr.io/civitatis/kubernetes-event-exporter);
only the chart itself still came from bitnami's OCI HelmRepository. Neither
maintained fork chart (itakurah, ownkube) clears this repo's bar - one has no
resources field and an unfixed template bug, the other is an unverified
rewrite - and upstream ships no chart at all, so the chart's own rendered
output is vendored as plain manifests instead. See ADR-0040.

Closes the last item of #1089.
verify-doc-paths.sh failed on the ADR's backticked
flux/sources/helmrepo-bitnami.yaml, deleted in the previous commit. Also:
ownkube was never rendered/diffed (it's a rewrite, judged on source and
maintenance record instead), Valkey and RabbitMQ are not "mid-migration" off
Bitnami (Valkey is done per SPEC-012, RabbitMQ's only user was removed with
grafana-oncall), the image tag is Renovate-tracked for the first time rather
than merely "only" tracked, and logs.md still described chart values
(metrics.enabled/serviceMonitor.enabled) that no longer exist.
…anges

The exporter reads -conf once at startup with no file watcher, and vendoring
the config as a plain ConfigMap (previous commit) dropped the HelmRelease's
checksum/config annotation that used to force a new ReplicaSet on a config
edit. Switch to Kustomize's configMapGenerator: its content-hashed name
still rolls the pod, and the built-in name-reference transformer updates the
Deployment's volume automatically. This is the repo's first
configMapGenerator, noted inline.
@github-actions

Copy link
Copy Markdown
Contributor

🔍 Rendered manifest diff — this PR vs main (desired state)

8 changed · 1 added · 3 removed

Rendered with kustomize build + helm template (source of truth = git), so Helm-expanded workloads are included. Shows what Flux will apply — not a diff against live cluster state (drift is alerted on separately), and not CRD-defaulted / webhook-mutated output. Secret values are redacted; per-render noise (webhook caBundles, checksum/* annotations, render timestamps) is normalized out.

🟡 changed — apps/v1/Deployment/observability/kubernetes-event-exporter
--- a/apps/v1/Deployment/observability/kubernetes-event-exporter
+++ b/apps/v1/Deployment/observability/kubernetes-event-exporter
@@ -4,10 +4,7 @@
   labels:
     app.kubernetes.io/component: kubernetes-event-exporter
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter
   namespace: observability
 spec:
@@ -21,19 +18,12 @@
     type: RollingUpdate
   template:
     metadata:
-      annotations:
-        checksum/config: <redacted by diff-bundles>
       labels:
         app.kubernetes.io/component: kubernetes-event-exporter
         app.kubernetes.io/instance: kubernetes-event-exporter
-        app.kubernetes.io/managed-by: Helm
         app.kubernetes.io/name: kubernetes-event-exporter
-        app.kubernetes.io/version: 1.7.0
-        helm.sh/chart: kubernetes-event-exporter-3.6.3
     spec:
       affinity:
-        nodeAffinity: null
-        podAffinity: null
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - podAffinityTerm:
@@ -106,5 +96,5 @@
       serviceAccountName: kubernetes-event-exporter
       volumes:
       - configMap:
-          name: kubernetes-event-exporter
+          name: kubernetes-event-exporter-m4mhb72d8m
         name: cfg
🔴 removed — helm.toolkit.fluxcd.io/v2/HelmRelease/observability/kubernetes-event-exporter
--- a/helm.toolkit.fluxcd.io/v2/HelmRelease/observability/kubernetes-event-exporter
+++ b/helm.toolkit.fluxcd.io/v2/HelmRelease/observability/kubernetes-event-exporter
@@ -1,48 +0,0 @@
-apiVersion: helm.toolkit.fluxcd.io/v2
-kind: HelmRelease
-metadata:
-  name: kubernetes-event-exporter
-  namespace: observability
-spec:
-  chart:
-    spec:
-      chart: kubernetes-event-exporter
-      interval: 12h
-      sourceRef:
-        kind: HelmRepository
-        name: bitnami
-        namespace: flux-system
-      version: 3.6.3
-  driftDetection:
-    mode: enabled
-  interval: 30m
-  values:
-    config:
-      clusterName: foobar
-      logFormat: json
-      logLevel: info
-      metricsNamePrefix: event_exporter_
-      receivers:
-      - loki:
-          streamLabels:
-            source: kubernetes-event-exporter
-          url: http://victoria-logs-victoria-logs-single-server.observability.svc.cluster.local:9428/insert/loki/api/v1/push
-        name: victorialogs
-      route:
-        routes:
-        - match:
-          - receiver: victorialogs
-    global:
-      security:
-        allowInsecureImages: true
-    image:
-      registry: ghcr.io
-      repository: civitatis/kubernetes-event-exporter
-      tag: '1.8'
-    metrics:
-      enabled: true
-      prometheusRule:
-        enabled: false
-      serviceMonitor:
-        enabled: true
-        namespace: observability
🟡 changed — monitoring.coreos.com/v1/ServiceMonitor/observability/kubernetes-event-exporter
--- a/monitoring.coreos.com/v1/ServiceMonitor/observability/kubernetes-event-exporter
+++ b/monitoring.coreos.com/v1/ServiceMonitor/observability/kubernetes-event-exporter
@@ -4,10 +4,7 @@
   labels:
     app.kubernetes.io/component: kubernetes-event-exporter
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter
   namespace: observability
 spec:
🟡 changed — networking.k8s.io/v1/NetworkPolicy/observability/kubernetes-event-exporter
--- a/networking.k8s.io/v1/NetworkPolicy/observability/kubernetes-event-exporter
+++ b/networking.k8s.io/v1/NetworkPolicy/observability/kubernetes-event-exporter
@@ -3,10 +3,7 @@
 metadata:
   labels:
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter
   namespace: observability
 spec:
🟡 changed — policy/v1/PodDisruptionBudget/observability/kubernetes-event-exporter
--- a/policy/v1/PodDisruptionBudget/observability/kubernetes-event-exporter
+++ b/policy/v1/PodDisruptionBudget/observability/kubernetes-event-exporter
@@ -3,10 +3,7 @@
 metadata:
   labels:
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter
   namespace: observability
 spec:
🟡 changed — rbac.authorization.k8s.io/v1/ClusterRole/-/kubernetes-event-exporter-observability
--- a/rbac.authorization.k8s.io/v1/ClusterRole/-/kubernetes-event-exporter-observability
+++ b/rbac.authorization.k8s.io/v1/ClusterRole/-/kubernetes-event-exporter-observability
@@ -3,10 +3,7 @@
 metadata:
   labels:
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter-observability
 rules:
 - apiGroups:
🟡 changed — rbac.authorization.k8s.io/v1/ClusterRoleBinding/-/kubernetes-event-exporter-observability
--- a/rbac.authorization.k8s.io/v1/ClusterRoleBinding/-/kubernetes-event-exporter-observability
+++ b/rbac.authorization.k8s.io/v1/ClusterRoleBinding/-/kubernetes-event-exporter-observability
@@ -3,10 +3,7 @@
 metadata:
   labels:
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter-observability
 roleRef:
   apiGroup: rbac.authorization.k8s.io
🔴 removed — source.toolkit.fluxcd.io/v1/HelmRepository/flux-system/bitnami
--- a/source.toolkit.fluxcd.io/v1/HelmRepository/flux-system/bitnami
+++ b/source.toolkit.fluxcd.io/v1/HelmRepository/flux-system/bitnami
@@ -1,9 +0,0 @@
-apiVersion: source.toolkit.fluxcd.io/v1
-kind: HelmRepository
-metadata:
-  name: bitnami
-  namespace: flux-system
-spec:
-  interval: 5m
-  type: oci
-  url: oci://registry-1.docker.io/bitnamicharts
🔴 removed — v1/ConfigMap/observability/kubernetes-event-exporter
--- a/v1/ConfigMap/observability/kubernetes-event-exporter
+++ b/v1/ConfigMap/observability/kubernetes-event-exporter
@@ -1,13 +0,0 @@
-apiVersion: v1
-data:
-  config.yaml: "clusterName: foobar\nleaderElection: {}\nlogFormat: json\nlogLevel: info\nmetricsNamePrefix: event_exporter_\nreceivers:\n- loki:\n    streamLabels:\n      source: kubernetes-event-exporter\n    url: http://victoria-logs-victoria-logs-single-server.observability.svc.cluster.local:9428/insert/loki/api/v1/push\n  name: victorialogs\nroute:\n  routes:\n  - match:\n    - receiver: victorialogs\n"
-kind: ConfigMap
-metadata:
-  labels:
-    app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
-  name: kubernetes-event-exporter
-  namespace: observability
🟢 added — v1/ConfigMap/observability/kubernetes-event-exporter-m4mhb72d8m
--- a/v1/ConfigMap/observability/kubernetes-event-exporter-m4mhb72d8m
+++ b/v1/ConfigMap/observability/kubernetes-event-exporter-m4mhb72d8m
@@ -0,0 +1,10 @@
+apiVersion: v1
+data:
+  config.yaml: "clusterName: foobar\nleaderElection: {}\nlogFormat: json\nlogLevel: info\nmetricsNamePrefix: event_exporter_\nreceivers:\n- loki:\n    streamLabels:\n      source: kubernetes-event-exporter\n    url: http://victoria-logs-victoria-logs-single-server.observability.svc.cluster.local:9428/insert/loki/api/v1/push\n  name: victorialogs\nroute:\n  routes:\n  - match:\n    - receiver: victorialogs\n"
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/instance: kubernetes-event-exporter
+    app.kubernetes.io/name: kubernetes-event-exporter
+  name: kubernetes-event-exporter-m4mhb72d8m
+  namespace: observability
🟡 changed — v1/Service/observability/kubernetes-event-exporter-metrics
--- a/v1/Service/observability/kubernetes-event-exporter-metrics
+++ b/v1/Service/observability/kubernetes-event-exporter-metrics
@@ -7,10 +7,7 @@
   labels:
     app.kubernetes.io/component: metrics
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter-metrics
   namespace: observability
 spec:
🟡 changed — v1/ServiceAccount/observability/kubernetes-event-exporter
--- a/v1/ServiceAccount/observability/kubernetes-event-exporter
+++ b/v1/ServiceAccount/observability/kubernetes-event-exporter
@@ -4,9 +4,6 @@
 metadata:
   labels:
     app.kubernetes.io/instance: kubernetes-event-exporter
-    app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-event-exporter
-    app.kubernetes.io/version: 1.7.0
-    helm.sh/chart: kubernetes-event-exporter-3.6.3
   name: kubernetes-event-exporter
   namespace: observability

@Smana
Smana merged commit 1c987b1 into main Sep 21, 2026
12 checks passed
@Smana
Smana deleted the chore/vendor-event-exporter branch September 21, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate from Bitnami's sources

1 participant