Skip to content

fix(truvami-monitoring)!: make Alertmanager optional and default it off #70

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
michaelbeutler wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix(truvami-monitoring)!: make Alertmanager optional and default it off #70

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3b0d1c6
fix(truvami-monitoring)!: make Alertmanager optional and default it off
michaelbeutler Jul 22, 2026
46904cf
Merge 3b0d1c67c6cef36225eea179032cf0b707f58d84 into 4df51d1d6d2ebbea1…
michaelbeutler Jul 22, 2026
57a8e8d
chore: add chart diagrams for PR review
github-actions[bot] Jul 22, 2026
ba4ca4d
fix(truvami-monitoring): preserve explicit empty alertmanagerConfigSe…
michaelbeutler Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/truvami-monitoring/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.4.0
version: 0.5.0
8 changes: 6 additions & 2 deletions charts/truvami-monitoring/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# truvami-monitoring

![Version: 0.4.0](https://img.shields.io/badge/Version-0.4.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)

This chart contains all CRD's for the truvami-stack monitoring and alerting.

Expand All @@ -17,7 +17,8 @@ This chart contains all CRD's for the truvami-stack monitoring and alerting.
| alertLabels.altravis_test | string | `"true"` | |
| alertLabels.code | string | `"SBCC"` | |
| alertLabels.customer | string | `"truvami"` | |
| alertmanager.enabled | bool | `true` | |
| alertmanager.configSelectorLabels | string | `nil` | |
| alertmanager.enabled | bool | `false` | |
Comment on lines +20 to +21

@coderabbitai coderabbitai Bot Jul 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document configSelectorLabels as a map, not a string.

The value accepts nil, {}, or a label map. Documenting it as string conflicts with values.yaml and can lead users to render an invalid matchLabels value.

Proposed documentation fix 
-| alertmanager.configSelectorLabels | string | `nil` |  |
+| alertmanager.configSelectorLabels | object | `nil` |  |
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
| alertmanager.configSelectorLabels | string | `nil` | |
| alertmanager.enabled | bool | `false` | |
| alertmanager.configSelectorLabels | object | `nil` | |
| alertmanager.enabled | bool | `false` | |
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@charts/truvami-monitoring/README.md` around lines 20 - 21, Update the
alertmanager.configSelectorLabels entry in the chart README table to document
its type as a map rather than a string, while preserving the existing default
value nil.

| alertmanager.global.http_config.enable_http2 | bool | `true` | |
| alertmanager.global.http_config.follow_redirects | bool | `true` | |
| alertmanager.global.resolve_timeout | string | `"5m"` | |
Expand All @@ -32,6 +33,7 @@ This chart contains all CRD's for the truvami-stack monitoring and alerting.
| alertmanager.inhibit_rules[1].equal[0] | string | `"service"` | |
| alertmanager.inhibit_rules[1].source_matchers[0] | string | `"maintenance=\"true\""` | |
| alertmanager.inhibit_rules[1].target_matchers[0] | string | `"service=~\".*\""` | |
| alertmanager.portName | string | `"web"` | |
| alertmanager.receivers.critical.enabled | bool | `true` | |
| alertmanager.receivers.critical.slack.channel | string | `"#critical-alerts"` | |
| alertmanager.receivers.critical.slack.enabled | bool | `false` | |
Expand Down Expand Up @@ -91,10 +93,12 @@ This chart contains all CRD's for the truvami-stack monitoring and alerting.
| alertmanager.receivers.teams.summary | string | `"{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}"` | |
| alertmanager.receivers.teams.title | string | `"{{ .GroupLabels.alertname }}"` | |
| alertmanager.receivers.teams.webhook_url | string | `""` | |
| alertmanager.replicas | int | `2` | |
| alertmanager.resources.limits.cpu | string | `"500m"` | |
| alertmanager.resources.limits.memory | string | `"512Mi"` | |
| alertmanager.resources.requests.cpu | string | `"50m"` | |
| alertmanager.resources.requests.memory | string | `"128Mi"` | |
| alertmanager.retention | string | `"120h"` | |
| alertmanager.route.group_by[0] | string | `"alertname"` | |
| alertmanager.route.group_by[1] | string | `"cluster"` | |
| alertmanager.route.group_by[2] | string | `"service"` | |
Expand Down
7 changes: 6 additions & 1 deletion charts/truvami-monitoring/templates/alertmanager-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,15 @@ spec:
repeatInterval: '1h'

# Watchdog alerts route
# NOTE: this must match the alert name emitted by templates/rule-watchdog.yaml,
# which is "TruvamiWatchdog". It previously read "Watchdog", and because the
# matcher is an exact match it never fired - so the dead-man's-switch
# heartbeat never reached watchdog.webhookUrl even when everything else in
# the alerting path was healthy.
- matchers:
- matchType: "="
name: "alertname"
value: "Watchdog"
value: "TruvamiWatchdog"
receiver: 'watchdog-heartbeat'
groupWait: '0s'
groupInterval: '1m'
Expand Down
39 changes: 39 additions & 0 deletions charts/truvami-monitoring/templates/alertmanager-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{{- if .Values.alertmanager.enabled }}
{{/*
Service fronting this release's Alertmanager pods.

The Prometheus CR's spec.alerting.alertmanagers[] entry addresses Alertmanager
by *Service* name - prometheus-operator resolves that Service's Endpoints to
build the alerting targets. Without this object the reference dangles and
Prometheus reports zero activeAlertmanagers, so every firing alert is silently
dropped. The operator's own `alertmanager-operated` Service is not a substitute:
it selects `app.kubernetes.io/name: alertmanager` across the whole namespace, so
where several Alertmanager CRs coexist it would fan alerts out to all of them.

The `alertmanager` label is set by prometheus-operator on the pods it creates
for an Alertmanager CR, and carries that CR's name - so this selector matches
only the Alertmanager owned by this release.
*/}}
apiVersion: v1
kind: Service
metadata:
name: {{ include "truvami-monitoring.fullname" . }}
namespace: {{ .Release.Namespace | quote }}
labels:
{{- include "truvami-monitoring.labels" . | nindent 4 }}
app.kubernetes.io/component: alertmanager
spec:
type: ClusterIP
# Alertmanager gossips over the mesh ports; peers must be discoverable before
# they are Ready, and Prometheus should keep addressing a peer that is still
# settling rather than lose its only alerting target during a rollout.
publishNotReadyAddresses: true
selector:
alertmanager: {{ include "truvami-monitoring.fullname" . }}
app.kubernetes.io/name: alertmanager
ports:
- name: {{ .Values.alertmanager.portName | default "web" }}
port: 9093
targetPort: {{ .Values.alertmanager.portName | default "web" }}
protocol: TCP
{{- end }}
48 changes: 45 additions & 3 deletions charts/truvami-monitoring/templates/prometheus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,17 @@ spec:
serviceMonitorNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: {{ .Release.Namespace }}
{{- if .Values.alertmanager.enabled }}
alerting:
alertmanagers:
# Addresses the Service created by templates/alertmanager-service.yaml.
# Only emitted when the Alertmanager is enabled - pointing Prometheus at a
# Service that is not rendered leaves it with zero activeAlertmanagers and
# silently discards every firing alert.
- namespace: {{ .Release.Namespace }}
name: "{{ include "truvami-monitoring.fullname" . }}"
port: web
port: {{ .Values.alertmanager.portName | default "web" }}
{{- end }}
enableRemoteWriteReceiver: {{ .Values.prometheus.enableRemoteWriteReceiver | default "false" }}
{{- if .Values.thanos.enabled }}
thanos:
Expand All @@ -106,12 +112,48 @@ spec:
{{- end }}
{{- end }}

{{- if .Values.alertmanager.enabled }}
---
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
name: "{{ include "truvami-monitoring.fullname" . }}"
labels:
{{- include "truvami-monitoring.labels" . | nindent 4 }}
spec:
replicas: 2
configSecret: "{{ include "truvami-monitoring.fullname" . }}-alertmanager-config"
replicas: {{ .Values.alertmanager.replicas | default 2 }}
portName: {{ .Values.alertmanager.portName | default "web" }}
{{- with .Values.alertmanager.retention }}
retention: {{ . | quote }}
{{- end }}
# Use the AlertmanagerConfig rendered by templates/alertmanager-config.yaml as
# this Alertmanager's top-level configuration.
#
# This replaces the previous `configSecret` reference, which named a Secret
# ("<fullname>-alertmanager-config") that no template in this chart has ever
# created. With that Secret absent, prometheus-operator fell back to its
# built-in default configuration, whose root route sends everything to the
# "null" receiver - so alerts were accepted and then discarded.
alertmanagerConfiguration:
name: {{ include "truvami-monitoring.fullname" . }}-config
# Additionally select any AlertmanagerConfig in this namespace that opts in,
# so teams can attach extra routes/receivers without editing this chart.
# Previously unset, which meant zero AlertmanagerConfigs were ever selected.
#
# An explicitly configured empty map is a meaningful value here -
# matchLabels: {} selects *every* AlertmanagerConfig in the namespace - so it
# must be distinguishable from "not configured".
#
# `default` cannot do that (it treats {} as empty), and neither can `hasKey`:
# the key is always present because values.yaml declares it. So the declared
# default is null, and only null means "not configured" - Helm lets a user
# map replace a null base wholesale, whereas a non-empty default map would be
# merged with the user's value and could never be reduced back to empty.
{{- $configSelector := .Values.alertmanager.configSelectorLabels }}
{{- if kindIs "invalid" $configSelector }}
{{- $configSelector = dict "truvami.com/alertmanager-config" "true" }}
{{- end }}
alertmanagerConfigSelector:
matchLabels: {{- toYaml $configSelector | nindent 6 }}
Comment on lines +152 to +157

@coderabbitai coderabbitai Bot Jul 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

values="$(mktemp)"
trap 'rm -f "$values"' EXIT

cat >"$values" <<'EOF'
alertmanager:
  enabled: true
  configSelectorLabels: invalid
EOF

if helm template review charts/truvami-monitoring -f "$values" >/dev/null 2>&1; then
  echo "Non-map configSelectorLabels was accepted" >&2
  exit 1
fi

Repository: truvami/helm

Length of output: 150

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Locate the relevant chart files and show the surrounding template code.
git ls-files charts/truvami-monitoring | sed -n '1,200p'
printf '\n--- template slice ---\n'
sed -n '130,190p' charts/truvami-monitoring/templates/prometheus.yaml

printf '\n--- values references ---\n'
rg -n "configSelectorLabels|alertmanager\.configSelectorLabels|alertmanager:" charts/truvami-monitoring -n

Repository: truvami/helm

Length of output: 6267

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Probe rendering behavior for several selector shapes to see what actually fails.
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT

cases=(
  "null|alertmanager:\n  enabled: true\n  configSelectorLabels: null\n"
  "empty-map|alertmanager:\n  enabled: true\n  configSelectorLabels: {}\n"
  "string|alertmanager:\n  enabled: true\n  configSelectorLabels: invalid\n"
  "list|alertmanager:\n  enabled: true\n  configSelectorLabels:\n    - a\n    - b\n"
)

for item in "${cases[@]}"; do
  name="${item%%|*}"
  body="${item#*|}"
  file="$tmpdir/$name.yaml"
  printf '%b' "$body" >"$file"
  printf '\n== %s ==\n' "$name"
  if out="$(helm template review charts/truvami-monitoring -f "$file" 2>&1)"; then
    echo "OK"
    printf '%s\n' "$out" | sed -n '1,40p'
  else
    echo "FAIL"
    printf '%s\n' "$out" | sed -n '1,80p'
  fi
done

Repository: truvami/helm

Length of output: 381

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Inspect docs/defaults to see whether the field is meant to be a map or string.
rg -n "configSelectorLabels|matchLabels|AlertmanagerConfig" README.md charts/truvami-monitoring -g '!**/Chart.lock'

Repository: truvami/helm

Length of output: 2954

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

# Show any chart schema or template-side validation relevant to this field.
for f in charts/truvami-monitoring/values.schema.json charts/truvami-monitoring/values.yaml charts/truvami-monitoring/templates/prometheus.yaml; do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    wc -l "$f"
    sed -n '1,240p' "$f" | sed -n '/configSelectorLabels/,+40p'
  fi
done

Repository: truvami/helm

Length of output: 839

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "--- values.schema.json ---"
cat -n charts/truvami-monitoring/values.schema.json

echo
echo "--- values.yaml around configSelectorLabels ---"
sed -n '404,428p' charts/truvami-monitoring/values.yaml

echo
echo "--- README table entry ---"
sed -n '14,26p' charts/truvami-monitoring/README.md

Repository: truvami/helm

Length of output: 2105

Reject non-map alertmanager.configSelectorLabels values.

matchLabels must be a map; strings or lists will render an invalid Alertmanager CR. Keep {} working, but fail on anything else.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@charts/truvami-monitoring/templates/prometheus.yaml` around lines 152 - 157,
Update the alertmanagerConfigSelector logic around $configSelector to accept
only map values, including an explicitly empty {} map, and call Helm fail for
strings, lists, or any other non-map value before rendering matchLabels.
Preserve the existing default selector behavior when the value is unset or
invalid.

resources: {{ toYaml .Values.alertmanager.resources | default "{}" | nindent 4 }}
{{- end }}
47 changes: 46 additions & 1 deletion charts/truvami-monitoring/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,52 @@ watchdog:

# AlertManager configuration
alertmanager:
enabled: true
# Alerting is handled by Grafana-managed alert rules, not by Alertmanager, so
# this defaults to off.
#
# When false, no Alertmanager CR, AlertmanagerConfig, Alertmanager secrets or
# Alertmanager Service are rendered, AND the Prometheus CR omits its
# spec.alerting block - so Prometheus is not left pointed at a Service that
# was never created.
#
# Prior to chart 0.5.0 the Alertmanager CR ignored this flag and was always
# created, so environments that set `enabled: false` still ran Alertmanager
# pods.
#
# From 0.5.0 the chart stops *rendering* those objects - it does not delete
# what is already deployed. Where ArgoCD pruning is disabled (it is on both
# the dev and prod truvami-monitoring apps), the existing Alertmanager CR and
# the pods it owns survive the upgrade and must be removed by hand:
# kubectl -n <namespace> delete alertmanager <release-name>
enabled: false

# Everything below applies only when `enabled: true`.

# Number of Alertmanager replicas. Keep >1 so a single node event cannot take
# the alerting path down.
replicas: 2

# Name of the Alertmanager container port, and the port name referenced by
# both the Service and the Prometheus spec.alerting entry. Changing this
# keeps all three in sync.
portName: web

# How long Alertmanager retains resolved alert state.
retention: 120h

# Labels an AlertmanagerConfig in this namespace must carry to be merged into
# this Alertmanager, via spec.alertmanagerConfigSelector. The chart's own
# config is attached separately as the top-level configuration and does not
# need this label.
#
# Left null, the chart selects on `truvami.com/alertmanager-config: "true"`.
# Set a map to select on your own labels, or `{}` to select EVERY
# AlertmanagerConfig in the namespace (rarely what you want).
#
# The default is expressed in the template rather than here on purpose: Helm
# merges map values, so a non-empty default here could be added to but never
# reduced back to `{}`.
configSelectorLabels:

# Resource configuration for Alertmanager
resources:
Expand Down
Binary file modified diagrams/truvami-monitoring.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added diagrams/truvami-monitoring.previous.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading