fix(chart): make GPU alerts exporter-agnostic instead of DCGM-only

[Tanguille]

Fixes #1356. Off main, independent of #1368, touches no dashboard.

All four GPU rules key on DCGM_FI_DEV_* while rules.gpu.enabled defaults to true, so a cluster without DCGM gets four rules that never fire and never error.

Change

Each rule ORs the normalized expression for every exporter the chart knows, so the one a cluster runs is the one that evaluates. rules.gpu.vendor is gone: nothing needs configuring.

Role	Expression
util	DCGM_FI_DEV_GPU_UTIL or amdgpu_gpu_busy_percent or drm_engine_utilization_ratio{engine="gpu"} * 100
temp	DCGM_FI_DEV_GPU_TEMP or node_hwmon_temp_celsius{sensor="temp1"} * on(chip, instance) group_left() node_hwmon_chip_names{chip_name="amdgpu"} or drm_temperature_celsius{sensor="edge"}
power	DCGM_FI_DEV_POWER_USAGE or node_hwmon_power_watt * on(chip, instance) group_left() node_hwmon_chip_names{chip_name="amdgpu"} or drm_power_watts
mem	(DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) * 100 or (amdgpu_vram_used_bytes / amdgpu_vram_total_bytes) * 100 or (drm_memory_used_bytes{pool="vram"} / drm_memory_total_bytes{pool="vram"}) * 100

Each union is parenthesised before the comparison: (a or b or c) > 90, not a or b or (c > 90). Rule bodies are written once from a role-keyed helper. GPUMetricsMissing is now absent() over all three families, ANDed.

GPUMemoryPressure ships off by default (rules.gpu.memoryPressure.enabled): on an inference host VRAM utilization reports the fraction the runtime was configured to reserve (mem-fraction-static, gpu-memory-utilization), not pressure. It is held for the process lifetime and does not fall under load, so no static threshold on it distinguishes healthy from saturated.

Normalization, measured on a live AMD cluster

Finding	Measurement	Fix
Sensor split	edge 69 C, junction 82 C, mem 79 C on one GPU; DCGM reports one edge-equivalent series	pin sensor="edge" / temp1
node_hwmon scope	unscoped, fired GPUHighTemperature at 89.25 C off a chipset sensor while the GPU sat at 69 C	join node_hwmon_chip_names{chip_name="amdgpu"}, as amd-gpu-observability.json does
Memory pool	drm carries pool={vram,smem}; unfiltered denominator worked only by implicit label matching	pin pool="vram" both sides
VRAM threshold	99% on this host, but the number is the configured reservation, not demand	GPUMemoryPressure opt-in

Evidence

Rendered from this branch, evaluated against live VictoriaMetrics:

Rule	Result
GPUHighUtilization	fires, 99% busy under load
GPUHighTemperature	does not fire; union resolves to GPU sensors only (39 / 69 / 44 C)
GPUPowerLimit	does not fire, 246 W vs 250
GPUMetricsMissing	does not fire
GPUMemoryPressure	absent from rendered output by default

Both exporters run on that cluster, so the temp union returns the node_hwmon and drm branches for the same GPU: they agree at 69 C.

helm unittest 67/67, go test ./internal/metrics/... ok, helm lint clean. The nvidia path renders byte-identical to the current release apart from GPUMetricsMissing.

If you would rather keep an explicit vendor knob, or disagree with any of the normalization choices, say so and I will revert it.

[Defilan]

Tanguille, excellent, well-researched change, thank you. I verified every AMD/DCGM/drm metric name against amd-gpu-observability.json and the dashboard_sync_test.go allowlist and they all check out, including the correct call to source temp/power from node_hwmon_* rather than a nonexistent amdgpu_temperature_celsius.

One blocker: GPUMetricsMissing (prometheusrule.yaml:75) never fires. PromQL and matches only on exactly-equal label sets, and absent(drm_engine_utilization_ratio{engine="gpu"}) produces {engine="gpu"} while the other two absent() terms produce {}, so the intersection is empty even when all three exporter families are absent, the exact silent no-op #1356 exists to kill. promtool test rules against an empty series set confirms it. Drop the {engine="gpu"} matcher inside that absent() call (keep it in the util alert, where it's a value comparison and unaffected), and update the expected string in prometheusrule_test.yaml:196 to match.

Two smaller things: please add the Assisted-by: disclosure line to the PR body per CONTRIBUTING, and consider keeping {{ $labels.instance }} in the descriptions so alerts still name a GPU (instance is present on all three families). I'll authorize the CI runs so we get a green signal.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Defilan]

CI heads-up, and it's a real consequence of the change rather than a flake. The Test Prometheus Integration job fails at .github/workflows/helm-chart.yml:276:

grep -q "DCGM_FI_DEV_GPU_UTIL > 95"   →   ❌ Custom GPU threshold not applied

That assertion pins the old DCGM-only alert shape. Your generalized GPUHighUtilization now renders as:

(DCGM_FI_DEV_GPU_UTIL or amdgpu_gpu_busy_percent or drm_engine_utilization_ratio{engine="gpu"} * 100) > 95

so the custom threshold is applied correctly, but the contiguous string DCGM_FI_DEV_GPU_UTIL > 95 no longer exists (the or terms split it). promtool itself passes ("SUCCESS: 12 rules found"), so the rules are valid, it's purely the test string that's stale.

Since this PR intentionally changes the alert shape, please update that assertion in the same pass, e.g.:

if ! grep -qE 'DCGM_FI_DEV_GPU_UTIL.*> 95' /tmp/custom-thresholds.yaml; then

which confirms the DCGM term is still present and the custom threshold applied, without requiring the old contiguous form. That, plus the GPUMetricsMissing absent() fix, should get you green.