feat(observability): AMD GPU tier — amdgpu-exporter DaemonSet, metric contract, dashboard reconcile

[joryirving]

What

AMD GPU observability slices for #700: the exporter DaemonSet + metric contract (#1185) and the dashboard reconcile for #1187 — consolidated onto the merged llmkube-amdgpu-exporter (#16 in llmkube-runtimes) as the canonical sysfs exporter.

Why

Fixes #1185; unblocks the remaining scrape-confirm halves of #1186/#1187.

The Jul-21 decomposition pinned a contract from the quick lab exporter (amdgpu_sclk_mhz, llmkubelab manifest). Since then the reviewed sysfs exporter merged into llmkube-runtimes with slightly different (richer) names — per-sensor labels, pci_slot as the stable key, native temp/power, pcie_replay_total. This PR makes that the single contract everywhere:

• config/monitoring/amdgpu-exporter.yaml — DaemonSet + headless Service mirroring dcgm-exporter.yaml (hostPort 9494, scrape annotations), with the exporter's hardened posture (nonroot 65534, read-only rootfs, caps dropped, /sys ro). Image pinned to the candidate digest — happy to swap to amdgpu-exporter-v0.1.0 in this PR if you cut the tag.
• prometheus-scrape-config.yaml — llmkube-amd-gpu static job, matching the doc's existing style.
• docs/observability/amd-gpu-metrics.md — the pinned contract [FEATURE] AMD observability slice 1/3: rocm-smi exporter DaemonSet + ServiceMonitor (owns metric contract) #1185 owns: full metric table, pci_slot-is-the-stable-key, the GTT note (model weights live there on APUs), node_hwmon_* cross-check for the health row, llamacpp:* inference signals (slice 2), and deliberate gaps (mclk via pp_dpm_* only, no per-process attribution).
• amd-gpu-observability.json — one expr fix: amdgpu_sclk_mhz → amdgpu_clock_hertz{sensor="sclk"} / 1e6 (unit stays MHz). busy/vram/gtt exprs and the node_hwmon health panels already matched the merged exporter, so they're untouched.

Checklist

• Tests added/updated — n/a (manifests/docs/dashboard); kustomize build config/monitoring validates, dashboard JSON validated
• make lint — no Go changes
• Conventional commits, DCO signed
• AI assistance disclosed — authored with Claude Code
• Documentation updated — the contract doc is the deliverable

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Defilan]

Jory, this is really clean, thank you. You took the naming feedback from the exporter PR and reconciled the whole contract to amdgpu_* (the rename, the dashboard exprs, and the docs), the DaemonSet mirrors dcgm-exporter with the hardened posture, the docs are accurate down to the mclk gap, and CI is fully green. Exactly the follow-through I was hoping for.

Two things before I merge:

1. nodeSelector won't match my cluster. The DaemonSet targets llmkube.dev/gpu-vendor: amd, but the actual AMD node here (shadowstrix, gfx1151) is labeled accelerator: amd — so as-shipped it schedules zero pods and the panels stay empty. Please switch the selector to accelerator: amd (the label that's already on the node) so it works out of the box.

2. Image is a candidate build. You already flagged this. I'll cut amdgpu-exporter-v0.1.0 on llmkube-runtimes so there's a released, attested digest, then let's swap the pin to that in this PR.

Fix the selector and I'll get the release tag cut, then we merge. Thanks again — great slice.

[joryirving]

Selector fixed — accelerator: amd (the label shadowstrix actually carries), pushed. Good catch; an exporter that schedules zero pods is the quietest possible failure.

Standing by on the image pin: the moment amdgpu-exporter-v0.1.0 publishes I'll swap the candidate digest for the released one in this PR.

[Defilan]

Selector fix confirmed, thanks — accelerator: amd is the label the gfx1151 node actually carries, so the DaemonSet will schedule.

I've cut the release on my side: amdgpu-exporter-v0.1.0 is tagged on llmkube-runtimes and the build published successfully (test + image + build all green, multi-arch amd64 + arm64, with SBOM and keyless provenance attestation).

The released digest to pin:

image: ghcr.io/defilantech/llmkube-amdgpu-exporter:0.1.0@sha256:150fc4ad15677fa43f747842404e23b23834f980b54151749048eb650bd5fcef

(or just the digest, if you'd rather not carry the tag alongside it — ghcr.io/defilantech/llmkube-amdgpu-exporter@sha256:150fc4ad.... Your call; I like the tag+digest form because the tag documents intent while the digest pins immutably.)

Swap the candidate pin for that and I'll merge. Nice work on this one — it closes out #700.