feat(dashboards): add an operator-health row to llmkube-inference - #1369
feat(dashboards): add an operator-health row to llmkube-inference#1369Tanguille wants to merge 5 commits into
Conversation
…eries color Three conventions made consistent across every dashboard, no panels or features added: - percentunit ratio-against-a-hard-limit panels get min:0/max:1 so they read against the limit instead of autoscaling (llmkube-quota GPU/VRAM utilization). - sum()/count() queries that can return an empty vector on a fresh cluster get noValue:"0" instead of "No data" (model-router request and fail-closed rates; sglang/vllm throughput, queue depth, and pool panels). - Panels whose series vary with a template variable (service/slo) use color.mode palette-classic-by-name so filtering doesn't repaint survivors (every timeseries panel in sglang-dashboard, vllm-dashboard, and the llmkube-slo over-time/burn-rate panels). amd-gpu-observability and llmkube-inference have no qualifying panels and are untouched. model-router's own $router variable isn't wired into any query, so its series aren't template-driven and its colors are left alone. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
…Conventions
Walks fieldConfig.defaults per panel (dashboardPanels), not the flat
expr list dashboardQueries returns, so a rule can see a panel's unit,
color and target queries together. Table-driven over three rules,
each scoped to timeseries panels (stat/gauge color by threshold,
tables have no per-series color or axis, so none of the three read on
them the way they do on a graph):
- percentunit + a bare metric/metric division -> min:0/max:1. Error
budgets ("1 - (...)") and burn-rate reference lines
("14 * (1 - $objective/100)") are excluded structurally: more than
one '/' or any '*'/'+' in the expr disqualifies it.
- sum()/count() as the outer operator -> noValue:"0". A ratio-of-sums
like sum(a)/sum(b) starts with "sum(" too, but its outer op is the
division; the sum() match must close at the very end of the trimmed
expr to count, which excludes it without an explicit list.
- a template variable inside a {...} label matcher ->
color.mode:"palette-classic-by-name". model-router's $router isn't
referenced in any query, so it correctly stays unmatched.
Scoped to charts/llmkube/dashboards only: config/grafana's standalone
pair predates the convention pass and was never touched by it.
Passes clean on the current tree, and a targeted sanity check
(stripping min/max from one panel) confirmed it fails when a
convention is actually violated.
Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Reconcile rate/duration, time-to-ready p95, fleet ready-vs-total, replica state, and GPU queue depth. Fixed-color overrides for success/error and ready/desired series, palette-classic-by-name for per-entity series, lastNotNull+max table legend calcs, no dual-axis panels (defilantech#1362). Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Normalize the six new panels to the file's own precedent: legend calcs to mean/max (matching the two pre-existing panels), and color back to palette-classic since llmkube-inference.json has no template variable wired into any query (by-name is for panels whose series repaint under a $service/$slo-style filter, per 6579758). Add noValue:"0" to "Services Ready vs total" so a fresh cluster reads 0 instead of "No data". Document that this row supersedes the equivalent panels in config/grafana/llmkube-inference-dashboard.json (ids 52, 51, 32, 3, 43, 21), a pre-chart standalone board that isn't kept in sync. Update both the dashboard's own description and its docs/grafana/README.md row, which still claimed latency-and-restarts-only coverage. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Panels 5 (reconcile rate by result) and 9 (fleet replicas ready vs desired) export no series until first observed, same as panel 4 on this board. Add noValue: 0 so they don't render No data on a fresh cluster. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Defilan
left a comment
Defilan
left a comment
There was a problem hiding this comment.
Tanguille, the operator-health row is a genuinely useful gap to close, and the four reconcile/ready/phase panels are wired correctly to real controller metrics, thank you. Two things block merge:
-
Fleet replicas: ready vs desired (
llmkube-inference.json:247) queriesllmkube_inferenceservice_replicas, which no collector, recording rule, or exporter in this repo emits (and there's nostatelabel anywhere). WithnoValue: "0"it renders a plausible-but-fake0forever rather than "No data", so it reads as "0 replicas ready" instead of "wired to nothing". Our ownTestDashboardMetricsAreEmittedguard should flag this. Please point it at a real series (e.g. a kube-state-metricskube_deployment_status_replicas/_availablejoin) or drop the panel until the operator emits replica counts. -
GPU queue depth by namespace (
llmkube-inference.json:275) readsllmkube_gpu_queue_depth, which is a label-less global gauge (Set to a single queue position instatus_builder.go:184). There's nonamespacelabel, so{{namespace}}renders empty and the "by namespace" title promises a breakdown that can't exist. Retitle to "GPU queue depth" and drop the{{namespace}}legend, or add anamespacelabel to the collector first.
Also: could you reconcile the "Live validation: fleet replicas 2" note? That panel's metric doesn't exist, so that number can't have come from this expr. And if any of this was AI-assisted, please add the Assisted-by: line to the PR body per CONTRIBUTING (DCO is clean). Happy to help wire the replica panel to a real series if that was the intent.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
2 participants
Fixes #1362. Stacked on #1368 — this branch contains that PR commits, so review the top commits only; the layer own change is 2 files, +191/-2.
What
An
Operatorrow onllmkube-inference.json: reconcile rate by result, reconcile p95, time-to-Ready p95, Services Ready (stat), fleet replicas ready vs desired, GPU queue depth.Why
The chart already scrapes the controller and alerts on it (
ControllerDown), but no board showed it, so an operator that is up and reconciling badly surfaced nowhere. All six series exist whenever the chart is installed, so the row is unconditional and the board correctly stays out of$runtimeDashboards: these are controller-scoped metrics, not an InferenceService fact.Overlap with config/grafana
Six panels overlap
config/grafana/llmkube-inference-dashboard.json(ids 52, 51, 32, 3, 43, 21), verified id-for-id. The chart board supersedes them, noted indocs/grafana/README.mdrather than left for a reviewer to find. Happy to instead narrow the row if you would rather keep that board canonical.Live validation
Applied to a production cluster behind a suspended HelmRelease, then reverted and proved clean.
Panel data: reconcile rate 2 series (by result), Services Ready 1+1, fleet replicas 2, GPU queue depth 1.
Reconcile duration p95andTime to Ready p95return empty on an idle operator, which is correct: they arehistogram_quantileover a 5m rate and populate during a rollout.