feat(dashboards): add a llama.cpp runtime dashboard - #1371
Open
Tanguille wants to merge 8 commits into
Open
Conversation
…eries color Three conventions made consistent across every dashboard, no panels or features added: - percentunit ratio-against-a-hard-limit panels get min:0/max:1 so they read against the limit instead of autoscaling (llmkube-quota GPU/VRAM utilization). - sum()/count() queries that can return an empty vector on a fresh cluster get noValue:"0" instead of "No data" (model-router request and fail-closed rates; sglang/vllm throughput, queue depth, and pool panels). - Panels whose series vary with a template variable (service/slo) use color.mode palette-classic-by-name so filtering doesn't repaint survivors (every timeseries panel in sglang-dashboard, vllm-dashboard, and the llmkube-slo over-time/burn-rate panels). amd-gpu-observability and llmkube-inference have no qualifying panels and are untouched. model-router's own $router variable isn't wired into any query, so its series aren't template-driven and its colors are left alone. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
…Conventions
Walks fieldConfig.defaults per panel (dashboardPanels), not the flat
expr list dashboardQueries returns, so a rule can see a panel's unit,
color and target queries together. Table-driven over three rules,
each scoped to timeseries panels (stat/gauge color by threshold,
tables have no per-series color or axis, so none of the three read on
them the way they do on a graph):
- percentunit + a bare metric/metric division -> min:0/max:1. Error
budgets ("1 - (...)") and burn-rate reference lines
("14 * (1 - $objective/100)") are excluded structurally: more than
one '/' or any '*'/'+' in the expr disqualifies it.
- sum()/count() as the outer operator -> noValue:"0". A ratio-of-sums
like sum(a)/sum(b) starts with "sum(" too, but its outer op is the
division; the sum() match must close at the very end of the trimmed
expr to count, which excludes it without an explicit list.
- a template variable inside a {...} label matcher ->
color.mode:"palette-classic-by-name". model-router's $router isn't
referenced in any query, so it correctly stays unmatched.
Scoped to charts/llmkube/dashboards only: config/grafana's standalone
pair predates the convention pass and was never touched by it.
Passes clean on the current tree, and a targeted sanity check
(stripping min/max from one panel) confirmed it fails when a
convention is actually violated.
Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Models the sglang board's schema version, row structure, DS_PROMETHEUS placeholder, and $service template variable. Covers only the metrics a live llama.cpp exposes: requests_processing/deferred, busy slots per decode, decode ops, prompt/generation throughput (both 5m rate and the runtime's own lifetime-avg gauges), prompt/decode duty cycle, and configured max context. No KV-cache panel since that series does not exist on this runtime. Registers llamacpp-dashboard.json in $runtimeDashboards so it defers to the GrafanaDashboard candidates ConfigMap in operator mode auto and applies directly in mode all. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Add the missing docs/grafana/README.md catalog row for llamacpp-dashboard.json, update charts/llmkube/README.md's runtime-only dashboard section to cover all three deferred boards, and rename the n_tokens_max panel and dashboard description from 'configured context length' to 'peak observed context length' since that series is a high-watermark of context actually used, not the --ctx-size setting. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
llama.cpp does expose a KV-cache series (docs/site/guides/metrics-driven -autoscaling.md already builds an HPA on it); the fixture just omitted it from internal/metrics/testdata/llamacpp-metrics.txt, which is the real reason the board skipped it. Add the metric to the fixture and a percentunit KV cache utilization panel (min 0, max 1) instead of the false claim. Also drop max:1 on the prompt/decode duty cycle panel: llama.cpp accumulates per-slot seconds, so rate() legitimately exceeds 1.0 with -np > 1 and the axis clipped exactly when the server was busiest. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Panels 1 (prompt vs generation tokens/sec) and 3 (processing vs deferred requests) are the direct analogues of sglang and vllm ids 1 and 6, which the base branch already gave noValue: 0. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
…n panels TestDashboardConventions flags any panel whose outermost operator is a single sum()/count() and that lacks noValue: the lifetime-average throughput and decode-rate panels read "No data" on a fresh cluster instead of 0. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
…xported Verified against ggml-org/llama.cpp:server-vulkan@31bca24 on a live cluster: /metrics exposes 11 llamacpp: series and kv_cache_usage_ratio is not among them. docs/site/guides/metrics-driven-autoscaling.md builds an HPA on that series, so the guide and the runtime disagree; the dashboard follows the runtime and the metrics fixture is restored. Also aligns the legend calcs with the [mean, max] used by the other boards. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
This was referenced Jul 30, 2026
Tanguille
marked this pull request as ready for review
July 30, 2026 21:37
Defilan
reviewed
Jul 31, 2026
Defilan
left a comment
Defilan
left a comment
Member
There was a problem hiding this comment.
Tanguille, the dashboard itself is in great shape, thank you. I verified all 11 llama.cpp metrics against the fixture and they match exactly, so no empty-panel risk, and the kv_cache_usage_ratio omission (with your live-verified 11-series enumeration and the self-correcting revert) is exactly right. Nice catch, too, on metrics-driven-autoscaling.md building an HPA on kv_cache_usage_ratio that this runtime doesn't expose, please do open that as its own issue so it isn't lost. One thing before I approve: add the Assisted-by: disclosure line to the PR body per CONTRIBUTING. I'll merge #1368 first, then this rebases down to the ~5-file change.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1359. Stacked on #1368 — this branch contains that PR commits; the layer own change is 5 files.
What
llamacpp-dashboard.json, registered in$runtimeDashboardsas"llamacpp-dashboard.json" "llamacpp"so auto mode publishes it only while a llamacpp InferenceService serves. That name is whatruntimeNameLabelreturns for an emptyspec.runtime(internal/controller/runtime.go:115), so the existing gate handles it with no controller change.Rows: load (in-flight vs deferred, busy slots per decode), throughput (recent 5m rate and lifetime average), decode activity, context.
Why
SGLang and vLLM each have a runtime board. llama.cpp, had none, and
llmkube-inference.jsonreadsvllm:*/sglang:*recording rules only, so a llamacpp-only cluster saw an almost empty board.requests_deferredis the saturation signal, so it sits on the first row.A doc bug this surfaced, worth its own issue
An earlier revision of this branch carried a KV-cache panel, added because
docs/site/guides/metrics-driven-autoscaling.mddocumentsllamacpp:kv_cache_usage_ratioand builds an HPA on it. Live testing againstghcr.io/ggml-org/llama.cpp:server-vulkan@31bca24shows the server exports 11llamacpp:series and that is not one of them:The panel and its metrics-fixture entry are removed. The autoscaling guide still needs fixing: as written, anyone following it gets an HPA on a series that never appears. I can open that separately.
Live validation
All 7 panels return live data across three llama.cpp InferenceServices (two embedders, one 2B chat model), 3 series each. Applied behind a suspended HelmRelease, reverted afterwards, cluster proved clean.
Happy to be told otherwise on any of this: if you disagree with the board's shape or the fixture change, say so and I will revert it.