feat(dashboards): SGLang queue-wait, stage latency, token mix and HiCache L3 panels - #1370
Open
Tanguille wants to merge 5 commits into
Open
feat(dashboards): SGLang queue-wait, stage latency, token mix and HiCache L3 panels#1370Tanguille wants to merge 5 commits into
Tanguille wants to merge 5 commits into
Conversation
…eries color Three conventions made consistent across every dashboard, no panels or features added: - percentunit ratio-against-a-hard-limit panels get min:0/max:1 so they read against the limit instead of autoscaling (llmkube-quota GPU/VRAM utilization). - sum()/count() queries that can return an empty vector on a fresh cluster get noValue:"0" instead of "No data" (model-router request and fail-closed rates; sglang/vllm throughput, queue depth, and pool panels). - Panels whose series vary with a template variable (service/slo) use color.mode palette-classic-by-name so filtering doesn't repaint survivors (every timeseries panel in sglang-dashboard, vllm-dashboard, and the llmkube-slo over-time/burn-rate panels). amd-gpu-observability and llmkube-inference have no qualifying panels and are untouched. model-router's own $router variable isn't wired into any query, so its series aren't template-driven and its colors are left alone. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
…Conventions
Walks fieldConfig.defaults per panel (dashboardPanels), not the flat
expr list dashboardQueries returns, so a rule can see a panel's unit,
color and target queries together. Table-driven over three rules,
each scoped to timeseries panels (stat/gauge color by threshold,
tables have no per-series color or axis, so none of the three read on
them the way they do on a graph):
- percentunit + a bare metric/metric division -> min:0/max:1. Error
budgets ("1 - (...)") and burn-rate reference lines
("14 * (1 - $objective/100)") are excluded structurally: more than
one '/' or any '*'/'+' in the expr disqualifies it.
- sum()/count() as the outer operator -> noValue:"0". A ratio-of-sums
like sum(a)/sum(b) starts with "sum(" too, but its outer op is the
division; the sum() match must close at the very end of the trimmed
expr to count, which excludes it without an explicit list.
- a template variable inside a {...} label matcher ->
color.mode:"palette-classic-by-name". model-router's $router isn't
referenced in any query, so it correctly stays unmatched.
Scoped to charts/llmkube/dashboards only: config/grafana's standalone
pair predates the convention pass and was never touched by it.
Passes clean on the current tree, and a targeted sanity check
(stripping min/max from one panel) confirmed it fails when a
convention is actually violated.
Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
…HiCache L3 panels Adds the panels from defilantech#1360 that were missing from the chart's SGLang board: queue wait p50/p90/p99, per-stage latency by stage, new token ratio, prompt length distribution, request rate, and a collapsed HiCache L3 row (backup/load-back rate, cached/evicted rate). Also applies the dataviz audit fixes for this file: color.mode palette-classic-by-name everywhere so color follows the service entity instead of series order, ordered-quantile hue shading on the new queue-wait panel, table legend calcs set to [lastNotNull, max], and fillOpacity on the aborted-requests bar panel dropped out of the saturated-fill range. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
…anel Panel 23 (token throughput by mode) was the only panel missing the $service filter. Add the matcher, group by service/namespace, switch to [5m] like its siblings, and set palette-classic-by-name + noValue. Also move the universally-applicable request rate / queue wait / per-stage latency row above the collapsed conditional rows (Mamba pool, HiCache) so core panels aren't buried under optional ones. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Queue wait p50/p90/p99 ran three histogram_quantile calls repeating the same inner aggregation on every refresh, unlike the TTFT and end-to-end panels beside it which read precomputed llmkube:inference:* records. Add matching p50/p95 recording rules (dropping p90/p99 to match the existing pair) and point the panel at them. Also add a description to the token-throughput-by-mode panel clarifying it splits the same prefill total by cache hit vs recompute rather than duplicating the generation-vs-prompt panel, and set noValue:"0" on four sole-aggregation panels the rebase onto refactor/dashboard-conventions now checks. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
This was referenced Jul 30, 2026
Tanguille
marked this pull request as ready for review
July 30, 2026 21:36
Defilan
approved these changes
Jul 31, 2026
Defilan
left a comment
Defilan
left a comment
Member
There was a problem hiding this comment.
Tanguille, the metric hygiene here is excellent, thank you. I pulled the current SGLang collector source and confirmed all ten metrics your panels and recording rules reference exist with the right types and labels: queue_time_seconds / per_stage_req_latency_seconds / prompt_tokens_histogram are histograms (correctly using _bucket), new_token_ratio is a gauge (correctly avg), realtime_tokens_total carries mode and per_stage carries stage. Nothing renders empty. Approving on the code.
Two small asks before I merge: please add the Assisted-by: disclosure line to the PR body per CONTRIBUTING, and heads-up your rule-count assertion (now 7) will collide with #1372 on prometheusrule_test.yaml, whoever lands second reconciles it. Optional nit: new_token_ratio's max:1 may clip, since it's a scheduler estimate rather than a strict 0-1 ratio.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1360. Stacked on #1368 — this branch contains that PR commits; the layer own change is 3 files, +281/-9.
What
Queue-wait quantiles are backed by new
llmkube:inference:queue_time_seconds:{p50,p95}_5mrecording rules that mirror the TTFT and e2e siblings exactly, rather than three inlinehistogram_quantilecalls recomputing one aggregation per refresh.Live validation
Production cluster, SGLang v0.5.16 on an AMD R9700 under real traffic. The recording rules materialise and carry signal no existing panel showed:
llmkube:inference:queue_time_seconds:p95_5mllmkube:inference:queue_time_seconds:p50_5mA 1 s median with an 18 s tail is queueing, not slow inference. Every other new panel returns live data: request rate 1 series, per-stage latency 3 (one per stage), new token ratio 1, prompt-length heatmap 36 buckets, L3 backup/load-back 2, L3 cached/evicted 2.
Token throughput by mode returns 3 series (
decode,prefill_cache,prefill_compute) and is not a duplicate of the generation-vs-prompt panel: it is the same prefill total split by cache hit versus recompute. Verified on the same cluster,prefill_cache + prefill_computeequalsprompt_tokens_total(45,898,768 + 5,503,461 vs 51,385,859 sampled moments apart).The L3 row is collapsed by default, matching the mamba and HiCache host rows, so a non-HiCache install does not open onto empty panels.
Merge-order note
This branch and the GPU-alerts work both edit
charts/llmkube/tests/prometheusrule_test.yamland conflict; whichever lands second needs the rule-count assertion reconciled.Happy to be told otherwise on any of this: if you would rather these panels lived elsewhere, or not at all, say so and I will revert it.