From ca50b218269082ee4fe141098e61b4e5f215e99e Mon Sep 17 00:00:00 2001 From: Dmitry Kondratyev Date: Tue, 8 Sep 2026 09:29:37 -0400 Subject: [PATCH 1/5] Replace the scaling metric with an occupancy ratio (in-flight / served) The default KEDA/Envoy load metric was mean inference queue time per batch execution. It is non-linear in load (near zero once capacity is adequate, explosive near saturation) and its scale depends on the model mixture, so a single threshold cannot serve different deployments: in A/B runs on Geddes it caused a premature mid-run scale-down under high load and six minutes of under-scaling at moderate load. The new default metric estimates the number of replicas the current in-flight work needs, via Little's law (L = rate of a cumulative time counter): R_needed = L_envoy / max(L_service / R_healthy, 1) L_envoy in-flight requests between Envoy and Triton (rate of envoy_cluster_upstream_rq_time_sum, ms -> /1e3) L_service requests actively executing across all models and pods (rate of request-duration minus queue-duration, us -> /1e6) R_healthy Triton endpoints Envoy routes to (max of envoy_cluster_membership_healthy across Envoy pods) Every term is a time integral, so models are weighted by the time they consume and the metric carries no model-specific constants; it reads ~1 with no queueing and grows linearly with the overload factor. KEDA now consumes it as metricType AverageValue (desired = ceil(metric/threshold), independent of pods that never scheduled); the Envoy rate limiter compares the per-replica form against a separate serverAdmissionThreshold so that new clients are only rejected when scaling can no longer keep up, not at the normal operating point. serverLoadThreshold now defaults to 2 (tolerated sojourn inflation); the rate() window is configurable via serverLoadRateInterval (keep >= 4x the Prometheus scrape interval). A custom serverLoadMetric is still used verbatim by both consumers, with keda.metricType available to restore the old Value semantics. Co-Authored-By: Claude Fable 5 --- docs/configuration-guide.rst | 99 ++++++++++++++--- helm/supersonic/templates/NOTES.txt | 4 +- .../templates/_helpers/_scaling-metric.tpl | 100 ++++++++++++++++-- .../templates/envoy/configmaps.yaml | 4 +- helm/supersonic/templates/keda/so.yaml | 4 +- helm/supersonic/values.schema.json | 12 +++ helm/supersonic/values.yaml | 30 +++++- values/values-geddes-cms.yaml | 2 - values/values-nautilus-atlas.yaml | 2 - values/values-nautilus-cms.yaml | 2 - values/values-nautilus-icecube.yaml | 2 - 11 files changed, 220 insertions(+), 41 deletions(-) diff --git a/docs/configuration-guide.rst b/docs/configuration-guide.rst index 8fe2d7fe..d11f5154 100644 --- a/docs/configuration-guide.rst +++ b/docs/configuration-guide.rst @@ -282,20 +282,91 @@ Prometheus is needed to scrape metrics for monitoring, as well as for the rate l 8. (Optional) Configure Metrics for Scaling and Rate Limiting =============================================================== -Both the rate limiter and the autoscaler are currently configured to use the same Prometheus metric and threshold. -They are defined in the ``serverLoadMetric`` and ``serverLoadThreshold`` parameters at the root level of the values file. -The default metric is the inference queue time at the Triton servers, as defined in -`here `_. - -When the metric value exceeds the threshold, the following happens: - -- Autoscaler scales up the number of Triton servers if possible. -- Envoy proxy rejects new ``RepositoryIndex`` requests. - -The pre-configured Grafana dashboard contains a graph of this metric, entitled "Server Load Metric". -The Prometheus query for the graph is automatically inferred from the value of ``serverLoadMetric`` parameter. -The graph also displays the threshold value defined in ``serverLoadThreshold`` parameter. - +The autoscaler and the Prometheus-based rate limiter are driven by one Prometheus +query, defined by the ``serverLoadMetric`` parameter at the root of the values file +(rendered in ``templates/_helpers/_scaling-metric.tpl``). + +The default metric: occupancy ratio +------------------------------------ + +By default, SuperSONIC estimates **how many Triton replicas the current in-flight +work needs**: + +.. math:: + + R_{needed} = \frac{L_{envoy}}{\max(L_{service} / R_{healthy},\ 1)} + +All three inputs are measured, with no model-specific constants: + +- :math:`L_{envoy}` — mean number of requests in flight between Envoy and the + Triton fleet (queued, executing, or on the wire). By Little's law + (:math:`L = \lambda W`), this equals the per-second rate of Envoy's cumulative + request-time counter: + ``sum(rate(envoy_cluster_upstream_rq_time_sum{...}[1m])) / 1e3`` (the counter is + in milliseconds). +- :math:`L_{service}` — mean number of requests being actively executed across all + models and pods, from Triton's cumulative counters: + ``sum(rate(nv_inference_request_duration_us - nv_inference_queue_duration_us)) / 1e6``. + Request duration minus queue duration covers every phase a replica spends working + on a request (input copy, inference, output copy, overhead), so models are + weighted by the time they consume rather than by request counts. +- :math:`R_{healthy}` — Triton endpoints Envoy currently routes to: + ``max(envoy_cluster_membership_healthy{...})`` (``max`` across Envoy pods, which + all report the same upstream cluster). + +In PromQL the division is spelled with ``clamp_min``, which is simply +:math:`\max(v, s)`: + +.. code-block:: text + + L_envoy * clamp_min(R_healthy, 1) / clamp_min(clamp_min(L_service, R_healthy), 1) + +The two floors encode physical facts rather than tuning: + +- ``clamp_min(L_service, R_healthy)`` — each healthy replica can execute at least + one request concurrently, so the fleet's serving capacity is never below its + replica count. This is what makes scale-down work at low load: an underutilized + fleet reads *below* 1 per replica instead of being stuck at the network floor. +- the outer ``clamp_min(..., 1)`` — at zero replicas (scale-from-zero) the metric + degrades to "requests in flight" instead of dividing by zero, and never yields + ``+Inf`` (which KEDA would treat as "scale to maximum"). + +Interpretation: divided by :math:`R_{healthy}`, the metric is the *sojourn-time +inflation* clients experience — 1.0 means every in-flight request is being +executed; 2.0 means requests spend as long waiting as being served. Because +in-flight work grows with offered load while serving capacity is bounded, the +metric is **linear in the overload factor**, which is exactly what the HPA's +proportional formula assumes; and it is invariant under the model mixture, since +every term is a time integral. + +Thresholds and how they are consumed +------------------------------------- + +- ``serverLoadThreshold`` (default ``2``) — KEDA consumes the metric with + ``metricType: AverageValue``: desired replicas = ``ceil(metric / threshold)``. + The threshold is the tolerated inflation: ``2`` targets "waiting ≈ serving"; + ``1.5`` buys lower latency at higher GPU cost. The unloaded floor is ~1.2–1.3 + (network transit), so values at or below ~1.3 over-provision. +- ``serverAdmissionThreshold`` (default ``3``) — the Envoy rate limiter compares + the *per-replica* form of the metric against this value and rejects new + ``RepositoryIndex`` requests above it. It is deliberately higher than + ``serverLoadThreshold``: the autoscaler settles the system near its threshold, + so gating admission at the same value would reject new clients during normal + operation. +- ``serverLoadRateInterval`` (default ``1m``) — the ``rate()`` window. Keep it at + or above 4× your Prometheus scrape interval; shorter windows add noise without + detecting load faster (the control loop is dominated by HPA sync and pod + startup), longer windows add lag. + +Custom metrics +--------------- + +If ``serverLoadMetric`` is set, it is used **verbatim** by both KEDA and the rate +limiter. KEDA compares it against ``serverLoadThreshold`` and the rate limiter +against ``serverAdmissionThreshold`` — set them to the same value if you want the +two consumers coupled. Also set ``keda.metricType`` to match your metric's +semantics (``Value`` for per-replica quantities, ``AverageValue`` for fleet-wide +ones). 9. (Optional) Deploy Grafana Dashboard ========================================== diff --git a/helm/supersonic/templates/NOTES.txt b/helm/supersonic/templates/NOTES.txt index dfa0b48d..d4495658 100644 --- a/helm/supersonic/templates/NOTES.txt +++ b/helm/supersonic/templates/NOTES.txt @@ -24,7 +24,9 @@ SuperSONIC chart successfully installed! Scaling metric:{{ if not ( eq .Values.serverLoadMetric "" ) }} {{ .Values.serverLoadMetric }}{{ else }}{{ include "supersonic.defaultMetric" . | nindent 4 }}{{ end }} -Scaling threshold: {{ include "supersonic.defaultThreshold" . }} +Scaling threshold (KEDA): {{ include "supersonic.defaultThreshold" . }} + +Admission threshold (Envoy rate limiter): {{ include "supersonic.admissionThreshold" . }} ┌-----------------------------------------------------------------------------┐ | Documentation: https://fastmachinelearning.org/SuperSONIC diff --git a/helm/supersonic/templates/_helpers/_scaling-metric.tpl b/helm/supersonic/templates/_helpers/_scaling-metric.tpl index a6d3e7bd..2c304168 100644 --- a/helm/supersonic/templates/_helpers/_scaling-metric.tpl +++ b/helm/supersonic/templates/_helpers/_scaling-metric.tpl @@ -1,23 +1,103 @@ {{/* -Get default scaling metric +Scaling and admission metrics. + +The default scaling metric is an "occupancy ratio" derived from Little's law +(L = lambda * W: mean occupancy equals the per-second rate of a cumulative +time counter). It is built from three measured quantities: + + L_envoy - mean number of requests in flight between Envoy and the Triton + fleet (queued + executing + on the wire), from Envoy's + cumulative upstream request-time counter (milliseconds). + L_service - mean number of requests being actively executed across all + models and pods, from Triton's cumulative request-duration + minus queue-duration counters (microseconds). + R_healthy - number of Triton endpoints Envoy currently routes to + (max across Envoy pods - each pod reports the same + upstream cluster membership). + +"supersonic.defaultMetric" renders the extensive form + + R_needed = L_envoy / max(L_service / R_healthy, 1) + +("how many replicas the current in-flight work needs"), spelled in PromQL as +L_envoy * R_healthy / max(L_service, R_healthy, 1) because clamp_min(v, s) +is PromQL for max(v, s). The floors encode two physical facts: each healthy +replica can serve at least one request concurrently (so serving capacity is +never below R_healthy), and at zero replicas the metric degrades to "requests +in flight" instead of dividing by zero. KEDA consumes this form with +metricType AverageValue: desired = ceil(R_needed / serverLoadThreshold). + +"supersonic.admissionMetric" is the same quantity per healthy replica +(R_needed / R_healthy, i.e. the sojourn-time inflation clients experience); +the Envoy Lua filter rejects RepositoryIndex requests when it exceeds +"supersonic.admissionThreshold". + +If .Values.serverLoadMetric is set, both helpers return it verbatim. +*/}} + +{{/* +Range-vector window for rate() in the default metric. +Keep it at >= 4x the Prometheus scrape interval. +*/}} +{{- define "supersonic.rateInterval" -}} +{{- default "1m" .Values.serverLoadRateInterval -}} +{{- end -}} + +{{/* +Healthy Triton endpoints as seen by Envoy. +*/}} +{{- define "supersonic.healthyReplicasExpr" -}} +max(envoy_cluster_membership_healthy{release=~"{{ include "supersonic.name" . }}", envoy_cluster_name="triton_grpc_service"}) +{{- end -}} + +{{/* +Get default scaling metric (extensive form: replicas needed) */}} {{- define "supersonic.defaultMetric" -}} {{- if not ( eq .Values.serverLoadMetric "" ) }} {{- printf "%s" .Values.serverLoadMetric -}} {{- else }} -sum by (release) ( - rate(nv_inference_queue_duration_us{release=~"{{ include "supersonic.name" . }}"}[30s]) -) - / -sum by (release) ( - (rate(nv_inference_exec_count{release=~"{{ include "supersonic.name" . }}"}[30s]) * 1000) + 0.001 +{{- $w := include "supersonic.rateInterval" . }} +sum(rate(envoy_cluster_upstream_rq_time_sum{release=~"{{ include "supersonic.name" . }}", envoy_cluster_name="triton_grpc_service"}[{{ $w }}])) / 1e3 +* scalar(clamp_min({{ include "supersonic.healthyReplicasExpr" . }}, 1)) +/ clamp_min( + clamp_min( + ( + sum(rate(nv_inference_request_duration_us{release=~"{{ include "supersonic.name" . }}"}[{{ $w }}])) + - + sum(rate(nv_inference_queue_duration_us{release=~"{{ include "supersonic.name" . }}"}[{{ $w }}])) + ) / 1e6, + scalar({{ include "supersonic.healthyReplicasExpr" . }}) + ), + 1 + ) +{{- end }} +{{- end }} + +{{/* +Get admission metric (intensive form: load per healthy replica) +*/}} +{{- define "supersonic.admissionMetric" -}} +{{- if not ( eq .Values.serverLoadMetric "" ) }} + {{- printf "%s" .Values.serverLoadMetric -}} +{{- else }} +( +{{- include "supersonic.defaultMetric" . }} ) +/ scalar(clamp_min({{ include "supersonic.healthyReplicasExpr" . }}, 1)) {{- end }} {{- end }} {{/* -Get server load threshold (defaults to 100 if not set) +Get scaling threshold (defaults to 2 if not set) */}} {{- define "supersonic.defaultThreshold" -}} -{{- default 100 .Values.serverLoadThreshold -}} -{{- end -}} \ No newline at end of file +{{- default 2 .Values.serverLoadThreshold -}} +{{- end -}} + +{{/* +Get admission threshold for the Envoy rate limiter (defaults to 3 if not set) +*/}} +{{- define "supersonic.admissionThreshold" -}} +{{- default 3 .Values.serverAdmissionThreshold -}} +{{- end -}} diff --git a/helm/supersonic/templates/envoy/configmaps.yaml b/helm/supersonic/templates/envoy/configmaps.yaml index fa74e4fb..6dc9ef5f 100644 --- a/helm/supersonic/templates/envoy/configmaps.yaml +++ b/helm/supersonic/templates/envoy/configmaps.yaml @@ -317,8 +317,8 @@ data: envoy-filter.lua: |- {{- /* Read and process the Lua configuration file */}} {{- $luaConfig := $.Files.Get .Values.envoy.rate_limiter.prometheus_based.luaConfig | nindent 4 }} - {{- $luaConfig = $luaConfig | replace "SERVER_LOAD_METRIC" (include "supersonic.defaultMetric" . | quote) }} - {{- $luaConfig = $luaConfig | replace "SERVER_LOAD_THRESHOLD" (quote .Values.serverLoadThreshold) }} + {{- $luaConfig = $luaConfig | replace "SERVER_LOAD_METRIC" (include "supersonic.admissionMetric" . | quote) }} + {{- $luaConfig = $luaConfig | replace "SERVER_LOAD_THRESHOLD" (include "supersonic.admissionThreshold" . | quote) }} {{- $luaConfig = $luaConfig | replace "PROMETHEUS_SCHEME" (include "supersonic.prometheusScheme" .) }} {{- $luaConfig = $luaConfig | replace "PROMETHEUS_HOST" (include "supersonic.prometheusHost" .) }} {{- $luaConfig = $luaConfig | replace "PROMETHEUS_PORT" (include "supersonic.prometheusPort" .) }} diff --git a/helm/supersonic/templates/keda/so.yaml b/helm/supersonic/templates/keda/so.yaml index 9d412962..31c137d7 100644 --- a/helm/supersonic/templates/keda/so.yaml +++ b/helm/supersonic/templates/keda/so.yaml @@ -47,11 +47,11 @@ spec: triggers: - type: prometheus - metricType: Value + metricType: {{ .Values.keda.metricType | default "AverageValue" }} metadata: serverAddress: {{ include "supersonic.prometheusUrl" . }} metricName: autoscaler-metric - threshold: {{ .Values.serverLoadThreshold | quote }} + threshold: {{ include "supersonic.defaultThreshold" . | quote }} ignoreNullValues: "true" query: |- {{ include "supersonic.defaultMetric" . | nindent 8 }} diff --git a/helm/supersonic/values.schema.json b/helm/supersonic/values.schema.json index 3d64c1d7..dda8f5b0 100644 --- a/helm/supersonic/values.schema.json +++ b/helm/supersonic/values.schema.json @@ -11,6 +11,12 @@ "serverLoadThreshold": { "type": "integer" }, + "serverLoadRateInterval": { + "type": "string" + }, + "serverAdmissionThreshold": { + "type": "integer" + }, "scaleFromZero": { "type": "object", "properties": { @@ -512,6 +518,9 @@ "enabled": { "type": "boolean" }, + "metricType": { + "type": "string" + }, "minReplicaCount": { "type": "integer" }, @@ -570,6 +579,7 @@ "cooldownPeriod", "enabled", "maxReplicaCount", + "metricType", "minReplicaCount", "pollingInterval", "scaleDown", @@ -1350,7 +1360,9 @@ "nameOverride", "prometheus", "scaleFromZero", + "serverAdmissionThreshold", "serverLoadMetric", + "serverLoadRateInterval", "serverLoadThreshold", "triton" ] diff --git a/helm/supersonic/values.yaml b/helm/supersonic/values.yaml index 037b89ac..f7171fa8 100644 --- a/helm/supersonic/values.yaml +++ b/helm/supersonic/values.yaml @@ -3,12 +3,29 @@ # -- Unique identifier of SuperSONIC instance (equal to release name by default) nameOverride: "" -# -- A metric used by both KEDA autoscaler and Envoy's prometheus-based rate limiter. -## Default metric (inference queue latency) is defined in templates/_helpers.tpl +# -- Prometheus query used by the KEDA autoscaler and (per healthy replica) by Envoy's +# prometheus-based rate limiter. Leave empty to use the default "occupancy ratio" metric: +# the mean number of requests in flight between Envoy and Triton (Little's law applied to +# Envoy's cumulative upstream request time) divided by per-replica serving occupancy +# (rate of Triton request time minus queue time). It reads ~1 when nothing queues, grows +# linearly with overload, and contains no model-specific constants, so one threshold works +# for any model mixture. Defined in templates/_helpers/_scaling-metric.tpl and documented +# in the configuration guide. A custom query set here is used verbatim by both consumers. serverLoadMetric: "" -# -- Threshold for the metric -serverLoadThreshold: 100 +# -- Threshold for the scaling metric. With the default metric this is the tolerated +# sojourn inflation per replica: KEDA scales to ceil(metric / threshold), so 2 targets +# "requests spend about as long waiting as being served"; 1.5 trades GPUs for latency. +serverLoadThreshold: 2 + +# -- Range-vector window used by rate() in the default metric. +# Keep it at or above 4x your Prometheus scrape interval. +serverLoadRateInterval: "1m" + +# -- Threshold for Envoy's prometheus-based rate limiter, compared against the scaling +# metric per healthy replica. Kept above serverLoadThreshold so that new clients are +# rejected only when scaling can no longer keep up, not at the normal operating point. +serverAdmissionThreshold: 3 # -- On RepositoryIndex, scale Triton to at least max(1, keda.minReplicaCount) replicas # and return the index only after Envoy has a healthy Triton upstream. @@ -226,6 +243,11 @@ keda: # new Triton servers will spawn if the metric exceeds the threshold set by ``serverLoadThreshold``. enabled: false + # -- How the HPA interprets the scaling metric. AverageValue (the default) treats it + # as fleet-wide "replicas needed": desired = ceil(metric / threshold). Set to Value + # for a custom per-replica metric: desired = ceil(current_replicas * metric / threshold). + metricType: AverageValue + # -- Minimum and maximum number of Triton servers. # Set minReplicaCount to 0 to release all resources when idle (requires scaleFromZero.enabled). # With scaleFromZero, a RepositoryIndex request scales Triton to max(1, minReplicaCount), diff --git a/values/values-geddes-cms.yaml b/values/values-geddes-cms.yaml index f113b11e..34258897 100644 --- a/values/values-geddes-cms.yaml +++ b/values/values-geddes-cms.yaml @@ -5,8 +5,6 @@ # Based on Purdue AF production values: # https://github.com/PurdueAF/purdue-af/blob/main/apps/sonic/supersonic/values.yaml -serverLoadThreshold: 100 - scaleFromZero: enabled: true readyTimeoutSeconds: 300 diff --git a/values/values-nautilus-atlas.yaml b/values/values-nautilus-atlas.yaml index 146963eb..709dbdec 100644 --- a/values/values-nautilus-atlas.yaml +++ b/values/values-nautilus-atlas.yaml @@ -1,5 +1,3 @@ -serverLoadThreshold: 100 - triton: name: triton-atlas image: milescb/traccc-aas:v1.1 diff --git a/values/values-nautilus-cms.yaml b/values/values-nautilus-cms.yaml index db519037..bcbcc669 100644 --- a/values/values-nautilus-cms.yaml +++ b/values/values-nautilus-cms.yaml @@ -1,5 +1,3 @@ -serverLoadThreshold: 100 - triton: replicas: 5 # image: fastml/triton-torchgeo:21.02-py3-geometric # run2 diff --git a/values/values-nautilus-icecube.yaml b/values/values-nautilus-icecube.yaml index 2686f64b..d17449d3 100644 --- a/values/values-nautilus-icecube.yaml +++ b/values/values-nautilus-icecube.yaml @@ -1,5 +1,3 @@ -serverLoadThreshold: 100 - triton: image: nvcr.io/nvidia/tritonserver:26.08-py3 affinity: From 0d0b902644edae82aaa562739f2ccdf60de33788 Mon Sep 17 00:00:00 2001 From: GitHub Actions Date: Tue, 8 Sep 2026 13:30:06 +0000 Subject: [PATCH 2/5] Update helm docs --- docs/.values-table.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/.values-table.md b/docs/.values-table.md index ad7c7e1e..6a9dcfde 100644 --- a/docs/.values-table.md +++ b/docs/.values-table.md @@ -3,8 +3,10 @@ | Key | Type | Default | Description | |-----|------|---------|-------------| | nameOverride | string | `""` | Unique identifier of SuperSONIC instance (equal to release name by default) | -| serverLoadMetric | string | `""` | A metric used by both KEDA autoscaler and Envoy's prometheus-based rate limiter. # Default metric (inference queue latency) is defined in templates/_helpers.tpl | -| serverLoadThreshold | int | `100` | Threshold for the metric | +| serverLoadMetric | string | `""` | Prometheus query used by the KEDA autoscaler and (per healthy replica) by Envoy's prometheus-based rate limiter. Leave empty to use the default "occupancy ratio" metric: the mean number of requests in flight between Envoy and Triton (Little's law applied to Envoy's cumulative upstream request time) divided by per-replica serving occupancy (rate of Triton request time minus queue time). It reads ~1 when nothing queues, grows linearly with overload, and contains no model-specific constants, so one threshold works for any model mixture. Defined in templates/_helpers/_scaling-metric.tpl and documented in the configuration guide. A custom query set here is used verbatim by both consumers. | +| serverLoadThreshold | int | `2` | Threshold for the scaling metric. With the default metric this is the tolerated sojourn inflation per replica: KEDA scales to ceil(metric / threshold), so 2 targets "requests spend about as long waiting as being served"; 1.5 trades GPUs for latency. | +| serverLoadRateInterval | string | `"1m"` | Range-vector window used by rate() in the default metric. Keep it at or above 4x your Prometheus scrape interval. | +| serverAdmissionThreshold | int | `3` | Threshold for Envoy's prometheus-based rate limiter, compared against the scaling metric per healthy replica. Kept above serverLoadThreshold so that new clients are rejected only when scaling can no longer keep up, not at the normal operating point. | | scaleFromZero | object | `{"admissionImage":"python:3.14-slim","enabled":false,"holdMinReplicasSeconds":300,"readyTimeoutSeconds":300}` | On RepositoryIndex, scale Triton to at least max(1, keda.minReplicaCount) replicas and return the index only after Envoy has a healthy Triton upstream. Requires keda.enabled and envoy.enabled. | | scaleFromZero.enabled | bool | `false` | Enable scale from zero | | scaleFromZero.readyTimeoutSeconds | int | `300` | Seconds to wait for a healthy Triton upstream before rejecting RepositoryIndex | @@ -60,6 +62,7 @@ | envoy.auth.url | string | `""` | | | envoy.auth.port | int | `443` | | | keda.enabled | bool | `false` | Enable autoscaling (requires Prometheus to also be enabled). Autoscaling will be based on the metric from parameter ``serverLoadMetric``; new Triton servers will spawn if the metric exceeds the threshold set by ``serverLoadThreshold``. | +| keda.metricType | string | `"AverageValue"` | How the HPA interprets the scaling metric. AverageValue (the default) treats it as fleet-wide "replicas needed": desired = ceil(metric / threshold). Set to Value for a custom per-replica metric: desired = ceil(current_replicas * metric / threshold). | | keda.minReplicaCount | int | `1` | Minimum and maximum number of Triton servers. Set minReplicaCount to 0 to release all resources when idle (requires scaleFromZero.enabled). With scaleFromZero, a RepositoryIndex request scales Triton to max(1, minReplicaCount), and upgrades keep the live ScaledObject minReplicaCount. | | keda.maxReplicaCount | int | `2` | | | keda.pollingInterval | int | `30` | How often KEDA polls Prometheus | From 444bdc334d9c9c83bde6c99b11b07abf6eab6491 Mon Sep 17 00:00:00 2001 From: Dmitry Kondratyev Date: Mon, 7 Sep 2026 10:21:08 -0400 Subject: [PATCH 3/5] Pin kube-prometheus-stack in CI and the README install steps The Prometheus Operator install floats on whatever prometheus-community/kube-prometheus-stack is latest. 90.0.0, published between 2026-09-05 and 2026-09-07, made the control-plane ServiceMonitors authenticate via a Secret that is only rendered when prometheus.enabled and prometheus.serviceAccount.create are both true. These call sites pass prometheus.enabled=false -- they want the operator and its CRDs, nothing else -- so templating now fails: The control-plane ServiceMonitors authenticate by default with the Secret created by prometheus.serviceAccount.createTokenSecret, which is only rendered when prometheus.enabled and prometheus.serviceAccount.create are also true. Bisected: 89.2.4 and every earlier release template fine with these flags; 90.0.0 is the first that does not. Pinning 89.2.4 keeps CI reproducible and matches how the chart's own dependencies are pinned. 89.2.4 still ships the servicemonitors CRD and the operator Deployment, which is all these steps need. The alternative -- tracking latest and disabling every control-plane exporter (kubelet, kubeApiServer, kubeControllerManager, kubeScheduler, kubeProxy, kubeEtcd, coreDns) -- also works on 90.0.0 but adds seven flags to each call site and would not protect against the next upstream change. This break is independent of this branch: it fails identically on main, which has not run CI since 90.0.0 was published. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci-external-config.yaml | 2 +- .github/workflows/ci-full.yaml | 2 +- .github/workflows/ci-local.sh | 1 + README.md | 2 +- helm/supersonic/README.md | 2 +- 5 files changed, 5 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci-external-config.yaml b/.github/workflows/ci-external-config.yaml index a0072e49..7fcf9af4 100644 --- a/.github/workflows/ci-external-config.yaml +++ b/.github/workflows/ci-external-config.yaml @@ -34,7 +34,7 @@ jobs: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update kubectl create namespace monitoring - helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false + helm install prometheus-operator prometheus-community/kube-prometheus-stack --version 89.2.4 --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false - name: Install KEDA Autoscaler run: | diff --git a/.github/workflows/ci-full.yaml b/.github/workflows/ci-full.yaml index 9416fc7d..6811db2a 100644 --- a/.github/workflows/ci-full.yaml +++ b/.github/workflows/ci-full.yaml @@ -34,7 +34,7 @@ jobs: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update kubectl create namespace monitoring - helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false + helm install prometheus-operator prometheus-community/kube-prometheus-stack --version 89.2.4 --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false - name: Install KEDA Autoscaler run: | diff --git a/.github/workflows/ci-local.sh b/.github/workflows/ci-local.sh index 7f255758..6bc72588 100644 --- a/.github/workflows/ci-local.sh +++ b/.github/workflows/ci-local.sh @@ -53,6 +53,7 @@ helm repo add prometheus-community https://prometheus-community.github.io/helm-c helm repo update kubectl create namespace monitoring helm install prometheus-operator prometheus-community/kube-prometheus-stack \ + --version 89.2.4 \ --namespace monitoring \ --set prometheusOperator.createCustomResource=false \ --set defaultRules.create=false \ diff --git a/README.md b/README.md index 45a9fe60..6d77631c 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ Currently, SuperSONIC supports the following functionality: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update kubectl create namespace monitoring - helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false + helm install prometheus-operator prometheus-community/kube-prometheus-stack --version 89.2.4 --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false ``` - [KEDA](https://keda.sh) CRDs (only if using autoscaling) diff --git a/helm/supersonic/README.md b/helm/supersonic/README.md index 45a9fe60..6d77631c 100644 --- a/helm/supersonic/README.md +++ b/helm/supersonic/README.md @@ -55,7 +55,7 @@ Currently, SuperSONIC supports the following functionality: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update kubectl create namespace monitoring - helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false + helm install prometheus-operator prometheus-community/kube-prometheus-stack --version 89.2.4 --namespace monitoring --set prometheusOperator.createCustomResource=false --set defaultRules.create=false --set alertmanager.enabled=false --set prometheus.enabled=false --set grafana.enabled=false ``` - [KEDA](https://keda.sh) CRDs (only if using autoscaling) From 087bb65a34102f918aa8a4b64fd341bcd4ff9fbe Mon Sep 17 00:00:00 2001 From: Dmitry Kondratyev Date: Tue, 8 Sep 2026 16:53:36 -0400 Subject: [PATCH 4/5] Tighten the scaling-metric explanations Trim the helper header, values comments, and configuration-guide section to what a reader needs: what each term measures, what the floors do, and how the thresholds are consumed. Drop the derivation asides and the repeated interpretation. Co-Authored-By: Claude Fable 5 --- docs/configuration-guide.rst | 123 +++++++----------- .../templates/_helpers/_scaling-metric.tpl | 42 +++--- helm/supersonic/values.yaml | 23 ++-- 3 files changed, 72 insertions(+), 116 deletions(-) diff --git a/docs/configuration-guide.rst b/docs/configuration-guide.rst index d11f5154..a92128a9 100644 --- a/docs/configuration-guide.rst +++ b/docs/configuration-guide.rst @@ -283,90 +283,59 @@ Prometheus is needed to scrape metrics for monitoring, as well as for the rate l =============================================================== The autoscaler and the Prometheus-based rate limiter are driven by one Prometheus -query, defined by the ``serverLoadMetric`` parameter at the root of the values file -(rendered in ``templates/_helpers/_scaling-metric.tpl``). +query, set by ``serverLoadMetric`` at the root of the values file and rendered in +``templates/_helpers/_scaling-metric.tpl``. -The default metric: occupancy ratio ------------------------------------- +The default metric +------------------- By default, SuperSONIC estimates **how many Triton replicas the current in-flight -work needs**: - -.. math:: - - R_{needed} = \frac{L_{envoy}}{\max(L_{service} / R_{healthy},\ 1)} - -All three inputs are measured, with no model-specific constants: - -- :math:`L_{envoy}` — mean number of requests in flight between Envoy and the - Triton fleet (queued, executing, or on the wire). By Little's law - (:math:`L = \lambda W`), this equals the per-second rate of Envoy's cumulative - request-time counter: - ``sum(rate(envoy_cluster_upstream_rq_time_sum{...}[1m])) / 1e3`` (the counter is - in milliseconds). -- :math:`L_{service}` — mean number of requests being actively executed across all - models and pods, from Triton's cumulative counters: - ``sum(rate(nv_inference_request_duration_us - nv_inference_queue_duration_us)) / 1e6``. - Request duration minus queue duration covers every phase a replica spends working - on a request (input copy, inference, output copy, overhead), so models are - weighted by the time they consume rather than by request counts. -- :math:`R_{healthy}` — Triton endpoints Envoy currently routes to: - ``max(envoy_cluster_membership_healthy{...})`` (``max`` across Envoy pods, which - all report the same upstream cluster). - -In PromQL the division is spelled with ``clamp_min``, which is simply -:math:`\max(v, s)`: - -.. code-block:: text - - L_envoy * clamp_min(R_healthy, 1) / clamp_min(clamp_min(L_service, R_healthy), 1) - -The two floors encode physical facts rather than tuning: - -- ``clamp_min(L_service, R_healthy)`` — each healthy replica can execute at least - one request concurrently, so the fleet's serving capacity is never below its - replica count. This is what makes scale-down work at low load: an underutilized - fleet reads *below* 1 per replica instead of being stuck at the network floor. -- the outer ``clamp_min(..., 1)`` — at zero replicas (scale-from-zero) the metric - degrades to "requests in flight" instead of dividing by zero, and never yields - ``+Inf`` (which KEDA would treat as "scale to maximum"). - -Interpretation: divided by :math:`R_{healthy}`, the metric is the *sojourn-time -inflation* clients experience — 1.0 means every in-flight request is being -executed; 2.0 means requests spend as long waiting as being served. Because -in-flight work grows with offered load while serving capacity is bounded, the -metric is **linear in the overload factor**, which is exactly what the HPA's -proportional formula assumes; and it is invariant under the model mixture, since -every term is a time integral. - -Thresholds and how they are consumed -------------------------------------- - -- ``serverLoadThreshold`` (default ``2``) — KEDA consumes the metric with - ``metricType: AverageValue``: desired replicas = ``ceil(metric / threshold)``. - The threshold is the tolerated inflation: ``2`` targets "waiting ≈ serving"; - ``1.5`` buys lower latency at higher GPU cost. The unloaded floor is ~1.2–1.3 - (network transit), so values at or below ~1.3 over-provision. -- ``serverAdmissionThreshold`` (default ``3``) — the Envoy rate limiter compares - the *per-replica* form of the metric against this value and rejects new - ``RepositoryIndex`` requests above it. It is deliberately higher than - ``serverLoadThreshold``: the autoscaler settles the system near its threshold, - so gating admission at the same value would reject new clients during normal - operation. -- ``serverLoadRateInterval`` (default ``1m``) — the ``rate()`` window. Keep it at - or above 4× your Prometheus scrape interval; shorter windows add noise without - detecting load faster (the control loop is dominated by HPA sync and pod - startup), longer windows add lag. +work needs**:: + + R_needed = L_envoy / max(L_service / R_healthy, 1) + +Each input is measured — the ``rate()`` of a cumulative time counter equals the +mean number of requests inside that stage: + +- ``L_envoy`` — requests in flight between Envoy and Triton (queued, executing, + or on the wire): ``sum(rate(envoy_cluster_upstream_rq_time_sum{...}[1m])) / 1e3``. +- ``L_service`` — requests being executed across all models and pods: + ``sum(rate(nv_inference_request_duration_us − nv_inference_queue_duration_us)) / 1e6``. + Models are weighted by the time they consume, so the metric has no + model-specific constants and one threshold works for any mixture. +- ``R_healthy`` — Triton endpoints Envoy routes to: + ``max(envoy_cluster_membership_healthy{...})``. + +In PromQL, ``clamp_min(v, s)`` spells ``max(v, s)``. The floors encode that a +healthy replica can always execute at least one request (which lets an +underutilized fleet scale down), and that at zero replicas the metric reads +"requests in flight" instead of dividing by zero. + +Per healthy replica, the metric is the inflation clients experience: 1 means +every in-flight request is being executed, 2 means requests wait as long as they +are served, and it keeps growing linearly with overload. + +Thresholds +----------- + +- ``serverLoadThreshold`` (default ``2``) — KEDA scales to + ``ceil(metric / threshold)`` (``metricType: AverageValue``). ``2`` targets + "waiting ≈ serving"; ``1.5`` trades GPUs for latency. The unloaded floor is + ~1.2–1.3 (network transit), so values below that over-provision. +- ``serverAdmissionThreshold`` (default ``3``) — Envoy rejects new + ``RepositoryIndex`` requests when the per-replica metric exceeds it. Kept + above ``serverLoadThreshold``: the autoscaler settles near its threshold, and + gating admission there would reject clients during normal operation. +- ``serverLoadRateInterval`` (default ``1m``) — the ``rate()`` window; keep it + at or above 4x the Prometheus scrape interval. Custom metrics --------------- -If ``serverLoadMetric`` is set, it is used **verbatim** by both KEDA and the rate -limiter. KEDA compares it against ``serverLoadThreshold`` and the rate limiter -against ``serverAdmissionThreshold`` — set them to the same value if you want the -two consumers coupled. Also set ``keda.metricType`` to match your metric's -semantics (``Value`` for per-replica quantities, ``AverageValue`` for fleet-wide -ones). +If ``serverLoadMetric`` is set, it is used **verbatim** by both consumers: KEDA +compares it against ``serverLoadThreshold`` and the rate limiter against +``serverAdmissionThreshold``. Set ``keda.metricType: Value`` for per-replica +custom metrics. 9. (Optional) Deploy Grafana Dashboard ========================================== diff --git a/helm/supersonic/templates/_helpers/_scaling-metric.tpl b/helm/supersonic/templates/_helpers/_scaling-metric.tpl index 2c304168..083f659a 100644 --- a/helm/supersonic/templates/_helpers/_scaling-metric.tpl +++ b/helm/supersonic/templates/_helpers/_scaling-metric.tpl @@ -1,36 +1,26 @@ {{/* Scaling and admission metrics. -The default scaling metric is an "occupancy ratio" derived from Little's law -(L = lambda * W: mean occupancy equals the per-second rate of a cumulative -time counter). It is built from three measured quantities: - - L_envoy - mean number of requests in flight between Envoy and the Triton - fleet (queued + executing + on the wire), from Envoy's - cumulative upstream request-time counter (milliseconds). - L_service - mean number of requests being actively executed across all - models and pods, from Triton's cumulative request-duration - minus queue-duration counters (microseconds). - R_healthy - number of Triton endpoints Envoy currently routes to - (max across Envoy pods - each pod reports the same - upstream cluster membership). - -"supersonic.defaultMetric" renders the extensive form +The default scaling metric estimates how many replicas the current in-flight +work needs: R_needed = L_envoy / max(L_service / R_healthy, 1) -("how many replicas the current in-flight work needs"), spelled in PromQL as -L_envoy * R_healthy / max(L_service, R_healthy, 1) because clamp_min(v, s) -is PromQL for max(v, s). The floors encode two physical facts: each healthy -replica can serve at least one request concurrently (so serving capacity is -never below R_healthy), and at zero replicas the metric degrades to "requests -in flight" instead of dividing by zero. KEDA consumes this form with -metricType AverageValue: desired = ceil(R_needed / serverLoadThreshold). + L_envoy - mean requests in flight between Envoy and Triton: rate of + Envoy's cumulative upstream request-time counter (ms -> /1e3). + L_service - mean requests being executed across all models and pods: rate + of Triton's request-duration minus queue-duration counters + (us -> /1e6). + R_healthy - Triton endpoints Envoy routes to (max across Envoy pods). + +clamp_min(v, s) is PromQL for max(v, s). The floors encode that a healthy +replica can always execute at least one request, and that at zero replicas +the metric reads "requests in flight" instead of dividing by zero. -"supersonic.admissionMetric" is the same quantity per healthy replica -(R_needed / R_healthy, i.e. the sojourn-time inflation clients experience); -the Envoy Lua filter rejects RepositoryIndex requests when it exceeds -"supersonic.admissionThreshold". +KEDA consumes "supersonic.defaultMetric" (the form above) with metricType +AverageValue: desired = ceil(R_needed / serverLoadThreshold). The Envoy Lua +filter consumes "supersonic.admissionMetric" (the same per healthy replica) +and rejects RepositoryIndex above "supersonic.admissionThreshold". If .Values.serverLoadMetric is set, both helpers return it verbatim. */}} diff --git a/helm/supersonic/values.yaml b/helm/supersonic/values.yaml index f7171fa8..508745ea 100644 --- a/helm/supersonic/values.yaml +++ b/helm/supersonic/values.yaml @@ -4,27 +4,24 @@ nameOverride: "" # -- Prometheus query used by the KEDA autoscaler and (per healthy replica) by Envoy's -# prometheus-based rate limiter. Leave empty to use the default "occupancy ratio" metric: -# the mean number of requests in flight between Envoy and Triton (Little's law applied to -# Envoy's cumulative upstream request time) divided by per-replica serving occupancy -# (rate of Triton request time minus queue time). It reads ~1 when nothing queues, grows -# linearly with overload, and contains no model-specific constants, so one threshold works -# for any model mixture. Defined in templates/_helpers/_scaling-metric.tpl and documented -# in the configuration guide. A custom query set here is used verbatim by both consumers. +# prometheus-based rate limiter. Leave empty for the default metric: requests in flight +# between Envoy and Triton divided by requests being executed, i.e. how many replicas +# the current load needs. Reads ~1 with no queueing, grows linearly with overload, and +# has no model-specific constants. See templates/_helpers/_scaling-metric.tpl and the +# configuration guide. A custom query set here is used verbatim by both consumers. serverLoadMetric: "" -# -- Threshold for the scaling metric. With the default metric this is the tolerated -# sojourn inflation per replica: KEDA scales to ceil(metric / threshold), so 2 targets -# "requests spend about as long waiting as being served"; 1.5 trades GPUs for latency. +# -- Threshold for the scaling metric. KEDA scales to ceil(metric / threshold): +# 2 targets "waiting = serving" per replica; 1.5 trades GPUs for latency. serverLoadThreshold: 2 # -- Range-vector window used by rate() in the default metric. # Keep it at or above 4x your Prometheus scrape interval. serverLoadRateInterval: "1m" -# -- Threshold for Envoy's prometheus-based rate limiter, compared against the scaling -# metric per healthy replica. Kept above serverLoadThreshold so that new clients are -# rejected only when scaling can no longer keep up, not at the normal operating point. +# -- Threshold for Envoy's prometheus-based rate limiter (scaling metric per healthy +# replica). Kept above serverLoadThreshold so new clients are rejected only when +# scaling can no longer keep up. serverAdmissionThreshold: 3 # -- On RepositoryIndex, scale Triton to at least max(1, keda.minReplicaCount) replicas From f80f24289efd2814a6f83d3583484b9b7e009de8 Mon Sep 17 00:00:00 2001 From: GitHub Actions Date: Tue, 8 Sep 2026 20:54:09 +0000 Subject: [PATCH 5/5] Update helm docs --- docs/.values-table.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/.values-table.md b/docs/.values-table.md index 6a9dcfde..cdf1a510 100644 --- a/docs/.values-table.md +++ b/docs/.values-table.md @@ -3,10 +3,10 @@ | Key | Type | Default | Description | |-----|------|---------|-------------| | nameOverride | string | `""` | Unique identifier of SuperSONIC instance (equal to release name by default) | -| serverLoadMetric | string | `""` | Prometheus query used by the KEDA autoscaler and (per healthy replica) by Envoy's prometheus-based rate limiter. Leave empty to use the default "occupancy ratio" metric: the mean number of requests in flight between Envoy and Triton (Little's law applied to Envoy's cumulative upstream request time) divided by per-replica serving occupancy (rate of Triton request time minus queue time). It reads ~1 when nothing queues, grows linearly with overload, and contains no model-specific constants, so one threshold works for any model mixture. Defined in templates/_helpers/_scaling-metric.tpl and documented in the configuration guide. A custom query set here is used verbatim by both consumers. | -| serverLoadThreshold | int | `2` | Threshold for the scaling metric. With the default metric this is the tolerated sojourn inflation per replica: KEDA scales to ceil(metric / threshold), so 2 targets "requests spend about as long waiting as being served"; 1.5 trades GPUs for latency. | +| serverLoadMetric | string | `""` | Prometheus query used by the KEDA autoscaler and (per healthy replica) by Envoy's prometheus-based rate limiter. Leave empty for the default metric: requests in flight between Envoy and Triton divided by requests being executed, i.e. how many replicas the current load needs. Reads ~1 with no queueing, grows linearly with overload, and has no model-specific constants. See templates/_helpers/_scaling-metric.tpl and the configuration guide. A custom query set here is used verbatim by both consumers. | +| serverLoadThreshold | int | `2` | Threshold for the scaling metric. KEDA scales to ceil(metric / threshold): 2 targets "waiting = serving" per replica; 1.5 trades GPUs for latency. | | serverLoadRateInterval | string | `"1m"` | Range-vector window used by rate() in the default metric. Keep it at or above 4x your Prometheus scrape interval. | -| serverAdmissionThreshold | int | `3` | Threshold for Envoy's prometheus-based rate limiter, compared against the scaling metric per healthy replica. Kept above serverLoadThreshold so that new clients are rejected only when scaling can no longer keep up, not at the normal operating point. | +| serverAdmissionThreshold | int | `3` | Threshold for Envoy's prometheus-based rate limiter (scaling metric per healthy replica). Kept above serverLoadThreshold so new clients are rejected only when scaling can no longer keep up. | | scaleFromZero | object | `{"admissionImage":"python:3.14-slim","enabled":false,"holdMinReplicasSeconds":300,"readyTimeoutSeconds":300}` | On RepositoryIndex, scale Triton to at least max(1, keda.minReplicaCount) replicas and return the index only after Envoy has a healthy Triton upstream. Requires keda.enabled and envoy.enabled. | | scaleFromZero.enabled | bool | `false` | Enable scale from zero | | scaleFromZero.readyTimeoutSeconds | int | `300` | Seconds to wait for a healthy Triton upstream before rejecting RepositoryIndex |