Replace the scaling metric with an occupancy ratio (in-flight / served requests)

[kondratyevd]

What

Replaces the default KEDA/Envoy scaling metric (mean inference queue time per batch execution, threshold 100) with an occupancy ratio: the mean number of requests in flight between Envoy and the Triton fleet, divided by the fleet's serving occupancy. The metric estimates how many replicas the current in-flight work needs, so the autoscaler can size the fleet in one proportional step.

Why

The queue-latency metric is non-linear in load and its scale depends on the model mixture:

• Once capacity is adequate it collapses toward zero, so the controller cannot tell "correctly sized" from "over-provisioned" — in a 30-client CMSSW A/B run on Geddes it scaled down mid-run and cost the remaining six minutes ~25% throughput.
• At moderate sustained queueing it under-reads — in a 15-client run it sat at 20–50 (threshold 100) for six minutes while clients spent 43% of their in-flight time queued.
• Its magnitude at saturation differs by ~25× between model mixtures observed on Geddes (UParT-scouting vs Run 3 MiniAOD), so no single threshold fits two deployments.

The occupancy ratio fixed both failure modes in back-to-back A/B runs (same jobs, same cluster): equal or lower GPU-minutes, −7–8% wall time, −17–21% p90 latency, steady fixed-point behaviour at both 5 replicas (capacity-capped) and 3 replicas (free choice).

How the PromQL query is constructed

Little's law (L = λW) turns cumulative time counters into mean occupancies: rate(<cumulative time counter>) is "seconds of request time per second of wall clock", i.e. the mean number of requests inside that system. The query combines three measured quantities:

Term	Meaning	Source
L_envoy	requests in flight between Envoy and Triton (queued + executing + on the wire)	sum(rate(envoy_cluster_upstream_rq_time_sum{...triton_grpc_service}[1m])) / 1e3 (counter in ms)
L_service	requests actively executing, all models and pods	sum(rate(nv_inference_request_duration_us − nv_inference_queue_duration_us)) / 1e6
R_healthy	Triton endpoints Envoy routes to	max(envoy_cluster_membership_healthy{...}) (max across Envoy pods — they report the same cluster)

and renders the extensive form

R_needed = L_envoy / max(L_service / R_healthy, 1)
         = L_envoy × R_healthy / max(L_service, R_healthy, 1)     # as written in PromQL

clamp_min(v, s) is PromQL for max(v, s). The two floors are physical, not tuning: each healthy replica can execute at least one request concurrently (serving capacity is never below the replica count — this is what makes scale-down work at low load), and at zero replicas the metric degrades to "requests in flight" rather than dividing by zero (never +Inf, which KEDA would read as "scale to max"). Because every term is a time integral, models are weighted by the time they consume — the metric contains no model-specific constants and reads the same for any mixture: ~1 when nothing queues, ~2 when requests wait as long as they are served, linear in the overload factor above that.

Consumers

• KEDA now uses metricType: AverageValue: desired = ceil(R_needed / serverLoadThreshold) — independent of pods that never scheduled (with the old Value type, pending pods inflate the request; observed as 3 unschedulable pods held for 10 minutes in testing). serverLoadThreshold defaults to 2 (tolerated sojourn inflation; 1.5 trades GPUs for latency).
• Envoy rate limiter compares the per-replica form (R_needed / R_healthy) against a new serverAdmissionThreshold (default 3). It is deliberately above the scaling threshold: the autoscaler settles the system near serverLoadThreshold, so gating admission at the same value would reject new clients during normal operation.
• serverLoadRateInterval (default 1m) exposes the rate() window — keep it ≥ 4× the Prometheus scrape interval; shorter windows double the noise without detecting load faster (measured: cv 18% vs 8% at 30s vs 1m, same threshold-crossing sample).

Compatibility

• A custom serverLoadMetric is still used verbatim by both consumers; keda.metricType: Value restores the old HPA semantics for per-replica custom metrics.
• Breaking for default users: the default serverLoadThreshold changes 100 → 2 with the new default metric; values files that only set serverLoadThreshold: 100 (without a custom metric) should drop it — the example values files in values/ are updated accordingly.
• The default metric now requires Envoy (envoy.enabled, the default) and Prometheus scraping of both Envoy and Triton, which the bundled Prometheus config already does.

Validation

• helm template verified for: defaults, values-minimal-full.yaml (scale-from-zero CI values), values-geddes-cms.yaml, custom-metric override, and the external-Envoy-config path; helm lint passes for every file in values/.
• Both rendered queries (KEDA and Lua forms) validated against the live Geddes Prometheus via promtool query instant.
• Live A/B evidence (4 runs, 2 load levels, real CMSSW clients) summarized above; happy to attach the full run reports.

🤖 Generated with Claude Code