feat(gateway): choose sites by published saturation per capacity

[hexfusion]

Summary

Stacked on #286. The gateway picks a site from grid_provider_saturation_ratio against its capacity, rather than ranking on a single queue-depth metric name, and sheds when no site has room. The operator keeps deriving the ratio: capacity is spec.maxRunning times ready endpoints, and maxRunning lives only in the peer site's Kubernetes API.

What changed

• Selection: with three or more sites with room, draw two by capacity and keep the lower saturation. With two, draw weighted, clamped to 0.1 and 0.9. With one, take it.
• No site with room: draw by capacity among the healthy sites, tied on the best polled queue depth.
• Shed above a saturation of 1.05, route again below 0.95, decided per snapshot.
• Load metric: accept llm_d_epp_average_queue_size as well as inference_pool_average_queue_size, the only name today's gateway reads.
• Readiness exclusion and the decision counters, with an opt-in metrics listener and its chart support, carried from the earlier draft.
• Demotion waits until praxis has reported cluster health, through a new health.observed().
• Errors: a shed answers 429 rate_limit_exceeded with code capacity_exhausted, no healthy site stays 503 no_healthy_site, an unknown model is 404 model_not_found.

Testing

Gateway and root workspaces: fmt, clippy and tests pass. 3122 root and 98 gateway tests.

Partition simulation against the gateway binary, three sites at 128, 128 and 64 slots, 200ms service, both binaries built on praxis 0.7.3 so selection is the only difference.

clients	build	hub, 0.40	west, 0.40	east, 0.20	peak 10s window	requests	overshoot
64	before	1.00	0.00	0.00	1.00	14272	0
64	after	0.38	0.39	0.23	0.46	14272	0
128	before	1.00	0.00	0.00	1.00	28544	0
128	after	0.40	0.37	0.24	0.48	28508	49
256	before	1.00	0.00	0.00	1.00	28655	24517
256	after	0.34	0.46	0.20	0.48	42290	15199

No request failed in any run. Saturation alone tracks capacity share at all three loads, and converges as load rises, so the published latency series stays observe-only and no delay term is added.

The lab matches the before column. Across a sweep from 1 to 12 requests per second every Qwen3 request went to hq-east, which queued up to 193 while two sites sat idle. Three of the four sites published no load signal, because the operator's EPP scrape used a token bound to another pod. With nothing to rank, candidates tie and the gateway takes the front of a local-first order. That scrape is fixed separately, but the deployed gateway reads only inference_pool_average_queue_size and the lab's router 0.11.0 EPP exports llm_d_epp_average_queue_size, so it would tie again.

Limitations

• Under capacity the throughput is the same. At 64 and 128 clients the clients are the limit, not the grid, so both builds serve the same requests. The 48 percent gain appears only at 256, where the old path caps at one site's 128 slots.
• At 128 clients this takes 49 overshoots where the old path takes none. Pinning 128 clients onto a 128-slot site is an exact fit by accident, and spreading puts a few requests on the 64-slot site past full. It is 0.17 percent of the run and nothing failed.
• Overshoot falls 38 percent at 256 clients but does not go away. Selection acts on a ratio up to one poll stale, so it cannot promise a site has a free slot on arrival. Removing overshoot needs admission control, not better selection.
• The two equal sites split 0.34 and 0.46 at 256 clients. Inside the bar the row sets, but the weighted draw is not tight between equals under heavy load.
• Saturation carries no delay term, and no KV or EPP-gate term.
• The overflow draw is unbounded.
• A signal is treated as fresh for two polls, not the wider bound the design derives from poll, scrape, poll timeout and one more poll.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}