feat(operator)!: publish provider readiness, load, capacity, and latency

[hexfusion]

Summary

The operator decides whether each provider can serve now and says why, as the provider's Ready condition and as grid_provider_ready. It also publishes how many requests each provider holds, its capacity, their ratio, and recent latency, on its signals endpoint and on /metrics. Every input is the EPP's /metrics. Stacked on #288.

What changed

• Readiness: Ready uses the status contract's reasons. NoEndpointsReady after two zero counts with no engine answer in 30s, since the EPP counts endpoints with fresh metrics and a saturated engine reads as zero, NoLivenessCheck when a scrape answers without the pool's ready-endpoint series, and ScrapeTimedOut, ScrapeUnauthorized, TLSHandshakeFailed or ScrapeFailed (class in the message) once nothing succeeds in the window. AwaitingFirstScrape (Unknown) before the first success. A change logs WARN when the provider turns not ready, INFO otherwise.
• Scrapes: grid_provider_scrape_total{grid_provider,result} and grid_provider_last_scrape_success_timestamp_seconds{grid_provider}.
• In-flight: grid_provider_in_flight_requests, the larger of the EPP's per-endpoint count and the pool averages times ready endpoints, plus flow control's queue.
• Capacity, breaking: spec.maxRunning counts one endpoint. grid_provider_capacity_requests is it times fresh ready endpoints. The grid-operator chart now passes grid.providers..maxRunning through.
• Saturation: grid_provider_saturation_ratio, in-flight over capacity, only for a finite non-negative count.
• Latency: TTFT p50 and p90, TPOT, prefill seconds per token, and error ratio over 30s, from the EPP's histograms, once 20 requests complete.
• /metrics: the provider series as gauges labeled grid_site and grid_provider, this site's and its peers'.
• Peer bounds: a hub keeps from a peer only the contract names and the EPP pool averages, DNS-1123 grid_provider values, finite non-negative values, and 64 providers. Refusals count in grid_peer_signals_refused_total{peer,reason}. A peer's custom signalNames are dropped.
• Overlay: a provider with an empty site selector is placed at its own site only.
• Serving config: candidates whose site is not a DNS-1123 label are refused, warned per GridNetwork on change.
• Load window: derived from the scrape, poll and poll timeout plus one poll of slack, 17s at a 5s scrape, instead of 30s.
• Mock provider: exposes a ready count, as an EPP does.

Testing

• Operator: clippy, fmt and tests, 1576 passed.
• Mock provider tests, 59 passed.
• Helm: lint --strict on every chart, unittest, and verify-helm-chart, 349 checks.

Limitations

• No e2e topology scrapes an EPP, so readiness, in-flight, capacity and latency are covered by unit tests only.
• The scraper token now reaches every provider with a metricsConfig. Narrowing its audience and targets is a separate change.
• A mock provider image older than this change reads NoLivenessCheck.
• Reinstall with per-endpoint maxRunning.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}