fix(model): stop advertising providers on stale egress-health evidence

[Ryanmello07]

The bug

GetAllProviderEgressHealthCounts reads provider_egress_health with no age bound. It selects the whole table, loads measured_at into ProviderEgressHealthCounts — and nothing ever reads that field again. passesHealth checks only the ratio.

So a health measurement, once taken, is permanent. A provider measured healthy months ago still passes the gate today.

The sibling loader has always been bounded. GetAllProviderEgressCountryCodes filters WHERE observed_at >= $1 against ProviderEgressLocationMaxAge. Location evidence expires; health evidence never did — and health is the half that detects a blackhole.

Why nothing caught it

The re-probe schedule keys off the wrong table. Both passes of GetProviderEgressLocationDueSharded are driven by provider_egress_location, so a provider with a fresh location is never re-offered no matter how old its health tally is. Location is trusted for 7 days; health for 1. A provider probed once and then quietly going dark keeps its passing tally and its place in the list until its location ages out, days later.

It fails silently by construction: such a provider stays connected and keeps accepting client connections. Nothing downstream can tell it apart from a working one.

Measured, not theorised

On a live deployment:

health evidence age	providers passing the gate
under 6h	36
6–24h	1,208
1–3 days	1,093
over 3 days	200

98.6% were advertised on evidence older than six hours. I took the twelve with the stalest evidence — all currently advertised — and probed them:

019fedb3-7baa…  ok=0/131
019fedb3-752e…  ok=0/131
019fedb3-7640…  ok=0/131
…  12 of 12

Total blackholes, in the public list, on the strength of a measurement taken days earlier.

The fix

1. Bound the evidence. GetAllProviderEgressHealthCounts filters on a new ProviderEgressHealthMaxAge. A provider whose measurement aged out is absent from the map and fails closed — identically to one never measured, which is the correct reading: both mean no current evidence this provider carries traffic.

2. Re-probe on health age. A third due-queue pass offers providers whose health has gone stale, keyed on provider_egress_health. Without it, half 1 is a slow drain rather than a fix — every gated provider would age out of the map and never be re-measured, and the list would empty.

24h for health against 7 days for location, because they decay differently. Where a provider egresses from is a property of its network and rarely changes. Whether it still carries traffic is a property of the moment and changes with no signal at all. The constant is a floor on how bad the list can get: the longest a provider can blackhole while still being advertised.

A defect the tests caught

My first version scoped pass 3 to "no fresh health row". That predicate is also true of a provider never measured, so it made every fixture in TestGetProviderEgressLocationDue due, including one probed an hour ago — re-probing on the absence of a row rather than on evidence going stale.

Pass 3 is now scoped to providers that have a health row which aged out. A provider with a location but no health row is left to passes 1 and 2: excluded from the list meanwhile (passesHealth fails closed on a missing row) and re-offered when its location goes stale — deferred, not stranded.

Passes 1 and 2 keep their predicates, ordering and index scans untouched.

Verification

Run against a live postgres:

• All six due-queue tests pass, including two new ones: stale health is offered, never-measured is not.
• TestGetProviderEgressLocationDueOffersStaleHealth was confirmed to fail when pass 3 is disabled (due = []), so it has teeth.
• Health, provider-count, find-providers and client-score tests pass unchanged (125s of execution).
• go build ./model/... ./api/... ./controller/..., go vet ./model/, gofmt -l all clean.

Deployed to beta ahead of this PR: 5,942 stale rows dropped out of the gate, the advertised set moved to tracking the fresh-and-passing population, and providers were observed expiring live — 1,207 → 1,186 as a cohort crossed 24h. Under the old code they would have stayed advertised indefinitely.

Related, not in this PR

The deployment where this was found had every provider going dark at exactly 24h — jwt.expiryDuration is 24h and those clients hold static tokens with no refresh. That's a client-side matter and is separately fixed in sdk (half-life rotation, 2026-08-06). This PR is the server-side defence: whatever the cause, a provider that has stopped carrying traffic should stop being advertised, and it now does.