diff --git a/CHANGELOG.md b/CHANGELOG.md index 39c61c142..26966117b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,62 @@ Semantic Versioning where the repository publishes a release. still named the removed `free_family_diversity` evidence field instead of its `free_account_diversity` replacement, which could send future monitoring work looking for a field that no longer exists. +- `scripts/ci/contextual_orchestrator_review_policy.py`'s catalog admission + cap and diversity evidence no longer conflate "independent credential + account" with "independent outage domain": `nvidia_nim`/`nvidia_nim_sub` + are independent accounts (may expose different models) but share one + physical upstream endpoint (`https://integrate.api.nvidia.com/v1`), so + they now share one admission-cap budget and count as one outage domain. A + new `free_outage_domain_diversity` report field (additive, alongside the + existing `free_account_diversity`) reflects this for callers deciding + whether a single provider outage could empty the free catalog. Outage- + domain grouping normalizes each row's `base_url` first (lowercasing + scheme/host, dropping an explicit default port, stripping a trailing + slash, and never raising even on a malformed IPv6-bracket URL), so a + formatting difference alone cannot split one physical endpoint into two + domains. Within a shared domain, the admission cap's bounded slots are + now split round-robin across the domain's contending accounts instead of + being consumed entirely by whichever account's rows happen to sort first + -- fixing a narrower starvation bug the outage-domain grouping itself + introduced (one credential could otherwise get zero admissions from a + shared domain even with rows available and cap budget nominally unused + by it). That fairness reordering is now strictly scoped to one admission- + priority tier (cost tier + ZDR status) at a time, never across tiers -- + an earlier revision grouped a whole outage domain's rows into one block + regardless of tier, which could drag a lower-priority route (paid, + non-ZDR) ahead of a higher-priority route (free, ZDR) belonging to a + different domain, sometimes dropping a free route for a paid one under a + tight catalog limit. IPv6 host normalization now re-brackets a + colon-bearing host before appending a port, so an explicit-port address + (`[::1]:8443`) and an unrelated literal that merely contains the same + digits (`[::1:8443]`) no longer collapse to one outage domain. The + priced-fallback catalog stage (`orchestrator/auto`'s post-primary-stage + fallback) gets its own domain-diversity fix: with both defaults at 4, + the fallback route budget coincidentally equaled the per-domain cap, so + a single dominant outage domain could exhaust the entire fallback stage + before a genuinely independent domain's row was ever considered (Devin + Review finding). `build_zdr_prioritized_catalog` gains an opt-in + `guarantee_domain_coverage` flag (only the priced-fallback call site + sets it) that admits in two passes instead of one: the first pass + admits at most one row per outage domain, guaranteeing representation; + the second fills any remaining budget from whichever domain's + next-highest-priority row comes first, still bounded by `account_cap`. + A first revision shrank the cap to `fallback_limit // domain_count` + instead, which fixed representation but wasted capacity whenever the + split was uneven (a second Devin Review finding, "fallback quota wastes + probe slots" -- `limit=4` across 3 domains admitted only 3 routes under + a floor of 1); the two-pass approach guarantees both properties at + once. The common single-domain case is unchanged. A third Devin Review + finding caught the identical gap reachable through the *primary* + `auto`-pool stage too, not just the fallback: the review sidecar's real + deployed default is `ORCHESTRATOR_CATALOG_ACCOUNT_CAP=8` (not the + launcher's `DEFAULT_ACCOUNT_CAP=4` fallback, which the sidecar never + leaves the env var unset for), and the primary stage's own route limit + for the `auto` pool is also capped at 8 + (`REVIEW_PREFLIGHT_PRIMARY_ROUTE_LIMIT`) -- the same cap-equals-limit + coincidence, just at 8 instead of 4. `guarantee_domain_coverage=True` + now applies to both `build_zdr_prioritized_catalog` call sites in + `main()`. - Noema, Strix, and OpenCode review sidecars now vendor contextual-orchestrator at `c107e3e52371993aa9c326fcc245e01c41fc3850` and treat every KV credential as an independent discovery account. Same-vendor credentials no longer diff --git a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md index 3a48cdf58..9be346477 100644 --- a/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md +++ b/docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md @@ -190,6 +190,29 @@ all five, and auto-optimize routing by cost. amendment" (above) are closed, without requiring a manual re-audit. `docs/doctoring/contextual-orchestrator-strix-free-diversity-evidence.md` records that PR's own reasoning trail. +- **2026-08-31 correction: account diversity is not outage-domain diversity.** + Review during this session found that #1468 (above), in correctly stopping + `nvidia_nim`/`nvidia_nim_sub` from being treated as one *model-catalog* + family, also let `free_account_diversity` and the catalog's admission cap + treat them as two fully independent *outage domains* — they are not: both + resolve to the identical `https://integrate.api.nvidia.com/v1` upstream + (see `PROVIDER_BASE_URLS` in `scripts/ci/zdr_policy.py`, and that table's + own `nvidia_nim_sub` ZDR-scope note). Conflating the two meant a discovery + report whose only free routes were these two credentials reported + `free_account_diversity == 2` — falsely reassuring for exactly the decision + this evidence exists to support (would a single physical outage empty the + free catalog) — and the admission cap let the pair jointly consume up to + twice its intended per-domain budget, crowding out a genuinely independent + provider even when one had free routes available. + `contextual_orchestrator_review_policy.py` now reports a second, distinct + field, `free_outage_domain_diversity`, grouped by each row's own `base_url` + evidence rather than a second hand-maintained provider-name table, and the + admission cap (`account_cap`; the name predates this fix and is kept for + CLI/environment stability) groups by outage domain, not by credential. A + caller deciding whether Strix can safely rely on a strict `orchestrator/free` + pool without the `orchestrator/auto` paid fallback (open PR #1437) should + read `free_outage_domain_diversity`, not `free_account_diversity`, for that + specific decision. - **2026-08-31 amendment: Noema reviews independently of OpenCode.** Noema no longer waits for an OpenCode approval, review-thread state, or other check conclusions before calling the gateway and submitting its current-head diff --git a/docs/product-goal-directive.md b/docs/product-goal-directive.md index ecb4f3b69..b5de58a23 100644 --- a/docs/product-goal-directive.md +++ b/docs/product-goal-directive.md @@ -66,7 +66,7 @@ Per this file's own conflict policy above: this note is the resolution, and `doc **Note (flagged by CodeRabbit on this PR, 2026-08-30):** section 8's quoted text describes `contextual-orchestrator`'s general product capability — broad model/modality support and all-five-secret auto model discovery as a *design principle for the orchestrator itself*. It does not specify, and must not be read as overriding, which pool each CI consumer routes through: that is governed exclusively by `docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md` and its doctoring records — `OpenCode` and `Noema` use the fail-closed, ZDR-prioritized `orchestrator/free` pool; only `Strix` security analysis uses the provider-diverse `orchestrator/auto` pool; private/internal review targets require an attested ZDR-only catalog and never fall back to a non-ZDR provider. Do not loosen any CI consumer's pool or credential scope on the strength of this section's general wording alone. -**Note (2026-08-30, superseded by the merged pin flip — see the correction below):** an earlier draft of this note said Strix stayed on `orchestrator/auto` pending `free_family_diversity` reaching `>= 2`. That is no longer true and must not be read as current: `.github/workflows/strix.yml` now hardcodes `STRIX_MODEL`/`CONTEXTUAL_ORCHESTRATOR_POOL` to `orchestrator/free` and fails closed on any other value, and ADR-0003's 2026-08-30 amendment records the owner's decision to accept the residual single-outage-domain risk immediately rather than wait for the evidence-gated threshold this note originally described. `free_account_diversity` (`scripts/ci/contextual_orchestrator_review_policy.py`; renamed from `free_family_diversity` once every KV credential became an independent discovery account rather than being grouped into a vendor "family", see #1468) remains useful as ongoing monitoring evidence for that accepted risk, not as a gate blocking the pin. +**Note (2026-08-30, superseded by the merged pin flip — see the correction below):** an earlier draft of this note said Strix stayed on `orchestrator/auto` pending `free_family_diversity` reaching `>= 2`. That is no longer true and must not be read as current: `.github/workflows/strix.yml` now hardcodes `STRIX_MODEL`/`CONTEXTUAL_ORCHESTRATOR_POOL` to `orchestrator/free` and fails closed on any other value, and ADR-0003's 2026-08-30 amendment records the owner's decision to accept the residual single-outage-domain risk immediately rather than wait for the evidence-gated threshold this note originally described. `free_account_diversity` (`scripts/ci/contextual_orchestrator_review_policy.py`; renamed from `free_family_diversity` once every KV credential became an independent discovery account rather than being grouped into a vendor "family", see #1468) remains useful as ongoing monitoring evidence for that accepted risk, not as a gate blocking the pin. **Correction (2026-08-31):** for *this specific* single-outage-domain risk, read `free_outage_domain_diversity`, not `free_account_diversity` — #1468's rename correctly made every KV credential an independent *account*, but `nvidia_nim`/`nvidia_nim_sub` remain one *outage domain* (both resolve to the identical `https://integrate.api.nvidia.com/v1` upstream), so `free_account_diversity` alone can read `2` for a catalog that is, in fact, still exposed to a single provider outage. `free_outage_domain_diversity` is the field that actually answers this note's question. ## 9. Reference libraries, tool invocations, and ecosystem repositories diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index ea56ced98..739f35426 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -1715,6 +1715,192 @@ string, a bare number) confirmed to fail against the pre-fix script (`KeyError: signature as the original round-4 bug) before passing after the fix. 1930 tests pass; 100% coverage and 100% docstring coverage on `scripts/ci/`. +## 2026-08-31 a second, subtler NVIDIA-independence gap: account diversity conflated with outage-domain diversity + +**Context.** This session investigated why `contextual-orchestrator` PR #941/#945's fix (independent +`nvidia_nim`/`nvidia_nim_sub` credentials must not be assumed to share one model catalog) was not +reflected in production review evidence, and found two bugs: a stale `ORCHESTRATOR_PIN_SHA` vendoring +pin, and this repo's own independent copy of the collapsing assumption in +`scripts/ci/contextual_orchestrator_review_policy.py`'s `PROVIDER_FAMILIES`. Both were superseded +mid-session by `.github#1468` ("fix(ci): keep sidecar credential accounts independent"), which the repo +owner merged directly and which covers both: it bumps the pin to `contextual-orchestrator`'s then-current +`main` tip (`0adca4703df67f8f31d3ea5b04a1e07ed775dd6c`, later advanced again by `.github#1469`) and +removes `PROVIDER_FAMILIES` entirely, renaming the concept from "provider family" to "provider account" +throughout (`free_family_diversity` → `free_account_diversity`, `family_cap` → `account_cap`). + +**What #1468 did not catch.** Review during this session (a Devin Review finding on the now-closed, +superseded PR #1470, checked directly against `main`'s actual merged code before acting) found that +#1468's fix, while correctly removing the wrong model-catalog assumption, introduced a second, more +subtle conflation on a genuinely different axis. Two independent questions exist for +`nvidia_nim`/`nvidia_nim_sub`: + +1. **Model-catalog identity** — may these two credentials be entitled to different models? Yes. This is + what #941/#945/#1468 correctly fixed. +2. **Outage-domain identity** — would one physical infrastructure outage take both credentials down + together? Also yes: both resolve to the identical `https://integrate.api.nvidia.com/v1` upstream (see + `PROVIDER_BASE_URLS` in `scripts/ci/zdr_policy.py`, and that table's own `nvidia_nim_sub` ZDR-scope + note, which already said as much). #1468's fix, in correcting axis 1, also flattened axis 2 to be + identical to axis 1 -- `provider_account()` (identity-only) became the *sole* grouping key for both + the `free_account_diversity` evidence field and the catalog's admission cap. + +This matters concretely: `free_account_diversity` exists specifically so a caller (open PR #1437, +draft, gating Strix's `orchestrator/free` eligibility) can tell whether a single provider outage could +empty the free catalog -- that is fundamentally an outage-domain question, not a credential-count +question. With the two axes conflated, a discovery report whose only free routes are these two NVIDIA +credentials reports `free_account_diversity == 2`, which would falsely read as "safe" for exactly the +decision this evidence exists to support. Separately, the admission cap (`account_cap`, sidecar default +8) let the two credentials jointly consume up to *twice* its intended per-endpoint budget, which +concretely re-creates a milder version of the 2026-08-30 `orchestrator/free` exhaustion incident this +cap exists to prevent (documented earlier in this file): a shared endpoint's rows could crowd out a +smaller, genuinely independent provider's free routes even when that provider had capacity available. + +**Fix (this PR, a small, focused follow-up against current `main`, not a revival of #1470).** +`scripts/ci/contextual_orchestrator_review_policy.py` gains a second, distinct grouping, +`_outage_domain(row)`, keyed on each row's own `base_url` evidence (not a second hand-maintained +provider-name table, so it cannot silently go stale independently of the `base_url` evidence the catalog +already serves from -- the exact failure mode that made the removed `PROVIDER_FAMILIES` mapping wrong). +The admission cap now groups by outage domain (two same-endpoint credentials share one cap budget, they +do not each get their own); a new report field, `free_outage_domain_diversity`, is added *alongside* the +existing `free_account_diversity` (additive, not a rename, to avoid another naming churn on top of +#1468's very recent one) so a caller like #1437 can read the field that actually answers its question. +`scripts/ci/contextual_orchestrator_review_launcher.py`'s `_with_discovery_counts` (which recomputes +diversity from full discovery-wide rows, not the narrower per-stage set) restores both fields the same +way. `account_cap`/`DEFAULT_ACCOUNT_CAP`/the CLI `--account-cap` flag/the sidecar's +`ORCHESTRATOR_CATALOG_ACCOUNT_CAP` env var names are all left unchanged (still meaningful as "the cap +value"; only its grouping was wrong) to minimize collision risk with `.github#1469`, which was +concurrently advancing the same sidecar's pin in the same active window. + +**Tests.** Two dedicated regressions reproduce the exact gaps: one asserting `nvidia_nim` + +`nvidia_nim_sub` alone report `free_account_diversity == 2` but `free_outage_domain_diversity == 1` +(the semantic-conflation bug), and one reproducing the crowding-out scenario concretely (a shared-endpoint +credential pair with far more free rows than an independent provider; before this fix the independent +provider could be admitted zero rows, after it the shared endpoint's admissions are capped to protect +room for independent providers). Existing tests (`test_build_catalog_applies_account_cap`, +`test_build_catalog_reports_free_account_diversity`, `test_build_catalog_counts_same_vendor_credentials_ +independently`, plus two launcher-facing tests in `test_contextual_orchestrator_review_runtime_ +preflight.py`) were updated to the corrected, domain-aware expectations. Full suite green; 100% coverage +and 100% docstring coverage on `scripts/ci/`. + +**Not touched:** open PR #1437's own gating logic. Its reviewer should read `free_outage_domain_ +diversity`, not `free_account_diversity`, when wiring the `>= 2` eligibility check this file documents. + +**Follow-up (same PR, same day): raw-string comparison would have reintroduced the same class of bug.** +A Devin Review finding on this PR pointed out that `_outage_domain(row)` as first written compared raw +`base_url` strings -- so a hostname-case difference, an explicit default port (`:443`), or a trailing +slash between two rows that are actually the *same* physical endpoint would split them into two outage +domains, silently reintroducing the exact diversity-overstating/cap-bypassing bug this PR set out to fix. +Verified against this codebase's actual code before acting, not assumed: every `DiscoveredModel.chat_ +base_url` in `contextual-orchestrator/contextual_orchestrator/model_discovery.py` traces to one of a +fixed set of hardcoded Python string literals (the `nvidia_nim`/`nvidia_nim_sub` entries are byte- +identical), and this repo's launcher (`_report_rows`) copies that value verbatim, falling back only to +`zdr_policy.PROVIDER_BASE_URLS` -- confirmed byte-identical to the same literals for all five tracked +providers. So the risk is **not reachable through this repo's one production caller (the sidecar/ +launcher) today**. It *is* reachable through `contextual_orchestrator_review_policy.py`'s own public, +independently invocable `--discovery-report` CLI, which reads an arbitrary JSON file and is not +restricted to the launcher's exact generation path -- not wired into any current production workflow +(`hourly-nvidia-nim-review-repair.yml` only runs tests/coverage against this file, never the CLI on live +input), so the risk is latent, not live, but real for that public surface. Given the fix is cheap and +behavior-neutral on every input this repo's sidecar produces today, it was applied rather than left as an +unstated assumption: `_outage_domain` now compares `_normalize_base_url(row["base_url"])`, which +lowercases scheme/host, drops an explicit default port, and strips a trailing slash, while preserving a +different host, non-default port, path, or scheme as genuinely distinct domains, and falling back to a +lowercased/stripped whole-string comparison (never raising) for anything it cannot parse into a scheme, +host, and numeric port. Five new tests cover the exact equivalent-spelling cases Devin named (case, +default port, trailing slash), confirm genuine distinctions still separate, confirm no-raise on malformed +input (including a non-numeric port, which `urlsplit(...).port` raises `ValueError` on), and one +end-to-end test through `build_zdr_prioritized_catalog` itself with two differently-spelled rows for the +same endpoint. Full suite green (2106 tests); 100% coverage (including the new fallback branch) and 100% +docstring coverage on `scripts/ci/`. + +**Second follow-up (same PR, same day): the outage-domain cap itself could starve one credential +entirely.** Two more Devin Review findings on this PR, one severe. + +- **Severe: shared-cap starvation within a domain.** Grouping the admission cap by outage domain (above) + fixed cross-domain crowding-out, but the admission loop still walks rows in one strict sorted + (cost-tier, ZDR, provider, model) order and admits greedily until a domain's cap is reached. Since + `"nvidia_nim" < "nvidia_nim_sub"` in every real fixture, `nvidia_nim`'s rows always sort first -- + meaning `nvidia_nim` alone could consume the *entire* shared cap before a single `nvidia_nim_sub` row + was ever considered. Verified concretely before fixing: 6 free `nvidia_nim` rows + 6 free + `nvidia_nim_sub` rows, `account_cap=4` -> `nvidia_nim_sub` was admitted **zero** rows. Not "prevented + from taking more than its fair share" (the bug already fixed), but "the alphabetically-first credential + can take the *entire* shared budget, the other gets nothing" -- a narrower but just-as-real version of + the same crowding-out problem, now happening *within* one domain instead of across domains. Fixed with + a new `_fair_admission_order()` reordering step, applied before the existing (otherwise unchanged) + greedy admission loop: rows are partitioned by outage domain (preserving each domain's original + position relative to other domains), and within any domain contributed to by more than one account, + rows are taken in round-robin turns across those accounts -- one from each account's own + priority-ordered queue per round -- instead of exhausting whichever account sorts first. A domain with + only one contributing account (every provider except the shared NVIDIA pair, as of this writing) is + returned completely untouched. Re-verified the same scenario after the fix: `nvidia_nim: 2, + nvidia_nim_sub: 2` -- both credentials now contribute. Two existing tests whose assertions had encoded + the starvation behavior (`test_build_catalog_applies_account_cap`, + `test_build_catalog_prevents_shared_endpoint_from_crowding_out_independent_providers`) were corrected + to the fair-split expectation; a new end-to-end regression + (`test_build_catalog_shared_domain_cap_does_not_starve_second_account`) and two unit-level tests + directly against `_fair_admission_order()` (untouched-single-account case; visible round-robin + reordering, including that a multi-account domain's block still starts at its original position among + other domains) were added. +- **Real: `urlsplit()` itself can raise, not only `.port`.** `_normalize_base_url`'s existing fallback + wrapped only the `.port` property access; `urlsplit()` itself raises `ValueError` for an unmatched + IPv6-literal bracket (e.g. `https://[::1/v1`, confirmed: `ValueError: Invalid IPv6 URL`), which happens + earlier, before any scheme/host is even available to inspect -- an uncaught exception past this + function's own "must never raise on evidence it merely groups" contract. Fixed by wrapping the + `urlsplit()` call itself in the same catch-and-fall-back-to-a-lowercased-copy pattern already used for + the `.port` case. One new regression test confirms both a malformed IPv6-bracket URL and its + differently-cased twin fall back to the same, non-raising, normalized value. +- **Noted, not chased further (info-level, optional):** hostname canonicalization stops at lowercasing -- + a trailing root-label dot, an IDN's Unicode vs. punycode form, and differently-compressed-but-equivalent + IPv6 literals are not folded together. None of these shapes occur in any `base_url` this codebase + produces today (every value traces to a fixed set of hardcoded, already-canonical HTTPS hostnames), so + this is documented as a deliberate residual gap in `_normalize_base_url`'s own docstring rather than + implemented prophylactically; a future provider whose entitled address genuinely takes one of these + forms should extend the function with evidence of that specific case. + +Full suite: 2111 passed, 1 skipped, 21 subtests passed. 100% coverage (including the new reordering +function and both new fallback branches) and 100% docstring coverage on `scripts/ci/`. + +**Third follow-up (same PR, same day): the round-robin fix itself broke tier priority, plus a real IPv6 +normalization collision.** Two more Devin Review findings, one a real correctness regression the previous +round introduced. + +- **Real regression: fairness reordering could drop a free route for a paid one.** The round-robin fix + above grouped every row belonging to one outage domain into a single contiguous block, emitted at the + position of that domain's *first* appearance in the (already tier-sorted) input -- but a domain's rows + can span multiple admission-priority tiers (e.g. `openai` contributes both a free and a priced route, + same single-account domain). Grouping by domain first, tier-blind, let a domain's lower-tier row (e.g. + priced) get pulled into the same block as its higher-tier row (free), ahead of a *different* domain's + higher-tier row that only sorted later because of the `(provider, model)` tie-break. Verified + concretely before fixing: sorted input `[free openai, free openrouter, priced openai]` reordered to + `[free openai, priced openai, free openrouter]`, and with `limit=2` the genuinely free `openrouter` + route was dropped in favor of the priced `openai` route -- a real correctness regression for a catalog + whose entire purpose is admitting free/ZDR routes preferentially. Fixed by scoping the round-robin + fairness pass strictly *within* one admission-priority tier at a time: `eligible_rows` and + `_fair_admission_order` now share one `_admission_priority_key()` function (the sort key and the + tier-boundary detector can no longer silently drift apart), the input is split into contiguous + same-tier runs (safe, since it is already tier-sorted), and the existing domain/account round-robin + logic (renamed `_fair_order_within_tier`) is applied independently to each run, then the runs are + concatenated back in their original order. Re-verified: the same scenario now correctly keeps + `[free openai, free openrouter]` under `limit=2`; the starvation-fix regression scenario from the + previous round still passes unchanged (both were verified together, programmatically, before + committing). Added a unit-level regression directly against `_fair_admission_order` and an end-to-end + regression through `build_zdr_prioritized_catalog`. +- **Real: IPv6 host normalization could collide two different endpoints.** `urlsplit().hostname` strips + IPv6 literal brackets (`[::1]` -> `::1`); appending a port without re-adding them meant an explicit-port + IPv6 URL (`https://[::1]:8443/v1`, host `::1` port `8443`) and an unrelated literal that merely contains + the same colon-digit sequence (`https://[::1:8443]/v1`, one IPv6 address, no separate port) both + normalized to the identical, syntactically-invalid `::1:8443` -- two genuinely different endpoints + undercounted as one outage domain, in addition to producing malformed reassembled URL syntax either + way. Fixed by re-wrapping a colon-bearing host in brackets before ever conditionally appending a port. + Re-verified: the two example URLs now normalize distinctly, a default IPv6 port is still correctly + dropped, and the malformed-IPv6-bracket fallback from the previous round still works unchanged. Two new + regression tests. +- **Optional perf nit, applied since already in this code:** the round-robin queues switched from + list-`pop(0)` (O(n) per pop) to `collections.deque.popleft()` (O(1)) -- current catalog sizes make this + immaterial, but the change was one import and two identifiers. + +Full suite: 2115 passed, 1 skipped, 21 subtests passed. 100% coverage and 100% docstring coverage on +`scripts/ci/`. + ## 2026-08-31 `ORCHESTRATOR_PIN_SHA` bumped to carry #925's stream_options/tools fix **Context**: `#1451` fixed a separate, org-wide `pingora_edge_policy.py` coverage diff --git a/scripts/ci/contextual_orchestrator_review_launcher.py b/scripts/ci/contextual_orchestrator_review_launcher.py index f115ef2b8..8c0989046 100644 --- a/scripts/ci/contextual_orchestrator_review_launcher.py +++ b/scripts/ci/contextual_orchestrator_review_launcher.py @@ -700,20 +700,30 @@ def _with_discovery_counts( rows: list[dict[str, Any]], *, provider_account: Any, + outage_domain: Any, ) -> dict[str, object]: """Copy a stage report while restoring full discovery-tier counts. - ``free_account_diversity`` is recomputed here from the full discovery-wide - ``rows``, not trusted from the stage report: the primary ``auto``-pool - stage may have selected only ZDR-admitted free rows (undercounting - diversity whenever ``--require-zdr`` excludes some free routes) and the - priced-fallback stage selects only priced rows (so its own internally - computed diversity is always zero) -- either stage report's - ``free_account_diversity``, as returned by ``build_zdr_prioritized_catalog`` - from whatever narrower row set it was given, would otherwise contradict - that field's documented "among *all* discovered free routes" contract. + ``free_account_diversity`` and ``free_outage_domain_diversity`` are both + recomputed here from the full discovery-wide ``rows``, not trusted from + the stage report: the primary ``auto``-pool stage may have selected only + ZDR-admitted free rows (undercounting diversity whenever ``--require-zdr`` + excludes some free routes) and the priced-fallback stage selects only + priced rows (so its own internally computed diversity is always zero) -- + either stage report's diversity fields, as returned by + ``build_zdr_prioritized_catalog`` from whatever narrower row set it was + given, would otherwise contradict those fields' documented "among *all* + discovered free routes" contract. + + ``provider_account`` and ``outage_domain`` are two deliberately distinct + groupings (see ``contextual_orchestrator_review_policy._outage_domain``'s + docstring): the former treats every credential as independent regardless + of vendor, the latter groups credentials that share one physical + upstream endpoint (e.g. ``nvidia_nim``/``nvidia_nim_sub``, both + ``https://integrate.api.nvidia.com/v1``) into one outage domain. """ enriched = dict(report) + free_rows = [row for row in rows if row.get("cost_evidence") == "free"] enriched.update( { "total_routes": len(rows), @@ -721,11 +731,10 @@ def _with_discovery_counts( "total_priced_routes": sum(row.get("cost_evidence") == "priced" for row in rows), "total_unknown_routes": sum(row.get("cost_evidence") == "unknown" for row in rows), "free_account_diversity": len( - { - provider_account(str(row["provider"])) - for row in rows - if row.get("cost_evidence") == "free" - } + {provider_account(str(row["provider"])) for row in free_rows} + ), + "free_outage_domain_diversity": len( + {outage_domain(row) for row in free_rows} ), } ) @@ -808,6 +817,7 @@ def main(argv: list[str] | None = None) -> int: DEFAULT_ACCOUNT_CAP, PolicyError, _load_zdr_endpoints, + _outage_domain, build_zdr_prioritized_catalog, is_zdr_model, parse_discovery_report, @@ -883,9 +893,13 @@ def main(argv: list[str] | None = None) -> int: zdr_endpoints=zdr_endpoints, require_zdr=args.require_zdr, pool=args.pool, + guarantee_domain_coverage=True, ) result["report"] = _with_discovery_counts( - result["report"], normalized_rows, provider_account=provider_account + result["report"], + normalized_rows, + provider_account=provider_account, + outage_domain=_outage_domain, ) Path(args.catalog_out).write_text( json.dumps({"agents": result["agents"]}, indent=2, sort_keys=True) + "\n", @@ -914,12 +928,16 @@ def main(argv: list[str] | None = None) -> int: zdr_endpoints=zdr_endpoints, require_zdr=args.require_zdr, pool="auto", + guarantee_domain_coverage=True, ) except PolicyError: fallback_result = None if fallback_result is not None: fallback_result["report"] = _with_discovery_counts( - fallback_result["report"], normalized_rows, provider_account=provider_account + fallback_result["report"], + normalized_rows, + provider_account=provider_account, + outage_domain=_outage_domain, ) fallback_result["report"]["primary_selected_count"] = primary_report[ "selected_count" diff --git a/scripts/ci/contextual_orchestrator_review_policy.py b/scripts/ci/contextual_orchestrator_review_policy.py index 1c8a17014..547b5e4b2 100644 --- a/scripts/ci/contextual_orchestrator_review_policy.py +++ b/scripts/ci/contextual_orchestrator_review_policy.py @@ -13,9 +13,10 @@ import math import re import sys -from collections import Counter +from collections import Counter, deque from pathlib import Path from typing import Any, Iterable, Mapping +from urllib.parse import urlsplit, urlunsplit from scripts.ci.zdr_policy import ( PROVIDER_AUTH_SCHEMES, @@ -30,6 +31,8 @@ DEFAULT_CATALOG_LIMIT = 12 DEFAULT_ACCOUNT_CAP = 4 +_DEFAULT_PORTS: Mapping[str, int] = {"http": 80, "https": 443} + COST_FREE = "free" COST_PRICED = "priced" COST_UNKNOWN = "unknown" @@ -51,6 +54,292 @@ def provider_account(provider_name: str) -> str: return provider_name +def _outage_domain(row: Mapping[str, Any]) -> str: + """Return the shared-infrastructure outage domain for a normalized row. + + This is a deliberately *different* axis from :func:`provider_account`. + ``provider_account`` answers "is this a distinct credential that may be + entitled to a distinct model catalog" (yes, for ``nvidia_nim`` vs. + ``nvidia_nim_sub`` -- see PR #941/#945 in ``contextual-orchestrator`` and + this repo's own matching fix, both of which correctly stopped assuming + those two independent NVIDIA NIM API keys share a catalog). This + function instead answers "would one physical upstream outage take both + of these routes down together" -- and for those same two credentials the + answer is yes: both resolve to the identical ``base_url``, + ``https://integrate.api.nvidia.com/v1`` (see ``PROVIDER_BASE_URLS`` in + ``scripts/ci/zdr_policy.py``, and that table's own ``nvidia_nim_sub`` + ZDR-scope note: "the same integrate.api.nvidia.com trial API"). + Conflating these two axes -- treating "independent credential" as + "independent outage domain" -- would let two same-endpoint credentials + jointly report full diversity and jointly fill an admission cap meant to + protect against exactly one endpoint's outage, silently recreating the + 2026-08-30 ``orchestrator/free`` exhaustion incident this cap exists to + prevent (see ``docs/product-technical-gap-baseline.md``), just on a + different axis than the one #941/#945/#1468 already fixed. + + Grouped by each row's own ``base_url`` evidence (already present on + every row ``parse_discovery_report``/the sidecar's live discovery + produces) rather than a second hand-maintained provider-name table, so + this cannot silently go stale independently of the ``base_url`` evidence + the catalog itself already serves from -- the same failure mode that + made the removed ``PROVIDER_FAMILIES`` mapping wrong in the first place. + + Compares :func:`_normalize_base_url`'s normalized form, not the raw + string: two spellings of the identical endpoint (a hostname cased + differently, an explicit default port, a trailing slash on one row but + not another) must not be read as two outage domains, or a pure + formatting accident could reintroduce exactly the diversity-overstating, + cap-bypassing bug this function exists to fix. Every KV-credentialed + provider in this codebase today resolves ``base_url`` from one of a + fixed set of hardcoded string literals (never a live, potentially + differently-formatted network response), so this normalization changes + nothing for any input this repository's sidecar currently produces -- + it exists to keep this public, independently invocable function (also + reachable through this script's own ``--discovery-report`` CLI, not only + the sidecar's exact generation path) correct for any future input, not + to compensate for an observed live discrepancy. + """ + return _normalize_base_url(str(row["base_url"])) + + +def _normalize_base_url(base_url: str) -> str: + """Return a case/port/trailing-slash-normalized identity for a base URL. + + Scheme and host are lowercased (both are case-insensitive per RFC 3986 + 3.1/3.2.2); an explicit port equal to the scheme's default (``:443`` for + ``https``, ``:80`` for ``http``) is dropped, since it is equivalent to + omitting it; exactly one trailing slash is stripped from the path, since + a base URL's trailing slash does not change which resource it addresses. + Every other distinction -- a different host, a different non-default + port, a different path, or a different query string -- is preserved + verbatim (routing evidence has no legitimate reason to carry a query + string; preserving rather than dropping it means an unexpected one + cannot silently vanish from the computed identity). The URL fragment is + the one deliberate exception: it is stripped, not preserved, because a + fragment is a client-side-only artifact that is never transmitted to + the server and therefore never identifies a different upstream endpoint + -- two base URLs differing only by fragment must collapse to the same + outage-domain key, sharing one diversity count and one admission-cap + budget, not report inflated diversity or a separately budgeted cap. Any + userinfo component present is dropped rather than preserved: + outage-domain identity is about the physical endpoint, not which + credential reaches it, and this codebase's base URLs never carry + userinfo (see ``configured_gateway_source`` in + ``contextual-orchestrator``, which rejects one outright). + + A string this cannot parse into a scheme, host, and numeric port -- + including an empty string (which would otherwise normalize to a value + indistinct from a real one-character path), a malformed IPv6 host (an + unmatched ``[``/``]`` bracket makes ``urlsplit()`` itself raise + ``ValueError``, before any scheme/host/port is even available to + inspect), and a non-numeric port substring (``urlsplit(...).port`` + raises ``ValueError`` for one, once splitting succeeds) -- falls back to + a simple lowercased, stripped copy of the whole string: grouping only + needs equal inputs to compare equal, not a validated URL, and this + function must never raise on evidence it merely groups. + + Known, deliberate residual gap: hostname canonicalization stops at + lowercasing. A trailing root-label dot (``host.``), an IDN written as + Unicode versus its ASCII/punycode form, or two differently-compressed + but equivalent literal IPv6 addresses (e.g. ``::1`` vs ``0:0:0:0:0:0:0:1``) + are not folded together, so such a pair could still read as two outage + domains. None of these shapes occur in any ``base_url`` this codebase + produces today (every value traces to a fixed set of hardcoded, + already-canonical HTTPS hostnames -- see ``_outage_domain``'s + docstring), so this is intentionally not chased further here; a future + provider whose entitled address genuinely takes one of these forms + should extend this function with evidence of the specific case, not + prophylactically. + """ + text = base_url.strip() + try: + parsed = urlsplit(text) + except ValueError: + return text.casefold() + if not parsed.scheme or not parsed.hostname: + return text.casefold() + try: + port = parsed.port + except ValueError: + return text.casefold() + scheme = parsed.scheme.casefold() + host = parsed.hostname.casefold() + # urlsplit().hostname strips IPv6 literal brackets (``[::1]`` -> ``::1``). + # Re-adding them whenever the host itself contains a colon -- before ever + # conditionally appending a port -- is required for two reasons: without + # it, an explicit-port IPv6 URL (``[::1]:8443``) and a bracketless, + # colon-bearing literal address that merely *looks* like host:port when + # flattened (``[::1:8443]``, port None) collapse to the identical + # ``::1:8443`` string despite being different addresses; and the + # reassembled ``netloc`` must stay valid host:port syntax regardless. + bracketed_host = f"[{host}]" if ":" in host else host + netloc = ( + bracketed_host + if port is None or port == _DEFAULT_PORTS.get(scheme) + else f"{bracketed_host}:{port}" + ) + path = parsed.path.rstrip("/") + return urlunsplit((scheme, netloc, path, parsed.query, "")) + + +def _admission_priority_key( + row: Mapping[str, Any], *, zdr_endpoints: frozenset[str] +) -> tuple[int, int, str, str]: + """Return the deterministic ``(cost tier, ZDR tier, provider, model)`` sort key. + + The single source of truth for admission priority: ``build_zdr_ + prioritized_catalog`` sorts ``eligible_rows`` with this key, and + :func:`_fair_admission_order` re-derives just its first two components + (the tier, excluding the ``(provider, model)`` tie-break) to find tier + boundaries in that same sorted sequence -- sharing one function instead + of two independently written key expressions means the two can never + silently drift out of sync with each other. + """ + return ( + _COST_EVIDENCE_RANK[_cost_evidence(row)], + 0 + if is_zdr_model( + str(row["provider"]), model=str(row["model"]), zdr_endpoints=zdr_endpoints + ) + else 1, + str(row["provider"]), + str(row["model"]), + ) + + +def _fair_admission_order( + rows: list[Mapping[str, Any]], *, zdr_endpoints: frozenset[str] +) -> list[Mapping[str, Any]]: + """Reorder rows so one outage domain's cap fills fairly across accounts. + + ``rows`` must already be sorted by :func:`_admission_priority_key` (see + ``build_zdr_prioritized_catalog``'s own sort, which uses the same key). + Grouping the admission cap by outage domain (:func:`_outage_domain`) + fixed one starvation bug -- two same-endpoint credentials sharing one + budget instead of each getting their own -- but introduced a second, + narrower one: the greedy admission loop consumes rows in sorted order, + so whichever account's rows happen to sort first (``"nvidia_nim"`` + before ``"nvidia_nim_sub"``, alphabetically, in every real fixture in + this file) could exhaust the *entire* shared cap before the domain's + other account was considered at all -- not "prevented from taking more + than its share", but shut out completely, even with rows of its own + available and cap budget nominally unused by it. + + Fairness is reordered strictly *within* one admission-priority tier + (the ``(cost tier, ZDR tier)`` pair -- the first two components of + :func:`_admission_priority_key`), never across tiers: an earlier + revision of this function grouped every row for one outage domain into + a single block at that domain's first appearance in the whole input, + regardless of tier, which could drag a lower-priority route (e.g. paid, + non-ZDR) from one domain ahead of a higher-priority route (e.g. free, + ZDR) belonging to a *different* domain that happened to appear later in + the original order -- a real correctness regression for a catalog whose + entire purpose is admitting free/ZDR routes preferentially. Splitting + ``rows`` into contiguous same-tier runs first (safe because the input + is already tier-sorted, so equal-tier rows are already contiguous) and + reordering fairness independently within each run, then concatenating + the runs back in their original order, makes tier priority strictly + non-negotiable: no row from a worse tier can ever end up ahead of a row + from a better tier, regardless of domain/account composition. + + Within one tier, a domain contributed to by only one account is + returned completely untouched, in its original relative position -- + this function changes nothing for the common case (every provider + except the shared ``nvidia_nim``/``nvidia_nim_sub`` pair, as of this + writing). Within a domain shared by more than one account, rows are + taken in round-robin turns across those accounts -- one row from + account A's own queue (which keeps A's rows in their original relative + order), then one from B's, cycling only over accounts that still have + an unconsumed row -- instead of admission naturally exhausting + whichever account's rows sort first. This guarantees every contending + account gets at least one turn before any account gets a second + admission from that domain, so the domain's cap is filled + proportionally across its accounts rather than by whichever one + happens to rank first within the tier; an account that runs out of rows + before the cap is reached simply stops participating in further + rounds, letting the domain's remaining accounts absorb the leftover + capacity. Crucially, a shared domain's own rows keep the exact global + positions they already occupied among ``rows`` -- round-robin only + decides which of the domain's own rows lands in which of its own + positions, never how far ahead or behind an unrelated domain's row + sits (see :func:`_fair_order_within_tier`'s docstring for the concrete + bug an earlier revision had here: collapsing a shared domain into one + contiguous block at its first appearance silently displaced an + unrelated domain's row that had been priority-ranked between two of + the shared domain's own occurrences). + """ + ordered: list[Mapping[str, Any]] = [] + tier_start = 0 + total = len(rows) + while tier_start < total: + tier = _admission_priority_key(rows[tier_start], zdr_endpoints=zdr_endpoints)[:2] + tier_end = tier_start + 1 + while ( + tier_end < total + and _admission_priority_key(rows[tier_end], zdr_endpoints=zdr_endpoints)[:2] + == tier + ): + tier_end += 1 + ordered.extend(_fair_order_within_tier(rows[tier_start:tier_end])) + tier_start = tier_end + return ordered + + +def _fair_order_within_tier( + rows: list[Mapping[str, Any]], +) -> list[Mapping[str, Any]]: + """Round-robin one already-single-tier run of rows across shared-domain accounts. + + See :func:`_fair_admission_order`'s docstring for why fairness must stay + scoped to one admission-priority tier at a time; this is that per-tier + reordering step, factored out so it never has visibility into rows from + a different tier to (mis)order against. + + This never collapses a domain's rows into one contiguous block. An + earlier revision grouped every row for one domain at that domain's + *first* appearance in ``rows``, which silently moved rows belonging to + *other* domains whenever a shared domain's own rows were not already + contiguous in the input: e.g. ``[A1, B1, A2]`` (domain A shared by two + accounts, with an unrelated domain B's row ranked between A's two + occurrences) became ``[A1, A2, B1]`` under that revision -- B1, an + independent domain's row that had outranked A2, was pushed behind + *both* of A's rows, which could drop B1 entirely under a tight + admission limit even though it was priority-ranked ahead of A2. + Instead, each domain keeps exactly the global positions its own rows + already occupy (recorded in ``domain_positions`` below); round-robining + a shared domain's accounts only decides which of *that domain's own* + rows fills each of its own positions, so a shared domain's Nth admitted + row can only displace what its own Nth-occurrence priority position + would have displaced, never a different domain's row. + """ + domain_positions: dict[str, list[int]] = {} + for index, row in enumerate(rows): + domain_positions.setdefault(_outage_domain(row), []).append(index) + + ordered: list[Mapping[str, Any]] = list(rows) + for positions in domain_positions.values(): + bucket = [rows[index] for index in positions] + account_order: list[str] = [] + queues: dict[str, deque[Mapping[str, Any]]] = {} + for row in bucket: + account = provider_account(str(row["provider"])) + if account not in queues: + account_order.append(account) + queues[account] = deque() + queues[account].append(row) + if len(account_order) <= 1: + continue + reordered: list[Mapping[str, Any]] = [] + while any(queues[account] for account in account_order): + for account in account_order: + queue = queues[account] + if queue: + reordered.append(queue.popleft()) + for index, row in zip(positions, reordered): + ordered[index] = row + return ordered + + def _normalize_agent_id(candidate: str, provider_name: str) -> str: """Return a two-or-more-word snake_case agent identifier.""" slug = re.sub(r"[^a-zA-Z0-9]+", "_", candidate).strip("_").lower() @@ -197,21 +486,80 @@ def build_zdr_prioritized_catalog( zdr_endpoints: frozenset[str] = frozenset(), require_zdr: bool = False, pool: str = "free", + guarantee_domain_coverage: bool = False, ) -> dict[str, Any]: - """Select a free-first, ZDR-aware, credential-account-diverse catalog. - - The returned report's ``free_account_diversity`` counts the distinct - credential accounts among *all* discovered free routes, independent of - ``pool`` or the per-account selection cap. Vendor identity is not model - equivalence; only an explicit contextual-orchestrator ``model_group`` may - share routing evidence across routes. + """Select a free-first, ZDR-aware, outage-domain-diverse catalog. + + The returned report carries two distinct diversity/admission signals, + deliberately kept separate (see :func:`_outage_domain`'s docstring for + the full rationale): + + - ``free_account_diversity`` counts the distinct credential accounts + (:func:`provider_account`) among *all* discovered free routes. Vendor + identity is not model equivalence -- ``nvidia_nim`` and + ``nvidia_nim_sub`` are independent here, since either may be entitled + to a different model catalog; only an explicit contextual-orchestrator + ``model_group`` may share routing evidence across routes. + - ``free_outage_domain_diversity`` counts the distinct shared- + infrastructure outage domains (:func:`_outage_domain`, keyed on each + row's own ``base_url``) among the same routes. ``nvidia_nim`` and + ``nvidia_nim_sub`` collapse to *one* domain here, since both resolve to + the identical upstream endpoint -- a caller deciding whether it is + safe to rely on a strict, fail-closed ``orchestrator/free`` pool + without an ``orchestrator/auto`` paid-route safety net (the actual + question ADR-0003 raised) should require at least two here, not on + ``free_account_diversity``: one shared endpoint's outage can empty the + free catalog even when two independent credentials both point at it. + + Both are computed independent of ``pool`` or the per-domain admission + cap below. The admission cap itself (``account_cap`` -- the name + predates this fix and is kept for CLI/environment stability, but its + grouping is by outage domain, matching the cap's original purpose: + preventing one physical endpoint from absorbing the bounded catalog, the + confirmed root cause of a real 2026-08-30 ``orchestrator/free`` + exhaustion incident recorded in ``docs/product-technical-gap- + baseline.md``) admits at most ``account_cap`` rows per outage domain, + not per credential -- two same-endpoint credentials share one cap + budget, they do not each get their own. This counts routes discovery reports as free, not routes runtime - preflight has confirmed are actually serving requests: a value of two or - more is evidence that one account failure cannot immediately empty the free - catalog, not proof that either account is presently reachable. A caller - needing readiness, not just discovery-time diversity, must combine this - with the runtime preflight report the sidecar already produces. + preflight has confirmed are actually serving requests: a + ``free_outage_domain_diversity`` of two or more is evidence that one + endpoint's outage cannot immediately empty the free catalog, not proof + that either domain is presently reachable. A caller needing readiness, + not just discovery-time diversity, must combine this with the runtime + preflight report the sidecar already produces. + + ``guarantee_domain_coverage`` (default ``False``, preserving every + existing caller's behavior unchanged) fixes a narrower gap a uniform + ``account_cap`` cannot: when ``limit`` is small relative to the number + of competing outage domains -- the review sidecar's priced-fallback + stage's own real shape, where ``limit`` and ``account_cap`` can both be + 4 -- a single scalar cap forces an uncomfortable choice between two + failure modes. A cap left at ``account_cap`` lets one dominant domain + exhaust ``limit`` before a second domain is ever considered (Devin + Review: "fallback remains single-domain"). Shrinking the cap to + ``limit // domain_count`` fixes that but wastes admittable capacity + whenever ``limit`` does not divide evenly (Devin Review, same PR: + "fallback quota wastes probe slots" -- concretely, ``limit=4`` across 3 + domains admits only 3 routes under a uniform floor of 1, even though a + 4th eligible row exists in one of those domains). When set, admission + runs in two passes instead of one: the first pass admits at most one + row per outage domain (bounded by ``account_cap`` and ``limit``, in the + same priority order the single-pass loop already uses), guaranteeing + every domain with an eligible row is represented before any domain + claims a second seat; the second pass then fills any remaining + ``limit`` budget from the rows the first pass did not pick, still + respecting each domain's ``account_cap`` ceiling (inclusive of what the + first pass already gave it), from whichever domain's next-highest- + priority row comes first -- so the full budget is used whenever enough + eligible rows exist anywhere, not artificially left idle. The picked + order places every first-pass (diversity) row ahead of every + second-pass (fill) row: for a fallback pool whose entire purpose is + outage-domain resilience, trying one candidate from each domain before + a second candidate from an already-represented domain is the more + useful preflight order, not merely a side effect of the two-pass + implementation. """ if pool not in {"free", "auto"}: raise PolicyError(f"unsupported review pool {pool!r}") @@ -234,30 +582,43 @@ def build_zdr_prioritized_catalog( ) ] eligible_rows.sort( - key=lambda row: ( - _COST_EVIDENCE_RANK[_cost_evidence(row)], - 0 - if is_zdr_model( - str(row["provider"]), - model=str(row["model"]), - zdr_endpoints=zdr_endpoints, - ) - else 1, - str(row["provider"]), - str(row["model"]), - ) + key=lambda row: _admission_priority_key(row, zdr_endpoints=zdr_endpoints) ) - per_account: Counter[str] = Counter() + per_domain: Counter[str] = Counter() picked: list[Mapping[str, Any]] = [] - for row in eligible_rows: - account = provider_account(str(row["provider"])) - if per_account[account] >= account_cap: - continue - per_account[account] += 1 - picked.append(row) - if len(picked) >= limit: - break + ordered_rows = _fair_admission_order(eligible_rows, zdr_endpoints=zdr_endpoints) + if guarantee_domain_coverage: + covered_domains: set[str] = set() + for row in ordered_rows: + if len(picked) >= limit: + break + domain = _outage_domain(row) + if domain in covered_domains or per_domain[domain] >= account_cap: + continue + covered_domains.add(domain) + per_domain[domain] += 1 + picked.append(row) + first_pass_ids = {id(row) for row in picked} + for row in ordered_rows: + if len(picked) >= limit: + break + if id(row) in first_pass_ids: + continue + domain = _outage_domain(row) + if per_domain[domain] >= account_cap: + continue + per_domain[domain] += 1 + picked.append(row) + else: + for row in ordered_rows: + domain = _outage_domain(row) + if per_domain[domain] >= account_cap: + continue + per_domain[domain] += 1 + picked.append(row) + if len(picked) >= limit: + break if not picked: route_kind = "attested ZDR" if require_zdr else pool @@ -304,6 +665,9 @@ def build_zdr_prioritized_catalog( free_account_diversity = len( {provider_account(str(row["provider"])) for row in all_free_rows} ) + free_outage_domain_diversity = len( + {_outage_domain(row) for row in all_free_rows} + ) selected_evidence = [_cost_evidence(row) for row in picked] return { @@ -315,6 +679,7 @@ def build_zdr_prioritized_catalog( "total_priced_routes": len(all_priced_rows), "total_unknown_routes": len(all_unknown_rows), "free_account_diversity": free_account_diversity, + "free_outage_domain_diversity": free_outage_domain_diversity, "zdr_required": require_zdr, "selected_count": len(catalog_rows), "free_selected_count": selected_evidence.count(COST_FREE), diff --git a/tests/test_contextual_orchestrator_review_policy.py b/tests/test_contextual_orchestrator_review_policy.py index b10fc4a0b..adde3d063 100644 --- a/tests/test_contextual_orchestrator_review_policy.py +++ b/tests/test_contextual_orchestrator_review_policy.py @@ -75,6 +75,361 @@ def test_provider_account_keeps_nvidia_keys_independent() -> None: assert policy.provider_account("openai") == "openai" +def test_outage_domain_groups_by_shared_base_url() -> None: + """Outage domain is keyed on a row's own base_url, not its provider name.""" + assert policy._outage_domain( + {"base_url": "https://integrate.api.nvidia.com/v1"} + ) == policy._outage_domain({"base_url": "https://integrate.api.nvidia.com/v1"}) + assert policy._outage_domain( + {"base_url": "https://api.openai.com/v1"} + ) != policy._outage_domain({"base_url": "https://integrate.api.nvidia.com/v1"}) + + +@pytest.mark.parametrize( + ("base_url", "equivalent_to"), + [ + ("HTTPS://Integrate.API.Nvidia.COM/v1", "https://integrate.api.nvidia.com/v1"), + ("https://integrate.api.nvidia.com:443/v1", "https://integrate.api.nvidia.com/v1"), + ("https://integrate.api.nvidia.com/v1/", "https://integrate.api.nvidia.com/v1"), + ("https://integrate.api.nvidia.com/v1//", "https://integrate.api.nvidia.com/v1"), + ("https://integrate.api.nvidia.com/v1#fragment", "https://integrate.api.nvidia.com/v1"), + ( + "https://integrate.api.nvidia.com/v1#fragment-a", + "https://integrate.api.nvidia.com/v1#fragment-b", + ), + ], +) +def test_normalize_base_url_treats_equivalent_spellings_as_one_domain( + base_url: str, equivalent_to: str +) -> None: + """Case, an explicit default port, a trailing slash, or a fragment don't split a domain. + + Regression for a Devin Review finding on this fix: comparing raw + ``base_url`` strings would let a hostname-case difference, an explicit + ``:443``, or a trailing slash split one physical endpoint into two + outage domains by formatting accident alone -- silently reintroducing + the diversity-overstating, cap-bypassing bug this module exists to fix, + for exactly the ``nvidia_nim``/``nvidia_nim_sub`` pair it was written to + protect. + + The two fragment cases are a second, later Devin Review finding: a URL + fragment is client-side only and never reaches the server, so it cannot + identify a different upstream endpoint -- two base URLs differing only + by fragment (including one with no fragment at all against one that has + one) must still normalize to the identical outage-domain key. + """ + assert policy._normalize_base_url(base_url) == policy._normalize_base_url(equivalent_to) + + +@pytest.mark.parametrize( + ("base_url", "distinct_from"), + [ + ("https://integrate.api.nvidia.com/v1", "https://api.openai.com/v1"), + ("https://integrate.api.nvidia.com:8443/v1", "https://integrate.api.nvidia.com/v1"), + ("https://integrate.api.nvidia.com/v2", "https://integrate.api.nvidia.com/v1"), + ("http://integrate.api.nvidia.com/v1", "https://integrate.api.nvidia.com/v1"), + ("https://integrate.api.nvidia.com/v1?tenant=a", "https://integrate.api.nvidia.com/v1"), + ], +) +def test_normalize_base_url_preserves_genuine_distinctions( + base_url: str, distinct_from: str +) -> None: + """A different host, non-default port, path, scheme, or query stays a different domain. + + The query-string case guards the fragment fix's scope: only the + fragment is dropped, the query string stays a real distinguishing + component (see :func:`_normalize_base_url`'s docstring). + """ + assert policy._normalize_base_url(base_url) != policy._normalize_base_url(distinct_from) + + +def test_normalize_base_url_falls_back_on_unparseable_input() -> None: + """A hostless or malformed-port URL groups by a stripped, lowercased copy. + + Never raises: this function only needs equal inputs to compare equal, + not a validated URL, since it groups audit evidence, not user input that + must be rejected. + """ + assert policy._normalize_base_url("") == policy._normalize_base_url("") + assert policy._normalize_base_url(" NOT-A-URL ") == policy._normalize_base_url("not-a-url") + assert policy._normalize_base_url( + "https://host:notaport/v1" + ) == policy._normalize_base_url("HTTPS://HOST:NOTAPORT/v1") + + +def test_normalize_base_url_falls_back_on_malformed_ipv6_bracket() -> None: + """An unmatched IPv6 bracket cannot raise past this function. + + Regression for a Devin Review finding: ``urlsplit()`` itself raises + ``ValueError`` for an unmatched ``[``/``]`` (e.g. ``https://[::1/v1``, + a missing closing bracket) -- before any scheme/host/port is even + available to inspect, so the earlier fallback (which only wrapped the + ``.port`` property access) did not cover it. + """ + # Would raise ValueError: Invalid IPv6 URL if urlsplit() itself were not + # also wrapped. + assert policy._normalize_base_url("https://[::1/v1") == "https://[::1/v1" + assert policy._normalize_base_url("HTTPS://[::1/V1") == policy._normalize_base_url( + "https://[::1/v1" + ) + + +def test_normalize_base_url_distinguishes_ipv6_port_from_literal_colon_digits() -> None: + """An IPv6 host:port pair and a differently-shaped literal stay distinct. + + Regression for a Devin Review finding: ``urlsplit().hostname`` strips + IPv6 literal brackets (``[::1]`` -> ``::1``), so appending a port + without re-adding them collapsed ``https://[::1]:8443/v1`` (host + ``::1``, port ``8443``) and ``https://[::1:8443]/v1`` (one IPv6 + literal, ``::1:8443``, with no separate port at all) to the identical + ``::1:8443`` string -- two different addresses undercounted as one + outage domain. + """ + explicit_port = policy._normalize_base_url("https://[::1]:8443/v1") + literal_colon_digits = policy._normalize_base_url("https://[::1:8443]/v1") + assert explicit_port != literal_colon_digits + # Both stay valid, bracketed netloc syntax, not the pre-fix bare form. + assert explicit_port == "https://[::1]:8443/v1" + assert literal_colon_digits == "https://[::1:8443]/v1" + + +def test_normalize_base_url_drops_default_port_for_ipv6_host() -> None: + """An explicit default port on an IPv6 host is still dropped, brackets intact.""" + assert policy._normalize_base_url( + "https://[::1]:443/v1" + ) == policy._normalize_base_url("https://[::1]/v1") + + +def test_outage_domain_uses_normalized_base_url() -> None: + """Two rows spelling one endpoint differently share one outage domain.""" + assert policy._outage_domain( + {"base_url": "https://integrate.api.nvidia.com/v1"} + ) == policy._outage_domain({"base_url": "https://Integrate.API.Nvidia.com:443/v1/"}) + + +def _row( + provider: str, model: str, *, cost_evidence: str = policy.COST_UNKNOWN +) -> dict[str, object]: + """Return a minimal normalized-shaped row for ``_fair_admission_order`` tests.""" + return { + "provider": provider, + "model": model, + "base_url": policy.PROVIDER_BASE_URLS[provider], + "cost_evidence": cost_evidence, + } + + +def test_fair_admission_order_untouched_for_single_account_domains() -> None: + """A domain with only one contributing account keeps its original order.""" + rows = [_row("openrouter", "a"), _row("openai", "b"), _row("bytez", "c")] + assert policy._fair_admission_order(rows, zdr_endpoints=frozenset()) == rows + + +def test_fair_admission_order_round_robins_a_shared_domain() -> None: + """Two accounts sharing a domain alternate instead of one exhausting first. + + Regression for the same Devin Review finding as + ``test_build_catalog_shared_domain_cap_does_not_starve_second_account``, + exercised directly against the reordering helper: unit-level coverage of + exactly which row is emitted in which position, not just the resulting + admission counts. + """ + rows = [ + _row("nvidia_nim", "m0"), + _row("nvidia_nim", "m1"), + _row("nvidia_nim", "m2"), + _row("nvidia_nim_sub", "s0"), + _row("nvidia_nim_sub", "s1"), + ] + ordered = policy._fair_admission_order(rows, zdr_endpoints=frozenset()) + assert [(row["provider"], row["model"]) for row in ordered] == [ + ("nvidia_nim", "m0"), + ("nvidia_nim_sub", "s0"), + ("nvidia_nim", "m1"), + ("nvidia_nim_sub", "s1"), + ("nvidia_nim", "m2"), + ] + + +def test_fair_admission_order_never_moves_a_row_across_priority_tiers() -> None: + """Fairness reordering never lets a worse-tier row outrank a better-tier one. + + Regression for a real correctness bug a Devin Review finding caught: an + earlier revision of ``_fair_admission_order`` grouped every row for one + outage domain into a single block at that domain's first appearance, + *regardless of tier* -- so a lower-priority row (here, priced OpenAI) + sharing a domain with a higher-priority row (free OpenAI) could get + dragged ahead of a higher-priority row from a *different* domain (free + OpenRouter) that happened to sort later only because of the + ``(provider, model)`` tie-break. Concretely: sorted input + ``[free OpenAI, free OpenRouter, priced OpenAI]`` must stay in that + exact order -- the free OpenRouter row must never be pushed behind the + priced OpenAI row merely because OpenAI's two rows share a domain. + """ + rows = [ + _row("openai", "free-model", cost_evidence=policy.COST_FREE), + _row("openrouter", "free-model", cost_evidence=policy.COST_FREE), + _row("openai", "priced-model", cost_evidence=policy.COST_PRICED), + ] + ordered = policy._fair_admission_order(rows, zdr_endpoints=frozenset()) + assert [(row["provider"], row["model"]) for row in ordered] == [ + ("openai", "free-model"), + ("openrouter", "free-model"), + ("openai", "priced-model"), + ] + + +def test_build_catalog_never_admits_a_priced_route_over_a_free_one_from_another_domain() -> None: + """End-to-end: a tight limit must never drop a free route for a paid one. + + Same Devin Review finding as ``test_fair_admission_order_never_moves_a_ + row_across_priority_tiers``, exercised through the full public API + rather than the internal reordering helper directly. + """ + report = { + "models": [ + { + "provider": "openai", + "model": "free-model", + "agent_id": "oa_free", + "is_free": True, + **FREE_PRICE, + }, + { + "provider": "openai", + "model": "priced-model", + "agent_id": "oa_priced", + "is_free": False, + "prompt_price_per_1k": 0.002, + "completion_price_per_1k": 0.008, + "currency_code": "USD", + }, + { + "provider": "openrouter", + "model": "free-model", + "agent_id": "or_free", + "is_free": True, + **FREE_PRICE, + }, + ] + } + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(report), limit=2, account_cap=4, pool="auto" + ) + assert [agent["model"] for agent in result["agents"]] == ["free-model", "free-model"] + assert [agent["provider_name"] for agent in result["agents"]] == ["openai", "openrouter"] + + +def test_fair_admission_order_preserves_domain_position_and_multiple_domains() -> None: + """Reordering stays local to each multi-account domain, in its original slot. + + A single-account domain on either side of a multi-account domain stays + exactly where it was, untouched; the multi-account domain's block still + starts where its first row originally appeared, with only its internal + order changed (``nvidia_nim``'s two consecutive rows are pulled apart to + give ``nvidia_nim_sub`` a turn between them, rather than staying + adjacent). + """ + rows = [ + _row("bytez", "b0"), + _row("nvidia_nim", "m0"), + _row("nvidia_nim", "m1"), + _row("nvidia_nim_sub", "s0"), + _row("openrouter", "r0"), + ] + ordered = policy._fair_admission_order(rows, zdr_endpoints=frozenset()) + assert [(row["provider"], row["model"]) for row in ordered] == [ + ("bytez", "b0"), + ("nvidia_nim", "m0"), + ("nvidia_nim_sub", "s0"), + ("nvidia_nim", "m1"), + ("openrouter", "r0"), + ] + + +def test_fair_admission_order_preserves_non_contiguous_shared_domain_positions() -> None: + """A shared domain's own rows never displace an interleaved independent row. + + Regression for a Devin Review finding: an earlier revision collapsed a + domain to one contiguous block at that domain's *first* appearance, + which was wrong whenever the domain's own rows were not already + contiguous in priority order. Here domain A (shared by two accounts) + contributes ``A1`` and ``A2``, with an unrelated domain B's ``B1`` + priority-ranked between them: ``[A1, B1, A2]``. The old code produced + ``[A1, A2, B1]`` -- B1, which had outranked A2, got pushed behind + *both* of A's rows. The fix must preserve B1's original slot between + A1 and A2. + """ + rows = [ + _row("nvidia_nim", "m0"), + _row("bytez", "b0"), + _row("nvidia_nim_sub", "s0"), + ] + ordered = policy._fair_admission_order(rows, zdr_endpoints=frozenset()) + assert [(row["provider"], row["model"]) for row in ordered] == [ + ("nvidia_nim", "m0"), + ("bytez", "b0"), + ("nvidia_nim_sub", "s0"), + ] + + +def test_build_catalog_does_not_drop_an_interleaved_independent_route_under_a_tight_limit() -> None: + """A tight global limit must not drop an independent-domain route. + + End-to-end regression for the same Devin Review finding as + ``test_fair_admission_order_preserves_non_contiguous_shared_domain_ + positions``, exercised through the full public API with a real, + sort-derived priority order rather than a hand-fed one. + + ``nvidia_nim`` and ``nvidia_nim_sub`` are this codebase's only shared + outage domain (both resolve to ``https://integrate.api.nvidia.com/v1``), + and no third registered provider name sorts alphabetically between them + -- so to reach a genuinely *sort-derived* interleaved order (not just a + hand-fed one) this reuses the ``nvidia_nim`` credential for a second row + with an explicit ``base_url`` override pointing at an unrelated, + independent endpoint. Account identity and outage-domain identity are + deliberately decoupled by this module's own design (see + ``_outage_domain``'s docstring), so one credential's discovery rows + spanning two different base URLs is a legitimate shape, not a + contrivance. Choosing a model name (``"z-indep"``) that sorts after + ``"m0"`` places the independent row's priority rank between the shared + domain's ``nvidia_nim`` and ``nvidia_nim_sub`` rows once + ``build_zdr_prioritized_catalog`` sorts by ``_admission_priority_key``. + """ + report = { + "models": [ + { + "provider": "nvidia_nim", + "model": "m0", + "agent_id": "nim_m0", + "is_free": True, + **FREE_PRICE, + }, + { + "provider": "nvidia_nim", + "model": "z-indep", + "agent_id": "nim_indep", + "is_free": True, + "base_url": "https://independent.example.com/v1", + **FREE_PRICE, + }, + { + "provider": "nvidia_nim_sub", + "model": "s0", + "agent_id": "nimsub_s0", + "is_free": True, + **FREE_PRICE, + }, + ] + } + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(report), limit=2, account_cap=4 + ) + admitted = {(agent["provider_name"], agent["model"]) for agent in result["agents"]} + assert ("nvidia_nim", "z-indep") in admitted + assert len(result["agents"]) == 2 + + @pytest.mark.parametrize( ("candidate", "provider", "expected"), [ @@ -275,7 +630,15 @@ def test_build_auto_catalog_keeps_private_targets_zdr_only() -> None: def test_build_catalog_reports_free_account_diversity() -> None: - """Diversity counts independently credentialed accounts with free routes.""" + """Diversity counts independently credentialed accounts with free routes. + + ``free_outage_domain_diversity`` is one lower than ``free_account_ + diversity`` here: ``nvidia_nim`` and ``nvidia_nim_sub`` are two + independent accounts (see ``test_build_catalog_counts_same_vendor_ + credentials_independently``) but share one physical upstream endpoint, + so they collapse to a single outage domain while the other three + providers (openrouter, openai, bytez) each keep their own. + """ result = policy.build_zdr_prioritized_catalog( policy.parse_discovery_report(_report()), limit=12, @@ -283,10 +646,24 @@ def test_build_catalog_reports_free_account_diversity() -> None: zdr_endpoints=ZDR_FEED, ) assert result["report"]["free_account_diversity"] == 5 + assert result["report"]["free_outage_domain_diversity"] == 4 def test_build_catalog_counts_same_vendor_credentials_independently() -> None: - """Same-vendor credentials remain distinct discovery accounts.""" + """Same-vendor credentials remain distinct discovery accounts. + + But they are *not* automatically distinct outage domains: + ``free_outage_domain_diversity`` reports 1 here, not 2, because both + rows' ``base_url`` (via ``PROVIDER_BASE_URLS``) resolve to the identical + ``https://integrate.api.nvidia.com/v1`` upstream. Regression for a real, + separate bug found by review during this session: #941/#945/#1468 + correctly stopped assuming these two credentials share a *model + catalog*, but a caller deciding whether a single physical outage could + empty the free catalog (e.g. open PR #1437's Strix ``orchestrator/free`` + eligibility gate) needs the outage-domain count, not the account count + -- conflating the two would let this exact pair report a falsely safe + diversity of 2 for that specific decision. + """ single_family_report = { "models": [ { @@ -311,6 +688,92 @@ def test_build_catalog_counts_same_vendor_credentials_independently() -> None: account_cap=4, ) assert result["report"]["free_account_diversity"] == 2 + assert result["report"]["free_outage_domain_diversity"] == 1 + + +def test_build_catalog_collapses_differently_spelled_equivalent_endpoints() -> None: + """A hostname-case/port/slash spelling difference cannot split one domain. + + End-to-end regression for the same Devin Review finding as + ``test_normalize_base_url_treats_equivalent_spellings_as_one_domain``, + exercised through ``parse_discovery_report``'s ``base_url`` override + (the field a discovery report -- including this script's own + ``--discovery-report`` CLI input, not only the sidecar's exact + generation path -- may supply explicitly) rather than the unit-level + helper directly. + """ + differently_spelled_report = { + "models": [ + { + "provider": "nvidia_nim", + "model": "nvidia/nemotron-3-nano-30b-a3b", + "agent_id": "nim_nano_free", + "is_free": True, + "base_url": "https://integrate.api.nvidia.com/v1", + **FREE_PRICE, + }, + { + "provider": "nvidia_nim_sub", + "model": "meta/llama-3.3-70b-instruct", + "agent_id": "nimsec_70b", + "is_free": True, + "base_url": "HTTPS://Integrate.API.Nvidia.com:443/v1/", + **FREE_PRICE, + }, + ] + } + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(differently_spelled_report), + limit=12, + account_cap=1, + ) + assert result["report"]["free_outage_domain_diversity"] == 1 + # The shared domain's cap of 1 admits only the first-sorted row, not one + # from each differently-spelled row. + assert len(result["agents"]) == 1 + + +def test_build_catalog_collapses_fragment_only_difference() -> None: + """A fragment-only spelling difference cannot split one domain. + + End-to-end regression for a Devin Review finding: a URL fragment is + client-side only and is never sent to the server, so it cannot + legitimately identify a different upstream endpoint. Two base URLs + differing only by fragment must still share one + ``free_outage_domain_diversity`` count and one admission-cap budget, + exercised through ``parse_discovery_report``'s ``base_url`` override + the same way as + ``test_build_catalog_collapses_differently_spelled_equivalent_endpoints``. + """ + fragment_only_report = { + "models": [ + { + "provider": "nvidia_nim", + "model": "nvidia/nemotron-3-nano-30b-a3b", + "agent_id": "nim_nano_free", + "is_free": True, + "base_url": "https://integrate.api.nvidia.com/v1#primary", + **FREE_PRICE, + }, + { + "provider": "nvidia_nim_sub", + "model": "meta/llama-3.3-70b-instruct", + "agent_id": "nimsec_70b", + "is_free": True, + "base_url": "https://integrate.api.nvidia.com/v1#secondary", + **FREE_PRICE, + }, + ] + } + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(fragment_only_report), + limit=12, + account_cap=1, + ) + assert result["report"]["free_outage_domain_diversity"] == 1 + # The shared domain's cap of 1 admits only the first-sorted row, not one + # from each differently-fragmented row. + assert len(result["agents"]) == 1 def test_build_catalog_rejects_unknown_pool() -> None: @@ -337,7 +800,21 @@ def test_build_catalog_assigns_unique_priorities() -> None: def test_build_catalog_applies_account_cap() -> None: - """An account cap keeps one credential from absorbing the pool.""" + """The admission cap is enforced per outage domain, split fairly within it. + + ``nvidia_nim`` and ``nvidia_nim_sub`` share one outage domain (both + ``https://integrate.api.nvidia.com/v1``), so they share one ``2``-slot + cap budget here rather than each getting their own -- with ``account_cap`` + still named for the credential-account concept it started as, but its + grouping fixed to outage domains (see ``test_build_catalog_prevents_ + shared_endpoint_from_crowding_out_independent_providers`` for the + concrete crowding-out scenario this exists to prevent). The shared + budget is split round-robin across the domain's accounts (see + ``test_build_catalog_shared_domain_cap_does_not_starve_second_account``), + not consumed entirely by whichever one sorts first: one slot each for + ``nvidia_nim``/``nvidia_nim_sub`` here, not two for one and zero for the + other. + """ report = { "models": [ {"provider": "nvidia_nim", "model": f"m{i}", "agent_id": f"nim_a{i}", "is_free": True, **FREE_PRICE} @@ -365,9 +842,276 @@ def test_build_catalog_applies_account_cap() -> None: for agent in result["agents"]: account = policy.provider_account(agent["provider_name"]) account_counts[account] = account_counts.get(account, 0) + 1 - assert account_counts["nvidia_nim"] == 2 - assert account_counts["nvidia_nim_sub"] == 2 - assert account_counts["openai"] == 2 + assert account_counts == {"nvidia_nim": 1, "nvidia_nim_sub": 1, "openai": 2} + assert len(result["agents"]) == 4 + + +def test_build_catalog_prevents_shared_endpoint_from_crowding_out_independent_providers() -> None: + """A shared-endpoint credential pair cannot out-compete independent providers. + + Regression for a real, still-open gap this session's own review found in + the already-merged #1468 fix: #1468 correctly stopped treating + ``nvidia_nim``/``nvidia_nim_sub`` as one *model-catalog* family, but in + doing so also let the admission cap treat them as two fully independent + *accounts* -- meaning the two credentials could jointly consume up to + ``2 * account_cap`` catalog slots, all from one physical endpoint, + crowding out a genuinely independent provider (``openrouter`` here) even + though it has its own free routes available. With the cap correctly + grouped by outage domain instead, the two NVIDIA credentials share one + domain's cap budget and cannot jointly exceed it. + """ + report = { + "models": [ + {"provider": "bytez", "model": f"b{i}", "agent_id": f"bytez_{i}", "is_free": True, **FREE_PRICE} + for i in range(2) + ] + + [ + {"provider": "nvidia_nim", "model": f"n{i}", "agent_id": f"nim_{i}", "is_free": True, **FREE_PRICE} + for i in range(10) + ] + + [ + {"provider": "nvidia_nim_sub", "model": f"n{i}", "agent_id": f"nimsub_{i}", "is_free": True, **FREE_PRICE} + for i in range(10) + ] + + [ + {"provider": "openrouter", "model": f"r{i}", "agent_id": f"or_{i}", "is_free": True, **FREE_PRICE} + for i in range(2) + ] + } + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(report), limit=20, account_cap=4 + ) + counts: dict[str, int] = {} + for agent in result["agents"]: + counts[agent["provider_name"]] = counts.get(agent["provider_name"], 0) + 1 + # NVIDIA's shared domain admits at most 4 total, split fairly (2 from + # each credential, not 4 from whichever sorts first and 0 from the + # other) -- leaving bytez and openrouter, each an independent domain, + # fully admitted. + assert counts == {"bytez": 2, "nvidia_nim": 2, "nvidia_nim_sub": 2, "openrouter": 2} + assert result["report"]["free_account_diversity"] == 4 + assert result["report"]["free_outage_domain_diversity"] == 3 + + +def test_build_catalog_shared_domain_cap_does_not_starve_second_account() -> None: + """A shared domain's cap admits from every contending account, not just one. + + Regression for a Devin Review finding on this fix: the admission loop + walks rows in strict sorted (cost-tier, ZDR, provider, model) order, so + grouping the cap by outage domain alone was not enough -- whichever + account's rows happened to sort first (``nvidia_nim`` before + ``nvidia_nim_sub`` in every fixture here) could exhaust the *entire* + shared cap before the domain's other account was considered at all, a + narrower but just-as-real version of the crowding-out bug this file + already fixes across domains. With both credentials offering far more + rows than the shared cap, both must still contribute. + """ + report = { + "models": [ + {"provider": "nvidia_nim", "model": f"n{i}", "agent_id": f"nim_{i}", "is_free": True, **FREE_PRICE} + for i in range(8) + ] + + [ + {"provider": "nvidia_nim_sub", "model": f"n{i}", "agent_id": f"nimsub_{i}", "is_free": True, **FREE_PRICE} + for i in range(8) + ] + } + result = policy.build_zdr_prioritized_catalog( + policy.parse_discovery_report(report), limit=20, account_cap=4 + ) + counts: dict[str, int] = {} + for agent in result["agents"]: + counts[agent["provider_name"]] = counts.get(agent["provider_name"], 0) + 1 + assert counts == {"nvidia_nim": 2, "nvidia_nim_sub": 2} + assert sum(counts.values()) == 4 + + +PRICED_PRICE = { + "prompt_price_per_1k": 0.01, + "completion_price_per_1k": 0.01, + "currency_code": "USD", +} + + +def test_build_catalog_guarantee_domain_coverage_leaves_single_domain_unchanged() -> None: + """A single competing domain behaves exactly as the unmodified admission loop did.""" + report = { + "models": [ + {"provider": "openrouter", "model": f"r{i}", "agent_id": f"or_{i}", "is_free": False, **PRICED_PRICE} + for i in range(6) + ] + } + rows = policy.parse_discovery_report(report) + without_flag = policy.build_zdr_prioritized_catalog( + rows, limit=4, account_cap=4, pool="auto" + ) + with_flag = policy.build_zdr_prioritized_catalog( + rows, limit=4, account_cap=4, pool="auto", guarantee_domain_coverage=True + ) + assert len(without_flag["agents"]) == len(with_flag["agents"]) == 4 + + +def test_build_catalog_guarantee_domain_coverage_fixes_single_domain_starvation() -> None: + """Regression for Devin Review's "fallback remains single-domain" finding on `.github#1474`. + + With both ``limit`` and ``account_cap`` at 4 (the review sidecar's real + priced-fallback shape), an outage domain with at least ``limit`` priced + rows used to exhaust the whole stage before a second, genuinely + independent domain's row was ever considered -- the per-domain cap + provided no diversity protection for this specific stage. Four + same-domain priced routes plus one independent priced route (Devin's + own suggested regression shape) must now leave room for the + independent route. + """ + report = { + "models": [ + {"provider": "nvidia_nim", "model": f"n{i}", "agent_id": f"nim_{i}", "is_free": False, **PRICED_PRICE} + for i in range(4) + ] + + [{"provider": "openrouter", "model": "independent", "agent_id": "or_0", "is_free": False, **PRICED_PRICE}] + } + rows = policy.parse_discovery_report(report) + result = policy.build_zdr_prioritized_catalog( + rows, limit=4, account_cap=4, pool="auto", guarantee_domain_coverage=True + ) + providers = {agent["provider_name"] for agent in result["agents"]} + assert providers == {"nvidia_nim", "openrouter"} + + +def test_build_catalog_guarantee_domain_coverage_uses_full_budget_on_uneven_split() -> None: + """Regression for Devin Review's "fallback quota wastes probe slots" finding. + + A uniform ``limit // domain_count`` floor (this fix's own first + revision) correctly guarantees every domain a seat but wastes capacity + whenever ``limit`` does not divide evenly: ``limit=4`` across 3 domains + floors to 1 each, admitting only 3 routes even though a 4th eligible + row exists. Three domains (bytez, openrouter, openai), each with 2 + priced rows, ``limit=4``, ``account_cap=4``: every domain must still be + represented, and the full 4-route budget must be used, not left at 3. + """ + report = { + "models": [ + {"provider": provider, "model": f"{provider}-{i}", "agent_id": f"{provider}_{i}", "is_free": False, **PRICED_PRICE} + for provider in ("bytez", "openrouter", "openai") + for i in range(2) + ] + } + rows = policy.parse_discovery_report(report) + result = policy.build_zdr_prioritized_catalog( + rows, limit=4, account_cap=4, pool="auto", guarantee_domain_coverage=True + ) + providers = {agent["provider_name"] for agent in result["agents"]} + assert providers == {"bytez", "openrouter", "openai"} + assert len(result["agents"]) == 4 + + +def test_build_catalog_guarantee_domain_coverage_uses_full_budget_on_eight_over_three() -> None: + """Devin Review's own second suggested non-divisible split (8 routes, 3 domains). + + Three domains, each with ample priced rows (5 each -- comfortably above + both ``account_cap`` and any per-domain share of ``limit``), ``limit=8``, + ``account_cap=4``: every domain represented, the full 8-route budget + used, and no domain exceeds ``account_cap``. + """ + report = { + "models": [ + {"provider": provider, "model": f"{provider}-{i}", "agent_id": f"{provider}_{i}", "is_free": False, **PRICED_PRICE} + for provider in ("bytez", "openrouter", "openai") + for i in range(5) + ] + } + rows = policy.parse_discovery_report(report) + result = policy.build_zdr_prioritized_catalog( + rows, limit=8, account_cap=4, pool="auto", guarantee_domain_coverage=True + ) + counts: dict[str, int] = {} + for agent in result["agents"]: + counts[agent["provider_name"]] = counts.get(agent["provider_name"], 0) + 1 + assert set(counts) == {"bytez", "openrouter", "openai"} + assert sum(counts.values()) == 8 + assert all(count <= 4 for count in counts.values()) + + +def test_build_catalog_guarantee_domain_coverage_fixes_auto_primary_stage_too() -> None: + """Regression for Devin Review's "auto primary catalog remains single-domain" finding. + + The *primary* ``auto``-pool stage has the identical single-scalar-cap- + equals-limit coincidence the priced-fallback stage already had fixed -- + not the launcher's ``DEFAULT_ACCOUNT_CAP`` (4) it might appear to use + at a glance, but the review sidecar's own real deployed default: the + sidecar script exports ``ORCHESTRATOR_CATALOG_ACCOUNT_CAP=8`` (see + ``contextual_orchestrator_review_sidecar.sh``), and the launcher's + ``REVIEW_PREFLIGHT_PRIMARY_ROUTE_LIMIT`` is also 8 for the ``auto`` + pool's primary stage. Eight free routes from one dominant outage + domain, ``limit=8`` and ``account_cap=8`` (the real deployed values, + not this file's usual ``account_cap=4`` fixtures), used to exclude + every independent free domain entirely. + """ + report = { + "models": [ + {"provider": "nvidia_nim", "model": f"n{i}", "agent_id": f"nim_{i}", "is_free": True, **FREE_PRICE} + for i in range(8) + ] + + [{"provider": "openrouter", "model": "independent", "agent_id": "or_0", "is_free": True, **FREE_PRICE}] + } + rows = policy.parse_discovery_report(report) + result = policy.build_zdr_prioritized_catalog( + rows, limit=8, account_cap=8, pool="auto", guarantee_domain_coverage=True + ) + providers = {agent["provider_name"] for agent in result["agents"]} + assert providers == {"nvidia_nim", "openrouter"} + assert len(result["agents"]) == 8 + + +def test_build_catalog_guarantee_domain_coverage_still_bounded_by_account_cap() -> None: + """The first-pass diversity guarantee never lets a domain skip its own cap. + + A single domain with far more rows than ``account_cap`` must still stop + at ``account_cap``, exactly as the unmodified admission loop already + guarantees -- ``guarantee_domain_coverage`` only changes *when* other + domains get a turn, never the per-domain ceiling itself. + """ + report = { + "models": [ + {"provider": "openrouter", "model": f"r{i}", "agent_id": f"or_{i}", "is_free": False, **PRICED_PRICE} + for i in range(10) + ] + } + rows = policy.parse_discovery_report(report) + result = policy.build_zdr_prioritized_catalog( + rows, limit=8, account_cap=4, pool="auto", guarantee_domain_coverage=True + ) + assert len(result["agents"]) == 4 + + +def test_build_catalog_guarantee_domain_coverage_caps_at_limit_when_domains_outnumber_it() -> None: + """More competing domains than ``limit`` still stops exactly at ``limit``. + + Five independent single-account domains only exist in this fixture set + via distinct providers, but this codebase registers only five providers + total (see ``PROVIDER_BASE_URLS``); ``nvidia_nim``/``nvidia_nim_sub`` + share one domain, so the maximum distinct domains available is four. + With ``limit=3`` and four competing domains, full domain coverage is + structurally impossible -- the first admission pass itself must stop at + ``limit`` before every domain gets a turn, exercising that pass's own + ``len(picked) >= limit`` bound (never reached by the other + ``guarantee_domain_coverage`` tests, which all keep ``limit >= + domain_count``). Exactly ``limit`` routes are admitted, each from a + different domain. + """ + report = { + "models": [ + {"provider": provider, "model": f"{provider}-0", "agent_id": f"{provider}_0", "is_free": False, **PRICED_PRICE} + for provider in ("bytez", "nvidia_nim", "openrouter", "openai") + ] + } + rows = policy.parse_discovery_report(report) + result = policy.build_zdr_prioritized_catalog( + rows, limit=3, account_cap=4, pool="auto", guarantee_domain_coverage=True + ) + assert len(result["agents"]) == 3 + providers = {agent["provider_name"] for agent in result["agents"]} + assert len(providers) == 3 def test_build_catalog_respects_limit() -> None: diff --git a/tests/test_contextual_orchestrator_review_runtime_preflight.py b/tests/test_contextual_orchestrator_review_runtime_preflight.py index 7093e8a3d..d29bfd02e 100644 --- a/tests/test_contextual_orchestrator_review_runtime_preflight.py +++ b/tests/test_contextual_orchestrator_review_runtime_preflight.py @@ -1488,6 +1488,29 @@ def test_catalog_account_cap_honors_an_explicit_override( assert namespace["_catalog_account_cap"](policy.DEFAULT_ACCOUNT_CAP) == 6 +def test_main_wires_guarantee_domain_coverage_for_both_catalog_stages() -> None: + """``main()``'s primary and priced-fallback catalog calls both opt into coverage. + + Devin Review finding on `.github#1474`: the primary stage's own real + deployment shape has the identical single-scalar-cap-equals-limit + coincidence the fallback stage already had fixed -- the sidecar's own + default `ORCHESTRATOR_CATALOG_ACCOUNT_CAP` is 8 (not + `DEFAULT_ACCOUNT_CAP`'s library-level fallback of 4, which the sidecar + never leaves the env var unset for), and `REVIEW_PREFLIGHT_PRIMARY_ + ROUTE_LIMIT` is also 8 for the ``auto`` pool's primary stage. Both + ``build_zdr_prioritized_catalog`` call sites in ``main()`` must pass + ``guarantee_domain_coverage=True``. + """ + source = _LAUNCHER.read_text(encoding="utf-8") + assert source.count("guarantee_domain_coverage=True") == 2 + primary_call_start = source.index("result = build_zdr_prioritized_catalog(") + primary_call = source[primary_call_start : primary_call_start + 400] + assert "guarantee_domain_coverage=True" in primary_call + fallback_call_start = source.index('pool == "auto"\n and admitted_free_rows') + fallback_call = source[fallback_call_start : fallback_call_start + 800] + assert "guarantee_domain_coverage=True" in fallback_call + + def test_main_sources_the_account_cap_default_from_policy_not_a_magic_number() -> None: """``main()`` must wire the cap default from ``policy.DEFAULT_ACCOUNT_CAP``. @@ -1498,9 +1521,16 @@ def test_main_sources_the_account_cap_default_from_policy_not_a_magic_number() - source-level contract test pins both ``build_zdr_prioritized_catalog`` call sites in ``main()`` to the single source of truth and forbids the total-routes constant from ever reappearing as the account-cap fallback. + Both the primary-stage and priced-fallback call sites pass + ``_catalog_account_cap(DEFAULT_ACCOUNT_CAP)`` directly as + ``account_cap=``; the fallback call site additionally sets + ``guarantee_domain_coverage=True`` (see + ``policy.build_zdr_prioritized_catalog``'s own regression tests for the + domain-diversity fix that flag adds), which does not change what value + the cap itself is sourced from. """ source = _LAUNCHER.read_text(encoding="utf-8") - assert source.count("account_cap=_catalog_account_cap(DEFAULT_ACCOUNT_CAP)") == 2 + assert source.count("_catalog_account_cap(DEFAULT_ACCOUNT_CAP)") == 2 assert "ORCHESTRATOR_CATALOG_FAMILY_CAP" not in source assert 'os.environ.get("ORCHESTRATOR_CATALOG_ACCOUNT_CAP", "4")' not in source @@ -1531,19 +1561,23 @@ def test_discovery_counts_survive_stage_specific_policy_reports() -> None: namespace = _load_launcher() base = {"selected_count": 1, "selected": [{"model": "priced/model"}]} rows = [ - {"cost_evidence": "free", "provider": "nvidia_nim"}, - {"cost_evidence": "priced", "provider": "openai"}, - {"cost_evidence": "priced", "provider": "openai"}, - {"cost_evidence": "unknown", "provider": "bytez"}, + {"cost_evidence": "free", "provider": "nvidia_nim", "base_url": "https://integrate.api.nvidia.com/v1"}, + {"cost_evidence": "priced", "provider": "openai", "base_url": "https://api.openai.com/v1"}, + {"cost_evidence": "priced", "provider": "openai", "base_url": "https://api.openai.com/v1"}, + {"cost_evidence": "unknown", "provider": "bytez", "base_url": "https://api.bytez.com/models/v2/openai/v1"}, ] enriched = namespace["_with_discovery_counts"]( - base, rows, provider_account=policy.provider_account + base, + rows, + provider_account=policy.provider_account, + outage_domain=policy._outage_domain, ) assert base == {"selected_count": 1, "selected": [{"model": "priced/model"}]} assert [enriched[key] for key in ( "total_routes", "total_free_routes", "total_priced_routes", "total_unknown_routes" )] == [4, 1, 2, 1] assert enriched["free_account_diversity"] == 1 + assert enriched["free_outage_domain_diversity"] == 1 def test_discovery_counts_recompute_diversity_from_full_discovery_not_the_stage() -> None: @@ -1551,24 +1585,60 @@ def test_discovery_counts_recompute_diversity_from_full_discovery_not_the_stage( Regression for a real bug: the ``auto``-pool primary stage only sees ZDR-admitted free rows, and the priced-fallback stage sees no free rows - at all, so either stage's internally computed ``free_account_diversity`` + at all, so either stage's internally computed diversity fields (whatever ``build_zdr_prioritized_catalog`` returned from its own narrower input) would undercount or read zero even when the full - discovery has multiple credential accounts with free routes. + discovery has multiple credential accounts (and outage domains) with + free routes. """ namespace = _load_launcher() - stage_report_from_priced_only_rows = {"free_account_diversity": 0} + stage_report_from_priced_only_rows = { + "free_account_diversity": 0, + "free_outage_domain_diversity": 0, + } full_discovery_rows = [ - {"cost_evidence": "free", "provider": "nvidia_nim"}, - {"cost_evidence": "free", "provider": "openrouter"}, - {"cost_evidence": "priced", "provider": "openai"}, + {"cost_evidence": "free", "provider": "nvidia_nim", "base_url": "https://integrate.api.nvidia.com/v1"}, + {"cost_evidence": "free", "provider": "openrouter", "base_url": "https://openrouter.ai/api/v1"}, + {"cost_evidence": "priced", "provider": "openai", "base_url": "https://api.openai.com/v1"}, ] enriched = namespace["_with_discovery_counts"]( stage_report_from_priced_only_rows, full_discovery_rows, provider_account=policy.provider_account, + outage_domain=policy._outage_domain, + ) + assert enriched["free_account_diversity"] == 2 + assert enriched["free_outage_domain_diversity"] == 2 + + +def test_discovery_counts_distinguish_account_from_outage_domain_diversity() -> None: + """Two same-endpoint NVIDIA credentials are 2 accounts but 1 outage domain. + + Regression for a real, separate bug found by review during this + session: #1468 correctly stopped treating ``nvidia_nim``/ + ``nvidia_nim_sub`` as one *model-catalog* family (they are independent + credentials that may expose different models), but a naive read of that + fix could also wrongly assume they are two independent *outage domains* + -- they are not: both resolve to the identical + ``https://integrate.api.nvidia.com/v1`` upstream. If one physical + endpoint's outage were mistaken for two independent domains, a caller + gating on diversity (e.g. open PR #1437's Strix ``orchestrator/free`` + eligibility check) could wrongly conclude the free catalog can survive + that single outage. + """ + namespace = _load_launcher() + full_discovery_rows = [ + {"cost_evidence": "free", "provider": "nvidia_nim", "base_url": "https://integrate.api.nvidia.com/v1"}, + {"cost_evidence": "free", "provider": "nvidia_nim_sub", "base_url": "https://integrate.api.nvidia.com/v1"}, + ] + enriched = namespace["_with_discovery_counts"]( + {}, + full_discovery_rows, + provider_account=policy.provider_account, + outage_domain=policy._outage_domain, ) assert enriched["free_account_diversity"] == 2 + assert enriched["free_outage_domain_diversity"] == 1 def test_temporary_fallback_catalog_is_removed_after_loading(tmp_path: Path) -> None: diff --git a/tests/test_opencode_agent_contract.py b/tests/test_opencode_agent_contract.py index 79fdba39a..746b750c1 100644 --- a/tests/test_opencode_agent_contract.py +++ b/tests/test_opencode_agent_contract.py @@ -2660,6 +2660,10 @@ def test_opencode_privileged_review_security_boundaries_are_fail_closed(): '^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]]' ) in metadata_step assert '[ "$live_head_repository" != "$TARGET_REPOSITORY" ]' not in metadata_step + # #1533 briefly relaxed this to a warn-and-proceed check, but #1540 + # reverted it back to the original strict fail-closed equality (no + # rationale recorded beyond the revert itself) -- confirmed against + # main's actual current content, not assumed from the PR history. assert '[ "$SUPPLIED_HEAD_SHA" = "$live_head_sha" ]' in metadata_step assert ( 'live_visibility="$(jq -r \'.base.repo.visibility // empty | ascii_downcase\''