fix(ci): source review catalog account-cap default from policy, not a literal

[seonghobae]

Summary

Investigated a reported "빈 깡통 경로 너무 많다" (too many empty-can routes) bug: the
review sidecar's bounded preflight budget was allegedly being wasted because
the per-account/family diversity cap silently fell back to the total routes
budget instead of a real per-account cap, letting two rate-limited NVIDIA NIM
credentials sharing one upstream jointly occupy the whole 12-slot preflight
budget (a live run showed probed_count: 12, ready_count: 2, rejected_count: 10 — 83% rejected via 429/404/timeout).

What I found investigating on current main (this PR's actual scope):

• The two specifically-hypothesized root causes named in the original report
  (_catalog_family_cap() defaulting to REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES,
  and PR fix(ci): keep sidecar credential accounts independent #1468 not being landed) do not exist on main — #1468 is
  already merged, and main's build_zdr_prioritized_catalog call sites
  already default ORCHESTRATOR_CATALOG_ACCOUNT_CAP to "4", matching
  contextual_orchestrator_review_policy.DEFAULT_ACCOUNT_CAP.
  git log --all -S _catalog_family_cap shows that function only ever
  existed on a different, unmerged branch
  (fix/noema-batched-preflight-413-evidence, PR fix(noema): batch sidecar route preflight #1415) — a real bug, but
  not one on main, and not one I can push a fix for (that branch belongs to
  another in-flight agent; I left a review comment on fix(noema): batch sidecar route preflight #1415 instead flagging
  the exact lines and evidence).
• A further, more precise refinement — capping by shared physical endpoint
  ("outage domain") rather than by credential account, since
  nvidia_nim/nvidia_nim_sub share integrate.api.nvidia.com — is already
  in progress as open PR fix(ci): group review catalog admission/diversity by outage domain, not account #1474. I did not duplicate it.
• The complementary contextual-orchestrator fix suggested in the original
  report (bound concurrency / stagger preflight requests sharing an account)
  does not apply: _preflight_review_agents in this repo's launcher already
  probes routes strictly sequentially, one proxy_send_once call at a time,
  zero retries — there is no concurrent burst to bound.
• The Bytez is_free gap (contextual-orchestrator PR [Fleet incident] Resolve independent-review bootstrap deadlock for sole-collaborator repositories #948) is confirmed
  still open/unmerged, but is explicitly a separate, already-tracked,
  already-PR'd issue — out of scope here, untouched.

What this PR actually fixes: a real, currently-existing gap on main
that let the above happen in the first place and can let it happen again:
main's two build_zdr_prioritized_catalog call sites hand-typed "4" as
the ORCHESTRATOR_CATALOG_ACCOUNT_CAP fallback instead of referencing
contextual_orchestrator_review_policy.DEFAULT_ACCOUNT_CAP (the module that
actually declares the intended default). The two values happen to agree on
main today, but nothing enforced that — and the sibling _catalog_family_cap()
bug on PR #1415 is live proof of exactly this drift class actually happening.

Changes

• scripts/ci/contextual_orchestrator_review_launcher.py: new
  _catalog_account_cap(default) helper; both build_zdr_prioritized_catalog
  call sites now pass account_cap=_catalog_account_cap(DEFAULT_ACCOUNT_CAP)
  (imported from the policy module) instead of a literal "4".
• tests/test_contextual_orchestrator_review_runtime_preflight.py: three new
  regression tests —
  • the cap helper defaults to policy.DEFAULT_ACCOUNT_CAP and is never equal
    to (or derived from) REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES;
  • an explicit ORCHESTRATOR_CATALOG_ACCOUNT_CAP override still applies;
  • a source-level contract test pinning both main() call sites to
    _catalog_account_cap(DEFAULT_ACCOUNT_CAP) and forbidding
    ORCHESTRATOR_CATALOG_FAMILY_CAP/the old "4" literal from reappearing.
• CHANGELOG.md: Unreleased entry.

Pre-existing coverage already proves the two properties the original report
asked to regression-test: test_routable_discovered_models_excludes_evidence_only_rows
(evidence_only/OpenRouter rows never reach the budget-consuming routing path)
and the absence of any cache/blacklist structure in
contextual_orchestrator/model_discovery.py (discovery is fully stateless
per run, so a recovered Bytez is trivially reconsidered next run — no state
to expire).

Test plan

• coverage run -m pytest tests -q — 2108 passed, 1 skipped, 21 subtests passed
• coverage report --show-missing — 100% on scripts/ci/
• interrogate — 100%
• bash -n scripts/ci/contextual_orchestrator_review_sidecar.sh (untouched, sanity-checked)
• git diff --check — clean
• Rebased onto latest origin/main (71633cd0) and re-ran the full gate suite after rebase

Related

• fix(noema): batch sidecar route preflight #1415 — the branch actually carrying the
  described _catalog_family_cap() bug; flagged via review comment, not
  touched directly (protected, another agent's in-flight work).
• fix(ci): keep sidecar credential accounts independent #1468 (merged) — provider_account rename this
  PR builds on.
• fix(ci): group review catalog admission/diversity by outage domain, not account #1474 (open) — the more precise outage-domain
  refinement; not duplicated here.
• fix(discovery): set is_free from Bytez's real meterPrice signal contextual-orchestrator#948 (open) — Bytez is_free
  gap; confirmed out of scope, untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

---

Generated by Claude Code

---

[coderabbitai]

Warning

Review limit reached

Next included review available in 48 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ce6496cd-cfd9-4e51-a7db-82d494ad0eeb

📥 Commits

Reviewing files that changed from the base of the PR and between 71633cd and 4211154.

📒 Files selected for processing (3)

• CHANGELOG.md
• scripts/ci/contextual_orchestrator_review_launcher.py
• tests/test_contextual_orchestrator_review_runtime_preflight.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.