feat(nim): add evidence-grade all-model benchmark (#86)

[seonghobae]

Summary

Implements issue #86 on current main.

• Dynamically discovers the OpenAI-compatible /v1/models catalog and deduplicates deterministic model evidence.
• Probes chat, completions, Responses, embeddings, image, video, audio understanding, transcription, and speech with bounded concurrency and a hard request budget.
• Reserves the complete catalog-probe and evaluation request plan before capability egress; the 30-task manifest is the evidence floor and undersized runs fail closed.
• Compares direct workers, route_once, conduct, hindsight-best-single, and reviewed cheapest-worker policies using equal task/scorer/call/depth/timeout/token budgets.
• Emits transactional secret-redacted JSON, CSV, and Markdown evidence with uncertainty, Pareto, cost provenance, skipped reasons, and production-default safety gates.
• Adds direct DNS-pinned HTTPS transport with original-host TLS, no proxy or redirect following, globally routable-address validation, and an 8 MiB response cap.
• Keeps dry runs secretless and deterministic; adds fuzzing, workflow contract tests, 100% NIM module coverage/docstring gates, wheel smoke, docs, APA/HELM paper evidence, and a one-frame MP4 fixture.

Validation

• python -m pytest -q: 2597 passed
• NIM focused and release/workflow tests: 112 passed
• NIM statement/branch coverage: 100% (991/991 statements, 374 branches)
• interrogate -f 100: passed
• ruff check: passed
• git diff --check: passed

No production routing defaults are changed.

[coderabbitai]

Warning

Review limit reached

Next included review available in 17 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: c05e61c4-bb57-4f28-b3f2-8b0024787007

📥 Commits

Reviewing files that changed from the base of the PR and between c6c3a0c and a2a9881.

⛔ Files ignored due to path filters (1)

• docs/papers/helm-holistic-evaluation-2211.09110.pdf is excluded by !**/*.pdf

📒 Files selected for processing (28)

• .github/workflows/ci.yml
• .github/workflows/fuzz.yml
• .github/workflows/nim-benchmark.yml
• .gitignore
• CHANGELOG.md
• README.md
• conductor/tracks.md
• contextual_orchestrator/__main__.py
• contextual_orchestrator/nim_benchmark.py
• contextual_orchestrator/provider_transport.py
• docs/architecture.md
• docs/doctoring/nim-benchmark-evidence-grade.md
• docs/nim_benchmark.md
• docs/papers/README.md
• docs/product-technical-gap-baseline.md
• examples/nim_pricing_scenario.json
• examples/nim_task_manifest.json
• fuzz/corpus/nim_catalog/hostile_entries.json
• fuzz/corpus/nim_catalog/valid_catalog.json
• fuzz/fuzz_nim_catalog.py
• fuzz/targets.py
• requirements-opencode-review-ci.in
• requirements-opencode-review-ci.txt
• tests/fuzz/test_fuzz_properties.py
• tests/test_cli_auth.py
• tests/test_nim_benchmark.py
• tests/test_nim_benchmark_release_acceptance.py
• tests/test_nim_benchmark_workflow_contract.py

✨ Finishing Touches 💡 1

⚔️ Resolve merge conflicts 💡

• Resolve merge conflict in branch feat/nim-benchmark-rebuild-20260828

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/nim-benchmark-rebuild-20260828

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Validated at exact HEAD 848418e36b8f6231e18c1f8eb22d208887796883:

• Quoted FUZZ_SECONDS in the NIM catalog fuzz step.
• Normalized socket.gaierror to CatalogDiscoveryError in catalog discovery and added regression coverage.
• Converted unknown trace-agent lookup from an uncaught KeyError/TypeError into BenchmarkContractError and added regression coverage.
• The single pending estimate is consumed by the benchmark's one-chat/one-usage call path; generated planning is not enabled by this benchmark, so no change was made there.
• The request reserve is intentionally conservative for the fixed route/conduct envelopes, while auth rejection is intentionally fail-closed during discovery/probes and per-cell during evaluation.

Validation: focused NIM suite 118 passed; Ruff, compile, and git diff --check passed.

[seonghobae]

Resolved against exact HEAD 65109a4.\n\n- Scheduled NIM runs now use 23 3 1 * *, so the next scheduled run falls inside the reviewed access-cost window ending 2026-09-04; _require_current_actual_cost_evidence remains fail-closed after that horizon. Added a workflow contract test and synchronized the benchmark docs and changelog.\n- The five-call conduct concern is not a current failure mode: evaluate_policies fixes workflow_planning to the default template (four steps), verifier_judge=model adds one verifier call, and tool_retry_attempts=0; realtime_judge is used by route cells, not conduct cells. The five-call limiter therefore matches the live benchmark envelope and the focused conduct contract asserts observed_budget_calls <= MAX_WORKFLOW_DEPTH.\n- The preflight nesting note is informational only; the existing exact-boundary tests cover it.\n\nValidation: NIM benchmark, release, and workflow suite: 119 passed. Ruff, compileall, and git diff --check passed.

[seonghobae]

Found and fixed the required-workflow-bootstrap failure on this head — it's unrelated to the tracked org-wide review-sidecar bug. The central Pingora edge policy checker (ContextualWisdomLab/.github's scripts/ci/pingora_edge_policy.py) fails closed on any binary file over 1 MiB that GitHub's Contents API can't return as base64, which is exactly what happens for the added docs/papers/helm-holistic-evaluation-2211.09110.pdf citation:

##[error]Pingora edge policy could not establish complete evidence: GitHub content evidence for docs/papers/helm-holistic-evaluation-2211.09110.pdf is not a regular base64 file

Fix: ContextualWisdomLab/.github#1435 (exempts documentation PDFs under docs/, tests + full suite green). Once that merges, rerunning this PR's checks should clear required-workflow-bootstrap.

Separately, opencode-review/noema-review on this PR are still blocked by the tracked org-wide contextual-orchestrator-review-sidecar startup preflight bug (docs/product-technical-gap-baseline.md's 2026-08-30 entry; a dedicated session is root-causing it) — independent of the above, no action needed here for that part. The previously-noted test_smoke_manifest_cannot_authorize_production_routing flakiness (live-provider-evidence-dependent) is left as-is per the prior cycle's call for the author's judgment.

---

Generated by Claude Code

[seonghobae]

설계 결정: 추상화는 사용하지 않습니다. 와 는 같은 endpoint를 쓰더라도 서로 다른 모델 catalog를 노출할 수 있는 독립 credential/account boundary이므로 각각 discovery·보존·실패 판정을 수행해야 합니다. 모델 간 동적 선택과 latency/quality evidence 공유는 provider 이름 추정이 아니라 오직 명시적 경계에서만 수행하십시오. 이 PR이 해당 전제를 포함하면 제거하거나 현재 head에서 재검증해 주세요.

[seonghobae]

일반화: 앞 코멘트의 NVIDIA는 예시일 뿐입니다. 모든 KV credential은 독립 provider-account/catalog boundary입니다. 같은 vendor/endpoint/model ID라도 서로 다른 key의 모델·권한·가격·privacy·가용성·실패 상태가 같다고 추정하거나 dedupe하지 마십시오. 개념은 제거하며, 논리적 모델 동등성과 latency/quality evidence 공유는 오직 operator가 명시한 으로만 성립합니다.

[seonghobae]

최종 일반 계약: 모든 KV credential은 독립 provider-account/catalog boundary입니다. 같은 vendor, endpoint, model ID라도 서로 다른 key의 모델, 권한, 가격, privacy, 가용성, 실패 상태를 같다고 추정하거나 dedupe하지 않습니다. provider family는 없으며, 논리적 모델 동등성과 latency/quality evidence 공유는 오직 operator가 명시한 model_group으로만 성립합니다.

[seonghobae]

Cross-PR integration contract: routing identity is provider-neutral model_group only; do not add or preserve a provider-family abstraction. OpenRouter discovery must retain concrete free model IDs, while the aggregate openrouter/free router is not a serving candidate. OpenCode, Noema, and Strix must call contextual-orchestrator. Do not impose fixed wall-clock deadlines on inference, initial ping, readiness/health, provider discovery, or OpenRouter ZDR-list retrieval; use explicit cancellation and evidence-backed transport failure instead. Reconcile this PR with #971 and central .github #1508 before merge.

[cwl-noema-review]

Noema LLM review

The PR implements a comprehensive, evidence-grade benchmark harness for NVIDIA NIM models. It introduces dynamic catalog discovery, multi-modality capability probing, and a fair-comparison framework for routing policies (direct, route-once, conduct) using equalized token and call budgets. The implementation adheres to strict security requirements, including DNS pinning for HTTPS connections, rejection of non-globally routable addresses, and a fail-closed budget system. Prior review concerns regarding budget accounting, transport pinning, and authentication handling have been resolved in the current implementation.

Reviewed changed lines

• contextual_orchestrator/__main__.py:497 (RIGHT): Correctly dispatches the 'nim-benchmark' subcommand using the normalized 'arguments' list rather than sys.argv, resolving a prior bug.
• contextual_orchestrator/nim_benchmark.py:325 (RIGHT): The _BudgetedModelClient explicitly sets max_retries=0 in init, ensuring the RequestBudget bounds actual egress calls rather than logical attempts.
• contextual_orchestrator/nim_benchmark.py:412 (RIGHT): EqualBudgetModelClient implements getattr to forward ModelClient capabilities, ensuring compatibility with orchestrator features like request_settings.
• contextual_orchestrator/nim_benchmark.py:175 (RIGHT): The transport implementation uses PinnedHTTPSConnection and validated_public_addresses to prevent DNS rebinding and SSRF, enforcing globally routable endpoints.

Adversarial validation

• contextual_orchestrator/nim_benchmark.py:325 (RIGHT) falsified: Logical chat calls could trigger hidden provider retries, exceeding the hard request budget. — The _BudgetedModelClient constructor sets kwargs['max_retries'] = 0, which is passed to the super ModelClient, disabling internal retries.
• contextual_orchestrator/nim_benchmark.py:175 (RIGHT) falsified: The benchmark could be used to probe internal network resources via DNS redirection. — require_public_https_endpoint calls validated_public_addresses, which rejects non-globally routable addresses (RFC 6598, etc.) before the connection is established.
• Residual risk: Low. The benchmark is an optional adapter and does not sit in the primary request path. The use of a hard request budget and DNS pinning mitigates the primary security and cost risks.

Findings

• No blocking findings.

• Result: APPROVE

• Head SHA: a2a9881d72c19aed90ede3001175208a14012490

• Reviewer credential: noema-review-github-app

• Actor: cwl-noema-review[bot]