Skip to content

feat: add free-first model fallback policy - #94

Closed
seonghobae wants to merge 35 commits into
fix/atheris-interpreter-lockfrom
feat/free-first-model-fallback-policy
Closed

feat: add free-first model fallback policy#94
seonghobae wants to merge 35 commits into
fix/atheris-interpreter-lockfrom
feat/free-first-model-fallback-policy

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Purpose

Provide a provider- and transport-neutral fallback policy that central workflows and standalone consumers can import without replacing their HTTP clients, reviewer identities, credential schemes, or output-validation gates.

Product boundary

  • validate a strict, versioned JSON manifest;
  • filter by repository visibility, required capabilities, and explicitly declared available credential names;
  • deterministically place every eligible free candidate before every paid candidate;
  • preserve declaration order as the final tie-breaker;
  • reject duplicate identities, unsafe control data, unknown keys, empty eligible pools, JSON/Python scalar type confusion, and mutable credential-control collections;
  • never open a socket, read a credential value, or serialize a credential value; and
  • expose a Python API and a standard-library CLI for immutable cross-repository integration.

Credential authority boundary

The trusted composition root that already owns transport and credential resolution declares only validated available credential names through FallbackContext.available_credentials or repeated --available-credential arguments. The policy never reads environment or KV values. The obsolete --credential-env selector remains rejected. A downstream transport remains responsible for resolving an actual value and failing closed when an availability declaration is incorrect.

FallbackCandidate.required_credentials is an immutable tuple. FallbackContext.available_credentials and required_capabilities are immutable frozensets. A caller cannot mutate credential availability after validation and silently change an already-created policy decision.

Current bounded hardening slice

The current branch preserves test-first lineage for two fail-closed repairs.

Strict manifest scalar types

  • RED e225934c2464535c95e5a8f35f9a8865b5bf0cc6 requires strict rejection of boolean/floating-point schema versions, unsafe programmatic agent selectors, and non-string candidate identifiers;
  • GREEN 344f478b3ac2c090f239909e10d7cde040b9bdb9 validates identifier scalar types before regular-expression matching; and
  • GREEN fde76c037ef467c0f5f983620f05af0dfe229d98 requires an exact integer schema version and validates the selected agent before mapping membership.

These changes normalize malformed control data as FallbackManifestError rather than allowing Python TypeError, and prevent JSON true or 1.0 from satisfying schema version 1 through Python equality semantics.

Immutable credential controls

  • RED a369e434173c09f99cceb7dc9610c895ff53009e requires candidates to reject mutable credential lists, contexts to reject mutable available-credential lists, and non-string repository visibility to fail as a controlled validation error;
  • GREEN 06f0010026d333a53593994b98f6b3b322f7a774 adds tuple/frozenset and visibility type boundaries;
  • diagnostic reconstruction then exposed an existing error-contract regression and a coverage regression: scalar strings no longer reached the established credential names must be a sequence branch;
  • 49a6d151a799d0cc4ea1f98b5994ca8fd4f9dd9e restores compatible diagnostics; and
  • final GREEN 5104ea1805ffb6a3bc82eb817cf74e4571e39c48 routes scalar strings/bytes through the existing fail-closed sequence validator, rejects mutable non-tuple sequences separately, and preserves complete branch exercise.

c99531039e3adb20f28eb82ce5efeb99277014f2 records both security boundaries in CHANGELOG.md.

Diagnostic verification

A networkless reconstruction from the exact current production modules and exact focused tests reports:

42 tests passed
279/279 statements: 100%
102/102 branches: 100%
missing public docstrings: 0
compileall: passed

This is diagnostic evidence only. The execution environment could not resolve github.com, so it does not replace a trusted checkout, repository GitHub Check, central coverage evidence, package build/install smoke, or independent review.

Exact integration identity

  • exact PR head: 73ed3a077f88a2f03cf734f1067bee2dcce2467f
  • exact tree: 31866c373cd0e51d2226a20373779c201b7a5c63
  • parents: fallback predecessor 5104ea1805ffb6a3bc82eb817cf74e4571e39c48 and current fix(security): pin provider egress and repair the Atheris lock #96 head 3703d0da9823b8258a0be94f1801aa5d61bfad9f
  • stacked base branch and exact live tip: fix/atheris-interpreter-lock at 3703d0da9823b8258a0be94f1801aa5d61bfad9f
  • state: Draft / mergeable

The ordinary two-parent merge had one semantic conflict: the fallback branch still activated provider transport through import-time ModelClient monkey-patching, while #96 made package import intentionally side-effect free and integrated the secure transport directly. The reconciliation preserves every fallback public export while retaining #96's side-effect-free transport authority.

Local verification of the exact published tree completed:

  • focused fallback, provider-pinning, and coverage-policy tests: 74 passed;
  • full suite: 608 passed;
  • production coverage: 100% across 3,947 statements and 1,036 branches, with zero misses or partial branches;
  • public docstrings: 100%;
  • git diff --check: passed.

The published remote tree is byte-identical to the locally tested tree. Local verification is useful evidence but does not replace GitHub exact-head checks or independent review.

Current-head review and check state

All repository-local workflows explicitly checked out exact contributor head 73ed3a077f88a2f03cf734f1067bee2dcce2467f with checkout credentials disabled:

  • Tests run 31323220333: success; 608 passed, 100% production statement/branch coverage (3,947 statements / 1,036 branches, zero misses/partials), and 100% public docstrings.
  • Security run 31323220336: success; CodeQL and the Python supply-chain/SBOM job completed on the exact head with no reported dependency vulnerability.
  • Fuzz run 31323220341: success; eight Hypothesis property tests and all four bounded Atheris targets completed without a crash artifact.
  • Model fallback policy quality run 31323220316: success; 42 focused tests passed, the five fallback modules achieved 279/279 statements and 102/102 branches (100%), and the public-docstring/secret-boundary contract passed.
  • CodeRabbit commit status: success, classified as status-only.
  • unresolved inline review threads: zero.
  • formal current-head reviews: absent; the three recorded OpenCode submissions are dismissed predecessor-head evidence.
  • qualifying independent non-author approval: absent.

The conditional Atheris crash-artifact upload was skipped because no crash file existed; the required Fuzz job concluded success. Exact-head repository-local behavior, security, fuzz, and focused fallback evidence are complete, but protected-base integration, required fresh automated reviews, and qualifying independent approval remain separate gates. No predecessor-head, dismissed, status-only, author-only, stale-base, synthetic-merge, or infrastructure-only evidence is promoted to acceptance.

Historical central prerequisites — read-only

  • Protected central .github main is exact tip 6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba.
  • Live .github#937 is Ready at exact head 8726df151e64eecb89d91a4c029e809a785ee126: ten terminal-success workflows, zero unresolved threads, COMMENTED-only reviews, and no qualifying approval.
  • Live .github#939 is Ready at exact head ac5665148bb113f92e97d2fc49a729bca2f050b5: nine terminal-success workflows, zero threads, and no formal review.
  • Neither central PR is protected-main authority; closed or superseded fix(security): enforce trace authority before disclosure #888/#906/#929 evidence is historical and does not transfer.
  • fix(security): pin provider egress and repair the Atheris lock #96 remains Draft at exact head 3703d0da9823b8258a0be94f1801aa5d61bfad9f, so this stacked PR remains Draft.

Integration boundary

Noema, OpenCode Agent, Strix, naruon, and standalone consumers retain their existing transports, authentication, schema validation, review identities, security gates, and failure semantics. This module derives an ordered eligible plan only. It never accepts provider output and does not change any review-agent credential name or scope.

Documentation

  • docs/model-fallback-policy.md
  • docs/doctoring/free-first-model-fallback.md
  • CHANGELOG.md

Merge acceptance

Do not mark Ready, merge, publish, or release until the exact integrated head has successful repository Tests, Fuzz, Security, Security Scan, SAST Semgrep, permanent fallback statement/branch/docstring coverage, package build/install smoke, trusted central coverage/package evidence, current-head CodeRabbit/OpenCode/Noema/Strix review where required, zero valid unresolved findings, qualifying independent non-author approval, and every branch-protection rule.

Queued, pending, skipped-required, cancelled, absent, failed, stale-head, predecessor-head, synthetic-merge, status-only, or author-only evidence is not success.

Refs #96 and #99.

Live dependency refresh (2026-08-12 05:10 UTC)

Read-only refetch superseding the historical prerequisite snapshot above:

  • This Draft PR remains exact head 73ed3a077f88a2f03cf734f1067bee2dcce2467f on exact fix(security): pin provider egress and repair the Atheris lock #96 base 3703d0da9823b8258a0be94f1801aa5d61bfad9f; protected repository main remains 6841b71935e0b7cb98fb52bcb4709cc5100c8d87.
  • Protected central .github/main remains 6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba.
  • Ready central #937 remains 67d834f510fe044dd9d53cd4f4b9783353e303bd with 11 terminal-success exact-head workflows, zero unresolved threads, and no qualifying approval.
  • Ready central #939 remains ac5665148bb113f92e97d2fc49a729bca2f050b5 with nine terminal-success exact-head workflows, zero unresolved threads, and no formal review.
  • Ready central #943 advanced to 601b254f3a8ea4cc593e7089d6baeadd9d8d3ee4 with nine terminal-success exact-head workflows, zero unresolved threads, and no formal review; it is the bounded trusted-download User-Agent repair and does not replace #939's separate review-status semantics.
  • This PR retains three formal predecessor-head review submissions, zero unresolved threads, and no qualifying independent approval.

All dependency identities remain planning evidence only; none is protected integration or transferable approval.

Live central review-authority correction (2026-08-12 06:29 UTC)

Read-only dependency refetch: protected central main remains 6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba. Central #937 remains open at 67d834f510fe044dd9d53cd4f4b9783353e303bd with eleven terminal-success workflows, zero unresolved threads, and one OpenCode APPROVED formal model review. That model review is not a qualifying independent human approval and does not make the central branch protected authority. Central #939 remains open at ac5665148bb113f92e97d2fc49a729bca2f050b5 with nine terminal-success workflows, zero threads, and no formal review; #943 remains open at 601b254f3a8ea4cc593e7089d6baeadd9d8d3ee4 with the same nine-success/zero-thread/no-review classification.

This repository loop did not mutate the central repository. This PR's unchanged source head and local workflow evidence remain active_pr; protected central integration and qualifying independent non-author approval remain absent. Keep the PR Draft.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1e8e383c-f6a0-47e6-99dd-23ce081150da

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae
seonghobae changed the base branch from main to claude/inkspan-pr-audit-ci-q1u4uj August 4, 2026 22:34

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 82ea37ee2673111b0a2f25642d637a305473f642.

  • Head SHA: 82ea37ee2673111b0a2f25642d637a305473f642

  • Workflow run: 30957074628

  • Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (7 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (7 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (2 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (2 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (5 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (5 files)"]
  R3 --> V3["targeted test run"]
Loading

@opencode-agent

opencode-agent Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

OpenCode Review Overview

  • Head SHA: 82ea37ee2673111b0a2f25642d637a305473f642
  • Workflow run: 30965451527
  • Workflow attempt: 1
  • Gate result: REQUEST_CHANGES (approval step)

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 82ea37ee2673111b0a2f25642d637a305473f642.

  • Head SHA: 82ea37ee2673111b0a2f25642d637a305473f642

  • Workflow run: 30965451527

  • Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (7 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (7 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (2 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (2 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (5 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (5 files)"]
  R3 --> V3["targeted test run"]
Loading

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@opencode-agent @cwl-noema-review exact-current-head review requested for 82ea37ee2673111b0a2f25642d637a305473f642. Tests, Fuzz, Security, Security Scan, and SAST Semgrep have completed successfully on this head. Please independently verify strict manifest validation, free-before-paid ordering, visibility/capability/credential-name filtering, secret non-retention, CLI boundary, 100% statement/branch coverage and docstrings, and compatibility with the DNS-pinned provider-transport base. Do not approve unless exact-head coverage evidence is independently proven.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected_head="82ea37ee2673111b0a2f25642d637a305473f642"

echo "== Checked-out commit =="
git rev-parse HEAD
git status --short
git show -s --format='commit=%H%nparents=%P%nsubject=%s' HEAD
if [ "$(git rev-parse HEAD)" != "$expected_head" ]; then
  echo "ERROR: checked-out HEAD does not match requested exact head" >&2
fi

echo "== Pull request metadata and exact-head checks =="
gh pr view 94 --repo ContextualWisdomLab/contextual-orchestrator \
  --json number,title,headRefOid,baseRefOid,baseRefName,headRefName,statusCheckRollup \
  --jq '.'
gh api "repos/ContextualWisdomLab/contextual-orchestrator/commits/${expected_head}/check-runs?per_page=100" \
  --jq '.check_runs[] | {name,status,conclusion,details_url}'
gh api "repos/ContextualWisdomLab/contextual-orchestrator/commits/${expected_head}/statuses" \
  --jq '.statuses[]? | {context,state,target_url,description}'

echo "== Files changed from first parent =="
base="$(git rev-parse HEAD^1)"
git diff --name-status "$base" HEAD
git diff --stat "$base" HEAD

echo "== Candidate fallback, transport, test, workflow, and coverage files =="
fd -HI -t f . \
  | rg -i '(^|/)(model.*fallback|fallback.*model|.*fallback.*|.*transport.*|.*provider.*|test.*fallback.*|.*coverage.*|pyproject\.toml|pytest\.ini|tox\.ini|\.coveragerc|.*workflow.*|CHANGELOG\.md)$' \
  | sort

echo "== Python structure map =="
fd -HI -e py . -x sh -c '
  case "$1" in
    *fallback*|*transport*|*provider*|*test*)
      echo "--- $1"
      ast-grep outline "$1" --items all --type class,function --view expanded || true
      ;;
  esac
' sh {} \;

echo "== Sensitive API and boundary patterns =="
rg -n -C 3 --glob '*.py' \
  '(socket|requests|urllib|httpx|aiohttp|subprocess|os\.environ|environ|getenv|argv|argparse|json\.(load|loads|dump|dumps)|open\(|print\(|logging|credential|secret|token|password|manifest|visibility|capabilit|free|paid)' \
  .

echo "== Public-symbol docstring candidates =="
rg -n -U --glob '*.py' \
  '^(def|class) [A-Za-z_][A-Za-z0-9_]*.*:\n[[:space:]]*(?!"""|'\'''\''|#)' \
  . || true

echo "== Coverage configuration and committed coverage reports =="
fd -HI -t f -g 'coverage.xml' -g '.coverage' -g 'htmlcov' -g '*coverage*.json' -g '*coverage*.txt' -g 'pyproject.toml' -g 'pytest.ini' -g 'tox.ini' -g '.coveragerc' \
  -x sh -c 'echo "--- $1"; if [ -f "$1" ]; then sed -n "1,260p" "$1"; fi' sh {} \;

echo "== Exact-head patch for fallback and transport-related files =="
git diff --find-renames --find-copies --unified=80 "$base" HEAD -- \
  ':(glob)**/*fallback*' ':(glob)**/*transport*' ':(glob)**/*provider*' \
  ':(glob)**/*test*' ':(glob)pyproject.toml' ':(glob)pytest.ini' \
  ':(glob)tox.ini' ':(glob).coveragerc' ':(glob)CHANGELOG.md' ':(glob)docs/**'

Length of output: 19534


🏁 Script executed:

#!/bin/bash
set -u

head="82ea37ee2673111b0a2f25642d637a305473f642"
p1="$(git rev-parse "${head}^1")"
p2="$(git rev-parse "${head}^2")"
merge_base="$(git merge-base "$p1" "$p2")"

echo "== Commit relationships =="
printf 'head=%s\nfirst_parent=%s\nsecond_parent=%s\nmerge_base=%s\n' "$head" "$p1" "$p2" "$merge_base"

echo "== Fallback changes introduced on first-parent branch =="
git diff --name-status "$p2" "$head" || true
echo "== Security transport changes merged from second parent =="
git diff --name-status "$p1" "$head" || true
echo "== Combined PR-range changes from merge base =="
git diff --name-status "$merge_base" "$head" || true

echo "== Repository candidates =="
find . -type f \
  \( -name '*.py' -o -name 'pyproject.toml' -o -name 'pytest.ini' -o -name 'tox.ini' -o -name '.coveragerc' -o -name 'coverage.xml' -o -name 'CHANGELOG.md' \) \
  -not -path './.git/*' -print \
  | sort \
  | grep -Ei '(fallback|transport|provider|test|coverage|pyproject|pytest|tox|coveragerc|changelog)' || true

echo "== Python source and test outlines =="
while IFS= read -r f; do
  case "$f" in
    *fallback*|*transport*|*provider*|*test*)
      echo "--- $f"
      ast-grep outline "$f" --items all --type class,function --view expanded 2>&1 || true
      ;;
  esac
done < <(find . -type f -name '*.py' -not -path './.git/*' -print | sort)

echo "== Fallback, CLI, validation, ordering, and secret-boundary excerpts =="
rg -n -i -C 8 --glob '*.py' \
  '(fallback|manifest|schema|version|additionalProperties|unknown.key|duplicate|visibility|capabilit|credential.name|free|paid|sort|order|socket|requests|urllib|httpx|aiohttp|os\.environ|getenv|secret|token|password|argparse|__main__|json\.)' \
  . || true

echo "== Exact fallback and transport patch =="
git diff --find-renames --unified=100 "$p2" "$head" -- \
  ':(glob)**/*fallback*' ':(glob)**/*transport*' ':(glob)**/*provider*' \
  ':(glob)**/*test*' ':(glob)pyproject.toml' ':(glob)pytest.ini' \
  ':(glob)tox.ini' ':(glob).coveragerc' || true

echo "== Coverage configuration =="
for f in pyproject.toml pytest.ini tox.ini .coveragerc coverage.xml; do
  if [ -f "$f" ]; then
    echo "--- $f"
    sed -n '1,320p' "$f"
  fi
done

echo "== Exact-head coverage workflow output and artifacts =="
gh run view 30956868345 --repo ContextualWisdomLab/contextual-orchestrator --job 92151972587 --log || true
gh run view 30956868345 --repo ContextualWisdomLab/contextual-orchestrator --json databaseId,headSha,event,conclusion,jobs,artifacts || true
gh api "repos/ContextualWisdomLab/contextual-orchestrator/actions/runs/30956868345/artifacts" \
  --jq '.artifacts[] | {name,size_in_bytes,expired,archive_download_url}' || true

Length of output: 152368


🏁 Script executed:

#!/bin/bash
set -u

repo="ContextualWisdomLab/contextual-orchestrator"
head="82ea37ee2673111b0a2f25642d637a305473f642"
p1="$(git rev-parse "${head}^1")"
p2="$(git rev-parse "${head}^2")"

echo "== Exact changed files, separated by merge parent =="
echo "-- fallback branch relative to security parent --"
git diff --name-status "$p2" "$head" | grep -Ei '(fallback|transport|provider|test|workflow|coverage|pyproject|changelog|docs)' || true
echo "-- security parent merged into fallback branch --"
git diff --name-status "$p1" "$head" | grep -Ei '(fallback|transport|provider|test|workflow|coverage|pyproject|changelog|docs)' || true

echo "== Complete fallback module source =="
for f in \
  contextual_orchestrator/_fallback_types.py \
  contextual_orchestrator/_fallback_manifest.py \
  contextual_orchestrator/_fallback_plan.py \
  contextual_orchestrator/_fallback_cli.py \
  contextual_orchestrator/model_fallback.py
do
  if git cat-file -e "${head}:${f}" 2>/dev/null; then
    echo "--- $f"
    git show "${head}:${f}"
  fi
done

echo "== Fallback import and transport-call audit =="
rg -n -C 2 --glob '*fallback*.py' \
  '^(import|from) |(socket|requests|urllib|httpx|aiohttp|http\.client|ssl|asyncio\.open_connection|subprocess|Popen|urlopen|urlretrieve)\b' \
  contextual_orchestrator tests || true

echo "== Changed fallback test functions and assertions =="
for f in tests/test_model_fallback_cli.py tests/test_model_fallback_manifest.py tests/test_model_fallback_plan.py; do
  if [ -f "$f" ]; then
    echo "--- $f"
    rg -n -C 2 '^def test_|^`@pytest`\.mark|assert |pytest\.raises' "$f" || true
  fi
done

echo "== Public fallback exports and package-level exposure =="
rg -n -C 4 '(model_fallback|FallbackCandidate|FallbackContext|build_fallback_plan|load_fallback_manifest)' \
  contextual_orchestrator/__init__.py contextual_orchestrator/model_fallback.py 2>/dev/null || true

echo "== Workflow files that produce fallback coverage evidence =="
find .github -type f \( -name '*.yml' -o -name '*.yaml' \) -not -path './.git/*' -print 2>/dev/null \
  | while IFS= read -r f; do
      if grep -qiE 'coverage-evidence|fallback|interrogate|coverage report|coverage xml' "$f"; then
        echo "--- $f"
        grep -niE -C 10 'coverage-evidence|fallback|interrogate|coverage report|coverage xml|pytest' "$f"
      fi
    done

echo "== Exact-head Action workflow runs and coverage job =="
gh api "repos/${repo}/actions/runs?head_sha=${head}&per_page=100" \
  --jq '.workflow_runs[] | {id,name,head_sha,conclusion,event,html_url}' || true
gh api "repos/${repo}/actions/runs/30956868345/jobs?per_page=100" \
  --jq '.jobs[] | select(.id == 92151972587 or (.name | test("coverage"; "i"))) | {id,name,conclusion,started_at,completed_at,steps,html_url}' || true
gh api "repos/${repo}/actions/runs/30956868345/artifacts?per_page=100" \
  --jq '.artifacts[] | {id,name,size_in_bytes,expired,created_at,archive_download_url}' || true

echo "== Coverage job log excerpts =="
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
if gh api -H 'Accept: application/vnd.github+json' \
  "repos/${repo}/actions/jobs/92151972587/logs" > "$tmpdir/job-logs.bin" 2>/dev/null; then
  file "$tmpdir/job-logs.bin" || true
  if unzip -Z1 "$tmpdir/job-logs.bin" >/dev/null 2>&1; then
    unzip -p "$tmpdir/job-logs.bin" \
      | grep -Ei -C 8 '(fallback|coverage|branch|statement|interrogate|100%|270|94)' \
      | tail -n 400 || true
  else
    grep -Ei -C 8 '(fallback|coverage|branch|statement|interrogate|100%|270|94)' \
      "$tmpdir/job-logs.bin" | tail -n 400 || true
  fi
else
  echo "Could not retrieve coverage job logs."
fi

Length of output: 55095


@seonghobae I completed the exact-head review for 82ea37ee2673111b0a2f25642d637a305473f642.

Static inspection supports these controls:

  • The manifest parser rejects unknown keys, unsupported versions, unsafe control data, duplicate identities, and empty candidate lists.
  • The planner filters repository visibility, credential names, capabilities, and disabled paid candidates before ordering.
  • The planner orders eligible candidates by free tier, then priority, then declaration order.
  • The fallback modules do not import or invoke provider transports or network APIs.
  • The fallback policy remains separate from the DNS-pinned provider_transport.py change.
  • The CLI serializes public metadata only. It does not retain or print credential values.

The Tests, Fuzz, Security, Security Scan, and SAST workflows succeeded on this exact head.

I cannot approve this PR yet. Exact-head 100% statement and branch coverage is not independently proven. The coverage-evidence job completed successfully, but it only runs an echo command that states it does not execute pull-request content. It provides no coverage command, report, artifact, or fallback-module metrics.

Please provide an exact-head, executable coverage workflow or artifact that shows branch coverage for the five fallback modules. The evidence must identify the measured 270 statements and 94 branches at 100%.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 82ea37ee2673111b0a2f25642d637a305473f642.

  • Head SHA: 82ea37ee2673111b0a2f25642d637a305473f642

  • Workflow run: 30961031762

  • Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (7 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (7 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (2 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (2 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (5 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (5 files)"]
  R3 --> V3["targeted test run"]
Loading

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head 82ea37ee2673111b0a2f25642d637a305473f642.

  • Head SHA: 82ea37ee2673111b0a2f25642d637a305473f642

  • Workflow run: 30965451527

  • Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (7 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (7 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (2 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (2 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (5 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (5 files)"]
  R3 --> V3["targeted test run"]
Loading

@seonghobae
seonghobae changed the base branch from claude/inkspan-pr-audit-ci-q1u4uj to fix/atheris-interpreter-lock August 5, 2026 01:31

Copy link
Copy Markdown
Contributor Author

@opencode-agent @cwl-noema-review

Please review exact head 40c6a4b419cdf8fa90c422acb5443a0e1cca5d16 against base cfd42f309ea39a189635ff8ba573c5d3b0e332ba only. The prior CHANGES_REQUESTED reviews are bound to stale head 82ea37ee2673111b0a2f25642d637a305473f642; the portable Atheris and coverage-evidence prerequisite is now present in the stacked base. Preserve the existing reviewer credential and authority boundaries, and do not reuse earlier-head evidence.

@seonghobae
seonghobae dismissed stale reviews from opencode-agent[bot] and opencode-agent[bot] August 5, 2026 02:19

Dismissed as stale exact-head evidence: this review is explicitly bound to head 82ea37e. The current head is 40c6a4b on the portable Atheris base cfd42f3. A fresh independent exact-head review remains required before merge.

@seonghobae
seonghobae marked this pull request as draft August 5, 2026 05:08
Require the policy-only CLI to accept an explicit trusted set of available credential names, reject the legacy environment selector, and prove that planning never reads environment values.
Add a permanent read-only exact-head workflow that runs the fallback policy behavior tests, 100% statement/branch coverage, 100% docstrings, compilation, and diff checks. The current test-first head is expected to fail before the CLI boundary is repaired.
@seonghobae
seonghobae marked this pull request as ready for review August 5, 2026 08:24
Run the permanent read-only fallback quality gate for path-matching pushes as well as pull requests so a stacked non-default-base branch produces exact-head evidence instead of waiting for its ancestor to merge.
Preserve ordinary locale/process environment access used by argparse while failing specifically on attempts to inspect configured provider credential values. The feature remains red until the CLI accepts declarative names and removes the environment selector.
@seonghobae
seonghobae marked this pull request as draft August 5, 2026 08:38

Copy link
Copy Markdown
Contributor Author

/oc Refetch the live PR and stop without writing unless exact head is 0b04029363438aa490c885ac978f51788e83e0dc. Keep Draft. The stacked base fix/atheris-interpreter-lock is exact prerequisite #96 head ecad520fc3bd3e2591fa79e473e0666223d6d427, but GitHub reports this PR non-mergeable and the PR body still names closed superseded central dependencies.

Reconcile the current #96 base into the existing branch with an ordinary non-force merge. Preserve the provider-egress, 8 MiB cumulative response bound, interpreter-portable Atheris lock, no-proxy/redirect/DNS snapshot/TLS authority, and all exact tests from #96 verbatim. Resolve overlap so this PR's effective diff contains only the transport-neutral free-first fallback policy, strict manifest/schema, composition-root credential-name contract, CLI/API, tests, doctoring, changelog, and permanent quality workflow. The fallback layer must never read credential values, open sockets, execute providers, or alter reviewer identities/transports.

Update the stack/body to the live order: central .github#816 and .github#812 are read-only prerequisites for trusted central evidence; contextual-orchestrator #96 must merge first; then retarget this PR to protected main and regenerate exact-head evidence; central .github#760 consumes the final integrated source only after both. Remove obsolete references that imply closed #759/#763 will merge.

Run the focused fallback suite, full contextual-orchestrator tests/fuzz/security/package checks, 100% production statement/branch coverage and public docstrings, Python compatibility, compileall, schema/CLI adversarial tests, and git diff --check. Do not mark Ready, approve, merge, release, retarget before #96 merges, force-push, create another branch/PR/workflow, read secrets, or introduce COPILOT_GITHUB_TOKEN.

Copy link
Copy Markdown
Contributor Author

@opencode-agent @cwl-noema-review

Review exact current contributor head 73ed3a077f88a2f03cf734f1067bee2dcce2467f against exact live base 3703d0da9823b8258a0be94f1801aa5d61bfad9f. All repository-local deterministic and product-specific workflows are terminal-success on this exact head. Treat predecessor-head reviews, approvals, requested changes, checks, and synthetic evidence as historical. Report only current-head actionable findings; model evidence is not qualifying independent approval. Keep this stacked PR Draft while its prerequisite remains unintegrated.

@seonghobae

Copy link
Copy Markdown
Contributor Author

CONFLICTING and directionally superseded by price-aware live routing in PR #111 (price_per_million tie-break among capability-matched agents). Free-first policy can re-land as an additive RoutingPolicy flag after #111 merges.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Closed by autonomous commercial loop: CONFLICTING and directionally superseded by price-aware live routing in PR #111 (price_per_million tie-break among capability-matched agents). Free-first policy can re-land as an additive RoutingPolicy flag after #111 merges.

@seonghobae seonghobae closed this Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

@opencode-agent address

Continue this existing stacked PR only after prerequisite #96's exact final head is integrated. Refetch the live stack and resolve all current-head reviews/checks without opening another PR or bypassing protection.

Preserve the transport-neutral free-first policy, deterministic eligible-model ordering, visibility/capability/credential filtering, strict manifest and immutable source-receipt validation, NVIDIA NIM provider policy, existing reviewer identities, and fail-closed output/security validators. Extend the permanent architecture, evaluation, and doctoring contracts so test-time computation is explicitly allocated between bounded single-model routing and deep multi-agent orchestration using workflow stage, recursion depth, task decomposition, tool/access allowlists, and role-specific reasoning effort; include ablations across reasoning/computation levels. Speed is not the optimization target. Ground the decisions in authoritative Fugu, Conductor, and TRINITY research with APA 7 citations and clearly separate evidence from architectural inference.

Require deterministic unit/contract tests plus realistic orchestration evaluations, Python 3.14, 100% production statement/branch coverage and public docstrings, security/supply-chain checks, ADR/architecture/CHANGELOG traceability, and clean-tree verification. Never use COPILOT_GITHUB_TOKEN. After #96 merges, reconcile the protected base, rerun every exact-head gate, request independent review, and merge only with zero unresolved findings, a qualifying non-author approval, and no protection bypass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant