Skip to content

fix(noema): add price-attested provider fallback #1410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seonghobae wants to merge 3 commits into from
Closed

fix(noema): add price-attested provider fallback #1410

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d10a333
test(noema): require free-first auto fallback route
seonghobae Aug 29, 2026
0c4f099
ci(noema): run gateway contracts on changed paths
seonghobae Aug 29, 2026
4741d8e
fix(noema): use free-first auto provider fallback
seonghobae Aug 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .github/workflows/noema-orchestrator-quality-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: Noema Orchestrator Quality CI

on:
pull_request:
branches: [main]
paths:
- ".github/workflows/noema-orchestrator-quality-ci.yml"
- ".github/workflows/noema-review.yml"
- "CHANGELOG.md"
- "docs/doctoring/noema-orchestrator-auto-fallback.md"
- "scripts/ci/contextual_orchestrator_review_launcher.py"
- "scripts/ci/contextual_orchestrator_review_policy.py"
- "scripts/ci/contextual_orchestrator_review_sidecar.sh"
- "scripts/ci/load_contextual_orchestrator_token.sh"
- "scripts/ci/zdr_policy.py"
- "tests/test_contextual_orchestrator_review_sidecar_contract.py"
- "tests/test_noema_orchestrator_workflow_contract.py"

permissions:
contents: read

concurrency:
group: noema-orchestrator-quality-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
exact-head-contract:
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Checkout exact source revision
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"

- name: Install exact hash-verified test runner dependencies
env:
PIP_DISABLE_PIP_VERSION_CHECK: "1"
PIP_NO_INPUT: "1"
shell: bash --noprofile --norc -e -o pipefail {0}
run: |
cat >"${RUNNER_TEMP}/noema-quality-requirements.txt" <<'EOF'
coverage==7.15.2 --hash=sha256:b9a6367e4aff723e8ee8190836836124284e8fcd4265e307c844010cfa074f3f
iniconfig==2.1.0 --hash=sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760
packaging==26.2 --hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e
pluggy==1.6.0 --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746
pygments==2.20.0 --hash=sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176
pytest==9.1.1 --hash=sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c
EOF
python -m pip install \
--only-binary=:all: \
--require-hashes \
-r "${RUNNER_TEMP}/noema-quality-requirements.txt"

- name: Verify exact-head Noema gateway contracts
shell: bash --noprofile --norc -e -o pipefail {0}
run: |
test "$(git rev-parse HEAD)" = "${{ github.event.pull_request.head.sha || github.sha }}"
python -m coverage run -m pytest \
tests/test_noema_orchestrator_workflow_contract.py \
tests/test_contextual_orchestrator_review_sidecar_contract.py \
-q
Comment on lines +65 to +68

@devin-ai-integration devin-ai-integration Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Focused CI remains predictably red

Existing tests still require orchestrator/free, while this workflow runs one unchanged suite. Every affected pull request fails before validating fallback behavior.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

python -m compileall -q \
tests/test_noema_orchestrator_workflow_contract.py \
tests/test_contextual_orchestrator_review_sidecar_contract.py \
scripts/ci/contextual_orchestrator_review_launcher.py
bash -n scripts/ci/contextual_orchestrator_review_sidecar.sh
git diff --exit-code
3 changes: 2 additions & 1 deletion .github/workflows/noema-review.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,7 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
CONTEXTUAL_ORCHESTRATOR_REQUIRE_ZDR: ${{ steps.target_visibility.outputs.require_zdr }}
CONTEXTUAL_ORCHESTRATOR_POOL: auto

@devin-ai-integration devin-ai-integration Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Noema reviews consume paid models

When free providers fail, CONTEXTUAL_ORCHESTRATOR_POOL admits priced routes for required Noema reviews. These reviews can incur unapproved provider charges.

Prompt for agents 
Restore the binding zero-cost Noema contract while addressing free-route outages. AGENTS.md and docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md require Noema to use orchestrator/free; orchestrator/auto admits fully price-attested paid routes after free-route preflight exhaustion. Keep Noema fail-closed on free-route exhaustion, or update the governing policy through a separately authorized decision before enabling paid fallback. Align the model exported later in this workflow and all affected contract tests with the selected governed route.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

run: |
set -euo pipefail
bash "$GITHUB_WORKSPACE/scripts/ci/contextual_orchestrator_review_sidecar.sh"
Expand All @@ -322,7 +323,7 @@ jobs:
fi
source "$GITHUB_WORKSPACE/scripts/ci/load_contextual_orchestrator_token.sh"
export NOEMA_LLM_API_URL="${CONTEXTUAL_ORCHESTRATOR_BASE_URL%/}/v1/chat/completions"
export NOEMA_LLM_MODEL="orchestrator/free"
export NOEMA_LLM_MODEL="orchestrator/auto"
export NOEMA_LLM_API_KEY="${CONTEXTUAL_ORCHESTRATOR_TOKEN}"
export NOEMA_LLM_VIA_ORCHESTRATOR=1
python3 scripts/ci/noema_review_gate.py \
Expand Down
10 changes: 6 additions & 4 deletions tests/test_noema_orchestrator_workflow_contract.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Noema review now uses the vendored orchestrator sidecar, not NVIDIA NIM."""
"""Noema review uses the vendored orchestrator auto pool with free-first fallback."""

from __future__ import annotations

Expand All @@ -11,8 +11,8 @@
from tests.test_required_workflow_queue_contract import workflow_step, workflow_text


def test_noema_review_credentials_and_llm_use_orchestrator_free() -> None:
"""Require reviewer credentials and the sidecar; the public NIM hardcode is gone."""
def test_noema_review_credentials_and_llm_use_orchestrator_auto() -> None:
"""Require reviewer credentials and the auto sidecar; direct NIM stays absent."""
workflow = workflow_text("noema-review.yml")

assert "fail_unavailable()" in workflow
Expand All @@ -32,6 +32,7 @@ def test_noema_review_credentials_and_llm_use_orchestrator_free() -> None:
assert "Resolve Noema target repository visibility" in workflow
assert "target_visibility.outputs.require_zdr" in workflow
assert "CONTEXTUAL_ORCHESTRATOR_REQUIRE_ZDR" in workflow
assert "CONTEXTUAL_ORCHESTRATOR_POOL: auto" in workflow
assert (
"NOEMA_LLM_API_KEY: ${{ secrets.NOEMA_LLM_API_KEY || secrets.OPENAI_API_KEY || '' }}"
not in workflow
Expand All @@ -42,7 +43,8 @@ def test_noema_review_credentials_and_llm_use_orchestrator_free() -> None:
assert "NVIDIA_NIM_API_KEY_SUB: ${{ secrets.NVIDIA_NIM_API_KEY_SUB }}" in workflow
assert "OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}" in workflow
assert "OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}" in workflow
assert 'export NOEMA_LLM_MODEL="orchestrator/free"' in workflow
assert 'export NOEMA_LLM_MODEL="orchestrator/auto"' in workflow
assert 'export NOEMA_LLM_MODEL="orchestrator/free"' not in workflow
assert (
"contextual-orchestrator review sidecar must be provisioned before Noema LLM review."
in workflow
Expand Down
Loading