feat(agents): evaluate subscription routing policy explicitly by devantler · Pull Request #210 · devantler-tech/agent-plugins

devantler · 2026-09-12T09:43:14Z

Why

Deployments need consistent model selection and escalation without treating uncertain quota or billing evidence as permission to run.

What

Adds routing-policy checks and bounded delegation procedures, together with the reviewed measurement skills. The 5.1.0 release keeps automatic execution dependent on verified native controls.

Fixes #209. Part of devantler-tech/monorepo#3314.

devantler · 2026-09-12T10:06:22Z

@coderabbitai review

coderabbitai · 2026-09-12T10:06:41Z

@devantler: I will review the changes.

✅ Action performed

Review finished.

coderabbitai · 2026-09-12T10:11:02Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ca01de2c-c9bb-4107-b224-1c3637233c3e

📥 Commits

Reviewing files that changed from the base of the PR and between a657cd5 and c99a54b.

📒 Files selected for processing (12)

.claude-plugin/marketplace.json
.github/plugin/marketplace.json
plugins/agentic-engineering/.claude-plugin/plugin.json
plugins/agentic-engineering/README.md
plugins/agentic-engineering/agents/agent-improver.agent.md
plugins/agentic-engineering/agents/agentic-engineer.agent.md
plugins/agentic-engineering/plugin.json
plugins/agentic-engineering/resources/inference-routing.md
plugins/agentic-engineering/resources/provider-neutral.desired-state.json
plugins/agentic-engineering/scripts/classify-default-branch-ci-runs.test.sh
plugins/agentic-engineering/scripts/evaluate-inference-routing.sh
plugins/agentic-engineering/scripts/evaluate-inference-routing.test.sh

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details

🧰 Additional context used

📓 Path-based instructions (3)

Any change to the plugin set updates both manifests in the same PR

📄 CodeRabbit inference engine (AGENTS.md)

Files:

.github/plugin/marketplace.json
.claude-plugin/marketplace.json

A plugin's version is its cache key — move it whenever its content changes.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

plugins/agentic-engineering/.claude-plugin/plugin.json
plugins/agentic-engineering/plugin.json

keep manifests/README cross-tool

📄 CodeRabbit inference engine (AGENTS.md)

Files:

.github/plugin/marketplace.json
.claude-plugin/marketplace.json
plugins/agentic-engineering/README.md

🧠 Learnings (1)

📚 Learning: 2026-07-25T12:40:46.973Z

Learnt from: devantler
Repo: devantler-tech/agent-plugins PR: 89
File: plugins/agentic-engineering/skills/portfolio-maintenance/SKILL.md:0-0
Timestamp: 2026-07-25T12:40:46.973Z
Learning: During code review in this repo: (1) For `plugins/agentic-engineering/agents/*.agent.md` (local agent-entrypoints), do NOT require upstream provenance frontmatter—assume these are authored locally; renames/edits to agent-entrypoints should remain local and not be forced through the upstream sync process. (2) For bundled skill artifacts at `plugins/*/skills/*/SKILL.md`, expect upstream provenance frontmatter to be present; do not make direct “local” edits for skill content—skill changes must originate from the upstream source and be brought in via the repo’s sync workflow.

Applied to files:

plugins/agentic-engineering/agents/agent-improver.agent.md

🔇 Additional comments (12)

📝 Walkthrough

Walkthrough

The plugin adds an optional consumer-owned inference-routing contract and documents its policy schema and runtime controls. A new offline evaluator returns RECOMMEND, HOLD, or INVALID decisions without launching inference or reserving quota. Desired-state metadata includes the evaluator and its checksum. Tests cover routing, escalation, quota, policy, and validation cases. Plugin metadata is updated to version 5.1.0.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to c99a5

The optional routing evaluator remains advisory and preserves existing behavior for consumers without the contract. No current merge-blocking risk was identified.

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Docstring Coverage	❌ Error	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 3 files. (9 skipped: 9 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Issue `#209` requires a portable deterministic evaluator with separate eligibility and readiness outcomes. The new evaluator reads one JSON request offline, validates the policy, task, snapshot, model t…
Out of Scope Changes check	✅ Passed	The changes remain within issue `#209`. The version updates maintain manifest parity. The routing contract and role-entrypoint text document the evaluator's optional consumer behavior. The desired-state…
Title check	✅ Passed	The title clearly and concisely describes the main change: explicit evaluation of inference subscription routing policy.
Description check	✅ Passed	The description explains the routing-policy checks, bounded delegation, verified controls, and linked issue. It is directly related to the changeset.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 3 files. (9 skipped: 9 unsupported.)

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

devantler · 2026-09-12T10:16:42Z

@coderabbitai review

The helper-function documentation flagged by the pre-merge check is now present; the runtime-asset digest is refreshed. Behavior is unchanged and all evaluator cases still pass.

coderabbitai · 2026-09-12T10:17:09Z

@devantler: I will review the updated changes.

⚠️ Action not completed

Review rate limited.

devantler · 2026-09-12T10:18:39Z

@codex review

chatgpt-codex-connector · 2026-09-12T10:19:01Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-12T11:09:35.238071Z	`1ddcdbc`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-09-12T10:21:50Z

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: e23fff0705

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

devantler · 2026-09-12T10:31:03Z

@coderabbitai review

coderabbitai · 2026-09-12T10:31:18Z

@devantler: I will review the updated changes.

⚠️ Action not completed

Review rate limited.

devantler · 2026-09-12T10:32:16Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7e3c1521ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

devantler · 2026-09-12T11:04:17Z

@coderabbitai review

coderabbitai · 2026-09-12T11:04:33Z

@devantler: I will review the updated changes.

⚠️ Action not completed

Review rate limited.

devantler · 2026-09-12T11:04:52Z

@codex review

chatgpt-codex-connector · 2026-09-12T11:09:33Z

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 1ddcdbc702

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-project-automation Bot added this to 🌊 Project Board Sep 12, 2026

github-project-automation Bot moved this to 🫴 Ready in 🌊 Project Board Sep 12, 2026

chatgpt-codex-connector Bot reviewed Sep 12, 2026

View reviewed changes

Comment thread plugins/agentic-engineering/agents/agent-improver.agent.md

Comment thread plugins/agentic-engineering/scripts/evaluate-inference-routing.test.sh Outdated

Comment thread plugins/agentic-engineering/scripts/evaluate-inference-routing.sh

botantler-1 Bot closed this Sep 12, 2026

botantler-1 Bot reopened this Sep 12, 2026

github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Sep 12, 2026

github-project-automation Bot moved this from ✅ Done to 🫴 Ready in 🌊 Project Board Sep 12, 2026

devantler added 4 commits September 12, 2026 12:58

feat(agents): evaluate subscription routing policy explicitly

4714e8d

docs(routing): document evaluator helper functions

3b2be86

fix: reject opaque best model alias in routing policy

a5abc2b

fix: make invalid-input regression assertion fail reliably

1ddcdbc

devantler force-pushed the codex/model-routing-209 branch from 7e3c152 to 1ddcdbc Compare September 12, 2026 11:00

devantler marked this pull request as ready for review September 12, 2026 11:11

devantler merged commit 314ca13 into main Sep 12, 2026
41 checks passed

devantler deleted the codex/model-routing-209 branch September 12, 2026 11:11

github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Sep 12, 2026

Uh oh!

Conversation

devantler commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Uh oh!

devantler commented Sep 12, 2026

Uh oh!

coderabbitai Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Pre-merge checks failed

❌ Failed checks (1 error)

Uh oh!

devantler commented Sep 12, 2026

Uh oh!

coderabbitai Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devantler commented Sep 12, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Sep 12, 2026

Uh oh!

devantler commented Sep 12, 2026

Uh oh!

coderabbitai Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devantler commented Sep 12, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devantler commented Sep 12, 2026

Uh oh!

coderabbitai Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devantler commented Sep 12, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devantler commented Sep 12, 2026 •

edited

Loading

coderabbitai Bot commented Sep 12, 2026 •

edited

Loading

coderabbitai Bot commented Sep 12, 2026 •

edited

Loading

coderabbitai Bot commented Sep 12, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 12, 2026 •

edited

Loading

coderabbitai Bot commented Sep 12, 2026 •

edited

Loading

coderabbitai Bot commented Sep 12, 2026 •

edited

Loading