feat(routing): add configurable per-prompt model routing - #1422
feat(routing): add configurable per-prompt model routing#1422mounir0672230294-alt wants to merge 2 commits into
Conversation
📝 WalkthroughWalkthroughPrompt routing adds deterministic classification into ChangesPrompt routing policy
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant RequestHandler
participant evidenceForModelRequest
participant classifyPromptComplexity
participant PolicyEvaluator
participant ModelProvider
Client->>RequestHandler: Send model request
RequestHandler->>evidenceForModelRequest: config, model, request body
evidenceForModelRequest->>classifyPromptComplexity: Extract latest user prompt
classifyPromptComplexity-->>evidenceForModelRequest: Task tier and signals
evidenceForModelRequest->>PolicyEvaluator: Submit routing evidence
PolicyEvaluator->>ModelProvider: Select eligible candidate
ModelProvider-->>Client: Return model response
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
✅ Deterministic PR hygiene checks passed. |
⏳ DRAFT
What to do
Review readiness checklist
0/4 boxes ticked. This PR stays in draft until every box above is ticked. |
mounir0672230294-alt
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@gui/src/i18n/en.ts`:
- Line 78: Update the routing.promptRoutingHelp locale strings to clarify that
prompt classification applies only to requests using the selected profile with
prompt routing enabled. Change the English entry in gui/src/i18n/en.ts at lines
78-78 and add the equivalent Japanese condition in gui/src/i18n/ja.ts at lines
35-35.
In `@src/server/management/routing-profile-routes.ts`:
- Around line 62-67: Update parseEvidence to return a specific parse error code
or message when taskTier is unsupported, rather than only marking validation as
failed. Propagate that error through the dry-run handler so invalid taskTier
requests report the taskTier issue instead of “evidence must be an object,” and
add a management-route test covering taskTier value “huge”.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 0c00d217-2333-4561-855d-09282616b061
⛔ Files ignored due to path filters (1)
docs-site/public/pr-screenshots/1422-smart-prompt-routing.pngis excluded by!**/*.png
📒 Files selected for processing (30)
docs-site/src/content/docs/guides/routing-profile-editor.mddocs-site/src/content/docs/reference/configuration/routing.mddocs-site/src/content/docs/zh-cn/reference/configuration/routing.mdgui/src/i18n/de.tsgui/src/i18n/en.tsgui/src/i18n/ja.tsgui/src/i18n/ko.tsgui/src/i18n/ru.tsgui/src/i18n/tr.tsgui/src/i18n/zh.tsgui/src/pages/RoutingProfiles.tsxgui/src/routing-profile-editor-data.tsgui/tests/routing-profiles.test.tsxsrc/cli/route-policy.tssrc/routing/evaluator.tssrc/routing/profile.tssrc/routing/prompt-classifier.tssrc/routing/request-evidence.tssrc/server/chat-completions.tssrc/server/claude-messages.tssrc/server/management/routing-profile-routes.tssrc/server/responses/compact.tssrc/server/responses/core.tssrc/types.tstests/policy-execution.test.tstests/prompt-classifier.test.tstests/request-evidence.test.tstests/route-explainability.test.tstests/routing-profile-editor-data.test.tstests/routing-profile.test.ts
| "routing.removeCandidate": "Remove candidate {provider}/{model}", | ||
| "routing.candidates": "Candidates", | ||
| "routing.promptRouting": "Automatic prompt routing", | ||
| "routing.promptRoutingHelp": "Classify each latest user prompt locally and select an eligible task tier for every request.", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Scope the prompt-routing help text to enabled profiles.
Both locale strings imply that classification runs for every request. The runtime only generates task-tier evidence when prompt routing is enabled for the selected profile.
gui/src/i18n/en.ts#L78-L78: change the wording to state that classification applies to requests using this profile.gui/src/i18n/ja.ts#L35-L35: add the equivalent Japanese condition.
📍 Affects 2 files
gui/src/i18n/en.ts#L78-L78(this comment)gui/src/i18n/ja.ts#L35-L35
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@gui/src/i18n/en.ts` at line 78, Update the routing.promptRoutingHelp locale
strings to clarify that prompt classification applies only to requests using the
selected profile with prompt routing enabled. Change the English entry in
gui/src/i18n/en.ts at lines 78-78 and add the equivalent Japanese condition in
gui/src/i18n/ja.ts at lines 35-35.
| if (record.taskTier !== undefined) { | ||
| if (record.taskTier !== "fast" && record.taskTier !== "balanced" && record.taskTier !== "powerful") { | ||
| return { evidence: {}, ok: false }; | ||
| } | ||
| evidence.taskTier = record.taskTier; | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Return an accurate validation error for an invalid task tier.
Line 64 rejects an unsupported taskTier, but Line 378 always returns "evidence must be an object". A request with { "taskTier": "huge" } has a valid evidence object, so the response directs the caller to fix the wrong field.
Return a parse error code or message from parseEvidence. Propagate it from the dry-run handler. Add a management-route test for an unsupported taskTier.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/server/management/routing-profile-routes.ts` around lines 62 - 67, Update
parseEvidence to return a specific parse error code or message when taskTier is
unsupported, rather than only marking validation as failed. Propagate that error
through the dry-run handler so invalid taskTier requests report the taskTier
issue instead of “evidence must be an object,” and add a management-route test
covering taskTier value “huge”.
Ingwannu
left a comment
Ingwannu
left a comment
There was a problem hiding this comment.
The direction is valuable, and I found no credential or destination-boundary problem in the submitted diff. I rechecked exact head 7d25ccde: the six focused suites pass 71/71, typecheck passes, and privacy scan passes. I am requesting changes because the execution boundary is not yet safe for a model-routing feature.
-
src/routing/request-evidence.ts:74-79synthesizesbalancedwhenever a request contains no user-text item, andsrc/server/responses/core.ts:1494-1499immediately uses that synthesized tier for route selection. A real Responses continuation can contain onlyprevious_response_idplusfunction_call_output. On this head, an initial complex request selects thepowerfulcandidate, while the tool-result continuation selectsbalanced. If local replay is unavailable but the selected API-key Responses provider can forward native continuation state, the prior response id can therefore be sent to a different model or provider. Preserve the prior tier/selected route for a continuation with no new user prompt, or fail closed when that affinity cannot be proved; do not invent a new balanced turn. Add server-level regressions for replay hit, native replay miss, tool-result-only input, and compact/continuation behavior. -
src/routing/prompt-classifier.ts:16-25,52-54recognizes intent only in English and Chinese, while short length alone subtracts enough to classify an otherwise unknown prompt asfast. I reproducedfastfor short, clearly repository-wide security work in Korean, Japanese, German, Russian, and Turkish—all languages exposed by this GUI. A short prompt is not positive evidence of simplicity. Make unmatched/unsupported-language input default conservatively to at leastbalancedunless a positive simple-intent or explicit low-effort signal exists, and add regressions across the shipped locales. Expanding a brittle keyword list is not required if the fallback contract is conservative. -
classifyPromptComplexity()computes bounded signal ids and a score, butrequest-evidence.ts:75-78discards both and persists only the tier. The PR describes explainability, yetocx logs explaincan show onlyprompt-tier-*, not why that tier was chosen. Either carry the bounded non-content signal ids/score through the route trace and dry-run contract, or narrow the product/docs claim. Do not persist raw prompt text. -
This is a new heuristic policy that changes provider selection, cost, and possibly continuation ownership, but there is no repository
[Decision Log]. Document the intent, alternatives, multilingual fallback, continuation/turn-stability rule, heuristic/versioning contract, and tradeoffs in the applicablestructure/document. The two current CodeRabbit threads also remain unresolved.
Finally, the head is 148 commits behind current dev@20c513198, the readiness checklist is 0/4, and there is no exact-head full CI. Rebase after the behavior blockers are fixed, rerun the focused GUI/runtime/docs checks, then require full exact-head CI before marking ready. This should remain a draft until those gates are complete.
2 participants
Summary
fast,balanced, andpowerfultask tiers.Verification
bun run typecheckbun run test tests/prompt-classifier.test.ts tests/request-evidence.test.ts tests/routing-profile.test.ts tests/policy-execution.test.ts tests/route-explainability.test.ts tests/routing-profile-editor-data.test.tscd gui && bun test tests/routing-profiles.test.tsx ../tests/routing-profile-editor-data.test.tscd gui && bun run lintcd gui && bun run lint:i18ncd gui && bun run buildcd docs-site && bun run buildbun x --yes react-doctor@0.9.11 --verbose --scope changed --base origin/dev --no-telemetrybun run privacy:scanbun run testScreenshot
Live local dashboard capture from opencodex v2.10.2 showing
Automatic prompt routingenabled, theautoalias, and explicit fast / balanced / powerful candidate assignments. The profile was also dry-run locally for all three tiers plus image input.Checklist
Review readiness checklist
This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:
All CI tests are green on my local testing.
I pushed my PR to the latest dev commit.
I resolved all correct Codex and CodeRabbit findings.
My PR is ready for review.
Summary by CodeRabbit
--task-tiersupport to policy dry runs and evaluation commands.