Repository navigation
Verification gate: pluggable deterministic checks as escalation trigger #1135
Description
Parent: #1134
The escalation trigger is deterministic verification, not model self-report:
- Define a
verify_gateinterface: named checks that return pass/fail + a machine-readable reason. v1 checks: project build succeeds; declared data tests pass; schema-contract diff (written models/columns match the task's declared contract when one exists). - Gate runs identically after primary AND escalation attempts (measurement needs both lanes' outcomes).
- Configurable per project:
route.gate: [build, tests, contract]; sensible defaults; a check that cannot run (e.g. no tests declared) reports "skipped", never blocks. - Gate results are part of the session record (consumed by measurement subtask).
- Tests: each check's pass/fail/skipped paths; gate composition; failure-reason payloads.
Activity
[-]Cascade routing core: two-model user selection + escalate-on-gate-failure flow[/-][+]Verification gate: pluggable deterministic checks as escalation trigger[/+]anandgupta42 commentedon Aug 24, 2026
Evidence note from the prior-art research (parent #1134): gate discriminativeness is an ongoing investment, not a one-time checkbox — CHASE-SQL's execution selector captures only part of the oracle headroom (73.01% vs 82.79% oracle, arxiv.org/html/2410.01943v1) and TestPrune shows improving test quality on an already-gated pipeline adds +2.4 to +12.9pts (arxiv.org/html/2510.18270v2). The gate interface must make adding/refining checks cheap. Also: prefer project-declared tests over model-generated ones — the literature's verifier-gaming failure mode applies to self-generated checks.
anandgupta42 commentedon Nov 5, 2026
Closing: after evaluating verifier-gated and cascade routing, we decided not to build an inferred per-request router. The direction is a strong default model with explicit, user-chosen modes. Reopen if routing is revisited.