diff --git a/CLAUDE.md b/CLAUDE.md index db8d2615..392c29e2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -141,8 +141,8 @@ action.yml # Published composite GitHub Action (coder-ev - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly. -- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `max_steps_to_decide`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step-budget breach to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. -- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm — "is this criterion type live-observable" is `models.LiveSuccessCriterion` subclassing (currently `skill_triggered`, `command_executed`), the single source of truth `validate_early_stop`/`EarlyStopWatcher` check directly via `isinstance`; each subclass implements the abstract, checker-independent `live_decidable_polarities()` (a pure function of its own fields) alongside the checker's `live_verdict` override, and lint rule CE025 (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`, a registry-based whole-tree check, not a per-file AST rule) keeps the two paired. `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing pre-weighting behavior byte-for-byte) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed subset's **ceiling** (best case for everything still undecided) can no longer reach the threshold, a pass-stop once the pass-armed subset's **floor** (worst case) already meets it — both **deferred while any pass-armed criterion is undecided**, so a distractor misfire never truncates a positive row's recall signal before the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. A per-criterion `max_steps_to_decide` (on `LiveSuccessCriterion` only, requires `stop_when`) caps tool-call steps spent still undecided — cumulative across retry attempts of the same turn — before `EarlyStopReason.DECISION_BUDGET_EXCEEDED` force-fails the run outright, bypassing the weighted gate (nothing to weigh a criterion that never decided against). Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally (even with `stop_early: true`) gates on the full set via the strict-AND `all_criteria_passed` — weight magnitude only forgives under the former, so the weighted gate is contingent on the watcher itself firing, not solely on the configured threshold. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged. +- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. +- **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged. ## Success Criteria (14 types) @@ -163,7 +163,7 @@ action.yml # Published composite GitHub Action (coder-ev | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) | | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. | -All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`/`auto`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only; `auto` arms the instance's own decidable polarities). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum. +All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus (on live criteria only) a `stop_early:` block (`on_pass`, `decide_within`) that arms the criterion for early stop by its presence. On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum. ## Evaluation Flow diff --git a/docs/AB_EXPERIMENTS.md b/docs/AB_EXPERIMENTS.md index 56d483ae..040d9b97 100644 --- a/docs/AB_EXPERIMENTS.md +++ b/docs/AB_EXPERIMENTS.md @@ -268,11 +268,12 @@ variant's, on the already-mutated prompt. ## Recipe: Smoke vs. e2e Flavors (Early Stop) Run the **same** task file as both a fast `smoke` flavor and a full `e2e` flavor -by flipping one boolean per variant — `run_limits.stop_early`. Arm the criteria -that define "the interesting thing happened" with `stop_when` in the task file; +with a one-line kill switch on the reference variant. Arm the criteria +that define "the interesting thing happened" with `stop_early:` blocks in the +task file; the `smoke` variant cuts off as soon as they're decided, while `e2e` runs to completion. Because the field merge is per-key, the variant sets only -`stop_early` without disturbing the task's `max_turns`. +`stop_early` (the run-level kill switch) without disturbing the task's `max_turns`. ```yaml experiment_id: early-stop-ab @@ -281,13 +282,12 @@ description: "Smoke vs. e2e from one file via opt-in early stop" variants: - variant_id: e2e run_limits: - stop_early: false # full run to completion (the reference flavor) + stop_early: false # kill switch: force-disarm the blocks (the reference flavor) - variant_id: smoke - run_limits: - stop_early: true # cut off once the armed criteria are decided + # no override needed: the task's stop_early: blocks arm the watcher ``` -The task file supplies the arming (`stop_when` on the criteria that gate the +The task file supplies the arming (`stop_early:` blocks on the criteria that gate the flavor) and a `max_turns` generous enough for `e2e`; see [`stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop). This recipe ships as `experiments/early-stop-ab.yaml`. diff --git a/docs/DIALOG_MODE.md b/docs/DIALOG_MODE.md index e9697a82..0b4bad29 100644 --- a/docs/DIALOG_MODE.md +++ b/docs/DIALOG_MODE.md @@ -69,9 +69,10 @@ calls the agent's `communicate()`, so Codex and plugin agents work. The *simulat always a Claude Code agent. A dialog run therefore needs the `claude` CLI on `PATH` and working Anthropic or Bedrock credentials **even when the agent under test is not Claude**. -**Early stop is not available here.** [`run_limits.stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop) -is single-shot-only; arming it alongside `simulation.enabled` is a hard error at resolution time, not -a silent no-op. Use `stop_on_criteria_pass` (below) for the dialog equivalent. +**Early stop is not available here.** [Criterion-level `stop_early:` arming](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop) +is single-shot-only; an armed criterion alongside `simulation.enabled` is a hard error at resolution +time, not a silent no-op (disarm with the `run_limits.stop_early: false` kill switch to run anyway). +Use `stop_on_criteria_pass` (below) for the dialog equivalent. Every field's default and constraint lives in one place — the [Task Definition Guide's simulation section](TASK_DEFINITION_GUIDE.md#simulation-multi-turn-user-dialog). diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index 288cabf2..6c681bcd 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -109,7 +109,7 @@ turn, and `ToolStart`/`ToolEnd` per tool call (close orphaned tools with returned `TurnRecord`, the single agent-agnostic capture path. Set `supports_cooperative_stop: ClassVar[bool] = True` only if your `communicate()` -actually honors `should_stop` (needed for `run_limits.stop_early`). Leaving it +actually honors `should_stop` (needed for criterion-level `stop_early:` arming). Leaving it `False` means early stop is rejected at resolution for your agent — which is correct if you can't stop cooperatively. @@ -146,7 +146,7 @@ class MyCriterion(BaseSuccessCriterion): Union membership is required — a run validates that every union member's `type` has a registered checker, and rejects unknown `type` tags in YAML. `BaseSuccessCriterion` gives you `description`, `weight` (default 1.0; `0` = informational/non-gating), -`pass_threshold` (default 0.9), `stop_when`, and `suite_thresholds` for free, with +`pass_threshold` (default 0.9) and `suite_thresholds` for free, with `extra="forbid"` so YAML typos are caught. ### Step 2 — the checker diff --git a/docs/REPORT_SCHEMA.md b/docs/REPORT_SCHEMA.md index 3fb492c5..df4bf740 100644 --- a/docs/REPORT_SCHEMA.md +++ b/docs/REPORT_SCHEMA.md @@ -189,8 +189,9 @@ backend), `num_turns`, `max_turns_exhausted`, Present (non-`null`) iff the run stopped early — there is no separate boolean. Fields: `reason` (`criterion_passed` / `criterion_failed` / -`decision_budget_exceeded` — the last forces `FinalStatus.FAILURE` outright, -bypassing the weighted gate), +`decision_budget_exceeded` — the last marks a fail-stop whose deciding +criterion timed out undecided past its `stop_early.decide_within`; it gates through +the same weighted armed gate as a native fail), `deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`, `sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call), `elapsed_seconds`, `turns_remaining_at_stop`, `gate_threshold` (the diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index 69cbde15..3da43187 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -243,8 +243,8 @@ run_limits: max_total_tokens: 200000 # cumulative input + output max_usd: 2.50 # cumulative cost - # Early stop - stop_early: true # end once the armed criteria are decided + # Early stop (kill switch only — arming lives on the criteria) + stop_early: false # force-disarm every criterion's stop_early: block ``` | Field | Default | Constraint | Description | @@ -259,8 +259,8 @@ run_limits: | `max_usd` | *unset* | `> 0.0` | Max cumulative cost in USD. Requires per-turn SDK cost reporting. | | `count_cached_input` | `false` | — | Count `cache_read_input_tokens` toward the input/total budgets. Off by default — cached reads are typically free. | | `count_cache_creation` | `false` | — | Count `cache_creation_input_tokens` toward the input/total budgets. Off by default. | -| `stop_early` | `false` | — | Opt-in master switch for early-stop-on-criterion. See [`stop_early`](#stop_early-opt-in-early-stop). | -| `stop_early_gate_threshold` | `1.0` | `[0.0, 1.0]` (but `> 0.0` is enforced at resolution when `stop_early: true`) | Minimum weighted score over the armed subset required to gate as a pass. See [`stop_early`](#stop_early-opt-in-early-stop). | +| `stop_early` | *unset* | `false` or unset | Run-level early-stop **kill switch** — there is no master arm. Unset: the criteria's own `stop_early:` blocks decide. `false`: force-disarm every block for this run. `true` (the removed master arm) is rejected at resolution. See [`stop_early`](#stop_early-opt-in-early-stop). | +| `stop_early_gate_threshold` | `1.0` | `[0.0, 1.0]` (but `> 0.0` is enforced at resolution on an armed task) | Minimum weighted score over the armed subset required for an **early-stopped** run to gate as a pass. See [`stop_early`](#stop_early-opt-in-early-stop). | The authoritative source is `src/coder_eval/models/limits.py`. A lint rule (CE030) fails the build if a field defined there goes undocumented in this guide, so the table can't quietly fall behind the @@ -326,125 +326,153 @@ default) to exclude a task from the metric entirely. ### `stop_early` (opt-in early stop) -`run_limits.stop_early` (default `false`) ends a single-shot run **early** once -the run's **armed** criteria are decided — so you can raise `max_turns` for the -full-run flavor without paying for turns the smoke flavor doesn't need. A -criterion is *armed* by giving it a `stop_when` (see the criterion-fields table); -`stop_early` is the master switch that turns arming on for the run. +Early stop ends a single-shot run **early** once the run's **armed** criteria +decide the outcome — so you can raise `max_turns` for the full-run flavor +without paying for turns the smoke flavor doesn't need. A criterion is *armed* +by attaching a **`stop_early:` block** to it — the block's presence IS the +arming, and it alone activates the run's watcher; there is **no run-level +master switch**. (Live-observable criteria only: the block field exists only on +`skill_triggered` / `command_executed`, so arming anything else is a schema +error, not a runtime surprise.) `run_limits.stop_early: false` is the run-level +**kill switch** that force-disarms every block — the one-line experiment/CLI +override that turns a smoke flavor back into an authoritative full run; +`run_limits.stop_early: true` (the removed master arm) is rejected at +resolution. + +Arming carries one **implicit** trigger — a definitive *effective* fail (a +native live-fail, or the `decide_within` timeout expiring) may end the run +under the weighted ceiling rule — plus two knobs inside the block: + +| Block | Meaning | +|-------|---------| +| `stop_early: {}` | armed: fail-stop on a native live-fail (the idiomatic distractor arming) | +| `stop_early: {on_pass: stop}` | …plus pass-stop the moment the criterion live-passes | +| `stop_early: {decide_within: N}` | …plus an *effective* fail if still **undecided** after N tool-call steps (reported as `decision_budget_exceeded`) | ```yaml run_limits: max_turns: 30 - stop_early: true # opt in; default false leaves behavior unchanged success_criteria: - type: skill_triggered skill_name: date-teller expected_skill: date-teller - stop_when: auto # arm whichever polarity this instance can decide - - type: file_exists # not armed → advisory on an early-stopped run + stop_early: + decide_within: 5 # not loaded within 5 steps → effective fail → stop + - type: file_exists # no block → unarmed (advisory on an early-stopped run) path: report.md ``` +The two intents compose cleanly: `decide_within` with the default +`on_pass: continue` means *"fail fast if the signal doesn't arrive in time, +but if it does arrive, keep running"* (a live PASS never stops the run — it +only **latches**, so the criterion is not re-checked). Set `on_pass: stop` +when the signal arriving makes the rest of the run redundant and you want to +bank the saved turns. + Semantics: -- **Opt-in, per run.** With `stop_early: false` (the default) the run behaves - exactly as before — `stop_when` is inert and every criterion gates normally. -- **Polarity.** `stop_when: pass` stops the moment all **pass-armed** criteria are - decided in the pass direction; `stop_when: fail` stops on a definitive - wrong-signal fail; `stop_when: decided` stops on either (the criterion instance - must be able to decide **both**). `stop_when: auto` arms whichever polarities - **this instance** can decide — use it when the decidable polarity is - instance-dependent, e.g. a `skill_triggered` activation suite where a positive - row (`skill_name == expected_skill`) can only live-pass and a distractor can only - live-fail, so one static value on a dataset-fanned criterion (whose - positive/distractor role flips per row) cannot fit every row. A **pass-stop** - needs every pass-armed criterion to pass — fail-armed distractors are not - required to, and a row with **zero** pass-armed criteria (e.g. a negative row) - never pass-stops; a **fail-stop** fires on the first fail-armed criterion that - live-fails, but is **deferred while any pass-armed criterion is still - undecided** — a distractor misfire on an early tool call must not cut a - positive row before its expected signal can appear (that would freeze a - would-be true positive as a false negative and deflate suite recall). The - misfire is latched, so the deferred fail-stop fires the moment every - pass-armed criterion decides; if none ever decides, the run simply continues - to the cap. Only criteria that can decide from a partial trajectory (currently - `skill_triggered`, `command_executed`) may be armed — arming any other criterion - is a hard error at resolution (plan *and* run), never a silent no-op. - Decidability can also depend on a criterion's own fields: `command_executed` can - live-**pass** only with `max_count` unset and `min_count > 0`, and live-**fail** - only with `max_count` set (which includes the `min_count: 0, max_count: 0` - "must-NOT-run" form). Arming a polarity the configured criterion can never reach - (e.g. `stop_when: pass` alongside a `max_count`, or `auto` on an instance that - can decide neither) is likewise a hard error at resolution, not a silent full - run. -- **Verdict.** Any task armed for early-stop (`stop_early: true`) is gated on - the **armed subset only** — the non-armed criteria become **advisory** and - are clearly marked (report badge + per-criterion note + `stopped_early` - row when the watcher actually fired) — whether or not the watcher actually - cut the run short; one task config maps to one gate semantic. Only a task - that never armed `stop_early` at all is gated on the **full** set, as - always. This is what lets one file serve both a `smoke` flavor - (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) — +- **Opt-in, per criterion.** With no `stop_early:` block anywhere the run + behaves exactly as before — there is no watcher at all. The + `run_limits.stop_early: false` kill switch force-disarms an armed task for + one run (e.g. an experiment's `e2e` variant, or + `-D run_limits.stop_early=false` from the CLI) without touching the + criteria. +- **Inert-by-design triggers (dataset fan-out).** A trigger whose polarity this + *instance* can never decide is silently inert, not an error: a positive + `skill_triggered` row (`skill_name == expected_skill`) can only live-pass, so + the implicit fail trigger does nothing on it; a distractor row can only + live-fail, so `on_pass: stop` and `decide_within` do nothing on it. That is + what lets **one** dataset-fanned YAML line — same block on every row — serve + both positive rows (pass/timeout live) and distractor rows (fail live) + without per-row conditionals. Decidability can also depend on a criterion's own + fields: `command_executed` can live-**pass** only with `max_count` unset and + `min_count > 0`, and live-**fail** only with `max_count` set (which includes + the `min_count: 0, max_count: 0` "must-NOT-run" form). +- **Verdict latching.** Once an armed criterion decides (pass or fail), its + live verdict is latched and never re-computed — the observable criteria are + monotonic (an engaged skill stays engaged), so re-polling is pure waste. +- **Fail-stop rule (weighted ceiling).** A fail-stop candidate is any armed + criterion whose *effective* verdict is fail — a native live-fail (the + implicit trigger every armed criterion carries), or an expired + `decide_within` timeout. The stop fires + only once the armed set's **ceiling** (best case: every still-undecided or + already-passed criterion ends up scoring 1.0, every failed one scores 0) can + no longer reach `stop_early_gate_threshold` — the gate is mathematically + guaranteed to fail regardless of how the trajectory continues. It is also + **deferred while any pass-capable armed criterion is still undecided** — a + distractor misfire on an early tool call must not cut a positive row before + its expected signal can appear (that would freeze a would-be true positive as + a false negative and deflate suite recall). The misfire is latched, so the + deferred fail-stop fires the moment every pass-capable criterion decides; if + none ever decides, the run simply continues to the cap. +- **Pass-stop rule (weighted floor).** A pass-stop fires once the + `on_pass: stop` subset's **floor** (worst case: every still-undecided member + scores 0) already meets the threshold. Distractors are excluded from this + bound (they can never live-pass); a task with **zero** `on_pass: stop` + criteria never pass-stops. Like the fail-stop, it is **deferred while any + pass-capable armed criterion outside the `on_pass: stop` subset is still + undecided** (subset members are already priced into the floor) — otherwise + an early pass would truncate a sibling `on_pass: continue` criterion's + expected signal out of the trajectory and freeze it as an unearned fail on + the armed gate. This deferral is what lets `on_pass: stop` and a sibling's + `decide_within` compose safely on the same task. +- **Verdict (fired-only gating).** A run the watcher actually **cut short** is + gated on the **armed subset only** — on a truncated trajectory the unarmed + criteria never had the chance to be satisfied, so they become **advisory** + and are clearly marked (report badge + per-criterion note + `stopped_early` + row). A run that **completes naturally** — armed or not — has a full + trajectory and gates strict-AND over the **full** set, as always: adding a + block (e.g. a `decide_within` fail-fast timeout) never changes the verdict + of a run it didn't cut. Precisely: the gate keys on the watcher having + **fired** (`result.early_stop is not None`), not on confirmed truncation — + an agent that ignores `should_stop`, or a stop that fires on the run's + final message, still gates armed-only. This is what lets one file serve both a `smoke` + flavor (blocks armed) and an `e2e` flavor (`stop_early: false` kill switch) — see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md). Verdict parity between the flavors is one-sided: a **fail-stop** is verdict-preserving (the deferral above - guarantees every pass-armed signal was allowed to resolve first), but a + guarantees every pass-capable signal was allowed to resolve first), but a **pass-stop** cuts the run once the positives are decided, so a distractor that would misfire on a *later* tool call is not observed (the frozen row scores as a clean pass) — the smoke flavor trades some precision completeness for budget, so - authoritative precision/recall belongs on the `stop_early: false` run. + authoritative precision/recall belongs on the kill-switched + (`run_limits.stop_early: false`) run. The same one-sidedness applies to an armed + criterion that is fail-only-decidable but still needs evidence to *pass* — e.g. + `command_executed` with `min_count: 1` **and** `max_count` set: the pass-stop + deferral holds only for pass-capable siblings, so an `on_pass: stop` sibling can + cut the run before the minimum count is reached and the armed gate scores that + criterion 0. Score such combinations authoritatively on the kill-switched run. - **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) — it can never silently disable a criterion or cause a false early stop. - **Weighting.** `run_limits.stop_early_gate_threshold` (default `1.0`) is the minimum weighted score (`Σ weight·score / Σ weight`, over the armed subset) required to gate as a pass — both for the post-hoc verdict and for the live - stop rule itself. A fail-stop fires once the armed subset's **ceiling** (best - case: every still-undecided or already-passed criterion ends up scoring 1.0, - every live-failed one scores 0) can no longer reach the threshold — the gate - is mathematically guaranteed to fail regardless of how the trajectory - continues. A pass-stop fires once the pass-armed subset's **floor** (worst - case: every still-undecided one scores 0) already meets it. At the default - `1.0` both bounds collapse to the pre-weighting rules above exactly (any - single armed criterion's live-fail already drops the ceiling below 1.0, and - the floor only reaches 1.0 once every pass-armed criterion has actually - passed) — lowering it lets a low-weight armed criterion's failure be absorbed - without truncating the run, at the cost of the gate becoming a genuine - weighted average rather than a strict AND. **The armed weighted gate applies - whenever `stop_early: true` is set — one task config, one gate semantic — - regardless of whether the watcher actually fired a stop.** A task armed for - early-stop that instead completes naturally (the agent finishes, or - `max_turns` is hit, before the bound ever trips) is gated on the *same* - weighted armed-subset formula as an actual early stop, not the full-run - `all_criteria_passed`; only a task that never armed `stop_early` at all uses - the strict full-set gate. Each armed criterion's own `pass_threshold` still - decides whether it individually passed (converted to a binary 1.0/0.0 + stop rules above. At the default `1.0` the bounds collapse to strict rules + exactly (any single armed criterion's effective fail already drops the + ceiling below 1.0, and the floor only reaches 1.0 once every `on_pass: stop` + criterion has actually passed) — lowering it lets a low-weight armed + criterion's failure **or timeout** be absorbed without truncating the run, + at the cost of the gate becoming a genuine weighted average rather than a + strict AND. **The armed weighted gate applies only to a run the watcher + actually cut** (fired-only gating, see *Verdict* above); a run that + completes naturally gates on the full-set `all_criteria_passed` regardless + of arming. Each armed criterion's own `pass_threshold` + still decides whether it individually passed (converted to a binary 1.0/0.0 before weighting) — only the combination rule (weighted average vs strict AND) changes, which is what makes the `gate_threshold=1.0` default an exact - equivalence with the pre-weighting `all(...)` rule. -- **Decision-step budget.** `max_steps_to_decide` (per armed criterion, only - on `skill_triggered` / `command_executed`, requires `stop_when`) caps how - many tool-call steps that criterion may spend still **undecided** before the - run gives up on it: - - ```yaml - success_criteria: - - type: skill_triggered - description: "date-teller must activate within 5 steps" - skill_name: date-teller - expected_skill: date-teller - stop_when: pass - max_steps_to_decide: 5 - ``` - - Once the cap is exceeded (checked AFTER the normal fail-/pass-stop checks - each round, so a criterion that decides on that very step is never - penalized), the watcher fires `reason: decision_budget_exceeded` and the run - is forced to `FinalStatus.FAILURE` outright — bypassing - `stop_early_gate_threshold`'s weighted gate entirely, since a criterion that - never reached a verdict has nothing meaningful to weigh against the others. - `None` (default) = no cap; the run relies solely on `run_limits.max_turns`. - The step count is **cumulative across every retry attempt** of the turn — - including an attempt that crashed or timed out before this criterion's own - investigation even began — so size the budget with that headroom in mind. + equivalence with the strict `all(...)` rule. +- **Decision-step timeout.** `stop_early: {decide_within: N}`. If the + criterion is still **undecided** + after N tool-call steps, the watcher latches an **effective fail** for it and + the normal fail-stop ceiling rule applies — reported as + `reason: decision_budget_exceeded` so an analysis can tell a timeout from a + native misfire, but gated identically (a low-weight criterion's timeout that + cannot doom the gate is absorbed, and the run continues). The timeout is + checked after the criterion's own verdict each round, so one that decides on + that very step is never penalized. `None` (default) = no timeout; the run + relies solely on `run_limits.max_turns`. The step count is **cumulative + across every retry attempt** of the turn — including an attempt that crashed + or timed out before this criterion's own investigation even began — so size + the budget with that headroom in mind. Observability (every early-stopped run is flagged everywhere so analysis never compares a truncated run against a full one): @@ -589,8 +617,7 @@ All criteria share these fields: | `description` | — | Human-readable description (required) | | `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate | | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass | -| `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`/`auto`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). `auto` arms whichever polarity this instance can decide (for dataset-fanned criteria whose positive/distractor role flips per row). See [`stop_early`](#stop_early-opt-in-early-stop). | -| `max_steps_to_decide` | `null` | **Only on live-observable criteria** (`skill_triggered`, `command_executed`) — requires `stop_when` to be set. Caps the tool-call steps this armed criterion may spend still undecided before the run gives up and force-fails. See [`stop_early`](#stop_early-opt-in-early-stop). | +| `stop_early` | `null` | **Only on live-observable criteria** (`skill_triggered`, `command_executed`). Presence arms the criterion for early stop (no run-level switch needed): an effective fail may end the run (weighted ceiling rule, recall deferral). Keys: `on_pass: stop\|continue` (default `continue`), `decide_within: N` (timeout → effective fail, reported as `decision_budget_exceeded`). Inert triggers by design on instances that can't decide their polarity (dataset fan-out support). See [`stop_early`](#stop_early-opt-in-early-stop). | **Scoring types:** - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered` @@ -600,7 +627,7 @@ All criteria share these fields: **Task success:** all *gating* criteria must score >= their `pass_threshold`. A criterion with `weight: 0` is informational — it is still checked, stored, and rendered in reports, but it neither contributes to the score nor fails the task. -(A `weight: 0` criterion may not set `stop_when` or `suite_thresholds`: arming a +(A `weight: 0` criterion may not set a `stop_early` block or `suite_thresholds`: arming a non-gating criterion for the early-stop or suite gate would let an "informational" check flip a run to failure.) diff --git a/docs/agents/CLAUDE_CODE.md b/docs/agents/CLAUDE_CODE.md index 7bc830a0..66670709 100644 --- a/docs/agents/CLAUDE_CODE.md +++ b/docs/agents/CLAUDE_CODE.md @@ -148,9 +148,10 @@ simulator force `[]` for the same reason.) ## Early stop Claude Code supports the cooperative early-stop seam, as do the -[Codex](CODEX.md) and [Antigravity](ANTIGRAVITY.md) agents. With -`run_limits.stop_early: true`, a single-shot run ends cleanly at the next -tool-call boundary once its **armed** criteria (`stop_when: pass|fail|decided`) are +[Codex](CODEX.md) and [Antigravity](ANTIGRAVITY.md) agents. When a criterion +carries a `stop_early:` block, a single-shot run ends cleanly at the next +tool-call boundary once its **armed** criteria (those carrying a +`stop_early:` block) are decided — so a raised `max_turns` isn't wasted on a smoke run. Early stop errors at resolution for any agent that does not declare `supports_cooperative_stop`. See the [Task Definition Guide](../TASK_DEFINITION_GUIDE.md) for the full contract. diff --git a/docs/tutorials/04-writing-a-task.md b/docs/tutorials/04-writing-a-task.md index dfb8a296..eece8ffa 100644 --- a/docs/tutorials/04-writing-a-task.md +++ b/docs/tutorials/04-writing-a-task.md @@ -102,7 +102,7 @@ created are preserved under `runs/latest////` (`task.json`, ## Where to go deeper - **All 14 criterion types, weights, thresholds** → [Task Definition Guide](../TASK_DEFINITION_GUIDE.md) -- **Stop a run early once the key criteria are decided** (opt-in `run_limits.stop_early` + `stop_when` on a criterion) → [Task Definition Guide → `stop_early`](../TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop) +- **Stop a run early once the key criteria are decided** (an opt-in `stop_early:` block on a criterion; `run_limits.stop_early: false` is the run-level kill switch) → [Task Definition Guide → `stop_early`](../TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop) - **Fan one task out over a dataset of rows** → [Bring Your Own Dataset](../DATASETS.md) - **Full CLI & config reference** → [User Guide](../USER_GUIDE.md) - **Compare two configurations on this task** → [Tutorial 05](05-comparing-models.md) diff --git a/experiments/early-stop-ab.yaml b/experiments/early-stop-ab.yaml index ba40e784..0edaefdb 100644 --- a/experiments/early-stop-ab.yaml +++ b/experiments/early-stop-ab.yaml @@ -1,11 +1,12 @@ experiment_id: early-stop-ab description: | Smoke vs. e2e flavors from ONE task file via opt-in early-stop. Both variants - share the same tasks, criteria, and max_turns; they differ only in the - run_limits.stop_early boolean (field-merged, so neither replaces the task's - run_limits block). The task's armed criteria (those carrying `stop_when`) - decide when the `smoke` variant cuts off — as soon as the designated criteria - are decided — while `e2e` runs every task to completion. + share the same tasks, criteria, and max_turns. Arming is per-criterion (the + `stop_early:` blocks in the task file), so the `smoke` variant needs NO + override at all — the armed criteria decide when it cuts off. The `e2e` + variant throws the run-level kill switch (`run_limits.stop_early: false`, + field-merged, so it does not replace the task's run_limits block) to + force-disarm every block and run every task to completion. Expect the `smoke` variant significantly lower on turns, duration, and tokens. Verdict parity is one-sided: the armed subset gates an early-stopped run @@ -20,6 +21,5 @@ variants: stop_early: false - variant_id: smoke - description: "Opt-in early stop — cut off once the armed criteria are decided." - run_limits: - stop_early: true + description: "Early stop as armed in the task — cut off once the armed criteria are decided." + # No override needed: the task's per-criterion stop_early: blocks arm the watcher. diff --git a/src/coder_eval/cli/plan_command.py b/src/coder_eval/cli/plan_command.py index 55778c92..264d863d 100644 --- a/src/coder_eval/cli/plan_command.py +++ b/src/coder_eval/cli/plan_command.py @@ -134,7 +134,7 @@ def plan_command( for variant in exp_def.variants: try: resolved, _lineage, _ = resolve_task_for_variant(default_exp, task, exp_def, variant) - # Early-stop guardrails (no-op unless run_limits.stop_early is armed). + # Early-stop guardrails (no-op unless a criterion carries a stop_early: block). validate_early_stop(resolved) agent_type = str(resolved.agent.type) if resolved.agent else "unknown" agent_model = resolved.agent.model if resolved.agent else None diff --git a/src/coder_eval/models/__init__.py b/src/coder_eval/models/__init__.py index 47927cd0..cf081ee1 100644 --- a/src/coder_eval/models/__init__.py +++ b/src/coder_eval/models/__init__.py @@ -47,6 +47,7 @@ RegexPattern, RunCommandCriterion, SkillTriggeredCriterion, + StopEarlyPolicy, SuccessCriterion, UiPathEvalCriterion, ) @@ -79,7 +80,7 @@ from coder_eval.models.judge_defaults import DEFAULT_JUDGE_MODEL # Limits -from coder_eval.models.limits import RunLimits +from coder_eval.models.limits import DEFAULT_STOP_EARLY_GATE_THRESHOLD, RunLimits # Merge strategy from coder_eval.models.merge_strategy import ( @@ -234,6 +235,7 @@ "LLMJudgeCriterion", "AgentJudgeCriterion", "SkillTriggeredCriterion", + "StopEarlyPolicy", "LiveSuccessCriterion", "LivePolarity", "SuccessCriterion", @@ -313,6 +315,7 @@ "simulator_cost_usd", # Judge defaults "DEFAULT_JUDGE_MODEL", + "DEFAULT_STOP_EARLY_GATE_THRESHOLD", # Judge "JudgeVerdict", # Limits diff --git a/src/coder_eval/models/criteria.py b/src/coder_eval/models/criteria.py index 26a1142c..77be9c9e 100644 --- a/src/coder_eval/models/criteria.py +++ b/src/coder_eval/models/criteria.py @@ -99,13 +99,15 @@ class BaseSuccessCriterion(BaseModel, ABC): weight: float = Field( default=1.0, ge=0.0, + allow_inf_nan=False, description=( "Relative importance of this criterion in the weighted score (default: 1.0). Set to 0 to " "make the criterion purely INFORMATIONAL -- useful for side-effect checks (e.g. a setup " "command): it is excluded from the weighted score AND from the pass/fail gate, so scoring " "below its pass_threshold no longer flips the task to FAILURE. The result is still " - "computed, stored, and rendered in reports. A weight=0 criterion may not set stop_when " - "or suite_thresholds (arming a non-gating criterion for a pass/fail gate is incoherent)." + "computed, stored, and rendered in reports. A weight=0 criterion may not set a stop_early " + "block or suite_thresholds (arming a non-gating criterion for a pass/fail gate is " + "incoherent)." ), ) @@ -123,26 +125,22 @@ class BaseSuccessCriterion(BaseModel, ABC): ), ) - stop_when: Literal["pass", "fail", "decided", "auto"] | None = Field( - default=None, - description=( - "Opt-in early-stop polarity for this criterion (membership in the run's 'armed set'). " - "None (default) = not a stop criterion. 'pass' = a live PASS may contribute to a stop; " - "'fail' = a live definitive FAIL may trigger a stop; 'decided' = either (the instance " - "must be able to decide BOTH polarities). 'auto' = arm whichever polarities THIS instance " - "can actually decide (its live_decidable_polarities) - use it when the decidable polarity " - "is instance-dependent, e.g. skill_triggered under any-engagement, where a positive row " - "(skill_name == expected_skill) can only live-pass and a distractor can only live-fail, " - "so no single static polarity fits every fanned-out row. Inert unless " - "run_limits.stop_early is True. Only valid on criteria observable mid-run (e.g. " - "skill_triggered, command_executed); an unobservable armed criterion is rejected at " - "resolution time." - ), - ) - requires_agent: ClassVar[bool] = False """True if this criterion requires agent turn records to evaluate correctly.""" + @property + def is_stop_armed(self) -> bool: + """True when this criterion participates in the run's early-stop armed set. + + Always ``False`` on the base: only live-observable criteria + (``LiveSuccessCriterion`` subclasses) carry stop triggers, so arming an + unobservable criterion is unrepresentable rather than a validation + error. The armed set drives both the runtime watcher + (``EarlyStopWatcher``) and the weighted armed gate + (``EvaluationResult.armed_criteria_passed``). + """ + return False + def model_post_init(self, context: Any, /) -> None: # Pin the discriminator tag into model_fields_set so it survives # model_dump(exclude_unset=True) → model_validate() round-trips even for @@ -156,21 +154,27 @@ def check_weight_zero_is_not_gating(self) -> Self: """Reject ``weight: 0`` combined with any gate-arming field. ``weight: 0`` makes a criterion informational — excluded from the score - and from the pass/fail gate (see ``is_gating``). Both ``stop_when`` (arms - the per-row early-stop gate) and ``suite_thresholds`` (arms the - across-row suite gate, which drives the run's exit code) would let an - "informational" criterion flip a run to failure — directly contradicting - the field's contract. So both combinations are authoring errors, caught - at load time rather than surfacing as a confusing exit code later. + and from the pass/fail gate (see ``is_gating``). Both the early-stop + triggers (``is_stop_armed``: arms the per-row early-stop gate) and + ``suite_thresholds`` (arms the across-row suite gate, which drives the + run's exit code) would let an "informational" criterion flip a run to + failure — directly contradicting the field's contract. So both + combinations are authoring errors, caught at load time rather than + surfacing as a confusing exit code later. """ if self.weight == 0.0: - for field, name in (("stop_when", "stop_when"), ("suite_thresholds", "suite_thresholds")): - if getattr(self, field) is not None: - raise ValueError( - f"criterion {self.type!r}: weight=0 makes the criterion informational (non-gating), " - + f"so it cannot also set {name} (which arms it for a pass/fail gate). " - + f"Give it a non-zero weight, or drop {name}." - ) + if self.is_stop_armed: + raise ValueError( + f"criterion {self.type!r}: weight=0 makes the criterion informational (non-gating), " + + "so it cannot also set a stop_early block, which arms it for a pass/fail gate. " + + "Give it a non-zero weight, or drop the block." + ) + if self.suite_thresholds is not None: + raise ValueError( + f"criterion {self.type!r}: weight=0 makes the criterion informational (non-gating), " + + "so it cannot also set suite_thresholds (which arms it for a pass/fail gate). " + + "Give it a non-zero weight, or drop suite_thresholds." + ) return self @property @@ -196,6 +200,52 @@ def is_gating(self) -> bool: LivePolarity = Literal["pass", "fail"] +class StopEarlyPolicy(BaseModel): + """Per-criterion early-stop policy — presence of the block IS the arming. + + Attaching ``stop_early:`` to a live-observable criterion arms it for the + run's early-stop watcher — there is no run-level master switch; the block + alone activates the watcher (``run_limits.stop_early: false`` is the + run-level veto). Arming carries ONE implicit trigger — a definitive *effective* fail + (a native live-fail, or the ``decide_within`` timeout expiring) may end the + run under the weighted ceiling rule — plus the two knobs below. A trigger + whose polarity the instance can never decide is inert by design, so one + dataset-fanned YAML line (same block on every row) serves both positive + rows (pass/timeout live) and distractor rows (fail live). + """ + + model_config = ConfigDict(extra="forbid") + + on_pass: Literal["stop", "continue"] = Field( + default="continue", + description=( + "What a live PASS does. 'continue' (default): the verdict latches (the criterion is " + "not re-checked) and the run proceeds untouched — use with decide_within for a " + "fail-fast timeout that does not cut successful runs short. 'stop': end the run (a " + "pass-stop) the moment this criterion live-passes — subject to the weighted floor " + "rule: the stop fires only once the on_pass=stop subset's worst-case weighted score " + "already meets run_limits.stop_early_gate_threshold. Inert on an instance that can " + "never live-pass (e.g. a distractor row)." + ), + ) + + decide_within: int | None = Field( + default=None, + ge=1, + description=( + "Timeout in tool-call steps: still 'undecided' after this many steps latches an " + "effective FAIL — fed through the same weighted ceiling fail-stop rule as a native " + "live-fail, reported as reason 'decision_budget_exceeded'. Inert on an instance " + "that can only ever live-fail (a distractor/guard, whose 'undecided' is its " + "success state). None (default) = no timeout. The step count is CUMULATIVE across " + "every retry attempt of the turn (the same EarlyStopWatcher instance, and its " + "counters, persist across retries) — including attempts that ultimately crashed or " + "timed out before this criterion's own investigation even began. Size it with that " + "headroom in mind." + ), + ) + + class LiveSuccessCriterion(BaseSuccessCriterion): """Base for criteria observable from a PARTIAL, mid-run trajectory (early-stop). @@ -205,8 +255,8 @@ class LiveSuccessCriterion(BaseSuccessCriterion): set). That makes it genuinely computable on the data model rather than the checker, unlike the checker's ``live_verdict`` (``criteria/base.py``), which reads the actual trajectory and stays checker-side logic. Moving - decidability here also gives early-stop-only config (e.g. - ``max_steps_to_decide``) a home that doesn't pollute ``BaseSuccessCriterion`` + decidability here also gives early-stop-only config (the + ``stop_early`` block) a home that doesn't pollute ``BaseSuccessCriterion`` with a field meaningless for every non-observable criterion type. Only ``SkillTriggeredCriterion`` / ``CommandExecutedCriterion`` subclass @@ -216,43 +266,35 @@ class LiveSuccessCriterion(BaseSuccessCriterion): checker-side flag to keep in sync). """ - max_steps_to_decide: int | None = Field( + stop_early: StopEarlyPolicy | None = Field( default=None, - ge=1, description=( - "Cap on tool-call steps this ARMED criterion (stop_when must be set) " - "may spend still 'undecided' before the run gives up on it. Once " - "exceeded, EarlyStopWatcher fires an early stop with reason " - "'decision_budget_exceeded' and the run is forced to FinalStatus." - "FAILURE outright — regardless of what any other armed criterion's " - "weighted score would otherwise gate to (this criterion never " - "reached a verdict at all, so there is nothing to weigh). None " - "(default) = no cap; the run relies solely on run_limits.max_turns. " - "Requires run_limits.stop_early and this criterion's own stop_when. " - "The step count is CUMULATIVE across every retry attempt of the " - "turn (the same EarlyStopWatcher instance, and its counters, " - "persist across retries) — including attempts that ultimately " - "crashed or timed out before this criterion's own investigation " - "even began. Size the budget with that headroom in mind." + "Opt-in early-stop policy block; its PRESENCE arms this criterion for the run's " + "early-stop watcher — the block alone activates the watcher, there is no run-level " + "master switch (run_limits.stop_early: false is the run-level veto). An armed " + "criterion's definitive effective FAIL — a native live-fail, or the decide_within " + "timeout expiring — may end the run under the weighted ceiling rule (deferred " + "while any pass-capable armed criterion is still undecided); set on_pass: stop to " + "also end the run on a live PASS. An empty block (stop_early: {}) is the idiomatic " + "distractor arming: fail-stop on misfire, nothing else. Triggers whose polarity " + "this instance cannot decide are inert by design (dataset fan-out support). " + "Unarmed criteria stay advisory on an early-stopped run. Only exists on " + "live-observable criteria, so arming anything else is a schema error." ), ) - @model_validator(mode="after") - def _check_max_steps_requires_armed(self) -> Self: - """Reject a decision-step cap on a criterion that isn't armed for early-stop. - - ``max_steps_to_decide`` only means anything relative to a criterion - that ``EarlyStopWatcher`` is actually tracking (``stop_when`` set); - setting it without ``stop_when`` is a dead field that silently does - nothing, so reject it at load time rather than let it rot unnoticed. + @property + def is_stop_armed(self) -> bool: + """True when the ``stop_early:`` block is present on this instance. + + Presence of the block IS the arming — the implicit fail trigger comes + with it, ``on_pass`` / ``decide_within`` refine it. A trigger whose + polarity this instance can never decide is inert by design (see + ``StopEarlyPolicy``) — that is what lets one dataset-fanned YAML + line serve both positive and distractor rows without per-row + conditionals. """ - if self.max_steps_to_decide is not None and self.stop_when is None: - raise ValueError( - f"criterion {self.type!r}: max_steps_to_decide requires stop_when to be set " - + "(the decision-step budget is meaningless for a criterion that isn't armed " - + "for early-stop)." - ) - return self + return self.stop_early is not None @abstractmethod def live_decidable_polarities(self) -> frozenset[LivePolarity]: @@ -260,10 +302,14 @@ def live_decidable_polarities(self) -> frozenset[LivePolarity]: Must return a subset of the polarities the corresponding checker's ``live_verdict`` can ever emit for this criterion type. Used by - ``validate_early_stop`` to reject arming a polarity this instance can - never reach, and by ``EarlyStopWatcher`` to resolve which polarities a - ``stop_when`` value actually arms for this instance (see - ``orchestration.early_stop._requested_polarities``). + ``EarlyStopWatcher`` to decide which triggers of an armed criterion's + ``stop_early`` block are live for this instance: ``on_pass: stop`` + needs ``"pass"``, the implicit fail trigger needs ``"fail"``, and + ``decide_within`` needs ``"pass"`` (a fail-only instance's 'undecided' + is its success state, so the timeout is inert there). A trigger whose + polarity is missing from this set is inert by design, not an error — + that is what lets one dataset-fanned YAML line serve both positive and + distractor rows. """ @@ -641,9 +687,10 @@ def live_decidable_polarities(self) -> frozenset[LivePolarity]: the moment the count exceeds it (this includes the ``min_count: 0, max_count: 0`` "must-NOT-run" form). - So these instance shapes are dead arms the class-level check misses: - ``stop_when: pass`` with ``max_count`` set (pass can never fire); - ``stop_when: fail`` with ``max_count: None`` (fail can never fire); + So these instance shapes leave a trigger inert (by design, so a + dataset-fanned line works across row roles): ``on_pass: stop`` / + ``decide_within`` with ``max_count`` set (pass can never fire); the + implicit fail trigger with ``max_count: None`` (fail can never fire); ``min_count: 0, max_count: None`` (neither can ever fire). """ decidable: set[LivePolarity] = set() @@ -765,10 +812,12 @@ def live_decidable_polarities(self) -> frozenset[LivePolarity]: live-``fail`` (a wrong skill engaging is a decidable miss; its absence is not). - ``validate_early_stop`` gates the requested ``stop_when`` on this set, - so arming a positive with ``fail`` / a distractor with ``pass`` — or - either with ``decided`` (which needs both) — is rejected at resolution - rather than silently degrading to a full run. + ``EarlyStopWatcher`` consults this set to decide which triggers are + live per instance: on a positive row ``on_pass: stop`` and + ``decide_within`` are live while the implicit fail trigger is inert; + on a distractor row the reverse. That per-row adaptivity is what lets + one dataset-fanned YAML line (same block on every row) serve both + roles. """ expected_yes = self.expected_skill == self.skill_name return frozenset({"pass"}) if expected_yes else frozenset({"fail"}) diff --git a/src/coder_eval/models/limits.py b/src/coder_eval/models/limits.py index 7f9184a3..e6febfea 100644 --- a/src/coder_eval/models/limits.py +++ b/src/coder_eval/models/limits.py @@ -2,9 +2,19 @@ from __future__ import annotations +from typing import Final + from pydantic import BaseModel, ConfigDict, Field +DEFAULT_STOP_EARLY_GATE_THRESHOLD: Final[float] = 1.0 +"""Default ``stop_early_gate_threshold``: reproduces strict-AND gating exactly. + +Single-sourced here so the field default below, the watcher's ``for_task`` +fallback, and the orchestrator's finalize fallback can never drift apart. +""" + + class RunLimits(BaseModel): """Run-time caps that abort a task when exceeded. @@ -87,51 +97,58 @@ class RunLimits(BaseModel): "existing behavior (Claude reports real cache-creation writes here)." ), ) - stop_early: bool = Field( - default=False, + stop_early: bool | None = Field( + default=None, description=( - "Opt-in master switch for early-stop-on-criterion. When True, the run ends early " - "once the armed criteria (those with stop_when set, incl. per-instance 'auto') " - "are decided mid-run: pass-stop when the armed subset's weighted score is " - "GUARANTEED to reach stop_early_gate_threshold regardless of any criterion still " - "undecided, fail-stop when it is GUARANTEED it never can (deferred while any " - "pass-armed criterion is undecided, so a misfire never truncates the recall " - "signal) - so a raised max_turns is not wasted once the outcome is locked in. " - "Default False keeps behavior identical. Requires a Claude single-shot task with " - "at least one observable armed criterion; every unsupported combination is " - "rejected at resolution time." + "Run-level early-stop KILL SWITCH — there is no run-level master arm. Arming is " + "per-criterion: a live-observable criterion's stop_early: block alone activates " + "the run's early-stop watcher. None (default): armed criteria decide; the run may " + "end early once they resolve mid-run (pass-stop when the on_pass=stop subset's " + "weighted score is GUARANTEED to reach stop_early_gate_threshold regardless of " + "any criterion still undecided, fail-stop when the armed set's weighted score is " + "GUARANTEED to never reach it — deferred while any pass-capable armed criterion " + "is undecided, so a misfire never truncates the recall signal; a decide_within " + "timeout latches an effective fail that feeds the same fail-stop rule). False: " + "force-disarm every criterion's block for this run — the one-line experiment/" + "variant override that turns a smoke flavor back into an authoritative, " + "non-truncated run. True is rejected at resolution time (the master arm was " + "removed; put a stop_early: block on the criterion instead). An armed run " + "requires a single-shot task on a cooperative-stop agent; every unsupported " + "combination is rejected at resolution time." ), ) stop_early_gate_threshold: float = Field( - default=1.0, + default=DEFAULT_STOP_EARLY_GATE_THRESHOLD, ge=0.0, le=1.0, description=( "Minimum weighted score (Σ weight_i·score_i / Σ weight_i, over the ARMED subset " - "only) required for an early-stopped run to gate as a pass. Also the bound " + "only) required for an EARLY-STOPPED run to gate as a pass (a run that completes " + "naturally gates strict-AND over the full criteria set, armed or not). Also the bound " "early-stop's trigger checks against: a fail-stop fires once no combination of " "still-undecided armed criteria could raise the weighted score to this " - "threshold; a pass-stop fires once it is already guaranteed to meet it regardless " - "of what's still undecided. Default 1.0 reproduces the pre-weighting behavior " - "exactly (every armed criterion's live-observable score is binary 0/1, so a " - "weighted score of 1.0 requires every armed criterion to have actually passed) - " - "lowering it lets a low-weight armed criterion's failure be absorbed without " - "truncating the run, at the cost of the gate/trigger becoming a genuine weighted " - "average rather than a strict AND." + "threshold (a decide_within timeout counts as a fail here); a pass-stop " + "fires once the on_pass=stop subset is already guaranteed to meet it regardless " + "of what's still undecided. Default 1.0 reproduces strict-AND behavior exactly " + "(a weighted score of 1.0 requires every armed criterion to have actually " + "passed) - lowering it lets a low-weight armed criterion's failure (or timeout) " + "be absorbed without truncating the run, at the cost of the gate/trigger " + "becoming a genuine weighted average rather than a strict AND." ), ) - # NOTE: stop_early_gate_threshold <= 0.0 together with stop_early: True is - # a degenerate, gate-neutralizing config (a threshold of 0 trivially - # passes the armed gate regardless of whether anything decided) and is - # rejected — but NOT here. RunLimits is field-merged across 5 layers, so a - # model-level validator has no visibility into which layer produced the - # merged value and cannot distinguish a real mistake from a value merged - # forward from a sibling layer (e.g. a task-level threshold inherited by a - # variant that only flips stop_early). That distinction requires seeing - # the whole resolved task, so the check lives in - # orchestration/early_stop.py::validate_early_stop instead, where it - # raises EarlyStopConfigError and gets the same hard-stop CLI treatment + # NOTE: stop_early_gate_threshold <= 0.0 on an ARMED task is a degenerate, + # gate-neutralizing config (a threshold of 0 trivially passes the armed + # gate regardless of whether anything decided) and is rejected — but NOT + # here, and neither is stop_early: True (the removed master arm). Whether a + # task is armed lives on the criteria, which RunLimits cannot see, and + # RunLimits is field-merged across 5 layers, so a model-level validator has + # no visibility into which layer produced the merged value and cannot + # distinguish a real mistake from a value merged forward from a sibling + # layer (e.g. a task-level threshold inherited by a variant that only + # toggles the kill switch). Both checks live in + # orchestration/early_stop.py::validate_early_stop instead, where they + # raise EarlyStopConfigError and get the same hard-stop CLI treatment # (flips the plan exit code, aborts run) as every other early-stop # guardrail — a plain pydantic ValueError here would instead land in # plan_command's generic per-variant "resolution failed" branch, which diff --git a/src/coder_eval/models/results.py b/src/coder_eval/models/results.py index 7591ba3c..e0f4f80b 100644 --- a/src/coder_eval/models/results.py +++ b/src/coder_eval/models/results.py @@ -21,6 +21,7 @@ from coder_eval.models.agent_config import ResolvedAgentConfig from coder_eval.models.criteria import SuccessCriterion from coder_eval.models.enums import FinalStatus +from coder_eval.models.limits import DEFAULT_STOP_EARLY_GATE_THRESHOLD from coder_eval.models.telemetry import ( CommandStatistics, CommandTelemetry, @@ -417,12 +418,12 @@ class EarlyStopReason(StrEnum): ``EvaluationResult`` carries it and ``models/`` is a leaf package that cannot import from ``simulation``. ``DialogStopReason`` is a stylistic reference only. Early-stop is orthogonal telemetry, NOT a ``FinalStatus`` — - the terminal-status set stays closed — with ONE deliberate exception: - ``DECISION_BUDGET_EXCEEDED`` forces ``FinalStatus.FAILURE`` directly at the - orchestrator finalize step (``orchestrator.py``), bypassing - ``armed_criteria_passed``'s weighted gate entirely — the criterion whose - budget expired never reached a verdict at all, so there is nothing - meaningful to weigh it against. + the terminal-status set stays closed, and every reason gates identically + through ``armed_criteria_passed``'s weighted gate. + ``DECISION_BUDGET_EXCEEDED`` is a reporting label only: it marks a + fail-stop whose deciding criterion timed out undecided past its + ``stop_early.decide_within`` (an *effective* fail latched by the watcher) + rather than live-failing natively. """ CRITERION_PASSED = "criterion_passed" @@ -441,8 +442,10 @@ class EarlyStopInfo(BaseModel): """ reason: EarlyStopReason = Field( - description="Why the run stopped: armed criteria passed, definitively failed, or an armed " - + "criterion's decision-step budget (max_steps_to_decide) expired unresolved." + description="Why the run stopped: the pass-stop floor locked in (criterion_passed), an armed " + + "criterion live-failed natively (criterion_failed), or an armed criterion's decision-step " + + "budget (stop_early.decide_within) expired while still undecided — an effective fail " + + "(decision_budget_exceeded). All reasons gate identically through the weighted armed gate." ) deciding_criterion_type: str = Field( description="Type of the criterion whose live verdict fired the stop (the failing one on " @@ -473,7 +476,7 @@ class EarlyStopInfo(BaseModel): + "saving); None when max_turns is unset.", ) gate_threshold: float = Field( - default=1.0, + default=DEFAULT_STOP_EARLY_GATE_THRESHOLD, ge=0.0, le=1.0, description="run_limits.stop_early_gate_threshold in effect for this stop — captured so a " @@ -673,7 +676,9 @@ def all_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: if c.is_gating ) - def armed_criteria_passed(self, criteria: list[SuccessCriterion], gate_threshold: float = 1.0) -> bool: + def armed_criteria_passed( + self, criteria: list[SuccessCriterion], gate_threshold: float = DEFAULT_STOP_EARLY_GATE_THRESHOLD + ) -> bool: """True iff the ARMED subset's weighted score meets ``gate_threshold``. The early-stop gate: on an early-stopped run only the armed subset gates @@ -684,8 +689,8 @@ def armed_criteria_passed(self, criteria: list[SuccessCriterion], gate_threshold single-sourced. Raises ``ValueError`` on an empty armed set — unreachable when a stop actually fired (a stop requires an armed criterion), so this is a defensive guard against misuse. No ``is_gating`` filter is needed - here: ``BaseSuccessCriterion`` rejects ``weight: 0`` together with - ``stop_when``, so every armed criterion is gating by construction. + here: ``BaseSuccessCriterion`` rejects ``weight: 0`` together with any + early-stop trigger, so every armed criterion is gating by construction. Each armed criterion's OWN ``pass_threshold`` still decides whether it individually passed — ``r.score`` is converted to a binary 1.0/0.0 via @@ -708,17 +713,15 @@ def armed_criteria_passed(self, criteria: list[SuccessCriterion], gate_threshold f"Results/criteria length mismatch for task {self.task_id}: " + f"{len(self.success_criteria_results)} results vs {len(criteria)} criteria." ) - armed = [ - (r, c) for r, c in zip(self.success_criteria_results, criteria, strict=True) if c.stop_when is not None - ] + armed = [(r, c) for r, c in zip(self.success_criteria_results, criteria, strict=True) if c.is_stop_armed] if not armed: raise ValueError( f"armed_criteria_passed called with no armed criteria for task {self.task_id}; " - + "the early-stop gate is only valid when at least one criterion sets stop_when." + + "the early-stop gate is only valid when at least one criterion sets a stop trigger." ) total_weight = sum(c.weight for _, c in armed) if total_weight <= 0.0: - # Unreachable today (weight=0 + stop_when is rejected at the model + # Unreachable today (weight=0 + a stop trigger is rejected at the model # layer, so every armed criterion carries weight > 0) — but a # defensive guard on a pass/fail gate must fail CLOSED, not open, # against a future criterion subclass that bypasses that diff --git a/src/coder_eval/orchestration/early_stop.py b/src/coder_eval/orchestration/early_stop.py index 379eac4d..68fa2f21 100644 --- a/src/coder_eval/orchestration/early_stop.py +++ b/src/coder_eval/orchestration/early_stop.py @@ -1,27 +1,57 @@ """Early-stop-on-criterion: resolution-time validation + runtime watcher. -Opt-in mechanism (``run_limits.stop_early``) that ends a single-shot run as soon -as its *armed* criteria (those with ``stop_when`` set) are decided mid-run — on -pass or on a definitive fail. This module owns the whole feature: +Opt-in mechanism that ends a single-shot run early once its *armed* criteria +decide the outcome mid-run. Arming lives ENTIRELY on the criterion — there is +no run-level master switch. A ``stop_early:`` block on a live-observable +criterion (``LiveSuccessCriterion`` only, so arming an unobservable criterion +is unrepresentable) alone activates the run's watcher; +``run_limits.stop_early: false`` is the run-level KILL SWITCH that force- +disarms every block (the one-line experiment-variant override for an +authoritative, non-truncated run), and ``run_limits.stop_early: true`` — the +removed master arm — is rejected at resolution: + +* ``stop_early: {}`` — the block's PRESENCE is the arming; it carries one + implicit trigger: a native live FAIL may end the run (fail-stop). +* ``stop_early: {on_pass: stop}`` — a live PASS may also end the run + (pass-stop); the default ``on_pass: continue`` just latches the verdict. +* ``stop_early: {decide_within: N}`` — still undecided after N tool-call steps + latches an *effective* FAIL, fed through the same fail-stop rule (reported + as ``decision_budget_exceeded``). + +A trigger whose polarity the instance can never decide (see +``live_decidable_polarities``) is INERT BY DESIGN, not an error — one +dataset-fanned YAML line (same block on every row) serves both positive rows +(pass/timeout live, fail inert) and distractor rows (fail live, pass/timeout +inert) without per-row conditionals. + +This module owns the whole feature: * ``validate_early_stop`` — resolution-time guardrails. Rejects every configuration v1 cannot honor as a hard error, so an unsupported arming is never a silent no-op. * ``EarlyStopWatcher`` — the runtime observer. A ``StreamCallback`` composed into the agent's event stream that maintains its own ``EventCollector``, - evaluates every armed criterion's ``live_verdict`` on each tool *call* (and on + evaluates the armed criteria's ``live_verdict`` on each tool *call* (and on its result), applies the stop rule, and exposes ``should_stop()`` (the - cooperative interrupt the agent polls) plus ``info`` (the ``EarlyStopInfo`` the - orchestrator records). Fail-open: a raising ``live_verdict`` disarms the + cooperative interrupt the agent polls) plus ``info`` (the ``EarlyStopInfo`` + the orchestrator records). Fail-open: a raising ``live_verdict`` disarms the watcher and degrades to a full run — a verdict bug can never cause a *false* early stop. +Verdicts LATCH: once an armed criterion decides (pass or fail) on a resolved +round, its ``live_verdict`` is never polled again for the rest of the run — +the checkers' documented monotonicity makes re-polling pure waste. Latching +happens only on resolved rounds (``ToolEndEvent``); an in-flight round's fresh +verdict can fire a stop but is not persisted, so a dispatched call that never +resolves (e.g. a crashed attempt) cannot leave a stale verdict behind across +retries. + Deciding on the tool *call* (``ToolStartEvent``), not the result, is what makes the stop robust: for an observable criterion the verdict is fully determined by the call's inputs (which skill / which command), so the watcher can latch the instant the call is dispatched — before a cut-short turn (e.g. a timeout) can strip the result and leave the call unresolved. The agent polls ``should_stop`` -immediately after dispatching each message, so a latch on the call breaks the +immediately after dispatching each message, so a stop on the call breaks the loop before the result message is ever pulled. Live verdicts only *trigger* the stop; the authoritative scores always come @@ -31,43 +61,58 @@ (``EvaluationResult.armed_criteria_passed``) consult ``run_limits. stop_early_gate_threshold`` (default ``1.0``) rather than treating every armed criterion's pass/fail as equally decisive. A fail-stop fires once the armed -subset's CEILING (best case: every still-``undecided``/``pass`` criterion -scores 1.0, every live-``fail``ed one scores 0) can no longer reach the -threshold — i.e. the gate is mathematically guaranteed to fail regardless of -how the trajectory continues. A pass-stop fires once the pass-armed subset's -FLOOR (worst case: every still-undecided one scores 0) already meets it. At the +set's CEILING (best case: every still-``undecided``/``pass`` criterion scores +1.0, every effectively-failed one scores 0) can no longer reach the threshold — +i.e. the gate is mathematically guaranteed to fail regardless of how the +trajectory continues. A ``decide_within`` timeout participates as an +ordinary weighted fail: a low-weight criterion's timeout that cannot doom the +gate does not stop the run (it is absorbed, exactly like a low-weight native +fail). A pass-stop fires once the ``on_pass: stop`` subset's FLOOR (worst +case: every still-undecided one scores 0) already meets the threshold. At the default threshold of 1.0 both bounds collapse exactly to "any single armed -criterion's live-fail stops the run" / "every pass-armed criterion has -live-passed" — byte-for-byte the pre-weighting behavior, since the only -live-observable criteria score binary 0/1. Lowering the threshold lets a -low-weight armed criterion's failure be absorbed without truncating the run. - -Precision trade-off: a pass-stop cuts the run the instant the pass-armed floor -locks in, so a *fail-armed* criterion (e.g. a distractor) that would only -misfire on a LATER tool call is never observed — the frozen trajectory then -scores that row as a clean pass. This is an intentional precision-for-budget -trade of the opt-in "smoke" flavor; the authoritative +criterion's effective fail stops the run" / "every on_pass=stop criterion has +live-passed". Lowering the threshold lets a low-weight armed criterion's +failure be absorbed without truncating the run. + +Precision trade-off: a pass-stop cuts the run the instant the on_pass=stop +floor locks in, so a fail-armed criterion (e.g. a distractor) that would +only misfire on a LATER tool call is never observed — the frozen trajectory +then scores that row as a clean pass. This is an intentional +precision-for-budget trade of the opt-in "smoke" flavor; the authoritative precision/recall must come from a non-early-stop (``stop_early: false``) run. -Recall, by contrast, is never truncated: the fail-stop is DEFERRED while any -*pass-armed* criterion is still undecided, so a distractor misfire on an early -tool call cannot cut a positive row before its expected signal has had the -chance to appear (which would freeze a would-be TP as an FN and deflate -recall/F1). The misfire is not lost — the observable criteria latch -monotonically, so the deferred fail fires the moment every pass-armed criterion -decides (fail-stop is evaluated before pass-stop each round), and if none ever -decides the run simply continues to the cap. A row with zero pass-armed -criteria (e.g. a negative row stacking only distractors) has nothing to defer -for and fail-stops on the first misfire. +Recall, by contrast, is never truncated — BOTH stops defer on it. The +fail-stop is DEFERRED while any pass-capable armed criterion is still +undecided (and within its budget), so a distractor misfire on an early tool +call cannot cut a positive row before its expected signal has had the chance +to appear (which would freeze a would-be TP as an FN and deflate recall/F1). +Symmetrically, the pass-stop is DEFERRED while any pass-capable armed +criterion OUTSIDE the on_pass=stop subset is still undecided (members of the +subset are already accounted for by the floor bound itself) — otherwise an +on_pass=stop criterion passing early would freeze a sibling ``on_pass: +continue`` criterion (e.g. one armed via ``decide_within``) as an unearned +fail on the truncated trajectory. Neither deferral loses the trigger — +verdicts latch monotonically, so the held stop fires the moment every +pass-capable armed criterion decides (fail-stop is evaluated before pass-stop +each round), and if none ever decides the run simply continues to the cap. A +row with zero pass-capable armed criteria (e.g. a negative row stacking only +distractors) has nothing to defer for and fail-stops on the first misfire. """ from __future__ import annotations import logging import time -from typing import TYPE_CHECKING, Any, Literal, assert_never - -from coder_eval.models import EarlyStopInfo, EarlyStopReason, LivePolarity, LiveSuccessCriterion +from typing import TYPE_CHECKING, Any + +from coder_eval.models import ( + DEFAULT_STOP_EARLY_GATE_THRESHOLD, + EarlyStopInfo, + EarlyStopReason, + LivePolarity, + LiveSuccessCriterion, + StopEarlyPolicy, +) from coder_eval.streaming.collector import EventCollector from coder_eval.streaming.events import ( AgentStartEvent, @@ -93,34 +138,8 @@ logger = logging.getLogger(__name__) -def _requested_polarities( - stop_when: Literal["pass", "fail", "decided", "auto"], decidable: frozenset[LivePolarity] -) -> frozenset[LivePolarity]: - """The polarities a ``stop_when`` value requests to arm, given what the instance can decide. - - The single source of truth for the ``stop_when`` -> polarity mapping — both - the resolution-time validator and the runtime watcher resolve through here, - so a value can never mean different things in the two places. ``pass``/ - ``fail`` request that single polarity; ``decided`` requests both; ``auto`` - requests exactly the instance's own decidable set (which is why it can - return the empty set: an instance that can decide neither polarity is a - dead arm, and the validator rejects it). ``assert_never`` makes widening the - ``stop_when`` Literal without updating this mapping a type error instead of - a silently inert arm. - """ - if stop_when == "auto": - return decidable - if stop_when == "decided": - return frozenset({"pass", "fail"}) - if stop_when == "pass" or stop_when == "fail": - return frozenset({stop_when}) - # `return` is redundant for control flow (assert_never never returns) but - # keeps every path explicit for analyzers that don't model `Never`. - return assert_never(stop_when) - - class EarlyStopConfigError(ValueError): - """Raised when a task arms ``run_limits.stop_early`` in a way v1 cannot honor. + """Raised when a task arms early-stop in a way v1 cannot honor. Subclasses ``ValueError`` so the run path's resolve -> ``typer.BadParameter`` conversion covers it transparently; the ``plan`` command catches this @@ -129,39 +148,75 @@ class EarlyStopConfigError(ValueError): """ +def early_stop_active(task: TaskDefinition) -> bool: + """True iff this run should build a watcher: >= 1 armed criterion, kill switch not thrown. + + The single arming predicate the orchestrator consults. Deliberately ignores + ``run_limits.stop_early is True`` (the removed master arm) — that value is + rejected by ``validate_early_stop``, which every path runs before watcher + creation, so it can never reach a live run. + """ + limits = task.run_limits + if limits is not None and limits.stop_early is False: + return False + return any(c.is_stop_armed for c in task.success_criteria) + + def validate_early_stop(task: TaskDefinition) -> None: """Validate an armed early-stop task at resolution time; no-op when unarmed. Called after the config layers have merged (``resolve_all_tasks`` post-CLI overrides, the ``plan`` per-variant loop, and defensively in - ``Orchestrator._setup``). All checks below are skipped unless - ``run_limits.stop_early`` is True, so default runs are entirely unaffected. + ``Orchestrator._setup``). ``run_limits.stop_early: true`` (the removed + master arm) is always rejected; everything else is skipped unless the task + is actually armed (``early_stop_active``), so default runs — and runs + force-disarmed via the ``stop_early: false`` kill switch — are entirely + unaffected. Raise order (matters for which error a multiply-invalid task reports first): - 5. armed together with ``simulation.enabled`` -> error - 1. agent does not declare ``supports_cooperative_stop`` -> error - 2. armed but no criterion sets ``stop_when`` -> error - then per armed criterion: - 3. criterion is not observable mid-run -> error - 4. requested ``stop_when`` polarity the criterion cannot decide -> error - (``auto`` errors only on a dead arm: an instance that can decide neither) + 1. ``run_limits.stop_early: true`` -> error (master arm removed) + 2. armed together with ``simulation.enabled`` -> error + 3. agent does not declare ``supports_cooperative_stop`` -> error + 4. degenerate ``stop_early_gate_threshold`` (``<= 0.0``) -> error + + There are deliberately NO per-instance polarity guards: a trigger whose + polarity this instance cannot decide is inert by design (documented on each + trigger field), which is what lets one dataset-fanned YAML line serve both + positive and distractor rows. Arming an unobservable criterion type is + structurally impossible — the ``stop_early`` block exists only on + ``LiveSuccessCriterion``, so a ``file_exists`` criterion carrying one is a + pydantic ``extra='forbid'`` error at load time, not a case this validator + needs to catch. And an armed-but-empty set needs no guard either: with no + blocks present there is simply no watcher, byte-for-byte default behavior. Raises: EarlyStopConfigError: on any unsupported armed configuration. """ limits = task.run_limits - if limits is None or not limits.stop_early: + # (1) The master arm no longer exists; arming moved onto the criteria. A + # hard error (not a deprecation no-op) because a task author writing + # `stop_early: true` expects arming to happen — silently ignoring it would + # run the full task and gate it differently than they intended. + if limits is not None and limits.stop_early is True: + raise EarlyStopConfigError( + "run_limits.stop_early: true has been removed — arming is per-criterion now. " + + "Put a stop_early: block on the live-observable criterion instead " + + "(e.g. stop_early: {} / {on_pass: stop} / {decide_within: N}); " + + "run_limits.stop_early: false remains available as the run-level kill switch." + ) + + if not early_stop_active(task): return - # (5) Simulation/dialog mode has its own criteria-driven stop. + # (2) Simulation/dialog mode has its own criteria-driven stop. if task.simulation is not None and task.simulation.enabled: raise EarlyStopConfigError( - "run_limits.stop_early is not supported together with simulation.enabled " + "criterion-level stop_early arming is not supported together with simulation.enabled " + "(early-stop v1 is single-shot only); use simulation.stop_on_criteria_pass " - + "for dialog-mode criteria stopping." + + "for dialog-mode criteria stopping, or disarm with run_limits.stop_early: false." ) - # (1) The agent must honor the cooperative interrupt. Lazily import the + # (3) The agent must honor the cooperative interrupt. Lazily import the # registry + plugin loader so this module stays free of runtime coder_eval # imports at load time. from coder_eval.agents.registry import AgentRegistry @@ -169,23 +224,36 @@ def validate_early_stop(task: TaskDefinition) -> None: ensure_plugins_loaded() agent_type = str(task.agent.type) if task.agent is not None and task.agent.type is not None else None - registration = AgentRegistry.get(agent_type) if agent_type is not None else None - supports = bool(getattr(registration.agent_class, "supports_cooperative_stop", False)) if registration else False - if not supports: + if agent_type is None: + # Distinct from an unregistered type: there is no agent block at all, + # so pointing at plugin loading would send the user the wrong way. raise EarlyStopConfigError( - "run_limits.stop_early requires an agent that supports cooperative stopping " - + f"(claude-code, codex, antigravity); agent type {agent_type!r} does not." + "criterion-level stop_early arming requires an agent block with a registered type; " + + "this task resolves without one. " + + "Disarm with run_limits.stop_early: false to bypass this check." + ) + registration = AgentRegistry.get(agent_type) + if registration is None: + # Not the same failure as an agent that opted out of cooperative stop: + # an unregistered type usually means a plugin is not installed/loaded. + raise EarlyStopConfigError( + f"criterion-level stop_early arming requires a registered agent type; {agent_type!r} is " + + "not registered (is the providing plugin installed and loaded?). " + + "Disarm with run_limits.stop_early: false to bypass this check." + ) + if not registration.agent_class.supports_cooperative_stop: + supporting = ", ".join( + kind + for kind in AgentRegistry.list_kinds() + if (reg := AgentRegistry.get(kind)) is not None and reg.agent_class.supports_cooperative_stop ) - - # (2) Arming requires at least one stop criterion. - armed = [c for c in task.success_criteria if c.stop_when is not None] - if not armed: raise EarlyStopConfigError( - "run_limits.stop_early is armed but no success criterion sets stop_when; " - + "arming requires at least one stop criterion (e.g. stop_when: auto)." + "criterion-level stop_early arming requires an agent that supports cooperative stopping " + + f"({supporting}); agent type {agent_type!r} does not. " + + "Disarm with run_limits.stop_early: false to run this agent anyway." ) - # (0) A threshold of exactly 0 trivially satisfies both the pass-stop + # (4) A threshold of exactly 0 trivially satisfies both the pass-stop # floor check and the final weighted gate regardless of whether any armed # criterion has actually decided — neutralizing the armed pass/fail gate # with one YAML line (coder-eval is used as a CI gate). Checked here @@ -194,76 +262,13 @@ def validate_early_stop(task: TaskDefinition) -> None: # aborts run, whereas a plain ValueError on the merged RunLimits model # would land in the CLI's generic "resolution failed" branch, which # prints red text but does not flip the exit code. - if limits.stop_early_gate_threshold <= 0.0: + if limits is not None and limits.stop_early_gate_threshold <= 0.0: raise EarlyStopConfigError( f"run_limits.stop_early_gate_threshold ({limits.stop_early_gate_threshold}) must be " - + "> 0.0 when stop_early is True (a threshold of 0 trivially passes the armed gate " + + "> 0.0 on an armed task (a threshold of 0 trivially passes the armed gate " + "regardless of whether any armed criterion actually decided)." ) - # (3)+(4) Per armed criterion: observable, then the requested polarity is - # decidable. - for c in armed: - # `armed` filtered on `stop_when is not None`; re-bind + assert so pyright - # narrows the Literal away from None for the set arithmetic below. - polarity = c.stop_when - assert polarity is not None - # (3) Type-level observability: a criterion type is live-observable iff - # its model is a ``LiveSuccessCriterion`` subclass (models/criteria.py) - # — the single source of truth, replacing a separate checker-side flag. - if not isinstance(c, LiveSuccessCriterion): - raise EarlyStopConfigError( - f"criterion type {c.type!r} is armed (stop_when={polarity!r}) but is not " - + "observable mid-run; early-stop supports only live-observable criteria " - + "(e.g. skill_triggered, command_executed)." - ) - # (4) Instance-level decidability: some criteria (e.g. command_executed) - # can decide fewer polarities than their type advertises depending on this - # instance's config, so gate on the per-instance set — otherwise a dead - # arm (a polarity this instance can never fire) would silently degrade to - # a full run instead of erroring here. - polarities = c.live_decidable_polarities() - requested = _requested_polarities(polarity, polarities) - # Dead arm: only `auto` can request the empty set (it requests exactly - # the instance's decidable polarities) — an instance that can decide - # neither has nothing to arm and would silently never fire. - if not requested: - raise EarlyStopConfigError( - f"criterion {c.type!r} ({c.description!r}) is armed (stop_when='auto') but this " - + "instance can decide no polarity mid-run; 'auto' requires at least one " - + "live-decidable polarity (its decidability can depend on the criterion's " - + "fields — e.g. command_executed can live-pass only with max_count unset + " - + "min_count>0, and live-fail only with max_count set)." - ) - missing = sorted(requested - polarities) - if missing: - supported = sorted(polarities) or "no polarities" - raise EarlyStopConfigError( - f"criterion {c.type!r} ({c.description!r}) cannot decide polarity {missing} mid-run " - + f"(stop_when={polarity!r}) for this configuration; it supports {supported}. " - + "Decidability can depend on the criterion's fields (e.g. command_executed " - + "can live-pass only with max_count unset + min_count>0, and live-fail only " - + "with max_count set)." - ) - # (6) A decision-step budget is meaningless for a fail-only-decidable - # instance (a pure distractor/guard, e.g. a "must-NOT-run" command or a - # negative-row skill_triggered): its "undecided" IS the success - # state — the forbidden event simply hasn't happened yet, and staying - # undecided forever is correct, not a stall. The budget only makes - # sense for an instance that can decide "pass": something it is - # actively waiting to observe. Rejecting this at resolution (rather - # than letting EarlyStopWatcher force-fail a clean run) matters most - # for a dataset-fanned `auto` criterion, where the same YAML line - # would force-fail every negative row. - if c.max_steps_to_decide is not None and "pass" not in polarities: - raise EarlyStopConfigError( - f"criterion {c.type!r} ({c.description!r}) sets max_steps_to_decide but this " - + f"instance can only ever live-decide {sorted(polarities) or 'no polarities'} — " - + "a fail-only-decidable instance's 'undecided' is its success state (the " - + "forbidden event hasn't happened), so a decision-step budget would force-fail " - + "a clean run. max_steps_to_decide requires an instance that can live-pass." - ) - class EarlyStopWatcher: """Observes the agent event stream and trips the cooperative interrupt. @@ -272,10 +277,10 @@ class EarlyStopWatcher: ``--stream`` is off, else beside the ``TaskScopedCallback``). It maintains its OWN ``EventCollector`` — independent of the one the agent builds its returned ``TurnRecord`` from — so each ``live_verdict`` sees a fresh, - single-element partial-trajectory list. On every tool completion it computes - all armed verdicts and applies the stop rule; once a stop fires (or the - watcher disarms on a raising verdict) the decision is latched and further - events are ignored. + single-element partial-trajectory list. On every tool call it evaluates the + armed criteria still undecided and applies the stop rule; once a stop fires + (or the watcher disarms on a raising verdict) the decision is latched and + further events are ignored. The orchestrator polls ``should_stop`` (passed to ``agent.communicate``) and, after the turn, reads ``info`` to populate ``EvaluationResult.early_stop``. @@ -292,29 +297,62 @@ def __init__( armed: list[_ArmedPair], *, max_turns: int | None, - gate_threshold: float = 1.0, + gate_threshold: float = DEFAULT_STOP_EARLY_GATE_THRESHOLD, ) -> None: self._task_id = task_id self._armed = armed self._gate_threshold = gate_threshold self._armed_weight = sum(c.weight for c, _ in armed) - # Per-instance resolved arming polarities, aligned with ``_armed``. Static - # for the run, resolved through ``_requested_polarities`` (the single - # stop_when -> polarity mapping). The stop rule consults this, not the raw - # ``stop_when`` string, so a distractor armed ``auto`` (fail only) is not - # required to live-pass for a pass-stop. - self._armed_polarities: list[frozenset[LivePolarity]] = [ - self._resolve_armed_polarities(criterion) for criterion, _checker in armed + # Per-instance decidable polarities, aligned with ``_armed``. Static for + # the run. Each trigger below is resolved against this set — a trigger + # whose polarity the instance cannot decide is inert by design. + self._decidable: list[frozenset[LivePolarity]] = [ + criterion.live_decidable_polarities() for criterion, _checker in armed + ] + # Effective per-instance triggers (inert ones already resolved away). + # Every armed pair carries a stop_early block by construction + # (is_stop_armed == block presence); an explicit raise (not an assert, + # which -O strips) keeps a blockless pair from slipping through. + blocks: list[StopEarlyPolicy] = [] + for criterion, _checker in armed: + if criterion.stop_early is None: + raise ValueError(f"criterion {criterion.type!r} passed to EarlyStopWatcher without a stop_early block") + blocks.append(criterion.stop_early) + self._pass_trigger: list[bool] = [ + block.on_pass == "stop" and "pass" in pol for block, pol in zip(blocks, self._decidable, strict=True) ] + # The fail trigger is IMPLICIT in arming: an armed criterion's native + # live-fail may always stop the run (ceiling-gated) — that is what the + # block's presence means. Inert when the instance can't live-fail. + self._fail_trigger: list[bool] = ["fail" in pol for pol in self._decidable] + # A timeout only means anything for an instance actively waiting to + # observe a pass — a fail-only instance's 'undecided' is its success + # state (the forbidden event hasn't happened), so its budget is inert. + self._budget: list[int | None] = [ + block.decide_within if "pass" in pol else None for block, pol in zip(blocks, self._decidable, strict=True) + ] + if not any(self._pass_trigger) and not any(self._fail_trigger) and all(b is None for b in self._budget): + # Legal (a fanned row whose armed lines are all inert for this row's + # role) but user-visible: on a non-fanned task this is dead config — + # the row can never stop early and will simply run to the cap, + # gating on the armed subset if the watcher somehow fires. + logger.warning("[%s] all armed stop triggers are inert for this row; run cannot stop early", task_id) self._max_turns = max_turns self._collector = EventCollector() self._sdk_turn_index = 0 self._tool_call_index = 0 self._started_monotonic: float | None = None - # Previous round's verdicts, for the "which criterion flipped to pass this - # round" attribution on pass-stop. Reassigned ONLY at the end of a - # non-firing evaluation, so it always holds the PREVIOUS round when a stop - # fires. Starts all-"undecided". + # Latched verdicts, aligned with ``_armed``. Once an entry leaves + # "undecided" (on a RESOLVED round) its checker is never polled again — + # the checkers' documented monotonicity makes re-polling pure waste. + # ``_budget_expired`` marks a latched fail as timeout-driven (reported + # as DECISION_BUDGET_EXCEEDED instead of CRITERION_FAILED). + self._latched: list[LiveVerdict] = ["undecided"] * len(armed) + self._budget_expired: list[bool] = [False] * len(armed) + # Previous round's verdicts, for the "which criterion flipped to pass + # this round" attribution on pass-stop. Reassigned ONLY at the end of a + # non-firing evaluation, so it always holds the PREVIOUS round when a + # stop fires. Starts all-"undecided". self._prev_verdicts: list[LiveVerdict] = ["undecided"] * len(armed) self._info: EarlyStopInfo | None = None self._disarmed = False @@ -324,30 +362,54 @@ def for_task(cls, task: TaskDefinition) -> EarlyStopWatcher: """Build a watcher for an armed task (instantiates the armed criteria's checkers). The criteria registry is imported lazily here — it is not initialized at - module import time. Checker classes take no ctor args. + module import time. Checker classes take no ctor args. Only + ``LiveSuccessCriterion`` instances can be armed (the trigger fields + exist nowhere else), so the ``isinstance`` filter is a pyright + narrowing aid, not a behavioral guard. """ from coder_eval.criteria import CriterionRegistry, init_criteria init_criteria(validate=False) - # The `isinstance` check is defense-in-depth, not load-bearing: every - # call site (`run`, `plan`) runs `validate_early_stop` first, which - # already hard-rejects an armed non-observable criterion at resolution - # time. Narrows `c` to `LiveSuccessCriterion` for pyright either way. armed: list[_ArmedPair] = [ (c, CriterionRegistry.get_checker(c.type)()) for c in task.success_criteria - if c.stop_when is not None and isinstance(c, LiveSuccessCriterion) + if isinstance(c, LiveSuccessCriterion) and c.is_stop_armed ] max_turns = task.run_limits.max_turns if task.run_limits is not None else None - gate_threshold = task.run_limits.stop_early_gate_threshold if task.run_limits is not None else 1.0 + gate_threshold = ( + task.run_limits.stop_early_gate_threshold + if task.run_limits is not None + else DEFAULT_STOP_EARLY_GATE_THRESHOLD + ) return cls(task.task_id, armed, max_turns=max_turns, gate_threshold=gate_threshold) # --- StreamCallback -------------------------------------------------- # def on_event(self, event: StreamEvent) -> None: + """Fail-open wrapper around ``_on_event_impl``: any unexpected exception + anywhere in the round — the collector reduction included, not just the + verdict-collection loop — disarms the watcher and degrades to a full + run. The agent-side ``safe_emit`` swallows callback exceptions, so + without disarming here a raising collector would leave the watcher + silently evaluating a corrupted partial trajectory on every subsequent + event with ``_disarmed`` still False. + """ + if self._info is not None or self._disarmed: + return + try: + self._on_event_impl(event) + except Exception: + self._disarmed = True + logger.error( + "[%s] early-stop event handling raised unexpectedly; disarming watcher, run degrades to a full run", + self._task_id, + exc_info=True, + ) + + def _on_event_impl(self, event: StreamEvent) -> None: """Forward the event to the internal collector; evaluate on each tool call. - Short-circuits once the decision is latched (fired or disarmed). Counts + Counts ``TurnStartEvent`` for ``sdk_turn_index`` and each dispatched tool call for the 1-based ``tool_call_index``, and stamps the wall-clock origin at the FIRST ``AgentStartEvent`` only (a retry's second AgentStart does not @@ -358,25 +420,29 @@ def on_event(self, event: StreamEvent) -> None: so latching here lets the agent's post-dispatch ``should_stop`` poll break the loop before a cut-short turn can strip the result. The call is not in the collector yet (it reduces commands from ``ToolEndEvent``), so it is - passed to ``_evaluate`` as the in-flight command, reported at + passed to ``_evaluate_impl`` as the in-flight command, reported at ``tool_call_index + 1`` (it has no ``ToolEndEvent`` to count yet). The matching ``ToolEndEvent`` still evaluates, which covers a verdict that only becomes decidable once the result is known and is a no-op once a call has already latched the stop. ``tool_call_index`` is incremented on the resolved ``ToolEndEvent`` so it stays a count of completed tool calls. - UNRESOLVED tool ends are ignored entirely (not counted or evaluated). + UNRESOLVED tool ends are RECORDED but never counted or evaluated on. ``_ClaudeTurnState.finalize`` force-closes orphaned tools as UNRESOLVED *after* the message loop has ended and the terminal status is already chosen (COMPLETED / TIMEOUT / crash) — those are not live tool activity and must not trip the stop rule, or a run that ran to completion (or timed out / crashed) without a real, in-loop decision would latch a false early - stop. A legitimate stop always fires on the in-loop call, so dropping - unresolved ends can never suppress a real stop; it also keeps a crashed - attempt's orphan tools out of the retry-persistent partial trajectory. + stop. A legitimate stop always fires on the in-loop call, so skipping the + evaluation can never suppress a real stop. They DO land in the collector: + the agent's own ``EventCollector`` records force-closed commands into the + ``TurnRecord`` that ``check_all_async`` later scores (including a crashed + attempt's drained partial turn), so the watcher must reduce the same + trajectory — otherwise its verdicts (and the fail-stop's ceiling bound) + would be computed over a strictly smaller command set than the + authoritative check, and a ``decide_within`` timeout could latch an + effective fail on a criterion the frozen trajectory scores as a pass. """ - if self._info is not None or self._disarmed: - return if isinstance(event, AgentStartEvent): if self._started_monotonic is None: self._started_monotonic = time.monotonic() @@ -385,14 +451,18 @@ def on_event(self, event: StreamEvent) -> None: elif isinstance(event, ToolStartEvent): # Decide on the call, evaluating with it appended as the in-flight # command (it has no ToolEnd to count yet, so report it as +1). - self._evaluate(in_flight=event.tool) + self._evaluate_impl(in_flight=event.tool) return elif isinstance(event, ToolEndEvent): if event.status == ToolEndStatus.UNRESOLVED: + # Trajectory parity with the agent's collector (see docstring): + # record, but don't count a round or evaluate — a force-closed + # orphan is not live tool activity and must not fire a stop. + self._collector.on_event(event) return self._tool_call_index += 1 self._collector.on_event(event) - self._evaluate() + self._evaluate_impl() return self._collector.on_event(event) @@ -412,30 +482,17 @@ def disarmed(self) -> bool: # --- Stop rule -------------------------------------------------- # - @staticmethod - def _resolve_armed_polarities(criterion: LiveSuccessCriterion) -> frozenset[LivePolarity]: - """The polarities this armed instance may fire, via ``_requested_polarities``. - - Validation has already guaranteed the resolved set is non-empty and - decidable for every armed criterion. ``None`` is never armed, but is - mapped to the empty set defensively so the caller need not special-case - it. - """ - sw = criterion.stop_when - if sw is None: - return frozenset() - return _requested_polarities(sw, criterion.live_decidable_polarities()) - def _ceiling(self, verdicts: list[LiveVerdict]) -> float: """Best-case weighted score over the WHOLE armed set, given current verdicts. - Every already-live-failed criterion is pinned at 0 (a monotonic - ``live_verdict`` guarantees it stays failed); every ``pass`` or still - ``undecided`` criterion is credited its full weight (the optimistic - assumption that it could still end up scoring 1.0). This is the same - weighting ``EvaluationResult.armed_criteria_passed`` uses for the real, - final gate, so ``ceiling < gate_threshold`` means the gate is - mathematically guaranteed to fail no matter how the trajectory continues. + Every already-failed criterion (native live-fail or expired budget) is + pinned at 0 (a monotonic ``live_verdict`` guarantees it stays failed); + every ``pass`` or still ``undecided`` criterion is credited its full + weight (the optimistic assumption that it could still end up scoring + 1.0). This is the same weighting ``EvaluationResult. + armed_criteria_passed`` uses for the real, final gate, so ``ceiling < + gate_threshold`` means the gate is mathematically guaranteed to fail no + matter how the trajectory continues. """ return sum(c.weight for (c, _checker), v in zip(self._armed, verdicts, strict=True) if v != "fail") / ( self._armed_weight @@ -457,28 +514,21 @@ def _floor(self, verdicts: list[LiveVerdict], indices: list[int]) -> float | Non return None return sum(self._armed[i][0].weight for i in indices if verdicts[i] == "pass") / weight - def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None: - """Fail-open wrapper: any unexpected exception anywhere in the round - disarms the watcher and degrades to a full run, exactly like a raising - ``live_verdict`` — not just the verdict-collection loop. The - ceiling/floor arithmetic below is currently guarded (an empty armed - set is unreachable, and every candidate short-circuits before - dividing), but a future change to that arithmetic — or to - ``_fire`` — must not be able to leave the watcher stuck re-raising on - every subsequent event with ``_disarmed`` still False (which is what a - narrower try/except would risk). - """ - try: - self._evaluate_impl(in_flight) - except Exception: - self._disarmed = True - logger.error( - "[%s] early-stop round raised unexpectedly; disarming watcher, run degrades to a full run", - self._task_id, - exc_info=True, - ) + def _collect_verdicts(self, in_flight: CommandTelemetry | None, tool_call_index: int) -> list[LiveVerdict]: + """One round of effective verdicts, latching decided ones on resolved rounds. - def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: + A latched (non-``undecided``) verdict is returned as-is — its checker is + never polled again (the checkers' monotonicity contract makes re-polling + pure waste). A fresh ``undecided`` on a pass-capable instance whose + ``decide_within`` budget has expired becomes an *effective* + ``fail`` (marked in ``_budget_expired`` for reason attribution). + + Latching only happens on RESOLVED rounds (``in_flight is None``): an + in-flight round's verdict may fire a stop this round, but is not + persisted — a dispatched call that never resolves (crashed attempt) + must not leave a stale verdict behind across retries. The verdict is + recomputed from the collector's resolved commands on the next round. + """ record = self._collector.build_turn_record() if in_flight is not None: # The in-flight call has no ToolEnd yet, so the collector (which @@ -487,15 +537,15 @@ def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: # partial trajectory in emission order. record.commands = sorted([*record.commands, in_flight], key=lambda c: c.sequence_number) records = [record] - # An in-flight call has not been counted by a ToolEnd yet, so report it as - # the next (1-based) tool call. - tool_call_index = self._tool_call_index + (1 if in_flight is not None else 0) verdicts: list[LiveVerdict] = [] - for criterion, checker in self._armed: + for i, (criterion, checker) in enumerate(self._armed): + if self._latched[i] != "undecided": + verdicts.append(self._latched[i]) + continue try: - verdicts.append(checker.live_verdict(criterion, records)) + verdict: LiveVerdict = checker.live_verdict(criterion, records) except Exception: - # Re-raise to the wrapping try/except in _evaluate, which sets + # Re-raise to the wrapping try/except in on_event, which sets # _disarmed — but log the specific criterion here first, since # that context (which criterion's live_verdict raised) would # otherwise be lost once the exception is caught generically. @@ -506,86 +556,135 @@ def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: exc_info=True, ) raise + budget = self._budget[i] + budget_expired = verdict == "undecided" and budget is not None and tool_call_index >= budget + if budget_expired: + verdict = "fail" + if in_flight is None and verdict != "undecided": + self._latched[i] = verdict + self._budget_expired[i] = budget_expired + verdicts.append(verdict) + return verdicts + + def _budget_drove(self, index: int, verdicts: list[LiveVerdict], tool_call_index: int) -> bool: + """True when ``index``'s ``fail`` is timeout-driven rather than a native live-fail. + + Reads the persistent ``_budget_expired`` latch when set; for a + transient (in-flight, not-yet-latched) fail it re-derives: a fail on an + instance that cannot natively live-fail, with an expired budget, can + only have come from the timeout. A native live-fail on an instance + whose budget also happens to be expired reports as a native fail — + ``_collect_verdicts`` only converts the verdict when the checker itself + returned ``undecided``. + """ + if self._budget_expired[index]: + return True + budget = self._budget[index] + return ( + verdicts[index] == "fail" + and self._latched[index] == "undecided" + and budget is not None + and tool_call_index >= budget + and "fail" not in self._decidable[index] + ) - pass_armed = [i for i, pol in enumerate(self._armed_polarities) if "pass" in pol] - - # Fail-stop: at least one armed criterion (criteria order) that live-fails - # AND whose resolved arming permits fail is a CANDIDATE — but the stop only - # actually fires once the ceiling bound (best case: every still-undecided - # or already-passed armed criterion ends up scoring 1.0, every live-failed - # one scores 0) can no longer reach ``gate_threshold``, i.e. the armed gate - # (``EvaluationResult.armed_criteria_passed``) is GUARANTEED to fail no - # matter what happens on the rest of the trajectory. At the default - # ``gate_threshold=1.0`` this is equivalent to firing on the first - # candidate (any armed criterion's weight is > 0 by construction, so a - # single fail already drops the ceiling below 1.0) — below 1.0 a - # low-weight candidate's failure may not be enough to doom the gate, so the - # run keeps going. This is DEFERRED while any pass-armed criterion is still - # undecided. Cutting a positive row on a distractor misfire before its - # expected signal could appear would freeze a would-be TP as an FN - # (truncating the suite's recall); the misfire is latched by the - # criterion's own monotone semantics, so the deferred fail still fires the - # moment every pass-armed criterion decides, and a row with zero - # pass-armed criteria (a negative row) defers nothing. - if not any(verdicts[i] == "undecided" for i in pass_armed): - candidate = next( - ( - criterion - for (criterion, _checker), verdict, armed_pol in zip( - self._armed, verdicts, self._armed_polarities, strict=True - ) - if verdict == "fail" and "fail" in armed_pol - ), - None, - ) - if candidate is not None and self._ceiling(verdicts) < self._gate_threshold: - self._fire(EarlyStopReason.CRITERION_FAILED, candidate, tool_call_index=tool_call_index) - return + def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: + # An in-flight call has not been counted by a ToolEnd yet, so report it as + # the next (1-based) tool call. + tool_call_index = self._tool_call_index + (1 if in_flight is not None else 0) + verdicts = self._collect_verdicts(in_flight, tool_call_index) + + # Recall deferral: while any pass-capable armed criterion is still + # undecided (within its budget — an expired budget is already an + # effective fail), a fail-stop is HELD. Cutting a positive row on a + # distractor misfire before its expected signal could appear would + # freeze a would-be TP as an FN (truncating the suite's recall); the + # misfire latches via the criterion's own monotone semantics, so the + # deferred fail still fires the moment every pass-capable criterion + # decides, and a row with zero pass-capable criteria (a negative row) + # defers nothing. + pass_capable_undecided = any( + v == "undecided" and "pass" in pol for v, pol in zip(verdicts, self._decidable, strict=True) + ) - # Pass-stop: the PASS-ARMED subset's own floor bound (worst case: every - # still-undecided pass-armed criterion ends up scoring 0, weighted against - # only the pass-armed subset's total weight) already meets - # ``gate_threshold`` — guaranteed regardless of what the rest of that - # subset still decides. Fail-armed criteria (e.g. distractors armed - # ``auto`` -> fail only) are excluded from both the numerator and the - # denominator: they can never live-pass and only guard the fail side - # above, so folding them in would veto every pass-stop (the mixed-arming - # bug) and penalize this bound for a criterion it was never scoped to - # cover. At the default ``gate_threshold=1.0`` this requires every - # pass-armed criterion to actually be "pass" (any non-pass drops the floor - # below 1.0) — identical to the pre-weighting ``all(...)`` rule. Guard the - # vacuous case: with zero pass-armed criteria (a negative row whose - # criteria are all distractors) there is nothing to pass-stop on, so the - # run must continue to the cap rather than firing on turn 0 with an empty - # numerator/denominator. - floor = self._floor(verdicts, pass_armed) - if floor is not None and floor >= self._gate_threshold: - # Deciding criterion = the last pass-armed (criteria order) whose verdict - # flipped vs the previous round; fall back to the last pass-armed. - deciding = self._armed[pass_armed[-1]][0] - for i in pass_armed: - if verdicts[i] != self._prev_verdicts[i]: - deciding = self._armed[i][0] - self._fire(EarlyStopReason.CRITERION_PASSED, deciding, tool_call_index=tool_call_index) - return + # Fail-stop: a criterion whose effective verdict is "fail" is a + # CANDIDATE — the fail trigger is implicit in arming (native fail on a + # fail-capable instance, or a decide_within timeout). The stop only fires once the ceiling bound (best case: + # every still-undecided or already-passed armed criterion ends up + # scoring 1.0, every failed one scores 0) can no longer reach + # ``gate_threshold``, i.e. the armed gate (``EvaluationResult. + # armed_criteria_passed``) is GUARANTEED to fail no matter what happens + # on the rest of the trajectory. At the default ``gate_threshold=1.0`` + # this is equivalent to firing on the first candidate (any armed + # criterion's weight is > 0 by construction, so a single fail already + # drops the ceiling below 1.0) — below 1.0 a low-weight candidate's + # failure (or timeout) may not be enough to doom the gate, so the run + # keeps going: the failure is absorbed. + if not pass_capable_undecided: + # Deterministic precedence: a native live-fail candidate always wins + # over a budget-driven one, so the persisted/telemetry reason cannot + # flip between CRITERION_FAILED and DECISION_BUDGET_EXCEEDED on a + # mere reorder of ``success_criteria`` when both resolve on the same + # round. Within each class, first criteria-order match wins. + native_fails = [ + i + for i, v in enumerate(verdicts) + if v == "fail" and self._fail_trigger[i] and not self._budget_drove(i, verdicts, tool_call_index) + ] + budget_fails = [ + i for i, v in enumerate(verdicts) if v == "fail" and self._budget_drove(i, verdicts, tool_call_index) + ] + candidate_index = native_fails[0] if native_fails else (budget_fails[0] if budget_fails else None) + if candidate_index is not None and self._ceiling(verdicts) < self._gate_threshold: + reason = EarlyStopReason.CRITERION_FAILED if native_fails else EarlyStopReason.DECISION_BUDGET_EXCEEDED + self._fire(reason, self._armed[candidate_index][0], tool_call_index=tool_call_index) + return - # Decision-step budget: an armed criterion with max_steps_to_decide set - # that is STILL "undecided" once that many tool-call steps have elapsed - # forces a hard fail — checked last, after the real fail-/pass-stop - # checks above, so a criterion that decides on this very round (however - # late) is never punished for a budget it technically exceeded. It never - # reached a verdict at all, so there is nothing meaningful to weigh it - # against — this is why DECISION_BUDGET_EXCEEDED bypasses the weighted - # gate entirely at the orchestrator finalize step rather than folding - # into the ceiling/floor bounds above. - for (criterion, _checker), verdict in zip(self._armed, verdicts, strict=True): - budget = criterion.max_steps_to_decide - if budget is not None and verdict == "undecided" and tool_call_index >= budget: - self._fire(EarlyStopReason.DECISION_BUDGET_EXCEEDED, criterion, tool_call_index=tool_call_index) + # Pass-stop: the on_pass=stop subset's own floor bound (worst case: + # every still-undecided member scores 0, weighted against only that + # subset's total weight) already meets ``gate_threshold`` — guaranteed + # regardless of what the rest of that subset still decides. Criteria + # armed only on the fail side (distractors) are excluded from both the + # numerator and the denominator: they can never live-pass and only + # guard the fail side above, so folding them in would veto every + # pass-stop and penalize this bound for a criterion it was never + # scoped to cover. At the default ``gate_threshold=1.0`` this requires + # every on_pass=stop criterion to actually be "pass" (any non-pass + # drops the floor below 1.0). The vacuous case (no on_pass=stop + # criteria at all) returns None — nothing to pass-stop on, the run + # continues to the cap. + # + # Recall deferral, mirrored from the fail-stop: the pass-stop is HELD + # while any pass-capable armed criterion OUTSIDE the on_pass=stop + # subset is still undecided (members of the subset are already priced + # into the floor). Cutting here would freeze a sibling + # ``on_pass: continue`` criterion's expected signal out of the + # trajectory — an unearned fail on the armed gate that a full run + # would not have produced. Once every such criterion decides (pass or + # fail), the still-satisfied floor fires the pass-stop on that round. + pass_stop_indices = [i for i, armed_pass in enumerate(self._pass_trigger) if armed_pass] + outside_pass_capable_undecided = any( + v == "undecided" and "pass" in pol and not armed_pass + for v, pol, armed_pass in zip(verdicts, self._decidable, self._pass_trigger, strict=True) + ) + if not outside_pass_capable_undecided: + floor = self._floor(verdicts, pass_stop_indices) + if floor is not None and floor >= self._gate_threshold: + # Deciding criterion = the last on_pass=stop (criteria order) whose + # verdict flipped vs the previous round; fall back to the last one. + deciding = self._armed[pass_stop_indices[-1]][0] + for i in pass_stop_indices: + if verdicts[i] != self._prev_verdicts[i]: + deciding = self._armed[i][0] + self._fire(EarlyStopReason.CRITERION_PASSED, deciding, tool_call_index=tool_call_index) return - # No stop this round — record the verdicts so the next round can detect flips. - self._prev_verdicts = verdicts + # No stop this round — record the verdicts so the next round can detect + # flips. Resolved rounds only: an in-flight round's verdicts are + # deliberately not latched (the call may never resolve), so persisting + # them here would let a transient round mask the real flip attribution. + if in_flight is None: + self._prev_verdicts = verdicts def _fire(self, reason: EarlyStopReason, criterion: LiveSuccessCriterion, *, tool_call_index: int) -> None: elapsed = 0.0 diff --git a/src/coder_eval/orchestration/experiment.py b/src/coder_eval/orchestration/experiment.py index f84a4a59..f75c585e 100644 --- a/src/coder_eval/orchestration/experiment.py +++ b/src/coder_eval/orchestration/experiment.py @@ -668,7 +668,7 @@ def resolve_all_tasks( _apply_cli_overrides(resolved_task, config, lineage) # Early-stop guardrails: run once the task is fully resolved (all 5 - # layers merged, incl. -D run_limits.stop_early). No-op unless armed; + # layers merged, incl. the -D run_limits.stop_early kill switch). No-op unless armed; # a bad arming raises EarlyStopConfigError (a ValueError). validate_early_stop(resolved_task) diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index 138ad722..8640dd7f 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -11,7 +11,7 @@ from datetime import datetime from inspect import isawaitable from pathlib import Path -from typing import Any, assert_never +from typing import Any from urllib.parse import urlparse from .agent import Agent @@ -30,6 +30,7 @@ from .evaluation.checker import SuccessChecker, _short_failure_reason from .litellm_cost import apply_actual_cost, load_cost_records from .models import ( + DEFAULT_STOP_EARLY_GATE_THRESHOLD, ROUTE_NAMES, AgentKind, ApiRoute, @@ -37,7 +38,6 @@ ConfigLineageEntry, CriterionResult, DirectRoute, - EarlyStopReason, EvaluationResult, FinalStatus, JudgeCriterionResult, @@ -56,7 +56,7 @@ resolve_evaluation_route, resolve_route, ) -from .orchestration.early_stop import EarlyStopWatcher, validate_early_stop +from .orchestration.early_stop import EarlyStopWatcher, early_stop_active, validate_early_stop from .orchestration.evaluation import load_reference from .path_utils import format_task_log_id, task_log_path from .sandbox import Sandbox @@ -390,8 +390,9 @@ def __init__( # Reference solution cache (loaded on-demand) self._reference_code: str | None = None - # Early-stop watcher (created in _setup only when run_limits.stop_early is - # armed; None otherwise, so the default path is entirely unaffected). + # Early-stop watcher (created in _setup only when a criterion carries a + # stop_early: block and the kill switch is not thrown; None otherwise, + # so the default path is entirely unaffected). self._early_stop_watcher: EarlyStopWatcher | None = None # One-shot flag: emit the "cost budget configured but no cost data" warning @@ -985,14 +986,15 @@ async def _setup(self) -> None: """ # Defensive early-stop guardrails for the library-use and in-container # paths (the CLI already validated during resolution). No-op unless - # run_limits.stop_early is armed. + # some criterion carries a stop_early: block. validate_early_stop(self.task) - # Build the early-stop watcher once, up front, when armed. This sits BEFORE - # the evaluate-only early return below, so an armed evaluate-only re-grade - # builds an inert (never-fed) watcher — harmless, and keeps a single - # creation point. - if self.task.run_limits is not None and self.task.run_limits.stop_early: + # Build the early-stop watcher once, up front, when armed (>= 1 criterion + # with a stop_early: block and the run_limits.stop_early kill switch not + # thrown). This sits BEFORE the evaluate-only early return below, so an + # armed evaluate-only re-grade builds an inert (never-fed) watcher — + # harmless, and keeps a single creation point. + if early_stop_active(self.task): self._early_stop_watcher = EarlyStopWatcher.for_task(self.task) if self.sandbox is not None: @@ -1568,60 +1570,43 @@ async def _evaluation_loop(self) -> bool: pairs = list(zip(criteria_results, self.task.success_criteria, strict=True)) passed_count = sum(1 for r, c in pairs if r.score >= c.pass_threshold) total_count = len(pairs) - # The armed weighted gate governs any task armed for early-stop - # (run_limits.stop_early: true), whether or not the watcher actually - # fired — one task config maps to one gate semantic. Without this, a - # run that happened to finish before the bound ever tripped would - # silently fall back to the strict full-set gate, re-failing on a - # low-weight criterion the weighted gate was configured to forgive, - # for a reason (incidental control flow) unrelated to configured - # intent. Only a task NOT armed for early-stop uses the full, - # strict-AND gate over every gating criterion. - if self.task.run_limits is not None and self.task.run_limits.stop_early: - if self.result.early_stop is not None: - reason = self.result.early_stop.reason - # Exhaustive on EarlyStopReason: adding a 4th member without a - # branch here is a type error (assert_never), not a silent - # fall-through into the weighted-gate branch below. - if reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED: - # Hard fail, bypassing the weighted gate entirely: the deciding - # criterion never reached a verdict within its max_steps_to_decide - # budget, so there is nothing meaningful to weigh it against. - all_passed = False - logger.info( - "Early-stopped run: decision-step budget exceeded for %r, forcing FAILURE.", - self.result.early_stop.deciding_criterion_description, - ) - elif reason == EarlyStopReason.CRITERION_PASSED or reason == EarlyStopReason.CRITERION_FAILED: - all_passed = self.result.armed_criteria_passed( - self.task.success_criteria, self.task.run_limits.stop_early_gate_threshold - ) - armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) - logger.info( - "Early-stopped run: gating on %d armed criteria (%d advisory, not gated).", - armed_count, - total_count - armed_count, - ) - else: - assert_never(reason) - else: - # Armed for early-stop but the watcher never fired (the agent - # finished, or max_turns was hit, before the bound tripped): - # only the armed subset gates final_status; the rest are - # advisory (recorded, never decisive) — same gate as an - # actual early stop, so a smoke flavor is not dragged to - # FAILURE by criteria whose work it deliberately skipped. - all_passed = self.result.armed_criteria_passed( - self.task.success_criteria, self.task.run_limits.stop_early_gate_threshold - ) - armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) - logger.info( - "stop_early armed but never fired (run completed naturally): gating on " - + "%d armed criteria (%d advisory, not gated).", - armed_count, - total_count - armed_count, - ) + # Gate selection is FIRED-ONLY: the weighted armed gate applies iff the + # watcher actually cut the run (early_stop is not None) — on a truncated + # trajectory the unarmed criteria never had the chance to be satisfied, + # so they stay advisory. A run that completed naturally (armed or not, + # watcher never fired or disarmed fail-open) has a full trajectory and + # gates strict-AND over every gating criterion, exactly like an unarmed + # run — arming a criterion (e.g. adding a decide_within fail-fast + # timeout) must never change the verdict of a run it didn't cut. + if self.result.early_stop is not None: + # One gate for every early-stopped run, no per-reason branches: a + # decision-budget stop is just a fail-stop whose deciding criterion + # timed out (the watcher only fires once the weighted ceiling + # proves the armed gate cannot pass). The ceiling is an upper bound + # on the authoritative armed score only because the watcher reduces + # the SAME trajectory the checker scores — it records UNRESOLVED + # tool ends exactly like the agent's EventCollector does (see + # EarlyStopWatcher._on_event_impl) — so the weighted armed gate is + # correct whether the watcher fired on a pass, a fail, or a timeout. + gate_threshold = ( + self.task.run_limits.stop_early_gate_threshold + if self.task.run_limits is not None + else DEFAULT_STOP_EARLY_GATE_THRESHOLD + ) + all_passed = self.result.armed_criteria_passed(self.task.success_criteria, gate_threshold) + armed_count = sum(1 for c in self.task.success_criteria if c.is_stop_armed) + logger.info( + "Early-stopped run (%s): gating on %d armed criteria (%d advisory, not gated).", + self.result.early_stop.reason.value, + armed_count, + total_count - armed_count, + ) else: + if self._early_stop_watcher is not None: + if self._early_stop_watcher.disarmed: + logger.info("early-stop watcher disarmed fail-open (verdict error): gating on the full set.") + else: + logger.info("early-stop armed but never fired (run completed naturally): gating on the full set.") all_passed = self.result.all_criteria_passed(self.task.success_criteria) # Reuse the model method for weighted score (single source of truth) diff --git a/src/coder_eval/reports.py b/src/coder_eval/reports.py index b55f6f8c..5ff79eab 100644 --- a/src/coder_eval/reports.py +++ b/src/coder_eval/reports.py @@ -6,11 +6,12 @@ from collections import defaultdict from collections.abc import Callable, Iterable, Sequence from pathlib import Path, PurePosixPath -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, assert_never from .models import ( CriterionAggregate, CriterionStats, + EarlyStopReason, FailedRowSummary, SuiteRollup, TaskResult, @@ -166,6 +167,38 @@ def _fmt_rate(rate: float | None) -> str: return f"{rate * 100:.1f}%" if rate is not None else "n/a" +def early_stop_gate_note(reason: str) -> str: + """The gate-explaining sentence for an early-stopped run, shared verbatim by + every report surface (markdown run-time note, HTML badge tooltip) so the + prose can never drift between renderers. + + ``reason`` is the ``EarlyStopReason`` string value (e.g. ``run.json``'s + ``early_stop_reason``); an unrecognized value (a legacy or hand-edited + record, or run.json's literal ``"unknown"``) gets the generic note. The + ``match`` is exhaustiveness-checked so a new ``EarlyStopReason`` member + cannot silently fall through to the wrong prose. + """ + generic = "gated on armed criteria only; other criteria are advisory" + try: + member = EarlyStopReason(reason) + except ValueError: + return generic + note = generic + match member: + case EarlyStopReason.DECISION_BUDGET_EXCEEDED: + # The deciding criterion timed out undecided (an effective fail); it + # gates through the same weighted armed gate as a native live-fail. + note = ( + "decision-step budget exceeded (criterion timed out undecided, treated as a failed " + "armed criterion); gated on armed criteria only; other criteria are advisory" + ) + case EarlyStopReason.CRITERION_PASSED | EarlyStopReason.CRITERION_FAILED: + pass # the generic note + case _: + assert_never(member) + return note + + def _pass_rate_lines(summary: RunSummary) -> list[str]: """The pass rate over every dispatched task, plus the error share when non-zero.""" lines = [f"- **Pass Rate**: {_fmt_rate(summary.pass_rate)} ({summary.tasks_succeeded}/{summary.tasks_run})"] @@ -438,14 +471,9 @@ def _runtime_notes_lines(summary: RunSummary) -> list[str]: reason = t.get("early_stop_reason") or "unknown" turns_remaining = t.get("turns_remaining_at_stop") avoided = f" <= {turns_remaining} turn(s) avoided —" if isinstance(turns_remaining, int) else "" - if reason == "decision_budget_exceeded": - # No criterion gated here at all — an armed criterion's - # decision-step budget expired unresolved, forcing FAILURE - # outright, bypassing the weighted gate entirely. - gate_note = " forced to FAILURE (decision-step budget exceeded, bypassing the gate)" - else: - gate_note = " gated on armed criteria only; other criteria are advisory" - notes.append(f"> **NOTE:** [{task_id}] stopped early ({reason});{avoided}" + gate_note) + notes.append( + f"> **NOTE:** [{task_id}] stopped early ({reason});{avoided} {early_stop_gate_note(reason)}" + ) if not notes: return [] return ["## Run-time Notes", "", *notes] diff --git a/src/coder_eval/reports_experiment.py b/src/coder_eval/reports_experiment.py index 8f2ec675..e141174b 100644 --- a/src/coder_eval/reports_experiment.py +++ b/src/coder_eval/reports_experiment.py @@ -202,7 +202,7 @@ def eval_result_to_task_dict( "visible_turns": visible_turn_count(result), "expected_turns": expected_turns_value, "has_final_reply": has_reply, - # Early-stop surfaces (opt-in run_limits.stop_early). None/False on the + # Early-stop surfaces (opt-in per-criterion stop_early: blocks). None/False on the # default path so downstream analysis never confuses a truncated run # with a full one. "stopped_early": result.early_stop is not None, diff --git a/src/coder_eval/reports_html.py b/src/coder_eval/reports_html.py index e5a2c048..48ab5519 100644 --- a/src/coder_eval/reports_html.py +++ b/src/coder_eval/reports_html.py @@ -17,7 +17,9 @@ from pathlib import Path from typing import TYPE_CHECKING, Any -from coder_eval.models import EarlyStopReason, FinalStatus, eval_result_total_cost, sum_costs +from coder_eval.models import FinalStatus, eval_result_total_cost, sum_costs + +from .reports import early_stop_gate_note if TYPE_CHECKING: @@ -345,14 +347,9 @@ def _render_header(result: EvaluationResult) -> str: expected_turns_badge = f'expected_turns exceeded ({actual}/{expected})' early_stop_badge = "" if result.early_stop is not None: - if result.early_stop.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED: - # No criterion gated here at all — forced to FAILURE outright, - # bypassing the weighted gate. - title = "Forced to FAILURE (decision-step budget exceeded, bypassing the gate)" - else: - title = "Gated on armed criteria only; other criteria are advisory" + title = early_stop_gate_note(result.early_stop.reason.value) early_stop_badge = ( - f'' + f'' + f"stopped early ({_esc(result.early_stop.reason.value)})" ) return f""" diff --git a/tasks/early_stop_decision_budget_exceeded.yaml b/tasks/early_stop_decision_budget_exceeded.yaml index 0d287f5a..f2786016 100644 --- a/tasks/early_stop_decision_budget_exceeded.yaml +++ b/tasks/early_stop_decision_budget_exceeded.yaml @@ -1,15 +1,19 @@ task_id: "early_stop_decision_budget_exceeded" description: > - Decision-step budget (GitHub issue #61, item 3): the armed command_executed - criterion caps at max_steps_to_decide: 3 — if the agent hasn't run the - script within its first 3 tool calls, EarlyStopWatcher fires - reason=decision_budget_exceeded and the run is force-failed outright, - bypassing the weighted stop_early_gate_threshold gate entirely (a criterion - that never reached a verdict has nothing meaningful to weigh against the - advisory file_exists criterion below). This is a NON-CI example task - (deliberately untagged for smoke-pass/smoke-fail): the agent's actual - exploration behavior is not deterministic enough for a live-agent CI - assertion — run it manually with `coder-eval run` to observe the mechanism. + Decision-step timeout (GitHub issue #61, item 3): the armed command_executed + criterion sets stop_early.decide_within: 3 — if the agent hasn't run the script + within its first 3 tool calls, EarlyStopWatcher latches an effective FAIL + for it (reported as reason=decision_budget_exceeded) and, since that fail + drops the armed ceiling below the default gate threshold of 1.0, the run + fail-stops immediately instead of burning the rest of run_limits.max_turns. + The timeout gates through the same weighted armed gate as a native + live-fail — no special force-fail path. Note the block's on_pass stays + 'continue', so a live PASS never stops the run + (if the agent runs the script within budget, the run simply continues to + natural completion). This is a NON-CI example task (deliberately untagged + for smoke-pass/smoke-fail): the agent's actual exploration behavior is not + deterministic enough for a live-agent CI assertion — run it manually with + `coder-eval run` to observe the mechanism. tags: [early-stop, decision-budget] initial_prompt: > @@ -25,7 +29,6 @@ agent: run_limits: max_turns: 20 - stop_early: true success_criteria: - type: "command_executed" @@ -33,8 +36,8 @@ success_criteria: tool_name: "Bash" command_pattern: "python app\\.py" min_count: 1 - stop_when: "auto" - max_steps_to_decide: 3 + stop_early: + decide_within: 3 - type: "file_exists" path: "app.py" description: "app.py must exist (advisory — not armed, only gates a full run)" diff --git a/tasks/early_stop_weighted_high_weight_kills_run.yaml b/tasks/early_stop_weighted_high_weight_kills_run.yaml index 141e7892..e9dcfe23 100644 --- a/tasks/early_stop_weighted_high_weight_kills_run.yaml +++ b/tasks/early_stop_weighted_high_weight_kills_run.yaml @@ -2,17 +2,17 @@ task_id: "early_stop_weighted_high_weight_kills_run" description: > Weighted early-stop (GitHub issue #61, item 1) — the mirror case of early_stop_weighted_low_weight_absorbed.yaml. Same two armed criteria, but - the weights are swapped: a LOW-weight (0.2) positive ("ran python app.py") - and a HIGH-weight (0.8) distractor ("never called curl"). If the distractor - misfires, the ceiling (best case: the low-weight positive still passes) is - only 0.2, which can never reach the 0.7 gate threshold — the run is - mathematically guaranteed to fail the armed gate no matter what happens - next. The fail-stop is DEFERRED, though, while the low-weight positive is - still undecided (a distractor misfire must not truncate recall) — with the - prompt ordering curl before python app.py, the fail-stop actually fires as - soon as the positive resolves (not "immediately" on the curl misfire - itself), rather than burning the rest of run_limits.max_turns on a doomed - run. This is a NON-CI example task + the weights are swapped: a LOW-weight (0.2) positive ("ran python app.py", + stop_early.on_pass: stop) and a HIGH-weight (0.8) distractor ("never + called curl", an empty stop_early block). If the distractor misfires, the ceiling (best case: the + low-weight positive still passes) is only 0.2, which can never reach the + 0.7 gate threshold — the run is mathematically guaranteed to fail the armed + gate no matter what happens next. The fail-stop is DEFERRED, though, while + the pass-capable positive is still undecided (a distractor misfire must not + truncate recall) — with the prompt ordering curl before python app.py, the + fail-stop actually fires as soon as the positive resolves (not + "immediately" on the curl misfire itself), rather than burning the rest of + run_limits.max_turns on a doomed run. This is a NON-CI example task (deliberately untagged for smoke-pass/smoke-fail): whether the agent actually calls curl as instructed is not deterministic enough for a live-agent CI assertion — run it manually with `coder-eval run` to observe @@ -31,7 +31,6 @@ agent: run_limits: max_turns: 20 - stop_early: true stop_early_gate_threshold: 0.7 success_criteria: @@ -41,7 +40,8 @@ success_criteria: command_pattern: "python app\\.py" min_count: 1 weight: 0.2 - stop_when: "auto" + stop_early: + on_pass: stop - type: "command_executed" description: "Agent did NOT call curl (the HIGH-weight distractor)" tool_name: "Bash" @@ -49,7 +49,7 @@ success_criteria: min_count: 0 max_count: 0 weight: 0.8 - stop_when: "auto" + stop_early: {} - type: "file_exists" path: "app.py" description: "app.py must exist (advisory — not armed, only gates a full run)" diff --git a/tasks/early_stop_weighted_low_weight_absorbed.yaml b/tasks/early_stop_weighted_low_weight_absorbed.yaml index a4e13a47..b97b3862 100644 --- a/tasks/early_stop_weighted_low_weight_absorbed.yaml +++ b/tasks/early_stop_weighted_low_weight_absorbed.yaml @@ -3,19 +3,20 @@ description: > Weighted early-stop (GitHub issue #61, item 1): a LOW-weight armed criterion misfiring must not unilaterally truncate the run once run_limits. stop_early_gate_threshold is below 1.0. Two armed criteria: a HIGH-weight - (0.8) positive ("ran python app.py") and a LOW-weight (0.2) distractor - ("never called curl"). If the distractor misfires alone, the ceiling - (best case: the positive still passes) is 0.8, which still clears the 0.7 - gate threshold — so the watcher must keep running instead of fail-stopping, - and the low-weight failure must not sink the final armed_criteria_passed - gate either (weighted score 0.8 >= 0.7). Note: the weighted forgiveness - applies only when the watcher itself fires the stop (pass-stops here once - the positive resolves, regardless of the distractor) — a stop_early: true - run that instead completes naturally still gates on the full strict-AND - set, unaffected by weight. This is a NON-CI example task (deliberately - untagged for smoke-pass/smoke-fail): agent behavior around curl is not - deterministic enough for a live-agent CI assertion — run it manually with - `coder-eval run` to observe the mechanism. + (0.8) positive ("ran python app.py", stop_early.on_pass: stop) and a LOW-weight (0.2) + distractor ("never called curl", an empty stop_early block). If the distractor misfires + alone, the ceiling (best case: the positive still passes) is 0.8, which + still clears the 0.7 gate threshold — so the watcher must keep running + instead of fail-stopping, and the low-weight failure must not sink the + final armed_criteria_passed gate either (weighted score 0.8 >= 0.7). Once + the positive resolves, its on_pass=stop floor (0.8/0.8 = 1.0 over the + on_pass=stop subset) meets the threshold and the run pass-stops. Gating is + FIRED-ONLY: the weighted armed gate applies only when the watcher actually + cuts the run; a run that completes naturally gates strict-AND over the full + criteria set (armed or not). This is a NON-CI example task + (deliberately untagged for smoke-pass/smoke-fail): agent behavior around + curl is not deterministic enough for a live-agent CI assertion — run it + manually with `coder-eval run` to observe the mechanism. tags: [early-stop, weighted-early-stop] initial_prompt: > @@ -30,7 +31,6 @@ agent: run_limits: max_turns: 20 - stop_early: true stop_early_gate_threshold: 0.7 success_criteria: @@ -40,7 +40,8 @@ success_criteria: command_pattern: "python app\\.py" min_count: 1 weight: 0.8 - stop_when: "auto" + stop_early: + on_pass: stop - type: "command_executed" description: "Agent did NOT call curl (the LOW-weight distractor)" tool_name: "Bash" @@ -48,7 +49,7 @@ success_criteria: min_count: 0 max_count: 0 weight: 0.2 - stop_when: "auto" + stop_early: {} - type: "file_exists" path: "app.py" description: "app.py must exist (advisory — not armed, only gates a full run)" diff --git a/tests/test_early_stop.py b/tests/test_early_stop.py index 9537853a..3f02aa83 100644 --- a/tests/test_early_stop.py +++ b/tests/test_early_stop.py @@ -39,12 +39,14 @@ from coder_eval.agents.codex_agent import CodexAgent, _CodexTurnState from coder_eval.agents.registry import AgentRegistry from coder_eval.cli.plan_command import plan_command +from coder_eval.config import settings from coder_eval.criteria import CriterionRegistry, init_criteria from coder_eval.criteria.command_executed import CommandExecutedChecker from coder_eval.criteria.skill_triggered import SkillTriggeredChecker, _engaged_skill_names from coder_eval.errors import AgentCrashError, TurnTimeoutError from coder_eval.models import ( AgentKind, + ApiBackend, BaseAgentConfig, CommandExecutedCriterion, CommandTelemetry, @@ -62,13 +64,19 @@ SandboxConfig, SimulationConfig, SkillTriggeredCriterion, + StopEarlyPolicy, TaskDefinition, TurnRecord, parse_agent_config, ) from coder_eval.orchestration.config import BatchRunConfig -from coder_eval.orchestration.early_stop import EarlyStopConfigError, EarlyStopWatcher, validate_early_stop -from coder_eval.orchestration.experiment import resolve_all_tasks +from coder_eval.orchestration.early_stop import ( + EarlyStopConfigError, + EarlyStopWatcher, + early_stop_active, + validate_early_stop, +) +from coder_eval.orchestration.experiment import load_experiment, resolve_all_tasks from coder_eval.orchestrator import Orchestrator, build_task_event from coder_eval.reports import ReportGenerator from coder_eval.reports_experiment import eval_result_to_task_dict @@ -109,12 +117,18 @@ def _turn(*commands: CommandTelemetry) -> TurnRecord: def _task( *, criteria: list[Any], - stop_early: bool = False, + stop_early: bool | None = None, agent_type: AgentKind | str = AgentKind.CLAUDE_CODE, simulation: SimulationConfig | None = None, gate_threshold: float = 1.0, ) -> TaskDefinition: - """Build a minimal resolved-style TaskDefinition for guardrail tests.""" + """Build a minimal resolved-style TaskDefinition for guardrail tests. + + ``stop_early`` is the run-level KILL SWITCH (None = criteria decide, + False = force-disarm, True = the removed master arm, rejected by + ``validate_early_stop``); arming comes from the criteria's own + ``stop_early:`` blocks. + """ return TaskDefinition( task_id="early-stop-test", description="early-stop test task", @@ -153,11 +167,28 @@ def dummy_no_stop_kind() -> Iterator[str]: AgentRegistry._registry.pop(kind, None) +def _block( + *, stop_on_pass: bool = False, stop_on_fail: bool = False, max_steps_to_decide: int | None = None +) -> StopEarlyPolicy | None: + """Compose a stop_early block from trigger intents (None = unarmed). + + stop_on_fail maps to the block's PRESENCE alone (the fail trigger is + implicit in arming) — an empty block is the idiomatic distractor arming. + """ + if not stop_on_pass and not stop_on_fail and max_steps_to_decide is None: + return None + return StopEarlyPolicy( + on_pass="stop" if stop_on_pass else "continue", + decide_within=max_steps_to_decide, + ) + + def _skill_crit( skill_name: str, expected_skill: str, *, - stop_when: str | None = None, + stop_on_pass: bool = False, + stop_on_fail: bool = False, weight: float = 1.0, max_steps_to_decide: int | None = None, pass_threshold: float = 0.9, @@ -167,9 +198,10 @@ def _skill_crit( description=f"{skill_name} activation", skill_name=skill_name, expected_skill=expected_skill, - stop_when=stop_when, # type: ignore[arg-type] + stop_early=_block( + stop_on_pass=stop_on_pass, stop_on_fail=stop_on_fail, max_steps_to_decide=max_steps_to_decide + ), weight=weight, - max_steps_to_decide=max_steps_to_decide, pass_threshold=pass_threshold, ) @@ -179,7 +211,8 @@ def _cmd_crit( min_count: int = 1, max_count: int | None = None, pattern: str | None = "curl", - stop_when: str | None = None, + stop_on_pass: bool = False, + stop_on_fail: bool = False, weight: float = 1.0, max_steps_to_decide: int | None = None, ) -> CommandExecutedCriterion: @@ -190,9 +223,10 @@ def _cmd_crit( command_pattern=pattern, min_count=min_count, max_count=max_count, - stop_when=stop_when, # type: ignore[arg-type] + stop_early=_block( + stop_on_pass=stop_on_pass, stop_on_fail=stop_on_fail, max_steps_to_decide=max_steps_to_decide + ), weight=weight, - max_steps_to_decide=max_steps_to_decide, ) @@ -276,22 +310,31 @@ def _unresolved_skill_end(skill: str, *, tool_id: str = "orphan-1") -> ToolEndEv class TestConfigSurface: - def test_stop_early_defaults_false(self) -> None: - assert RunLimits().stop_early is False - - def test_stop_early_settable(self) -> None: + def test_stop_early_defaults_none(self) -> None: + # None = no run-level opinion: the criteria's own blocks decide arming. + assert RunLimits().stop_early is None + + def test_stop_early_kill_switch_settable(self) -> None: + assert RunLimits(stop_early=False).stop_early is False + + def test_stop_early_true_constructs_at_the_model_level(self) -> None: + # True (the removed master arm) is NOT rejected by RunLimits itself — + # RunLimits is field-merged across 5 layers, so the hard rejection + # lives on the whole-task surface (validate_early_stop), where it gets + # the exit-code-flipping EarlyStopConfigError treatment. See + # TestValidateEarlyStop. assert RunLimits(stop_early=True).stop_early is True def test_gate_threshold_out_of_bounds_rejected(self) -> None: with pytest.raises(ValueError, match="less than or equal to 1"): - RunLimits(stop_early=True, stop_early_gate_threshold=1.5) + RunLimits(stop_early_gate_threshold=1.5) with pytest.raises(ValueError, match="greater than or equal to 0"): - RunLimits(stop_early=True, stop_early_gate_threshold=-0.1) + RunLimits(stop_early_gate_threshold=-0.1) - def test_gate_threshold_nondefault_without_stop_early_allowed(self) -> None: + def test_gate_threshold_nondefault_with_kill_switch_allowed(self) -> None: # A non-default threshold with stop_early=False is inert, not # rejected: RunLimits is field-merged across 5 layers, so a variant - # that flips only stop_early: false must be able to legitimately + # that only throws the kill switch must be able to legitimately # inherit a threshold value set on a sibling layer (e.g. the # early-stop-ab e2e variant) without that being a resolution error. limits = RunLimits(stop_early=False, stop_early_gate_threshold=0.7) @@ -303,47 +346,102 @@ def test_gate_threshold_zero_constructs_at_the_model_level(self) -> None: # a model-level validator can't distinguish it from a value merged # forward from a sibling layer. See TestValidateEarlyStop for the # actual hard-stop rejection. - limits = RunLimits(stop_early=True, stop_early_gate_threshold=0.0) + limits = RunLimits(stop_early_gate_threshold=0.0) assert limits.stop_early_gate_threshold == 0.0 def test_gate_threshold_default_is_valid_either_way(self) -> None: assert RunLimits(stop_early=False).stop_early_gate_threshold == 1.0 - assert RunLimits(stop_early=True).stop_early_gate_threshold == 1.0 + assert RunLimits().stop_early_gate_threshold == 1.0 - def test_stop_when_defaults_none(self) -> None: - assert _skill_crit("s", "s").stop_when is None + @pytest.mark.parametrize("bad", [float("inf"), float("-inf"), float("nan")]) + def test_weight_rejects_non_finite(self, bad: float) -> None: + # weight: .inf would satisfy ge=0.0 and then poison every weighted-gate + # sum (inf/inf = nan compares False against any threshold), so + # non-finite weights are rejected at the model layer. + with pytest.raises(ValueError, match="finite"): + _skill_crit("s", "s", weight=bad) - @pytest.mark.parametrize("value", ["pass", "fail", "decided", "auto"]) - def test_stop_when_accepts_valid_polarities(self, value: str) -> None: - assert _skill_crit("s", "s", stop_when=value).stop_when == value + def test_block_absent_by_default(self) -> None: + crit = _skill_crit("s", "s") + assert crit.stop_early is None + assert crit.is_stop_armed is False - def test_stop_when_rejects_invalid_polarity(self) -> None: - with pytest.raises(ValueError): - _skill_crit("s", "s", stop_when="maybe") + @pytest.mark.parametrize( + ("kwargs", "expected"), + [ + ({"stop_on_pass": True}, True), + ({"stop_on_fail": True}, True), + ({"max_steps_to_decide": 5}, True), + ({}, False), + ], + ) + def test_any_trigger_arms(self, kwargs: dict[str, Any], expected: bool) -> None: + assert _skill_crit("s", "s", **kwargs).is_stop_armed is expected - def test_stop_when_auto_roundtrips(self) -> None: - # The new `auto` value survives model_dump -> model_validate with its - # model_fields_set intact (Pydantic round-trip integrity). - crit = _skill_crit("s", "s", stop_when="auto") - restored = SkillTriggeredCriterion.model_validate_json(crit.model_dump_json()) - assert restored.stop_when == "auto" - assert "stop_when" in restored.model_fields_set + def test_block_unrepresentable_on_unobservable_criterion(self) -> None: + # The early_stop block lives on LiveSuccessCriterion only, so arming an + # unobservable criterion is a schema error (extra='forbid'), not a + # runtime validation case. + with pytest.raises(ValueError): + FileExistsCriterion( + type="file_exists", + path="x", + description="d", + stop_early=StopEarlyPolicy(), # type: ignore[call-arg] + ) - def test_max_steps_to_decide_defaults_none(self) -> None: - assert _skill_crit("s", "s", stop_when="pass").max_steps_to_decide is None + def test_empty_block_arms(self) -> None: + # The idiomatic distractor arming: presence alone (implicit fail trigger). + crit = SkillTriggeredCriterion( + type="skill_triggered", + description="d", + skill_name="wrong", + expected_skill="s", + stop_early=StopEarlyPolicy(), + ) + assert crit.is_stop_armed is True + assert crit.stop_early is not None + assert crit.stop_early.on_pass == "continue" + assert crit.stop_early.decide_within is None - def test_max_steps_to_decide_requires_stop_when(self) -> None: - with pytest.raises(ValueError, match="max_steps_to_decide requires stop_when"): - _skill_crit("s", "s", max_steps_to_decide=5) + def test_block_rejects_unknown_keys(self) -> None: + with pytest.raises(ValueError): + StopEarlyPolicy(on_fail="stop") # type: ignore[call-arg] - def test_max_steps_to_decide_allowed_with_stop_when(self) -> None: - crit = _skill_crit("s", "s", stop_when="pass", max_steps_to_decide=5) - assert crit.max_steps_to_decide == 5 + def test_decide_within_bounds(self) -> None: + with pytest.raises(ValueError): + StopEarlyPolicy(decide_within=0) - def test_max_steps_to_decide_requires_stop_when_command_executed(self) -> None: - # The other LiveSuccessCriterion subclass — same validator, same error. - with pytest.raises(ValueError, match="max_steps_to_decide requires stop_when"): - _cmd_crit(max_steps_to_decide=5) + def test_block_roundtrips(self) -> None: + # The block survives model_dump -> model_validate (round-trip integrity). + crit = _skill_crit("s", "s", stop_on_pass=True, max_steps_to_decide=5) + restored = SkillTriggeredCriterion.model_validate_json(crit.model_dump_json()) + assert restored.stop_early is not None + assert restored.stop_early.on_pass == "stop" + assert restored.stop_early.decide_within == 5 + assert restored.is_stop_armed is True + + def test_decide_within_defaults_none(self) -> None: + crit = _skill_crit("s", "s", stop_on_pass=True) + assert crit.stop_early is not None + assert crit.stop_early.decide_within is None + + def test_decide_within_arms_via_block(self) -> None: + # A timeout-only block: on_pass stays continue; the block's presence arms. + crit = _skill_crit("s", "s", max_steps_to_decide=5) + assert crit.is_stop_armed is True + assert crit.stop_early is not None + assert crit.stop_early.on_pass == "continue" + + def test_decide_within_combines_with_on_pass_stop(self) -> None: + crit = _skill_crit("s", "s", stop_on_pass=True, max_steps_to_decide=5) + assert crit.stop_early is not None + assert crit.stop_early.decide_within == 5 + assert crit.is_stop_armed is True + + def test_block_arms_command_executed(self) -> None: + # The other LiveSuccessCriterion subclass — same block field. + assert _cmd_crit(max_steps_to_decide=5).is_stop_armed is True # --------------------------------------------------------------------------- # @@ -550,159 +648,207 @@ def test_skill_triggered_decidable_is_subset_of_type_universe(self) -> None: class TestValidateEarlyStop: def test_unarmed_is_noop_even_with_bad_shape(self) -> None: - # stop_early=False → validator never inspects anything. - task = _task(criteria=[_skill_crit("s", "s")], stop_early=False, agent_type=AgentKind.CODEX) + # No blocks anywhere → validator never inspects anything. + task = _task(criteria=[_skill_crit("s", "s")], agent_type=AgentKind.CODEX) validate_early_stop(task) # no raise - def test_unarmed_with_stop_when_is_inert(self) -> None: - # A criterion may declare stop_when; without stop_early it stays inert. - task = _task(criteria=[_skill_crit("s", "s", stop_when="decided")], stop_early=False) + def test_master_arm_true_rejected(self) -> None: + # run_limits.stop_early: true is the REMOVED master arm — a hard error + # (not a silent no-op) whether or not any criterion carries a block. + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], stop_early=True) + with pytest.raises(EarlyStopConfigError, match="has been removed"): + validate_early_stop(task) + + def test_master_arm_true_rejected_even_unarmed(self) -> None: + task = _task(criteria=[_skill_crit("s", "s")], stop_early=True) + with pytest.raises(EarlyStopConfigError, match="has been removed"): + validate_early_stop(task) + + def test_kill_switch_disarms_armed_criteria(self) -> None: + # stop_early=False force-disarms every block — validator no-ops. + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True, stop_on_fail=True)], stop_early=False) + validate_early_stop(task) # no raise + assert early_stop_active(task) is False + + def test_kill_switch_skips_all_guards(self) -> None: + # A disarmed run needs NO guard: armed blocks + simulation would be + # rejected, but the kill switch makes the combination legitimately + # runnable (this is exactly the e2e experiment-variant escape hatch). + sim = SimulationConfig(enabled=True, persona="user", goal="g") + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], stop_early=False, simulation=sim) validate_early_stop(task) # no raise def test_armed_happy_path_accepts(self) -> None: - # A positive skill_triggered decides only "pass", so arm it with pass. - task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True) + # A positive skill_triggered decides only "pass"; its block alone arms + # the task — no run-level switch involved. + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)]) validate_early_stop(task) # no raise + assert early_stop_active(task) is True def test_gate_threshold_zero_rejected(self) -> None: # This is the hard-stop rejection for a degenerate threshold — moved # here (not a RunLimits model validator) so it flips the plan exit # code / aborts run like every other early-stop guardrail. - task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True, gate_threshold=0.0) + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], gate_threshold=0.0) with pytest.raises(EarlyStopConfigError, match=r"must be > 0\.0"): validate_early_stop(task) + def test_gate_threshold_zero_unarmed_accepted(self) -> None: + # The degeneracy only matters on an armed task; unarmed, the threshold + # is inert and must not block resolution. + task = _task(criteria=[_skill_crit("s", "s")], gate_threshold=0.0) + validate_early_stop(task) # no raise + def test_gate_threshold_positive_accepted(self) -> None: - task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True, gate_threshold=0.7) + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], gate_threshold=0.7) validate_early_stop(task) # no raise - def test_max_steps_to_decide_rejected_for_fail_only_criterion(self) -> None: - # A distractor (fail-only decidable) with a decision-step budget would - # force-fail a clean run whose "undecided" is its success state. + def test_max_steps_to_decide_inert_on_fail_only_criterion(self) -> None: + # A distractor (fail-only decidable) with a timeout is ACCEPTED — the + # timeout is inert on it (its "undecided" is its success state). This + # tolerance is what lets one dataset-fanned YAML line carry a timeout + # for both positive rows (applies) and distractor rows (ignored). The + # runtime inertness itself is asserted in TestEarlyStopWatcher. task = _task( - criteria=[_skill_crit("weather-teller", "date-teller", stop_when="fail", max_steps_to_decide=3)], - stop_early=True, + criteria=[_skill_crit("weather-teller", "date-teller", stop_on_fail=True, max_steps_to_decide=3)], ) - with pytest.raises(EarlyStopConfigError, match="fail-only-decidable"): - validate_early_stop(task) + validate_early_stop(task) # no raise - def test_max_steps_to_decide_rejected_for_fail_only_command_executed(self) -> None: + def test_max_steps_to_decide_inert_on_fail_only_command_executed(self) -> None: # The "must-NOT-run" shape (min_count=0, max_count=0) is fail-only - # decidable too — same rejection. + # decidable too — same inert-by-design tolerance. task = _task( - criteria=[_cmd_crit(min_count=0, max_count=0, stop_when="fail", max_steps_to_decide=3)], - stop_early=True, + criteria=[_cmd_crit(min_count=0, max_count=0, stop_on_fail=True, max_steps_to_decide=3)], ) - with pytest.raises(EarlyStopConfigError, match="fail-only-decidable"): - validate_early_stop(task) + validate_early_stop(task) # no raise def test_max_steps_to_decide_accepted_for_pass_decidable_criterion(self) -> None: task = _task( - criteria=[_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=3)], - stop_early=True, + criteria=[_skill_crit("date-teller", "date-teller", stop_on_pass=True, max_steps_to_decide=3)], ) validate_early_stop(task) # no raise + def test_max_steps_to_decide_alone_arms_the_task(self) -> None: + # The timeout is an arming trigger in its own right — a task whose only + # trigger is a timeout passes the at-least-one-armed guard. + task = _task(criteria=[_skill_crit("s", "s", max_steps_to_decide=5)]) + validate_early_stop(task) # no raise + def test_armed_distractor_fail_accepts(self) -> None: # A distractor (skill_name != expected_skill) decides only "fail". - task = _task(criteria=[_skill_crit("wrong", "s", stop_when="fail")], stop_early=True) + task = _task(criteria=[_skill_crit("wrong", "s", stop_on_fail=True)]) validate_early_stop(task) # no raise - def test_skill_triggered_positive_fail_arm_rejected(self) -> None: - # A positive criterion can never live-fail; arming it with fail is a dead arm. - task = _task(criteria=[_skill_crit("s", "s", stop_when="fail")], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="cannot decide polarity"): - validate_early_stop(task) + def test_skill_triggered_positive_fail_trigger_inert_but_accepted(self) -> None: + # A positive criterion can never live-fail; stop_on_fail is inert on it, + # not an error — the fanning idiom (same triggers on every dataset row) + # depends on this tolerance. + task = _task(criteria=[_skill_crit("s", "s", stop_on_fail=True)]) + validate_early_stop(task) # no raise - def test_skill_triggered_decided_arm_rejected(self) -> None: - # A single skill_triggered instance decides only one polarity, so - # stop_when=decided (which needs both) can never be honored. - task = _task(criteria=[_skill_crit("s", "s", stop_when="decided")], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="cannot decide polarity"): - validate_early_stop(task) + def test_skill_triggered_both_triggers_accepted(self) -> None: + # Both triggers on one instance: whichever polarity the instance can + # decide is live, the other is inert — valid on any row role. + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True, stop_on_fail=True)]) + validate_early_stop(task) # no raise def test_armed_command_executed_accepts(self) -> None: # A decidable fail arm: must-NOT-run (max_count set) can live-fail. - task = _task(criteria=[_cmd_crit(stop_when="fail", min_count=0, max_count=0)], stop_early=True) + task = _task(criteria=[_cmd_crit(stop_on_fail=True, min_count=0, max_count=0)]) validate_early_stop(task) # no raise def test_armed_command_executed_pass_accepts(self) -> None: # A decidable pass arm: min_count>0 with no upper bound can live-pass. - task = _task(criteria=[_cmd_crit(stop_when="pass", min_count=1, max_count=None)], stop_early=True) + task = _task(criteria=[_cmd_crit(stop_on_pass=True, min_count=1, max_count=None)]) validate_early_stop(task) # no raise - def test_dead_arm_pass_with_max_count_rejected(self) -> None: - # stop_when=pass but max_count is set → live pass can never fire. - task = _task(criteria=[_cmd_crit(stop_when="pass", min_count=1, max_count=3)], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="cannot decide polarity"): - validate_early_stop(task) - - def test_dead_arm_fail_without_max_count_rejected(self) -> None: - # stop_when=fail but max_count is None → live fail can never fire. - task = _task(criteria=[_cmd_crit(stop_when="fail", min_count=1, max_count=None)], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="cannot decide polarity"): - validate_early_stop(task) + def test_inert_pass_trigger_with_max_count_accepted(self) -> None: + # stop_on_pass with max_count set: live pass can never fire, so the + # trigger is inert — accepted by design (fanning tolerance). The + # criterion is still armed (gates via the armed subset). + task = _task(criteria=[_cmd_crit(stop_on_pass=True, min_count=1, max_count=3)]) + validate_early_stop(task) # no raise - def test_dead_arm_zero_min_no_max_rejected(self) -> None: - # min_count=0, max_count=None → neither polarity can ever fire (empty set). - task = _task(criteria=[_cmd_crit(stop_when="decided", min_count=0, max_count=None)], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="cannot decide polarity"): - validate_early_stop(task) + def test_inert_fail_trigger_without_max_count_accepted(self) -> None: + # stop_on_fail with max_count unset: live fail can never fire — inert. + task = _task(criteria=[_cmd_crit(stop_on_fail=True, min_count=1, max_count=None)]) + validate_early_stop(task) # no raise - def test_dead_arm_decided_with_max_count_rejected(self) -> None: - # stop_when=decided needs BOTH polarities; max_count set gives only fail. - task = _task(criteria=[_cmd_crit(stop_when="decided", min_count=1, max_count=3)], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="cannot decide polarity"): - validate_early_stop(task) + def test_all_triggers_inert_still_accepted(self) -> None: + # min_count=0, max_count=None decides NEITHER polarity — every trigger + # is inert, the run can never stop early, and the watcher just logs a + # debug breadcrumb. Accepted: on a fanned dataset some rows + # legitimately end up with all-inert triggers. + task = _task(criteria=[_cmd_crit(stop_on_pass=True, stop_on_fail=True, min_count=0, max_count=None)]) + validate_early_stop(task) # no raise def test_guardrail5_simulation_rejected(self) -> None: sim = SimulationConfig(enabled=True, persona="user", goal="get it done") - task = _task(criteria=[_skill_crit("s", "s", stop_when="decided")], stop_early=True, simulation=sim) + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True, stop_on_fail=True)], simulation=sim) with pytest.raises(EarlyStopConfigError, match="simulation"): validate_early_stop(task) def test_guardrail1_non_supporting_agent_rejected(self, dummy_no_stop_kind: str) -> None: # Codex/antigravity now support the cooperative interrupt, so guardrail 1 # is exercised with a dummy agent that leaves the flag at False. - task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True, agent_type=dummy_no_stop_kind) + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], agent_type=dummy_no_stop_kind) with pytest.raises(EarlyStopConfigError, match="cooperative stopping"): validate_early_stop(task) + def test_guardrail3_agentless_task_rejected(self) -> None: + # An armed task with no agent block at all: the diagnosis must point at + # the missing agent block, not at plugin loading. + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)]).model_copy(update={"agent": None}) + with pytest.raises(EarlyStopConfigError, match="agent block"): + validate_early_stop(task) + + def test_guardrail3_unregistered_agent_type_rejected(self) -> None: + # An armed task whose agent type vanished from the registry (plugin not + # installed/loaded) must fail with the plugin-pointing diagnosis. + kind = "vanishing-agent" + AgentRegistry.register(kind, _DummyNoStopConfig)(_DummyNoStopAgent) + try: + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], agent_type=kind) + finally: + AgentRegistry._registry.pop(kind, None) + with pytest.raises(EarlyStopConfigError, match="not registered"): + validate_early_stop(task) + def test_guardrail1_armed_codex_accepts(self) -> None: - task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True, agent_type=AgentKind.CODEX) + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], agent_type=AgentKind.CODEX) validate_early_stop(task) # no raise def test_guardrail1_armed_antigravity_accepts(self) -> None: - task = _task( - criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True, agent_type=AgentKind.ANTIGRAVITY - ) + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], agent_type=AgentKind.ANTIGRAVITY) validate_early_stop(task) # no raise - def test_guardrail2_no_stop_criterion_rejected(self) -> None: - task = _task(criteria=[_skill_crit("s", "s")], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="at least one stop criterion"): - validate_early_stop(task) - - def test_guardrail3_unobservable_criterion_rejected(self) -> None: - crit = FileExistsCriterion(type="file_exists", path="x.txt", description="x", stop_when="pass") - task = _task(criteria=[crit], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="observable"): - validate_early_stop(task) - - def test_guardrail4_unsupported_polarity_rejected(self, monkeypatch: pytest.MonkeyPatch) -> None: - # Force command_executed to be pass-only, then arm it with stop_when="fail". - init_criteria(validate=False) - monkeypatch.setattr(CommandExecutedCriterion, "live_decidable_polarities", lambda self: frozenset({"pass"})) - task = _task(criteria=[_cmd_crit(stop_when="fail")], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="polarity"): - validate_early_stop(task) + def test_unarmed_task_is_plain_noop(self) -> None: + # No blocks -> no watcher, byte-for-byte default behavior. The old + # "at least one criterion" guard is gone with the master arm: there is + # nothing left to arm a task that has no blocks. + task = _task(criteria=[_skill_crit("s", "s")]) + validate_early_stop(task) # no raise + assert early_stop_active(task) is False + + def test_guardrail3_unobservable_criterion_unrepresentable(self) -> None: + # The stop_early block exists only on LiveSuccessCriterion, so an armed + # unobservable criterion cannot even be constructed (extra='forbid') — + # the old runtime "observable" guard is now a schema property. Match on + # the field name so the rejection is provably about stop_early, not + # some other typo'd kwarg. + with pytest.raises(ValueError, match="stop_early"): + FileExistsCriterion( + type="file_exists", + path="x.txt", + description="x", + stop_early=StopEarlyPolicy(), # type: ignore[call-arg] + ) def test_raise_order_simulation_before_agent(self, dummy_no_stop_kind: str) -> None: # Both simulation AND a non-supporting agent are invalid; simulation reports first. sim = SimulationConfig(enabled=True, persona="user", goal="g") task = _task( - criteria=[_skill_crit("s", "s", stop_when="decided")], - stop_early=True, + criteria=[_skill_crit("s", "s", stop_on_pass=True, stop_on_fail=True)], agent_type=dummy_no_stop_kind, simulation=sim, ) @@ -714,48 +860,40 @@ def test_stacked_activation_criteria_accept(self) -> None: # criterion arms pass, a distractor arms fail. `decided` is invalid for # either because a single instance decides only one polarity. crits = [ - _skill_crit("skill-a", "skill-a", stop_when="pass"), # positive -> pass - _skill_crit("skill-b", "skill-a", stop_when="fail"), # distractor -> fail + _skill_crit("skill-a", "skill-a", stop_on_pass=True), # positive -> pass + _skill_crit("skill-b", "skill-a", stop_on_fail=True), # distractor -> fail ] - task = _task(criteria=crits, stop_early=True) + task = _task(criteria=crits) validate_early_stop(task) # no raise - def test_auto_positive_accepts(self) -> None: - # `auto` on a positive resolves to the pass polarity it can decide. - task = _task(criteria=[_skill_crit("s", "s", stop_when="auto")], stop_early=True) + def test_fanned_positive_row_accepts_both_triggers(self) -> None: + # The fanning idiom: both triggers on every row. On a positive row the + # pass trigger is live and the fail trigger inert. + task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True, stop_on_fail=True)]) validate_early_stop(task) # no raise - def test_auto_distractor_accepts(self) -> None: - # `auto` on a distractor resolves to the fail polarity it can decide. - task = _task(criteria=[_skill_crit("wrong", "s", stop_when="auto")], stop_early=True) + def test_fanned_distractor_row_accepts_both_triggers(self) -> None: + # On a distractor row the fail trigger is live and the pass trigger inert. + task = _task(criteria=[_skill_crit("wrong", "s", stop_on_pass=True, stop_on_fail=True)]) validate_early_stop(task) # no raise - def test_auto_negative_row_distractor_accepts(self) -> None: - # A negative row's criterion (expected_skill == "") is a distractor -> fail. - task = _task(criteria=[_skill_crit("wrong", "", stop_when="auto")], stop_early=True) + def test_fanned_negative_row_accepts_both_triggers(self) -> None: + # A negative row's criterion (expected_skill == "") is a distractor -> fail live. + task = _task(criteria=[_skill_crit("wrong", "", stop_on_pass=True, stop_on_fail=True)]) validate_early_stop(task) # no raise - def test_auto_stacked_activation_accepts(self) -> None: - # The real activation shape: ONE uniform `stop_when: auto` across every - # stacked criterion, which resolves per-instance to pass (the positive) or - # fail (each distractor). This is what a single fanned-out `stop_when` value - # can express and `pass`/`fail`/`decided` cannot, since the role flips per row. + def test_fanned_stacked_activation_accepts(self) -> None: + # The real activation shape: ONE uniform trigger pair across every + # stacked criterion; per-instance decidability makes the right trigger + # live on each (pass on the positive, fail on each distractor). crits = [ - _skill_crit("skill-a", "skill-a", stop_when="auto"), # positive -> pass - _skill_crit("skill-b", "skill-a", stop_when="auto"), # distractor -> fail - _skill_crit("skill-c", "skill-a", stop_when="auto"), # distractor -> fail + _skill_crit("skill-a", "skill-a", stop_on_pass=True, stop_on_fail=True), # positive -> pass live + _skill_crit("skill-b", "skill-a", stop_on_pass=True, stop_on_fail=True), # distractor -> fail live + _skill_crit("skill-c", "skill-a", stop_on_pass=True, stop_on_fail=True), # distractor -> fail live ] - task = _task(criteria=crits, stop_early=True) + task = _task(criteria=crits) validate_early_stop(task) # no raise - def test_auto_dead_arm_rejected(self) -> None: - # `auto` on an instance that can decide NEITHER polarity is a dead arm and - # must be rejected, not silently degrade to a full run. command_executed - # with min_count=0 + max_count=None supports no live polarity. - task = _task(criteria=[_cmd_crit(stop_when="auto", min_count=0, max_count=None)], stop_early=True) - with pytest.raises(EarlyStopConfigError, match="no polarity"): - validate_early_stop(task) - # --------------------------------------------------------------------------- # # Guardrail integration: the plan and run resolution surfaces actually invoke @@ -764,11 +902,11 @@ def test_auto_dead_arm_rejected(self) -> None: # CLI-level error on BOTH surfaces, never a silent no-op. # --------------------------------------------------------------------------- # -_ARMED_UNOBSERVABLE_CRITERION = """\ - - type: file_exists - description: out exists - path: out.txt - stop_when: pass +_UNARMED_CRITERION = """\ + - type: skill_triggered + description: date-teller activation + skill_name: date-teller + expected_skill: date-teller """ _ARMED_OBSERVABLE_CRITERION = """\ @@ -776,12 +914,14 @@ def test_auto_dead_arm_rejected(self) -> None: description: date-teller activation skill_name: date-teller expected_skill: date-teller - stop_when: pass + stop_early: + on_pass: stop """ -def _write_task_yaml(tmp_path: Path, *, criterion_yaml: str, stop_early: bool) -> Path: +def _write_task_yaml(tmp_path: Path, *, criterion_yaml: str, stop_early: bool | None = None) -> Path: task_file = tmp_path / "es_task.yaml" + stop_early_line = "" if stop_early is None else f" stop_early: {str(stop_early).lower()}\n" task_file.write_text( "task_id: es-guardrail-task\n" + "description: early-stop guardrail surface test\n" @@ -792,7 +932,7 @@ def _write_task_yaml(tmp_path: Path, *, criterion_yaml: str, stop_early: bool) - + " driver: tempdir\n" + "run_limits:\n" + " max_turns: 20\n" - + f" stop_early: {str(stop_early).lower()}\n" + + stop_early_line + "success_criteria:\n" + criterion_yaml ) @@ -813,36 +953,51 @@ def _resolve_surface(task_file: Path, tmp_path: Path, *, overrides: dict[str, An class TestGuardrailResolutionSurfaces: """A bad arming is rejected by the real plan/run wiring, not only the helper.""" - def test_run_surface_rejects_bad_armed_task(self, tmp_path: Path) -> None: - # The armed unobservable criterion propagates out of resolve_all_tasks as - # EarlyStopConfigError (a ValueError, so the run CLI converts it to a - # clean BadParameter) instead of being demoted to a skipped task. - task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_UNOBSERVABLE_CRITERION, stop_early=True) - with pytest.raises(EarlyStopConfigError, match="observable"): + def test_run_surface_rejects_master_arm(self, tmp_path: Path) -> None: + # run_limits.stop_early: true (the removed master arm) propagates out + # of resolve_all_tasks as EarlyStopConfigError (a ValueError, so the + # run CLI converts it to a clean BadParameter) instead of being + # demoted to a skipped task. (An armed UNOBSERVABLE criterion no + # longer reaches this validator at all — the block exists only on + # LiveSuccessCriterion, so it is a pydantic schema error at load.) + task_file = _write_task_yaml(tmp_path, criterion_yaml=_UNARMED_CRITERION, stop_early=True) + with pytest.raises(EarlyStopConfigError, match="has been removed"): _resolve_surface(task_file, tmp_path) def test_run_surface_accepts_valid_armed_task(self, tmp_path: Path) -> None: - task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_OBSERVABLE_CRITERION, stop_early=True) + # The criterion's block alone arms — no run_limits.stop_early line. + task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_OBSERVABLE_CRITERION) resolved, skipped = _resolve_surface(task_file, tmp_path) assert not skipped assert len(resolved) == 1 - limits = resolved[0].task.run_limits - assert limits is not None and limits.stop_early is True - - def test_run_surface_validates_cli_override_arming(self, tmp_path: Path) -> None: - # The YAML alone is inert (stop_when set, stop_early false) and must be - # accepted; arming via the layer-5 -D override must then be validated, - # proving the guardrails run AFTER _apply_cli_overrides. - task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_UNOBSERVABLE_CRITERION, stop_early=False) + task = resolved[0].task + assert early_stop_active(task) is True + limits = task.run_limits + assert limits is not None and limits.stop_early is None + + def test_run_surface_validates_cli_override_master_arm(self, tmp_path: Path) -> None: + # The YAML alone is inert (no blocks, no stop_early line) and must be + # accepted; the removed master arm smuggled in via the layer-5 -D + # override must then be rejected, proving the guardrails run AFTER + # _apply_cli_overrides. + task_file = _write_task_yaml(tmp_path, criterion_yaml=_UNARMED_CRITERION) resolved, _ = _resolve_surface(task_file, tmp_path) assert len(resolved) == 1 # inert without the override - with pytest.raises(EarlyStopConfigError, match="observable"): + with pytest.raises(EarlyStopConfigError, match="has been removed"): _resolve_surface(task_file, tmp_path, overrides={"run_limits.stop_early": True}) + def test_run_surface_cli_kill_switch_disarms(self, tmp_path: Path) -> None: + # -D run_limits.stop_early=false force-disarms an armed task file — + # the one-line authoritative-run override. + task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_OBSERVABLE_CRITERION) + resolved, _ = _resolve_surface(task_file, tmp_path, overrides={"run_limits.stop_early": False}) + assert len(resolved) == 1 + assert early_stop_active(resolved[0].task) is False + def test_run_surface_variant_inherits_task_threshold_with_stop_early_false(self, tmp_path: Path) -> None: - # Mirrors the shipped early-stop-ab experiment: a task sets both - # stop_early: true and a non-default stop_early_gate_threshold; a - # variant flips ONLY stop_early to false (field-merged, so it + # Mirrors the shipped early-stop-ab experiment: a task arms via its + # criterion block and sets a non-default stop_early_gate_threshold; the + # e2e variant throws ONLY the kill switch (field-merged, so it # inherits the task's threshold). This must resolve cleanly — the # inherited-but-inert threshold is not a misconfiguration. task_file = tmp_path / "es_layered_task.yaml" @@ -856,14 +1011,13 @@ def test_run_surface_variant_inherits_task_threshold_with_stop_early_false(self, + " driver: tempdir\n" + "run_limits:\n" + " max_turns: 20\n" - + " stop_early: true\n" + " stop_early_gate_threshold: 0.7\n" + "success_criteria:\n" + _ARMED_OBSERVABLE_CRITERION ) variants = [ ExperimentVariant(variant_id="e2e", run_limits=RunLimits(stop_early=False)), - ExperimentVariant(variant_id="smoke", run_limits=RunLimits(stop_early=True)), + ExperimentVariant(variant_id="smoke"), ] resolved, skipped = resolve_all_tasks( task_files=[task_file], @@ -875,10 +1029,13 @@ def test_run_surface_variant_inherits_task_threshold_with_stop_early_false(self, ) assert not skipped assert len(resolved) == 2 - by_variant = {r.variant_id: r.task.run_limits for r in resolved} - assert by_variant["e2e"] is not None and by_variant["e2e"].stop_early is False - assert by_variant["e2e"].stop_early_gate_threshold == 0.7 # inherited, inert - assert by_variant["smoke"] is not None and by_variant["smoke"].stop_early is True + by_variant = {r.variant_id: r.task for r in resolved} + e2e_limits = by_variant["e2e"].run_limits + assert e2e_limits is not None and e2e_limits.stop_early is False + assert e2e_limits.stop_early_gate_threshold == 0.7 # inherited, inert + assert early_stop_active(by_variant["e2e"]) is False + # The smoke variant needs no override at all: the task's block arms it. + assert early_stop_active(by_variant["smoke"]) is True def _run_plan(self, task_file: Path, exp_dir: Path) -> tuple[str, int]: """Invoke the real plan_command against a minimal single-variant experiment. @@ -904,20 +1061,43 @@ def _run_plan(self, task_file: Path, exp_dir: Path) -> tuple[str, int]: printed = " ".join(str(call) for call in mock_console.print.call_args_list) return printed, exit_code - def test_plan_surface_flips_exit_code_on_bad_armed_task(self, tmp_path: Path) -> None: - task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_UNOBSERVABLE_CRITERION, stop_early=True) + def test_plan_surface_flips_exit_code_on_master_arm(self, tmp_path: Path) -> None: + task_file = _write_task_yaml(tmp_path, criterion_yaml=_UNARMED_CRITERION, stop_early=True) printed, exit_code = self._run_plan(task_file, tmp_path) assert exit_code == 1 assert "early-stop config error" in printed - assert "observable" in printed + assert "has been removed" in printed def test_plan_surface_accepts_valid_armed_task(self, tmp_path: Path) -> None: - task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_OBSERVABLE_CRITERION, stop_early=True) + task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_OBSERVABLE_CRITERION) printed, exit_code = self._run_plan(task_file, tmp_path) assert exit_code == 0 assert "All tasks are valid!" in printed +class TestShippedEarlyStopExperiment: + """The checked-in experiments/early-stop-ab.yaml loads and its two flavors + resolve as documented: the e2e variant's kill switch disarms an armed task, + the smoke variant (no override) stays armed.""" + + def test_early_stop_ab_variants_disarm_and_arm(self, tmp_path: Path) -> None: + experiment = load_experiment(Path("experiments/early-stop-ab.yaml")) + task_file = _write_task_yaml(tmp_path, criterion_yaml=_ARMED_OBSERVABLE_CRITERION) + resolved, skipped = resolve_all_tasks( + task_files=[task_file], + experiment=experiment, + default_experiment=ExperimentDefinition( + experiment_id="default", variants=[ExperimentVariant(variant_id="default")] + ), + config=BatchRunConfig(run_dir=tmp_path / "runs", overrides={}), + ) + assert not skipped + by_variant = {r.variant_id: r.task for r in resolved} + assert set(by_variant) == {"e2e", "smoke"} + assert early_stop_active(by_variant["e2e"]) is False # kill switch force-disarms + assert early_stop_active(by_variant["smoke"]) is True # the task's block alone arms + + # --------------------------------------------------------------------------- # # Cooperative should_stop seam on ClaudeCodeAgent — still UNWIRED: the # orchestrator does not pass should_stop yet, so these drive the agent directly. @@ -1053,8 +1233,7 @@ def test_fixture_resolves_without_error(self, task_file: Path, tmp_path: Path) - resolved, skipped = _resolve_surface(task_file, tmp_path) assert not skipped assert len(resolved) == 1 - limits = resolved[0].task.run_limits - assert limits is not None and limits.stop_early is True + assert early_stop_active(resolved[0].task) is True class TestCooperativeStopSeam: @@ -1164,7 +1343,7 @@ def test_evaluation_result_roundtrip_with_early_stop(self) -> None: def test_armed_criteria_passed_gates_armed_only(self) -> None: # Armed skill passes; advisory file_exists fails. armed gate -> True. criteria = [ - _skill_crit("date-teller", "date-teller", stop_when="pass"), + _skill_crit("date-teller", "date-teller", stop_on_pass=True), FileExistsCriterion(path="x", description="x must exist"), ] result = _result(criteria_results=[_crit_result("skill_triggered", 1.0), _crit_result("file_exists", 0.0)]) @@ -1174,7 +1353,7 @@ def test_armed_criteria_passed_gates_armed_only(self) -> None: def test_armed_criteria_passed_fails_when_armed_fails(self) -> None: criteria = [ - _skill_crit("date-teller", "date-teller", stop_when="pass"), + _skill_crit("date-teller", "date-teller", stop_on_pass=True), FileExistsCriterion(path="x", description="x must exist"), ] result = _result(criteria_results=[_crit_result("skill_triggered", 0.0), _crit_result("file_exists", 1.0)]) @@ -1186,13 +1365,34 @@ def test_armed_criteria_passed_raises_on_empty_armed(self) -> None: with pytest.raises(ValueError, match="no armed criteria"): result.armed_criteria_passed(criteria) + def test_armed_criteria_passed_raises_on_length_mismatch(self) -> None: + # The gate shares all_criteria_passed's length pre-check: a + # results/criteria mismatch raises rather than silently truncating. + criteria = [ + _skill_crit("date-teller", "date-teller", stop_on_pass=True), + FileExistsCriterion(path="x", description="x must exist"), + ] + result = _result(criteria_results=[_crit_result("skill_triggered", 1.0)]) + with pytest.raises(ValueError, match="length mismatch"): + result.armed_criteria_passed(criteria) + + def test_armed_criteria_passed_zero_total_weight_fails_closed(self) -> None: + # weight=0 + a stop trigger is rejected at the model layer, so this is + # reachable only by bypassing validation — which is exactly the case + # the guard exists for: a pass/fail gate must fail CLOSED, not + # trivially pass, on a degenerate zero-weight armed set. + crit = _skill_crit("date-teller", "date-teller", stop_on_pass=True) + crit.weight = 0.0 # bypass model validation deliberately + result = _result(criteria_results=[_crit_result("skill_triggered", 1.0)]) + assert result.armed_criteria_passed([crit]) is False + def test_armed_criteria_passed_default_threshold_still_requires_all(self) -> None: # gate_threshold=1.0 (the default) must reproduce the old all()-must-pass # rule exactly: one armed criterion at 0.0 fails the gate regardless of # the other armed criterion's weight. criteria = [ - _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.8), - _skill_crit("weather-teller", "date-teller", stop_when="fail", weight=0.2), + _skill_crit("date-teller", "date-teller", stop_on_pass=True, weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_on_fail=True, weight=0.2), ] low_weight_fails = _result( criteria_results=[_crit_result("skill_triggered", 1.0), _crit_result("skill_triggered", 0.0)] @@ -1212,8 +1412,8 @@ def test_armed_criteria_passed_low_weight_failure_absorbed_below_threshold(self) # LOW-weight criterion failing (weighted score 0.8) still clears 0.7; # the HIGH-weight one failing (weighted score 0.2) does not. criteria = [ - _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.8), - _skill_crit("weather-teller", "date-teller", stop_when="fail", weight=0.2), + _skill_crit("date-teller", "date-teller", stop_on_pass=True, weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_on_fail=True, weight=0.2), ] low_weight_fails = _result( criteria_results=[_crit_result("skill_triggered", 1.0), _crit_result("skill_triggered", 0.0)] @@ -1231,11 +1431,11 @@ def test_armed_criteria_passed_still_honors_pass_threshold(self) -> None: # score of 0.5 fails a pass_threshold of 0.99, so it must NOT clear # even a low gate_threshold: pass_threshold is not bypassable by # lowering gate_threshold. - criteria = [_skill_crit("date-teller", "date-teller", stop_when="pass", pass_threshold=0.99)] + criteria = [_skill_crit("date-teller", "date-teller", stop_on_pass=True, pass_threshold=0.99)] result = _result(criteria_results=[_crit_result("skill_triggered", 0.5)]) assert result.armed_criteria_passed(criteria, gate_threshold=0.1) is False # A score that DOES clear its own pass_threshold (0.5 >= 0.4) passes. - criteria_lenient = [_skill_crit("date-teller", "date-teller", stop_when="pass", pass_threshold=0.4)] + criteria_lenient = [_skill_crit("date-teller", "date-teller", stop_on_pass=True, pass_threshold=0.4)] assert result.armed_criteria_passed(criteria_lenient, gate_threshold=0.1) is True def test_armed_criteria_passed_gate_equivalence_at_default_threshold(self) -> None: @@ -1250,7 +1450,7 @@ def test_armed_criteria_passed_gate_equivalence_at_default_threshold(self) -> No (0.5, 0.4, 0.2), # clears its own threshold despite a low score (0.0, 0.0, 1.0), # pass_threshold: 0.0 — the non-gating-arming escape hatch ]: - criteria = [_skill_crit("s", "s", stop_when="pass", weight=weight, pass_threshold=pass_threshold)] + criteria = [_skill_crit("s", "s", stop_on_pass=True, weight=weight, pass_threshold=pass_threshold)] result = _result(criteria_results=[_crit_result("skill_triggered", score)]) expected = score >= pass_threshold assert result.armed_criteria_passed(criteria) is expected, (score, pass_threshold, weight) @@ -1259,8 +1459,8 @@ def test_armed_criteria_passed_weighted_gate_with_command_executed(self) -> None # The other LiveSuccessCriterion subclass exercised through the same # weighted gate — command_executed, not just skill_triggered. criteria = [ - _cmd_crit(min_count=1, max_count=None, stop_when="pass", weight=0.8), - _cmd_crit(min_count=0, max_count=0, stop_when="fail", weight=0.2), + _cmd_crit(min_count=1, max_count=None, stop_on_pass=True, weight=0.8), + _cmd_crit(min_count=0, max_count=0, stop_on_fail=True, weight=0.2), ] low_weight_fails = _result( criteria_results=[_crit_result("command_executed", 1.0), _crit_result("command_executed", 0.0)] @@ -1278,7 +1478,7 @@ def test_armed_criteria_passed_weighted_gate_with_command_executed(self) -> None def _watcher(criteria: list[Any], *, max_turns: int | None = 20, gate_threshold: float = 1.0) -> EarlyStopWatcher: - task = _task(criteria=criteria, stop_early=True) + task = _task(criteria=criteria) assert task.run_limits is not None task.run_limits.max_turns = max_turns task.run_limits.stop_early_gate_threshold = gate_threshold @@ -1294,7 +1494,7 @@ class TestEarlyStopWatcher: def test_for_task_arms_only_stop_criteria(self) -> None: watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="pass"), + _skill_crit("date-teller", "date-teller", stop_on_pass=True), FileExistsCriterion(path="x", description="x must exist"), ] ) @@ -1302,13 +1502,13 @@ def test_for_task_arms_only_stop_criteria(self) -> None: assert len(watcher._armed) == 1 def test_undecided_before_engagement_no_stop(self) -> None: - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, [_agent_start(), _turn_start()]) assert watcher.should_stop() is False assert watcher.info is None def test_pass_stop_fires_on_expected_skill(self) -> None: - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, _skill_events("date-teller")) assert watcher.should_stop() is True assert watcher.info is not None @@ -1317,7 +1517,7 @@ def test_pass_stop_fires_on_expected_skill(self) -> None: def test_fail_stop_fires_on_distractor_skill(self) -> None: # A distractor criterion (its skill != the expected skill) fail-stops the # instant its skill is engaged — the per-skill precision signal. - watcher = _watcher([_skill_crit("weather-teller", "date-teller", stop_when="fail")]) + watcher = _watcher([_skill_crit("weather-teller", "date-teller", stop_on_fail=True)]) _feed(watcher, _skill_events("weather-teller")) assert watcher.should_stop() is True assert watcher.info is not None @@ -1326,7 +1526,7 @@ def test_fail_stop_fires_on_distractor_skill(self) -> None: def test_wrong_skill_does_not_stop_positive_row(self) -> None: # Item 1: a positive row (armed pass) engaging the WRONG skill must NOT # stop — the run keeps going so the expected skill can still load later. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, _skill_events("weather-teller")) assert watcher.should_stop() is False assert watcher.info is None @@ -1337,8 +1537,8 @@ def test_stacked_pass_stop_requires_all(self) -> None: # second (both now passed) fires the pass-stop. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="pass"), - _skill_crit("weather-teller", "weather-teller", stop_when="pass"), + _skill_crit("date-teller", "date-teller", stop_on_pass=True), + _skill_crit("weather-teller", "weather-teller", stop_on_pass=True), ] ) _feed(watcher, _skill_events("date-teller")) @@ -1356,8 +1556,8 @@ def test_stacked_wrong_skill_defers_fail_stop_until_positive_decides(self) -> No # (no pass-armed criterion left undecided) the deferred fail-stop fires. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="pass"), - _skill_crit("weather-teller", "date-teller", stop_when="fail"), + _skill_crit("date-teller", "date-teller", stop_on_pass=True), + _skill_crit("weather-teller", "date-teller", stop_on_fail=True), ] ) _feed(watcher, _skill_events("weather-teller")) @@ -1377,8 +1577,10 @@ def test_fail_stop_precedes_pass_stop_same_round(self) -> None: # before pass-stop, so the round must record CRITERION_FAILED. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="auto"), # positive -> pass - _skill_crit("weather-teller", "date-teller", stop_when="auto"), # distractor -> fail + _skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True), # positive -> pass + _skill_crit( + "weather-teller", "date-teller", stop_on_pass=True, stop_on_fail=True + ), # distractor -> fail ] ) both = _cmd("Bash", {"command": "cat skills/date-teller/SKILL.md skills/weather-teller/SKILL.md"}) @@ -1395,8 +1597,10 @@ def test_auto_positive_row_misfire_alone_never_stops(self) -> None: # never a truncated FN. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="auto"), # positive -> pass - _skill_crit("weather-teller", "date-teller", stop_when="auto"), # distractor -> fail + _skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True), # positive -> pass + _skill_crit( + "weather-teller", "date-teller", stop_on_pass=True, stop_on_fail=True + ), # distractor -> fail ] ) _feed(watcher, _skill_events("weather-teller")) @@ -1406,8 +1610,8 @@ def test_auto_positive_row_misfire_alone_never_stops(self) -> None: def test_auto_positive_pass_stops(self) -> None: # `auto` on a positive resolves to pass-armed: engaging the expected skill - # pass-stops, identically to an explicit stop_when="pass". - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="auto")]) + # pass-stops, identically to an explicit stop_on_pass=True. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True)]) _feed(watcher, _skill_events("date-teller")) assert watcher.should_stop() is True assert watcher.info is not None @@ -1421,9 +1625,11 @@ def test_auto_mixed_pass_stops_ignoring_undecided_distractors(self) -> None: # fire, since a distractor can never live-pass.) watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="auto"), # positive -> pass - _skill_crit("weather-teller", "date-teller", stop_when="auto"), # distractor -> fail - _skill_crit("news-teller", "date-teller", stop_when="auto"), # distractor -> fail + _skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True), # positive -> pass + _skill_crit( + "weather-teller", "date-teller", stop_on_pass=True, stop_on_fail=True + ), # distractor -> fail + _skill_crit("news-teller", "date-teller", stop_on_pass=True, stop_on_fail=True), # distractor -> fail ] ) _feed(watcher, _skill_events("date-teller")) @@ -1440,8 +1646,8 @@ def test_auto_negative_row_no_pass_stop_on_benign_call(self) -> None: # True); the run continues to the cap as intended. watcher = _watcher( [ - _skill_crit("date-teller", "", stop_when="auto"), # distractor -> fail - _skill_crit("weather-teller", "", stop_when="auto"), # distractor -> fail + _skill_crit("date-teller", "", stop_on_pass=True, stop_on_fail=True), # distractor -> fail + _skill_crit("weather-teller", "", stop_on_pass=True, stop_on_fail=True), # distractor -> fail ] ) _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) @@ -1454,8 +1660,8 @@ def test_auto_negative_row_misfire_fail_stops(self) -> None: # never pass-stop. watcher = _watcher( [ - _skill_crit("date-teller", "", stop_when="auto"), # distractor -> fail - _skill_crit("weather-teller", "", stop_when="auto"), # distractor -> fail + _skill_crit("date-teller", "", stop_on_pass=True, stop_on_fail=True), # distractor -> fail + _skill_crit("weather-teller", "", stop_on_pass=True, stop_on_fail=True), # distractor -> fail ] ) _feed(watcher, _skill_events("date-teller")) @@ -1468,8 +1674,8 @@ def test_mixed_static_arming_pass_stops_ignoring_fail_armed(self) -> None: # pass-positive + fail-distractor mix also pass-stops on the positive alone. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="pass"), # pass-armed - _skill_crit("weather-teller", "date-teller", stop_when="fail"), # fail-armed + _skill_crit("date-teller", "date-teller", stop_on_pass=True), # pass-armed + _skill_crit("weather-teller", "date-teller", stop_on_fail=True), # fail-armed ] ) _feed(watcher, _skill_events("date-teller")) @@ -1484,8 +1690,8 @@ def test_ceiling_bound_defers_fail_stop_below_default_gate_threshold(self) -> No # positive comes through, so the run must NOT stop yet. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="auto", weight=0.8), - _skill_crit("weather-teller", "date-teller", stop_when="auto", weight=0.2), + _skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True, weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_on_pass=True, stop_on_fail=True, weight=0.2), ], gate_threshold=0.7, ) @@ -1500,8 +1706,8 @@ def test_ceiling_bound_fires_fail_stop_when_high_weight_criterion_fails(self) -> # even though it's the "small" criterion still undecided. watcher = _watcher( [ - _skill_crit("weather-teller", "date-teller", stop_when="auto", weight=0.8), - _skill_crit("news-teller", "date-teller", stop_when="auto", weight=0.2), + _skill_crit("weather-teller", "date-teller", stop_on_pass=True, stop_on_fail=True, weight=0.8), + _skill_crit("news-teller", "date-teller", stop_on_pass=True, stop_on_fail=True, weight=0.2), ], gate_threshold=0.7, ) @@ -1515,8 +1721,8 @@ def test_default_gate_threshold_fires_fail_stop_on_any_weight(self) -> None: # failure alone must still fire — byte-for-byte the pre-weighting rule. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="auto", weight=0.8), - _skill_crit("weather-teller", "date-teller", stop_when="auto", weight=0.2), + _skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True, weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_on_pass=True, stop_on_fail=True, weight=0.2), ] ) _feed(watcher, _skill_events("weather-teller")) @@ -1534,7 +1740,7 @@ def test_floor_bound_pass_stops_before_low_weight_distractor_decides(self) -> No # pass-armed floor by design either way). watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.9), + _skill_crit("date-teller", "date-teller", stop_on_pass=True, weight=0.9), ], gate_threshold=0.7, ) @@ -1549,8 +1755,8 @@ def test_floor_bound_pass_stop_requires_full_pass_armed_subset_below_default(sel # one's weight share would drop the floor below the threshold. watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.5), - _skill_crit("weather-teller", "weather-teller", stop_when="pass", weight=0.5), + _skill_crit("date-teller", "date-teller", stop_on_pass=True, weight=0.5), + _skill_crit("weather-teller", "weather-teller", stop_on_pass=True, weight=0.5), ], gate_threshold=0.7, ) @@ -1562,7 +1768,7 @@ def test_decision_budget_exceeded_when_still_undecided(self) -> None: # An armed criterion capped at max_steps_to_decide=1 that is still # "undecided" after its first tool call forces a budget-exceeded stop. # Full-field EarlyStopInfo parity, matching every other stop-reason test. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=1)]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True, max_steps_to_decide=1)]) _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) assert watcher.should_stop() is True assert watcher.info is not None @@ -1573,18 +1779,18 @@ def test_decision_budget_exceeded_when_still_undecided(self) -> None: assert watcher.info.tool_call_index == 1 def test_decision_budget_exceeded_names_the_right_criterion_among_several(self) -> None: - # Multiple armed criteria with different budgets: only the SECOND - # one's budget has expired (cap=1, undecided after 1 call); the first - # has a longer budget (cap=5) and is also still undecided. The - # deciding criterion reported must be the one whose budget actually - # tripped, not just the first armed criterion in list order. + # Two armed criteria; the first resolves (pass) on the very call that + # expires the second's budget (cap=1). The deciding criterion reported + # must be the one whose budget actually tripped — not just the first + # armed criterion in list order — and the timeout-driven fail-stop + # wins over the first criterion's pass (fail-stop is evaluated first). watcher = _watcher( [ - _skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=5), - _skill_crit("weather-teller", "weather-teller", stop_when="pass", max_steps_to_decide=1), + _skill_crit("date-teller", "date-teller", stop_on_pass=True, max_steps_to_decide=5), + _skill_crit("weather-teller", "weather-teller", stop_on_pass=True, max_steps_to_decide=1), ] ) - _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) + _feed(watcher, _skill_events("date-teller")) assert watcher.should_stop() is True assert watcher.info is not None assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED @@ -1594,7 +1800,7 @@ def test_decision_budget_exceeded_with_command_executed(self) -> None: # The other LiveSuccessCriterion subclass: a command_executed pass-armed # criterion (min_count=1, no upper bound) capped at max_steps_to_decide=1 # that never sees a matching command force-fails identically. - watcher = _watcher([_cmd_crit(min_count=1, max_count=None, stop_when="pass", max_steps_to_decide=1)]) + watcher = _watcher([_cmd_crit(min_count=1, max_count=None, stop_on_pass=True, max_steps_to_decide=1)]) _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) assert watcher.should_stop() is True assert watcher.info is not None @@ -1604,7 +1810,7 @@ def test_decision_budget_exceeded_with_command_executed(self) -> None: def test_decision_budget_not_exceeded_below_cap(self) -> None: # Same cap, but only reached on the FIRST tool call (index 1) — a cap of # 2 must not fire yet. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=2)]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True, max_steps_to_decide=2)]) _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) assert watcher.should_stop() is False assert watcher.info is None @@ -1612,7 +1818,7 @@ def test_decision_budget_not_exceeded_below_cap(self) -> None: def test_real_decision_within_budget_wins_over_budget_check(self) -> None: # The criterion decides (pass-stops) on the SAME tool call that would # otherwise have tripped its budget — the real decision takes priority. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=1)]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True, max_steps_to_decide=1)]) _feed(watcher, _skill_events("date-teller")) assert watcher.should_stop() is True assert watcher.info is not None @@ -1621,32 +1827,227 @@ def test_real_decision_within_budget_wins_over_budget_check(self) -> None: def test_decision_budget_ignored_when_unset(self) -> None: # No max_steps_to_decide -> no budget check, run continues indefinitely # (up to run_limits.max_turns) while undecided. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) assert watcher.should_stop() is False assert watcher.info is None + def test_timeout_only_arming_pass_within_budget_never_stops(self) -> None: + # THE fail-fast-without-success-stop intent: max_steps_to_decide alone + # (no stop_on_pass). The skill engages on call 1 — well within the + # budget of 3 — so the verdict latches pass and the run continues + # untouched: no pass-stop (not armed for one), and the timeout can + # never fire again (the verdict is no longer undecided). Extra calls + # beyond the budget prove the latch holds. + watcher = _watcher([_skill_crit("date-teller", "date-teller", max_steps_to_decide=3)]) + _feed(watcher, _skill_events("date-teller")) + assert watcher.should_stop() is False + for i in range(4): # sail past the budget — still no stop + watcher.on_event(_tool_end(_cmd("Bash", {"command": f"echo {i}"}))) + assert watcher.should_stop() is False + assert watcher.info is None + + def test_timeout_only_arming_undecided_past_budget_stops(self) -> None: + # The other half of the same intent: not engaged within the budget → + # effective fail → fail-stop (default gate threshold 1.0). + watcher = _watcher([_skill_crit("date-teller", "date-teller", max_steps_to_decide=2)]) + _feed( + watcher, + [ + _agent_start(), + _turn_start(), + _tool_end(_cmd("Bash", {"command": "ls"})), + _tool_end(_cmd("Bash", {"command": "cat x"})), + ], + ) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + + def test_timeout_inert_on_fail_only_distractor(self) -> None: + # A distractor (fail-only decidable) carrying a timeout — the fanned + # line case. Its "undecided" is its success state: sailing past the + # budget with no misfire must NOT stop the run. + watcher = _watcher([_skill_crit("weather-teller", "date-teller", stop_on_fail=True, max_steps_to_decide=1)]) + _feed( + watcher, + [ + _agent_start(), + _turn_start(), + _tool_end(_cmd("Bash", {"command": "ls"})), + _tool_end(_cmd("Bash", {"command": "cat x"})), + ], + ) + assert watcher.should_stop() is False + assert watcher.info is None + + def test_low_weight_timeout_absorbed_below_threshold(self) -> None: + # A timeout is an ORDINARY weighted fail: a low-weight (0.2) criterion + # timing out cannot drop the ceiling (0.8) below a 0.7 gate threshold, + # so the run continues — the timeout is absorbed exactly like a + # low-weight native fail. (The high-weight positive resolves first so + # the deferral is not what's holding the stop.) + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_on_pass=True, weight=0.8), + _skill_crit("todo-lister", "todo-lister", weight=0.2, max_steps_to_decide=1), + ], + gate_threshold=0.7, + ) + _feed(watcher, _skill_events("date-teller")) + # date-teller passed (0.8 locked in); todo-lister timed out (0.2 lost). + # Ceiling = 0.8 >= 0.7 → no fail-stop. Pass-stop floor over the + # stop_on_pass subset = 0.8/0.8 = 1.0 >= 0.7 → pass-stop fires instead. + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + + def test_low_weight_timeout_absorbed_no_pass_stop_continues(self) -> None: + # Same absorption, but with no stop_on_pass anywhere (both criteria + # armed via timeouts only): the low-weight timeout alone cannot doom + # the 0.7 gate — ceiling 0.8/1.0 after the high-weight positive + # latches pass — and nothing else can stop, so the run continues. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", weight=0.8, max_steps_to_decide=50), + _skill_crit("todo-lister", "todo-lister", weight=0.2, max_steps_to_decide=1), + ], + gate_threshold=0.7, + ) + _feed(watcher, _skill_events("date-teller")) + for i in range(3): + watcher.on_event(_tool_end(_cmd("Bash", {"command": f"echo {i}"}))) + assert watcher.should_stop() is False + assert watcher.info is None + + def test_pass_stop_deferred_while_outside_pass_capable_undecided(self) -> None: + # Recall deferral on the PASS side (mixed arming): A (on_pass: stop) + # passes on call 1 while B — pass-capable but armed on_pass: continue + # via decide_within — is still undecided and within budget. Firing the + # pass-stop here would truncate B's expected signal out of the + # trajectory, so the stop is HELD. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_on_pass=True), + _skill_crit("todo-lister", "todo-lister", max_steps_to_decide=5), + ] + ) + _feed(watcher, _skill_events("date-teller")) + assert watcher.should_stop() is False # deferred: todo-lister undecided + assert watcher.info is None + # Once B decides (pass), the on_pass=stop floor (over A alone) still + # holds, so the deferred pass-stop fires on that round. + _feed(watcher, [_tool_end(_skill_cmd("todo-lister", tool_id="td"))]) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + + def test_pass_stop_fires_after_outside_criterion_fails_below_threshold(self) -> None: + # The other resolution of the deferral: B (0.2, decide_within=2) times + # out AFTER A (0.8, on_pass: stop) has passed. Under a 0.7 threshold + # the low-weight fail cannot doom the ceiling (0.8 >= 0.7), so no + # fail-stop — and with B decided, the deferral clears and the floor + # (1.0 over the on_pass=stop subset) fires the pass-stop. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_on_pass=True, weight=0.8), + _skill_crit("todo-lister", "todo-lister", weight=0.2, max_steps_to_decide=2), + ], + gate_threshold=0.7, + ) + _feed(watcher, _skill_events("date-teller")) # call 1: A passes, B undecided (budget 2) + assert watcher.should_stop() is False # deferred while B is in budget + watcher.on_event(_tool_end(_cmd("Bash", {"command": "ls"}))) # call 2: B's budget expires + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + + def test_decision_budget_exceeded_on_in_flight_call(self) -> None: + # The budget expires on the in-flight round: an AgentStart + TurnStart + + # a dispatched ToolStart with NO ToolEnd. The in-flight call reports as + # tool call 1, which meets decide_within=1 — the timeout fail-stop must + # fire on the call itself, before any result resolves. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True, max_steps_to_decide=1)]) + start = ToolStartEvent(task_id="t", tool=_cmd("Bash", {"command": "echo hi"})) + _feed(watcher, [_agent_start(), _turn_start(), start]) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert watcher.info.tool_call_index == 1 + + def test_timeout_fail_deferred_while_sibling_positive_in_budget(self) -> None: + # Criterion B times out (budget 1) while criterion A — pass-capable, + # no budget — is still undecided: the fail-stop is DEFERRED (recall + # protection). It fires the moment A decides. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_on_pass=True), + _skill_crit("todo-lister", "todo-lister", max_steps_to_decide=1), + ] + ) + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "ls"}))]) + assert watcher.should_stop() is False # deferred: date-teller undecided + watcher.on_event(_tool_end(_skill_cmd("date-teller", tool_id="sk-9"))) + # A resolved (pass) → deferral clears → B's latched timeout fail fires + # (ceiling 0.5 < 1.0). Fail-stop precedes pass-stop in the same round. + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + + def test_verdicts_latch_and_are_not_repolled(self) -> None: + # Once a criterion decides on a resolved round, its live_verdict is + # never called again — count the checker's calls directly. + watcher = _watcher([_skill_crit("date-teller", "date-teller", max_steps_to_decide=10)]) + checker = watcher._armed[0][1] + calls = {"n": 0} + original = type(checker).live_verdict + + def counting(self_, criterion, records): + calls["n"] += 1 + return original(self_, criterion, records) + + type(checker).live_verdict = counting # type: ignore[method-assign] + try: + _feed(watcher, _skill_events("date-teller")) # decides pass on call 1 + decided_at = calls["n"] + for i in range(5): + watcher.on_event(_tool_end(_cmd("Bash", {"command": f"echo {i}"}))) + assert calls["n"] == decided_at # latched: zero further polls + finally: + type(checker).live_verdict = original # type: ignore[method-assign] + assert watcher.should_stop() is False # and still no stop (no stop_on_pass) + + def test_pass_without_stop_on_pass_never_stops(self) -> None: + # A stop_on_fail-armed positive... cannot exist (fail is inert on a + # positive); the realistic shape is both-trigger fanning. On a positive + # row with only stop_on_fail, NOTHING can ever fire — engaging the + # skill latches pass silently and the run continues. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_fail=True)]) + _feed(watcher, _skill_events("date-teller")) + assert watcher.should_stop() is False + assert watcher.info is None + def test_records_turn_and_tool_index(self) -> None: - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, _skill_events("date-teller")) assert watcher.info is not None assert watcher.info.sdk_turn_index == 1 assert watcher.info.tool_call_index == 1 def test_turns_remaining_from_max_turns(self) -> None: - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")], max_turns=15) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)], max_turns=15) _feed(watcher, _skill_events("date-teller")) assert watcher.info is not None assert watcher.info.turns_remaining_at_stop == 14 # 15 - sdk_turn_index(1) def test_turns_remaining_none_when_max_turns_unset(self) -> None: - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")], max_turns=None) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)], max_turns=None) _feed(watcher, _skill_events("date-teller")) assert watcher.info is not None assert watcher.info.turns_remaining_at_stop is None def test_fail_open_on_raising_verdict(self) -> None: - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) with patch.object(SkillTriggeredChecker, "live_verdict", side_effect=RuntimeError("boom")): _feed(watcher, _skill_events("date-teller")) # Fail-open: disarmed, no false stop, degrades to a full run. @@ -1659,16 +2060,16 @@ def test_unresolved_tool_end_does_not_latch(self) -> None: # loop ends and the terminal status is chosen. Such an orphan Skill # engagement must NOT trip a stop, else a naturally-completed (or # timed-out / crashed) run gets recorded as early-stopped. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, [_agent_start(), _turn_start(), _unresolved_skill_end("date-teller")]) assert watcher.should_stop() is False assert watcher.info is None assert watcher._tool_call_index == 0 # the unresolved end is not even counted def test_resolved_after_unresolved_still_decides(self) -> None: - # An UNRESOLVED end is dropped, but a later RESOLVED engagement still fires - # the stop (dropping orphans never suppresses a real, observed stop). - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + # An UNRESOLVED end never evaluates, but a later RESOLVED engagement still + # fires the stop (skipping orphan rounds never suppresses a real stop). + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, [_agent_start(), _turn_start(), _unresolved_skill_end("date-teller")]) assert watcher.info is None _feed(watcher, [_tool_end(_skill_cmd("date-teller", tool_id="sk-real"))]) @@ -1676,8 +2077,78 @@ def test_resolved_after_unresolved_still_decides(self) -> None: assert watcher.info is not None assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + def test_unresolved_end_recorded_for_trajectory_parity(self) -> None: + # TRAJECTORY PARITY: the agent's EventCollector records force-closed + # (UNRESOLVED) commands into the TurnRecord that check_all_async later + # scores — e.g. a crashed attempt's drained partial turn. The watcher + # must reduce the SAME trajectory: the orphan is recorded (visible to + # the next evaluation round), just never counted or evaluated on. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) + _feed(watcher, [_agent_start(), _turn_start(), _unresolved_skill_end("date-teller")]) + assert watcher._tool_call_index == 0 # no round counted + assert watcher.info is None # no stop fired on the orphan itself + record = watcher._collector.build_turn_record() + assert any(c.tool_name == "Skill" for c in record.commands) # ...but it IS in the trajectory + # The next real round evaluates over the parity trajectory: an unrelated + # Bash call decides the criterion pass from the recorded orphan. + _feed(watcher, [_tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + + def test_budget_timeout_not_latched_when_orphan_already_decided(self) -> None: + # The verdict-preserving half of trajectory parity: a decide_within + # timeout must never latch an effective fail on a criterion the frozen + # trajectory scores as a pass. The deciding engagement arrived as a + # force-closed orphan (recorded, not evaluated); the budget expiring on + # the next round must see it as a live-pass, not fabricate a fail. + watcher = _watcher([_skill_crit("date-teller", "date-teller", max_steps_to_decide=1)]) + _feed(watcher, [_agent_start(), _turn_start(), _unresolved_skill_end("date-teller")]) + # Round 1 (tool_call_index == 1 >= decide_within): without parity this + # would latch a synthetic fail and fire DECISION_BUDGET_EXCEEDED. + _feed(watcher, [_tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.should_stop() is False + assert watcher.info is None + + def test_pass_stop_cuts_undecided_fail_only_sibling_documented_gap(self) -> None: + # KNOWN one-sided trade, pinned so a future deferral redesign flips it + # consciously: the pass-stop deferral holds only for PASS-CAPABLE + # siblings. An armed fail-only-decidable criterion that still needs + # evidence (command_executed with min_count>=1 AND max_count set — + # polarities == {"fail"} but the frozen score needs the command run) + # is NOT deferred on, so an on_pass=stop sibling can cut before its + # minimum count is reached and the armed gate scores it 0. Documented + # in TASK_DEFINITION_GUIDE.md § stop_early: authoritative scoring for + # such combinations belongs on the kill-switched run. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_on_pass=True), + _cmd_crit(min_count=1, max_count=3, stop_on_fail=True), + ] + ) + _feed(watcher, _skill_events("date-teller")) + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + + def test_fail_stop_reason_precedence_is_criteria_order_invariant(self) -> None: + # A native live-fail (distractor misfire) and a decide_within timeout + # resolving on the SAME round must report the same persisted/telemetry + # reason in either YAML order: the native fail always wins. + def build(order: str) -> EarlyStopWatcher: + distractor = _skill_crit("weather-teller", "date-teller", stop_on_fail=True) + timed = _skill_crit("date-teller", "date-teller", max_steps_to_decide=1) + criteria = [distractor, timed] if order == "distractor-first" else [timed, distractor] + return _watcher(criteria) + + for order in ("distractor-first", "timed-first"): + watcher = build(order) + # One resolved misfire round: the distractor natively fails AND the + # timed criterion's budget (1) expires on the same tool call. + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_skill_cmd("weather-teller", tool_id="w1"))]) + assert watcher.info is not None, order + assert watcher.info.reason == EarlyStopReason.CRITERION_FAILED, order + def test_decision_latched_after_fire(self) -> None: - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, _skill_events("date-teller")) fired = watcher.info # A subsequent (wrong-skill) engagement must not overwrite the latched decision. @@ -1690,7 +2161,7 @@ def test_tool_call_fires_before_result(self) -> None: # The decision latches on the tool CALL (ToolStartEvent): a Skill call # whose result never arrives (a cut-short turn would strip it) still stops. # No ToolEndEvent is ever fed. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, [_agent_start(), _turn_start(), _skill_start("date-teller")]) assert watcher.should_stop() is True assert watcher.info is not None @@ -1701,7 +2172,7 @@ def test_tool_call_fires_before_result(self) -> None: def test_tool_call_distractor_fail_fires(self) -> None: # A distractor (armed fail) fail-stops on the tool CALL that engages its # skill, before any result arrives. - watcher = _watcher([_skill_crit("weather-teller", "date-teller", stop_when="fail")]) + watcher = _watcher([_skill_crit("weather-teller", "date-teller", stop_on_fail=True)]) _feed(watcher, [_agent_start(), _turn_start(), _skill_start("weather-teller")]) assert watcher.info is not None assert watcher.info.reason == EarlyStopReason.CRITERION_FAILED @@ -1711,7 +2182,7 @@ def test_tool_call_latches_on_file_read_engagement(self) -> None: # (skills//...), not via a Skill tool call. The watcher must latch on # that Read ToolStart — the file-path parameter carries the signal on the # call itself, so early-stop fires off-Claude just as it does for Claude. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) read = CommandTelemetry( tool_name="Read", tool_id="r1", @@ -1727,7 +2198,7 @@ def test_tool_call_latches_before_unresolved_end(self) -> None: # The call fires the stop in-loop; a later finalize() UNRESOLVED end for # the SAME call is short-circuited (decision already latched) — no relabel, # no double count. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, [_agent_start(), _turn_start(), _skill_start("date-teller", tool_id="sk-1")]) fired = watcher.info _feed(watcher, [_unresolved_skill_end("date-teller", tool_id="sk-1")]) @@ -1738,7 +2209,7 @@ def test_tool_call_latches_before_unresolved_end(self) -> None: def test_tool_call_index_counts_prior_resolved_calls(self) -> None: # A prior resolved, non-deciding tool is counted at its ToolEnd; the # deciding in-flight call is then reported as the next (2nd) call. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) prior = _cmd("Bash", {"command": "ls"}) # not a skill engagement _feed(watcher, [_agent_start(), _turn_start(), _tool_end(prior)]) assert watcher.info is None @@ -1751,7 +2222,7 @@ def test_second_agent_start_does_not_reset_origin(self) -> None: # retry's second AgentStart must NOT reset it (the documented no-op branch # in on_event). Exercised deterministically via _started_monotonic rather # than the time-based elapsed_seconds field. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True)]) _feed(watcher, [_agent_start()]) origin = watcher._started_monotonic assert origin is not None @@ -1770,7 +2241,7 @@ def test_decision_budget_accumulates_across_retry_attempts(self) -> None: # AgentStartEvent (as on a retry) must NOT reset tool_call_index. A # future per-attempt reset would silently change scoring with this # test catching it. - watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=2)]) + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_on_pass=True, max_steps_to_decide=2)]) _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) assert watcher.should_stop() is False # 1 call so far, budget is 2 # A retry: a second AgentStartEvent must not reset the counter. @@ -1823,7 +2294,6 @@ async def _run_wiring( criteria: list[Any], events: list[Any], scores: list[float], - stop_early: bool, tmp_path, agent_type: AgentKind = AgentKind.CLAUDE_CODE, gate_threshold: float = 1.0, @@ -1833,7 +2303,7 @@ async def _run_wiring( ``scores`` are positional CriterionResult scores matching ``criteria``. The early-stop watcher is built directly (_setup is not invoked here). """ - task = _task(criteria=criteria, stop_early=stop_early, agent_type=agent_type, gate_threshold=gate_threshold) + task = _task(criteria=criteria, agent_type=agent_type, gate_threshold=gate_threshold) run_dir = tmp_path / "run" run_dir.mkdir(parents=True) orch = Orchestrator(task=task, run_dir=run_dir, variant_id="default") @@ -1858,7 +2328,7 @@ async def _run_wiring( ) orch.success_checker = checker - if stop_early: + if early_stop_active(task): orch._early_stop_watcher = EarlyStopWatcher.for_task(task) turn = TurnRecord(iteration=1, user_input="p", agent_output="done") @@ -1874,27 +2344,27 @@ async def _run_wiring( class TestOrchestratorEarlyStopWiring: _SKILL = "date-teller" - def _criteria(self, *, expected: str = "date-teller", stop_when: str | None = "pass") -> list[Any]: - # Armed positive skill_triggered + advisory file_exists (deliberately failing). + def _criteria(self, *, expected: str = "date-teller", armed: bool = True) -> list[Any]: + # Armed (stop_on_pass) positive skill_triggered + advisory file_exists + # (deliberately failing). return [ - _skill_crit(self._SKILL, expected, stop_when=stop_when), + _skill_crit(self._SKILL, expected, stop_on_pass=armed), FileExistsCriterion(path="artifact.txt", description="artifact must exist"), ] def _distractor_criteria(self) -> list[Any]: # A distractor (armed fail) + advisory file_exists, for the fail-stop path. return [ - _skill_crit("weather-teller", self._SKILL, stop_when="fail"), + _skill_crit("weather-teller", self._SKILL, stop_on_fail=True), FileExistsCriterion(path="artifact.txt", description="artifact must exist"), ] async def test_default_off_full_gate_no_early_stop(self, tmp_path) -> None: # Unarmed: no watcher, all criteria gate, advisory 0.0 drags to FAILURE. result, agent, _success = await _run_wiring( - criteria=self._criteria(stop_when=None), + criteria=self._criteria(armed=False), events=_skill_events(self._SKILL), scores=[1.0, 0.0], - stop_early=False, tmp_path=tmp_path, ) assert result.early_stop is None @@ -1907,7 +2377,6 @@ async def test_pass_stop_cuts_the_stream(self, tmp_path) -> None: criteria=self._criteria(), events=events, scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) assert agent.delivered == 3 @@ -1920,7 +2389,6 @@ async def test_fail_stop_wiring(self, tmp_path) -> None: criteria=self._distractor_criteria(), events=_skill_events("weather-teller"), scores=[0.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) assert result.early_stop is not None @@ -1931,7 +2399,6 @@ async def test_early_stop_info_fields_populated(self, tmp_path) -> None: criteria=self._criteria(), events=_skill_events(self._SKILL), scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) assert result.early_stop is not None @@ -1945,22 +2412,26 @@ async def test_advisory_not_gated_on_early_stop(self, tmp_path) -> None: criteria=self._criteria(), events=_skill_events(self._SKILL), scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) assert result.early_stop is not None assert result.all_criteria_passed(self._criteria()) is False # full gate would fail assert result.armed_criteria_passed(self._criteria()) is True # armed gate passes - async def test_decision_budget_exceeded_forces_failure_bypassing_gate(self, tmp_path) -> None: + async def test_decision_budget_exceeded_gates_through_armed_gate(self, tmp_path) -> None: # A criterion capped at max_steps_to_decide=1 that never engages its - # skill forces a hard fail — even though BOTH mocked criterion scores - # are 1.0 (the weighted gate, if consulted, would pass). + # skill fires a timeout fail-stop — and then gates through the SAME + # weighted armed gate as any other stop (no force-fail bypass). The + # mocked checker deliberately scores everything 1.0, so the armed gate + # passes: the stop truncates the run, the gate decides the verdict. + # (In a real run the frozen trajectory would score the undecided + # criterion 0.0 and the gate would fail — asserted separately below in + # test_decision_budget_exceeded_real_scores_fail_the_gate.) criteria = [ - _skill_crit(self._SKILL, self._SKILL, stop_when="pass", max_steps_to_decide=1), + _skill_crit(self._SKILL, self._SKILL, stop_on_pass=True, max_steps_to_decide=1), FileExistsCriterion(path="artifact.txt", description="artifact must exist"), ] - task = _task(criteria=criteria, stop_early=True) + task = _task(criteria=criteria) run_dir = tmp_path / "run" run_dir.mkdir(parents=True) orch = Orchestrator(task=task, run_dir=run_dir, variant_id="default") @@ -1992,73 +2463,119 @@ async def test_decision_budget_exceeded_forces_failure_bypassing_gate(self, tmp_ assert orch.result.early_stop is not None assert orch.result.early_stop.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert success is True + + async def test_decision_budget_exceeded_real_scores_fail_the_gate(self, tmp_path) -> None: + # Same timeout stop, but with realistic frozen-trajectory scores: the + # timed-out criterion never engaged its skill, so the standard checker + # scores it 0.0 and the armed gate fails — the timeout leads to + # FAILURE through the gate, not around it. + result, _agent, success = await _run_wiring( + criteria=[ + _skill_crit(self._SKILL, self._SKILL, stop_on_pass=True, max_steps_to_decide=1), + FileExistsCriterion(path="artifact.txt", description="artifact must exist"), + ], + events=[_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))], + scores=[0.0, 1.0], + tmp_path=tmp_path, + ) + assert result.early_stop is not None + assert result.early_stop.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED assert success is False - async def test_completed_naturally_still_uses_armed_gate(self, tmp_path) -> None: - # Armed for early-stop, but the skill is never engaged -> watcher never - # fires -> the run completes naturally. The armed subset STILL gates - # final_status (not the full set): the armed criterion itself scored - # 0.0, so _evaluation_loop's real return value is False — a genuine - # armed-gate failure, not a full-gate one (though both agree here). + async def test_decision_budget_exceeded_advisory_demotion(self, tmp_path) -> None: + # The advisory-demotion consequence of fired-only gating, pinned where + # the two gates DISAGREE: the timed-out armed criterion scores 1.0 on + # the frozen trajectory while the unarmed advisory criterion scores + # 0.0. The armed gate passes (SUCCESS) even though the full strict-AND + # gate would fail — the advisory criterion never had the chance to be + # satisfied on a truncated trajectory, so it must not gate. + criteria = [ + _skill_crit(self._SKILL, self._SKILL, stop_on_pass=True, max_steps_to_decide=1), + FileExistsCriterion(path="artifact.txt", description="artifact must exist"), + ] + result, _agent, success = await _run_wiring( + criteria=criteria, + events=[_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))], + scores=[1.0, 0.0], + tmp_path=tmp_path, + ) + assert result.early_stop is not None + assert result.early_stop.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert result.all_criteria_passed(criteria) is False # full gate would fail + assert success is True # armed gate decides: advisory 0.0 is demoted + + async def test_completed_naturally_weighted_armed_gate_does_not_run(self, tmp_path) -> None: + # FIRED-ONLY gating, diverging in the other direction from the sibling + # test below: two ARMED criteria (0.8 passing / 0.2 failing) under a + # 0.7 threshold. The weighted armed gate WOULD pass (0.8 >= 0.7), but + # the run completed naturally (watcher never fired), so the strict + # full-set gate applies and the failing 0.2 criterion drags the run to + # failure — proving the armed gate did not run. + criteria = [ + _skill_crit(self._SKILL, self._SKILL, stop_on_pass=True, weight=0.8), + _skill_crit("weather-teller", self._SKILL, stop_on_fail=True, weight=0.2), + ] result, agent, success = await _run_wiring( - criteria=self._criteria(), - events=[_agent_start(), _turn_start()], # no skill engagement - scores=[0.0, 0.0], - stop_early=True, + criteria=criteria, + events=[_agent_start(), _turn_start()], # no skill engagement -> watcher never fires + scores=[1.0, 0.0], tmp_path=tmp_path, + gate_threshold=0.7, ) assert result.early_stop is None assert agent.delivered == 2 # full (short) stream consumed - assert success is False - - async def test_completed_naturally_armed_gate_forgives_advisory_failure(self, tmp_path) -> None: - # THE fix this test pins: same never-fired scenario, but the ARMED - # criterion passes (1.0) while the ADVISORY one fails (0.0). Under the - # old full-set gate this would be FAILURE (the advisory 0.0 drags it - # down); under the fixed armed-gate-always-applies-when-stop_early - # semantics _evaluation_loop's real return value is True — one task - # config, one gate semantic, regardless of whether the watcher - # physically fired. + assert result.armed_criteria_passed(criteria, 0.7) is True # the armed gate WOULD pass + assert success is False # the strict full-set gate is what decided this run + + async def test_completed_naturally_full_gate_applies_even_when_armed(self, tmp_path) -> None: + # FIRED-ONLY gating: an armed run whose watcher never fires (the agent + # completed naturally) has a FULL trajectory, so the strict full-set + # gate applies — the advisory 0.0 drags it to FAILURE exactly as it + # would on an unarmed run. Arming a criterion (e.g. adding a + # decide_within fail-fast timeout) must never change the verdict of a + # run it didn't cut; the weighted armed gate is reserved for runs the + # watcher actually truncated. result, agent, success = await _run_wiring( criteria=self._criteria(), events=[_agent_start(), _turn_start()], # no skill engagement -> watcher never fires scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) assert result.early_stop is None assert agent.delivered == 2 - assert result.all_criteria_passed(self._criteria()) is False # the full gate WOULD fail - assert success is True # but the armed gate is what actually decided this run + assert result.armed_criteria_passed(self._criteria()) is True # the armed gate WOULD pass + assert success is False # but the full gate is what actually decided this run async def test_gate_threshold_plumbing_end_to_end(self, tmp_path) -> None: - # Mutation-resistant pin for the two plumbing hops the reviewer - # flagged as untested: YAML stop_early_gate_threshold -> the final - # gate (orchestrator.py) -> _evaluation_loop's real return value, AND - # -> the persisted EarlyStopInfo.gate_threshold. Weighted criteria - # (0.8/0.2), watcher never fires (never touches either skill), so - # this exercises the natural-completion armed-gate path directly. + # Mutation-resistant pin for the plumbing hop: YAML + # stop_early_gate_threshold -> the final gate (orchestrator.py) -> + # _evaluation_loop's real return value. Weighted criteria (0.8/0.2); + # the positive engages its skill so the watcher FIRES a pass-stop + # (fired-only gating means the armed gate only ever applies to a fired + # run), and the mocked frozen-trajectory scores fail the low-weight + # distractor — so the threshold alone decides the verdict. criteria = [ - _skill_crit(self._SKILL, self._SKILL, stop_when="pass", weight=0.8), - _skill_crit("weather-teller", self._SKILL, stop_when="fail", weight=0.2), + _skill_crit(self._SKILL, self._SKILL, stop_on_pass=True, weight=0.8), + _skill_crit("weather-teller", self._SKILL, stop_on_fail=True, weight=0.2), ] - _result_default, _agent, success_default = await _run_wiring( + result_default, _agent, success_default = await _run_wiring( criteria=criteria, - events=[_agent_start(), _turn_start()], + events=_skill_events(self._SKILL), scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path / "a", gate_threshold=1.0, ) + assert result_default.early_stop is not None # the armed gate only applies to a fired run assert success_default is False # 0.8 < 1.0 - _result_low, _agent2, success_low = await _run_wiring( + result_low, _agent2, success_low = await _run_wiring( criteria=criteria, - events=[_agent_start(), _turn_start()], + events=_skill_events(self._SKILL), scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path / "b", gate_threshold=0.7, ) + assert result_low.early_stop is not None assert success_low is True # 0.8 >= 0.7 — a mutation to a literal 1.0 would flip this async def test_gate_threshold_persisted_on_early_stop_info(self, tmp_path) -> None: @@ -2068,7 +2585,6 @@ async def test_gate_threshold_persisted_on_early_stop_info(self, tmp_path) -> No criteria=self._criteria(), events=_skill_events(self._SKILL), scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, gate_threshold=0.7, ) @@ -2085,7 +2601,6 @@ async def test_completed_run_with_orphan_tool_not_early_stopped(self, tmp_path) criteria=self._criteria(), events=events, scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) assert result.early_stop is None @@ -2101,7 +2616,6 @@ async def test_tool_call_cut_without_tool_end(self, tmp_path) -> None: criteria=self._criteria(), events=events, scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) assert result.early_stop is not None @@ -2115,13 +2629,59 @@ async def test_fail_open_wiring_degrades_to_full_run(self, tmp_path) -> None: criteria=self._criteria(), events=_skill_events(self._SKILL), scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, ) # Fail-open: no early_stop recorded, full gate applies. assert result.early_stop is None +class TestOrchestratorSetupActivation: + """The REAL ``Orchestrator._setup`` builds (or withholds) the watcher. + + The wiring tests above inject the watcher by hand; these drive ``_setup`` + itself on its evaluate-only path (sandbox pre-set, so no agent/sandbox + creation is reached) to pin the activation seam: armed -> watcher built, + kill-switched -> watcher stays None. + """ + + def _orchestrator(self, tmp_path: Path, *, stop_early: bool | None) -> Orchestrator: + task = _task( + criteria=[_skill_crit("date-teller", "date-teller", stop_on_pass=True)], + stop_early=stop_early, + ) + run_dir = tmp_path / "run" + run_dir.mkdir(parents=True) + orch = Orchestrator(task=task, run_dir=run_dir, variant_id="default") + orch.result = EvaluationResult( + task_id=task.task_id, + task_description=task.description, + variant_id="default", + agent_type=AgentKind.CLAUDE_CODE, + started_at=datetime.now(), + final_status=FinalStatus.FAILURE, + iteration_count=0, + environment_info={}, + ) + sandbox = MagicMock() + sandbox.sandbox_dir = tmp_path / "sandbox" + sandbox.sandbox_dir.mkdir() + orch.sandbox = sandbox # evaluate-only: _setup skips agent/sandbox creation + return orch + + async def test_setup_builds_watcher_when_armed(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(settings, "api_backend", ApiBackend.DIRECT) + orch = self._orchestrator(tmp_path, stop_early=None) + await orch._setup() + assert orch._early_stop_watcher is not None + assert len(orch._early_stop_watcher._armed) == 1 + + async def test_setup_kill_switch_leaves_watcher_none(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(settings, "api_backend", ApiBackend.DIRECT) + orch = self._orchestrator(tmp_path, stop_early=False) + await orch._setup() + assert orch._early_stop_watcher is None + + # --------------------------------------------------------------------------- # # Report / telemetry surfaces # --------------------------------------------------------------------------- # @@ -2181,17 +2741,18 @@ def test_runtime_note_rendered_with_turns_avoided(self) -> None: assert "<= 14 turn(s) avoided" in blob assert "gated on armed criteria only; other criteria are advisory" in blob - def test_runtime_note_for_decision_budget_exceeded_is_not_misleading(self) -> None: - # The budget-exceeded reason forces FAILURE outright — NO criterion - # gated here, unlike a real early stop. The note must say so, not the - # generic "gated on armed criteria" text (which would tell the reader - # the opposite of what happened). + def test_runtime_note_for_decision_budget_exceeded_names_the_timeout(self) -> None: + # The budget-exceeded reason is an effective fail gated through the + # armed gate like any other — the note must say the criterion timed + # out (so a reader can tell a timeout from a native misfire) AND that + # the armed gate applied (it did — no bypass). result = _stopped_result(reason=EarlyStopReason.DECISION_BUDGET_EXCEEDED) lines = ReportGenerator._runtime_notes_lines(_run_summary([eval_result_to_task_dict(result)])) blob = "\n".join(lines) assert "stopped early (decision_budget_exceeded)" in blob - assert "gated on armed criteria only" not in blob - assert "forced to FAILURE" in blob + assert "timed out undecided" in blob + assert "gated on armed criteria only" in blob + assert "forced to FAILURE" not in blob def test_runtime_note_absent_for_unarmed_run(self) -> None: lines = ReportGenerator._runtime_notes_lines(_run_summary([eval_result_to_task_dict(_result())])) @@ -2200,9 +2761,18 @@ def test_runtime_note_absent_for_unarmed_run(self) -> None: def test_html_header_shows_early_stop_badge(self) -> None: html = _render_header(_stopped_result()) assert "stopped early (criterion_passed)" in html + assert "gated on armed criteria only" in html # shared gate note as the tooltip # No badge on a normal run. assert "stopped early" not in _render_header(_result()) + def test_html_badge_tooltip_for_decision_budget_exceeded(self) -> None: + # The decision-budget flavor of the shared gate note reaches the HTML + # tooltip too — and survives _esc (the prose has no markup, so escaping + # must be a no-op on it). + html = _render_header(_stopped_result(reason=EarlyStopReason.DECISION_BUDGET_EXCEEDED)) + assert "stopped early (decision_budget_exceeded)" in html + assert "decision-step budget exceeded" in html + def test_html_criteria_marks_only_advisory_rows(self) -> None: # Armed skill_triggered (matches _info.armed_criteria) + advisory file_exists. armed = _crit_result("skill_triggered", 1.0) @@ -2678,7 +3248,7 @@ class TestOrchestratorEarlyStopWiringCodex: def _criteria(self) -> list[Any]: return [ - _skill_crit(self._SKILL, self._SKILL, stop_when="pass"), + _skill_crit(self._SKILL, self._SKILL, stop_on_pass=True), FileExistsCriterion(path="artifact.txt", description="artifact must exist"), ] @@ -2689,7 +3259,6 @@ async def test_pass_stop_populates_early_stop_and_armed_gate(self, tmp_path) -> criteria=self._criteria(), events=events, scores=[1.0, 0.0], - stop_early=True, tmp_path=tmp_path, agent_type=AgentKind.CODEX, ) diff --git a/tests/test_threshold_enforcement.py b/tests/test_threshold_enforcement.py index 3f59ffa4..4a31a738 100644 --- a/tests/test_threshold_enforcement.py +++ b/tests/test_threshold_enforcement.py @@ -17,6 +17,7 @@ EvaluationResult, FileExistsCriterion, FinalStatus, + StopEarlyPolicy, ) @@ -134,12 +135,12 @@ def test_all_zero_weight_criteria_leave_an_empty_gate(self): assert _make_result([0.0, 0.0]).all_criteria_passed(criteria) def test_zero_weight_cannot_be_armed_for_early_stop(self): - """weight=0 + stop_when is incoherent: it would leave the early-stop gate empty.""" + """weight=0 + a stop_early block is incoherent: it would leave the early-stop gate empty.""" with pytest.raises(ValidationError, match="weight=0"): CommandExecutedCriterion( description="informational + armed", weight=0.0, - stop_when="pass", + stop_early=StopEarlyPolicy(on_pass="stop"), tool_name="Bash", command_pattern="pytest", )