UiPath
diff --git a/‎CLAUDE.md‎
Lines changed: 1 addition & 1 deletion b/‎CLAUDE.md‎
Lines changed: 1 addition & 1 deletion
diff --git a/‎docs/EXTENDING.md‎
Lines changed: 17 additions & 3 deletions b/‎docs/EXTENDING.md‎
Lines changed: 17 additions & 3 deletions
diff --git a/‎src/coder_eval/criteria/base.py‎
Lines changed: 110 additions & 45 deletions b/‎src/coder_eval/criteria/base.py‎
Lines changed: 110 additions & 45 deletions
@@ -141,7 +141,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
 
@@ -174,19 +174,33 @@ class MyChecker(BaseCriterion[MyCriterion]):
 
 Notes:
 
-- **Do not override `check()` / `check_async()`** — both are final and wrap
+- **Do not override `check()` / `check_async()`** — both are `@final` and wrap
   `_check_impl` / `_check_impl_async` with error handling (an exception becomes
   a score-0.0 result with the error captured; a `JudgeInfrastructureError`
   escalates instead).
 - Implement exactly ONE of `_check_impl` (plain sync — the common case, shown
   above) or `_check_impl_async` (genuine async I/O — an async HTTP client or
   subprocess bridge; see `llm_judge`/`agent_judge`). Whichever you implement,
   `BaseCriterion` derives the other for free (`asyncio.to_thread` / `asyncio.run`),
-  so there is no need to hand-maintain both. Overriding neither raises
-  `TypeError` immediately at class-definition time. `SuccessChecker.check_all_async`
+  so there is no need to hand-maintain both. Overriding neither, or overriding
+  BOTH, raises `TypeError` immediately at class-definition time (a shared
+  abstract base for a family of checkers that intentionally implements neither
+  can opt out with the `abstract=True` class keyword — every one of ITS
+  subclasses is still checked normally). `SuccessChecker.check_all_async`
   — the orchestrator's entry point — awaits every `_check_impl_async`-native
   checker directly on the event loop (concurrently with its siblings), and runs
   everything else through one `asyncio.to_thread` slot.
+- If your checker overrides `_check_impl_async`, it MUST NOT do blocking work
+  (file I/O, subprocess calls) directly on the event loop — that would stall
+  every sibling judge criterion gathered alongside it. Offload blocking calls
+  with `await asyncio.to_thread(...)` (see `llm_judge`/`agent_judge`, which do
+  this for their sandbox/reference file reads).
+- The derived sync bridge (`_check_impl`'s default `asyncio.run(...)` call) can
+  only run when no event loop is already running — calling the public sync
+  `check()`/`check_all()` on an async-only checker from inside a running loop
+  raises `CheckerMisuseError` (escalates, like `JudgeInfrastructureError`)
+  rather than returning a wrong score. Always reach for `check_async()` /
+  `check_all_async()` from async code.
 - Return `score` in `[0.0, 1.0]` — binary criteria use `0.0`/`1.0`; fractional ones
   anything in between.
 - For **suite-level metrics** on dataset-backed tasks, override
 
@@ -9,9 +9,9 @@
 from collections.abc import Awaitable, Callable
 from dataclasses import dataclass
 from functools import wraps
-from typing import TYPE_CHECKING, Any, ClassVar, Concatenate, Literal, ParamSpec
+from typing import TYPE_CHECKING, Any, ClassVar, Concatenate, Literal, ParamSpec, final
 
-from coder_eval.errors import JudgeInfrastructureError
+from coder_eval.errors import CheckerMisuseError, JudgeInfrastructureError
 from coder_eval.models import BaseSuccessCriterion, CriterionAggregate, CriterionResult
 
 
@@ -52,6 +52,36 @@ class CheckContext:
 # uninitialized local; a plain `typing.ParamSpec` is unambiguous to both tools.
 P = ParamSpec("P")
 
+# Exceptions that must escalate rather than be captured into a scored-0.0
+# CriterionResult — a judge-infra outage or a checker-contract misuse is not an
+# agent failure. Shared by both handle_criterion_errors(_async) wrappers below.
+_ESCALATING_EXCEPTIONS: tuple[type[Exception], ...] = (JudgeInfrastructureError, CheckerMisuseError)
+
+
+def _failed_result(owner: Any, criterion: BaseSuccessCriterion, exc: Exception, method: str) -> CriterionResult:
+    """Build the failed ``CriterionResult`` for a captured (non-escalating)
+    checker exception, and log it. Shared by the sync/async
+    ``handle_criterion_errors(_async)`` wrapper tails so the two decorators
+    differ only in ``def``/``async def`` and ``return``/``return await``.
+    """
+    exc_info = f"{exc.__class__.__name__}: {exc}"
+    tb = ""
+    if os.getenv("CODER_EVAL_DEBUG") == "1":
+        tb = "\n" + "".join(traceback.format_exc(limit=5))
+
+    criterion_type = criterion.type
+    logger.error(
+        f"Error in {owner.__class__.__name__}.{method}() for criterion type '{criterion_type}': {exc_info}",
+        exc_info=True,  # Adds full stack trace to logs
+    )
+    return CriterionResult(
+        criterion_type=criterion_type,
+        description=criterion.description,
+        score=0.0,
+        details=f"Error during check: {exc_info}{tb}",
+        error=exc_info,  # Include exception type and message
+    )
+
 
 def handle_criterion_errors(  # noqa: UP047
     func: Callable[Concatenate[Any, BaseSuccessCriterion, P], CriterionResult],
@@ -78,28 +108,13 @@ def wrapper(
     ) -> CriterionResult:
         try:
             return func(self, criterion, *args, **kwargs)
-        except JudgeInfrastructureError:
-            # Judge infra failure is NOT an agent failure — do not score it 0.0.
-            # Propagates to Orchestrator.run()'s broad except → FinalStatus.ERROR.
+        except _ESCALATING_EXCEPTIONS:
+            # Judge infra failure / checker-contract misuse is NOT an agent
+            # failure — do not score it 0.0. Propagates to Orchestrator.run()'s
+            # broad except → FinalStatus.ERROR.
             raise
         except Exception as e:
-            exc_info = f"{e.__class__.__name__}: {e}"
-            tb = ""
-            if os.getenv("CODER_EVAL_DEBUG") == "1":
-                tb = "\n" + "".join(traceback.format_exc(limit=5))
-
-            criterion_type = criterion.type
-            logger.error(
-                f"Error in {self.__class__.__name__}.check() for criterion type '{criterion_type}': {exc_info}",
-                exc_info=True,  # Adds full stack trace to logs
-            )
-            return CriterionResult(
-                criterion_type=criterion_type,
-                description=criterion.description,
-                score=0.0,
-                details=f"Error during check: {exc_info}{tb}",
-                error=exc_info,  # Include exception type and message
-            )
+            return _failed_result(self, criterion, e, "check")
 
     return wrapper
 
@@ -123,26 +138,10 @@ async def wrapper(
     ) -> CriterionResult:
         try:
             return await func(self, criterion, *args, **kwargs)
-        except JudgeInfrastructureError:
+        except _ESCALATING_EXCEPTIONS:
             raise
         except Exception as e:
-            exc_info = f"{e.__class__.__name__}: {e}"
-            tb = ""
-            if os.getenv("CODER_EVAL_DEBUG") == "1":
-                tb = "\n" + "".join(traceback.format_exc(limit=5))
-
-            criterion_type = criterion.type
-            logger.error(
-                f"Error in {self.__class__.__name__}.check_async() for criterion type '{criterion_type}': {exc_info}",
-                exc_info=True,
-            )
-            return CriterionResult(
-                criterion_type=criterion_type,
-                description=criterion.description,
-                score=0.0,
-                details=f"Error during check: {exc_info}{tb}",
-                error=exc_info,
-            )
+            return _failed_result(self, criterion, e, "check_async")
 
     return wrapper
 
@@ -169,10 +168,12 @@ class derives the other automatically:
       base's default ``_check_impl`` derives a sync call by running the async
       one to completion on a fresh event loop (``asyncio.run``).
 
-    ``__init_subclass__`` enforces that a checker overrides at least one of
+    ``__init_subclass__`` enforces that a checker overrides EXACTLY ONE of
     the two, at class-definition time — overriding neither would recurse
     forever between the defaults (``asyncio.run`` <-> ``asyncio.to_thread``)
-    the first time either is called.
+    the first time either is called, and overriding both would let the two
+    implementations silently drift into different scores depending on which
+    entry point (``check`` vs ``check_async``) ran.
 
     ``check()`` / ``check_async()`` are FINAL — they apply centralized error
     handling and must not be overridden; implement ``_check_impl`` /
@@ -206,19 +207,67 @@ def _check_impl(
     # live_verdict; CE025 enforces that the two stay consistent.
     live_stop_polarities: ClassVar[frozenset[str]] = frozenset()
 
-    def __init_subclass__(cls, **kwargs: Any) -> None:
+    def __new__(cls, *args: Any, **kwargs: Any) -> "BaseCriterion[C]":
+        """Block direct instantiation of ``BaseCriterion`` itself.
+
+        ``__init_subclass__`` below only runs for SUBCLASSES, so with no
+        ``@abstractmethod`` left on this class (both ``_check_impl*`` methods
+        have concrete default bodies, by design — that's what lets each derive
+        the other), plain ``ABCMeta`` no longer blocks ``BaseCriterion()``
+        directly. This restores that guarantee without reintroducing an
+        abstract method that would break the "override at least one" contract.
+        """
+        if cls is BaseCriterion:
+            raise TypeError("BaseCriterion is abstract and cannot be instantiated directly")
+        return super().__new__(cls)
+
+    def __init_subclass__(cls, *, abstract: bool = False, **kwargs: Any) -> None:
         """Enforce the ``_check_impl`` / ``_check_impl_async`` override contract
         at class-definition time (module import), regardless of which entry
         point later registers the class — closing the gap where a subclass
         registered via ``CriterionRegistry.register`` directly (bypassing the
         ``register_criterion`` decorator) escaped the check, and turning the
         mutual-recursion failure mode (``asyncio.run`` <-> ``asyncio.to_thread``
         exhausting OS threads) into an immediate, clearly-named ``TypeError``.
+
+        Enforces "exactly one", not just "at least one": overriding BOTH is
+        also rejected — a checker with two live implementations (sync-path
+        `_check_impl` and async-path `_check_impl_async`) is free to have them
+        drift into different scores for identical agent output depending on
+        which entry point (``check`` vs ``check_async``) happened to run it,
+        which is exactly the class of bug this derivation design exists to
+        eliminate.
+
+        Pass ``abstract=True`` on a class that intentionally implements
+        neither (e.g. a shared abstract base for a family of related
+        checkers) to opt out of the check for that one class; every one of
+        ITS subclasses is still checked normally.
         """
         super().__init_subclass__(**kwargs)
-        if cls._check_impl is BaseCriterion._check_impl and cls._check_impl_async is BaseCriterion._check_impl_async:
+        if abstract:
+            return
+        overrides_sync = cls._check_impl is not BaseCriterion._check_impl
+        overrides_async = cls._check_impl_async is not BaseCriterion._check_impl_async
+        if not overrides_sync and not overrides_async:
             raise TypeError(f"{cls.__name__} must override _check_impl or _check_impl_async")
+        if overrides_sync and overrides_async:
+            msg = f"{cls.__name__} must override exactly one of _check_impl / _check_impl_async, not both"
+            raise TypeError(msg)
+
+    @classmethod
+    def is_native_async(cls) -> bool:
+        """Whether this checker class makes genuine async I/O — i.e. overrides
+        ``_check_impl_async`` itself rather than inheriting the base's
+        to-thread-wrapped-sync default.
+
+        Public + typed so dispatch code (``SuccessChecker._is_native_async``)
+        and anything else that needs to classify a checker doesn't have to
+        reach into ``_check_impl_async`` (a "protected" name) on another
+        class from a different package.
+        """
+        return cls._check_impl_async is not BaseCriterion._check_impl_async
 
+    @final
     @handle_criterion_errors
     def check(
         self,
@@ -283,8 +332,23 @@ def _check_impl(
             CriterionResult with score (0.0-1.0), details, and error info
 
         Raises:
-            Any exception - will be caught by @handle_criterion_errors decorator
+            CheckerMisuseError: this bridge is called from inside a running
+                event loop (``asyncio.run`` cannot start a nested loop) — this
+                is a caller mistake (the async-primary surface should have
+                been awaited instead), not an agent failure, so it escalates
+                rather than silently scoring 0.0.
+            Any other exception - will be caught by @handle_criterion_errors
         """
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            pass
+        else:
+            msg = (
+                f"{type(self).__name__} implements only _check_impl_async; "
+                f"call check_async()/check_all_async() from an event loop, not check()/check_all()"
+            )
+            raise CheckerMisuseError(msg)
         return asyncio.run(
             self._check_impl_async(
                 criterion,
@@ -295,6 +359,7 @@ def _check_impl(
             )
         )
 
+    @final
     @handle_criterion_errors_async
     async def check_async(
         self,