fix(command-executed): match patterns against shell-normalized commands

.claude/harness-candidates.md

@@ -107,3 +107,23 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   caught them. The cleanup plan explicitly deferred this as YAGNI for the
   one-time purge, but any future doc rename/deletion re-opens the same blind
   spot — caught in the 2026-07-03 open-source-docs-cleanup implementation run.
+
+## From PR #77 (command-executed shell-normalize) — CE030-to-criteria deferred
+
+- [ ] **Extend CE030 doc/schema-parity to the `SuccessCriterion` union** so a new
+  criterion (or field) can't ship undocumented. Attempted in PR #77 and reverted:
+  CI installs `--extra uipath`, and in that environment `coder_eval.models.criteria`
+  gains a `CliCalledCriterion` (fields `log`/`positional`) that is NOT present in a
+  plain checkout (it did not reproduce on macOS, whose lockfile resolution omits the
+  contributing linux-only component). It defeated every discriminator tried — union
+  membership, a `__module__` string filter (it is spoofed to `coder_eval.models.criteria`),
+  a genuine-module-attribute scan (it is `setattr` onto the module), and even an AST
+  parse of the `SuccessCriterion` union literal in `criteria.py` source (CI's imported
+  criteria module resolves to a file whose union literal already contains it). No
+  runtime OR source signal available in the lint could separate the injected criterion
+  from an in-tree one. Revisit only with a way to identify the in-tree criterion set that
+  is provably immune to the uipath integration — e.g. a hardcoded name allowlist of the
+  in-tree criteria (losing auto-coverage of new ones), or first understanding exactly how
+  that environment injects the criterion. Until then CE030 stays scoped to the four
+  top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is
+  documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless.

docs/TASK_DEFINITION_GUIDE.md

@@ -796,6 +796,7 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary
 | `expected_exit_code` | 0 | Expected exit code |
 | `expected_stdout` | `null` | When set, stdout is also checked |
 | `stdout_match` | `"exact"` | Match mode: `exact` (stripped), `contains` (substring), `regex` (pattern) |
+| `score_from_stdout` | `false` | Read a float score (0.0–1.0) from the first stdout line (remaining lines become details); a non-zero exit code or a parse failure scores 0.0. Mutually exclusive with `expected_stdout`. |
 
 ### `file_matches_regex`
 
@@ -823,6 +824,12 @@ Compares agent's code with a reference solution using similarity scoring. **Cont
   weight: 2.0
 ```
 
+| Field | Default | Description |
+|-------|---------|-------------|
+| `agent_file` | *required* | Path to the agent's generated file (relative to the sandbox root). |
+| `comparison_method` | `"ast"` | `ast` (structure), `token` (text), or `complexity` (metrics). |
+| `similarity_threshold` | 0.8 | Minimum similarity score to pass (0.0–1.0). |
+
 **Comparison methods:**
 - `ast` — Abstract Syntax Tree similarity (structure-based)
 - `token` — Token-based similarity (implementation details)
@@ -841,6 +848,17 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins
   description: "Agent must use curl to fetch weather"
 ```
 
+| Field | Default | Description |
+|-------|---------|-------------|
+| `tool_name` | `null` | Tool-name filter (e.g. `Bash`); `null` counts any tool. |
+| `command_pattern` | `null` | Regex to match the command; `null` matches any command. Matched with shell normalization (see below). |
+| `min_count` | 1 | Minimum matching commands required. `0` permits zero matches — combine with `max_count: 0` to assert a command must **NOT** run. |
+| `max_count` | `null` | Optional inclusive upper bound. When set, the criterion passes iff `min_count <= matches <= max_count`. |
+| `require_success` | `false` | Only count commands that completed successfully. |
+| `exclude_pattern` | `null` | Regex that must NOT match; a command matching both `command_pattern` and `exclude_pattern` is skipped. Also matched with shell normalization (see below). |
+
+**Shell normalization.** For a Bash command, both `command_pattern` and `exclude_pattern` are matched against the raw command text **and** its shell-normalized form — the `bash`/`sh`/`zsh -lc "..."` wrapper stripped and shell quoting resolved with `shlex` — and a hit on *either* form counts. So a pattern like `curated_channels` matches whether the agent wrote the argument bare, `'single'`-quoted, `"double"`-quoted, or `\"escaped\"`; you do **not** hand-encode shell quoting. Because the same haystacks also feed `exclude_pattern` and the `max_count` gate, normalization is **not** purely additive: a quote-obfuscated call can now be caught by an exclusion or a `max_count: 0` gate that the raw text alone would have missed — and, conversely, an unedited `exclude_pattern` may now exclude a call it previously let through. Cross-repo suites that hand-encoded quote tolerance in their patterns should re-baseline.
+
 **Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry.
 
 ### `cli_called`
@@ -1026,6 +1044,8 @@ Have an LLM grade the task against a rubric written in the task YAML. **Continuo
 | `temperature` | `0.0` | Sampling temperature (0.0 = deterministic) |
 | `max_tokens` | `2000` | Maximum tokens in the judge's response |
 | `max_file_chars` | `20000` | Per-file (and agent_output) truncation applied before building the prompt |
+| `capture_transcript` | `true` | Persist a `JudgeTranscript` (raw verdict + rendered prompts + token usage) to a sibling `judge-<idx>.yaml`. Set `false` to drop it when on-disk size matters (e.g. 1000-row datasets); the `findings` on the result persist regardless. |
+| `max_transcript_chars` | `100000` | Aggregate cap on captured transcript text (verdict + prompt + system, split 60/30/10). Exceeding it marks the transcript `truncated=True`. |
 
 **Transport selection.** The judge call is routed by the active `API_BACKEND`:
 
@@ -1099,6 +1119,8 @@ Spawn a full Claude Code SDK agent as the judge. Unlike `llm_judge` (a single LL
 | `max_turns` | `50` | Judge's inner-loop turn limit |
 | `turn_timeout` | `300` | Wall-clock timeout (seconds) |
 | `agent` | hardened judge defaults | Nested `AgentConfig` — `model`, `permission_mode`, `allowed_tools`, `disallowed_tools`, `ignore_patterns`, `sdk_options`. A partial block (e.g. only `model:`) still applies the judge security defaults for missing fields, and the security floor (`.claude` / `.mcp.json` / `_reference` ignore patterns, `setting_sources=[]`) is always enforced. |
+| `capture_transcript` | `true` | Persist a `JudgeTranscript` (tool calls + token usage + raw verdict + rendered prompts) to a sibling `judge-<idx>.yaml`. Set `false` to drop the trajectory log when on-disk size matters; the `findings` on the result persist regardless. |
+| `max_transcript_chars` | `100000` | Aggregate cap on captured transcript text (verdict + prompt + system + tool detail/result-preview lines, split 60/30/10 with tool calls prioritized). Exceeding it marks the transcript `truncated=True`. |
 
 **Security**
 

src/coder_eval/criteria/command_executed.py

@@ -3,6 +3,8 @@
 import json
 import logging
 import re
+import shlex
+from functools import lru_cache
 from typing import TYPE_CHECKING
 
 from coder_eval.criteria.base import BaseCriterion, CheckContext, LiveVerdict, register_criterion
@@ -16,10 +18,127 @@
 
 logger = logging.getLogger(__name__)
 
-# Limit regex search input length to mitigate ReDoS on large command strings
+# Limit regex search input length to mitigate ReDoS on large command strings.
+# Normalization runs over this same truncated window (see _match_haystacks), so
+# shlex never sees more than this many chars and needs no separate size guard.
 _MAX_PATTERN_SEARCH_LEN = 2000
 
 
+def _is_shell_program(arg0: str) -> bool:
+    """True if argv[0]'s basename looks like a POSIX shell.
+
+    A predicate rather than an enumerated allowlist: the set of shells is open
+    (``bash``/``sh`` on Linux, ``zsh`` on macOS — Codex shells through the host's
+    login shell, see codex_agent.py — plus ``dash``/``ksh``/…), and every common
+    shell basename ends in ``sh``. Matching is additive (the raw text stays a
+    haystack), so favouring recall over a hand-maintained list is safe.
+    """
+    return arg0.rsplit("/", 1)[-1].endswith("sh")
+
+
+def _is_command_flag(tok: str) -> bool:
+    """True for a short-option cluster carrying a ``-c`` command string.
+
+    Covers ``-c``, ``-lc``, ``-ic``, ``-lic`` (login/interactive + command) in
+    any order — a single ``-``-prefixed token whose option letters are all
+    alphabetic and include ``c``. ``--long`` options and ``-o=val`` forms are
+    rejected, so this is the first flag that actually introduces the command
+    string. The combined and split (``bash -l -c``) forms both work: a non-``c``
+    short flag like ``-l`` simply isn't the command flag and the scan continues.
+    """
+    return len(tok) >= 2 and tok[0] == "-" and tok[1] != "-" and tok[1:].isalpha() and "c" in tok[1:]
+
+
+@lru_cache(maxsize=1024)
+def _normalize_shell(cmd_text: str) -> str | None:
+    """Quote-resolved, wrapper-stripped form of a shell command, or None.
+
+    ``command_pattern`` regexes are written against the *logical* command
+    (``uip is resources run list <key> <resource>``), but telemetry records the
+    raw ``bash -lc "..."`` wrapper — so whichever way the agent happened to quote
+    an argument (bare, ``"double"``, ``'single'``, ``\\"escaped\\"``) leaks into
+    the pattern. Authors then hand-model that escaping and get it subtly wrong
+    (e.g. allowing ``"`` but not ``'``), silently under-counting correct calls.
+
+    This unwraps a ``bash``/``sh``/``zsh -c`` wrapper and resolves shell quoting
+    with ``shlex`` so a pattern can match argv semantics regardless of quoting.
+    Shell operators (``&&``, ``|``, ``>``) survive as their own tokens, so
+    patterns that reference them keep working, and embedded newlines collapse to
+    single spaces. Returns ``None`` when the text can't be parsed (an odd quote
+    count — NOT heredocs, which tokenize fine); the caller then keeps only the
+    raw text as a haystack.
+
+    Adding a second (normalized) haystack is *not* purely additive: a
+    ``command_pattern`` can only gain matches, but the same haystacks feed
+    ``exclude_pattern`` and the ``max_count`` gate, so a normalized form can
+    newly satisfy an exclusion or trip a ``max_count`` cap — i.e. a command that
+    counted on the raw text alone can stop counting. See
+    ``CommandExecutedChecker._matching_commands``.
+
+    Memoized (pure function of ``cmd_text``): the early-stop watcher re-scans the
+    whole accumulated trajectory on every tool-call event, so the same command is
+    normalized many times per run — the cache collapses that to once per distinct
+    (already-truncated) command string.
+    """
+    try:
+        tokens = shlex.split(cmd_text, posix=True)
+    except ValueError:
+        return None
+    if not tokens:
+        return None
+    # Unwrap `bash -lc "<script>"` / `sh -c "<script>"`: the real command is
+    # everything after the -c/-lc flag.
+    if _is_shell_program(tokens[0]):
+        for i in range(1, len(tokens) - 1):
+            tok = tokens[i]
+            if _is_command_flag(tok):
+                rest = tokens[i + 1 :]
+                if len(rest) == 1:
+                    # Quoted whole-script form (`bash -lc "uip ... 'arg' ..."`):
+                    # the script is a single token that may still hold inner
+                    # quotes — re-split to resolve them.
+                    try:
+                        tokens = shlex.split(rest[0], posix=True)
+                    except ValueError:
+                        return None
+                else:
+                    # Argv-joined form: Codex rollout recovery joins argv WITHOUT
+                    # re-quoting (codex_agent.py), so `bash -lc uip is resources ...`
+                    # already arrives split — keep every token instead of
+                    # collapsing to the first word.
+                    tokens = rest
+                break
+            if not tok.startswith("-"):
+                break  # first positional before any -c: not a command wrapper
+    return " ".join(tokens)
+
+
+def _match_haystacks(cmd_text: str, *, is_shell: bool) -> list[str]:
+    """Strings a pattern may match against for one command.
+
+    Always the raw ``cmd_text`` truncated to the ReDoS bound; when ``is_shell``,
+    additionally the quote-resolved, wrapper-stripped form of that **same
+    truncated window** (see :func:`_normalize_shell`). Normalizing the already-
+    truncated slice keeps both haystacks describing the same window, so quote-
+    stripping can never slide content from past the cap into the match, and
+    caps ``shlex`` input at ``_MAX_PATTERN_SEARCH_LEN`` for free. Matching is
+    "either" — a pattern hits the command if it matches ANY haystack.
+
+    ``is_shell`` is decided once by the caller (a Bash tool whose ``command`` is
+    a non-empty ``str``) and passed in, rather than re-derived here from
+    ``tool_name`` alone: a Bash record with a missing/empty ``command`` serializes
+    its params to JSON, where shell tokenization is meaningless, and must NOT be
+    normalized (else stripped JSON quotes could newly satisfy an exclusion).
+    """
+    window = cmd_text[:_MAX_PATTERN_SEARCH_LEN]
+    haystacks = [window]
+    if is_shell:
+        normalized = _normalize_shell(window)
+        if normalized is not None and normalized != window:
+            haystacks.append(normalized)
+    return haystacks
+
+
 @register_criterion
 class CommandExecutedChecker(BaseCriterion[CommandExecutedCriterion]):
     """Checker for CommandExecutedCriterion.
@@ -54,27 +173,39 @@ def _matching_commands(
             if criterion.require_success and cmd.result_status != "success":
                 continue
 
-            # Extract text for pattern matching (Bash: command param; others: JSON-serialized params)
-            if cmd.tool_name == "Bash" and cmd.parameters.get("command"):
-                cmd_text = cmd.parameters["command"]
+            # Extract text for pattern matching (Bash: command param; others: JSON-serialized params).
+            # ``parameters`` is ``dict[str, Any]``, and a ``command`` value is not
+            # guaranteed to be a ``str`` — Codex sub-agent rollout recovery can carry
+            # it as an argv *list* (codex_agent.py). Narrow with ``isinstance`` so a
+            # non-``str`` value never reaches ``shlex.split``/slicing (which would raise
+            # ``AttributeError`` and zero the whole criterion); fall back to the JSON blob.
+            raw_command = cmd.parameters.get("command")
+            if cmd.tool_name == "Bash" and isinstance(raw_command, str) and raw_command:
+                cmd_text = raw_command
+                is_shell = True
             else:
                 cmd_text = json.dumps(cmd.parameters)
+                is_shell = False
 
-            # Truncate to mitigate ReDoS on large command strings
-            if len(cmd_text) > _MAX_PATTERN_SEARCH_LEN:
-                cmd_text = cmd_text[:_MAX_PATTERN_SEARCH_LEN]
+            # Match the pattern against the raw command AND its quote-resolved
+            # form, so authors need not encode shell quoting/escaping (which they
+            # do inconsistently, silently under-counting correctly-quoted calls).
+            # ``is_shell`` (decided once, above) tells the helper whether cmd_text
+            # is a shell command — it must not re-derive that from tool_name alone.
+            haystacks = _match_haystacks(cmd_text, is_shell=is_shell)
 
             # Filter by command pattern
-            if pattern is not None and not pattern.search(cmd_text):
+            if pattern is not None and not any(pattern.search(h) for h in haystacks):
                 continue
 
-            # Apply exclusion pattern (skip commands matching the exclusion)
-            if exclude_re is not None and exclude_re.search(cmd_text):
+            # Apply exclusion pattern (skip commands matching the exclusion).
+            # Same both-haystacks logic so exclusion can't be dodged by quoting.
+            if exclude_re is not None and any(exclude_re.search(h) for h in haystacks):
                 continue
 
-            # Build a display label for the matched command
-            if cmd.tool_name == "Bash" and cmd.parameters.get("command"):
-                label = cmd.parameters["command"]
+            # Build a display label for the matched command (same narrowing as above)
+            if cmd.tool_name == "Bash" and isinstance(raw_command, str) and raw_command:
+                label = raw_command
             else:
                 label = f"{cmd.tool_name}({json.dumps(cmd.parameters)[:80]})"
             matching.append(label)

src/coder_eval/models/criteria.py

@@ -906,7 +906,14 @@ class CommandExecutedCriterion(LiveSuccessCriterion):
     type: Literal["command_executed"] = "command_executed"
     tool_name: str | None = Field(default=None, description="Tool name filter (e.g., 'Bash'). None = any tool.")
     command_pattern: str | None = Field(
-        default=None, description="Regex to match command parameters. None = any command."
+        default=None,
+        description=(
+            "Regex to match command parameters. None = any command. For a Bash "
+            "command the pattern is matched against the raw command text OR its "
+            "shell-normalized form — `shlex`-resolved quoting with a "
+            "`bash`/`zsh -lc` wrapper stripped — whichever hits, so you need not "
+            "hand-encode shell quoting/escaping (`'single'`, `\"double\"`, bare)."
+        ),
     )
     min_count: int = Field(
         default=1,
@@ -930,7 +937,9 @@ class CommandExecutedCriterion(LiveSuccessCriterion):
     exclude_pattern: str | None = Field(
         default=None,
         description=(
-            "Regex that must NOT match. Commands matching both command_pattern and exclude_pattern are skipped."
+            "Regex that must NOT match. Commands matching both command_pattern and exclude_pattern are skipped. "
+            "Like command_pattern, this is matched against the raw Bash command OR its shell-normalized form, so "
+            "it also excludes quote-obfuscated variants of the same call."
         ),
     )