feat(sandbox): per-source template exclude_patterns and host-side grading convention

docs/TASK_DEFINITION_GUIDE.md

@@ -19,6 +19,7 @@ Complete reference for defining evaluation tasks in Coder Eval.
 - [Sandbox Configuration](#sandbox-configuration)
   - [Recording CLI Invocations](#recording-cli-invocations)
 - [Template Sources](#template-sources)
+- [Grading Assets and the Sandbox Boundary](#grading-assets-and-the-sandbox-boundary)
 - [Success Criteria](#success-criteria)
   - [Continuous Scoring](#continuous-scoring)
   - [file_exists](#file_exists)
@@ -565,6 +566,8 @@ template_sources:
   - type: "template_dir"
     path: "../templates/python-starter"  # Relative to task YAML file
     mount_point: "."                      # Optional: subdir inside sandbox to copy into (default ".")
+    include_patterns: []                  # Optional: copy these even if default-ignored
+    exclude_patterns: []                  # Optional: never copy these
 ```
 
 The framework automatically ignores `.venv`, `.git`, `__pycache__`, `node_modules`, `dist`, `build`, and other common build/cache artifacts (full list: `coder_eval/resources/default_ignore_patterns.yaml`).
@@ -576,6 +579,23 @@ Override the defaults via `sandbox.ignore_patterns` (or `agent.ignore_patterns`
 
 `mount_point` controls where inside the sandbox the template contents land. With `mount_point: "."` (default) files are copied to the sandbox root. With `mount_point: "c"` everything from the source directory ends up under `<sandbox>/c/`. The mount point must be a relative path that stays within the sandbox.
 
+#### Per-source `include_patterns` and `exclude_patterns`
+
+Two optional per-source lists refine which template paths reach the sandbox. Both take template-relative glob patterns matched with `fnmatch`, where `*` does **not** stop at `/`, so `grading/*` also matches `grading/fixtures/expected.json`. A leading `./` is stripped. Absolute paths, `..` segments, and empty entries are rejected at YAML load.
+
+- `include_patterns` - copy a path **even though** it matches a default ignore pattern. Useful for a template that commits a build output under `dist/`. This is author-controlled, so re-including sensitive directories such as `.git` or `.env` is possible; review the patterns you write.
+- `exclude_patterns` - **never** copy a path. Exclusion is terminal: it beats `include_patterns` and beats a `!`-negated `ignore_patterns` entry, so an excluded path cannot be brought back by either.
+
+```yaml
+template_sources:
+  - type: "template_dir"
+    path: "./project"
+    # The agent gets the project; the grading oracle stays on the host.
+    exclude_patterns: ["grading", "grading/*", "**/*.expected"]
+```
+
+`exclude_patterns` is the mechanism for a template that doubles as both the agent's starting point and the source of grading material: ship the working files, withhold the expected outputs. See [Grading Assets and the Sandbox Boundary](#grading-assets-and-the-sandbox-boundary) for where grading material should live in the first place.
+
 ### Inline Starter Files
 
 Define files directly in YAML (ideal for 1–3 files):
@@ -633,6 +653,44 @@ In this example, the `with-context-hint` variant gets the same sandbox as `basel
 
 This pattern is especially useful for A/B testing whether additional context improves agent performance.
 
+## Grading Assets and the Sandbox Boundary
+
+Anything copied into the sandbox is material the agent can read. If a criterion's answer is sitting in the working directory, the task stops measuring the skill it was written for: the agent can satisfy the criterion by reading grading material rather than by doing the work.
+
+**Convention: grading assets live next to the task YAML, not in a template.** Expected outputs, reference data, rubrics, and verifier scripts belong in the task's own directory. The task directory is never copied into the sandbox, so the agent has no path to it, while the framework, which runs host-side, does:
+
+| Surface | How it reaches the task directory |
+|---------|-----------------------------------|
+| `run_command` criteria (and `pre_run` / `post_run`) | The `TASK_DIR` environment variable is set for every command, so `"$TASK_DIR/verifier/check.py"` resolves host-side while the command's working directory stays the sandbox. Commands run through the platform shell, so `$TASK_DIR` expands under `sh` but not under `cmd.exe`; a command that must grade identically on both should read the variable from inside the program it launches instead. |
+| `llm_judge` / `agent_judge` `files:` entries | An entry prefixed with `$TASK_DIR/` is read from the host filesystem relative to the task YAML's parent directory instead of from the sandbox. |
+| `reference:` | `reference.file` is resolved relative to the task YAML and loaded host-side; the reference solution never enters the sandbox. |
+
+A task directory therefore looks like this:
+
+```text
+tasks/my-task/
+├── task.yaml            # references ./verifier and ../templates/my-project
+├── verifier/            # grading only - stays on the host
+│   └── check.py
+└── expected/
+    └── report.json
+```
+
+**When the sandbox copy and the grading copy are the same files.** Some tasks ship tests on purpose - TDD-style, the tests *are* the spec the agent codes against. That is fine, but grade against the pristine host copy, not the sandbox copy the agent can edit. `tasks/fibonacci_with_template.yaml` is the worked example: the template ships `tests/` into the sandbox, and the criterion resolves the same tests under `TASK_DIR` and runs that host copy against the sandbox's `src/`, so rewriting the sandbox copy cannot move the bar.
+
+Where a template mixes working files and grading files in one tree, [`exclude_patterns`](#per-source-include_patterns-and-exclude_patterns) withholds the grading paths from the copy while the rest of the template still ships.
+
+### Test-data separation and its limits
+
+These mechanisms are hygiene, not containment. Under the `tempdir` driver the agent runs as the host user with the host filesystem in reach, so nothing here prevents a determined agent from reading a file outside its working directory. What they do is remove every path the agent is *pointed at*: the criterion's answer is no longer in the working directory, no longer named in the prompt, and no longer a plausible thing to stumble over while working.
+
+Two separate efforts cover what this does not:
+
+- **Containment** - the `docker` driver's UID/GID isolation, which makes host paths genuinely unreachable rather than merely unadvertised.
+- **Detection** - transcript-level analysis of what the agent actually read during a run.
+
+Treat unintended access to grading material as a task-authoring defect: if a criterion can be satisfied without doing the work, the exposure is in the task, and moving the asset host-side is the fix.
+
 ## Success Criteria
 
 Every task needs at least one success criterion. The framework supports 14 criterion types.

src/coder_eval/models/templates.py

@@ -6,7 +6,7 @@
 from abc import ABC
 from typing import Annotated, Any, Literal
 
-from pydantic import BaseModel, ConfigDict, Field, field_validator
+from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator
 
 
 class StarterFile(BaseModel):
@@ -56,6 +56,16 @@ class TemplateDirSource(BaseTemplateSource):
             "see `_matches_template_include_pattern`."
         ),
     )
+    exclude_patterns: list[str] = Field(
+        default_factory=list,
+        description=(
+            "Template-relative glob patterns that are never copied into the sandbox. Exclusion is "
+            "terminal: it beats `include_patterns` and beats a `!`-negated default ignore pattern, "
+            "so an excluded path cannot be brought back by either. Use it to keep grading material "
+            "(expected outputs, reference solutions, verifier fixtures) out of the agent's working "
+            "copy. `*` does not stop at `/`, see `_matches_template_exclude_pattern`."
+        ),
+    )
 
     @field_validator("mount_point")
     @classmethod
@@ -69,17 +79,18 @@ def _validate_mount_point(cls, v: str) -> str:
             raise ValueError(f"mount_point must not contain '..' segments, got: {v!r}")
         return v
 
-    @field_validator("include_patterns")
+    @field_validator("include_patterns", "exclude_patterns")
     @classmethod
-    def _validate_include_patterns(cls, v: list[str]) -> list[str]:
+    def _validate_patterns(cls, v: list[str], info: ValidationInfo) -> list[str]:
+        field = info.field_name
         for pattern in v:
             if not pattern:
-                raise ValueError("include_patterns entries must not be empty")
+                raise ValueError(f"{field} entries must not be empty")
             if os.path.isabs(pattern) or pattern.startswith(("/", "\\")):
-                raise ValueError(f"include_patterns entries must be relative, got: {pattern!r}")
+                raise ValueError(f"{field} entries must be relative, got: {pattern!r}")
             parts = pattern.replace("\\", "/").split("/")
             if any(p == ".." for p in parts):
-                raise ValueError(f"include_patterns entries must not contain '..' segments, got: {pattern!r}")
+                raise ValueError(f"{field} entries must not contain '..' segments, got: {pattern!r}")
         return v
 
 

src/coder_eval/sandbox.py

@@ -354,6 +354,12 @@ def _apply_template_dir_source(self, source: TemplateDirSource) -> None:
         for item in template_path.rglob("*"):
             # Calculate relative path
             rel_path = item.relative_to(template_path)
+            # Exclusion is terminal and is therefore checked first: an excluded path is
+            # never copied, so it beats `include_patterns` and beats a `!`-negated
+            # default ignore pattern. This is how a task keeps grading material out of
+            # the agent's working copy while still shipping the rest of the template.
+            if self._matches_template_exclude_pattern(rel_path, source.exclude_patterns):
+                continue
             # Match ignore patterns against the template-relative path only —
             # checking the absolute path would let an ancestor directory named
             # `dist`, `build`, `env`, `venv`, or `node_modules` filter out the
@@ -620,6 +626,17 @@ def _matches_template_include_pattern(self, rel_path: Path, include_patterns: li
                 return True
         return False
 
+    def _matches_template_exclude_pattern(self, rel_path: Path, exclude_patterns: list[str]) -> bool:
+        """Return whether a template-relative path is withheld from the sandbox copy.
+
+        Matching is identical to :meth:`_matches_template_include_pattern`: the
+        same ``fnmatchcase`` form where ``*`` does not stop at ``/``, so
+        ``grading/*`` also withholds every descendant of ``grading/``. Only the
+        decision differs: an exclude match is terminal, so the path is dropped
+        regardless of ``include_patterns`` or ignore-pattern negations.
+        """
+        return self._matches_template_include_pattern(rel_path, exclude_patterns)
+
     def _setup_virtualenv(self) -> None:
         """Create a Python virtual environment in the sandbox."""
         if not self.sandbox_dir:

tasks/fibonacci_with_template.yaml

@@ -26,7 +26,20 @@ initial_prompt: |
   Make all tests pass.
 
 success_criteria:
+  # The template ships `tests/` into the sandbox on purpose -- they are the spec
+  # the agent codes against. Grading therefore runs the PRISTINE host copy of the
+  # same tests against the sandbox's `src/` (cwd stays the sandbox), so edits to
+  # the sandbox copy cannot move the bar.
+  #
+  # TASK_DIR is read via os.environ rather than shell `$TASK_DIR` so the command
+  # behaves identically under sh and cmd.exe. `-c` pins the template's own
+  # pytest.ini as the config, so the enclosing repo's pytest settings are not
+  # inherited; `-B` and `-p no:cacheprovider` keep the run from writing bytecode
+  # or a cache directory into the host template.
   - type: run_command
-    description: "All tests pass"
-    command: "python -m pytest tests/"
+    description: "All tests pass (graded against the pristine host copy of tests/)"
+    command: >-
+      python -B -c "import os, pathlib, sys, pytest;
+      tpl = pathlib.Path(os.environ['TASK_DIR']).parent / 'templates' / 'fibonacci-starter';
+      sys.exit(pytest.main(['-p', 'no:cacheprovider', '-c', str(tpl / 'pytest.ini'), str(tpl / 'tests')]))"
     timeout: 60

tasks/mock_path_dirs_template_dir/mock-cli-bins/README.md

@@ -18,3 +18,8 @@ every plain file under `mocks/` executable and prepends the resolved absolute pa
 to the agent subprocess `PATH`. The agent can then invoke the bare command names
 (`say_hello`, `echo_args`) and the mock implementations win the lookup ahead of any
 real binary on the host.
+
+Each mock composes its output line at runtime from the invocation (argument count
+and argument lengths) rather than echoing a fixed signature. The task's criteria
+assert on those computed receipts, so the expected text is not present verbatim in
+any file that ships into the sandbox and the mocks have to actually be executed.

tasks/mock_path_dirs_template_dir/mock-cli-bins/mocks/echo_args

@@ -1,5 +1,13 @@
 #!/bin/sh
-echo "ECHO_ARGS_OK count=$#"
+# Same runtime-derived receipt shape as say_hello: the header counts the
+# arguments and their total length, and each argument is echoed with its own
+# length. None of the asserted lines exist verbatim in this file.
+argc=$#
+argchars=0
 for arg in "$@"; do
-    echo "  arg: $arg"
+    argchars=$((argchars + ${#arg}))
+done
+printf 'ECHO_ARGS_OK argc=%s argchars=%s\n' "$argc" "$argchars"
+for arg in "$@"; do
+    printf 'arg[%s]=%s\n' "${#arg}" "$arg"
 done

tasks/mock_path_dirs_template_dir/mock-cli-bins/mocks/say_hello

@@ -1,2 +1,11 @@
 #!/bin/sh
-echo "MOCK_PATH_OK from say_hello, args=$*"
+# The receipt line is assembled at RUNTIME from the invocation (argument count
+# and total argument length), so the exact text the task's criteria assert on
+# appears nowhere in this file. Reading the source is not a substitute for
+# executing the mock.
+argc=$#
+argchars=0
+for arg in "$@"; do
+    argchars=$((argchars + ${#arg}))
+done
+printf 'MOCK_PATH_OK say_hello argc=%s argchars=%s\n' "$argc" "$argchars"

tasks/mock_path_dirs_template_dir/task.yaml

@@ -19,8 +19,7 @@ sandbox:
 
 initial_prompt: |
   Two mock CLI binaries (`say_hello`, `echo_args`) have been placed on the agent's
-  PATH via the sandbox's mock_path_dirs feature. Their sources live under `mocks/`
-  in the working directory.
+  PATH via the sandbox's mock_path_dirs feature.
 
   Run, in this order, capturing stdout to the named files (do not write the files
   yourself -- they must be produced by shell redirection from the command):
@@ -30,18 +29,22 @@ initial_prompt: |
 
   Confirm afterwards that both output files exist.
 
+# Each mock composes its receipt at runtime from the invocation (argument count
+# and argument lengths), so none of the asserted lines exist verbatim in the
+# scripts that ship into the sandbox. Reading the mock sources is therefore not a
+# shortcut to satisfying these criteria -- the mocks have to be executed.
 success_criteria:
   - type: "file_exists"
     path: "say_hello.out"
     description: "say_hello.out must be produced by the say_hello mock's stdout."
   - type: "file_contains"
     path: "say_hello.out"
-    includes: ["MOCK_PATH_OK from say_hello, args=world"]
-    description: "say_hello.out must contain the mock's signature line with the passed arg."
+    includes: ["MOCK_PATH_OK say_hello argc=1 argchars=5"]
+    description: "say_hello.out must contain the receipt say_hello computes from its arguments."
   - type: "file_exists"
     path: "echo_args.out"
     description: "echo_args.out must be produced by the echo_args mock."
   - type: "file_contains"
     path: "echo_args.out"
-    includes: ["ECHO_ARGS_OK count=3", "arg: one", "arg: two", "arg: three"]
-    description: "echo_args.out must contain the mock's args breakdown."
+    includes: ["ECHO_ARGS_OK argc=3 argchars=11", "arg[3]=one", "arg[3]=two", "arg[5]=three"]
+    description: "echo_args.out must contain the receipt echo_args computes from its arguments."

templates/fibonacci-starter/pytest.ini

@@ -0,0 +1,4 @@
+; Pins this template as its own pytest rootdir so a pytest run rooted at these
+; tests never inherits configuration from an enclosing project (the coder_eval
+; repo's own pyproject.toml, for example). Intentionally empty otherwise.
+[pytest]