feat(tasks): task-name variants, UGMathBench as _fixed, and grading off the shared event loop

.claude/rules/tasks.md

@@ -7,14 +7,31 @@ paths:
 
 ## Naming & Model Type
 
-- File naming must follow `<task>_<N>shot_<mode>.py` pattern (authoritative table in `sieval/tasks/CLAUDE.md`):
+- File naming must follow `<task>_<N>shot_<mode>[_<variant>].py` pattern (authoritative table in `sieval/tasks/CLAUDE.md`):
     - `_gen.py` → `model_type = "chat"`
     - `_base_gen.py` → `model_type = "gen"` (base model, uses GenModel)
     - `_ppl.py` → `model_type = "gen"` (perplexity, uses GenModel)
     - `_clp.py` → `model_type = "gen"` (conditional next-token log-prob, uses GenModel)
-- Class naming: `<Benchmark><ShotType><Mode>Task` — words for shot count (`ZeroShot`, `FewShot`)
+- Class naming: `<Benchmark><ShotType><Mode>[<Variant>]Task` — words for shot count (`ZeroShot`, `FewShot`)
 - `ppl` vs `clp` distinction: see `sieval/tasks/CLAUDE.md`.
 
+### Variants
+
+An optional trailing segment lets two readings of one benchmark coexist as
+separate registered tasks (full rationale in `sieval/tasks/CLAUDE.md`).
+
+- The unqualified name means **what upstream measures, bugs included**. Never
+  repurpose it for a local change.
+- `_fixed` requires a **defect** in upstream's data or grader, not a preference,
+  and owes both: every divergence in `reference_impl.notes`, and a **quantified**
+  score impact.
+- A variant may not spell a mode — `..._clp_gen.py` is rejected.
+- A different **measurement regime** is a mode, not a variant.
+- A fix to **problem text or reference answers** is a `datasets/` concern: a
+  dataset variant applying a patch table over the same pinned revision, never a
+  forked copy. See `sieval/datasets/CLAUDE.md`.
+- Do not coin a new variant name speculatively.
+
 ## Checklist for New Benchmarks
 
 - Add benchmark-specific dependencies to `pyproject.toml` optional dependency groups (e.g., `[project.optional-dependencies.benchmark_name]`)

CONTRIBUTING.md

@@ -92,7 +92,9 @@ See [tests/README.md](tests/README.md) for mock infrastructure and full command
 1. Create dataset in `sieval/datasets/` — keep upstream field names, and cast a column's dtype
    only if the pinned revision requires it (each Task binds 1:1 to its own sample `TypedDict`, so
    uniformity with a sibling loader buys nothing)
-2. Create task in `sieval/tasks/` — file naming: `<task>_<N>shot_<mode>.py` (see `sieval/tasks/CLAUDE.md`)
+2. Create task in `sieval/tasks/` — file naming: `<task>_<N>shot_<mode>[_<variant>].py`
+   (see `sieval/tasks/CLAUDE.md`). The unqualified name tracks upstream, bugs included; a
+   local correction takes a `_fixed` variant and owes a quantified score impact
 3. If the reference implementation repeats sampling (`n_repeats`, `--n 4`) and your task's default
    `n` differs, record that in `reference_impl.notes` with how to match it
 4. Add unit tests under `tests/unit/datasets/` and `tests/unit/tasks/` mirroring the source layout

pyproject.toml

@@ -218,11 +218,23 @@ paths_to_mutate = ["sieval/core"]
 tests_dir = ["tests/unit"]
 exclude = ["sieval/core/**/__init__.py"]
 also_copy = [
+  # `sieval/__init__.py` is load-bearing: without it `mutants/sieval` is a plain
+  # directory rather than a package, so every test in the copy resolves `sieval`
+  # through the editable install instead, and mutmut dies on an unrelated import
+  # before running a single mutant.
+  "sieval/__init__.py",
+  # `scripts/` is not a package, so `tests/unit/scripts/` puts it on `sys.path`
+  # by walking up from its own `__file__`. Inside the copy that resolves to
+  # `mutants/scripts`, so without this entry stats collection dies on
+  # `ModuleNotFoundError: No module named 'check_layer_imports'` — the same
+  # class of failure as the entry above, and just as invisible.
+  "scripts",
   "sieval/community",
   "sieval/datasets",
   "sieval/tasks",
   "sieval/infer",
   "sieval/cli",
+  "sieval/meta",
   "sieval/__main__.py",
 ]
 

scripts/check_preflight.py

@@ -56,8 +56,15 @@
 
 _MAX_DRIFT_DETAILS = 20
 
+# `<task>_<N>shot_<mode>[_<variant>].py`. The mode alternation is anchored right
+# after the shot segment, so `model_type` stays readable off the name even when a
+# variant follows. Multi-token modes lead the alternation, and a variant may not
+# spell a mode (`..._clp_gen.py`) — that name has two readings, so it is rejected
+# rather than resolved by regex precedence.
+_TASK_MODE_ALTERNATION = "base_gen|llmjudge_gen|gen|ppl|clp"
 _TASK_FILE_PATTERN = re.compile(
-    r"^[a-z][a-z0-9_]*_(\d+|k)shot_(gen|base_gen|ppl|clp|llmjudge_gen)\.py$"
+    rf"^[a-z][a-z0-9_]*_(?:\d+|k)shot_(?:{_TASK_MODE_ALTERNATION})"
+    rf"(?:_(?!(?:{_TASK_MODE_ALTERNATION})\.py$)[a-z][a-z0-9_]*)?\.py$"
 )
 _DATASET_SUFFIX_PATTERN = re.compile(r"(Dataset|DatasetSample|CSVSample)$")
 
@@ -576,6 +583,7 @@ class PreflightRunner:
         "check_imports",
         "check_examples",
         "check_meta_index_sync",
+        "check_mutmut_config",
         "check_version",
     ]
 
@@ -1914,6 +1922,75 @@ def _get_latest_git_tag(self) -> str | None:
         except (subprocess.CalledProcessError, FileNotFoundError):
             return None
 
+    def check_mutmut_config(self) -> list[CheckResult]:
+        """Verify ``[tool.mutmut]`` still produces an importable ``mutants/`` copy.
+
+        mutmut runs the suite from a copy of the tree built out of
+        ``paths_to_mutate`` + ``also_copy``. Anything the suite imports that the
+        copy omits makes the run die during stats collection, before a single
+        mutant executes — and it surfaces as an unrelated broken test rather
+        than as a configuration error, which is why this went unnoticed twice.
+
+        Two entries are load-bearing, for the same reason and with the same
+        symptom:
+
+        * ``sieval/__init__.py`` — without it ``mutants/sieval`` is a plain
+          directory rather than a package, so every test in the copy resolves
+          ``sieval`` through the editable install instead.
+        * ``scripts`` — ``scripts/`` is not a package, so ``tests/unit/scripts/``
+          puts it on ``sys.path`` by walking up from its own ``__file__``. In the
+          copy that resolves to ``mutants/scripts``.
+
+        Cheap to assert, invisible otherwise: it made the mutation-score
+        requirement in ``sieval/core/CLAUDE.md`` unsatisfiable for as long as
+        either entry was missing.
+        """
+        check = "check_mutmut_config"
+        pyproject = self.project_root / "pyproject.toml"
+        if not pyproject.exists():
+            return [CheckResult("FAIL", check, "pyproject.toml not found")]
+
+        import tomllib
+
+        config = tomllib.loads(pyproject.read_text(encoding="utf-8"))
+        mutmut = config.get("tool", {}).get("mutmut")
+        if not mutmut:
+            return [CheckResult("PASS", check, "no [tool.mutmut] section to check")]
+
+        copied = list(mutmut.get("also_copy", [])) + list(
+            mutmut.get("paths_to_mutate", [])
+        )
+        # A parent entry ("sieval") carries the file; an exact entry is the
+        # normal case. Anything else means the path is not in the copy.
+        required = {
+            "sieval/__init__.py": "mutants/sieval is not an importable package",
+            "scripts": "tests/unit/scripts/ cannot import the module it tests",
+        }
+        missing = [
+            f"{path!r} ({why})"
+            for path, why in required.items()
+            if not any(
+                path == entry or path.startswith(f"{entry}/") for entry in copied
+            )
+        ]
+        if missing:
+            return [
+                CheckResult(
+                    "FAIL",
+                    check,
+                    "[tool.mutmut] omits a path the suite imports, so every "
+                    "mutation run dies during stats collection",
+                    [*missing, f"also_copy + paths_to_mutate = {copied}"],
+                )
+            ]
+        return [
+            CheckResult(
+                "PASS",
+                check,
+                "[tool.mutmut] copies every path the suite imports",
+            )
+        ]
+
     def check_version(self) -> list[CheckResult]:
         """Check CHANGELOG / git tag / Dockerfile version alignment."""
         results: list[CheckResult] = []