feat(tasks): task-name variants, UGMathBench as _fixed, and grading off the shared event loop

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Three things. The first two were the original scope; the third came out of review and grew to touch every math grader in the repo, so it is called out here rather than left to the diff.

1. a naming convention for task variants;
2. UGMathBench as its first user, ugmathbench_0shot_gen_fixed;
3. grading moved off the shared event loop — a new core/utils/offload.py, and 16 task modules rewired onto it or onto the existing anyio.to_thread pattern.

Scope: 33 files, +3,296 / −85. If review would rather see (3) split out, the seam is clean — it is the last four commits (41c10e59, fa1232b0, 0b218994, 33f0905f) and touches no UGMathBench behaviour.

1 · Task names gain an optional trailing variant segment

<task>_<N>shot_<mode>[_<variant>].py        class <Benchmark><ShotType><Mode>[<Variant>]Task

Upstream's buggy reading of a benchmark and our corrected one need to coexist as separate registered tasks, for a long time. The task name is the registry key and the run-directory name, so a distinct name is the only place that distinction can live — a tasks/fixed/ subpackage would only move the module while @sieval_task(name=...) still decides the key.

• The unqualified name always means "what upstream measures", bugs included, and is never repurposed. Anyone comparing a run against a published number reads the name and stops thinking.
• _fixed is licensed by a defect in upstream's grader or data — not by preference — and owes two things: every divergence enumerated in reference_impl.notes, and its score impact quantified. An unmeasured fork does not get to call itself a fix.
• Two exclusions. A different measurement regime is not a variant (it gets a descriptive mode name, the way arc_challenge_kshot_clp vs _ppl already differ). Fixing problem text or reference answers is a datasets/ concern: a <name>_fixed dataset applying an explicit patch table (id, field, old → new, why) over the same pinned revision, which must fail loudly when a targeted row no longer matches the recorded old. Never a forked copy of the data — a patch table shrinks to empty when upstream fixes the row, and that is the only exit condition a local fix can have.

Purely additive: every existing task name stays valid, and no existing task is renamed. _TASK_FILE_PATTERN anchors the mode alternation right after the shot segment (multi-token modes leading) so model_type stays readable off the name, plus a negative lookahead rejecting a variant that spells a mode — foo_0shot_clp_gen.py has two readings and is rejected outright rather than settled by regex precedence. Nothing else in the tree derives the mode from a filename, so the segment is contained to this lint.

2 · UGMathBench, as ugmathbench_0shot_gen_fixed

• Adds UGMathBench (paper, ICLR 2025 · repo · dataset): dataset + ugmathbench_0shot_gen_fixed, deps_group="math" (no new dependencies), status="stable".
• Each of the 5,061 problems ships three randomized versions, so a sample is a (problem, version) pair and a full run is 15,183 inferences. EAcc (share of problems correct in all three versions) is score, reported with AAcc, CAcc, Delta, relative_delta (upstream's RE, ×100) and per-subject EAcc.
• Grading is per answer slot: a problem declares how many [ANS] placeholders it has and what type each takes (10 types: NV/EX/EQ/INT/TF/MCS/MCM/OE/OL/UOL). A sample is correct only when every slot is.
• The grader is an independent implementation, not a port — a licensing constraint, not a shortcut. UGMathBench is GPL-3.0 and sieval is Apache-2.0, so judge_rule.py/utils.py cannot be vendored the way matharena / DeepSeek-Math / IMO-Bench were. community/ugmathbench.py reimplements the 10 comparison rules on math-verify + sympy-source parsing + substitution equivalence.
• It therefore ships as _fixed, and the unqualified ugmathbench_0shot_gen stays permanently vacant — a faithful port would have to vendor that GPL-3.0 grader.
• The prompt, by contrast, is exact — byte-identical to upstream's raw template on all 15,183 rows.

Related Issues

• [Bug]: theoremqa evaluates model output with bare eval(); the sandbox is escapable, and closing it is a _fixed-shaped change #77 — theoremqa evaluates model output with bare eval() and the guard is escapable. Same defect class as the parse_expr fix here, but closing it changes that benchmark's grading semantics, so it lands as a _fixed variant and is deliberately gated on this PR's convention being exercised first.

Live run — the headline, and how to read it

Qwen3-30B-A3B, thinking on, temperature 0.6 / top_p 0.95 / top_k 20, one rollout per version. All 15,183 versions / 5,061 problems, sglang tp2×dp4 on 8×H100, 75.7M output tokens.

metric	value
EAcc (score)	38.49
AAcc	45.59
CAcc	53.53
Delta	7.10
fails / incomplete_problems	0 / 0
extracted=False	32 / 15,183 (0.21%)

38.49 is protocol-faithful, not this model's mathematical ability. The two are separated by a format tax of roughly 33 EAcc points, and the tax is the benchmark's rather than this port's: only the last \boxed{} counts and a slot-count mismatch scores every slot wrong, while Qwen3 ends multi-part problems with \boxed{a}, \boxed{b}, \boxed{c}. Single-answer rows mismatch 0.30%, multi-answer rows 86.31% — same model, same subjects, so this is formatting, not difficulty. Upstream's judge_rule.py has the identical rule, so reproducing it is the point. Repairing extraction on the same stored responses lifts EAcc to 74.43, which is where a model scoring 72.5 on AIME 2026 in this harness belongs.

The _fixed divergence, measured two ways

Live head-to-head (the one that carries the weight). Upstream's judge run as a local instrument over the same 15,183 stored responses — GPL-3.0 restricts distribution, not use; nothing vendored:

	value
per-sample agreement	95.51%
sieval right, upstream wrong	591 ← where a lenient grader would show up
upstream right, sieval wrong	91 (0.60% of samples)
upstream's own EAcc on these responses	35.55 (below this task's 38.49)

The 591 is the direction that matters. Sampled eyeball: 12 of 12 genuinely correct — 2√(2t+9) vs sqrt(2*4*t+36), 4^20 vs 1.09951E+12, ln(2)/2 vs 0.346573590279973. No false positives. Residual misses have named, non-systematic causes: LaTeX interval notation (\cup, \infty), absolute-value bars, a y = prefix on an EX answer.

Reference replay (weaker, and it is worth saying why). Replaying each row's own reference back as a boxed answer: upstream accepts its own reference on 14,616 / 15,183 (96.27%), this module on 15,160 (99.85%); the 552 disagreeing rows span 192 of 5,061 problems — an EAcc ceiling difference of 3.79 pp, and 548 of the 552 are rows upstream rejects its own reference on, so the gap is repair rather than drift.

But this measurement could not see the largest defect the live run found: replaying a gold as its own answer short-circuits on _squash(pred) == _squash(gold) and never reaches the symbolic path. A self-replay canary exercises the fast path and is silent about exactly the comparison logic it appears to certify. Worth remembering beyond this task.

What promotion to stable rested on

The gate was never "matches upstream" — unreachable by construction here, and it would pin this task to experimental forever. It was evidence that this grader is right:

#	criterion	result
1	extracted=False rate low	met — 0.21%, truncation 0.19%
2	fails is 0	met — 0
3	plausible band vs siblings on same model	explained, not met — the whole remaining gap is the format tax above, attributed two independent ways (the 0.30% vs 86.31% split, and tracking upstream's verifier to within 3 points). This criterion exists to catch an unexplained anomaly
4	sampled false-negative rate	met — via the independent-instrument audit above

Criterion 4 failed at 34.9% on the first run, which is what caught the math_verify.parse LaTeX-reader defect (the dataset's plain-sympy gold 7*sin(pi*x/5)+1 read as 7*s*i*n*(i*p*x)/5 + 1). Fixing it was worth EAcc 34.46 → 38.49, 716 verdicts wrong-to-right and 0 right-to-wrong.

Review fixes (c2f299b5)

Three of these share one cause: the grader treated model output as trusted input, and the report treated a missing grouping key as a missing sample.

Grader

• parse_expr executed model output. Its default global namespace is built by exec("from sympy import *", ...), which injects __builtins__ — so a boxed `__import__('os').system(...)` ran, on the ordinary path (every wrong free-form answer falls through to the substitution pass), while the slot was still reported wrong so nothing looked unusual. _sympy_globals() removes the builtins and keeps the sympy names, which auto_symbol needs to read sin(...) as a function at all.
• A boxed 9^9^9^9 hung the run. ^ is rewritten to ** and sympy exponentiates eagerly, so it asked for a 370-million-digit integer. Grading is synchronous on the event loop, so one such sample froze every concurrent sample, and max_retries cannot recover a hang. _evaluable() screens it with an unevaluated pre-parse (microseconds) and grades the answer wrong. The largest exponent in the pinned references is three digits, so no reachable comparison changes.
• math_equal's numeric retry returned instead of falling through, making the substitution pass unreachable for every pair the LaTeX mangling turns into a number — 2**100 reads as 2, so a correct 2^100 was compared against 2 and lost without the pass this PR added ever running.

Report

• EAcc could exceed AAcc, silently. feedback()'s raw-sample-less branch emitted a judgement with no extra, so report() could not name the problem and dropped it from EAcc's denominator while its wrong verdict stayed in AAcc's — the same upward bias already fixed for failed samples, in the path that had no guard. It now recovers the grouping keys from the prompt record, as _identify does. EAcc <= AAcc is checked and logged, and unattributed_finals records what could not be recovered.

Docs / metadata, no behaviour change

• the two modules disagreed on the same measurement (705/38.43 vs 716/38.49) — the arcsin-alias commit updated only one;
• reference_impl.notes still said the task was experimental, and shipped that way in meta/index.json next to status: stable. Replaced with the live head-to-head, which was missing there;
• the task docstring had grown into a changelog and stated current numbers three contradictory ways. Rewritten to describe what ships;
• two extraction divergences were missing from a list that enumerates them deliberately: upstream's herefore pre-split, and an empty box returning the whole response;
• build_prompt derived the joined type description once for both branches, where upstream's single-answer branch reads answer_type[0] alone. Equal on every pinned row, so this pins the shape rather than the coincidence.

Note: moving grading off the event loop is now done — see the section below. sieval/community/deepseek_math.py has the same parse_expr exposure, but it is a vendored port of GPL upstream where fidelity is the constraint — separate issue.

3 · Grading no longer blocks the session

(41c10e59 · fa1232b0 · 0b218994 · 33f0905f)

Every runner in a session shares one event loop — MultiTaskRunner.arun starts each TaskRunner with tg.start_soon inside a single anyio.run, and EvalSession uses it. Synchronous grading in feedback() therefore stalled every other task too, not just its own samples.

Measured by mirroring that structure (a co-running benchmark doing 50 ms awaited round trips):

	round trips in 12 s
sibling benchmark alone	239
sibling + ugmathbench grading, before	1 (0.4%)
sibling + ugmathbench grading, after	237 (99.2%)

A heartbeat probe puts it plainly: 0 ticks during a grade, where a free loop manages 100/s. Not only on pathological input — an ordinary wrong answer holds the loop ~23 ms, and a full run is ~8,200 of them (~190 s). After the change ugmathbench graded slightly more samples (533 vs 517), since grading now parallelises across workers instead of serialising on the loop.

sieval/core/utils/offload.py adds run_cpu_bound, running a picklable module-level callable in a shared process pool.

A process and not a thread, which was not the first choice:

• math-verify bounds its own parse/verify with signal.SIGALRM, which only arms on the main thread. In a worker thread it does not degrade — it raises ValueError: ... doesn't support threaded environment. Callers wrap that in a broad except, so the whole math-verify strategy would vanish silently and verdicts flip: \frac{1}{2} against 0.5 measured True → False, as did \frac{\pi}{4} and \sqrt{2}.
• Disabling its timeout does make it thread-safe, but math-verify then warns the caller "must provide the logic for timeout interuption yourself" — which a thread cannot do, being uninterruptible. That trades blocking for unbounded.
• A worker process is the main thread of its own process, so timeouts keep working and verdicts are unchanged. spawn, not fork: the parent is an async process with live worker threads.

Degrades rather than fails — if the pool cannot start, work runs inline (the previous behaviour, slow but correct). SIEVAL_OFFLOAD_WORKERS=0 forces that path.

One rule, applied to every grader. Grading does not run on the event loop, and the mechanism follows the grader's own constraint:

grader	tasks	mechanism	why
math-verify	14 tasks — ugmathbench, IMO-AnswerBench, and the 12 competition/MATH tasks (AIME ×3, HMMT ×3, Apex ×2, BRUMO, CMIMC, SMT, MATH-500)	process	its timeouts are signal.SIGALRM-based and it raises off the main thread
deepseek_math (pure sympy)	gsm8k, hendrycks_math	process	thread-safe, but reached with math_equal(..., timeout=False), so nothing else bounds it — criterion 2

An earlier revision wired only ugmathbench, on the grounds that the other twelve cost 0.2–0.7 s of blocked loop per run against its ~190 s. That was a weak line: 0.2 s assumes n=4 while Apex publishes at n=16, so the number moves with the run config — and it came from short integer answers (median 1.4 ms) when the same sample already reached 89 ms, with math-verify's own timeout at 5 s. A single 1.7 s spike was exactly the argument for wiring the DeepSeek graders. ugmathbench was never special; it was just the one measured first.

The edit is mechanical and bounded: that comparison block was already duplicated verbatim in all twelve (verified — one digest across the twelve), so it is extracted once into sieval/tasks/_math_verify.py and imported by all twelve, module-level because a worker pickles the callable by reference. The twelve sites shared a contract that has to change together (gold first, both sides $-wrapped), which is this repo's bar for extracting — but the cost is real and is now stated in that module's docstring: one edit there rotates twelve benchmarks' verdicts at once, where the duplication it replaced let them drift apart deliberately.

One of the fourteen was missed on the first pass: imo_answer_bench grades through verify_math_answer, a community wrapper around math_verify.verify, so a completeness check matching the names parse / verify inside feedback walked straight past it and reported clean. The check is now wrapper-aware — it resolves which sieval/community functions reach math-verify transitively, then asks whether any async stage calls one without an await.

Verdicts checked directly rather than inferred from green tests: extracted function against the old inline logic over integers, fractions, radicals, equivalent-but-differently-written pairs and both empty-string cases — 0 mismatches / 12.

The one honest cost is pool cold start (~2.7 s): a session running only a 30-problem AIME pays more than the 0.2 s it saves. The pool is lazy and shared, so any multi-task session amortises it, and against runs measured in minutes of inference both numbers are noise — not a reason to keep a rule that needs a paragraph to explain.

No wrapper over anyio.to_thread.run_sync. The ten thread-offload sites do unrelated things — shard-index collection, session setup, a subprocess env probe, scicode's target reads — and share no contract that must change together, which is this repo's stated bar for extracting. One real drift did surface while inventorying them: scicode_0shot_gen.py used asyncio.to_thread, which runs on the loop's own executor and so escapes anyio's CapacityLimiter, putting those reads outside the session's thread budget. Switched to anyio.to_thread.run_sync, so all ten share one mechanism and one bound.

Review fixes, round 2 (22b77195 · 98900034 · 2a972dd9 · a66e0af1)

Rebased onto main @ bba57b79 (#81). Two behavioural fixes, two that only
narrow a claim to what the code does. Both behavioural fixes ship a test that
fails against the unfixed code, not just a passing one.

A grading timeout failed the sample instead of scoring it wrong — gsm8k_0shot_gen
and hendrycks_math_kshot_base_gen were the only 2 of the 16 run_cpu_bound call sites with
no except around the await, so TimeoutError propagated out of feedback and the runner
turned it into a failed sample. The accuracy is identical either way — both report() methods
count fails in the denominator — so what changed is what the run says: fails goes non-zero
and an ERROR is logged, which reads as infrastructure breakage, and fails == 0 is one of the
signals this very PR promotes a task on. The other 14 sites, and ugmathbench_0shot_gen_fixed's
own explicit handler, all keep the opposite contract.

A dead offload pool was retried once per sample. _mark_unusable set _pool_failed but left
_pool set, and _get_pool's guard is if _pool is not None or _pool_failed: return _pool — so
the flag only ever stopped the pool being rebuilt, never stopped the dead one being handed back.
Measured: 4 calls after the pool died made 4 submit attempts; now 1. Results were always correct
(the inline fallback runs regardless), so this is the module agreeing with its own warning —
"falls back to the event loop for the rest of this run" — rather than a wrong answer. Same guard,
second case: an executor that outlived a failed CapacityLimiter would have run against anyio's
shared 40-token default instead of its own, silently undoing the admission control that makes
timeout mean "one grade" rather than "grade plus queue". Now both are built, or neither.

_evaluable promised more than it screens. Its docstring asked "would evaluating this
terminate in useful time?" while the first line returns True for anything without **.
from sympy import * puts eagerly-evaluating callables in the parse namespace that need no
exponent to be expensive — measured on the pinned parser, primepi(10**12) takes 48 s and
factorial(1000000) 3.8 s. Enumerating those callees is the same losing game as allowlisting
them in _quotes_free, so the docstring now names the two shapes it does screen and points at
the bound that actually holds (the worker process and GRADE_TIMEOUT). No behaviour change —
the module docstring's narrower wording was already correct.

[tool.mutmut] still could not collect. See the caveat in the Test Plan. also_copy gains
scripts, and check_mutmut_config now asserts both required paths rather than one, since the
failure mode and symptom are identical and only the path differs.

Also corrected in this description, no code involved: the reference-replay figures were the
pre-arcsin-alias ones (the modules had the current numbers, this did not); the mechanism table
still said the DeepSeek graders use anyio.to_thread.run_sync, where 109b0f1c moved them to a
process; the thread-site inventory still counted grading among them (ten sites now, none of them
grading); and the twelve math-verify graders share one sieval/tasks/_math_verify.py rather than
each carrying a copy.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check — 409 files)
• Type check clean (ty check)
• Unit tests pass — 3425 passed, 0 failed (28 new tests added by this PR), rebased onto main @ bba57b79
• Coverage for the new core/ module — 97% (sieval/core/utils/offload.py: 59 stmts,
  1 miss, 14 branches). The single miss is the double-checked-lock re-check inside with _lock:,
  reachable only when another thread wins the race between the two checks; not worth a contrived
  test. Meets the ≥95% bar in sieval/core/CLAUDE.md.
  (pytest --cov dies in this environment on a pyarrow double-registration — it reproduces on
  untouched modules like core/utils/hf.py, so it is the pytest-cov plugin, not this change.
  python -m coverage run --source=sieval -m pytest ... works; both notes added to
  sieval/core/CLAUDE.md.)
• Mutation score for the new core/ module — 71/93 = 76.3%, over the ≥70% bar.
  Caveat, found in review: this number cannot currently be reproduced from a clean
  checkout — mutmut run still dies during stats collection, because also_copy omitted
  scripts as well as sieval/__init__.py (tests/unit/scripts/ puts scripts/ on
  sys.path by walking up from __file__, which inside the copy is mutants/scripts).
  Fixed below, which moves collection from 0 tests to 2,847 — but one blocker remains and it
  is structural, not configuration: test_import_does_not_pull_math_verify spawns a fresh
  interpreter, which re-imports mutmut's injected trampoline and dies on it. So no mutation
  score is claimed for offload.py on this branch, and the ≥70% bar is currently
  unsatisfiable for any core/ change rather than failing. The paragraph below records what
  the assertions were written to pin, which stands on its own.
  Started at 38.7%. The gain is entirely assertions, not new behaviour: the suite exercised the
  module without pinning what it promises. Now asserted — the pool is spawned, never forked
  (a mutation to "fork" fails no behavioural test, it just occasionally deadlocks a run), built
  once and reused, sized by _worker_count(), not reconsidered once disabled, and not retried per
  sample after a failed start; the degradation warnings name their cause, since a diagnostic that
  drops it is what makes a silent fallback silent; and _worker_count pins its clamps (negative
  floors to 0, caps at 8, floors at 1, survives a None cpu_count, leaves a core for the loop).
  The residual 22 are genuinely worker-internal — pool.submit(func, …) → submit(None, …) is
  answered identically by the inline fallback mutmut forces, since it cannot instrument a spawned
  worker.
  Two pre-existing tooling bugs found on the way, both fixed here. [tool.mutmut].also_copy
  omitted sieval/__init__.py, so mutants/sieval was never an importable package (8beafef0);
  and python -m mutmut — the form sieval/core/CLAUDE.md prescribed — double-executes
  mutmut/__main__.py and dies on the first mutant of every module, which is why this
  requirement had never been runnable. Verified by control on core/utils/concurrency.py, which
  predates this branch: identical failure under python -m, 12/16 killed under the console script.
  Neither was introduced by this PR.

Manual

• sieval dataset download ugmathbench succeeds — 16 configs, unpacks to 15,183 rows; sieval task show / sieval dataset show resolve.
• Prompt is byte-identical to upstream on all 15,183 rows, verified by extracting upstream's make_prompt via AST and diffing against build_prompt with the raw template: 15,183 / 15,183, 0 mismatches.
• Full live run — 15,183 versions, fails 0; numbers above.
• Head-to-head against upstream's judge on the same stored responses — 95.51% agreement, 591-to-91.
• Guards verified end-to-end through judge_answers() — the RCE payload creates no file and the power tower returns in 0.11 s, with 0 regressions on the equivalence cases the grader fix was worth.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files (sanitize.sh clean)
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — nothing deleted; the two renames are git mv (task module + its test)

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table — N/A by construction, and that is the point of _fixed. The grader that produced the published numbers is GPL-3.0 and cannot ship here, so comparing this task's EAcc to the paper's would compare two different rulers. The live head-to-head above is the substitute, and it is stronger: it measures this grader against upstream's directly, on real model prose rather than replayed references.
• Dataset loading tested; hf: source revision-pinned (8ab16f0c…, verified to resolve; card licence gpl-3.0 matches the declared license)
• Task registered in package-level __init__.py (lazy export + regenerated __init__.pyi and meta/index.json)

If: community/ Changes

• Upstream diff documented — every known delta enumerated in community/ugmathbench.py's module docstring, summarized in reference_impl.notes
• License attribution preserved — the module header states UGMathBench's GPL-3.0 licence, why no harness code is vendored, and what is reproduced (the prompt protocol), against a commit-pinned URL

If: Engineering Infrastructure (rules / enforcer / hooks)

• Enforcer and rule agree — _TASK_FILE_PATTERN in scripts/check_preflight.py matches the documented convention, with its mirror test in tests/unit/scripts/test_check_preflight.py
• Coupled sites updated in the same commit — .claude/rules/tasks.md, sieval/tasks/CLAUDE.md, sieval/datasets/CLAUDE.md, CONTRIBUTING.md; pre-commit files: scope unchanged
• One existing reject case became legal (foo_5shot_clp_extra.py is now a valid variant) — removed rather than left asserting the old behaviour, and replaced with positive coverage plus mode-shadowing rejects

---

Deliberate data decisions

• Two upstream-corrupt rows are kept, not dropped (Financial_mathematics_0132 v2, Linear_algebra_0306 v3 — problem text replaced by ERROR caught by Translator, empty answer sequence). They are prompted and graded like any other sample and score 0, exactly as upstream does. Dropping them would let the two problems they belong to reach EAcc on 2 of 3 versions. These are data defects, and per the new convention a data fix would be a <name>_fixed dataset with a patch table — deliberately not done here.
• Sample ordering is problem-major (upstream emits version-major), so a problem's three versions are adjacent and slice(n) keeps whole problems. Score is order-independent, so this costs nothing.
• report.json carries incomplete_problems, so a problem judged on fewer than three versions is visible rather than silently counted as a miss.

Caveat on Delta

At temperature 0.6 the reasoning gap is mostly the sampler. Control on Geometry with n=3: Delta 8.70 across the three randomized versions vs 7.25 across three rollouts of one fixed version — only 1.45 pp is version sensitivity, 83% is sampling noise. Upstream generates greedily, where this does not arise. Read Delta only against the sampling settings that produced it.

Budget note

A full run is 15,183 inferences. Grading a wrong answer costs ~25 ms of synchronous sympy per sample; a correct one short-circuits on string equality and is effectively free. datasets.<name>.args.subjects selects a subject subset for smoke tests.

🤖 Generated with Claude Code

[ethan-scitix]

First live run — results, and why the task stays experimental

The docstring gated promotion on a live run and said "Record the numbers here." Done (e0ce467). Docstring + status comment only; no behaviour change.

Setup. Qwen3-30B-A3B, thinking on, temperature 0.6 / top_p 0.95 / top_k 20, one rollout per version. All 15,183 versions / 5,061 problems, sglang tp2×dp4 on 8×H100, 75.7M output tokens. fails 0, incomplete_problems 0.

metric	value
EAcc (score)	34.46
AAcc	40.87
CAcc	48.07
Delta	6.42

Promotion criteria

#	criterion	result
1	extracted=False rate low	pass — 32/15,183 (0.21%); truncation 29 (0.19%)
2	fails is 0	pass — 0
3	plausible band vs siblings on same model	fired — 34.46 on undergrad coursework vs this model's AIME 2026 72.5 / HMMT Feb 2026 51.5 in this harness. Undergrad coursework is not harder than AIME; the criterion caught a measurement problem
4	sampled false-negative rate	FAIL — 34.9%

Criterion 4, and the fix it asks for

Of 1,634 wrong slots in the bucket where extraction and reference agree on slot count, 570 (34.9%) are the grader's error, confirmed by substituting the same random values into both sides' free symbols. Entirely in free-form types — EX 59.6%, NV 25.4%, OE 7/10 — and exactly 0% in every structured type (OL, UOL, MCS, MCM, TF, INT, EQ).

Root cause is in math_equal: math_verify.parse routes everything through a LaTeX reader, so the dataset's plain-sympy gold is mangled —

gold "7*sin(pi*x/5)+1"           -> 7*s*i*n*(i*p*x)/5 + 1     # sin as s·i·n, pi as p·i
pred "7\sin(\frac{\pi}{5}x)+1"   -> 7*sin(pi*x/5) + 1         # parsed fine

A gold containing any function name can then only match by exact string equality.

This is why the 99.81% replay figure could not see it. Replaying a reference as its own answer short-circuits on _squash(pred) == _squash(gold) and never reaches the symbolic path. A self-replay canary validates the fast path only — it is silent about precisely the comparison logic it appears to certify. Worth remembering beyond this task.

Suggested fix: parse the gold as sympy source, or add a numeric-substitution fallback in math_equal. Happy to push it here or as a follow-up — say which. It is worth ~+6.5 EAcc.

How to read 34.46

It is protocol-faithful and paper-comparable, but it is not this model's math ability. Two effects, both quantified off the same stored responses with no extra inference:

• Format tax ≈ +33.5 EAcc — the benchmark's, not ours. Only the last \boxed{} is kept, and a slot-count mismatch scores every slot wrong. Qwen3 writes \boxed{a}, \boxed{b}, \boxed{c} instead of \boxed{a, b, c}. Single-answer rows mismatch 0.30%, multi-answer rows 86.31% — same model, same subjects, so this is formatting, not difficulty. Upstream judge_rule.py has the identical rule, so we reproduce it deliberately.
• The criterion-4 defect ≈ +6.5 EAcc.

Repairing both, exactly (not extrapolated — EAcc is per-problem): 34.46 → 67.97 → 74.43, which does sit next to AIME 72.5 as criterion 3 expects.

Swapping in upstream's verifier is not the answer

Run as a local instrument over the same 15,183 responses (GPL-3.0 restricts distribution, not use; nothing vendored):

grader	EAcc	AAcc	CAcc
this task	34.46	40.87	48.07
upstream official	35.55	42.23	49.63
diff	+1.09	+1.36	+1.56

94.95% per-sample agreement, disagreeing both ways (upstream wins 486, this grader wins 280 — the latter is the unwinnable-slot repair the module documents). It carries the same format tax, recovers little of the equivalence gap, and is unshippable here on licence.

Caveat on Delta

At temperature 0.6 the reasoning gap is mostly the sampler. Control on Geometry with n=3: Delta 8.70 across the three randomized versions vs 7.25 across three rollouts of one fixed version — only 1.45 pp is version sensitivity, 83% is sampling noise. Upstream generates greedily, where this does not arise. Read Delta only against the sampling settings that produced it.

_{Run artifacts, configs and analysis scripts: /volume/ai4s-data/ylsun/ugmathbench-run/. Tests: 72 passed (-p no:asyncio; the plugin collection error in this env is pre-existing on main).}

[ethan-scitix]

Grader fix pushed — bb80669

The criterion-4 defect from the previous comment is fixed and measured.

What was wrong

math_verify.parse runs a LaTeX reader over both sides, and the dataset does not store LaTeX. In LaTeX an unescaped sin is the product s·i·n and pi is p·i:

gold "7*sin(pi*x/5)+1"          ->  7*s*i*n*(i*p*x)/5 + 1     # mangled
pred "7\sin(\frac{\pi}{5}x)+1"  ->  7*sin(pi*x/5) + 1         # fine

So any gold naming a function could only match by exact string equality.

The fix

• _parse_sympy_source — read the gold with sympy's own parser ([...] is grouping in this dialect, ^ is power, e/pi/ln/infinity mean what they look like), plus an implicit-multiplication reading as an extra candidate for predictions like 5 - 5c. Both parsers are tried on both sides; neither side is guaranteed to be in one syntax.
• _same_function — equivalence by numeric substitution, closing the one gap where upstream was genuinely stronger. Catches what no normalizer reaches, e.g. 3\cos(2\sqrt{35}t) vs 3*cos(sqrt(980/7)*t).

Conservative by construction: it runs only after every other strategy said "not equal", so the only verdict it can change is wrong→right. Symbol-name sets must match (an answer in x never matches one in t); every cleanly-evaluating probe must agree and ≥3 must evaluate; rounding past the benchmark's own tolerance stays wrong.

Two details worth review:

• the probe ladder is fixed, not a seeded RNG — a grader must return the same verdict for the same pair regardless of how many comparisons preceded it;
• it is deliberately small-valued: e^{\cosh(4x)} overflows above x≈2, and on a larger ladder every overflowed probe got discarded, leaving too few points and marking a correct exponential answer wrong. That was a real bug in my first cut, caught by re-auditing the residuals.

Measured, on the same 15,183 stored responses (extraction held fixed, so this is the grader's delta alone)

	EAcc	AAcc	CAcc
before	34.46	40.87	48.07
after	38.43	45.52	53.45
delta	+3.97	+4.64	+5.37

705 samples wrong → right, and 0 right → wrong. The zero matters more than the +3.97 — it's the structural safety property, confirmed empirically rather than argued. For reference, upstream's own verifier scores 35.55 on these same responses, so the repaired grader is now above both its previous self and the reference implementation.

13 regression tests added: the LaTeX-vs-sympy pairs, rounding and different-variable cases that must stay wrong, OL/UOL element dispatch, structured types left strict, and determinism across repeated calls. 97 passed, ruff clean.

Still experimental — and this is a maintainer call, not a measurement

1. Criterion (3) is unmet on the raw number. 38.43 vs AIME 72.5, and the entire remaining gap is the format tax, which is the benchmark's last-box rule faithfully reproduced. Whether a protocol-faithful score dominated by a formatting convention counts as "a plausible band" is a decision about what this task is for — no further run settles it.
2. Criterion (4) can no longer be measured the way it was. The audit that found this defect used numeric substitution; the grader now uses numeric substitution. Auditing it with the same method measures a method against itself. A residual audit needs an independent instrument (manual sample, or upstream's judge). A sampled eyeball of what remains wrong is dominated by genuine model errors, with a thin tail of prediction-side parse failures — a y = prefix on an EX answer, a trailing dx.

Happy to take either further if you want it in this PR.

[ethan-scitix]

Promoted to stable — 3c79d86

You were right on the format tax, and it changed the call. Recording why.

Criterion 3 does not block

The tax is upstream's own rule, and reproducing it is this task's design goal. Criterion 3 exists to catch an unexplained anomaly — a mis-wired prompt, a mis-joined gold, a broken extractor. This gap is explained, quantified, and attributed, and the attribution is checked two independent ways:

• single-answer rows mismatch 0.30% vs multi-answer rows 86.31%, same model and subjects — formatting, not difficulty;
• upstream's own verifier scores 35.55 on these responses, i.e. below this task. A broken harness would not track the reference implementation that closely.

So 38.49 is protocol-faithful and paper-comparable; it is just not the model's math ability. That is a caveat in the docstring, not a status blocker.

Criterion 4, re-measured with an independent instrument

My "it can't be measured" was too quick — the circularity is real (the audit that found the defect used substitution; the grader now uses substitution), but upstream's judge_rule.py is an independent implementation of the same spec and can play that role. Over the same 15,183 responses:

	count
agreement	95.51% (was 94.95% pre-fix)
sieval right, upstream wrong	591 ← where a lenient grader would show up
upstream right, sieval wrong	91 ← residual misses, 0.60% of samples

The 591 is the direction that matters. Sampled eyeball: 12 of 12 genuinely correct — 2√(2t+9) vs sqrt(2*4*t+36), 4^20 vs 1.09951E+12, 7/108 vs 0.0648148, ln(2)/2 vs 0.346573590279973, x^0*e^(-8x)cos(9x) vs e^{-8x}cos(9x). No false positives.

Residual misses have named, non-systematic causes: LaTeX interval notation (\cup, \infty), absolute-value bars, a y = prefix on an EX answer.

Also in this commit

Inverse-trig aliases for the sympy-source parser — the dataset spells it arcsin, sympy calls it asin, so without the alias it parsed as a symbol and silently changed the free-symbol set. Worth 11 more samples.

Final grader delta on the stored run (extraction held fixed):

	EAcc	AAcc	CAcc
before	34.46	40.87	48.07
after	38.49	45.59	53.53

716 wrong → right, 0 right → wrong.

meta/index.json regenerated via scripts/sync_meta_index.py (CI checks it). Full suite: 3677 passed; the 2 CLI JSON failures reproduce on pristine HEAD, so they predate this branch.