fix(deepseek-math): stop symbolic_equal executing the answer it grades

[ethan-scitix]

Type

• fix — bug fix or alignment correction

Was stacked on #83, which is now merged; this branch has been rebased onto main and the base is main. #83 established the "fidelity stops at execution safety" carve-out this PR applies. The diff is this change only.

Summary

• symbolic_equal executed the answer it was grading, reachable from an extracted model answer via math_equal / is_correct / eval_math — so from both gsm8k_0shot_gen and hendrycks_math_kshot_base_gen. Verified: __import__('os').system(...) supplied as an answer ran, and the sample still graded wrong, so nothing in a run looked unusual.
• There were two independent paths, not one. The issue named the first; the second is the one that matters:
  1. bare parse_expr(s) — default namespace built by exec("from sympy import *", ...), which injects __builtins__;
  2. the raw-string fallback — when both parsers fail upstream returns s, which then reaches N(a), and N sympifies a string with sympy's own default namespace, not the caller's. (Not simplify(a - b) — see Review fixes below.)
• (2) defeats any fix to (1) on its own, because once __import__ resolves, a payload needs no quote — so the quote screen doesn't help either. Guarding only the parse would have moved the hole rather than closed it. Fixed by refusing: an unparseable answer becomes None and the comparison returns False, instead of being handed to sympify under another name.
• Guards are now shared. The three written for the UGMathBench grader move to sieval/community/_sympy_guards.py. Licensed by coupling, not call count: both graders hand model output to the same library under the same threat model, and a new escape route must close in both or one is left open. The ugmathbench side is a pure move — its 82 tests pass unchanged.
• Not a _fixed variant, per the carve-out in fix(theoremqa): evaluate answers instead of executing them #83: the divergence is execution safety only, and it is quantified below.

Related Issues

Refs #77 (the third known site it lists). Also corrects the issue on two points:

• [Bug]: theoremqa evaluates model output with bare eval(); the sandbox is escapable, and closing it is a _fixed-shaped change #77 says deepseek_math is "vendored GPL upstream, so fidelity is the constraint". DeepSeek-Math's code is MIT (LICENSE-CODE) — GPL-3.0 is UGMathBench's judge_rule.py. Nothing here was license-constrained.
• The unbounded-compute half is already handled: both tasks route grading through run_cpu_bound with GRADE_TIMEOUT (from feat(tasks): task-name variants, UGMathBench as _fixed, and grading off the shared event loop #68), and deepseek's own call_with_timeout path is dead code (math_equal is only ever reached with timeout=False). Executing in a worker is still an RCE, but it is not a session stall.

Test Plan

Automated

• Lint/format clean on the files this PR owns (sieval/community/ is pre-commit-excluded; the 1747 pre-existing findings there are untouched)
• Type check clean (ty check)
• Unit tests pass — full tests/unit, 3558 passed (post-rebase onto main); new test__sympy_guards.py, plus execution-safety cases in test_deepseek_math.py
• scripts/check_preflight.py — all pass

Tests pin the premises, not just the fix: that a cleared namespace alone still executes a nested-parse payload (so the quote screen isn't kept on faith), and that a quoteless chr()-built payload is stopped only by the raw-string refusal.

Manual — replay against two full stored runs, in two environments each

corpus	environment	upstream	guarded	verdicts differing
GSM8K 1319 (deepseek-llm-7b-chat)	parse_latex on	63.3813	63.3813	0
GSM8K 1319	parse_latex off	63.3055	63.3055	0
MATH 5000 (Qwen2.5-72B)	parse_latex on	61.2600	61.2600	0
MATH 5000	parse_latex off	60.0200	60.0200	0

12,638 gradings, zero divergences. The parse_latex off rows are the adversarial case, not padding: with the LaTeX parser dead every comparison falls through to the guarded path — 1622 of 5000 on MATH reach the raw-string refusal specifically, and none of them changes a verdict.

Incidental finding worth its own attention

Both stored runs were produced without antlr4-python3-runtime, so parse_latex raised into upstream's bare except: and the symbolic path silently never ran. Disabling it reproduces each stored report.json score exactly (MATH 60.0200 = its recorded 60.02; GSM8K 63.3055 = its recorded 63.3055), which is how the cause was confirmed rather than guessed. It is worth 1.24 pp on MATH, with no signal in any log.

Already pinned in the [math] extra since #31, so main is correct and no change is needed — those runs simply predate the pin. Recorded in both tasks' reference_impl.notes so the next person reading an old number knows why it is low, and pinned by four test_symbolic_equality_still_works cases drawn from the real disagreements.

Review fixes (24ac2c85 + 5ddcdbae)

Prose and metadata only — the guards and the refusal are untouched, and behaviour is unchanged.

• The fallback reached N, not simplify and N. simplify(a - b) never sees the raw text: the subtraction runs first and sympy's arithmetic dunders sympify strictly, so str - str and str - Expr both raise TypeError before simplify is entered (verified both). N alone is the whole vector, and alone it is enough to defeat a parse-only guard — so the case for refusing the fallback is unchanged, but "both sympify" would have pointed the next person hardening this code at the wrong call. Corrected in the module docstring, the inline comment, both tasks' reference_impl.notes and index.json.
• Named the guard edge that can actually flip a verdict. The notes quantified the raw-string refusal but said nothing about the exponent pre-parse, which declines a right-nested ** tower (2**3**2) and an integer exponent above MAX_EXPONENT — both of which upstream evaluates. Unreachable while the antlr4 pin holds, since parse_latex resolves those spellings first, and the measured zero covers the disabled-parse_latex cells, so neither occurs in either stored run. Worth naming because it is reachable in exactly the environment the same note warns about.
• quotes_free now carries evidence for both dialects. It generalized to both graders on the move but cited only UGMathBench's 42,064 quote-free gold slots; deepseek reads LaTeX, where the evidence is the replay above.
• sieval/community/CLAUDE.md no longer contradicts its own contents. The charter said the directory holds "not original code" while _sympy_guards.py is exactly that, and the package-wide ruff/mypy/pre-commit exclusions — which exist to keep vendored code byte-identical — silently cover it, which is the wrong default for the module holding a security boundary. Documents the coupling that earns its place here, how to lint it by hand meanwhile, and the bar for adding more original code. Narrowing the exclusions to the vendored paths is left to its own change.

Checklist

Required (all PRs)

• PR title follows conventional format
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — the guards moved, and ugmathbench.py's call sites were updated with them; no other referent existed

If: community/ Changes

• Upstream diff documented — new "Deviations from upstream" entry in deepseek_math.py, with the measurement
• License attribution preserved — header and pinned-commit reference untouched

If: Breaking Change

Not breaking — no schema change, no metric moves, old result directories read the same.

🤖 Generated with Claude Code