feat(tasks): add GSM-Plus (dataset + 0-shot CoT task)

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Important

Blocking question for review: upstream's code repo has no license.
qtli/GSM-Plus ships no LICENSE/COPYING/NOTICE file, no SPDX metadata
(gh api repos/qtli/GSM-Plus → "license": null), and no license statement in
its README — so by default it is all-rights-reserved, and vendoring it into
Apache-2.0 sieval is not clearly permitted. This is the same class of gate as the
UGMathBench GPL case, where the answer was "reimplement, don't vendor".
Scope, so the call is informed: of the 286 lines in
sieval/community/gsm_plus.py, the bulk carries independent permissive
provenance — SUBSTITUTIONS / REMOVED_EXPRESSIONS / normalize_final_answer
are Minerva (Lewkowycz et al., per upstream's own attribution comment, and also
shipped in lm-evaluation-harness under Apache-2.0), and delete_extra_zero is
MetaMath-lineage (MIT). The GSM-Plus-original surface is ~80 lines:
extract_gold_ans, extract_pred_ans's cot branch, check_sympy_equivalence,
and extract_pred_ans_none + its _NONE_PATTERNS refusal list (the genuinely
creative part). Reimplementing just that surface is tractable if we want the
UGMathBench treatment. Do not merge until this is decided. Note the dataset
is separately and clearly CC-BY-SA-4.0, and is referenced, not redistributed.

• Adds GSM-Plus (paper, ACL 2024; code
  qtli/GSM-Plus@3474129e;
  data qintongli/GSM-Plus@3b708db5):
  GSM8K's 1319 test problems rewritten under 8 adversarial perturbations, so a
  model's GSM8K score can be compared against the same problems perturbed.
  10552 rows; upstream's testmini (2400) via eval_split.
• Why it needs its own extractor, not GSM8K's: extraction dispatches on
  perturbation_type, because the critical thinking perturbation deletes a
  quantity the question needs — its gold answer is the literal string "None"
  ("unanswerable"), scored on refusal phrasing rather than on a number. A
  numeric-only scorer silently zeroes all 1319 of those rows (⅛ of the benchmark)
  while still reporting a plausible overall score.
• report.json carries overall accuracy, a per-perturbation breakdown, and
  score_wo_critical_thinking (upstream's gsmplus_wo_ncr) — the paper leads
  with both.
• status="experimental", for two independent reasons: the licensing question
  above, and no live-model run yet (validation is replay-based, see Manual below).
• No new dependencies — reuses the existing math extra, whose pinned ANTLR
  runtime is load-bearing here (see the score table).

Related Issues

None.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean — ty check (the project's primary checker) passes.
  mypy sieval reports 28 unannotated-def findings in the new files, in line
  with the precedent it mirrors (deepseek_math.py 52, gsm8k.py 1,
  gsm8k_0shot_gen.py 7) against 1347 pre-existing repo-wide.
• Unit tests pass — 2954 passed, including 32 new ones
  (tests/unit/tasks/test_gsm_plus_0shot_gen.py,
  tests/unit/datasets/test_gsm_plus.py).
• scripts/check_preflight.py --level deep exits 0 (sole WARN is a
  pre-existing Wikipedia 403, unrelated).
• sync_meta_index.py --check and sync_package_stubs.py --check both clean.

Manual

• Replay validation against upstream's own stored predictions. Upstream
  ships results/gpt-3.5-turbo.json, which persists its own gold, pred
  and result per sample. Replaying all 10552 items through this PR's real
  extraction + grading reproduces upstream's gold and pred on
  10552/10552 and its verdict on 10527/10552 (99.76%).
• sieval dataset download gsm_plus succeeds; sieval task show gsm_plus_0shot_gen reports Ready: yes.
• End-to-end on the real snapshot: correct splits (test 10552 / testmini
  2400), 8×1319 type balance, interleaved order confirmed (so a slice of a
  multiple of 8 stays perturbation-balanced); prompts render; a #### 27
  rollout and a refusal rollout each score correctly on their row.
• Live-model run — not done. No served model in this environment. This is
  the reason status="experimental" stays until a run reproduces a published
  model's full 8-cell table within a stated band.

Score comparison (GPT-3.5-Turbo, 0-shot CoT — replay-derived, not a live run)

Cell	Upstream published	This port	Diff
Overall (gsmplus)	61.19	61.43	+0.24
Excl. critical thinking	63.18	63.45	+0.27
integer-decimal-fraction conversion	62.32	63.84	+1.52
reversing operation	55.19	55.42	+0.23
numerical substitution	69.52	69.60	+0.08
digit expansion	70.36	70.43	+0.07
adding operation	48.45	48.45	0.00
distraction insertion	62.17	62.17	0.00
critical thinking	47.31	47.31	0.00
problem understanding	74.22	74.22	0.00

Max diff +1.52pp, inside the template's <3% target, and one-directional.

Every diff is upstream's environment, not its logic. All 25 disagreements have
upstream False / this port True, and all 25 are pairs that are genuinely equal
(3/1 vs 3, 7/20 vs 0.35, 2.45 vs 2450/1000). Upstream's
requirements.txt pins sympy==1.12 and no antlr4-python3-runtime, which
parse_latex requires — so in the authors' environment every parse_latex call
raised, and check_sympy_equivalence's bare except: silently degraded it to
string equality. The published table is therefore a string-equality table. sieval's
[math] extra does pin that runtime, so the same vendored code reaches its
symbolic branch. 20 of the 25 land on integer-decimal-fraction conversion, which
is exactly the perturbation whose purpose is rewriting integers as decimals and
fractions — i.e. where string vs. symbolic equality disagree most.

Reproducing the published digits exactly would mean deliberately breaking
parse_latex — bespoke logic diverging from upstream source, worse on both the
fidelity and the correctness axis — so the port stays faithful and the gap is
documented in the module docstring and reference_impl.notes.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ — core/ untouched
• Deleted code verified — nothing deleted; this PR is additive only

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included (model, expected, actual, diff)
• Dataset loading tested (sieval dataset download gsm_plus succeeds)
• Task registered in package-level __init__.py — via the auto-discovery
  registry; generated __init__.pyi stubs and meta/index.json regenerated
  and committed.

If: community/ Changes

• Upstream diff documented — the module docstring enumerates every deviation:
  only the prompt_type == "cot" branch ported (the other ten multiplex
  prompting techniques, three of which exec() model-generated Python);
  mv == 1 only (no cot_sc 5-sample majority vote); extract_gold_ans
  raises ValueError where upstream calls pdb.set_trace(); regex literals
  spelled as raw strings (identical string values, no SyntaxWarning).
  The task docstring additionally documents the unported confusion
  matrix / decay rate, which need a paired GSM8K run — the seed_* columns
  they need are preserved on every sample.
• License attribution preserved — CANNOT BE CHECKED. There is no upstream
  license to attribute; see the blocking note at the top of Summary. The
  module header does pin the exact source commit and file, and upstream's own
  Minerva attribution comment is preserved verbatim.

If: New Dependency

N/A — no new dependencies.