feat(gsm1k): add GSM1k dataset + 5-shot base and 0-shot chat tasks

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• GSM1k (paper, eval repo) is Scale AI's from-scratch re-do of GSM8K: 1205 human-written problems, no LLM assistance, matched to GSM8K's difficulty and answer-magnitude distribution. It is a paired instrument — the GSM8K − GSM1k gap estimates how much of a GSM8K score is memorization — so this lands both halves of the pair, not one protocol.
• gsm1k dataset — hf:ScaleAI/gsm1k@bc09569d, test split only, 1205 rows.
• gsm1k_kshot_base_gen — port of Scale's own lm-eval-harness fork task (gsm1k_scale.yaml, pinned 39294c6f): Question:/Answer: prompt, upstream's single flexible-extract filter, exact_match with its four regexes_to_ignore, its stop list, n_shot=5.
• gsm1k_0shot_gen — the chat half, reusing gsm8k_0shot_gen's DeepSeek-Math protocol verbatim so the two form a prompt-exact pair. This is a different measurement regime from upstream's, not a port of it: no published GSM1k number corresponds to a 0-shot chat score, so read the diff, not the absolute.
• Both status="experimental" — faithful by construction, no alignment run yet (see Test Plan).

Three decisions worth reviewing

1. license="MIT", on a snapshot with an empty card. The paper says the data is withheld, which does not match a public HF snapshot existing, so the release was reconstructed rather than assumed: the paper precommits to release on the earlier of "three open-source models of different lineages reach 95%" or June 2025, its datasheet states "The dataset (yet unreleased) will be released with the MIT license", and the snapshot was uploaded to the ScaleAI org 2025-03-31/04-01. MIT is therefore the datasheet's commitment for the data, not the eval repo's code license.
2. Provenance verified, because an empty card is not evidence. All 50 questions in Scale's gsm1k_public_50.csv (published while the full set was still withheld) appear verbatim in the snapshot's test split with identical answers — 50/50 found, 0 answer mismatches — and the row count matches the paper's stated 1205 exactly.
3. No bare exact_match key. gsm8k_kshot_base_gen spells its strict (#### N) metric exact_match, while upstream GSM1k's only metric is the flexible one. Sharing the name would let a paired diff compare two different extraction rules and read the extraction gap as a memorization gap. Both rules are reported under rule-named keys instead: flexible_exact_match (upstream's filter, and the headline score/correct) and strict_exact_match.

Documented deviations from upstream

• Few-shot exemplars are fixed, not resampled per question. GSM1k ships final answers only (no worked solutions) and has no train split, so it cannot supply chain-of-thought exemplars; upstream draws them from GSM8k train, "which vary for each new question". This task vendors 5 openai/gsm8k train rows, verified byte-identical to shuffle(seed=1234)[:5] — the same 5 gsm8k_kshot_base_gen draws at n_shot=5, fewshot_seed=1234, so the pair can be run prompt-controlled. Holding the prefix fixed also keeps exemplar variance out of the diff, which is what upstream's single shared prompt is reaching for. It does move absolute scores relative to the published column.
• report() divides by len(finals) + len(fails) (this repo's convention, 19 of 22 tasks). gsm8k_kshot_base_gen divides by len(finals), so a paired diff must be read with both fails counts in view; identical when fails == 0.
• max_tokens=1000 is upstream's one deliberate change to lm-eval defaults (raised from 256 so CoT is not truncated). It is a model-layer asset here, recorded in reference_impl.notes.

Both deviations live in the shipped module docstring and reference_impl.notes, not only in this PR.

Test Plan

Automated

• Lint/format clean (ruff check + ruff format)
• Type check clean — ty check passes on all three modules. mypy --strict reports 14 unannotated-stage-method / untyped-community-call errors, the same classes the GSM8K siblings produce (28 on the same axis), i.e. the existing house baseline rather than new debt.
• Unit tests pass — 16 new tests; full suite 3113 passed.
• scripts/check_preflight.py — all checks PASS (registry, naming, shot knobs, record-key access, meta-index sync, imports, examples). --level deep link check: 52/53 reachable, the one WARN is a pre-existing Wikipedia 403.
• scripts/check_layer_imports.py and scripts/sanitize.sh clean.

Manual

• sieval dataset download gsm1k succeeds (6 files from the pinned revision).
• Real-data checks over all 1205 rows: test is the only split; _normalize_exact_match alters 0 golds (upstream's four regexes are no-ops on GSM1k's bare integers); the flexible rule recovers 1205/1205 golds from a plain restatement.
• Rendered 5-shot prompt on real row 0: 2669 chars, 6 Question: blocks, 5 #### N exemplar answers, ends \nAnswer:, reference 133 matches the row's gold.
• Score comparison table — NOT included. This is the one checklist item this PR cannot satisfy: it needs an inference run, which is why both tasks ship status="experimental" rather than stable. Alignment targets to run against, from the paper's Table 1 (GSM8k → GSM1k, 5-shot, temperature 0, no chat template): Meta-Llama-3-8B-Instruct 0.752 → 0.690 (diff 0.062), Meta-Llama-3-70B-Instruct 0.914 → 0.900 (0.014), Mistral-7B-Instruct-v0.2 0.428 → 0.419 (0.009), phi-2 0.566 → 0.504 (0.063). The diff is the quantity to align on; the fixed-exemplar deviation moves the absolute columns.

Checklist

Required (all PRs)

• PR title follows conventional format
• No internal paths, credentials, or personal info in committed files (sanitize.sh clean)
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — nothing deleted

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table — see Test Plan; needs a run, tasks are experimental until then
• Dataset loading tested (sieval dataset download gsm1k succeeds)
• Task registered in package-level __init__.py (stubs + meta/index.json regenerated)

If: New Dependency

• No new dependencies. gsm1k_0shot_gen reuses the existing math extra (README's extras line updated to name GSM1k); gsm1k_kshot_base_gen needs none.

🤖 Generated with Claude Code