feat(tasks): add AGIEval (21 subsets, 0-shot two-stage) with subset selection

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• Adds AGIEval v1.1 (paper, repo): agieval dataset + agieval_0shot_gen task, all 21 subsets in one class (19 MCQ + 2 math cloze, 7,272 problems — Gaokao, SAT, LSAT, AQuA-RAT, JEC-QA, LogiQA, MATH). The README's "20 tasks" counts sat-en / sat-en-without-passage as one; upstream's own driver evaluates and averages all 21 files.
• Subset selection is the point of the shape, since nobody runs all 21 by accident: args: {group: math} (the 5 math subsets, 1,943 problems) / {group: en-mcq|zh-mcq|all} (upstream's leaderboard groups) / {subsets: [math, sat-math]} (exact names). Selection always concatenates in canonical subset order, so the same choice spelled two ways yields identical sample ids.
• Two model calls per sample, because that is upstream's zero-shot protocol — answer freely, then re-read the reply under an extraction cue and parse that. Upstream's parser is "first A-F character in the reply", which on a chain of thought returns the letter of the first option it restated; dropping stage 2 is a different, worse AGIEval, not a cheaper one. Both ModelOutputs come back from infer() as a list, so both are profiled and stage 1's reasoning stays on disk.
• New task arg extractor (model-config dict or Model) pins stage 2 to a separate model, as upstream did (gpt-35-turbo regardless of the stage-1 model). Unset, stage 2 runs on the model under test — the one protocol-level divergence, and it moves scores; spelled out in reference_impl.notes.
• Data is 21 commit-pinned, per-file-checksummed url: sources: there is no HF mirror that carries the two cloze subsets (hails/*, dmayhem93/* are MCQ-only; lighteval/agi_eval_en is English-only and omits sat-en-without-passage).
• report.json: per-subset score_<subset>, score = macro over the subsets that ran (upstream's "average for all datasets"), and macro_math / macro_en_mcq / macro_zh_mcq only when a whole group ran — a partial macro is not the published number. macro_math stays distinct from score_math, the MATH subset's own accuracy.
• status="experimental": faithful to upstream line by line, but there is no reproduction run of our own yet (see Manual below). Targets recorded in the notes.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass — 43 new (community 21 / datasets 9 / tasks 13); full unit suite 2965 passed
• scripts/check_preflight.py — 21 PASS, no FAIL/WARN (incl. check_datasets pinning/checksums, check_meta_index_sync, check_examples)

Manual

• sieval dataset download agieval stages all 21 files (8.2 MB) and checksum verification passes; re-run is idempotent (already present)
• End-to-end run over the real staged data (group: math, stub chat model, TaskRunner): prompt built from real rows, 2 infer entries persisted per sample, prediction extracted from stage 2 only, judgement + report.json keys as expected (score, score_sat_math)
• Stage-2 spend is profiled, not silently dropped: profile.json records count: 6 inferred calls for 3 samples
• sieval task show agieval_0shot_gen / sieval dataset show agieval resolve and render
• Score comparison against published numbers — NOT DONE, and the reason this ships status="experimental" rather than stable. Upstream's v1.1 zero-shot leaderboard targets are in reference_impl.notes (GPT-4o 62.3 all / 65.2 en / 63.3 zh; GPT-3.5-Turbo 46.0 / 54.1 / 45.0). A validating run needs a real model with extractor pinned, since the stage-2 model is score-relevant.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring (community/agieval/__init__.py is deliberately empty, matching community/ruler/__init__.py — no code to attribute)
• No new upper-layer dependencies added to core/ — core/ untouched
• Deleted code verified — nothing deleted, additive only

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table — absent on purpose; see Manual above. This is the gap between experimental and stable.
• Dataset loading tested (sieval dataset download agieval succeeds)
• Task registered in package-level __init__.py (lazy export map + regenerated __init__.pyi and meta/index.json)

If: community/ Changes

• Upstream diff documented — each vendored module carries the pinned upstream URL plus a "Deltas from upstream" block. Substantive ones:
  • zero-shot only: the few-shot paths (combine_prompt / concat_prompt / convert_few_shot) and their tiktoken budget trimming are not vendored, along with the setting_name parameter and the few-shot-CoT extract_last_line branches it gated. remove_few_shot_prefix stays — upstream calls it unconditionally inside parse_math_answer.
  • unknown subset raises ValueError instead of upstream's try/except NameError returning None (which surfaces later as a TypeError on the prompt).
  • a failed extraction returns None, not "" / [], per the PredictionRecord contract. Verdicts unchanged: neither can equal a gold answer. Same for convert_to_set(None) returning an empty set where upstream returns an empty dict ({}, a typo) — every comparison it feeds reaches the same verdict.
  • regexes are raw strings, byte-identical patterns (upstream's "\$(.*)\$" etc. are plain strings whose invalid escapes Python leaves untouched). The one place this could have gone wrong is called out inline: upstream's second percent-strip is written .replace("\%", ""), i.e. a redundant repeat of the first, not a bare-% strip.
  • is_equiv is vendored again rather than reusing sieval.community.math, which is a trimmed copy of the same hendrycks/math file. The steps it drops (linebreak/space/degree/percent strip, _remove_right_units, leading "k = ", 0.5 -> \frac{1}{2}) change verdicts, so AGIEval scores must come from AGIEval's copy. Pinned by test_is_equiv_applies_the_normalizations_sieval_community_math_drops.
  • Also kept verbatim on purpose: 7 of 351 gaokao-mathqa rows carry multi-letter gold ('AD', 'ACD', 'A B D', …) while the subset is not on upstream's multi_choice list and is scored by exact single-letter compare, so those rows are unwinnable (~2% of that subset). Fixing it would diverge from every published AGIEval number.
• License attribution preserved — Microsoft MIT headers on the three files that carry them upstream; math_equivalence.py credits both AGIEval and the hendrycks/math original it vendors.

Dataset shape notes (not a template section, but the reviewable bit)

Rows keep upstream's field names and nullability (passage / question / options / label / answer / other) and gain subset, which is absent from the raw rows but decides prompt, parsing and scoring. Two normalizations were unavoidable to concatenate 21 files whose per-file inferred schemas disagree, both verdict-neutral and both guarded:

• jec-qa-kd / jec-qa-ca ship label as a 1-element list where every other subset ships a string; the list is unwrapped. A longer list raises rather than silently re-scoring — v1.0 had genuine multi-label rows, and set-comparison semantics would change.
• other.level is an int64 in math.jsonl and absent elsewhere; stringified so the struct has one dtype across subsets. No other column is cast — upstream already ships them as strings.

🤖 Generated with Claude Code