feat(tasks): add AGIEval (21 subsets, 0-shot two-stage) with subset selection

examples/agieval-math.yaml

@@ -0,0 +1,66 @@
+# ------------------------------------------------------------------------------
+# AGIEval — math subsets only (0-shot, two-stage)
+# ------------------------------------------------------------------------------
+# AGIEval ships 21 subsets across two languages and two answer formats. This runs
+# the 5 drawn from math exams — sat-math, aqua-rat, gaokao-mathqa, math,
+# gaokao-mathcloze (1,943 problems) — selected with `group: math`.
+#
+# Other selections:
+#   args: {group: all}                      # all 21 (7,272 problems)
+#   args: {group: en-mcq}                   # AGIEval-en leaderboard group (8)
+#   args: {group: zh-mcq}                   # AGIEval-zh leaderboard group (11)
+#   args: {subsets: [math, sat-math]}       # exact names, any combination
+#
+# Two-step flow:
+#   1. sieval dataset download agieval
+#   2. sieval eval agieval-math.yaml
+#
+# Each sample costs TWO model calls: the model answers, then re-reads its own
+# answer under an extraction cue and that short reply is what gets scored. This
+# is AGIEval's own zero-shot protocol, not an add-on — see the task's
+# reference_impl.notes.
+#
+# Edit these fields to match your setup:
+#   models.local-model.infer.checkpoint   — your model's on-disk path
+#   models.local-model.infer_meta.image   — swap to your own image if you need
+#                                           custom deps, or drop the block to run
+#                                           without a container
+# ------------------------------------------------------------------------------
+
+result_dir: ./outputs/agieval-math
+
+models:
+  local-model:
+    args:
+      concurrency_limit: 32
+      temperature: 0.0
+    infer:
+      backend: sglang
+      checkpoint: /path/to/your/model        # EDIT ME
+    infer_meta:
+      gpu: H100-80G
+      image: lmsysorg/sglang:latest
+
+datasets:
+  agieval_math:
+    class: AGIEvalDataset
+    path: "${SIEVAL_DATA_DIR}/agieval"   # `sieval dataset download agieval` stages this
+    args:
+      group: math
+
+tasks:
+  agieval_0shot_gen:
+    class: AGIEvalZeroShotGenTask
+    dataset: agieval_math
+    model: local-model
+    # Answer extraction (stage 2) runs on the model under test by default.
+    # Upstream instead ran it on a fixed cheap model; pin one here to match, and
+    # note the score depends on which you choose:
+    # args:
+    #   extractor:
+    #     model: gpt-3.5-turbo
+    #     api_base: https://api.openai.com/v1
+    #     temperature: 0.0
+    infer_args:
+      # Cloze subsets need room to work; MCQ subsets stop long before this.
+      max_tokens: 4096

sieval/community/agieval/dataset_loader.py

@@ -0,0 +1,169 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+# adapted from https://github.com/ruixiangcui/AGIEval/blob/84ab72d94318290aad2e4ec820d535a95a1f7552/src/dataset_loader.py
+"""AGIEval subset taxonomy + zero-shot prompt construction.
+
+Two things upstream keeps in ``src/dataset_loader.py`` and every other AGIEval
+module keys off:
+
+* the **subset families** — four disjoint tuples (english/chinese × qa/cloze)
+  that decide prompt language and answer-parsing rules, plus the two scoring
+  overrides (``MULTI_CHOICE_SUBSETS``, ``MATH_OUTPUT_SUBSETS``);
+* the **zero-shot prompt** (``convert_zero_shot``) and the **second-stage
+  answer-extraction prompt** (``generate_second_stage_input``). Upstream's
+  zero-shot protocol is two calls: the model answers freely, then a second call
+  re-reads its own answer under a "the answer is" cue so a short, parseable
+  letter/value can be extracted. Both stages are needed to reproduce AGIEval's
+  published zero-shot numbers.
+
+Kept out on purpose: the few-shot paths (``combine_prompt`` / ``concat_prompt``
+/ ``convert_few_shot``) and their ``tiktoken`` budget trimming — sieval ships
+the zero-shot task only, and dead vendored code rots.
+
+Deltas from upstream, all deliberate:
+
+* ``convert_zero_shot`` / ``generate_second_stage_input`` are per-sample
+  functions here (upstream's operate on whole files) and take the subset name
+  first; the emitted strings are byte-identical.
+* An unknown subset raises ``ValueError``. Upstream wraps the family dispatch in
+  ``try/except NameError`` and returns ``None`` for a name in no family, which
+  surfaces much later as a ``TypeError`` on the prompt.
+* ``MATH_SUBSETS`` is a **sieval** grouping, not upstream's; see its comment.
+
+AI-Generated Code - Claude Opus 5 (1M context) (Anthropic)
+"""
+
+from collections.abc import Mapping
+
+# Subset families, verbatim from upstream (order included: it is the order the
+# families are declared in, and sieval reuses it as the canonical subset order).
+# `gaokao-english` sits in the ENGLISH family because its *prompt* is English —
+# the exam is Chinese, and upstream's own leaderboard averages count it as
+# Chinese. Do not conflate the two groupings; see evaluation.py.
+ENGLISH_QA_SUBSETS: tuple[str, ...] = (
+    "lsat-ar",
+    "lsat-lr",
+    "lsat-rc",
+    "logiqa-en",
+    "sat-math",
+    "sat-en",
+    "aqua-rat",
+    "sat-en-without-passage",
+    "gaokao-english",
+)
+CHINESE_QA_SUBSETS: tuple[str, ...] = (
+    "logiqa-zh",
+    "jec-qa-kd",
+    "jec-qa-ca",
+    "gaokao-chinese",
+    "gaokao-geography",
+    "gaokao-history",
+    "gaokao-biology",
+    "gaokao-chemistry",
+    "gaokao-physics",
+    "gaokao-mathqa",
+)
+ENGLISH_CLOZE_SUBSETS: tuple[str, ...] = ("math",)
+CHINESE_CLOZE_SUBSETS: tuple[str, ...] = ("gaokao-mathcloze",)
+
+#: Answers are compared as *sets* of letters, not strings (upstream's
+#: `multi_choice_datasets`). `gaokao-physics` stayed on this list after v1.1
+#: made its labels single-answer — kept, because set-vs-set and string-vs-string
+#: agree on single letters.
+MULTI_CHOICE_SUBSETS: tuple[str, ...] = ("jec-qa-kd", "jec-qa-ca", "gaokao-physics")
+
+#: Answers are compared by math equivalence, not string equality (upstream's
+#: `math_output_datasets`). Same membership as the two cloze families.
+MATH_OUTPUT_SUBSETS: tuple[str, ...] = ("gaokao-mathcloze", "math")
+
+#: All 21 data files under `data/v1_1`, in upstream's family-declaration order.
+#: The AGIEval paper and README say "20 tasks" — they count `sat-en` and
+#: `sat-en-without-passage` as one task in two prompt variants, while upstream's
+#: own driver script evaluates and averages over all 21 files.
+SUBSETS: tuple[str, ...] = (
+    ENGLISH_QA_SUBSETS + CHINESE_QA_SUBSETS + ENGLISH_CLOZE_SUBSETS + CHINESE_CLOZE_SUBSETS
+)
+
+#: sieval-defined grouping — upstream has no "math" group. The five subsets
+#: drawn from mathematics exams: SAT math, AQuA-RAT algebraic word problems,
+#: Gaokao math (MCQ + cloze), and MATH competition problems. Everything else in
+#: AGIEval is language/logic/law/science.
+MATH_SUBSETS: tuple[str, ...] = (
+    "sat-math",
+    "aqua-rat",
+    "gaokao-mathqa",
+    "math",
+    "gaokao-mathcloze",
+)
+
+_OPTION_LETTERS = "ABCDEFG"
+
+# Second-stage cue per family, verbatim from `generate_second_stage_input`
+# (with_format_prompt=False, the setting upstream's run_prediction.py uses).
+# The hardcoded "A through E" / "A到D" do not track the actual option count —
+# upstream's text, kept as-is.
+_SECOND_STAGE_CUES: tuple[tuple[tuple[str, ...], str], ...] = (
+    (ENGLISH_QA_SUBSETS, "Therefore, among A through E, the answer is"),
+    (CHINESE_QA_SUBSETS, "因此，从A到D, 我们应选择"),
+    (ENGLISH_CLOZE_SUBSETS, "Therefore, the answer is"),
+    (CHINESE_CLOZE_SUBSETS, "因此，答案是"),
+)
+
+
+def zero_shot_prompt(subset: str, row: Mapping) -> str:
+    """Upstream ``convert_zero_shot(line, dataset_name)`` for one row.
+
+    *row* needs ``passage`` / ``question`` / ``options`` (the AGIEval sample
+    fields); ``options`` may be empty for the cloze subsets, which do not use it.
+    """
+    passage = row["passage"] if row["passage"] is not None else ""
+    question = row["question"]
+    options = row["options"] or []
+
+    if subset in ENGLISH_QA_SUBSETS:
+        count = len(options)
+        if count == 1:
+            count = 5
+        return (
+            passage
+            + "Q: "
+            + question
+            + " "
+            + "Answer Choices: "
+            + " ".join(options)
+            + "\n"
+            + "A: Among A through {}, the answer is".format(_OPTION_LETTERS[count - 1])
+        )
+    if subset in CHINESE_QA_SUBSETS:
+        count = len(options)
+        if count == 1:
+            count = 4
+        return (
+            passage
+            + "问题："
+            + question
+            + " "
+            + "选项："
+            + " ".join(options)
+            + "\n"
+            + "答案：从A到{}, 我们应选择".format(_OPTION_LETTERS[count - 1])
+        )
+    if subset in ENGLISH_CLOZE_SUBSETS:
+        return passage + "Q: " + question + "\nA: The answer is"
+    if subset in CHINESE_CLOZE_SUBSETS:
+        return passage + "问题：" + question + "\n答案："
+    raise ValueError(f"Unknown AGIEval subset {subset!r}; expected one of {SUBSETS}")
+
+
+def second_stage_prompt(subset: str, context: str, first_stage_output: str) -> str:
+    """Upstream ``generate_second_stage_input`` for one row.
+
+    *context* is the first-stage prompt and *first_stage_output* the model's
+    reply to it; the cue that follows asks for the answer alone, which is what
+    :func:`sieval.community.agieval.post_process.post_process` parses.
+    """
+    for subsets, cue in _SECOND_STAGE_CUES:
+        if subset in subsets:
+            return "{0}\n{1}\n{2}".format(context, first_stage_output, cue)
+    raise ValueError(f"Unknown AGIEval subset {subset!r}; expected one of {SUBSETS}")

sieval/community/agieval/evaluation.py

@@ -0,0 +1,75 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+# adapted from https://github.com/ruixiangcui/AGIEval/blob/84ab72d94318290aad2e4ec820d535a95a1f7552/src/evaluation.py
+# leaderboard groupings from https://github.com/ruixiangcui/AGIEval/blob/84ab72d94318290aad2e4ec820d535a95a1f7552/post_process_and_evaluation.py
+"""AGIEval per-sample verdict + the groupings its leaderboard averages over.
+
+Three comparison rules (upstream ``evaluate_single_sample``): set-of-letters for
+the multi-answer MCQ subsets, math equivalence for the two cloze subsets, exact
+string equality for everything else.
+
+The leaderboard groups here are **not** the prompt-language families in
+:mod:`.dataset_loader`, and the difference is not cosmetic: ``gaokao-english`` is
+prompted in English but averaged as Chinese (it is a Chinese Gaokao paper), and
+``gaokao-mathqa`` is Chinese in both. Upstream's driver keeps two separate lists
+for exactly this reason.
+
+Deltas from upstream: ``convert_to_set(None)`` returns an empty *set* (upstream
+returns an empty *dict* — a typo, ``{}``); every comparison it feeds reaches the
+same verdict either way. A ``None`` prediction (sieval's "could not extract")
+needs no special case: it compares unequal under all three rules.
+
+AI-Generated Code - Claude Opus 5 (1M context) (Anthropic)
+"""
+
+from .dataset_loader import MATH_OUTPUT_SUBSETS, MULTI_CHOICE_SUBSETS
+from .math_equivalence import is_equiv
+
+#: The 8 English MCQ subsets upstream averages for its AGIEval-en leaderboard.
+#: `gaokao-english` is excluded here and counted with the Chinese group below.
+LEADERBOARD_EN_MCQ_SUBSETS: tuple[str, ...] = (
+    "aqua-rat",
+    "logiqa-en",
+    "lsat-ar",
+    "lsat-lr",
+    "lsat-rc",
+    "sat-math",
+    "sat-en",
+    "sat-en-without-passage",
+)
+
+#: The 11 Chinese MCQ subsets upstream averages for its AGIEval-zh leaderboard.
+LEADERBOARD_ZH_MCQ_SUBSETS: tuple[str, ...] = (
+    "logiqa-zh",
+    "jec-qa-kd",
+    "jec-qa-ca",
+    "gaokao-chinese",
+    "gaokao-english",
+    "gaokao-geography",
+    "gaokao-history",
+    "gaokao-biology",
+    "gaokao-chemistry",
+    "gaokao-physics",
+    "gaokao-mathqa",
+)
+
+
+def convert_to_set(item: str | list | None) -> set[str]:
+    if isinstance(item, list):
+        return set(item)
+    if isinstance(item, str):
+        return {item}
+    if item is None:
+        return set()
+    raise ValueError("Input can't parse:", item)
+
+
+def evaluate_single_sample(
+    subset: str, prediction: str | list | None, label: str | None
+) -> bool:
+    if subset in MULTI_CHOICE_SUBSETS:
+        return convert_to_set(prediction) == convert_to_set(label)
+    if subset in MATH_OUTPUT_SUBSETS:
+        return is_equiv(prediction, label)
+    return prediction == label