feat(tasks): add Multi-IF multi-turn multilingual instruction-following

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Adds Multi-IF (paper · code · data) — IFEval extended into three-turn conversations across eight languages, 4,501 conversations.

• sieval's first multi-turn task. One sample is one whole conversation: infer walks its turns in order, appending the model's own reply before sending the next user turn, and returns list[ModelOutput] — a shape the runner already special-cases, so every turn's tokens reach profile.json. Upstream instead re-runs the whole set once per --steps value, writing the growing history back to a CSV; one pass makes a conversation a single unit of work and is what lets one run report all three turns.
• Upstream's own multilingual fork of the IFEval checkers is vendored in sieval/community/multi_if/, not reused from sieval/community/instruction_following_eval/. Both carry the same 25 instruction ids, but Multi-IF's implementations are language-aware (langdetect-routed word/sentence counting, CJK and Hindi counters, Thai tokenizers), so they are not interchangeable — please don't "deduplicate" them later.
• Grading is verified against upstream's own metrics_gen, not merely faithful by construction — see the table below.
• Two upstream defects are tracked, not repaired, per the unqualified-name rule. Both make upstream's grader unable to reproduce itself.
• status="experimental": Multi-IF publishes paper scores only, with no per-model inference dump to replay, so no published number has been reproduced yet.

Related Issues

None.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check) — ty is the primary checker per CLAUDE.md. mypy --strict is repo-wide noisy (1,426 pre-existing errors in 135 files); the one diagnostic genuinely introduced here (a list invariance error on the prediction payload) is fixed, and the remainder are the house-style ones every task shares (unannotated @override stage methods, bare dict).
• Unit tests pass — 10 new tests; full tests/unit/ suite green at 3,181 passed, no regressions.
• python scripts/check_preflight.py — 22/22 PASS, including check_dep_coverage, check_meta_index_sync, and check_deps (no unrequested lock drift).

Manual

Grader fidelity — the port vs. upstream's own evaluator. Both were run on identical (row, response) pairs and compared per-constraint, not just on the aggregate:

Check	Coverage	Result
strict/loose follow_instruction_list	3,098 comparisons	0 mismatches
per-language turn_{t}_{lang}_overall	every cell, all 8 languages	0 differences
turn_{t}_prompts_number	turns 1/2/3	exact (535 / 535 / 479)

Sample: 535 conversations, language-stratified across all eight languages, including all 56 two-turn rows. The two conversations described below are excluded — they cannot match anything, upstream included.

No score-vs-published table is included, and that is why status="experimental". The paper reports o1-preview 0.877 → 0.707 (turn 1 → turn 3); reproducing that needs a live run against a checkpoint that is no longer served, and Multi-IF publishes no inference cache to replay the way PlatinumBench and MathArena ports do. This PR ships the half that can be verified offline — the grader and the aggregation — exactly.

Also verified manually:

• sieval dataset download multi_if succeeds; sieval task show multi_if_0shot_gen then reports Ready: yes.
• Constraints accumulate across turns — turn t's instruction_id_list is a prefix-extension of turn t-1's on every row of the pinned revision, 0 violations.
• Vendored file confirmed byte-identical to the pinned upstream commit before adaptations (sha256 match against raw.githubusercontent.com at 1cdb53ed).

Two upstream defects, tracked rather than repaired

Both are documented in reference_impl.notes and at their source sites. Fixing either would be a grader change, which belongs in a _fixed variant with a measured delta — not under the unqualified name.

1. Rejected kwargs fall back to unseeded random.choice. Two conversations (6 of 13,447 turn-cells, 0.04%) carry kwargs the checker rejects — keywords:letter_frequency with letter="#", and keywords:frequency with no keyword. build_description then draws a random letter and grades the response against a constraint nobody asked for, redrawn on every call.
2. langdetect is unseeded, and detection selects the word- and sentence-counting algorithm behind every length constraint (~2–3% of calls flip on short or mixed-script text). Measured exposure is narrower than that rate suggests: observed flips stayed within Latin-script languages, which share a counting branch.

Other deviations from upstream

• The 56 conversations without a third turn get no third generation. Upstream sends them a literal "None" prompt and then drops the row when scoring turn 3 — identical metrics, tokens not spent. Turn-3 denominators are 4,445, and report.json carries turn_{t}_prompts_number so this is visible rather than inferred.
• Bootstrap confidence intervals are not computed. Upstream resamples every language cell 10,000× via scipy; nothing in sieval consumes the interval, and the per-sample verdicts needed to recompute one are all on disk.
• score is the mean of the three turns' all-language overalls. Upstream emits one report per turn and never reduces them; a task needs one headline, and every component is in the report.
• Percentages, not fractions, matching the IFEval/IFBench siblings — the paper's 0.877 reads as 87.7 here.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring (the vendored upstream files carry Meta's Apache-2.0 header instead, as they are not AI-generated)
• No new upper-layer dependencies added to core/ — core/ is untouched
• Deleted code verified — nothing deleted; the change is purely additive

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included (model, expected, actual, diff) — not possible for this benchmark; the grader-fidelity table above is the substitute, and status="experimental" records the gap. See the Manual section.
• Dataset loading tested (sieval dataset download multi_if succeeds)
• Task registered in package-level __init__.py (lazy registry + regenerated __init__.pyi and meta/index.json)

If: community/ Changes

• Upstream diff documented — sieval/community/multi_if/__init__.py enumerates every adaptation. Only one behavioural change: pythainlp is imported lazily instead of at module scope. evaluation_lib.py takes only the two per-response graders from upstream's metrics.py (its surrounding pipeline is a pandas/scipy CSV driver sieval does not use). The unseeded langdetect is deliberately left as-is, with a note explaining why the obvious fix is absent.
• License attribution preserved — Meta's Apache-2.0 header kept on both vendored files, with commit-pinned permalinks to their sources.

If: New Dependency

• Added to correct PDM dependency group — new multi-if extra; group name inserted into pdm.lock [metadata].groups first, then pdm lock --update-reuse, per .claude/rules/deps.md. Lock diff is +27/−12 with pythainlp the only new package; check_deps confirms no unrequested drift.
• Justified in Summary — deliberately not the ifeval group: the fork uses stdlib logging and MappingProxyType, so absl-py/immutabledict are not needed, while emoji is required rather than optional because the CJK word counter behind every Chinese length constraint counts emoji as words. pythainlp is declared but lazily imported: it is only reached if langdetect reports Thai for a model response (Thai is not one of the dataset's eight languages), and declaring it keeps that path from dying on ImportError mid-run.

Note on the dataset license

license="CC-BY-NC-2.0" — the Hub dataset card's license, which is not the Apache-2.0 of the facebookresearch/Multi-IF code repo that ships the evaluator. The data is non-commercial; the two licenses differ and DatasetMeta.license records the data's.

🤖 Generated with Claude Code