feat(tasks): add AdvancedIF rubric-graded instruction-following benchmark

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• Adds AdvancedIF (Meta, arXiv:2511.10507, upstream facebookresearch/AdvancedIF): 1,645 expert-written prompts with human-curated rubrics across complex single-turn (402), system steerability (507) and carried-context multi-turn (736) instruction following. A grader LLM answers every rubric question; headline score = overall pass rate, with the co-published micro_pass_rate and a per-aspect breakdown.
• The judge prompts are not vendored. Upstream ships every file under CC-BY-NC-4.0 (LICENSE file, not just the README), which cannot be redistributed inside this Apache-2.0 tree — there is currently no NC-licensed code in sieval/, only NC license= metadata on dataset pointers. The operator stages their own checkout and points SIEVAL_ADVANCED_IF_SRC at it; the loader sha256-pins judge.py to f9d3013 and refuses a drifted revision. The data is CC-BY-NC-4.0 too and likewise only referenced, so running this benchmark accepts the upstream terms either way. Side benefit: the prompts are byte-exact by construction rather than by review.
• Reproduces an upstream defect on purpose. Upstream selects the system-steerability judge on benchmark_name == "if_system_steerability_oss", a value the released dataset never contains (it ships system_steerability_v2) — so all 507 system-prompt rows are graded by the plain user-instruction judge, and the CLI's --task choices match zero rows. processor.process_file's own docstring uses the released spelling, so the if_*_oss literals are what went stale. Per sieval/tasks/CLAUDE.md, the unqualified name tracks upstream including defects; correcting the routing moves scores on a third of the benchmark and belongs in a _fixed variant carrying a measured delta. A test pins the behaviour so it cannot be "cleaned up" silently.
• The two published rates deliberately do not share a denominator: the per-sample rubric rate divides by the rubric count the data carries, while the pooled micro_pass_rate counts every answer key the grader emitted. Both are kept, each matching its own upstream definition, with the raw counts a pooled rate cannot reconstruct persisted per rollout.
• The whole rubric goes to the grader in one indexed call per rollout, so its ModelOutput is stored as a single mapping and its spend reaches profile.json (iter_grader_outputs skips a list).
• status="experimental" — faithful port, no published number reproduced (see Manual below).

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check — clean across sieval/ and tests/)
• Type check clean (ty check clean; mypy --strict emits only the same pre-existing no-untyped-def/type-arg classes as the sibling grader task simpleqa_verified_0shot_gen)
• Unit tests pass (3215 passed, 44 new)

Also clean: scripts/check_preflight.py (all checks, no FAIL/WARN), and sync_meta_index.py --check / sync_package_stubs.py --check.

Manual

• Real upstream prompts, end to end (offline). Staged a checkout at the pinned commit; the sha256 gate passes, judge.py imports, and both judge prompts compose with every slot filled (judge_prompt 1,048 chars; system-steer 28,587 chars incl. the 27,227-char few-shot block). A realistic grader reply then parses and scores correctly. The 27KB few-shot block is also the concrete reason vendoring was not an option.
• Real dataset at the pinned revision. sieval dataset download advanced_if succeeds and sieval task show advanced_if_0shot_gen reports Ready: yes. Loaded rows match the modelled schema exactly: 1,645 rows, three str columns, train mirrored to test, and benchmark_name counts 402 / 507 / 736 as documented.
• Score comparison table — deliberately absent, not skipped. No published number has been reproduced. Grading all 1,645 prompts requires a paid grader endpoint, and the paper's own figures come from Meta's internal pipeline rather than the released CLI, so there is no directly comparable target to diff against. This is why the task ships status="experimental"; validation is follow-up work, and any _fixed routing variant is blocked on the same measurement.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it (nothing deleted; additive only)

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included — see Manual above for why this is absent rather than pending
• Dataset loading tested (sieval dataset download advanced_if succeeds)
• Task registered in package-level __init__.py (lazy auto-discovering registry; __init__.pyi stubs + meta/index.json regenerated, and name resolution verified via sieval task show)

If: community/ Changes

• Upstream diff documented — module docstring of sieval/community/advanced_if.py enumerates the reproduced routing defect plus two deviations: a fenced-JSON parse fallback (sieval cannot force response_format=json_object on an arbitrary grader endpoint the way upstream's OpenAI client does — without it a grader that fences its JSON would score zero everywhere, a harness artifact rather than a model property), and stringifying non-string rubric answers instead of aborting the row.
• License attribution preserved — upstream commit-pinned permalink and the CC-BY-NC-4.0 terms are stated in the module header, the dataset license= field, and the task's reference_impl.notes.

Running it

git clone https://github.com/facebookresearch/AdvancedIF && \
  git -C AdvancedIF checkout f9d30137c4139d4d9af260ae28108b5afae828c0
export SIEVAL_ADVANCED_IF_SRC=$PWD/AdvancedIF
sieval dataset download advanced_if
# pass `grader:` in the task args — upstream uses o3-mini-2025-01-31,
# temperature 0, max_completion_tokens 32768, response_format {type: json_object}

🤖 Generated with Claude Code