feat(tasks): add ComplexConstraints (dataset + 0-shot rubric-judged task)

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• Adds ComplexConstraints (Surge AI, arXiv:2606.09118,
  surgeai/ComplexConstraints, CC-BY-4.0):
  75 multi-constraint instruction-following prompts, each carrying 10–40 atomic rubric criteria
  (1,559 total), graded by an LLM judge rather than exact match. Two new registry entries
  (complex_constraints dataset, complex_constraints_0shot_gen task), no new dependencies.
• Headline metric = task pass rate (response satisfies every criterion) — the metric the
  paper's public 75-prompt leaderboard reports (its Table 1). The mean per-criterion pass rate is
  reported in both its published macro form (Table 3 caption) and the pooled micro form;
  the two genuinely differ because rubric sizes vary 10–40 per prompt.
• status="experimental", and the reason is score-affecting: upstream publishes no evaluation
  code and no judge prompt. The paper names GPT-5-mini as the judge and defines the metrics,
  nothing more, and the dataset card adds nothing — so the rubric prompt and verdict parsing are
  authored by this port. Same posture as aa_lcr, one notch weaker (aa_lcr's templates are at
  least the upstream card's own snippets).
• ⚠️ No leaderboard alignment run yet — see Manual test plan below. reference_impl.notes says
  NOT YET VALIDATED and carries the Table 1 baselines to align against.
• Loader detail: the Hub repo ships one wide CSV whose 40 sparse criterion_{i} columns are
  collapsed into a single criteria list (a reshape, not a rename-for-uniformity — a 40-key
  mostly-absent TypedDict is unusable as a sample type). The card's configs.data_files.path
  names ComplexConstraints_Benchmark_Set.csv, which does not exist (casing), so
  load_dataset("surgeai/ComplexConstraints") cannot resolve the file at all; the loader reads the
  staged snapshot by its real name.

Two design calls worth reviewing

One judge call per rollout, not one per criterion. Upstream never states its call structure, and
"atomic criteria" would argue for per-criterion calls. The framework decides it: iter_grader_outputs
collects extra[GRADER_OUTPUT_KEY] only when it is a Mapping, so a list of N grader outputs is
skipped silently and that spend vanishes from profile.json — and tasks may not modify core/.
Batching keeps a rollout's whole verdict set in the single ModelOutput the profiler expects. The
verdicts are emitted as an indexed PASS/FAIL list so misalignment is detectable rather than
silently shifting neighbours.

Unreadable verdicts are counted, not just scored. A criterion the judge returns no readable
verdict for scores not-satisfied (an unreadable verdict must never inflate a score) but is counted
separately as n_unparsed, per rollout and pooled in the report — so judge format drift stays
distinguishable from a model that genuinely failed the rubric. Empty/whitespace responses satisfy
zero criteria without invoking the judge (grader_output absent there, no call made).

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass — 48 new tests across the three modules; full tests/unit suite 3219 passed
• scripts/check_preflight.py — 22 PASS / 0 FAIL (incl. check_meta_index_sync, dataset
  revision pinning, task naming/tags, import policy)
• sync_meta_index.py --check and sync_package_stubs.py --check both clean

Manual

• Loader run against the real CSV: 75 rows, 1,559 criteria total — matches the paper's
  stated counts exactly; every collapsed criterion a non-empty string.

• sieval dataset download complex_constraints succeeds; sieval task show complex_constraints_0shot_gen then reports Ready: yes.

• End-to-end preprocess → infer → postprocess → feedback → report over real samples with a
  scripted judge, on the smallest (10), largest (40) and a middle (19) rubric: verdicts
  index-aligned with 0 unparsed, and macro (68.64) ≠ micro (68.12) as expected on uneven
  rubrics.

• Score comparison table vs the published leaderboard — NOT DONE. This needs a real judge
  plus a real candidate model (API spend), so no (model, expected, actual, diff) row exists
  yet. It is the reason the task ships experimental, and the alignment targets are recorded
  in reference_impl.notes for whoever runs it:

  Model	Expected (Table 1, task pass %)	Actual	Diff
  Gemini 3.1 Pro	40.4	not run	—
  GPT-5.5	38.7	not run	—
  Claude Opus 4.8	34.9	not run	—

  Note the leaderboard snapshot (2026-06-03) does not state a repeat count, and the paper does not
  say whether Table 1 used the same GPT-5-mini judge as its training pipeline — so even a matched
  run has an unpinned degree of freedom. The port defaults to n=1.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ (core/ untouched)
• Deleted code verified — no code deleted

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included — table present but unpopulated; see Manual above
• Dataset loading tested (sieval dataset download complex_constraints succeeds)
• Task registered — auto-discovered by the lazy registry; generated __init__.pyi stubs and
  sieval/meta/index.json regenerated and committed

If: community/ Changes

• Upstream diff documented — sieval/community/complex_constraints.py states in its docstring
  that upstream ships no eval code and no judge prompt, so the template and parser are authored
  by this port (contrasted explicitly with sieval.community.aa_lcr)
• License attribution preserved — dataset is CC-BY-4.0, recorded on @sieval_dataset(license=…);
  no upstream code was vendored, so there is none to attribute