Build provenance, typed rendering, safety-task leaks, answer-first MC fix, privacy-200k

[sileod]

Addresses the six points of the static review of feb44ee, my audit of the new safety tasks, and adds privacy-200k. There is one commit per topic, so any of them can be dropped.

1. Review fixes (f8f5740)

• Resume can't mix shards from different generations. A build fingerprint covers the commit, a hash of uncommitted changes to src/ and scripts/ (untracked files included), and every setting that shapes shards (row caps, augmentation and packing rates). It is stored in each task's report record.
  • Resuming onto shards with another fingerprint stops with an explanation unless --reuse-incompatible-shards is passed.
  • build-manifest.json keeps earlier runs' provenance, and build-summary.json lists the fingerprints the selected shards were built with.
  • Checked end to end: a rerun with a different --max-rows stops, an unchanged rerun skips finished tasks, and the flag overrides the stop.
• Exact source revisions. Before loading, each source's Hub repos are resolved to a commit at the revision it requests (refs/convert/parquet and refs embedded in hf:// URLs included), and Hub-repo loads are pinned to that commit.
  • The commit is recorded in the report, and sources.yaml reads it from there instead of querying the default branch at publish time. Tasks built before this change are marked revisions_unrecorded.
  • Example: dream loads dataset-org/dream@refs/convert/parquet (ca4c45f…), where the old sources.yaml would have named main (78b128b…).
  • task_provenance now reports revision and data_file_revisions; graded and procedural sources report theirs.
• Typed renderer. render_typed_decision(_group) renders choice, score (ordered levels, 2–10) and noul by the row's kind, instead of always choice. Unknown kinds raise.
• IntentGrasp now rejects negative answer indices (0 <= index < len(options)).
• list_tasks accepts a list for excluded and returns a copy of the cached catalog. Before, select_tasks added columns to the cached frame.
• Task lookup filters directly: an unknown task raises KeyError, and a selector matching several tasks raises ValueError. The unused dict_to_query is removed.

2. Safety-task data (bb9241b)

• WildGuardMix (the bogdanminko/wildguardmix-cleaned mirror): it has only a train split, with two responses per prompt, so the random validation/test split put 90% of evaluation prompts in train. All three tasks now split by a stable hash of the prompt (new split_by_group), and prompt_harm keeps one row per prompt: 0% overlap now. ORIGINALS names allenai/wildguardmix.
  • The gated original has a real test split, but using it would make the task fail for anyone who hasn't accepted AI2's terms, so the mirror stays.
• BeaverTails: each (prompt, response) was a separate row per annotator, three rows per pair, and 27.6% of pairs have split votes. The task now keeps one row per pair with the majority vote, which also removes the 89% validation-in-train overlap.
  • New graded/beavertails gives the share of annotators who judged the response unsafe (0, ⅓, ⅔, 1): another real calibration source. Family gains a pre_process hook for it.
• neuralchemy (no change): the near-duplicates I flagged earlier are the source's deliberate case, whitespace and Unicode augmentations sharing a group_id, and its splits are group-aware as documented.

3. load_task put the gold answer first (dc04ccb)

In plain load_task, the gold answer was option 0 in 100% of rows for:

• multiple-choice tasks built from option lists (hellaswag, ARC, IntentGrasp…);
• fixed-column tasks whose annotation lists the gold answer first (labels=constant(0): sciq, hh-rlhf, UltraFeedback, prm800k, the DPO pairs… 16 tasks).

Kept options are now permuted deterministically from their text. Rows whose options refer to each other by position keep the source order. Choice columns are read in numeric order, which fixes a latent mislabel when truncating rows with 10+ options.

• Checked with caching disabled: gold is spread about evenly across slots for sciq, hh-rlhf, UltraFeedback, cosmos_qa, hellaswag, ARC and IntentGrasp, and sciq's gold matches the source's correct answer in 500/500 rows.
• The Jev and instruct recasts already shuffled and are unchanged.

4. privacy-200k-Mistral-Large-3 (02267a6)

Privacy sensitivity rated 1–5 over ten domains (Loiseau et al., 2026).

• The question and level names follow the annotation prompt's scale ("direct or indirect identifiers", "should not be made public").
• ordinal=True: in Jev, about half the rows become 5-level score questions.
• The domain splits are merged (a domain column is kept) and split 90/5/5 by text.
• Card conflict: these labels come from a teacher LLM, while the Jev card says labels come "from established datasets, not a teacher model". Either amend the card or exclude it from the Jev build.

Tests

181 pass. New tests cover the fingerprint and staleness, revision reporting, each rendered kind, the IntentGrasp bounds, list_tasks copies and lists, the lookup errors, the grouped split, and both BeaverTails aggregations. Catalogs are regenerated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EVp7VRXPWTGT6HsHHWJghc

---

Generated by Claude Code