[Feature]: give JudgementRecord.reference a durable signal for why it is absent

[ethan-scitix]

🚀 The feature, motivation and pitch

JudgementRecord.reference has no durable signal for why it is absent, so a stored judgement cannot tell "this task's ground truth is a procedure" apart from "this sample's gold went missing".

build_judgement_record writes reference unconditionally, and obj_to_dict drops None-valued keys, so reference=None is present in memory and gone on disk:

procedural (human_eval):  {"rollouts":[...], "n_rollouts":1, "n_correct":1, "extra":{"entry_point":"..."}}
value      (gpqa):        {"reference":"B", "rollouts":[...], "n_rollouts":1, "n_correct":1}

The only way to perceive "no value-form ground truth" is to notice a missing key — and key absence collapses two causes that want different responses:

	cause	cardinality	wanted response
(a)	the task's truth is a procedure (test suite, rubric)	constant for the whole task	declared, visible before reading any shard
(b)	this sample's gold is missing or malformed	per sample	an anomaly, reported

prediction does not have this problem, and the contrast is the pitch. Its None has exactly one cause (extraction failed), so a single derived boolean carries it — and records.py calls extracted "The durable signal" precisely because None does not survive serialization. Crucially, extracted is not valuable on its own: it is valuable because detect_extraction_failure reads it. reference has neither half.

This is the same hazard _checked_metrics already guards for metrics, in its own words — "a None metric would be absent on disk, turning 'not measured' into 'never existed'". That argument was never applied to reference.

Proposal, in this order (the order is a dependency, not a preference):

1. (a) → task metadata. Add something like reference_kind: Literal["value", "procedure"] to TaskMeta. It is a task-level constant — storing it per sample would repeat one fact 164 times for HumanEval, and JudgementRecord's own docstring already argues this shape ("Sample-level facts live here rather than being repeated per rollout, so a ground truth is stored once regardless of n"); the same logic escalates one level. Declaring it makes the fact machine-readable in the registry, meta/index.json and the leaderboard before any shard is read. Default "value"; only the four procedural-reference tasks (human_eval, mbpp, livecodebench, scicode — each already carrying a hand-written None, # the reference is the test suite, not a value comment) declare "procedure".
2. (b) → anomaly rule. A value-reference task whose sample carries no reference is a real defect and should surface in anomalies.json. None of the five existing rules (empty_infer_gen, empty_infer_ppl, truncated_output, empty_postprocess, extraction_failure) covers reference.

(b) depends on (a): the rule's signature is (ctx: TaskContext) -> set[int], and deciding whether absence is legitimate requires knowing the task's reference_kind. So (a) has to land first, or the rule has no way to tell the two causes apart either.

Alternatives

• A per-sample boolean (has_reference / "was gold saved"). Considered and rejected: it is exactly equivalent to key presence, so it adds no information — only the "self-describing row" property that n_rollouts has over len(rollouts). More importantly it does not fix the actual problem, because false still collapses (a) and (b). It would make absence explicit without making it meaningful.
• Gate reference in check_record_key_access (i.e. force [] → .get()). Rejected in fix(tasks): stop indexing a rollout key that is absent on disk #70: the sole [] read, ruler_0shot_gen.py:266, consumes the value as an iterable, so .get() yields list(None) → TypeError — the same trap that keeps extra ungated. It also would not make anything perceivable, only non-crashing. reference is classified as ungated with a recorded reason as of 69f0d82c.
• Do nothing. Defensible for now, which is why this is not urgent: nothing crashes today. See below.

Additional context

Not urgent, and deliberately not part of #70. This is an observability gap, not a defect:

• Nothing crashes. The only [] read of a record reference on a JudgementRecord is ruler_0shot_gen.py:266, and ruler's reference is always list(ctx.raw_sample["outputs"]) — never None, so the key is never absent there.
• There is no cross-task consumer to be confused: repo-wide there are exactly three reads of a record reference (ruler_0shot_gen.py:266, gpqa_diamond_0shot_gen.py:110, hellaswag_kshot_ppl.py:256), all inside sieval/tasks/, and core/ has no generic reference reader. (cli/leaderboard/card.py:91's meta.get("reference") is leaderboard metadata, unrelated.)

PromptRecord.reference is a different, milder case and does not need this. build_prompt_record omits the key when None, so it is absent in memory too — fresh and resumed runs fail identically rather than diverging, i.e. loudly and in development. The two [] reads (gpqa_diamond_0shot_gen.py:110, hellaswag_kshot_ppl.py:256) are correct as written.

Note when implementing (a): adding a TaskMeta field changes sieval/meta/index.json, so scripts/sync_meta_index.py has to be re-run and the result committed, or check_meta_index_sync fails preflight.

Surfaced while reviewing #70, which fixed the prediction half of this hazard (39 call sites) and added check_record_key_access to enforce it. 69f0d82c there widened that check's classification to all five record TypedDicts, which is what forced reference to be classified at all and surfaced this gap.

Before submitting a new issue...

• Make sure you already searched for relevant issues and documentation.