compute_acc_by_diff() in evaluation_ex.py divides by len(simple_results), len(moderate_results) and len(challenging_results). If any of those buckets is empty — a --diff_json_path whose labels are outside {simple, moderate, challenging}, or a set that simply has no questions of one difficulty — the evaluator raises ZeroDivisionError after every query has been executed, and nothing is written.
This bites on the train split (which has no difficulty field at all): a full train evaluation runs for 20+ minutes and then dies in the summary step.
File "evaluation_ex.py", line 147, in <module>
simple_acc, moderate_acc, challenging_acc, acc, count_lists = compute_acc_by_diff(
File "evaluation_ex.py", line 88, in compute_acc_by_diff
simple_acc = sum([res["res"] for res in simple_results]) / len(simple_results)
ZeroDivisionError: division by zero
Reproduce: any prediction file with a difficulty JSONL of {"difficulty": "n/a"} lines (or a subset containing only one difficulty).
Current workaround: cycle the three labels through the JSONL purely to keep the divisors non-zero and read only the total — the per-bucket numbers are then meaningless. Suggested fix: guard each division (report n/a for an empty bucket) and accept an unlabelled set, reporting the total only.
compute_acc_by_diff()inevaluation_ex.pydivides bylen(simple_results),len(moderate_results)andlen(challenging_results). If any of those buckets is empty — a--diff_json_pathwhose labels are outside {simple, moderate, challenging}, or a set that simply has no questions of one difficulty — the evaluator raisesZeroDivisionErrorafter every query has been executed, and nothing is written.This bites on the train split (which has no
difficultyfield at all): a full train evaluation runs for 20+ minutes and then dies in the summary step.Reproduce: any prediction file with a difficulty JSONL of
{"difficulty": "n/a"}lines (or a subset containing only one difficulty).Current workaround: cycle the three labels through the JSONL purely to keep the divisors non-zero and read only the total — the per-bucket numbers are then meaningless. Suggested fix: guard each division (report
n/afor an empty bucket) and accept an unlabelled set, reporting the total only.