The flag name reads like a diff report the evaluator will write. It is the opposite: compute_acc_by_diff() calls load_jsonl(diff_json_path) to read each question's difficulty. A path that does not exist raises FileNotFoundError only after all queries have executed, and dev.json (a JSON array) cannot be used directly because the loader expects JSONL.
Suggested fix: rename to --difficulty_jsonl_path, validate it before running, and accept dev.json.
The flag name reads like a diff report the evaluator will write. It is the opposite:
compute_acc_by_diff()callsload_jsonl(diff_json_path)to read each question'sdifficulty. A path that does not exist raisesFileNotFoundErroronly after all queries have executed, anddev.json(a JSON array) cannot be used directly because the loader expects JSONL.Suggested fix: rename to
--difficulty_jsonl_path, validate it before running, and acceptdev.json.