Skip to content

evaluation_ex.py crashes with ZeroDivisionError after executing every query when a difficulty bucket is empty (e.g. the train split) — results lost #47

Description

@christian-gheorghe

compute_acc_by_diff() in evaluation_ex.py divides by len(simple_results), len(moderate_results) and len(challenging_results). If any of those buckets is empty — a --diff_json_path whose labels are outside {simple, moderate, challenging}, or a set that simply has no questions of one difficulty — the evaluator raises ZeroDivisionError after every query has been executed, and nothing is written.

This bites on the train split (which has no difficulty field at all): a full train evaluation runs for 20+ minutes and then dies in the summary step.

File "evaluation_ex.py", line 147, in <module>
    simple_acc, moderate_acc, challenging_acc, acc, count_lists = compute_acc_by_diff(
File "evaluation_ex.py", line 88, in compute_acc_by_diff
    simple_acc = sum([res["res"] for res in simple_results]) / len(simple_results)
ZeroDivisionError: division by zero

Reproduce: any prediction file with a difficulty JSONL of {"difficulty": "n/a"} lines (or a subset containing only one difficulty).

Current workaround: cycle the three labels through the JSONL purely to keep the divisors non-zero and read only the total — the per-bucket numbers are then meaningless. Suggested fix: guard each division (report n/a for an empty bucket) and accept an unlabelled set, reporting the total only.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions