Issues

Five of the eleven dev databases differ between dev.zip and minidev.zip

#50

· ivermin1123 opened

on Sep 7, 2026

BIRD dev (bird_sql_dev_20251106): 23 golds still cut at a tie, sort NULL first, or sort numbers as text

#49

· ivermin1123 opened

on Sep 7, 2026

PostgreSQL: q1473 scores differently on two runs of evaluation_ex.py against the same database

#48

· ivermin1123 opened

on Sep 7, 2026

evaluation_ex.py crashes with ZeroDivisionError after executing every query when a difficulty bucket is empty (e.g. the train split) — results lost

#47

· christian-gheorghe opened

on Sep 5, 2026

Train split: 372 of 9,428 gold queries cannot run against the shipped train databases (gold-as-prediction scores 0.9605; retail_world schema mismatch)

#46

· christian-gheorghe opened

on Sep 5, 2026

Dev set: 10 gold/evidence annotation defects with executed evidence (129, 1113, 646, 635, 1199, 271, 1006/1011/1014, 1279)

#45

· christian-gheorghe opened

on Sep 5, 2026

--diff_json_path is an INPUT (JSONL difficulty labels), not an output; a missing file crashes after the full evaluation has run

#44

· christian-gheorghe opened

on Sep 5, 2026

Two dev gold queries exceed the default --meta_time_out 30.0: dev has a hard ceiling of 0.9987 and the ceiling depends on the grader's hardware

#43

· christian-gheorghe opened

on Sep 5, 2026

A malformed prediction entry is silently scored against the "financial" database instead of raising

#42

· christian-gheorghe opened

on Sep 5, 2026

evaluation_ex.py pairs predictions to gold POSITIONALLY and ignores the keys — a sort_keys=True prediction file scores 0.0013 with no error

#41

· christian-gheorghe opened

on Sep 5, 2026

In fact, gt file obtained through gmail account is broken

#35

· jamesunnc opened

on May 22, 2026

Boundary-case data coverage for numeric predicates

#31

· tohpinren opened

on Jan 18, 2026

Navigation Menu

All issues

Five of the eleven dev databases differ between dev.zip and minidev.zip

BIRD dev (bird_sql_dev_20251106): 23 golds still cut at a tie, sort NULL first, or sort numbers as text

PostgreSQL: q1473 scores differently on two runs of evaluation_ex.py against the same database

evaluation_ex.py crashes with ZeroDivisionError after executing every query when a difficulty bucket is empty (e.g. the train split) — results lost

Train split: 372 of 9,428 gold queries cannot run against the shipped train databases (gold-as-prediction scores 0.9605; retail_world schema mismatch)

Dev set: 10 gold/evidence annotation defects with executed evidence (129, 1113, 646, 635, 1199, 271, 1006/1011/1014, 1279)

--diff_json_path is an INPUT (JSONL difficulty labels), not an output; a missing file crashes after the full evaluation has run

Two dev gold queries exceed the default --meta_time_out 30.0: dev has a hard ceiling of 0.9987 and the ceiling depends on the grader's hardware

A malformed prediction entry is silently scored against the "financial" database instead of raising

evaluation_ex.py pairs predictions to gold POSITIONALLY and ignores the keys — a sort_keys=True prediction file scores 0.0013 with no error

In fact, gt file obtained through gmail account is broken

Boundary-case data coverage for numeric predicates

Search results