evaluation_ex.py scores a timed-out query as incorrect. Two dev gold queries cannot finish in 30 s on the shipped SQLite databases (measured on an M1 Pro laptop):
| question_id |
database |
gold runtime |
difficulty |
| 701 |
codebase_community |
~192 s |
challenging |
| 518 |
card_games |
~39 s |
moderate |
Consequence: submitting dev.json's own SQL as the prediction scores 1532/1534 = 0.9987, not 1.0. Both queries execute correctly and deterministically without the timeout — this is a budget problem, not bad SQL. Because the limit is wall-clock, the maximum reachable dev score varies with the grading machine, which makes entrants' dev numbers not strictly comparable.
Suggested fix: either exempt gold from the timeout when it is the gold that is slow, raise the default, or document the ceiling. Could you also confirm whether the same two questions breach the timeout in the hidden-test scoring environment?
evaluation_ex.pyscores a timed-out query as incorrect. Two dev gold queries cannot finish in 30 s on the shipped SQLite databases (measured on an M1 Pro laptop):Consequence: submitting
dev.json's own SQL as the prediction scores 1532/1534 = 0.9987, not 1.0. Both queries execute correctly and deterministically without the timeout — this is a budget problem, not bad SQL. Because the limit is wall-clock, the maximum reachable dev score varies with the grading machine, which makes entrants' dev numbers not strictly comparable.Suggested fix: either exempt gold from the timeout when it is the gold that is slow, raise the default, or document the ceiling. Could you also confirm whether the same two questions breach the timeout in the hidden-test scoring environment?