In evaluation_utils.package_sqls(), a prediction is split on "\t----- bird -----\t" to recover (sql, db_id). When the split fails the code does not raise — it falls back to db_name = "financial":
try:
sql, db_name = sql_str.split("\t----- bird -----\t")
except ValueError:
sql = sql_str.strip()
db_name = "financial"
So a prediction file that loses its separators (a formatting bug, a bad join, a truncated write) is quietly evaluated entirely against financial, producing a low-but-plausible score. Nothing in the output says the file was invalid.
Reproduce: pass {"0": "SELECT 1 FROM frpm"} (no separator) through package_sqls — it returns normally with db_name == "financial".
Suggested fix: raise on a failed split (or at minimum count and print the number of fallbacks).
In
evaluation_utils.package_sqls(), a prediction is split on"\t----- bird -----\t"to recover(sql, db_id). When the split fails the code does not raise — it falls back todb_name = "financial":So a prediction file that loses its separators (a formatting bug, a bad join, a truncated write) is quietly evaluated entirely against
financial, producing a low-but-plausible score. Nothing in the output says the file was invalid.Reproduce: pass
{"0": "SELECT 1 FROM frpm"}(no separator) throughpackage_sqls— it returns normally withdb_name == "financial".Suggested fix: raise on a failed split (or at minimum count and print the number of fallbacks).