evaluation_ex.py pairs predictions to gold POSITIONALLY and ignores the keys — a sort_keys=True prediction file scores 0.0013 with no error

[christian-gheorghe]

package_sqls() in evaluation_utils.py iterates for _, sql_str in sql_data.items(), so the "0".."N-1" keys are discarded and the i-th VALUE in file order is paired with the i-th gold query.

Any writer that emits keys in lexicographic order — e.g. json.dump(preds, f, sort_keys=True), which yields "0","1","10","100","1000",… — produces a file that is complete and well-formed but mis-paired from position 2 onward. The evaluator reports a near-zero score with no warning; it looks like a broken model rather than a mis-ordered file.

Reproduce with BIRD's own gold (dev_20240627):

1. Build predictions from dev.json gold: {str(i): f"{SQL}\t----- bird -----\t{db_id}"} in dev order → evaluation_ex.py scores it 0.9987 (see the separate timeout issue for the missing 0.13%).
2. Re-serialize the same dict with sort_keys=True → evaluation_ex.py scores 0.0013 (0.13%).
3. Re-emit in numeric key order → 0.9987 again.

Suggested fix: sort by int(key) before pairing, or assert list(keys) == [str(i) for i in range(n)]. Either is a two-line change and removes a silent failure mode from every submission.