The two zips the README links carry the same eleven dev_databases, but only six of the files are the same. I checked https://bird-bench.oss-cn-beijing.aliyuncs.com/dev.zip (sha256 cdd6d19faeb45a23970b98d3ef6c40a87987c95459c2cf12076897a60cf5a630, Last-Modified 2024-06-29) against https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-02) by file digest first, then by schema, row count per table and a digest of each table's rows where the files differed. card_games, codebase_community, debit_card_specializing, financial, student_club and superhero are byte-identical. The other five are not, and neither README says so.
| Database |
Table |
dev.zip |
minidev.zip |
| formula_1 |
lapTimes |
420,369 rows |
400,524 rows |
| formula_1 |
pitStops |
6,070 |
5,815 |
| formula_1 |
qualifying |
7,397 |
6,967 |
| formula_1 |
races |
976 |
954 |
| formula_1 |
results |
23,657 |
23,179 |
| thrombosis_prediction |
Examination |
806 |
106 |
| toxicology |
atom |
12,333 |
9,111 |
| toxicology |
bond |
12,379 |
9,156 |
| toxicology |
connected |
24,758 |
18,312 |
| california_schools |
satscores |
same count, 211 of 2,269 cds codes without their leading zero |
the same codes with the zero |
| european_football_2 |
Player |
same count, height stored as REAL |
height stored as a truncated integer |
The schemas agree in all five. In formula_1, thrombosis_prediction and toxicology the dev copy simply holds more rows. In california_schools the row counts agree and 211 of the 2,269 satscores.cds values lost their leading zero in dev's copy, so those 211 rows join to schools in Mini-Dev's copy and do not in dev's. In european_football_2 the counts agree and Player.height is a REAL in dev and a truncated integer in Mini-Dev.
Why it matters: a Mini-Dev question is a dev question, and its gold runs against whichever copy the user unpacked. A gold over formula_1, toxicology or thrombosis_prediction can return other rows on the two copies, and a gold that joins satscores to schools returns other rows on california_schools. Two people scoring the same predictions against "the dev databases" can get different EX for that reason alone, and the README of neither repository says which copy is the one the golds were written against.
Two questions, then: which copy is the intended one for Mini-Dev, and could the README of each zip state the sha256 of the other and the five differences? The check itself is a short script; I am happy to send it, or the per-table row digests, if that helps.
The two zips the README links carry the same eleven
dev_databases, but only six of the files are the same. I checked https://bird-bench.oss-cn-beijing.aliyuncs.com/dev.zip (sha256cdd6d19faeb45a23970b98d3ef6c40a87987c95459c2cf12076897a60cf5a630, Last-Modified 2024-06-29) against https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-02) by file digest first, then by schema, row count per table and a digest of each table's rows where the files differed.card_games,codebase_community,debit_card_specializing,financial,student_clubandsuperheroare byte-identical. The other five are not, and neither README says so.cdscodes without their leading zeroheightstored as REALheightstored as a truncated integerThe schemas agree in all five. In
formula_1,thrombosis_predictionandtoxicologythe dev copy simply holds more rows. Incalifornia_schoolsthe row counts agree and 211 of the 2,269satscores.cdsvalues lost their leading zero in dev's copy, so those 211 rows join toschoolsin Mini-Dev's copy and do not in dev's. Ineuropean_football_2the counts agree andPlayer.heightis a REAL in dev and a truncated integer in Mini-Dev.Why it matters: a Mini-Dev question is a dev question, and its gold runs against whichever copy the user unpacked. A gold over
formula_1,toxicologyorthrombosis_predictioncan return other rows on the two copies, and a gold that joinssatscorestoschoolsreturns other rows oncalifornia_schools. Two people scoring the same predictions against "the dev databases" can get different EX for that reason alone, and the README of neither repository says which copy is the one the golds were written against.Two questions, then: which copy is the intended one for Mini-Dev, and could the README of each zip state the sha256 of the other and the five differences? The check itself is a short script; I am happy to send it, or the per-table row digests, if that helps.