Skip to content

Five of the eleven dev databases differ between dev.zip and minidev.zip #50

Description

@ivermin1123

The two zips the README links carry the same eleven dev_databases, but only six of the files are the same. I checked https://bird-bench.oss-cn-beijing.aliyuncs.com/dev.zip (sha256 cdd6d19faeb45a23970b98d3ef6c40a87987c95459c2cf12076897a60cf5a630, Last-Modified 2024-06-29) against https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-02) by file digest first, then by schema, row count per table and a digest of each table's rows where the files differed. card_games, codebase_community, debit_card_specializing, financial, student_club and superhero are byte-identical. The other five are not, and neither README says so.

Database Table dev.zip minidev.zip
formula_1 lapTimes 420,369 rows 400,524 rows
formula_1 pitStops 6,070 5,815
formula_1 qualifying 7,397 6,967
formula_1 races 976 954
formula_1 results 23,657 23,179
thrombosis_prediction Examination 806 106
toxicology atom 12,333 9,111
toxicology bond 12,379 9,156
toxicology connected 24,758 18,312
california_schools satscores same count, 211 of 2,269 cds codes without their leading zero the same codes with the zero
european_football_2 Player same count, height stored as REAL height stored as a truncated integer

The schemas agree in all five. In formula_1, thrombosis_prediction and toxicology the dev copy simply holds more rows. In california_schools the row counts agree and 211 of the 2,269 satscores.cds values lost their leading zero in dev's copy, so those 211 rows join to schools in Mini-Dev's copy and do not in dev's. In european_football_2 the counts agree and Player.height is a REAL in dev and a truncated integer in Mini-Dev.

Why it matters: a Mini-Dev question is a dev question, and its gold runs against whichever copy the user unpacked. A gold over formula_1, toxicology or thrombosis_prediction can return other rows on the two copies, and a gold that joins satscores to schools returns other rows on california_schools. Two people scoring the same predictions against "the dev databases" can get different EX for that reason alone, and the README of neither repository says which copy is the one the golds were written against.

Two questions, then: which copy is the intended one for Mini-Dev, and could the README of each zip state the sha256 of the other and the five differences? The check itself is a short script; I am happy to send it, or the per-table row digests, if that helps.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions