While validating the dev set (dev.zip, sha256 cdd6d19faeb45a23970b98d3ef6c40a87987c95459c2cf12076897a60cf5a630) we executed every gold query and diffed a sample of results against the questions as written. The items below look like annotation defects rather than modelling disagreements: the gold answers a different question than the one asked, or is internally broken.
| question_id |
defect |
| 129 |
"top ten withdrawals … by district names": gold sorts alphabetically (ORDER BY A2 ASC), not by amount. |
| 1113 |
question asks the defence aggression class; gold selects chanceCreationShootingClass. |
| 646 |
gold contains T2.PostId = T2.Id (a self-join typo) and filters posts.Score where the evidence defines comment Score. |
| 635 |
evidence states "more than 4 votes refers to PostId > 4" — the wrong column. |
| 1199 |
evidence defines the normal range on TP; gold filters ALB. |
| 271 |
gold filters element = 'c1' (typo for 'cl') and returns raw rows for a yes/no question. |
| 1006, 1011, 1014 |
gold re-derives lap durations by string-parsing the time column and disagrees with the sibling milliseconds column. |
| 1279 |
COUNT(CASE WHEN Diagnosis LIKE '%SLE%' THEN T1.ID ELSE 0 END) — COUNT counts non-NULLs and ELSE 0 is not NULL, so the numerator equals the denominator for every possible database state; with integer division the published value cannot express a percentage. |
Related evaluator reports filed separately: positional pairing of predictions, the silent financial fallback, the 30 s timeout ceiling of 0.9987, and the --diff_json_path flag.
Two questions: (1) do corrections to dev propagate to the annotation process for the hidden test set, and (2) is this tracker the preferred channel for further reports of this kind?
While validating the dev set (
dev.zip, sha256cdd6d19faeb45a23970b98d3ef6c40a87987c95459c2cf12076897a60cf5a630) we executed every gold query and diffed a sample of results against the questions as written. The items below look like annotation defects rather than modelling disagreements: the gold answers a different question than the one asked, or is internally broken.ORDER BY A2 ASC), not by amount.chanceCreationShootingClass.T2.PostId = T2.Id(a self-join typo) and filtersposts.Scorewhere the evidence defines comment Score.TP; gold filtersALB.element = 'c1'(typo for'cl') and returns raw rows for a yes/no question.timecolumn and disagrees with the siblingmillisecondscolumn.COUNT(CASE WHEN Diagnosis LIKE '%SLE%' THEN T1.ID ELSE 0 END)—COUNTcounts non-NULLs andELSE 0is not NULL, so the numerator equals the denominator for every possible database state; with integer division the published value cannot express a percentage.Related evaluator reports filed separately: positional pairing of predictions, the silent
financialfallback, the 30 s timeout ceiling of 0.9987, and the--diff_json_pathflag.Two questions: (1) do corrections to dev propagate to the annotation process for the hidden test set, and (2) is this tracker the preferred channel for further reports of this kind?