Skip to content

fix(analysis): break ties between number readings by the sheet's evidence, and flag disagreement - #62

Merged
vaceslav merged 1 commit into
mainfrom
fix/ambiguous-numbers
Sep 29, 2026
Merged

vaceslav merged 1 commit into
mainfrom
fix/ambiguous-numbers

Conversation

@vaceslav

Copy link
Copy Markdown
Contributor

Closes #61. This is the numeric twin of #58.

A column of values with exactly three digits after one separator (48.137) reads completely both as decimals and as German grouped integers. The narrowest type used to win the tie (Integer/de-DE, 48137), while MinNumeric/MaxNumeric were measured under the first culture listed (48.137). The top hypothesis and the extremes described different numbers.

Changes

  • New fact ColumnFacts.NumberReadingsDisagree. It is true when the cultures that read the most values as numbers read some row as different numbers. It is measured with the same per-row, order-free fingerprint as DateReadingsDisagree.
  • Sheet-level evidence for the decimal separator, in this order:
    • the sheet's columns whose values are not in doubt, meaning a comma culture reads more of them as numbers than a point culture does, or the other way round;
    • a csv's delimiter: ; implies a comma, and , implies a point, because a comma-delimited file cannot use an unquoted comma for decimals.
  • Ranking, only where the readings disagree: the favoured separator's readings come first, and a decimal ranks above a grouped integer, because one group is weak evidence of grouping. Where the readings agree, the order is unchanged.
  • Extremes: MinNumeric/MaxNumeric are measured under the same choice, so they describe the reading that ranks first.

This also covers the 41,181 observation in #47.

Docs: docs/concepts.md gains "Which columns a sheet has", which notes that a value to the right of the header row makes a column with an empty header. It also gains "When readings tie", which explains the tie rules and both flags, and that a leading group may be any length (1234.567 under de-DE is 1234567).

Performance: on the corpus, within −3.2 % to +2.3 % of main (5M-row csv +2.3 %). The fingerprint hashes the decimal's raw bits, and skips a culture that reads numbers exactly as an earlier one does.

Tests: AmbiguousNumberTests, six cases: a comma-delimited csv, a semicolon-delimited csv, evidence from another column in both directions, xlsx text cells with no evidence, and columns where the readings agree.

…he sheet's evidence favours, and the facts say when number readings disagree
@vaceslav
vaceslav merged commit 8b14042 into main Sep 29, 2026
9 checks passed
@vaceslav
vaceslav deleted the fix/ambiguous-numbers branch September 29, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Three-digit decimals rank a grouped-integer reading first, and MinNumeric/MaxNumeric describe a different reading

1 participant