fix(analysis): break ties between number readings by the sheet's evidence, and flag disagreement - #62
Merged
Merged
Conversation
…he sheet's evidence favours, and the facts say when number readings disagree
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #61. This is the numeric twin of #58.
A column of values with exactly three digits after one separator (
48.137) reads completely both as decimals and as German grouped integers. The narrowest type used to win the tie (Integer/de-DE, 48137), whileMinNumeric/MaxNumericwere measured under the first culture listed (48.137). The top hypothesis and the extremes described different numbers.Changes
ColumnFacts.NumberReadingsDisagree. It istruewhen the cultures that read the most values as numbers read some row as different numbers. It is measured with the same per-row, order-free fingerprint asDateReadingsDisagree.;implies a comma, and,implies a point, because a comma-delimited file cannot use an unquoted comma for decimals.MinNumeric/MaxNumericare measured under the same choice, so they describe the reading that ranks first.This also covers the
41,181observation in #47.Docs:
docs/concepts.mdgains "Which columns a sheet has", which notes that a value to the right of the header row makes a column with an empty header. It also gains "When readings tie", which explains the tie rules and both flags, and that a leading group may be any length (1234.567under de-DE is 1234567).Performance: on the corpus, within −3.2 % to +2.3 % of
main(5M-row csv +2.3 %). The fingerprint hashes the decimal's raw bits, and skips a culture that reads numbers exactly as an earlier one does.Tests:
AmbiguousNumberTests, six cases: a comma-delimited csv, a semicolon-delimited csv, evidence from another column in both directions, xlsx text cells with no evidence, and columns where the readings agree.