feat: count and, per binding, import a decimal written with the other separator - #56
Merged
Merged
Conversation
…separator where it can be read no other way
…bers, with the separator looked up once
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #47.
A value that writes the decimal separator the other way from the culture, and that can be read no other way, is now counted in the profile. It is imported only when the binding asks for it.
NumberReading.TryReadOtherSeparator): optional sign, digits, exactly one.or,(whichever separator the culture does not use for decimals), then digits that are not exactly three long, since three would make it a group. At most 28 digits.34.020367under de-DE reads as 34.020367.1.234is left to the culture, and so is anything holding a second separator, a space or an exponent.CultureParseCounts.OtherSeparatorDecimals. Such values also stay among the numeric outliers, and the hypothesis confidence is unchanged: the facts stay measured.ColumnBinding.AcceptOtherDecimalSeparator, off by default. When it is on, a decimal field reads such values, andExtractionSummary.OtherSeparatorDecimalscounts them, so the leniency is never silent.ValueReadingoverload as the import.Documented in
docs/importing.md.Performance. The profile asks the question of every value that failed as a number, in every culture. The first version cost +53 % on the 5M-row csv. After a single-pass, allocation-free shape check that runs only on values that could be numbers, with the separator looked up once per culture, the analysis timings on the corpus are within −4 % to +4 % of
main(5M-row csv +2.3 %; median of 3 alternating runs, on a loaded machine).Not done: the ranking tie-break for
41,181-style values mentioned in the issue. It would change which hypothesis comes first for existing files, and it needs its own discussion.Tests:
OtherDecimalSeparatorTests, nine cases.