Skip to content

feat: count and, per binding, import a decimal written with the other separator - #56

Merged
vaceslav merged 2 commits into
mainfrom
feat/other-decimal-separator
Sep 28, 2026
Merged

vaceslav merged 2 commits into
mainfrom
feat/other-decimal-separator

Conversation

@vaceslav

Copy link
Copy Markdown
Contributor

Closes #47.

A value that writes the decimal separator the other way from the culture, and that can be read no other way, is now counted in the profile. It is imported only when the binding asks for it.

  • The rule (NumberReading.TryReadOtherSeparator): optional sign, digits, exactly one . or , (whichever separator the culture does not use for decimals), then digits that are not exactly three long, since three would make it a group. At most 28 digits. 34.020367 under de-DE reads as 34.020367. 1.234 is left to the culture, and so is anything holding a second separator, a space or an exponent.
  • Profile: CultureParseCounts.OtherSeparatorDecimals. Such values also stay among the numeric outliers, and the hypothesis confidence is unchanged: the facts stay measured.
  • Import: ColumnBinding.AcceptOtherDecimalSeparator, off by default. When it is on, a decimal field reads such values, and ExtractionSummary.OtherSeparatorDecimals counts them, so the leniency is never silent.
  • Precheck: it judges the plan with the setting. Both the type check and the rule checks read through the same ValueReading overload as the import.

Documented in docs/importing.md.

Performance. The profile asks the question of every value that failed as a number, in every culture. The first version cost +53 % on the 5M-row csv. After a single-pass, allocation-free shape check that runs only on values that could be numbers, with the separator looked up once per culture, the analysis timings on the corpus are within −4 % to +4 % of main (5M-row csv +2.3 %; median of 3 alternating runs, on a loaded machine).

Not done: the ranking tie-break for 41,181-style values mentioned in the issue. It would change which hypothesis comes first for existing files, and it needs its own discussion.

Tests: OtherDecimalSeparatorTests, nine cases.

@vaceslav
vaceslav merged commit b9ba1c5 into main Sep 28, 2026
9 checks passed
@vaceslav
vaceslav deleted the feat/other-decimal-separator branch September 28, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A column mixing decimal separators: one unambiguous value fails import

1 participant