Skip to content

feat: DistinctValueCounts, and exact AffectedRows for precheck rule findings - #64

Merged
vaceslav merged 1 commit into
mainfrom
feat/distinct-value-counts
Sep 29, 2026
Merged

vaceslav merged 1 commit into
mainfrom
feat/distinct-value-counts

Conversation

@vaceslav

Copy link
Copy Markdown
Contributor

Closes #63.

  • ColumnFacts.DistinctValueCounts: every value of DistinctValues with the exact number of rows holding it, in the same (first-seen) order. It is empty whenever DistinctValuesAreComplete is false, and when complete the counts add up to NonEmptyCount. Values are counted as written and trimmed, so Wood and wood are two entries.
  • MappingPrecheck rule findings (allowed values, ranges, patterns, lengths, custom rules) now carry AffectedRows, the rows whose value fails, taken from those counts. The bound is RowCountBound.Exact. Without complete counts, AffectedRows stays null as before.

How it works: the profiler already does one frequency-dictionary lookup per non-empty value. While the distinct set is complete, that dictionary now counts every kept value, up to RetainedDistinctValues. The moment the set stops being complete, the dictionary shrinks back to the 64-key sample of its most frequent entries, so a high-cardinality column costs what it did before.

Performance. Analysis, median of 3 alternating runs against main on a quiet machine (load average about 3–7), with /usr/bin/time for peak memory:

file time peak memory
5M-row csv, 17 columns 13,009 → 13,002 ms (−0.1 %) 155 → 164 MB
3M-row csv, 17 columns 9,475 → 9,668 ms (+2.0 %) 154 → 155 MB
1M-row workbook, 17 columns 7,232 → 7,084 ms (−2.0 %) 204 → 204 MB
100k-row workbook, 28 columns 1,551 → 1,504 ms (−3.0 %) 94 → 102 MB

A first version, before the shrink, cost +2.6 % on the 5M-row csv.

Tests: DistinctValueCountsTests covers:

  • counts in first-seen order;
  • counts past the 64-key sample;
  • no counts once the set is incomplete;
  • exact AffectedRows for a case-insensitive allowed-values rule and for AtLeast/AtMost.

@vaceslav
vaceslav merged commit 52ef740 into main Sep 29, 2026
9 checks passed
@vaceslav
vaceslav deleted the feat/distinct-value-counts branch September 29, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Row counts per distinct value, and exact affected rows in precheck rule findings

1 participant