diff --git a/PLAN.md b/PLAN.md index 535371c..d4fcfc7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -464,7 +464,7 @@ So parallel training alone was worth **1.06×** — real work, drowned by a bott The baseline scores 0.594, so the headroom above it collapses from 0.406 to 0.225: **45% of the achievable gain, lost in silence**. `parseNumber` ends in `Number(cleaned)`, `Number('12,5')` is `NaN`, the column falls through, and V24's TF-IDF cheerfully tokenises digits into a hundred word features. Nothing warned, nothing refused — the pipeline just produced a worse model. **Three of this row's own predictions were wrong, and the measurement corrected them**: the mis-typed column becomes `text` on high cardinality but `categorical` (one-hot) on low, not `text` alone; a windows-1252 file does not display `Québec` — that is the reverse case (UTF-8 read as cp1252) — it yields U+FFFD replacement characters; and day-first dates were already handled by V8, `31/12/2025` parsing correctly all along, with only the dash form `31-12-2025` returning null. **The fix.** One reader, shared by the ML Lab and the Data Studio, that decides by evidence rather than by guessing the user's locale: the browser's language says nothing about the file someone dragged in. Encoding is settled by **trying and failing** — UTF-8 in `fatal` mode throws on cp1252 accents, so the fallback is a certainty, not a preference; the delimiter is the candidate that splits every sampled line into the same number of columns, quotes respected; and the decimal separator is decided **per column**, never per file. A column is rewritten only when at least 90% of its values are numbers in that form AND it carries a comma AND it would otherwise not be numeric at all — so a text column containing « vis, tête plate » comes out untouched, and a column of bare integers is left alone because rewriting it would be a change with no cause. Values that do not match the pattern are never rewritten: a stray « n/d » stays « n/d » rather than becoming a plausible-looking number. Detection reads only the **head** of the file, so V25's streaming abort survives intact and a 2 GB file still stops at the cell budget. The reading is then **announced with its evidence** — « virgule décimale détectée et convertie dans 2 colonnes : surface (400/400), prix (400/400) » — above a five-row preview, and only when the reading was not the plain default: an ordinary UTF-8 comma file gets no card, no confirmation step, no friction. Batch scoring, drift and join files go through the same reader, because comparing a French export against a normalised dataset would otherwise report drift that is nothing but a decimal separator. **What this wave deliberately does not do**: guess a locale from `navigator.language` (the file has no relationship to the browser's language), rewrite a column on a bare majority (below the 90% floor the evidence is not evidence), or touch values individually inside a column the evidence does not cover. 446 unit tests, 73 e2e — of the 21 new reader tests, five assert that it **refuses** to act. | Owner request (22/08/2026): what to improve in /data. The audit found a defect first, and the wave began by reproducing it end to end: a French-locale CSV — the single most likely file this owner's users will open — silently loses every numeric column. The repair is exact rather than approximate: the French file now trains to the same types, the same feature count and the same score, to ten decimal places, as the file that never had the problem. | | **V39 — delivered** | **Data Studio: a recipe that works column by column.** `RecipeOptions` applied `missing` and `clipOutliers` to the **whole file** — one strategy for every column, however different they are. A median makes sense for an age and none at all for a postcode. The recipe is now an ordered list of **per-column steps**, with the file-wide settings demoted to **defaults a column may override**: a column with no entry behaves exactly as it did before V39, which is what keeps every previously exported recipe valid and replayable. The missing strategies that were absent are there: **median, mean, most-frequent, a constant used verbatim, and a « MANQUANT » category** for categorical columns — absence becoming a level of its own rather than a guess. Clipping became a per-column decision on the same terms, so a column can opt in or out of the global flag. **The rule the wave exists to enforce: imputing without marking destroys information.** A blank field is rarely blank at random, and the fact of the blank is frequently predictive on its own, so every column may add a `_absent` indicator — and the ordering is the whole point: **every indicator is written before any blank is filled**, so it records where the blanks really were, never where some other column's rule left them. Three consequences fall out of that and are tested: a column with nothing missing gets no indicator (an all-zero column is noise, not information); rows dropped by one column's rule are gathered and removed **once**, so indicators stay aligned instead of being shifted by a sibling column; and when a strategy cannot be honoured — a median over a column holding no parseable number, a constant with no value typed — the blanks are **left blank** rather than filled with something invented. Because the indicator is optional, the studio **announces the columns it filled without marking**, by name, instead of quietly producing a tidier table. **What this wave deliberately does not do**: make the indicator mandatory (it would silently add columns to every existing recipe, and imposing is not the same as announcing), impute from a model (opaque, and it fabricates values that look plausible — already refused for V40), or reorder the recipe's fixed stages, which is what keeps toggling an option from compounding with the last one. 461 unit tests, 75 e2e. | A single global strategy is the kind of default that looks tidy and quietly makes the data worse; per-column steps cost little to build because the recipe was already an object, not a pile of checkboxes. The build confirmed it: the engine change is contained in one stage of `applyRecipe`, and the 17 pre-existing recipe tests passed untouched. | -| V40 | **Data Studio: validity, drift, and an auditable diff.** Quality is measured today as completeness and consistency of type; what is missing is **validity** — a value can be present, well-typed and still impossible. Named rules, each stated in plain language: an age of 200, a date in the future, a percentage at 130, a malformed postcode. Then **cross-column consistency** (`date_fin < date_debut`, `total ≠ quantité × prix`), for which **V29's DuckDB is already the engine** — the rules are SQL, and they run on the file that is already registered. Then three things that make the studio auditable rather than merely helpful: a **replayable reference profile** so a second file can be checked for drift against the first (the same idea as the V22 model manifest), a **before/after diff of the rows a recipe modified** — which rows, which columns, which values, not just a count — and a **breakdown of the quality score** so the number is explained by its parts instead of being asserted. Ends with **Parquet export**, nearly free now that DuckDB is loaded (`COPY … TO 'x.parquet'`). **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). | Comes last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would flag the parser, not the data. | +| **V40 — delivered** | **Data Studio: validity, drift, and an auditable diff.** Quality was measured as completeness and consistency of type; what was missing is **validity** — a value can be present, correctly typed and still impossible. Five named rules now say so in plain language: an age outside 0–120, a date in the future, a percentage outside 0–100, a negative amount, a malformed postcode. Then **cross-column consistency**: an end date before its start, a total that is not quantity × price — every cell fine, the ROW impossible. **The plan's choice of engine was wrong, and building it showed why.** It proposed running the consistency rules through V29's DuckDB, since the file is already registered there. But DuckDB is an announced, opt-in 18–22 MB download: routing these checks through it would have made a universally applicable check conditional on a large download most users will decline, leaving the panel empty for them. Comparing two columns of a table already in memory is a loop, so it is a loop, and every user gets it. DuckDB keeps the job it is genuinely needed for — arbitrary SQL, and the new **Parquet export** (`COPY … TO`, one call, bytes straight to a download). Both families of rules obey the two laws V38's reader established: **they fire on evidence, never on a column's name alone** — a column called `age` holding 20 000 is a duration in days, so the rule checks that most of the column is plausible before it flags the rest, and stays silent otherwise — and **they report without ever repairing**, because V39's recipe is the one record of what was done to the data. Three things then make the studio auditable rather than merely helpful. A **before/after diff** naming which rows, which columns and which values changed: the hard part is that a recipe drops rows and adds columns, so `applyRecipe` now returns which SOURCE row each surviving row came from — without it the diff would pair row 7 with a different row 7 and report a screen full of changes that never happened. A **replayable reference profile**, the V22 manifest idea applied to data: bin edges and shares, never rows, so a profile of a payroll file describes the shape of the salary distribution and nobody's salary — which is what makes it safe to commit beside the code, and it scores a new file to the same PSI, to six decimal places, as V11's live two-file comparison. And a **score broken into its parts**, each with its weight and what it actually cost. The weights now sum to **105, not 100**, deliberately: validity brought its own 5 points rather than taking them from an existing part, because redistributing would have quietly changed what every previously published score meant. **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). 502 unit tests, 78 e2e. | Came last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would have flagged the parser, not the data. Closes the Data Studio group. Of the 36 new unit tests, eleven assert that a rule REFUSES to fire — a rule that flags a good file is worse than no rule, because it teaches the reader to ignore the panel. | **Ordering**: V38 came before V39 and V40, and for the same reason V35 came first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35, V36 and V37 are delivered. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 and V31 both start with a bench, because neither « a bigger model » nor « it still makes mistakes » is a measurable statement today; no wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped — V25 (delivered) chose announced sampling and a named memory guard over the diff --git a/README.md b/README.md index 6434d69..2046208 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,16 @@ The project follows three non-negotiable principles: column can add a `_absent` **missing indicator**, written before anything is filled; columns filled _without_ one are named out loud, because a blank field is rarely blank at random and filling it silently erases that. +- **Validity rules**: a value can be present, correctly typed and still impossible — an + age of 200, a date in the future, a percentage at 130, a malformed postcode. Plus + **cross-column consistency**: an end date before its start, a total that is not + quantity × price. Every rule fires on evidence rather than on a column's name, and + reports without ever repairing — the recipe is where data changes. +- **The quality score, broken into its parts**: each with its weight, what it found and + what it cost, instead of a number asserted without explanation. +- **An auditable before/after diff** of what the recipe did — which rows, which columns, + which values — and a **replayable reference profile** (bin edges and shares, never + rows) so a new file can be checked for drift against a snapshot you no longer hold. - **Left-join a second file** on a shared key: match rate, duplicates and orphans are named, never silent; the joined result becomes the working dataset. - **Drift check**: compare a new batch against the reference — schema diff, PSI per @@ -124,7 +134,7 @@ The project follows three non-negotiable principles: - **Performance.** Every section serves a prerendered static shell (hero paints before JavaScript); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 461 unit tests, 75 Playwright end-to-end tests (including offline PWA, +- **Quality bar.** 502 unit tests, 78 Playwright end-to-end tests (including offline PWA, fake-webcam and axe-core WCAG A/AA accessibility checks), strict TypeScript, ESLint, Prettier, and Lighthouse budgets — all enforced in CI. diff --git a/e2e/validity.spec.ts b/e2e/validity.spec.ts new file mode 100644 index 0000000..fa1ea3b --- /dev/null +++ b/e2e/validity.spec.ts @@ -0,0 +1,71 @@ +import { expect, test } from '@playwright/test'; + +test.use({ locale: 'en-US' }); +test.setTimeout(120_000); + +/** + * V40 — validity, consistency, and a score that explains itself. + */ +function impossibleCsv(): Buffer { + const lines = ['nom,age,taux_reussite,date_debut,date_fin,quantite,prix_unitaire,total']; + for (let i = 0; i < 60; i++) { + // Two impossible ages, one impossible percentage, one reversed date pair, + // one total that is not quantity × price — everything else is plausible. + const age = i === 3 ? '200' : i === 7 ? '-4' : String(20 + (i % 50)); + const rate = i === 11 ? '130' : String(i % 100); + const start = `2025-0${(i % 9) + 1}-0${(i % 8) + 1}`; + const end = i === 5 ? '2024-01-01' : `2025-0${(i % 9) + 1}-0${(i % 8) + 2}`; + const qty = String(1 + (i % 5)); + const unit = String(2 + (i % 4)); + const total = i === 9 ? '999' : String((1 + (i % 5)) * (2 + (i % 4))); + lines.push(`ligne${i},${age},${rate},${start},${end},${qty},${unit},${total}`); + } + return Buffer.from(lines.join('\n'), 'utf-8'); +} + +test('impossible values and contradictory rows are named, with their rules', async ({ page }) => { + await page.goto('/data'); + await page.getByTestId('data-file-input').setInputFiles({ + name: 'commandes.csv', + mimeType: 'text/csv', + buffer: impossibleCsv(), + }); + await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 }); + + // Values that are present and correctly typed, and still impossible. + const validity = page.getByTestId('issue-validity'); + await expect(validity).toBeVisible(); + await expect(validity).toContainText('age'); + await expect(validity).toContainText('age outside 0–120'); + await expect(validity).toContainText('percentage outside 0–100'); + + // A row where two columns contradict each other. + const consistency = page.getByTestId('issue-consistency'); + await expect(consistency).toBeVisible(); + await expect(consistency).toContainText('the end comes before the start'); + await expect(consistency).toContainText('the total is not quantity × price'); +}); + +test('the quality score is explained by its parts, not asserted', async ({ page }) => { + await page.goto('/data'); + await page.getByRole('button', { name: /cafe-sales\.csv/ }).click(); + await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 }); + + await page.getByText('How this score was computed', { exact: true }).click(); + const breakdown = page.getByTestId('score-breakdown'); + await expect(breakdown).toContainText('Missing cells'); + await expect(breakdown).toContainText('Duplicate rows'); + await expect(breakdown).toContainText('Impossible values'); + // The weights are shown, including the deliberate 105 total. + await expect(breakdown).toContainText('105, not 100'); +}); + +test('a clean file raises no validity or consistency card at all', async ({ page }) => { + await page.goto('/data'); + // titanic has missing values and messy spellings, but every age is a real + // age and no two columns contradict each other. + await page.getByRole('button', { name: /titanic\.csv/ }).click(); + await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 }); + await expect(page.getByTestId('issue-validity')).toHaveCount(0); + await expect(page.getByTestId('issue-consistency')).toHaveCount(0); +}); diff --git a/src/features/data/components/DataDropZone.tsx b/src/features/data/components/DataDropZone.tsx index 9561bd4..bdde50c 100644 --- a/src/features/data/components/DataDropZone.tsx +++ b/src/features/data/components/DataDropZone.tsx @@ -46,6 +46,7 @@ export function DataDropZone() { {t('data.drop.hint')} @@ -166,8 +168,76 @@ export function QualitySummary() { )} + + {report.validity.length > 0 && ( + + + )} + + {report.consistency.length > 0 && ( + + + )} )} + +
+ + {t('data.score.breakdownTitle')} + +

{t('data.score.breakdownHint')}

+ + + + + + + + + + + {report.breakdown.map((part) => ( + + + + + + + ))} + +
{t('data.score.part')}{t('data.score.found')}{t('data.score.weight')}{t('data.score.cost')}
{t(`data.score.parts.${part.part}`)} + {NUMBER.format(part.count)} + {part.ratio !== undefined && ` (${percent(part.ratio)} %)`} + {part.weight}−{part.penalty}
+
); } diff --git a/src/features/data/quality/checks.ts b/src/features/data/quality/checks.ts index 5d539ef..422510b 100644 --- a/src/features/data/quality/checks.ts +++ b/src/features/data/quality/checks.ts @@ -1,3 +1,5 @@ +import { checkValidity, invalidCellCount } from '@/features/data/quality/validity'; +import { checkConsistency, inconsistentRowCount } from '@/features/data/quality/consistency'; import { inferColumnType, isMissing, parseNumber } from '@/features/ml/data/infer'; import type { Cell, ColumnType } from '@/features/ml/data/types'; import type { @@ -114,19 +116,83 @@ function missingCountOf(values: Cell[]): number { } /** - * Deterministic 0–100 score. Each ratio saturates at 12.5% (×8) so a modest - * amount of dirt is already visible; weights sum to 100. + * V40: the score, decomposed. + * + * The number was asserted before: 62 out of 100, with nothing saying why. Each + * part now carries its own weight, the ratio that drove it, and the points it + * actually cost — so the score is explained by its parts instead of being + * announced. The arithmetic is unchanged apart from the new validity part; a + * file with no validity findings scores exactly what it scored before V40. */ -export function qualityScore(report: Omit): number { +export type ScorePart = 'missing' | 'duplicates' | 'messy' | 'outliers' | 'structural' | 'validity'; + +export interface ScoreBreakdown { + part: ScorePart; + /** The most this part can ever cost. */ + weight: number; + /** What it cost here, rounded to one decimal. */ + penalty: number; + /** The ratio that drove it — absent for the structural count. */ + ratio?: number; + /** The raw count behind the ratio, for the plain-language line. */ + count: number; +} + +/** Each ratio saturates at 12.5% (×8) so a modest amount of dirt is visible. */ +const SATURATION = 8; +/** Sum of every weight below. Above 100 on purpose — see the validity part. */ +export const TOTAL_WEIGHT = 105; + +export function scoreBreakdown( + report: Omit, + invalidCells = 0, +): ScoreBreakdown[] { const cells = Math.max(1, report.cellCount); const rows = Math.max(1, report.rowCount); - const saturate = (ratio: number) => Math.min(1, ratio * 8); - const penalty = - 35 * saturate(report.missingCells / cells) + - 20 * saturate(report.duplicateRows / rows) + - 20 * saturate(report.messyCells / cells) + - 15 * saturate(report.outlierCells / cells) + - Math.min(10, 2.5 * report.structural.length); + const saturate = (ratio: number) => Math.min(1, ratio * SATURATION); + const round = (value: number) => Math.round(value * 10) / 10; + const ratioPart = ( + part: ScorePart, + weight: number, + count: number, + total: number, + ): ScoreBreakdown => { + const ratio = count / total; + return { part, weight, penalty: round(weight * saturate(ratio)), ratio, count }; + }; + return [ + ratioPart('missing', 35, report.missingCells, cells), + ratioPart('duplicates', 20, report.duplicateRows, rows), + ratioPart('messy', 20, report.messyCells, cells), + ratioPart('outliers', 15, report.outlierCells, cells), + { + part: 'structural', + weight: 10, + penalty: round(Math.min(10, 2.5 * report.structural.length)), + count: report.structural.length, + }, + // V40: validity is a NEW dimension, so it brings its own 5 points rather + // than taking them from an existing part. The weights therefore sum to 105, + // not 100, and that is deliberate: redistributing would have quietly + // changed what every previously published score meant. A file with no + // validity findings scores exactly what it scored before V40 — the total + // still floors at 0, which the previous formula could already reach. + ratioPart('validity', 5, invalidCells, cells), + ]; +} + +/** + * Deterministic 0–100 score, now the sum of its published parts rather than a + * separate formula that could drift away from them. + */ +export function qualityScore( + report: Omit, + invalidCells = 0, +): number { + const penalty = scoreBreakdown(report, invalidCells).reduce( + (total, part) => total + part.penalty, + 0, + ); return Math.max(0, Math.round(100 - penalty)); } @@ -198,7 +264,14 @@ export function buildQualityReport(header: string[], columns: Cell[][]): Quality messyColumns.sort((a, b) => b.cellCount - a.cellCount || a.column.localeCompare(b.column)); outlierColumns.sort((a, b) => b.count - a.count || a.column.localeCompare(b.column)); - const partial: Omit = { + // V40: the two families of impossible data — one column at a time, then + // rows where two columns contradict each other. + const validity = checkValidity(header, columns); + const invalidCells = invalidCellCount(validity); + const consistency = checkConsistency(header, columns); + const inconsistentRows = inconsistentRowCount(consistency); + + const partial: Omit = { rowCount, columnCount, cellCount, @@ -210,6 +283,14 @@ export function buildQualityReport(header: string[], columns: Cell[][]): Quality outlierColumns, outlierCells, structural, + validity, + invalidCells, + consistency, + inconsistentRows, + }; + return { + ...partial, + score: qualityScore(partial, invalidCells), + breakdown: scoreBreakdown(partial, invalidCells), }; - return { ...partial, score: qualityScore(partial) }; } diff --git a/src/features/data/quality/clean.ts b/src/features/data/quality/clean.ts index 2bf2210..b000460 100644 --- a/src/features/data/quality/clean.ts +++ b/src/features/data/quality/clean.ts @@ -135,9 +135,13 @@ export function applyRecipe( header: string[], source: Cell[][], options: RecipeOptions, -): { header: string[]; columns: Cell[][]; stats: CleanStats } { +): { header: string[]; columns: Cell[][]; stats: CleanStats; survivingRows: number[] } { let outHeader = [...header]; let columns = source.map((column) => [...column]); + // V40: the index in `source` that each current row came from. Every step + // that drops rows filters this alongside the data, so the diff can attribute + // a change to the row it actually came from. + let surviving = Array.from({ length: source[0]?.length ?? 0 }, (_, index) => index); // Forced types steer every type-sensitive step; inference is the fallback. const typeOf = (name: string, values: Cell[]) => { const overrides = options.types as Partial> | undefined; @@ -247,7 +251,10 @@ export function applyRecipe( if (options.dropDuplicates && columns.length > 0) { const duplicates = new Set(duplicateRowIndices(columns, columns[0].length)); stats.droppedDuplicateRows = duplicates.size; - if (duplicates.size > 0) columns = dropRows(columns, duplicates); + if (duplicates.size > 0) { + columns = dropRows(columns, duplicates); + surviving = surviving.filter((_, index) => !duplicates.has(index)); + } } // V39: missing values, column by column. Two passes, and the order between @@ -325,6 +332,7 @@ export function applyRecipe( stats.droppedByColumn[source.column] = source.rows.length; } columns = dropRows(columns, toDrop); + surviving = surviving.filter((_, index) => !toDrop.has(index)); for (const indicator of indicators) { indicator.values = indicator.values.filter((_, index) => !toDrop.has(index)); } @@ -377,11 +385,15 @@ export function applyRecipe( } if (stats.droppedAnomalyRows > 0) { columns = columns.map((column) => keep.map((r) => column[r])); + surviving = keep.map((r) => surviving[r]); } } } stats.rowCount = columns[0]?.length ?? 0; stats.columnCount = outHeader.length; - return { header: outHeader, columns, stats }; + // V40: which SOURCE row each surviving row came from. Without this the diff + // would pair row 7 with a different row 7 the moment anything was dropped, + // and report every subsequent row as changed. + return { header: outHeader, columns, stats, survivingRows: surviving }; } diff --git a/src/features/data/quality/consistency.test.ts b/src/features/data/quality/consistency.test.ts new file mode 100644 index 0000000..d8ee42c --- /dev/null +++ b/src/features/data/quality/consistency.test.ts @@ -0,0 +1,106 @@ +/** + * V40 — cross-column consistency, in JavaScript rather than in SQL. + */ +import { describe, expect, it } from 'vitest'; +import { + checkConsistency, + inconsistentRowCount, + PRODUCT_TOLERANCE, +} from '@/features/data/quality/consistency'; +import type { Cell } from '@/features/ml/data/types'; + +describe('V40 — a row can be impossible even when every cell is fine', () => { + it('flags an end date before its start date', () => { + const header = ['date_debut', 'date_fin']; + const columns: Cell[][] = [ + ['2025-01-04', '2025-02-01', '2025-03-10', '2025-04-01', '2025-05-01', '2025-06-01'], + ['2025-01-09', '2025-02-05', '2025-02-28', '2025-04-10', '2025-05-06', '2025-06-08'], + ]; + const [found] = checkConsistency(header, columns); + expect(found.rule).toBe('dateOrder'); + expect(found.columns).toEqual(['date_debut', 'date_fin']); + expect(found.rows).toEqual([2]); + expect(found.examples).toEqual(['2025-03-10 → 2025-02-28']); + }); + + it('flags a total that is not quantity × unit price', () => { + const header = ['quantite', 'prix_unitaire', 'total']; + const columns: Cell[][] = [ + ['2', '3', '1', '4', '5', '2'], + ['5', '2.5', '10', '1.25', '2', '3'], + ['10', '7.5', '99', '5', '10', '6'], + ]; + const [found] = checkConsistency(header, columns); + expect(found.rule).toBe('productMismatch'); + expect(found.rows).toEqual([2]); + expect(found.examples).toEqual(['1 × 10 ≠ 99']); + }); + + it('tolerates ordinary rounding on money', () => { + const header = ['qty', 'unit_price', 'total']; + const columns: Cell[][] = [ + ['3', '3', '3', '3', '3', '3'], + ['0.3333', '0.3333', '0.3333', '0.3333', '0.3333', '0.3333'], + ['1', '1', '1', '1', '1', '1'], + ]; + // 3 × 0.3333 = 0.9999, rounded to 1 — a rounded total is not a wrong total. + expect(checkConsistency(header, columns)).toEqual([]); + expect(PRODUCT_TOLERANCE).toBeLessThanOrEqual(0.01); + }); +}); + +describe('V40 — consistency rules refuse to fire on a file they misread', () => { + it('REFUSES when most rows disagree — the columns are not what it assumed', () => { + // `total` here is a running balance, not quantity × price. + const header = ['quantite', 'prix_unitaire', 'total']; + const columns: Cell[][] = [ + ['2', '3', '1', '4', '5', '2'], + ['5', '2.5', '10', '1.25', '2', '3'], + ['100', '250', '380', '420', '600', '750'], + ]; + expect(checkConsistency(header, columns)).toEqual([]); + }); + + it('REFUSES a date pair whose values are not dates', () => { + const header = ['debut', 'fin']; + const columns: Cell[][] = [ + ['tôt', 'tôt', 'tard', 'tôt', 'tard', 'tôt'], + ['tard', 'tard', 'tôt', 'tard', 'tôt', 'tard'], + ]; + expect(checkConsistency(header, columns)).toEqual([]); + }); + + it('does not charge a row twice when one endpoint is simply missing', () => { + const header = ['date_debut', 'date_fin']; + const columns: Cell[][] = [ + ['2025-01-04', '2025-02-01', '2025-03-10', '2025-04-01', '2025-05-01', '2025-06-01'], + ['2025-01-09', '', '2025-04-01', '2025-04-10', '2025-05-06', '2025-06-08'], + ]; + // Row 1 is missing, not inconsistent — the missing report already has it. + expect(checkConsistency(header, columns)).toEqual([]); + }); + + it('says nothing when the columns it needs are absent', () => { + expect( + checkConsistency( + ['a', 'b'], + [ + ['1', '2'], + ['3', '4'], + ], + ), + ).toEqual([]); + expect(inconsistentRowCount([])).toBe(0); + }); + + it('never modifies the data it inspects', () => { + const header = ['date_debut', 'date_fin']; + const columns: Cell[][] = [ + ['2025-03-10', '2025-01-04', '2025-02-01', '2025-04-01', '2025-05-01', '2025-06-01'], + ['2025-02-28', '2025-01-09', '2025-02-05', '2025-04-10', '2025-05-06', '2025-06-08'], + ]; + const snapshot = columns.map((column) => [...column]); + checkConsistency(header, columns); + expect(columns).toEqual(snapshot); + }); +}); diff --git a/src/features/data/quality/consistency.ts b/src/features/data/quality/consistency.ts new file mode 100644 index 0000000..8c873ec --- /dev/null +++ b/src/features/data/quality/consistency.ts @@ -0,0 +1,191 @@ +/** + * V40: cross-column consistency — rows where two columns contradict each other. + * + * A delivery date before its order date, a total that is not quantity × price: + * every value is present, correctly typed, individually plausible, and the ROW + * is still impossible. Validity checks one column at a time and cannot see it. + * + * **Why this is not SQL.** The plan proposed running these rules through V29's + * DuckDB, since the file is already registered there. Building it made the cost + * obvious: DuckDB is an announced, opt-in 18–22 MB download, so routing these + * checks through it would make a universally applicable check conditional on a + * large download the user may well decline — a quality panel that is empty for + * most people. Comparing two columns of a table already in memory is a loop, so + * it is a loop, and every user gets it. DuckDB keeps the job it is genuinely + * needed for: arbitrary SQL, and the Parquet export. + * + * The rules follow the same two laws as `validity.ts`: they fire on evidence + * rather than on column names alone, and they report without ever repairing. + */ +import { isMissing, parseNumber } from '@/features/ml/data/infer'; +import { parseDate } from '@/features/ml/timeseries/series'; +import { MAX_EXAMPLE_ROWS, APPLICABILITY } from '@/features/data/quality/validity'; +import type { Cell } from '@/features/ml/data/types'; + +export type ConsistencyRule = + /** An "end" date strictly before its matching "start" date. */ + | 'dateOrder' + /** A total that does not match quantity × unit price. */ + | 'productMismatch'; + +export interface ConsistencyFinding { + rule: ConsistencyRule; + /** The columns the rule compared, in the order it compared them. */ + columns: string[]; + rows: number[]; + count: number; + /** One rendered example per listed row, e.g. « 2025-01-04 → 2024-12-30 ». */ + examples: string[]; +} + +/** Relative tolerance on the product rule — money is rounded, not exact. */ +export const PRODUCT_TOLERANCE = 0.01; + +function normalize(column: string): string { + return column.toLowerCase().normalize('NFD').replace(/[̀-ͯ]/g, ''); +} + +const START_HINTS = ['debut', 'start', 'commande', 'order', 'creation', 'created', 'from'] as const; +const END_HINTS = ['fin', 'end', 'livraison', 'delivery', 'cloture', 'closed', 'to'] as const; +const QUANTITY_HINTS = ['quantity', 'quantite', 'qte', 'qty', 'nombre'] as const; +const UNIT_PRICE_HINTS = [ + 'unit_price', + 'prix_unitaire', + 'unitprice', + 'prixunitaire', + 'pu', +] as const; +const TOTAL_HINTS = ['total', 'montant', 'amount', 'sum'] as const; + +function findColumn(header: readonly string[], hints: readonly string[]): number { + return header.findIndex((name) => hints.some((hint) => normalize(name).includes(hint))); +} + +/** Parses a whole column as dates, or returns null when it is not one. */ +function asDates(values: Cell[]): (number | null)[] | null { + const parsed: (number | null)[] = []; + let usable = 0; + let ok = 0; + for (const value of values) { + if (isMissing(value)) { + parsed.push(null); + continue; + } + usable += 1; + const at = parseDate((value as string).trim()); + parsed.push(at); + if (at !== null) ok += 1; + } + if (usable === 0 || ok / usable < APPLICABILITY) return null; + return parsed; +} + +function asNumbers(values: Cell[]): (number | null)[] | null { + const parsed: (number | null)[] = []; + let usable = 0; + let ok = 0; + for (const value of values) { + if (isMissing(value)) { + parsed.push(null); + continue; + } + usable += 1; + const number = parseNumber((value as string).trim()); + parsed.push(number); + if (number !== null) ok += 1; + } + if (usable === 0 || ok / usable < APPLICABILITY) return null; + return parsed; +} + +function build( + rule: ConsistencyRule, + columns: string[], + failures: { row: number; example: string }[], +): ConsistencyFinding | null { + if (failures.length === 0) return null; + const capped = failures.slice(0, MAX_EXAMPLE_ROWS); + return { + rule, + columns, + rows: capped.map((f) => f.row), + examples: capped.map((f) => f.example), + count: failures.length, + }; +} + +export function checkConsistency( + header: readonly string[], + columns: Cell[][], +): ConsistencyFinding[] { + const found: ConsistencyFinding[] = []; + + const startAt = findColumn(header, START_HINTS); + const endAt = findColumn(header, END_HINTS); + if (startAt >= 0 && endAt >= 0 && startAt !== endAt) { + const start = asDates(columns[startAt] ?? []); + const end = asDates(columns[endAt] ?? []); + if (start && end) { + const failures: { row: number; example: string }[] = []; + for (let row = 0; row < start.length; row++) { + const from = start[row]; + const to = end[row]; + // A missing endpoint is a MISSING value, already reported as one: + // counting it here too would charge the same row twice. + if (from === null || to === null) continue; + if (to < from) { + failures.push({ + row, + example: `${String(columns[startAt][row])} → ${String(columns[endAt][row])}`, + }); + } + } + const finding = build('dateOrder', [header[startAt], header[endAt]], failures); + if (finding) found.push(finding); + } + } + + const qtyAt = findColumn(header, QUANTITY_HINTS); + const unitAt = findColumn(header, UNIT_PRICE_HINTS); + const totalAt = findColumn(header, TOTAL_HINTS); + if (qtyAt >= 0 && unitAt >= 0 && totalAt >= 0 && totalAt !== unitAt) { + const qty = asNumbers(columns[qtyAt] ?? []); + const unit = asNumbers(columns[unitAt] ?? []); + const total = asNumbers(columns[totalAt] ?? []); + if (qty && unit && total) { + const failures: { row: number; example: string }[] = []; + let comparable = 0; + for (let row = 0; row < qty.length; row++) { + const q = qty[row]; + const u = unit[row]; + const t = total[row]; + if (q === null || u === null || t === null) continue; + comparable += 1; + const expected = q * u; + // Relative tolerance, because a rounded total is not a wrong total. + const slack = Math.max(Math.abs(expected) * PRODUCT_TOLERANCE, 0.01); + if (Math.abs(t - expected) > slack) { + failures.push({ row, example: `${q} × ${u} ≠ ${t}` }); + } + } + // If most rows disagree, these three columns are not quantity, price and + // total — the rule has misread the file and says nothing rather than + // flagging every row of it. + if (comparable > 0 && failures.length / comparable <= 1 - APPLICABILITY) { + const finding = build( + 'productMismatch', + [header[qtyAt], header[unitAt], header[totalAt]], + failures, + ); + if (finding) found.push(finding); + } + } + } + + return found.sort((a, b) => b.count - a.count); +} + +/** Impossible rows across every consistency rule — what the score charges for. */ +export function inconsistentRowCount(findings: readonly ConsistencyFinding[]): number { + return findings.reduce((total, finding) => total + finding.count, 0); +} diff --git a/src/features/data/quality/diff.test.ts b/src/features/data/quality/diff.test.ts new file mode 100644 index 0000000..63173e6 --- /dev/null +++ b/src/features/data/quality/diff.test.ts @@ -0,0 +1,103 @@ +/** + * V40 — the diff is evidence, where a count was only a claim. + * + * The test that matters most is the alignment one: a diff that pairs the wrong + * rows after a drop is worse than no diff at all, because it reports a screen + * full of changes that never happened. + */ +import { describe, expect, it } from 'vitest'; +import { applyRecipe } from '@/features/data/quality/clean'; +import { diffRecipe } from '@/features/data/quality/diff'; +import { DEFAULT_RECIPE, type RecipeOptions } from '@/features/data/quality/types'; +import type { Cell } from '@/features/ml/data/types'; + +const QUIET: RecipeOptions = { + ...DEFAULT_RECIPE, + trimWhitespace: false, + mergeVariants: false, + dropDuplicates: false, + missing: 'keep', +}; + +function run(header: string[], columns: Cell[][], options: RecipeOptions) { + const result = applyRecipe(header, columns, options); + return { + result, + diff: diffRecipe(header, columns, result.header, result.columns, result.survivingRows), + }; +} + +describe('V40 — which rows, which columns, which values', () => { + it('names the cell that changed, with its value before and after', () => { + const header = ['ville', 'age']; + const columns: Cell[][] = [ + [' Québec ', 'Montréal', 'Laval'], + ['34', '41', '29'], + ]; + const { diff } = run(header, columns, { ...QUIET, trimWhitespace: true }); + expect(diff.changedRowCount).toBe(1); + expect(diff.changedRows[0]).toEqual({ + sourceRow: 0, + changes: [{ column: 'ville', before: ' Québec ', after: 'Québec' }], + }); + expect(diff.changedCells).toBe(1); + }); + + it('reports nothing at all when the recipe changed nothing', () => { + const header = ['a']; + const columns: Cell[][] = [['1', '2', '3']]; + const { diff } = run(header, columns, QUIET); + expect(diff.changedRowCount).toBe(0); + expect(diff.droppedRowCount).toBe(0); + expect(diff.addedColumns).toEqual([]); + expect(diff.removedColumns).toEqual([]); + }); + + it('attributes changes to the SOURCE row even after rows were dropped', () => { + // Row 1 is a duplicate of row 0 and disappears. Without the surviving-row + // map, every later row would be compared against its predecessor and the + // diff would claim changes that never happened. + const header = ['ville']; + const columns: Cell[][] = [['Québec', 'Québec', ' Laval ', 'Montréal']]; + const { result, diff } = run(header, columns, { + ...QUIET, + dropDuplicates: true, + trimWhitespace: true, + }); + expect(result.survivingRows).toEqual([0, 2, 3]); + expect(diff.droppedRows).toEqual([1]); + expect(diff.droppedRowCount).toBe(1); + // Exactly one real change, and it is attributed to source row 2. + expect(diff.changedRowCount).toBe(1); + expect(diff.changedRows[0].sourceRow).toBe(2); + expect(diff.changedRows[0].changes[0]).toEqual({ + column: 'ville', + before: ' Laval ', + after: 'Laval', + }); + }); + + it('names the columns a recipe added and removed', () => { + const header = ['age', 'constante']; + const columns: Cell[][] = [ + ['34', '', '29', '41', '52', '61'], + ['x', 'x', 'x', 'x', 'x', 'x'], + ]; + const { diff } = run(header, columns, { + ...QUIET, + dropStructural: true, + columns: { age: { missing: 'median', indicator: true } }, + }); + expect(diff.addedColumns).toEqual(['age_absent']); + expect(diff.removedColumns).toEqual(['constante']); + }); + + it('caps the listed rows without ever capping the counts', () => { + const header = ['ville']; + const columns: Cell[][] = [Array.from({ length: 200 }, (_, i) => ` ville${i} `)]; + const { diff } = run(header, columns, { ...QUIET, trimWhitespace: true }); + expect(diff.changedRowCount).toBe(200); + expect(diff.changedRows.length).toBe(50); + expect(diff.changedCells).toBe(200); + }); +}); diff --git a/src/features/data/quality/diff.ts b/src/features/data/quality/diff.ts new file mode 100644 index 0000000..e67585a --- /dev/null +++ b/src/features/data/quality/diff.ts @@ -0,0 +1,108 @@ +/** + * V40: the before/after diff — which rows, which columns, which values. + * + * The studio already reported counts: « 412 cells trimmed, 6 duplicate rows + * dropped ». A count is a claim; a diff is evidence. This is what makes the + * recipe auditable rather than merely helpful — someone can check what it did + * instead of trusting that it did the right thing. + * + * The hard part is that a recipe does not only edit cells: it drops rows and + * adds columns, so « row 7 » before and « row 7 » after are not the same row. + * The alignment below is therefore explicit rather than positional — the + * recipe reports which source rows survived, and every change is attributed to + * the row it actually came from. + */ +import { isMissing } from '@/features/ml/data/infer'; +import type { Cell } from '@/features/ml/data/types'; + +/** Rows listed in the diff — enough to audit, not enough to freeze the tab. */ +export const MAX_DIFF_ROWS = 50; + +export interface CellChange { + column: string; + before: string; + after: string; +} + +export interface RowChange { + /** Index in the ORIGINAL dataset, so it can be found in the source file. */ + sourceRow: number; + changes: CellChange[]; +} + +export interface RecipeDiff { + /** Rows whose values changed, in source order, capped at MAX_DIFF_ROWS. */ + changedRows: RowChange[]; + /** How many rows changed in total — `changedRows` may be a prefix. */ + changedRowCount: number; + /** Source rows the recipe removed, capped for display. */ + droppedRows: number[]; + droppedRowCount: number; + /** Columns the recipe removed and added, by name. */ + removedColumns: string[]; + addedColumns: string[]; + /** How many individual cells changed value. */ + changedCells: number; +} + +function display(value: Cell): string { + return isMissing(value) ? '' : (value as string); +} + +/** + * Compares the dataset before and after a recipe. + * + * `survivingRows` maps each row of the AFTER dataset back to its index in the + * BEFORE dataset. Without it the comparison would silently pair row 7 with a + * different row 7 the moment anything was dropped, and report every column of + * every subsequent row as changed — a diff that is worse than no diff. + */ +export function diffRecipe( + beforeHeader: readonly string[], + beforeColumns: readonly Cell[][], + afterHeader: readonly string[], + afterColumns: readonly Cell[][], + survivingRows: readonly number[], +): RecipeDiff { + const beforeIndex = new Map(beforeHeader.map((name, i) => [name, i])); + const afterIndex = new Map(afterHeader.map((name, i) => [name, i])); + + const removedColumns = beforeHeader.filter((name) => !afterIndex.has(name)); + const addedColumns = afterHeader.filter((name) => !beforeIndex.has(name)); + const shared = beforeHeader.filter((name) => afterIndex.has(name)); + + const kept = new Set(survivingRows); + const droppedAll: number[] = []; + const beforeRowCount = beforeColumns[0]?.length ?? 0; + for (let row = 0; row < beforeRowCount; row++) { + if (!kept.has(row)) droppedAll.push(row); + } + + const changedRows: RowChange[] = []; + let changedRowCount = 0; + let changedCells = 0; + + for (let afterRow = 0; afterRow < survivingRows.length; afterRow++) { + const sourceRow = survivingRows[afterRow]; + const changes: CellChange[] = []; + for (const name of shared) { + const before = display(beforeColumns[beforeIndex.get(name)!]?.[sourceRow] ?? null); + const after = display(afterColumns[afterIndex.get(name)!]?.[afterRow] ?? null); + if (before !== after) changes.push({ column: name, before, after }); + } + if (changes.length === 0) continue; + changedRowCount += 1; + changedCells += changes.length; + if (changedRows.length < MAX_DIFF_ROWS) changedRows.push({ sourceRow, changes }); + } + + return { + changedRows, + changedRowCount, + droppedRows: droppedAll.slice(0, MAX_DIFF_ROWS), + droppedRowCount: droppedAll.length, + removedColumns, + addedColumns, + changedCells, + }; +} diff --git a/src/features/data/quality/reference.test.ts b/src/features/data/quality/reference.test.ts new file mode 100644 index 0000000..ec71c18 --- /dev/null +++ b/src/features/data/quality/reference.test.ts @@ -0,0 +1,163 @@ +/** + * V40 — the reference profile: drift against a file you no longer have. + */ +import { describe, expect, it } from 'vitest'; +import { + PROFILE_FORMAT, + buildProfile, + compareToProfile, + parseProfile, +} from '@/features/data/quality/reference'; +import { buildDriftReport } from '@/features/data/quality/drift'; +import { mulberry32 } from '@/features/ml/train/random'; +import type { Cell } from '@/features/ml/data/types'; + +function numericColumn(n: number, seed: number, shift = 0): Cell[] { + const rng = mulberry32(seed); + return Array.from({ length: n }, () => String(Math.round((rng() * 100 + shift) * 100) / 100)); +} + +function categoryColumn(n: number, weights: [string, number][]): Cell[] { + const out: Cell[] = []; + const total = weights.reduce((sum, [, w]) => sum + w, 0); + for (let i = 0; i < n; i++) { + let position = (i % total) + 1; + for (const [name, weight] of weights) { + position -= weight; + if (position <= 0) { + out.push(name); + break; + } + } + } + return out; +} + +describe('V40 — a profile describes the shape, never the rows', () => { + it('stores edges and shares, and no data', () => { + const header = ['salaire', 'ville']; + const columns: Cell[][] = [ + numericColumn(200, 42), + categoryColumn(200, [ + ['Québec', 3], + ['Montréal', 2], + ]), + ]; + const profile = buildProfile(header, columns, 'paie.csv'); + const serialized = JSON.stringify(profile); + // The exact values of the file must not appear anywhere in the profile: + // this is what makes it safe to commit beside the code it describes. + for (const value of columns[0].slice(0, 20)) { + expect(serialized).not.toContain(`"${String(value)}"`); + } + expect(profile.format).toBe(PROFILE_FORMAT); + expect(profile.rowCount).toBe(200); + const salary = profile.columns.find((c) => c.column === 'salaire')!; + expect(salary.kind).toBe('numeric'); + }); + + it('round-trips through JSON', () => { + const profile = buildProfile( + ['a', 'b'], + [ + numericColumn(120, 7), + categoryColumn(120, [ + ['x', 1], + ['y', 1], + ]), + ], + 'ref.csv', + ); + const parsed = parseProfile(JSON.stringify(profile)); + expect(parsed).not.toBeNull(); + expect(parsed!.columns.map((c) => c.column)).toEqual(['a', 'b']); + expect(parsed!.source).toBe('ref.csv'); + }); + + it('refuses anything that is not a LabML profile', () => { + expect(parseProfile('{}')).toBeNull(); + expect(parseProfile('not json')).toBeNull(); + expect(parseProfile(JSON.stringify({ format: 'something-else', columns: [] }))).toBeNull(); + // A profile whose columns all failed validation describes nothing. + expect(parseProfile(JSON.stringify({ format: PROFILE_FORMAT, columns: [{}] }))).toBeNull(); + }); +}); + +describe('V40 — a profile scores a new file the way V11 would have', () => { + it('finds no drift when the new file is the reference itself', () => { + const header = ['a', 'b']; + const columns: Cell[][] = [ + numericColumn(300, 42), + categoryColumn(300, [ + ['x', 3], + ['y', 2], + ]), + ]; + const profile = buildProfile(header, columns, 'ref.csv'); + const comparison = compareToProfile(profile, header, columns); + expect(comparison.worst).toBe('stable'); + for (const drift of comparison.columns) expect(drift.psi).toBeCloseTo(0, 6); + }); + + it('agrees with the live two-file comparison on the same pair', () => { + const header = ['a']; + const reference: Cell[][] = [numericColumn(400, 42)]; + const current: Cell[][] = [numericColumn(400, 99, 40)]; + + const live = buildDriftReport(header, reference, header, current); + const viaProfile = compareToProfile( + buildProfile(header, reference, 'ref.csv'), + header, + current, + ); + // Same maths, same bins, same verdict — the profile is a stored reference, + // not a second opinion. + expect(viaProfile.columns[0].psi).toBeCloseTo(live.columns[0].psi, 6); + expect(viaProfile.columns[0].severity).toBe(live.columns[0].severity); + }); + + it('detects a shifted numeric distribution', () => { + const header = ['montant']; + const profile = buildProfile(header, [numericColumn(400, 42)], 'ref.csv'); + const shifted = compareToProfile(profile, header, [numericColumn(400, 99, 60)]); + expect(shifted.columns[0].psi).toBeGreaterThan(0.2); + expect(shifted.worst).toBe('strong'); + }); + + it('puts an unseen category in OTHER instead of inventing a bucket', () => { + const header = ['ville']; + const profile = buildProfile( + header, + [ + categoryColumn(200, [ + ['Québec', 3], + ['Montréal', 2], + ]), + ], + 'ref.csv', + ); + const withNew = compareToProfile(profile, header, [ + categoryColumn(200, [ + ['Québec', 1], + ['Laval', 4], + ]), + ]); + expect(withNew.columns[0].psi).toBeGreaterThan(0); + expect(withNew.columns[0].kind).toBe('categorical'); + }); + + it('names the columns that appeared and disappeared', () => { + const profile = buildProfile( + ['a', 'b'], + [numericColumn(60, 1), numericColumn(60, 2)], + 'ref.csv', + ); + const comparison = compareToProfile( + profile, + ['a', 'c'], + [numericColumn(60, 1), numericColumn(60, 3)], + ); + expect(comparison.missingColumns).toEqual(['b']); + expect(comparison.newColumns).toEqual(['c']); + }); +}); diff --git a/src/features/data/quality/reference.ts b/src/features/data/quality/reference.ts new file mode 100644 index 0000000..1f0df88 --- /dev/null +++ b/src/features/data/quality/reference.ts @@ -0,0 +1,304 @@ +/** + * V40: the reference profile — drift without the original file. + * + * V11 compares two datasets that are both open in the tab. That answers « has + * this month's export drifted from last month's? » only for as long as you + * still hold last month's file. A profile is the same idea as V22's model + * manifest applied to data: a small, exportable, replayable description of what + * the reference LOOKED like, so a new file can be checked against a snapshot + * taken months ago and long since deleted. + * + * What it deliberately is not: a copy of the data. It stores bin EDGES and + * SHARES, never rows — a profile of a payroll file reveals the shape of the + * salary distribution, not anybody's salary. That is what makes it safe to + * commit next to the code it describes, which is the whole point of having it. + */ +import { isMissing, inferColumnType, parseNumber } from '@/features/ml/data/infer'; +import { + psi, + severityOf, + type ColumnDrift, + type DriftSeverity, +} from '@/features/data/quality/drift'; +import type { Cell } from '@/features/ml/data/types'; + +/** Same bin count as V11, so a profile and a live comparison agree. */ +const NUMERIC_BINS = 10; +/** Categories kept by name; the rest collapse into OTHER. */ +const TOP_CATEGORIES = 12; +export const PROFILE_FORMAT = 'labml-data-profile/1'; + +export interface NumericProfile { + kind: 'numeric'; + column: string; + missingRatio: number; + mean: number; + /** Quantile edges of the reference — the bins a new file is scored into. */ + edges: number[]; + /** Reference share per bin, length edges.length + 1. */ + shares: number[]; +} + +export interface CategoricalProfile { + kind: 'categorical'; + column: string; + missingRatio: number; + /** Share per named category, plus OTHER — never the rows themselves. */ + shares: Record; +} + +export type ColumnProfileEntry = NumericProfile | CategoricalProfile; + +export interface DataProfile { + format: typeof PROFILE_FORMAT; + source: string; + createdAt: string; + rowCount: number; + columns: ColumnProfileEntry[]; +} + +const OTHER = '__other__'; + +function numbersOf(values: Cell[]): number[] { + const numbers: number[] = []; + for (const value of values) { + if (isMissing(value)) continue; + const parsed = parseNumber((value as string).trim()); + if (parsed !== null) numbers.push(parsed); + } + return numbers; +} + +function missingRatio(values: Cell[]): number { + if (values.length === 0) return 0; + let missing = 0; + for (const value of values) if (isMissing(value)) missing += 1; + return missing / values.length; +} + +function quantile(sorted: number[], q: number): number { + if (sorted.length === 0) return 0; + const position = (sorted.length - 1) * q; + const low = Math.floor(position); + const high = Math.ceil(position); + return low === high ? sorted[low] : sorted[low] + (position - low) * (sorted[high] - sorted[low]); +} + +function sharesOverEdges(edges: number[], numbers: number[]): number[] { + const counts = new Array(edges.length + 1).fill(0); + for (const value of numbers) { + let bin = 0; + while (bin < edges.length && value > edges[bin]) bin += 1; + counts[bin] += 1; + } + return counts.map((count) => (numbers.length > 0 ? count / numbers.length : 0)); +} + +function categoryShares(values: Cell[]): { shares: Record; names: string[] } { + const counts = new Map(); + let total = 0; + for (const value of values) { + if (isMissing(value)) continue; + const key = (value as string).trim(); + counts.set(key, (counts.get(key) ?? 0) + 1); + total += 1; + } + const top = [...counts.entries()] + .sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])) + .slice(0, TOP_CATEGORIES); + const shares: Record = {}; + let named = 0; + for (const [name, count] of top) { + shares[name] = total > 0 ? count / total : 0; + named += count; + } + shares[OTHER] = total > 0 ? (total - named) / total : 0; + return { shares, names: top.map(([name]) => name) }; +} + +/** Builds the exportable profile of a dataset. */ +export function buildProfile( + header: readonly string[], + columns: Cell[][], + source: string, + now = new Date(), +): DataProfile { + const entries: ColumnProfileEntry[] = []; + for (let i = 0; i < header.length; i++) { + const name = header[i]; + const values = columns[i] ?? []; + const type = inferColumnType(name, values); + if (type === 'numeric') { + const sorted = numbersOf(values).sort((a, b) => a - b); + const edges: number[] = []; + for (let b = 1; b < NUMERIC_BINS; b++) { + const edge = quantile(sorted, b / NUMERIC_BINS); + if (edges.length === 0 || edge > edges[edges.length - 1]) edges.push(edge); + } + entries.push({ + kind: 'numeric', + column: name, + missingRatio: missingRatio(values), + mean: sorted.length > 0 ? sorted.reduce((a, v) => a + v, 0) / sorted.length : 0, + edges, + shares: sharesOverEdges(edges, sorted), + }); + continue; + } + // Text and dates are profiled as categories: a share table is meaningful + // for both, and PSI over shares is the same computation either way. + entries.push({ + kind: 'categorical', + column: name, + missingRatio: missingRatio(values), + shares: categoryShares(values).shares, + }); + } + return { + format: PROFILE_FORMAT, + source, + createdAt: now.toISOString(), + rowCount: columns[0]?.length ?? 0, + columns: entries, + }; +} + +export interface ProfileComparison { + /** Per-column drift against the stored profile, worst first. */ + columns: ColumnDrift[]; + /** Columns the profile describes that the new file does not have. */ + missingColumns: string[]; + /** Columns the new file has that the profile does not describe. */ + newColumns: string[]; + worst: DriftSeverity; +} + +/** + * Scores a new file against a stored profile. Every number is computed the way + * V11 computes it, so a profile comparison and a live two-file comparison give + * the same PSI for the same pair of datasets. + */ +export function compareToProfile( + profile: DataProfile, + header: readonly string[], + columns: Cell[][], +): ProfileComparison { + const index = new Map(header.map((name, i) => [name, i])); + const described = new Set(profile.columns.map((entry) => entry.column)); + const drifts: ColumnDrift[] = []; + const missingColumns: string[] = []; + + for (const entry of profile.columns) { + const at = index.get(entry.column); + if (at === undefined) { + missingColumns.push(entry.column); + continue; + } + const values = columns[at] ?? []; + if (entry.kind === 'numeric') { + const numbers = numbersOf(values); + const value = + numbers.length === 0 ? 0 : psi(entry.shares, sharesOverEdges(entry.edges, numbers)); + drifts.push({ + column: entry.column, + kind: 'numeric', + psi: value, + severity: severityOf(value), + refMissingRatio: entry.missingRatio, + newMissingRatio: missingRatio(values), + refMean: entry.mean, + newMean: numbers.length > 0 ? numbers.reduce((a, v) => a + v, 0) / numbers.length : 0, + }); + continue; + } + // Categorical: score the new file into the profile's own buckets, so a + // category the reference never saw lands in OTHER rather than inventing + // a bucket the reference has no share for. + const names = Object.keys(entry.shares).filter((name) => name !== OTHER); + const nameSet = new Set(names); + const counts = new Map(names.map((name) => [name, 0])); + counts.set(OTHER, 0); + let total = 0; + for (const value of values) { + if (isMissing(value)) continue; + const key = (value as string).trim(); + const bucket = nameSet.has(key) ? key : OTHER; + counts.set(bucket, (counts.get(bucket) ?? 0) + 1); + total += 1; + } + const order = [...names, OTHER]; + const referenceShares = order.map((name) => entry.shares[name] ?? 0); + const newShares = order.map((name) => (total > 0 ? (counts.get(name) ?? 0) / total : 0)); + const value = total === 0 ? 0 : psi(referenceShares, newShares); + drifts.push({ + column: entry.column, + kind: 'categorical', + psi: value, + severity: severityOf(value), + refMissingRatio: entry.missingRatio, + newMissingRatio: missingRatio(values), + }); + } + + const newColumns = header.filter((name) => !described.has(name)); + drifts.sort((a, b) => b.psi - a.psi || a.column.localeCompare(b.column)); + // The worst severity present, using V11's own three levels. Ordered + // explicitly rather than by array position, so adding a level later cannot + // silently make this return the wrong one. + const worst: DriftSeverity = drifts.some((d) => d.severity === 'strong') + ? 'strong' + : drifts.some((d) => d.severity === 'moderate') + ? 'moderate' + : 'stable'; + return { columns: drifts, missingColumns, newColumns, worst }; +} + +/** + * Parses an exported profile. Strict on the format marker and on the shapes it + * needs — a profile that cannot be trusted to describe anything is refused by + * name rather than half-applied. + */ +export function parseProfile(json: string): DataProfile | null { + let parsed: unknown; + try { + parsed = JSON.parse(json); + } catch { + return null; + } + if (typeof parsed !== 'object' || parsed === null) return null; + const record = parsed as Record; + if (record.format !== PROFILE_FORMAT) return null; + if (!Array.isArray(record.columns)) return null; + + const columns: ColumnProfileEntry[] = []; + for (const raw of record.columns) { + if (typeof raw !== 'object' || raw === null) continue; + const entry = raw as Record; + if (typeof entry.column !== 'string') continue; + const missing = typeof entry.missingRatio === 'number' ? entry.missingRatio : 0; + if (entry.kind === 'numeric' && Array.isArray(entry.edges) && Array.isArray(entry.shares)) { + columns.push({ + kind: 'numeric', + column: entry.column, + missingRatio: missing, + mean: typeof entry.mean === 'number' ? entry.mean : 0, + edges: entry.edges.filter((value): value is number => typeof value === 'number'), + shares: entry.shares.filter((value): value is number => typeof value === 'number'), + }); + } else if (entry.kind === 'categorical' && typeof entry.shares === 'object' && entry.shares) { + const shares: Record = {}; + for (const [name, share] of Object.entries(entry.shares as Record)) { + if (typeof share === 'number') shares[name] = share; + } + columns.push({ kind: 'categorical', column: entry.column, missingRatio: missing, shares }); + } + } + if (columns.length === 0) return null; + return { + format: PROFILE_FORMAT, + source: typeof record.source === 'string' ? record.source : '', + createdAt: typeof record.createdAt === 'string' ? record.createdAt : '', + rowCount: typeof record.rowCount === 'number' ? record.rowCount : 0, + columns, + }; +} diff --git a/src/features/data/quality/types.ts b/src/features/data/quality/types.ts index 67a22ef..c0e4253 100644 --- a/src/features/data/quality/types.ts +++ b/src/features/data/quality/types.ts @@ -1,3 +1,6 @@ +import type { ValidityFinding } from '@/features/data/quality/validity'; +import type { ConsistencyFinding } from '@/features/data/quality/consistency'; +import type { ScoreBreakdown } from '@/features/data/quality/checks'; /** Issue report and cleaning recipe types for the Data Studio. */ export interface MissingColumn { @@ -51,8 +54,19 @@ export interface QualityReport { outlierColumns: OutlierColumn[]; outlierCells: number; structural: StructuralIssue[]; + /** + * V40: values that are present, correctly typed and still impossible — + * an age of 200, a date in the future. Reported, never repaired. + */ + validity: ValidityFinding[]; + invalidCells: number; + /** V40: rows where two columns contradict each other. */ + consistency: ConsistencyFinding[]; + inconsistentRows: number; /** 0–100; deterministic function of the ratios above. */ score: number; + /** V40: the score, explained by its parts instead of asserted. */ + breakdown: ScoreBreakdown[]; } /** Forceable column types — they steer the cleaning, not the "before" report. */ diff --git a/src/features/data/quality/v40.test.ts b/src/features/data/quality/v40.test.ts new file mode 100644 index 0000000..c99804c --- /dev/null +++ b/src/features/data/quality/v40.test.ts @@ -0,0 +1,92 @@ +/** + * V40 — the score, explained by its parts. + */ +import { describe, expect, it } from 'vitest'; +import { + TOTAL_WEIGHT, + buildQualityReport, + qualityScore, + scoreBreakdown, +} from '@/features/data/quality/checks'; +import type { Cell } from '@/features/ml/data/types'; + +/** The report without the two fields the score functions compute themselves. */ +function report(header: string[], columns: Cell[][]) { + const built = buildQualityReport(header, columns); + const rest: Record = { ...built }; + delete rest.score; + delete rest.breakdown; + return rest as Parameters[0]; +} + +describe('V40 — the quality score is the sum of its published parts', () => { + it('adds up to exactly the score shown', () => { + const header = ['a', 'b']; + const columns: Cell[][] = [ + ['1', '', '3', '4', '', '6'], + ['x', 'X', 'y', 'y', 'z', ''], + ]; + const partial = report(header, columns); + const parts = scoreBreakdown(partial, 2); + const total = parts.reduce((sum, part) => sum + part.penalty, 0); + expect(qualityScore(partial, 2)).toBe(Math.max(0, Math.round(100 - total))); + }); + + it('names every part, with its weight, its ratio and what it cost', () => { + const partial = report(['a'], [['1', '', '3', '4']]); + const parts = scoreBreakdown(partial); + expect(parts.map((p) => p.part)).toEqual([ + 'missing', + 'duplicates', + 'messy', + 'outliers', + 'structural', + 'validity', + ]); + const missing = parts.find((p) => p.part === 'missing')!; + expect(missing.weight).toBe(35); + expect(missing.count).toBe(1); + expect(missing.ratio).toBeCloseTo(0.25, 10); + // 35 × min(1, 0.25 × 8) = 35, saturated. + expect(missing.penalty).toBe(35); + }); + + it('keeps every pre-V40 weight untouched, so old scores still mean the same', () => { + // Validity brought its own 5 points instead of taking them from a + // neighbour: a file with no validity findings must score exactly what it + // scored before this wave. + const partial = report( + ['a', 'b'], + [ + ['1', '', '3', '4'], + ['x', 'X', 'y', 'y'], + ], + ); + expect(qualityScore(partial, 0)).toBe(qualityScore(partial)); + const weights = scoreBreakdown(partial).map((p) => p.weight); + expect(weights).toEqual([35, 20, 20, 15, 10, 5]); + expect(weights.reduce((a, b) => a + b, 0)).toBe(TOTAL_WEIGHT); + }); + + it('charges for impossible values, and only when there are some', () => { + const partial = report(['a'], [['1', '2', '3', '4', '5', '6', '7', '8']]); + expect(scoreBreakdown(partial, 0).find((p) => p.part === 'validity')!.penalty).toBe(0); + expect(scoreBreakdown(partial, 4).find((p) => p.part === 'validity')!.penalty).toBeGreaterThan( + 0, + ); + expect(qualityScore(partial, 4)).toBeLessThan(qualityScore(partial, 0)); + }); + + it('still gives a clean file 100 and never goes below 0', () => { + const clean = report( + ['a', 'b'], + [ + ['1', '2', '3', '4'], + ['x', 'y', 'x', 'y'], + ], + ); + expect(qualityScore(clean, 0)).toBe(100); + const filthy = report(['a'], [['', '', '', '']]); + expect(qualityScore(filthy, 4)).toBeGreaterThanOrEqual(0); + }); +}); diff --git a/src/features/data/quality/validity.test.ts b/src/features/data/quality/validity.test.ts new file mode 100644 index 0000000..0141dfc --- /dev/null +++ b/src/features/data/quality/validity.test.ts @@ -0,0 +1,157 @@ +/** + * V40 — validity. + * + * As in V38's reader, the tests that matter most are the ones asserting the + * module REFUSES to fire: a rule that flags a good file is worse than no rule, + * because it teaches the user to ignore the panel. + */ +import { describe, expect, it } from 'vitest'; +import { + APPLICABILITY, + checkColumn, + checkValidity, + invalidCellCount, +} from '@/features/data/quality/validity'; +import type { Cell } from '@/features/ml/data/types'; + +/** A fixed "now" so "the future" never depends on when the suite runs. */ +const NOW = Date.UTC(2026, 7, 23); + +describe('V40 — a value can be present, well-typed and still impossible', () => { + it('flags an age outside 0–120, naming the rows and the bound', () => { + const values: Cell[] = ['34', '41', '200', '29', '52', '61', '18', '77', '-3', '45']; + const [found] = checkColumn('age', values, NOW); + expect(found.rule).toBe('ageRange'); + expect(found.count).toBe(2); + expect(found.rows).toEqual([2, 8]); + expect(found.examples).toEqual(['200', '-3']); + expect(found.bound).toEqual({ min: 0, max: 120 }); + }); + + it('flags a date in the future', () => { + const values: Cell[] = [ + '2025-01-04', + '2025-06-11', + '2087-03-02', + '2024-12-31', + '2025-02-02', + '2025-03-03', + ]; + const [found] = checkColumn('date_vente', values, NOW); + expect(found.rule).toBe('futureDate'); + expect(found.rows).toEqual([2]); + }); + + it('flags a percentage above 100 and a negative amount', () => { + const pct = checkColumn('taux_reussite', ['12', '54', '130', '77', '90', '31'], NOW); + expect(pct[0].rule).toBe('percentRange'); + expect(pct[0].examples).toEqual(['130']); + + const amount = checkColumn('prix', ['12.5', '3.25', '-8', '48.9', '7.1', '99'], NOW); + expect(amount[0].rule).toBe('negativeAmount'); + expect(amount[0].examples).toEqual(['-8']); + }); + + it('flags a malformed postcode among well-formed ones', () => { + const values: Cell[] = ['H2X 1Y4', '75008', 'G1R2B5', '13001', 'ABC', 'H3Z 2Y7']; + const [found] = checkColumn('code_postal', values, NOW); + expect(found.rule).toBe('postcodeShape'); + expect(found.examples).toEqual(['ABC']); + }); +}); + +describe('V40 — a rule fires on evidence, never on a column name alone', () => { + it('REFUSES an "age" column that is plainly a duration, not a person', () => { + // Ages in days: every value is above 120, so the rule is not applicable — + // flagging all ten rows would be the rule being wrong, not the data. + const values: Cell[] = [ + '3200', + '5400', + '900', + '12000', + '7800', + '4300', + '6100', + '2200', + '9900', + '15000', + ]; + expect(checkColumn('age_jours', values, NOW)).toEqual([]); + }); + + it('REFUSES a rate written on a 0–1 scale', () => { + const values: Cell[] = ['0.12', '0.54', '0.9', '0.77', '0.31', '0.66']; + expect(checkColumn('taux_conversion', values, NOW)).toEqual([]); + }); + + it('REFUSES a ledger column that is routinely negative', () => { + const values: Cell[] = ['-12', '-3', '45', '-8', '-99', '-40']; + expect(checkColumn('montant_ajustement', values, NOW)).toEqual([]); + }); + + it('REFUSES a column whose name matches but whose values are not that thing', () => { + // "date_label" holds free text, not dates: nothing to check. + const values: Cell[] = ['hiver', 'été', 'printemps', 'automne', 'hiver', 'été']; + expect(checkColumn('date_label', values, NOW)).toEqual([]); + }); + + it('REFUSES every rule on a column whose name matches nothing', () => { + expect(checkColumn('ville', ['Québec', 'Montréal', '200', '-8'], NOW)).toEqual([]); + }); + + it('keeps the applicability floor high enough to mean something', () => { + expect(APPLICABILITY).toBeGreaterThanOrEqual(0.8); + }); + + it('stops applying a rule once too much of the column contradicts it', () => { + // 8 of 10 plausible: exactly at the floor, so the rule still fires. + const atFloor: Cell[] = ['34', '200', '201', '29', '52', '61', '18', '77', '45', '38']; + expect(checkColumn('age', atFloor, NOW)[0].count).toBe(2); + + // 7 of 10 plausible: below the floor. A column a third of which is + // "invalid" is a column the rule has misunderstood, so it says nothing + // rather than flagging thirty rows of a good file. + const belowFloor: Cell[] = ['34', '200', '201', '202', '29', '52', '61', '18', '77', '45']; + expect(checkColumn('age', belowFloor, NOW)).toEqual([]); + }); +}); + +describe('V40 — findings are reported, never repaired', () => { + it('does not touch the values it flags', () => { + const values: Cell[] = ['34', '41', '200', '29', '52', '61']; + const before = [...values]; + checkColumn('age', values, NOW); + // V39's recipe is where data changes; a check that quietly repaired what it + // found would put edits outside the one record of what was done. + expect(values).toEqual(before); + }); + + it('caps the listed rows but never the count', () => { + const values: Cell[] = Array.from({ length: 100 }, (_, i) => (i < 20 ? '999' : '40')); + const [found] = checkColumn('age', values, NOW); + expect(found.count).toBe(20); + expect(found.rows.length).toBe(10); + }); + + it('orders findings worst first and totals the impossible cells', () => { + const header = ['age', 'prix']; + const columns: Cell[][] = [ + ['34', '200', '201', '29', '52', '61', '18', '77', '45', '38'], + ['12.5', '-8', '48.9', '7.1', '99', '3.25', '5', '6', '7', '8'], + ]; + const findings = checkValidity(header, columns, NOW); + expect(findings.map((f) => f.column)).toEqual(['age', 'prix']); + expect(invalidCellCount(findings)).toBe(3); + }); + + it('returns nothing at all for a clean file', () => { + const header = ['age', 'prix', 'date_vente']; + const columns: Cell[][] = [ + ['34', '41', '29', '52', '61', '18'], + ['12.5', '3.25', '48.9', '7.1', '99', '5'], + ['2025-01-04', '2025-06-11', '2024-12-31', '2025-02-02', '2025-03-03', '2025-04-04'], + ]; + expect(checkValidity(header, columns, NOW)).toEqual([]); + expect(invalidCellCount([])).toBe(0); + }); +}); diff --git a/src/features/data/quality/validity.ts b/src/features/data/quality/validity.ts new file mode 100644 index 0000000..cc7fdb1 --- /dev/null +++ b/src/features/data/quality/validity.ts @@ -0,0 +1,203 @@ +/** + * V40: validity — the third question, after completeness and type. + * + * The studio already asks « is the value there? » (missing) and « is it the + * right shape? » (type). Neither catches a value that is present, correctly + * typed, and still impossible: an age of 200, a delivery date in 2087, a + * percentage at 130, a postcode of four characters. Those pass every check + * the report performs today and go straight into a model. + * + * Two rules shape this module, and they are the same two that shaped V38's + * reader: + * + * 1. **A rule fires on evidence, never on a column's name alone.** A column + * called `age` holding 20 000 is a duration in days, not a person; a rule + * that trusted the name would flag every row of a perfectly good file. So + * a name makes a rule *applicable*, and the values decide whether it fires. + * 2. **Flagging is not cleaning.** Every finding names the rule, the column, + * how many rows failed and which ones — and then stops. V39's recipe is + * where data gets changed, deliberately and reproducibly; a check that + * quietly repaired what it found would put edits outside the recipe, which + * is the one record of what was done. + */ +import { isMissing, parseNumber } from '@/features/ml/data/infer'; +import { parseDate } from '@/features/ml/timeseries/series'; +import type { Cell } from '@/features/ml/data/types'; + +/** Rows listed per finding — enough to inspect, not enough to flood the UI. */ +export const MAX_EXAMPLE_ROWS = 10; +/** + * A rule must apply to enough of a column to be about the column rather than + * about a handful of odd cells: below this share of usable values matching the + * rule's own shape, the rule does not consider itself applicable at all. + */ +export const APPLICABILITY = 0.8; + +export type ValidityRule = + /** A human age outside 0–120. */ + | 'ageRange' + /** A date after today — a birth date or a sale cannot be in the future. */ + | 'futureDate' + /** A percentage outside 0–100. */ + | 'percentRange' + /** A quantity, price or amount below zero. */ + | 'negativeAmount' + /** A Canadian or French postcode that does not have the right shape. */ + | 'postcodeShape'; + +export interface ValidityFinding { + rule: ValidityRule; + column: string; + /** Rows that failed, 0-based, capped at MAX_EXAMPLE_ROWS. */ + rows: number[]; + /** How many rows failed in total — `rows` may be a prefix of this. */ + count: number; + /** Values that failed, aligned with `rows`, for display. */ + examples: string[]; + /** The bound the rule enforced, when it has one — shown in the message. */ + bound?: { min?: number; max?: number }; +} + +/** Column names a rule is willing to look at, matched loosely and accent-free. */ +function nameMatches(column: string, needles: readonly string[]): boolean { + const normalized = column.toLowerCase().normalize('NFD').replace(/[̀-ͯ]/g, ''); + return needles.some((needle) => normalized.includes(needle)); +} + +function usableValues(values: Cell[]): { row: number; value: string }[] { + const out: { row: number; value: string }[] = []; + for (let row = 0; row < values.length; row++) { + const value = values[row]; + if (isMissing(value)) continue; + out.push({ row, value: (value as string).trim() }); + } + return out; +} + +function finding( + rule: ValidityRule, + column: string, + failures: { row: number; value: string }[], + bound?: { min?: number; max?: number }, +): ValidityFinding | null { + if (failures.length === 0) return null; + const capped = failures.slice(0, MAX_EXAMPLE_ROWS); + return { + rule, + column, + rows: capped.map((f) => f.row), + examples: capped.map((f) => f.value), + count: failures.length, + ...(bound !== undefined && { bound }), + }; +} + +const AGE_NAMES = ['age', 'âge'] as const; +const PERCENT_NAMES = ['percent', 'pourcent', 'pct', 'taux', 'rate'] as const; +const AMOUNT_NAMES = [ + 'price', + 'prix', + 'montant', + 'amount', + 'total', + 'quantity', + 'quantite', +] as const; +const POSTCODE_NAMES = ['postcode', 'postal', 'zip', 'cp'] as const; +const DATE_NAMES = ['date', 'jour', 'day'] as const; + +/** Canadian `H2X 1Y4` or French `75008`. */ +const CANADIAN = /^[A-Za-z]\d[A-Za-z][ -]?\d[A-Za-z]\d$/; +const FRENCH_POSTCODE = /^\d{5}$/; + +/** + * Every validity finding for one column. `now` is injected so a test can pin + * "the future" to a fixed instant rather than depending on the clock. + */ +export function checkColumn(column: string, values: Cell[], now = Date.now()): ValidityFinding[] { + const usable = usableValues(values); + if (usable.length === 0) return []; + const found: ValidityFinding[] = []; + + const numbers = usable + .map((entry) => ({ ...entry, parsed: parseNumber(entry.value) })) + .filter((entry): entry is typeof entry & { parsed: number } => entry.parsed !== null); + const numericEnough = numbers.length / usable.length >= APPLICABILITY; + + if (numericEnough && nameMatches(column, AGE_NAMES)) { + // A column of ages that is mostly above 120 is not ages — it is a duration + // in months or days, and flagging every row of it would be the rule being + // wrong, not the data. + const plausible = numbers.filter((n) => n.parsed >= 0 && n.parsed <= 120).length; + if (plausible / numbers.length >= APPLICABILITY) { + const bad = numbers.filter((n) => n.parsed < 0 || n.parsed > 120); + const result = finding('ageRange', column, bad, { min: 0, max: 120 }); + if (result) found.push(result); + } + } + + if (numericEnough && nameMatches(column, PERCENT_NAMES)) { + // Same guard: a rate written as 0–1 is not a percentage out of range. + const plausible = numbers.filter((n) => n.parsed >= 0 && n.parsed <= 100).length; + if (plausible / numbers.length >= APPLICABILITY) { + const bad = numbers.filter((n) => n.parsed < 0 || n.parsed > 100); + const result = finding('percentRange', column, bad, { min: 0, max: 100 }); + if (result) found.push(result); + } + } + + if (numericEnough && nameMatches(column, AMOUNT_NAMES)) { + // A ledger column that is routinely negative is a balance, not an error. + const nonNegative = numbers.filter((n) => n.parsed >= 0).length; + if (nonNegative / numbers.length >= APPLICABILITY) { + const bad = numbers.filter((n) => n.parsed < 0); + const result = finding('negativeAmount', column, bad, { min: 0 }); + if (result) found.push(result); + } + } + + if (nameMatches(column, DATE_NAMES)) { + const dates = usable + .map((entry) => ({ ...entry, at: parseDate(entry.value) })) + .filter((entry): entry is typeof entry & { at: number } => entry.at !== null); + if (dates.length / usable.length >= APPLICABILITY) { + const bad = dates.filter((d) => d.at > now); + const result = finding('futureDate', column, bad); + if (result) found.push(result); + } + } + + if (nameMatches(column, POSTCODE_NAMES)) { + const shaped = usable.filter( + (entry) => CANADIAN.test(entry.value) || FRENCH_POSTCODE.test(entry.value), + ); + // Only a column that is mostly postcodes gets to have malformed ones. + if (shaped.length / usable.length >= APPLICABILITY) { + const bad = usable.filter( + (entry) => !CANADIAN.test(entry.value) && !FRENCH_POSTCODE.test(entry.value), + ); + const result = finding('postcodeShape', column, bad); + if (result) found.push(result); + } + } + + return found; +} + +/** Every finding across the dataset, worst first. */ +export function checkValidity( + header: readonly string[], + columns: Cell[][], + now = Date.now(), +): ValidityFinding[] { + const found: ValidityFinding[] = []; + for (let i = 0; i < header.length; i++) { + found.push(...checkColumn(header[i], columns[i] ?? [], now)); + } + return found.sort((a, b) => b.count - a.count || a.column.localeCompare(b.column)); +} + +/** Total impossible cells — what the quality score charges for. */ +export function invalidCellCount(findings: readonly ValidityFinding[]): number { + return findings.reduce((total, finding) => total + finding.count, 0); +} diff --git a/src/features/data/sql/engine.ts b/src/features/data/sql/engine.ts index 88abb59..068ebfe 100644 --- a/src/features/data/sql/engine.ts +++ b/src/features/data/sql/engine.ts @@ -26,6 +26,13 @@ export interface SqlEngine { /** Makes a file readable by SQL under `name`; the bytes stay in the tab. */ register(name: string, bytes: Uint8Array): Promise; run(sql: string, cap: number): Promise; + /** + * V40: runs a query and returns the result as Parquet bytes. Nearly free + * here — DuckDB is already loaded and the file already registered, so this + * is one `COPY … TO` and a read of the buffer it wrote. The bytes never + * leave the tab: the caller turns them into a download. + */ + toParquet(sql: string): Promise; close(): Promise; } @@ -66,6 +73,16 @@ export async function openEngine(): Promise { const records = result.toArray().map((row) => row.toJSON()); return toSqlTable(columns, records, cap); }, + async toParquet(sql) { + // A per-call name so two exports can never collide on the virtual FS. + const name = `export-${Date.now()}.parquet`; + await connection.query(`COPY (${sql}) TO '${name}' (FORMAT PARQUET, COMPRESSION ZSTD)`); + const bytes = await db.copyFileToBuffer(name); + // Registered files live in the Wasm heap: dropping it keeps a session of + // repeated exports from growing without bound. + await db.dropFile(name); + return bytes; + }, async close() { await connection.close(); await db.terminate(); diff --git a/src/locales/en.json b/src/locales/en.json index eae2e70..3eafd3c 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -702,6 +702,20 @@ "good": "Good", "fair": "Needs attention", "poor": "Poor" + }, + "breakdownTitle": "How this score was computed", + "breakdownHint": "Each part carries its own weight and the points it actually cost. The weights sum to 105, not 100: validity brought its own 5 points in V40 rather than taking them from an existing part, so a file with no impossible values scores exactly what it scored before.", + "part": "Part", + "found": "Found", + "weight": "Max", + "cost": "Cost", + "parts": { + "missing": "Missing cells", + "duplicates": "Duplicate rows", + "messy": "Messy spellings", + "outliers": "Outlier values", + "structural": "Useless columns", + "validity": "Impossible values" } }, "issues": { @@ -734,6 +748,27 @@ "constant": "constant", "nearEmpty": "near-empty", "id": "identifier" + }, + "validity": { + "title_one": "{{count}} impossible value", + "title_other": "{{count}} impossible values", + "body": "Present, correctly typed, and still impossible. Flagged, never repaired — the recipe is where data changes.", + "rules": { + "ageRange": "age outside 0–120", + "futureDate": "date in the future", + "percentRange": "percentage outside 0–100", + "negativeAmount": "negative amount", + "postcodeShape": "malformed postcode" + } + }, + "consistency": { + "title_one": "{{count}} contradictory row", + "title_other": "{{count}} contradictory rows", + "body": "Every cell is fine; the row is not. Two columns disagree with each other.", + "rules": { + "dateOrder": "{{columns}}: the end comes before the start", + "productMismatch": "{{columns}}: the total is not quantity × price" + } } }, "recipe": { diff --git a/src/locales/fr.json b/src/locales/fr.json index 8bebb53..c93ab6b 100644 --- a/src/locales/fr.json +++ b/src/locales/fr.json @@ -702,6 +702,20 @@ "good": "Bon", "fair": "À surveiller", "poor": "Faible" + }, + "breakdownTitle": "Comment ce score a été calculé", + "breakdownHint": "Chaque partie porte son poids et les points qu'elle a réellement coûtés. Les poids totalisent 105 et non 100 : la validité a apporté ses 5 points en V40 au lieu de les prendre à une partie existante, donc un fichier sans valeur impossible obtient exactement le score qu'il obtenait avant.", + "part": "Partie", + "found": "Constaté", + "weight": "Max", + "cost": "Coût", + "parts": { + "missing": "Cellules manquantes", + "duplicates": "Lignes en double", + "messy": "Orthographes désordonnées", + "outliers": "Valeurs aberrantes", + "structural": "Colonnes inutiles", + "validity": "Valeurs impossibles" } }, "issues": { @@ -734,6 +748,27 @@ "constant": "constante", "nearEmpty": "quasi vide", "id": "identifiant" + }, + "validity": { + "title_one": "{{count}} valeur impossible", + "title_other": "{{count}} valeurs impossibles", + "body": "Présente, bien typée, et pourtant impossible. Signalée, jamais réparée — c'est la recette qui modifie les données.", + "rules": { + "ageRange": "âge hors 0–120", + "futureDate": "date dans le futur", + "percentRange": "pourcentage hors 0–100", + "negativeAmount": "montant négatif", + "postcodeShape": "code postal malformé" + } + }, + "consistency": { + "title_one": "{{count}} ligne contradictoire", + "title_other": "{{count}} lignes contradictoires", + "body": "Chaque cellule est correcte ; la ligne ne l'est pas. Deux colonnes se contredisent.", + "rules": { + "dateOrder": "{{columns}} : la fin précède le début", + "productMismatch": "{{columns}} : le total n'est pas quantité × prix" + } } }, "recipe": {