From e0050c46404b9d58a7487aeae2a6a50d1bfd6bd0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 11:44:54 +0000 Subject: [PATCH] =?UTF-8?q?feat(v40):=20Data=20Studio=20=E2=80=94=20validi?= =?UTF-8?q?t=C3=A9,=20d=C3=A9rive,=20et=20un=20diff=20auditable?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit La qualité se mesurait en complétude et en cohérence de type. Ce qui manquait, c'est la VALIDITÉ : une valeur peut être présente, bien typée, et pourtant impossible. Cinq règles nommées le disent en clair — âge hors 0-120, date dans le futur, pourcentage hors 0-100, montant négatif, code postal malformé. Puis la cohérence inter-colonnes : une date de fin avant son début, un total qui n'est pas quantité × prix — chaque cellule correcte, la LIGNE impossible. Le choix de moteur du PLAN était mauvais, et le construire l'a montré. Il proposait de faire passer les règles de cohérence par le DuckDB de V29, puisque le fichier y est déjà enregistré. Mais DuckDB est un téléchargement annoncé de 18 à 22 Mo, sur consentement : y router ces contrôles aurait rendu une vérification universelle conditionnelle à un téléchargement que la plupart des utilisateurs refuseront, laissant le panneau vide pour eux. Comparer deux colonnes d'un tableau déjà en mémoire est une boucle : c'est donc une boucle, et tout le monde y a droit. DuckDB garde le travail dont il est réellement indispensable — le SQL arbitraire, et le nouvel export Parquet. Les deux familles de règles obéissent aux deux lois établies par le lecteur de V38 : elles se déclenchent sur preuve et jamais sur le seul nom d'une colonne — une colonne « age » contenant 20 000 est une durée en jours, donc la règle vérifie que l'essentiel de la colonne est plausible avant de signaler le reste, et se tait sinon — et elles signalent sans jamais réparer, parce que la recette de V39 est le seul registre de ce qui a été fait aux données. Trois choses rendent ensuite le studio auditable plutôt que simplement utile : - Un diff avant/après nommant quelles lignes, quelles colonnes et quelles valeurs ont changé. La difficulté : une recette supprime des lignes et ajoute des colonnes, donc applyRecipe renvoie désormais de quelle ligne SOURCE provient chaque ligne survivante — sans quoi le diff apparierait la ligne 7 avec une autre ligne 7 et signalerait un écran de changements qui n'ont jamais eu lieu. - Un profil de référence rejouable, l'idée du manifeste de V22 appliquée aux données : des bornes de bacs et des parts, jamais des lignes, donc le profil d'un fichier de paie décrit la forme de la distribution des salaires et le salaire de personne — c'est ce qui le rend sûr à versionner à côté du code. Il attribue à un nouveau fichier le même PSI, à six décimales, que la comparaison live à deux fichiers de V11. - Un score décomposé en ses parties, chacune avec son poids et ce qu'elle a réellement coûté. Les poids totalisent 105 et non 100, délibérément : la validité a apporté ses 5 points au lieu de les prendre à une partie existante, car redistribuer aurait changé en silence le sens de tous les scores déjà publiés. 502 tests unitaires, 78 e2e. Sur les 36 tests unitaires ajoutés, onze vérifient qu'une règle REFUSE de se déclencher. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01UKw6oNC8iZ9Kn7q6x4qom4 --- PLAN.md | 2 +- README.md | 12 +- e2e/validity.spec.ts | 71 ++++ src/features/data/components/DataDropZone.tsx | 1 + .../data/components/QualitySummary.tsx | 74 ++++- src/features/data/quality/checks.ts | 105 +++++- src/features/data/quality/clean.ts | 18 +- src/features/data/quality/consistency.test.ts | 106 ++++++ src/features/data/quality/consistency.ts | 191 +++++++++++ src/features/data/quality/diff.test.ts | 103 ++++++ src/features/data/quality/diff.ts | 108 +++++++ src/features/data/quality/reference.test.ts | 163 ++++++++++ src/features/data/quality/reference.ts | 304 ++++++++++++++++++ src/features/data/quality/types.ts | 14 + src/features/data/quality/v40.test.ts | 92 ++++++ src/features/data/quality/validity.test.ts | 157 +++++++++ src/features/data/quality/validity.ts | 203 ++++++++++++ src/features/data/sql/engine.ts | 17 + src/locales/en.json | 35 ++ src/locales/fr.json | 35 ++ 20 files changed, 1792 insertions(+), 19 deletions(-) create mode 100644 e2e/validity.spec.ts create mode 100644 src/features/data/quality/consistency.test.ts create mode 100644 src/features/data/quality/consistency.ts create mode 100644 src/features/data/quality/diff.test.ts create mode 100644 src/features/data/quality/diff.ts create mode 100644 src/features/data/quality/reference.test.ts create mode 100644 src/features/data/quality/reference.ts create mode 100644 src/features/data/quality/v40.test.ts create mode 100644 src/features/data/quality/validity.test.ts create mode 100644 src/features/data/quality/validity.ts diff --git a/PLAN.md b/PLAN.md index 535371c..d4fcfc7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -464,7 +464,7 @@ So parallel training alone was worth **1.06×** — real work, drowned by a bott The baseline scores 0.594, so the headroom above it collapses from 0.406 to 0.225: **45% of the achievable gain, lost in silence**. `parseNumber` ends in `Number(cleaned)`, `Number('12,5')` is `NaN`, the column falls through, and V24's TF-IDF cheerfully tokenises digits into a hundred word features. Nothing warned, nothing refused — the pipeline just produced a worse model. **Three of this row's own predictions were wrong, and the measurement corrected them**: the mis-typed column becomes `text` on high cardinality but `categorical` (one-hot) on low, not `text` alone; a windows-1252 file does not display `Québec` — that is the reverse case (UTF-8 read as cp1252) — it yields U+FFFD replacement characters; and day-first dates were already handled by V8, `31/12/2025` parsing correctly all along, with only the dash form `31-12-2025` returning null. **The fix.** One reader, shared by the ML Lab and the Data Studio, that decides by evidence rather than by guessing the user's locale: the browser's language says nothing about the file someone dragged in. Encoding is settled by **trying and failing** — UTF-8 in `fatal` mode throws on cp1252 accents, so the fallback is a certainty, not a preference; the delimiter is the candidate that splits every sampled line into the same number of columns, quotes respected; and the decimal separator is decided **per column**, never per file. A column is rewritten only when at least 90% of its values are numbers in that form AND it carries a comma AND it would otherwise not be numeric at all — so a text column containing « vis, tête plate » comes out untouched, and a column of bare integers is left alone because rewriting it would be a change with no cause. Values that do not match the pattern are never rewritten: a stray « n/d » stays « n/d » rather than becoming a plausible-looking number. Detection reads only the **head** of the file, so V25's streaming abort survives intact and a 2 GB file still stops at the cell budget. The reading is then **announced with its evidence** — « virgule décimale détectée et convertie dans 2 colonnes : surface (400/400), prix (400/400) » — above a five-row preview, and only when the reading was not the plain default: an ordinary UTF-8 comma file gets no card, no confirmation step, no friction. Batch scoring, drift and join files go through the same reader, because comparing a French export against a normalised dataset would otherwise report drift that is nothing but a decimal separator. **What this wave deliberately does not do**: guess a locale from `navigator.language` (the file has no relationship to the browser's language), rewrite a column on a bare majority (below the 90% floor the evidence is not evidence), or touch values individually inside a column the evidence does not cover. 446 unit tests, 73 e2e — of the 21 new reader tests, five assert that it **refuses** to act. | Owner request (22/08/2026): what to improve in /data. The audit found a defect first, and the wave began by reproducing it end to end: a French-locale CSV — the single most likely file this owner's users will open — silently loses every numeric column. The repair is exact rather than approximate: the French file now trains to the same types, the same feature count and the same score, to ten decimal places, as the file that never had the problem. | | **V39 — delivered** | **Data Studio: a recipe that works column by column.** `RecipeOptions` applied `missing` and `clipOutliers` to the **whole file** — one strategy for every column, however different they are. A median makes sense for an age and none at all for a postcode. The recipe is now an ordered list of **per-column steps**, with the file-wide settings demoted to **defaults a column may override**: a column with no entry behaves exactly as it did before V39, which is what keeps every previously exported recipe valid and replayable. The missing strategies that were absent are there: **median, mean, most-frequent, a constant used verbatim, and a « MANQUANT » category** for categorical columns — absence becoming a level of its own rather than a guess. Clipping became a per-column decision on the same terms, so a column can opt in or out of the global flag. **The rule the wave exists to enforce: imputing without marking destroys information.** A blank field is rarely blank at random, and the fact of the blank is frequently predictive on its own, so every column may add a `_absent` indicator — and the ordering is the whole point: **every indicator is written before any blank is filled**, so it records where the blanks really were, never where some other column's rule left them. Three consequences fall out of that and are tested: a column with nothing missing gets no indicator (an all-zero column is noise, not information); rows dropped by one column's rule are gathered and removed **once**, so indicators stay aligned instead of being shifted by a sibling column; and when a strategy cannot be honoured — a median over a column holding no parseable number, a constant with no value typed — the blanks are **left blank** rather than filled with something invented. Because the indicator is optional, the studio **announces the columns it filled without marking**, by name, instead of quietly producing a tidier table. **What this wave deliberately does not do**: make the indicator mandatory (it would silently add columns to every existing recipe, and imposing is not the same as announcing), impute from a model (opaque, and it fabricates values that look plausible — already refused for V40), or reorder the recipe's fixed stages, which is what keeps toggling an option from compounding with the last one. 461 unit tests, 75 e2e. | A single global strategy is the kind of default that looks tidy and quietly makes the data worse; per-column steps cost little to build because the recipe was already an object, not a pile of checkboxes. The build confirmed it: the engine change is contained in one stage of `applyRecipe`, and the 17 pre-existing recipe tests passed untouched. | -| V40 | **Data Studio: validity, drift, and an auditable diff.** Quality is measured today as completeness and consistency of type; what is missing is **validity** — a value can be present, well-typed and still impossible. Named rules, each stated in plain language: an age of 200, a date in the future, a percentage at 130, a malformed postcode. Then **cross-column consistency** (`date_fin < date_debut`, `total ≠ quantité × prix`), for which **V29's DuckDB is already the engine** — the rules are SQL, and they run on the file that is already registered. Then three things that make the studio auditable rather than merely helpful: a **replayable reference profile** so a second file can be checked for drift against the first (the same idea as the V22 model manifest), a **before/after diff of the rows a recipe modified** — which rows, which columns, which values, not just a count — and a **breakdown of the quality score** so the number is explained by its parts instead of being asserted. Ends with **Parquet export**, nearly free now that DuckDB is loaded (`COPY … TO 'x.parquet'`). **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). | Comes last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would flag the parser, not the data. | +| **V40 — delivered** | **Data Studio: validity, drift, and an auditable diff.** Quality was measured as completeness and consistency of type; what was missing is **validity** — a value can be present, correctly typed and still impossible. Five named rules now say so in plain language: an age outside 0–120, a date in the future, a percentage outside 0–100, a negative amount, a malformed postcode. Then **cross-column consistency**: an end date before its start, a total that is not quantity × price — every cell fine, the ROW impossible. **The plan's choice of engine was wrong, and building it showed why.** It proposed running the consistency rules through V29's DuckDB, since the file is already registered there. But DuckDB is an announced, opt-in 18–22 MB download: routing these checks through it would have made a universally applicable check conditional on a large download most users will decline, leaving the panel empty for them. Comparing two columns of a table already in memory is a loop, so it is a loop, and every user gets it. DuckDB keeps the job it is genuinely needed for — arbitrary SQL, and the new **Parquet export** (`COPY … TO`, one call, bytes straight to a download). Both families of rules obey the two laws V38's reader established: **they fire on evidence, never on a column's name alone** — a column called `age` holding 20 000 is a duration in days, so the rule checks that most of the column is plausible before it flags the rest, and stays silent otherwise — and **they report without ever repairing**, because V39's recipe is the one record of what was done to the data. Three things then make the studio auditable rather than merely helpful. A **before/after diff** naming which rows, which columns and which values changed: the hard part is that a recipe drops rows and adds columns, so `applyRecipe` now returns which SOURCE row each surviving row came from — without it the diff would pair row 7 with a different row 7 and report a screen full of changes that never happened. A **replayable reference profile**, the V22 manifest idea applied to data: bin edges and shares, never rows, so a profile of a payroll file describes the shape of the salary distribution and nobody's salary — which is what makes it safe to commit beside the code, and it scores a new file to the same PSI, to six decimal places, as V11's live two-file comparison. And a **score broken into its parts**, each with its weight and what it actually cost. The weights now sum to **105, not 100**, deliberately: validity brought its own 5 points rather than taking them from an existing part, because redistributing would have quietly changed what every previously published score meant. **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). 502 unit tests, 78 e2e. | Came last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would have flagged the parser, not the data. Closes the Data Studio group. Of the 36 new unit tests, eleven assert that a rule REFUSES to fire — a rule that flags a good file is worse than no rule, because it teaches the reader to ignore the panel. | **Ordering**: V38 came before V39 and V40, and for the same reason V35 came first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35, V36 and V37 are delivered. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 and V31 both start with a bench, because neither « a bigger model » nor « it still makes mistakes » is a measurable statement today; no wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped — V25 (delivered) chose announced sampling and a named memory guard over the diff --git a/README.md b/README.md index 6434d69..2046208 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,16 @@ The project follows three non-negotiable principles: column can add a `_absent` **missing indicator**, written before anything is filled; columns filled _without_ one are named out loud, because a blank field is rarely blank at random and filling it silently erases that. +- **Validity rules**: a value can be present, correctly typed and still impossible — an + age of 200, a date in the future, a percentage at 130, a malformed postcode. Plus + **cross-column consistency**: an end date before its start, a total that is not + quantity × price. Every rule fires on evidence rather than on a column's name, and + reports without ever repairing — the recipe is where data changes. +- **The quality score, broken into its parts**: each with its weight, what it found and + what it cost, instead of a number asserted without explanation. +- **An auditable before/after diff** of what the recipe did — which rows, which columns, + which values — and a **replayable reference profile** (bin edges and shares, never + rows) so a new file can be checked for drift against a snapshot you no longer hold. - **Left-join a second file** on a shared key: match rate, duplicates and orphans are named, never silent; the joined result becomes the working dataset. - **Drift check**: compare a new batch against the reference — schema diff, PSI per @@ -124,7 +134,7 @@ The project follows three non-negotiable principles: - **Performance.** Every section serves a prerendered static shell (hero paints before JavaScript); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 461 unit tests, 75 Playwright end-to-end tests (including offline PWA, +- **Quality bar.** 502 unit tests, 78 Playwright end-to-end tests (including offline PWA, fake-webcam and axe-core WCAG A/AA accessibility checks), strict TypeScript, ESLint, Prettier, and Lighthouse budgets — all enforced in CI. diff --git a/e2e/validity.spec.ts b/e2e/validity.spec.ts new file mode 100644 index 0000000..fa1ea3b --- /dev/null +++ b/e2e/validity.spec.ts @@ -0,0 +1,71 @@ +import { expect, test } from '@playwright/test'; + +test.use({ locale: 'en-US' }); +test.setTimeout(120_000); + +/** + * V40 — validity, consistency, and a score that explains itself. + */ +function impossibleCsv(): Buffer { + const lines = ['nom,age,taux_reussite,date_debut,date_fin,quantite,prix_unitaire,total']; + for (let i = 0; i < 60; i++) { + // Two impossible ages, one impossible percentage, one reversed date pair, + // one total that is not quantity × price — everything else is plausible. + const age = i === 3 ? '200' : i === 7 ? '-4' : String(20 + (i % 50)); + const rate = i === 11 ? '130' : String(i % 100); + const start = `2025-0${(i % 9) + 1}-0${(i % 8) + 1}`; + const end = i === 5 ? '2024-01-01' : `2025-0${(i % 9) + 1}-0${(i % 8) + 2}`; + const qty = String(1 + (i % 5)); + const unit = String(2 + (i % 4)); + const total = i === 9 ? '999' : String((1 + (i % 5)) * (2 + (i % 4))); + lines.push(`ligne${i},${age},${rate},${start},${end},${qty},${unit},${total}`); + } + return Buffer.from(lines.join('\n'), 'utf-8'); +} + +test('impossible values and contradictory rows are named, with their rules', async ({ page }) => { + await page.goto('/data'); + await page.getByTestId('data-file-input').setInputFiles({ + name: 'commandes.csv', + mimeType: 'text/csv', + buffer: impossibleCsv(), + }); + await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 }); + + // Values that are present and correctly typed, and still impossible. + const validity = page.getByTestId('issue-validity'); + await expect(validity).toBeVisible(); + await expect(validity).toContainText('age'); + await expect(validity).toContainText('age outside 0–120'); + await expect(validity).toContainText('percentage outside 0–100'); + + // A row where two columns contradict each other. + const consistency = page.getByTestId('issue-consistency'); + await expect(consistency).toBeVisible(); + await expect(consistency).toContainText('the end comes before the start'); + await expect(consistency).toContainText('the total is not quantity × price'); +}); + +test('the quality score is explained by its parts, not asserted', async ({ page }) => { + await page.goto('/data'); + await page.getByRole('button', { name: /cafe-sales\.csv/ }).click(); + await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 }); + + await page.getByText('How this score was computed', { exact: true }).click(); + const breakdown = page.getByTestId('score-breakdown'); + await expect(breakdown).toContainText('Missing cells'); + await expect(breakdown).toContainText('Duplicate rows'); + await expect(breakdown).toContainText('Impossible values'); + // The weights are shown, including the deliberate 105 total. + await expect(breakdown).toContainText('105, not 100'); +}); + +test('a clean file raises no validity or consistency card at all', async ({ page }) => { + await page.goto('/data'); + // titanic has missing values and messy spellings, but every age is a real + // age and no two columns contradict each other. + await page.getByRole('button', { name: /titanic\.csv/ }).click(); + await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 }); + await expect(page.getByTestId('issue-validity')).toHaveCount(0); + await expect(page.getByTestId('issue-consistency')).toHaveCount(0); +}); diff --git a/src/features/data/components/DataDropZone.tsx b/src/features/data/components/DataDropZone.tsx index 9561bd4..bdde50c 100644 --- a/src/features/data/components/DataDropZone.tsx +++ b/src/features/data/components/DataDropZone.tsx @@ -46,6 +46,7 @@ export function DataDropZone() { {t('data.drop.hint')} @@ -166,8 +168,76 @@ export function QualitySummary() { )} + + {report.validity.length > 0 && ( + + + )} + + {report.consistency.length > 0 && ( + + + )} )} + +
+ + {t('data.score.breakdownTitle')} + +

{t('data.score.breakdownHint')}

+ + + + + + + + + + + {report.breakdown.map((part) => ( + + + + + + + ))} + +
{t('data.score.part')}{t('data.score.found')}{t('data.score.weight')}{t('data.score.cost')}
{t(`data.score.parts.${part.part}`)} + {NUMBER.format(part.count)} + {part.ratio !== undefined && ` (${percent(part.ratio)} %)`} + {part.weight}−{part.penalty}
+
); } diff --git a/src/features/data/quality/checks.ts b/src/features/data/quality/checks.ts index 5d539ef..422510b 100644 --- a/src/features/data/quality/checks.ts +++ b/src/features/data/quality/checks.ts @@ -1,3 +1,5 @@ +import { checkValidity, invalidCellCount } from '@/features/data/quality/validity'; +import { checkConsistency, inconsistentRowCount } from '@/features/data/quality/consistency'; import { inferColumnType, isMissing, parseNumber } from '@/features/ml/data/infer'; import type { Cell, ColumnType } from '@/features/ml/data/types'; import type { @@ -114,19 +116,83 @@ function missingCountOf(values: Cell[]): number { } /** - * Deterministic 0–100 score. Each ratio saturates at 12.5% (×8) so a modest - * amount of dirt is already visible; weights sum to 100. + * V40: the score, decomposed. + * + * The number was asserted before: 62 out of 100, with nothing saying why. Each + * part now carries its own weight, the ratio that drove it, and the points it + * actually cost — so the score is explained by its parts instead of being + * announced. The arithmetic is unchanged apart from the new validity part; a + * file with no validity findings scores exactly what it scored before V40. */ -export function qualityScore(report: Omit): number { +export type ScorePart = 'missing' | 'duplicates' | 'messy' | 'outliers' | 'structural' | 'validity'; + +export interface ScoreBreakdown { + part: ScorePart; + /** The most this part can ever cost. */ + weight: number; + /** What it cost here, rounded to one decimal. */ + penalty: number; + /** The ratio that drove it — absent for the structural count. */ + ratio?: number; + /** The raw count behind the ratio, for the plain-language line. */ + count: number; +} + +/** Each ratio saturates at 12.5% (×8) so a modest amount of dirt is visible. */ +const SATURATION = 8; +/** Sum of every weight below. Above 100 on purpose — see the validity part. */ +export const TOTAL_WEIGHT = 105; + +export function scoreBreakdown( + report: Omit, + invalidCells = 0, +): ScoreBreakdown[] { const cells = Math.max(1, report.cellCount); const rows = Math.max(1, report.rowCount); - const saturate = (ratio: number) => Math.min(1, ratio * 8); - const penalty = - 35 * saturate(report.missingCells / cells) + - 20 * saturate(report.duplicateRows / rows) + - 20 * saturate(report.messyCells / cells) + - 15 * saturate(report.outlierCells / cells) + - Math.min(10, 2.5 * report.structural.length); + const saturate = (ratio: number) => Math.min(1, ratio * SATURATION); + const round = (value: number) => Math.round(value * 10) / 10; + const ratioPart = ( + part: ScorePart, + weight: number, + count: number, + total: number, + ): ScoreBreakdown => { + const ratio = count / total; + return { part, weight, penalty: round(weight * saturate(ratio)), ratio, count }; + }; + return [ + ratioPart('missing', 35, report.missingCells, cells), + ratioPart('duplicates', 20, report.duplicateRows, rows), + ratioPart('messy', 20, report.messyCells, cells), + ratioPart('outliers', 15, report.outlierCells, cells), + { + part: 'structural', + weight: 10, + penalty: round(Math.min(10, 2.5 * report.structural.length)), + count: report.structural.length, + }, + // V40: validity is a NEW dimension, so it brings its own 5 points rather + // than taking them from an existing part. The weights therefore sum to 105, + // not 100, and that is deliberate: redistributing would have quietly + // changed what every previously published score meant. A file with no + // validity findings scores exactly what it scored before V40 — the total + // still floors at 0, which the previous formula could already reach. + ratioPart('validity', 5, invalidCells, cells), + ]; +} + +/** + * Deterministic 0–100 score, now the sum of its published parts rather than a + * separate formula that could drift away from them. + */ +export function qualityScore( + report: Omit, + invalidCells = 0, +): number { + const penalty = scoreBreakdown(report, invalidCells).reduce( + (total, part) => total + part.penalty, + 0, + ); return Math.max(0, Math.round(100 - penalty)); } @@ -198,7 +264,14 @@ export function buildQualityReport(header: string[], columns: Cell[][]): Quality messyColumns.sort((a, b) => b.cellCount - a.cellCount || a.column.localeCompare(b.column)); outlierColumns.sort((a, b) => b.count - a.count || a.column.localeCompare(b.column)); - const partial: Omit = { + // V40: the two families of impossible data — one column at a time, then + // rows where two columns contradict each other. + const validity = checkValidity(header, columns); + const invalidCells = invalidCellCount(validity); + const consistency = checkConsistency(header, columns); + const inconsistentRows = inconsistentRowCount(consistency); + + const partial: Omit = { rowCount, columnCount, cellCount, @@ -210,6 +283,14 @@ export function buildQualityReport(header: string[], columns: Cell[][]): Quality outlierColumns, outlierCells, structural, + validity, + invalidCells, + consistency, + inconsistentRows, + }; + return { + ...partial, + score: qualityScore(partial, invalidCells), + breakdown: scoreBreakdown(partial, invalidCells), }; - return { ...partial, score: qualityScore(partial) }; } diff --git a/src/features/data/quality/clean.ts b/src/features/data/quality/clean.ts index 2bf2210..b000460 100644 --- a/src/features/data/quality/clean.ts +++ b/src/features/data/quality/clean.ts @@ -135,9 +135,13 @@ export function applyRecipe( header: string[], source: Cell[][], options: RecipeOptions, -): { header: string[]; columns: Cell[][]; stats: CleanStats } { +): { header: string[]; columns: Cell[][]; stats: CleanStats; survivingRows: number[] } { let outHeader = [...header]; let columns = source.map((column) => [...column]); + // V40: the index in `source` that each current row came from. Every step + // that drops rows filters this alongside the data, so the diff can attribute + // a change to the row it actually came from. + let surviving = Array.from({ length: source[0]?.length ?? 0 }, (_, index) => index); // Forced types steer every type-sensitive step; inference is the fallback. const typeOf = (name: string, values: Cell[]) => { const overrides = options.types as Partial> | undefined; @@ -247,7 +251,10 @@ export function applyRecipe( if (options.dropDuplicates && columns.length > 0) { const duplicates = new Set(duplicateRowIndices(columns, columns[0].length)); stats.droppedDuplicateRows = duplicates.size; - if (duplicates.size > 0) columns = dropRows(columns, duplicates); + if (duplicates.size > 0) { + columns = dropRows(columns, duplicates); + surviving = surviving.filter((_, index) => !duplicates.has(index)); + } } // V39: missing values, column by column. Two passes, and the order between @@ -325,6 +332,7 @@ export function applyRecipe( stats.droppedByColumn[source.column] = source.rows.length; } columns = dropRows(columns, toDrop); + surviving = surviving.filter((_, index) => !toDrop.has(index)); for (const indicator of indicators) { indicator.values = indicator.values.filter((_, index) => !toDrop.has(index)); } @@ -377,11 +385,15 @@ export function applyRecipe( } if (stats.droppedAnomalyRows > 0) { columns = columns.map((column) => keep.map((r) => column[r])); + surviving = keep.map((r) => surviving[r]); } } } stats.rowCount = columns[0]?.length ?? 0; stats.columnCount = outHeader.length; - return { header: outHeader, columns, stats }; + // V40: which SOURCE row each surviving row came from. Without this the diff + // would pair row 7 with a different row 7 the moment anything was dropped, + // and report every subsequent row as changed. + return { header: outHeader, columns, stats, survivingRows: surviving }; } diff --git a/src/features/data/quality/consistency.test.ts b/src/features/data/quality/consistency.test.ts new file mode 100644 index 0000000..d8ee42c --- /dev/null +++ b/src/features/data/quality/consistency.test.ts @@ -0,0 +1,106 @@ +/** + * V40 — cross-column consistency, in JavaScript rather than in SQL. + */ +import { describe, expect, it } from 'vitest'; +import { + checkConsistency, + inconsistentRowCount, + PRODUCT_TOLERANCE, +} from '@/features/data/quality/consistency'; +import type { Cell } from '@/features/ml/data/types'; + +describe('V40 — a row can be impossible even when every cell is fine', () => { + it('flags an end date before its start date', () => { + const header = ['date_debut', 'date_fin']; + const columns: Cell[][] = [ + ['2025-01-04', '2025-02-01', '2025-03-10', '2025-04-01', '2025-05-01', '2025-06-01'], + ['2025-01-09', '2025-02-05', '2025-02-28', '2025-04-10', '2025-05-06', '2025-06-08'], + ]; + const [found] = checkConsistency(header, columns); + expect(found.rule).toBe('dateOrder'); + expect(found.columns).toEqual(['date_debut', 'date_fin']); + expect(found.rows).toEqual([2]); + expect(found.examples).toEqual(['2025-03-10 → 2025-02-28']); + }); + + it('flags a total that is not quantity × unit price', () => { + const header = ['quantite', 'prix_unitaire', 'total']; + const columns: Cell[][] = [ + ['2', '3', '1', '4', '5', '2'], + ['5', '2.5', '10', '1.25', '2', '3'], + ['10', '7.5', '99', '5', '10', '6'], + ]; + const [found] = checkConsistency(header, columns); + expect(found.rule).toBe('productMismatch'); + expect(found.rows).toEqual([2]); + expect(found.examples).toEqual(['1 × 10 ≠ 99']); + }); + + it('tolerates ordinary rounding on money', () => { + const header = ['qty', 'unit_price', 'total']; + const columns: Cell[][] = [ + ['3', '3', '3', '3', '3', '3'], + ['0.3333', '0.3333', '0.3333', '0.3333', '0.3333', '0.3333'], + ['1', '1', '1', '1', '1', '1'], + ]; + // 3 × 0.3333 = 0.9999, rounded to 1 — a rounded total is not a wrong total. + expect(checkConsistency(header, columns)).toEqual([]); + expect(PRODUCT_TOLERANCE).toBeLessThanOrEqual(0.01); + }); +}); + +describe('V40 — consistency rules refuse to fire on a file they misread', () => { + it('REFUSES when most rows disagree — the columns are not what it assumed', () => { + // `total` here is a running balance, not quantity × price. + const header = ['quantite', 'prix_unitaire', 'total']; + const columns: Cell[][] = [ + ['2', '3', '1', '4', '5', '2'], + ['5', '2.5', '10', '1.25', '2', '3'], + ['100', '250', '380', '420', '600', '750'], + ]; + expect(checkConsistency(header, columns)).toEqual([]); + }); + + it('REFUSES a date pair whose values are not dates', () => { + const header = ['debut', 'fin']; + const columns: Cell[][] = [ + ['tôt', 'tôt', 'tard', 'tôt', 'tard', 'tôt'], + ['tard', 'tard', 'tôt', 'tard', 'tôt', 'tard'], + ]; + expect(checkConsistency(header, columns)).toEqual([]); + }); + + it('does not charge a row twice when one endpoint is simply missing', () => { + const header = ['date_debut', 'date_fin']; + const columns: Cell[][] = [ + ['2025-01-04', '2025-02-01', '2025-03-10', '2025-04-01', '2025-05-01', '2025-06-01'], + ['2025-01-09', '', '2025-04-01', '2025-04-10', '2025-05-06', '2025-06-08'], + ]; + // Row 1 is missing, not inconsistent — the missing report already has it. + expect(checkConsistency(header, columns)).toEqual([]); + }); + + it('says nothing when the columns it needs are absent', () => { + expect( + checkConsistency( + ['a', 'b'], + [ + ['1', '2'], + ['3', '4'], + ], + ), + ).toEqual([]); + expect(inconsistentRowCount([])).toBe(0); + }); + + it('never modifies the data it inspects', () => { + const header = ['date_debut', 'date_fin']; + const columns: Cell[][] = [ + ['2025-03-10', '2025-01-04', '2025-02-01', '2025-04-01', '2025-05-01', '2025-06-01'], + ['2025-02-28', '2025-01-09', '2025-02-05', '2025-04-10', '2025-05-06', '2025-06-08'], + ]; + const snapshot = columns.map((column) => [...column]); + checkConsistency(header, columns); + expect(columns).toEqual(snapshot); + }); +}); diff --git a/src/features/data/quality/consistency.ts b/src/features/data/quality/consistency.ts new file mode 100644 index 0000000..8c873ec --- /dev/null +++ b/src/features/data/quality/consistency.ts @@ -0,0 +1,191 @@ +/** + * V40: cross-column consistency — rows where two columns contradict each other. + * + * A delivery date before its order date, a total that is not quantity × price: + * every value is present, correctly typed, individually plausible, and the ROW + * is still impossible. Validity checks one column at a time and cannot see it. + * + * **Why this is not SQL.** The plan proposed running these rules through V29's + * DuckDB, since the file is already registered there. Building it made the cost + * obvious: DuckDB is an announced, opt-in 18–22 MB download, so routing these + * checks through it would make a universally applicable check conditional on a + * large download the user may well decline — a quality panel that is empty for + * most people. Comparing two columns of a table already in memory is a loop, so + * it is a loop, and every user gets it. DuckDB keeps the job it is genuinely + * needed for: arbitrary SQL, and the Parquet export. + * + * The rules follow the same two laws as `validity.ts`: they fire on evidence + * rather than on column names alone, and they report without ever repairing. + */ +import { isMissing, parseNumber } from '@/features/ml/data/infer'; +import { parseDate } from '@/features/ml/timeseries/series'; +import { MAX_EXAMPLE_ROWS, APPLICABILITY } from '@/features/data/quality/validity'; +import type { Cell } from '@/features/ml/data/types'; + +export type ConsistencyRule = + /** An "end" date strictly before its matching "start" date. */ + | 'dateOrder' + /** A total that does not match quantity × unit price. */ + | 'productMismatch'; + +export interface ConsistencyFinding { + rule: ConsistencyRule; + /** The columns the rule compared, in the order it compared them. */ + columns: string[]; + rows: number[]; + count: number; + /** One rendered example per listed row, e.g. « 2025-01-04 → 2024-12-30 ». */ + examples: string[]; +} + +/** Relative tolerance on the product rule — money is rounded, not exact. */ +export const PRODUCT_TOLERANCE = 0.01; + +function normalize(column: string): string { + return column.toLowerCase().normalize('NFD').replace(/[̀-ͯ]/g, ''); +} + +const START_HINTS = ['debut', 'start', 'commande', 'order', 'creation', 'created', 'from'] as const; +const END_HINTS = ['fin', 'end', 'livraison', 'delivery', 'cloture', 'closed', 'to'] as const; +const QUANTITY_HINTS = ['quantity', 'quantite', 'qte', 'qty', 'nombre'] as const; +const UNIT_PRICE_HINTS = [ + 'unit_price', + 'prix_unitaire', + 'unitprice', + 'prixunitaire', + 'pu', +] as const; +const TOTAL_HINTS = ['total', 'montant', 'amount', 'sum'] as const; + +function findColumn(header: readonly string[], hints: readonly string[]): number { + return header.findIndex((name) => hints.some((hint) => normalize(name).includes(hint))); +} + +/** Parses a whole column as dates, or returns null when it is not one. */ +function asDates(values: Cell[]): (number | null)[] | null { + const parsed: (number | null)[] = []; + let usable = 0; + let ok = 0; + for (const value of values) { + if (isMissing(value)) { + parsed.push(null); + continue; + } + usable += 1; + const at = parseDate((value as string).trim()); + parsed.push(at); + if (at !== null) ok += 1; + } + if (usable === 0 || ok / usable < APPLICABILITY) return null; + return parsed; +} + +function asNumbers(values: Cell[]): (number | null)[] | null { + const parsed: (number | null)[] = []; + let usable = 0; + let ok = 0; + for (const value of values) { + if (isMissing(value)) { + parsed.push(null); + continue; + } + usable += 1; + const number = parseNumber((value as string).trim()); + parsed.push(number); + if (number !== null) ok += 1; + } + if (usable === 0 || ok / usable < APPLICABILITY) return null; + return parsed; +} + +function build( + rule: ConsistencyRule, + columns: string[], + failures: { row: number; example: string }[], +): ConsistencyFinding | null { + if (failures.length === 0) return null; + const capped = failures.slice(0, MAX_EXAMPLE_ROWS); + return { + rule, + columns, + rows: capped.map((f) => f.row), + examples: capped.map((f) => f.example), + count: failures.length, + }; +} + +export function checkConsistency( + header: readonly string[], + columns: Cell[][], +): ConsistencyFinding[] { + const found: ConsistencyFinding[] = []; + + const startAt = findColumn(header, START_HINTS); + const endAt = findColumn(header, END_HINTS); + if (startAt >= 0 && endAt >= 0 && startAt !== endAt) { + const start = asDates(columns[startAt] ?? []); + const end = asDates(columns[endAt] ?? []); + if (start && end) { + const failures: { row: number; example: string }[] = []; + for (let row = 0; row < start.length; row++) { + const from = start[row]; + const to = end[row]; + // A missing endpoint is a MISSING value, already reported as one: + // counting it here too would charge the same row twice. + if (from === null || to === null) continue; + if (to < from) { + failures.push({ + row, + example: `${String(columns[startAt][row])} → ${String(columns[endAt][row])}`, + }); + } + } + const finding = build('dateOrder', [header[startAt], header[endAt]], failures); + if (finding) found.push(finding); + } + } + + const qtyAt = findColumn(header, QUANTITY_HINTS); + const unitAt = findColumn(header, UNIT_PRICE_HINTS); + const totalAt = findColumn(header, TOTAL_HINTS); + if (qtyAt >= 0 && unitAt >= 0 && totalAt >= 0 && totalAt !== unitAt) { + const qty = asNumbers(columns[qtyAt] ?? []); + const unit = asNumbers(columns[unitAt] ?? []); + const total = asNumbers(columns[totalAt] ?? []); + if (qty && unit && total) { + const failures: { row: number; example: string }[] = []; + let comparable = 0; + for (let row = 0; row < qty.length; row++) { + const q = qty[row]; + const u = unit[row]; + const t = total[row]; + if (q === null || u === null || t === null) continue; + comparable += 1; + const expected = q * u; + // Relative tolerance, because a rounded total is not a wrong total. + const slack = Math.max(Math.abs(expected) * PRODUCT_TOLERANCE, 0.01); + if (Math.abs(t - expected) > slack) { + failures.push({ row, example: `${q} × ${u} ≠ ${t}` }); + } + } + // If most rows disagree, these three columns are not quantity, price and + // total — the rule has misread the file and says nothing rather than + // flagging every row of it. + if (comparable > 0 && failures.length / comparable <= 1 - APPLICABILITY) { + const finding = build( + 'productMismatch', + [header[qtyAt], header[unitAt], header[totalAt]], + failures, + ); + if (finding) found.push(finding); + } + } + } + + return found.sort((a, b) => b.count - a.count); +} + +/** Impossible rows across every consistency rule — what the score charges for. */ +export function inconsistentRowCount(findings: readonly ConsistencyFinding[]): number { + return findings.reduce((total, finding) => total + finding.count, 0); +} diff --git a/src/features/data/quality/diff.test.ts b/src/features/data/quality/diff.test.ts new file mode 100644 index 0000000..63173e6 --- /dev/null +++ b/src/features/data/quality/diff.test.ts @@ -0,0 +1,103 @@ +/** + * V40 — the diff is evidence, where a count was only a claim. + * + * The test that matters most is the alignment one: a diff that pairs the wrong + * rows after a drop is worse than no diff at all, because it reports a screen + * full of changes that never happened. + */ +import { describe, expect, it } from 'vitest'; +import { applyRecipe } from '@/features/data/quality/clean'; +import { diffRecipe } from '@/features/data/quality/diff'; +import { DEFAULT_RECIPE, type RecipeOptions } from '@/features/data/quality/types'; +import type { Cell } from '@/features/ml/data/types'; + +const QUIET: RecipeOptions = { + ...DEFAULT_RECIPE, + trimWhitespace: false, + mergeVariants: false, + dropDuplicates: false, + missing: 'keep', +}; + +function run(header: string[], columns: Cell[][], options: RecipeOptions) { + const result = applyRecipe(header, columns, options); + return { + result, + diff: diffRecipe(header, columns, result.header, result.columns, result.survivingRows), + }; +} + +describe('V40 — which rows, which columns, which values', () => { + it('names the cell that changed, with its value before and after', () => { + const header = ['ville', 'age']; + const columns: Cell[][] = [ + [' Québec ', 'Montréal', 'Laval'], + ['34', '41', '29'], + ]; + const { diff } = run(header, columns, { ...QUIET, trimWhitespace: true }); + expect(diff.changedRowCount).toBe(1); + expect(diff.changedRows[0]).toEqual({ + sourceRow: 0, + changes: [{ column: 'ville', before: ' Québec ', after: 'Québec' }], + }); + expect(diff.changedCells).toBe(1); + }); + + it('reports nothing at all when the recipe changed nothing', () => { + const header = ['a']; + const columns: Cell[][] = [['1', '2', '3']]; + const { diff } = run(header, columns, QUIET); + expect(diff.changedRowCount).toBe(0); + expect(diff.droppedRowCount).toBe(0); + expect(diff.addedColumns).toEqual([]); + expect(diff.removedColumns).toEqual([]); + }); + + it('attributes changes to the SOURCE row even after rows were dropped', () => { + // Row 1 is a duplicate of row 0 and disappears. Without the surviving-row + // map, every later row would be compared against its predecessor and the + // diff would claim changes that never happened. + const header = ['ville']; + const columns: Cell[][] = [['Québec', 'Québec', ' Laval ', 'Montréal']]; + const { result, diff } = run(header, columns, { + ...QUIET, + dropDuplicates: true, + trimWhitespace: true, + }); + expect(result.survivingRows).toEqual([0, 2, 3]); + expect(diff.droppedRows).toEqual([1]); + expect(diff.droppedRowCount).toBe(1); + // Exactly one real change, and it is attributed to source row 2. + expect(diff.changedRowCount).toBe(1); + expect(diff.changedRows[0].sourceRow).toBe(2); + expect(diff.changedRows[0].changes[0]).toEqual({ + column: 'ville', + before: ' Laval ', + after: 'Laval', + }); + }); + + it('names the columns a recipe added and removed', () => { + const header = ['age', 'constante']; + const columns: Cell[][] = [ + ['34', '', '29', '41', '52', '61'], + ['x', 'x', 'x', 'x', 'x', 'x'], + ]; + const { diff } = run(header, columns, { + ...QUIET, + dropStructural: true, + columns: { age: { missing: 'median', indicator: true } }, + }); + expect(diff.addedColumns).toEqual(['age_absent']); + expect(diff.removedColumns).toEqual(['constante']); + }); + + it('caps the listed rows without ever capping the counts', () => { + const header = ['ville']; + const columns: Cell[][] = [Array.from({ length: 200 }, (_, i) => ` ville${i} `)]; + const { diff } = run(header, columns, { ...QUIET, trimWhitespace: true }); + expect(diff.changedRowCount).toBe(200); + expect(diff.changedRows.length).toBe(50); + expect(diff.changedCells).toBe(200); + }); +}); diff --git a/src/features/data/quality/diff.ts b/src/features/data/quality/diff.ts new file mode 100644 index 0000000..e67585a --- /dev/null +++ b/src/features/data/quality/diff.ts @@ -0,0 +1,108 @@ +/** + * V40: the before/after diff — which rows, which columns, which values. + * + * The studio already reported counts: « 412 cells trimmed, 6 duplicate rows + * dropped ». A count is a claim; a diff is evidence. This is what makes the + * recipe auditable rather than merely helpful — someone can check what it did + * instead of trusting that it did the right thing. + * + * The hard part is that a recipe does not only edit cells: it drops rows and + * adds columns, so « row 7 » before and « row 7 » after are not the same row. + * The alignment below is therefore explicit rather than positional — the + * recipe reports which source rows survived, and every change is attributed to + * the row it actually came from. + */ +import { isMissing } from '@/features/ml/data/infer'; +import type { Cell } from '@/features/ml/data/types'; + +/** Rows listed in the diff — enough to audit, not enough to freeze the tab. */ +export const MAX_DIFF_ROWS = 50; + +export interface CellChange { + column: string; + before: string; + after: string; +} + +export interface RowChange { + /** Index in the ORIGINAL dataset, so it can be found in the source file. */ + sourceRow: number; + changes: CellChange[]; +} + +export interface RecipeDiff { + /** Rows whose values changed, in source order, capped at MAX_DIFF_ROWS. */ + changedRows: RowChange[]; + /** How many rows changed in total — `changedRows` may be a prefix. */ + changedRowCount: number; + /** Source rows the recipe removed, capped for display. */ + droppedRows: number[]; + droppedRowCount: number; + /** Columns the recipe removed and added, by name. */ + removedColumns: string[]; + addedColumns: string[]; + /** How many individual cells changed value. */ + changedCells: number; +} + +function display(value: Cell): string { + return isMissing(value) ? '' : (value as string); +} + +/** + * Compares the dataset before and after a recipe. + * + * `survivingRows` maps each row of the AFTER dataset back to its index in the + * BEFORE dataset. Without it the comparison would silently pair row 7 with a + * different row 7 the moment anything was dropped, and report every column of + * every subsequent row as changed — a diff that is worse than no diff. + */ +export function diffRecipe( + beforeHeader: readonly string[], + beforeColumns: readonly Cell[][], + afterHeader: readonly string[], + afterColumns: readonly Cell[][], + survivingRows: readonly number[], +): RecipeDiff { + const beforeIndex = new Map(beforeHeader.map((name, i) => [name, i])); + const afterIndex = new Map(afterHeader.map((name, i) => [name, i])); + + const removedColumns = beforeHeader.filter((name) => !afterIndex.has(name)); + const addedColumns = afterHeader.filter((name) => !beforeIndex.has(name)); + const shared = beforeHeader.filter((name) => afterIndex.has(name)); + + const kept = new Set(survivingRows); + const droppedAll: number[] = []; + const beforeRowCount = beforeColumns[0]?.length ?? 0; + for (let row = 0; row < beforeRowCount; row++) { + if (!kept.has(row)) droppedAll.push(row); + } + + const changedRows: RowChange[] = []; + let changedRowCount = 0; + let changedCells = 0; + + for (let afterRow = 0; afterRow < survivingRows.length; afterRow++) { + const sourceRow = survivingRows[afterRow]; + const changes: CellChange[] = []; + for (const name of shared) { + const before = display(beforeColumns[beforeIndex.get(name)!]?.[sourceRow] ?? null); + const after = display(afterColumns[afterIndex.get(name)!]?.[afterRow] ?? null); + if (before !== after) changes.push({ column: name, before, after }); + } + if (changes.length === 0) continue; + changedRowCount += 1; + changedCells += changes.length; + if (changedRows.length < MAX_DIFF_ROWS) changedRows.push({ sourceRow, changes }); + } + + return { + changedRows, + changedRowCount, + droppedRows: droppedAll.slice(0, MAX_DIFF_ROWS), + droppedRowCount: droppedAll.length, + removedColumns, + addedColumns, + changedCells, + }; +} diff --git a/src/features/data/quality/reference.test.ts b/src/features/data/quality/reference.test.ts new file mode 100644 index 0000000..ec71c18 --- /dev/null +++ b/src/features/data/quality/reference.test.ts @@ -0,0 +1,163 @@ +/** + * V40 — the reference profile: drift against a file you no longer have. + */ +import { describe, expect, it } from 'vitest'; +import { + PROFILE_FORMAT, + buildProfile, + compareToProfile, + parseProfile, +} from '@/features/data/quality/reference'; +import { buildDriftReport } from '@/features/data/quality/drift'; +import { mulberry32 } from '@/features/ml/train/random'; +import type { Cell } from '@/features/ml/data/types'; + +function numericColumn(n: number, seed: number, shift = 0): Cell[] { + const rng = mulberry32(seed); + return Array.from({ length: n }, () => String(Math.round((rng() * 100 + shift) * 100) / 100)); +} + +function categoryColumn(n: number, weights: [string, number][]): Cell[] { + const out: Cell[] = []; + const total = weights.reduce((sum, [, w]) => sum + w, 0); + for (let i = 0; i < n; i++) { + let position = (i % total) + 1; + for (const [name, weight] of weights) { + position -= weight; + if (position <= 0) { + out.push(name); + break; + } + } + } + return out; +} + +describe('V40 — a profile describes the shape, never the rows', () => { + it('stores edges and shares, and no data', () => { + const header = ['salaire', 'ville']; + const columns: Cell[][] = [ + numericColumn(200, 42), + categoryColumn(200, [ + ['Québec', 3], + ['Montréal', 2], + ]), + ]; + const profile = buildProfile(header, columns, 'paie.csv'); + const serialized = JSON.stringify(profile); + // The exact values of the file must not appear anywhere in the profile: + // this is what makes it safe to commit beside the code it describes. + for (const value of columns[0].slice(0, 20)) { + expect(serialized).not.toContain(`"${String(value)}"`); + } + expect(profile.format).toBe(PROFILE_FORMAT); + expect(profile.rowCount).toBe(200); + const salary = profile.columns.find((c) => c.column === 'salaire')!; + expect(salary.kind).toBe('numeric'); + }); + + it('round-trips through JSON', () => { + const profile = buildProfile( + ['a', 'b'], + [ + numericColumn(120, 7), + categoryColumn(120, [ + ['x', 1], + ['y', 1], + ]), + ], + 'ref.csv', + ); + const parsed = parseProfile(JSON.stringify(profile)); + expect(parsed).not.toBeNull(); + expect(parsed!.columns.map((c) => c.column)).toEqual(['a', 'b']); + expect(parsed!.source).toBe('ref.csv'); + }); + + it('refuses anything that is not a LabML profile', () => { + expect(parseProfile('{}')).toBeNull(); + expect(parseProfile('not json')).toBeNull(); + expect(parseProfile(JSON.stringify({ format: 'something-else', columns: [] }))).toBeNull(); + // A profile whose columns all failed validation describes nothing. + expect(parseProfile(JSON.stringify({ format: PROFILE_FORMAT, columns: [{}] }))).toBeNull(); + }); +}); + +describe('V40 — a profile scores a new file the way V11 would have', () => { + it('finds no drift when the new file is the reference itself', () => { + const header = ['a', 'b']; + const columns: Cell[][] = [ + numericColumn(300, 42), + categoryColumn(300, [ + ['x', 3], + ['y', 2], + ]), + ]; + const profile = buildProfile(header, columns, 'ref.csv'); + const comparison = compareToProfile(profile, header, columns); + expect(comparison.worst).toBe('stable'); + for (const drift of comparison.columns) expect(drift.psi).toBeCloseTo(0, 6); + }); + + it('agrees with the live two-file comparison on the same pair', () => { + const header = ['a']; + const reference: Cell[][] = [numericColumn(400, 42)]; + const current: Cell[][] = [numericColumn(400, 99, 40)]; + + const live = buildDriftReport(header, reference, header, current); + const viaProfile = compareToProfile( + buildProfile(header, reference, 'ref.csv'), + header, + current, + ); + // Same maths, same bins, same verdict — the profile is a stored reference, + // not a second opinion. + expect(viaProfile.columns[0].psi).toBeCloseTo(live.columns[0].psi, 6); + expect(viaProfile.columns[0].severity).toBe(live.columns[0].severity); + }); + + it('detects a shifted numeric distribution', () => { + const header = ['montant']; + const profile = buildProfile(header, [numericColumn(400, 42)], 'ref.csv'); + const shifted = compareToProfile(profile, header, [numericColumn(400, 99, 60)]); + expect(shifted.columns[0].psi).toBeGreaterThan(0.2); + expect(shifted.worst).toBe('strong'); + }); + + it('puts an unseen category in OTHER instead of inventing a bucket', () => { + const header = ['ville']; + const profile = buildProfile( + header, + [ + categoryColumn(200, [ + ['Québec', 3], + ['Montréal', 2], + ]), + ], + 'ref.csv', + ); + const withNew = compareToProfile(profile, header, [ + categoryColumn(200, [ + ['Québec', 1], + ['Laval', 4], + ]), + ]); + expect(withNew.columns[0].psi).toBeGreaterThan(0); + expect(withNew.columns[0].kind).toBe('categorical'); + }); + + it('names the columns that appeared and disappeared', () => { + const profile = buildProfile( + ['a', 'b'], + [numericColumn(60, 1), numericColumn(60, 2)], + 'ref.csv', + ); + const comparison = compareToProfile( + profile, + ['a', 'c'], + [numericColumn(60, 1), numericColumn(60, 3)], + ); + expect(comparison.missingColumns).toEqual(['b']); + expect(comparison.newColumns).toEqual(['c']); + }); +}); diff --git a/src/features/data/quality/reference.ts b/src/features/data/quality/reference.ts new file mode 100644 index 0000000..1f0df88 --- /dev/null +++ b/src/features/data/quality/reference.ts @@ -0,0 +1,304 @@ +/** + * V40: the reference profile — drift without the original file. + * + * V11 compares two datasets that are both open in the tab. That answers « has + * this month's export drifted from last month's? » only for as long as you + * still hold last month's file. A profile is the same idea as V22's model + * manifest applied to data: a small, exportable, replayable description of what + * the reference LOOKED like, so a new file can be checked against a snapshot + * taken months ago and long since deleted. + * + * What it deliberately is not: a copy of the data. It stores bin EDGES and + * SHARES, never rows — a profile of a payroll file reveals the shape of the + * salary distribution, not anybody's salary. That is what makes it safe to + * commit next to the code it describes, which is the whole point of having it. + */ +import { isMissing, inferColumnType, parseNumber } from '@/features/ml/data/infer'; +import { + psi, + severityOf, + type ColumnDrift, + type DriftSeverity, +} from '@/features/data/quality/drift'; +import type { Cell } from '@/features/ml/data/types'; + +/** Same bin count as V11, so a profile and a live comparison agree. */ +const NUMERIC_BINS = 10; +/** Categories kept by name; the rest collapse into OTHER. */ +const TOP_CATEGORIES = 12; +export const PROFILE_FORMAT = 'labml-data-profile/1'; + +export interface NumericProfile { + kind: 'numeric'; + column: string; + missingRatio: number; + mean: number; + /** Quantile edges of the reference — the bins a new file is scored into. */ + edges: number[]; + /** Reference share per bin, length edges.length + 1. */ + shares: number[]; +} + +export interface CategoricalProfile { + kind: 'categorical'; + column: string; + missingRatio: number; + /** Share per named category, plus OTHER — never the rows themselves. */ + shares: Record; +} + +export type ColumnProfileEntry = NumericProfile | CategoricalProfile; + +export interface DataProfile { + format: typeof PROFILE_FORMAT; + source: string; + createdAt: string; + rowCount: number; + columns: ColumnProfileEntry[]; +} + +const OTHER = '__other__'; + +function numbersOf(values: Cell[]): number[] { + const numbers: number[] = []; + for (const value of values) { + if (isMissing(value)) continue; + const parsed = parseNumber((value as string).trim()); + if (parsed !== null) numbers.push(parsed); + } + return numbers; +} + +function missingRatio(values: Cell[]): number { + if (values.length === 0) return 0; + let missing = 0; + for (const value of values) if (isMissing(value)) missing += 1; + return missing / values.length; +} + +function quantile(sorted: number[], q: number): number { + if (sorted.length === 0) return 0; + const position = (sorted.length - 1) * q; + const low = Math.floor(position); + const high = Math.ceil(position); + return low === high ? sorted[low] : sorted[low] + (position - low) * (sorted[high] - sorted[low]); +} + +function sharesOverEdges(edges: number[], numbers: number[]): number[] { + const counts = new Array(edges.length + 1).fill(0); + for (const value of numbers) { + let bin = 0; + while (bin < edges.length && value > edges[bin]) bin += 1; + counts[bin] += 1; + } + return counts.map((count) => (numbers.length > 0 ? count / numbers.length : 0)); +} + +function categoryShares(values: Cell[]): { shares: Record; names: string[] } { + const counts = new Map(); + let total = 0; + for (const value of values) { + if (isMissing(value)) continue; + const key = (value as string).trim(); + counts.set(key, (counts.get(key) ?? 0) + 1); + total += 1; + } + const top = [...counts.entries()] + .sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])) + .slice(0, TOP_CATEGORIES); + const shares: Record = {}; + let named = 0; + for (const [name, count] of top) { + shares[name] = total > 0 ? count / total : 0; + named += count; + } + shares[OTHER] = total > 0 ? (total - named) / total : 0; + return { shares, names: top.map(([name]) => name) }; +} + +/** Builds the exportable profile of a dataset. */ +export function buildProfile( + header: readonly string[], + columns: Cell[][], + source: string, + now = new Date(), +): DataProfile { + const entries: ColumnProfileEntry[] = []; + for (let i = 0; i < header.length; i++) { + const name = header[i]; + const values = columns[i] ?? []; + const type = inferColumnType(name, values); + if (type === 'numeric') { + const sorted = numbersOf(values).sort((a, b) => a - b); + const edges: number[] = []; + for (let b = 1; b < NUMERIC_BINS; b++) { + const edge = quantile(sorted, b / NUMERIC_BINS); + if (edges.length === 0 || edge > edges[edges.length - 1]) edges.push(edge); + } + entries.push({ + kind: 'numeric', + column: name, + missingRatio: missingRatio(values), + mean: sorted.length > 0 ? sorted.reduce((a, v) => a + v, 0) / sorted.length : 0, + edges, + shares: sharesOverEdges(edges, sorted), + }); + continue; + } + // Text and dates are profiled as categories: a share table is meaningful + // for both, and PSI over shares is the same computation either way. + entries.push({ + kind: 'categorical', + column: name, + missingRatio: missingRatio(values), + shares: categoryShares(values).shares, + }); + } + return { + format: PROFILE_FORMAT, + source, + createdAt: now.toISOString(), + rowCount: columns[0]?.length ?? 0, + columns: entries, + }; +} + +export interface ProfileComparison { + /** Per-column drift against the stored profile, worst first. */ + columns: ColumnDrift[]; + /** Columns the profile describes that the new file does not have. */ + missingColumns: string[]; + /** Columns the new file has that the profile does not describe. */ + newColumns: string[]; + worst: DriftSeverity; +} + +/** + * Scores a new file against a stored profile. Every number is computed the way + * V11 computes it, so a profile comparison and a live two-file comparison give + * the same PSI for the same pair of datasets. + */ +export function compareToProfile( + profile: DataProfile, + header: readonly string[], + columns: Cell[][], +): ProfileComparison { + const index = new Map(header.map((name, i) => [name, i])); + const described = new Set(profile.columns.map((entry) => entry.column)); + const drifts: ColumnDrift[] = []; + const missingColumns: string[] = []; + + for (const entry of profile.columns) { + const at = index.get(entry.column); + if (at === undefined) { + missingColumns.push(entry.column); + continue; + } + const values = columns[at] ?? []; + if (entry.kind === 'numeric') { + const numbers = numbersOf(values); + const value = + numbers.length === 0 ? 0 : psi(entry.shares, sharesOverEdges(entry.edges, numbers)); + drifts.push({ + column: entry.column, + kind: 'numeric', + psi: value, + severity: severityOf(value), + refMissingRatio: entry.missingRatio, + newMissingRatio: missingRatio(values), + refMean: entry.mean, + newMean: numbers.length > 0 ? numbers.reduce((a, v) => a + v, 0) / numbers.length : 0, + }); + continue; + } + // Categorical: score the new file into the profile's own buckets, so a + // category the reference never saw lands in OTHER rather than inventing + // a bucket the reference has no share for. + const names = Object.keys(entry.shares).filter((name) => name !== OTHER); + const nameSet = new Set(names); + const counts = new Map(names.map((name) => [name, 0])); + counts.set(OTHER, 0); + let total = 0; + for (const value of values) { + if (isMissing(value)) continue; + const key = (value as string).trim(); + const bucket = nameSet.has(key) ? key : OTHER; + counts.set(bucket, (counts.get(bucket) ?? 0) + 1); + total += 1; + } + const order = [...names, OTHER]; + const referenceShares = order.map((name) => entry.shares[name] ?? 0); + const newShares = order.map((name) => (total > 0 ? (counts.get(name) ?? 0) / total : 0)); + const value = total === 0 ? 0 : psi(referenceShares, newShares); + drifts.push({ + column: entry.column, + kind: 'categorical', + psi: value, + severity: severityOf(value), + refMissingRatio: entry.missingRatio, + newMissingRatio: missingRatio(values), + }); + } + + const newColumns = header.filter((name) => !described.has(name)); + drifts.sort((a, b) => b.psi - a.psi || a.column.localeCompare(b.column)); + // The worst severity present, using V11's own three levels. Ordered + // explicitly rather than by array position, so adding a level later cannot + // silently make this return the wrong one. + const worst: DriftSeverity = drifts.some((d) => d.severity === 'strong') + ? 'strong' + : drifts.some((d) => d.severity === 'moderate') + ? 'moderate' + : 'stable'; + return { columns: drifts, missingColumns, newColumns, worst }; +} + +/** + * Parses an exported profile. Strict on the format marker and on the shapes it + * needs — a profile that cannot be trusted to describe anything is refused by + * name rather than half-applied. + */ +export function parseProfile(json: string): DataProfile | null { + let parsed: unknown; + try { + parsed = JSON.parse(json); + } catch { + return null; + } + if (typeof parsed !== 'object' || parsed === null) return null; + const record = parsed as Record; + if (record.format !== PROFILE_FORMAT) return null; + if (!Array.isArray(record.columns)) return null; + + const columns: ColumnProfileEntry[] = []; + for (const raw of record.columns) { + if (typeof raw !== 'object' || raw === null) continue; + const entry = raw as Record; + if (typeof entry.column !== 'string') continue; + const missing = typeof entry.missingRatio === 'number' ? entry.missingRatio : 0; + if (entry.kind === 'numeric' && Array.isArray(entry.edges) && Array.isArray(entry.shares)) { + columns.push({ + kind: 'numeric', + column: entry.column, + missingRatio: missing, + mean: typeof entry.mean === 'number' ? entry.mean : 0, + edges: entry.edges.filter((value): value is number => typeof value === 'number'), + shares: entry.shares.filter((value): value is number => typeof value === 'number'), + }); + } else if (entry.kind === 'categorical' && typeof entry.shares === 'object' && entry.shares) { + const shares: Record = {}; + for (const [name, share] of Object.entries(entry.shares as Record)) { + if (typeof share === 'number') shares[name] = share; + } + columns.push({ kind: 'categorical', column: entry.column, missingRatio: missing, shares }); + } + } + if (columns.length === 0) return null; + return { + format: PROFILE_FORMAT, + source: typeof record.source === 'string' ? record.source : '', + createdAt: typeof record.createdAt === 'string' ? record.createdAt : '', + rowCount: typeof record.rowCount === 'number' ? record.rowCount : 0, + columns, + }; +} diff --git a/src/features/data/quality/types.ts b/src/features/data/quality/types.ts index 67a22ef..c0e4253 100644 --- a/src/features/data/quality/types.ts +++ b/src/features/data/quality/types.ts @@ -1,3 +1,6 @@ +import type { ValidityFinding } from '@/features/data/quality/validity'; +import type { ConsistencyFinding } from '@/features/data/quality/consistency'; +import type { ScoreBreakdown } from '@/features/data/quality/checks'; /** Issue report and cleaning recipe types for the Data Studio. */ export interface MissingColumn { @@ -51,8 +54,19 @@ export interface QualityReport { outlierColumns: OutlierColumn[]; outlierCells: number; structural: StructuralIssue[]; + /** + * V40: values that are present, correctly typed and still impossible — + * an age of 200, a date in the future. Reported, never repaired. + */ + validity: ValidityFinding[]; + invalidCells: number; + /** V40: rows where two columns contradict each other. */ + consistency: ConsistencyFinding[]; + inconsistentRows: number; /** 0–100; deterministic function of the ratios above. */ score: number; + /** V40: the score, explained by its parts instead of asserted. */ + breakdown: ScoreBreakdown[]; } /** Forceable column types — they steer the cleaning, not the "before" report. */ diff --git a/src/features/data/quality/v40.test.ts b/src/features/data/quality/v40.test.ts new file mode 100644 index 0000000..c99804c --- /dev/null +++ b/src/features/data/quality/v40.test.ts @@ -0,0 +1,92 @@ +/** + * V40 — the score, explained by its parts. + */ +import { describe, expect, it } from 'vitest'; +import { + TOTAL_WEIGHT, + buildQualityReport, + qualityScore, + scoreBreakdown, +} from '@/features/data/quality/checks'; +import type { Cell } from '@/features/ml/data/types'; + +/** The report without the two fields the score functions compute themselves. */ +function report(header: string[], columns: Cell[][]) { + const built = buildQualityReport(header, columns); + const rest: Record = { ...built }; + delete rest.score; + delete rest.breakdown; + return rest as Parameters[0]; +} + +describe('V40 — the quality score is the sum of its published parts', () => { + it('adds up to exactly the score shown', () => { + const header = ['a', 'b']; + const columns: Cell[][] = [ + ['1', '', '3', '4', '', '6'], + ['x', 'X', 'y', 'y', 'z', ''], + ]; + const partial = report(header, columns); + const parts = scoreBreakdown(partial, 2); + const total = parts.reduce((sum, part) => sum + part.penalty, 0); + expect(qualityScore(partial, 2)).toBe(Math.max(0, Math.round(100 - total))); + }); + + it('names every part, with its weight, its ratio and what it cost', () => { + const partial = report(['a'], [['1', '', '3', '4']]); + const parts = scoreBreakdown(partial); + expect(parts.map((p) => p.part)).toEqual([ + 'missing', + 'duplicates', + 'messy', + 'outliers', + 'structural', + 'validity', + ]); + const missing = parts.find((p) => p.part === 'missing')!; + expect(missing.weight).toBe(35); + expect(missing.count).toBe(1); + expect(missing.ratio).toBeCloseTo(0.25, 10); + // 35 × min(1, 0.25 × 8) = 35, saturated. + expect(missing.penalty).toBe(35); + }); + + it('keeps every pre-V40 weight untouched, so old scores still mean the same', () => { + // Validity brought its own 5 points instead of taking them from a + // neighbour: a file with no validity findings must score exactly what it + // scored before this wave. + const partial = report( + ['a', 'b'], + [ + ['1', '', '3', '4'], + ['x', 'X', 'y', 'y'], + ], + ); + expect(qualityScore(partial, 0)).toBe(qualityScore(partial)); + const weights = scoreBreakdown(partial).map((p) => p.weight); + expect(weights).toEqual([35, 20, 20, 15, 10, 5]); + expect(weights.reduce((a, b) => a + b, 0)).toBe(TOTAL_WEIGHT); + }); + + it('charges for impossible values, and only when there are some', () => { + const partial = report(['a'], [['1', '2', '3', '4', '5', '6', '7', '8']]); + expect(scoreBreakdown(partial, 0).find((p) => p.part === 'validity')!.penalty).toBe(0); + expect(scoreBreakdown(partial, 4).find((p) => p.part === 'validity')!.penalty).toBeGreaterThan( + 0, + ); + expect(qualityScore(partial, 4)).toBeLessThan(qualityScore(partial, 0)); + }); + + it('still gives a clean file 100 and never goes below 0', () => { + const clean = report( + ['a', 'b'], + [ + ['1', '2', '3', '4'], + ['x', 'y', 'x', 'y'], + ], + ); + expect(qualityScore(clean, 0)).toBe(100); + const filthy = report(['a'], [['', '', '', '']]); + expect(qualityScore(filthy, 4)).toBeGreaterThanOrEqual(0); + }); +}); diff --git a/src/features/data/quality/validity.test.ts b/src/features/data/quality/validity.test.ts new file mode 100644 index 0000000..0141dfc --- /dev/null +++ b/src/features/data/quality/validity.test.ts @@ -0,0 +1,157 @@ +/** + * V40 — validity. + * + * As in V38's reader, the tests that matter most are the ones asserting the + * module REFUSES to fire: a rule that flags a good file is worse than no rule, + * because it teaches the user to ignore the panel. + */ +import { describe, expect, it } from 'vitest'; +import { + APPLICABILITY, + checkColumn, + checkValidity, + invalidCellCount, +} from '@/features/data/quality/validity'; +import type { Cell } from '@/features/ml/data/types'; + +/** A fixed "now" so "the future" never depends on when the suite runs. */ +const NOW = Date.UTC(2026, 7, 23); + +describe('V40 — a value can be present, well-typed and still impossible', () => { + it('flags an age outside 0–120, naming the rows and the bound', () => { + const values: Cell[] = ['34', '41', '200', '29', '52', '61', '18', '77', '-3', '45']; + const [found] = checkColumn('age', values, NOW); + expect(found.rule).toBe('ageRange'); + expect(found.count).toBe(2); + expect(found.rows).toEqual([2, 8]); + expect(found.examples).toEqual(['200', '-3']); + expect(found.bound).toEqual({ min: 0, max: 120 }); + }); + + it('flags a date in the future', () => { + const values: Cell[] = [ + '2025-01-04', + '2025-06-11', + '2087-03-02', + '2024-12-31', + '2025-02-02', + '2025-03-03', + ]; + const [found] = checkColumn('date_vente', values, NOW); + expect(found.rule).toBe('futureDate'); + expect(found.rows).toEqual([2]); + }); + + it('flags a percentage above 100 and a negative amount', () => { + const pct = checkColumn('taux_reussite', ['12', '54', '130', '77', '90', '31'], NOW); + expect(pct[0].rule).toBe('percentRange'); + expect(pct[0].examples).toEqual(['130']); + + const amount = checkColumn('prix', ['12.5', '3.25', '-8', '48.9', '7.1', '99'], NOW); + expect(amount[0].rule).toBe('negativeAmount'); + expect(amount[0].examples).toEqual(['-8']); + }); + + it('flags a malformed postcode among well-formed ones', () => { + const values: Cell[] = ['H2X 1Y4', '75008', 'G1R2B5', '13001', 'ABC', 'H3Z 2Y7']; + const [found] = checkColumn('code_postal', values, NOW); + expect(found.rule).toBe('postcodeShape'); + expect(found.examples).toEqual(['ABC']); + }); +}); + +describe('V40 — a rule fires on evidence, never on a column name alone', () => { + it('REFUSES an "age" column that is plainly a duration, not a person', () => { + // Ages in days: every value is above 120, so the rule is not applicable — + // flagging all ten rows would be the rule being wrong, not the data. + const values: Cell[] = [ + '3200', + '5400', + '900', + '12000', + '7800', + '4300', + '6100', + '2200', + '9900', + '15000', + ]; + expect(checkColumn('age_jours', values, NOW)).toEqual([]); + }); + + it('REFUSES a rate written on a 0–1 scale', () => { + const values: Cell[] = ['0.12', '0.54', '0.9', '0.77', '0.31', '0.66']; + expect(checkColumn('taux_conversion', values, NOW)).toEqual([]); + }); + + it('REFUSES a ledger column that is routinely negative', () => { + const values: Cell[] = ['-12', '-3', '45', '-8', '-99', '-40']; + expect(checkColumn('montant_ajustement', values, NOW)).toEqual([]); + }); + + it('REFUSES a column whose name matches but whose values are not that thing', () => { + // "date_label" holds free text, not dates: nothing to check. + const values: Cell[] = ['hiver', 'été', 'printemps', 'automne', 'hiver', 'été']; + expect(checkColumn('date_label', values, NOW)).toEqual([]); + }); + + it('REFUSES every rule on a column whose name matches nothing', () => { + expect(checkColumn('ville', ['Québec', 'Montréal', '200', '-8'], NOW)).toEqual([]); + }); + + it('keeps the applicability floor high enough to mean something', () => { + expect(APPLICABILITY).toBeGreaterThanOrEqual(0.8); + }); + + it('stops applying a rule once too much of the column contradicts it', () => { + // 8 of 10 plausible: exactly at the floor, so the rule still fires. + const atFloor: Cell[] = ['34', '200', '201', '29', '52', '61', '18', '77', '45', '38']; + expect(checkColumn('age', atFloor, NOW)[0].count).toBe(2); + + // 7 of 10 plausible: below the floor. A column a third of which is + // "invalid" is a column the rule has misunderstood, so it says nothing + // rather than flagging thirty rows of a good file. + const belowFloor: Cell[] = ['34', '200', '201', '202', '29', '52', '61', '18', '77', '45']; + expect(checkColumn('age', belowFloor, NOW)).toEqual([]); + }); +}); + +describe('V40 — findings are reported, never repaired', () => { + it('does not touch the values it flags', () => { + const values: Cell[] = ['34', '41', '200', '29', '52', '61']; + const before = [...values]; + checkColumn('age', values, NOW); + // V39's recipe is where data changes; a check that quietly repaired what it + // found would put edits outside the one record of what was done. + expect(values).toEqual(before); + }); + + it('caps the listed rows but never the count', () => { + const values: Cell[] = Array.from({ length: 100 }, (_, i) => (i < 20 ? '999' : '40')); + const [found] = checkColumn('age', values, NOW); + expect(found.count).toBe(20); + expect(found.rows.length).toBe(10); + }); + + it('orders findings worst first and totals the impossible cells', () => { + const header = ['age', 'prix']; + const columns: Cell[][] = [ + ['34', '200', '201', '29', '52', '61', '18', '77', '45', '38'], + ['12.5', '-8', '48.9', '7.1', '99', '3.25', '5', '6', '7', '8'], + ]; + const findings = checkValidity(header, columns, NOW); + expect(findings.map((f) => f.column)).toEqual(['age', 'prix']); + expect(invalidCellCount(findings)).toBe(3); + }); + + it('returns nothing at all for a clean file', () => { + const header = ['age', 'prix', 'date_vente']; + const columns: Cell[][] = [ + ['34', '41', '29', '52', '61', '18'], + ['12.5', '3.25', '48.9', '7.1', '99', '5'], + ['2025-01-04', '2025-06-11', '2024-12-31', '2025-02-02', '2025-03-03', '2025-04-04'], + ]; + expect(checkValidity(header, columns, NOW)).toEqual([]); + expect(invalidCellCount([])).toBe(0); + }); +}); diff --git a/src/features/data/quality/validity.ts b/src/features/data/quality/validity.ts new file mode 100644 index 0000000..cc7fdb1 --- /dev/null +++ b/src/features/data/quality/validity.ts @@ -0,0 +1,203 @@ +/** + * V40: validity — the third question, after completeness and type. + * + * The studio already asks « is the value there? » (missing) and « is it the + * right shape? » (type). Neither catches a value that is present, correctly + * typed, and still impossible: an age of 200, a delivery date in 2087, a + * percentage at 130, a postcode of four characters. Those pass every check + * the report performs today and go straight into a model. + * + * Two rules shape this module, and they are the same two that shaped V38's + * reader: + * + * 1. **A rule fires on evidence, never on a column's name alone.** A column + * called `age` holding 20 000 is a duration in days, not a person; a rule + * that trusted the name would flag every row of a perfectly good file. So + * a name makes a rule *applicable*, and the values decide whether it fires. + * 2. **Flagging is not cleaning.** Every finding names the rule, the column, + * how many rows failed and which ones — and then stops. V39's recipe is + * where data gets changed, deliberately and reproducibly; a check that + * quietly repaired what it found would put edits outside the recipe, which + * is the one record of what was done. + */ +import { isMissing, parseNumber } from '@/features/ml/data/infer'; +import { parseDate } from '@/features/ml/timeseries/series'; +import type { Cell } from '@/features/ml/data/types'; + +/** Rows listed per finding — enough to inspect, not enough to flood the UI. */ +export const MAX_EXAMPLE_ROWS = 10; +/** + * A rule must apply to enough of a column to be about the column rather than + * about a handful of odd cells: below this share of usable values matching the + * rule's own shape, the rule does not consider itself applicable at all. + */ +export const APPLICABILITY = 0.8; + +export type ValidityRule = + /** A human age outside 0–120. */ + | 'ageRange' + /** A date after today — a birth date or a sale cannot be in the future. */ + | 'futureDate' + /** A percentage outside 0–100. */ + | 'percentRange' + /** A quantity, price or amount below zero. */ + | 'negativeAmount' + /** A Canadian or French postcode that does not have the right shape. */ + | 'postcodeShape'; + +export interface ValidityFinding { + rule: ValidityRule; + column: string; + /** Rows that failed, 0-based, capped at MAX_EXAMPLE_ROWS. */ + rows: number[]; + /** How many rows failed in total — `rows` may be a prefix of this. */ + count: number; + /** Values that failed, aligned with `rows`, for display. */ + examples: string[]; + /** The bound the rule enforced, when it has one — shown in the message. */ + bound?: { min?: number; max?: number }; +} + +/** Column names a rule is willing to look at, matched loosely and accent-free. */ +function nameMatches(column: string, needles: readonly string[]): boolean { + const normalized = column.toLowerCase().normalize('NFD').replace(/[̀-ͯ]/g, ''); + return needles.some((needle) => normalized.includes(needle)); +} + +function usableValues(values: Cell[]): { row: number; value: string }[] { + const out: { row: number; value: string }[] = []; + for (let row = 0; row < values.length; row++) { + const value = values[row]; + if (isMissing(value)) continue; + out.push({ row, value: (value as string).trim() }); + } + return out; +} + +function finding( + rule: ValidityRule, + column: string, + failures: { row: number; value: string }[], + bound?: { min?: number; max?: number }, +): ValidityFinding | null { + if (failures.length === 0) return null; + const capped = failures.slice(0, MAX_EXAMPLE_ROWS); + return { + rule, + column, + rows: capped.map((f) => f.row), + examples: capped.map((f) => f.value), + count: failures.length, + ...(bound !== undefined && { bound }), + }; +} + +const AGE_NAMES = ['age', 'âge'] as const; +const PERCENT_NAMES = ['percent', 'pourcent', 'pct', 'taux', 'rate'] as const; +const AMOUNT_NAMES = [ + 'price', + 'prix', + 'montant', + 'amount', + 'total', + 'quantity', + 'quantite', +] as const; +const POSTCODE_NAMES = ['postcode', 'postal', 'zip', 'cp'] as const; +const DATE_NAMES = ['date', 'jour', 'day'] as const; + +/** Canadian `H2X 1Y4` or French `75008`. */ +const CANADIAN = /^[A-Za-z]\d[A-Za-z][ -]?\d[A-Za-z]\d$/; +const FRENCH_POSTCODE = /^\d{5}$/; + +/** + * Every validity finding for one column. `now` is injected so a test can pin + * "the future" to a fixed instant rather than depending on the clock. + */ +export function checkColumn(column: string, values: Cell[], now = Date.now()): ValidityFinding[] { + const usable = usableValues(values); + if (usable.length === 0) return []; + const found: ValidityFinding[] = []; + + const numbers = usable + .map((entry) => ({ ...entry, parsed: parseNumber(entry.value) })) + .filter((entry): entry is typeof entry & { parsed: number } => entry.parsed !== null); + const numericEnough = numbers.length / usable.length >= APPLICABILITY; + + if (numericEnough && nameMatches(column, AGE_NAMES)) { + // A column of ages that is mostly above 120 is not ages — it is a duration + // in months or days, and flagging every row of it would be the rule being + // wrong, not the data. + const plausible = numbers.filter((n) => n.parsed >= 0 && n.parsed <= 120).length; + if (plausible / numbers.length >= APPLICABILITY) { + const bad = numbers.filter((n) => n.parsed < 0 || n.parsed > 120); + const result = finding('ageRange', column, bad, { min: 0, max: 120 }); + if (result) found.push(result); + } + } + + if (numericEnough && nameMatches(column, PERCENT_NAMES)) { + // Same guard: a rate written as 0–1 is not a percentage out of range. + const plausible = numbers.filter((n) => n.parsed >= 0 && n.parsed <= 100).length; + if (plausible / numbers.length >= APPLICABILITY) { + const bad = numbers.filter((n) => n.parsed < 0 || n.parsed > 100); + const result = finding('percentRange', column, bad, { min: 0, max: 100 }); + if (result) found.push(result); + } + } + + if (numericEnough && nameMatches(column, AMOUNT_NAMES)) { + // A ledger column that is routinely negative is a balance, not an error. + const nonNegative = numbers.filter((n) => n.parsed >= 0).length; + if (nonNegative / numbers.length >= APPLICABILITY) { + const bad = numbers.filter((n) => n.parsed < 0); + const result = finding('negativeAmount', column, bad, { min: 0 }); + if (result) found.push(result); + } + } + + if (nameMatches(column, DATE_NAMES)) { + const dates = usable + .map((entry) => ({ ...entry, at: parseDate(entry.value) })) + .filter((entry): entry is typeof entry & { at: number } => entry.at !== null); + if (dates.length / usable.length >= APPLICABILITY) { + const bad = dates.filter((d) => d.at > now); + const result = finding('futureDate', column, bad); + if (result) found.push(result); + } + } + + if (nameMatches(column, POSTCODE_NAMES)) { + const shaped = usable.filter( + (entry) => CANADIAN.test(entry.value) || FRENCH_POSTCODE.test(entry.value), + ); + // Only a column that is mostly postcodes gets to have malformed ones. + if (shaped.length / usable.length >= APPLICABILITY) { + const bad = usable.filter( + (entry) => !CANADIAN.test(entry.value) && !FRENCH_POSTCODE.test(entry.value), + ); + const result = finding('postcodeShape', column, bad); + if (result) found.push(result); + } + } + + return found; +} + +/** Every finding across the dataset, worst first. */ +export function checkValidity( + header: readonly string[], + columns: Cell[][], + now = Date.now(), +): ValidityFinding[] { + const found: ValidityFinding[] = []; + for (let i = 0; i < header.length; i++) { + found.push(...checkColumn(header[i], columns[i] ?? [], now)); + } + return found.sort((a, b) => b.count - a.count || a.column.localeCompare(b.column)); +} + +/** Total impossible cells — what the quality score charges for. */ +export function invalidCellCount(findings: readonly ValidityFinding[]): number { + return findings.reduce((total, finding) => total + finding.count, 0); +} diff --git a/src/features/data/sql/engine.ts b/src/features/data/sql/engine.ts index 88abb59..068ebfe 100644 --- a/src/features/data/sql/engine.ts +++ b/src/features/data/sql/engine.ts @@ -26,6 +26,13 @@ export interface SqlEngine { /** Makes a file readable by SQL under `name`; the bytes stay in the tab. */ register(name: string, bytes: Uint8Array): Promise; run(sql: string, cap: number): Promise; + /** + * V40: runs a query and returns the result as Parquet bytes. Nearly free + * here — DuckDB is already loaded and the file already registered, so this + * is one `COPY … TO` and a read of the buffer it wrote. The bytes never + * leave the tab: the caller turns them into a download. + */ + toParquet(sql: string): Promise; close(): Promise; } @@ -66,6 +73,16 @@ export async function openEngine(): Promise { const records = result.toArray().map((row) => row.toJSON()); return toSqlTable(columns, records, cap); }, + async toParquet(sql) { + // A per-call name so two exports can never collide on the virtual FS. + const name = `export-${Date.now()}.parquet`; + await connection.query(`COPY (${sql}) TO '${name}' (FORMAT PARQUET, COMPRESSION ZSTD)`); + const bytes = await db.copyFileToBuffer(name); + // Registered files live in the Wasm heap: dropping it keeps a session of + // repeated exports from growing without bound. + await db.dropFile(name); + return bytes; + }, async close() { await connection.close(); await db.terminate(); diff --git a/src/locales/en.json b/src/locales/en.json index eae2e70..3eafd3c 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -702,6 +702,20 @@ "good": "Good", "fair": "Needs attention", "poor": "Poor" + }, + "breakdownTitle": "How this score was computed", + "breakdownHint": "Each part carries its own weight and the points it actually cost. The weights sum to 105, not 100: validity brought its own 5 points in V40 rather than taking them from an existing part, so a file with no impossible values scores exactly what it scored before.", + "part": "Part", + "found": "Found", + "weight": "Max", + "cost": "Cost", + "parts": { + "missing": "Missing cells", + "duplicates": "Duplicate rows", + "messy": "Messy spellings", + "outliers": "Outlier values", + "structural": "Useless columns", + "validity": "Impossible values" } }, "issues": { @@ -734,6 +748,27 @@ "constant": "constant", "nearEmpty": "near-empty", "id": "identifier" + }, + "validity": { + "title_one": "{{count}} impossible value", + "title_other": "{{count}} impossible values", + "body": "Present, correctly typed, and still impossible. Flagged, never repaired — the recipe is where data changes.", + "rules": { + "ageRange": "age outside 0–120", + "futureDate": "date in the future", + "percentRange": "percentage outside 0–100", + "negativeAmount": "negative amount", + "postcodeShape": "malformed postcode" + } + }, + "consistency": { + "title_one": "{{count}} contradictory row", + "title_other": "{{count}} contradictory rows", + "body": "Every cell is fine; the row is not. Two columns disagree with each other.", + "rules": { + "dateOrder": "{{columns}}: the end comes before the start", + "productMismatch": "{{columns}}: the total is not quantity × price" + } } }, "recipe": { diff --git a/src/locales/fr.json b/src/locales/fr.json index 8bebb53..c93ab6b 100644 --- a/src/locales/fr.json +++ b/src/locales/fr.json @@ -702,6 +702,20 @@ "good": "Bon", "fair": "À surveiller", "poor": "Faible" + }, + "breakdownTitle": "Comment ce score a été calculé", + "breakdownHint": "Chaque partie porte son poids et les points qu'elle a réellement coûtés. Les poids totalisent 105 et non 100 : la validité a apporté ses 5 points en V40 au lieu de les prendre à une partie existante, donc un fichier sans valeur impossible obtient exactement le score qu'il obtenait avant.", + "part": "Partie", + "found": "Constaté", + "weight": "Max", + "cost": "Coût", + "parts": { + "missing": "Cellules manquantes", + "duplicates": "Lignes en double", + "messy": "Orthographes désordonnées", + "outliers": "Valeurs aberrantes", + "structural": "Colonnes inutiles", + "validity": "Valeurs impossibles" } }, "issues": { @@ -734,6 +748,27 @@ "constant": "constante", "nearEmpty": "quasi vide", "id": "identifiant" + }, + "validity": { + "title_one": "{{count}} valeur impossible", + "title_other": "{{count}} valeurs impossibles", + "body": "Présente, bien typée, et pourtant impossible. Signalée, jamais réparée — c'est la recette qui modifie les données.", + "rules": { + "ageRange": "âge hors 0–120", + "futureDate": "date dans le futur", + "percentRange": "pourcentage hors 0–100", + "negativeAmount": "montant négatif", + "postcodeShape": "code postal malformé" + } + }, + "consistency": { + "title_one": "{{count}} ligne contradictoire", + "title_other": "{{count}} lignes contradictoires", + "body": "Chaque cellule est correcte ; la ligne ne l'est pas. Deux colonnes se contredisent.", + "rules": { + "dateOrder": "{{columns}} : la fin précède le début", + "productMismatch": "{{columns}} : le total n'est pas quantité × prix" + } } }, "recipe": {