V40 — Data Studio : validité, dérive, et un diff auditable

PLAN.md

@@ -464,7 +464,7 @@ So parallel training alone was worth **1.06×** — real work, drowned by a bott
 
 The baseline scores 0.594, so the headroom above it collapses from 0.406 to 0.225: **45% of the achievable gain, lost in silence**. `parseNumber` ends in `Number(cleaned)`, `Number('12,5')` is `NaN`, the column falls through, and V24's TF-IDF cheerfully tokenises digits into a hundred word features. Nothing warned, nothing refused — the pipeline just produced a worse model. **Three of this row's own predictions were wrong, and the measurement corrected them**: the mis-typed column becomes `text` on high cardinality but `categorical` (one-hot) on low, not `text` alone; a windows-1252 file does not display `Québec` — that is the reverse case (UTF-8 read as cp1252) — it yields U+FFFD replacement characters; and day-first dates were already handled by V8, `31/12/2025` parsing correctly all along, with only the dash form `31-12-2025` returning null. **The fix.** One reader, shared by the ML Lab and the Data Studio, that decides by evidence rather than by guessing the user's locale: the browser's language says nothing about the file someone dragged in. Encoding is settled by **trying and failing** — UTF-8 in `fatal` mode throws on cp1252 accents, so the fallback is a certainty, not a preference; the delimiter is the candidate that splits every sampled line into the same number of columns, quotes respected; and the decimal separator is decided **per column**, never per file. A column is rewritten only when at least 90% of its values are numbers in that form AND it carries a comma AND it would otherwise not be numeric at all — so a text column containing « vis, tête plate » comes out untouched, and a column of bare integers is left alone because rewriting it would be a change with no cause. Values that do not match the pattern are never rewritten: a stray « n/d » stays « n/d » rather than becoming a plausible-looking number. Detection reads only the **head** of the file, so V25's streaming abort survives intact and a 2 GB file still stops at the cell budget. The reading is then **announced with its evidence** — « virgule décimale détectée et convertie dans 2 colonnes : surface (400/400), prix (400/400) » — above a five-row preview, and only when the reading was not the plain default: an ordinary UTF-8 comma file gets no card, no confirmation step, no friction. Batch scoring, drift and join files go through the same reader, because comparing a French export against a normalised dataset would otherwise report drift that is nothing but a decimal separator. **What this wave deliberately does not do**: guess a locale from `navigator.language` (the file has no relationship to the browser's language), rewrite a column on a bare majority (below the 90% floor the evidence is not evidence), or touch values individually inside a column the evidence does not cover. 446 unit tests, 73 e2e — of the 21 new reader tests, five assert that it **refuses** to act. | Owner request (22/08/2026): what to improve in /data. The audit found a defect first, and the wave began by reproducing it end to end: a French-locale CSV — the single most likely file this owner's users will open — silently loses every numeric column. The repair is exact rather than approximate: the French file now trains to the same types, the same feature count and the same score, to ten decimal places, as the file that never had the problem. |
 | **V39 — delivered** | **Data Studio: a recipe that works column by column.** `RecipeOptions` applied `missing` and `clipOutliers` to the **whole file** — one strategy for every column, however different they are. A median makes sense for an age and none at all for a postcode. The recipe is now an ordered list of **per-column steps**, with the file-wide settings demoted to **defaults a column may override**: a column with no entry behaves exactly as it did before V39, which is what keeps every previously exported recipe valid and replayable. The missing strategies that were absent are there: **median, mean, most-frequent, a constant used verbatim, and a « MANQUANT » category** for categorical columns — absence becoming a level of its own rather than a guess. Clipping became a per-column decision on the same terms, so a column can opt in or out of the global flag. **The rule the wave exists to enforce: imputing without marking destroys information.** A blank field is rarely blank at random, and the fact of the blank is frequently predictive on its own, so every column may add a `<column>_absent` indicator — and the ordering is the whole point: **every indicator is written before any blank is filled**, so it records where the blanks really were, never where some other column's rule left them. Three consequences fall out of that and are tested: a column with nothing missing gets no indicator (an all-zero column is noise, not information); rows dropped by one column's rule are gathered and removed **once**, so indicators stay aligned instead of being shifted by a sibling column; and when a strategy cannot be honoured — a median over a column holding no parseable number, a constant with no value typed — the blanks are **left blank** rather than filled with something invented. Because the indicator is optional, the studio **announces the columns it filled without marking**, by name, instead of quietly producing a tidier table. **What this wave deliberately does not do**: make the indicator mandatory (it would silently add columns to every existing recipe, and imposing is not the same as announcing), impute from a model (opaque, and it fabricates values that look plausible — already refused for V40), or reorder the recipe's fixed stages, which is what keeps toggling an option from compounding with the last one. 461 unit tests, 75 e2e. | A single global strategy is the kind of default that looks tidy and quietly makes the data worse; per-column steps cost little to build because the recipe was already an object, not a pile of checkboxes. The build confirmed it: the engine change is contained in one stage of `applyRecipe`, and the 17 pre-existing recipe tests passed untouched. |
-| V40 | **Data Studio: validity, drift, and an auditable diff.** Quality is measured today as completeness and consistency of type; what is missing is **validity** — a value can be present, well-typed and still impossible. Named rules, each stated in plain language: an age of 200, a date in the future, a percentage at 130, a malformed postcode. Then **cross-column consistency** (`date_fin < date_debut`, `total ≠ quantité × prix`), for which **V29's DuckDB is already the engine** — the rules are SQL, and they run on the file that is already registered. Then three things that make the studio auditable rather than merely helpful: a **replayable reference profile** so a second file can be checked for drift against the first (the same idea as the V22 model manifest), a **before/after diff of the rows a recipe modified** — which rows, which columns, which values, not just a count — and a **breakdown of the quality score** so the number is explained by its parts instead of being asserted. Ends with **Parquet export**, nearly free now that DuckDB is loaded (`COPY … TO 'x.parquet'`). **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). | Comes last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would flag the parser, not the data. |
+| **V40 — delivered** | **Data Studio: validity, drift, and an auditable diff.** Quality was measured as completeness and consistency of type; what was missing is **validity** — a value can be present, correctly typed and still impossible. Five named rules now say so in plain language: an age outside 0–120, a date in the future, a percentage outside 0–100, a negative amount, a malformed postcode. Then **cross-column consistency**: an end date before its start, a total that is not quantity × price — every cell fine, the ROW impossible. **The plan's choice of engine was wrong, and building it showed why.** It proposed running the consistency rules through V29's DuckDB, since the file is already registered there. But DuckDB is an announced, opt-in 18–22 MB download: routing these checks through it would have made a universally applicable check conditional on a large download most users will decline, leaving the panel empty for them. Comparing two columns of a table already in memory is a loop, so it is a loop, and every user gets it. DuckDB keeps the job it is genuinely needed for — arbitrary SQL, and the new **Parquet export** (`COPY … TO`, one call, bytes straight to a download). Both families of rules obey the two laws V38's reader established: **they fire on evidence, never on a column's name alone** — a column called `age` holding 20 000 is a duration in days, so the rule checks that most of the column is plausible before it flags the rest, and stays silent otherwise — and **they report without ever repairing**, because V39's recipe is the one record of what was done to the data. Three things then make the studio auditable rather than merely helpful. A **before/after diff** naming which rows, which columns and which values changed: the hard part is that a recipe drops rows and adds columns, so `applyRecipe` now returns which SOURCE row each surviving row came from — without it the diff would pair row 7 with a different row 7 and report a screen full of changes that never happened. A **replayable reference profile**, the V22 manifest idea applied to data: bin edges and shares, never rows, so a profile of a payroll file describes the shape of the salary distribution and nobody's salary — which is what makes it safe to commit beside the code, and it scores a new file to the same PSI, to six decimal places, as V11's live two-file comparison. And a **score broken into its parts**, each with its weight and what it actually cost. The weights now sum to **105, not 100**, deliberately: validity brought its own 5 points rather than taking them from an existing part, because redistributing would have quietly changed what every previously published score meant. **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). 502 unit tests, 78 e2e. | Came last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would have flagged the parser, not the data. Closes the Data Studio group. Of the 36 new unit tests, eleven assert that a rule REFUSES to fire — a rule that flags a good file is worse than no rule, because it teaches the reader to ignore the panel. |
 
 **Ordering**: V38 came before V39 and V40, and for the same reason V35 came first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35, V36 and V37 are delivered. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 and V31 both start with a bench, because neither « a bigger model » nor « it still makes mistakes » is a measurable statement today; no wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped —
 V25 (delivered) chose announced sampling and a named memory guard over the

README.md

@@ -68,6 +68,16 @@ The project follows three non-negotiable principles:
   column can add a `<column>_absent` **missing indicator**, written before anything is
   filled; columns filled _without_ one are named out loud, because a blank field is
   rarely blank at random and filling it silently erases that.
+- **Validity rules**: a value can be present, correctly typed and still impossible — an
+  age of 200, a date in the future, a percentage at 130, a malformed postcode. Plus
+  **cross-column consistency**: an end date before its start, a total that is not
+  quantity × price. Every rule fires on evidence rather than on a column's name, and
+  reports without ever repairing — the recipe is where data changes.
+- **The quality score, broken into its parts**: each with its weight, what it found and
+  what it cost, instead of a number asserted without explanation.
+- **An auditable before/after diff** of what the recipe did — which rows, which columns,
+  which values — and a **replayable reference profile** (bin edges and shares, never
+  rows) so a new file can be checked for drift against a snapshot you no longer hold.
 - **Left-join a second file** on a shared key: match rate, duplicates and orphans are
   named, never silent; the joined result becomes the working dataset.
 - **Drift check**: compare a new batch against the reference — schema diff, PSI per
@@ -124,7 +134,7 @@ The project follows three non-negotiable principles:
 - **Performance.** Every section serves a prerendered static shell (hero paints before
   JavaScript); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy
   dependencies (Dexie, SheetJS, ONNX Runtime) load lazily.
-- **Quality bar.** 461 unit tests, 75 Playwright end-to-end tests (including offline PWA,
+- **Quality bar.** 502 unit tests, 78 Playwright end-to-end tests (including offline PWA,
   fake-webcam and axe-core WCAG A/AA accessibility checks), strict TypeScript, ESLint,
   Prettier, and Lighthouse budgets — all enforced in CI.
 

e2e/validity.spec.ts

@@ -0,0 +1,71 @@
+import { expect, test } from '@playwright/test';
+
+test.use({ locale: 'en-US' });
+test.setTimeout(120_000);
+
+/**
+ * V40 — validity, consistency, and a score that explains itself.
+ */
+function impossibleCsv(): Buffer {
+  const lines = ['nom,age,taux_reussite,date_debut,date_fin,quantite,prix_unitaire,total'];
+  for (let i = 0; i < 60; i++) {
+    // Two impossible ages, one impossible percentage, one reversed date pair,
+    // one total that is not quantity × price — everything else is plausible.
+    const age = i === 3 ? '200' : i === 7 ? '-4' : String(20 + (i % 50));
+    const rate = i === 11 ? '130' : String(i % 100);
+    const start = `2025-0${(i % 9) + 1}-0${(i % 8) + 1}`;
+    const end = i === 5 ? '2024-01-01' : `2025-0${(i % 9) + 1}-0${(i % 8) + 2}`;
+    const qty = String(1 + (i % 5));
+    const unit = String(2 + (i % 4));
+    const total = i === 9 ? '999' : String((1 + (i % 5)) * (2 + (i % 4)));
+    lines.push(`ligne${i},${age},${rate},${start},${end},${qty},${unit},${total}`);
+  }
+  return Buffer.from(lines.join('\n'), 'utf-8');
+}
+
+test('impossible values and contradictory rows are named, with their rules', async ({ page }) => {
+  await page.goto('/data');
+  await page.getByTestId('data-file-input').setInputFiles({
+    name: 'commandes.csv',
+    mimeType: 'text/csv',
+    buffer: impossibleCsv(),
+  });
+  await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 });
+
+  // Values that are present and correctly typed, and still impossible.
+  const validity = page.getByTestId('issue-validity');
+  await expect(validity).toBeVisible();
+  await expect(validity).toContainText('age');
+  await expect(validity).toContainText('age outside 0–120');
+  await expect(validity).toContainText('percentage outside 0–100');
+
+  // A row where two columns contradict each other.
+  const consistency = page.getByTestId('issue-consistency');
+  await expect(consistency).toBeVisible();
+  await expect(consistency).toContainText('the end comes before the start');
+  await expect(consistency).toContainText('the total is not quantity × price');
+});
+
+test('the quality score is explained by its parts, not asserted', async ({ page }) => {
+  await page.goto('/data');
+  await page.getByRole('button', { name: /cafe-sales\.csv/ }).click();
+  await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 });
+
+  await page.getByText('How this score was computed', { exact: true }).click();
+  const breakdown = page.getByTestId('score-breakdown');
+  await expect(breakdown).toContainText('Missing cells');
+  await expect(breakdown).toContainText('Duplicate rows');
+  await expect(breakdown).toContainText('Impossible values');
+  // The weights are shown, including the deliberate 105 total.
+  await expect(breakdown).toContainText('105, not 100');
+});
+
+test('a clean file raises no validity or consistency card at all', async ({ page }) => {
+  await page.goto('/data');
+  // titanic has missing values and messy spellings, but every age is a real
+  // age and no two columns contradict each other.
+  await page.getByRole('button', { name: /titanic\.csv/ }).click();
+  await expect(page.getByTestId('recipe-result')).toBeVisible({ timeout: 60_000 });
+  await expect(page.getByTestId('issue-validity')).toHaveCount(0);
+  await expect(page.getByTestId('issue-consistency')).toHaveCount(0);
+});

src/features/data/components/DataDropZone.tsx

@@ -46,6 +46,7 @@ export function DataDropZone() {
         <span className="max-w-md text-sm text-muted">{t('data.drop.hint')}</span>
       </button>
       <input
+        data-testid="data-file-input"
         ref={inputRef}
         type="file"
         accept=".csv,.tsv,.txt,.xlsx,.xls,text/csv"