From 978b53f8bc4772be3a678088b5b7dcc895446b5d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 17:13:34 +0000 Subject: [PATCH] =?UTF-8?q?docs(v30):=20la=20question=20du=20mod=C3=A8le?= =?UTF-8?q?=20plus=20gros,=20mesur=C3=A9e=20=E2=80=94=201,43=20Go=20ach?= =?UTF-8?q?=C3=A8te=20cinq=20r=C3=A9ponses=20fausses=20de=20plus?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Le banc prend le modèle en paramètre, donc « un modèle plus gros lirait-il mieux ? » a été posée à l'instrument plutôt qu'à l'intuition. Mêmes 55 questions, même runtime CPU, même prompt, même décodage contraint : Qwen3-0.6B-DQ 355 Mo — appli 42 justes / 7 fausses / 6 refus Qwen3-1.7B 1,43 Go — appli 40 justes / 12 fausses / 3 refus Quatre fois la bande passante, cinq réponses fausses de plus. Le gros modèle lit mieux les questions difficiles pour un petit — il répond aux quatre comparaisons de groupes que le 0.6B refusait — et moins bien les faciles : il répond à « count the passengers older than 60 » par aggregate op=count column=age, qui compte les lignes AYANT un âge plutôt que les lignes. Deux limites écrites plutôt qu'enfouies : le prompt a été réglé contre le 0.6B, donc ce chiffre décrit la paire et non le modèle ; et c'est un seul barreau — 764 Mo et 1,09 Go n'ont pas été essayés, leurs licences étant plus strictes que l'Apache-2.0 du reste de LabML. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01UKw6oNC8iZ9Kn7q6x4qom4 --- PLAN.md | 61 ++++++++++++++++++++++++++++++++++++++++--------------- README.md | 13 ++++++++++++ 2 files changed, 58 insertions(+), 16 deletions(-) diff --git a/PLAN.md b/PLAN.md index 9c2c7f9..fe73ddc 100644 --- a/PLAN.md +++ b/PLAN.md @@ -428,25 +428,25 @@ production on 21/08/2026. Cap 6's guiding thread: the lab meets the real world — real photos, real text, real file sizes, and the question every data budget asks. -| Wave | Content | Why | -| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **V23 — delivered** | **Vision 2**: SqueezeNet (2012) retired for three self-hosted ONNX models — **EfficientNet-Lite4 int8** classification (1,000 ImageNet classes, 77.6% top-1), **YOLOX-Nano** object detection (80 COCO classes; the stronger-but-AGPL YOLOs were ruled out, Apache-2.0 kept) and **UltraFace RFB-320** face detection — boxes drawn on the image, FR/EN class names, plain-language counts ("1 person · 1 face"). Box decoding (grids/strides, exp, IoU, per-class NMS) is hand-written and unit-tested; letterboxed inputs (aspect squashing measurably mislocated face boxes); named thresholds (objects 35%; faces 0.9 — real faces score ≥ 0.95, measured false positives top out at 0.85); ~19 MB total, runtime-cached, offline after first use; verified on real photos (NASA portrait → 1 person + 1 face; German Shepherd → dog + breed at 99.9%) | Owner request (21/08/2026): portraits have no ImageNet class, so the old model answered off-target — and the detector must recognize a whole range of things, not just faces | -| **V24 — delivered** | **Text columns**: free text stops being skipped and enters the pipeline as a hand-written **TF-IDF** block — accent-folding bilingual tokenizer, merged FR/EN stop words, vocabulary capped at 256 terms ranked by document frequency (ties alphabetical, terms seen in a single training document dropped), smoothed IDF, L2-normalized vectors, fitted on the training split only. Features are named `column:word`, so importance, Shapley and the report speak in words; `encodedBlocks` now measures a text block by its real width (counting it as one column silently shifted every block after it). Explanations gained **signed word effects** by occlusion — erase one word from the reviews containing it and average the shift of the answer — because permutation is blind to a redundant vocabulary, and multiclass is refused rather than faked. Export bumped to **format v3** (v2 files still import). Demo `reviews.csv`: 240 bilingual orders where the text carries the signal — baseline 0.52 → 0.92, `review` top of the importance chart, `fast`/`excellent`/`avance` pushing up, `refund`/`cheap` pushing down | Real CSVs have text columns (comments, descriptions) — the lab used to drop them on the floor | +| Wave | Content | Why | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **V23 — delivered** | **Vision 2**: SqueezeNet (2012) retired for three self-hosted ONNX models — **EfficientNet-Lite4 int8** classification (1,000 ImageNet classes, 77.6% top-1), **YOLOX-Nano** object detection (80 COCO classes; the stronger-but-AGPL YOLOs were ruled out, Apache-2.0 kept) and **UltraFace RFB-320** face detection — boxes drawn on the image, FR/EN class names, plain-language counts ("1 person · 1 face"). Box decoding (grids/strides, exp, IoU, per-class NMS) is hand-written and unit-tested; letterboxed inputs (aspect squashing measurably mislocated face boxes); named thresholds (objects 35%; faces 0.9 — real faces score ≥ 0.95, measured false positives top out at 0.85); ~19 MB total, runtime-cached, offline after first use; verified on real photos (NASA portrait → 1 person + 1 face; German Shepherd → dog + breed at 99.9%) | Owner request (21/08/2026): portraits have no ImageNet class, so the old model answered off-target — and the detector must recognize a whole range of things, not just faces | +| **V24 — delivered** | **Text columns**: free text stops being skipped and enters the pipeline as a hand-written **TF-IDF** block — accent-folding bilingual tokenizer, merged FR/EN stop words, vocabulary capped at 256 terms ranked by document frequency (ties alphabetical, terms seen in a single training document dropped), smoothed IDF, L2-normalized vectors, fitted on the training split only. Features are named `column:word`, so importance, Shapley and the report speak in words; `encodedBlocks` now measures a text block by its real width (counting it as one column silently shifted every block after it). Explanations gained **signed word effects** by occlusion — erase one word from the reviews containing it and average the shift of the answer — because permutation is blind to a redundant vocabulary, and multiclass is refused rather than faked. Export bumped to **format v3** (v2 files still import). Demo `reviews.csv`: 240 bilingual orders where the text carries the signal — baseline 0.52 → 0.92, `review` top of the importance chart, `fast`/`excellent`/`avance` pushing up, `refund`/`cheap` pushing down | Real CSVs have text columns (comments, descriptions) — the lab used to drop them on the floor | | **V25 — delivered** | **Scale**: the lab now takes 100k–1M-row files without dying, on a measure-first design. Before: a stack overflow killed every run past ~65k rows (`push(...spread)` in the split), and the slow families made big runs unusable anyway (random forest alone: **535 s at 5 000 rows**). Measured first, then fixed: (1) the split rewritten with plain loops; (2) the planned typed-array pipeline rewrite was **descoped on measurement** — the pipeline was never the bottleneck (parse 2.6 s + profile 4.9 s + fit/transform 3.1 s at 1M rows); (3) **announced seeded sampling, never silent** — a global cap of 100 000 usable rows (seeded, stratified, `summary.sampledFrom`) plus measured per-family training caps (forest 1 000 · tree 2 000 · logistic/linear/MLP 20 000 · k-NN 5 000 · GBDT 50 000) drawn as nested prefixes of one seeded order, every capped model scored on the same full test set, and every sample printed on the leaderboard, in the tuning panel and in the HTML report; k-NN's old **silent** internal 5 000-row subsample was folded into the announced mechanism; (4) a **named memory guard**: parsing streams and refuses past 20M cells (rows × columns) with the numbers spelled out, instead of letting the tab die. After, measured: a 1M-row file trains the whole 8-model zoo in **~130 s** (and a 120k-row file in the same ~126 s — cost is flat past the cap), gbdt reaches 0.991 accuracy on the 50 000-row announced sample. Every demo dataset sits under every cap: existing behavior unchanged. 283 unit tests (announced-sampling determinism, stratified nesting, cap recording), 54 e2e (a generated 120k-row CSV trains with the announcement asserted; a 21M-cell file is refused by name) | | **V26 — delivered** | **Learning curves**: the lab answers the classic budget question — "would more data help this model, or is it time to work on features?" — with one new chart. On demand (like tuning), one model is retrained on growing seeded fractions of the train split: the SAME nested prefixes V25's announced caps draw from (same seed, same order), so for a capped family the last point is exactly the leaderboard model's diet and the verdict says out loud whether the announced cap costs accuracy. A geometric ladder of up to 6 sizes (each at least 16 rows, refused entirely when only one rung fits — one point is not a curve); at every size the pipeline is REFITTED on that prefix only (imputation, encoding, IDF, scaling all see just those rows — the strict no-leakage reading of a learning curve) and the model is scored on the same full held-out test set with a V20 bootstrap 95% band. The verdict is the V20 paired bootstrap applied to the last size step: a decisive gain reads "still climbing — more data would probably help", anything else "flattened — work on features or the model", each with the capped variant ("the cap costs accuracy" / "the cap costs nothing here"). The chart (log-spaced sizes, CI band, one dot per announced size) ships with its numbers table, joins the run artifacts (history, HTML report, share links) and refuses the baseline by name — flat by definition. Measured on Titanic/gbdt: 45 → 713 rows traces 0.753 → 0.820 with the plateau verdict at the last step. 293 unit tests, 55 e2e | | **V27 — delivered** | **Local chat, upgraded**: a real language model — **Qwen3-0.6B-DQ, 355 MB, Apache-2.0** — running entirely in the browser, offered beside the V6 deterministic interpreter, which stays the DEFAULT and the fallback. The model never computes: it translates a question into a V6 query, and every number still comes from the deterministic engine. Its output is checked against the closed Intent grammar — invented columns, unknown operators, absurd k, a correlation of a column with itself are all REFUSED, and a refusal falls back to the keyword parser, with a badge under each answer naming which engine produced the query. Three constraints shaped the build, all discovered by measurement: (1) Cloudflare Pages refuses assets over 25 MiB, so `scripts/prepare-llm.mjs` fetches the weights at DEPLOY time (never committed — 355 MB in git would slow every clone) and splits them into **15 parts of ≤ 24 MiB**, glued back in the browser through transformers.js's `customCache` hook, with per-part size checks and named refusals (`llm-part-missing`, `llm-part-size`, `llm-short`); (2) the strict CSP forbids the library's CDN default, so its pinned ONNX Runtime build is self-hosted under `/ort-llm/` — the jsep (WebGPU) variant clears the 25 MiB limit by only 0.1 MiB, so re-check it on upgrades; (3) **WebGPU is required, not preferred**: the model's `GatherBlockQuantized` embedding kernel has no WASM implementation and needs `shader-f16`, so a device without it gets a named refusal instead of an unusable download. Verified end to end in a browser: the sharded weights download with correct cumulative progress, reassemble, and build an ONNX Runtime WebGPU session (18 s warm). **Honest limit, stated rather than hidden**: the interpretation-quality bench could NOT be run here — it needs a GPU with `shader-f16`, which the CI and dev runners lack. It ships instead as a repo tool (`npm run llm:bench`, 16 FR/EN questions over Titanic, half of them phrasings the keyword grammar cannot catch) that exercises the real production path, so the number can be measured on real hardware before the model is promoted beyond opt-in. 314 unit tests, 57 e2e | -| **V27.1 — delivered** | **The model earns its place, it does not take it**: the V27 order was wrong, and the measurement said so. With the local model selected it read EVERY question first and won whenever its JSON passed the grammar check — even when the keyword parser had a correct reading of its own. Measured in production on six reference questions over Titanic: the model turned « combien de personnes sont montées à Cherbourg ? » into `embarked = Cherbourg` → **0 rows**, where the deterministic parser had `embark_town = Cherbourg` → **168**; and read « est-ce que les femmes payaient plus cher que les hommes ? » as a plain count (314 female) instead of mean fare grouped by sex. Tally: 2 right, 2 confidently wrong, 2 refusals. The order is now **deterministic first, model as a rescue** (`resolveIntent`, unit-tested): the keyword grammar can only ever name a column that exists and a value that actually occurs in it, so when it understands, nothing overrides it — and the model is asked only about what it gives up on, which is exactly the gap that justifies its 355 MB. On the same six, **measured on the owner's GPU after deploy**: 5 right, 1 wrong, 0 refusals — up from 2 right, 2 wrong, 2 refusals. Two further defects fixed: (1) a refusal was badged « question read by the local model », claiming a reading nobody had made — refusals now name nobody and say whether the model was even consulted; (2) the system prompt had **no groupBy and no top-k example at all**, and no rule tying a filter value to the column whose value list contains it — both added, with FR phrasings and a numeric-threshold example. The bench gains the two shapes that failed (`age < 10`, a top-k) and now reports the **shipped order** as its headline number instead of the two engines separately. 326 unit tests, 57 e2e. **The number that justifies the download**: « combien d'enfants de moins de 10 ans ? » → `count age < 10` = 62 and « à quel âge moyen voyageaient les passagers ? » → `mean age` = 29.699, both of which the keyword grammar refuses outright; and « combien de personnes sont montées à Cherbourg ? » came back as the deterministic engine's 168, the model never consulted. **Still open**: one question of the six — « est-ce que les femmes payaient plus cher que les hommes ? » — is still read as a correlation (fare↔age, a column the question never names); addressed in V27.2. The full bench remains un-runnable here (no `shader-f16`), so its number still has to come from real hardware. | Measured by the owner in production (22/08/2026), the day V27 shipped. A confidently wrong answer costs more trust than a refusal — and V27 produced two of them, including a 0 where the deterministic engine already had the right 168. | -| **V27.2 — delivered** | **Two honesty defects, one measured, one found while reading the measurement**: (1) the comparison question V27.1 left wrong — « est-ce que les femmes payaient plus cher que les hommes ? » read as a correlation between `fare` and `age` — gets a rule that names both halves of the mistake: a question comparing two groups is an aggregate with `groupBy` on the column whose values name them, NEVER a correlation; and never pick a column the question does not mention. A second FR comparison example ships with it, in a **different phrasing** from the failing one, which stays a held-out bench case rather than becoming a memorised answer. (2) The answer sentence said « (sur 891 lignes) » under a mean built from 714 values: `rowsConsidered` counts rows after the filter, while `numericAt` skips missing and unparseable cells. The number was right, the sentence around it was not. Aggregates now carry `valuesUsed` (scalar) and `used` per group, set only when they differ from the row count, and the UI says « 714 valeurs utilisables sur 891 lignes » — matching what the correlation branch already did. This one predates V27 entirely: it has been there since V6. 330 unit tests, 57 e2e. **Measured after deploy**: the rule did stop the correlation — but the model then read the same question as `count fare >= 0`, still wrong. Two prompt attempts, two failure modes; see V27.3 for where that stops. | Measured by the owner on real hardware (22/08/2026): 5 of 6 reference questions right after V27.1. The sixth is a confidently wrong answer to a different question than the one asked, and the « sur 891 lignes » wording was found by reading that same screenshot closely — a right number inside a wrong sentence is exactly what this project refuses to ship. | -| **V27.3 — delivered** | **A `>=` that was quietly an `=`**: retesting the comparison question after V27.2 produced « 0 ligne correspond où fare >= 0 » — impossible on a table where all 891 fares clear zero. Root cause found by reproduction, not by reading: the model emitted `"value": "0"` as a **string**, `asFilter` accepted a string for any operator, and `matchesFilter` took the numeric branch only for `typeof value === 'number'` — so `>=` fell through to the equality branch and tested `fare == "0"` against a column whose zero fares are written `0.0`. Same intent with a real number: 891 rows. The hole is closed on both sides: `asFilter` converts a numeric string and refuses anything else on `<`, `<=`, `>`, `>=` (equality keeps text — that is how categorical filters work), and `matchesFilter` handles the ordering operators apart, throwing the named `filter-not-numeric` rather than passing a bug off as a query with no matches. V6 code, reachable only through the model: the keyword parser always built numbers. **And a limit, recorded rather than papered over**: « est-ce que les femmes payaient plus cher que les hommes ? » is still read wrong — a correlation before V27.2, a vacuous count after. The model finds `fare` every time and the shape never. Two prompt attempts are enough; a third would be sewing the prompt around one sentence, which buys a flattering bench and nothing else. The measured score stands at **5 of 6**, and the sixth is written down as what a 0.6B does not do. 337 unit tests, 57 e2e. | Found by retesting in production (22/08/2026). An arithmetically impossible answer — zero rows for a condition every row satisfies — is worse than a refusal and worse than a wrong reading: it makes the engine itself untrustworthy, which is the one thing LabML sells. | -| **V28 — delivered** | **« Ne nous croyez pas sur parole »** — a `/privacy` route that states the local-only promise once, in full, and then hands the reader the means to check it without trusting a word of it. Four verification steps, ordered by how hard they are to fake: cut the network (DevTools → Network → Offline, or the Wi-Fi switch) and watch the whole lab keep working; watch the Network tab while loading a file and training, and see nothing happen; read `Content-Security-Policy` on the document itself; open Application → IndexedDB and see exactly what was kept. The served policy is **quoted verbatim on the page and pinned to `public/_headers` by a unit test** — a page that claims a protection the site quietly dropped is worse than no page. A schematic of the Network panel is drawn rather than screenshotted (DevTools chrome differs per browser and per locale) and captioned as a diagram, not a capture. A live audit panel counts this page's own resource timings by origin and says, in the same breath, what it cannot see: worker timelines and requests the CSP blocked — a proof that oversells itself is worth less than none. Last section lists what _does_ cross the network (app files, demo datasets on click, vision models on entering Vision, LLM weights on explicit consent) and what never does. FR/EN, prerendered shell, WCAG AA verified by axe including the audit result. 344 unit tests, 60 e2e. | Owner request (22/08/2026): the promise is repeated across the site, but a user has no way to tell a true claim from a comforting one. Verifiability is the product here — anyone can write « your data stays local » in a footer. | -| **V29 — delivered** | **Analytical SQL in the browser (DuckDB-Wasm, MIT)**: the Data Studio gains a real OLAP engine — joins, window functions, aggregations — over the file you just loaded, with no server and no upload. The file is queried **as dropped, before the cleaning recipe**: the recipe belongs to the studio, and a result traceable to nothing the user can reopen would be worse than no SQL at all. Extra CSV / **Parquet** / JSON files can be attached in the same session (Parquet is a new input format for the lab), each exposed as a view named after the file; a result exports to CSV or goes to the ML Lab in one click, through the handoff path V4 already built. Errors show **DuckDB's own message** — it names the line and the token, which no paraphrase of ours would. **The measurement that set the version**: `@duckdb/duckdb-wasm` is pinned to **1.28.0**, not `latest`. From 1.29 the binaries cross Cloudflare Pages' hard 25 MiB per-file limit (eh 34.2 MiB, mvp 39.4 MiB); at 1.28.0 they are **17.3 and 21.1 MiB** and fit. Newer would have meant sharding the wasm and either widening `connect-src` to `blob:` — days after publishing a page that quotes that very directive — or rebuilding the service worker in injectManifest mode. An older engine was the cheaper honest trade, and it is written here so the next upgrade re-measures instead of rediscovering. Self-hosted under `/duckdb/` (the library defaults to jsDelivr, which the CSP refuses), **never precached** — cached on first use like the vision models, so nobody pays 18 MiB before opening the console — and the `coi` threaded build is left out entirely: no COOP/COEP, no SharedArrayBuffer, single-threaded as the assumed mode. Remote S3/HTTP querying stays out, by CSP and by intent. 352 unit tests, 61 e2e. | Owner request (21/08/2026): real analytical SQL on ~100 MB files with zero backend. Delivered after the /privacy page at the owner's request (22/08/2026). | -| **V30 — delivered** | **Chat that reads better, measured before it is made bigger.** The wave began by building the instrument, because V27's stood on 18 cases that needed a GPU with `shader-f16` — one laptop's worth of evidence, re-runnable by nobody. It now stands on **55 reference questions**, French and English, over every shape of the query grammar plus three that no query can answer, where refusing is the only correct outcome. Two harnesses run it: one in CI on every commit with no model at all (the deterministic parser, the grammar automaton, the token mask), and one against the REAL pinned q4f16 weights on a CPU through onnxruntime-node — same files, same prompt, same decoding path as production, minus the GPU. « Measurable » stopped meaning « on one machine ». **The instrument immediately contradicted the wave's own premise.** The failure was not mainly the model: on those 55 questions the shipped app answered **33 right, 15 WRONG, 7 refused** — and **seven of the fifteen wrong came from the deterministic parser**, which runs first and can never be overridden. « Combien de femmes ? » answered 891 instead of 314: the grammar knows `combien`, knows nothing about `femmes`, kept the count and dropped the condition — under the badge that is supposed to mean exact. Four of those seven the local model reads correctly, and never got asked. **So the parser now checks its own coverage**: every word of the question must be accounted for by a lexicon phrase, a column the answer uses, a value it filters on, or one of three closed lists (the table's own furniture, generic row nouns, grammatical filler). A leftover word is a refusal. Measured: **19 right, 0 wrong, 36 refused** — the seven wrong answers became refusals and not one correct answer was lost. `wrong === 0` is now asserted in CI, and the trade is one-directional by construction: an unknown word can cost a refusal where an answer was possible, never a wrong answer where a refusal was right. **(B) Constrained decoding**, hand-written: an automaton over the query grammar and a `LogitsProcessor` that masks, at every token, everything that would leave it. It walks UTF-8 **bytes**, not characters, because Qwen's vocabulary is byte-level BPE and 1 457 of its 151 669 tokens are fragments of a character — a character-level automaton would have made « Île-de-France » unwritable as a filter value. It cost about 16 ms at its most expensive step (`{"kind":"` masks seven letters against seven large buckets) after the first-byte step was hoisted out of the per-token loop, down from 53 ms. **And on its own it made things worse**: model refusals fell 14 → 2 and correct answers rose 29 → 34, but **wrong answers rose 12 → 19**. Forcing a valid answer turns « I could not parse that » into a confident wrong number. That is why the grammar keeps `{"kind":"none"}` reachable — a shape whose only meaning is « I cannot express this », which maps to the refusal V27 already had. **(C) Examples drawn from the user's own columns**, for 0 MB — and the reason turned out to be sharper than the plan's. V27's nine examples were frozen Titanic, and **seven of the 55 corpus questions appear in them verbatim**: the prompt had been fitted to the bench across V27.1 and V27.2, so on those questions the old bench could not tell reading from recitation. (Checked rather than assumed: on those seven, before and after score identically, 5 right / 1 wrong / 1 refused. The defect is methodological, and its measured effect on this comparison is zero.) Generating the examples from the loaded file removes the contamination structurally and deletes the rule that asked the model to ignore what it had just been shown. **Two versions of them were worse than the frozen ones, and both reasons are now in the code.** The first left out the aggregate-WITH-FILTER shape: under constraint, a shape the model has not been shown comes out as a confident wrong answer rather than a refusal, and « prix moyen payé par les survivants » became `count where fare = 1000000000`. The second still picked the FIRST numeric column, which on Titanic is `survived` — so the examples read « average survived » and the model duly reached for that column on questions that never mention it. `ColumnInfo` gained a capped `distinct` count so an example averages a **quantity**, not a 0/1 flag. **(D) Two samples, one vote — dropped, on this wave's own measurements.** Both halves of its tie-break died with (B): constrained decoding guarantees every candidate validates, so « keep the one that validates » no longer discriminates; and « invents no column the question never names » is contradicted by the corpus, where « did women pay more than men? » is correctly answered with `fare`, a column the question never names. Two samples for a vote with no criterion, at twice the latency, is not a trade. **What the wave deliberately does not do**: ship a second, bigger model as a download (the cheap levers were not exhausted when the plan proposed it, and now they are), guess a column by fuzzy name-matching (the refusal is the honest outcome), or let the grammar automaton replace `validateIntent` — it over-approximates in two named places and is a filter, not the authority. 557 unit tests, 79 e2e. | Owner question (22/08/2026): would a bigger model raise the share of correct answers? The wave answers with a measurement rather than an estimate, and the answer has two halves. For **0 MB**, the app went from **33 right / 15 wrong** to **42 right / 7 wrong** out of 55 — nine more correct answers and **fifty-three percent fewer wrong ones**, the single largest piece of which came from the deterministic parser rather than the model. And the bench now takes the model as a parameter (`LABML_LLM_REPO=… npm run llm:bench:node`), so the second half is a command, not an opinion. | -| V31 | **Vision that stops being asked the impossible.** Today's three models weigh **18.6 MB total** (EfficientNet-Lite4 int8 13.6, YOLOX-Nano 3.7, UltraFace 1.3) against 370 MB for the chat model — the headroom is enormous. **The main cause of the mistakes is not the network**: ImageNet-1k has **no « person » class** — 1000 labels, ~120 of them dog breeds, none for a human being — so a photo of someone comes back as « suit » or « jersey ». The model is not wrong; it is being asked a question whose answer is absent from its vocabulary. **(A) Measure first**: 30–50 public-domain images with expected label and expected boxes, replayed in e2e, so « it still makes mistakes » becomes a percentage. **(B) Fix the label space — the real correction**: CLIP ViT-B/32 zero-shot, vision q4f16 **126 MB** + text int8 **64 MB** ≈ **190 MB**, letting the visitor type their own labels (« a cat », « an invoice », « a houseplant »). It repairs the defect and makes a far better demonstration than 1000 frozen classes; open weights, self-hosted, local execution — the doctrine holds. **(C) What costs no download**: check the crop (squashing a 16:9 photo into a square skews everything — `preprocess.ts` is the suspect), average over two crops, recalibrate `OBJECT_THRESHOLD` (0.35) and `FACE_THRESHOLD` (0.9), and above all **refuse below a confidence floor** — « I am not sure » rather than a label picked at random, which is the chat's doctrine applied to pixels. **(D) A better detector**: YOLOX-S (Apache-2.0), ~35 MB, roughly +14 mAP over Nano — with acquisition and licence verified first, as in V23: the YOLOX ONNX files on the Hub are community re-uploads, not official releases. | Owner report (22/08/2026): the vision playground is better than the chat but still makes mistakes. Naming the label-space mismatch is what turns a vague complaint into a fixable defect. | -| V32 | **Documentation, the scaffolding and one finished tutorial.** A `/docs` route, linked from the footer beside « Comment ça marche », built on the **Diátaxis** split — tutorial (learning), how-to (a task), reference (lookup), explanation (the why) — because the usual failure of documentation is mixing all four on one page: a tutorial that pauses to weigh an alternative loses the beginner it was written for. A tutorial offers **no choices** and **guarantees the result**. Five rules specific to this project: **(1) the docs are tested like the code** — everything here is seeded at 42, so « you will get 0.821 accuracy » becomes an assertion in `e2e/docs.spec.ts` and a drifting page **breaks the build**; a documentation that cannot lie is the same promise as the rest of the site. **(2) Screenshots are generated** with Playwright, never hand-taken — one that cannot be regenerated does not ship. **(3) Better than a screenshot, a link that does the thing**: « try it » deep-links landing on the panel with the demo already loaded (needs small URL-parameter support), which never goes stale. **(4) Markdown lives in the repo** (`src/content/docs/**`), compiled at build with prerendered shells like every other route; no Algolia, no third-party doc host — a third-party call on a site that publishes `/privacy` would be indefensible, so search is a local index. **(5) The docs are not PLAN.md**: this file is the engineering record in English with the trade-offs; the docs are for users, FR/EN. Scope of this wave: the route, the Markdown pipeline, the table of contents, local search, and **one** complete tutorial — « premier modèle en 10 minutes » — tested end to end. It is the template every later page copies: tone, length, how figures are quoted. | Owner request (22/08/2026): document every shipped feature across /ml, /data and /ai, linked from the footer. One finished tutorial first, on purpose — writing the full reference before the template is settled means rewriting all of it. | -| V33 | **The reference, and the table of refusals.** Page-per-panel coverage of the three sections: ML Lab (leaderboard, tuning, thresholds, segments, uncertainty, learning curves, run comparison, model export/import, batch scoring), Data Studio (quality score, recipe, forced types, join, drift, anomalies, SQL console) and AI (vision, assistant, the two interpreters). Reference is dry, exhaustive and structured like the software — not prose. The page that no competitor has: **a complete table of the named refusals** — `filter-not-numeric`, `llm-part-missing`, `too-large`, `no-webgpu`, « neither interpreter understood », « the interval is not conclusive » — with what triggers each one, what it means and what to do about it. Refusing well is this project's distinguishing feature; documenting the refusals is the most honest page it can publish. Plus a formats page (CSV, Parquet, JSON, the model manifest). **Honest sizing**: this is 1–2 days of _writing_ for ~25 features in two languages. It does not automate into anything but mush. | A feature nobody can look up is a feature that does not exist for the reader; and a refusal nobody can decode reads as a bug rather than as the design it is. | -| V34 | **Explanations, how-to guides, and the pages that make the project readable as engineering.** The why-pages: why a baseline before anything else, why intervals instead of a single figure, why seed 42 everywhere, why everything runs locally (pointing at `/privacy` rather than repeating it — duplicated prose diverges), and **what LabML does not do, and why** — a project that names its limits reads as a serious one, and the limits are already measured here (the comparison question a 0.6B model does not read, the bench that needs `shader-f16`, SQL over the file before the recipe). Task-shaped how-to guides for readers who already know their way around: score a new batch, compare two runs, read a learning curve, hand a SQL result to the lab. Generated screenshots and the « try it » deep-links land here too. Every page ends with « et ensuite ? » — documentation without a next step is a dead end. | Three audiences, deliberately: the curious visitor (five minutes), the practitioner (one task), and the evaluator judging whether the engineering is rigorous. The explanation pages are what the third one reads. | -| **V35 — delivered** | **ML Lab: the number stops flattering itself.** Two method defects in shipped code, fixed, plus the two additions that follow from them. **(1) The winner was picked on the test set** — the leaderboard sorted nine models by the metric computed on test and crowned `sorted[0]`, which makes the headline figure the optimistic maximum of nine draws. There is now a **third split**: validation is carved out of the train side (64/16/20 with the default ratios), ranking and crowning happen on validation, and the champion line spells out both numbers and the gap between them — « selected on validation at 0.974, scores 0.917 on the untouched test set ». The **test indices are byte-identical** to what the same config produced before V35, so every panel that reads the test set (segments, thresholds, uncertainty, batch compare) is unchanged; below 60 usable rows the third split is refused by name and the lab ranks on test as before. Ranking now lives in ONE module (`ranking.ts`) used by the leaderboard, the history, the run comparison, the report and the auto-selected insights model — the bug that shipped mid-wave was exactly that a fourth site still sorted on test and opened a different model than the one crowned. **(2) The split was always random, even on dated data.** A chronological split (oldest rows train, newest test, rows without a parseable date dropped and counted) and a group split (no group on both sides — the same customer in train and test is the same leak) are offered when a column supports one, and **announced** in the run info. **(3) A predictive leak detector**: V6 caught columns that MAP to the target; this catches the merely predictive one — a one-column stump fitted on train and scored on validation, and a lone column reading the target at ≥ 99% shows as a copper warning with its measured score, never as a victory. **(4) A robust leaderboard on demand**: 5×2 repeated cross-validation over train+validation (the pipeline refitted inside every fold, the test set never touched), reporting a mean, a spread, and how often the leader actually beat the runner-up — « 10 of 10 folds: the order is stable » or « 6 of 10: treat them as tied ». **A defect the wave exposed and named**: with the smaller train split, Gaussian Naive Bayes on ~150 TF-IDF features saturates to exactly 0/1, so V24's word-effect occlusion measured exactly zero for every word and the card simply vanished — reading as « no word matters », which is false. Measured (2 distinct probabilities out of 48 test rows, against 48 for logistic and gbdt), the card now **refuses by name** and points at a model that can answer. 369 unit tests, 65 e2e. | Owner request (22/08/2026), launched 23/08/2026. Two of the four items were defects rather than gaps: a lab that sells honest evaluation cannot ship a headline figure it knows to be optimistic, nor a split that leaks on dated data. | -| **V36 — delivered** | **ML Lab: the gaps that were deliberately left open.** Each item was consciously deferred in an earlier wave rather than forgotten; delivering them together keeps the descopes visible instead of letting them quietly become permanent. **(1) Class imbalance**, descoped by name in V16. Two mechanisms, each NAMED per family rather than hidden behind one word: the loss is weighted where the loss is ours (logistic regression, gradient boosting — the gradient AND the hessian are scaled, since scaling only the gradient inflates leaf values instead of rebalancing), and a **seeded balanced resample** is used for the ml-cart families (tree, forest), which take no sample weights. The minority is upsampled to the majority's size — never the reverse: balancing by trimming the common class throws away real observations to fix a ratio. Off by default, because on a balanced target it changes nothing and a knob that does nothing is worse than no knob; the run announces the majority share, and the leaderboard says so when it crosses 60%. **(2) The ranking metric is a choice.** Accuracy and RMSE were imposed, which is the wrong criterion on an imbalanced problem — a model that never predicts the rare class can top an accuracy ranking and be useless. Rank on F1, recall, precision or ROC-AUC and the order genuinely changes; a model that cannot produce the chosen metric sorts last rather than being dropped. Ranking stays in the single V35 module, so the leaderboard, the history, the comparison, the report and the auto-inspected model all move together. **(3) Multiclass thresholds**, open since V16, read **one-vs-rest**: pick a class, score it against all the others, same PR and calibration curves. What the panel refuses to imply is a complete multiclass decision rule — two classes can both clear their thresholds and nothing here says which wins, so it says that instead. **(4) An ensemble of the best**: the average of the top three, built from models already fitted, so it costs one pass over the test set. The baseline is never a member (averaging a constant predictor drags the result toward the majority class), members are picked by the same V35 ranking rule, and **probabilistic members are preferred** — a mid-wave measurement showed the ensemble winning on iris with a bare vote because k-NN was in the top three, which silently closed the threshold, calibration and word-effect panels on the champion. **What this wave deliberately does not do**: add a tenth model family (nine is plenty; a tenth improves neither honesty nor understanding), build an AutoML « we handle everything » mode (the opposite of a lab that shows its decisions), or bring in tabular deep learning (high cost, no gain at this scale, and no longer hand-written). 387 unit tests, 69 e2e. | Launched 23/08/2026, right after V35. Each item was a named descope, not an oversight — and the ensemble exposed one more silent-disappearance defect, of the same family as the one V35 found. | +| **V27.1 — delivered** | **The model earns its place, it does not take it**: the V27 order was wrong, and the measurement said so. With the local model selected it read EVERY question first and won whenever its JSON passed the grammar check — even when the keyword parser had a correct reading of its own. Measured in production on six reference questions over Titanic: the model turned « combien de personnes sont montées à Cherbourg ? » into `embarked = Cherbourg` → **0 rows**, where the deterministic parser had `embark_town = Cherbourg` → **168**; and read « est-ce que les femmes payaient plus cher que les hommes ? » as a plain count (314 female) instead of mean fare grouped by sex. Tally: 2 right, 2 confidently wrong, 2 refusals. The order is now **deterministic first, model as a rescue** (`resolveIntent`, unit-tested): the keyword grammar can only ever name a column that exists and a value that actually occurs in it, so when it understands, nothing overrides it — and the model is asked only about what it gives up on, which is exactly the gap that justifies its 355 MB. On the same six, **measured on the owner's GPU after deploy**: 5 right, 1 wrong, 0 refusals — up from 2 right, 2 wrong, 2 refusals. Two further defects fixed: (1) a refusal was badged « question read by the local model », claiming a reading nobody had made — refusals now name nobody and say whether the model was even consulted; (2) the system prompt had **no groupBy and no top-k example at all**, and no rule tying a filter value to the column whose value list contains it — both added, with FR phrasings and a numeric-threshold example. The bench gains the two shapes that failed (`age < 10`, a top-k) and now reports the **shipped order** as its headline number instead of the two engines separately. 326 unit tests, 57 e2e. **The number that justifies the download**: « combien d'enfants de moins de 10 ans ? » → `count age < 10` = 62 and « à quel âge moyen voyageaient les passagers ? » → `mean age` = 29.699, both of which the keyword grammar refuses outright; and « combien de personnes sont montées à Cherbourg ? » came back as the deterministic engine's 168, the model never consulted. **Still open**: one question of the six — « est-ce que les femmes payaient plus cher que les hommes ? » — is still read as a correlation (fare↔age, a column the question never names); addressed in V27.2. The full bench remains un-runnable here (no `shader-f16`), so its number still has to come from real hardware. | Measured by the owner in production (22/08/2026), the day V27 shipped. A confidently wrong answer costs more trust than a refusal — and V27 produced two of them, including a 0 where the deterministic engine already had the right 168. | +| **V27.2 — delivered** | **Two honesty defects, one measured, one found while reading the measurement**: (1) the comparison question V27.1 left wrong — « est-ce que les femmes payaient plus cher que les hommes ? » read as a correlation between `fare` and `age` — gets a rule that names both halves of the mistake: a question comparing two groups is an aggregate with `groupBy` on the column whose values name them, NEVER a correlation; and never pick a column the question does not mention. A second FR comparison example ships with it, in a **different phrasing** from the failing one, which stays a held-out bench case rather than becoming a memorised answer. (2) The answer sentence said « (sur 891 lignes) » under a mean built from 714 values: `rowsConsidered` counts rows after the filter, while `numericAt` skips missing and unparseable cells. The number was right, the sentence around it was not. Aggregates now carry `valuesUsed` (scalar) and `used` per group, set only when they differ from the row count, and the UI says « 714 valeurs utilisables sur 891 lignes » — matching what the correlation branch already did. This one predates V27 entirely: it has been there since V6. 330 unit tests, 57 e2e. **Measured after deploy**: the rule did stop the correlation — but the model then read the same question as `count fare >= 0`, still wrong. Two prompt attempts, two failure modes; see V27.3 for where that stops. | Measured by the owner on real hardware (22/08/2026): 5 of 6 reference questions right after V27.1. The sixth is a confidently wrong answer to a different question than the one asked, and the « sur 891 lignes » wording was found by reading that same screenshot closely — a right number inside a wrong sentence is exactly what this project refuses to ship. | +| **V27.3 — delivered** | **A `>=` that was quietly an `=`**: retesting the comparison question after V27.2 produced « 0 ligne correspond où fare >= 0 » — impossible on a table where all 891 fares clear zero. Root cause found by reproduction, not by reading: the model emitted `"value": "0"` as a **string**, `asFilter` accepted a string for any operator, and `matchesFilter` took the numeric branch only for `typeof value === 'number'` — so `>=` fell through to the equality branch and tested `fare == "0"` against a column whose zero fares are written `0.0`. Same intent with a real number: 891 rows. The hole is closed on both sides: `asFilter` converts a numeric string and refuses anything else on `<`, `<=`, `>`, `>=` (equality keeps text — that is how categorical filters work), and `matchesFilter` handles the ordering operators apart, throwing the named `filter-not-numeric` rather than passing a bug off as a query with no matches. V6 code, reachable only through the model: the keyword parser always built numbers. **And a limit, recorded rather than papered over**: « est-ce que les femmes payaient plus cher que les hommes ? » is still read wrong — a correlation before V27.2, a vacuous count after. The model finds `fare` every time and the shape never. Two prompt attempts are enough; a third would be sewing the prompt around one sentence, which buys a flattering bench and nothing else. The measured score stands at **5 of 6**, and the sixth is written down as what a 0.6B does not do. 337 unit tests, 57 e2e. | Found by retesting in production (22/08/2026). An arithmetically impossible answer — zero rows for a condition every row satisfies — is worse than a refusal and worse than a wrong reading: it makes the engine itself untrustworthy, which is the one thing LabML sells. | +| **V28 — delivered** | **« Ne nous croyez pas sur parole »** — a `/privacy` route that states the local-only promise once, in full, and then hands the reader the means to check it without trusting a word of it. Four verification steps, ordered by how hard they are to fake: cut the network (DevTools → Network → Offline, or the Wi-Fi switch) and watch the whole lab keep working; watch the Network tab while loading a file and training, and see nothing happen; read `Content-Security-Policy` on the document itself; open Application → IndexedDB and see exactly what was kept. The served policy is **quoted verbatim on the page and pinned to `public/_headers` by a unit test** — a page that claims a protection the site quietly dropped is worse than no page. A schematic of the Network panel is drawn rather than screenshotted (DevTools chrome differs per browser and per locale) and captioned as a diagram, not a capture. A live audit panel counts this page's own resource timings by origin and says, in the same breath, what it cannot see: worker timelines and requests the CSP blocked — a proof that oversells itself is worth less than none. Last section lists what _does_ cross the network (app files, demo datasets on click, vision models on entering Vision, LLM weights on explicit consent) and what never does. FR/EN, prerendered shell, WCAG AA verified by axe including the audit result. 344 unit tests, 60 e2e. | Owner request (22/08/2026): the promise is repeated across the site, but a user has no way to tell a true claim from a comforting one. Verifiability is the product here — anyone can write « your data stays local » in a footer. | +| **V29 — delivered** | **Analytical SQL in the browser (DuckDB-Wasm, MIT)**: the Data Studio gains a real OLAP engine — joins, window functions, aggregations — over the file you just loaded, with no server and no upload. The file is queried **as dropped, before the cleaning recipe**: the recipe belongs to the studio, and a result traceable to nothing the user can reopen would be worse than no SQL at all. Extra CSV / **Parquet** / JSON files can be attached in the same session (Parquet is a new input format for the lab), each exposed as a view named after the file; a result exports to CSV or goes to the ML Lab in one click, through the handoff path V4 already built. Errors show **DuckDB's own message** — it names the line and the token, which no paraphrase of ours would. **The measurement that set the version**: `@duckdb/duckdb-wasm` is pinned to **1.28.0**, not `latest`. From 1.29 the binaries cross Cloudflare Pages' hard 25 MiB per-file limit (eh 34.2 MiB, mvp 39.4 MiB); at 1.28.0 they are **17.3 and 21.1 MiB** and fit. Newer would have meant sharding the wasm and either widening `connect-src` to `blob:` — days after publishing a page that quotes that very directive — or rebuilding the service worker in injectManifest mode. An older engine was the cheaper honest trade, and it is written here so the next upgrade re-measures instead of rediscovering. Self-hosted under `/duckdb/` (the library defaults to jsDelivr, which the CSP refuses), **never precached** — cached on first use like the vision models, so nobody pays 18 MiB before opening the console — and the `coi` threaded build is left out entirely: no COOP/COEP, no SharedArrayBuffer, single-threaded as the assumed mode. Remote S3/HTTP querying stays out, by CSP and by intent. 352 unit tests, 61 e2e. | Owner request (21/08/2026): real analytical SQL on ~100 MB files with zero backend. Delivered after the /privacy page at the owner's request (22/08/2026). | +| **V30 — delivered** | **Chat that reads better, measured before it is made bigger.** The wave began by building the instrument, because V27's stood on 18 cases that needed a GPU with `shader-f16` — one laptop's worth of evidence, re-runnable by nobody. It now stands on **55 reference questions**, French and English, over every shape of the query grammar plus three that no query can answer, where refusing is the only correct outcome. Two harnesses run it: one in CI on every commit with no model at all (the deterministic parser, the grammar automaton, the token mask), and one against the REAL pinned q4f16 weights on a CPU through onnxruntime-node — same files, same prompt, same decoding path as production, minus the GPU. « Measurable » stopped meaning « on one machine ». **The instrument immediately contradicted the wave's own premise.** The failure was not mainly the model: on those 55 questions the shipped app answered **33 right, 15 WRONG, 7 refused** — and **seven of the fifteen wrong came from the deterministic parser**, which runs first and can never be overridden. « Combien de femmes ? » answered 891 instead of 314: the grammar knows `combien`, knows nothing about `femmes`, kept the count and dropped the condition — under the badge that is supposed to mean exact. Four of those seven the local model reads correctly, and never got asked. **So the parser now checks its own coverage**: every word of the question must be accounted for by a lexicon phrase, a column the answer uses, a value it filters on, or one of three closed lists (the table's own furniture, generic row nouns, grammatical filler). A leftover word is a refusal. Measured: **19 right, 0 wrong, 36 refused** — the seven wrong answers became refusals and not one correct answer was lost. `wrong === 0` is now asserted in CI, and the trade is one-directional by construction: an unknown word can cost a refusal where an answer was possible, never a wrong answer where a refusal was right. **(B) Constrained decoding**, hand-written: an automaton over the query grammar and a `LogitsProcessor` that masks, at every token, everything that would leave it. It walks UTF-8 **bytes**, not characters, because Qwen's vocabulary is byte-level BPE and 1 457 of its 151 669 tokens are fragments of a character — a character-level automaton would have made « Île-de-France » unwritable as a filter value. It cost about 16 ms at its most expensive step (`{"kind":"` masks seven letters against seven large buckets) after the first-byte step was hoisted out of the per-token loop, down from 53 ms. **And on its own it made things worse**: model refusals fell 14 → 2 and correct answers rose 29 → 34, but **wrong answers rose 12 → 19**. Forcing a valid answer turns « I could not parse that » into a confident wrong number. That is why the grammar keeps `{"kind":"none"}` reachable — a shape whose only meaning is « I cannot express this », which maps to the refusal V27 already had. **(C) Examples drawn from the user's own columns**, for 0 MB — and the reason turned out to be sharper than the plan's. V27's nine examples were frozen Titanic, and **seven of the 55 corpus questions appear in them verbatim**: the prompt had been fitted to the bench across V27.1 and V27.2, so on those questions the old bench could not tell reading from recitation. (Checked rather than assumed: on those seven, before and after score identically, 5 right / 1 wrong / 1 refused. The defect is methodological, and its measured effect on this comparison is zero.) Generating the examples from the loaded file removes the contamination structurally and deletes the rule that asked the model to ignore what it had just been shown. **Two versions of them were worse than the frozen ones, and both reasons are now in the code.** The first left out the aggregate-WITH-FILTER shape: under constraint, a shape the model has not been shown comes out as a confident wrong answer rather than a refusal, and « prix moyen payé par les survivants » became `count where fare = 1000000000`. The second still picked the FIRST numeric column, which on Titanic is `survived` — so the examples read « average survived » and the model duly reached for that column on questions that never mention it. `ColumnInfo` gained a capped `distinct` count so an example averages a **quantity**, not a 0/1 flag. **(D) Two samples, one vote — dropped, on this wave's own measurements.** Both halves of its tie-break died with (B): constrained decoding guarantees every candidate validates, so « keep the one that validates » no longer discriminates; and « invents no column the question never names » is contradicted by the corpus, where « did women pay more than men? » is correctly answered with `fare`, a column the question never names. Two samples for a vote with no criterion, at twice the latency, is not a trade. **What the wave deliberately does not do**: ship a second, bigger model as a download (the cheap levers were not exhausted when the plan proposed it, and now they are), guess a column by fuzzy name-matching (the refusal is the honest outcome), or let the grammar automaton replace `validateIntent` — it over-approximates in two named places and is a filter, not the authority. 557 unit tests, 79 e2e. | Owner question (22/08/2026): would a bigger model raise the share of correct answers? The wave answers with a measurement rather than an estimate, and the answer has two halves. For **0 MB**, the app went from **33 right / 15 wrong** to **42 right / 7 wrong** out of 55 — nine more correct answers and **fifty-three percent fewer wrong ones**, the single largest piece of which came from the deterministic parser rather than the model. And the second half was measured too, not deferred: **Qwen3-1.7B at 1.43 GB — four times the download — scores worse** (40 right / 12 wrong against 42 / 7). It reads the hard questions better and the easy ones worse. « Bigger » is not a direction of improvement on this task; it is a trade whose sign has to be measured, and the bench now measures it in one command. | +| V31 | **Vision that stops being asked the impossible.** Today's three models weigh **18.6 MB total** (EfficientNet-Lite4 int8 13.6, YOLOX-Nano 3.7, UltraFace 1.3) against 370 MB for the chat model — the headroom is enormous. **The main cause of the mistakes is not the network**: ImageNet-1k has **no « person » class** — 1000 labels, ~120 of them dog breeds, none for a human being — so a photo of someone comes back as « suit » or « jersey ». The model is not wrong; it is being asked a question whose answer is absent from its vocabulary. **(A) Measure first**: 30–50 public-domain images with expected label and expected boxes, replayed in e2e, so « it still makes mistakes » becomes a percentage. **(B) Fix the label space — the real correction**: CLIP ViT-B/32 zero-shot, vision q4f16 **126 MB** + text int8 **64 MB** ≈ **190 MB**, letting the visitor type their own labels (« a cat », « an invoice », « a houseplant »). It repairs the defect and makes a far better demonstration than 1000 frozen classes; open weights, self-hosted, local execution — the doctrine holds. **(C) What costs no download**: check the crop (squashing a 16:9 photo into a square skews everything — `preprocess.ts` is the suspect), average over two crops, recalibrate `OBJECT_THRESHOLD` (0.35) and `FACE_THRESHOLD` (0.9), and above all **refuse below a confidence floor** — « I am not sure » rather than a label picked at random, which is the chat's doctrine applied to pixels. **(D) A better detector**: YOLOX-S (Apache-2.0), ~35 MB, roughly +14 mAP over Nano — with acquisition and licence verified first, as in V23: the YOLOX ONNX files on the Hub are community re-uploads, not official releases. | Owner report (22/08/2026): the vision playground is better than the chat but still makes mistakes. Naming the label-space mismatch is what turns a vague complaint into a fixable defect. | +| V32 | **Documentation, the scaffolding and one finished tutorial.** A `/docs` route, linked from the footer beside « Comment ça marche », built on the **Diátaxis** split — tutorial (learning), how-to (a task), reference (lookup), explanation (the why) — because the usual failure of documentation is mixing all four on one page: a tutorial that pauses to weigh an alternative loses the beginner it was written for. A tutorial offers **no choices** and **guarantees the result**. Five rules specific to this project: **(1) the docs are tested like the code** — everything here is seeded at 42, so « you will get 0.821 accuracy » becomes an assertion in `e2e/docs.spec.ts` and a drifting page **breaks the build**; a documentation that cannot lie is the same promise as the rest of the site. **(2) Screenshots are generated** with Playwright, never hand-taken — one that cannot be regenerated does not ship. **(3) Better than a screenshot, a link that does the thing**: « try it » deep-links landing on the panel with the demo already loaded (needs small URL-parameter support), which never goes stale. **(4) Markdown lives in the repo** (`src/content/docs/**`), compiled at build with prerendered shells like every other route; no Algolia, no third-party doc host — a third-party call on a site that publishes `/privacy` would be indefensible, so search is a local index. **(5) The docs are not PLAN.md**: this file is the engineering record in English with the trade-offs; the docs are for users, FR/EN. Scope of this wave: the route, the Markdown pipeline, the table of contents, local search, and **one** complete tutorial — « premier modèle en 10 minutes » — tested end to end. It is the template every later page copies: tone, length, how figures are quoted. | Owner request (22/08/2026): document every shipped feature across /ml, /data and /ai, linked from the footer. One finished tutorial first, on purpose — writing the full reference before the template is settled means rewriting all of it. | +| V33 | **The reference, and the table of refusals.** Page-per-panel coverage of the three sections: ML Lab (leaderboard, tuning, thresholds, segments, uncertainty, learning curves, run comparison, model export/import, batch scoring), Data Studio (quality score, recipe, forced types, join, drift, anomalies, SQL console) and AI (vision, assistant, the two interpreters). Reference is dry, exhaustive and structured like the software — not prose. The page that no competitor has: **a complete table of the named refusals** — `filter-not-numeric`, `llm-part-missing`, `too-large`, `no-webgpu`, « neither interpreter understood », « the interval is not conclusive » — with what triggers each one, what it means and what to do about it. Refusing well is this project's distinguishing feature; documenting the refusals is the most honest page it can publish. Plus a formats page (CSV, Parquet, JSON, the model manifest). **Honest sizing**: this is 1–2 days of _writing_ for ~25 features in two languages. It does not automate into anything but mush. | A feature nobody can look up is a feature that does not exist for the reader; and a refusal nobody can decode reads as a bug rather than as the design it is. | +| V34 | **Explanations, how-to guides, and the pages that make the project readable as engineering.** The why-pages: why a baseline before anything else, why intervals instead of a single figure, why seed 42 everywhere, why everything runs locally (pointing at `/privacy` rather than repeating it — duplicated prose diverges), and **what LabML does not do, and why** — a project that names its limits reads as a serious one, and the limits are already measured here (the comparison question a 0.6B model does not read, the bench that needs `shader-f16`, SQL over the file before the recipe). Task-shaped how-to guides for readers who already know their way around: score a new batch, compare two runs, read a learning curve, hand a SQL result to the lab. Generated screenshots and the « try it » deep-links land here too. Every page ends with « et ensuite ? » — documentation without a next step is a dead end. | Three audiences, deliberately: the curious visitor (five minutes), the practitioner (one task), and the evaluator judging whether the engineering is rigorous. The explanation pages are what the third one reads. | +| **V35 — delivered** | **ML Lab: the number stops flattering itself.** Two method defects in shipped code, fixed, plus the two additions that follow from them. **(1) The winner was picked on the test set** — the leaderboard sorted nine models by the metric computed on test and crowned `sorted[0]`, which makes the headline figure the optimistic maximum of nine draws. There is now a **third split**: validation is carved out of the train side (64/16/20 with the default ratios), ranking and crowning happen on validation, and the champion line spells out both numbers and the gap between them — « selected on validation at 0.974, scores 0.917 on the untouched test set ». The **test indices are byte-identical** to what the same config produced before V35, so every panel that reads the test set (segments, thresholds, uncertainty, batch compare) is unchanged; below 60 usable rows the third split is refused by name and the lab ranks on test as before. Ranking now lives in ONE module (`ranking.ts`) used by the leaderboard, the history, the run comparison, the report and the auto-selected insights model — the bug that shipped mid-wave was exactly that a fourth site still sorted on test and opened a different model than the one crowned. **(2) The split was always random, even on dated data.** A chronological split (oldest rows train, newest test, rows without a parseable date dropped and counted) and a group split (no group on both sides — the same customer in train and test is the same leak) are offered when a column supports one, and **announced** in the run info. **(3) A predictive leak detector**: V6 caught columns that MAP to the target; this catches the merely predictive one — a one-column stump fitted on train and scored on validation, and a lone column reading the target at ≥ 99% shows as a copper warning with its measured score, never as a victory. **(4) A robust leaderboard on demand**: 5×2 repeated cross-validation over train+validation (the pipeline refitted inside every fold, the test set never touched), reporting a mean, a spread, and how often the leader actually beat the runner-up — « 10 of 10 folds: the order is stable » or « 6 of 10: treat them as tied ». **A defect the wave exposed and named**: with the smaller train split, Gaussian Naive Bayes on ~150 TF-IDF features saturates to exactly 0/1, so V24's word-effect occlusion measured exactly zero for every word and the card simply vanished — reading as « no word matters », which is false. Measured (2 distinct probabilities out of 48 test rows, against 48 for logistic and gbdt), the card now **refuses by name** and points at a model that can answer. 369 unit tests, 65 e2e. | Owner request (22/08/2026), launched 23/08/2026. Two of the four items were defects rather than gaps: a lab that sells honest evaluation cannot ship a headline figure it knows to be optimistic, nor a split that leaks on dated data. | +| **V36 — delivered** | **ML Lab: the gaps that were deliberately left open.** Each item was consciously deferred in an earlier wave rather than forgotten; delivering them together keeps the descopes visible instead of letting them quietly become permanent. **(1) Class imbalance**, descoped by name in V16. Two mechanisms, each NAMED per family rather than hidden behind one word: the loss is weighted where the loss is ours (logistic regression, gradient boosting — the gradient AND the hessian are scaled, since scaling only the gradient inflates leaf values instead of rebalancing), and a **seeded balanced resample** is used for the ml-cart families (tree, forest), which take no sample weights. The minority is upsampled to the majority's size — never the reverse: balancing by trimming the common class throws away real observations to fix a ratio. Off by default, because on a balanced target it changes nothing and a knob that does nothing is worse than no knob; the run announces the majority share, and the leaderboard says so when it crosses 60%. **(2) The ranking metric is a choice.** Accuracy and RMSE were imposed, which is the wrong criterion on an imbalanced problem — a model that never predicts the rare class can top an accuracy ranking and be useless. Rank on F1, recall, precision or ROC-AUC and the order genuinely changes; a model that cannot produce the chosen metric sorts last rather than being dropped. Ranking stays in the single V35 module, so the leaderboard, the history, the comparison, the report and the auto-inspected model all move together. **(3) Multiclass thresholds**, open since V16, read **one-vs-rest**: pick a class, score it against all the others, same PR and calibration curves. What the panel refuses to imply is a complete multiclass decision rule — two classes can both clear their thresholds and nothing here says which wins, so it says that instead. **(4) An ensemble of the best**: the average of the top three, built from models already fitted, so it costs one pass over the test set. The baseline is never a member (averaging a constant predictor drags the result toward the majority class), members are picked by the same V35 ranking rule, and **probabilistic members are preferred** — a mid-wave measurement showed the ensemble winning on iris with a bare vote because k-NN was in the top three, which silently closed the threshold, calibration and word-effect panels on the champion. **What this wave deliberately does not do**: add a tenth model family (nine is plenty; a tenth improves neither honesty nor understanding), build an AutoML « we handle everything » mode (the opposite of a lab that shows its decisions), or bring in tabular deep learning (high cost, no gain at this scale, and no longer hand-written). 387 unit tests, 69 e2e. | Launched 23/08/2026, right after V35. Each item was a named descope, not an oversight — and the ensemble exposed one more silent-disappearance defect, of the same family as the one V35 found. | | **V37 — delivered** | **ML Lab: speed and the comfort of long sessions.** The wave opened, as the plan demanded, with a measurement — and the measurement moved the wave. **(1) Parallel training** was the headline: the zoo trains sequentially in one worker, so the heavy families were shipped to helper cores. Models cannot cross a worker boundary — `predict` is a closure and structured clone drops functions — so each helper returns `toJSON()` **as a JSON string** and the caller rebuilds through the V22 import path, which makes a parallel model byte-identical to an imported one. That string is not a detail: posting the object instead let structured clone keep shapes JSON drops, and ml-cart's `load()` then rebuilt a tree whose first prediction threw `this.root.classify(...).maxRowIndex is not a function` — a defect that would only have surfaced later, when the user opened insights. Helpers are split by measured cost, heaviest first, to the lightest helper (greedy longest-processing-time); k-NN never leaves the main worker, being the one family with no `toJSON` and also the one that fits in 0 ms; any failure — no Worker support, a helper that throws, an unserialisable family — silently falls back to the sequential trainer, because parallelism may change how long a run takes and never which models it produces. Every family's inference latency is still measured **here**, on the rebuilt predictor: a helper's timing would describe another core under contention, and the column would otherwise read 0 ms for exactly the families that ran in parallel. **(2) The measurement then found the real bottleneck, which was not training at all.** On a 60 000-row run, k-NN inference cost **59.6 s of a 68.8 s wall time — 87% of the whole run** — because the neighbour search allocated 5 000 objects and sorted them for every single prediction, and the scorer asked for labels and then for probabilities, searching twice. Fixed with a bounded top-k insertion over a flat `Float64Array` (ties keep the row seen first, exactly as the stable sort did) and an explicit `predictWithProba` for families where both answers come out of one computation. The old sorted implementation is kept verbatim in the tests as the oracle: the fast path is asserted to predict identically, row for row. **(3) Comparing more than two runs** — V21 compares two; three to six read against the **oldest** of the selection, so the deltas say what the session's changes did rather than what the newest run happens to be. Deliberately not a second diff engine: the matrix is the same V35 ranking and the feature columns are set algebra over the same `summary.featureColumns` V21 reads. **Measured, same machine, same 60 000-row file, same seed** — four arms: | | | k-NN as shipped before | k-NN fixed | @@ -491,6 +491,35 @@ concurrent load, which says nothing about a visitor's WebGPU run. What was measured in isolation is the mask itself — 16 ms at its most expensive step, about 1 ms at every other — and that cost does not shrink on a GPU. +**And the owner's question, answered with the instrument rather than an +estimate.** The bench takes the model as a parameter, so the same 55 questions +were put to **Qwen3-1.7B q4f16 — 1.43 GB, four times the download**, on the same +CPU runtime, with the same prompt and the same constrained decoding: + +| model | download | model alone | app | +| --------------------------- | -------- | -------------- | --------------- | +| Qwen3-0.6B-DQ — **shipped** | 355 MB | **41 / 7 / 7** | **42 / 7 / 6** | +| Qwen3-1.7B | 1.43 GB | 39 / 12 / 4 | 40 / **12** / 3 | + +**Four times the bandwidth buys five MORE wrong answers.** The bigger model is +genuinely better at the questions that are hard for a small one — it reads all +four grouped comparisons (« did women pay more than men? », « survival rate by +sex ») that the 0.6B refused. It is worse at the easy ones: it answers « count +the passengers older than 60 » with `aggregate op=count column=age`, which +counts the rows that HAVE an age rather than the rows, and it reads « average +age of women » as a group-by rather than a filter. Net, on this task, it loses. + +Two limits, stated rather than buried. **The prompt was tuned against the 0.6B** +— these examples, this refusal example, this choice of measure column were all +settled on the small model's failures, and a prompt fitted to the 1.7B would +plausibly score differently; the number above is a property of _this pair_, not +of the model. And it is **one rung**: 764 MB (gemma-3-1b-it) and 1.09 GB +(Llama-3.2-1B) were not tried, both because of licence terms stricter than the +Apache-2.0 the rest of LabML ships under, and because the rung that was tried +went the wrong way. What the measurement does settle is the shape of the +decision: « bigger » is not a direction of improvement here, it is a trade whose +sign has to be measured — and now can be, in one command. + **Ordering**: V38 came before V39 and V40, and for the same reason V35 came first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35, V36, V37 and V30 are delivered. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 is delivered, and it started with a bench for exactly the reason written here: « a bigger model » was not a measurable statement. It is one now — the bench takes the model as a parameter — and V31 still has its own instrument to build. No wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped — V25 (delivered) chose announced sampling and a named memory guard over the typed-array rewrite, which measurement showed unnecessary; widening the diff --git a/README.md b/README.md index 7d3a5ec..cbf0736 100644 --- a/README.md +++ b/README.md @@ -192,6 +192,19 @@ deterministic parser produces **zero wrong answers** on the corpus. The model ha separate on-demand workflow (`.github/workflows/llm-bench.yml`) — it downloads 355 MB and takes minutes, which is not a cost worth adding to every pull request. +The bench takes the model as a parameter, which is how « would a bigger model read +better? » stops being an opinion: + +``` +LABML_LLM_REPO=onnx-community/Qwen3-1.7B-ONNX npm run llm:bench:node +``` + +Measured on the same 55 questions: **Qwen3-1.7B (1.43 GB, four times the download) +scores worse** — 40 right / 12 wrong against 42 / 7 for the 355 MB model that ships. It +reads the grouped comparisons a small model refuses, and misreads simple counts. The +prompt was tuned against the small model, so that number is a property of the pair, not +of the model; see PLAN.md § N for the full result and its limits. + ## Deployment CI builds, tests and deploys on every push: pull requests get a Cloudflare Pages preview,