V30 — Chat : un banc qui tient debout, un parseur qui avoue, un décodage sous grammaire

.github/workflows/llm-bench.yml

@@ -0,0 +1,59 @@
+# V30 — the interpretation bench, on demand.
+#
+# It is NOT part of `ci.yml` on purpose: it downloads 355 MB of weights and
+# spends several minutes answering 55 questions on a CPU, which would add that
+# cost to every pull request for a number that only moves when the chat's
+# prompt, grammar or parser change. Everything about the corpus that does NOT
+# need the model — how the deterministic parser reads all 55 questions, the
+# grammar automaton, the token mask — runs in `ci.yml` on every commit, in
+# seconds (see src/features/ai/llm/corpus.test.ts).
+#
+# Run it from the Actions tab, or let it fire on a change to the chat.
+name: LLM bench
+
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - 'src/features/ai/llm/**'
+      - 'src/features/ai/chat/**'
+      - 'scripts/prepare-llm.mjs'
+
+permissions:
+  contents: read
+
+concurrency:
+  group: llm-bench-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  bench:
+    name: Interpretation bench (CPU)
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+      - run: npm ci
+      # The same pinned files production deploys, laid out flat: onnxruntime-node
+      # reads them from disk and has no 25 MiB limit to work around.
+      - name: Cache the pinned weights
+        id: weights
+        uses: actions/cache@v4
+        with:
+          path: .llm-cache
+          key: llm-cache-qwen3-0.6b-dq-q4f16-v1
+      - if: steps.weights.outputs.cache-hit != 'true'
+        run: npm run llm:fetch
+      - name: Bench, decoding inside the grammar
+        env:
+          LABML_LLM_OUT: bench-report.json
+        run: npm run llm:bench:node
+      - uses: actions/upload-artifact@v4
+        with:
+          name: llm-bench-report
+          path: bench-report.json
+          retention-days: 30

.gitignore

@@ -11,3 +11,6 @@ test-results
 
 # V27: the local language model is fetched at deploy time, never committed.
 public/llm/
+
+# V30: the Node bench keeps a flat copy of the same pinned weights here.
+.llm-cache/

README.md

@@ -98,16 +98,21 @@ The project follows three non-negotiable principles:
   reason: from 1.29 its binaries exceed Cloudflare's 25 MiB per-file limit.
 - **Data assistant** (`/ai/chat`): plain French or English questions about a loaded
   dataset (averages, counts, top-N, correlations…) answered by a deterministic local
-  interpreter — when it does not understand, it says so. A **real local language model**
-  (Qwen3-0.6B, 355 MB, Apache-2.0, self-hosted and split into 24 MiB parts to clear
-  Cloudflare's limit) can be downloaded on explicit consent to read free-form phrasings:
-  it only _translates_ the question into a query — the deterministic engine still
-  computes every number, the translation is validated against a closed grammar, and a
-  badge under each answer names which engine produced it. The deterministic parser reads
-  **first** and is never overridden: it can only name a column that exists and a value
-  that occurs in it, so the model is asked only about the questions it gives up on.
-  WebGPU required; without it the refusal is named and the deterministic interpreter
-  stays fully available.
+  interpreter — when it does not understand, it says so. It only claims to understand
+  once it has read the **whole** question: a word it cannot account for is a refusal, not
+  an answer to a shorter question. A **real local language model** (Qwen3-0.6B, 355 MB,
+  Apache-2.0, self-hosted and split into 24 MiB parts to clear Cloudflare's limit) can be
+  downloaded on explicit consent to read free-form phrasings: it only _translates_ the
+  question into a query — the deterministic engine still computes every number, and a
+  badge under each answer names which engine produced it. The translation is decoded
+  **inside** the query grammar: a hand-written logits processor masks, at every token,
+  everything that would leave the grammar, so an invented column, an operator that does
+  not exist or a category the column does not hold cannot be written in the first place.
+  One shape stays reachable on purpose — `{"kind":"none"}`, the model's way of saying it
+  cannot express the question — because forcing a valid answer turns a refusal into a
+  wrong number. The reading of all 55 reference questions is measured, not asserted: see
+  **Measuring the assistant** below. WebGPU required; without it the refusal is named and
+  the deterministic interpreter stays fully available.
 
 ## Engineering notes
 
@@ -170,6 +175,23 @@ llm:prepare` downloads it into `public/llm/` and splits it into parts under Clou
 else works — the assistant simply falls back to its deterministic interpreter, which is
 the default in any case.
 
+### Measuring the assistant
+
+`src/features/ai/llm/corpus.ts` holds **55 reference questions**, French and English,
+across every shape of the query grammar plus three that no query can answer — where
+refusing is the only correct outcome. Two harnesses run the same corpus:
+
+| Command                                                       | Needs                             | Measures                                                        |
+| ------------------------------------------------------------- | --------------------------------- | --------------------------------------------------------------- |
+| `npm run test` (`corpus.test.ts`)                             | nothing                           | the deterministic parser, the grammar automaton, the token mask |
+| `npm run llm:fetch && npm run llm:bench:node`                 | 355 MB on disk, a few CPU minutes | the real model, end to end                                      |
+| `V27_BENCH=1 npm run build && node scripts/run-llm-bench.mjs` | a GPU with `shader-f16`           | the same, on the shipped WebGPU runtime                         |
+
+The CI half runs on every commit and asserts the number that matters most: the
+deterministic parser produces **zero wrong answers** on the corpus. The model half is a
+separate on-demand workflow (`.github/workflows/llm-bench.yml`) — it downloads 355 MB and
+takes minutes, which is not a cost worth adding to every pull request.
+
 ## Deployment
 
 CI builds, tests and deploys on every push: pull requests get a Cloudflare Pages preview,

e2e/chat.spec.ts

@@ -60,3 +60,37 @@ test('suggestion chips ask a real question and the hub links here', async ({ pag
   await expect(answer).toContainText('Missing cells', { timeout: 15000 });
   await expect(answer).toContainText('deck'); // titanic's famously incomplete column
 });
+
+test('V30 — a question it only half reads is refused, not answered short', async ({ page }) => {
+  await page.goto('/ai/chat');
+  await page.getByRole('button', { name: /titanic\.csv/ }).click();
+  await expect(page.getByText('891 rows · 15 columns')).toBeVisible();
+
+  const input = page.getByLabel('Your question');
+  const ask = page.getByRole('button', { name: 'Ask' });
+
+  // Before V30 this answered 891 — the whole table — because the keyword
+  // grammar knows "how many" and knows nothing about "women", so it kept the
+  // count and dropped the condition. The number was wrong and the badge said
+  // the deterministic interpreter had read the question.
+  await input.fill('how many women?');
+  await ask.click();
+  const refused = page.getByTestId('chat-assistant').last();
+  await expect(refused).toContainText('did not understand', { timeout: 15000 });
+  await expect(refused).not.toContainText('891 rows match');
+
+  // Same shape in French, and the same refusal.
+  await input.fill('average age of women');
+  await ask.click();
+  await expect(page.getByTestId('chat-assistant').last()).toContainText('did not understand', {
+    timeout: 15000,
+  });
+
+  // What it DOES read, it still answers — the guard did not cost the questions
+  // the grammar genuinely understands.
+  await input.fill('How many rows where sex is female?');
+  await ask.click();
+  await expect(page.getByTestId('chat-assistant').last()).toContainText('314 rows match', {
+    timeout: 15000,
+  });
+});

package.json

@@ -13,6 +13,8 @@
     "preview": "vite preview --host 127.0.0.1 --port 4173 --strictPort",
     "llm:prepare": "node scripts/prepare-llm.mjs",
     "llm:bench": "node scripts/run-llm-bench.mjs",
+    "llm:bench:node": "LABML_LLM_BENCH=1 vitest run src/features/ai/llm/bench.node.test.ts",
+    "llm:fetch": "node scripts/prepare-llm.mjs .llm-cache --flat",
     "lint": "eslint .",
     "format": "prettier --write .",
     "format:check": "prettier --check .",

scripts/prepare-llm.mjs

@@ -6,7 +6,14 @@
  * weights are NOT committed: 355 MB of binaries would slow every clone and
  * every CI checkout, for a model only the opt-in chat engine downloads.
  *
- *   node scripts/prepare-llm.mjs <outDir>      # default: dist/llm
+ *   node scripts/prepare-llm.mjs <outDir>          # default: dist/llm
+ *   node scripts/prepare-llm.mjs <outDir> --flat  # unsharded, for the Node bench
+ *
+ * V30 — `--flat` writes the same pinned files without splitting them, which is
+ * what `npm run llm:bench:node` needs: onnxruntime-node reads the weights from
+ * disk directly and has no 25 MiB limit to work around. It is the same
+ * download, the same size checks, and the same revision as the deployed copy —
+ * so the bench measures the model production actually ships, not a lookalike.
  *
  * Every file is checked against the byte sizes pinned below. A mismatch is a
  * hard failure: shipping a truncated model would fail in the browser, later,
@@ -72,15 +79,17 @@ async function write(outDir, relative, bytes) {
 }
 
 async function main() {
-  const outDir = process.argv[2] ?? 'dist/llm';
+  const args = process.argv.slice(2).filter((a) => a !== '--flat');
+  const flat = process.argv.includes('--flat');
+  const outDir = args[0] ?? 'dist/llm';
   const root = join(outDir, REPO);
   const files = [];
   let totalBytes = 0;
 
   for (const path of [...Object.keys(FILES), 'LICENSE']) {
     const bytes = await download(path);
     totalBytes += bytes.byteLength;
-    if (bytes.byteLength <= SHARD_BYTES) {
+    if (flat || bytes.byteLength <= SHARD_BYTES) {
       await write(root, path, bytes);
       console.log(`  entier  ${path} (${(bytes.byteLength / 1e6).toFixed(1)} Mo)`);
       continue;
@@ -103,6 +112,14 @@ async function main() {
   await write(outDir, 'manifest.json', JSON.stringify(manifest, null, 2));
   console.log(`\nmanifeste écrit — ${(totalBytes / 1e6).toFixed(0)} Mo au total`);
 
+  // The 25 MiB guard below is about Cloudflare Pages. A flat copy never goes
+  // there — it is read from local disk by the bench — so the guard would fail
+  // on a layout that is correct for its purpose.
+  if (flat) {
+    console.log('copie à plat (banc Node) — garde des 25 Mio sans objet ✓');
+    return;
+  }
+
   // A last guard: nothing we just wrote may exceed the platform limit.
   const oversized = [];
   const walk = async (dir) => {

src/features/ai/chat/EnginePicker.tsx

@@ -16,6 +16,7 @@ export function EnginePicker() {
   const { t, i18n } = useTranslation();
   const lang = i18n.resolvedLanguage ?? 'en';
   const llmStatus = useChatStore((s) => s.llmStatus);
+  const llmConstrained = useChatStore((s) => s.llmConstrained);
   const llmBytes = useChatStore((s) => s.llmBytes);
   const llmProgress = useChatStore((s) => s.llmProgress);
   const llmError = useChatStore((s) => s.llmError);
@@ -121,12 +122,19 @@ export function EnginePicker() {
       )}
 
       {llmStatus === 'ready' && (
-        <p className="text-xs text-muted">
-          <Badge variant="outline" className="mr-2 text-[0.62rem]">
-            {t('ai.chat.engine.readyTag')}
-          </Badge>
-          {t('ai.chat.engine.readyNote')}
-        </p>
+        <div className="space-y-2">
+          <p className="text-xs text-muted" data-testid="llm-constrained">
+            <Badge variant="outline" className="mr-2 text-[0.62rem]">
+              {t(llmConstrained ? 'ai.chat.engine.constrainedTag' : 'ai.chat.engine.readyTag')}
+            </Badge>
+            {t(
+              llmConstrained
+                ? 'ai.chat.engine.constrainedNote'
+                : 'ai.chat.engine.unconstrainedNote',
+            )}
+          </p>
+          <p className="text-xs text-muted">{t('ai.chat.engine.readyNote')}</p>
+        </div>
       )}
 
       {llmStatus === 'failed' && (

src/features/ai/chat/chat-store.ts

@@ -42,6 +42,12 @@ interface ChatState {
   llmBytes: number;
   llmProgress: { loaded: number; total: number } | null;
   llmError: string | null;
+  /**
+   * V30 — true when the model's answer is decoded INSIDE the query grammar.
+   * Shown rather than assumed: if the tokenizer does not expose its vocabulary
+   * the guard cannot run, and the badge must not claim it did.
+   */
+  llmConstrained: boolean;
   /** The user's choice of interpreter; deterministic stays the default. */
   engine: ChatEngine;
   loadFile: (file: File) => void;
@@ -71,6 +77,7 @@ const initialState = {
   llmBytes: 0,
   llmProgress: null,
   llmError: null,
+  llmConstrained: false,
   engine: 'deterministic' as ChatEngine,
 };
 
@@ -107,7 +114,12 @@ export const useChatStore = create<ChatState>((set, get) => {
         } else if (message.kind === 'llm-progress') {
           set({ llmProgress: { loaded: message.loaded, total: message.total } });
         } else if (message.kind === 'llm-ready') {
-          set({ llmStatus: 'ready', llmProgress: null, engine: 'llm' });
+          set({
+            llmStatus: 'ready',
+            llmProgress: null,
+            engine: 'llm',
+            llmConstrained: message.constrained,
+          });
         } else if (message.kind === 'llm-failed') {
           set({ llmStatus: 'failed', llmProgress: null, llmError: message.reason });
         } else if (message.kind === 'error') {

src/features/ai/chat/chat.worker.ts

@@ -31,7 +31,7 @@ export type ChatWorkerResponse =
   | { kind: 'unknown'; by: 'none' | 'none-both' }
   | { kind: 'llm-capability'; available: boolean; webgpu: boolean; totalBytes: number }
   | { kind: 'llm-progress'; loaded: number; total: number }
-  | { kind: 'llm-ready' }
+  | { kind: 'llm-ready'; constrained: boolean }
   /** Named refusal: 'no-manifest' | 'no-webgpu' | anything the loader threw. */
   | { kind: 'llm-failed'; reason: string }
   | { kind: 'error'; message: string };
@@ -72,21 +72,21 @@ function resetState() {
   columnInfo = [];
 }
 
+/** Enough to tell a 0/1 flag from a quantity; counting further changes nothing. */
+const DISTINCT_CAP = 12;
+
 function buildColumnInfo(): ColumnInfo[] {
   return header.map((name, i) => {
     const type = inferColumnType(name, columns[i]);
     const isNumeric = type === 'numeric' || type === 'id';
-    let values: string[] = [];
-    if (!isNumeric) {
-      const distinct = new Set<string>();
-      for (const cell of columns[i]) {
-        if (isMissing(cell)) continue;
-        distinct.add((cell as string).trim());
-        if (distinct.size > MAX_VALUES) break;
-      }
-      if (distinct.size <= MAX_VALUES) values = [...distinct].sort();
+    const distinct = new Set<string>();
+    for (const cell of columns[i]) {
+      if (isMissing(cell)) continue;
+      distinct.add((cell as string).trim());
+      if (distinct.size > Math.max(MAX_VALUES, DISTINCT_CAP)) break;
     }
-    return { name, isNumeric, values };
+    const values = !isNumeric && distinct.size <= MAX_VALUES ? [...distinct].sort() : [];
+    return { name, isNumeric, values, distinct: distinct.size };
   });
 }
 
@@ -173,7 +173,7 @@ self.onmessage = async (event: MessageEvent<ChatWorkerRequest>) => {
         model = await loadModel(capability.manifest, {
           onProgress: ({ loaded, total }) => post({ kind: 'llm-progress', loaded, total }),
         });
-        post({ kind: 'llm-ready' });
+        post({ kind: 'llm-ready', constrained: model.constrained });
       } catch (error) {
         model = null;
         post({

src/features/ai/chat/parser.test.ts

@@ -176,17 +176,16 @@ describe('a condition it cannot read is REFUSED, never dropped', () => {
     expect(parseQuestion('average age for kids born before 1900', columns, 'en')).toBeNull();
   });
 
-  // The guard targets DROPPED conditions, not guessed columns. Here the
-  // condition IS applied — to `fare`, the only numeric column named — which
-  // also happens to be the legitimate reading of "average fare below 12".
-  // Refusing this would cost a real question to catch a fuzzy one.
-  it('keeps a condition it can attach, even when the wording is loose', () => {
-    expect(parseQuestion('average fare for kids below 12', columns, 'en')).toMatchObject({
-      kind: 'aggregate',
-      op: 'mean',
-      column: 'fare',
-      filter: { column: 'fare', op: '<', value: 12 },
-    });
+  // V30 reverses this case, deliberately. It used to assert that the parser
+  // ANSWERS here, on the grounds that `fare < 12` is « also a legitimate
+  // reading » — but the question says *kids*, and the only reading that word
+  // supports is `age < 12`. Attaching the threshold to the nearest numeric
+  // column is a guess, and the guess is delivered under the deterministic
+  // badge, which is supposed to mean exact. Refusing hands the question to the
+  // model, which reads it; answering hands the user a number for a question
+  // nobody asked.
+  it('refuses when the condition is attached to a column the wording never named', () => {
+    expect(parseQuestion('average fare for kids below 12', columns, 'en')).toBeNull();
   });
 
   it('still answers the same question when the condition names a real column', () => {