test: evals per-case QA tests and inlined suites (#16424)

# Overview

Refactors the eval suite to surface per-question pass/fail in vitest
output and reduces boilerplate by inlining suite registration into the
spec files.

## Key Changes

- **Per-question test reporting**
- QA datasets previously batched all questions into a single `it()` that
asserted on aggregate accuracy. Each question now registers as its own
`it()` block inside `describe.concurrent`, so vitest reports pass/fail
per question and `-t` filtering targets individual cases.
- Extracted `runQACase` from `runDataset` for per-case invocation.
`runDataset` still exists as a wrapper that maps over cases and
aggregates accuracy for the snapshot output.

- **Inlined suite registration**
- The 10 `register*Suite()` functions in `suites/` were thin wrappers
around two patterns. Replaced with `registerQACases` and
`registerCodegenCases` helpers in `suites/helpers.ts`; each spec file
calls these directly with its dataset and label.
- Deleted the per-suite files. The `suites/` directory now contains only
`helpers.ts` and `types.ts`.
- Dropped redundant nested wrappers in graphql/local-api/rest-api specs
(single-child `describe('Collections')` / `describe('CRUD')`).

- **API key check in globalSetup**
- The duplicated `beforeAll` API key guard from 10 spec files moved into
`globalSetup.setup()`. Spec files no longer import `beforeAll` or repeat
the check.

- **Richer failure output**
- `failureMessage` and `caseFailureMessage` now include the model answer
and scorer reasoning inline in the vitest assertion message.
- Switched all `assert(condition, message)` calls in suites to
`expect(value, message).toBe(...)` for native vitest reporter
integration.

- **Path alias support in fixture type-checking**
- Added `fixtures/ambient.d.ts` with wildcard module declarations for
`@/*` and `~/*`. The per-invocation tsconfig in `validate.ts` includes
this file so LLM-generated configs that import from common path aliases
type-check structurally without requiring real stub files per fixture.

- **Removed unused ACCURACY_THRESHOLD**
- The dataset-level accuracy gate is no longer applied since each case
asserts independently. `SCORE_THRESHOLD` (per-case pass criterion for
the LLM judge) remains.

- **Cleanup**
- Removed redundant `dotenv.config()` from `runner/systemPrompts.ts`
(vitest setup already loads `.env`).
  - Switched bare `fs`/`path` imports to `node:` prefixed.

## Design Decisions

- **Per-case over batched assertions**: The batched `it()` masked
individual failures behind aggregate accuracy. With 30+ questions per
dataset, the visibility tradeoff favored splitting. Each case now
asserts independently and aggregate metrics are computed at
snapshot/reporting time rather than in test code.

- **Inlining over indirection**: With variant selection moved to a
runtime env var (`EVAL_VARIANT` via `resolveVariantOptions()`), each
spec calls register once. The `register*Suite()` function abstraction
added a layer with no remaining purpose. Inlining puts the suite
definition (name, datasets, structure) in one file the reader can scan
top to bottom.

- **`expect` over `assert`**: Both work, but `expect` integrates with
vitest's reporter so the custom failure message is rendered alongside
the standard test output rather than as a raw `AssertionError`.

- **Ambient module declarations over per-fixture stub files**: LLMs
commonly generate imports from `@/components/...` paths. Maintaining
real stub files per import path would not scale. Wildcard module
declarations let any `@/*` import resolve to `any`, which is sufficient
for structural type-checking.

## Overall Flow

```mermaid
flowchart TB
    subgraph Before
      A1[spec file]
      A1 --> A2[register Suite function]
      A2 --> A3[runDataset]
      A3 --> A4[Promise.all over cases]
      A4 --> A5[Single it asserts aggregate accuracy]
    end

    subgraph After
      B1[spec file]
      B1 --> B2[registerQACases helper]
      B2 --> B3[describe.concurrent]
      B3 --> B4[it per case]
      B4 --> B5[runQACase]
      B5 --> B6[expect with rich message]
    end
```

Author: denolfe

test/evals/datasets/config/codegen.ts

@@ -1,14 +1,6 @@
 import type { CodegenEvalCase } from '../../types.js'
 
 export const configCodegenDataset: CodegenEvalCase[] = [
-  {
-    input:
-      'Add the SEO plugin from "@payloadcms/plugin-seo" to the config. Pass a generateTitle function that returns a string combining the document title and the site name "Acme Corp".',
-    expected:
-      'import seoPlugin from "@payloadcms/plugin-seo", seoPlugin() added to plugins array, generateTitle function returning a string that includes the doc title and "Acme Corp"',
-    category: 'config',
-    fixturePath: 'config/codegen/seo-plugin',
-  },
   {
     input:
       'Configure the admin panel to use a custom Logo component imported from "@/components/Logo" and set the admin meta titleSuffix to " | My CMS".',

test/evals/eval.building-plugins.spec.ts

@@ -1,12 +1,14 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerBuildingPluginsSuite } from './suites/index.js'
+import { pluginsCodegenDataset } from './datasets/plugins/codegen.js'
+import { pluginsQADataset } from './datasets/plugins/qa.js'
+import { registerCodegenCases, registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerBuildingPluginsSuite(resolveVariantOptions())
+describe(`Building Plugins${labelSuffix}`, () => {
+  registerQACases(pluginsQADataset, 'Building Plugins: QA', options)
+  registerCodegenCases(pluginsCodegenDataset, 'Building Plugins: Codegen', options)
+})

test/evals/eval.collections.spec.ts

@@ -1,12 +1,14 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerCollectionsSuite } from './suites/index.js'
+import { collectionsCodegenDataset } from './datasets/collections/codegen.js'
+import { collectionsQADataset } from './datasets/collections/qa.js'
+import { registerCodegenCases, registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerCollectionsSuite(resolveVariantOptions())
+describe(`Collections${labelSuffix}`, () => {
+  registerQACases(collectionsQADataset, 'Collections: QA', options)
+  registerCodegenCases(collectionsCodegenDataset, 'Collections: Codegen', options)
+})

test/evals/eval.config.spec.ts

@@ -1,12 +1,14 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerConfigSuite } from './suites/index.js'
+import { configCodegenDataset } from './datasets/config/codegen.js'
+import { configQADataset } from './datasets/config/qa.js'
+import { registerCodegenCases, registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerConfigSuite(resolveVariantOptions())
+describe(`Config${labelSuffix}`, () => {
+  registerQACases(configQADataset, 'Config: QA', options)
+  registerCodegenCases(configCodegenDataset, 'Config: Codegen', options)
+})

test/evals/eval.conventions.spec.ts

@@ -1,12 +1,12 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerConventionsSuite } from './suites/index.js'
+import { conventionsQADataset } from './datasets/conventions/qa.js'
+import { registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerConventionsSuite(resolveVariantOptions())
+describe(`Conventions${labelSuffix}`, () => {
+  registerQACases(conventionsQADataset, 'Conventions: QA', options)
+})

test/evals/eval.fields.spec.ts

@@ -1,12 +1,14 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerFieldsSuite } from './suites/index.js'
+import { fieldsCodegenDataset } from './datasets/fields/codegen.js'
+import { fieldsQADataset } from './datasets/fields/qa.js'
+import { registerCodegenCases, registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerFieldsSuite(resolveVariantOptions())
+describe(`Fields${labelSuffix}`, () => {
+  registerQACases(fieldsQADataset, 'Fields: QA', options)
+  registerCodegenCases(fieldsCodegenDataset, 'Fields: Codegen', options)
+})

test/evals/eval.graphql.spec.ts

@@ -1,12 +1,12 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerGraphQLSuite } from './suites/index.js'
+import { graphqlCollectionsQADataset } from './datasets/graphql/collections/qa.js'
+import { registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerGraphQLSuite(resolveVariantOptions())
+describe(`GraphQL${labelSuffix}`, () => {
+  registerQACases(graphqlCollectionsQADataset, 'GraphQL: Collections QA', options)
+})

test/evals/eval.local-api.spec.ts

@@ -1,12 +1,12 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerLocalApiSuite } from './suites/index.js'
+import { localApiCollectionsQADataset } from './datasets/local-api/collections/qa.js'
+import { registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerLocalApiSuite(resolveVariantOptions())
+describe(`Local API${labelSuffix}`, () => {
+  registerQACases(localApiCollectionsQADataset, 'Local API: Collections QA', options)
+})

test/evals/eval.negative.spec.ts

@@ -1,12 +1,29 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerNegativeSuite } from './suites/index.js'
+import {
+  negativeCorrectionCodegenDataset,
+  negativeInvalidInstructionDataset,
+} from './datasets/negative/codegen.js'
+import { negativeQADataset } from './datasets/negative/qa.js'
+import { registerCodegenCases, registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerNegativeSuite(resolveVariantOptions())
+describe(`Negative Tests${labelSuffix}`, () => {
+  registerQACases(negativeQADataset, 'Negative: Detection', {
+    ...options,
+    systemPromptKeyOverride: 'configReview',
+  })
+  registerCodegenCases(negativeCorrectionCodegenDataset, 'Negative: Correction', {
+    ...options,
+    groupName: 'Correction: Codegen',
+  })
+  registerCodegenCases(negativeInvalidInstructionDataset, 'Negative: Invalid Instruction', {
+    ...options,
+    expectPass: false,
+    groupName: 'Invalid Instruction: Codegen',
+    testNamePrefix: 'should fail: ',
+  })
+})

test/evals/eval.official-plugins.spec.ts

@@ -1,12 +1,14 @@
-import { beforeAll } from 'vitest'
+import { describe } from 'vitest'
 
-import { registerOfficialPluginsSuite } from './suites/index.js'
+import { pluginsOfficialCodegenDataset } from './datasets/plugins/official/codegen.js'
+import { pluginsOfficialQADataset } from './datasets/plugins/official/qa.js'
+import { registerCodegenCases, registerQACases } from './suites/helpers.js'
 import { resolveVariantOptions } from './variantOptions.js'
 
-beforeAll(() => {
-  if (!process.env.OPENAI_API_KEY) {
-    throw new Error('OPENAI_API_KEY must be set to run eval tests')
-  }
-})
+const options = resolveVariantOptions()
+const { labelSuffix = '' } = options
 
-registerOfficialPluginsSuite(resolveVariantOptions())
+describe(`Official Plugins${labelSuffix}`, () => {
+  registerQACases(pluginsOfficialQADataset, 'Official Plugins: QA', options)
+  registerCodegenCases(pluginsOfficialCodegenDataset, 'Official Plugins: Codegen', options)
+})