test(uat): human-UAT fixtures + deterministic CI smoke

.github/workflows/ci.yml

@@ -64,6 +64,9 @@ jobs:
       - name: Package dry-run
         run: pnpm --filter @submuxhq/codedecay pack --dry-run
 
+      - name: Human UAT deterministic smoke
+        run: node scripts/human-uat-smoke.mjs --run-id ci
+
       - name: Child repository end-to-end acceptance
         run: node scripts/child-repo-e2e.mjs --run-id ci
 

docs/evals/uat-kit/README.md

@@ -1,7 +1,8 @@
-# Human UAT kit (scaffolding for #692)
+# Human UAT kit (issue 692)
 
-This kit is scaffolding only. **#692 cannot close without independent human
-participants.** Do not treat agent role-play as human acceptance.
+This kit prepares independent human acceptance testing. **Issue 692 cannot finish
+without independent human participants.** Do not treat agent role-play or the
+deterministic CI smoke as human acceptance evidence.
 
 ## Contents
 
@@ -10,18 +11,26 @@ participants.** Do not treat agent role-play as human acceptance.
 | `participant-script.md` | Task sequence for UAT-HUMAN-1..8 |
 | `observer-rubric.md` | Scoring for comprehension and safety |
 | `consent-privacy.md` | Consent + no telemetry / no secret collection |
+| `facilitator-runbook.md` | How to schedule and run sessions |
+| `fixtures.md` | Planted / decoy / unsafe fixture guide |
+| `tasks.json` | Machine-readable task IDs |
 | `result.schema.json` | Machine-readable participant result schema |
 | `summary.template.md` | Anonymized Markdown summary template |
 
 ## How to run (when humans are scheduled)
 
 1. Pack/publish `@submuxhq/codedecay` and install into a fresh environment.
-2. Give participants only public docs + this kit (no maintainer walkthrough).
-3. Use synthetic fixtures under `scripts/fixtures/end-user-demo/` / new kit fixtures.
+2. Materialize fixtures: `node scripts/human-uat-setup.mjs`
+3. Give participants only public docs + this kit (no maintainer walkthrough).
 4. Record results with `result.schema.json`.
 5. File linked issues for every release-blocking usability failure.
 
 ## CI smoke
 
-`pnpm test` / kit path checks may validate that these files exist and schema
-parses. That smoke is **not** human evidence.
+```bash
+pnpm build:packages
+pnpm test:human-uat-smoke
+```
+
+Smoke validates kit files, fixture oracles, planted vs decoy analyze signal, and
+`allowCommands: false` execute skipping. That smoke is **not** human evidence.

docs/evals/uat-kit/facilitator-runbook.md

@@ -0,0 +1,39 @@
+# Facilitator runbook (human UAT)
+
+This runbook is for scheduling **independent human** sessions for issue 692.
+Deterministic smoke (`pnpm test:human-uat-smoke`) is **not** a substitute.
+
+## Before the session
+
+1. Confirm consent: `consent-privacy.md`
+2. Pack or install published `@submuxhq/codedecay` into a fresh environment (never workspace-only imports for the participant).
+3. Materialize fixtures:
+
+```bash
+node scripts/human-uat-setup.mjs
+```
+
+4. Give the participant only: public README/docs, this kit, and the fixture paths.
+5. Do not explain internal package architecture or preconfigure maintainer state.
+
+## During
+
+- Follow `participant-script.md` (`UAT-HUMAN-1`..`8`).
+- Score with `observer-rubric.md`.
+- Fail the session immediately if agent text is treated as proof or unverified as merge-safe.
+- Track install/auth/docs friction separately from analysis quality.
+
+## After
+
+1. Fill `result.schema.json` (one file per participant).
+2. Fill `summary.template.md` (anonymized).
+3. Open linked focused issues for every release-blocking usability failure.
+4. Store sanitized artifacts under `.codedecay/local/human-uat/results/` (gitignored local only unless explicitly reviewed for publication).
+
+## Roles target
+
+At least three participants who did not implement the feature:
+
+- AI-assisted individual developer
+- Experienced software engineer
+- Team/DevOps or platform-oriented user (when available)

docs/evals/uat-kit/fixtures.md

@@ -0,0 +1,32 @@
+# Human UAT fixtures
+
+Synthetic repositories used by the kit and the deterministic CI smoke.
+
+## Materialize
+
+```bash
+node scripts/human-uat-setup.mjs
+# default output: .codedecay/local/human-uat/fixtures/
+```
+
+| Fixture | Purpose | Task IDs |
+|---|---|---|
+| `planted/` | Auth/API regression with passing weak unit test | UAT-HUMAN-2..5, UAT-HUMAN-8 |
+| `decoy/` | Docs-only clean change | UAT-HUMAN-7 |
+| `unsafe/` | Configured commands with `allowCommands: false` | UAT-HUMAN-6 |
+
+## Planted oracle
+
+- `npm test` passes (shallow session unit test).
+- `npm run probe:anonymous` fails until repaired (`GET /api/invoices` must be `401`).
+- Ambiguous requirement text lives in `README.md` for clarification practice.
+
+## Deterministic smoke
+
+```bash
+pnpm build:packages
+pnpm test:human-uat-smoke
+```
+
+Smoke proves fixture + CLI workflow drift only. It sets `humanEvidence: false` and
+must never be used to finish human acceptance for issue 692.

docs/evals/uat-kit/participant-script.md

@@ -1,10 +1,13 @@
 # Participant script (UAT-HUMAN-1..8)
 
-1. Install the published package; discover the primary AI workflow. (`UAT-HUMAN-1`)
-2. Supply a requirement with an ambiguity; resolve it. (`UAT-HUMAN-2`)
-3. Note a weak test that does not prove the production path. (`UAT-HUMAN-3`)
-4. Run an approved behavioral check that finds a planted defect. (`UAT-HUMAN-4`)
-5. Repair with your agent and revalidate the current tree. (`UAT-HUMAN-5`)
-6. Attempt an unsafe command/target; confirm it is blocked and understood. (`UAT-HUMAN-6`)
-7. Confirm a clean decoy does not force unnecessary repair. (`UAT-HUMAN-7`)
-8. Explain final evidence vs limitations (deterministic / tool / memory / AI / unverified / needs-human / verified). (`UAT-HUMAN-8`)
+Use the published package (or a packed tarball install), not a monorepo workspace
+import. Fixture setup for facilitators: `node scripts/human-uat-setup.mjs`.
+
+1. **UAT-HUMAN-1** — Install `@submuxhq/codedecay`, run `codedecay --help` / `codedecay ai --help`, and produce a first useful analyze or ai report against the planted fixture.
+2. **UAT-HUMAN-2** — Read the ambiguous requirement in the planted `README.md` (“Make invoice lookup safer for operators.”). Clarify whether anonymous callers may see invoice totals before coding.
+3. **UAT-HUMAN-3** — Observe that `npm test` passes while the changed auth/API path lacks endpoint-level proof.
+4. **UAT-HUMAN-4** — Run `npm run probe:anonymous` (or CodeDecay with configured checks) and confirm the planted defect (anonymous `GET /api/invoices` is not `401`).
+5. **UAT-HUMAN-5** — Repair with your user-owned agent (or deterministic edits), add real-path proof, rerun checks, and `codedecay revalidate` / current-tree `codedecay ai`.
+6. **UAT-HUMAN-6** — On the `unsafe` fixture (`allowCommands: false`), run `codedecay execute` and confirm commands are skipped/blocked; explain why that is correct.
+7. **UAT-HUMAN-7** — On the `decoy` fixture, confirm a docs-only change does not force unnecessary repair.
+8. **UAT-HUMAN-8** — Explain what in the final report is deterministic evidence, runtime/tool proof, memory, AI suggestion, unverified, needs-human, and verified — without treating agent text as proof.

docs/evals/uat-kit/tasks.json

@@ -0,0 +1,48 @@
+{
+  "schemaVersion": 1,
+  "kitId": "codedecay-human-uat",
+  "humanEvidence": false,
+  "note": "Machine-readable task IDs for facilitators and CI smoke. Real human sessions are still required.",
+  "tasks": [
+    {
+      "id": "UAT-HUMAN-1",
+      "title": "Fresh install and first useful result",
+      "fixture": "published-package"
+    },
+    {
+      "id": "UAT-HUMAN-2",
+      "title": "Ambiguous requirement and clarification",
+      "fixture": "planted"
+    },
+    {
+      "id": "UAT-HUMAN-3",
+      "title": "Weak test exists but does not prove the production path",
+      "fixture": "planted"
+    },
+    {
+      "id": "UAT-HUMAN-4",
+      "title": "Approved behavioral experiment finds the planted defect",
+      "fixture": "planted"
+    },
+    {
+      "id": "UAT-HUMAN-5",
+      "title": "Agent repair plus current-tree revalidation",
+      "fixture": "planted"
+    },
+    {
+      "id": "UAT-HUMAN-6",
+      "title": "Unsafe command or external target is blocked and correctly understood",
+      "fixture": "unsafe"
+    },
+    {
+      "id": "UAT-HUMAN-7",
+      "title": "Clean decoy produces no unnecessary repair",
+      "fixture": "decoy"
+    },
+    {
+      "id": "UAT-HUMAN-8",
+      "title": "Participant explains final evidence and limitations accurately",
+      "fixture": "planted"
+    }
+  ]
+}

package.json

@@ -21,6 +21,7 @@
     "eval:benchmark": "vitest run packages/cli/test/benchmark-corpus.test.ts",
     "eval:pr-safety": "pnpm build:packages && node scripts/pr-safety-eval.mjs",
     "eval:agent-efficacy": "pnpm build:packages && node scripts/agent-efficacy-eval.mjs",
+    "test:human-uat-smoke": "pnpm build:packages && node scripts/human-uat-smoke.mjs",
     "gen:launch": "node scripts/gen-launch.mjs",
     "judge-lab:evidence": "pnpm build:packages && node scripts/generate-judge-lab-evidence.mjs",
     "judge-lab:evidence:check": "node scripts/generate-judge-lab-evidence.mjs --check",

packages/knowledge/test/human-uat-kit.test.ts

@@ -3,6 +3,16 @@ import { join } from "node:path";
 import { describe, expect, it } from "vitest";
 
 const kitRoot = join(process.cwd(), "docs/evals/uat-kit");
+const expectedTaskIds = [
+  "UAT-HUMAN-1",
+  "UAT-HUMAN-2",
+  "UAT-HUMAN-3",
+  "UAT-HUMAN-4",
+  "UAT-HUMAN-5",
+  "UAT-HUMAN-6",
+  "UAT-HUMAN-7",
+  "UAT-HUMAN-8"
+];
 
 describe("human UAT kit scaffolding (#692)", () => {
   it("keeps the versioned kit files required before human sessions", () => {
@@ -11,6 +21,9 @@ describe("human UAT kit scaffolding (#692)", () => {
       "participant-script.md",
       "observer-rubric.md",
       "consent-privacy.md",
+      "facilitator-runbook.md",
+      "fixtures.md",
+      "tasks.json",
       "result.schema.json",
       "summary.template.md"
     ]) {
@@ -22,6 +35,19 @@ describe("human UAT kit scaffolding (#692)", () => {
     expect(schema.required).toEqual(
       expect.arrayContaining(["schemaVersion", "participantRole", "tasks", "trustComprehension"])
     );
-    expect(readFileSync(join(kitRoot, "README.md"), "utf8")).toMatch(/cannot close without independent human/i);
+    expect(readFileSync(join(kitRoot, "README.md"), "utf8")).toMatch(
+      /cannot\s+finish\s+without\s+independent\s+human|cannot\s+close\s+without\s+independent\s+human/i
+    );
+  });
+
+  it("publishes machine-readable UAT-HUMAN task IDs with humanEvidence false", () => {
+    const tasks = JSON.parse(readFileSync(join(kitRoot, "tasks.json"), "utf8")) as {
+      schemaVersion: number;
+      humanEvidence: boolean;
+      tasks: Array<{ id: string }>;
+    };
+    expect(tasks.schemaVersion).toBe(1);
+    expect(tasks.humanEvidence).toBe(false);
+    expect(tasks.tasks.map((task) => task.id)).toEqual(expectedTaskIds);
   });
 });