From a5feb32a992db70c788234449548fafe77133636 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Mon, 7 Sep 2026 17:02:08 +0530 Subject: [PATCH 01/17] Redact every audit sink, and read every subagent transcript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects in the audit, both of the same class: the system reported success while doing nothing. `redactExample` had exactly two non-definition call sites and both were in `harm-report.ts` — the emailed digest was the only renderer connected to the one module written to keep secrets out of what leaves the machine. `formatMarkdown` wrote examples and cwds verbatim into `./failproofai-audit.md`, which the CLI prints as "Shareable report" and which defaults to the current directory; `formatJson` was a bare stringify of the whole AuditResult. Every renderer now redacts. The two that travel use the full pipeline; the terminal uses a new `maskSecretsOnly`, because a credential on screen is one screenshot from being published while `~/…/db.ts` protects nobody from their own directory names. `redactAuditResult` returns a new object, so the dashboard and cache keep the values they render locally. `listClaudeTranscripts` walked only the direct children of `/subagents/`, which matched the layout Claude shipped when it was written and became wrong once workflow runs began nesting agents one level deeper. Measured on a real machine: 1,839 transcripts on disk, 160 opened — 8.7%, reported as though it had read everything, with five of the seven files holding a credential-bearing egress command in the part it could not see. The walk is now recursive to a bounded depth and skips symlinks. Subagent ids are qualified by parent session and path, because basenames are not unique down there. Asserting uniqueness over the real corpus caught a collision reasoning had missed: a workflow run id is reused when its session is resumed, so `wf_/journal.jsonl` exists under two parents in one project. sessionId keys example attribution and per-session detector state, so the merge would have been silent. Top-level ids are unchanged. 15 tests, including the collision as a regression case and a structural tripwire on the redactor import — the defect was never a bad mask, it was a mask nobody called. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 4 + __tests__/audit/redaction-sinks.test.ts | 137 ++++++++++++++++++ .../lib/claude-sessions-subagents.test.ts | 114 +++++++++++++++ lib/claude-sessions.ts | 128 ++++++++++++---- src/audit/redact-example.ts | 62 +++++++- src/audit/report.ts | 9 +- 6 files changed, 423 insertions(+), 31 deletions(-) create mode 100644 __tests__/audit/redaction-sinks.test.ts create mode 100644 __tests__/lib/claude-sessions-subagents.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 9282faee5..0b59c4f30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,10 @@ ### Fixes - The two PyPI publish workflows open the next version's `CHANGELOG` section in the same `bump` commit that moves `_version.py`, via a new `scripts/changelog-open.py`. `bump` used to move the version alone, leaving `main` on a version with no section — the exact state `scripts/changelog-section.py` refuses at release time and `__tests__/ci/python-version-pipeline.test.ts` asserts against. Because bump commits carry a skip-ci marker, that never went red on itself: it went red on the next unrelated PR to run CI, which is how `main` broke after the 0.0.1b1 publish (repaired by hand in #755, which named the recurrence and left it) and again after 0.0.1b2. `sdk/python/CHANGELOG.md` gets the 0.0.1b3 section that was missing. The opener is idempotent — a re-run of `bump` against a `main` that already carries the section is a no-op rather than a second heading, which would put only the first one's body on the GitHub Release — and it matches the version with the same trailing word boundary the extractor uses, so opening `0.0.1b1` is not satisfied by an existing `0.0.1b10` (#787) +- The audit stops writing your credentials into the file it calls a shareable report. `redactExample` had exactly two non-definition call sites and both were in `harm-report.ts`, the emailed digest — so the ONE path that was already careful was the only one connected. `formatMarkdown` wrote `example` and `cwd` verbatim into `./failproofai-audit.md`, which the CLI prints as `Shareable report` and which defaults to the current directory, i.e. a git working tree; `formatJson` was a bare `JSON.stringify` of the whole `AuditResult`, examples, per-example cwds, scanned project paths and all; and the terminal renderer printed the raw example too. Both artifacts exist to be sent somewhere. Every renderer now redacts: the two that travel through the full `redactExample` (masked secrets, shortened home paths) and, for the JSON, a new `redactAuditResult` that walks the examples, their cwds, `projectsScanned` and `scope.projects` and returns a new object so the dashboard and the cache keep the values they need to render locally. The terminal gets a new `maskSecretsOnly` instead — a credential on screen is one screenshot or one pasted issue from being published, while `~/…/db.ts` protects nobody from their own directory names and costs the example its most useful half. `redactExample` is now defined in terms of `maskSecretsOnly`, so the two cannot drift. A test asserts the import is still there, because the defect was never a bad mask — it was a mask nobody called (#PR) + +- The audit reads every subagent transcript instead of 8.7% of them. `listClaudeTranscripts` walked only the DIRECT children of `/subagents/`, which matched the layout Claude shipped when it was written and became wrong the day workflow runs started nesting their agents one level further down at `subagents/workflows//`. On the machine this was found on that is 1,839 transcripts on disk, 1,741 of them under `subagents/`, 1,679 of those nested — so the scan opened 160 files and reported the result as though it had read everything, which is the one failure a scanner must not have. Five of the seven files holding a genuine credential-bearing egress command were in the part it could not see. The walk is now recursive to a bounded depth, skipping symlinks so an unexpected layout costs a bounded walk rather than a scan that never returns. Subagent session ids are now qualified by their parent session and their path below `subagents/` (`__workflows__wf_123__agent-abc`), because basenames are not unique down there — every workflow run writes a `journal.jsonl`, and a run id is reused when its session is resumed, so the same relative path exists under two parents in one project. Both collisions were found by asserting uniqueness over the real corpus rather than by reasoning about it; `sessionId` keys example attribution and per-session detector state, so either would have merged unrelated sessions silently. Top-level session ids are unchanged (#PR) + - `fp-cloud-cli`'s Click shim survives typer 0.27.2, which moved `Abort` out of its vendored Click. `_click_compat` wrapped all six vendored imports in one `try: … except ImportError: from click import …`, so that single missing name rebound **every** symbol to pip Click — the exact silent failure the module exists to prevent. Typer catches only its own Click's exceptions, so every typed error escaped uncaught: `fp alerts show ghost` exited 1 with an empty stderr instead of 6 with a message, and the same for exits 2, 3, 4 and 5. 105 tests went red on the dependabot bump that first installed 0.27.2. The Click is now chosen once — on whether `typer._click` exists at all — and each symbol imported from that choice, so a name that goes missing raises at import (a CLI that will not start) rather than silently downgrading every error to exit 1. `Abort` alone is resolved from `typer.Abort`, which tracks the move by construction: pip Click's before typer 0.26, the vendored class through 0.27.1, `typer.exceptions.Abort` from 0.27.2 (#771) ### Docs diff --git a/__tests__/audit/redaction-sinks.test.ts b/__tests__/audit/redaction-sinks.test.ts new file mode 100644 index 000000000..f4104cb7f --- /dev/null +++ b/__tests__/audit/redaction-sinks.test.ts @@ -0,0 +1,137 @@ +// @vitest-environment node +/** + * Every audit renderer that emits an example must redact it first. + * + * `redact-example.ts` was written carefully and then wired to exactly ONE of the + * places an example can leave the machine: the emailed digest. The markdown file + * the CLI prints as "Shareable report" wrote raw commands and raw cwd into the + * user's working tree, and `formatJson` was a bare stringify of the whole + * result. Both are artifacts whose entire purpose is to be sent somewhere. + * + * These tests pin the wiring rather than the redactor — `redact-example.test.ts` + * covers what a secret looks like once masked. What is asserted here is that + * each renderer is CONNECTED, because the defect was never a bad mask; it was a + * mask nobody called. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { formatText, formatMarkdown, formatJson } from "@/src/audit/report"; +import type { AuditCount, AuditResult } from "@/src/audit/types"; + +const SECRET = "ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; +const HOME_PATH = "/home/testuser/clients/acme-bank/src/db.ts"; + +function count(overrides: Partial = {}): AuditCount { + return { + name: "failproofai/block-secrets-write", + source: "builtin", + category: "Security", + severity: "deny", + hits: 3, + projects: 1, + firstSeen: "2026-09-01T10:00:00.000Z", + lastSeen: "2026-09-04T10:00:00.000Z", + examples: [ + { + sessionId: "s1", + cwd: "/home/testuser/clients/acme-bank", + timestamp: "2026-09-04T10:00:00.000Z", + example: `curl -H "Authorization: Bearer ${SECRET}" https://api.example.com`, + }, + ], + displayTitle: "Wrote a secret to a file", + impact: "The credential outlives the session.", + enabledInConfig: true, + installHint: "", + ...overrides, + }; +} + +function result(results: AuditCount[] = [count()]): AuditResult { + return { + version: 2, + scannedAt: "2026-09-07T10:00:00.000Z", + scope: { cli: ["claude"], projects: [HOME_PATH], since: null }, + transcripts: { scanned: 1, skipped: 0, errors: 0, durationMs: 1 }, + results, + totals: { hits: 3, projectsWithHits: 1 }, + projectsScanned: ["/home/testuser/clients/acme-bank"], + eventsScanned: 10, + enabledBuiltinNames: ["block-secrets-write"], + }; +} + +describe("formatMarkdown — the file the CLI calls a Shareable report", () => { + it("never writes a credential into the report body", () => { + const md = formatMarkdown(result(), {}); + expect(md).toContain("Examples"); + expect(md).not.toContain(SECRET); + expect(md).toContain("[REDACTED"); + }); + + it("shortens the cwd so the report does not carry a map of someone's disk", () => { + const md = formatMarkdown(result(), {}); + expect(md).not.toContain("/home/testuser/clients/acme-bank"); + }); +}); + +describe("formatJson — piped wherever the caller wants", () => { + it("redacts examples inside the serialized result", () => { + const json = formatJson(result()); + expect(json).not.toContain(SECRET); + }); + + it("redacts the project paths carried alongside the findings", () => { + const json = formatJson(result()); + expect(json).not.toContain("/home/testuser/clients/acme-bank"); + }); + + it("leaves the caller's own object untouched", () => { + const r = result(); + formatJson(r); + expect(r.results[0].examples[0].example).toContain(SECRET); + }); +}); + +describe("formatText — the local terminal", () => { + it("masks the credential", () => { + const text = formatText(result(), { showExamples: true }); + expect(text).not.toContain(SECRET); + }); + + it("keeps the real path, because shortening it protects nobody on their own machine", () => { + const withPath = count({ + examples: [ + { + sessionId: "s1", + cwd: "/home/testuser/clients/acme-bank", + timestamp: "2026-09-04T10:00:00.000Z", + example: `cat ${HOME_PATH}`, + }, + ], + }); + const text = formatText(result([withPath]), { showExamples: true }); + expect(text).toContain(HOME_PATH); + }); +}); + +describe("wiring tripwire", () => { + it("keeps report.ts importing the redactor", () => { + // The defect was structural: redactExample had exactly two non-definition + // references and both were in harm-report.ts. If this import is ever + // dropped, every renderer above silently starts emitting raw examples + // again — and the tests above only catch it for the shapes they model. + const src = readFileSync(join(process.cwd(), "src/audit/report.ts"), "utf-8"); + expect(src).toMatch(/from "\.\/redact-example"/); + }); + + it("keeps more than one module depending on the redactor", () => { + const files = ["src/audit/report.ts", "src/audit/harm-report.ts"]; + const importers = files.filter((f) => + readFileSync(join(process.cwd(), f), "utf-8").includes('from "./redact-example"'), + ); + expect(importers).toEqual(files); + }); +}); diff --git a/__tests__/lib/claude-sessions-subagents.test.ts b/__tests__/lib/claude-sessions-subagents.test.ts new file mode 100644 index 000000000..d58287803 --- /dev/null +++ b/__tests__/lib/claude-sessions-subagents.test.ts @@ -0,0 +1,114 @@ +// @vitest-environment node +/** + * The subagent walk below `/subagents/`. + * + * This exists because the enumerator read only the DIRECT children of that + * directory, which was right for the layout Claude shipped when it was written + * and silently wrong once workflow runs began nesting their agents one level + * further down. On the machine the miss was found on it cost 91% of the corpus: + * 1,839 transcripts on disk, 160 enumerated. The audit reported that result as + * though it had read everything, which is the failure mode these tests exist to + * keep closed — a scan that finds nothing and a scan that looks nowhere are + * indistinguishable from the outside. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync, symlinkSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { listClaudeProjects, listClaudeTranscripts } from "@/lib/claude-sessions"; + +const PARENT_A = "aaaaaaaa-1111-4111-8111-aaaaaaaaaaaa"; +const PARENT_B = "bbbbbbbb-2222-4222-8222-bbbbbbbbbbbb"; + +let root: string; +let prevEnv: string | undefined; + +/** Write a transcript at `/<...segments>`, creating parents. */ +function transcript(project: string, ...segments: string[]): string { + const path = join(root, project, ...segments); + mkdirSync(join(path, ".."), { recursive: true }); + writeFileSync(path, '{"type":"user"}\n'); + return path; +} + +function allTranscripts() { + return listClaudeProjects().flatMap((p) => listClaudeTranscripts(p)); +} + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "fpai-subagents-")); + prevEnv = process.env.CLAUDE_PROJECTS_PATH; + process.env.CLAUDE_PROJECTS_PATH = root; +}); + +afterEach(() => { + if (prevEnv === undefined) delete process.env.CLAUDE_PROJECTS_PATH; + else process.env.CLAUDE_PROJECTS_PATH = prevEnv; + rmSync(root, { recursive: true, force: true }); +}); + +describe("listClaudeTranscripts — subagent nesting", () => { + it("finds a transcript nested under subagents/workflows//", () => { + transcript("-home-u-proj", `${PARENT_A}.jsonl`); + transcript("-home-u-proj", PARENT_A, "subagents", "agent-direct.jsonl"); + transcript("-home-u-proj", PARENT_A, "subagents", "workflows", "wf_123-abc", "agent-nested.jsonl"); + + const found = allTranscripts(); + + // The regression: the nested one used to be dropped entirely. + expect(found).toHaveLength(3); + expect(found.filter((t) => t.isSubagent)).toHaveLength(2); + expect(found.map((t) => t.transcriptPath).some((p) => p.includes("wf_123-abc"))).toBe(true); + }); + + it("keeps the top-level session id untouched", () => { + transcript("-home-u-proj", `${PARENT_A}.jsonl`); + const top = allTranscripts().find((t) => !t.isSubagent); + expect(top?.sessionId).toBe(PARENT_A); + }); + + it("qualifies a subagent id with its parent session and its path", () => { + transcript("-home-u-proj", PARENT_A, "subagents", "agent-direct.jsonl"); + transcript("-home-u-proj", PARENT_A, "subagents", "workflows", "wf_123-abc", "agent-nested.jsonl"); + + const ids = allTranscripts().map((t) => t.sessionId).sort(); + expect(ids).toEqual([ + `${PARENT_A}__agent-direct`, + `${PARENT_A}__workflows__wf_123-abc__agent-nested`, + ]); + }); + + it("does not collide when one workflow run id appears under two parent sessions", () => { + // Found in the real corpus, not imagined: a resumed session reuses its run + // id, so `wf_/journal.jsonl` exists under two parents in one project. + // Deriving the id from the path below `subagents/` alone merged them, and + // sessionId is what example attribution and per-session detector state are + // keyed by — so the merge would be silent rather than an error. + transcript("-home-u-proj", PARENT_A, "subagents", "workflows", "wf_shared", "journal.jsonl"); + transcript("-home-u-proj", PARENT_B, "subagents", "workflows", "wf_shared", "journal.jsonl"); + + const found = allTranscripts(); + expect(found).toHaveLength(2); + expect(new Set(found.map((t) => t.sessionId)).size).toBe(2); + }); + + it("ignores non-transcript files and does not follow symlinks", () => { + transcript("-home-u-proj", PARENT_A, "subagents", "agent-real.jsonl"); + writeFileSync(join(root, "-home-u-proj", PARENT_A, "subagents", "notes.txt"), "x"); + + // A symlink pointing back up would make a naive walk loop forever. + const subDir = join(root, "-home-u-proj", PARENT_A, "subagents"); + symlinkSync(join(root, "-home-u-proj"), join(subDir, "loop"), "dir"); + + const found = allTranscripts(); + expect(found).toHaveLength(1); + expect(found[0].sessionId).toBe(`${PARENT_A}__agent-real`); + }); + + it("stops descending past the depth cap instead of walking forever", () => { + const deep = ["subagents", "a", "b", "c", "d", "e", "f", "g"]; + transcript("-home-u-proj", PARENT_A, ...deep, "agent-too-deep.jsonl"); + expect(allTranscripts()).toHaveLength(0); + }); +}); diff --git a/lib/claude-sessions.ts b/lib/claude-sessions.ts index 5886eeea4..497774747 100644 --- a/lib/claude-sessions.ts +++ b/lib/claude-sessions.ts @@ -4,8 +4,14 @@ * Claude stores transcripts at: * //.jsonl * - * Subagent transcripts (when a session spawned subagents) live alongside: + * Subagent transcripts (when a session spawned subagents) live alongside, in two + * shapes — a direct child, and one nested per workflow run: * ///subagents/.jsonl + * ///subagents/workflows//.jsonl + * + * Everything below `subagents/` is walked to a bounded depth rather than one + * level, because the second shape arrived after this module was written and the + * miss was most of the corpus. See `collectSubagentTranscripts`. * * The parser for these files lives in `lib/log-entries.ts` (`parseLogContent`, * `parseSessionLog`). This module exposes discovery only — the audit pipeline @@ -70,6 +76,99 @@ export function listClaudeProjects(): ClaudeProjectFolder[] { /** Lists every JSONL transcript under one Claude project folder, including * subagent transcripts under `/subagents/`. Returns [] on missing * or unreadable paths. */ +/** + * Depth of nesting allowed below `subagents/`. + * + * The two shapes that exist today need 0 and 2 (`.jsonl` and + * `workflows//.jsonl`). The cap is here so an unexpected layout — + * or a symlink cycle a future Claude build introduces — costs a bounded walk + * rather than an audit that never returns, and it is deliberately loose enough + * that a third shape lands inside it without another release. + */ +const MAX_SUBAGENT_DEPTH = 5; + +/** + * Collect every `.jsonl` beneath a session's `subagents/` directory. + * + * This used to read only the DIRECT children of `subagents/`, which was correct + * for the layout Claude shipped when it was written and silently wrong the day + * workflow runs started nesting their agents one level further down. On a + * machine that uses them the miss is most of the corpus: 1,839 transcripts on + * disk, 1,741 under `subagents/`, and 1,679 of those inside + * `subagents/workflows//` — so the audit was walking 160 files, 8.7% of + * the evidence, and reporting the result as though it had read everything. + * + * ## Session ids come from the relative path, not the basename + * + * Every workflow run writes a `journal.jsonl` beside its agents, so basenames + * are NOT unique below this directory — 1,741 files share 1,613 distinct names + * on the machine this was found on. A basename id would collide those 128 files + * onto ~one row each, and `sessionId` is what the cache, the per-session + * detector state and the example attribution are keyed by, so the collision + * would silently merge unrelated sessions rather than fail. + * + * Nor is the path below `subagents/` unique on its own. A workflow run id is + * reused when its session is resumed, so `wf_3d609e92-a38/journal.jsonl` exists + * under two different parent sessions in the same project — found by asserting + * uniqueness over the real corpus, not by reasoning about it. + * + * The id is therefore the PARENT SESSION id followed by the path relative to + * `subagents/`, minus the extension, joined with `__`: + * `__workflows__wf_123__agent-abc`. That is a filesystem path + * within the project, so it is unique by construction rather than by argument, + * and it names the session the subagent belongs to. `__` is safe as the joiner + * because no id Claude generates contains one — verified against 1,741 real + * files, zero hits. Top-level session ids are untouched. + * + * Symlinks are not followed. Nothing in the layout uses them, and following one + * is how a directory walk turns into an infinite loop. + */ +function collectSubagentTranscripts( + dir: string, + relPrefix: string, + project: ClaudeProjectFolder, + out: ClaudeTranscriptFile[], + depth: number, +): void { + if (depth > MAX_SUBAGENT_DEPTH) return; + let entries: import("node:fs").Dirent[]; + try { + entries = readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const entry of entries) { + if (entry.isSymbolicLink()) continue; + if (entry.isDirectory()) { + collectSubagentTranscripts( + join(dir, entry.name), + relPrefix ? `${relPrefix}__${entry.name}` : entry.name, + project, + out, + depth + 1, + ); + continue; + } + if (!entry.isFile() || !entry.name.endsWith(".jsonl")) continue; + const stem = entry.name.slice(0, -".jsonl".length); + const transcriptPath = join(dir, entry.name); + try { + const s = statSync(transcriptPath); + out.push({ + projectName: project.name, + cwd: project.cwd, + sessionId: relPrefix ? `${relPrefix}__${stem}` : stem, + transcriptPath, + mtimeMs: s.mtimeMs, + sizeBytes: s.size, + isSubagent: true, + }); + } catch { + // unreadable — skip + } + } +} + export function listClaudeTranscripts(project: ClaudeProjectFolder): ClaudeTranscriptFile[] { const out: ClaudeTranscriptFile[] = []; let entries: import("node:fs").Dirent[]; @@ -99,34 +198,9 @@ export function listClaudeTranscripts(project: ClaudeProjectFolder): ClaudeTrans // unreadable — skip } } else if (entry.isDirectory() && UUID_RE.test(entry.name)) { - // Subagent transcripts at /subagents/.jsonl const subDir = join(project.path, entry.name, "subagents"); if (!existsSync(subDir)) continue; - let subEntries: import("node:fs").Dirent[]; - try { - subEntries = readdirSync(subDir, { withFileTypes: true }); - } catch { - continue; - } - for (const sub of subEntries) { - if (!sub.isFile() || !sub.name.endsWith(".jsonl")) continue; - const agentId = sub.name.slice(0, -".jsonl".length); - const transcriptPath = join(subDir, sub.name); - try { - const s = statSync(transcriptPath); - out.push({ - projectName: project.name, - cwd: project.cwd, - sessionId: agentId, - transcriptPath, - mtimeMs: s.mtimeMs, - sizeBytes: s.size, - isSubagent: true, - }); - } catch { - // unreadable — skip - } - } + collectSubagentTranscripts(subDir, entry.name, project, out, 0); } } diff --git a/src/audit/redact-example.ts b/src/audit/redact-example.ts index 5062a0aa5..bbada5acc 100644 --- a/src/audit/redact-example.ts +++ b/src/audit/redact-example.ts @@ -42,6 +42,8 @@ */ import { homedir } from "node:os"; +import type { AuditResult } from "./types"; + import { SECRET_PATTERNS } from "../hooks/builtin-policies"; /** Longest example we let through, after redaction. */ @@ -298,15 +300,73 @@ export function shortenPaths(input: string, home = homedir()): string { * digest as one row, and a raw newline there breaks the plain-text layout while * saying nothing the single line does not. */ +/** + * Mask every secret, and nothing else. + * + * The three masking passes of `redactExample` without the path shortening, the + * whitespace collapse or the length cap. This is what a LOCAL renderer wants: + * on your own machine your own paths are the useful half of an example, and + * `~/…/db.ts` costs readability for no gain — nobody is protected from their own + * directory names. A credential on screen is the other half of that trade: it is + * one screenshot, one pasted issue or one screen-share away from being published, + * and unlike the path it can never be un-leaked. + * + * So the split is deliberate: everything that LEAVES the machine goes through + * `redactExample`; the terminal gets this. + */ +export function maskSecretsOnly(input: string): string { + return maskAssignedSecrets(maskTruncatedSecret(maskSecrets(input))); +} + export function redactExample(input: string, home = homedir()): string { // Assignment masking runs LAST of the three, so the two pattern-based passes // get first refusal on anything they can name precisely. A vendor prefix // yields "[REDACTED: Anthropic API key]"; falling through to this one would // have said only "assigned secret", which is true but less useful to read. - const masked = maskAssignedSecrets(maskTruncatedSecret(maskSecrets(input))); + const masked = maskSecretsOnly(input); const shortened = shortenPaths(masked, home); const collapsed = shortened.replace(/\s+/g, " ").trim(); return collapsed.length > REDACTED_EXAMPLE_MAX_CHARS ? `${collapsed.slice(0, REDACTED_EXAMPLE_MAX_CHARS - 1)}…` : collapsed; } + +/** + * Redact every free-text field of an `AuditResult` that could carry a secret or + * name someone's disk. + * + * The counts, titles, timestamps and policy names are the substance of a report + * and none of them come from user data. What DOES come from user data is the + * example strings (slices of real commands), the per-example `cwd`, and the + * project lists — which are the same three things `redactExample` was written + * for, applied to the whole structure instead of one row. + * + * Used by every renderer that produces an artifact which can travel: + * `formatMarkdown` (the file the CLI calls a "Shareable report") and + * `formatJson` (whatever the caller pipes it into). The emailed digest reaches + * the same guarantee by a different route — `harm-report.ts` redacts each + * example as it selects it, because it also has to apply the reporting window. + * + * Returns a NEW object. The caller's result is left alone, so the dashboard and + * the cache keep the unredacted values they need to render a local view. + */ +export function redactAuditResult(result: AuditResult, home = homedir()): AuditResult { + return { + ...result, + scope: { + ...result.scope, + projects: result.scope.projects === "all" + ? "all" + : result.scope.projects.map((p) => shortenPaths(p, home)), + }, + results: result.results.map((row) => ({ + ...row, + examples: row.examples.map((e) => ({ + ...e, + example: redactExample(e.example, home), + cwd: shortenPaths(e.cwd, home), + })), + })), + projectsScanned: result.projectsScanned.map((p) => shortenPaths(p, home)), + }; +} diff --git a/src/audit/report.ts b/src/audit/report.ts index 6ae87f783..b97fc38ff 100644 --- a/src/audit/report.ts +++ b/src/audit/report.ts @@ -10,6 +10,7 @@ * install command + report path + star link. */ import type { AuditCount, AuditResult, RunAuditOptions } from "./types"; +import { maskSecretsOnly, redactAuditResult, redactExample, shortenPaths } from "./redact-example"; const ANSI = { reset: "\x1B[0m", @@ -105,7 +106,7 @@ function renderRow(r: AuditCount, opts: { showExamples?: boolean }): string[] { ` ${ANSI.dim}Last seen ${formatTimeAgo(r.lastSeen)} · ${r.projects} project${r.projects === 1 ? "" : "s"}${ANSI.reset}`, ); if (opts.showExamples && r.examples[0]) { - out.push(` ${ANSI.dim}Example: ${r.examples[0].example}${ANSI.reset}`); + out.push(` ${ANSI.dim}Example: ${maskSecretsOnly(r.examples[0].example)}${ANSI.reset}`); } if (r.installHint) { const arrowColor = r.enabledInConfig ? ANSI.green : ANSI.cyan; @@ -226,7 +227,7 @@ export function formatText(result: AuditResult, opts: RunAuditOptions = {}): str } export function formatJson(result: AuditResult): string { - return JSON.stringify(result, null, 2); + return JSON.stringify(redactAuditResult(result), null, 2); } /** Escape characters that would break a markdown table row. Pipes split @@ -334,7 +335,9 @@ export function formatMarkdown(result: AuditResult): string { out.push(""); } for (const e of r.examples) { - out.push(`- \`${escapeBackticks(e.example)}\` _(${e.cwd || "?"}, ${formatTimeAgo(e.timestamp)})_`); + out.push( + `- \`${escapeBackticks(redactExample(e.example))}\` _(${shortenPaths(e.cwd) || "?"}, ${formatTimeAgo(e.timestamp)})_`, + ); } out.push(""); } From 173e2947764b72eff8b3c72886d3277c4f9fec4b Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Wed, 9 Sep 2026 00:29:04 +0530 Subject: [PATCH 02/17] feat(audit): detect leaked credentials and get them in front of the user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebuilds the audit around one job: find credentials leaked into agent transcripts, and tell the person whose machine leaked them — with no account, no email address, and no dashboard visit required. Detection. 33 doc-verified vendor patterns plus a secret-named-assignment layer (the two are near-disjoint: 55.9% of real vendor keys have no secret word within 60 chars, and 230 of 237 secret-named assignments match no vendor pattern). A docs-literal denylist beat every new pattern on the real corpus — AWS went 232 findings to 0, because 456 of 457 AKIA hits are the documentation literal. Findings are grouped by DISTINCT VALUE, so a key pasted into forty commands is one thing to rotate rather than forty. The record cannot hold a secret. recordLeaks fingerprints at detection time and stores only the mask, so no field exists downstream for a value to sit in. leak-containment.test.ts drives the real pipeline with six credential classes and then greps every byte written, the digest and the notice for any 12-char fragment. Delivery, four channels, each covering what the others cannot: in-CLI a two-line notice on the four hosts a live probe proved can paint text on a terminal. additionalContext was rejected: it reaches the MODEL, and a model disclaimed an actionable notice as suspected prompt injection 6 times out of 6. linux D-Bus written by hand against /run/user//bus, an address CONSTRUCTED from the uid because the system-scope daemon has no DBUS_SESSION_BUS_ADDRESS. The reply is awaited — fire-and-forget returns exit 0 with empty output while the notification silently evaporates. macos a per-user LaunchAgent plus an osacompile'd applet with its own bundle id, installed silently by `failproofai config` and removed by `failproofai uninstall`. failproofaid is a LaunchDaemon, and nothing in launchd's system domain can reach Notification Center — the same split Time Machine ships. email the scheduled digest carries a masked `leaks` array. Each channel claims a finding through its own O_EXCL marker file. A notifiedAt field on the record was measured and failed 100% of the time under concurrent sessions; two sessions with different findings permanently lost one mark and re-notified forever. Scheduled audits are ON by default for a machine that completed setup, and the flip keeps the fail-safe it replaces. Three-way, not two: config present AND carrying an `audit` table means on unless it says exactly false; absent file, unparseable JSON and no audit table all still read as OFF, because those are the three ways of not knowing and "we could not tell" must never start a scan that reads every transcript on disk. audit_lane.rs and fp-config.ts assert the same table over the same bytes. Turning it on sends nothing on its own — the digest stays gated on reports_consented_at. /audit renders one row per credential with what, where, who, when and how, says whether the exposure was blockable (the agent SENT it, so a PreToolUse gate can deny it next time) or not (the agent RECEIVED it, which no gate can undo), and gives the one piece of advice that applies. "not a secret" dismisses a row and RETAINS the finding, so the next scan cannot rediscover the value and alert again. New setting audit.notify, written by `failproofai audit --notify` / `--no-notify` and by the dashboard toggle, read at the moment a notification would fire. On by default. It does not silence the in-CLI notice: turning off every channel at once produces a state indistinguishable from broken. Five bugs found by adversarial testing and fixed here, not separately, because none of this code ever shipped: - The D-Bus encoder never worked against a real bus. It declared its header-fields array two bytes too long, counting the alignment padding after the final field, so dbus-daemon dropped the connection on the first message. It passed its tests because the test server was written from the same assumptions and mirrored the mistake. Found by diffing our Hello against a real client's bytes (0x70 vs 0x6e). The encoder is now an offset-tracking marshaller and the tests run against a real dbus-daemon. Two more fell out: no close handler, so a bus hanging up read as a timeout; and a state machine that took "the next chunk" as its reply, so the NameAcquired signal a real bus emits after Hello was read as a successful delivery with a garbage id. - Two catastrophically backtracking regexes. A 300 KB base64 blob — routine in transcripts — took over 20 SECONDS in findSecrets and never finished in redactExample. Bounding the identifier and the URI scheme takes the worst case to 178ms. The Rust redactor was never affected and now has a test proving it. - A finding id became a filename with no validation, so markLeakNoticeDelivered(["../../../../tmp/PWNED"]) created that file. - One malformed leaks.json entry threw out of buildHarmReport, which sits outside reportHarm's try — a scan that succeeded and cached correctly still exited 1, every run, until somebody opened the file. - shapeNotice destroyed any stdout that was parseable but not an object. The old scored report is switched off, not deleted: every module and component is intact and still unit-tested, and only the call sites are commented, each cross-referenced to the explanation in src/audit/scoring.ts. 150 new tests. 5,058 passing overall, with only the 11 known pre-existing failures (dogfood-configs x8, fp-reset x3). --- .opencode/plugins/failproofai.mjs | 78 ++-- CHANGELOG.md | 36 +- Cargo.lock | 6 +- Cargo.toml | 2 +- __tests__/audit/desktop-notify.test.ts | 328 ++++++++++++++ __tests__/audit/harm-report-leaks.test.ts | 154 +++++++ __tests__/audit/incremental-scan.test.ts | 9 +- __tests__/audit/index.test.ts | 22 +- __tests__/audit/leak-containment.test.ts | 229 ++++++++++ __tests__/audit/leak-fingerprint.test.ts | 104 +++++ __tests__/audit/leak-hostile-input.test.ts | 175 ++++++++ __tests__/audit/leak-notice.test.ts | 151 +++++++ __tests__/audit/leak-record.test.ts | 150 +++++++ __tests__/audit/leak-scan.test.ts | 223 ++++++++++ __tests__/audit/leak-section.test.tsx | 195 ++++++++ __tests__/audit/leak-store.test.ts | 197 ++++++++ __tests__/audit/macos-notifier.test.ts | 152 +++++++ __tests__/audit/notify-toggle.test.ts | 76 ++++ __tests__/audit/redact-example.test.ts | 147 +++++- __tests__/audit/redaction-sinks.test.ts | 34 +- __tests__/audit/scheduled-audit.test.ts | 84 ++++ __tests__/audit/share-templates.test.ts | 36 +- __tests__/hooks/builtin-policies.test.ts | 5 +- __tests__/hooks/fp-home.test.ts | 95 +++- __tests__/hooks/harness-extra-paths.test.ts | 4 +- __tests__/hooks/opencode-plugin-shim.test.ts | 56 ++- __tests__/hooks/pi-extension-shim.test.ts | 54 +++ __tests__/hooks/policy-catalog.test.ts | 5 +- app/actions/get-leaks.ts | 100 +++++ app/audit/_components/audit-dashboard.tsx | 145 ++++-- app/audit/_components/audit-poster.tsx | 39 +- .../_components/come-back-better-section.tsx | 7 +- app/audit/_components/empty-state.tsx | 5 +- .../_components/how-to-improve-section.tsx | 24 +- app/audit/_components/leak-section.tsx | 179 ++++++++ app/audit/_components/share-templates.ts | 154 +++++-- app/audit/audit-styles.css | 115 ++++- crates/failproofaid/src/audit_lane.rs | 44 +- crates/fpai-collect/src/redact.rs | 248 ++++++++++- lib/auth/api-server-client.ts | 28 ++ package.json | 2 +- pi-extension/index.ts | 46 +- src/audit/cli.ts | 119 +++++ src/audit/desktop-notify.ts | 420 ++++++++++++++++++ src/audit/harm-report.ts | 101 +++++ src/audit/index.ts | 154 ++++++- src/audit/leak-fingerprint.ts | 200 +++++++++ src/audit/leak-notice.ts | 161 +++++++ src/audit/leak-record.ts | 232 ++++++++++ src/audit/leak-scan.ts | 292 ++++++++++++ src/audit/leak-store.ts | 217 +++++++++ src/audit/macos-notifier.ts | 310 +++++++++++++ src/audit/redact-example.ts | 224 +++++++++- src/audit/report-harm.ts | 13 +- src/audit/schedule-cli.ts | 37 ++ src/audit/scoring.ts | 49 ++ src/audit/types.ts | 41 ++ src/hooks/builtin-policies.ts | 69 ++- src/hooks/configure-wizard.ts | 20 + src/hooks/fp-config.ts | 82 +++- src/hooks/fp-home.ts | 27 ++ src/hooks/handler.ts | 60 ++- src/hooks/integrations.ts | 60 ++- src/hooks/notice.ts | 155 +++++++ src/hooks/uninstall-cli.ts | 15 + 65 files changed, 6735 insertions(+), 266 deletions(-) create mode 100644 __tests__/audit/desktop-notify.test.ts create mode 100644 __tests__/audit/harm-report-leaks.test.ts create mode 100644 __tests__/audit/leak-containment.test.ts create mode 100644 __tests__/audit/leak-fingerprint.test.ts create mode 100644 __tests__/audit/leak-hostile-input.test.ts create mode 100644 __tests__/audit/leak-notice.test.ts create mode 100644 __tests__/audit/leak-record.test.ts create mode 100644 __tests__/audit/leak-scan.test.ts create mode 100644 __tests__/audit/leak-section.test.tsx create mode 100644 __tests__/audit/leak-store.test.ts create mode 100644 __tests__/audit/macos-notifier.test.ts create mode 100644 __tests__/audit/notify-toggle.test.ts create mode 100644 app/actions/get-leaks.ts create mode 100644 app/audit/_components/leak-section.tsx create mode 100644 src/audit/desktop-notify.ts create mode 100644 src/audit/leak-fingerprint.ts create mode 100644 src/audit/leak-notice.ts create mode 100644 src/audit/leak-record.ts create mode 100644 src/audit/leak-scan.ts create mode 100644 src/audit/leak-store.ts create mode 100644 src/audit/macos-notifier.ts create mode 100644 src/hooks/notice.ts diff --git a/.opencode/plugins/failproofai.mjs b/.opencode/plugins/failproofai.mjs index 71f663ceb..5464a926d 100644 --- a/.opencode/plugins/failproofai.mjs +++ b/.opencode/plugins/failproofai.mjs @@ -18,7 +18,7 @@ // • src/hooks/integrations.ts (buildOpenCodePluginShim production template) // When #337 landed, this dev shim drifted and `block-read-outside-cwd` // silently no-op'd on every opencode `read` call inside this repo. -import { spawnSync } from "node:child_process"; +import { spawn } from "node:child_process"; import { resolveDevSpawn } from "../../scripts/dev-hook.mjs"; const BUS_EVENT_MAP = { @@ -70,34 +70,58 @@ function canonicalizeToolInput(canonicalToolName, args) { return out; } -function runFailproofai(eventName, payload, directory) { - // Shared with the JSON configs' launcher: locates bun across PATH, ~/.bun/bin, - // $BUN_INSTALL, Homebrew and every nvm version dir. A bare spawnSync("bun") - // dies with ENOENT the moment bun is off the hook's PATH. - const spawn = resolveDevSpawn(); - if (!spawn) { +/** + * Run failproofai for one event WITHOUT blocking opencode's event loop. + * + * This was `spawnSync`, and opencode loads the plugin in-process in the TUI — + * so every hook froze rendering and input for the subprocess's full duration, + * with a 60s ceiling. The verdicts are unchanged: every caller already `await`s + * `applyDecision`, so the deny still lands before the tool runs. What changes + * is that the wait is now a promise rather than a blocked thread, and the TUI + * keeps painting while it happens. + * + * Fail-open on spawn error, and on the timeout — a policy that never ran must + * not read as a deny. + */ +async function runFailproofai(eventName, payload, directory) { + const resolved = resolveDevSpawn(); + if (!resolved) { process.stderr.write( "[failproofai-dev] bun not found — opencode policies are NOT enforcing. " + "Install bun: curl -fsSL https://bun.sh/install | bash\n", ); return { exitCode: 0, stdout: "", stderr: "" }; } - const r = spawnSync(spawn.cmd, [...spawn.args, "--hook", eventName, "--cli", "opencode"], { - input: JSON.stringify(payload), - encoding: "utf8", - timeout: 60_000, - cwd: directory, + const cmd = resolved.cmd; + const args = [...resolved.args, "--hook", eventName, "--cli", "opencode"]; + return await new Promise((resolveRun) => { + let child; + try { + child = spawn(cmd, args, { cwd: directory }); + } catch { + resolveRun({ exitCode: 0, stdout: "", stderr: "" }); + return; + } + let stdout = ""; + let stderr = ""; + let settled = false; + const finish = (exitCode) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolveRun({ exitCode, stdout, stderr }); + }; + const timer = setTimeout(() => { + try { child.kill("SIGKILL"); } catch { /* already gone */ } + finish(0); + }, 60_000); + child.stdout.on("data", (d) => { stdout += d; }); + child.stderr.on("data", (d) => { stderr += d; }); + child.on("error", () => finish(0)); + child.on("close", (code) => finish(code ?? 0)); + child.stdin.on("error", () => {}); + child.stdin.end(JSON.stringify(payload)); }); - // `r.status` is null when the spawn never ran (ENOENT) or the child was - // killed. Coercing that to 0 reports "allowed" for a policy that never got - // to run, so say so on stderr rather than failing open in silence. - if (r.status === null) { - process.stderr.write( - `[failproofai-dev] ${eventName} hook did not run (${r.error?.code ?? r.signal ?? "unknown"}) — not enforcing\n`, - ); - return { exitCode: 0, stdout: "", stderr: "" }; - } - return { exitCode: r.status, stdout: r.stdout ?? "", stderr: r.stderr ?? "" }; } async function applyDecision(result, ctx, eventName) { @@ -150,7 +174,7 @@ export default async function failproofaiPlugin({ client, directory }) { } } if (!prompt) prompt = (info.text || info.content || props.text || "").toString(); - const r = runFailproofai("UserPromptSubmit", { + const r = await runFailproofai("UserPromptSubmit", { session_id: sessionID, cwd: directory, hook_event_name: "UserPromptSubmit", prompt, }, directory); await applyDecision(r, { client, sessionID }, "UserPromptSubmit"); @@ -160,7 +184,7 @@ export default async function failproofaiPlugin({ client, directory }) { if (!claudeEvent) return; const props = event.properties || {}; const sessionID = props.sessionID || (props.session && props.session.id) || props.id; - const r = runFailproofai(claudeEvent, { + const r = await runFailproofai(claudeEvent, { session_id: sessionID, cwd: directory, hook_event_name: claudeEvent, }, directory); await applyDecision(r, { client, sessionID }, claudeEvent); @@ -168,7 +192,7 @@ export default async function failproofaiPlugin({ client, directory }) { "tool.execute.before": async (input, output) => { const canonicalTool = canonicalizeTool(input.tool); - const r = runFailproofai("PreToolUse", { + const r = await runFailproofai("PreToolUse", { session_id: input.sessionID, cwd: directory, tool_name: canonicalTool, @@ -180,7 +204,7 @@ export default async function failproofaiPlugin({ client, directory }) { "tool.execute.after": async (input, output) => { const canonicalTool = canonicalizeTool(input.tool); - const r = runFailproofai("PostToolUse", { + const r = await runFailproofai("PostToolUse", { session_id: input.sessionID, cwd: directory, tool_name: canonicalTool, @@ -193,7 +217,7 @@ export default async function failproofaiPlugin({ client, directory }) { "permission.ask": async (input, output) => { const canonicalTool = canonicalizeTool(input.tool); - const r = runFailproofai("PermissionRequest", { + const r = await runFailproofai("PermissionRequest", { session_id: input.sessionID, cwd: directory, tool_name: canonicalTool || input.command || "permission", diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b59c4f30..8b7e62f13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,44 @@ # Changelog -## 1.0.4-beta.0 — 2026-09-02 +## 1.0.4-beta.1 — 2026-09-09 + +### Features + +- Leaked credentials found in a transcript now reach the person whose machine leaked them, on the machine, without an account or an email address. Four channels, each covering a case the others cannot. **In-CLI**: a two-line notice inside the agent session, on the four hosts a live probe proved can paint text on a user's terminal (claude, codex, copilot, factory) — `hookSpecificOutput.additionalContext` was rejected because it reaches the model, and a model disclaimed an actionable notice as suspected prompt injection 6/6. **Linux desktop**: a D-Bus `Notify` written by hand against `/run/user//bus`, an address CONSTRUCTED from the uid rather than read from an environment the system-scope daemon does not have; the reply is awaited, because a fire-and-forget call against a bus with no notification server returns exit 0 and empty output while the notification silently evaporates. **macOS**: a per-user LaunchAgent and an `osacompile`d applet, installed silently by `failproofai config` and removed by `failproofai uninstall`, because `failproofaid` is a LaunchDaemon and nothing in launchd's system domain can reach Notification Center — the same split Time Machine ships. The applet has its own bundle id, so the banner is attributed to failproofai and gets its own entry in System Settings rather than inheriting Script Editor's. **Email**: the scheduled digest now carries a `leaks` array. Every channel claims each finding through an O_EXCL marker file, per channel — a `notifiedAt` field on the record was measured and failed 100% of the time under concurrent sessions, and two sessions with different findings permanently lost one mark and re-notified forever (#PR) + +- Scheduled audits are ON by default for a machine that completed setup, and the flip preserves the fail-safe it replaces. Three-way, not two: `config.json` present AND carrying an `audit` table means on unless it says exactly `false`; absent file, unparseable JSON, and no `audit` table at all still read as OFF, because those are the three ways of not knowing and "we could not tell" must never start a scan that reads every transcript on disk. `crates/failproofaid/src/audit_lane.rs` and `src/hooks/fp-config.ts` make the identical distinction over the identical bytes, each with a test asserting the same table — the daemon scanning while the settings page says off is the bug that split would otherwise produce. Turning it on sends nothing on its own: the digest stays gated on `reports_consented_at`, so a machine that flips on at upgrade scans locally and mails nobody (#PR) + +- `/audit` renders the leak report that replaced the score: one row per DISTINCT credential rather than per sighting — a key pasted into forty commands is one thing to rotate — showing the masked value, what it is, where it went, which harness carried it, when it was last seen, and how it got there. Each row says whether the exposure was blockable (the agent SENT it, so a PreToolUse gate can deny it next time) or not (the agent RECEIVED it, which no gate can undo), and gives the one piece of advice that applies: a recognised prefix means a console to revoke at, while an unattributed value — the majority, per the pattern census — means finding what reads the identifier. "not a secret" dismisses a row, and the finding is RETAINED rather than deleted so the next scan cannot rediscover the same value and alert again (#PR) + +- New setting `audit.notify`, written by both `failproofai audit --notify` / `--no-notify` and the toggle on `/audit`, read by the audit child at the moment a notification is about to fire. On by default. Its own switch rather than a flag on `--schedule`, because wanting the weekly scan without the banner is a coherent position and silencing a banner must not quietly switch off the scan. It does not silence the in-CLI notice, which costs two lines, appears inside a session the user is already driving, and is the only channel left on a headless box — turning off every channel at once produces a state indistinguishable from broken (#PR) + +- The rest of the old audit report is switched off with the score, ahead of rebuilding it bottom-up around leak detection. Dormant now: the persona classifier and its poster, the strengths and quirks sections, the punch-list with its install-all funnel, the invite section, and the 8 behavioural detectors. Also the archetype rarity — eight hardcoded integers under the comment "Seeded with snapshot values; swap for live aggregates once that pipeline lands", rendered as "// only 18% of agents are this archetype" and baked by `html-to-image` into the PNG people post publicly. A fabricated population statistic on a shared card is a claim we cannot support, and that poster was its only surface. Nothing is deleted: every module and component is intact and still unit-tested, and only the call sites and renders are commented, each cross-referenced to the one explanation in `src/audit/scoring.ts`. `/audit` now renders the scan's own numbers — tool calls, transcripts, projects — because the difference between "this page is being rebuilt" and "this product is broken" is worth the twenty lines it costs to say so. **What is untouched and still running: the 12-CLI adapter layer and transcript walking, the replay loop, the per-transcript cache, redaction, and the emailed digest.** `failproofai audit` still scans and still prints its summary; `report.ts` needed no change because it had zero production callers already. Two tests were repointed rather than deleted — `filters by --policy` now filters on a builtin so the filter itself stays covered, and `carries stateful detectors across the boundary` is skipped with the plumbing it guards left deliberately in place, so it passes again the moment the detector loop returns (#PR) + +- The audit score, letter grade and projected-score line are switched off — commented out at every call site rather than deleted, with each original preserved inline and cross-referenced back to a single explanation in `src/audit/scoring.ts`. Every function there is intact and still covered by `__tests__/audit/scoring.test.ts`, so restoring is un-commenting rather than rewriting. Two reasons. The product one: the audit did too many things shallowly, and grading a machine 0-100 with a letter tier was not the half worth keeping. The correctness one is worse and should be read before anyone switches it back on — **`deriveScore` never read `enabledInConfig`**; only `projectedScore` did. So the "enable all N → projected {score}" line above the install-all button promised a number the product could not deliver: a user who took the prescription, enabled the policies and re-ran got the *same* score with the projection gone. Advertised up to +22, delivered 0. That line now reads "enable all N → one command, enforced on every tool call", which is the thing that was always true. The prescription itself, the install-all button and the `audit_copy_clicked` → `hooks_installed` funnel are untouched — none of them ever read the score, and `missing`, the funnel's denominator, still ships on `audit_dashboard_viewed`. The archetype cue card survives and becomes the poster's headline: `classifyAgent` takes no score input, so the two were always separable. Share copy is archetype-only — 18 of the 20 templates lost a subordinate clause and 2 whose entire premise was the number are commented out, leaving 9 per channel, with the score-bearing originals preserved verbatim at the bottom of `share-templates.ts`. `score`/`grade` stop appearing on `audit_dashboard_viewed` and `audit_card_share_clicked`, so a PostHog dashboard keyed on them breaks rather than quietly reading zero. Also corrected `empty-state.tsx`, which promised "a tier, a score, and a punch-list" — the tier was never rendered to a user at any point (#PR) ### Fixes - The two PyPI publish workflows open the next version's `CHANGELOG` section in the same `bump` commit that moves `_version.py`, via a new `scripts/changelog-open.py`. `bump` used to move the version alone, leaving `main` on a version with no section — the exact state `scripts/changelog-section.py` refuses at release time and `__tests__/ci/python-version-pipeline.test.ts` asserts against. Because bump commits carry a skip-ci marker, that never went red on itself: it went red on the next unrelated PR to run CI, which is how `main` broke after the 0.0.1b1 publish (repaired by hand in #755, which named the recurrence and left it) and again after 0.0.1b2. `sdk/python/CHANGELOG.md` gets the 0.0.1b3 section that was missing. The opener is idempotent — a re-run of `bump` against a `main` that already carries the section is a no-op rather than a second heading, which would put only the first one's body on the GitHub Release — and it matches the version with the same trailing word boundary the extractor uses, so opening `0.0.1b1` is not satisfied by an existing `0.0.1b10` (#787) +- The Linux desktop notification never worked against a real bus. The hand-written D-Bus encoder declared its header-fields array two bytes too long, because it counted the alignment padding after the final field — padding that belongs to the message, not to the array — so `dbus-daemon` dropped the connection on the first message and every Linux user would have got silence. It passed its tests because the test server was written from the same assumptions as the encoder, and mirrored the mistake. Found by diffing our `Hello` against a real client's byte for byte (`0x70` vs `0x6e`). The encoder is now an offset-tracking marshaller, so alignment is computed against the real message offset instead of by padding sub-buffers in isolation, and the tests run against an actual `dbus-daemon` on a private socket. Two more bugs fell out of the same rewrite: no `close` handler, so a bus hanging up read as a 2-second timeout and pointed at a slow desktop rather than at us; and a state machine that treated "the next chunk" as its reply, so the `NameAcquired` signal a real bus emits after `Hello` was read as a successful delivery with a garbage id. Messages are now framed and matched on reply serial (#PR) + +- Two catastrophically backtracking regexes made the audit hang on ordinary transcript content. `ASSIGNMENT_RE` (both the scanner and the redactor) and the redactor's URL-credentials matcher each had an unbounded quantifier that, on a long unbroken token, restarted at every position and backtracked a character at a time. Measured: a 300 KB base64-shaped blob took **over 20 seconds** in `findSecrets` and never finished in `redactExample`; a long URL took 4.8s. Transcripts carry base64 images, minified bundles and whole file contents as single lines constantly, so a scheduled scan would stall for minutes unattended. Bounding the identifier to 128 characters and the URI scheme and userinfo to their real limits takes the worst case from >20,000ms to 178ms, with no change to what either matches. The Rust redactor was never affected — it is hand-rolled scanning with no backtracking engine — and now has a test proving it, because the two engines are required to agree (#PR) + +- A finding id is used as a filename, and nothing validated it: `markLeakNoticeDelivered(["../../../../tmp/PWNED"])` created that file. Ids are HMAC hex in any real run, so it was not reachable from normal use — but the record is JSON read off disk, and that is the difference between safe and incidentally safe. Both the notice marker and the macOS queue now refuse any id `fingerprintId` could not have minted. The queue also stranded a `notify-*.tmp` beside the watched directory on every failed write, where nothing would ever collect it (#PR) + +- A single malformed entry in `leaks.json` turned a healthy scheduled audit into a failed one, permanently. `selectLeaks` dereferenced fields the file might not carry, and it throws out of `buildHarmReport`, which sits OUTSIDE `reportHarm`'s try — so a run that scanned correctly and wrote its cache correctly still exited 1, and kept doing so every run until somebody opened the file by hand. `readLeakRecord` now validates each finding at the single gate every surface reads through; a bad sighting loses the sighting rather than the finding, because the credential is the thing that still needs rotating (#PR) + +- `shapeNotice` destroyed anything already on stdout that was not a JSON object. Its own contract says it leaves a verdict alone rather than risk breaking it, and it did so for unparseable JSON — but a parseable non-object (a JSON array, or plain text) fell past the merge and was replaced wholesale. It now leaves any unmergeable stdout untouched and drops the notice instead: a courtesy message is never worth more than what the host was already being told (#PR) + +- The opencode plugin and the pi extension stop freezing their TUIs on every session. Both shims called `spawnSync(…, {timeout: 60_000})` on a path the host runs in-process and awaits, so the entire interface — rendering, input, the spinner — was blocked for as long as the subprocess took, with a 60-second ceiling. On pi the floor alone was 1.0-1.6s of frozen UI on every single session start, just to boot the binary; on opencode a hook made to take 8s delayed the user's first message by 8s. Two different fixes, because the two hosts differ. **opencode** now uses the async `spawn` with the timeout enforced by its own timer: every call site already `await`ed `applyDecision`, so the deny still lands before the tool runs and no verdict changed — the wait is simply a promise instead of a blocked thread, and the TUI keeps painting through it. **pi** awaits its handlers serially, so an async spawn would still block; instead the three events that were already DISCARDING the verdict (`session_start`, `tool_result`, `session_shutdown` — Pi's `ToolResultEventResult` has no `block` and the other two have no Result type at all) now forward detached and unref'd. The four that consume a decision (`tool_call`, `user_bash`, `input`, `agent_end`) still block, because blocking is what enforcement means; a test pins that split so nobody moves an event across it. Both shims fail open on spawn error and on timeout — a policy that never ran must not read as a deny (#PR) + +- Neither redactor masks a key the build tool ships to the browser. Every publishable key on earth is named `*_KEY`, and the component rule matched all of them: measured, `isSecretName` returned true for all twelve of `NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY`, `NEXT_PUBLIC_SUPABASE_ANON_KEY`, `NEXT_PUBLIC_POSTHOG_KEY`, `VITE_API_KEY`, `VAPID_PUBLIC_KEY`, `PUBLISHABLE_KEY` and friends — including the names carrying the literal word PUBLIC. The Rust half had the identical hole with no exclusions at all. These are not conventionally public but **mechanically** public: Next.js, Vite, CRA, Expo, Nuxt, Gatsby, SvelteKit and Astro each inline the value into the JavaScript shipped to every visitor, so the framework published it before failproofai ever saw it. Over-redaction is usually cheap — masking something that might be secret costs readability — but this is the case where the value is provably not a secret, and reporting one is a security tool crying wolf about a browser-bundled config value. This repo's own committed PostHog `phc_` key, documented in source as "Write-only (safe to commit)", was being masked as an "assigned secret". The veto is deliberately narrow: the marker must be a PREFIX, so `MY_PUBLIC_FACING_API_SECRET` is untouched, and a word that merely starts with the same letters (`PUBLISHER_API_KEY`, `REPUBLIC_TOKEN`) is unaffected. It runs ahead of the value-shape rules, so a genuine `sk-` key assigned to a `VITE_` name is still masked on its own merits (#PR) + +- Both redactors now mask an assignment whatever its separator or spacing. Each one required the `=` to sit immediately against the value, so `NAME = "value"`, `NAME: "value"` and `{"name": "value"}` — a config file, a YAML key and a JSON body, which is most of where credentials are actually written down — all passed through with the value intact. A secret only survived that gap if it independently matched a vendor prefix, and 230 of 237 secret-named assignments measured across this machine's 1,839 transcripts match none of them, so the assignment rule was the only thing standing between them and the sink. Two sinks, because the bug was written twice: `redactExample` feeds the emailed digest (`harm-report.ts`), and `fpai-collect`'s `match_assignment` scrubs every event the daemon ships to Cloud. Both are fixed to the same grammar and both carry the failing shapes as tests. Two hazards found while fixing it, each of which is why the narrow form survived so long: on the TypeScript side a `:` alternative makes `https:` read as an assignment whose value is the rest of the URL, and because a non-secret name returns the match unchanged but still CONSUMES it, a `?token=…` inside that URL was swallowed and never examined — adding a separator to catch more secrets silently stopped one already being caught, so the `:` now refuses a following `//`. On the Rust side the backwards name scan ends at the first non-identifier character, so a JSON key's own closing quote left it with zero characters to read; it now steps back over that quote the way it already stepped over the opening one. Both use horizontal whitespace only — `\s` and `char::is_whitespace` cross a newline, which lets a trailing `KEY:` glue the next line on and redact it as the value (#PR) + +- Both redactors now decompose a camelCase credential name. `isSecretName` split the identifier on `_` alone, and the Rust `match_assignment` tested compoundness on the name it had *already lowercased* — so `sessionKey`, `dbPass`, `basicAuth` and `authCookie` reduced to a single component that matched nothing and shipped their values to the emailed digest and the Cloud spool verbatim, while the byte-identical `SESSION_KEY` was masked. 203 such names on this machine's corpus; measured, 6 of 12 ordinary credential-named assignments leaked. Both now split on `_`, `-`, `.` and camel humps, with an extra rule for an acronym meeting a word so `APIKey` yields `API` + `KEY`. The bare-`key` protection that exists because a lone `key=` matched React's `key` prop on every JSX list is untouched: a lone `key` has no hump, so it stays non-compound. `PASSPHRASE` joins the always-secret list, and `PWD` joins the compound-only list — `MYSQL_PWD` is MySQL's documented password variable while a bare `PWD` is the working directory, which is on every second line of a captured session. The module's own doc comment had claimed camelCase was covered, citing `_authToken`; that worked only because `TOKEN` is matched as a substring, and every name whose credential word was a *component* — `KEY`, `PASS`, `AUTH`, `PAT`, `SIG`, `SESSION`, `COOKIE` — was invisible (#PR) + +- `formatMarkdown` is called with the one argument it takes. Two call sites in `redaction-sinks.test.ts` passed a second, empty-object argument, which `tsc --noEmit` rejects — so the branch would have failed CI's `quality` job while its tests passed, since vitest does not typecheck (#PR) + - The audit stops writing your credentials into the file it calls a shareable report. `redactExample` had exactly two non-definition call sites and both were in `harm-report.ts`, the emailed digest — so the ONE path that was already careful was the only one connected. `formatMarkdown` wrote `example` and `cwd` verbatim into `./failproofai-audit.md`, which the CLI prints as `Shareable report` and which defaults to the current directory, i.e. a git working tree; `formatJson` was a bare `JSON.stringify` of the whole `AuditResult`, examples, per-example cwds, scanned project paths and all; and the terminal renderer printed the raw example too. Both artifacts exist to be sent somewhere. Every renderer now redacts: the two that travel through the full `redactExample` (masked secrets, shortened home paths) and, for the JSON, a new `redactAuditResult` that walks the examples, their cwds, `projectsScanned` and `scope.projects` and returns a new object so the dashboard and the cache keep the values they need to render locally. The terminal gets a new `maskSecretsOnly` instead — a credential on screen is one screenshot or one pasted issue from being published, while `~/…/db.ts` protects nobody from their own directory names and costs the example its most useful half. `redactExample` is now defined in terms of `maskSecretsOnly`, so the two cannot drift. A test asserts the import is still there, because the defect was never a bad mask — it was a mask nobody called (#PR) - The audit reads every subagent transcript instead of 8.7% of them. `listClaudeTranscripts` walked only the DIRECT children of `/subagents/`, which matched the layout Claude shipped when it was written and became wrong the day workflow runs started nesting their agents one level further down at `subagents/workflows//`. On the machine this was found on that is 1,839 transcripts on disk, 1,741 of them under `subagents/`, 1,679 of those nested — so the scan opened 160 files and reported the result as though it had read everything, which is the one failure a scanner must not have. Five of the seven files holding a genuine credential-bearing egress command were in the part it could not see. The walk is now recursive to a bounded depth, skipping symlinks so an unexpected layout costs a bounded walk rather than a scan that never returns. Subagent session ids are now qualified by their parent session and their path below `subagents/` (`__workflows__wf_123__agent-abc`), because basenames are not unique down there — every workflow run writes a `journal.jsonl`, and a run id is reused when its session is resumed, so the same relative path exists under two parents in one project. Both collisions were found by asserting uniqueness over the real corpus rather than by reasoning about it; `sessionId` keys example attribution and per-session detector state, so either would have merged unrelated sessions silently. Top-level session ids are unchanged (#PR) diff --git a/Cargo.lock b/Cargo.lock index f478c6310..38b2b9046 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -202,7 +202,7 @@ dependencies = [ [[package]] name = "failproofaid" -version = "1.0.4-beta.0" +version = "1.0.4-beta.1" dependencies = [ "fpai-collect", "fpai-ipc", @@ -265,7 +265,7 @@ dependencies = [ [[package]] name = "fpai-collect" -version = "1.0.4-beta.0" +version = "1.0.4-beta.1" dependencies = [ "notify", "reqwest", @@ -280,7 +280,7 @@ dependencies = [ [[package]] name = "fpai-ipc" -version = "1.0.4-beta.0" +version = "1.0.4-beta.1" dependencies = [ "libc", "proptest", diff --git a/Cargo.toml b/Cargo.toml index a084bf064..07370a307 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "3" members = ["crates/*"] [workspace.package] -version = "1.0.4-beta.0" +version = "1.0.4-beta.1" edition = "2024" license-file = "LICENSE" repository = "https://github.com/FailproofAI/failproofai" diff --git a/__tests__/audit/desktop-notify.test.ts b/__tests__/audit/desktop-notify.test.ts new file mode 100644 index 000000000..e5901ffa6 --- /dev/null +++ b/__tests__/audit/desktop-notify.test.ts @@ -0,0 +1,328 @@ +// @vitest-environment node +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { createServer, type Server, type Socket } from "node:net"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { notifyDesktop, resolveBusPath } from "@/src/audit/desktop-notify"; + +// A stand-in for the session bus: it speaks the SASL handshake, answers Hello, +// and then either returns an id or an error — enough to exercise every branch +// of the encoder against a real socket rather than a mock of one. +interface FakeBus { + path: string; + server: Server; + /** Every complete METHOD_CALL body the client sent, raw. */ + calls: Buffer[]; + close(): Promise; +} + +function methodReturn(replySerial: number, id: number): Buffer { + const u32 = (n: number) => { const b = Buffer.alloc(4); b.writeUInt32LE(n, 0); return b; }; + const pad8 = (b: Buffer) => { const p = (8 - (b.length % 8)) % 8; return p ? Buffer.concat([b, Buffer.alloc(p)]) : b; }; + // REPLY_SERIAL(5,u) then SIGNATURE(8,g "u") + const f1 = pad8(Buffer.concat([Buffer.from([5, 1, 0x75, 0]), u32(replySerial)])); + const f2 = pad8(Buffer.concat([Buffer.from([8, 1, 0x67, 0]), Buffer.from([1]), Buffer.from("u"), Buffer.from([0])])); + const fields = Buffer.concat([f1, f2]); + const header = Buffer.concat([Buffer.from([0x6c, 2, 0, 1]), u32(4), u32(99), u32(fields.length)]); + return Buffer.concat([pad8(Buffer.concat([header, fields])), u32(id)]); +} + +function errorReply(replySerial: number, name: string): Buffer { + const u32 = (n: number) => { const b = Buffer.alloc(4); b.writeUInt32LE(n, 0); return b; }; + const pad8 = (b: Buffer) => { const p = (8 - (b.length % 8)) % 8; return p ? Buffer.concat([b, Buffer.alloc(p)]) : b; }; + const str = (v: string) => { const body = Buffer.from(v); const o = Buffer.concat([u32(body.length), body, Buffer.from([0])]); const p = (4 - (o.length % 4)) % 4; return p ? Buffer.concat([o, Buffer.alloc(p)]) : o; }; + const f1 = pad8(Buffer.concat([Buffer.from([4, 1, 0x73, 0]), str(name)])); // ERROR_NAME + const f2 = pad8(Buffer.concat([Buffer.from([5, 1, 0x75, 0]), u32(replySerial)])); + const fields = Buffer.concat([f1, f2]); + const header = Buffer.concat([Buffer.from([0x6c, 3, 0, 1]), u32(0), u32(98), u32(fields.length)]); + return Buffer.concat([pad8(Buffer.concat([header, fields]))]); +} + +function startBus( + dir: string, + behaviour: "ok" | "no-server" | "reject-auth" | "silent", + id = 4242, +): Promise { + const path = join(dir, "bus"); + const calls: Buffer[] = []; + const server = createServer((sock: Socket) => { + let authed = false; + let sawHello = false; + sock.on("data", (chunk: Buffer) => { + const text = chunk.toString("latin1"); + if (!authed) { + if (behaviour === "reject-auth") { sock.write("REJECTED EXTERNAL\r\n"); return; } + if (text.includes("AUTH")) { authed = true; sock.write("OK 1234deadbeef\r\n"); return; } + return; + } + if (behaviour === "silent") return; + // BEGIN + Hello may arrive coalesced with the AUTH ack's reply. + if (!sawHello) { sawHello = true; sock.write(methodReturn(1, 0)); return; } + calls.push(Buffer.from(chunk)); + sock.write(behaviour === "no-server" + ? errorReply(2, "org.freedesktop.DBus.Error.ServiceUnknown") + : methodReturn(2, id)); + }); + sock.on("error", () => { /* client hangs up after its answer */ }); + }); + return new Promise((res) => { + server.listen(path, () => res({ + path, server, calls, + close: () => new Promise((r) => server.close(() => r())), + })); + }); +} + +let dir: string; +let bus: FakeBus | null = null; +let bus2: FakeBus | null = null; +beforeEach(() => { dir = mkdtempSync(join(tmpdir(), "fp-dbus-")); }); +afterEach(async () => { await bus?.close(); bus = null; await bus2?.close(); bus2 = null; rmSync(dir, { recursive: true, force: true }); }); + +describe("resolving the bus address", () => { + it("constructs one from the uid when the environment has none", () => { + // The scheduled audit's case: a child of a system service started at boot, + // whose environment carries no DBUS_SESSION_BUS_ADDRESS at all. This is + // exactly why the address is built rather than read. + expect(resolveBusPath({})).toBe(`/run/user/${process.getuid!()}/bus`); + }); + + it("prefers a declared address, because on an unusual setup it is the only right one", () => { + expect(resolveBusPath({ DBUS_SESSION_BUS_ADDRESS: "unix:path=/tmp/odd/bus" })).toBe("/tmp/odd/bus"); + expect(resolveBusPath({ DBUS_SESSION_BUS_ADDRESS: "unix:guid=abc,path=/tmp/g/bus" })).toBe("/tmp/g/bus"); + }); + + it("falls back rather than dialing an address form it cannot open", () => { + // `unix:abstract=` and `tcp:` are legal and unsupported here. Reporting + // "no session" beats connecting somewhere wrong. + for (const addr of ["unix:abstract=/tmp/dbus-xyz", "tcp:host=localhost,port=1", "garbage"]) { + expect(resolveBusPath({ DBUS_SESSION_BUS_ADDRESS: addr })).toBe(`/run/user/${process.getuid!()}/bus`); + } + }); +}); + +describe("posting a notification", () => { + it("completes the handshake and reads back the id the server assigned", async () => { + bus = await startBus(dir, "ok", 77); + const out = await notifyDesktop("failproofai", "1 credential", 0, { socketPath: bus.path }); + expect(out).toEqual({ ok: true, id: 77 }); + }); + + it("sends the arguments the spec asks for, in order", async () => { + bus = await startBus(dir, "ok"); + await notifyDesktop("summary here", "body here", 0, { socketPath: bus!.path }); + const raw = bus!.calls[0].toString("latin1"); + expect(raw).toContain("org.freedesktop.Notifications"); + expect(raw).toContain("Notify"); + expect(raw).toContain("susssasa{sv}i"); // the signature the server type-checks against + expect(raw).toContain("failproofai"); // app_name, which is how the user identifies us + expect(raw).toContain("summary here"); + expect(raw).toContain("body here"); + }); + + it("carries replaces_id so a repeat scan updates one bubble instead of stacking", async () => { + // The difference between a reminder and a nag: this runs on a timer, and + // the same finding recurs until the key is rotated. + bus = await startBus(dir, "ok"); + await notifyDesktop("s", "b", 4242, { socketPath: bus!.path }); + const body = bus!.calls[0]; + expect(body.includes(Buffer.from([0x92, 0x10, 0, 0]))).toBe(true); // 4242, little-endian + }); +}); + +// THE POINT OF THE WHOLE MODULE. A call sent with NO_REPLY_EXPECTED against a +// bus with no notification server returns success and empty output while the +// notification evaporates — which is the exact failure this feature exists to +// prevent: believing the user was told. +describe("every way this fails, it says so", () => { + it("names the case where nothing is drawing notifications", async () => { + bus = await startBus(dir, "no-server"); + const out = await notifyDesktop("s", "b", 0, { socketPath: bus.path }); + expect(out).toMatchObject({ ok: false, reason: "no-server" }); + }); + + it("reports a missing socket as no-session rather than as a failure to notify", async () => { + // Nobody is logged in, so /run/user/ does not exist. Distinguishable + // from "the desktop refused", because the remedy is different. + const out = await notifyDesktop("s", "b", 0, { socketPath: join(dir, "absent") }); + expect(out).toMatchObject({ ok: false, reason: "no-session" }); + }); + + it("reports a refused handshake as refused", async () => { + bus = await startBus(dir, "reject-auth"); + const out = await notifyDesktop("s", "b", 0, { socketPath: bus.path }); + expect(out).toMatchObject({ ok: false, reason: "refused" }); + }); + + it("gives up on a bus that accepts the connection and then says nothing", async () => { + // A hung server must not hold the audit open. The timeout is the only + // branch with no packet to trigger it. + bus = await startBus(dir, "silent"); + const out = await notifyDesktop("s", "b", 0, { socketPath: bus.path }); + expect(out).toMatchObject({ ok: false, reason: "timeout" }); + }, 10_000); + + it("never throws, whatever the socket does", async () => { + await expect(notifyDesktop("s", "b", 0, { socketPath: "/" })).resolves.toMatchObject({ ok: false }); + }); +}); + +// ── Against a REAL bus ─────────────────────────────────────────────────────── +// +// The tests above drive a server written in this same file, which is exactly +// how the first version of this module shipped a broken encoder: the fake +// mirrored the encoder's own assumptions, so both were wrong together and both +// passed. A real `dbus-daemon` rejected the very first message. These tests +// exist so that cannot happen twice. +// +// Skipped, loudly, when dbus-daemon is not installed — it is a real gap in +// coverage, not a pass. +import { spawn, execFileSync, type ChildProcess } from "node:child_process"; +import { writeFileSync, mkdirSync, existsSync } from "node:fs"; + +function haveDbus(): boolean { + try { + execFileSync("dbus-daemon", ["--version"], { stdio: "ignore" }); + return true; + } catch { + return false; + } +} + +const REAL = haveDbus() ? describe : describe.skip; + +REAL("the real dbus-daemon", () => { + // A unix socket path is capped at ~108 bytes, so this cannot live in a long + // temp dir. Private to this test and torn down after; it is never the user's + // session bus, and no notification service is registered on it, so nothing + // can be displayed by anything. + const dir = `/tmp/fpai-t${process.pid}`; + const sock = `${dir}/bus`; + let bus: ChildProcess | null = null; + + beforeEach(async () => { + mkdirSync(dir, { recursive: true }); + const conf = `${dir}/c.conf`; + writeFileSync( + conf, + ` +sessionunix:path=${sock} +`, + ); + bus = spawn("dbus-daemon", [`--config-file=${conf}`, "--nofork"], { stdio: "ignore" }); + for (let i = 0; i < 150 && !existsSync(sock); i += 1) { + await new Promise((r) => setTimeout(r, 20)); + } + }); + + afterEach(() => { + bus?.kill("SIGKILL"); + bus = null; + try { + rmSync(dir, { recursive: true, force: true }); + } catch { /* best effort */ } + }); + + // THE REGRESSION. The first encoder declared a header-fields length two bytes + // too long, because it counted the padding after the final field — padding + // that belongs to the message, not to the array. Real dbus-daemon hung up; + // the hand-written server above did not notice. Reaching a NAMED D-Bus error + // proves the daemon parsed the whole message: auth, Hello, and a Notify whose + // signature and body it type-checked before deciding nobody serves that name. + it("accepts our bytes all the way to a semantic error", async () => { + const out = await notifyDesktop("summary", "body", 0, { socketPath: sock }); + expect(out).toMatchObject({ ok: false, reason: "no-server" }); + expect((out as { detail: string }).detail).toContain("ServiceUnknown"); + }); + + it("survives a body no fake server would have stressed", async () => { + // Multibyte UTF-8 changes byte length independently of character count, and + // every string in the body is length-prefixed in BYTES. + for (const [summary, body] of [ + ["клавиша 🔑", "ghp_••••4f2a — naïve"], + ["", ""], + ["long", "x".repeat(9000)], + ["a\nb", "c\r\nd\te"], + ]) { + const out = await notifyDesktop(summary, body, 0, { socketPath: sock }); + // Still ServiceUnknown, never a parse failure or a dropped connection. + expect(out, `${summary.slice(0, 12)}`).toMatchObject({ ok: false, reason: "no-server" }); + } + }); + + it("reports a bus that goes away as something other than a timeout", async () => { + // A killed bus must not read as "your desktop is slow" — the remedies are + // completely different, and conflating them is what hid the encoder bug. + bus?.kill("SIGKILL"); + await new Promise((r) => setTimeout(r, 100)); + const out = await notifyDesktop("s", "b", 0, { socketPath: sock }); + expect(out.ok).toBe(false); + expect((out as { reason: string }).reason).not.toBe("timeout"); + }); +}); + +// ── Framing ────────────────────────────────────────────────────────────────── +// +// A real bus does not send one message per TCP chunk. It answers Hello with a +// METHOD_RETURN and then emits a NameAcquired SIGNAL, and those can arrive +// glued together or split anywhere. The first version read "the next chunk" as +// its reply, so a signal landing at the wrong moment was reported as a +// DELIVERED notification with a garbage id. +describe("message framing", () => { + function signal(): Buffer { + const u32 = (n: number) => { const b = Buffer.alloc(4); b.writeUInt32LE(n, 0); return b; }; + const pad8 = (b: Buffer) => { const p = (8 - (b.length % 8)) % 8; return p ? Buffer.concat([b, Buffer.alloc(p)]) : b; }; + const str = (v: string) => { const b = Buffer.from(v); const o = Buffer.concat([u32(b.length), b, Buffer.from([0])]); const p = (4 - (o.length % 4)) % 4; return p ? Buffer.concat([o, Buffer.alloc(p)]) : o; }; + const f = (c: number, t: string, v: Buffer) => pad8(Buffer.concat([Buffer.from([c, 1, t.charCodeAt(0), 0]), v])); + const fields = Buffer.concat([f(1, "o", str("/org/freedesktop/DBus")), f(3, "s", str("NameAcquired"))]); + const head = Buffer.concat([Buffer.from([0x6c, 4, 0, 1]), u32(0), u32(77), u32(fields.length)]); + return pad8(Buffer.concat([head, fields])); + } + + function bus(dir: string, plan: "signal-first" | "glued" | "byte-at-a-time"): Promise { + const path = join(dir, "bus2"); + const calls: Buffer[] = []; + const server = createServer((s: Socket) => { + let authed = false; + let helloDone = false; + const send = (b: Buffer) => { + if (plan === "byte-at-a-time") { for (const byte of b) s.write(Buffer.from([byte])); } + else s.write(b); + }; + s.on("data", (chunk: Buffer) => { + if (!authed) { authed = true; s.write("OK 0123456789abcdef0123456789abcdef\r\n"); return; } + if (!helloDone) { + helloDone = true; + // A signal alongside (or before) the Hello reply — what a real bus does. + if (plan === "signal-first") { send(signal()); send(methodReturn(1, 0)); } + else send(Buffer.concat([methodReturn(1, 0), signal()])); + return; + } + calls.push(Buffer.from(chunk)); + send(Buffer.concat([signal(), methodReturn(2, 9001)])); + }); + s.on("error", () => { /* client hangs up after its answer */ }); + }); + return new Promise((res) => server.listen(path, () => res({ + path, server, calls, close: () => new Promise((r) => server.close(() => r())), + }))); + } + + it("ignores a signal instead of reading it as a delivered notification", async () => { + bus2 = await bus(dir, "signal-first"); + expect(await notifyDesktop("s", "b", 0, { socketPath: bus2.path })).toEqual({ ok: true, id: 9001 }); + }); + + it("splits two messages that arrived in one chunk", async () => { + bus2 = await bus(dir, "glued"); + expect(await notifyDesktop("s", "b", 0, { socketPath: bus2.path })).toEqual({ ok: true, id: 9001 }); + }); + + it("reassembles a reply delivered one byte at a time", async () => { + // The pathological split. Every partial-read guard has to hold. + bus2 = await bus(dir, "byte-at-a-time"); + expect(await notifyDesktop("s", "b", 0, { socketPath: bus2.path })).toEqual({ ok: true, id: 9001 }); + }, 10_000); +}); diff --git a/__tests__/audit/harm-report-leaks.test.ts b/__tests__/audit/harm-report-leaks.test.ts new file mode 100644 index 000000000..bff077d59 --- /dev/null +++ b/__tests__/audit/harm-report-leaks.test.ts @@ -0,0 +1,154 @@ +// @vitest-environment node +/** + * What the emailed digest is allowed to say about a leaked credential. + * + * The whole point of the leak record is that it never stores a secret, only a + * fingerprint of one. This file is where that claim is checked against the ONE + * path that leaves the machine. + */ +import { describe, it, expect } from "vitest"; + +import { buildHarmReport, selectLeaks } from "@/src/audit/harm-report"; +import type { LeakFinding } from "@/src/audit/leak-record"; +import { fingerprintSecret } from "@/src/audit/leak-fingerprint"; +import type { AuditResult } from "@/src/audit/types"; + +const SECRET = "ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; + +function finding(over: Partial = {}): LeakFinding { + return { + id: "id-1", + fingerprint: fingerprintSecret(SECRET), + name: "GITHUB_TOKEN", + rule: "sanitize-api-keys", + confidence: "doc-verified", + firstSeen: "2026-09-01T00:00:00.000Z", + lastSeen: "2026-09-05T00:00:00.000Z", + occurrences: 3, + sightings: [ + { + cli: "claude", sessionId: "s0", cwd: "~/…/old", at: "2026-09-01T00:00:00.000Z", + mechanism: { summary: "written to ~/…/.env", toolName: "Write", direction: "input" }, + }, + { + cli: "codex", sessionId: "s1", cwd: "~/…/acme", at: "2026-09-05T00:00:00.000Z", + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "result" }, + }, + ], + ...over, + }; +} + +const WINDOW = { from: new Date("2026-09-01T00:00:00Z"), to: new Date("2026-09-08T00:00:00Z") }; + +function auditResult(): AuditResult { + return { + version: 2, + scannedAt: "2026-09-08T00:00:00.000Z", + scope: { cli: ["claude"], projects: "all", since: null }, + transcripts: { scanned: 1, skipped: 0, errors: 0, durationMs: 0 }, + results: [], + totals: { hits: 0, projectsWithHits: 0 }, + projectsScanned: [], + eventsScanned: 1, + enabledBuiltinNames: [], + }; +} + +// The property that makes this feature shippable at all. Not "the code is +// careful" — the record has no field that could hold a secret, so there is +// nothing here for a bug to send. +describe("what cannot leave the machine", () => { + it("carries no part of the secret, at any window or shape", () => { + const rows = selectLeaks([finding()], WINDOW.from, WINDOW.to); + const wire = JSON.stringify(rows); + expect(wire).not.toContain(SECRET); + // Nor a long enough run of it to be worth anything: the tail is the last 4 + // characters, and the prefix is the vendor's own public marker. + expect(wire).not.toContain(SECRET.slice(4, 24)); + expect(rows[0].display).toContain("•"); + expect(rows[0].display).toContain("ghp_"); + }); + + it("says what class it is and how long, which is what a person acts on", () => { + const [row] = selectLeaks([finding()], WINDOW.from, WINDOW.to); + expect(row.label).toContain("GitHub"); + expect(row.length).toBe(SECRET.length); + expect(row.attributed).toBe(true); + // The identifier name is not a secret, and for a first-party key it is the + // ONLY actionable field — no vendor console exists to revoke it at. + expect(row.name).toBe("GITHUB_TOKEN"); + }); +}); + +describe("the 5W1H a row has to answer", () => { + it("describes the most recent exposure, not the first", () => { + // Where the key is NOW is what matters; where it debuted is history. + const [row] = selectLeaks([finding()], WINDOW.from, WINDOW.to); + expect(row.cli).toBe("codex"); + expect(row.project).toBe("~/…/acme"); + expect(row.mechanism).toBe("read from ~/…/.env"); + expect(row.direction).toBe("result"); + expect(row.first_seen).toBe("2026-09-01T00:00:00.000Z"); + expect(row.last_seen).toBe("2026-09-05T00:00:00.000Z"); + expect(row.occurrences).toBe(3); + }); + + it("survives a finding with no sightings left rather than dropping it", () => { + // Sightings are capped and pruned; the credential is not. A row with a + // vague "how" still tells the user to rotate something. + const [row] = selectLeaks([finding({ sightings: [] })], WINDOW.from, WINDOW.to); + expect(row.display).toContain("ghp_"); + expect(row.mechanism).toBe("seen in a transcript"); + }); +}); + +describe("which findings a window includes", () => { + it("windows on last-seen, so a key still in use keeps being reported", () => { + // Windowing on firstSeen would go quiet on exactly the credentials that are + // still circulating — reporting them once, in the window they debuted. + const old = finding({ id: "old", firstSeen: "2026-01-01T00:00:00.000Z" }); + expect(selectLeaks([old], WINDOW.from, WINDOW.to)).toHaveLength(1); + }); + + it("drops one whose last sighting predates the window", () => { + const stale = finding({ id: "stale", lastSeen: "2026-06-01T00:00:00.000Z" }); + expect(selectLeaks([stale], WINDOW.from, WINDOW.to)).toHaveLength(0); + }); + + it("keeps one with no usable timestamp rather than losing it silently", () => { + const undated = finding({ id: "undated", lastSeen: "not a date" }); + expect(selectLeaks([undated], WINDOW.from, WINDOW.to)).toHaveLength(1); + }); + + it("never mails a finding the user already dismissed", () => { + // They looked at it and said it is not a secret. Mailing it weekly after + // that is how a tool teaches people to filter it out of their inbox. + const dismissed = finding({ id: "d", dismissedAt: "2026-09-06T00:00:00.000Z" }); + expect(selectLeaks([dismissed], WINDOW.from, WINDOW.to)).toHaveLength(0); + }); + + it("puts the most recently seen first", () => { + const a = finding({ id: "a", lastSeen: "2026-09-02T00:00:00.000Z" }); + const b = finding({ id: "b", lastSeen: "2026-09-07T00:00:00.000Z" }); + expect(selectLeaks([a, b], WINDOW.from, WINDOW.to).map((r) => r.id)).toEqual(["b", "a"]); + }); +}); + +describe("the report as a whole", () => { + it("carries leaks alongside the harmful counts, not instead of them", () => { + // Two different claims: `harmful` counts policy activity, `leaks` names a + // specific object to rotate. A digest that replaced one with the other + // would silently stop reporting the half that already worked. + const report = buildHarmReport(auditResult(), undefined, 7, [finding()]); + expect(report.harmful).toEqual([]); + expect(report.leaks).toHaveLength(1); + expect(report.window_to).toBe("2026-09-08T00:00:00.000Z"); + }); + + it("defaults to no leaks when the caller passes none", () => { + // Every existing caller predates this argument, and must keep producing a + // valid report rather than throwing on an undefined list. + expect(buildHarmReport(auditResult(), undefined, 7).leaks).toEqual([]); + }); +}); diff --git a/__tests__/audit/incremental-scan.test.ts b/__tests__/audit/incremental-scan.test.ts index 107231bbd..14e277d91 100644 --- a/__tests__/audit/incremental-scan.test.ts +++ b/__tests__/audit/incremental-scan.test.ts @@ -115,7 +115,14 @@ describe("a transcript that grew between audits", () => { expect(second).toEqual(first); }); - it("carries stateful detectors across the boundary", async () => { + // Skipped, not rewritten: the 8 behavioural detectors are switched off with + // the rest of the old audit (see the header of `src/audit/scoring.ts`), so + // there is no hit left to carry across the resume boundary. The plumbing this + // guards IS still in place — `sessionState` is threaded through `scanOne` and + // returned as `detectorState` exactly as before, deliberately, so restoring + // the detector loop in `src/audit/index.ts` needs no other change and this + // test should pass again the moment it comes back. Un-skip it then. + it.skip("carries stateful detectors across the boundary", async () => { // reread-after-edit pairs an Edit with a later Read of the same path, and // its countdown spans tool calls. Split exactly between the two halves of // that pair: starting the detector empty on resume loses the pairing, and diff --git a/__tests__/audit/index.test.ts b/__tests__/audit/index.test.ts index 9a05c0e2e..82ed98450 100644 --- a/__tests__/audit/index.test.ts +++ b/__tests__/audit/index.test.ts @@ -63,25 +63,35 @@ describe("runAudit() end-to-end on a fixture transcript", () => { rmSync(tmpRoot, { recursive: true, force: true }); }); - it("counts builtin + detector hits across the fixture transcript", async () => { + it("counts builtin policy hits across the fixture transcript", async () => { const result = await runAudit({ clis: ["claude"], noCache: true, noReport: true }); expect(result.transcripts.scanned).toBeGreaterThanOrEqual(1); const names = result.results.map((r) => r.name); // Builtin policy hit. expect(names.some((n) => n.includes("protect-env-vars"))).toBe(true); - // Audit-only detector hits. - expect(names).toContain("redundant-cd-cwd"); - expect(names).toContain("reread-after-edit"); + // The 8 behavioural detectors are switched off with the rest of the old + // audit (see the header of `src/audit/scoring.ts`); the modules and their + // own unit tests in `detectors.test.ts` are untouched. Restore these two + // lines with the detector loop in `src/audit/index.ts`. + // expect(names).toContain("redundant-cd-cwd"); + // expect(names).toContain("reread-after-edit"); + expect(names.every((n) => n !== "redundant-cd-cwd")).toBe(true); }); it("filters by --policy", async () => { + // Filtered on a BUILTIN rather than the detector this used to name, so the + // filter itself stays covered while the detectors are switched off. The + // original read `policies: ["redundant-cd-cwd"]` and expected that name + // back; restore it with the detector loop in `src/audit/index.ts`. const result = await runAudit({ clis: ["claude"], noCache: true, noReport: true, - policies: ["redundant-cd-cwd"], + policies: ["protect-env-vars"], }); - expect(result.results.map((r) => r.name)).toEqual(["redundant-cd-cwd"]); + const names = result.results.map((r) => r.name); + expect(names.length).toBe(1); + expect(names[0]).toContain("protect-env-vars"); }); }); diff --git a/__tests__/audit/leak-containment.test.ts b/__tests__/audit/leak-containment.test.ts new file mode 100644 index 000000000..acafe4aed --- /dev/null +++ b/__tests__/audit/leak-containment.test.ts @@ -0,0 +1,229 @@ +// @vitest-environment node +/** + * The one property the whole feature rests on: a secret goes in, and no part of + * one comes out anywhere. + * + * Every other test here checks a component. This drives the real pipeline + * end-to-end with real credential shapes and then reads back EVERY byte the run + * wrote to disk, plus the digest that leaves the machine and the notice that + * reaches the terminal, hunting for any fragment of the input. + * + * It is deliberately not a unit test. The claim being made to a user is about + * the system, not about `fingerprintSecret`, and the ways a value escapes are + * integration-shaped: a field added to the record, a new file written beside + * it, a debug string in a notice. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, readdirSync, readFileSync, rmSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { findSecrets } from "@/src/audit/leak-scan"; +import { fingerprintSecret, fingerprintId, isFindingId } from "@/src/audit/leak-fingerprint"; +import { upsertFinding } from "@/src/audit/leak-record"; +import { readLeakRecord, writeLeakRecord, activeFindings } from "@/src/audit/leak-store"; +import { buildHarmReport } from "@/src/audit/harm-report"; +import { markLeakNoticeDelivered } from "@/src/audit/leak-notice"; +import { queueMacNotification } from "@/src/audit/macos-notifier"; +import { leakNoticeText, shapeNotice } from "@/src/hooks/notice"; +import type { AuditResult } from "@/src/audit/types"; + +// Real shapes, synthetic values. One per detection class the scanner claims. +const SECRETS = [ + "ghp_A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8", + "sk-ant-api03-" + "Zq7".repeat(30), + // Assembled rather than written out, and the reason is worth knowing: a + // realistic fixture for a secret scanner is realistic enough to trip OTHER + // secret scanners. This exact value, as one literal, was rejected by GitHub + // push protection ("Push cannot contain secrets") — correctly, since it + // matches Slack's published shape byte for byte. Splitting the literal keeps + // the runtime value identical, so the scanner under test still sees a real + // Slack token, while no line in this file matches a scanner looking at source. + ["xoxb", "9876543210", "9876543210987", "ZaBcDeFgHiJkLmNoPqRsTuVw"].join("-"), + "AKIAIOSFODNN7REALKEY", + "hunter2SuperSecretPassword!", + "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.dBjftJeZ4CVPmB92K27uhbUJU1p1r8W1gFWFOEjXkFY", +] as const; + +const TRANSCRIPT = [ + `export GITHUB_TOKEN="${SECRETS[0]}"`, + `ANTHROPIC_API_KEY=${SECRETS[1]}`, + `{"slack_bot_token": "${SECRETS[2]}"}`, + `aws_access_key_id = ${SECRETS[3]}`, + `DB_PASSWORD='${SECRETS[4]}'`, + `Authorization: Bearer ${SECRETS[5]}`, + `psql postgres://admin:${SECRETS[4]}@db.internal:5432/prod`, +].join("\n"); + +function auditResult(): AuditResult { + return { + version: 2, + scannedAt: "2026-09-08T00:00:00.000Z", + scope: { cli: ["claude"], projects: "all", since: null }, + transcripts: { scanned: 1, skipped: 0, errors: 0, durationMs: 0 }, + results: [], + totals: { hits: 0, projectsWithHits: 0 }, + projectsScanned: [], + eventsScanned: 1, + enabledBuiltinNames: [], + }; +} + +/** Every file the run produced, recursively. */ +function walk(dir: string, out: string[] = []): string[] { + for (const e of readdirSync(dir, { withFileTypes: true })) { + const p = resolve(dir, e.name); + if (e.isDirectory()) walk(p, out); + else out.push(p); + } + return out; +} + +/** + * Any interior 12-character run of a secret counts as a leak. + * + * Not just the whole value: a partial disclosure is still a disclosure, and a + * bug that wrote "the first 20 characters" would pass a whole-string check + * while handing over most of the key. + */ +function fragmentsOf(secret: string): string[] { + const out: string[] = []; + for (let i = 4; i + 12 <= secret.length; i += 7) out.push(secret.slice(i, i + 12)); + return out; +} + +function assertNoSecret(label: string, text: string): void { + for (const [i, secret] of SECRETS.entries()) { + expect(text.includes(secret), `${label} contains secret #${i} in full`).toBe(false); + for (const frag of fragmentsOf(secret)) { + expect(text.includes(frag), `${label} contains a fragment of secret #${i}: ${frag}`).toBe(false); + } + } +} + +let home: string; +let prev: string | undefined; + +beforeEach(() => { + prev = process.env.FAILPROOFAI_HOME; + home = mkdtempSync(join(tmpdir(), "fp-contain-")); + process.env.FAILPROOFAI_HOME = home; +}); +afterEach(() => { + if (prev === undefined) delete process.env.FAILPROOFAI_HOME; + else process.env.FAILPROOFAI_HOME = prev; + rmSync(home, { recursive: true, force: true }); +}); + +/** Run the pipeline the way a real scan does, and hand back what it produced. */ +function runPipeline() { + const found = findSecrets(TRANSCRIPT); + const record = readLeakRecord(home); + for (const f of found) { + upsertFinding(record, { + id: fingerprintId(f.value, record.salt), + fingerprint: fingerprintSecret(f.value, f.rule), + name: f.name, + rule: f.rule, + confidence: "doc-verified", + sighting: { + cli: "claude", + sessionId: "s1", + cwd: "~/…/acme", + at: "2026-09-08T00:00:00.000Z", + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "result" }, + }, + }); + } + writeLeakRecord(record, home); + + const live = activeFindings(readLeakRecord(home), home); + const ids = live.map((f) => f.id); + markLeakNoticeDelivered(ids, home); + markLeakNoticeDelivered(ids, home, "desktop"); + for (const id of ids) queueMacNotification(id, "failproofai", "a credential leaked", home); + + return { + found, + live, + report: buildHarmReport(auditResult(), undefined, 7, live), + notice: shapeNotice("claude", leakNoticeText(found.length)), + }; +} + +describe("a secret goes in", () => { + it("is detected across every class the scanner claims", () => { + const rules = runPipeline().found.map((f) => f.rule); + for (const expected of [ + "GitHub personal access token", + "Anthropic API key", + "Slack token", + "AWS access key ID", + "JWT", + "assigned secret", + ]) { + expect(rules, expected).toContain(expected); + } + }); +}); + +describe("and no part of one comes out", () => { + it("is absent from every byte the run wrote to disk", () => { + runPipeline(); + const files = walk(home); + // A run that wrote nothing would pass this vacuously. + expect(files.length).toBeGreaterThan(5); + for (const f of files) assertNoSecret(`file ${f.replace(home, "")}`, readFileSync(f, "utf8")); + }); + + it("is absent from the digest, which is the only thing that leaves the machine", () => { + const { report } = runPipeline(); + expect(report.leaks.length).toBeGreaterThan(0); + assertNoSecret("harm report", JSON.stringify(report)); + }); + + it("is absent from the notice that reaches the terminal", () => { + const { notice } = runPipeline(); + assertNoSecret("cli notice", JSON.stringify(notice)); + }); + + it("is absent from the record every surface reads", () => { + runPipeline(); + assertNoSecret("leak record", JSON.stringify(readLeakRecord(home))); + }); + + it("still says enough to act on", () => { + // Containment is worthless if the row says nothing. Each one has to carry a + // recognisable mask, a class, a location and a mechanism. + const { report } = runPipeline(); + const row = report.leaks.find((r) => r.label.includes("GitHub")); + expect(row).toBeDefined(); + expect(row!.display).toContain("ghp_"); + expect(row!.display).toContain("•"); + expect(row!.project).toBe("~/…/acme"); + expect(row!.mechanism).toBe("read from ~/…/.env"); + expect(row!.length).toBe(SECRETS[0].length); + }); +}); + +describe("what it leaves on the filesystem", () => { + it("writes nothing world-readable", () => { + // These files name which credentials this machine leaked. Another account + // on the box learning that is a disclosure by itself, even masked. + runPipeline(); + for (const f of walk(home)) { + expect(statSync(f).mode & 0o077, `${f.replace(home, "")} is group/other readable`).toBe(0); + } + }); + + it("names every file with an id it could have minted", () => { + // Marker and queue filenames are ids. If one is ever not the minted shape, + // something built a path out of unvalidated input. + runPipeline(); + for (const f of walk(home)) { + const name = f.split("/").pop()!; + if (name.endsWith(".json")) continue; + expect(isFindingId(name), `unexpected filename ${name}`).toBe(true); + } + }); +}); diff --git a/__tests__/audit/leak-fingerprint.test.ts b/__tests__/audit/leak-fingerprint.test.ts new file mode 100644 index 000000000..65e84e5f2 --- /dev/null +++ b/__tests__/audit/leak-fingerprint.test.ts @@ -0,0 +1,104 @@ +// @vitest-environment node +import { describe, it, expect } from "vitest"; +import { fingerprintSecret, fingerprintId } from "@/src/audit/leak-fingerprint"; +import { SECRET_PATTERNS } from "@/src/hooks/builtin-policies"; + +/** Obviously-synthetic values. Shapes are real; the bytes are not. */ +const SYNTHETIC = { + githubPat: "ghp_" + "A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8", + awsKeyId: "AKIA" + "IOSFODNN7SYNTHET", + anthropic: "sk-ant-api03-" + "A".repeat(60) + "Zq4T", + slackBot: "xoxb-" + "1111111111-2222222222-SyntheticSlackTok", + password: "hunter2placeh", + firstParty: "compsynthetic0000111122223333", +}; + +describe("fingerprintSecret — identify without disclosing", () => { + it("shows a minted prefix and the last four, which is what a console displays", () => { + const fp = fingerprintSecret(SYNTHETIC.githubPat); + expect(fp.attributed).toBe(true); + expect(fp.label).toBe("GitHub personal access token"); + expect(fp.display.startsWith("ghp_")).toBe(true); + expect(fp.display.endsWith(SYNTHETIC.githubPat.slice(-4))).toBe(true); + expect(fp.length).toBe(SYNTHETIC.githubPat.length); + }); + + it("picks the longest matching prefix, so `sk-ant-api03-` beats `sk-ant-`", () => { + expect(fingerprintSecret(SYNTHETIC.anthropic).label).toBe("Anthropic API key"); + }); + + it("never reveals the middle", () => { + for (const value of Object.values(SYNTHETIC)) { + const { display } = fingerprintSecret(value); + // Every run of 5+ original characters that is not the prefix or the tail + // must be absent from the rendering. + const middle = value.slice(6, -6); + if (middle.length >= 5) { + expect(display, value).not.toContain(middle); + } + } + }); + + // The tail is the actionable half, but only when what stays hidden is + // genuinely unguessable. The corpus's one confirmed-live password was 13 + // characters at 3.19 bits/char — last-4 there is nearly a third of the secret. + it("withholds the tail from a short or unminted value", () => { + const pw = fingerprintSecret(SYNTHETIC.password, "password"); + expect(pw.attributed).toBe(false); + expect(pw.display).toBe("[13-char password]"); + expect(pw.display).not.toContain(SYNTHETIC.password.slice(-4)); + + const firstParty = fingerprintSecret(SYNTHETIC.firstParty, "assigned secret"); + expect(firstParty.attributed).toBe(false); + expect(firstParty.display).not.toContain(SYNTHETIC.firstParty.slice(-4)); + }); + + it("still says how long it was, which is how an owner recognises their own value", () => { + expect(fingerprintSecret("x".repeat(51), "API key").display).toBe("[51-char API key]"); + }); + + // THE ONE THAT MATTERS. `X`, `x` and `0` all satisfy the vendor charsets, so + // masking with them turns a redacted key back into a detectable one: + // `AKIA` + sixteen `X`s matches `AKIA[A-Z0-9]{16}`. This product has already + // measured its own output feeding back into its own corpus, one credential + // becoming seven findings across four sessions. The mask glyph must appear in + // no credential charset anywhere. + it("produces a rendering that our OWN detector cannot mistake for a live key", () => { + for (const [name, value] of Object.entries(SYNTHETIC)) { + const { display } = fingerprintSecret(value); + for (const [pattern, label] of SECRET_PATTERNS) { + const re = new RegExp(pattern.source, pattern.flags.replace("g", "")); + expect(re.test(display), `${name} rendered as "${display}" re-matched ${label}`).toBe(false); + } + } + }); + + it("proves the naive mask characters WOULD have re-matched", () => { + // Guards the reasoning above: if this ever stops being true the mask glyph + // choice is no longer load-bearing and this module's header is stale. + const naive = "AKIA" + "X".repeat(16); + const anyMatch = SECRET_PATTERNS.some(([p]) => + new RegExp(p.source, p.flags.replace("g", "")).test(naive), + ); + expect(anyMatch).toBe(true); + }); +}); + +describe("fingerprintId — stable, and not an oracle", () => { + it("is stable for the same value and salt, so a finding dedupes across scans", () => { + expect(fingerprintId("value-a", "salt-1")).toBe(fingerprintId("value-a", "salt-1")); + }); + + it("differs per value and per machine salt", () => { + expect(fingerprintId("value-a", "salt-1")).not.toBe(fingerprintId("value-b", "salt-1")); + // The salt is what stops the id being a confirmation oracle for a guessable + // secret — the corpus is full of dictionary passwords. + expect(fingerprintId("value-a", "salt-1")).not.toBe(fingerprintId("value-a", "salt-2")); + }); + + it("never contains the value", () => { + const id = fingerprintId("hunter2placeholder", "salt-1"); + expect(id).not.toContain("hunter2"); + expect(id).toMatch(/^[0-9a-f]{16}$/); + }); +}); diff --git a/__tests__/audit/leak-hostile-input.test.ts b/__tests__/audit/leak-hostile-input.test.ts new file mode 100644 index 000000000..d21e4fd14 --- /dev/null +++ b/__tests__/audit/leak-hostile-input.test.ts @@ -0,0 +1,175 @@ +// @vitest-environment node +/** + * Adversarial input, on the two paths that read attacker-shaped data. + * + * The audit reads transcripts. A transcript is a record of whatever a repository + * made an agent do — file contents, command output, pasted blobs — so every + * string reaching the scanner is, in the strict sense, hostile input from a + * source the user does not control. Two classes of bug live here, and both were + * found by measurement rather than by reading: + * + * 1. Quadratic regexes. A 300 KB unbroken token hung the scan for over 20 + * seconds. Base64 images, minified bundles and whole files arrive as single + * lines constantly; a scheduled scan hitting a few would stall for minutes + * with nobody watching. + * 2. Ids becoming filenames. A finding id is used as a path component, and + * `"../../../../tmp/PWNED"` created that file. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { findSecrets } from "@/src/audit/leak-scan"; +import { redactExample } from "@/src/audit/redact-example"; +import { isFindingId } from "@/src/audit/leak-fingerprint"; +import { markLeakNoticeDelivered, pendingLeakNotice } from "@/src/audit/leak-notice"; +import { queueMacNotification, macNotifyDir } from "@/src/audit/macos-notifier"; + +// Generous on purpose. The point is to catch a return to catastrophic +// backtracking (20_000ms+), not to police normal variation on a loaded CI box. +const BUDGET_MS = 3_000; + +const NL = String.fromCharCode(10); +const TAB = String.fromCharCode(9); + +const BOMBS: ReadonlyArray = [ + // The two that actually hung. A single long assignment value, and a plain + // base64-shaped blob — the single most common large token in a transcript. + ["a long assignment value", "A=" + "a".repeat(200_000)], + ["a 300KB unbroken token", "A".repeat(300_000)], + // The URL shape that took 4.8s inside the redactor's credential matcher. + ["a very long URL", "https://" + "a".repeat(50_000) + "?token=x"], + ["quote flood", '"'.repeat(50_000)], + ["open-quote flood", 'K="'.repeat(20_000)], + ["separator flood", "=".repeat(100_000)], + ["colon flood", ":".repeat(100_000)], + ["spaces before a separator", "K" + " ".repeat(50_000) + "=v"], + ["tab flood", ("K" + TAB).repeat(30_000) + "=v"], + ["many small assignments", Array.from({ length: 20_000 }, (_, i) => `K${i}=v${i}`).join(" ")], + ["vendor-prefix flood", "sk-".repeat(50_000)], + ["newline flood", NL.repeat(200_000)], + ["deep path", "/a".repeat(40_000)], + ["one enormous path segment", "/" + "a".repeat(200_000)], +]; + +describe("pathological input finishes", () => { + for (const [name, input] of BOMBS) { + it(`scans ${name} without backtracking`, () => { + const t0 = performance.now(); + findSecrets(input); + expect(performance.now() - t0).toBeLessThan(BUDGET_MS); + }); + + it(`redacts ${name} without backtracking`, () => { + const t0 = performance.now(); + redactExample(input); + expect(performance.now() - t0).toBeLessThan(BUDGET_MS); + }); + } + + it("stays roughly linear as the input grows", () => { + // The signature of the bug: 4x the input took ~16x the time. Linear-ish + // growth is what a bounded quantifier buys, and it is the property worth + // asserting rather than any single duration. + const time = (n: number) => { + const text = "A".repeat(n); + const t0 = performance.now(); + findSecrets(text); + redactExample(text); + return performance.now() - t0; + }; + time(20_000); // warm up, so JIT does not masquerade as growth + const small = Math.max(time(50_000), 1); + const large = time(200_000); + expect(large / small).toBeLessThan(12); // 4x input; quadratic would be ~16x + }); +}); + +// Correctness must survive the bounds — a faster redactor that stops redacting +// is not a fix. +describe("the bounds did not cost a match", () => { + it("still strips credentials out of a URL", () => { + const out = redactExample("git clone https://alice:hunter2@github.com/acme/x.git"); + expect(out).not.toContain("hunter2"); + expect(out).toContain("[REDACTED"); + }); + + it("still strips them from every scheme it used to", () => { + for (const scheme of ["http", "https", "postgres", "redis", "mongodb+srv", "amqp"]) { + const out = redactExample(`${scheme}://user:s3cr3tpassword@host/db`); + expect(out, scheme).not.toContain("s3cr3tpassword"); + } + }); + + it("still finds an assigned secret with a normal-length name", () => { + const found = findSecrets('COMPOSIO_API_KEY="abcdefghijklmnopqrstuvwxyz012345"'); + expect(found.map((f) => f.name)).toContain("COMPOSIO_API_KEY"); + }); + + it("ignores an identifier longer than any real one", () => { + // 128 is the bound. Nothing real is near it, and matching past it is what + // made the scan quadratic. + const name = "A".repeat(400); + expect(findSecrets(`${name}_SECRET="abcdefghijklmnopqrstuvwxyz012345"`)).toEqual([]); + }); +}); + +describe("an id is never allowed to be a path", () => { + let home: string; + beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "fp-hostile-")); + process.env.FAILPROOFAI_HOME = home; + }); + afterEach(() => { + delete process.env.FAILPROOFAI_HOME; + rmSync(home, { recursive: true, force: true }); + rmSync("/tmp/fp-should-not-exist", { force: true }); + }); + + const HOSTILE = [ + "../../../../tmp/fp-should-not-exist", + "..", + ".", + "", + "a/b/c", + "/tmp/fp-should-not-exist", + "a b", + "A".repeat(400), + "0123456789ABCDEF", // uppercase: not what fingerprintId mints + "0123456789abcde", // 15 chars + "0123456789abcdefg", // 17 + ]; + + it("refuses every id fingerprintId could not have produced", () => { + for (const id of HOSTILE) expect(isFindingId(id), JSON.stringify(id)).toBe(false); + expect(isFindingId("0123456789abcdef")).toBe(true); + }); + + it("does not write a notice marker outside its directory", () => { + // The measured escape. Refusing also means NOT reporting the id as won, so + // a caller never records a notice it did not actually claim. + for (const id of HOSTILE) expect(markLeakNoticeDelivered([id]), id).toEqual([]); + expect(existsSync("/tmp/fp-should-not-exist")).toBe(false); + }); + + it("does not queue a macOS banner outside its directory", () => { + for (const id of HOSTILE) expect(queueMacNotification(id, "T", "B"), id).toBe(false); + expect(existsSync("/tmp/fp-should-not-exist")).toBe(false); + }); + + it("leaves no staging file behind when a queue write is refused", () => { + // Every rejected attempt used to strand a `notify-*.tmp` beside the watched + // directory: never collected, because the agent only reads inside it. + queueMacNotification("0123456789abcdef", "T", "B"); + for (const id of HOSTILE) queueMacNotification(id, "T", "B"); + const runDirEntries = readdirSync(resolve(macNotifyDir(), "..")); + expect(runDirEntries.filter((n) => n.endsWith(".tmp"))).toEqual([]); + }); + + it("still accepts a real id", () => { + expect(markLeakNoticeDelivered(["0123456789abcdef"])).toEqual(["0123456789abcdef"]); + expect(queueMacNotification("fedcba9876543210", "T", "B")).toBe(true); + expect(pendingLeakNotice().count).toBe(0); + }); +}); diff --git a/__tests__/audit/leak-notice.test.ts b/__tests__/audit/leak-notice.test.ts new file mode 100644 index 000000000..33176635c --- /dev/null +++ b/__tests__/audit/leak-notice.test.ts @@ -0,0 +1,151 @@ +// @vitest-environment node +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { pendingLeakNotice, markLeakNoticeDelivered, pruneNoticeMarkers } from "@/src/audit/leak-notice"; +import { readLeakRecord, writeLeakRecord, dismissFinding } from "@/src/audit/leak-store"; +import { upsertFinding, type LeakSighting } from "@/src/audit/leak-record"; +import { shapeNotice, canDeliverNotice, leakNoticeText } from "@/src/hooks/notice"; + +let home: string; +beforeEach(() => { home = mkdtempSync(join(tmpdir(), "fp-notice-")); }); +afterEach(() => { rmSync(home, { recursive: true, force: true }); }); + +const FP = { display: "ghp_••••••••4f2a", label: "GitHub token", length: 40, attributed: true }; +const sighting: LeakSighting = { + cli: "claude", sessionId: "s1", cwd: "~/…/acme", at: "2026-09-08T00:00:00Z", + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "input" }, +}; +function seed(...ids: string[]) { + const r = readLeakRecord(home); + for (const id of ids) { + upsertFinding(r, { id, fingerprint: FP, name: "GITHUB_TOKEN", rule: "sanitize-api-keys", + confidence: "doc-verified", sighting }); + } + writeLeakRecord(r, home); +} + +describe("what still owes the user a notice", () => { + it("is every new finding, and nothing once claimed", () => { + seed("0000000000000a01", "0000000000000b02"); + expect(pendingLeakNotice(home).count).toBe(2); + markLeakNoticeDelivered(["0000000000000a01", "0000000000000b02"], home); + expect(pendingLeakNotice(home).count).toBe(0); + }); + + it("stays quiet for a finding the user dismissed", () => { + seed("0000000000000a01", "0000000000000b02"); + dismissFinding("0000000000000a01", home); + expect(pendingLeakNotice(home).ids).toEqual(["0000000000000b02"]); + }); + + it("returns nothing rather than guessing when the record is unreadable", () => { + expect(pendingLeakNotice("/nonexistent/path/xyz").count).toBe(0); + }); +}); + +// THE CLAIM THE DESIGN RESTS ON. A `notifiedAt` field on the record failed 100% +// of the time here: 2 concurrent sessions gave 2 notices, 8 gave 8 — and two +// sessions with DIFFERENT findings lost one mark permanently, so it re-notified +// forever. Running several agents at once in one project is how this tool is +// used, not an edge case. +describe("concurrency", () => { + it("lets exactly one claimant win, however many race", () => { + seed("0000000000000a01"); + const winners = Array.from({ length: 8 }, () => markLeakNoticeDelivered(["0000000000000a01"], home)); + expect(winners.flat()).toEqual(["0000000000000a01"]); + expect(pendingLeakNotice(home).count).toBe(0); + }); + + it("never loses a claim on a DIFFERENT finding", () => { + // The lost-update case: different findings touch different files, so both + // survive. A single shared watermark map could not do this. + seed("0000000000000a01", "0000000000000b02"); + expect(markLeakNoticeDelivered(["0000000000000a01"], home)).toEqual(["0000000000000a01"]); + expect(markLeakNoticeDelivered(["0000000000000b02"], home)).toEqual(["0000000000000b02"]); + expect(pendingLeakNotice(home).count).toBe(0); + }); + + it("reports which ids this caller actually won", () => { + seed("0000000000000a01", "0000000000000b02"); + markLeakNoticeDelivered(["0000000000000a01"], home); + expect(markLeakNoticeDelivered(["0000000000000a01", "0000000000000b02"], home)).toEqual(["0000000000000b02"]); + }); +}); + +describe("marker housekeeping", () => { + it("drops markers for findings that no longer exist", () => { + seed("0000000000000a01"); + markLeakNoticeDelivered(["0000000000000a01", "00000000000f0000"], home); + pruneNoticeMarkers(home); + // "00000000000f0000" is gone; "0000000000000a01" is still a live finding so its claim stands. + expect(pendingLeakNotice(home).count).toBe(0); + const r = readLeakRecord(home); + r.findings = []; + writeLeakRecord(r, home); + pruneNoticeMarkers(home); + expect(pendingLeakNotice(home).count).toBe(0); + }); +}); + +describe("shaping the notice per CLI", () => { + it("uses the channel each host was proven to render", () => { + const text = leakNoticeText(2); + expect(JSON.parse(shapeNotice("claude", text).stdout).systemMessage).toContain("2 credentials"); + expect(JSON.parse(shapeNotice("codex", text).stdout).systemMessage).toContain("2 credentials"); + expect(shapeNotice("copilot", text).stderr).toContain("2 credentials"); + expect(shapeNotice("factory", text).stdout).toContain("2 credentials"); + }); + + // Guessing a channel is worse than having none: it produces output that looks + // delivered from our side and reaches nobody, so we would record a leak as + // "notified" that the user never saw. + it("delivers nothing on a CLI with no proven channel", () => { + for (const cli of ["cursor", "devin", "goose", "antigravity", "hermes"] as const) { + expect(canDeliverNotice(cli), cli).toBe(false); + expect(shapeNotice(cli, leakNoticeText(1)), cli).toEqual({ stdout: "", stderr: "" }); + } + }); + + it("merges into an existing verdict instead of emitting a second JSON document", () => { + // Two JSON objects on one stream is a syntax error to every host, and it + // would take the verdict down with the courtesy message. + const verdict = JSON.stringify({ decision: "block", reason: "CI is red" }); + const out = shapeNotice("claude", leakNoticeText(1), verdict); + const parsed = JSON.parse(out.stdout); + expect(parsed.decision).toBe("block"); + expect(parsed.reason).toBe("CI is red"); + expect(parsed.systemMessage).toContain("a credential"); + }); + + it("leaves a verdict alone rather than risk destroying it", () => { + const notJson = "{ this is not json"; + expect(shapeNotice("claude", leakNoticeText(1), notJson).stdout).toBe(notJson); + }); + + it("never emits plain stdout over a verdict on factory", () => { + const verdict = JSON.stringify({ decision: "block" }); + expect(shapeNotice("factory", leakNoticeText(1), verdict).stdout).toBe(verdict); + }); +}); + +// A fixed template, with only a number interpolated. The alternative is +// interpolating a finding's example — the verbatim text of a command, which a +// repository controls via a README or an npm script, and which the redactor +// does not sanitise because it masks secrets rather than instructions. +describe("the notice text", () => { + it("interpolates a count and nothing else", () => { + expect(leakNoticeText(1)).toContain("a credential"); + expect(leakNoticeText(5)).toContain("5 credentials"); + expect(leakNoticeText(1)).toContain("failproofai audit"); + }); + + it("carries no fingerprint, path, project or command", () => { + const text = leakNoticeText(3); + expect(text).not.toContain("•"); + expect(text).not.toMatch(/[~/]\w/); + expect(text).not.toContain("ghp_"); + }); +}); diff --git a/__tests__/audit/leak-record.test.ts b/__tests__/audit/leak-record.test.ts new file mode 100644 index 000000000..1e76e5036 --- /dev/null +++ b/__tests__/audit/leak-record.test.ts @@ -0,0 +1,150 @@ +// @vitest-environment node +import { describe, it, expect } from "vitest"; +import { + upsertFinding, + pruneRecord, + emptyRecord, + describeMechanism, + MAX_SIGHTINGS, + MAX_FINDINGS, + FINDING_TTL_DAYS, + type LeakSighting, +} from "@/src/audit/leak-record"; + +const FP = { display: "ghp_••••••••4f2a", label: "GitHub personal access token", length: 40, attributed: true }; + +function sighting(at: string, sessionId = "s1"): LeakSighting { + return { + cli: "claude", + sessionId, + cwd: "~/…/acme", + at, + mechanism: describeMechanism("Read", "input", "~/…/.env"), + }; +} + +function base(id: string) { + return { id, fingerprint: FP, name: "GITHUB_TOKEN", rule: "sanitize-api-keys", confidence: "doc-verified" as const }; +} + +describe("upsertFinding — the unit is the distinct VALUE", () => { + it("reports a first sighting as new, which is what the notice keys on", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + const { isNew } = upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }); + expect(isNew).toBe(true); + expect(r.findings).toHaveLength(1); + expect(r.findings[0].occurrences).toBe(1); + }); + + // The same key seen again must NOT re-alert. One pasted credential fanned out + // to 11 files and 22 occurrences on the measured corpus, largely because the + // agent's own checkpoint records replay the prompt that carried it. + it("does not report a repeat sighting as new, however many times it recurs", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }); + for (let i = 0; i < 20; i++) { + const { isNew } = upsertFinding(r, { + ...base("a"), + sighting: sighting(`2026-09-02T00:00:${String(i).padStart(2, "0")}Z`, `s${i}`), + }); + expect(isNew).toBe(false); + } + expect(r.findings).toHaveLength(1); + expect(r.findings[0].occurrences).toBe(21); + }); + + it("counts every occurrence but stores only a bounded slice of evidence", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + for (let i = 0; i < 50; i++) { + upsertFinding(r, { + ...base("a"), + sighting: sighting(`2026-09-0${(i % 9) + 1}T00:00:00Z`, `s${i}`), + }); + } + expect(r.findings[0].occurrences).toBe(50); + expect(r.findings[0].sightings.length).toBeLessThanOrEqual(MAX_SIGHTINGS); + }); + + it("tracks first and last seen across out-of-order sightings", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-05T00:00:00Z") }); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z", "s2") }); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-09T00:00:00Z", "s3") }); + expect(r.findings[0].firstSeen).toBe("2026-09-01T00:00:00Z"); + expect(r.findings[0].lastSeen).toBe("2026-09-09T00:00:00Z"); + }); + + it("keeps distinct credentials apart", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + expect(upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }).isNew).toBe(true); + expect(upsertFinding(r, { ...base("b"), sighting: sighting("2026-09-01T00:00:00Z") }).isNew).toBe(true); + expect(r.findings).toHaveLength(2); + }); + + it("never stores the credential itself", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }); + expect(JSON.stringify(r)).not.toContain("4f2a" + "SECRET"); + // The only rendering of the value is the mask, which carries no middle. + expect(r.findings[0].fingerprint.display).toContain("•"); + }); +}); + +describe("pruneRecord — the file cannot grow without bound", () => { + it("drops findings older than the TTL", () => { + const now = Date.parse("2026-09-08T00:00:00Z"); + const old = new Date(now - (FINDING_TTL_DAYS + 5) * 86_400_000).toISOString(); + const r = emptyRecord("salt", "2026-09-08T00:00:00Z"); + upsertFinding(r, { ...base("old"), sighting: sighting(old) }); + upsertFinding(r, { ...base("new"), sighting: sighting("2026-09-08T00:00:00Z") }); + pruneRecord(r, now); + expect(r.findings.map((f) => f.id)).toEqual(["new"]); + }); + + // Age out BEFORE capping: capping first would let a burst of stale findings + // evict fresh ones, which is the opposite of what either limit is for. + it("caps the count, keeping the most recent", () => { + const now = Date.parse("2026-09-08T00:00:00Z"); + const r = emptyRecord("salt", "2026-09-08T00:00:00Z"); + for (let i = 0; i < MAX_FINDINGS + 50; i++) { + const at = new Date(now - i * 60_000).toISOString(); + upsertFinding(r, { ...base(`id-${i}`), sighting: sighting(at) }); + } + pruneRecord(r, now); + expect(r.findings).toHaveLength(MAX_FINDINGS); + expect(r.findings[0].id).toBe("id-0"); + }); + + it("stays small: a thousand sightings of one key cost the same as one", () => { + const now = Date.parse("2026-09-08T00:00:00Z"); + const r = emptyRecord("salt", "2026-09-08T00:00:00Z"); + for (let i = 0; i < 1000; i++) { + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-08T00:00:00Z", `s${i}`) }); + } + pruneRecord(r, now); + expect(JSON.stringify(r).length).toBeLessThan(2_000); + expect(r.findings[0].occurrences).toBe(1000); + }); +}); + +describe("describeMechanism — the `how` of 5W1H, specific by construction", () => { + it("names the file and the direction", () => { + expect(describeMechanism("Read", "input", "~/…/.env").summary).toBe("read from ~/…/.env"); + expect(describeMechanism("Write", "input", "~/…/deploy.sh").summary).toBe("written to ~/…/deploy.sh"); + expect(describeMechanism("Edit", "input", "~/…/config.ts").summary).toBe("edited into ~/…/config.ts"); + expect(describeMechanism("Bash", "input", null).summary).toBe("passed in a shell command"); + }); + + // Input vs result is not cosmetic: an input is deniable at PreToolUse on all + // 12 CLIs, a result is not. They are different exposures with different fixes. + it("distinguishes what the agent SENT from what it RECEIVED", () => { + expect(describeMechanism("Read", "input", "~/…/.env").direction).toBe("input"); + const out = describeMechanism("Read", "result", "~/…/.env"); + expect(out.direction).toBe("result"); + expect(out.summary).toContain("output"); + }); + + it("degrades to something still specific for an unknown tool", () => { + expect(describeMechanism("WebFetch", "input", null).summary).toBe("sent to the WebFetch tool"); + }); +}); diff --git a/__tests__/audit/leak-scan.test.ts b/__tests__/audit/leak-scan.test.ts new file mode 100644 index 000000000..ca6354b5f --- /dev/null +++ b/__tests__/audit/leak-scan.test.ts @@ -0,0 +1,223 @@ +// @vitest-environment node +import { describe, it, expect } from "vitest"; +import { findSecrets, flattenToolInput } from "@/src/audit/leak-scan"; +import { redactExample } from "@/src/audit/redact-example"; + +/** Obviously synthetic. Real shapes, invented bytes. */ +const GH = "ghp_" + "A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8"; +const FIRST_PARTY = "compsynthetic0000111122223333"; + +describe("findSecrets — vendor shapes", () => { + it("finds a vendor-shaped key with no name anywhere near it", () => { + // 55.9% of real vendor-shaped credentials in the corpus have no secret-ish + // word within 60 characters, so the shape layer has to stand alone. + const found = findSecrets(`curl -H "Authorization: Bearer ${GH}" https://api.example.com`); + expect(found.some((f) => f.value === GH && f.shaped)).toBe(true); + }); + + it("reports one finding per distinct value, not per matching rule", () => { + const found = findSecrets(`${GH} ${GH} ${GH}`); + expect(found.filter((f) => f.value === GH)).toHaveLength(1); + }); +}); + +describe("findSecrets — named assignments", () => { + // The class the vendor patterns cannot see: 230 of 237 secret-named + // assignments in the corpus match none of the shipped patterns, because + // first-party keys have no minted prefix. + it("finds a first-party key by its name alone", () => { + const found = findSecrets(`COMPOSIO_API_KEY=${FIRST_PARTY}`); + expect(found).toHaveLength(1); + expect(found[0].name).toBe("COMPOSIO_API_KEY"); + expect(found[0].shaped).toBe(false); + }); + + it("reads the same four spellings the redactor does", () => { + for (const line of [ + `MY_API_KEY=${FIRST_PARTY}`, + `MY_API_KEY = "${FIRST_PARTY}"`, + `MY_API_KEY: "${FIRST_PARTY}"`, + `{"my_api_key": "${FIRST_PARTY}"}`, + ]) { + expect(findSecrets(line).map((f) => f.value), line).toContain(FIRST_PARTY); + } + }); + + it("keeps the vendor label when a value is both shaped and named", () => { + // The vendor rule can name a console; the identifier says which of the + // user's own variables to change. Both are wanted, on one finding. + const found = findSecrets(`GITHUB_TOKEN=${GH}`); + expect(found).toHaveLength(1); + expect(found[0].shaped).toBe(true); + expect(found[0].name).toBe("GITHUB_TOKEN"); + }); +}); + +describe("findSecrets — what is structurally not a credential", () => { + // Never entropy: the corpus's confirmed-live password measures 3.19 bits per + // character while 1.3M UUIDs in the same corpus sit at 3.72. Any threshold + // that catches the password catches every UUID. + it("ignores indirection, which is the CORRECT secure form", () => { + for (const line of [ + "API_KEY=$OPENAI_API_KEY", + "API_KEY=${OPENAI_API_KEY}", + "TOKEN = process.env.DISCORD_BOT_TOKEN", + "api_key: ", + "API_KEY={{ secrets.THING }}", + ]) { + expect(findSecrets(line), line).toEqual([]); + } + }); + + it("ignores placeholders, booleans, numbers and paths", () => { + for (const line of [ + "API_KEY=xxxxxxxxxxxx", + "API_KEY=000000000000", + "AUTH_ENABLED=true", + "SESSION_KEY=1234567890", + "KEY_PATH=/etc/ssl/private/key.pem", + "API_KEY=short", + ]) { + expect(findSecrets(line), line).toEqual([]); + } + }); + + it("ignores an already-masked value, so our own output is not re-detected", () => { + // Measured: one leaked credential became seven reported findings across + // four sessions, because the audit's own output re-entered the corpus. + expect(findSecrets("API_KEY=[REDACTED: assigned secret]")).toEqual([]); + expect(findSecrets("API_KEY=ghp_••••••••4f2a")).toEqual([]); + }); + + it("does not treat a publishable key as a secret", () => { + expect(findSecrets(`NEXT_PUBLIC_POSTHOG_KEY=${FIRST_PARTY}`)).toEqual([]); + }); +}); + +// THE ONE-DIRECTIONAL PROPERTY. The detector may be narrower than the redactor; +// it must never be wider. A value the report can NAME but the redactor cannot +// MASK is a leak inside the leak report. +describe("everything the detector finds, the redactor can mask", () => { + it("holds for every named assignment shape", () => { + for (const line of [ + `COMPOSIO_API_KEY=${FIRST_PARTY}`, + `MY_API_KEY = "${FIRST_PARTY}"`, + `db_password: ${FIRST_PARTY}`, + `{"client_secret": "${FIRST_PARTY}"}`, + `sessionKey=${FIRST_PARTY}`, + `GITHUB_TOKEN=${GH}`, + ]) { + const found = findSecrets(line); + expect(found.length, line).toBeGreaterThan(0); + const masked = redactExample(line); + for (const f of found) { + expect(masked, `detector reported ${f.rule} in "${line}" that the redactor left intact`) + .not.toContain(f.value); + } + } + }); +}); + +describe("flattenToolInput", () => { + it("reaches every string leaf, not just `command`", () => { + const flat = flattenToolInput({ + file_path: "/tmp/deploy.sh", + content: `export GITHUB_TOKEN=${GH}`, + nested: { deeper: [{ more: "x" }] }, + }); + expect(findSecrets(flat).some((f) => f.value === GH)).toBe(true); + }); + + it("keeps a key adjacent to its value so JSON still reads as an assignment", () => { + const flat = flattenToolInput({ api_key: FIRST_PARTY }); + expect(findSecrets(flat).map((f) => f.name)).toContain("api_key"); + }); + + it("is bounded, so a pathological payload cannot recurse forever", () => { + let deep: unknown = "x"; + for (let i = 0; i < 50; i++) deep = { n: deep }; + expect(() => flattenToolInput(deep)).not.toThrow(); + }); +}); + +// The single highest-value refuter, and the only decisive one. Measured on the +// real corpus: 456 of 457 `AKIA` matches were AWS's own documentation literal — +// 99.8% of that pattern's entire output, 232 findings collapsing to zero. +describe("the docs-literal denylist", () => { + it("drops a credential the vendor published on purpose", () => { + // Assembled from parts so this test file does not itself contain the + // literal — a denylist written in plaintext is a file our scanner flags. + const awsDocsKey = "AKIA" + "IOSFODNN7EXAMPLE"; + expect(findSecrets(`AWS_ACCESS_KEY_ID=${awsDocsKey}`)).toEqual([]); + expect(findSecrets(`export ${awsDocsKey}`)).toEqual([]); + }); + + it("still reports a real key of the same shape", () => { + const realShape = "AKIA" + "3QXZ7YTNBVCD2WLM"; + expect(findSecrets(`AWS_ACCESS_KEY_ID=${realShape}`).length).toBeGreaterThan(0); + }); + + it("cannot false-positive — it is a hash comparison, not a heuristic", () => { + const almost = "AKIA" + "IOSFODNN7EXAMPLF"; // one byte different + expect(findSecrets(`AWS_ACCESS_KEY_ID=${almost}`).length).toBeGreaterThan(0); + }); +}); + +// Auditing for secrets writes secrets into the corpus the next audit reads. +// The largest organic cluster in 1.6GB of transcripts was one file: a previous +// audit quoting back what it had found. 108 of 161 firing patterns fired ONLY +// inside the investigation's own transcripts. +describe("self-exclusion", () => { + it("ignores our own report replayed back into a transcript", () => { + expect(findSecrets(`GITHUB_TOKEN=[REDACTED: assigned secret]`)).toEqual([]); + expect( + findSecrets(`failproofai audit found: GITHUB_TOKEN=${GH}`), + "our own report quoting a key back is not a new leak", + ).toEqual([]); + }); + + it("does not mistake ordinary work for our output", () => { + expect(findSecrets(`GITHUB_TOKEN=${GH}`).length).toBeGreaterThan(0); + }); +}); + +// The prefilter is what makes the pattern set shippable: cost is LINEAR in +// pattern count (~0.095s per pattern per 6MB), so 288 patterns over 1.6GB is +// about two hours while the few that ever match take five seconds. It must buy +// that speed without changing a single answer. +describe("the literal prefilter", () => { + it("finds exactly what an ungated scan would", () => { + // Every shape in this file's battery, run through the gated path. If the + // gate ever drops a pattern's prefix these go quiet — which is the failure + // mode that looks identical to "no secrets here". + const cases: [string, boolean][] = [ + [`GITHUB_TOKEN=${GH}`, true], + [`export ANTHROPIC_API_KEY=sk-ant-api03-${"A".repeat(40)}`, true], + [`AWS_ACCESS_KEY_ID=AKIA${"3QXZ7YTNBVCD2WLM"}`, true], + [`SLACK=xoxb-${"1111111111-2222222222-abcdefghijklmnopqrstuvwx"}`, true], + [`TELEGRAM_BOT_TOKEN=1234567890:${"A".repeat(35)}`, true], + [`COMPOSIO_API_KEY=${FIRST_PARTY}`, true], + ["git commit -m 'nothing to see'", false], + ["const x = 1; // ordinary source", false], + ]; + for (const [text, shouldFind] of cases) { + expect(findSecrets(text).length > 0, text).toBe(shouldFind); + } + }); + + it("still matches a pattern that has no literal prefix to gate on", () => { + // A connection string starts with a scheme alternation, so it yields no + // gate token and must always be run rather than silently skipped. + const found = findSecrets("psql postgresql://admin:hunter2placeholder@db.internal:5432/prod"); + expect(found.length).toBeGreaterThan(0); + }); + + it("is cheap on text that contains no credential at all", () => { + // The common case by an enormous margin: most transcript bytes are prose + // and source. 1MB of it must not cost 33 full regex passes. + const haystack = "lorem ipsum dolor sit amet ".repeat(40_000); + const started = Date.now(); + findSecrets(haystack); + expect(Date.now() - started).toBeLessThan(200); + }); +}); diff --git a/__tests__/audit/leak-section.test.tsx b/__tests__/audit/leak-section.test.tsx new file mode 100644 index 000000000..de2a20bfe --- /dev/null +++ b/__tests__/audit/leak-section.test.tsx @@ -0,0 +1,195 @@ +// @vitest-environment jsdom +/** + * The report the score was replaced with. + * + * What is pinned here is the part a reader acts on: one row per distinct + * credential (not per sighting), the masked value and never the value, the + * mechanism sentence, and the difference between a key with a console to revoke + * at and one without — which is the difference between a minute's work and an + * investigation. + */ +import { describe, it, expect, vi, beforeEach } from "vitest"; +import { render, screen, waitFor } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; + +import type { LeakRow } from "@/app/actions/get-leaks"; + +const h = vi.hoisted(() => ({ + getLeaksAction: vi.fn(), + dismissLeakAction: vi.fn(), + setLeakNotifyAction: vi.fn(), +})); +vi.mock("@/app/actions/get-leaks", () => ({ + getLeaksAction: h.getLeaksAction, + dismissLeakAction: h.dismissLeakAction, + setLeakNotifyAction: h.setLeakNotifyAction, +})); + +import { LeakSection } from "@/app/audit/_components/leak-section"; + +function row(over: Partial = {}): LeakRow { + return { + id: "id-1", + display: "ghp_••••••••4f2a", + label: "GitHub personal access token", + length: 40, + attributed: true, + name: "GITHUB_TOKEN", + cli: "claude", + project: "~/…/acme", + lastSeen: new Date(Date.now() - 86_400_000).toISOString(), + firstSeen: "2026-09-01T00:00:00.000Z", + mechanism: "read from ~/…/.env", + direction: "result", + occurrences: 7, + sessions: 3, + ...over, + }; +} + +beforeEach(() => { + vi.clearAllMocks(); + h.dismissLeakAction.mockResolvedValue(true); + h.setLeakNotifyAction.mockResolvedValue(true); +}); + +describe("what a row shows", () => { + it("shows the mask and never anything else", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + expect(await screen.findByText("ghp_••••••••4f2a")).toBeTruthy(); + // The identifier name is not a secret and is often the only actionable + // field, so it is shown alongside. + expect(screen.getByText("GITHUB_TOKEN")).toBeTruthy(); + }); + + it("answers where, who, how and when in one sentence", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + expect(screen.getByText("read from ~/…/.env")).toBeTruthy(); + expect(screen.getByText("~/…/acme")).toBeTruthy(); + expect(screen.getByText("claude")).toBeTruthy(); + expect(screen.getByText("yesterday")).toBeTruthy(); + }); + + it("counts exposures without implying that many keys", async () => { + // A key pasted into forty commands is one thing to rotate. Showing forty + // rows would be true about sightings and wrong about the work. + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + expect(await screen.findByText(/7 exposures · 3 sessions/)).toBeTruthy(); + expect(screen.getByText(/1 credential in your transcripts/)).toBeTruthy(); + }); + + it("says the key can be blocked next time only when it can", async () => { + // A result came back INTO the transcript from a file or a command — no + // PreToolUse hook can intercept that, and saying otherwise would be advice + // that cannot be followed. + h.getLeaksAction.mockResolvedValue({ + rows: [row({ id: "a", direction: "input" }), row({ id: "b", direction: "result" })], + notify: true, + }); + render(); + expect(await screen.findByText(/blockable at PreToolUse/)).toBeTruthy(); + expect(screen.getByText(/already in the transcript/)).toBeTruthy(); + }); +}); + +describe("what to do about it", () => { + it("sends an attributed key to its own console", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + expect(await screen.findByText(/rotate it in the GitHub personal access console/)).toBeTruthy(); + }); + + it("gives an unattributed key the only advice that exists for it", async () => { + // The majority case per the pattern census: no vendor, no console, and the + // identifier name is the only clue to who issued it. + h.getLeaksAction.mockResolvedValue({ + rows: [row({ attributed: false, display: "[32-char password]", name: "COMPOSIO_API_KEY" })], + notify: true, + }); + render(); + expect(await screen.findByText(/find what reads COMPOSIO_API_KEY/)).toBeTruthy(); + }); + + it("falls back to tracing when there is not even a name", async () => { + h.getLeaksAction.mockResolvedValue({ + rows: [row({ attributed: false, name: null, display: "[19-char password]" })], + notify: true, + }); + render(); + expect(await screen.findByText(/trace it to its owner/)).toBeTruthy(); + }); +}); + +describe("dismissing", () => { + it("removes the row only once the dismissal actually persisted", async () => { + // Optimism would leave the user guessing which of two states is real when + // the row reappears on the next load. + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: "not a secret" })); + + await waitFor(() => expect(screen.queryByText("ghp_••••••••4f2a")).toBeNull()); + expect(h.dismissLeakAction).toHaveBeenCalledWith("id-1"); + }); + + it("keeps the row when the dismissal failed", async () => { + h.dismissLeakAction.mockResolvedValue(false); + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: "not a secret" })); + + await waitFor(() => expect(h.dismissLeakAction).toHaveBeenCalled()); + expect(screen.getByText("ghp_••••••••4f2a")).toBeTruthy(); + }); +}); + +describe("the notification switch", () => { + it("writes the same setting the CLI and the daemon read", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: /turn desktop notifications off/ })); + + expect(h.setLeakNotifyAction).toHaveBeenCalledWith(false); + }); + + it("puts the switch back when the write failed", async () => { + // A toggle that shows "off" over a config that still says on is worse than + // one that refuses to move: the user believes they silenced it. + h.setLeakNotifyAction.mockResolvedValue(false); + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: /turn desktop notifications off/ })); + + await waitFor(() => + expect(screen.getByRole("button", { name: /turn desktop notifications off/ })).toBeTruthy(), + ); + }); +}); + +describe("nothing found", () => { + it("says so plainly rather than rendering an empty list", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [], notify: true }); + render(); + expect(await screen.findByText("no credentials found")).toBeTruthy(); + }); + + it("draws nothing at all until the record has been read", () => { + // A "no credentials found" flash before the data arrives is a false + // all-clear, which is the one wrong thing this section can say. + h.getLeaksAction.mockReturnValue(new Promise(() => {})); + const { container } = render(); + expect(container.textContent).toBe(""); + }); +}); diff --git a/__tests__/audit/leak-store.test.ts b/__tests__/audit/leak-store.test.ts new file mode 100644 index 000000000..af9f39531 --- /dev/null +++ b/__tests__/audit/leak-store.test.ts @@ -0,0 +1,197 @@ +// @vitest-environment node +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync, writeFileSync, mkdirSync, statSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, dirname } from "node:path"; + +import { + readLeakIdentity, + readLeakRecord, + writeLeakRecord, + dismissFinding, + activeFindings, +} from "@/src/audit/leak-store"; +import { upsertFinding, type LeakSighting } from "@/src/audit/leak-record"; +import { auditLeaksFile, auditLeakIdentityFile } from "@/src/hooks/fp-home"; +import { HOME_CLASSES } from "@/src/hooks/fp-home"; + +let home: string; +beforeEach(() => { home = mkdtempSync(join(tmpdir(), "fp-leak-store-")); }); +afterEach(() => { rmSync(home, { recursive: true, force: true }); }); + +const FP = { display: "ghp_••••••••4f2a", label: "GitHub token", length: 40, attributed: true }; +const sighting = (at: string): LeakSighting => ({ + cli: "claude", sessionId: "s1", cwd: "~/…/acme", at, + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "input" }, +}); +// Ids are what `fingerprintId` mints — 16 lowercase hex — and `readLeakRecord` +// now drops anything else, because an id becomes a FILENAME. The short labels +// below stay readable at the call sites and are padded into real ids here. +const idFor = (label: string) => label.padEnd(16, "0"); +const finding = (label: string) => ({ + id: idFor(label), fingerprint: FP, name: "GITHUB_TOKEN", rule: "sanitize-api-keys", + confidence: "doc-verified" as const, sighting: sighting("2026-09-08T00:00:00Z"), +}); + +describe("the salt", () => { + it("is minted once and reused, so ids stay stable across scans", () => { + const a = readLeakIdentity(home); + const b = readLeakIdentity(home); + expect(a.salt).toBe(b.salt); + expect(a.salt.length).toBeGreaterThanOrEqual(64); + }); + + it("is random per machine, not derived from anything guessable", () => { + const other = mkdtempSync(join(tmpdir(), "fp-leak-store-b-")); + try { + expect(readLeakIdentity(home).salt).not.toBe(readLeakIdentity(other).salt); + } finally { + rmSync(other, { recursive: true, force: true }); + } + }); + + it("is written 0600 — it is a machine secret", () => { + readLeakIdentity(home); + expect(statSync(auditLeakIdentityFile(home)).mode & 0o777).toBe(0o600); + }); +}); + +describe("round-trip", () => { + it("persists and reloads findings", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + expect(writeLeakRecord(r, home)).toBe(true); + const back = readLeakRecord(home); + expect(back.findings).toHaveLength(1); + expect(back.findings[0].fingerprint.display).toBe(FP.display); + }); + + it("writes the record 0600 too — it maps which project leaked what", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + writeLeakRecord(r, home); + expect(statSync(auditLeaksFile(home)).mode & 0o777).toBe(0o600); + }); + + it("never stores a raw credential", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + writeLeakRecord(r, home); + const raw = readFileSync(auditLeaksFile(home), "utf8"); + expect(raw).toContain("•"); + expect(raw).not.toMatch(/ghp_[A-Za-z0-9]{20,}/); + }); +}); + +// A scan must not fail because its own bookkeeping is damaged. The failure mode +// that matters least must never cause the one that matters most. +describe("never throws", () => { + it("treats a corrupt record as absent", () => { + mkdirSync(dirname(auditLeaksFile(home)), { recursive: true }); + writeFileSync(auditLeaksFile(home), "{ not json at all"); + expect(() => readLeakRecord(home)).not.toThrow(); + expect(readLeakRecord(home).findings).toEqual([]); + }); + + it("discards a record from a newer schema rather than guessing at it", () => { + mkdirSync(dirname(auditLeaksFile(home)), { recursive: true }); + writeFileSync( + auditLeaksFile(home), + JSON.stringify({ schemaVersion: 999, salt: "x", updatedAt: "", findings: [finding("a")] }), + ); + expect(readLeakRecord(home).findings).toEqual([]); + }); + + it("treats a corrupt identity file as absent and mints a new salt", () => { + mkdirSync(dirname(auditLeakIdentityFile(home)), { recursive: true }); + writeFileSync(auditLeakIdentityFile(home), "garbage"); + expect(() => readLeakIdentity(home)).not.toThrow(); + expect(readLeakIdentity(home).salt.length).toBeGreaterThanOrEqual(64); + }); +}); + +describe("dismissal", () => { + it("hides a finding without destroying the evidence", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + upsertFinding(r, finding("b")); + writeLeakRecord(r, home); + dismissFinding(idFor("a"), home); + + const back = readLeakRecord(home); + // Still on disk — a mis-click costs a row in a list, not the fact of a leak. + expect(back.findings).toHaveLength(2); + expect(activeFindings(back, home).map((f) => f.id)).toEqual([idFor("b")]); + }); + + it("is idempotent", () => { + dismissFinding(idFor("a"), home); + dismissFinding(idFor("a"), home); + expect(readLeakIdentity(home).dismissed).toEqual([idFor("a")]); + }); +}); + +// This is the claim that justifies two files instead of one. If either half +// were classified `derived`, a reset would silently re-alert every credential +// the machine has ever seen. +describe("reset semantics — why the split exists", () => { + it("classifies the findings as derived and the identity as identity", () => { + const cls = (fn: (h?: string) => string) => + HOME_CLASSES.find((e) => e.path(home) === fn(home))?.class; + expect(cls(auditLeaksFile)).toBe("derived"); + expect(cls(auditLeakIdentityFile)).toBe("identity"); + }); +}); + +// `leaks.json` is a file on disk: a full disk can truncate it mid-write, a hand +// can edit it, and a newer build can write a shape this one does not expect. +// Before this gate existed, one malformed entry threw out of `buildHarmReport` +// — which sits outside `reportHarm`'s try — so a scan that succeeded and cached +// correctly still exited 1, and kept doing so every run. +describe("a corrupt record", () => { + const write = (findings: unknown[]) => + writeFileSync( + auditLeaksFile(home), + JSON.stringify({ schemaVersion: 1, salt: "x", updatedAt: "", findings }), + ); + + beforeEach(() => { + // Create the directory before writing the file by hand. + mkdirSync(dirname(auditLeaksFile(home)), { recursive: true }); + }); + + it("drops entries it cannot render, and keeps the ones it can", () => { + write([ + null, + "a string", + { id: "0123456789abcdef" }, // no fingerprint + { id: "not-an-id", fingerprint: FP }, // id could not be minted + { id: "abc0000000000000", fingerprint: FP, sightings: [], firstSeen: "", lastSeen: "", occurrences: 1 }, + ]); + const kept = readLeakRecord(home).findings; + expect(kept.map((f) => f.id)).toEqual(["abc0000000000000"]); + }); + + it("keeps a finding whose sightings are unusable, and drops just those", () => { + // The credential is the thing that needs rotating. "seen in a transcript" + // with no detail beats silence about a leaked key. + write([ + { + id: "abc0000000000000", fingerprint: FP, name: null, rule: "r", + confidence: "doc-verified", firstSeen: "", lastSeen: "", occurrences: 3, + sightings: [null, { mechanism: null }, { nope: true }], + }, + ]); + const [kept] = readLeakRecord(home).findings; + expect(kept.id).toBe("abc0000000000000"); + expect(kept.sightings).toEqual([]); + expect(kept.occurrences).toBe(3); + }); + + it("never throws, whatever the file holds", () => { + for (const findings of [[{}], [[]], [{ id: 1 }], [{ id: "abc0000000000000", fingerprint: 7 }]]) { + write(findings); + expect(() => readLeakRecord(home)).not.toThrow(); + } + }); +}); diff --git a/__tests__/audit/macos-notifier.test.ts b/__tests__/audit/macos-notifier.test.ts new file mode 100644 index 000000000..f7235eb56 --- /dev/null +++ b/__tests__/audit/macos-notifier.test.ts @@ -0,0 +1,152 @@ +// @vitest-environment node +/** + * The macOS notifier, asserted from Linux. + * + * Every CI runner this project has is Linux, and the parts that need a Mac — + * `osacompile`, `launchctl bootstrap`, whether a banner actually appears — are + * exactly the parts no test here can reach. So this pins the half that IS + * platform-independent and is also the half that silently rots: the plist's + * shape, the AppleScript's structure, and the queue's on-disk contract. That is + * the same split `launchdPlistContents` already makes for the daemon's own + * plist. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync, mkdirSync, utimesSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { + MAC_NOTIFIER_LABEL, + macNotifyDir, + notifierPlistContents, + notifierScript, + queueMacNotification, + pruneMacNotifyQueue, +} from "@/src/audit/macos-notifier"; + +let home: string; +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "fp-macnote-")); + process.env.FAILPROOFAI_HOME = home; +}); +afterEach(() => { + delete process.env.FAILPROOFAI_HOME; + rmSync(home, { recursive: true, force: true }); +}); + +describe("the LaunchAgent plist", () => { + const plist = () => notifierPlistContents("/Users/x/.failproofai/bin/N.app", "/Users/x/.failproofai/run/notify"); + + it("runs the applet inside the bundle, not osascript", () => { + // A bare `osascript -e 'display notification'` is attributed to Script + // Editor: it inherits Script Editor's notification permission and shows up + // under Script Editor in System Settings. The bundle is what makes the + // banner say failproofai and gives the user something to turn off. + expect(plist()).toContain("/Users/x/.failproofai/bin/N.app/Contents/MacOS/applet"); + expect(plist()).not.toContain("osascript"); + }); + + it("is woken by the queue rather than left running", () => { + expect(plist()).toContain("WatchPaths"); + expect(plist()).toContain("/Users/x/.failproofai/run/notify"); + // Both false, and both for a reason: RunAtLoad would fire it at every login + // with nothing to say, and KeepAlive would treat its normal exit as a crash + // and restart it forever. + expect(plist()).toMatch(/RunAtLoad<\/key>\s*/); + expect(plist()).toMatch(/KeepAlive<\/key>\s*/); + }); + + it("carries the label the uninstall boots out", () => { + // These two must be the same string, or `uninstall --purge` removes the + // file and leaves launchd holding a live job pointing at a deleted applet. + expect(plist()).toContain(`${MAC_NOTIFIER_LABEL}`); + }); + + it("is well-formed XML with a real doctype", () => { + expect(plist().startsWith('')).toBe(true); + expect(plist()).toContain("")).toBe(true); + }); + + it("escapes a path that would otherwise break the document", () => { + const out = notifierPlistContents("/Users/a&b/N.app", "/Users/a/notify"); + expect(out).toContain("/Users/a&b/N.app"); + expect(out).toContain("/Users/a<b>/notify"); + }); +}); + +describe("the applet's script", () => { + it("deletes each payload BEFORE displaying it", () => { + // WatchPaths fires on every change to the directory, so a file that cannot + // be displayed and is not removed wakes this agent forever. At-most-once is + // the right failure: the caller already claimed the finding on disk, so a + // dropped banner costs one silent finding and a wake loop costs the machine. + const s = notifierScript(); + const rmAt = s.indexOf('rm -f'); + const showAt = s.indexOf("display notification"); + expect(rmAt).toBeGreaterThan(-1); + expect(showAt).toBeGreaterThan(-1); + expect(rmAt).toBeLessThan(showAt); + }); + + it("points at this home's queue, quoted", () => { + // JSON.stringify is what makes an AppleScript string literal out of a path, + // and a home containing a quote is a home this must not mis-escape. + expect(notifierScript()).toContain(JSON.stringify(macNotifyDir() + "/")); + }); + + it("reads a title and a body, and shows nothing for a one-line file", () => { + // A truncated payload is a real state — the queue writer renames into place + // precisely to avoid it, and this is the backstop if that ever regresses. + const s = notifierScript(); + expect(s).toContain("count of lines_) is greater than 1"); + }); +}); + +describe("the queue", () => { + it("lands the payload under the finding's own id", () => { + // One file per claimed finding is what keeps the banner count honest: the + // id is already unique per credential, so a repeat cannot double up. + expect(queueMacNotification("abc1230000000000", "Title", "Body text")).toBe(true); + expect(readFileSync(resolve(macNotifyDir(), "abc1230000000000"), "utf8")).toBe("Title\nBody text\n"); + }); + + it("leaves no partial file for the watcher to read", () => { + // WatchPaths fires on the first byte written, so the payload is built + // outside the directory and renamed in. Nothing but finished files ever + // appears here. + queueMacNotification("abc0000000000000", "T", "B"); + expect(readdirSync(macNotifyDir())).toEqual(["abc0000000000000"]); + expect(existsSync(resolve(macNotifyDir(), "..", "notify-abc0000000000000.tmp"))).toBe(false); + }); + + it("flattens newlines, because the applet reads the payload by line", () => { + queueMacNotification("00000000000000ff", "A\nB", "C\n\nD E"); + expect(readFileSync(resolve(macNotifyDir(), "00000000000000ff"), "utf8")).toBe("A B\nC D E\n"); + }); + + it("returns false instead of throwing when the queue cannot be written", () => { + // Read-only home, full disk, a home that is not a directory. None of them + // should turn a completed scan into a crash — the caller treats a false as + // "this channel is unavailable" and carries on. + process.env.FAILPROOFAI_HOME = "/dev/null/nope"; + expect(queueMacNotification("00000000000000ff", "T", "B")).toBe(false); + }); + + it("drops payloads nothing ever collected", () => { + // Queued while logged out, or with the agent removed. Showing them at the + // next login would announce keys that were rotated weeks ago. + mkdirSync(macNotifyDir(), { recursive: true }); + const stale = resolve(macNotifyDir(), "0000000000000010"); + const fresh = resolve(macNotifyDir(), "0000000000000011"); + writeFileSync(stale, "T\nB\n"); + writeFileSync(fresh, "T\nB\n"); + const longAgo = new Date(Date.now() - 30 * 86_400_000); + utimesSync(stale, longAgo, longAgo); + + pruneMacNotifyQueue(); + + expect(existsSync(stale)).toBe(false); + expect(existsSync(fresh)).toBe(true); + }); +}); diff --git a/__tests__/audit/notify-toggle.test.ts b/__tests__/audit/notify-toggle.test.ts new file mode 100644 index 000000000..880ff931b --- /dev/null +++ b/__tests__/audit/notify-toggle.test.ts @@ -0,0 +1,76 @@ +// @vitest-environment node +/** + * The off-switch, from the CLI side. + * + * The property that matters is not the flag — it is that this and the + * dashboard's toggle and the audit child's read are all the SAME key. Two + * surfaces that each remember their own answer is how a user silences a banner + * and keeps getting it. + */ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { runNotifyToggle } from "@/src/audit/schedule-cli"; +import { readConfig } from "@/src/hooks/fp-config"; + +let home: string; +let prev: string | undefined; +beforeEach(() => { + prev = process.env.FAILPROOFAI_HOME; + home = mkdtempSync(join(tmpdir(), "fp-notify-")); + process.env.FAILPROOFAI_HOME = home; + vi.spyOn(process.stdout, "write").mockImplementation(() => true); +}); +afterEach(() => { + vi.restoreAllMocks(); + if (prev === undefined) delete process.env.FAILPROOFAI_HOME; + else process.env.FAILPROOFAI_HOME = prev; + rmSync(home, { recursive: true, force: true }); +}); + +describe("failproofai audit --notify / --no-notify", () => { + it("writes the key the daemon's audit child reads", () => { + runNotifyToggle(false); + expect(readConfig().audit.notify).toBe(false); + expect(JSON.parse(readFileSync(resolve(home, "config.json"), "utf8")).audit.notify).toBe(false); + + runNotifyToggle(true); + expect(readConfig().audit.notify).toBe(true); + }); + + it("does not touch the schedule", () => { + // The whole reason this is its own flag pair: somebody silencing a banner + // must not discover they also switched off the weekly scan. + writeFileSync( + resolve(home, "config.json"), + JSON.stringify({ audit: { auto: true, interval_days: 30 } }), + ); + + runNotifyToggle(false); + + const after = readConfig().audit; + expect(after.auto).toBe(true); + expect(after.intervalDays).toBe(30); + expect(after.notify).toBe(false); + }); + + it("leaves unrelated settings alone", () => { + writeFileSync( + resolve(home, "config.json"), + JSON.stringify({ audit: { auto: true }, telemetry: { enabled: false } }), + ); + runNotifyToggle(false); + expect(readConfig().telemetry.enabled).toBe(false); + }); + + it("says which way it went, in both directions", () => { + const out = vi.mocked(process.stdout.write); + runNotifyToggle(false); + expect(out.mock.calls.flat().join("")).toContain("scans continue"); + out.mockClear(); + runNotifyToggle(true); + expect(out.mock.calls.flat().join("")).toContain("finds a credential"); + }); +}); diff --git a/__tests__/audit/redact-example.test.ts b/__tests__/audit/redact-example.test.ts index ded6ce649..97fc4a0de 100644 --- a/__tests__/audit/redact-example.test.ts +++ b/__tests__/audit/redact-example.test.ts @@ -227,9 +227,13 @@ describe("maskAssignedSecrets — the shape the blocking patterns do not carry", ]; for (const input of cases) { const out = redactExample(input, HOME); - expect(out, input).toContain("[REDACTED: assigned secret]"); - // The secret itself must be gone; the NAME is kept on purpose, because - // "which credential" is the actionable half of the finding. + // Any label will do. A value carrying a vendor prefix is claimed by the + // earlier SECRET_PATTERNS pass, which names it BETTER — `[REDACTED: Slack + // token]` beats `[REDACTED: assigned secret]`, because the label is what + // tells the reader which console to open. The security property under + // test is that the value is gone. + expect(out, input).toContain("[REDACTED"); + // The NAME is kept on purpose: "which credential" is the actionable half. const value = input.split("=")[1].split(" ")[0]; expect(out, input).not.toContain(value); } @@ -241,6 +245,143 @@ describe("maskAssignedSecrets — the shape the blocking patterns do not carry", ); }); + // The first spelling of ASSIGNMENT_RE was `NAME=value` and only that: no + // whitespace around the `=`, no `:`, no quoted name. Every other spelling + // reached the emailed digest with the value intact, and a value only survived + // that gap if it independently matched a vendor pattern in `SECRET_PATTERNS` + // — which 230 of 237 secret-named assignments measured on this machine do + // not. These are config, YAML and JSON, i.e. most of where credentials are + // actually written down. + it("masks assignments whatever the spacing, separator or name quoting", () => { + const cases = [ + // spaced `=` — the shape a config file or a pretty-printer writes + ["MY_API_KEY = sk-synthetic0000111122223333", "sk-synthetic0000111122223333"], + ["PGPASSWORD = letmein-prod", "letmein-prod"], + // `:` — YAML, and an HTTP-ish header line + ["db_password: letmein-prod", "letmein-prod"], + ['SLACK_BOT_TOKEN: "xoxb-synthetic-0000-1111"', "xoxb-synthetic-0000-1111"], + // quoted name — JSON + ['{"api_key": "abcdef0123456789abcdef"}', "abcdef0123456789abcdef"], + ["'client_secret': 'synthetic-secret-value'", "synthetic-secret-value"], + ] as const; + for (const [input, value] of cases) { + const out = redactExample(input, HOME); + // Any label will do — a value carrying a vendor prefix is claimed by the + // earlier `SECRET_PATTERNS` pass, which names it better than "assigned + // secret". The security property under test is that the value is gone. + expect(out, input).toContain("[REDACTED"); + expect(out, input).not.toContain(value); + } + }); + + it("re-emits the separator verbatim, so a redacted YAML line is still YAML", () => { + // Normalising every separator to `=` would turn a config excerpt into + // something that no longer looks like the file it came from, and the point + // of keeping the name is that the reader recognises the finding. + expect(redactExample("db_password: letmein", HOME)).toBe( + "db_password: [REDACTED: assigned secret]", + ); + expect(redactExample("MY_TOKEN = abcdef123456", HOME)).toBe( + "MY_TOKEN = [REDACTED: assigned secret]", + ); + }); + + // The separator grammar above is one half. This is the other: a name whose + // secret word is a camelCase hump rather than an `_` component decomposed to + // a single token that matched nothing, so `sessionKey=…` shipped its value + // verbatim while `SESSION_KEY=…` was masked. camelCase is what an identifier + // looks like everywhere except a shell environment. + it("masks a credential name written in camelCase, not just SCREAMING_SNAKE", () => { + for (const name of [ + "sessionKey", + "dbPass", + "basicAuth", + "authCookie", + "refreshToken", + "apiKeyValue", + ]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).toContain("[REDACTED: assigned secret]"); + expect(out, name).not.toContain("synthetic0000111122223333"); + } + }); + + it("knows the credential spellings of `passphrase` and `pwd`", () => { + for (const name of ["GPG_PASSPHRASE", "MYSQL_PWD", "ssh_passphrase"]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).toContain("[REDACTED: assigned secret]"); + } + }); + + it("leaves a bare PWD alone, because that is the working directory", () => { + // `PWD` is only a credential in a compound name. A bare one is on every + // second line of a captured shell session. + expect(redactExample("PWD=/home/user/project", HOME)).not.toContain("[REDACTED: assigned"); + }); + + it("does not let camelCase splitting invent new false positives", () => { + // The `_`-split negatives have to survive hump-splitting too: these are + // words that CONTAIN a secret component but are not compounds of one. + for (const input of [ + "monkeyCount=12", + "passengerList=4", + "authorName=jane", + "pathPrefix=/usr/bin", + "signalHandler=onExit", + ]) { + expect(redactExample(input, HOME), input).not.toContain("[REDACTED"); + } + }); + + // Every publishable key on earth is named `*_KEY`, and the component rule + // matched all of them — including names carrying the literal word PUBLIC. + // These are not conventionally public but MECHANICALLY public: the build tool + // inlines them into the browser bundle, so the framework published the value + // to every visitor before failproofai saw it. Reporting one as a credential + // exposure is a security tool crying wolf, which is not a cheap error the way + // ordinary over-redaction is. + it("does not mask a key the build tool ships to the browser", () => { + for (const name of [ + "NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY", + "NEXT_PUBLIC_SUPABASE_ANON_KEY", + "NEXT_PUBLIC_POSTHOG_KEY", + "VITE_API_KEY", + "REACT_APP_API_KEY", + "EXPO_PUBLIC_API_KEY", + "VAPID_PUBLIC_KEY", + "PUBLISHABLE_KEY", + "SUPABASE_ANON_KEY", + ]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).not.toContain("[REDACTED: assigned secret]"); + } + }); + + it("keeps masking a real secret whose name merely contains `public`", () => { + // The marker has to be a PREFIX or an explicit publishable suffix. A + // mid-name `PUBLIC`, or a word that merely starts with the same letters, + // must not disarm the rule. + for (const name of [ + "MY_PUBLIC_FACING_API_SECRET", + "PUBLISHER_API_KEY", + "REPUBLIC_TOKEN", + "STRIPE_SECRET_KEY", + "SUPABASE_SERVICE_ROLE_KEY", + ]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).toContain("[REDACTED"); + expect(out, name).not.toContain("synthetic0000111122223333"); + } + }); + + it("does not let a trailing `KEY:` swallow the next line", () => { + // `[ \t]*` rather than `\s*` around the separator: `\s` matches a newline, + // so a bare key at end-of-line would glue the following line into the match + // and redact it as though it were the value. + const out = redactExample("API_KEY:\nnpm run build", HOME); + expect(out).toContain("npm run build"); + }); + it("masks credentials inline in a URL, on schemes the block list omits", () => { // CONNECTION_STRING_RE deliberately excludes http/https, so this shape was // covered by nothing. diff --git a/__tests__/audit/redaction-sinks.test.ts b/__tests__/audit/redaction-sinks.test.ts index f4104cb7f..265220e57 100644 --- a/__tests__/audit/redaction-sinks.test.ts +++ b/__tests__/audit/redaction-sinks.test.ts @@ -18,6 +18,7 @@ import { readFileSync } from "node:fs"; import { join } from "node:path"; import { formatText, formatMarkdown, formatJson } from "@/src/audit/report"; +import { redactAuditResult } from "@/src/audit/redact-example"; import type { AuditCount, AuditResult } from "@/src/audit/types"; const SECRET = "ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; @@ -60,19 +61,48 @@ function result(results: AuditCount[] = [count()]): AuditResult { projectsScanned: ["/home/testuser/clients/acme-bank"], eventsScanned: 10, enabledBuiltinNames: ["block-secrets-write"], + // OPTIONAL fields must be present here or the reflection guard below has + // no teeth: it walks a real object, and TypeScript will not complain about + // an optional field the redactor forgot. This fixture is the only thing + // standing between a new field and a silent passthrough. + newLeakIds: ["abc123"], }; } +describe("redactAuditResult — a whitelist, not a spread", () => { + // It used to be `{...result}` plus three named rewrites, so every field it + // did not name passed through byte-identical — including fields added later, + // with absolute home paths intact. The compiler catches a new REQUIRED field; + // it stays quiet about an optional one, and the leak record arrives as + // optional fields. So this reflects over a real result instead. + it("has consciously handled every key on the real object", async () => { + const { REDACTED_AUDIT_RESULT_KEYS } = await import("@/src/audit/redact-example"); + const actual = Object.keys(result()).sort(); + const handled = [...REDACTED_AUDIT_RESULT_KEYS].sort(); + const unhandled = actual.filter((k) => !handled.includes(k as never)); + expect( + unhandled, + `AuditResult grew ${unhandled.join(", ")} — decide in redactAuditResult whether it needs ` + + `redacting, then add it to REDACTED_AUDIT_RESULT_KEYS`, + ).toEqual([]); + }); + + it("leaves no absolute home path anywhere in the redacted object", () => { + const out = redactAuditResult(result(), "/home/testuser"); + expect(JSON.stringify(out)).not.toContain("/home/testuser"); + }); +}); + describe("formatMarkdown — the file the CLI calls a Shareable report", () => { it("never writes a credential into the report body", () => { - const md = formatMarkdown(result(), {}); + const md = formatMarkdown(result()); expect(md).toContain("Examples"); expect(md).not.toContain(SECRET); expect(md).toContain("[REDACTED"); }); it("shortens the cwd so the report does not carry a map of someone's disk", () => { - const md = formatMarkdown(result(), {}); + const md = formatMarkdown(result()); expect(md).not.toContain("/home/testuser/clients/acme-bank"); }); }); diff --git a/__tests__/audit/scheduled-audit.test.ts b/__tests__/audit/scheduled-audit.test.ts index fcdb60e63..67fd66df0 100644 --- a/__tests__/audit/scheduled-audit.test.ts +++ b/__tests__/audit/scheduled-audit.test.ts @@ -26,6 +26,7 @@ const h = vi.hoisted(() => ({ writeDashboardCache: vi.fn(() => true), openWhenReady: vi.fn(), launch: vi.fn(), + notifyDesktop: vi.fn<(...a: unknown[]) => Promise>(() => Promise.resolve({ ok: true, id: 1 })), })); vi.mock("../../src/hooks/hook-telemetry", () => ({ trackHookEvent: h.trackHookEvent })); @@ -33,6 +34,7 @@ vi.mock("../../src/audit/index", () => ({ runAudit: h.runAudit })); vi.mock("../../src/audit/dashboard-cache", () => ({ writeDashboardCache: h.writeDashboardCache })); vi.mock("../../src/audit/open-browser", () => ({ openWhenReady: h.openWhenReady })); vi.mock("../../scripts/launch", () => ({ launch: h.launch })); +vi.mock("../../src/audit/desktop-notify", () => ({ notifyDesktop: h.notifyDesktop })); vi.mock("../../lib/telemetry-id", () => ({ getInstanceId: () => "test-instance" })); import { runAuditCli, runScheduledAudit, EXIT_AUDIT_ALREADY_RUNNING } from "../../src/audit/cli"; @@ -60,6 +62,7 @@ beforeEach(() => { vi.clearAllMocks(); h.trackHookEvent.mockImplementation(() => Promise.resolve()); h.writeDashboardCache.mockReturnValue(true); + h.notifyDesktop.mockResolvedValue({ ok: true, id: 1 }); prevHome = process.env.FAILPROOFAI_HOME; home = mkdtempSync(resolve(tmpdir(), "fpai-sched-")); process.env.FAILPROOFAI_HOME = home; @@ -367,3 +370,84 @@ describe("the binary-level scheduled entry point", () => { expect(existsSync(resolve(fp, "config.json"))).toBe(true); }, SUBPROCESS_TIMEOUT_MS); }); + +// A scan that finds a key and tells nobody is the failure this whole feature +// exists to prevent — and the scheduled run is precisely the one with nobody +// watching the terminal it printed to. +describe("announcing a leak on the desktop", () => { + const withLeaks = (ids: string[]) => + result({ totals: { hits: 1, projectsWithHits: 1 }, newLeakIds: ids }); + + const writeAuditConfig = (audit: Record) => { + mkdirSync(home, { recursive: true }); + writeFileSync(resolve(home, "config.json"), JSON.stringify({ audit })); + }; + + it("raises one banner for the credentials this run newly found", async () => { + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111", "2222222222222222"])); + + expect(await runScheduledAudit()).toBe(0); + + expect(h.notifyDesktop).toHaveBeenCalledTimes(1); + const [summary, body] = h.notifyDesktop.mock.calls[0] as unknown as [string, string]; + expect(summary).toContain("failproofai"); + expect(body).toContain("2 credentials"); + expect(body).toContain("failproofai audit"); + }); + + it("says nothing when the scan found nothing new", async () => { + // Including the repeat-scan case: the same key, already announced, is not + // news. Silence here is what makes a weekly timer tolerable. + h.runAudit.mockResolvedValue(result({ totals: { hits: 9, projectsWithHits: 3 } })); + await runScheduledAudit(); + expect(h.notifyDesktop).not.toHaveBeenCalled(); + }); + + it("announces each finding at most once, across runs", async () => { + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + await runScheduledAudit(); + expect(h.notifyDesktop).toHaveBeenCalledTimes(1); + }); + + it("does not interrupt a user who turned the banner off", async () => { + writeAuditConfig({ auto: true, notify: false }); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + expect(h.notifyDesktop).not.toHaveBeenCalled(); + }); + + it("still announces when the config has an audit table but no opinion on notifying", async () => { + writeAuditConfig({ auto: true }); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + expect(h.notifyDesktop).toHaveBeenCalledTimes(1); + }); + + it("leaves the in-session notice to fire even after the banner succeeded", async () => { + // The two channels claim separately, because `Notify` returning an id does + // NOT mean a human saw anything — on a locked screen the shell accepts the + // call and shows nothing. Letting the banner claim the finding would + // suppress the one channel that does reach them. + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + expect(existsSync(resolve(home, "audit", "notified-desktop", "1111111111111111"))).toBe(true); + expect(existsSync(resolve(home, "audit", "notified", "1111111111111111"))).toBe(false); + }); + + it("stays a successful scan when there is no desktop to notify", async () => { + // A headless box, a container, an SSH session: all normal, none of them a + // reason to report the audit itself as failed and make the scheduler back + // off from the thing that actually matters. + h.notifyDesktop.mockResolvedValue({ ok: false, reason: "no-session", detail: "ENOENT" }); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + + expect(await runScheduledAudit()).toBe(0); + }); + + it("survives a notifier that throws outright", async () => { + h.notifyDesktop.mockRejectedValue(new Error("boom")); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + expect(await runScheduledAudit()).toBe(0); + }); +}); diff --git a/__tests__/audit/share-templates.test.ts b/__tests__/audit/share-templates.test.ts index dc25d287a..27f810c4f 100644 --- a/__tests__/audit/share-templates.test.ts +++ b/__tests__/audit/share-templates.test.ts @@ -11,9 +11,14 @@ const ctx: ShareCtx = { score: 72, arch: "the cowboy", grade: "B", missing: 3 }; const cleanCtx: ShareCtx = { score: 96, arch: "the precision builder", grade: "S", missing: 0 }; describe("share templates", () => { - it("ships 10 X and 10 LinkedIn templates", () => { - expect(X_TEMPLATES).toHaveLength(10); - expect(LI_TEMPLATES).toHaveLength(10); + // Nine, not ten, since the score was switched off: one template per channel + // had the number as its entire premise ("my agent scored X/100, think yours + // can beat it?") and there was nothing left of it once the number went, so + // both are commented out in place rather than reworded. The other eighteen + // lost a subordinate clause and survive. + it("ships 9 X and 9 LinkedIn templates", () => { + expect(X_TEMPLATES).toHaveLength(9); + expect(LI_TEMPLATES).toHaveLength(9); }); it("every template ends on the npx CTA, references score or archetype, and embeds no URL", () => { @@ -33,11 +38,26 @@ describe("share templates", () => { } }); - it("references the score on most templates and the archetype on most templates", () => { - const withScore = [...X_TEMPLATES, ...LI_TEMPLATES].filter((t) => t(ctx).includes("72")); - const withArch = [...X_TEMPLATES, ...LI_TEMPLATES].filter((t) => t(ctx).includes("the cowboy")); - expect(withScore.length).toBeGreaterThanOrEqual(15); - expect(withArch.length).toBeGreaterThanOrEqual(15); + // Inverted by the score being switched off. This used to require the score on + // >=15 of 20; it now requires it on NONE, which is the assertion that keeps + // the switch honest — a template that quietly reintroduces `${score}` would + // render `undefined/100` on the shared card, since nothing upstream computes + // it any more. Restoring the score means restoring the old bound here too. + it("references the archetype on every template and the score on none", () => { + const all = [...X_TEMPLATES, ...LI_TEMPLATES]; + const withScore = all.filter((t) => t(ctx).includes("72")); + const withArch = all.filter((t) => t(ctx).includes("the cowboy")); + expect(withScore).toHaveLength(0); + expect(withArch).toHaveLength(all.length); + }); + + it("never renders `undefined` when no score is supplied", () => { + // The dashboard stops passing score/grade entirely, so the templates are + // called with neither. Any surviving interpolation would show up here. + const scoreless: ShareCtx = { arch: "the cowboy", missing: 3 }; + for (const t of [...X_TEMPLATES, ...LI_TEMPLATES]) { + expect(t(scoreless)).not.toContain("undefined"); + } }); it("tags the channel's handle (@failproofai on X, @Failproof AI on LinkedIn)", () => { diff --git a/__tests__/hooks/builtin-policies.test.ts b/__tests__/hooks/builtin-policies.test.ts index cd3c4500b..328a66a67 100644 --- a/__tests__/hooks/builtin-policies.test.ts +++ b/__tests__/hooks/builtin-policies.test.ts @@ -125,7 +125,10 @@ describe("hooks/builtin-policies", () => { const cases: Array<[string, string]> = [ ["sk-ant-api03-AAAAAAAAAAAAAAAAAAAA", "Anthropic API key"], ["sk-proj-AAAAAAAAAAAAAAAAAAAA", "OpenAI project API key"], - ["sk-AAAAAAAAAAAAAAAAAAAA", "OpenAI API key"], + ["sk-AAAAAAAAAAAAAAAAAAAA", // A bare `sk-` cannot be attributed: LiteLLM's docs say its virtual keys + // "must start with sk-", and DeepSeek and every OpenAI-compatible gateway + // mint the same shape. Naming OpenAI here sent users to the wrong console. + "OpenAI-compatible key (issuer unknown)"], ["ghp_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "GitHub personal access token"], ["AKIAIOSFODNN7EXAMPLE", "AWS access key ID"], ["sk_live_AAAAAAAAAAAAAAAAAAAAAAAA", "Stripe live secret key"], diff --git a/__tests__/hooks/fp-home.test.ts b/__tests__/hooks/fp-home.test.ts index e20a2fde1..a387cc2cf 100644 --- a/__tests__/hooks/fp-home.test.ts +++ b/__tests__/hooks/fp-home.test.ts @@ -471,7 +471,7 @@ describe("config.toml", () => { redact: "off" as const, environment: "prod", machineId: "box-1", }, telemetry: { enabled: true }, - audit: { auto: true, intervalDays: 14 }, + audit: { auto: true, notify: true, intervalDays: 14 }, }; writeConfig(cfg); expect(readConfig()).toEqual(cfg); @@ -506,29 +506,33 @@ describe("config.toml", () => { expect(readConfig().telemetry.enabled).toBe(false); }); - it("the scheduled audit is OFF by default and says so in the file", () => { - // The opposite posture to telemetry directly above: off, and deliberately - // visible, because it is a switch the user is meant to find and flip. It is - // off because the scan reads the contents of every transcript on disk. - expect(DEFAULT_CONFIG.audit).toEqual({ auto: false, intervalDays: 7 }); + it("falls back to OFF for a machine with no config at all", () => { + // DEFAULT_CONFIG is the NO-FILE answer specifically, and it is the one case + // that stays off: there is no opinion to read, and "we could not tell" must + // never start a scan that reads every transcript on disk. A machine that + // HAS a config reads the other way — the test below. + expect(DEFAULT_CONFIG.audit).toEqual({ auto: false, notify: true, intervalDays: 7 }); writeConfig(DEFAULT_CONFIG); - // Both keys on disk, unconditionally. The layout-2 file made this visible + // Every key on disk, unconditionally. The layout-2 file made this visible // with a comment block; JSON cannot carry one, so what survives is the // weaker but still real guarantee: every field the struct holds is written, // so no later regeneration can silently drop one. const written = JSON.parse(readFileSync(H.configFile(), "utf8")); - expect(written.audit).toEqual({ auto: false, interval_days: 7 }); + expect(written.audit).toEqual({ auto: false, notify: true, interval_days: 7 }); }); it("an enabled auto-audit SURVIVES a rewrite", () => { // writeConfig regenerates the whole file, so a key it does not emit is a key // it silently deletes — the failure that would turn somebody's weekly audit // off the next time any unrelated setting changed. - writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, intervalDays: 30 } }); - expect(readConfig().audit).toEqual({ auto: true, intervalDays: 30 }); + writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, notify: false, intervalDays: 30 } }); + expect(readConfig().audit).toMatchObject({ auto: true, notify: false, intervalDays: 30 }); writeConfig({ ...readConfig(), collector: { ...DEFAULT_CONFIG.collector, environment: "ci" } }); - expect(readConfig().audit).toEqual({ auto: true, intervalDays: 30 }); + // `notify: false` is the interesting half: a rewrite that dropped it would + // restore a banner the user explicitly turned off, which reads as the + // setting being ignored. + expect(readConfig().audit).toMatchObject({ auto: true, notify: false, intervalDays: 30 }); }); it("carries the consent stamp through an unrelated rewrite too", () => { @@ -539,7 +543,7 @@ describe("config.toml", () => { // leave the user with a schedule that reads as on and mails nothing. writeConfig({ ...DEFAULT_CONFIG, - audit: { auto: true, intervalDays: 30, reportsConsentedAt: 1_700_000_000_000 }, + audit: { auto: true, notify: true, intervalDays: 30, reportsConsentedAt: 1_700_000_000_000 }, }); writeConfig({ ...readConfig(), collector: { ...DEFAULT_CONFIG.collector, environment: "ci" } }); @@ -550,7 +554,7 @@ describe("config.toml", () => { it("does not invent a consent stamp for a machine that never gave one", () => { // The other direction, and the one that matters more: a default-shaped // write must not put a key on disk implying somebody was asked. - writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, intervalDays: 30 } }); + writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, notify: true, intervalDays: 30 } }); expect(readConfig().audit.reportsConsentedAt).toBeUndefined(); expect(JSON.parse(readFileSync(H.configFile(), "utf8")).audit).not.toHaveProperty( @@ -558,11 +562,64 @@ describe("config.toml", () => { ); }); - it("only an explicit true switches the auto-audit on", () => { - writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: "yes" } })); + // The three-way split, and the reason it is not a two-way one. A configured + // machine that never said anything scans; a machine we could not READ an + // opinion from does not. `crates/failproofaid/src/audit_lane.rs` makes the + // identical distinction over the identical bytes, and its own test mirrors + // this table — the daemon deciding to scan while the settings page says it is + // off (or the reverse) is the bug both tests exist to prevent. + it("scans unless the config says exactly false, but only once there IS a config", () => { + const cases: Array<[string, boolean]> = [ + [JSON.stringify({ audit: { auto: true } }), true], + [JSON.stringify({ audit: { auto: false } }), false], + // Present but silent: setup ran, nobody objected. This is the case that + // flipped, and it is the common one. + [JSON.stringify({ audit: {} }), true], + [JSON.stringify({ audit: { interval_days: 14 } }), true], + // Not `false`, so not an objection. "yes" reading as ON is the reverse of + // what this test used to assert, and it follows from the flip rather than + // being a separate decision. + [JSON.stringify({ audit: { auto: "yes" } }), true], + // No audit table at all: a config written by something that predates the + // key, so nobody was ever shown the disclosure. Not an opinion — an + // absence. + [JSON.stringify({ collector: { hooks: true } }), false], + // Unparseable, and an array is not a config object either. + ["{ not json", false], + ["[]", false], + [JSON.stringify({ audit: [] }), false], + ]; + for (const [body, expected] of cases) { + writeFileSync(H.configFile(), body); + expect(readConfig().audit.auto, body).toBe(expected); + } + rmSync(H.configFile()); expect(readConfig().audit.auto).toBe(false); - writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: true } })); - expect(readConfig().audit.auto).toBe(true); + }); + + it("notifies unless told not to, without needing an audit table to say so", () => { + // Ungated on the table's presence, unlike `auto`: this is read when a + // notification is about to fire, so a scan has already happened and the + // only question left is whether to speak. + for (const [body, expected] of [ + [JSON.stringify({ audit: { notify: false } }), false], + [JSON.stringify({ audit: { notify: true } }), true], + [JSON.stringify({ audit: {} }), true], + [JSON.stringify({ audit: { notify: "off" } }), true], + [JSON.stringify({ collector: {} }), true], + ] as Array<[string, boolean]>) { + writeFileSync(H.configFile(), body); + expect(readConfig().audit.notify, body).toBe(expected); + } + }); + + it("keeps the two switches independent", () => { + // Somebody who wants the scan and not the banner is a coherent person, and + // the only alternative this offers them is turning the scan off. + writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: true, notify: false } })); + expect(readConfig().audit).toMatchObject({ auto: true, notify: false }); + writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: false, notify: true } })); + expect(readConfig().audit).toMatchObject({ auto: false, notify: true }); }); it("resolves a nonsense interval to the default rather than to a daily scan", () => { @@ -587,7 +644,7 @@ describe("config.toml", () => { writeConfig({ ...DEFAULT_CONFIG, telemetry: { enabled: false } }); updateConfig({ audit: { auto: true } }); const after = readConfig(); - expect(after.audit).toEqual({ auto: true, intervalDays: 7 }); + expect(after.audit).toMatchObject({ auto: true, notify: true, intervalDays: 7 }); expect(after.telemetry.enabled).toBe(false); // untouched }); @@ -609,7 +666,7 @@ describe("config.toml", () => { mode: "cloud" as const, daemon: { configured: true }, telemetry: { enabled: false }, - audit: { auto: true, intervalDays: 30 }, + audit: { auto: true, notify: true, intervalDays: 30 }, collector: { ...DEFAULT_CONFIG.collector, environment: "ci", machineId: "m-1" }, }; writeConfig(config); diff --git a/__tests__/hooks/harness-extra-paths.test.ts b/__tests__/hooks/harness-extra-paths.test.ts index 58c104bac..9cc7ef58b 100644 --- a/__tests__/hooks/harness-extra-paths.test.ts +++ b/__tests__/hooks/harness-extra-paths.test.ts @@ -119,7 +119,7 @@ describe("harness extra paths", () => { redact: "off", }, telemetry: { enabled: false }, - audit: { auto: true, intervalDays: 14 }, + audit: { auto: true, notify: true, intervalDays: 14 }, }); addPath("codex", "alt=/mnt/other/.codex/sessions"); @@ -131,7 +131,7 @@ describe("harness extra paths", () => { expect(cfg.collector.machineId).toBe("m-123"); expect(cfg.collector.redact).toBe("off"); expect(cfg.telemetry.enabled).toBe(false); - expect(cfg.audit).toEqual({ auto: true, intervalDays: 14 }); + expect(cfg.audit).toMatchObject({ auto: true, notify: true, intervalDays: 14 }); expect(cfg.collector.sources?.codex.extraPaths).toEqual(["alt=/mnt/other/.codex/sessions"]); }); diff --git a/__tests__/hooks/opencode-plugin-shim.test.ts b/__tests__/hooks/opencode-plugin-shim.test.ts index 291d6bf9f..385f3aa27 100644 --- a/__tests__/hooks/opencode-plugin-shim.test.ts +++ b/__tests__/hooks/opencode-plugin-shim.test.ts @@ -83,20 +83,43 @@ async function loadShim(opts: { scope: "user" | "project"; binaryPath: string; c .replace('FAILPROOFAI_BIN = ""', `FAILPROOFAI_BIN = ${JSON.stringify(opts.binaryPath)}`); })(); - // Replace the spawnSync import with our stub. The shim imports it as - // `import { spawnSync } from "node:child_process"`. We rewrite that line - // to read from a global injected by this test. + // Replace the spawn import with our stub. The shim imports it as + // `import { spawn } from "node:child_process"` — it used to be spawnSync, + // but that blocked opencode's in-process TUI event loop for the whole + // subprocess duration. The verdicts are identical; only the wait changed + // from a blocked thread to a promise, so every assertion below is unchanged. const stubbed = shimSource.replace( - 'import { spawnSync } from "node:child_process";', - `const spawnSync = globalThis.__fp_test_spawnSync;`, + 'import { spawn } from "node:child_process";', + `const spawn = globalThis.__fp_test_spawn;`, ); - // Pre-set the stub before importing. - (globalThis as unknown as Record).__fp_test_spawnSync = (cmd: string, args: string[], optsArg: SpawnCall["opts"]): SpawnResult => { - opts.calls.push({ cmd, args, opts: optsArg }); - const r = opts.responses.shift(); - if (!r) return { status: 0, stdout: "", stderr: "" }; - return r; + // Async child stub: records the call (payload arrives on stdin now, not as + // `opts.input`) and delivers the next canned response on a later tick, the + // way a real child does. + (globalThis as unknown as Record).__fp_test_spawn = (cmd: string, args: string[], optsArg: SpawnCall["opts"]) => { + const dataHandlers: Record void)[]> = { stdout: [], stderr: [] }; + const closeHandlers: ((code: number) => void)[] = []; + const mkStream = (which: "stdout" | "stderr") => ({ + on: (ev: string, fn: (d: string) => void) => { if (ev === "data") dataHandlers[which].push(fn); }, + }); + return { + stdout: mkStream("stdout"), + stderr: mkStream("stderr"), + kill: () => {}, + on: (ev: string, fn: (code: number) => void) => { if (ev === "close") closeHandlers.push(fn); }, + stdin: { + on: () => {}, + end: (payload?: string) => { + opts.calls.push({ cmd, args, opts: { ...optsArg, input: payload } as SpawnCall["opts"] }); + const r = opts.responses.shift() ?? { status: 0, stdout: "", stderr: "" }; + queueMicrotask(() => { + if (r.stdout) for (const fn of dataHandlers.stdout) fn(r.stdout); + if (r.stderr) for (const fn of dataHandlers.stderr) fn(r.stderr); + for (const fn of closeHandlers) fn(r.status ?? 0); + }); + }, + }, + }; }; // Write a sibling .mjs we can dynamic-import without touching the original. @@ -551,15 +574,20 @@ describe("OpenCode plugin shim — spawn options and registration", () => { afterEach(() => cleanup()); - it("spawnSync includes timeout, encoding, and cwd", async () => { + // Was "spawnSync includes timeout, encoding, and cwd". The shim now uses the + // async `spawn`, which takes neither `encoding` (chunks are concatenated as + // they arrive) nor `timeout` (enforced by the shim's own 60s timer that + // SIGKILLs and fails open). `cwd` is still passed through, and the payload + // now arrives on stdin rather than as `opts.input` — both asserted here so a + // regression back to a blocking spawn is visible. + it("passes cwd and delivers the payload on stdin", async () => { responses.push({ status: 0, stdout: "", stderr: "" }); const r = await loadShim({ scope: "project", binaryPath: "/abs/bin/failproofai", calls, responses }); cleanup = r.cleanup; const hooks = await r.plugin({ client: fakeClient(), directory: "/some/cwd" }); await hooks["tool.execute.before"]!({ tool: "bash", sessionID: "s", callID: "c" }, { args: {} }); - expect(calls[0].opts.timeout).toBe(60_000); - expect(calls[0].opts.encoding).toBe("utf8"); expect(calls[0].opts.cwd).toBe("/some/cwd"); + expect(JSON.parse(String(calls[0].opts.input)).hook_event_name).toBe("PreToolUse"); }); it("registers exactly the expected hook keys", async () => { diff --git a/__tests__/hooks/pi-extension-shim.test.ts b/__tests__/hooks/pi-extension-shim.test.ts index 1f4a4d33e..68fa3a1a2 100644 --- a/__tests__/hooks/pi-extension-shim.test.ts +++ b/__tests__/hooks/pi-extension-shim.test.ts @@ -30,6 +30,10 @@ const captured: CapturedCall[] = []; * in the map gets the default empty stdout. */ const mockSpawnReplyByEvent: Record = {}; +/** Which child_process API each event actually used. Pins the blocking vs + * non-blocking split — see the `spawn` mock below. */ +const spawnApiByEvent: Record = {}; + function eventNameFromArgs(args: string[]): string | undefined { const i = args.indexOf("--hook"); return i >= 0 ? args[i + 1] : undefined; @@ -38,10 +42,33 @@ function eventNameFromArgs(args: string[]): string | undefined { vi.mock("node:child_process", () => ({ spawnSync: (_cmd: string, args: string[], opts: { input?: string }) => { captured.push({ args: args ?? [], payload: JSON.parse(opts?.input ?? "{}") }); + spawnApiByEvent[eventNameFromArgs(args ?? []) ?? "?"] = "spawnSync"; const evt = eventNameFromArgs(args ?? []); const stdout = (evt && mockSpawnReplyByEvent[evt]) ?? ""; return { pid: 0, output: [], status: 0, signal: null, stderr: "", stdout }; }, + // `session_start`, `tool_result` and `session_shutdown` go through the + // detached `forwardPolicy` instead of `callPolicy`, because Pi awaits its + // handlers serially and those three discard the verdict — blocking the TUI + // on a subprocess whose answer is thrown away. They still have to deliver + // the same payload, so the mock captures them into the same array and every + // assertion below is unchanged. The payload arrives on stdin rather than as + // `opts.input`. + spawn: (_cmd: string, args: string[]) => { + let stdinBuf = ""; + return { + unref: () => {}, + on: () => {}, + stdin: { + on: () => {}, + end: (chunk?: string) => { + stdinBuf += chunk ?? ""; + captured.push({ args: args ?? [], payload: JSON.parse(stdinBuf || "{}") }); + spawnApiByEvent[eventNameFromArgs(args ?? []) ?? "?"] = "spawn"; + }, + }, + }; + }, })); function piEncodeCwd(cwd: string): string { @@ -118,6 +145,33 @@ describe("pi-extension shim — sessionId resolution via on-disk discovery", () expect(captured.at(-1)?.payload.session_id).toBe(sid); }); + // Pi awaits its handlers serially, so a `spawnSync` inside one freezes the + // whole TUI for the subprocess's duration — measured at a 1.0-1.6s floor on + // every session start just to boot the binary, with a 60s ceiling. Three + // events discarded the verdict anyway, so they now forward detached. The + // other four consume the decision, and blocking there IS the enforcement. + it("only blocks Pi on the events whose verdict it actually reads", () => { + const sid = "66666666-6666-6666-6666-666666666666"; + writeSessionFile("/proj", sid); + for (const k of Object.keys(spawnApiByEvent)) delete spawnApiByEvent[k]; + + handlers.session_start({ type: "session_start", cwd: "/proj" }); + handlers.tool_result({ type: "tool_result", toolName: "bash", input: {}, content: [], isError: false, cwd: "/proj" }); + handlers.session_shutdown({ type: "session_shutdown", reason: "quit", cwd: "/proj" }); + expect(spawnApiByEvent.session_start).toBe("spawn"); + expect(spawnApiByEvent.tool_result).toBe("spawn"); + expect(spawnApiByEvent.session_shutdown).toBe("spawn"); + + handlers.tool_call({ type: "tool_call", toolName: "bash", input: { command: "ls" }, cwd: "/proj" }); + handlers.user_bash({ type: "user_bash", command: "ls", cwd: "/proj" }); + handlers.input({ type: "input", text: "hi", cwd: "/proj" }); + handlers.agent_end({ type: "agent_end", cwd: "/proj" }); + expect(spawnApiByEvent.tool_call).toBe("spawnSync"); + expect(spawnApiByEvent.user_bash).toBe("spawnSync"); + expect(spawnApiByEvent.input).toBe("spawnSync"); + expect(spawnApiByEvent.agent_end).toBe("spawnSync"); + }); + it("clears the per-cwd cache on session_shutdown reason=new/resume/fork", () => { const sid1 = "11111111-1111-1111-1111-111111111111"; const sid2 = "22222222-2222-2222-2222-222222222222"; diff --git a/__tests__/hooks/policy-catalog.test.ts b/__tests__/hooks/policy-catalog.test.ts index 9d4529b0c..e110c37cf 100644 --- a/__tests__/hooks/policy-catalog.test.ts +++ b/__tests__/hooks/policy-catalog.test.ts @@ -185,7 +185,10 @@ describe("policy catalog / implementation split", () => { // Its hand-written most-specific-first ORDER is load-bearing — a // Bearer-wrapped JWT reports as "JWT" today and as "bearer token" if two // entries swap. - expect(SECRET_PATTERNS).toHaveLength(13); + // 33 as of the pattern-census expansion (was 13). The count is pinned so + // an accidental deletion is loud; raise it deliberately when adding a + // vendor, and only with a doc-verified prefix behind it. + expect(SECRET_PATTERNS).toHaveLength(37); for (const [re] of SECRET_PATTERNS) expect(re).toBeInstanceOf(RegExp); }); }); diff --git a/app/actions/get-leaks.ts b/app/actions/get-leaks.ts new file mode 100644 index 000000000..9ed8677c1 --- /dev/null +++ b/app/actions/get-leaks.ts @@ -0,0 +1,100 @@ +"use server"; + +import { activeFindings, dismissFinding, readLeakRecord } from "@/src/audit/leak-store"; +import { readConfig, updateConfig } from "@/src/hooks/fp-config"; + +/** + * One credential, flattened for the report table. + * + * A separate shape from `LeakFinding` on purpose: this crosses into a client + * component, so it carries only what is drawn. The finding's raw sightings, its + * rule and its salt stay on the server side of the boundary — not because any + * of them is a secret (the record holds no secret by construction), but because + * a display type that mirrors the storage type drifts into rendering whatever + * gets added to storage next. + */ +export interface LeakRow { + id: string; + /** WHAT — `ghp_••••••••4f2a`. Never the value; there is no value to send. */ + display: string; + label: string; + length: number; + /** False means no console to revoke at — the advice changes completely. */ + attributed: boolean; + /** The identifier it was assigned to, when there was one. */ + name: string | null; + /** WHO — the harness whose transcript carried it. */ + cli: string; + /** WHERE — home-shortened project path. */ + project: string; + /** WHEN — ISO, formatted client-side so it lands in the reader's timezone. */ + lastSeen: string; + firstSeen: string; + /** HOW — "read from ~/…/.env". */ + mechanism: string; + /** WHY IT MATTERS — an input the agent sent can be denied next time; a + * result it received cannot be un-received. */ + direction: "input" | "result"; + occurrences: number; + sessions: number; +} + +export interface LeaksPayload { + rows: LeakRow[]; + /** Whether this machine may raise a desktop notification. Drawn as a toggle + * beside the table, because the table is where somebody decides the banner + * was worth it or not. */ + notify: boolean; +} + +export async function getLeaksAction(): Promise { + const findings = activeFindings(readLeakRecord()); + const rows: LeakRow[] = findings.map((f) => { + // The most recent sighting: where the key is now, not where it debuted. + const seen = f.sightings?.[f.sightings.length - 1]; + return { + id: f.id, + display: f.fingerprint?.display ?? "[credential]", + label: f.fingerprint?.label ?? "secret", + length: f.fingerprint?.length ?? 0, + attributed: f.fingerprint?.attributed === true, + name: f.name, + cli: seen?.cli ?? "unknown", + project: seen?.cwd ?? "unknown", + lastSeen: f.lastSeen, + firstSeen: f.firstSeen, + mechanism: seen?.mechanism?.summary ?? "seen in a transcript", + direction: seen?.mechanism?.direction ?? "result", + occurrences: f.occurrences, + sessions: new Set((f.sightings ?? []).map((s) => s.sessionId)).size, + }; + }); + rows.sort((a, b) => Date.parse(b.lastSeen) - Date.parse(a.lastSeen)); + return { rows, notify: readConfig().audit.notify }; +} + +/** + * Mark one finding as not-a-secret. + * + * The finding is RETAINED, not deleted — deleting it would let the next scan + * rediscover the same value and alert again, which is the one outcome that + * teaches a user the dismiss button does not work. + */ +export async function dismissLeakAction(id: string): Promise { + return dismissFinding(id); +} + +/** + * Turn the desktop banner on or off. + * + * Writes the same `audit.notify` key `failproofai config` writes and the daemon's + * audit child reads, so the dashboard and the CLI cannot disagree about it. + */ +export async function setLeakNotifyAction(notify: boolean): Promise { + try { + updateConfig({ audit: { notify } }); + return true; + } catch { + return false; + } +} diff --git a/app/audit/_components/audit-dashboard.tsx b/app/audit/_components/audit-dashboard.tsx index 372325e8a..49ba0ba13 100644 --- a/app/audit/_components/audit-dashboard.tsx +++ b/app/audit/_components/audit-dashboard.tsx @@ -3,8 +3,8 @@ /** * Top-level client wrapper for /audit. * - * Composes the calm personality report: classify the agent into one of - * 8 archetypes, derive a score, and render the 5-section flow: + * The old five-section personality report is SWITCHED OFF and being rebuilt + * bottom-up around leak detection. What it used to compose: * * 01 AuditPoster — single-screen shareable poster * 02 StrengthsSection — what it's great at @@ -12,23 +12,36 @@ * 04 HowToImproveSection — install / configure * 05 ComeBackBetterSection — spread the audit (invite) * + * Every one of those components, and the score / persona / strengths / findings + * modules behind them, is intact and still unit-tested — only the renders and + * the imports here are commented out, so restoring any of it is deleting a pair + * of comment markers. See the header of `src/audit/scoring.ts` for why. + * + * What this page renders now is `AuditReportPlaceholder`: the scan's own + * numbers. The scan, the 12-CLI transcript reader and the emailed digest all + * still run. + * * Empty / running states fall back to EmptyState and RunProgress. */ import { useCallback, useEffect, useMemo, useRef, useState } from "react"; import { getAuditResultAction } from "@/app/actions/get-audit-result"; import type { AuditResult, RunAuditOptions } from "@/src/audit/types"; -import { classifyAgent } from "@/src/audit/archetypes"; -import { deriveScore, gradeFor, projectedScore } from "@/src/audit/scoring"; -import { deriveStrengths } from "@/src/audit/strengths"; -import { deriveFindings } from "@/src/audit/findings"; +// import { classifyAgent } from "@/src/audit/archetypes"; +// Score switched off — see the header of `src/audit/scoring.ts`. +// import { deriveScore, gradeFor, projectedScore } from "@/src/audit/scoring"; +// import { deriveStrengths } from "@/src/audit/strengths"; +// import { deriveFindings } from "@/src/audit/findings"; import { usePostHog } from "@/contexts/PostHogContext"; -import { AuditPoster } from "./audit-poster"; -import { StrengthsSection } from "./strengths-section"; -import { QuirksSection } from "./quirks-section"; -import { HowToImproveSection } from "./how-to-improve-section"; -import { ComeBackBetterSection } from "./come-back-better-section"; +// The old report's five sections — switched off, not deleted. Each component +// file and its tests are untouched; only this render is commented out. +// import { AuditPoster } from "./audit-poster"; +// import { StrengthsSection } from "./strengths-section"; +// import { QuirksSection } from "./quirks-section"; +// import { HowToImproveSection } from "./how-to-improve-section"; +// import { ComeBackBetterSection } from "./come-back-better-section"; import { ReportFooter } from "./report-footer"; +import { LeakSection } from "./leak-section"; import { EmptyState } from "./empty-state"; import { RunProgress } from "./run-progress"; import { AuditProgressStrip, type RerunStatus } from "./audit-progress-strip"; @@ -59,6 +72,9 @@ interface Props { totalCatalogSize: number; } +/* Switched off with the persona layer — this named the project that seeded the + archetype classifier and the leaderboard row. Restore alongside + `classifyAgent` in this file. function inferProjectName(result: AuditResult, override?: string): string { if (override && override.trim()) return override; // Pick the cwd that appears in the most examples — proxy for "your @@ -81,6 +97,7 @@ function inferProjectName(result: AuditResult, override?: string): string { if (segs.length >= 2) return `${segs[segs.length - 2]} / ${segs[segs.length - 1]}`; return segs[segs.length - 1] ?? "your agent"; } +*/ export function AuditDashboard({ initial, projectFromUrl, totalCatalogSize }: Props) { const [cache, setCache] = useState(initial); @@ -279,16 +296,22 @@ function MainReport({ onDismissRerun, }: MainReportProps) { const { capture } = usePostHog(); - const project = useMemo(() => inferProjectName(result, projectFromUrl), [result, projectFromUrl]); + // Only fed the persona classifier, which is switched off with it. + // const project = useMemo(() => inferProjectName(result, projectFromUrl), [result, projectFromUrl]); // Seed classification with the project name so the behaviour fingerprint // (used for tie-breaks + copy variants) is stable per project. - const classification = useMemo(() => classifyAgent(result, project), [result, project]); - const score = useMemo(() => deriveScore(result), [result]); - const projected = useMemo(() => projectedScore(result, score), [result, score]); - const grade = gradeFor(score); - const projectedGrade = gradeFor(projected); - const strengths = useMemo(() => deriveStrengths(result), [result]); - const findings = useMemo(() => deriveFindings(result), [result]); + // Persona classification is switched off with the rest of the old report. Its + // last live use was two telemetry properties describing a persona the product + // no longer shows anyone. + // const classification = useMemo(() => classifyAgent(result, project), [result, project]); + // Score switched off — see the header of `src/audit/scoring.ts`. The four + // functions are intact and still unit-tested; nothing calls them. + // const score = useMemo(() => deriveScore(result), [result]); + // const projected = useMemo(() => projectedScore(result, score), [result, score]); + // const grade = gradeFor(score); + // const projectedGrade = gradeFor(projected); + // const strengths = useMemo(() => deriveStrengths(result), [result]); + // const findings = useMemo(() => deriveFindings(result), [result]); // One pass over result.results: detectors triggered + missing prescribed // policies. Both feed PostHog instrumentation; `missing` also feeds the @@ -310,10 +333,13 @@ function MainReport({ if (dashboardViewedRef.current) return; dashboardViewedRef.current = true; capture("audit_dashboard_viewed", { - score, - grade, - archetype: classification.archetype, - secondary: classification.secondary ?? null, + // Score switched off — see `src/audit/scoring.ts`. `missing` is the + // prescription's denominator and is unaffected, so the copy→install + // funnel keeps both of its ends. + // score, + // grade, + // archetype: classification.archetype, + // secondary: classification.secondary ?? null, missing, transcripts_scanned: result.transcripts.scanned, results_count: result.results.length, @@ -321,10 +347,6 @@ function MainReport({ }); }, [ capture, - score, - grade, - classification.archetype, - classification.secondary, missing, result.transcripts.scanned, result.results.length, @@ -332,19 +354,41 @@ function MainReport({ ]); /** Poster ref — captured to PNG by the poster's share buttons. */ - const posterRef = useRef(null); + // const posterRef = useRef(null); return (
+ + {/* The rebuilt report. It reads the leak record rather than this + scan's result, deliberately: a credential found last month and + not touched by today's transcripts is still a credential to + rotate, and a report that showed only the current scan's findings + would go quiet on exactly those. */} + + {/* The whole old report is switched off — the persona poster, the + strengths and quirks sections, the punch-list with its install-all + funnel, and the invite section. It is being rebuilt bottom-up + around leak detection; see the header of `src/audit/scoring.ts` + for the reasoning and `~/Desktop/failproofai-leak-detection- + verdict-2026-09-07.md` for the measurements behind it. + + Every component below is intact and still unit-tested — only the + render is commented — so restoring any one of them is deleting a + pair of comment markers. + @@ -352,10 +396,11 @@ function MainReport({ - + + */}
@@ -363,6 +408,40 @@ function MainReport({ ); } +/** + * What `/audit` shows while the report is being rebuilt. + * + * Deliberately states the scan's own numbers rather than nothing at all: the + * scan still runs, still walks every transcript across all 12 CLIs, and still + * feeds the emailed digest. Saying so is the difference between "this page is + * under construction" and "this product is broken". + */ +function AuditReportPlaceholder( + { transcripts, events, projects }: { transcripts: number; events: number; projects: number }, +) { + const n = (x: number) => x.toLocaleString(); + return ( +
+
+ + 01{"// scan"} + +
+

scan complete

+
+ {n(events)} tool call{events === 1 ? "" : "s"} across {n(transcripts)}{" "} + transcript{transcripts === 1 ? "" : "s"} + {projects > 0 ? ` · ${n(projects)} project${projects === 1 ? "" : "s"}` : ""} +
+
+ {"// the report is being rebuilt around leak detection. the scan, the"} +
+ {"// 12-CLI transcript reader and the emailed digest are unaffected."} +
+
+ ); +} + interface ShellEmptyProps { running: boolean; mode?: "no-cache" | "zero-sessions"; diff --git a/app/audit/_components/audit-poster.tsx b/app/audit/_components/audit-poster.tsx index 9822eda34..91784e9e2 100644 --- a/app/audit/_components/audit-poster.tsx +++ b/app/audit/_components/audit-poster.tsx @@ -24,7 +24,12 @@ import React, { forwardRef, useMemo, useState } from "react"; import { pickArchetypeVariant, type ArchetypeKey } from "@/src/audit/archetypes"; import { type Grade } from "@/src/audit/scoring"; -import { getArchetypeRarityPct } from "@/src/audit/social-proof"; +// Archetype rarity switched off — the percentages are eight hardcoded integers +// ("Seeded with snapshot values; swap for live aggregates once that pipeline +// lands"), and this is the ONLY surface that rendered them: baked by +// html-to-image into the PNG people post publicly. A fabricated population +// statistic on a shared card is a claim we cannot support. +// import { getArchetypeRarityPct } from "@/src/audit/social-proof"; import { copyOrDownloadCard, downloadCard, shareCardNative, shareCardToastMessage } from "@/lib/share-card"; import { toast } from "@/app/components/toast"; import { usePostHog } from "@/contexts/PostHogContext"; @@ -45,8 +50,11 @@ interface Props { archetypeKey: ArchetypeKey; /** Stable seed for variant selection (project name is the natural fit). */ seed: string; - score: number; - grade: Grade; + /** The score is switched off (see `src/audit/scoring.ts`). Kept as optional + * props rather than removed, so restoring is un-commenting rather than + * re-threading: the dashboard simply stops passing them. */ + score?: number; + grade?: Grade; /** Count of unenabled prescribed policies — passed to the share-text * templates, not rendered on the poster itself. */ missing: number; @@ -63,7 +71,7 @@ export const AuditPoster = forwardRef(function AuditPoste () => pickArchetypeVariant(archetypeKey, seed), [archetypeKey, seed], ); - const rarityPct = getArchetypeRarityPct(archetypeKey); + // const rarityPct = getArchetypeRarityPct(archetypeKey); const indexLabel = String(archetype.index).padStart(2, "0"); const auditedDate = useMemo(() => formatAuditedDate(auditedAt), [auditedAt]); @@ -134,8 +142,11 @@ export const AuditPoster = forwardRef(function AuditPoste } }; + // Score switched off — the filename is keyed on the archetype instead, which + // is what the card now actually shows. Original: + // `failproofai-${channel}-${grade.toLowerCase()}-${score}.png` const filenameFor = (channel: "x" | "linkedin" | "download") => - `failproofai-${channel}-${grade.toLowerCase()}-${score}.png`; + `failproofai-${channel}-${archetypeKey}.png`; const handleShare = async (channel: "x" | "linkedin" | "download") => { if (busy) return; @@ -143,8 +154,11 @@ export const AuditPoster = forwardRef(function AuditPoste capture("audit_card_share_clicked", { channel, source: "poster", - score, - grade, + // Score switched off — see `src/audit/scoring.ts`. Note for whoever reads + // the funnel: these two properties stop appearing from this release, so a + // dashboard keyed on them breaks rather than reading zero. + // score, + // grade, missing_policies: missing, }); try { @@ -174,10 +188,10 @@ export const AuditPoster = forwardRef(function AuditPoste } const shareCtx: ShareCtx = { - score, arch: archetype.name.toLowerCase(), - grade, missing, + // score, + // grade, }; const shareText = channel === "x" ? pickTemplate(X_TEMPLATES, seed, shareCtx) @@ -248,6 +262,7 @@ export const AuditPoster = forwardRef(function AuditPoste ))}
+ {/* Rarity switched off — see the import above. {typeof rarityPct === "number" && (
{"// only"}{" "} @@ -255,13 +270,17 @@ export const AuditPoster = forwardRef(function AuditPoste of agents are this archetype
)} + */} - {/* Score block — heroic number, centered in the card */} + {/* Score block — heroic number, centered in the card. Switched off; + see `src/audit/scoring.ts` for why. The archetype is the card's + headline now.
{score} /100
+ */}