diff --git a/.agents/skills/policy-author/SKILL.md b/.agents/skills/policy-author/SKILL.md new file mode 100644 index 000000000..d9b98c711 --- /dev/null +++ b/.agents/skills/policy-author/SKILL.md @@ -0,0 +1,520 @@ +--- +name: policy-author +description: |- + The way to turn what an agent keeps doing wrong into enforcement — failproofai policies that fire on every tool call. Reach for it on vague phrasing: "my agent keeps force-pushing", "it deleted my files again" — a complaint, not a policy request. + + Trigger when the user wants to: + • act on an audit — turn `failproofai audit` findings into fixes, or ask which policies actually work; + • stop a recurring behaviour, in plain words or as "write a policy that blocks X"; + • enforce a rules file — make a AGENTS.md / AGENTS.md real instead of advisory; + • enable an existing builtin — usually the right answer, checked before authoring; + • work from an AgentEye deployment — findings, plus which hooks fail or over-deny. + + Served by the `failproofai` CLI and the policy files it loads. + + NOT for reading telemetry or operating an AgentEye deployment (that's `agenteye-cli`), designing evaluator scoring logic (`agenteye-evaluator`), fixing the bug an agent introduced, or repo invariants that belong in tests. +--- + +# Authoring failproofai policies + +> Source pointers below are paths inside the failproofai package. In a project that +> installed it, they live under `node_modules/failproofai/`; in a source checkout, +> at the repo root. The `grep` anchors work either way. + + +Five ways you get here. All converge on the same authoring core (§2). + +**1. Automatically, after an audit.** A `PostToolUse` policy fires when `failproofai audit` +runs and instructs the agent to come here. The findings are already sitting in the cache — +**start at §1** and triage them. This is the primary path: the audit knows what went wrong, +so it should not be on the user to notice and ask. + +**2. The user describes a problem.** *"My agent keeps force-pushing."* *"It deleted my files +again."* *"How do I stop it committing to main?"* They are reporting a failure, not +requesting a policy — most users do not know policies exist. Translate the complaint into a +rule, then **go to §2**. Do not make them specify events or tools; infer from the behavior +and confirm at the end. + +**3. The user asks for a policy outright.** *"Write a policy that blocks X."* Straight to §2. + +**4. Findings live in AgentEye.** The org runs [AgentEye](https://app.befailproof.ai) +and wants its findings enforced — or wants to know which of their policies are actually +working. AgentEye sees the whole fleet, so it answers things a local audit cannot: which +hooks are *failing* (and therefore enforcing nothing), and which are denying so often the +rule is probably mis-scoped. **§4** has the procedure. + +**5. The user has a rules file agents keep ignoring.** *"Agents keep skipping what's in my +AGENTS.md."* Prose rules are advisory — the agent reads them (maybe) and forgets them under +context pressure. Policies are enforcement. Extract the rules, classify what is enforceable, +and turn that subset into policies — **§3** has the procedure and the classification table. + +For 2, 3, 4 and 5, still check the audit cache if one exists — it often shows the behavior is +already happening and gives you real commands to use as test cases (§2.4). + +The single most common mistake is writing a custom policy when a builtin already covers +the case and just needs enabling. Always check coverage before authoring. + +--- + +## 1. Audit-driven triage + +**One finding, or all of them?** If the user names a single finding — by its policy name +(`git-commit-no-verify`), by description ("the one about `--no-verify`"), or by pointing at +a row in the dashboard — do **not** run the full triage. Handle just that one: + +1. Look it up in the cache by name, or by matching its `displayTitle` / `examples[]` + against what they described. +2. Read its `examples[]`. These are **real commands from their machine** — the single most + valuable thing the audit gives you. They become the should-fire cases in §2.4 (should-deny + for a block-mode policy, should-instruct for oversight), which means the finished policy + is proven against what actually happened rather than against invented input. +3. Pick the mode from the finding's severity class (§2's mode table): deny-class findings + get `deny()`, warn-class get `instruct()` oversight. **State the choice in one line and + offer the other mode** — "went with oversight since this has legitimate uses; say the + word for a hard block." The user asked for enforcement; which flavor is their call. +4. Check whether a builtin covers it (§2.1). If yes and it is off, that is a config line, + not a policy. +5. Otherwise author it (§2), test against those examples, report back. + +Mention what else is unaddressed in one line at the end — do not expand into a full triage +they did not ask for. + +Only run the full §1.1–1.4 sweep when they ask broadly: "what should I do about my audit", +"harden this repo", "fix these findings". + +### 1.1 Read the findings + +There is **no machine-readable output flag**. `RunAuditOptions` declares `--json`, `--since`, +`--cli`, `--project`, `--limit` and more, but `runAuditCli` parses only `--help` — every +other argument is rejected outright (`failproofai audit --json` errors). The one programmatic +path is the dashboard cache, which every audit run writes: + +```bash +cat ~/.failproofai/audit-dashboard.json +``` + +**The cache is the only source, and it can be arbitrarily old.** Check `cachedAt` before +trusting anything in it: + +```bash +bun -e 'const j = await Bun.file(process.env.HOME + "/.failproofai/audit-dashboard.json").json(); +const age = (Date.now() - Date.parse(j.cachedAt)) / 86400000; +console.log(`cached ${j.cachedAt} (${age.toFixed(1)} days ago), ${j.result.results.length} findings`);' +``` + +More than a day or two old and the findings describe past behavior, not current — say so +rather than presenting it as the live picture. + +**Refreshing it is awkward.** `failproofai audit` writes the cache and then starts a +dashboard server on port 8020 and opens a browser, so it never exits on its own. Prefer +asking the user to run it. If you must refresh unattended, the cache is written *before* the +server starts, so a timeout gets you fresh data without leaving a server running: + +```bash +timeout 180 failproofai audit >/dev/null 2>&1 || true # exits 124; cache is written +``` + +Do that only when the user has asked for fresh findings — it can take minutes on a large +history (423 transcripts took ~50s here) and it opens a browser tab. + +If the file does not exist at all, the user has never run an audit. Ask them to, rather than +running it for them. + +**The `AuditResult` is nested under a cache envelope.** The file is +`{schemaVersion, cachedAt, params, result}` — the findings are at **`.result.results[]`**, +not `.results[]`: + +```bash +bun -e 'const j = await Bun.file(process.env.HOME + "/.failproofai/audit-dashboard.json").json(); +for (const c of j.result.results.sort((a,b)=>b.hits-a.hits)) + console.log([c.name.replace("failproofai/",""), c.source, c.hits, c.projects, c.enabledInConfig].join(" | "));' +``` + +Each entry in `.result.results[]` is an `AuditCount`. The fields that matter for triage: + +| Field | Use | +|---|---| +| `name` | Builtins are **canonical-prefixed** (`failproofai/block-rm-rf`); detectors are bare. Strip the prefix before matching against config | +| `source` | `"builtin"` = a real policy that *would have* fired; `"audit-detector"` = audit-only pattern | +| `hits`, `projects` | How much this actually happens — prioritize by this | +| `examples[]` | **Real payloads from this machine.** These become your test cases in §2.4 | +| `enabledInConfig` | **Do not trust this** — see below. Always `false` for detectors | + +**`enabledInConfig` is a stale snapshot.** It records the merged config as it was *at audit +time*, and the cache persists indefinitely. Observed on this machine: the audit cached at +20:14 reported all 39 builtins enabled; the global config was emptied at 20:19, five minutes +later. Every finding still claims `enabledInConfig: true`. + +Always re-read the current config before classifying: + +```bash +cat .failproofai/policies-config.json 2>/dev/null # project scope +cat ~/.failproofai/policies-config.json 2>/dev/null # global scope +``` + +`enabledPolicies` is a **union** across project, local and global +(`hooks-config.ts`, grep `enabledSet`) — not precedence. A policy is on if *any* scope lists it. + +**Scope matters more than it looks.** The audit spans every project on the machine, but a +project-scope config protects only one. If findings span many projects and enforcement lives +in a single project config, the honest conclusion is that most of those hits are still +unprotected — and the fix belongs in the **global** config, not a project one. + +### 1.2 Sort every finding into one of three buckets + +**Bucket A — a builtin covers it and is off.** Do not write code. Add the short name to +`enabledPolicies` in `.failproofai/policies-config.json`. This is the cheapest and most +maintainable fix, and it is the right answer for most `source: "builtin"` findings. + +**Bucket B — a builtin covers it and is already on.** No action. Report it so the user knows +the finding is historical, not ongoing. + +**Bucket C — nothing genuinely covers it.** Author a custom policy (§2). + +### 1.3 Do not trust `DETECTOR_TO_POLICY` + +`src/audit/findings.ts`, grep `DETECTOR_TO_POLICY` maps every audit-only detector to some builtin, and its own +header comment explains why: so that "every finding looks like it has a failproofai fix." +Several of those mappings do not actually prevent the behavior. Judge coverage yourself: + +| Detector | Maps to | Real coverage | +|---|---|---| +| `redundant-cd-cwd` | `warn-repeated-tool-calls` | **None** — unrelated heuristic. Bucket C | +| `prefer-edit-over-sed-awk` | `warn-repeated-tool-calls` | **None**. Bucket C | +| `prefer-edit-over-read-cat` | `block-read-outside-cwd` | **None** for in-cwd reads. Bucket C | +| `prefer-write-over-heredoc` | `block-env-files` | Only the `.env` subset. Bucket C for the rest | +| `find-from-root` | `block-read-outside-cwd` | Partial — that policy is about file reads, not `find` | +| `sleep-polling-loop` | `warn-background-process` | Partial | +| `git-commit-no-verify` | `warn-git-amend` | **None** — different command | +| `reread-after-edit` | `warn-repeated-tool-calls` | Partial — only if params are identical | + +So most audit-only detectors are Bucket C. That is the gap this skill exists to fill. + +### 1.4 Present the triage before acting + +Show the user the three buckets with hit counts, then act. For Bucket A, propose the +config diff rather than silently editing — enabling enforcement changes what their agent is +allowed to do. For a single explicit request ("turn on block-rm-rf"), just edit it. + +**Never widen scope on your own initiative.** These three are off-limits without the user +asking for them in the current request: + +| Action | Why it needs asking | +|---|---| +| `failproofai policies --install` at **user scope** | Wires hooks into *every* project on the machine, not the one they are in | +| Editing `~/.failproofai/policies-config.json` | Global config; a deny there fires everywhere | +| Setting `customPoliciesPath` globally | Silently activates policy files across all projects | + +A question — *"what should I do about my findings?"*, *"is this protected?"* — asks for an +answer, not a change. Recommend the machine-wide fix in words and let them decide. Being +right about what should happen is not authorization to make it happen. + +Project-scope edits inside the repo the user is working in are fine when they asked for a +fix. The line is **scope**: their repo, yes; their machine, ask first. + +--- + +## 2. The authoring core + +**Arriving from a complaint?** Translate it into a concrete tool call first — a policy can +only match what actually crosses the wire. Common mappings: + +| What the user says | What to match on | +|---|---| +| "keeps force-pushing" | `Bash`, `command` =~ `git push --force` / `-f` | +| "deleted my files" | `Bash`, `command` =~ `rm -rf` | +| "commits straight to main" | `Bash`, `git commit` + current branch check | +| "reads my secrets / .env" | `Read`/`Bash`, `file_path` or `command` =~ `.env` | +| "installs junk globally" | `Bash`, `command` =~ `npm i -g`, `pip install`, … | +| "edits generated files" | `Write`/`Edit`, `file_path` =~ lockfile / `dist/` | +| "leaks keys into chat" | `sanitize-api-keys` + `additionalPatterns` param first; else a `PostToolUse` sanitizer (blocks the output — `message` is inert, `traps.md` §9) | + +If the mapping is not obvious, ask for the command they actually saw, or pull it from the +audit cache — `examples[]` holds real invocations and doubles as your test cases (§2.4). + +Two judgment calls to make before writing, and to state back to the user at the end: + +- **Which of the three modes?** failproofai does not only block. Match the builtins: + + | Mode | Helper | Name it | When | + |---|---|---|---| + | block | `deny()` | `block-*` | irreversible, no legitimate use | + | **oversight** | `instruct()` | `warn-*` | risky but sometimes right — *"STOP: … Confirm with the user before executing."* | + | sanitize | raw deny object (blocks output; `message` inert — traps.md §9) | `sanitize-*` | secrets in tool output | + + Default to **oversight** when the action has any legitimate use. Blocking those just gets + the policy disabled. See `patterns.md` for the exact voice the builtins use — copy it. +- **Scope.** Project config protects one repo; user scope (`~/.failproofai/policies/`) + applies everywhere. "My agent keeps doing X" usually means *everywhere*, not *here*. + +### 2.1 Check the builtins first + +Read `references/builtins.md`. All 39 builtins with their categories, default state, events +and parameters. If one matches, enabling it beats writing a new file every time. + +Many builtins take `params` (allowlists, thresholds, protected branches) that go in the +`policyParams` map — a parameterized builtin often covers a case that looks custom. + +Then check the project's **existing custom policies** — `ls .failproofai/policies/` and +read their `name`/`description` lines. Coverage is not only builtins: a hand-written policy +may already enforce exactly what you were about to author, and a duplicate means two +policies firing on every matching event. + +### 2.2 Choose the event and tool + +Read `references/api.md` for `PolicyContext`, the decision helpers, and the full event list. + +Rules of thumb: +- Blocking an action before it happens → `PreToolUse` +- Redacting or reacting to output → `PostToolUse` +- Gating the end of a turn → `Stop` (but see `references/traps.md` §6 — unsatisfiable Stop gates — before using this in + this repo) + +### 2.3 Write the file + +Location: `.failproofai/policies/` in the project. + +**The filename must end in `policies.js`, `policies.mjs`, or `policies.ts`.** A file named +`block-foo.mjs` is silently skipped and enforces nothing. Name it `block-foo-policies.mjs`. +This is the highest-frequency failure in the whole system — see `references/traps.md` §1. + +See `references/patterns.md` for worked examples per event type. + +### 2.4 Verify it actually fires + +Loading and execution are both fail-open — a broken policy is indistinguishable from a +working one unless you test it. Never report a policy as done without this step. + +Use the bundled runner. `$SKILL_DIR` below is **this skill's own folder** — the +agent harness reports it when the skill loads; substitute the real path: + +```bash +node "$SKILL_DIR/scripts/test-policy.mjs" \ + --policy .failproofai/policies/my-policies.mjs \ + --event PreToolUse --tool Bash \ + --input '{"command":"sudo rm -rf /tmp/x"}' --expect deny +``` + +Or a batch, which is what you want once there is more than one rule — `{{cwd}}` in an input +expands to the sandbox path: + +```json +[{ "name": "blocks sudo", "event": "PreToolUse", "tool": "Bash", + "input": {"command":"sudo ls"}, "expect": "deny" }, + { "name": "allows plain ls", "event": "PreToolUse", "tool": "Bash", + "input": {"command":"ls"}, "expect": "allow" }] +``` + +```bash +node "$SKILL_DIR/scripts/test-policy.mjs" --policy --cases cases.json +``` + +With `--policy`, the file is copied into a throwaway directory that acts as both project and +HOME — so `customPoliciesEnabled: false`, the real project config, and user-scope policies +cannot affect the result. It also **renames the file if it violates the loader convention** +(§2.3) and says so. Omit `--policy` to test the current directory's real config instead. + +Exit code is 1 if any `--expect` fails, so it drops straight into a script. + +Underneath it is just the documented stdin protocol, if you need it by hand: + +```bash +echo '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"sudo rm -rf /tmp/x"},"session_id":"test","transcript_path":"/dev/null","cwd":"'"$PWD"'"}' \ + | npx -y failproofai --hook PreToolUse +``` + +(Inside a failproofai source checkout, `node scripts/dev-hook.mjs --hook …` is the +dev-only equivalent — `npx -y failproofai` is what every other project uses.) + +**Read stdout, not the exit code.** Both outcomes exit 0: + +| Outcome | stdout | +|---|---| +| Denied (`PreToolUse`) | `{"hookSpecificOutput":{…,"permissionDecision":"deny","permissionDecisionReason":"…"}}` | +| Denied (`PermissionRequest`) | `{"hookSpecificOutput":{…,"decision":{"behavior":"deny","message":"…"}}}` | +| Instructed | `{"hookSpecificOutput":{…,"additionalContext":"Instruction from failproofai: …"}}` | +| Allowed | *(empty)* | + +**Deny is not one shape.** `PreToolUse` uses `permissionDecision`; `PermissionRequest` uses +`decision.behavior`; `instruct` uses `additionalContext`; the non-Codex CLIs use flat +`{decision:"block"}` or `{permission:"deny"}`, and Factory signals deny by exit code 2 with +the reason on stderr. Grepping for a single key will under-report blocks — this exact +mistake produced a false FAIL during testing. `test-policy.mjs` handles all of them. + +To test without touching the project's own config, build a throwaway project directory with +its own `.failproofai/policies/` and `policies-config.json`, and point the payload's `cwd` +at it. Project-scope discovery keys off that `cwd`, so the policy loads in isolation. + +Test both directions: a payload that **should** be denied, and a near-miss that **should** +be allowed. A policy that denies everything passes the first test. + +Two ways this test can lie to you: + +- **Empty output proves nothing on its own.** It is the same result whether your policy + allowed correctly, threw, timed out, or was never loaded. Always pair it with a + should-deny case that actually produces output. +- **A deny does not prove *your* policy denied.** Every enabled policy sees the event, so + another one may have fired — a rule matching nothing can hide behind a green suite. Assert + on text unique to your policy's reason, or test with `enabledPolicies: []`. See + `traps.md` §4; this happens more often than it sounds. + +Use the `examples[]` from the audit finding as your should-deny cases — they are real +commands from this machine, so they prove the policy catches what actually happened. + +### 2.5 Confirm the policy is actually live + +Do **not** rely on `customPoliciesEnabled` in `.failproofai/policies-config.json` — that flag +is a no-op and disables nothing (`traps.md` §2). The only reliable check is the one you +already did in §2.4: run the hook and look at stdout. + +What genuinely stops a policy from running: the filename convention (§2.3), a load-time +throw, or hooks not being installed for the CLI at all. Verify with: + +```bash +failproofai policies --list +``` + +--- + +## 3. Enforcing a rules file (AGENTS.md / AGENTS.md / system prompts) + +Prose rules are advisory: the agent reads them at session start and drops them under +context pressure. Policies fire on every tool call regardless of what the model remembers. +This path converts the enforceable subset of a rules file into policies — and is honest +about the rest. + +### 3.1 Extract + +Read the file. Pull out every rule stated as *behavior* — quote each verbatim and note its +section heading. Skip background prose, architecture notes, and anything descriptive. + +### 3.2 Classify + +Sort each rule using the table in `references/rules-files.md`. The short version: + +| Rule language | Class | Becomes | +|---|---|---| +| "never X" / "do not X" — tool-shaped | hard rule | `block-*` deny, or a builtin | +| "always X before Y" — ordering | workflow gate | PreToolUse check **on Y** (prefer over Stop gates) | +| "prefer X over Y" / "avoid Y" | preference | `warn-*` `instruct()` nudge | +| "file/config must contain Z" | repo invariant | a **test in the test suite** — not a policy | +| style, judgment, tone | unenforceable | stays prose; say so | + +Two classes here are easy to get wrong. Repo invariants ("configs must use the launcher +form") are about file *states*, and policies see tool *calls* — recommend a test, don't +force a policy. And workflow rules enforce best at the **action they gate** (`gh pr create`, +`git commit`), not as Stop gates — tighter feedback, no loop risk (`traps.md` §6). + +### 3.3 Check coverage — builtins AND existing custom policies + +Rules files are exactly where hand-written policies come from, so the rule you are about to +enforce may already be enforced. Check `references/builtins.md` (with params — §2.1), then +read the project's existing custom policies: + +```bash +ls .failproofai/policies/ && grep -h "name:\|description:" .failproofai/policies/*policies.mjs +``` + +A rule already covered goes in the report as covered — writing a duplicate policy means two +policies fire on every matching event forever. + +### 3.4 Present the extraction before enforcing + +Show the table — rule → class → action — before writing a batch. Turning a page of prose +into enforcement changes what the user's agents are allowed to do everywhere in the +project; that is a decision they confirm, not a side effect (§1.4 discipline). + +The table is **not optional and not summarizable into prose**: even when the user +explicitly asked you to enforce the file, your report opens with the table. It is the one +artifact that lets them audit your classification at a glance — prose bullets hide a +misclassified rule; a table row does not. + +### 3.5 Author and verify + +Route the uncovered enforceable subset through §2. In each generated file, cite provenance +so the policy can be traced back and re-synced when the md changes: + +```js +// derived-from: AGENTS.md § "Workflow rules" — "Never push a branch that is +// missing commits from main" (extracted 2026-07-24) +``` + +### 3.6 Report the honest split + +End with four lists: **enforced now** (new policies + enabled builtins), **already +covered** (by what), **nudged** (instruct-only), **left as prose** (and why). A rules file +never converts 100%. Claiming it does means something above was misclassified. + +--- + +## 4. Sourcing findings from AgentEye + +AgentEye is the observability half: failproofai enforces inside the loop, AgentEye records +what happened across the whole fleet. Full procedure and verified commands are in +`references/agenteye.md` — read it before running anything. The shape: + +### 4.1 Preflight + +`agenteye whoami`. Exit 4 means not logged in, and **you cannot fix that** — login needs a +code emailed to the user. Ask them to run `agenteye login --email ` and stop. Note the +permissions it prints; they decide what you may do in §4.5. Pass `--json` to every command. + +### 4.2 Ask all three questions + +AgentEye answers four different things, and they produce different work: + +| Question | Command shape | Produces | +|---|---|---| +| What is enforceable? | `audits findings` → filter `kind == "policy"` | policies to write or builtins to enable | +| What enforcement is **broken**? | `query run` over `hook_completed` where outcome not in (ok, approved) | **hooks failing = enforcing nothing** | +| What is **too strict**? | same, outcome in (denied, blocked) | deny-mode rules that should be oversight | +| What is on the issues board? | `issues list` → branch on `source` | `audit`-born → follow to the finding; `alert` → metric breach, not policy work; `manual` → free text, classify like §3 | + +`kind: policy` is AgentEye's own classification — `improvement` and `failure` findings are +code and instrumentation work, not yours. The issues board is a **human attention queue**, +not a behaviour log: on a live deployment 0 of 11 issues were policy-actionable, because an +alert fires on a number (latency, error rate, score) while a policy gates a tool call. Read +it, classify it, and report what you skipped — do not manufacture enforcement for a metric. + +The middle row is the one no local audit can answer. failproofai fails open (`traps.md` §3), +so a policy that throws returns allow and nothing surfaces locally. AgentEye records every +hook outcome, so a failing hook is a query away — and a hook failing hundreds of times is a +live gap that outranks any new policy you might write. Report it first. + +### 4.3 Get the actual commands + +`audits finding ` needs the **full UUID** from `--show-id`; the short id in the table is +rejected, despite the CLI docs. Its `evidence.queries` are runnable and often carry the exact +regex. For raw payloads: `events --full --session-id ` — `--full` is the only way to get +`payload` and it is expensive, so always bound it to one session. + +If an evidence query returns nothing, do not assume the finding is wrong — say in your report +that the policy came from the finding's description rather than observed payloads. That is a +real difference in confidence and the user should know which one they got. + +### 4.4 Author as normal + +Straight into §2 — builtins and their params first, then mode, then filename, then test both +directions. AgentEye changes where the work comes from, not how a policy is written. +`agenteye list tools` is worth one call: a policy matching a tool the fleet never emits is +dead on arrival. + +### 4.5 Propose the close-out; do not run it + +Print the `issues comment-add` and `audits resolve` commands and let the user run them. +AgentEye's confirms **auto-skip on a non-TTY, which is how you run it**, so a wrong id +resolves someone else's finding on a shared board with no prompt — and triage needs +`audits:write`, which a read-only account lacks. See `references/agenteye.md`. + +--- + +## 5. Reference files + +| File | Read it when | +|---|---| +| `references/api.md` | Writing any policy — types, helpers, context, events | +| `references/builtins.md` | Triaging coverage, or before authoring anything | +| `references/traps.md` | **Always.** Nine documented silent failures | +| `references/patterns.md` | Worked examples per event type | +| `references/rules-files.md` | Enforcing a AGENTS.md / AGENTS.md (§3) — classification and worked examples | +| `references/agenteye.md` | Sourcing findings from an AgentEye deployment (§4) — verified commands, gotchas, close-out | diff --git a/.agents/skills/policy-author/references/agenteye.md b/.agents/skills/policy-author/references/agenteye.md new file mode 100644 index 000000000..1f193369e --- /dev/null +++ b/.agents/skills/policy-author/references/agenteye.md @@ -0,0 +1,274 @@ +# Sourcing policy work from AgentEye + +[AgentEye](https://app.befailproof.ai) is the observability side of the same story: failproofai +enforces inside the agent loop, AgentEye records what happened across the fleet. Where the +local `failproofai audit` reads one machine's transcripts, AgentEye holds every session from +every agent in an org — so it sees things a local audit structurally cannot. + +Everything here was verified against a live deployment (org `demo`, `agenteye` 0.1.13). + +## Preflight + +```bash +agenteye whoami # exit 4 = not logged in; you cannot fix this — see below +``` + +If not logged in, **stop and ask the user to run `agenteye login --email `** — it needs +a one-time code emailed to them, so you cannot complete it. Note the permissions `whoami` +prints; they decide what you may do at the end (see "Closing the loop"). + +Add `--json` to **every** command — it prints machine-readable output and nothing else. + +## Four sources of policy work + +AgentEye answers four different questions. Check all four; they produce different work — and +two of them produce no policy at all, which is a result worth reporting rather than a dead end. + +### 1. Findings already classified as policy-shaped + +AgentEye tags each finding with a `kind`. **`kind: policy` means it is enforceable** — +`improvement` and `failure` are code or instrumentation work and are not yours. + +```bash +agenteye --json audits findings --limit 100 --show-id \ + | jq '.findings[] | select(.kind == "policy") | {id, title, severity, status}' +``` + +`--show-id` is not optional: `audits finding ` **requires the full UUID**. The short id +in the table looks usable and is rejected with `no finding `, exit 6 — the CLI reference +says short ids are accepted, and for this subcommand they are not. + +Then read one in full: + +```bash +agenteye --json audits finding +``` + +The fields that matter: + +| Field | Use | +|---|---| +| `description` / `root_cause_hypothesis` | what happened and why — the policy's rationale | +| `recommendation` | AgentEye's own fix. Often names the mechanism ("add a pre-tool-use hook that rejects…") | +| `evidence.queries` | **runnable SQL, often containing the exact regex** | +| `evidence.policy_id` | AgentEye's internal rule id, e.g. `secret.credential_in_tool_args` | +| `scope` | narrows it — e.g. `{"tool": "db.query"}` | +| `occurrences`, `evidence.matched_sessions` | how much this actually happens | + +A real one, verbatim: *"Credential-shaped strings passed to db.query in production"*, whose +evidence query carries `match(payload, '(AKIA[0-9A-Z]{16}|Bearer\s+[A-Za-z0-9._-]{20,})')`. +That regex is the pattern to enforce — but **run it through §2.1 builtin-first anyway**: +`sanitize-api-keys` and `sanitize-bearer-tokens` already cover those two shapes, so the +answer is mostly config, and a custom policy only covers what the builtins miss. + +### 2. Hook failures — policies that are silently enforcing nothing + +**The one thing a local audit can never tell you.** failproofai fails open (`traps.md` §3): +a policy that throws or times out returns allow, and nothing surfaces. Locally that is +invisible. AgentEye records every hook outcome, so the gap is a query: + +```bash +agenteye --json query run --sql " + SELECT JSONExtractString(payload,'hook_name') AS hook, + JSONExtractString(payload,'outcome') AS outcome, + count() AS c + FROM events + WHERE event_type = 'hook_completed' + AND JSONExtractString(payload,'outcome') NOT IN ('ok','approved') + GROUP BY hook, outcome ORDER BY c DESC LIMIT 20" +``` + +Live demo output included `pre_tool_use failed 877` and `pii_redactor failed 702`. Every one +of those is an action that ran **unguarded** while the dashboard showed a hook installed. + +This is not a policy to author — it is a **broken policy to report**. Tell the user which +hook is failing and how often, and treat fixing it as higher priority than any new policy: +a policy that fails 877 times is worth less than nothing, because it looks like protection. + +### 3. Denial patterns — policies that are too strict + +A high denial count means one of two very different things, and **the raw count cannot tell +you which** — you must measure the *distribution across sessions* before drawing any +conclusion. AgentEye's own finding text says denials mean an agent is "burning a turn each +time", which presumes a retry loop; that is one of the two cases, not the default. + +| Shape | Meaning | Fix | +|---|---|---| +| Denials **concentrated** — few sessions, many denials each | an agent stuck retrying something it will never be allowed | the rule is right; the agent needs to be *told* what to do instead → `instruct` | +| Denials **spread** — many sessions, ~1 denial each | the rule is contested broadly, i.e. **mis-scoped** | narrow the rule, or downgrade to oversight | + +Measure it rather than assuming — this query gives the distribution, not just the total: + +```bash +agenteye --json query run --sql " + SELECT denials_per_session, count() AS sessions FROM ( + SELECT session_id, count() AS denials_per_session + FROM events + WHERE event_type = 'hook_completed' + AND JSONExtractString(payload,'outcome') IN ('denied','blocked') + GROUP BY session_id) + GROUP BY denials_per_session ORDER BY denials_per_session" +``` + +Live result from the demo org: **76.3% of affected sessions had exactly one denial**, 21.6% +had two, and the maximum was four. That is the *spread* shape — a mis-scoped rule — and a +report calling it a retry loop would be wrong on the mechanism even while landing on the +right fix. A blind agent made exactly that error against this data, so state the distribution +in your report, not just the total. + +One caveat before recommending any change: a human-approval gate (`approval.tool_use`) is +*supposed* to deny sometimes. A non-zero denial rate there is the control working, not a +defect. Ask whether the denials cluster on a handful of tool/argument shapes before +proposing anything. + +### 4. The issues board — read it, but classify before believing it + +Issues are AgentEye's **human attention queue**, not a behaviour log. Three sources feed it, +and only one reliably contains policy work: + +```bash +agenteye --json issues list --limit 100 --show-id +``` + +| `source` | What it is | Policy material? | +|---|---|---| +| `audit` | a finding that graduated — same content, back-linked | **Yes** — follow `source_finding_id` and treat it as §1 | +| `alert` | a configured metric crossed a threshold | **Almost never** — see below | +| `manual` | a person opened it, free text | **Sometimes** — depends entirely on what they wrote | + +**Why alert-born issues are usually not yours.** An alert fires on a *number*: p95 latency, +error rate, eval score, token burn. A policy gates a *tool call before it runs*. Those are +different axes, and no policy can move a latency percentile. Measured on a live deployment: +**0 of 11 issues were policy-actionable** — every one was a metric breach whose resolution +comments read "revert the prompt", "tune down retries", "fix the downstream service". + +Check `trigger_kind` and `breach_summary` before spending any effort: `eval_compound`, +`error_rate`, `latency_p95`, `token_burn` are all metric triggers and none of them describes +an action a hook could have stopped. + +**Manual issues are free text**, so treat them exactly like the complaint path — does the +text describe a *behaviour* ("the agent keeps writing to prod config") or a *measurement* +("checkout latency is up")? Behaviour goes through the §3 rules-file classification; a +measurement does not become a policy no matter how it is phrased. + +**Audit-born issues are the ones worth chasing**, because they are findings in a different +wrapper: + +```bash +agenteye --json issues list --limit 100 --show-id \ + | jq '.issues[] | select(.source == "audit") | {id, title, source_finding_id}' +``` + +Then `agenteye --json audits finding ` and you are back in §1. + +**A caveat worth checking in the target org.** The CLI reference states every finding +graduates to an issue, but on the live deployment tested here **no issue carried a +`source_finding_id` and no `kind: policy` finding carried an `issue_id`** — the two surfaces +were entirely disconnected. If the same is true wherever you are running, do not conclude +there is no policy work because the issues board looks operational; go to `audits findings` +directly. Say so in the report, because a disconnect there means audit findings are never +reaching the board the team actually watches. + +## Getting the actual commands + +A policy must match real input, and its tests need real payloads (§2.4). Two routes: + +```bash +# a) the finding's own evidence query, bounded +agenteye --json query run --sql " LIMIT 20" + +# b) raw payloads for a cited session +agenteye --json events --full --session-id --all --limit 1000 \ + | jq '.events[].payload' +``` + +**`--full` is the only way to get `payload`** and it hits a heavy endpoint — always bound it +to a session, never sweep unbounded. + +**An evidence query returning zero rows does not mean the finding is wrong.** Verified live: +the credential finding cites 1424 matched events, yet its evidence query returns nothing, +because the demo's seeded findings and seeded events were generated separately. When that +happens, fall back to the patterns named in the finding text itself, and say in your report +that the policy was built from the finding's description rather than from observed payloads — +that is a real difference in confidence. + +## Then: author it and plug it in + +From here it is §2 unchanged — **check builtins and their params first**, pick a mode, name +the file `*policies.mjs`, test both directions with `scripts/test-policy.mjs`. AgentEye +changes where the work comes from, not how a policy gets written. + +"Plugging it in" is two concrete edits in the target project, and neither touches AgentEye: + +``` +.failproofai/policies/-policies.mjs the custom policy (filename convention — traps.md §1) +.failproofai/policies-config.json `enabledPolicies` for any builtin that covers a finding +``` + +Nothing about this needs an AgentEye permission — a failproofai policy is a local file plus a +config entry. Then prove both took effect, because neither is self-evident: + +```bash +# the custom file actually loads (fail-open hides a file that never loaded — traps.md §3) +node "$SKILL_DIR/scripts/test-policy.mjs" --policy .failproofai/policies/-policies.mjs \ + --cwd . --event PreToolUse --tool Bash --input '{"command":""}' --expect deny + +# an enabled builtin fires against the REAL project config (omit --policy) +node "$SKILL_DIR/scripts/test-policy.mjs" --cwd . \ + --event PreToolUse --tool Bash --input '{"command":"sudo ls"}' --expect deny +``` + +Record which AgentEye finding each policy came from, in the file itself, so the two systems +stay traceable to each other: + +```js +// derived-from: AgentEye finding a41243c4-01fe-4f88-871e-a41bc148906d +// "Credential-shaped strings passed to db.query in production" (org demo, 2026-07-27) +``` + +One extra step worth taking: AgentEye knows which tools actually exist in the fleet. + +```bash +agenteye --json list tools # also: agents, envs, event_types, hooks, error_types +``` + +A policy matching a tool name no tool ever emits is dead on arrival, and this is the cheapest +way to catch that before shipping it. + +## Closing the loop + +After a policy is installed and tested, the finding should not sit open forever. But: + +**Do not run the triage commands yourself.** Two independent reasons: + +1. `agenteye`'s confirm prompts **auto-skip on a non-TTY — which is exactly how you run it**. + `audits resolve ` executes immediately, with no chance to catch a wrong id, on a board + the user's whole team shares. +2. Triage needs `audits:write`, which a read-only account does not have (the live demo + account has `audits:read` only — check `whoami` before assuming). + +So **print the commands and let the user run them**: + +```bash +agenteye issues comment-add --body "Enforced by failproofai policy \`\` in .failproofai/policies/ — denies , verified with N cases." +agenteye audits resolve +``` + +`resolve` leaves no suppression, so a genuine recurrence reopens as new — the right verb once +enforcement exists. `mute`/`dismiss` suppress the pattern permanently and are the wrong choice +here. Every finding carries an `issue_id` linking to its issue; triage on either surface +mirrors onto the other. + +## What to report + +Same honest split as everywhere else, plus one AgentEye-specific line: + +- **enforced now** — new policies + builtins enabled, with the finding each came from +- **already covered** — findings a builtin already handles +- **broken enforcement** — hooks failing (source 2). Call these out first; they are live gaps. + Give the **fleet-wide aggregate** (`failed / total = N%`) alongside per-hook counts — raw + counts alone undersell the systemic scale (live: 10,736 of 202,645 = 5.3%) +- **too strict** — deny-mode policies that should be oversight (source 3) +- **not policy work** — `kind: improvement` / `failure` findings, and alert/metric issues. + Name them and say why, so the user can see they were read rather than skipped +- **to close** — the exact `comment-add` / `resolve` commands, for the user to run diff --git a/.agents/skills/policy-author/references/api.md b/.agents/skills/policy-author/references/api.md new file mode 100644 index 000000000..8140e55ef --- /dev/null +++ b/.agents/skills/policy-author/references/api.md @@ -0,0 +1,173 @@ +# Policy API reference + +> Source pointers below are paths inside the failproofai package. In a project that +> installed it, they live under `node_modules/failproofai/`; in a source checkout, +> at the repo root. The `grep` anchors work either way. + + +Everything here is exported from `src/index.ts` (19 lines — that is the entire public +surface): + +```ts +export { customPolicies, getCustomHooks, clearCustomHooks } from "./hooks/custom-hooks-registry"; +export { allow, deny, instruct } from "./hooks/policy-helpers"; +export type { PolicyContext, PolicyResult, CustomHook, PolicyDecision, PolicyFunction } from "./hooks/policy-types"; +``` + +## The policy object + +`CustomHook` — `src/hooks/policy-types.ts`, grep `interface CustomHook`: + +```ts +export interface CustomHook { + name: string; + description?: string; + match?: { + events?: HookEventType[]; + }; + fn: (ctx: PolicyContext) => PolicyResult | Promise; +} +``` + +Registered with `customPolicies.add(hook)`. That is the whole registration surface — no +remove, no update, and **no validation of any kind** — `custom-hooks-registry.ts` (grep +`getRegistry`) is a bare array push. + +## Context + +`PolicyContext` — `policy-types.ts`, grep `interface PolicyContext`: + +```ts +export interface PolicyContext { + eventType: HookEventType; + payload: Record; + toolName?: string; + toolInput?: Record; + session?: SessionMetadata; + params?: Record; + cli?: IntegrationType; // which agent CLI fired this; mirrors session.cli +} +``` + +- `ctx.toolInput` holds the tool's arguments — `command` for Bash, `file_path`/`content` for + Write, `old_string`/`new_string` for Edit, `pattern` for Grep. These keys are already + canonicalized across all 11 supported CLIs, so you write them once. +- `ctx.session?.cwd` is the working directory. +- `ctx.payload` is the raw hook payload if you need something not surfaced above. + +## Decisions + +`PolicyResult` — `policy-types.ts`, grep `interface PolicyResult`: + +```ts +export interface PolicyResult { + decision: "allow" | "deny" | "instruct"; + reason?: string; + message?: string; +} +``` + +Helpers (`policy-helpers.ts`, the complete file): + +```ts +export function allow(reason?: string): PolicyResult { ... } // reason optional +export function deny(reason: string): PolicyResult { ... } // reason required +export function instruct(reason: string): PolicyResult { ... } // reason required +``` + +- **`allow`** — let it through. Also the correct return when your policy does not apply. +- **`deny`** — block it. `reason` is shown to the agent and the user. +- **`instruct`** — let it through but inject guidance into the agent's next turn. Only + properly supported on Claude Code, Devin and Antigravity; degrades to a stderr note on + Hermes, Goose, OpenClaw and Pi. Fine for a local skill; do not rely on it if the policy + will be distributed. + +### The `message` field is currently inert — sanitize works by blocking, not replacing + +The builtin sanitizers return `deny` plus a `message` that *looks like* a replacement — +`sanitizeJwt` in `builtin-policies.ts`: + +```ts +return { + decision: "deny", + reason: "JWT token detected in tool output", + message: "[REDACTED: JWT token removed by failproofai]", +}; +``` + +**But the evaluator never consumes `PolicyResult.message`** (verified 2026-07-24: the deny +response in `policy-evaluator.ts` is built from `reason` alone; the only `.message` read in +that file is `err.message`). What actually happens on a sanitize deny: the tool output is +**blocked entirely** and the model sees the block reason instead. The secret is still +protected — by omission, not redaction — but the model also loses the rest of that output. +Do not promise users "the output is scrubbed and the rest passes through"; it is not. + +Two practical consequences: +- Put everything useful into `reason` — it is the only channel that surfaces. +- For output scrubbing, check `sanitize-api-keys`'s **`additionalPatterns` param first** + (`{regex, label}` entries) — builtin-first applies to sanitizers too. A custom sanitizer + is only needed when the pattern-per-output-shape doesn't fit that param. + +The `deny()` helper cannot set `message` anyway; if you do set it (future-proofing for when +the evaluator honors it), return the raw object literal. + +## Events + +`HookEventType` — `src/hooks/types.ts`, grep `HOOK_EVENT_TYPES`. The ones worth knowing: + +| Event | Fires | Can block? | +|---|---|---| +| `PreToolUse` | Before a tool runs | **Yes** — the main enforcement point | +| `PostToolUse` | After a tool returns | Yes — a deny blocks the whole output (see the `message` note above) | +| `UserPromptSubmit` | On user input | Yes | +| `Stop` | Agent about to finish its turn | Yes — deny forces another turn | +| `SessionStart` / `SessionEnd` | Session boundaries | Observation | +| `SubagentStop` | Subagent returns | Yes on most CLIs | +| `PermissionRequest` / `PermissionDenied` | Permission flow | Yes | + +Full list also includes `PostToolUseFailure`, `StopFailure`, `Notification`, +`SubagentStart`, `TaskCreated`, `TaskCompleted`, `PreCompact`, `PostCompact`, `FileChanged`, +`CwdChanged`, `ConfigChange`, `WorktreeCreate`, `WorktreeRemove`, `Elicitation`, +`ElicitationResult`, `UserPromptExpansion`, `PostToolBatch`, `InstructionsLoaded`, +`TeammateIdle`, `Setup`. + +## Filtering by tool + +`CustomHook.match` publicly declares **only `events`**. Filter on the tool inside `fn`: + +```js +fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + ... +} +``` + +An undocumented `match.toolNames` does work at runtime (`handler.ts`, grep `hook.match ??` passes `match` +straight through, and `policy-registry.ts`, grep `getPoliciesForEvent` filters on it), but it is absent from the +public type and could be typed away. Prefer filtering in `fn`. + +## Execution model + +- Custom policies run at **priority -1**, i.e. after all builtins. +- Each `fn` gets a **10-second timeout**. Timeout or throw → `{decision: "allow"}`. +- Namespaced as `custom/`, `.failproofai-project/` or `.failproofai-user/`. + +## Configuration + +`.failproofai/policies-config.json` — `HooksConfig`, `policy-types.ts`, grep `interface HooksConfig`: + +```ts +export interface HooksConfig { + enabledPolicies: string[]; // builtins, by short name + llm?: LlmConfig; + policyParams?: Record>; // keyed by short name + customPoliciesPath?: string; + customPoliciesEnabled?: boolean; // absent = enabled +} +``` + +Builtins are enabled **purely by presence** of the short name in `enabledPolicies` — there +is no per-policy enabled/disabled object, and omission means off. + +Merged across three scopes, in precedence order: project `{cwd}/.failproofai/` → local → +global `~/.failproofai/` (`hooks-config.ts`, grep `readMergedHooksConfig`). diff --git a/.agents/skills/policy-author/references/builtins.md b/.agents/skills/policy-author/references/builtins.md new file mode 100644 index 000000000..ca789a75d --- /dev/null +++ b/.agents/skills/policy-author/references/builtins.md @@ -0,0 +1,132 @@ +# Builtin policies (39) + +**Generated — do not hand-edit.** Regenerate with: + +```bash +bun "$SKILL_DIR/scripts/sync-builtins.mjs" +``` + +This snapshot goes stale whenever a builtin is added, renamed, or has its default +flipped. When it matters, ask the CLI instead — it is always current: + +```bash +failproofai policies # every policy, with enabled status and params +``` + +## How to use this for triage + +Enabling a builtin beats writing a custom policy: nothing to maintain, no naming +trap, no fail-open risk, and it ships with tests. + +Before concluding "no builtin covers this", check whether a **parameterized** one +does — several take allowlists or thresholds that widen their scope considerably. +Params go in the `policyParams` map, keyed by short name. + +To enable: add the short name to `enabledPolicies` in `.failproofai/policies-config.json`. + +--- + +### Sanitize + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `sanitize-jwt` | **on** | PostToolUse | Stop Claude from reading JWTs in tool responses | +| `sanitize-api-keys` | **on** | PostToolUse | Stop Claude from reading API keys (OpenAI, Anthropic, GitHub, AWS, Stripe, Google) in tool responses _(params: additionalPatterns)_ | +| `sanitize-connection-strings` | **on** | PostToolUse | Stop Claude from reading database connection strings with embedded credentials in tool responses | +| `sanitize-private-key-content` | **on** | PostToolUse | Stop Claude from reading PEM private key content in tool responses | +| `sanitize-bearer-tokens` | **on** | PostToolUse | Stop Claude from reading Authorization Bearer tokens in tool responses | + +### Environment + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `protect-env-vars` | **on** | PreToolUse | Prevent commands that read environment variables | +| `block-env-files` | **on** | PreToolUse | Block reading/writing .env files | +| `block-read-outside-cwd` | off | PreToolUse | Block file reads outside the session working directory _(params: allowPaths)_ | + +### Dangerous Commands + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `block-sudo` | **on** | PreToolUse, PermissionRequest | Block sudo commands _(params: allowPatterns)_ | +| `block-curl-pipe-sh` | **on** | PreToolUse | Block piping downloads to shell | +| `block-rm-rf` | off | PreToolUse | Prevent catastrophic deletions _(params: allowPaths)_ | +| `block-failproofai-commands` | **on** | PreToolUse | Block failproofai CLI commands and uninstallation | +| `block-secrets-write` | off | PreToolUse | Block writing secret key files _(params: additionalPatterns)_ | + +### Infra Commands + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `block-kubectl` | off | PreToolUse | Block kubectl commands (Kubernetes cluster mutations) _(params: allowPatterns)_ | +| `block-terraform` | off | PreToolUse | Block terraform and tofu (OpenTofu) commands _(params: allowPatterns)_ | +| `block-aws-cli` | off | PreToolUse | Block aws CLI commands _(params: allowPatterns)_ | +| `block-gcloud` | off | PreToolUse | Block gcloud (Google Cloud) CLI commands _(params: allowPatterns)_ | +| `block-az-cli` | off | PreToolUse | Block az (Azure) CLI commands _(params: allowPatterns)_ | +| `block-helm` | off | PreToolUse | Block helm commands _(params: allowPatterns)_ | +| `block-gh-pipeline` | off | PreToolUse | Block gh CLI pipeline-trigger subcommands (workflow run, run rerun/cancel, pr merge, release create/delete, cache delete, secret set/delete) _(params: allowPatterns)_ | + +### Git + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `block-push-master` | **on** | PreToolUse | Block pushing to main/master _(params: protectedBranches)_ | +| `block-force-push` | off | PreToolUse | Prevent force-pushing to any branch | +| `block-work-on-main` | off | PreToolUse | Block git commits and merges on main/master branch _(params: protectedBranches)_ | +| `warn-git-amend` | off | PreToolUse | Warns before amending git commits, which rewrites history | +| `warn-git-stash-drop` | off | PreToolUse | Warns before permanently deleting stashed changes | +| `warn-all-files-staged` | off | PreToolUse | Warns before staging all working tree files with git add -A / . / --all | + +### Database + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `warn-destructive-sql` | off | PreToolUse | Warn before executing destructive SQL (DROP/TRUNCATE/DELETE without WHERE) via database clients | +| `warn-schema-alteration` | off | PreToolUse | Warns before SQL schema changes (ALTER TABLE with column or rename operations) | + +### Packages & System + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `warn-package-publish` | off | PreToolUse | Warn before publishing packages to public registries (npm, PyPI, crates.io, RubyGems, etc.) | +| `warn-global-package-install` | off | PreToolUse | Warns before installing packages globally (npm -g, cargo install, etc.) | +| `prefer-package-manager` | off | PreToolUse | Blocks non-preferred package managers and tells Claude to use an allowed one (e.g., uv instead of pip) _(params: allowed, blocked)_ | +| `warn-large-file-write` | off | PreToolUse | Warn before writing files larger than 1MB (configurable via thresholdKb param) _(params: thresholdKb)_ | +| `warn-background-process` | off | PreToolUse | Warns before starting detached or background processes | + +### AI Behavior + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `warn-repeated-tool-calls` | off | PreToolUse | Warn when the same tool is called 3+ times with identical parameters | + +### Workflow + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `require-commit-before-stop` | off | Stop | Require all changes to be committed before Claude stops | +| `require-push-before-stop` | off | Stop | Require all commits to be pushed to remote before Claude stops _(params: remote, baseBranch)_ | +| `require-pr-before-stop` | off | Stop | Require a pull request to exist for the current branch before Claude stops _(params: baseBranch)_ | +| `require-no-conflicts-before-stop` | off | Stop | Require the current branch to merge cleanly with the base branch before Claude stops _(params: baseBranch)_ | +| `require-ci-green-before-stop` | off | Stop | Require CI checks to pass on the current HEAD commit before Claude stops (ignores stale runs on prior commits) | + +> The five `require-*-before-stop` policies gate the end of a turn. A gate whose +> condition cannot be met in the current project loops forever — see `traps.md` §6 +> before enabling one. + +--- + +## Audit-only detectors + +These have no real-time builtin equivalent, so they are the prime candidates for +custom policies. List them from source with: + +```bash +bun -e 'const {AUDIT_DETECTORS}=await import("./src/audit/detectors/index.ts"); +for (const d of AUDIT_DETECTORS) console.log(d.name, "|", d.category+"/"+d.severity, "|", d.description)' +``` + +All but `reread-after-edit` are Bash-command patterns, so a `PreToolUse` policy +filtering on `ctx.toolName === "Bash"` and matching `ctx.toolInput.command` covers +most of them. `reread-after-edit` needs cross-call session state, which hooks cannot +see — that one needs a builtin, not a custom policy. diff --git a/.agents/skills/policy-author/references/patterns.md b/.agents/skills/policy-author/references/patterns.md new file mode 100644 index 000000000..992eb6932 --- /dev/null +++ b/.agents/skills/policy-author/references/patterns.md @@ -0,0 +1,224 @@ +# Worked patterns + +All examples go in `.failproofai/policies/-policies.mjs` — the filename **must** +end in `policies.mjs`. See `traps.md` §1. + +Multiple `customPolicies.add()` calls per file are fine and are the normal way to group +related rules. + +--- + +## Block a Bash command pattern + +The workhorse. Seven of the eight audit detectors are Bash-command patterns, so this shape +covers most Bucket C findings. + +```js +import { customPolicies, allow, deny } from "failproofai"; + +const NO_VERIFY_RE = /\bgit\s+commit\b[^\n]*\s(--no-verify|-n)\b/; + +customPolicies.add({ + name: "block-commit-no-verify", + description: "Block git commit --no-verify, which skips pre-commit hooks", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? ""); + if (!NO_VERIFY_RE.test(command)) return allow(); + return deny( + "git commit --no-verify skips the pre-commit hooks. Run the checks, or commit without the flag.", + ); + }, +}); +``` + +Points that matter: +- Return `allow()` early for tools you do not handle. A policy that only cares about Bash + still runs on every `PreToolUse`. +- Coerce with `String(... ?? "")`. `toolInput` is `Record`. +- Write the `reason` as an instruction to the agent, not just a complaint. It is what the + agent reads next, so tell it what to do instead. + +## Block by file path + +```js +import { customPolicies, allow, deny } from "failproofai"; + +customPolicies.add({ + name: "block-lockfile-edits", + description: "Lockfiles are generated — block hand-edits", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (!["Write", "Edit"].includes(ctx.toolName ?? "")) return allow(); + const path = String(ctx.toolInput?.file_path ?? ""); + if (!/\b(bun\.lock|package-lock\.json|yarn\.lock)$/.test(path)) return allow(); + return deny("Lockfiles are generated. Run the package manager instead of editing by hand."); + }, +}); +``` + +`file_path` is canonical across all 11 CLIs — the per-CLI input maps normalize Copilot's +`path`, Hermes's `path`, etc. before your policy sees it. + +## Three modes — pick one deliberately + +failproofai is not only a blocker. The builtins split three ways, and the name prefix +signals which: + +| Prefix | Helper | Effect | Use when | +|---|---|---|---| +| `block-*` (17) | `deny()` | action never runs | irreversible or unsafe, no legitimate case | +| `warn-*` (10) | `instruct()` | action runs; agent told to check with the human first | risky but sometimes correct — needs a human, not a wall | +| `sanitize-*` (5) | raw deny object | output **blocked** before the model sees it (`message` is inert — traps.md §9) | secrets in tool output | + +Name your policy with the matching prefix. A reader should know the mode from the name. + +Most requests that sound like "block X" are really `warn-*`. Blocking something with a +legitimate use just teaches people to disable the policy. + +## Oversight — stop and confirm with the human + +The `warn-*` builtins share one voice, and it is worth copying exactly. They do **not** nudge +toward a better tool — they halt the agent and hand the decision back to the person: + +> **STOP:** This command permanently deletes stashed changes (git stash drop/clear). Stash +> entries cannot be recovered after deletion. Confirm with the user before executing. + +Three parts: **STOP:** + what the command actually does + *Confirm with the user before +executing.* The middle part explains why it matters, so the human can decide in one read. + +```js +import { customPolicies, allow, instruct } from "failproofai"; + +const PROD_DEPLOY_RE = /\b(kubectl\s+apply|helm\s+upgrade|serverless\s+deploy)\b[^\n]*\bprod/; + +customPolicies.add({ + name: "warn-prod-deploy", + description: "Require human confirmation before a production deploy", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? ""); + if (!PROD_DEPLOY_RE.test(command)) return allow(); + return instruct( + "STOP: This command deploys to production. It affects live traffic and is not " + + "trivially reversible. Confirm with the user before executing.", + ); + }, +}); +``` + +This is the mode to reach for when the answer to "should this ever be allowed?" is +"sometimes, and a human should decide." + +**Caveat:** `instruct` reaches the model properly on Claude Code, Devin and Antigravity. On +Hermes, Goose, OpenClaw and Pi it degrades to a stderr note the agent never sees — so on +those, oversight silently becomes no oversight. If the policy must hold everywhere, use +`deny()` with a reason explaining how to proceed. + +## Nudge toward a better tool + +The other use of `instruct()` — the action is not dangerous, just wasteful. No "STOP", no +human needed, just a better way to do it. + +```js +import { customPolicies, allow, instruct } from "failproofai"; + +customPolicies.add({ + name: "prefer-read-over-cat", + description: "Nudge toward the Read tool instead of shelling out to cat", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? "").trim(); + if (!/^(cat|head|tail|less|more)\s+\S+$/.test(command)) return allow(); + return instruct("Use the Read tool for single files — it is cheaper and gives line numbers."); + }, +}); +``` + +Caveat: `instruct` only injects properly on Claude Code, Devin and Antigravity. On Hermes, +Goose, OpenClaw and Pi it degrades to a stderr note. Fine for a local skill. + +## Sanitize tool output + +**Check the builtin param first:** `sanitize-api-keys` takes `additionalPatterns` +(`[{regex, label}]`), which covers most "scrub token X from output" requests with a config +edit and no code. Author a custom sanitizer only when that param can't express the need. + +**Know what sanitize actually does today:** the evaluator never consumes +`PolicyResult.message` (`traps.md` §9), so a sanitize deny **blocks the whole output** — +the model sees the block reason, not a redacted version. The secret stays protected; the +rest of that output is lost with it. Setting `message` anyway is harmless future-proofing, +and requires the raw object literal — the `deny()` helper cannot set it. + +```js +import { customPolicies, allow } from "failproofai"; + +const INTERNAL_URL_RE = /https:\/\/[a-z0-9-]+\.internal\.example\.com\/\S*/gi; + +customPolicies.add({ + name: "sanitize-internal-urls", + description: "Strip internal hostnames from tool output", + match: { events: ["PostToolUse"] }, + fn: async (ctx) => { + if (!INTERNAL_URL_RE.test(JSON.stringify(ctx.payload))) return allow(); + return { + decision: "deny", + reason: "Internal URL detected in tool output", + message: "[REDACTED: internal URL removed by failproofai]", + }; + }, +}); +``` + +Note `INTERNAL_URL_RE` has the `g` flag, which makes `.test()` stateful via `lastIndex`. +Either drop `g` or reset `lastIndex` between calls — a classic source of every-other-call +misses. + +## Gate the end of a turn + +```js +import { customPolicies, allow, deny } from "failproofai"; +import { execSync } from "node:child_process"; + +customPolicies.add({ + name: "require-changelog-entry", + description: "Every change must touch CHANGELOG.md before the turn ends", + match: { events: ["Stop"] }, + fn: async (ctx) => { + const cwd = ctx.session?.cwd; + if (!cwd) return allow(); + try { + const changed = execSync("git diff --name-only HEAD", { cwd, encoding: "utf8" }); + if (!changed.trim()) return allow(); + if (changed.includes("CHANGELOG.md")) return allow(); + return deny("Add a CHANGELOG.md entry under the current version heading before finishing."); + } catch { + return allow(); + } + }, +}); +``` + +**Stop policies are the dangerous kind.** A deny forces the agent to take another turn. If +the condition can never be satisfied in the current environment, it loops. Always: + +- wrap external calls in `try/catch` and `return allow()` on failure (fail-open) +- return `allow()` when there is nothing to check +- confirm the condition is actually reachable — this is exactly how the + `require-push-before-stop` loop happened in this repo (`traps.md` §6) + +Also mind the **10-second timeout**. A `Stop` policy shelling out to `gh` or the network can +blow it, and a timeout silently becomes `allow()`. + +## Configuration values + +`ctx.params` is always `{}` for custom policies (`traps.md` §8). Use module constants: + +```js +const PROTECTED = ["main", "master", "release"]; +``` + +Or read your own file inside `fn` if the values need to change without a code edit. diff --git a/.agents/skills/policy-author/references/rules-files.md b/.agents/skills/policy-author/references/rules-files.md new file mode 100644 index 000000000..6a2981461 --- /dev/null +++ b/.agents/skills/policy-author/references/rules-files.md @@ -0,0 +1,110 @@ +# Turning a rules file into enforcement + +The full classification guide for SKILL.md §3. The governing idea: **a policy can only +match a tool call.** Every rule must first be translated into "which tool, carrying what +input, in what state" — a rule that cannot be phrased that way cannot be a policy, no +matter how important it is. + +## Classification by language cue + +| The file says | Class | Enforcement | Mode | +|---|---|---|---| +| "never run X", "do not use X" | hard rule | builtin if one matches, else `block-*` | `deny()` | +| "only use X (not Y)" | hard rule, param-shaped | usually a **parameterized builtin** — check before authoring | `deny()` via builtin | +| "always do X before Y" | workflow gate | PreToolUse on **Y**, checking X happened | `deny()` with instructions | +| "X must accompany Y" (changelog with PR) | workflow gate | PreToolUse on Y's command | `deny()` with instructions | +| "prefer X over Y", "avoid Y" | preference | `warn-*` nudge | `instruct()` | +| "confirm with me before X" | oversight | `warn-*`, STOP voice (`patterns.md`) | `instruct()` | +| "file/config must contain Z" | repo invariant | **a test in the test suite** | not a policy | +| "write clear code", tone, style | judgment | stays prose | not enforceable | + +## Worked examples + +**"Use bun. Do not use npm/yarn to install deps."** Param-shaped hard rule — the first +candidate is `prefer-package-manager` with params: + +```json +{ + "enabledPolicies": ["prefer-package-manager"], + "policyParams": { + "prefer-package-manager": { "allowed": ["bun"], "blocked": ["npm", "yarn"] } + } +} +``` + +(Remember: params for a policy absent from `enabledPolicies` do nothing — `traps.md` §7.) + +**But check the builtin's matching breadth before enabling it.** A builtin can be broader +than the rule. `prefer-package-manager`'s npm matcher is a bare `\bnpm\b`, so with +`blocked: ["npm"]` it also denies `npm pack`, `npm view`, `npm ls` — and if the repo's own +docs mandate one of those (this repo's testing protocol requires `npm pack +--ignore-scripts`), the builtin over-blocks a documented workflow. The test is cheap: run +the repo's own legitimate commands through the hook with the builtin enabled in a sandbox. +If a legitimate use gets denied and no param can carve it out, a scoped custom policy +(match the install-family subcommands only) is the right call — say why in the report, +since it is an exception to builtin-first, not the norm. + +**"Every PR must include a CHANGELOG.md update."** Workflow gate. Enforce at the action it +gates — `gh pr create` — not at Stop: + +```js +// derived-from: CLAUDE.md § "Changelog" — "Every PR must include an update to +// CHANGELOG.md" (extracted 2026-07-24) +customPolicies.add({ + name: "require-changelog-in-pr", + description: "gh pr create must not run unless the branch touches CHANGELOG.md", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? ""); + if (!/\bgh\s+pr\s+create\b/.test(command)) return allow(); + const cwd = ctx.session?.cwd; + if (!cwd) return allow(); + try { + const changed = execSync("git diff --name-only origin/main...HEAD", { + cwd, encoding: "utf8", timeout: 5000, + }); + if (changed.includes("CHANGELOG.md")) return allow(); + return deny("This PR has no CHANGELOG.md entry. Add one under the current version heading, commit it, then create the PR."); + } catch { + return allow(); // can't tell -> fail open, never trap the agent + } + }, +}); +``` + +Why PreToolUse-on-the-action beats a Stop gate for these: the denial lands at the exact +moment of the violation with a precise fix, and there is no retry-loop risk when the +condition is unsatisfiable (`traps.md` §6). + +**"Configs must use the launcher form / `$CLAUDE_PROJECT_DIR` paths."** Repo invariant — +it describes file *contents*, not agent *behavior*. A policy would have to intercept every +Write and re-parse the config; a unit test reads the file directly and fails CI. Recommend +the test. (This repo's `dogfood-configs.test.ts` is exactly that solution.) + +## Extraction discipline + +- Quote each rule **verbatim** and keep its section heading — paraphrases drift, and the + provenance comment needs the exact source. +- One rule can produce two enforcements (a deny AND a nudge for the softer half). One + enforcement can cover several rules. Do not force 1:1. +- Rules about *people* ("ask the team lead before…") are out of scope — policies gate + agents, not humans. + +## Provenance and drift + +Every generated file carries, per rule: + +```js +// derived-from: § "" — "" (extracted ) +``` + +When the rules file changes, the comments say which policies to revisit. There is no +automatic sync — say so in the report rather than implying the enforcement tracks the file. + +## The honest split + +The report ends with four lists — enforced / already covered / nudged / left as prose. +Typical real-world files convert well under half their rules. If your extraction claims +everything became enforceable, the invariants and judgment calls were misclassified, and +the user now believes prose is protection. diff --git a/.agents/skills/policy-author/references/traps.md b/.agents/skills/policy-author/references/traps.md new file mode 100644 index 000000000..329e96a80 --- /dev/null +++ b/.agents/skills/policy-author/references/traps.md @@ -0,0 +1,207 @@ +# Silent failure modes + +Every item here is a documented, already-been-hit failure where a policy looks installed and +enforces nothing. Read this before reporting any policy as working. + +## 1. The filename convention + +`CONVENTION_FILE_RE = /policies\.(js|mjs|ts)$/` — `custom-hooks-loader.ts`, grep `CONVENTION_FILE_RE`. + +A file in `.failproofai/policies/` named `block-foo.mjs` is **silently skipped**. Only names +ending in `policies.js`, `policies.mjs` or `policies.ts` load. + +| Name | Loads? | +|---|---| +| `block-foo-policies.mjs` | yes | +| `security-policies.mjs` | yes | +| `policies.mjs` | yes | +| `block-foo.mjs` | **no** | +| `foo-policy.mjs` | **no** (singular) | + +This is not hypothetical. From the source comment at custom-hooks-loader.ts — grep findSkippedPolicyFiles): + +> This repo shipped `block-version-bumps.mjs` that way, so the guard written after a bad +> version bump had never once run. + +`findSkippedPolicyFiles()` (custom-hooks-loader.ts — grep findSkippedPolicyFiles)) exists solely to catch this, but its warnings go +only to the hook log, which nobody reads. **Always check the filename before anything else.** + +## 2. `customPoliciesEnabled: false` is a silent no-op (does NOT disable anything) + +The config key exists, `configure-wizard` writes it, and the loader checks it — but the +value never reaches the loader, so **setting it to `false` has no effect**. Convention +policies load regardless. + +The break is in `readMergedHooksConfig` (`hooks-config.ts` (grep `return {` inside `readMergedHooksConfig`)), which hand-builds its +return object from four keys and omits this one: + +```ts +return { + enabledPolicies: [...enabledSet], + ...(… ? { policyParams: mergedParams } : {}), + ...(customPoliciesPath !== undefined ? { customPoliciesPath } : {}), + ...(llm !== undefined ? { llm } : {}), +}; // customPoliciesEnabled is never propagated +``` + +So `handler.ts`, grep `customPoliciesEnabled:` passes `undefined`, and `custom-hooks-loader.ts`, grep `conventionEnabled` +(`opts?.customPoliciesEnabled !== false`) evaluates **true**, always. + +Verified empirically: a temp project with `"customPoliciesEnabled": false` plus a canary +policy still fires the canary. Do not tell a user their policies are disabled because this +flag is set — check by running the hook. + +The practical consequence for triage: a repo with this flag set is **still enforcing** its +custom policies. Treat the flag as unreliable in both directions until it is fixed. + +Note the explicit `customPoliciesPath` config key is **not** gated by this flag +(custom-hooks-loader.ts — grep customPoliciesPath)) — only convention discovery is, in intent. + +## 3. Everything is fail-open + +| Failure | Result | +|---|---| +| File not found | zero hooks, logged only | +| Syntax error | zero hooks, logged only | +| Throw at import time | zero hooks, logged only | +| Throw inside `fn` | `{decision: "allow"}` | +| `fn` exceeds 10s | `{decision: "allow"}` | + +Classified for telemetry as `module_not_found` / `syntax_error` / `runtime_error` +(custom-hooks-loader.ts — grep errorType)), then swallowed. Nothing fails loudly. + +The consequence: **silence is not success.** A policy that was never loaded and a policy +that correctly allowed an action produce identical observable behavior. This is why the +verification step in SKILL.md §2.4 is mandatory, not optional. + +## 4. A deny in your test does not prove *your* policy denied + +Every enabled policy sees every matching event. When a test case blocks, some policy blocked +— not necessarily yours. A rule that matches nothing can sit behind a green suite +indefinitely. + +Observed live: a heredoc case written to test a "no hardcoded localhost" policy passed, +but the deny came from the builtin `protect-env-vars` reacting to the word `export` in the +payload. The localhost rule was entirely unproven while reporting green. + +``` +cat < a.js +export const API = "http://localhost:3000"; +EOF +→ deny: "Command exports environment variable" ← protect-env-vars, not your policy +``` + +**Always attribute the deny.** The reason string names the responsible policy's message, so +assert on it rather than on the fact that something blocked: + +```js +if (got.decision !== "deny" || !got.reason.includes("")) + fail("blocked, but not by this policy"); +``` + +Two cheap ways to avoid the trap: + +- Write test payloads that avoid unrelated triggers — no `export`, no `sudo`, no `.env`, no + `curl … | sh` unless that is what you are testing. +- Test against a config with `enabledPolicies: []` so nothing else can fire. This is what + `test-policy.mjs --policy ` does — its sandbox config enables zero builtins, so any + deny is necessarily yours. + +## 5. Validation is essentially nil + +`failproofai p -i -c ` (`manager.ts`, grep `Validated`) executes the file and counts +`customPolicies.add()` calls. It does **not** check: + +- that the hook object has the right shape +- that `name` is unique (duplicates silently coexist) +- that `match.events` are real event names +- that `fn` returns a valid `PolicyResult` + +`customPolicies.add()` is a bare array push with no validation +(`custom-hooks-registry.ts`, grep `add(hook`). "Validated 1 custom hook(s)" means "the file ran and +called add() once" — nothing more. + +## 6. A Stop gate that cannot be satisfied loops forever + +Denying at `Stop` forces another turn. If the condition can never become true in the current +environment, the agent retries, fails, and never exits. Before enabling any +`require-*-before-stop` builtin — or authoring a Stop policy — check the condition is +actually reachable **in the project you are working in**: + +| Gate | Reachable only if | +|---|---| +| `require-commit-before-stop` | always (local git) | +| `require-push-before-stop` | a remote exists **and** credentials work | +| `require-pr-before-stop` | `gh` authenticated, remote is GitHub | +| `require-no-conflicts-before-stop` | base branch fetchable | +| `require-ci-green-before-stop` | CI configured **and** runs on push | + +Cheap check: + +```bash +git remote -v # no output → push/PR/CI gates cannot pass +git push --dry-run 2>&1 | head -2 # auth failure → same +gh auth status 2>&1 | head -3 +``` + +**This does not generalise across projects.** A gate that loops in one repo is correct in +another. Evaluate per project rather than carrying a verdict over. + +Concrete instance: the failproofai repo itself has no push credentials, so its five Stop +gates were deliberately removed from `.failproofai/policies-config.json` after exactly this +loop. That is a fact about that repo, not a rule about Stop gates. + +The same reachability rule applies to custom Stop policies — always `try/catch` external +calls and `return allow()` on failure, so an unavailable tool degrades to letting the turn +end rather than trapping it. + +## 7. Builtins are enabled by presence, not by a flag + +`enabledPolicies` is a `string[]`. Omission means off. There is no +`{"block-rm-rf": false}` form — to disable, remove the string. + +Params live in a **sibling** `policyParams` object keyed by the same short name, not nested +inside the policy entry: + +```json +{ + "enabledPolicies": ["block-read-outside-cwd"], + "policyParams": { + "block-read-outside-cwd": { "allowPaths": ["/tmp"] } + } +} +``` + +A param set for a policy that is not in `enabledPolicies` does nothing. + +## 8. `ctx.params` is always empty for custom policies + +`PolicyContext` exposes `params`, and `policies-config.json` has a `policyParams` map — so it +looks like a custom policy can be configured from config. It cannot. + +`POLICY_PARAMS_MAP` (`policy-evaluator.ts`, grep `POLICY_PARAMS_MAP`) is built **only** from `BUILTIN_POLICIES` +entries that declare a `params` schema. A custom hook has no schema, so it falls to the +else-branch at policy-evaluator.ts — grep without schema get empty params): + +```ts +// Custom hooks and policies without schema get empty params +ctx = { ...baseCtx, params: {} }; +``` + +Adding `policyParams: { "my-custom-policy": {...} }` to config does nothing — silently. + +**Workaround:** hardcode the values as module constants in the policy file, or read your own +config file / env var inside `fn`. The file is real JS, so anything is available. + +## 9. Sanitizers block, they do not redact — and `deny()` cannot set `message` anyway + +Two layered surprises. First, the `message` field of `PolicyResult` is not settable through +the exported `deny()` helper — a sanitizer needs the raw object literal. Second, and bigger: +**the evaluator never consumes `message` at all** (verified live 2026-07-24 — the deny +response in `policy-evaluator.ts` is built from `reason` only). The builtins' own +`[REDACTED: …]` messages are dead code. + +So a sanitize deny on `PostToolUse` blocks the *entire* tool output; the model sees the +block reason, never a redacted version. Protection holds — by omission — but do not tell a +user "the token is scrubbed and the rest passes through." Put anything the agent needs into +`reason`, and prefer `sanitize-api-keys.additionalPatterns` over authoring. See `api.md`. diff --git a/.agents/skills/policy-author/scripts/sync-builtins.mjs b/.agents/skills/policy-author/scripts/sync-builtins.mjs new file mode 100644 index 000000000..670863d37 --- /dev/null +++ b/.agents/skills/policy-author/scripts/sync-builtins.mjs @@ -0,0 +1,144 @@ +#!/usr/bin/env node +/** + * Regenerate references/builtins.md from the live BUILTIN_POLICIES registry. + * + * node sync-builtins.mjs rewrite the reference file + * node sync-builtins.mjs --check exit 1 if it is out of date (for CI) + * + * The reference file is a convenience snapshot. It drifts the moment a builtin + * is added, renamed, or has its default flipped — and a stale list is worse than + * no list, because it is quietly authoritative. Run this after any change to + * builtin-policies.ts. + * + * Resolves the registry from the repo checkout first, then from an installed + * failproofai package, so it works inside the repo and from a user's project. + */ +import { readFileSync, writeFileSync, existsSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const OUT = resolve(HERE, "..", "references", "builtins.md"); +const check = process.argv.includes("--check"); + +/** Walk up for a repo checkout, then fall back to node_modules. */ +function findRegistry() { + let dir = HERE; + for (let i = 0; i < 8; i++) { + const src = join(dir, "src", "hooks", "builtin-policies.ts"); + if (existsSync(src)) return src; + const dep = join(dir, "node_modules", "failproofai", "src", "hooks", "builtin-policies.ts"); + if (existsSync(dep)) return dep; + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + return null; +} + +const registryPath = findRegistry(); +if (!registryPath) { + console.error("Could not locate builtin-policies.ts (looked in the repo and node_modules)."); + console.error("Run this from inside the failproofai repo, or a project with failproofai installed."); + process.exit(2); +} + +// TypeScript source — needs a TS-capable runtime. Bun handles it directly. +let BUILTIN_POLICIES; +try { + ({ BUILTIN_POLICIES } = await import(pathToFileURL(registryPath).href)); +} catch (err) { + console.error(`Could not import ${registryPath}: ${err.message}`); + console.error("This script needs a TypeScript-capable runtime — try: bun sync-builtins.mjs"); + process.exit(2); +} + +const byCategory = new Map(); +for (const p of BUILTIN_POLICIES) { + if (!byCategory.has(p.category)) byCategory.set(p.category, []); + byCategory.get(p.category).push(p); +} + +const lines = []; +lines.push(`# Builtin policies (${BUILTIN_POLICIES.length})`); +lines.push(""); +lines.push("**Generated — do not hand-edit.** Regenerate with:"); +lines.push(""); +lines.push("```bash"); +lines.push('bun "$SKILL_DIR/scripts/sync-builtins.mjs" # $SKILL_DIR = this skill\'s folder'); +lines.push("```"); +lines.push(""); +lines.push("This snapshot goes stale whenever a builtin is added, renamed, or has its default"); +lines.push("flipped. When it matters, ask the CLI instead — it is always current:"); +lines.push(""); +lines.push("```bash"); +lines.push("failproofai policies # every policy, with enabled status and params"); +lines.push("```"); +lines.push(""); +lines.push("## How to use this for triage"); +lines.push(""); +lines.push("Enabling a builtin beats writing a custom policy: nothing to maintain, no naming"); +lines.push("trap, no fail-open risk, and it ships with tests."); +lines.push(""); +lines.push("Before concluding \"no builtin covers this\", check whether a **parameterized** one"); +lines.push("does — several take allowlists or thresholds that widen their scope considerably."); +lines.push("Params go in the `policyParams` map, keyed by short name."); +lines.push(""); +lines.push("To enable: add the short name to `enabledPolicies` in `.failproofai/policies-config.json`."); +lines.push(""); +lines.push("---"); + +for (const [category, policies] of byCategory) { + lines.push(""); + lines.push(`### ${category}`); + lines.push(""); + lines.push("| Policy | Default | Events | What it catches |"); + lines.push("|---|---|---|---|"); + for (const p of policies) { + const events = (p.match?.events ?? []).join(", ") || "—"; + const params = p.params ? ` _(params: ${Object.keys(p.params).join(", ")})_` : ""; + const beta = p.beta ? " _(beta)_" : ""; + lines.push( + `| \`${p.name}\` | ${p.defaultEnabled ? "**on**" : "off"} | ${events} | ${p.description}${params}${beta} |`, + ); + } +} + +lines.push(""); +lines.push("> The five `require-*-before-stop` policies gate the end of a turn. A gate whose"); +lines.push("> condition cannot be met in the current project loops forever — see `traps.md` §6"); +lines.push("> before enabling one."); +lines.push(""); +lines.push("---"); +lines.push(""); +lines.push("## Audit-only detectors"); +lines.push(""); +lines.push("These have no real-time builtin equivalent, so they are the prime candidates for"); +lines.push("custom policies. List them from source with:"); +lines.push(""); +lines.push("```bash"); +lines.push("bun -e 'const {AUDIT_DETECTORS}=await import(\"./src/audit/detectors/index.ts\");"); +lines.push("for (const d of AUDIT_DETECTORS) console.log(d.name, \"|\", d.category+\"/\"+d.severity, \"|\", d.description)'"); +lines.push("```"); +lines.push(""); +lines.push("All but `reread-after-edit` are Bash-command patterns, so a `PreToolUse` policy"); +lines.push("filtering on `ctx.toolName === \"Bash\"` and matching `ctx.toolInput.command` covers"); +lines.push("most of them. `reread-after-edit` needs cross-call session state, which hooks cannot"); +lines.push("see — that one needs a builtin, not a custom policy."); +lines.push(""); + +const generated = lines.join("\n"); + +if (check) { + const current = existsSync(OUT) ? readFileSync(OUT, "utf8") : ""; + if (current !== generated) { + console.error("references/builtins.md is OUT OF DATE."); + console.error(`Registry has ${BUILTIN_POLICIES.length} builtins. Run: bun ${process.argv[1]}`); + process.exit(1); + } + console.log(`references/builtins.md is current (${BUILTIN_POLICIES.length} builtins).`); + process.exit(0); +} + +writeFileSync(OUT, generated); +console.log(`Wrote ${OUT} — ${BUILTIN_POLICIES.length} builtins across ${byCategory.size} categories.`); diff --git a/.agents/skills/policy-author/scripts/test-policy.mjs b/.agents/skills/policy-author/scripts/test-policy.mjs new file mode 100644 index 000000000..0afe7d829 --- /dev/null +++ b/.agents/skills/policy-author/scripts/test-policy.mjs @@ -0,0 +1,207 @@ +#!/usr/bin/env node +/** + * Run a policy against a synthetic hook payload and report the decision. + * + * Exists because loading and execution are both fail-open: a policy that was + * never loaded, threw, or timed out is indistinguishable from one that + * correctly allowed. The only way to know a policy works is to make it fire. + * + * node test-policy.mjs --policy --event PreToolUse --tool Bash \ + * --input '{"command":"cd /x && ls"}' --expect instruct + * + * node test-policy.mjs --cases cases.json + * + * With --policy the policy is copied into a throwaway project (which also acts + * as HOME) so neither the real project config, `customPoliciesEnabled: false`, + * nor user-scope policies can affect the result. Without it, the payload runs + * against the current directory's real config — useful for testing builtins. + */ +import { spawnSync } from "node:child_process"; +import { mkdtempSync, mkdirSync, copyFileSync, writeFileSync, existsSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { resolve, dirname, basename, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +// ---------------------------------------------------------------- arg parsing + +function parseArgs(argv) { + const out = {}; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (!a.startsWith("--")) continue; + const key = a.slice(2); + const next = argv[i + 1]; + if (next === undefined || next.startsWith("--")) out[key] = true; + else { out[key] = next; i++; } + } + return out; +} + +const args = parseArgs(process.argv.slice(2)); + +if (args.help || (!args.cases && !args.event)) { + console.log(` +Usage: + test-policy.mjs --policy --event [--tool ] \\ + --input '' [--expect deny|allow|instruct] [--cwd ] + test-policy.mjs --cases + + --policy Policy file to test in isolation. Omit to test the current + directory's real config (builtins). + --event Hook event, e.g. PreToolUse, PostToolUse, Stop. + --tool Canonical tool name: Bash, Read, Write, Edit, Grep. + --input Tool input as JSON, e.g. '{"command":"sudo ls"}'. + --expect Assert the outcome. Exits 1 on mismatch. + --cwd cwd reported to the policy. Defaults to the sandbox / process cwd. + +cases.json is an array of objects with the same keys minus --policy: + [{ "name": "blocks sudo", "event": "PreToolUse", "tool": "Bash", + "input": { "command": "sudo ls" }, "expect": "deny" }] +`.trim()); + process.exit(args.help ? 0 : 1); +} + +// ------------------------------------------------------------------- runner + +/** Walk up looking for this repo's dev launcher; fall back to the published CLI. */ +function findRunner() { + let dir = dirname(fileURLToPath(import.meta.url)); + for (let i = 0; i < 8; i++) { + const candidate = join(dir, "scripts", "dev-hook.mjs"); + if (existsSync(candidate)) return { cmd: "node", pre: [candidate] }; + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + return { cmd: "npx", pre: ["-y", "failproofai"] }; +} + +const runner = findRunner(); + +/** + * Build a throwaway project that is also HOME, so project config, global + * config and user-scope policies all resolve inside it and nothing on the + * real machine leaks into the result. + */ +function makeSandbox(policyPath) { + const root = mkdtempSync(join(tmpdir(), "failproofai-policy-test-")); + mkdirSync(join(root, ".failproofai", "policies"), { recursive: true }); + + // The convention regex is /policies\.(js|mjs|ts)$/ — a file that does not + // match is silently skipped, so normalize the name rather than trusting it. + let name = basename(policyPath); + if (!/policies\.(js|mjs|ts)$/.test(name)) { + name = name.replace(/\.(js|mjs|ts)$/, "-policies.$1"); + console.log(` note: renamed to ${name} to satisfy the loader convention`); + } + copyFileSync(policyPath, join(root, ".failproofai", "policies", name)); + writeFileSync( + join(root, ".failproofai", "policies-config.json"), + JSON.stringify({ enabledPolicies: [] }, null, 2), + ); + return root; +} + +/** + * Deny is reported in several different shapes depending on event and CLI. + * Missing one silently downgrades a real deny to "allow", which is the exact + * failure this script exists to catch — so handle all of them. + * + * PreToolUse (Claude) hookSpecificOutput.permissionDecision === "deny" + * PermissionRequest hookSpecificOutput.decision.behavior === "deny" + * instruct hookSpecificOutput.additionalContext + * Copilot/Goose/Devin/… { decision: "block" } + * Cursor/Pi flat { permission: "deny" } + * Factory non-Stop exit code 2, reason on stderr + */ +function classify(stdout, exitCode, stderr) { + const text = stdout.trim(); + if (!text) { + if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() }; + return { decision: "allow", reason: "" }; + } + let parsed; + try { parsed = JSON.parse(text); } catch { return { decision: "unparseable", reason: text }; } + const h = parsed.hookSpecificOutput ?? {}; + if (h.permissionDecision === "deny") return { decision: "deny", reason: h.permissionDecisionReason ?? "" }; + if (h.decision?.behavior === "deny") return { decision: "deny", reason: h.decision.message ?? "" }; + if (h.additionalContext) return { decision: "instruct", reason: h.additionalContext }; + if (parsed.decision === "block") return { decision: "deny", reason: parsed.reason ?? "" }; + if (parsed.permission === "deny") return { decision: "deny", reason: parsed.reason ?? "" }; + if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() }; + return { decision: "allow", reason: "" }; +} + +function runCase(c, sandbox) { + const cwd = c.cwd ?? sandbox ?? process.cwd(); + const payload = { + hook_event_name: c.event, + session_id: "policy-test", + transcript_path: "/dev/null", + cwd, + }; + if (c.tool) payload.tool_name = c.tool; + if (c.input) { + // `{{cwd}}` expands to the effective cwd, so cases can reference the + // sandbox path (generated per run) in commands like `cd {{cwd}} && ls`. + const raw = typeof c.input === "string" ? c.input : JSON.stringify(c.input); + payload.tool_input = JSON.parse(raw.split("{{cwd}}").join(cwd)); + } + if (c.event === "Stop") payload.stop_hook_active = false; + + const env = { ...process.env, FAILPROOFAI_TELEMETRY_DISABLED: "1" }; + if (sandbox) env.HOME = sandbox; + + const res = spawnSync(runner.cmd, [...runner.pre, "--hook", c.event], { + input: JSON.stringify(payload), + encoding: "utf8", + cwd, + env, + timeout: 20000, + }); + + if (res.error) return { decision: "error", reason: String(res.error) }; + return classify(res.stdout ?? "", res.status, res.stderr); +} + +// --------------------------------------------------------------------- main + +let cases = args.cases + ? JSON.parse(await import("node:fs").then((fs) => fs.readFileSync(resolve(args.cases), "utf8"))) + : [{ name: `${args.event}${args.tool ? " " + args.tool : ""}`, event: args.event, tool: args.tool, + input: args.input, expect: args.expect, cwd: args.cwd }]; + +// A command-line --cwd is the default for every case that does not set its own. +// Without this, --cwd is silently ignored alongside --cases, and policies that +// inspect real filesystem or git state quietly evaluate against the empty +// sandbox instead — producing allow-everything results that look like failures +// of the policy rather than of the harness. +if (args.cwd) cases = cases.map((c) => ({ ...c, cwd: c.cwd ?? args.cwd })); + +let sandbox = null; +if (args.policy) { + const p = resolve(args.policy); + if (!existsSync(p)) { console.error(`Policy file not found: ${p}`); process.exit(1); } + sandbox = makeSandbox(p); +} + +let failed = 0; +try { + for (const c of cases) { + const got = runCase(c, sandbox); + const label = c.name ?? `${c.event} ${c.tool ?? ""}`.trim(); + if (!c.expect) { + console.log(` ${label}\n → ${got.decision}${got.reason ? ": " + got.reason : ""}`); + continue; + } + const ok = got.decision === c.expect; + if (!ok) failed++; + console.log(` ${ok ? "PASS" : "FAIL"} ${label} (expected ${c.expect}, got ${got.decision})`); + if (got.reason) console.log(` ${got.reason.slice(0, 160)}`); + } +} finally { + if (sandbox) rmSync(sandbox, { recursive: true, force: true }); +} + +if (failed) { console.log(`\n${failed} of ${cases.length} failed.`); process.exit(1); } +if (cases.some((c) => c.expect)) console.log(`\nAll ${cases.length} passed.`); diff --git a/.opencode/plugins/failproofai.mjs b/.opencode/plugins/failproofai.mjs index 71f663ceb..5464a926d 100644 --- a/.opencode/plugins/failproofai.mjs +++ b/.opencode/plugins/failproofai.mjs @@ -18,7 +18,7 @@ // • src/hooks/integrations.ts (buildOpenCodePluginShim production template) // When #337 landed, this dev shim drifted and `block-read-outside-cwd` // silently no-op'd on every opencode `read` call inside this repo. -import { spawnSync } from "node:child_process"; +import { spawn } from "node:child_process"; import { resolveDevSpawn } from "../../scripts/dev-hook.mjs"; const BUS_EVENT_MAP = { @@ -70,34 +70,58 @@ function canonicalizeToolInput(canonicalToolName, args) { return out; } -function runFailproofai(eventName, payload, directory) { - // Shared with the JSON configs' launcher: locates bun across PATH, ~/.bun/bin, - // $BUN_INSTALL, Homebrew and every nvm version dir. A bare spawnSync("bun") - // dies with ENOENT the moment bun is off the hook's PATH. - const spawn = resolveDevSpawn(); - if (!spawn) { +/** + * Run failproofai for one event WITHOUT blocking opencode's event loop. + * + * This was `spawnSync`, and opencode loads the plugin in-process in the TUI — + * so every hook froze rendering and input for the subprocess's full duration, + * with a 60s ceiling. The verdicts are unchanged: every caller already `await`s + * `applyDecision`, so the deny still lands before the tool runs. What changes + * is that the wait is now a promise rather than a blocked thread, and the TUI + * keeps painting while it happens. + * + * Fail-open on spawn error, and on the timeout — a policy that never ran must + * not read as a deny. + */ +async function runFailproofai(eventName, payload, directory) { + const resolved = resolveDevSpawn(); + if (!resolved) { process.stderr.write( "[failproofai-dev] bun not found — opencode policies are NOT enforcing. " + "Install bun: curl -fsSL https://bun.sh/install | bash\n", ); return { exitCode: 0, stdout: "", stderr: "" }; } - const r = spawnSync(spawn.cmd, [...spawn.args, "--hook", eventName, "--cli", "opencode"], { - input: JSON.stringify(payload), - encoding: "utf8", - timeout: 60_000, - cwd: directory, + const cmd = resolved.cmd; + const args = [...resolved.args, "--hook", eventName, "--cli", "opencode"]; + return await new Promise((resolveRun) => { + let child; + try { + child = spawn(cmd, args, { cwd: directory }); + } catch { + resolveRun({ exitCode: 0, stdout: "", stderr: "" }); + return; + } + let stdout = ""; + let stderr = ""; + let settled = false; + const finish = (exitCode) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolveRun({ exitCode, stdout, stderr }); + }; + const timer = setTimeout(() => { + try { child.kill("SIGKILL"); } catch { /* already gone */ } + finish(0); + }, 60_000); + child.stdout.on("data", (d) => { stdout += d; }); + child.stderr.on("data", (d) => { stderr += d; }); + child.on("error", () => finish(0)); + child.on("close", (code) => finish(code ?? 0)); + child.stdin.on("error", () => {}); + child.stdin.end(JSON.stringify(payload)); }); - // `r.status` is null when the spawn never ran (ENOENT) or the child was - // killed. Coercing that to 0 reports "allowed" for a policy that never got - // to run, so say so on stderr rather than failing open in silence. - if (r.status === null) { - process.stderr.write( - `[failproofai-dev] ${eventName} hook did not run (${r.error?.code ?? r.signal ?? "unknown"}) — not enforcing\n`, - ); - return { exitCode: 0, stdout: "", stderr: "" }; - } - return { exitCode: r.status, stdout: r.stdout ?? "", stderr: r.stderr ?? "" }; } async function applyDecision(result, ctx, eventName) { @@ -150,7 +174,7 @@ export default async function failproofaiPlugin({ client, directory }) { } } if (!prompt) prompt = (info.text || info.content || props.text || "").toString(); - const r = runFailproofai("UserPromptSubmit", { + const r = await runFailproofai("UserPromptSubmit", { session_id: sessionID, cwd: directory, hook_event_name: "UserPromptSubmit", prompt, }, directory); await applyDecision(r, { client, sessionID }, "UserPromptSubmit"); @@ -160,7 +184,7 @@ export default async function failproofaiPlugin({ client, directory }) { if (!claudeEvent) return; const props = event.properties || {}; const sessionID = props.sessionID || (props.session && props.session.id) || props.id; - const r = runFailproofai(claudeEvent, { + const r = await runFailproofai(claudeEvent, { session_id: sessionID, cwd: directory, hook_event_name: claudeEvent, }, directory); await applyDecision(r, { client, sessionID }, claudeEvent); @@ -168,7 +192,7 @@ export default async function failproofaiPlugin({ client, directory }) { "tool.execute.before": async (input, output) => { const canonicalTool = canonicalizeTool(input.tool); - const r = runFailproofai("PreToolUse", { + const r = await runFailproofai("PreToolUse", { session_id: input.sessionID, cwd: directory, tool_name: canonicalTool, @@ -180,7 +204,7 @@ export default async function failproofaiPlugin({ client, directory }) { "tool.execute.after": async (input, output) => { const canonicalTool = canonicalizeTool(input.tool); - const r = runFailproofai("PostToolUse", { + const r = await runFailproofai("PostToolUse", { session_id: input.sessionID, cwd: directory, tool_name: canonicalTool, @@ -193,7 +217,7 @@ export default async function failproofaiPlugin({ client, directory }) { "permission.ask": async (input, output) => { const canonicalTool = canonicalizeTool(input.tool); - const r = runFailproofai("PermissionRequest", { + const r = await runFailproofai("PermissionRequest", { session_id: input.sessionID, cwd: directory, tool_name: canonicalTool || input.command || "permission", diff --git a/CHANGELOG.md b/CHANGELOG.md index 9282faee5..e3a5a774a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,84 @@ # Changelog -## 1.0.4-beta.0 — 2026-09-02 +## 1.0.4-beta.4 — 2026-09-09 + +### Docs + +- Add the repository-local policy-author skill and its reference material for maintaining failproofai enforcement policies (#789) + +### Features + +- The desktop banner now asks for review without claiming every detector match is a confirmed leak: "Possible credential exposure was found in your agent transcripts. Run failproofai audit to verify the matches and rotate anything real." It omits both the noisy count and the unrelated email setup pitch. No action buttons, deliberately: `Notify` delivers clicks back to the short-lived audit child after it has exited, so the button would be dead (#789) + +- Leaked credentials found in a transcript now reach the person whose machine leaked them, on the machine, without an account or an email address. Four channels, each covering a case the others cannot. **In-CLI**: a two-line notice inside the agent session, on six hosts with a real user-visible surface (claude, codex, copilot, factory, OpenCode and Pi). OpenCode uses `client.tui.showToast(...)` and Pi uses `ctx.ui.notify(...)`; `hookSpecificOutput.additionalContext` was rejected because it reaches the model rather than the person. **Linux desktop**: a D-Bus `Notify` written by hand against `/run/user//bus`, an address CONSTRUCTED from the uid rather than read from an environment the system-scope daemon does not have; the reply is awaited, because a fire-and-forget call against a bus with no notification server returns exit 0 and empty output while the notification silently evaporates. **macOS**: a per-user LaunchAgent and an `osacompile`d applet, installed silently by `failproofai config` and removed by `failproofai uninstall`, because `failproofaid` is a LaunchDaemon and nothing in launchd's system domain can reach Notification Center — the same split Time Machine ships. The applet has its own bundle id, so the banner is attributed to failproofai and gets its own entry in System Settings rather than inheriting Script Editor's. **Email**: the scheduled digest now carries a `leaks` array. Every channel claims each finding through an O_EXCL marker file, per channel — a `notifiedAt` field on the record was measured and failed 100% of the time under concurrent sessions, and two sessions with different findings permanently lost one mark and re-notified forever (#789) + +- Scheduled audits are ON by default for a machine that completed setup, and the flip preserves the fail-safe it replaces. Three-way, not two: `config.json` present AND carrying an `audit` table means on unless it says exactly `false`; absent file, unparseable JSON, and no `audit` table at all still read as OFF, because those are the three ways of not knowing and "we could not tell" must never start a scan that reads every transcript on disk. `crates/failproofaid/src/audit_lane.rs` and `src/hooks/fp-config.ts` make the identical distinction over the identical bytes, each with a test asserting the same table — the daemon scanning while the settings page says off is the bug that split would otherwise produce. Turning it on sends nothing on its own: the digest stays gated on `reports_consented_at`, so a machine that flips on at upgrade scans locally and mails nobody (#789) + +- `/audit` renders the leak report that replaced the score: one row per DISTINCT credential rather than per sighting — a key pasted into forty commands is one thing to rotate — showing the masked value, what it is, where it went, which harness carried it, when it was last seen, and how it got there. Each row says whether the exposure was blockable (the agent SENT it, so a PreToolUse gate can deny it next time) or not (the agent RECEIVED it, which no gate can undo), and gives the one piece of advice that applies: a recognised prefix means a console to revoke at, while an unattributed value — the majority, per the pattern census — means finding what reads the identifier. "not a secret" dismisses a row, and the finding is RETAINED rather than deleted so the next scan cannot rediscover the same value and alert again (#789) + +- New setting `audit.notify`, written by both `failproofai audit --notify` / `--no-notify` and the toggle on `/audit`, read by the audit child at the moment a notification is about to fire. On by default. Its own switch rather than a flag on `--schedule`, because wanting the weekly scan without the banner is a coherent position and silencing a banner must not quietly switch off the scan. It does not silence the in-CLI notice, which costs two lines, appears inside a session the user is already driving, and is the only channel left on a headless box — turning off every channel at once produces a state indistinguishable from broken (#789) + +- The rest of the old audit report is switched off with the score, ahead of rebuilding it bottom-up around leak detection. Dormant now: the persona classifier and its poster, the strengths and quirks sections, the punch-list with its install-all funnel, the invite section, and the 8 behavioural detectors. Also the archetype rarity — eight hardcoded integers under the comment "Seeded with snapshot values; swap for live aggregates once that pipeline lands", rendered as "// only 18% of agents are this archetype" and baked by `html-to-image` into the PNG people post publicly. A fabricated population statistic on a shared card is a claim we cannot support, and that poster was its only surface. Nothing is deleted: every module and component is intact and still unit-tested, and only the call sites and renders are commented, each cross-referenced to the one explanation in `src/audit/scoring.ts`. `/audit` now renders the scan's own numbers — tool calls, transcripts, projects — because the difference between "this page is being rebuilt" and "this product is broken" is worth the twenty lines it costs to say so. **What is untouched and still running: the 12-CLI adapter layer and transcript walking, the replay loop, the per-transcript cache, redaction, and the emailed digest.** `failproofai audit` still scans and still prints its summary; `report.ts` needed no change because it had zero production callers already. Two tests were repointed rather than deleted — `filters by --policy` now filters on a builtin so the filter itself stays covered, and `carries stateful detectors across the boundary` is skipped with the plumbing it guards left deliberately in place, so it passes again the moment the detector loop returns (#789) + +- The audit score, letter grade and projected-score line are switched off — commented out at every call site rather than deleted, with each original preserved inline and cross-referenced back to a single explanation in `src/audit/scoring.ts`. Every function there is intact and still covered by `__tests__/audit/scoring.test.ts`, so restoring is un-commenting rather than rewriting. Two reasons. The product one: the audit did too many things shallowly, and grading a machine 0-100 with a letter tier was not the half worth keeping. The correctness one is worse and should be read before anyone switches it back on — **`deriveScore` never read `enabledInConfig`**; only `projectedScore` did. So the "enable all N → projected {score}" line above the install-all button promised a number the product could not deliver: a user who took the prescription, enabled the policies and re-ran got the *same* score with the projection gone. Advertised up to +22, delivered 0. That line now reads "enable all N → one command, enforced on every tool call", which is the thing that was always true. The prescription itself, the install-all button and the `audit_copy_clicked` → `hooks_installed` funnel are untouched — none of them ever read the score, and `missing`, the funnel's denominator, still ships on `audit_dashboard_viewed`. The archetype cue card survives and becomes the poster's headline: `classifyAgent` takes no score input, so the two were always separable. Share copy is archetype-only — 18 of the 20 templates lost a subordinate clause and 2 whose entire premise was the number are commented out, leaving 9 per channel, with the score-bearing originals preserved verbatim at the bottom of `share-templates.ts`. `score`/`grade` stop appearing on `audit_dashboard_viewed` and `audit_card_share_clicked`, so a PostHog dashboard keyed on them breaks rather than quietly reading zero. Also corrected `empty-state.tsx`, which promised "a tier, a score, and a punch-list" — the tier was never rendered to a user at any point (#789) ### Fixes +- Leak-detector precision fixes now reach warm machines and repair the report they already wrote. The per-transcript cache previously keyed builtin policies and behavioural detectors but not `findSecrets`, so changing credential matching kept serving old `result.leaks`; the cache now carries an explicit leak-scanner version. A successful all-CLI, all-history scan also rebuilds the persistent finding set from its authoritative result instead of only merging, so matches the current detector rejects disappear immediately rather than remaining active for 90 days. Scoped or incomplete scans still merge, because absence outside their coverage proves nothing. New-notification ids are compared with the previous record during a rebuild, so correcting stale data does not re-alert for credentials the user already knew about (#789) + +- The credential audit no longer creates findings from a variable name alone. Measured on a real 2,673-session machine, that lane produced hundreds of ordinary code literals and realistic scanner fixtures (`keyType`, `max_output_tokens`, synthetic `API_KEY` samples) that looked alarming but were not leaked credentials. Findings now require a recognizable credential shape; an assignment name can still label an already-recognized value so the report remains actionable. Sessions deliberately researching or generating secret-scanner fixtures are also suppressed only after multiple independent intent signals, preventing the scanner from auditing its own test corpus while leaving ordinary sessions that merely mention credentials untouched. The leak-scanner cache version is bumped again, and incremental scans merge new leak sightings while propagating research suppression across the cached prefix and appended tail (#789) + +- Failed desktop notification delivery is retryable. Linux D-Bus failures and thrown notifier errors now release only the desktop-channel O_EXCL claims won by that run, so the next scheduled audit tries again instead of converting a transient missing session or notification server into permanent silence. Successful Linux delivery and durable macOS queueing retain their claims (#789) + +- `--email` stops promising something it cannot do. Its help read "With `--schedule`, skips the sign-in prompt", which says no interaction is needed — so a non-interactive run with the flag set looked broken. It does not skip signing in and cannot: it fills in the address so you are not asked for it, and a one-time code is still emailed to you to paste. The flag itself parses correctly in all four forms (`--schedule 7 --email x`, `--schedule --email x`, `--email=x`, and either order). The help row now says what it does, and the non-interactive error — the one you actually hit — names `--email` explicitly and explains why it does not help, because that is where somebody is standing when they need to know (#789) + +- `failproofai audit --help` and the top-level `failproofai --help` describe what the audit is now. The audit's tagline still read "review your agent CLIs for risky and wasteful patterns" — the scored report that was switched off — and the top-level row still said "Scan your agents' history, then open the audit view". They now name credentials, and `--schedule` says it notifies on this machine as well as emailing. The footer keeps the local-only promise the docs also make (a test asserts that phrase) while naming all three channels a scan tells you through (#789) + +- The audit was silently dropping session databases, and the `Aborted(OOM)` lines on stderr were the only sign. `lib/sqlite-reader.ts` falls through to sql.js whenever `node:sqlite` is absent — every bun process and every supported Node below 22.5, so `engines.node: ">=20.9.0"` puts real users there. That build's heap is a **fixed 22,151,168-byte ArrayBuffer** compiled with `ALLOW_MEMORY_GROWTH` off, so growing it is `abort("OOM")`; and opening a WAL-flagged image makes SQLite build a wal-index shared-memory region that sql.js's MEMFS VFS never reclaims on `close()`. Measured: the **128th** open exhausts the heap — identical threshold for a 319 KB database and a 10 MB one, because the leak is a fixed per-connection allocation, not data. Worse, `initSqlJs` memoizes one module per process, so after the abort **every later `openSqliteReadonly` silently returns null**. A full audit opens ~150 databases (devin, goose and opencode each open theirs once per session), so it crossed the line and then read nothing further, while still reporting success. The fix clears the WAL flag (header bytes 18/19) on our own in-memory copy — sql.js never sees the `-wal` sidecar anyway, which is the snapshot caveat that module already documents, so the flag bought nothing. On the machine this was found on: 22 aborts → 0, and sessions scanned 2,661 → **2,670**. The regression test mocks `node:sqlite` away to force tier 2, because without that it is vacuously green: it passed identically with the fix reverted, never reaching the code under test (#789) + +- The three intermittent `fp-reset` failures were reading the developer's real machine. The file isolates `FAILPROOFAI_HOME` and nothing else, so `checkLayoutForCli()` → `healDaemonFlag()` asked the REAL `daemonServiceStatus()` — an `existsSync` on `/etc/systemd/system` plus `systemctl is-active`. On any machine that has actually run `failproofai config` that answers "running", and the next line awaits `probeDaemonEndToEnd()`: a 10s poll (`DAEMON_PROBE_READY_TIMEOUT_MS`) against a socket inside the test's own temp home that nothing will ever listen on. Twice vitest's 5s default, so the two `daemon.configured: true` tests time out on a developer box and pass in CI. The timeout was the smaller half: vitest fails the test but cannot cancel the promise, so the probe finished ~5s later and ran `updateConfig({daemon:{configured:false}})` — and because every path helper resolves `FAILPROOFAI_HOME` at CALL time, that write landed in a LATER test's home. A stray `config.json` is a layout-4 landmark that `detectLayout()` checks before `config.toml`, so the layout-2 home the spool-drain test seeds read as "current" and the migration it asserts never ran. One unisolated read, three failures, two describes apart. `FAILPROOFAI_SYSTEMD_DIR` is now redirected per test, and the two daemon tests pin `daemonServiceStatus` to the state they name — without that pin they pass off the self-heal message, which contains every string they assert, while exercising a different branch entirely (#789) + +- The eight "known pre-existing" dogfood-config failures were never pre-existing, and the reason they survived is worse than the failures. Nine config files had been emptied on disk to `{}` / `{"version":1}` — and all nine carried git's `skip-worktree` bit, so `git status` reported a completely clean tree, `git checkout -- ` **failed silently** (exit 1, "pathspec did not match") leaving the empty file in place, and `git stash` said "No local changes to save". Every assertion in `dogfood-configs.test.ts` reads the working tree, so all eight failed, and every attempt to restore from git was a no-op that reported success. The committed content was correct the whole time and satisfies every assertion — verified by re-running the test's own extractor against the HEAD blobs. Clearing the bit and restoring takes the file from 8 failures to **65 passing**. The real cost was never the red tests: with those files empty, failproofai enforced **nothing** in this repo for codex, copilot, cursor, factory, devin, antigravity, goose, opencode and pi, silently, while git insisted everything was fine. Two guards are added so it cannot hide again — one asserting no dogfood config is skip-worktree'd or assume-unchanged, one asserting none is empty JSON. The first deliberately matches on anything that is not exactly `H`: `git ls-files -v` tags skip-worktree with an UPPERCASE `S` and assume-unchanged with a lowercase letter, so the obvious `/^[a-z]/` filter would have passed vacuously against the actual damage (#789) + +- The in-CLI notice keys on whether the USER has looked, not on whether we emitted. The per-finding "delivered once" marker is right for a desktop banner and wrong here, and it failed twice in one day on the same machine: once because the notice was attached to `SessionStart`, whose channel the host ignores (Claude Code documents `hookSpecificOutput.additionalContext` there, not `systemMessage`), and once because `disableAllHooks` was set in the project under test. Both marked every finding delivered and showed nothing, permanently, with no retry — 499 findings in the measured case. The notice now shows while findings exist that are newer than the last time the report was opened, bounded to once per session, and stops when the user opens it. A dropped notice costs one session's silence instead of the alert; the only thing that silences it for good is the outcome the notice exists to produce. `reportViewedAt` joins `dismissed` in the identity file, for the same reason: losing it re-alerts about credentials somebody already reviewed (#789) + +- The transcript scan bounds MEMORY as well as file count. Concurrency counted files, which is the wrong unit — the reader materialises every event of a transcript into JS objects, so a 57 MB JSONL becomes several hundred MB of them, and on a real 1.15 GB corpus (2,107 files, 7 holding 224 MB) eight workers could put a quarter of the corpus in memory at once. Admission is now weighted by size against a 48 MB budget, with a file larger than the whole budget admitted alone rather than refused — otherwise the largest transcript on a machine, the one most likely to hold something, could never be scanned. This is explicitly NOT the fix for the `Aborted(OOM)` lines that machine prints: those are ~22 per run, appear in the first two seconds, are unchanged at 48 MB or 16 MB, and do not fail the run. Their source is still unidentified and the comment says so (#789) + +- Two thirds of the leak report was structural JSON, not credentials. Measured on a real 2,664-transcript machine: 500 findings, of which **254 were `SESSION` names** — `DBUS_SESSION_BUS_ADDRESS`, `XDG_SESSION_TYPE`, `SESSION_MANAGER`, `session_id`, `sessionUpdate` — and another 80 were a bare `key`. `SESSION` is removed from the name lists outright rather than demoted to compound-only, because demoting would have changed nothing: `DBUS_SESSION_BUS_ADDRESS` is already a compound. A session IDENTIFIER is not a rotatable credential, and a session SECRET still matches through `SECRET`, `TOKEN`, or `KEY` in `sessionKey`. `KEY` becomes compound-only for the same reason `PWD` already was — a bare `key=` is a React list prop or a map entry. This is affordable precisely because the SHAPE layer catches real vendor keys regardless of name; the name layer exists for first-party secrets, and those essentially always carry a qualified name. `SIG` was demoted alongside them and put back: a bare `sig=` in a URL query IS a request signature, an existing test says so, and it never appeared in the measured noise — a demotion needs evidence, not a plausible story. Also: a UUID is now never a credential, which is the dominant shape of the `session_id` values that remained. Re-measured against the same record: 500 findings → 170 (#789) + +- The in-CLI notice was claimed and never shown. `notice.ts` derived each host's channel from a live probe on **Stop**, and `handler.ts` then allowed SessionStart too, on the assumption that a channel proven for one event works for another. It does not: Claude Code documents `hookSpecificOutput.additionalContext` for SessionStart, not `systemMessage`. On a real machine 499 findings were marked delivered and the user saw nothing — SessionStart fires first in a session, consumed every claim, and the host dropped the field. That is the worst outcome this design has: a finding recorded as delivered that reached nobody and is never retried. The notice now fires on Stop only, which happens at the end of every assistant turn, so nothing is lost by waiting for it (#789) + +- `/audit` has a way to run the audit again. The re-scan control lived in the old report's sections and went dormant when they were commented out, leaving `onRerun` wired up, passed down, and called by nothing — so the only way to re-scan was to go back to the terminal. It is now a **run again** button in the scan section header (#789) + +- A row says when the credential LEAKED, not when a scan last noticed it. Every row read "today", which was true and useless: the record keeps the 500 most recently seen findings, so everything surviving the cap necessarily carried a recent `lastSeen`. The date is now `firstSeen`, with `last seen` shown beside it only when the two differ (#789) + +- The desktop banner's count matches the report it points at. `writeLeakRecord` prunes to `MAX_FINDINGS`, so on a large history the announced count and the stored one diverge wildly — a real run announced "5703 credentials" while the dashboard it sends you to showed 500. A banner that disagrees with its own page teaches the reader that the number is noise (#789) + - The two PyPI publish workflows open the next version's `CHANGELOG` section in the same `bump` commit that moves `_version.py`, via a new `scripts/changelog-open.py`. `bump` used to move the version alone, leaving `main` on a version with no section — the exact state `scripts/changelog-section.py` refuses at release time and `__tests__/ci/python-version-pipeline.test.ts` asserts against. Because bump commits carry a skip-ci marker, that never went red on itself: it went red on the next unrelated PR to run CI, which is how `main` broke after the 0.0.1b1 publish (repaired by hand in #755, which named the recurrence and left it) and again after 0.0.1b2. `sdk/python/CHANGELOG.md` gets the 0.0.1b3 section that was missing. The opener is idempotent — a re-run of `bump` against a `main` that already carries the section is a no-op rather than a second heading, which would put only the first one's body on the GitHub Release — and it matches the version with the same trailing word boundary the extractor uses, so opening `0.0.1b1` is not satisfied by an existing `0.0.1b10` (#787) +- The Linux desktop notification never worked against a real bus. The hand-written D-Bus encoder declared its header-fields array two bytes too long, because it counted the alignment padding after the final field — padding that belongs to the message, not to the array — so `dbus-daemon` dropped the connection on the first message and every Linux user would have got silence. It passed its tests because the test server was written from the same assumptions as the encoder, and mirrored the mistake. Found by diffing our `Hello` against a real client's byte for byte (`0x70` vs `0x6e`). The encoder is now an offset-tracking marshaller, so alignment is computed against the real message offset instead of by padding sub-buffers in isolation, and the tests run against an actual `dbus-daemon` on a private socket. Two more bugs fell out of the same rewrite: no `close` handler, so a bus hanging up read as a 2-second timeout and pointed at a slow desktop rather than at us; and a state machine that treated "the next chunk" as its reply, so the `NameAcquired` signal a real bus emits after `Hello` was read as a successful delivery with a garbage id. Messages are now framed and matched on reply serial (#789) + +- Two catastrophically backtracking regexes made the audit hang on ordinary transcript content. `ASSIGNMENT_RE` (both the scanner and the redactor) and the redactor's URL-credentials matcher each had an unbounded quantifier that, on a long unbroken token, restarted at every position and backtracked a character at a time. Measured: a 300 KB base64-shaped blob took **over 20 seconds** in `findSecrets` and never finished in `redactExample`; a long URL took 4.8s. Transcripts carry base64 images, minified bundles and whole file contents as single lines constantly, so a scheduled scan would stall for minutes unattended. Bounding the identifier to 128 characters and the URI scheme and userinfo to their real limits takes the worst case from >20,000ms to 178ms, with no change to what either matches. The Rust redactor was never affected — it is hand-rolled scanning with no backtracking engine — and now has a test proving it, because the two engines are required to agree (#789) + +- A finding id is used as a filename, and nothing validated it: `markLeakNoticeDelivered(["../../../../tmp/PWNED"])` created that file. Ids are HMAC hex in any real run, so it was not reachable from normal use — but the record is JSON read off disk, and that is the difference between safe and incidentally safe. Both the notice marker and the macOS queue now refuse any id `fingerprintId` could not have minted. The queue also stranded a `notify-*.tmp` beside the watched directory on every failed write, where nothing would ever collect it (#789) + +- A single malformed entry in `leaks.json` turned a healthy scheduled audit into a failed one, permanently. `selectLeaks` dereferenced fields the file might not carry, and it throws out of `buildHarmReport`, which sits OUTSIDE `reportHarm`'s try — so a run that scanned correctly and wrote its cache correctly still exited 1, and kept doing so every run until somebody opened the file by hand. `readLeakRecord` now validates each finding at the single gate every surface reads through; a bad sighting loses the sighting rather than the finding, because the credential is the thing that still needs rotating (#789) + +- `shapeNotice` destroyed anything already on stdout that was not a JSON object. Its own contract says it leaves a verdict alone rather than risk breaking it, and it did so for unparseable JSON — but a parseable non-object (a JSON array, or plain text) fell past the merge and was replaced wholesale. It now leaves any unmergeable stdout untouched and drops the notice instead: a courtesy message is never worth more than what the host was already being told (#789) + +- The opencode plugin and the pi extension stop freezing their TUIs on every session. Both shims called `spawnSync(…, {timeout: 60_000})` on a path the host runs in-process and awaits, so the entire interface — rendering, input, the spinner — was blocked for as long as the subprocess took, with a 60-second ceiling. On pi the floor alone was 1.0-1.6s of frozen UI on every single session start, just to boot the binary; on opencode a hook made to take 8s delayed the user's first message by 8s. Two different fixes, because the two hosts differ. **opencode** now uses the async `spawn` with the timeout enforced by its own timer: every call site already `await`ed `applyDecision`, so the deny still lands before the tool runs and no verdict changed — the wait is simply a promise instead of a blocked thread, and the TUI keeps painting through it. **pi** awaits its handlers serially, so an async spawn would still block; instead the three events that were already DISCARDING the verdict (`session_start`, `tool_result`, `session_shutdown` — Pi's `ToolResultEventResult` has no `block` and the other two have no Result type at all) now forward detached and unref'd. The four that consume a decision (`tool_call`, `user_bash`, `input`, `agent_end`) still block, because blocking is what enforcement means; a test pins that split so nobody moves an event across it. Both shims fail open on spawn error and on timeout — a policy that never ran must not read as a deny (#789) + +- Neither redactor masks a key the build tool ships to the browser. Every publishable key on earth is named `*_KEY`, and the component rule matched all of them: measured, `isSecretName` returned true for all twelve of `NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY`, `NEXT_PUBLIC_SUPABASE_ANON_KEY`, `NEXT_PUBLIC_POSTHOG_KEY`, `VITE_API_KEY`, `VAPID_PUBLIC_KEY`, `PUBLISHABLE_KEY` and friends — including the names carrying the literal word PUBLIC. The Rust half had the identical hole with no exclusions at all. These are not conventionally public but **mechanically** public: Next.js, Vite, CRA, Expo, Nuxt, Gatsby, SvelteKit and Astro each inline the value into the JavaScript shipped to every visitor, so the framework published it before failproofai ever saw it. Over-redaction is usually cheap — masking something that might be secret costs readability — but this is the case where the value is provably not a secret, and reporting one is a security tool crying wolf about a browser-bundled config value. This repo's own committed PostHog `phc_` key, documented in source as "Write-only (safe to commit)", was being masked as an "assigned secret". The veto is deliberately narrow: the marker must be a PREFIX, so `MY_PUBLIC_FACING_API_SECRET` is untouched, and a word that merely starts with the same letters (`PUBLISHER_API_KEY`, `REPUBLIC_TOKEN`) is unaffected. It runs ahead of the value-shape rules, so a genuine `sk-` key assigned to a `VITE_` name is still masked on its own merits (#789) + +- Both redactors now mask an assignment whatever its separator or spacing. Each one required the `=` to sit immediately against the value, so `NAME = "value"`, `NAME: "value"` and `{"name": "value"}` — a config file, a YAML key and a JSON body, which is most of where credentials are actually written down — all passed through with the value intact. A secret only survived that gap if it independently matched a vendor prefix, and 230 of 237 secret-named assignments measured across this machine's 1,839 transcripts match none of them, so the assignment rule was the only thing standing between them and the sink. Two sinks, because the bug was written twice: `redactExample` feeds the emailed digest (`harm-report.ts`), and `fpai-collect`'s `match_assignment` scrubs every event the daemon ships to Cloud. Both are fixed to the same grammar and both carry the failing shapes as tests. Two hazards found while fixing it, each of which is why the narrow form survived so long: on the TypeScript side a `:` alternative makes `https:` read as an assignment whose value is the rest of the URL, and because a non-secret name returns the match unchanged but still CONSUMES it, a `?token=…` inside that URL was swallowed and never examined — adding a separator to catch more secrets silently stopped one already being caught, so the `:` now refuses a following `//`. On the Rust side the backwards name scan ends at the first non-identifier character, so a JSON key's own closing quote left it with zero characters to read; it now steps back over that quote the way it already stepped over the opening one. Both use horizontal whitespace only — `\s` and `char::is_whitespace` cross a newline, which lets a trailing `KEY:` glue the next line on and redact it as the value (#789) + +- Both redactors now decompose a camelCase credential name. `isSecretName` split the identifier on `_` alone, and the Rust `match_assignment` tested compoundness on the name it had *already lowercased* — so `sessionKey`, `dbPass`, `basicAuth` and `authCookie` reduced to a single component that matched nothing and shipped their values to the emailed digest and the Cloud spool verbatim, while the byte-identical `SESSION_KEY` was masked. 203 such names on this machine's corpus; measured, 6 of 12 ordinary credential-named assignments leaked. Both now split on `_`, `-`, `.` and camel humps, with an extra rule for an acronym meeting a word so `APIKey` yields `API` + `KEY`. The bare-`key` protection that exists because a lone `key=` matched React's `key` prop on every JSX list is untouched: a lone `key` has no hump, so it stays non-compound. `PASSPHRASE` joins the always-secret list, and `PWD` joins the compound-only list — `MYSQL_PWD` is MySQL's documented password variable while a bare `PWD` is the working directory, which is on every second line of a captured session. The module's own doc comment had claimed camelCase was covered, citing `_authToken`; that worked only because `TOKEN` is matched as a substring, and every name whose credential word was a *component* — `KEY`, `PASS`, `AUTH`, `PAT`, `SIG`, `SESSION`, `COOKIE` — was invisible (#789) + +- `formatMarkdown` is called with the one argument it takes. Two call sites in `redaction-sinks.test.ts` passed a second, empty-object argument, which `tsc --noEmit` rejects — so the branch would have failed CI's `quality` job while its tests passed, since vitest does not typecheck (#789) + +- The audit stops writing your credentials into the file it calls a shareable report. `redactExample` had exactly two non-definition call sites and both were in `harm-report.ts`, the emailed digest — so the ONE path that was already careful was the only one connected. `formatMarkdown` wrote `example` and `cwd` verbatim into `./failproofai-audit.md`, which the CLI prints as `Shareable report` and which defaults to the current directory, i.e. a git working tree; `formatJson` was a bare `JSON.stringify` of the whole `AuditResult`, examples, per-example cwds, scanned project paths and all; and the terminal renderer printed the raw example too. Both artifacts exist to be sent somewhere. Every renderer now redacts: the two that travel through the full `redactExample` (masked secrets, shortened home paths) and, for the JSON, a new `redactAuditResult` that walks the examples, their cwds, `projectsScanned` and `scope.projects` and returns a new object so the dashboard and the cache keep the values they need to render locally. The terminal gets a new `maskSecretsOnly` instead — a credential on screen is one screenshot or one pasted issue from being published, while `~/…/db.ts` protects nobody from their own directory names and costs the example its most useful half. `redactExample` is now defined in terms of `maskSecretsOnly`, so the two cannot drift. A test asserts the import is still there, because the defect was never a bad mask — it was a mask nobody called (#789) + +- The audit reads every subagent transcript instead of 8.7% of them. `listClaudeTranscripts` walked only the DIRECT children of `/subagents/`, which matched the layout Claude shipped when it was written and became wrong the day workflow runs started nesting their agents one level further down at `subagents/workflows//`. On the machine this was found on that is 1,839 transcripts on disk, 1,741 of them under `subagents/`, 1,679 of those nested — so the scan opened 160 files and reported the result as though it had read everything, which is the one failure a scanner must not have. Five of the seven files holding a genuine credential-bearing egress command were in the part it could not see. The walk is now recursive to a bounded depth, skipping symlinks so an unexpected layout costs a bounded walk rather than a scan that never returns. Subagent session ids are now qualified by their parent session and their path below `subagents/` (`__workflows__wf_123__agent-abc`), because basenames are not unique down there — every workflow run writes a `journal.jsonl`, and a run id is reused when its session is resumed, so the same relative path exists under two parents in one project. Both collisions were found by asserting uniqueness over the real corpus rather than by reasoning about it; `sessionId` keys example attribution and per-session detector state, so either would have merged unrelated sessions silently. Top-level session ids are unchanged (#789) + - `fp-cloud-cli`'s Click shim survives typer 0.27.2, which moved `Abort` out of its vendored Click. `_click_compat` wrapped all six vendored imports in one `try: … except ImportError: from click import …`, so that single missing name rebound **every** symbol to pip Click — the exact silent failure the module exists to prevent. Typer catches only its own Click's exceptions, so every typed error escaped uncaught: `fp alerts show ghost` exited 1 with an empty stderr instead of 6 with a message, and the same for exits 2, 3, 4 and 5. 105 tests went red on the dependabot bump that first installed 0.27.2. The Click is now chosen once — on whether `typer._click` exists at all — and each symbol imported from that choice, so a name that goes missing raises at import (a CLI that will not start) rather than silently downgrading every error to exit 1. `Abort` alone is resolved from `typer.Abort`, which tracks the move by construction: pip Click's before typer 0.26, the vendored class through 0.27.1, `typer.exceptions.Abort` from 0.27.2 (#771) ### Docs @@ -604,7 +678,7 @@ disappearing quietly. - Drop the Status link from the docs sidebar. It was a `navigation.global.anchors` entry, which Mintlify pins above the page tree on every page in every tab — permanent real estate for a link that answers a question almost no reader of a docs page is asking. Support stays, since that one is reached from anywhere in the docs by someone who is already stuck. (#718) -- Drop a duplicated entry and a repeated `### Fixes` heading from this release's own section. The canary-images entry was committed twice — once carrying an unfilled `(#PR)` placeholder and once as `(#705)` — and the section then opened a second `### Fixes` block a few entries after the first. Both were invisible while the changelog was only ever read on GitHub; the release announcement renders straight from these sections, so a duplicated headline and a heading appearing twice were about to show up in a public channel. (#721) +- Drop a duplicated entry and a repeated `### Fixes` heading from this release's own section. The canary-images entry was committed twice — once carrying an unfilled `(#789)` placeholder and once as `(#705)` — and the section then opened a second `### Fixes` block a few entries after the first. Both were invisible while the changelog was only ever read on GitHub; the release announcement renders straight from these sections, so a duplicated headline and a heading appearing twice were about to show up in a public channel. (#721) ## 1.0.1-beta.0 — 2026-08-14 diff --git a/Cargo.lock b/Cargo.lock index f478c6310..2fe668690 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -202,7 +202,7 @@ dependencies = [ [[package]] name = "failproofaid" -version = "1.0.4-beta.0" +version = "1.0.4-beta.4" dependencies = [ "fpai-collect", "fpai-ipc", @@ -265,7 +265,7 @@ dependencies = [ [[package]] name = "fpai-collect" -version = "1.0.4-beta.0" +version = "1.0.4-beta.4" dependencies = [ "notify", "reqwest", @@ -280,7 +280,7 @@ dependencies = [ [[package]] name = "fpai-ipc" -version = "1.0.4-beta.0" +version = "1.0.4-beta.4" dependencies = [ "libc", "proptest", diff --git a/Cargo.toml b/Cargo.toml index a084bf064..91be4953d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "3" members = ["crates/*"] [workspace.package] -version = "1.0.4-beta.0" +version = "1.0.4-beta.4" edition = "2024" license-file = "LICENSE" repository = "https://github.com/FailproofAI/failproofai" diff --git a/__tests__/audit/byte-gate.test.ts b/__tests__/audit/byte-gate.test.ts new file mode 100644 index 000000000..0b9709437 --- /dev/null +++ b/__tests__/audit/byte-gate.test.ts @@ -0,0 +1,101 @@ +// @vitest-environment node +/** + * Admission control for the transcript scan. + * + * The scan's concurrency counts FILES, which is the wrong unit: the reader + * materialises every event of a transcript into JS objects, so a 57 MB JSONL + * becomes several hundred MB of them. On a real 1.15 GB corpus — 2,107 files, + * 7 of them holding 224 MB — eight workers put a quarter of the corpus in + * memory at once and the run printed `Aborted(OOM)` 22 times. + */ +import { describe, it, expect } from "vitest"; + +// The gate is internal, so it is exercised through the behaviour that matters: +// a synthetic scheduler with the same shape, asserting the invariant. +class ByteGate { + private inFlight = 0; + private waiting: Array<() => void> = []; + constructor(private readonly budget: number) {} + async acquire(bytes: number): Promise { + const want = Math.max(0, bytes); + while (this.inFlight > 0 && this.inFlight + want > this.budget) { + await new Promise((r) => this.waiting.push(r)); + } + this.inFlight += want; + } + release(bytes: number): void { + this.inFlight -= Math.max(0, bytes); + if (this.inFlight < 0) this.inFlight = 0; + const w = this.waiting; + this.waiting = []; + for (const wake of w) wake(); + } +} + +async function run(sizes: number[], budget: number, workers: number) { + const gate = new ByteGate(budget); + let inFlight = 0; + let peak = 0; + let concurrentPeak = 0; + let running = 0; + let next = 0; + const worker = async () => { + while (next < sizes.length) { + const size = sizes[next++]; + await gate.acquire(size); + inFlight += size; + running += 1; + peak = Math.max(peak, inFlight); + concurrentPeak = Math.max(concurrentPeak, running); + await new Promise((r) => setTimeout(r, 1)); + inFlight -= size; + running -= 1; + gate.release(size); + } + }; + await Promise.all(Array.from({ length: workers }, worker)); + return { peak, concurrentPeak }; +} + +const MB = 1024 * 1024; + +describe("the memory budget", () => { + it("never lets the big files bunch, however many workers there are", async () => { + // The measured shape: a long tail of small files and a handful of huge ones. + const sizes = [ + ...Array.from({ length: 40 }, () => 300 * 1024), + 57 * MB, 40 * MB, 34 * MB, 27 * MB, 24 * MB, 22 * MB, 20 * MB, + ]; + const { peak } = await run(sizes, 48 * MB, 8); + // Two 57MB files at once is the crash. One plus small change is fine. + expect(peak).toBeLessThanOrEqual(57 * MB + 48 * MB); + expect(peak).toBeLessThan(224 * MB); + }); + + it("admits a file larger than the whole budget rather than deadlocking", async () => { + // Refusing it would make the largest transcript on the machine permanently + // unscannable — precisely the one most likely to hold something. + const { peak } = await run([500 * MB], 48 * MB, 8); + expect(peak).toBe(500 * MB); + }); + + it("still runs the small files in parallel", async () => { + // The budget must not turn a 2,000-file scan into a serial one. + const { concurrentPeak } = await run( + Array.from({ length: 200 }, () => 100 * 1024), + 48 * MB, + 8, + ); + expect(concurrentPeak).toBe(8); + }); + + it("frees weight even when a task fails", async () => { + // Holding a failed task's weight shrinks the budget permanently over a long + // scan, until nothing can be admitted at all. + const gate = new ByteGate(10); + await gate.acquire(10); + gate.release(10); + await expect(gate.acquire(10)).resolves.toBeUndefined(); + }); + +}); diff --git a/__tests__/audit/cache.test.ts b/__tests__/audit/cache.test.ts index 04eabc6e9..91d801782 100644 --- a/__tests__/audit/cache.test.ts +++ b/__tests__/audit/cache.test.ts @@ -13,6 +13,7 @@ import { import { DEFAULT_AUDIT_INTERVAL_DAYS } from "../../src/hooks/fp-config"; import type { TranscriptAuditResult } from "../../src/audit/types"; import { auditCacheDir } from "../../src/hooks/fp-home"; +import { LEAK_SCAN_VERSION } from "../../src/audit/leak-scan"; const TRANSCRIPT_PATH = "/tmp/fake-transcript.jsonl"; const MTIME = 1_700_000_000_000; @@ -74,6 +75,16 @@ describe("per-transcript audit cache", () => { expect(typeof entry.cachedAt).toBe("number"); expect(entry.cachedAt).toBeGreaterThanOrEqual(before); expect(entry.cachedAt).toBeLessThanOrEqual(after); + expect(entry.leakScanVersion).toBe(LEAK_SCAN_VERSION); + }); + + it("rejects entries from an older leak scanner", () => { + writeCachedTranscriptResult(TRANSCRIPT_PATH, MTIME, SIZE, FAKE_RESULT); + const path = cachePathFor(TRANSCRIPT_PATH); + const entry = JSON.parse(readFileSync(path, "utf-8")); + entry.leakScanVersion = LEAK_SCAN_VERSION - 1; + writeFileSync(path, JSON.stringify(entry)); + expect(readCachedTranscriptResult(TRANSCRIPT_PATH, MTIME, SIZE)).toBeNull(); }); it("skips zero-byte transcripts (OpenCode DB-backed sources)", () => { diff --git a/__tests__/audit/claude-adapter-metadata.test.ts b/__tests__/audit/claude-adapter-metadata.test.ts new file mode 100644 index 000000000..dc3b1d53e --- /dev/null +++ b/__tests__/audit/claude-adapter-metadata.test.ts @@ -0,0 +1,39 @@ +// @vitest-environment node +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { listClaudeTranscriptMetadata } from "@/src/audit/cli-adapters/claude"; + +const PARENT = "aaaaaaaa-1111-4111-8111-aaaaaaaaaaaa"; + +describe("Claude audit metadata", () => { + let root: string; + let previous: string | undefined; + + beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "fp-claude-meta-")); + previous = process.env.CLAUDE_PROJECTS_PATH; + process.env.CLAUDE_PROJECTS_PATH = root; + }); + + afterEach(() => { + if (previous === undefined) delete process.env.CLAUDE_PROJECTS_PATH; + else process.env.CLAUDE_PROJECTS_PATH = previous; + rmSync(root, { recursive: true, force: true }); + }); + + it("carries a subagent's host-authored purpose into the audit", async () => { + const dir = join(root, "-tmp-proj", PARENT, "subagents"); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, "agent-a.jsonl"), '{"type":"user"}\n'); + writeFileSync(join(dir, "agent-a.meta.json"), JSON.stringify({ + description: "Hunt secrets at rest in transcripts", + })); + + const found = await listClaudeTranscriptMetadata(); + expect(found).toHaveLength(1); + expect(found[0].sessionDescription).toBe("Hunt secrets at rest in transcripts"); + }); +}); diff --git a/__tests__/audit/desktop-notify.test.ts b/__tests__/audit/desktop-notify.test.ts new file mode 100644 index 000000000..e5901ffa6 --- /dev/null +++ b/__tests__/audit/desktop-notify.test.ts @@ -0,0 +1,328 @@ +// @vitest-environment node +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { createServer, type Server, type Socket } from "node:net"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { notifyDesktop, resolveBusPath } from "@/src/audit/desktop-notify"; + +// A stand-in for the session bus: it speaks the SASL handshake, answers Hello, +// and then either returns an id or an error — enough to exercise every branch +// of the encoder against a real socket rather than a mock of one. +interface FakeBus { + path: string; + server: Server; + /** Every complete METHOD_CALL body the client sent, raw. */ + calls: Buffer[]; + close(): Promise; +} + +function methodReturn(replySerial: number, id: number): Buffer { + const u32 = (n: number) => { const b = Buffer.alloc(4); b.writeUInt32LE(n, 0); return b; }; + const pad8 = (b: Buffer) => { const p = (8 - (b.length % 8)) % 8; return p ? Buffer.concat([b, Buffer.alloc(p)]) : b; }; + // REPLY_SERIAL(5,u) then SIGNATURE(8,g "u") + const f1 = pad8(Buffer.concat([Buffer.from([5, 1, 0x75, 0]), u32(replySerial)])); + const f2 = pad8(Buffer.concat([Buffer.from([8, 1, 0x67, 0]), Buffer.from([1]), Buffer.from("u"), Buffer.from([0])])); + const fields = Buffer.concat([f1, f2]); + const header = Buffer.concat([Buffer.from([0x6c, 2, 0, 1]), u32(4), u32(99), u32(fields.length)]); + return Buffer.concat([pad8(Buffer.concat([header, fields])), u32(id)]); +} + +function errorReply(replySerial: number, name: string): Buffer { + const u32 = (n: number) => { const b = Buffer.alloc(4); b.writeUInt32LE(n, 0); return b; }; + const pad8 = (b: Buffer) => { const p = (8 - (b.length % 8)) % 8; return p ? Buffer.concat([b, Buffer.alloc(p)]) : b; }; + const str = (v: string) => { const body = Buffer.from(v); const o = Buffer.concat([u32(body.length), body, Buffer.from([0])]); const p = (4 - (o.length % 4)) % 4; return p ? Buffer.concat([o, Buffer.alloc(p)]) : o; }; + const f1 = pad8(Buffer.concat([Buffer.from([4, 1, 0x73, 0]), str(name)])); // ERROR_NAME + const f2 = pad8(Buffer.concat([Buffer.from([5, 1, 0x75, 0]), u32(replySerial)])); + const fields = Buffer.concat([f1, f2]); + const header = Buffer.concat([Buffer.from([0x6c, 3, 0, 1]), u32(0), u32(98), u32(fields.length)]); + return Buffer.concat([pad8(Buffer.concat([header, fields]))]); +} + +function startBus( + dir: string, + behaviour: "ok" | "no-server" | "reject-auth" | "silent", + id = 4242, +): Promise { + const path = join(dir, "bus"); + const calls: Buffer[] = []; + const server = createServer((sock: Socket) => { + let authed = false; + let sawHello = false; + sock.on("data", (chunk: Buffer) => { + const text = chunk.toString("latin1"); + if (!authed) { + if (behaviour === "reject-auth") { sock.write("REJECTED EXTERNAL\r\n"); return; } + if (text.includes("AUTH")) { authed = true; sock.write("OK 1234deadbeef\r\n"); return; } + return; + } + if (behaviour === "silent") return; + // BEGIN + Hello may arrive coalesced with the AUTH ack's reply. + if (!sawHello) { sawHello = true; sock.write(methodReturn(1, 0)); return; } + calls.push(Buffer.from(chunk)); + sock.write(behaviour === "no-server" + ? errorReply(2, "org.freedesktop.DBus.Error.ServiceUnknown") + : methodReturn(2, id)); + }); + sock.on("error", () => { /* client hangs up after its answer */ }); + }); + return new Promise((res) => { + server.listen(path, () => res({ + path, server, calls, + close: () => new Promise((r) => server.close(() => r())), + })); + }); +} + +let dir: string; +let bus: FakeBus | null = null; +let bus2: FakeBus | null = null; +beforeEach(() => { dir = mkdtempSync(join(tmpdir(), "fp-dbus-")); }); +afterEach(async () => { await bus?.close(); bus = null; await bus2?.close(); bus2 = null; rmSync(dir, { recursive: true, force: true }); }); + +describe("resolving the bus address", () => { + it("constructs one from the uid when the environment has none", () => { + // The scheduled audit's case: a child of a system service started at boot, + // whose environment carries no DBUS_SESSION_BUS_ADDRESS at all. This is + // exactly why the address is built rather than read. + expect(resolveBusPath({})).toBe(`/run/user/${process.getuid!()}/bus`); + }); + + it("prefers a declared address, because on an unusual setup it is the only right one", () => { + expect(resolveBusPath({ DBUS_SESSION_BUS_ADDRESS: "unix:path=/tmp/odd/bus" })).toBe("/tmp/odd/bus"); + expect(resolveBusPath({ DBUS_SESSION_BUS_ADDRESS: "unix:guid=abc,path=/tmp/g/bus" })).toBe("/tmp/g/bus"); + }); + + it("falls back rather than dialing an address form it cannot open", () => { + // `unix:abstract=` and `tcp:` are legal and unsupported here. Reporting + // "no session" beats connecting somewhere wrong. + for (const addr of ["unix:abstract=/tmp/dbus-xyz", "tcp:host=localhost,port=1", "garbage"]) { + expect(resolveBusPath({ DBUS_SESSION_BUS_ADDRESS: addr })).toBe(`/run/user/${process.getuid!()}/bus`); + } + }); +}); + +describe("posting a notification", () => { + it("completes the handshake and reads back the id the server assigned", async () => { + bus = await startBus(dir, "ok", 77); + const out = await notifyDesktop("failproofai", "1 credential", 0, { socketPath: bus.path }); + expect(out).toEqual({ ok: true, id: 77 }); + }); + + it("sends the arguments the spec asks for, in order", async () => { + bus = await startBus(dir, "ok"); + await notifyDesktop("summary here", "body here", 0, { socketPath: bus!.path }); + const raw = bus!.calls[0].toString("latin1"); + expect(raw).toContain("org.freedesktop.Notifications"); + expect(raw).toContain("Notify"); + expect(raw).toContain("susssasa{sv}i"); // the signature the server type-checks against + expect(raw).toContain("failproofai"); // app_name, which is how the user identifies us + expect(raw).toContain("summary here"); + expect(raw).toContain("body here"); + }); + + it("carries replaces_id so a repeat scan updates one bubble instead of stacking", async () => { + // The difference between a reminder and a nag: this runs on a timer, and + // the same finding recurs until the key is rotated. + bus = await startBus(dir, "ok"); + await notifyDesktop("s", "b", 4242, { socketPath: bus!.path }); + const body = bus!.calls[0]; + expect(body.includes(Buffer.from([0x92, 0x10, 0, 0]))).toBe(true); // 4242, little-endian + }); +}); + +// THE POINT OF THE WHOLE MODULE. A call sent with NO_REPLY_EXPECTED against a +// bus with no notification server returns success and empty output while the +// notification evaporates — which is the exact failure this feature exists to +// prevent: believing the user was told. +describe("every way this fails, it says so", () => { + it("names the case where nothing is drawing notifications", async () => { + bus = await startBus(dir, "no-server"); + const out = await notifyDesktop("s", "b", 0, { socketPath: bus.path }); + expect(out).toMatchObject({ ok: false, reason: "no-server" }); + }); + + it("reports a missing socket as no-session rather than as a failure to notify", async () => { + // Nobody is logged in, so /run/user/ does not exist. Distinguishable + // from "the desktop refused", because the remedy is different. + const out = await notifyDesktop("s", "b", 0, { socketPath: join(dir, "absent") }); + expect(out).toMatchObject({ ok: false, reason: "no-session" }); + }); + + it("reports a refused handshake as refused", async () => { + bus = await startBus(dir, "reject-auth"); + const out = await notifyDesktop("s", "b", 0, { socketPath: bus.path }); + expect(out).toMatchObject({ ok: false, reason: "refused" }); + }); + + it("gives up on a bus that accepts the connection and then says nothing", async () => { + // A hung server must not hold the audit open. The timeout is the only + // branch with no packet to trigger it. + bus = await startBus(dir, "silent"); + const out = await notifyDesktop("s", "b", 0, { socketPath: bus.path }); + expect(out).toMatchObject({ ok: false, reason: "timeout" }); + }, 10_000); + + it("never throws, whatever the socket does", async () => { + await expect(notifyDesktop("s", "b", 0, { socketPath: "/" })).resolves.toMatchObject({ ok: false }); + }); +}); + +// ── Against a REAL bus ─────────────────────────────────────────────────────── +// +// The tests above drive a server written in this same file, which is exactly +// how the first version of this module shipped a broken encoder: the fake +// mirrored the encoder's own assumptions, so both were wrong together and both +// passed. A real `dbus-daemon` rejected the very first message. These tests +// exist so that cannot happen twice. +// +// Skipped, loudly, when dbus-daemon is not installed — it is a real gap in +// coverage, not a pass. +import { spawn, execFileSync, type ChildProcess } from "node:child_process"; +import { writeFileSync, mkdirSync, existsSync } from "node:fs"; + +function haveDbus(): boolean { + try { + execFileSync("dbus-daemon", ["--version"], { stdio: "ignore" }); + return true; + } catch { + return false; + } +} + +const REAL = haveDbus() ? describe : describe.skip; + +REAL("the real dbus-daemon", () => { + // A unix socket path is capped at ~108 bytes, so this cannot live in a long + // temp dir. Private to this test and torn down after; it is never the user's + // session bus, and no notification service is registered on it, so nothing + // can be displayed by anything. + const dir = `/tmp/fpai-t${process.pid}`; + const sock = `${dir}/bus`; + let bus: ChildProcess | null = null; + + beforeEach(async () => { + mkdirSync(dir, { recursive: true }); + const conf = `${dir}/c.conf`; + writeFileSync( + conf, + ` +sessionunix:path=${sock} +`, + ); + bus = spawn("dbus-daemon", [`--config-file=${conf}`, "--nofork"], { stdio: "ignore" }); + for (let i = 0; i < 150 && !existsSync(sock); i += 1) { + await new Promise((r) => setTimeout(r, 20)); + } + }); + + afterEach(() => { + bus?.kill("SIGKILL"); + bus = null; + try { + rmSync(dir, { recursive: true, force: true }); + } catch { /* best effort */ } + }); + + // THE REGRESSION. The first encoder declared a header-fields length two bytes + // too long, because it counted the padding after the final field — padding + // that belongs to the message, not to the array. Real dbus-daemon hung up; + // the hand-written server above did not notice. Reaching a NAMED D-Bus error + // proves the daemon parsed the whole message: auth, Hello, and a Notify whose + // signature and body it type-checked before deciding nobody serves that name. + it("accepts our bytes all the way to a semantic error", async () => { + const out = await notifyDesktop("summary", "body", 0, { socketPath: sock }); + expect(out).toMatchObject({ ok: false, reason: "no-server" }); + expect((out as { detail: string }).detail).toContain("ServiceUnknown"); + }); + + it("survives a body no fake server would have stressed", async () => { + // Multibyte UTF-8 changes byte length independently of character count, and + // every string in the body is length-prefixed in BYTES. + for (const [summary, body] of [ + ["клавиша 🔑", "ghp_••••4f2a — naïve"], + ["", ""], + ["long", "x".repeat(9000)], + ["a\nb", "c\r\nd\te"], + ]) { + const out = await notifyDesktop(summary, body, 0, { socketPath: sock }); + // Still ServiceUnknown, never a parse failure or a dropped connection. + expect(out, `${summary.slice(0, 12)}`).toMatchObject({ ok: false, reason: "no-server" }); + } + }); + + it("reports a bus that goes away as something other than a timeout", async () => { + // A killed bus must not read as "your desktop is slow" — the remedies are + // completely different, and conflating them is what hid the encoder bug. + bus?.kill("SIGKILL"); + await new Promise((r) => setTimeout(r, 100)); + const out = await notifyDesktop("s", "b", 0, { socketPath: sock }); + expect(out.ok).toBe(false); + expect((out as { reason: string }).reason).not.toBe("timeout"); + }); +}); + +// ── Framing ────────────────────────────────────────────────────────────────── +// +// A real bus does not send one message per TCP chunk. It answers Hello with a +// METHOD_RETURN and then emits a NameAcquired SIGNAL, and those can arrive +// glued together or split anywhere. The first version read "the next chunk" as +// its reply, so a signal landing at the wrong moment was reported as a +// DELIVERED notification with a garbage id. +describe("message framing", () => { + function signal(): Buffer { + const u32 = (n: number) => { const b = Buffer.alloc(4); b.writeUInt32LE(n, 0); return b; }; + const pad8 = (b: Buffer) => { const p = (8 - (b.length % 8)) % 8; return p ? Buffer.concat([b, Buffer.alloc(p)]) : b; }; + const str = (v: string) => { const b = Buffer.from(v); const o = Buffer.concat([u32(b.length), b, Buffer.from([0])]); const p = (4 - (o.length % 4)) % 4; return p ? Buffer.concat([o, Buffer.alloc(p)]) : o; }; + const f = (c: number, t: string, v: Buffer) => pad8(Buffer.concat([Buffer.from([c, 1, t.charCodeAt(0), 0]), v])); + const fields = Buffer.concat([f(1, "o", str("/org/freedesktop/DBus")), f(3, "s", str("NameAcquired"))]); + const head = Buffer.concat([Buffer.from([0x6c, 4, 0, 1]), u32(0), u32(77), u32(fields.length)]); + return pad8(Buffer.concat([head, fields])); + } + + function bus(dir: string, plan: "signal-first" | "glued" | "byte-at-a-time"): Promise { + const path = join(dir, "bus2"); + const calls: Buffer[] = []; + const server = createServer((s: Socket) => { + let authed = false; + let helloDone = false; + const send = (b: Buffer) => { + if (plan === "byte-at-a-time") { for (const byte of b) s.write(Buffer.from([byte])); } + else s.write(b); + }; + s.on("data", (chunk: Buffer) => { + if (!authed) { authed = true; s.write("OK 0123456789abcdef0123456789abcdef\r\n"); return; } + if (!helloDone) { + helloDone = true; + // A signal alongside (or before) the Hello reply — what a real bus does. + if (plan === "signal-first") { send(signal()); send(methodReturn(1, 0)); } + else send(Buffer.concat([methodReturn(1, 0), signal()])); + return; + } + calls.push(Buffer.from(chunk)); + send(Buffer.concat([signal(), methodReturn(2, 9001)])); + }); + s.on("error", () => { /* client hangs up after its answer */ }); + }); + return new Promise((res) => server.listen(path, () => res({ + path, server, calls, close: () => new Promise((r) => server.close(() => r())), + }))); + } + + it("ignores a signal instead of reading it as a delivered notification", async () => { + bus2 = await bus(dir, "signal-first"); + expect(await notifyDesktop("s", "b", 0, { socketPath: bus2.path })).toEqual({ ok: true, id: 9001 }); + }); + + it("splits two messages that arrived in one chunk", async () => { + bus2 = await bus(dir, "glued"); + expect(await notifyDesktop("s", "b", 0, { socketPath: bus2.path })).toEqual({ ok: true, id: 9001 }); + }); + + it("reassembles a reply delivered one byte at a time", async () => { + // The pathological split. Every partial-read guard has to hold. + bus2 = await bus(dir, "byte-at-a-time"); + expect(await notifyDesktop("s", "b", 0, { socketPath: bus2.path })).toEqual({ ok: true, id: 9001 }); + }, 10_000); +}); diff --git a/__tests__/audit/harm-report-leaks.test.ts b/__tests__/audit/harm-report-leaks.test.ts new file mode 100644 index 000000000..bff077d59 --- /dev/null +++ b/__tests__/audit/harm-report-leaks.test.ts @@ -0,0 +1,154 @@ +// @vitest-environment node +/** + * What the emailed digest is allowed to say about a leaked credential. + * + * The whole point of the leak record is that it never stores a secret, only a + * fingerprint of one. This file is where that claim is checked against the ONE + * path that leaves the machine. + */ +import { describe, it, expect } from "vitest"; + +import { buildHarmReport, selectLeaks } from "@/src/audit/harm-report"; +import type { LeakFinding } from "@/src/audit/leak-record"; +import { fingerprintSecret } from "@/src/audit/leak-fingerprint"; +import type { AuditResult } from "@/src/audit/types"; + +const SECRET = "ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; + +function finding(over: Partial = {}): LeakFinding { + return { + id: "id-1", + fingerprint: fingerprintSecret(SECRET), + name: "GITHUB_TOKEN", + rule: "sanitize-api-keys", + confidence: "doc-verified", + firstSeen: "2026-09-01T00:00:00.000Z", + lastSeen: "2026-09-05T00:00:00.000Z", + occurrences: 3, + sightings: [ + { + cli: "claude", sessionId: "s0", cwd: "~/…/old", at: "2026-09-01T00:00:00.000Z", + mechanism: { summary: "written to ~/…/.env", toolName: "Write", direction: "input" }, + }, + { + cli: "codex", sessionId: "s1", cwd: "~/…/acme", at: "2026-09-05T00:00:00.000Z", + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "result" }, + }, + ], + ...over, + }; +} + +const WINDOW = { from: new Date("2026-09-01T00:00:00Z"), to: new Date("2026-09-08T00:00:00Z") }; + +function auditResult(): AuditResult { + return { + version: 2, + scannedAt: "2026-09-08T00:00:00.000Z", + scope: { cli: ["claude"], projects: "all", since: null }, + transcripts: { scanned: 1, skipped: 0, errors: 0, durationMs: 0 }, + results: [], + totals: { hits: 0, projectsWithHits: 0 }, + projectsScanned: [], + eventsScanned: 1, + enabledBuiltinNames: [], + }; +} + +// The property that makes this feature shippable at all. Not "the code is +// careful" — the record has no field that could hold a secret, so there is +// nothing here for a bug to send. +describe("what cannot leave the machine", () => { + it("carries no part of the secret, at any window or shape", () => { + const rows = selectLeaks([finding()], WINDOW.from, WINDOW.to); + const wire = JSON.stringify(rows); + expect(wire).not.toContain(SECRET); + // Nor a long enough run of it to be worth anything: the tail is the last 4 + // characters, and the prefix is the vendor's own public marker. + expect(wire).not.toContain(SECRET.slice(4, 24)); + expect(rows[0].display).toContain("•"); + expect(rows[0].display).toContain("ghp_"); + }); + + it("says what class it is and how long, which is what a person acts on", () => { + const [row] = selectLeaks([finding()], WINDOW.from, WINDOW.to); + expect(row.label).toContain("GitHub"); + expect(row.length).toBe(SECRET.length); + expect(row.attributed).toBe(true); + // The identifier name is not a secret, and for a first-party key it is the + // ONLY actionable field — no vendor console exists to revoke it at. + expect(row.name).toBe("GITHUB_TOKEN"); + }); +}); + +describe("the 5W1H a row has to answer", () => { + it("describes the most recent exposure, not the first", () => { + // Where the key is NOW is what matters; where it debuted is history. + const [row] = selectLeaks([finding()], WINDOW.from, WINDOW.to); + expect(row.cli).toBe("codex"); + expect(row.project).toBe("~/…/acme"); + expect(row.mechanism).toBe("read from ~/…/.env"); + expect(row.direction).toBe("result"); + expect(row.first_seen).toBe("2026-09-01T00:00:00.000Z"); + expect(row.last_seen).toBe("2026-09-05T00:00:00.000Z"); + expect(row.occurrences).toBe(3); + }); + + it("survives a finding with no sightings left rather than dropping it", () => { + // Sightings are capped and pruned; the credential is not. A row with a + // vague "how" still tells the user to rotate something. + const [row] = selectLeaks([finding({ sightings: [] })], WINDOW.from, WINDOW.to); + expect(row.display).toContain("ghp_"); + expect(row.mechanism).toBe("seen in a transcript"); + }); +}); + +describe("which findings a window includes", () => { + it("windows on last-seen, so a key still in use keeps being reported", () => { + // Windowing on firstSeen would go quiet on exactly the credentials that are + // still circulating — reporting them once, in the window they debuted. + const old = finding({ id: "old", firstSeen: "2026-01-01T00:00:00.000Z" }); + expect(selectLeaks([old], WINDOW.from, WINDOW.to)).toHaveLength(1); + }); + + it("drops one whose last sighting predates the window", () => { + const stale = finding({ id: "stale", lastSeen: "2026-06-01T00:00:00.000Z" }); + expect(selectLeaks([stale], WINDOW.from, WINDOW.to)).toHaveLength(0); + }); + + it("keeps one with no usable timestamp rather than losing it silently", () => { + const undated = finding({ id: "undated", lastSeen: "not a date" }); + expect(selectLeaks([undated], WINDOW.from, WINDOW.to)).toHaveLength(1); + }); + + it("never mails a finding the user already dismissed", () => { + // They looked at it and said it is not a secret. Mailing it weekly after + // that is how a tool teaches people to filter it out of their inbox. + const dismissed = finding({ id: "d", dismissedAt: "2026-09-06T00:00:00.000Z" }); + expect(selectLeaks([dismissed], WINDOW.from, WINDOW.to)).toHaveLength(0); + }); + + it("puts the most recently seen first", () => { + const a = finding({ id: "a", lastSeen: "2026-09-02T00:00:00.000Z" }); + const b = finding({ id: "b", lastSeen: "2026-09-07T00:00:00.000Z" }); + expect(selectLeaks([a, b], WINDOW.from, WINDOW.to).map((r) => r.id)).toEqual(["b", "a"]); + }); +}); + +describe("the report as a whole", () => { + it("carries leaks alongside the harmful counts, not instead of them", () => { + // Two different claims: `harmful` counts policy activity, `leaks` names a + // specific object to rotate. A digest that replaced one with the other + // would silently stop reporting the half that already worked. + const report = buildHarmReport(auditResult(), undefined, 7, [finding()]); + expect(report.harmful).toEqual([]); + expect(report.leaks).toHaveLength(1); + expect(report.window_to).toBe("2026-09-08T00:00:00.000Z"); + }); + + it("defaults to no leaks when the caller passes none", () => { + // Every existing caller predates this argument, and must keep producing a + // valid report rather than throwing on an undefined list. + expect(buildHarmReport(auditResult(), undefined, 7).leaks).toEqual([]); + }); +}); diff --git a/__tests__/audit/incremental-scan.test.ts b/__tests__/audit/incremental-scan.test.ts index 107231bbd..14e277d91 100644 --- a/__tests__/audit/incremental-scan.test.ts +++ b/__tests__/audit/incremental-scan.test.ts @@ -115,7 +115,14 @@ describe("a transcript that grew between audits", () => { expect(second).toEqual(first); }); - it("carries stateful detectors across the boundary", async () => { + // Skipped, not rewritten: the 8 behavioural detectors are switched off with + // the rest of the old audit (see the header of `src/audit/scoring.ts`), so + // there is no hit left to carry across the resume boundary. The plumbing this + // guards IS still in place — `sessionState` is threaded through `scanOne` and + // returned as `detectorState` exactly as before, deliberately, so restoring + // the detector loop in `src/audit/index.ts` needs no other change and this + // test should pass again the moment it comes back. Un-skip it then. + it.skip("carries stateful detectors across the boundary", async () => { // reread-after-edit pairs an Edit with a later Read of the same path, and // its countdown spans tool calls. Split exactly between the two halves of // that pair: starting the detector empty on resume loses the pairing, and diff --git a/__tests__/audit/index.test.ts b/__tests__/audit/index.test.ts index 9a05c0e2e..82ed98450 100644 --- a/__tests__/audit/index.test.ts +++ b/__tests__/audit/index.test.ts @@ -63,25 +63,35 @@ describe("runAudit() end-to-end on a fixture transcript", () => { rmSync(tmpRoot, { recursive: true, force: true }); }); - it("counts builtin + detector hits across the fixture transcript", async () => { + it("counts builtin policy hits across the fixture transcript", async () => { const result = await runAudit({ clis: ["claude"], noCache: true, noReport: true }); expect(result.transcripts.scanned).toBeGreaterThanOrEqual(1); const names = result.results.map((r) => r.name); // Builtin policy hit. expect(names.some((n) => n.includes("protect-env-vars"))).toBe(true); - // Audit-only detector hits. - expect(names).toContain("redundant-cd-cwd"); - expect(names).toContain("reread-after-edit"); + // The 8 behavioural detectors are switched off with the rest of the old + // audit (see the header of `src/audit/scoring.ts`); the modules and their + // own unit tests in `detectors.test.ts` are untouched. Restore these two + // lines with the detector loop in `src/audit/index.ts`. + // expect(names).toContain("redundant-cd-cwd"); + // expect(names).toContain("reread-after-edit"); + expect(names.every((n) => n !== "redundant-cd-cwd")).toBe(true); }); it("filters by --policy", async () => { + // Filtered on a BUILTIN rather than the detector this used to name, so the + // filter itself stays covered while the detectors are switched off. The + // original read `policies: ["redundant-cd-cwd"]` and expected that name + // back; restore it with the detector loop in `src/audit/index.ts`. const result = await runAudit({ clis: ["claude"], noCache: true, noReport: true, - policies: ["redundant-cd-cwd"], + policies: ["protect-env-vars"], }); - expect(result.results.map((r) => r.name)).toEqual(["redundant-cd-cwd"]); + const names = result.results.map((r) => r.name); + expect(names.length).toBe(1); + expect(names[0]).toContain("protect-env-vars"); }); }); diff --git a/__tests__/audit/leak-containment.test.ts b/__tests__/audit/leak-containment.test.ts new file mode 100644 index 000000000..7e661bf47 --- /dev/null +++ b/__tests__/audit/leak-containment.test.ts @@ -0,0 +1,226 @@ +// @vitest-environment node +/** + * The one property the whole feature rests on: a secret goes in, and no part of + * one comes out anywhere. + * + * Every other test here checks a component. This drives the real pipeline + * end-to-end with real credential shapes and then reads back EVERY byte the run + * wrote to disk, plus the digest that leaves the machine and the notice that + * reaches the terminal, hunting for any fragment of the input. + * + * It is deliberately not a unit test. The claim being made to a user is about + * the system, not about `fingerprintSecret`, and the ways a value escapes are + * integration-shaped: a field added to the record, a new file written beside + * it, a debug string in a notice. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, readdirSync, readFileSync, rmSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { findSecrets } from "@/src/audit/leak-scan"; +import { fingerprintSecret, fingerprintId, isFindingId } from "@/src/audit/leak-fingerprint"; +import { upsertFinding } from "@/src/audit/leak-record"; +import { readLeakRecord, writeLeakRecord, activeFindings } from "@/src/audit/leak-store"; +import { buildHarmReport } from "@/src/audit/harm-report"; +import { markLeakNoticeDelivered } from "@/src/audit/leak-notice"; +import { queueMacNotification } from "@/src/audit/macos-notifier"; +import { leakNoticeText, shapeNotice } from "@/src/hooks/notice"; +import type { AuditResult } from "@/src/audit/types"; + +// Real shapes, synthetic values. One per detection class the scanner claims. +const SECRETS = [ + "ghp_A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8", + "sk-ant-api03-" + "Zq7".repeat(30), + // Assembled rather than written out, and the reason is worth knowing: a + // realistic fixture for a secret scanner is realistic enough to trip OTHER + // secret scanners. This exact value, as one literal, was rejected by GitHub + // push protection ("Push cannot contain secrets") — correctly, since it + // matches Slack's published shape byte for byte. Splitting the literal keeps + // the runtime value identical, so the scanner under test still sees a real + // Slack token, while no line in this file matches a scanner looking at source. + ["xoxb", "9876543210", "9876543210987", "ZaBcDeFgHiJkLmNoPqRsTuVw"].join("-"), + "AKIAIOSFODNN7REALKEY", + "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.dBjftJeZ4CVPmB92K27uhbUJU1p1r8W1gFWFOEjXkFY", +] as const; + +const TRANSCRIPT = [ + `export GITHUB_TOKEN="${SECRETS[0]}"`, + `ANTHROPIC_API_KEY=${SECRETS[1]}`, + `{"slack_bot_token": "${SECRETS[2]}"}`, + `aws_access_key_id = ${SECRETS[3]}`, + `Authorization: Bearer ${SECRETS[4]}`, + "psql postgres://admin:hunter2SuperSecretPassword!@db.internal:5432/prod", +].join("\n"); + +function auditResult(): AuditResult { + return { + version: 2, + scannedAt: "2026-09-08T00:00:00.000Z", + scope: { cli: ["claude"], projects: "all", since: null }, + transcripts: { scanned: 1, skipped: 0, errors: 0, durationMs: 0 }, + results: [], + totals: { hits: 0, projectsWithHits: 0 }, + projectsScanned: [], + eventsScanned: 1, + enabledBuiltinNames: [], + }; +} + +/** Every file the run produced, recursively. */ +function walk(dir: string, out: string[] = []): string[] { + for (const e of readdirSync(dir, { withFileTypes: true })) { + const p = resolve(dir, e.name); + if (e.isDirectory()) walk(p, out); + else out.push(p); + } + return out; +} + +/** + * Any interior 12-character run of a secret counts as a leak. + * + * Not just the whole value: a partial disclosure is still a disclosure, and a + * bug that wrote "the first 20 characters" would pass a whole-string check + * while handing over most of the key. + */ +function fragmentsOf(secret: string): string[] { + const out: string[] = []; + for (let i = 4; i + 12 <= secret.length; i += 7) out.push(secret.slice(i, i + 12)); + return out; +} + +function assertNoSecret(label: string, text: string): void { + for (const [i, secret] of SECRETS.entries()) { + expect(text.includes(secret), `${label} contains secret #${i} in full`).toBe(false); + for (const frag of fragmentsOf(secret)) { + expect(text.includes(frag), `${label} contains a fragment of secret #${i}: ${frag}`).toBe(false); + } + } +} + +let home: string; +let prev: string | undefined; + +beforeEach(() => { + prev = process.env.FAILPROOFAI_HOME; + home = mkdtempSync(join(tmpdir(), "fp-contain-")); + process.env.FAILPROOFAI_HOME = home; +}); +afterEach(() => { + if (prev === undefined) delete process.env.FAILPROOFAI_HOME; + else process.env.FAILPROOFAI_HOME = prev; + rmSync(home, { recursive: true, force: true }); +}); + +/** Run the pipeline the way a real scan does, and hand back what it produced. */ +function runPipeline() { + const found = findSecrets(TRANSCRIPT); + const record = readLeakRecord(home); + for (const f of found) { + upsertFinding(record, { + id: fingerprintId(f.value, record.salt), + fingerprint: fingerprintSecret(f.value, f.rule), + name: f.name, + rule: f.rule, + confidence: "doc-verified", + sighting: { + cli: "claude", + sessionId: "s1", + cwd: "~/…/acme", + at: "2026-09-08T00:00:00.000Z", + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "result" }, + }, + }); + } + writeLeakRecord(record, home); + + const live = activeFindings(readLeakRecord(home), home); + const ids = live.map((f) => f.id); + markLeakNoticeDelivered(ids, home); + markLeakNoticeDelivered(ids, home, "desktop"); + for (const id of ids) queueMacNotification(id, "failproofai", "a credential leaked", home); + + return { + found, + live, + report: buildHarmReport(auditResult(), undefined, 7, live), + notice: shapeNotice("claude", leakNoticeText(found.length)), + }; +} + +describe("a secret goes in", () => { + it("is detected across every class the scanner claims", () => { + const rules = runPipeline().found.map((f) => f.rule); + for (const expected of [ + "GitHub personal access token", + "Anthropic API key", + "Slack token", + "AWS access key ID", + "JWT", + ]) { + expect(rules, expected).toContain(expected); + } + }); +}); + +describe("and no part of one comes out", () => { + it("is absent from every byte the run wrote to disk", () => { + runPipeline(); + const files = walk(home); + // A run that wrote nothing would pass this vacuously. + expect(files.length).toBeGreaterThan(5); + for (const f of files) assertNoSecret(`file ${f.replace(home, "")}`, readFileSync(f, "utf8")); + }); + + it("is absent from the digest, which is the only thing that leaves the machine", () => { + const { report } = runPipeline(); + expect(report.leaks.length).toBeGreaterThan(0); + assertNoSecret("harm report", JSON.stringify(report)); + }); + + it("is absent from the notice that reaches the terminal", () => { + const { notice } = runPipeline(); + assertNoSecret("cli notice", JSON.stringify(notice)); + }); + + it("is absent from the record every surface reads", () => { + runPipeline(); + assertNoSecret("leak record", JSON.stringify(readLeakRecord(home))); + }); + + it("still says enough to act on", () => { + // Containment is worthless if the row says nothing. Each one has to carry a + // recognisable mask, a class, a location and a mechanism. + const { report } = runPipeline(); + const row = report.leaks.find((r) => r.label.includes("GitHub")); + expect(row).toBeDefined(); + expect(row!.display).toContain("ghp_"); + expect(row!.display).toContain("•"); + expect(row!.project).toBe("~/…/acme"); + expect(row!.mechanism).toBe("read from ~/…/.env"); + expect(row!.length).toBe(SECRETS[0].length); + }); +}); + +describe("what it leaves on the filesystem", () => { + it("writes nothing world-readable", () => { + // These files name which credentials this machine leaked. Another account + // on the box learning that is a disclosure by itself, even masked. + runPipeline(); + for (const f of walk(home)) { + expect(statSync(f).mode & 0o077, `${f.replace(home, "")} is group/other readable`).toBe(0); + } + }); + + it("names every file with an id it could have minted", () => { + // Marker and queue filenames are ids. If one is ever not the minted shape, + // something built a path out of unvalidated input. + runPipeline(); + for (const f of walk(home)) { + const name = f.split("/").pop()!; + if (name.endsWith(".json")) continue; + expect(isFindingId(name), `unexpected filename ${name}`).toBe(true); + } + }); +}); diff --git a/__tests__/audit/leak-fingerprint.test.ts b/__tests__/audit/leak-fingerprint.test.ts new file mode 100644 index 000000000..65e84e5f2 --- /dev/null +++ b/__tests__/audit/leak-fingerprint.test.ts @@ -0,0 +1,104 @@ +// @vitest-environment node +import { describe, it, expect } from "vitest"; +import { fingerprintSecret, fingerprintId } from "@/src/audit/leak-fingerprint"; +import { SECRET_PATTERNS } from "@/src/hooks/builtin-policies"; + +/** Obviously-synthetic values. Shapes are real; the bytes are not. */ +const SYNTHETIC = { + githubPat: "ghp_" + "A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8", + awsKeyId: "AKIA" + "IOSFODNN7SYNTHET", + anthropic: "sk-ant-api03-" + "A".repeat(60) + "Zq4T", + slackBot: "xoxb-" + "1111111111-2222222222-SyntheticSlackTok", + password: "hunter2placeh", + firstParty: "compsynthetic0000111122223333", +}; + +describe("fingerprintSecret — identify without disclosing", () => { + it("shows a minted prefix and the last four, which is what a console displays", () => { + const fp = fingerprintSecret(SYNTHETIC.githubPat); + expect(fp.attributed).toBe(true); + expect(fp.label).toBe("GitHub personal access token"); + expect(fp.display.startsWith("ghp_")).toBe(true); + expect(fp.display.endsWith(SYNTHETIC.githubPat.slice(-4))).toBe(true); + expect(fp.length).toBe(SYNTHETIC.githubPat.length); + }); + + it("picks the longest matching prefix, so `sk-ant-api03-` beats `sk-ant-`", () => { + expect(fingerprintSecret(SYNTHETIC.anthropic).label).toBe("Anthropic API key"); + }); + + it("never reveals the middle", () => { + for (const value of Object.values(SYNTHETIC)) { + const { display } = fingerprintSecret(value); + // Every run of 5+ original characters that is not the prefix or the tail + // must be absent from the rendering. + const middle = value.slice(6, -6); + if (middle.length >= 5) { + expect(display, value).not.toContain(middle); + } + } + }); + + // The tail is the actionable half, but only when what stays hidden is + // genuinely unguessable. The corpus's one confirmed-live password was 13 + // characters at 3.19 bits/char — last-4 there is nearly a third of the secret. + it("withholds the tail from a short or unminted value", () => { + const pw = fingerprintSecret(SYNTHETIC.password, "password"); + expect(pw.attributed).toBe(false); + expect(pw.display).toBe("[13-char password]"); + expect(pw.display).not.toContain(SYNTHETIC.password.slice(-4)); + + const firstParty = fingerprintSecret(SYNTHETIC.firstParty, "assigned secret"); + expect(firstParty.attributed).toBe(false); + expect(firstParty.display).not.toContain(SYNTHETIC.firstParty.slice(-4)); + }); + + it("still says how long it was, which is how an owner recognises their own value", () => { + expect(fingerprintSecret("x".repeat(51), "API key").display).toBe("[51-char API key]"); + }); + + // THE ONE THAT MATTERS. `X`, `x` and `0` all satisfy the vendor charsets, so + // masking with them turns a redacted key back into a detectable one: + // `AKIA` + sixteen `X`s matches `AKIA[A-Z0-9]{16}`. This product has already + // measured its own output feeding back into its own corpus, one credential + // becoming seven findings across four sessions. The mask glyph must appear in + // no credential charset anywhere. + it("produces a rendering that our OWN detector cannot mistake for a live key", () => { + for (const [name, value] of Object.entries(SYNTHETIC)) { + const { display } = fingerprintSecret(value); + for (const [pattern, label] of SECRET_PATTERNS) { + const re = new RegExp(pattern.source, pattern.flags.replace("g", "")); + expect(re.test(display), `${name} rendered as "${display}" re-matched ${label}`).toBe(false); + } + } + }); + + it("proves the naive mask characters WOULD have re-matched", () => { + // Guards the reasoning above: if this ever stops being true the mask glyph + // choice is no longer load-bearing and this module's header is stale. + const naive = "AKIA" + "X".repeat(16); + const anyMatch = SECRET_PATTERNS.some(([p]) => + new RegExp(p.source, p.flags.replace("g", "")).test(naive), + ); + expect(anyMatch).toBe(true); + }); +}); + +describe("fingerprintId — stable, and not an oracle", () => { + it("is stable for the same value and salt, so a finding dedupes across scans", () => { + expect(fingerprintId("value-a", "salt-1")).toBe(fingerprintId("value-a", "salt-1")); + }); + + it("differs per value and per machine salt", () => { + expect(fingerprintId("value-a", "salt-1")).not.toBe(fingerprintId("value-b", "salt-1")); + // The salt is what stops the id being a confirmation oracle for a guessable + // secret — the corpus is full of dictionary passwords. + expect(fingerprintId("value-a", "salt-1")).not.toBe(fingerprintId("value-a", "salt-2")); + }); + + it("never contains the value", () => { + const id = fingerprintId("hunter2placeholder", "salt-1"); + expect(id).not.toContain("hunter2"); + expect(id).toMatch(/^[0-9a-f]{16}$/); + }); +}); diff --git a/__tests__/audit/leak-hostile-input.test.ts b/__tests__/audit/leak-hostile-input.test.ts new file mode 100644 index 000000000..b6274c69f --- /dev/null +++ b/__tests__/audit/leak-hostile-input.test.ts @@ -0,0 +1,175 @@ +// @vitest-environment node +/** + * Adversarial input, on the two paths that read attacker-shaped data. + * + * The audit reads transcripts. A transcript is a record of whatever a repository + * made an agent do — file contents, command output, pasted blobs — so every + * string reaching the scanner is, in the strict sense, hostile input from a + * source the user does not control. Two classes of bug live here, and both were + * found by measurement rather than by reading: + * + * 1. Quadratic regexes. A 300 KB unbroken token hung the scan for over 20 + * seconds. Base64 images, minified bundles and whole files arrive as single + * lines constantly; a scheduled scan hitting a few would stall for minutes + * with nobody watching. + * 2. Ids becoming filenames. A finding id is used as a path component, and + * `"../../../../tmp/PWNED"` created that file. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { findSecrets } from "@/src/audit/leak-scan"; +import { redactExample } from "@/src/audit/redact-example"; +import { isFindingId } from "@/src/audit/leak-fingerprint"; +import { markLeakNoticeDelivered, pendingLeakNotice } from "@/src/audit/leak-notice"; +import { queueMacNotification, macNotifyDir } from "@/src/audit/macos-notifier"; + +// Generous on purpose. The point is to catch a return to catastrophic +// backtracking (20_000ms+), not to police normal variation on a loaded CI box. +const BUDGET_MS = 3_000; + +const NL = String.fromCharCode(10); +const TAB = String.fromCharCode(9); + +const BOMBS: ReadonlyArray = [ + // The two that actually hung. A single long assignment value, and a plain + // base64-shaped blob — the single most common large token in a transcript. + ["a long assignment value", "A=" + "a".repeat(200_000)], + ["a 300KB unbroken token", "A".repeat(300_000)], + // The URL shape that took 4.8s inside the redactor's credential matcher. + ["a very long URL", "https://" + "a".repeat(50_000) + "?token=x"], + ["quote flood", '"'.repeat(50_000)], + ["open-quote flood", 'K="'.repeat(20_000)], + ["separator flood", "=".repeat(100_000)], + ["colon flood", ":".repeat(100_000)], + ["spaces before a separator", "K" + " ".repeat(50_000) + "=v"], + ["tab flood", ("K" + TAB).repeat(30_000) + "=v"], + ["many small assignments", Array.from({ length: 20_000 }, (_, i) => `K${i}=v${i}`).join(" ")], + ["vendor-prefix flood", "sk-".repeat(50_000)], + ["newline flood", NL.repeat(200_000)], + ["deep path", "/a".repeat(40_000)], + ["one enormous path segment", "/" + "a".repeat(200_000)], +]; + +describe("pathological input finishes", () => { + for (const [name, input] of BOMBS) { + it(`scans ${name} without backtracking`, () => { + const t0 = performance.now(); + findSecrets(input); + expect(performance.now() - t0).toBeLessThan(BUDGET_MS); + }); + + it(`redacts ${name} without backtracking`, () => { + const t0 = performance.now(); + redactExample(input); + expect(performance.now() - t0).toBeLessThan(BUDGET_MS); + }); + } + + it("stays roughly linear as the input grows", () => { + // The signature of the bug: 4x the input took ~16x the time. Linear-ish + // growth is what a bounded quantifier buys, and it is the property worth + // asserting rather than any single duration. + const time = (n: number) => { + const text = "A".repeat(n); + const t0 = performance.now(); + findSecrets(text); + redactExample(text); + return performance.now() - t0; + }; + time(20_000); // warm up, so JIT does not masquerade as growth + const small = Math.max(time(50_000), 1); + const large = time(200_000); + expect(large / small).toBeLessThan(12); // 4x input; quadratic would be ~16x + }); +}); + +// Correctness must survive the bounds — a faster redactor that stops redacting +// is not a fix. +describe("the bounds did not cost a match", () => { + it("still strips credentials out of a URL", () => { + const out = redactExample("git clone https://alice:hunter2@github.com/acme/x.git"); + expect(out).not.toContain("hunter2"); + expect(out).toContain("[REDACTED"); + }); + + it("still strips them from every scheme it used to", () => { + for (const scheme of ["http", "https", "postgres", "redis", "mongodb+srv", "amqp"]) { + const out = redactExample(`${scheme}://user:s3cr3tpassword@host/db`); + expect(out, scheme).not.toContain("s3cr3tpassword"); + } + }); + + it("still annotates a recognized credential with a normal-length name", () => { + const found = findSecrets('GITHUB_TOKEN="ghp_A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8"'); + expect(found.map((f) => f.name)).toContain("GITHUB_TOKEN"); + }); + + it("ignores an identifier longer than any real one", () => { + // 128 is the bound. Nothing real is near it, and matching past it is what + // made the scan quadratic. + const name = "A".repeat(400); + expect(findSecrets(`${name}_SECRET="abcdefghijklmnopqrstuvwxyz012345"`)).toEqual([]); + }); +}); + +describe("an id is never allowed to be a path", () => { + let home: string; + beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "fp-hostile-")); + process.env.FAILPROOFAI_HOME = home; + }); + afterEach(() => { + delete process.env.FAILPROOFAI_HOME; + rmSync(home, { recursive: true, force: true }); + rmSync("/tmp/fp-should-not-exist", { force: true }); + }); + + const HOSTILE = [ + "../../../../tmp/fp-should-not-exist", + "..", + ".", + "", + "a/b/c", + "/tmp/fp-should-not-exist", + "a b", + "A".repeat(400), + "0123456789ABCDEF", // uppercase: not what fingerprintId mints + "0123456789abcde", // 15 chars + "0123456789abcdefg", // 17 + ]; + + it("refuses every id fingerprintId could not have produced", () => { + for (const id of HOSTILE) expect(isFindingId(id), JSON.stringify(id)).toBe(false); + expect(isFindingId("0123456789abcdef")).toBe(true); + }); + + it("does not write a notice marker outside its directory", () => { + // The measured escape. Refusing also means NOT reporting the id as won, so + // a caller never records a notice it did not actually claim. + for (const id of HOSTILE) expect(markLeakNoticeDelivered([id]), id).toEqual([]); + expect(existsSync("/tmp/fp-should-not-exist")).toBe(false); + }); + + it("does not queue a macOS banner outside its directory", () => { + for (const id of HOSTILE) expect(queueMacNotification(id, "T", "B"), id).toBe(false); + expect(existsSync("/tmp/fp-should-not-exist")).toBe(false); + }); + + it("leaves no staging file behind when a queue write is refused", () => { + // Every rejected attempt used to strand a `notify-*.tmp` beside the watched + // directory: never collected, because the agent only reads inside it. + queueMacNotification("0123456789abcdef", "T", "B"); + for (const id of HOSTILE) queueMacNotification(id, "T", "B"); + const runDirEntries = readdirSync(resolve(macNotifyDir(), "..")); + expect(runDirEntries.filter((n) => n.endsWith(".tmp"))).toEqual([]); + }); + + it("still accepts a real id", () => { + expect(markLeakNoticeDelivered(["0123456789abcdef"])).toEqual(["0123456789abcdef"]); + expect(queueMacNotification("fedcba9876543210", "T", "B")).toBe(true); + expect(pendingLeakNotice().count).toBe(0); + }); +}); diff --git a/__tests__/audit/leak-notice.test.ts b/__tests__/audit/leak-notice.test.ts new file mode 100644 index 000000000..d7e136be5 --- /dev/null +++ b/__tests__/audit/leak-notice.test.ts @@ -0,0 +1,276 @@ +// @vitest-environment node +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { existsSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + pendingLeakNotice, + markLeakNoticeDelivered, + releaseLeakNoticeClaims, + pruneNoticeMarkers, + sessionNoticePending, + markSessionNoticed, +} from "@/src/audit/leak-notice"; +import { + readLeakRecord, + writeLeakRecord, + dismissFinding, + markLeakReportViewed, +} from "@/src/audit/leak-store"; +import { upsertFinding, type LeakSighting } from "@/src/audit/leak-record"; +import { shapeNotice, canDeliverNotice, leakNoticeText, desktopLeakNotice } from "@/src/hooks/notice"; + +let home: string; +beforeEach(() => { home = mkdtempSync(join(tmpdir(), "fp-notice-")); }); +afterEach(() => { rmSync(home, { recursive: true, force: true }); }); + +const FP = { display: "ghp_••••••••4f2a", label: "GitHub token", length: 40, attributed: true }; +const sighting: LeakSighting = { + cli: "claude", sessionId: "s1", cwd: "~/…/acme", at: "2026-09-08T00:00:00Z", + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "input" }, +}; +function seed(...ids: string[]) { + const r = readLeakRecord(home); + for (const id of ids) { + upsertFinding(r, { id, fingerprint: FP, name: "GITHUB_TOKEN", rule: "sanitize-api-keys", + confidence: "doc-verified", sighting }); + } + writeLeakRecord(r, home); +} + +describe("what still owes the user a notice", () => { + it("is every new finding, and nothing once claimed", () => { + seed("0000000000000a01", "0000000000000b02"); + expect(pendingLeakNotice(home).count).toBe(2); + markLeakNoticeDelivered(["0000000000000a01", "0000000000000b02"], home); + expect(pendingLeakNotice(home).count).toBe(0); + }); + + it("stays quiet for a finding the user dismissed", () => { + seed("0000000000000a01", "0000000000000b02"); + dismissFinding("0000000000000a01", home); + expect(pendingLeakNotice(home).ids).toEqual(["0000000000000b02"]); + }); + + it("returns nothing rather than guessing when the record is unreadable", () => { + expect(pendingLeakNotice("/nonexistent/path/xyz").count).toBe(0); + }); +}); + +// THE CLAIM THE DESIGN RESTS ON. A `notifiedAt` field on the record failed 100% +// of the time here: 2 concurrent sessions gave 2 notices, 8 gave 8 — and two +// sessions with DIFFERENT findings lost one mark permanently, so it re-notified +// forever. Running several agents at once in one project is how this tool is +// used, not an edge case. +describe("concurrency", () => { + it("lets exactly one claimant win, however many race", () => { + seed("0000000000000a01"); + const winners = Array.from({ length: 8 }, () => markLeakNoticeDelivered(["0000000000000a01"], home)); + expect(winners.flat()).toEqual(["0000000000000a01"]); + expect(pendingLeakNotice(home).count).toBe(0); + }); + + it("never loses a claim on a DIFFERENT finding", () => { + // The lost-update case: different findings touch different files, so both + // survive. A single shared watermark map could not do this. + seed("0000000000000a01", "0000000000000b02"); + expect(markLeakNoticeDelivered(["0000000000000a01"], home)).toEqual(["0000000000000a01"]); + expect(markLeakNoticeDelivered(["0000000000000b02"], home)).toEqual(["0000000000000b02"]); + expect(pendingLeakNotice(home).count).toBe(0); + }); + + it("reports which ids this caller actually won", () => { + seed("0000000000000a01", "0000000000000b02"); + markLeakNoticeDelivered(["0000000000000a01"], home); + expect(markLeakNoticeDelivered(["0000000000000a01", "0000000000000b02"], home)).toEqual(["0000000000000b02"]); + }); + + it("can release a failed delivery claim so the next attempt retries", () => { + seed("0000000000000a01"); + const won = markLeakNoticeDelivered(["0000000000000a01"], home, "desktop"); + expect(pendingLeakNotice(home, "desktop").count).toBe(0); + releaseLeakNoticeClaims(won, home, "desktop"); + expect(pendingLeakNotice(home, "desktop").ids).toEqual(["0000000000000a01"]); + }); +}); + +describe("marker housekeeping", () => { + it("drops markers for findings that no longer exist", () => { + seed("0000000000000a01"); + markLeakNoticeDelivered(["0000000000000a01", "00000000000f0000"], home); + pruneNoticeMarkers(home); + // "00000000000f0000" is gone; "0000000000000a01" is still a live finding so its claim stands. + expect(pendingLeakNotice(home).count).toBe(0); + const r = readLeakRecord(home); + r.findings = []; + writeLeakRecord(r, home); + pruneNoticeMarkers(home); + expect(pendingLeakNotice(home).count).toBe(0); + }); +}); + +describe("shaping the notice per CLI", () => { + it("uses the channel each host was proven to render", () => { + const text = leakNoticeText(2); + expect(JSON.parse(shapeNotice("claude", text).stdout).systemMessage).toContain("2 possible credential exposures"); + expect(JSON.parse(shapeNotice("codex", text).stdout).systemMessage).toContain("2 possible credential exposures"); + expect(shapeNotice("copilot", text).stderr).toContain("2 possible credential exposures"); + expect(shapeNotice("factory", text).stdout).toContain("2 possible credential exposures"); + for (const cli of ["opencode", "pi"] as const) { + expect(canDeliverNotice(cli)).toBe(true); + expect(JSON.parse(shapeNotice(cli, text).stdout).failproofaiNotice) + .toContain("2 possible credential exposures"); + } + }); + + // Guessing a channel is worse than having none: it produces output that looks + // delivered from our side and reaches nobody, so we would record a leak as + // "notified" that the user never saw. + it("delivers nothing on a CLI with no proven channel", () => { + for (const cli of ["cursor", "devin", "goose", "antigravity", "hermes"] as const) { + expect(canDeliverNotice(cli), cli).toBe(false); + expect(shapeNotice(cli, leakNoticeText(1)), cli).toEqual({ stdout: "", stderr: "" }); + } + }); + + it("merges into an existing verdict instead of emitting a second JSON document", () => { + // Two JSON objects on one stream is a syntax error to every host, and it + // would take the verdict down with the courtesy message. + const verdict = JSON.stringify({ decision: "block", reason: "CI is red" }); + const out = shapeNotice("claude", leakNoticeText(1), verdict); + const parsed = JSON.parse(out.stdout); + expect(parsed.decision).toBe("block"); + expect(parsed.reason).toBe("CI is red"); + expect(parsed.systemMessage).toContain("a possible credential exposure"); + }); + + it("leaves a verdict alone rather than risk destroying it", () => { + const notJson = "{ this is not json"; + expect(shapeNotice("claude", leakNoticeText(1), notJson).stdout).toBe(notJson); + }); + + it("never emits plain stdout over a verdict on factory", () => { + const verdict = JSON.stringify({ decision: "block" }); + expect(shapeNotice("factory", leakNoticeText(1), verdict).stdout).toBe(verdict); + }); +}); + +// A fixed template, with only a number interpolated. The alternative is +// interpolating a finding's example — the verbatim text of a command, which a +// repository controls via a README or an npm script, and which the redactor +// does not sanitise because it masks secrets rather than instructions. +describe("the notice text", () => { + it("interpolates a count and nothing else", () => { + expect(leakNoticeText(1)).toContain("a possible credential exposure"); + expect(leakNoticeText(5)).toContain("5 possible credential exposures"); + expect(leakNoticeText(1)).toContain("failproofai audit"); + }); + + it("keeps the desktop banner factual and focused on the next action", () => { + const notice = desktopLeakNotice(); + expect(notice.title).toBe("failproofai audit needs review"); + expect(notice.body).toContain("Possible credential exposure"); + expect(notice.body).toContain("failproofai audit"); + expect(notice.body).not.toMatch(/email|schedule|500|leaked credential/i); + }); + + it("carries no fingerprint, path, project or command", () => { + const text = leakNoticeText(3); + expect(text).not.toContain("•"); + expect(text).not.toMatch(/[~/]\w/); + expect(text).not.toContain("ghp_"); + }); +}); + +// ── The in-CLI notice's delivery model ─────────────────────────────────────── +// +// The per-finding claim above is right for the desktop banner and WRONG for the +// in-CLI notice, and the difference was learned the expensive way. "We emitted +// it once" is not "it arrived": on a real machine 499 findings were marked +// delivered and nothing was ever shown — once because the notice was attached +// to an event whose channel the host ignores, and once because hooks were +// disabled in the project under test. Neither is detectable from inside, and a +// claimed finding is never retried. +// +// So this channel keys on the USER's action instead of ours. +describe("the in-CLI notice keys on whether the user has looked", () => { + const seedOld = (id: string) => { + const r = readLeakRecord(home); + const past = new Date(Date.now() - 3 * 86_400_000).toISOString(); + upsertFinding(r, { + id, fingerprint: FP, name: "GITHUB_TOKEN", rule: "sanitize-api-keys", + confidence: "doc-verified", sighting: { ...sighting, at: past }, + }); + writeLeakRecord(r, home); + }; + + it("tells one session once, not once per turn", () => { + seedOld("0000000000000a01"); + expect(sessionNoticePending("sess-a", home).count).toBe(1); + expect(markSessionNoticed("sess-a", home)).toBe(true); + expect(sessionNoticePending("sess-a", home).count).toBe(0); + }); + + it("tells the NEXT session too, because the first one may never have shown it", () => { + // The whole point. A dropped notice costs one session's silence rather than + // the alert itself. + seedOld("0000000000000a01"); + markSessionNoticed("sess-a", home); + expect(sessionNoticePending("sess-b", home).count).toBe(1); + }); + + it("goes quiet once the user opens the report", () => { + seedOld("0000000000000a01"); + markLeakReportViewed(home); + expect(sessionNoticePending("sess-c", home).count).toBe(0); + }); + + it("speaks again when something NEW leaks after they looked", () => { + seedOld("0000000000000a01"); + markLeakReportViewed(home); + // A credential first seen after the view is news; one they already reviewed + // is not, however recently the scan tripped over it again. + const r = readLeakRecord(home); + upsertFinding(r, { + id: "0000000000000b02", fingerprint: FP, name: "N", rule: "r", + confidence: "doc-verified", + sighting: { ...sighting, at: new Date(Date.now() + 1000).toISOString() }, + }); + writeLeakRecord(r, home); + expect(sessionNoticePending("sess-d", home).count).toBe(1); + }); + + it("stays silent when a finding is merely seen again", () => { + // lastSeen moving is not news — only firstSeen decides. + seedOld("0000000000000a01"); + markLeakReportViewed(home); + const r = readLeakRecord(home); + upsertFinding(r, { + id: "0000000000000a01", fingerprint: FP, name: "GITHUB_TOKEN", rule: "r", + confidence: "doc-verified", sighting: { ...sighting, at: new Date().toISOString() }, + }); + writeLeakRecord(r, home); + expect(sessionNoticePending("sess-e", home).count).toBe(0); + }); + + it("says nothing at all when there are no findings", () => { + expect(sessionNoticePending("sess-f", home).count).toBe(0); + }); + + it("refuses a session id that could escape the marker directory", () => { + // Same rule as a finding id: it becomes a filename. + seedOld("0000000000000a01"); + for (const bad of ["../../../../tmp/PWNED", "a/b", ""]) { + expect(markSessionNoticed(bad, home), bad).toBe(false); + expect(sessionNoticePending(bad, home).count, bad).toBe(0); + } + expect(existsSync("/tmp/PWNED")).toBe(false); + }); + + it("lets exactly one concurrent turn claim a session", () => { + seedOld("0000000000000a01"); + const wins = Array.from({ length: 8 }, () => markSessionNoticed("sess-race", home)); + expect(wins.filter(Boolean)).toHaveLength(1); + }); +}); diff --git a/__tests__/audit/leak-reconciliation.test.ts b/__tests__/audit/leak-reconciliation.test.ts new file mode 100644 index 000000000..859177a46 --- /dev/null +++ b/__tests__/audit/leak-reconciliation.test.ts @@ -0,0 +1,58 @@ +// @vitest-environment node +import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { persistLeaks } from "@/src/audit"; +import { readLeakRecord, writeLeakRecord } from "@/src/audit/leak-store"; +import { describeMechanism, upsertFinding } from "@/src/audit/leak-record"; + +const GHOST_ID = "0000000000000bad"; + +describe("full-scan leak reconciliation", () => { + let root: string; + let oldFpHome: string | undefined; + + beforeAll(() => { + root = mkdtempSync(join(tmpdir(), "fp-leak-reconcile-")); + oldFpHome = process.env.FAILPROOFAI_HOME; + process.env.FAILPROOFAI_HOME = join(root, "fp-home"); + }); + + beforeEach(() => { + const record = readLeakRecord(); + record.findings = []; + upsertFinding(record, { + id: GHOST_ID, + fingerprint: { display: "ghp_••••••••1234", label: "GitHub token", length: 40, attributed: true }, + name: "GITHUB_TOKEN", + rule: "GitHub personal access token", + confidence: "doc-verified", + sighting: { + cli: "claude", + sessionId: "old", + cwd: "~/…/old", + at: "2026-09-08T00:00:00.000Z", + mechanism: describeMechanism("Read", "result", "~/…/.env"), + }, + }); + writeLeakRecord(record); + }); + + afterAll(() => { + if (oldFpHome === undefined) delete process.env.FAILPROOFAI_HOME; + else process.env.FAILPROOFAI_HOME = oldFpHome; + rmSync(root, { recursive: true, force: true }); + }); + + it("removes findings the current detector no longer sees after a complete scan", () => { + persistLeaks([], true); + expect(readLeakRecord().findings).toEqual([]); + }); + + it("preserves unseen findings when the scan is explicitly scoped", () => { + persistLeaks([], false); + expect(readLeakRecord().findings.map((f) => f.id)).toContain(GHOST_ID); + }); +}); diff --git a/__tests__/audit/leak-record.test.ts b/__tests__/audit/leak-record.test.ts new file mode 100644 index 000000000..1e76e5036 --- /dev/null +++ b/__tests__/audit/leak-record.test.ts @@ -0,0 +1,150 @@ +// @vitest-environment node +import { describe, it, expect } from "vitest"; +import { + upsertFinding, + pruneRecord, + emptyRecord, + describeMechanism, + MAX_SIGHTINGS, + MAX_FINDINGS, + FINDING_TTL_DAYS, + type LeakSighting, +} from "@/src/audit/leak-record"; + +const FP = { display: "ghp_••••••••4f2a", label: "GitHub personal access token", length: 40, attributed: true }; + +function sighting(at: string, sessionId = "s1"): LeakSighting { + return { + cli: "claude", + sessionId, + cwd: "~/…/acme", + at, + mechanism: describeMechanism("Read", "input", "~/…/.env"), + }; +} + +function base(id: string) { + return { id, fingerprint: FP, name: "GITHUB_TOKEN", rule: "sanitize-api-keys", confidence: "doc-verified" as const }; +} + +describe("upsertFinding — the unit is the distinct VALUE", () => { + it("reports a first sighting as new, which is what the notice keys on", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + const { isNew } = upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }); + expect(isNew).toBe(true); + expect(r.findings).toHaveLength(1); + expect(r.findings[0].occurrences).toBe(1); + }); + + // The same key seen again must NOT re-alert. One pasted credential fanned out + // to 11 files and 22 occurrences on the measured corpus, largely because the + // agent's own checkpoint records replay the prompt that carried it. + it("does not report a repeat sighting as new, however many times it recurs", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }); + for (let i = 0; i < 20; i++) { + const { isNew } = upsertFinding(r, { + ...base("a"), + sighting: sighting(`2026-09-02T00:00:${String(i).padStart(2, "0")}Z`, `s${i}`), + }); + expect(isNew).toBe(false); + } + expect(r.findings).toHaveLength(1); + expect(r.findings[0].occurrences).toBe(21); + }); + + it("counts every occurrence but stores only a bounded slice of evidence", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + for (let i = 0; i < 50; i++) { + upsertFinding(r, { + ...base("a"), + sighting: sighting(`2026-09-0${(i % 9) + 1}T00:00:00Z`, `s${i}`), + }); + } + expect(r.findings[0].occurrences).toBe(50); + expect(r.findings[0].sightings.length).toBeLessThanOrEqual(MAX_SIGHTINGS); + }); + + it("tracks first and last seen across out-of-order sightings", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-05T00:00:00Z") }); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z", "s2") }); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-09T00:00:00Z", "s3") }); + expect(r.findings[0].firstSeen).toBe("2026-09-01T00:00:00Z"); + expect(r.findings[0].lastSeen).toBe("2026-09-09T00:00:00Z"); + }); + + it("keeps distinct credentials apart", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + expect(upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }).isNew).toBe(true); + expect(upsertFinding(r, { ...base("b"), sighting: sighting("2026-09-01T00:00:00Z") }).isNew).toBe(true); + expect(r.findings).toHaveLength(2); + }); + + it("never stores the credential itself", () => { + const r = emptyRecord("salt", "2026-09-01T00:00:00Z"); + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-01T00:00:00Z") }); + expect(JSON.stringify(r)).not.toContain("4f2a" + "SECRET"); + // The only rendering of the value is the mask, which carries no middle. + expect(r.findings[0].fingerprint.display).toContain("•"); + }); +}); + +describe("pruneRecord — the file cannot grow without bound", () => { + it("drops findings older than the TTL", () => { + const now = Date.parse("2026-09-08T00:00:00Z"); + const old = new Date(now - (FINDING_TTL_DAYS + 5) * 86_400_000).toISOString(); + const r = emptyRecord("salt", "2026-09-08T00:00:00Z"); + upsertFinding(r, { ...base("old"), sighting: sighting(old) }); + upsertFinding(r, { ...base("new"), sighting: sighting("2026-09-08T00:00:00Z") }); + pruneRecord(r, now); + expect(r.findings.map((f) => f.id)).toEqual(["new"]); + }); + + // Age out BEFORE capping: capping first would let a burst of stale findings + // evict fresh ones, which is the opposite of what either limit is for. + it("caps the count, keeping the most recent", () => { + const now = Date.parse("2026-09-08T00:00:00Z"); + const r = emptyRecord("salt", "2026-09-08T00:00:00Z"); + for (let i = 0; i < MAX_FINDINGS + 50; i++) { + const at = new Date(now - i * 60_000).toISOString(); + upsertFinding(r, { ...base(`id-${i}`), sighting: sighting(at) }); + } + pruneRecord(r, now); + expect(r.findings).toHaveLength(MAX_FINDINGS); + expect(r.findings[0].id).toBe("id-0"); + }); + + it("stays small: a thousand sightings of one key cost the same as one", () => { + const now = Date.parse("2026-09-08T00:00:00Z"); + const r = emptyRecord("salt", "2026-09-08T00:00:00Z"); + for (let i = 0; i < 1000; i++) { + upsertFinding(r, { ...base("a"), sighting: sighting("2026-09-08T00:00:00Z", `s${i}`) }); + } + pruneRecord(r, now); + expect(JSON.stringify(r).length).toBeLessThan(2_000); + expect(r.findings[0].occurrences).toBe(1000); + }); +}); + +describe("describeMechanism — the `how` of 5W1H, specific by construction", () => { + it("names the file and the direction", () => { + expect(describeMechanism("Read", "input", "~/…/.env").summary).toBe("read from ~/…/.env"); + expect(describeMechanism("Write", "input", "~/…/deploy.sh").summary).toBe("written to ~/…/deploy.sh"); + expect(describeMechanism("Edit", "input", "~/…/config.ts").summary).toBe("edited into ~/…/config.ts"); + expect(describeMechanism("Bash", "input", null).summary).toBe("passed in a shell command"); + }); + + // Input vs result is not cosmetic: an input is deniable at PreToolUse on all + // 12 CLIs, a result is not. They are different exposures with different fixes. + it("distinguishes what the agent SENT from what it RECEIVED", () => { + expect(describeMechanism("Read", "input", "~/…/.env").direction).toBe("input"); + const out = describeMechanism("Read", "result", "~/…/.env"); + expect(out.direction).toBe("result"); + expect(out.summary).toContain("output"); + }); + + it("degrades to something still specific for an unknown tool", () => { + expect(describeMechanism("WebFetch", "input", null).summary).toBe("sent to the WebFetch tool"); + }); +}); diff --git a/__tests__/audit/leak-research-suppression.test.ts b/__tests__/audit/leak-research-suppression.test.ts new file mode 100644 index 000000000..020208f93 --- /dev/null +++ b/__tests__/audit/leak-research-suppression.test.ts @@ -0,0 +1,94 @@ +// @vitest-environment node +import { describe, expect, it } from "vitest"; + +import { isCredentialResearchSession, mergeIncremental } from "@/src/audit"; +import type { NormalizedToolEvent, TranscriptAuditResult, TranscriptLeak } from "@/src/audit/types"; + +function event(command: string): NormalizedToolEvent { + return { + cli: "claude", + sessionId: "session-1", + transcriptPath: "/tmp/session.jsonl", + cwd: "/repo", + timestamp: "2026-09-09T00:00:00.000Z", + toolName: "Bash", + rawToolName: "Bash", + toolInput: { command }, + }; +} + +function result(over: Partial = {}): TranscriptAuditResult { + return { + transcriptPath: "/tmp/session.jsonl", + cli: "claude", + projectName: "repo", + sessionId: "session-1", + mtimeMs: 1, + sizeBytes: 10, + cwd: "/repo", + eventsScanned: 1, + hitsByName: {}, + examplesByName: {}, + rangeByName: {}, + ...over, + }; +} + +const LEAK: TranscriptLeak = { + id: "1111111111111111", + fingerprint: { display: "ghp_••••••••4f2a", label: "GitHub token", length: 40, attributed: true }, + name: "GITHUB_TOKEN", + rule: "GitHub personal access token", + shaped: true, + timestamp: "2026-09-09T00:00:00.000Z", + cwd: "/repo", + toolName: "Bash", + direction: "input", + path: null, +}; + +describe("credential research session suppression", () => { + it("requires multiple independent signs of deliberate detector work", () => { + expect(isCredentialResearchSession([ + event("rg SECRET_PATTERNS src/audit/leak-scan.ts"), + event("bun test secret scanner with synthetic credential fixtures"), + ])).toBe(true); + }); + + it("recognizes repeated command-line hunting even without scanner source names", () => { + expect(isCredentialResearchSession([ + event("grep -lF token *.json"), + event("rg 'secret|credential' corpus/"), + event("python scan.py --search-token-shapes"), + ])).toBe(true); + }); + + it("uses a host-authored subagent purpose when the tool commands are opaque", () => { + expect(isCredentialResearchSession( + [event("python3 /tmp/job.py")], + "Hunt secrets at rest in transcripts", + )).toBe(true); + }); + + it("does not suppress ordinary sessions that merely mention credentials", () => { + expect(isCredentialResearchSession([ + event("rotate the leaked credential in production"), + event("git status --short"), + ])).toBe(false); + }); + + it("clears cached leaks when a resumed tail proves the session is research", () => { + const merged = mergeIncremental( + result({ leaks: [LEAK] }), + result({ leakScanSuppressed: "credential-research", leaks: [] }), + ); + expect(merged.leakScanSuppressed).toBe("credential-research"); + expect(merged.leaks).toEqual([]); + }); + + it("retains leaks from both halves of an ordinary resumed session", () => { + const second = { ...LEAK, id: "2222222222222222" }; + expect(mergeIncremental(result({ leaks: [LEAK] }), result({ leaks: [second] })).leaks) + .toEqual([LEAK, second]); + }); +}); diff --git a/__tests__/audit/leak-scan.test.ts b/__tests__/audit/leak-scan.test.ts new file mode 100644 index 000000000..75fc31829 --- /dev/null +++ b/__tests__/audit/leak-scan.test.ts @@ -0,0 +1,240 @@ +// @vitest-environment node +import { describe, it, expect } from "vitest"; +import { findSecrets, flattenToolInput } from "@/src/audit/leak-scan"; +import { redactExample } from "@/src/audit/redact-example"; + +/** Obviously synthetic. Real shapes, invented bytes. */ +const GH = "ghp_" + "A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8"; +const FIRST_PARTY = "compsynthetic0000111122223333"; + +describe("findSecrets — vendor shapes", () => { + it("finds a vendor-shaped key with no name anywhere near it", () => { + // 55.9% of real vendor-shaped credentials in the corpus have no secret-ish + // word within 60 characters, so the shape layer has to stand alone. + const found = findSecrets(`curl -H "Authorization: Bearer ${GH}" https://api.example.com`); + expect(found.some((f) => f.value === GH && f.shaped)).toBe(true); + }); + + it("reports one finding per distinct value, not per matching rule", () => { + const found = findSecrets(`${GH} ${GH} ${GH}`); + expect(found.filter((f) => f.value === GH)).toHaveLength(1); + }); +}); + +describe("findSecrets — named assignments", () => { + it("does not turn a secret-looking variable name into a finding", () => { + for (const line of [ + `MY_API_KEY=${FIRST_PARTY}`, + `MY_API_KEY = "${FIRST_PARTY}"`, + `MY_API_KEY: "${FIRST_PARTY}"`, + `{"my_api_key": "${FIRST_PARTY}"}`, + ]) { + expect(findSecrets(line), line).toEqual([]); + } + }); + + it("keeps the vendor label when a value is both shaped and named", () => { + // The vendor rule can name a console; the identifier says which of the + // user's own variables to change. Both are wanted, on one finding. + const found = findSecrets(`GITHUB_TOKEN=${GH}`); + expect(found).toHaveLength(1); + expect(found[0].shaped).toBe(true); + expect(found[0].name).toBe("GITHUB_TOKEN"); + }); +}); + +describe("findSecrets — what is structurally not a credential", () => { + // Never entropy: the corpus's confirmed-live password measures 3.19 bits per + // character while 1.3M UUIDs in the same corpus sit at 3.72. Any threshold + // that catches the password catches every UUID. + it("ignores indirection, which is the CORRECT secure form", () => { + for (const line of [ + "API_KEY=$OPENAI_API_KEY", + "API_KEY=${OPENAI_API_KEY}", + "TOKEN = process.env.DISCORD_BOT_TOKEN", + "api_key: ", + "API_KEY={{ secrets.THING }}", + ]) { + expect(findSecrets(line), line).toEqual([]); + } + }); + + it("ignores placeholders, booleans, numbers and paths", () => { + for (const line of [ + "API_KEY=xxxxxxxxxxxx", + "API_KEY=000000000000", + "AUTH_ENABLED=true", + "SESSION_KEY=1234567890", + "KEY_PATH=/etc/ssl/private/key.pem", + "API_KEY=short", + ]) { + expect(findSecrets(line), line).toEqual([]); + } + }); + + it("ignores an already-masked value, so our own output is not re-detected", () => { + // Measured: one leaked credential became seven reported findings across + // four sessions, because the audit's own output re-entered the corpus. + expect(findSecrets("API_KEY=[REDACTED: assigned secret]")).toEqual([]); + expect(findSecrets("API_KEY=ghp_••••••••4f2a")).toEqual([]); + }); + + it("does not treat a publishable key as a secret", () => { + expect(findSecrets(`NEXT_PUBLIC_POSTHOG_KEY=${FIRST_PARTY}`)).toEqual([]); + }); +}); + +// THE ONE-DIRECTIONAL PROPERTY. The detector may be narrower than the redactor; +// it must never be wider. A value the report can NAME but the redactor cannot +// MASK is a leak inside the leak report. +describe("everything the detector finds, the redactor can mask", () => { + it("holds for every recognized credential shape", () => { + for (const line of [ + `GITHUB_TOKEN=${GH}`, + `ANTHROPIC_API_KEY=sk-ant-api03-${"A".repeat(40)}`, + `AWS_ACCESS_KEY_ID=AKIA${"3QXZ7YTNBVCD2WLM"}`, + ]) { + const found = findSecrets(line); + expect(found.length, line).toBeGreaterThan(0); + const masked = redactExample(line); + for (const f of found) { + expect(masked, `detector reported ${f.rule} in "${line}" that the redactor left intact`) + .not.toContain(f.value); + } + } + }); +}); + +describe("flattenToolInput", () => { + it("reaches every string leaf, not just `command`", () => { + const flat = flattenToolInput({ + file_path: "/tmp/deploy.sh", + content: `export GITHUB_TOKEN=${GH}`, + nested: { deeper: [{ more: "x" }] }, + }); + expect(findSecrets(flat).some((f) => f.value === GH)).toBe(true); + }); + + it("keeps a key adjacent to a shaped value so its name can be attached", () => { + const flat = flattenToolInput({ api_key: GH }); + expect(findSecrets(flat).map((f) => f.name)).toContain("api_key"); + }); + + it("is bounded, so a pathological payload cannot recurse forever", () => { + let deep: unknown = "x"; + for (let i = 0; i < 50; i++) deep = { n: deep }; + expect(() => flattenToolInput(deep)).not.toThrow(); + }); +}); + +// The single highest-value refuter, and the only decisive one. Measured on the +// real corpus: 456 of 457 `AKIA` matches were AWS's own documentation literal — +// 99.8% of that pattern's entire output, 232 findings collapsing to zero. +describe("the docs-literal denylist", () => { + it("drops a credential the vendor published on purpose", () => { + // Assembled from parts so this test file does not itself contain the + // literal — a denylist written in plaintext is a file our scanner flags. + const awsDocsKey = "AKIA" + "IOSFODNN7EXAMPLE"; + expect(findSecrets(`AWS_ACCESS_KEY_ID=${awsDocsKey}`)).toEqual([]); + expect(findSecrets(`export ${awsDocsKey}`)).toEqual([]); + }); + + it("still reports a real key of the same shape", () => { + const realShape = "AKIA" + "3QXZ7YTNBVCD2WLM"; + expect(findSecrets(`AWS_ACCESS_KEY_ID=${realShape}`).length).toBeGreaterThan(0); + }); + + it("cannot false-positive — it is a hash comparison, not a heuristic", () => { + const almost = "AKIA" + "IOSFODNN7EXAMPLF"; // one byte different + expect(findSecrets(`AWS_ACCESS_KEY_ID=${almost}`).length).toBeGreaterThan(0); + }); +}); + +// Auditing for secrets writes secrets into the corpus the next audit reads. +// The largest organic cluster in 1.6GB of transcripts was one file: a previous +// audit quoting back what it had found. 108 of 161 firing patterns fired ONLY +// inside the investigation's own transcripts. +describe("self-exclusion", () => { + it("ignores our own report replayed back into a transcript", () => { + expect(findSecrets(`GITHUB_TOKEN=[REDACTED: assigned secret]`)).toEqual([]); + expect( + findSecrets(`failproofai audit found: GITHUB_TOKEN=${GH}`), + "our own report quoting a key back is not a new leak", + ).toEqual([]); + }); + + it("does not mistake ordinary work for our output", () => { + expect(findSecrets(`GITHUB_TOKEN=${GH}`).length).toBeGreaterThan(0); + }); +}); + +// The prefilter is what makes the pattern set shippable: cost is LINEAR in +// pattern count (~0.095s per pattern per 6MB), so 288 patterns over 1.6GB is +// about two hours while the few that ever match take five seconds. It must buy +// that speed without changing a single answer. +describe("the literal prefilter", () => { + it("finds exactly what an ungated scan would", () => { + // Every shape in this file's battery, run through the gated path. If the + // gate ever drops a pattern's prefix these go quiet — which is the failure + // mode that looks identical to "no secrets here". + const cases: [string, boolean][] = [ + [`GITHUB_TOKEN=${GH}`, true], + [`export ANTHROPIC_API_KEY=sk-ant-api03-${"A".repeat(40)}`, true], + [`AWS_ACCESS_KEY_ID=AKIA${"3QXZ7YTNBVCD2WLM"}`, true], + [`SLACK=xoxb-${"1111111111-2222222222-abcdefghijklmnopqrstuvwx"}`, true], + [`TELEGRAM_BOT_TOKEN=1234567890:${"A".repeat(35)}`, true], + [`COMPOSIO_API_KEY=${FIRST_PARTY}`, false], + ["git commit -m 'nothing to see'", false], + ["const x = 1; // ordinary source", false], + ]; + for (const [text, shouldFind] of cases) { + expect(findSecrets(text).length > 0, text).toBe(shouldFind); + } + }); + + it("still matches a pattern that has no literal prefix to gate on", () => { + // A connection string starts with a scheme alternation, so it yields no + // gate token and must always be run rather than silently skipped. + const found = findSecrets("psql postgresql://admin:hunter2placeholder@db.internal:5432/prod"); + expect(found.length).toBeGreaterThan(0); + }); + + it("is cheap on text that contains no credential at all", () => { + // The common case by an enormous margin: most transcript bytes are prose + // and source. 1MB of it must not cost 33 full regex passes. + const haystack = "lorem ipsum dolor sit amet ".repeat(40_000); + const started = Date.now(); + findSecrets(haystack); + expect(Date.now() - started).toBeLessThan(200); + }); +}); + +// Name-only matching produced hundreds of ordinary code literals and synthetic +// scanner fixtures on the measured machine. Names now annotate only values a +// recognizable credential shape already established. +describe("names never create findings", () => { + const found = (text: string) => findSecrets(text).length > 0; + + it("drops both explicit secret names and ambiguous programming vocabulary", () => { + for (const text of [ + "DB_PASSWORD=hunter2secret", + 'apiKey: "Zk7Qw2Lm9Xr4Tp8Vb1Nc6Hs3"', + 'authToken="Zq7Kp2Lm9Xr4Tv8Nb1Hc6Ws3Ee5"', + "keyType=primary", + "tokenLimitCancelled=false", + "max_output_tokens=4096", + "resultKey=someLongCamelCaseFieldName", + "configDirKey=user-config-directory-path", + "sig=abcdefgh", + "tokens=1024", + "keyType=standardIssueValueHere", + ]) { + expect(found(text), text).toBe(false); + } + }); + + it("still attaches a name to a recognizable vendor key", () => { + expect(found("token=ghp_A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8")).toBe(true); + expect(found("keyType=" + "sk-ant-api03-" + "Zq7".repeat(30))).toBe(true); + }); +}); diff --git a/__tests__/audit/leak-section.test.tsx b/__tests__/audit/leak-section.test.tsx new file mode 100644 index 000000000..de2a20bfe --- /dev/null +++ b/__tests__/audit/leak-section.test.tsx @@ -0,0 +1,195 @@ +// @vitest-environment jsdom +/** + * The report the score was replaced with. + * + * What is pinned here is the part a reader acts on: one row per distinct + * credential (not per sighting), the masked value and never the value, the + * mechanism sentence, and the difference between a key with a console to revoke + * at and one without — which is the difference between a minute's work and an + * investigation. + */ +import { describe, it, expect, vi, beforeEach } from "vitest"; +import { render, screen, waitFor } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; + +import type { LeakRow } from "@/app/actions/get-leaks"; + +const h = vi.hoisted(() => ({ + getLeaksAction: vi.fn(), + dismissLeakAction: vi.fn(), + setLeakNotifyAction: vi.fn(), +})); +vi.mock("@/app/actions/get-leaks", () => ({ + getLeaksAction: h.getLeaksAction, + dismissLeakAction: h.dismissLeakAction, + setLeakNotifyAction: h.setLeakNotifyAction, +})); + +import { LeakSection } from "@/app/audit/_components/leak-section"; + +function row(over: Partial = {}): LeakRow { + return { + id: "id-1", + display: "ghp_••••••••4f2a", + label: "GitHub personal access token", + length: 40, + attributed: true, + name: "GITHUB_TOKEN", + cli: "claude", + project: "~/…/acme", + lastSeen: new Date(Date.now() - 86_400_000).toISOString(), + firstSeen: "2026-09-01T00:00:00.000Z", + mechanism: "read from ~/…/.env", + direction: "result", + occurrences: 7, + sessions: 3, + ...over, + }; +} + +beforeEach(() => { + vi.clearAllMocks(); + h.dismissLeakAction.mockResolvedValue(true); + h.setLeakNotifyAction.mockResolvedValue(true); +}); + +describe("what a row shows", () => { + it("shows the mask and never anything else", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + expect(await screen.findByText("ghp_••••••••4f2a")).toBeTruthy(); + // The identifier name is not a secret and is often the only actionable + // field, so it is shown alongside. + expect(screen.getByText("GITHUB_TOKEN")).toBeTruthy(); + }); + + it("answers where, who, how and when in one sentence", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + expect(screen.getByText("read from ~/…/.env")).toBeTruthy(); + expect(screen.getByText("~/…/acme")).toBeTruthy(); + expect(screen.getByText("claude")).toBeTruthy(); + expect(screen.getByText("yesterday")).toBeTruthy(); + }); + + it("counts exposures without implying that many keys", async () => { + // A key pasted into forty commands is one thing to rotate. Showing forty + // rows would be true about sightings and wrong about the work. + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + expect(await screen.findByText(/7 exposures · 3 sessions/)).toBeTruthy(); + expect(screen.getByText(/1 credential in your transcripts/)).toBeTruthy(); + }); + + it("says the key can be blocked next time only when it can", async () => { + // A result came back INTO the transcript from a file or a command — no + // PreToolUse hook can intercept that, and saying otherwise would be advice + // that cannot be followed. + h.getLeaksAction.mockResolvedValue({ + rows: [row({ id: "a", direction: "input" }), row({ id: "b", direction: "result" })], + notify: true, + }); + render(); + expect(await screen.findByText(/blockable at PreToolUse/)).toBeTruthy(); + expect(screen.getByText(/already in the transcript/)).toBeTruthy(); + }); +}); + +describe("what to do about it", () => { + it("sends an attributed key to its own console", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + expect(await screen.findByText(/rotate it in the GitHub personal access console/)).toBeTruthy(); + }); + + it("gives an unattributed key the only advice that exists for it", async () => { + // The majority case per the pattern census: no vendor, no console, and the + // identifier name is the only clue to who issued it. + h.getLeaksAction.mockResolvedValue({ + rows: [row({ attributed: false, display: "[32-char password]", name: "COMPOSIO_API_KEY" })], + notify: true, + }); + render(); + expect(await screen.findByText(/find what reads COMPOSIO_API_KEY/)).toBeTruthy(); + }); + + it("falls back to tracing when there is not even a name", async () => { + h.getLeaksAction.mockResolvedValue({ + rows: [row({ attributed: false, name: null, display: "[19-char password]" })], + notify: true, + }); + render(); + expect(await screen.findByText(/trace it to its owner/)).toBeTruthy(); + }); +}); + +describe("dismissing", () => { + it("removes the row only once the dismissal actually persisted", async () => { + // Optimism would leave the user guessing which of two states is real when + // the row reappears on the next load. + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: "not a secret" })); + + await waitFor(() => expect(screen.queryByText("ghp_••••••••4f2a")).toBeNull()); + expect(h.dismissLeakAction).toHaveBeenCalledWith("id-1"); + }); + + it("keeps the row when the dismissal failed", async () => { + h.dismissLeakAction.mockResolvedValue(false); + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: "not a secret" })); + + await waitFor(() => expect(h.dismissLeakAction).toHaveBeenCalled()); + expect(screen.getByText("ghp_••••••••4f2a")).toBeTruthy(); + }); +}); + +describe("the notification switch", () => { + it("writes the same setting the CLI and the daemon read", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: /turn desktop notifications off/ })); + + expect(h.setLeakNotifyAction).toHaveBeenCalledWith(false); + }); + + it("puts the switch back when the write failed", async () => { + // A toggle that shows "off" over a config that still says on is worse than + // one that refuses to move: the user believes they silenced it. + h.setLeakNotifyAction.mockResolvedValue(false); + h.getLeaksAction.mockResolvedValue({ rows: [row()], notify: true }); + render(); + await screen.findByText("ghp_••••••••4f2a"); + + await userEvent.click(screen.getByRole("button", { name: /turn desktop notifications off/ })); + + await waitFor(() => + expect(screen.getByRole("button", { name: /turn desktop notifications off/ })).toBeTruthy(), + ); + }); +}); + +describe("nothing found", () => { + it("says so plainly rather than rendering an empty list", async () => { + h.getLeaksAction.mockResolvedValue({ rows: [], notify: true }); + render(); + expect(await screen.findByText("no credentials found")).toBeTruthy(); + }); + + it("draws nothing at all until the record has been read", () => { + // A "no credentials found" flash before the data arrives is a false + // all-clear, which is the one wrong thing this section can say. + h.getLeaksAction.mockReturnValue(new Promise(() => {})); + const { container } = render(); + expect(container.textContent).toBe(""); + }); +}); diff --git a/__tests__/audit/leak-store.test.ts b/__tests__/audit/leak-store.test.ts new file mode 100644 index 000000000..af9f39531 --- /dev/null +++ b/__tests__/audit/leak-store.test.ts @@ -0,0 +1,197 @@ +// @vitest-environment node +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync, writeFileSync, mkdirSync, statSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, dirname } from "node:path"; + +import { + readLeakIdentity, + readLeakRecord, + writeLeakRecord, + dismissFinding, + activeFindings, +} from "@/src/audit/leak-store"; +import { upsertFinding, type LeakSighting } from "@/src/audit/leak-record"; +import { auditLeaksFile, auditLeakIdentityFile } from "@/src/hooks/fp-home"; +import { HOME_CLASSES } from "@/src/hooks/fp-home"; + +let home: string; +beforeEach(() => { home = mkdtempSync(join(tmpdir(), "fp-leak-store-")); }); +afterEach(() => { rmSync(home, { recursive: true, force: true }); }); + +const FP = { display: "ghp_••••••••4f2a", label: "GitHub token", length: 40, attributed: true }; +const sighting = (at: string): LeakSighting => ({ + cli: "claude", sessionId: "s1", cwd: "~/…/acme", at, + mechanism: { summary: "read from ~/…/.env", toolName: "Read", direction: "input" }, +}); +// Ids are what `fingerprintId` mints — 16 lowercase hex — and `readLeakRecord` +// now drops anything else, because an id becomes a FILENAME. The short labels +// below stay readable at the call sites and are padded into real ids here. +const idFor = (label: string) => label.padEnd(16, "0"); +const finding = (label: string) => ({ + id: idFor(label), fingerprint: FP, name: "GITHUB_TOKEN", rule: "sanitize-api-keys", + confidence: "doc-verified" as const, sighting: sighting("2026-09-08T00:00:00Z"), +}); + +describe("the salt", () => { + it("is minted once and reused, so ids stay stable across scans", () => { + const a = readLeakIdentity(home); + const b = readLeakIdentity(home); + expect(a.salt).toBe(b.salt); + expect(a.salt.length).toBeGreaterThanOrEqual(64); + }); + + it("is random per machine, not derived from anything guessable", () => { + const other = mkdtempSync(join(tmpdir(), "fp-leak-store-b-")); + try { + expect(readLeakIdentity(home).salt).not.toBe(readLeakIdentity(other).salt); + } finally { + rmSync(other, { recursive: true, force: true }); + } + }); + + it("is written 0600 — it is a machine secret", () => { + readLeakIdentity(home); + expect(statSync(auditLeakIdentityFile(home)).mode & 0o777).toBe(0o600); + }); +}); + +describe("round-trip", () => { + it("persists and reloads findings", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + expect(writeLeakRecord(r, home)).toBe(true); + const back = readLeakRecord(home); + expect(back.findings).toHaveLength(1); + expect(back.findings[0].fingerprint.display).toBe(FP.display); + }); + + it("writes the record 0600 too — it maps which project leaked what", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + writeLeakRecord(r, home); + expect(statSync(auditLeaksFile(home)).mode & 0o777).toBe(0o600); + }); + + it("never stores a raw credential", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + writeLeakRecord(r, home); + const raw = readFileSync(auditLeaksFile(home), "utf8"); + expect(raw).toContain("•"); + expect(raw).not.toMatch(/ghp_[A-Za-z0-9]{20,}/); + }); +}); + +// A scan must not fail because its own bookkeeping is damaged. The failure mode +// that matters least must never cause the one that matters most. +describe("never throws", () => { + it("treats a corrupt record as absent", () => { + mkdirSync(dirname(auditLeaksFile(home)), { recursive: true }); + writeFileSync(auditLeaksFile(home), "{ not json at all"); + expect(() => readLeakRecord(home)).not.toThrow(); + expect(readLeakRecord(home).findings).toEqual([]); + }); + + it("discards a record from a newer schema rather than guessing at it", () => { + mkdirSync(dirname(auditLeaksFile(home)), { recursive: true }); + writeFileSync( + auditLeaksFile(home), + JSON.stringify({ schemaVersion: 999, salt: "x", updatedAt: "", findings: [finding("a")] }), + ); + expect(readLeakRecord(home).findings).toEqual([]); + }); + + it("treats a corrupt identity file as absent and mints a new salt", () => { + mkdirSync(dirname(auditLeakIdentityFile(home)), { recursive: true }); + writeFileSync(auditLeakIdentityFile(home), "garbage"); + expect(() => readLeakIdentity(home)).not.toThrow(); + expect(readLeakIdentity(home).salt.length).toBeGreaterThanOrEqual(64); + }); +}); + +describe("dismissal", () => { + it("hides a finding without destroying the evidence", () => { + const r = readLeakRecord(home); + upsertFinding(r, finding("a")); + upsertFinding(r, finding("b")); + writeLeakRecord(r, home); + dismissFinding(idFor("a"), home); + + const back = readLeakRecord(home); + // Still on disk — a mis-click costs a row in a list, not the fact of a leak. + expect(back.findings).toHaveLength(2); + expect(activeFindings(back, home).map((f) => f.id)).toEqual([idFor("b")]); + }); + + it("is idempotent", () => { + dismissFinding(idFor("a"), home); + dismissFinding(idFor("a"), home); + expect(readLeakIdentity(home).dismissed).toEqual([idFor("a")]); + }); +}); + +// This is the claim that justifies two files instead of one. If either half +// were classified `derived`, a reset would silently re-alert every credential +// the machine has ever seen. +describe("reset semantics — why the split exists", () => { + it("classifies the findings as derived and the identity as identity", () => { + const cls = (fn: (h?: string) => string) => + HOME_CLASSES.find((e) => e.path(home) === fn(home))?.class; + expect(cls(auditLeaksFile)).toBe("derived"); + expect(cls(auditLeakIdentityFile)).toBe("identity"); + }); +}); + +// `leaks.json` is a file on disk: a full disk can truncate it mid-write, a hand +// can edit it, and a newer build can write a shape this one does not expect. +// Before this gate existed, one malformed entry threw out of `buildHarmReport` +// — which sits outside `reportHarm`'s try — so a scan that succeeded and cached +// correctly still exited 1, and kept doing so every run. +describe("a corrupt record", () => { + const write = (findings: unknown[]) => + writeFileSync( + auditLeaksFile(home), + JSON.stringify({ schemaVersion: 1, salt: "x", updatedAt: "", findings }), + ); + + beforeEach(() => { + // Create the directory before writing the file by hand. + mkdirSync(dirname(auditLeaksFile(home)), { recursive: true }); + }); + + it("drops entries it cannot render, and keeps the ones it can", () => { + write([ + null, + "a string", + { id: "0123456789abcdef" }, // no fingerprint + { id: "not-an-id", fingerprint: FP }, // id could not be minted + { id: "abc0000000000000", fingerprint: FP, sightings: [], firstSeen: "", lastSeen: "", occurrences: 1 }, + ]); + const kept = readLeakRecord(home).findings; + expect(kept.map((f) => f.id)).toEqual(["abc0000000000000"]); + }); + + it("keeps a finding whose sightings are unusable, and drops just those", () => { + // The credential is the thing that needs rotating. "seen in a transcript" + // with no detail beats silence about a leaked key. + write([ + { + id: "abc0000000000000", fingerprint: FP, name: null, rule: "r", + confidence: "doc-verified", firstSeen: "", lastSeen: "", occurrences: 3, + sightings: [null, { mechanism: null }, { nope: true }], + }, + ]); + const [kept] = readLeakRecord(home).findings; + expect(kept.id).toBe("abc0000000000000"); + expect(kept.sightings).toEqual([]); + expect(kept.occurrences).toBe(3); + }); + + it("never throws, whatever the file holds", () => { + for (const findings of [[{}], [[]], [{ id: 1 }], [{ id: "abc0000000000000", fingerprint: 7 }]]) { + write(findings); + expect(() => readLeakRecord(home)).not.toThrow(); + } + }); +}); diff --git a/__tests__/audit/macos-notifier.test.ts b/__tests__/audit/macos-notifier.test.ts new file mode 100644 index 000000000..f7235eb56 --- /dev/null +++ b/__tests__/audit/macos-notifier.test.ts @@ -0,0 +1,152 @@ +// @vitest-environment node +/** + * The macOS notifier, asserted from Linux. + * + * Every CI runner this project has is Linux, and the parts that need a Mac — + * `osacompile`, `launchctl bootstrap`, whether a banner actually appears — are + * exactly the parts no test here can reach. So this pins the half that IS + * platform-independent and is also the half that silently rots: the plist's + * shape, the AppleScript's structure, and the queue's on-disk contract. That is + * the same split `launchdPlistContents` already makes for the daemon's own + * plist. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync, mkdirSync, utimesSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { + MAC_NOTIFIER_LABEL, + macNotifyDir, + notifierPlistContents, + notifierScript, + queueMacNotification, + pruneMacNotifyQueue, +} from "@/src/audit/macos-notifier"; + +let home: string; +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "fp-macnote-")); + process.env.FAILPROOFAI_HOME = home; +}); +afterEach(() => { + delete process.env.FAILPROOFAI_HOME; + rmSync(home, { recursive: true, force: true }); +}); + +describe("the LaunchAgent plist", () => { + const plist = () => notifierPlistContents("/Users/x/.failproofai/bin/N.app", "/Users/x/.failproofai/run/notify"); + + it("runs the applet inside the bundle, not osascript", () => { + // A bare `osascript -e 'display notification'` is attributed to Script + // Editor: it inherits Script Editor's notification permission and shows up + // under Script Editor in System Settings. The bundle is what makes the + // banner say failproofai and gives the user something to turn off. + expect(plist()).toContain("/Users/x/.failproofai/bin/N.app/Contents/MacOS/applet"); + expect(plist()).not.toContain("osascript"); + }); + + it("is woken by the queue rather than left running", () => { + expect(plist()).toContain("WatchPaths"); + expect(plist()).toContain("/Users/x/.failproofai/run/notify"); + // Both false, and both for a reason: RunAtLoad would fire it at every login + // with nothing to say, and KeepAlive would treat its normal exit as a crash + // and restart it forever. + expect(plist()).toMatch(/RunAtLoad<\/key>\s*/); + expect(plist()).toMatch(/KeepAlive<\/key>\s*/); + }); + + it("carries the label the uninstall boots out", () => { + // These two must be the same string, or `uninstall --purge` removes the + // file and leaves launchd holding a live job pointing at a deleted applet. + expect(plist()).toContain(`${MAC_NOTIFIER_LABEL}`); + }); + + it("is well-formed XML with a real doctype", () => { + expect(plist().startsWith('')).toBe(true); + expect(plist()).toContain("")).toBe(true); + }); + + it("escapes a path that would otherwise break the document", () => { + const out = notifierPlistContents("/Users/a&b/N.app", "/Users/a/notify"); + expect(out).toContain("/Users/a&b/N.app"); + expect(out).toContain("/Users/a<b>/notify"); + }); +}); + +describe("the applet's script", () => { + it("deletes each payload BEFORE displaying it", () => { + // WatchPaths fires on every change to the directory, so a file that cannot + // be displayed and is not removed wakes this agent forever. At-most-once is + // the right failure: the caller already claimed the finding on disk, so a + // dropped banner costs one silent finding and a wake loop costs the machine. + const s = notifierScript(); + const rmAt = s.indexOf('rm -f'); + const showAt = s.indexOf("display notification"); + expect(rmAt).toBeGreaterThan(-1); + expect(showAt).toBeGreaterThan(-1); + expect(rmAt).toBeLessThan(showAt); + }); + + it("points at this home's queue, quoted", () => { + // JSON.stringify is what makes an AppleScript string literal out of a path, + // and a home containing a quote is a home this must not mis-escape. + expect(notifierScript()).toContain(JSON.stringify(macNotifyDir() + "/")); + }); + + it("reads a title and a body, and shows nothing for a one-line file", () => { + // A truncated payload is a real state — the queue writer renames into place + // precisely to avoid it, and this is the backstop if that ever regresses. + const s = notifierScript(); + expect(s).toContain("count of lines_) is greater than 1"); + }); +}); + +describe("the queue", () => { + it("lands the payload under the finding's own id", () => { + // One file per claimed finding is what keeps the banner count honest: the + // id is already unique per credential, so a repeat cannot double up. + expect(queueMacNotification("abc1230000000000", "Title", "Body text")).toBe(true); + expect(readFileSync(resolve(macNotifyDir(), "abc1230000000000"), "utf8")).toBe("Title\nBody text\n"); + }); + + it("leaves no partial file for the watcher to read", () => { + // WatchPaths fires on the first byte written, so the payload is built + // outside the directory and renamed in. Nothing but finished files ever + // appears here. + queueMacNotification("abc0000000000000", "T", "B"); + expect(readdirSync(macNotifyDir())).toEqual(["abc0000000000000"]); + expect(existsSync(resolve(macNotifyDir(), "..", "notify-abc0000000000000.tmp"))).toBe(false); + }); + + it("flattens newlines, because the applet reads the payload by line", () => { + queueMacNotification("00000000000000ff", "A\nB", "C\n\nD E"); + expect(readFileSync(resolve(macNotifyDir(), "00000000000000ff"), "utf8")).toBe("A B\nC D E\n"); + }); + + it("returns false instead of throwing when the queue cannot be written", () => { + // Read-only home, full disk, a home that is not a directory. None of them + // should turn a completed scan into a crash — the caller treats a false as + // "this channel is unavailable" and carries on. + process.env.FAILPROOFAI_HOME = "/dev/null/nope"; + expect(queueMacNotification("00000000000000ff", "T", "B")).toBe(false); + }); + + it("drops payloads nothing ever collected", () => { + // Queued while logged out, or with the agent removed. Showing them at the + // next login would announce keys that were rotated weeks ago. + mkdirSync(macNotifyDir(), { recursive: true }); + const stale = resolve(macNotifyDir(), "0000000000000010"); + const fresh = resolve(macNotifyDir(), "0000000000000011"); + writeFileSync(stale, "T\nB\n"); + writeFileSync(fresh, "T\nB\n"); + const longAgo = new Date(Date.now() - 30 * 86_400_000); + utimesSync(stale, longAgo, longAgo); + + pruneMacNotifyQueue(); + + expect(existsSync(stale)).toBe(false); + expect(existsSync(fresh)).toBe(true); + }); +}); diff --git a/__tests__/audit/notify-toggle.test.ts b/__tests__/audit/notify-toggle.test.ts new file mode 100644 index 000000000..880ff931b --- /dev/null +++ b/__tests__/audit/notify-toggle.test.ts @@ -0,0 +1,76 @@ +// @vitest-environment node +/** + * The off-switch, from the CLI side. + * + * The property that matters is not the flag — it is that this and the + * dashboard's toggle and the audit child's read are all the SAME key. Two + * surfaces that each remember their own answer is how a user silences a banner + * and keeps getting it. + */ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { runNotifyToggle } from "@/src/audit/schedule-cli"; +import { readConfig } from "@/src/hooks/fp-config"; + +let home: string; +let prev: string | undefined; +beforeEach(() => { + prev = process.env.FAILPROOFAI_HOME; + home = mkdtempSync(join(tmpdir(), "fp-notify-")); + process.env.FAILPROOFAI_HOME = home; + vi.spyOn(process.stdout, "write").mockImplementation(() => true); +}); +afterEach(() => { + vi.restoreAllMocks(); + if (prev === undefined) delete process.env.FAILPROOFAI_HOME; + else process.env.FAILPROOFAI_HOME = prev; + rmSync(home, { recursive: true, force: true }); +}); + +describe("failproofai audit --notify / --no-notify", () => { + it("writes the key the daemon's audit child reads", () => { + runNotifyToggle(false); + expect(readConfig().audit.notify).toBe(false); + expect(JSON.parse(readFileSync(resolve(home, "config.json"), "utf8")).audit.notify).toBe(false); + + runNotifyToggle(true); + expect(readConfig().audit.notify).toBe(true); + }); + + it("does not touch the schedule", () => { + // The whole reason this is its own flag pair: somebody silencing a banner + // must not discover they also switched off the weekly scan. + writeFileSync( + resolve(home, "config.json"), + JSON.stringify({ audit: { auto: true, interval_days: 30 } }), + ); + + runNotifyToggle(false); + + const after = readConfig().audit; + expect(after.auto).toBe(true); + expect(after.intervalDays).toBe(30); + expect(after.notify).toBe(false); + }); + + it("leaves unrelated settings alone", () => { + writeFileSync( + resolve(home, "config.json"), + JSON.stringify({ audit: { auto: true }, telemetry: { enabled: false } }), + ); + runNotifyToggle(false); + expect(readConfig().telemetry.enabled).toBe(false); + }); + + it("says which way it went, in both directions", () => { + const out = vi.mocked(process.stdout.write); + runNotifyToggle(false); + expect(out.mock.calls.flat().join("")).toContain("scans continue"); + out.mockClear(); + runNotifyToggle(true); + expect(out.mock.calls.flat().join("")).toContain("finds a credential"); + }); +}); diff --git a/__tests__/audit/redact-example.test.ts b/__tests__/audit/redact-example.test.ts index ded6ce649..97fc4a0de 100644 --- a/__tests__/audit/redact-example.test.ts +++ b/__tests__/audit/redact-example.test.ts @@ -227,9 +227,13 @@ describe("maskAssignedSecrets — the shape the blocking patterns do not carry", ]; for (const input of cases) { const out = redactExample(input, HOME); - expect(out, input).toContain("[REDACTED: assigned secret]"); - // The secret itself must be gone; the NAME is kept on purpose, because - // "which credential" is the actionable half of the finding. + // Any label will do. A value carrying a vendor prefix is claimed by the + // earlier SECRET_PATTERNS pass, which names it BETTER — `[REDACTED: Slack + // token]` beats `[REDACTED: assigned secret]`, because the label is what + // tells the reader which console to open. The security property under + // test is that the value is gone. + expect(out, input).toContain("[REDACTED"); + // The NAME is kept on purpose: "which credential" is the actionable half. const value = input.split("=")[1].split(" ")[0]; expect(out, input).not.toContain(value); } @@ -241,6 +245,143 @@ describe("maskAssignedSecrets — the shape the blocking patterns do not carry", ); }); + // The first spelling of ASSIGNMENT_RE was `NAME=value` and only that: no + // whitespace around the `=`, no `:`, no quoted name. Every other spelling + // reached the emailed digest with the value intact, and a value only survived + // that gap if it independently matched a vendor pattern in `SECRET_PATTERNS` + // — which 230 of 237 secret-named assignments measured on this machine do + // not. These are config, YAML and JSON, i.e. most of where credentials are + // actually written down. + it("masks assignments whatever the spacing, separator or name quoting", () => { + const cases = [ + // spaced `=` — the shape a config file or a pretty-printer writes + ["MY_API_KEY = sk-synthetic0000111122223333", "sk-synthetic0000111122223333"], + ["PGPASSWORD = letmein-prod", "letmein-prod"], + // `:` — YAML, and an HTTP-ish header line + ["db_password: letmein-prod", "letmein-prod"], + ['SLACK_BOT_TOKEN: "xoxb-synthetic-0000-1111"', "xoxb-synthetic-0000-1111"], + // quoted name — JSON + ['{"api_key": "abcdef0123456789abcdef"}', "abcdef0123456789abcdef"], + ["'client_secret': 'synthetic-secret-value'", "synthetic-secret-value"], + ] as const; + for (const [input, value] of cases) { + const out = redactExample(input, HOME); + // Any label will do — a value carrying a vendor prefix is claimed by the + // earlier `SECRET_PATTERNS` pass, which names it better than "assigned + // secret". The security property under test is that the value is gone. + expect(out, input).toContain("[REDACTED"); + expect(out, input).not.toContain(value); + } + }); + + it("re-emits the separator verbatim, so a redacted YAML line is still YAML", () => { + // Normalising every separator to `=` would turn a config excerpt into + // something that no longer looks like the file it came from, and the point + // of keeping the name is that the reader recognises the finding. + expect(redactExample("db_password: letmein", HOME)).toBe( + "db_password: [REDACTED: assigned secret]", + ); + expect(redactExample("MY_TOKEN = abcdef123456", HOME)).toBe( + "MY_TOKEN = [REDACTED: assigned secret]", + ); + }); + + // The separator grammar above is one half. This is the other: a name whose + // secret word is a camelCase hump rather than an `_` component decomposed to + // a single token that matched nothing, so `sessionKey=…` shipped its value + // verbatim while `SESSION_KEY=…` was masked. camelCase is what an identifier + // looks like everywhere except a shell environment. + it("masks a credential name written in camelCase, not just SCREAMING_SNAKE", () => { + for (const name of [ + "sessionKey", + "dbPass", + "basicAuth", + "authCookie", + "refreshToken", + "apiKeyValue", + ]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).toContain("[REDACTED: assigned secret]"); + expect(out, name).not.toContain("synthetic0000111122223333"); + } + }); + + it("knows the credential spellings of `passphrase` and `pwd`", () => { + for (const name of ["GPG_PASSPHRASE", "MYSQL_PWD", "ssh_passphrase"]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).toContain("[REDACTED: assigned secret]"); + } + }); + + it("leaves a bare PWD alone, because that is the working directory", () => { + // `PWD` is only a credential in a compound name. A bare one is on every + // second line of a captured shell session. + expect(redactExample("PWD=/home/user/project", HOME)).not.toContain("[REDACTED: assigned"); + }); + + it("does not let camelCase splitting invent new false positives", () => { + // The `_`-split negatives have to survive hump-splitting too: these are + // words that CONTAIN a secret component but are not compounds of one. + for (const input of [ + "monkeyCount=12", + "passengerList=4", + "authorName=jane", + "pathPrefix=/usr/bin", + "signalHandler=onExit", + ]) { + expect(redactExample(input, HOME), input).not.toContain("[REDACTED"); + } + }); + + // Every publishable key on earth is named `*_KEY`, and the component rule + // matched all of them — including names carrying the literal word PUBLIC. + // These are not conventionally public but MECHANICALLY public: the build tool + // inlines them into the browser bundle, so the framework published the value + // to every visitor before failproofai saw it. Reporting one as a credential + // exposure is a security tool crying wolf, which is not a cheap error the way + // ordinary over-redaction is. + it("does not mask a key the build tool ships to the browser", () => { + for (const name of [ + "NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY", + "NEXT_PUBLIC_SUPABASE_ANON_KEY", + "NEXT_PUBLIC_POSTHOG_KEY", + "VITE_API_KEY", + "REACT_APP_API_KEY", + "EXPO_PUBLIC_API_KEY", + "VAPID_PUBLIC_KEY", + "PUBLISHABLE_KEY", + "SUPABASE_ANON_KEY", + ]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).not.toContain("[REDACTED: assigned secret]"); + } + }); + + it("keeps masking a real secret whose name merely contains `public`", () => { + // The marker has to be a PREFIX or an explicit publishable suffix. A + // mid-name `PUBLIC`, or a word that merely starts with the same letters, + // must not disarm the rule. + for (const name of [ + "MY_PUBLIC_FACING_API_SECRET", + "PUBLISHER_API_KEY", + "REPUBLIC_TOKEN", + "STRIPE_SECRET_KEY", + "SUPABASE_SERVICE_ROLE_KEY", + ]) { + const out = redactExample(`${name}=synthetic0000111122223333`, HOME); + expect(out, name).toContain("[REDACTED"); + expect(out, name).not.toContain("synthetic0000111122223333"); + } + }); + + it("does not let a trailing `KEY:` swallow the next line", () => { + // `[ \t]*` rather than `\s*` around the separator: `\s` matches a newline, + // so a bare key at end-of-line would glue the following line into the match + // and redact it as though it were the value. + const out = redactExample("API_KEY:\nnpm run build", HOME); + expect(out).toContain("npm run build"); + }); + it("masks credentials inline in a URL, on schemes the block list omits", () => { // CONNECTION_STRING_RE deliberately excludes http/https, so this shape was // covered by nothing. diff --git a/__tests__/audit/redaction-sinks.test.ts b/__tests__/audit/redaction-sinks.test.ts new file mode 100644 index 000000000..265220e57 --- /dev/null +++ b/__tests__/audit/redaction-sinks.test.ts @@ -0,0 +1,167 @@ +// @vitest-environment node +/** + * Every audit renderer that emits an example must redact it first. + * + * `redact-example.ts` was written carefully and then wired to exactly ONE of the + * places an example can leave the machine: the emailed digest. The markdown file + * the CLI prints as "Shareable report" wrote raw commands and raw cwd into the + * user's working tree, and `formatJson` was a bare stringify of the whole + * result. Both are artifacts whose entire purpose is to be sent somewhere. + * + * These tests pin the wiring rather than the redactor — `redact-example.test.ts` + * covers what a secret looks like once masked. What is asserted here is that + * each renderer is CONNECTED, because the defect was never a bad mask; it was a + * mask nobody called. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { formatText, formatMarkdown, formatJson } from "@/src/audit/report"; +import { redactAuditResult } from "@/src/audit/redact-example"; +import type { AuditCount, AuditResult } from "@/src/audit/types"; + +const SECRET = "ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; +const HOME_PATH = "/home/testuser/clients/acme-bank/src/db.ts"; + +function count(overrides: Partial = {}): AuditCount { + return { + name: "failproofai/block-secrets-write", + source: "builtin", + category: "Security", + severity: "deny", + hits: 3, + projects: 1, + firstSeen: "2026-09-01T10:00:00.000Z", + lastSeen: "2026-09-04T10:00:00.000Z", + examples: [ + { + sessionId: "s1", + cwd: "/home/testuser/clients/acme-bank", + timestamp: "2026-09-04T10:00:00.000Z", + example: `curl -H "Authorization: Bearer ${SECRET}" https://api.example.com`, + }, + ], + displayTitle: "Wrote a secret to a file", + impact: "The credential outlives the session.", + enabledInConfig: true, + installHint: "", + ...overrides, + }; +} + +function result(results: AuditCount[] = [count()]): AuditResult { + return { + version: 2, + scannedAt: "2026-09-07T10:00:00.000Z", + scope: { cli: ["claude"], projects: [HOME_PATH], since: null }, + transcripts: { scanned: 1, skipped: 0, errors: 0, durationMs: 1 }, + results, + totals: { hits: 3, projectsWithHits: 1 }, + projectsScanned: ["/home/testuser/clients/acme-bank"], + eventsScanned: 10, + enabledBuiltinNames: ["block-secrets-write"], + // OPTIONAL fields must be present here or the reflection guard below has + // no teeth: it walks a real object, and TypeScript will not complain about + // an optional field the redactor forgot. This fixture is the only thing + // standing between a new field and a silent passthrough. + newLeakIds: ["abc123"], + }; +} + +describe("redactAuditResult — a whitelist, not a spread", () => { + // It used to be `{...result}` plus three named rewrites, so every field it + // did not name passed through byte-identical — including fields added later, + // with absolute home paths intact. The compiler catches a new REQUIRED field; + // it stays quiet about an optional one, and the leak record arrives as + // optional fields. So this reflects over a real result instead. + it("has consciously handled every key on the real object", async () => { + const { REDACTED_AUDIT_RESULT_KEYS } = await import("@/src/audit/redact-example"); + const actual = Object.keys(result()).sort(); + const handled = [...REDACTED_AUDIT_RESULT_KEYS].sort(); + const unhandled = actual.filter((k) => !handled.includes(k as never)); + expect( + unhandled, + `AuditResult grew ${unhandled.join(", ")} — decide in redactAuditResult whether it needs ` + + `redacting, then add it to REDACTED_AUDIT_RESULT_KEYS`, + ).toEqual([]); + }); + + it("leaves no absolute home path anywhere in the redacted object", () => { + const out = redactAuditResult(result(), "/home/testuser"); + expect(JSON.stringify(out)).not.toContain("/home/testuser"); + }); +}); + +describe("formatMarkdown — the file the CLI calls a Shareable report", () => { + it("never writes a credential into the report body", () => { + const md = formatMarkdown(result()); + expect(md).toContain("Examples"); + expect(md).not.toContain(SECRET); + expect(md).toContain("[REDACTED"); + }); + + it("shortens the cwd so the report does not carry a map of someone's disk", () => { + const md = formatMarkdown(result()); + expect(md).not.toContain("/home/testuser/clients/acme-bank"); + }); +}); + +describe("formatJson — piped wherever the caller wants", () => { + it("redacts examples inside the serialized result", () => { + const json = formatJson(result()); + expect(json).not.toContain(SECRET); + }); + + it("redacts the project paths carried alongside the findings", () => { + const json = formatJson(result()); + expect(json).not.toContain("/home/testuser/clients/acme-bank"); + }); + + it("leaves the caller's own object untouched", () => { + const r = result(); + formatJson(r); + expect(r.results[0].examples[0].example).toContain(SECRET); + }); +}); + +describe("formatText — the local terminal", () => { + it("masks the credential", () => { + const text = formatText(result(), { showExamples: true }); + expect(text).not.toContain(SECRET); + }); + + it("keeps the real path, because shortening it protects nobody on their own machine", () => { + const withPath = count({ + examples: [ + { + sessionId: "s1", + cwd: "/home/testuser/clients/acme-bank", + timestamp: "2026-09-04T10:00:00.000Z", + example: `cat ${HOME_PATH}`, + }, + ], + }); + const text = formatText(result([withPath]), { showExamples: true }); + expect(text).toContain(HOME_PATH); + }); +}); + +describe("wiring tripwire", () => { + it("keeps report.ts importing the redactor", () => { + // The defect was structural: redactExample had exactly two non-definition + // references and both were in harm-report.ts. If this import is ever + // dropped, every renderer above silently starts emitting raw examples + // again — and the tests above only catch it for the shapes they model. + const src = readFileSync(join(process.cwd(), "src/audit/report.ts"), "utf-8"); + expect(src).toMatch(/from "\.\/redact-example"/); + }); + + it("keeps more than one module depending on the redactor", () => { + const files = ["src/audit/report.ts", "src/audit/harm-report.ts"]; + const importers = files.filter((f) => + readFileSync(join(process.cwd(), f), "utf-8").includes('from "./redact-example"'), + ); + expect(importers).toEqual(files); + }); +}); diff --git a/__tests__/audit/scheduled-audit.test.ts b/__tests__/audit/scheduled-audit.test.ts index fcdb60e63..767c14a4a 100644 --- a/__tests__/audit/scheduled-audit.test.ts +++ b/__tests__/audit/scheduled-audit.test.ts @@ -26,6 +26,7 @@ const h = vi.hoisted(() => ({ writeDashboardCache: vi.fn(() => true), openWhenReady: vi.fn(), launch: vi.fn(), + notifyDesktop: vi.fn<(...a: unknown[]) => Promise>(() => Promise.resolve({ ok: true, id: 1 })), })); vi.mock("../../src/hooks/hook-telemetry", () => ({ trackHookEvent: h.trackHookEvent })); @@ -33,9 +34,11 @@ vi.mock("../../src/audit/index", () => ({ runAudit: h.runAudit })); vi.mock("../../src/audit/dashboard-cache", () => ({ writeDashboardCache: h.writeDashboardCache })); vi.mock("../../src/audit/open-browser", () => ({ openWhenReady: h.openWhenReady })); vi.mock("../../scripts/launch", () => ({ launch: h.launch })); +vi.mock("../../src/audit/desktop-notify", () => ({ notifyDesktop: h.notifyDesktop })); vi.mock("../../lib/telemetry-id", () => ({ getInstanceId: () => "test-instance" })); import { runAuditCli, runScheduledAudit, EXIT_AUDIT_ALREADY_RUNNING } from "../../src/audit/cli"; +import { leakNoticeText } from "../../src/hooks/notice"; function result(over: Partial = {}): AuditResult { return { @@ -60,6 +63,7 @@ beforeEach(() => { vi.clearAllMocks(); h.trackHookEvent.mockImplementation(() => Promise.resolve()); h.writeDashboardCache.mockReturnValue(true); + h.notifyDesktop.mockResolvedValue({ ok: true, id: 1 }); prevHome = process.env.FAILPROOFAI_HOME; home = mkdtempSync(resolve(tmpdir(), "fpai-sched-")); process.env.FAILPROOFAI_HOME = home; @@ -367,3 +371,96 @@ describe("the binary-level scheduled entry point", () => { expect(existsSync(resolve(fp, "config.json"))).toBe(true); }, SUBPROCESS_TIMEOUT_MS); }); + +// A scan that finds a key and tells nobody is the failure this whole feature +// exists to prevent — and the scheduled run is precisely the one with nobody +// watching the terminal it printed to. +describe("announcing a leak on the desktop", () => { + const withLeaks = (ids: string[]) => + result({ totals: { hits: 1, projectsWithHits: 1 }, newLeakIds: ids }); + + const writeAuditConfig = (audit: Record) => { + mkdirSync(home, { recursive: true }); + writeFileSync(resolve(home, "config.json"), JSON.stringify({ audit })); + }; + + it("raises one factual review banner when this run finds new matches", async () => { + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111", "2222222222222222"])); + + expect(await runScheduledAudit()).toBe(0); + + expect(h.notifyDesktop).toHaveBeenCalledTimes(1); + const [summary, body] = h.notifyDesktop.mock.calls[0] as unknown as [string, string]; + expect(summary).toContain("failproofai"); + expect(body).toContain("Possible credential exposure"); + expect(body).toContain("failproofai audit"); + expect(body).not.toMatch(/email|schedule|2 credentials|leaked credential/i); + }); + + it("keeps the in-CLI notice focused on review rather than email setup", () => { + expect(leakNoticeText(2)).toContain("review the matches"); + expect(leakNoticeText(2)).not.toMatch(/email|--schedule/); + }); + + it("says nothing when the scan found nothing new", async () => { + // Including the repeat-scan case: the same key, already announced, is not + // news. Silence here is what makes a weekly timer tolerable. + h.runAudit.mockResolvedValue(result({ totals: { hits: 9, projectsWithHits: 3 } })); + await runScheduledAudit(); + expect(h.notifyDesktop).not.toHaveBeenCalled(); + }); + + it("announces each finding at most once, across runs", async () => { + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + await runScheduledAudit(); + expect(h.notifyDesktop).toHaveBeenCalledTimes(1); + }); + + it("does not interrupt a user who turned the banner off", async () => { + writeAuditConfig({ auto: true, notify: false }); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + expect(h.notifyDesktop).not.toHaveBeenCalled(); + }); + + it("still announces when the config has an audit table but no opinion on notifying", async () => { + writeAuditConfig({ auto: true }); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + expect(h.notifyDesktop).toHaveBeenCalledTimes(1); + }); + + it("leaves the in-session notice to fire even after the banner succeeded", async () => { + // The two channels claim separately, because `Notify` returning an id does + // NOT mean a human saw anything — on a locked screen the shell accepts the + // call and shows nothing. Letting the banner claim the finding would + // suppress the one channel that does reach them. + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + await runScheduledAudit(); + expect(existsSync(resolve(home, "audit", "notified-desktop", "1111111111111111"))).toBe(true); + expect(existsSync(resolve(home, "audit", "notified", "1111111111111111"))).toBe(false); + }); + + it("stays a successful scan when there is no desktop to notify", async () => { + // A headless box, a container, an SSH session: all normal, none of them a + // reason to report the audit itself as failed and make the scheduler back + // off from the thing that actually matters. + h.notifyDesktop.mockResolvedValue({ ok: false, reason: "no-session", detail: "ENOENT" }); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + + expect(await runScheduledAudit()).toBe(0); + expect(existsSync(resolve(home, "audit", "notified-desktop", "1111111111111111"))).toBe(false); + await runScheduledAudit(); + expect(h.notifyDesktop).toHaveBeenCalledTimes(2); + }); + + it("survives a notifier that throws outright", async () => { + h.notifyDesktop.mockRejectedValue(new Error("boom")); + h.runAudit.mockResolvedValue(withLeaks(["1111111111111111"])); + expect(await runScheduledAudit()).toBe(0); + expect(existsSync(resolve(home, "audit", "notified-desktop", "1111111111111111"))).toBe(false); + expect(await runScheduledAudit()).toBe(0); + expect(h.notifyDesktop).toHaveBeenCalledTimes(2); + }); +}); diff --git a/__tests__/audit/share-templates.test.ts b/__tests__/audit/share-templates.test.ts index dc25d287a..27f810c4f 100644 --- a/__tests__/audit/share-templates.test.ts +++ b/__tests__/audit/share-templates.test.ts @@ -11,9 +11,14 @@ const ctx: ShareCtx = { score: 72, arch: "the cowboy", grade: "B", missing: 3 }; const cleanCtx: ShareCtx = { score: 96, arch: "the precision builder", grade: "S", missing: 0 }; describe("share templates", () => { - it("ships 10 X and 10 LinkedIn templates", () => { - expect(X_TEMPLATES).toHaveLength(10); - expect(LI_TEMPLATES).toHaveLength(10); + // Nine, not ten, since the score was switched off: one template per channel + // had the number as its entire premise ("my agent scored X/100, think yours + // can beat it?") and there was nothing left of it once the number went, so + // both are commented out in place rather than reworded. The other eighteen + // lost a subordinate clause and survive. + it("ships 9 X and 9 LinkedIn templates", () => { + expect(X_TEMPLATES).toHaveLength(9); + expect(LI_TEMPLATES).toHaveLength(9); }); it("every template ends on the npx CTA, references score or archetype, and embeds no URL", () => { @@ -33,11 +38,26 @@ describe("share templates", () => { } }); - it("references the score on most templates and the archetype on most templates", () => { - const withScore = [...X_TEMPLATES, ...LI_TEMPLATES].filter((t) => t(ctx).includes("72")); - const withArch = [...X_TEMPLATES, ...LI_TEMPLATES].filter((t) => t(ctx).includes("the cowboy")); - expect(withScore.length).toBeGreaterThanOrEqual(15); - expect(withArch.length).toBeGreaterThanOrEqual(15); + // Inverted by the score being switched off. This used to require the score on + // >=15 of 20; it now requires it on NONE, which is the assertion that keeps + // the switch honest — a template that quietly reintroduces `${score}` would + // render `undefined/100` on the shared card, since nothing upstream computes + // it any more. Restoring the score means restoring the old bound here too. + it("references the archetype on every template and the score on none", () => { + const all = [...X_TEMPLATES, ...LI_TEMPLATES]; + const withScore = all.filter((t) => t(ctx).includes("72")); + const withArch = all.filter((t) => t(ctx).includes("the cowboy")); + expect(withScore).toHaveLength(0); + expect(withArch).toHaveLength(all.length); + }); + + it("never renders `undefined` when no score is supplied", () => { + // The dashboard stops passing score/grade entirely, so the templates are + // called with neither. Any surviving interpolation would show up here. + const scoreless: ShareCtx = { arch: "the cowboy", missing: 3 }; + for (const t of [...X_TEMPLATES, ...LI_TEMPLATES]) { + expect(t(scoreless)).not.toContain("undefined"); + } }); it("tags the channel's handle (@failproofai on X, @Failproof AI on LinkedIn)", () => { diff --git a/__tests__/audit/sqlite-experimental-warning.test.ts b/__tests__/audit/sqlite-experimental-warning.test.ts new file mode 100644 index 000000000..d58500662 --- /dev/null +++ b/__tests__/audit/sqlite-experimental-warning.test.ts @@ -0,0 +1,82 @@ +// @vitest-environment node +/** + * Node's SQLite warning must never reach the terminal. + * + * Not a cosmetic concern. `startProgress()` in `src/audit/cli.ts` redraws its + * four stage lines by moving the cursor up a FIXED number of rows, so anything + * else printed mid-run pushes the cursor down and the next redraw repaints the + * block lower — leaving the top of the previous frame stranded above it. The + * warning is two lines, printed on first use of `node:sqlite`, which lands it + * squarely inside the block. Every stage appeared twice and the audit looked + * like it had run twice. + * + * There is no defensive fix in the renderer (erasing to end of screen leaves + * the stranded lines, which are ABOVE the cursor), so the fix is to keep the + * terminal quiet — and this is the guard on that. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { openSqliteReadonly } from "@/lib/sqlite-reader"; + +let dir: string; +let dbPath: string; +let original: typeof process.emitWarning; + +beforeEach(async () => { + original = process.emitWarning; + dir = mkdtempSync(join(tmpdir(), "fp-warn-")); + dbPath = join(dir, "s.db"); + const initSqlJs = (await import("sql.js/dist/sql-asm.js")).default; + const SQL = await initSqlJs(); + const db = new SQL.Database(); + db.run("CREATE TABLE t (a TEXT);"); + db.run("INSERT INTO t VALUES ('x')"); + writeFileSync(dbPath, Buffer.from(db.export())); + db.close(); +}); + +afterEach(() => { + process.emitWarning = original; + rmSync(dir, { recursive: true, force: true }); +}); + +describe("the SQLite experimental warning", () => { + it("never reaches the terminal when a database is opened", async () => { + const seen: string[] = []; + // Stand in for Node's own emitter so the assertion is about what WOULD be + // printed, not about capturing stderr from another process. + process.emitWarning = ((w: unknown) => { + seen.push(typeof w === "string" ? w : String((w as Error)?.message ?? "")); + }) as typeof process.emitWarning; + + const reader = await openSqliteReadonly(dbPath); + expect(reader).not.toBeNull(); + reader!.query("SELECT * FROM t"); + reader!.close?.(); + + // Re-emit through whatever the module installed, to prove the filter is in + // place rather than that Node simply did not warn on this run. + process.emitWarning("SQLite is an experimental feature and might change at any time"); + expect(seen.filter((m) => m.includes("SQLite is an experimental feature"))).toEqual([]); + }); + + it("still lets every other warning through", async () => { + // A blanket silence would hide deprecations and real problems, in a tool + // that installs into other people's machines. + const seen: string[] = []; + process.emitWarning = ((w: unknown) => { + seen.push(typeof w === "string" ? w : String((w as Error)?.message ?? "")); + }) as typeof process.emitWarning; + + const reader = await openSqliteReadonly(dbPath); + reader?.close?.(); + + process.emitWarning("something that actually matters"); + process.emitWarning(new Error("a real deprecation")); + expect(seen).toContain("something that actually matters"); + expect(seen).toContain("a real deprecation"); + }); +}); diff --git a/__tests__/audit/sqlite-wal-heap.test.ts b/__tests__/audit/sqlite-wal-heap.test.ts new file mode 100644 index 000000000..e8d4b80ec --- /dev/null +++ b/__tests__/audit/sqlite-wal-heap.test.ts @@ -0,0 +1,115 @@ +// @vitest-environment node +/** + * sql.js dies after 128 WAL-mode opens, and takes the rest of the run with it. + * + * `lib/sqlite-reader.ts` falls through to sql.js whenever `node:sqlite` is + * absent — which is every bun process and every supported Node below 22.5, so + * `engines.node: ">=20.9.0"` puts real users on this path. + * + * That build's heap is a FIXED 22,151,168-byte ArrayBuffer compiled with + * ALLOW_MEMORY_GROWTH off, so growing it is `abort("OOM")`. Opening a + * WAL-flagged image makes SQLite build a wal-index shared-memory region that + * sql.js's MEMFS VFS never reclaims on close, and the 128th open exhausts the + * heap — the same threshold for a 319 KB database and a 10 MB one, because the + * leak is a fixed per-connection allocation rather than data. + * + * The abort is not confined to that open: `initSqlJs` memoizes one module for + * the process, so afterwards EVERY `openSqliteReadonly` silently returns null. + * A full audit opens ~150 databases (devin, goose and opencode each open theirs + * once per session), so it crossed the line and then dropped whatever it had + * not yet read — with `Aborted(OOM)` on stderr as the only symptom, and a run + * that still reported success. Fixing it recovered 9 sessions on the machine + * this was found on. + */ +import { describe, it, expect, beforeAll, afterAll, vi } from "vitest"; +import { mkdtempSync, rmSync, writeFileSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// FORCE TIER 2. `sqlite-reader` prefers `node:sqlite` and only falls through to +// sql.js when that import rejects — which is every bun process and every Node +// below 22.5, but NOT the Node this suite runs on. Without this mock the whole +// file is vacuously green: it passed identically with the fix reverted, because +// it never reached the code under test. +vi.mock("node:sqlite", () => { + throw new Error("No such built-in module: node:sqlite"); +}); + +import { openSqliteReadonly } from "@/lib/sqlite-reader"; + +let dir: string; +let dbPath: string; + +// A minimal real SQLite file, built through sql.js itself and then flagged WAL +// in its header — which is exactly the shape the goose/opencode/devin +// databases arrive in on a live machine. +beforeAll(async () => { + dir = mkdtempSync(join(tmpdir(), "fp-wal-")); + dbPath = join(dir, "sessions.db"); + + // Same specifier the reader uses — `lib/sql-js-asm.d.ts` types this one; the + // bare "sql.js" is untyped and fails `tsc --noEmit`, which gates CI. + const initSqlJs = (await import("sql.js/dist/sql-asm.js")).default; + const SQL = await initSqlJs(); + const db = new SQL.Database(); + db.run("CREATE TABLE sessions (id TEXT, body TEXT);"); + for (let i = 0; i < 50; i += 1) db.run("INSERT INTO sessions VALUES (?, ?)", [`s${i}`, "x".repeat(200)]); + const bytes = Buffer.from(db.export()); + db.close(); + + // Header bytes 18/19 are the file-format write/read versions. 2 = WAL. + bytes[18] = 2; + bytes[19] = 2; + writeFileSync(dbPath, bytes); +}); + +afterAll(() => rmSync(dir, { recursive: true, force: true })); + +describe("opening a WAL-mode database many times", () => { + it("is what the audit actually does, and it must not exhaust the heap", async () => { + // 150 is the measured open count of a full audit on a real machine; 128 is + // where the unfixed build died. Anything at or above 129 proves the fix. + const OPENS = 150; + let lastRowCount = -1; + + for (let i = 0; i < OPENS; i += 1) { + const reader = await openSqliteReadonly(dbPath); + expect(reader, `open #${i + 1} returned null — sql.js is dead for this process`).not.toBeNull(); + const rows = reader!.query<{ id: string }>("SELECT id FROM sessions"); + // Not just "it opened": the silent failure mode returns a LIVE reader + // whose queries come back empty, which is indistinguishable from a + // machine with no sessions. + expect(rows.length, `open #${i + 1} read no rows`).toBe(50); + if (lastRowCount >= 0) expect(rows.length).toBe(lastRowCount); + lastRowCount = rows.length; + reader!.close?.(); + } + }, 120_000); + + it("reads the same rows a rollback-journal image gives", async () => { + // Clearing the WAL flag must not change what is read. sql.js never sees the + // -wal sidecar either way, so the two are the same snapshot. + const plain = join(dir, "plain.db"); + const bytes = readFileSync(dbPath); + bytes[18] = 1; + bytes[19] = 1; + writeFileSync(plain, bytes); + + const a = await openSqliteReadonly(dbPath); + const b = await openSqliteReadonly(plain); + expect(a!.query("SELECT * FROM sessions")).toEqual(b!.query("SELECT * FROM sessions")); + a!.close?.(); + b!.close?.(); + }); + + it("leaves the file on disk untouched", async () => { + // The flag is cleared on our own in-memory copy. Rewriting a user's real + // session database to make our reader happy would be indefensible. + const before = readFileSync(dbPath); + const r = await openSqliteReadonly(dbPath); + r!.query("SELECT 1"); + r!.close?.(); + expect(readFileSync(dbPath).equals(before)).toBe(true); + expect(readFileSync(dbPath)[18]).toBe(2); // still flagged WAL on disk + }); +}); diff --git a/__tests__/hooks/builtin-policies.test.ts b/__tests__/hooks/builtin-policies.test.ts index cd3c4500b..328a66a67 100644 --- a/__tests__/hooks/builtin-policies.test.ts +++ b/__tests__/hooks/builtin-policies.test.ts @@ -125,7 +125,10 @@ describe("hooks/builtin-policies", () => { const cases: Array<[string, string]> = [ ["sk-ant-api03-AAAAAAAAAAAAAAAAAAAA", "Anthropic API key"], ["sk-proj-AAAAAAAAAAAAAAAAAAAA", "OpenAI project API key"], - ["sk-AAAAAAAAAAAAAAAAAAAA", "OpenAI API key"], + ["sk-AAAAAAAAAAAAAAAAAAAA", // A bare `sk-` cannot be attributed: LiteLLM's docs say its virtual keys + // "must start with sk-", and DeepSeek and every OpenAI-compatible gateway + // mint the same shape. Naming OpenAI here sent users to the wrong console. + "OpenAI-compatible key (issuer unknown)"], ["ghp_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "GitHub personal access token"], ["AKIAIOSFODNN7EXAMPLE", "AWS access key ID"], ["sk_live_AAAAAAAAAAAAAAAAAAAAAAAA", "Stripe live secret key"], diff --git a/__tests__/hooks/dogfood-configs.test.ts b/__tests__/hooks/dogfood-configs.test.ts index f0b07c15d..a0a4866e3 100644 --- a/__tests__/hooks/dogfood-configs.test.ts +++ b/__tests__/hooks/dogfood-configs.test.ts @@ -14,6 +14,7 @@ */ import { describe, it, expect } from "vitest"; import { existsSync, readFileSync } from "node:fs"; +import { execFileSync } from "node:child_process"; import { resolve } from "node:path"; import { CLAUDE_INSTALL_EVENT_TYPES } from "@/src/hooks/types"; @@ -172,6 +173,62 @@ describe("the surfaces that spawn bun from JS rather than a config", () => { }); }); +describe("git must not be able to hide a gutted config", () => { + /** + * `git update-index --skip-worktree` on a dogfood config is a trap with no + * bottom, and it sprang: nine of these files were emptied on disk to `{}` / + * `{"version":1}` while `git status` reported a completely clean tree, + * `git checkout -- ` FAILED SILENTLY (exit 1, "pathspec did not match") + * leaving the empty file in place, and `git stash` said "No local changes to + * save". Every assertion in this file reads the WORKING TREE, so all eight + * per-file tests failed — and were written off as "known pre-existing + * failures" for weeks, because every attempt to restore from git was a no-op + * that reported success. + * + * The real cost was not the red tests. With those files empty, failproofai + * enforced NOTHING in this repo for codex, copilot, cursor, factory, devin, + * antigravity, goose, opencode and pi — silently, while `git status` said + * everything was fine. + */ + it("marks no dogfood config skip-worktree or assume-unchanged", () => { + const paths = [ + ...CONFIGS.map((c) => c.file), + ".opencode/opencode.json", + ".opencode/plugins/failproofai.mjs", + ".pi/settings.json", + ".failproofai/policies-config.json", + ].filter((f) => existsSync(resolve(ROOT, f))); + + const out = execFileSync("git", ["ls-files", "-v", "--", ...paths], { + cwd: ROOT, + encoding: "utf8", + }); + + // `git ls-files -v` tags a normal tracked file `H`. UPPERCASE `S` is + // skip-worktree; a LOWERCASE letter is assume-unchanged. Both hide edits, + // so anything that is not exactly `H` fails — matching on case alone (the + // obvious `/^[a-z]/`) would have passed vacuously against the real damage, + // which was tagged `S`. + const hidden = out + .split(String.fromCharCode(10)) + .filter((line) => line.trim() && !line.startsWith("H ")) + .map((line) => line.trim()); + + expect(hidden, `hidden from git — these can be gutted invisibly:\n${hidden.join("\n")}`) + .toEqual([]); + }); + + it("has no dogfood config that is empty JSON", () => { + // The state the bit was hiding. Cheap, and it catches the damage directly + // rather than only the mechanism that concealed it. + for (const { file } of CONFIGS) { + const raw = readFileSync(resolve(ROOT, file), "utf8").trim(); + expect(raw.length, `${file} is empty`).toBeGreaterThan(20); + expect(raw, `${file} has been gutted`).not.toBe("{}"); + } + }); +}); + describe("repo-wide invariants", () => { it("the launcher every config points at exists", () => { expect(existsSync(resolve(ROOT, LAUNCHER))).toBe(true); diff --git a/__tests__/hooks/fp-home.test.ts b/__tests__/hooks/fp-home.test.ts index e20a2fde1..a387cc2cf 100644 --- a/__tests__/hooks/fp-home.test.ts +++ b/__tests__/hooks/fp-home.test.ts @@ -471,7 +471,7 @@ describe("config.toml", () => { redact: "off" as const, environment: "prod", machineId: "box-1", }, telemetry: { enabled: true }, - audit: { auto: true, intervalDays: 14 }, + audit: { auto: true, notify: true, intervalDays: 14 }, }; writeConfig(cfg); expect(readConfig()).toEqual(cfg); @@ -506,29 +506,33 @@ describe("config.toml", () => { expect(readConfig().telemetry.enabled).toBe(false); }); - it("the scheduled audit is OFF by default and says so in the file", () => { - // The opposite posture to telemetry directly above: off, and deliberately - // visible, because it is a switch the user is meant to find and flip. It is - // off because the scan reads the contents of every transcript on disk. - expect(DEFAULT_CONFIG.audit).toEqual({ auto: false, intervalDays: 7 }); + it("falls back to OFF for a machine with no config at all", () => { + // DEFAULT_CONFIG is the NO-FILE answer specifically, and it is the one case + // that stays off: there is no opinion to read, and "we could not tell" must + // never start a scan that reads every transcript on disk. A machine that + // HAS a config reads the other way — the test below. + expect(DEFAULT_CONFIG.audit).toEqual({ auto: false, notify: true, intervalDays: 7 }); writeConfig(DEFAULT_CONFIG); - // Both keys on disk, unconditionally. The layout-2 file made this visible + // Every key on disk, unconditionally. The layout-2 file made this visible // with a comment block; JSON cannot carry one, so what survives is the // weaker but still real guarantee: every field the struct holds is written, // so no later regeneration can silently drop one. const written = JSON.parse(readFileSync(H.configFile(), "utf8")); - expect(written.audit).toEqual({ auto: false, interval_days: 7 }); + expect(written.audit).toEqual({ auto: false, notify: true, interval_days: 7 }); }); it("an enabled auto-audit SURVIVES a rewrite", () => { // writeConfig regenerates the whole file, so a key it does not emit is a key // it silently deletes — the failure that would turn somebody's weekly audit // off the next time any unrelated setting changed. - writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, intervalDays: 30 } }); - expect(readConfig().audit).toEqual({ auto: true, intervalDays: 30 }); + writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, notify: false, intervalDays: 30 } }); + expect(readConfig().audit).toMatchObject({ auto: true, notify: false, intervalDays: 30 }); writeConfig({ ...readConfig(), collector: { ...DEFAULT_CONFIG.collector, environment: "ci" } }); - expect(readConfig().audit).toEqual({ auto: true, intervalDays: 30 }); + // `notify: false` is the interesting half: a rewrite that dropped it would + // restore a banner the user explicitly turned off, which reads as the + // setting being ignored. + expect(readConfig().audit).toMatchObject({ auto: true, notify: false, intervalDays: 30 }); }); it("carries the consent stamp through an unrelated rewrite too", () => { @@ -539,7 +543,7 @@ describe("config.toml", () => { // leave the user with a schedule that reads as on and mails nothing. writeConfig({ ...DEFAULT_CONFIG, - audit: { auto: true, intervalDays: 30, reportsConsentedAt: 1_700_000_000_000 }, + audit: { auto: true, notify: true, intervalDays: 30, reportsConsentedAt: 1_700_000_000_000 }, }); writeConfig({ ...readConfig(), collector: { ...DEFAULT_CONFIG.collector, environment: "ci" } }); @@ -550,7 +554,7 @@ describe("config.toml", () => { it("does not invent a consent stamp for a machine that never gave one", () => { // The other direction, and the one that matters more: a default-shaped // write must not put a key on disk implying somebody was asked. - writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, intervalDays: 30 } }); + writeConfig({ ...DEFAULT_CONFIG, audit: { auto: true, notify: true, intervalDays: 30 } }); expect(readConfig().audit.reportsConsentedAt).toBeUndefined(); expect(JSON.parse(readFileSync(H.configFile(), "utf8")).audit).not.toHaveProperty( @@ -558,11 +562,64 @@ describe("config.toml", () => { ); }); - it("only an explicit true switches the auto-audit on", () => { - writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: "yes" } })); + // The three-way split, and the reason it is not a two-way one. A configured + // machine that never said anything scans; a machine we could not READ an + // opinion from does not. `crates/failproofaid/src/audit_lane.rs` makes the + // identical distinction over the identical bytes, and its own test mirrors + // this table — the daemon deciding to scan while the settings page says it is + // off (or the reverse) is the bug both tests exist to prevent. + it("scans unless the config says exactly false, but only once there IS a config", () => { + const cases: Array<[string, boolean]> = [ + [JSON.stringify({ audit: { auto: true } }), true], + [JSON.stringify({ audit: { auto: false } }), false], + // Present but silent: setup ran, nobody objected. This is the case that + // flipped, and it is the common one. + [JSON.stringify({ audit: {} }), true], + [JSON.stringify({ audit: { interval_days: 14 } }), true], + // Not `false`, so not an objection. "yes" reading as ON is the reverse of + // what this test used to assert, and it follows from the flip rather than + // being a separate decision. + [JSON.stringify({ audit: { auto: "yes" } }), true], + // No audit table at all: a config written by something that predates the + // key, so nobody was ever shown the disclosure. Not an opinion — an + // absence. + [JSON.stringify({ collector: { hooks: true } }), false], + // Unparseable, and an array is not a config object either. + ["{ not json", false], + ["[]", false], + [JSON.stringify({ audit: [] }), false], + ]; + for (const [body, expected] of cases) { + writeFileSync(H.configFile(), body); + expect(readConfig().audit.auto, body).toBe(expected); + } + rmSync(H.configFile()); expect(readConfig().audit.auto).toBe(false); - writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: true } })); - expect(readConfig().audit.auto).toBe(true); + }); + + it("notifies unless told not to, without needing an audit table to say so", () => { + // Ungated on the table's presence, unlike `auto`: this is read when a + // notification is about to fire, so a scan has already happened and the + // only question left is whether to speak. + for (const [body, expected] of [ + [JSON.stringify({ audit: { notify: false } }), false], + [JSON.stringify({ audit: { notify: true } }), true], + [JSON.stringify({ audit: {} }), true], + [JSON.stringify({ audit: { notify: "off" } }), true], + [JSON.stringify({ collector: {} }), true], + ] as Array<[string, boolean]>) { + writeFileSync(H.configFile(), body); + expect(readConfig().audit.notify, body).toBe(expected); + } + }); + + it("keeps the two switches independent", () => { + // Somebody who wants the scan and not the banner is a coherent person, and + // the only alternative this offers them is turning the scan off. + writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: true, notify: false } })); + expect(readConfig().audit).toMatchObject({ auto: true, notify: false }); + writeFileSync(H.configFile(), JSON.stringify({ audit: { auto: false, notify: true } })); + expect(readConfig().audit).toMatchObject({ auto: false, notify: true }); }); it("resolves a nonsense interval to the default rather than to a daily scan", () => { @@ -587,7 +644,7 @@ describe("config.toml", () => { writeConfig({ ...DEFAULT_CONFIG, telemetry: { enabled: false } }); updateConfig({ audit: { auto: true } }); const after = readConfig(); - expect(after.audit).toEqual({ auto: true, intervalDays: 7 }); + expect(after.audit).toMatchObject({ auto: true, notify: true, intervalDays: 7 }); expect(after.telemetry.enabled).toBe(false); // untouched }); @@ -609,7 +666,7 @@ describe("config.toml", () => { mode: "cloud" as const, daemon: { configured: true }, telemetry: { enabled: false }, - audit: { auto: true, intervalDays: 30 }, + audit: { auto: true, notify: true, intervalDays: 30 }, collector: { ...DEFAULT_CONFIG.collector, environment: "ci", machineId: "m-1" }, }; writeConfig(config); diff --git a/__tests__/hooks/fp-reset.test.ts b/__tests__/hooks/fp-reset.test.ts index 5e82044ec..0429e736b 100644 --- a/__tests__/hooks/fp-reset.test.ts +++ b/__tests__/hooks/fp-reset.test.ts @@ -42,15 +42,57 @@ import { let home: string; let prev: string | undefined; +let prevUnitDir: string | undefined; +let prevDaemonBinary: string | undefined; + beforeEach(() => { prev = process.env.FAILPROOFAI_HOME; home = mkdtempSync(resolve(tmpdir(), "fpai-reset-")); process.env.FAILPROOFAI_HOME = home; + + // FAILPROOFAI_HOME alone does not isolate this file, and the gap made three + // tests here fail on a developer's machine while passing in CI — which is + // exactly how they came to be written off as "known pre-existing". + // + // `checkLayoutForCli()` calls `healDaemonFlag()`, which asks the REAL + // `daemonServiceStatus()` — an `existsSync` on /etc/systemd/system plus a + // `systemctl is-active`. On the machine of anyone who has actually run + // `failproofai config`, that answers "running", and the next line awaits + // `probeDaemonEndToEnd()`: a poll against a socket inside THIS temp home that + // nothing will ever listen on, bounded by DAEMON_PROBE_READY_TIMEOUT_MS = + // 10s. That is twice vitest's 5s default, so the two `daemon.configured:true` + // tests time out. + // + // The timeout is the smaller half. Vitest fails the test but cannot cancel + // the promise, so the probe finishes ~5s later and runs + // `updateConfig({daemon:{configured:false}})`. Every path helper resolves + // FAILPROOFAI_HOME at CALL time, and beforeEach has repointed it by then — so + // that write lands in a LATER test's home. A stray config.json is a layout-4 + // landmark that `detectLayout()` checks before config.toml, so the layout-2 + // home the spool-drain test seeds reads as "current" and the migration it + // asserts never runs. One unisolated read, three failures, two files apart. + prevUnitDir = process.env.FAILPROOFAI_SYSTEMD_DIR; + process.env.FAILPROOFAI_SYSTEMD_DIR = resolve(home, "systemd"); + mkdirSync(resolve(home, "systemd"), { recursive: true }); + + // Defensive, and honestly labelled: `daemonVersionSkew()` returns null + // outright when this is set ("someone named a binary explicitly"), which + // would delete the hint the tests below assert. Removing this block and + // exporting the variable did NOT change the outcome here — the tests that + // care now pin `daemonServiceStatus` themselves — so this is isolation + // against a read that could matter, not a demonstrated fix. It costs nothing + // and removes one more way this file can depend on whose machine it runs on. + prevDaemonBinary = process.env.FAILPROOFAI_DAEMON_BINARY; + delete process.env.FAILPROOFAI_DAEMON_BINARY; }); afterEach(() => { if (prev === undefined) delete process.env.FAILPROOFAI_HOME; else process.env.FAILPROOFAI_HOME = prev; + if (prevUnitDir === undefined) delete process.env.FAILPROOFAI_SYSTEMD_DIR; + else process.env.FAILPROOFAI_SYSTEMD_DIR = prevUnitDir; + if (prevDaemonBinary === undefined) delete process.env.FAILPROOFAI_DAEMON_BINARY; + else process.env.FAILPROOFAI_DAEMON_BINARY = prevDaemonBinary; rmSync(home, { recursive: true, force: true }); }); @@ -551,8 +593,16 @@ describe("checkLayoutForCli", () => { installedDaemon("0.0.1-old"); writeVersionFile({ daemon: "0.0.1-old" }); updateConfig({ daemon: { configured: true } }); + // The unit dir is redirected to a scratch path in `beforeEach`, so the + // real status would read `not-installed` — which clears the flag and + // produces the SELF-HEAL message instead. That message happens to contain + // every string asserted below, so the test would pass while exercising a + // different branch entirely. Pin the status to the state this test names. + const svc = await import("../../src/hooks/daemon-service"); + const stat = vi.spyOn(svc, "daemonServiceStatus").mockReturnValue("stopped"); const text = (await checkLayoutForCli()).lines.join("\n"); + stat.mockRestore(); expect(text).toContain("0.0.1-old"); // Must name the consequence, not just the mismatch: the reason to act now @@ -580,8 +630,13 @@ describe("checkLayoutForCli", () => { it("says nothing about the daemon when there is no skew", async () => { seedLayoutOne(); updateConfig({ daemon: { configured: true } }); + // Same reason as above: without pinning, this asserts silence from a + // machine with no daemon rather than from one whose daemon is fine. + const svc = await import("../../src/hooks/daemon-service"); + const stat = vi.spyOn(svc, "daemonServiceStatus").mockReturnValue("stopped"); const text = (await checkLayoutForCli()).lines.join("\n"); + stat.mockRestore(); expect(text).not.toContain("failproofai update"); }); diff --git a/__tests__/hooks/harness-extra-paths.test.ts b/__tests__/hooks/harness-extra-paths.test.ts index 58c104bac..9cc7ef58b 100644 --- a/__tests__/hooks/harness-extra-paths.test.ts +++ b/__tests__/hooks/harness-extra-paths.test.ts @@ -119,7 +119,7 @@ describe("harness extra paths", () => { redact: "off", }, telemetry: { enabled: false }, - audit: { auto: true, intervalDays: 14 }, + audit: { auto: true, notify: true, intervalDays: 14 }, }); addPath("codex", "alt=/mnt/other/.codex/sessions"); @@ -131,7 +131,7 @@ describe("harness extra paths", () => { expect(cfg.collector.machineId).toBe("m-123"); expect(cfg.collector.redact).toBe("off"); expect(cfg.telemetry.enabled).toBe(false); - expect(cfg.audit).toEqual({ auto: true, intervalDays: 14 }); + expect(cfg.audit).toMatchObject({ auto: true, notify: true, intervalDays: 14 }); expect(cfg.collector.sources?.codex.extraPaths).toEqual(["alt=/mnt/other/.codex/sessions"]); }); diff --git a/__tests__/hooks/opencode-plugin-shim.test.ts b/__tests__/hooks/opencode-plugin-shim.test.ts index 291d6bf9f..2cead6ccf 100644 --- a/__tests__/hooks/opencode-plugin-shim.test.ts +++ b/__tests__/hooks/opencode-plugin-shim.test.ts @@ -83,20 +83,43 @@ async function loadShim(opts: { scope: "user" | "project"; binaryPath: string; c .replace('FAILPROOFAI_BIN = ""', `FAILPROOFAI_BIN = ${JSON.stringify(opts.binaryPath)}`); })(); - // Replace the spawnSync import with our stub. The shim imports it as - // `import { spawnSync } from "node:child_process"`. We rewrite that line - // to read from a global injected by this test. + // Replace the spawn import with our stub. The shim imports it as + // `import { spawn } from "node:child_process"` — it used to be spawnSync, + // but that blocked opencode's in-process TUI event loop for the whole + // subprocess duration. The verdicts are identical; only the wait changed + // from a blocked thread to a promise, so every assertion below is unchanged. const stubbed = shimSource.replace( - 'import { spawnSync } from "node:child_process";', - `const spawnSync = globalThis.__fp_test_spawnSync;`, + 'import { spawn } from "node:child_process";', + `const spawn = globalThis.__fp_test_spawn;`, ); - // Pre-set the stub before importing. - (globalThis as unknown as Record).__fp_test_spawnSync = (cmd: string, args: string[], optsArg: SpawnCall["opts"]): SpawnResult => { - opts.calls.push({ cmd, args, opts: optsArg }); - const r = opts.responses.shift(); - if (!r) return { status: 0, stdout: "", stderr: "" }; - return r; + // Async child stub: records the call (payload arrives on stdin now, not as + // `opts.input`) and delivers the next canned response on a later tick, the + // way a real child does. + (globalThis as unknown as Record).__fp_test_spawn = (cmd: string, args: string[], optsArg: SpawnCall["opts"]) => { + const dataHandlers: Record void)[]> = { stdout: [], stderr: [] }; + const closeHandlers: ((code: number) => void)[] = []; + const mkStream = (which: "stdout" | "stderr") => ({ + on: (ev: string, fn: (d: string) => void) => { if (ev === "data") dataHandlers[which].push(fn); }, + }); + return { + stdout: mkStream("stdout"), + stderr: mkStream("stderr"), + kill: () => {}, + on: (ev: string, fn: (code: number) => void) => { if (ev === "close") closeHandlers.push(fn); }, + stdin: { + on: () => {}, + end: (payload?: string) => { + opts.calls.push({ cmd, args, opts: { ...optsArg, input: payload } as SpawnCall["opts"] }); + const r = opts.responses.shift() ?? { status: 0, stdout: "", stderr: "" }; + queueMicrotask(() => { + if (r.stdout) for (const fn of dataHandlers.stdout) fn(r.stdout); + if (r.stderr) for (const fn of dataHandlers.stderr) fn(r.stderr); + for (const fn of closeHandlers) fn(r.status ?? 0); + }); + }, + }, + }; }; // Write a sibling .mjs we can dynamic-import without touching the original. @@ -116,7 +139,10 @@ async function loadShim(opts: { scope: "user" | "project"; binaryPath: string; c } function fakeClient() { - return { session: { prompt: vi.fn().mockResolvedValue(undefined) } }; + return { + session: { prompt: vi.fn().mockResolvedValue(undefined) }, + tui: { showToast: vi.fn().mockResolvedValue(undefined) }, + }; } describe("OpenCode plugin shim — translation of plugin events to binary stdin", () => { @@ -416,6 +442,42 @@ describe("OpenCode plugin shim — translation of binary response to plugin acti expect(callArg.body.parts[0]).toEqual({ type: "text", text: "Note: hello" }); }); + it("failproofaiNotice → uses OpenCode's visible TUI toast API", async () => { + responses.push({ + status: 0, + stdout: JSON.stringify({ failproofaiNotice: "Review possible credential exposure" }), + stderr: "", + }); + const client = fakeClient(); + const { plugin } = await setup(); + const hooks = await plugin({ client, directory: "/repo" }); + await hooks.event!({ event: { type: "session.idle", properties: { sessionID: "ses_1" } } }); + expect(client.tui.showToast).toHaveBeenCalledWith({ + body: { + title: "failproofai audit", + message: "Review possible credential exposure", + variant: "warning", + duration: 12_000, + }, + }); + expect(client.session.prompt).not.toHaveBeenCalled(); + }); + + it("swallows an OpenCode toast failure", async () => { + responses.push({ + status: 0, + stdout: JSON.stringify({ failproofaiNotice: "Review possible credential exposure" }), + stderr: "", + }); + const client = fakeClient(); + client.tui.showToast.mockRejectedValue(new Error("TUI closed")); + const { plugin } = await setup(); + const hooks = await plugin({ client, directory: "/repo" }); + await expect( + hooks.event!({ event: { type: "session.idle", properties: { sessionID: "ses_1" } } }), + ).resolves.toBeUndefined(); + }); + it("SDK rejection on session.prompt is swallowed (fire-and-forget)", async () => { responses.push({ status: 0, @@ -551,15 +613,20 @@ describe("OpenCode plugin shim — spawn options and registration", () => { afterEach(() => cleanup()); - it("spawnSync includes timeout, encoding, and cwd", async () => { + // Was "spawnSync includes timeout, encoding, and cwd". The shim now uses the + // async `spawn`, which takes neither `encoding` (chunks are concatenated as + // they arrive) nor `timeout` (enforced by the shim's own 60s timer that + // SIGKILLs and fails open). `cwd` is still passed through, and the payload + // now arrives on stdin rather than as `opts.input` — both asserted here so a + // regression back to a blocking spawn is visible. + it("passes cwd and delivers the payload on stdin", async () => { responses.push({ status: 0, stdout: "", stderr: "" }); const r = await loadShim({ scope: "project", binaryPath: "/abs/bin/failproofai", calls, responses }); cleanup = r.cleanup; const hooks = await r.plugin({ client: fakeClient(), directory: "/some/cwd" }); await hooks["tool.execute.before"]!({ tool: "bash", sessionID: "s", callID: "c" }, { args: {} }); - expect(calls[0].opts.timeout).toBe(60_000); - expect(calls[0].opts.encoding).toBe("utf8"); expect(calls[0].opts.cwd).toBe("/some/cwd"); + expect(JSON.parse(String(calls[0].opts.input)).hook_event_name).toBe("PreToolUse"); }); it("registers exactly the expected hook keys", async () => { diff --git a/__tests__/hooks/pi-extension-shim.test.ts b/__tests__/hooks/pi-extension-shim.test.ts index 1f4a4d33e..05aaf26e1 100644 --- a/__tests__/hooks/pi-extension-shim.test.ts +++ b/__tests__/hooks/pi-extension-shim.test.ts @@ -18,8 +18,12 @@ interface CapturedCall { args: string[]; } +interface PiExtensionContext { + ui?: { notify(message: string, type?: "info" | "warning" | "error"): void }; +} + interface PiExtensionApi { - on(event: string, handler: (event: unknown) => unknown): void; + on(event: string, handler: (event: unknown, ctx?: PiExtensionContext) => unknown): void; } const captured: CapturedCall[] = []; @@ -30,6 +34,10 @@ const captured: CapturedCall[] = []; * in the map gets the default empty stdout. */ const mockSpawnReplyByEvent: Record = {}; +/** Which child_process API each event actually used. Pins the blocking vs + * non-blocking split — see the `spawn` mock below. */ +const spawnApiByEvent: Record = {}; + function eventNameFromArgs(args: string[]): string | undefined { const i = args.indexOf("--hook"); return i >= 0 ? args[i + 1] : undefined; @@ -38,10 +46,33 @@ function eventNameFromArgs(args: string[]): string | undefined { vi.mock("node:child_process", () => ({ spawnSync: (_cmd: string, args: string[], opts: { input?: string }) => { captured.push({ args: args ?? [], payload: JSON.parse(opts?.input ?? "{}") }); + spawnApiByEvent[eventNameFromArgs(args ?? []) ?? "?"] = "spawnSync"; const evt = eventNameFromArgs(args ?? []); const stdout = (evt && mockSpawnReplyByEvent[evt]) ?? ""; return { pid: 0, output: [], status: 0, signal: null, stderr: "", stdout }; }, + // `session_start`, `tool_result` and `session_shutdown` go through the + // detached `forwardPolicy` instead of `callPolicy`, because Pi awaits its + // handlers serially and those three discard the verdict — blocking the TUI + // on a subprocess whose answer is thrown away. They still have to deliver + // the same payload, so the mock captures them into the same array and every + // assertion below is unchanged. The payload arrives on stdin rather than as + // `opts.input`. + spawn: (_cmd: string, args: string[]) => { + let stdinBuf = ""; + return { + unref: () => {}, + on: () => {}, + stdin: { + on: () => {}, + end: (chunk?: string) => { + stdinBuf += chunk ?? ""; + captured.push({ args: args ?? [], payload: JSON.parse(stdinBuf || "{}") }); + spawnApiByEvent[eventNameFromArgs(args ?? []) ?? "?"] = "spawn"; + }, + }, + }; + }, })); function piEncodeCwd(cwd: string): string { @@ -49,7 +80,7 @@ function piEncodeCwd(cwd: string): string { } describe("pi-extension shim — sessionId resolution via on-disk discovery", () => { - let handlers: Record unknown> = {}; + let handlers: Record unknown> = {}; let bridge: (pi: PiExtensionApi) => void; let piRoot: string; let originalEnv: string | undefined; @@ -118,6 +149,33 @@ describe("pi-extension shim — sessionId resolution via on-disk discovery", () expect(captured.at(-1)?.payload.session_id).toBe(sid); }); + // Pi awaits its handlers serially, so a `spawnSync` inside one freezes the + // whole TUI for the subprocess's duration — measured at a 1.0-1.6s floor on + // every session start just to boot the binary, with a 60s ceiling. Three + // events discarded the verdict anyway, so they now forward detached. The + // other four consume the decision, and blocking there IS the enforcement. + it("only blocks Pi on the events whose verdict it actually reads", () => { + const sid = "66666666-6666-6666-6666-666666666666"; + writeSessionFile("/proj", sid); + for (const k of Object.keys(spawnApiByEvent)) delete spawnApiByEvent[k]; + + handlers.session_start({ type: "session_start", cwd: "/proj" }); + handlers.tool_result({ type: "tool_result", toolName: "bash", input: {}, content: [], isError: false, cwd: "/proj" }); + handlers.session_shutdown({ type: "session_shutdown", reason: "quit", cwd: "/proj" }); + expect(spawnApiByEvent.session_start).toBe("spawn"); + expect(spawnApiByEvent.tool_result).toBe("spawn"); + expect(spawnApiByEvent.session_shutdown).toBe("spawn"); + + handlers.tool_call({ type: "tool_call", toolName: "bash", input: { command: "ls" }, cwd: "/proj" }); + handlers.user_bash({ type: "user_bash", command: "ls", cwd: "/proj" }); + handlers.input({ type: "input", text: "hi", cwd: "/proj" }); + handlers.agent_end({ type: "agent_end", cwd: "/proj" }); + expect(spawnApiByEvent.tool_call).toBe("spawnSync"); + expect(spawnApiByEvent.user_bash).toBe("spawnSync"); + expect(spawnApiByEvent.input).toBe("spawnSync"); + expect(spawnApiByEvent.agent_end).toBe("spawnSync"); + }); + it("clears the per-cwd cache on session_shutdown reason=new/resume/fork", () => { const sid1 = "11111111-1111-1111-1111-111111111111"; const sid2 = "22222222-2222-2222-2222-222222222222"; @@ -260,7 +318,7 @@ describe("pi-extension shim — sessionId resolution via on-disk discovery", () * suffix on the next `before_agent_start`. These tests cover that handoff. */ describe("pi-extension shim — agent_end → before_agent_start stop-block handoff", () => { - let handlers: Record unknown> = {}; + let handlers: Record unknown> = {}; let piRoot: string; let originalEnv: string | undefined; const SID = "ffffffff-ffff-ffff-ffff-ffffffffffff"; @@ -305,6 +363,19 @@ describe("pi-extension shim — agent_end → before_agent_start stop-block hand ); }); + it("agent_end notice uses Pi's visible UI notification API", () => { + mockSpawnReplyByEvent["agent_end"] = JSON.stringify({ + permission: "allow", + failproofaiNotice: "Review possible credential exposure", + }); + const notify = vi.fn(); + handlers.agent_end( + { type: "agent_end", cwd: "/proj" }, + { ui: { notify } }, + ); + expect(notify).toHaveBeenCalledWith("Review possible credential exposure", "warning"); + }); + it("before_agent_start with no pending block returns undefined", () => { const result = handlers.before_agent_start({ type: "before_agent_start", diff --git a/__tests__/hooks/policy-catalog.test.ts b/__tests__/hooks/policy-catalog.test.ts index 9d4529b0c..e110c37cf 100644 --- a/__tests__/hooks/policy-catalog.test.ts +++ b/__tests__/hooks/policy-catalog.test.ts @@ -185,7 +185,10 @@ describe("policy catalog / implementation split", () => { // Its hand-written most-specific-first ORDER is load-bearing — a // Bearer-wrapped JWT reports as "JWT" today and as "bearer token" if two // entries swap. - expect(SECRET_PATTERNS).toHaveLength(13); + // 33 as of the pattern-census expansion (was 13). The count is pinned so + // an accidental deletion is loud; raise it deliberately when adding a + // vendor, and only with a doc-verified prefix behind it. + expect(SECRET_PATTERNS).toHaveLength(37); for (const [re] of SECRET_PATTERNS) expect(re).toBeInstanceOf(RegExp); }); }); diff --git a/__tests__/lib/claude-sessions-subagents.test.ts b/__tests__/lib/claude-sessions-subagents.test.ts new file mode 100644 index 000000000..d58287803 --- /dev/null +++ b/__tests__/lib/claude-sessions-subagents.test.ts @@ -0,0 +1,114 @@ +// @vitest-environment node +/** + * The subagent walk below `/subagents/`. + * + * This exists because the enumerator read only the DIRECT children of that + * directory, which was right for the layout Claude shipped when it was written + * and silently wrong once workflow runs began nesting their agents one level + * further down. On the machine the miss was found on it cost 91% of the corpus: + * 1,839 transcripts on disk, 160 enumerated. The audit reported that result as + * though it had read everything, which is the failure mode these tests exist to + * keep closed — a scan that finds nothing and a scan that looks nowhere are + * indistinguishable from the outside. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync, symlinkSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { listClaudeProjects, listClaudeTranscripts } from "@/lib/claude-sessions"; + +const PARENT_A = "aaaaaaaa-1111-4111-8111-aaaaaaaaaaaa"; +const PARENT_B = "bbbbbbbb-2222-4222-8222-bbbbbbbbbbbb"; + +let root: string; +let prevEnv: string | undefined; + +/** Write a transcript at `/<...segments>`, creating parents. */ +function transcript(project: string, ...segments: string[]): string { + const path = join(root, project, ...segments); + mkdirSync(join(path, ".."), { recursive: true }); + writeFileSync(path, '{"type":"user"}\n'); + return path; +} + +function allTranscripts() { + return listClaudeProjects().flatMap((p) => listClaudeTranscripts(p)); +} + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "fpai-subagents-")); + prevEnv = process.env.CLAUDE_PROJECTS_PATH; + process.env.CLAUDE_PROJECTS_PATH = root; +}); + +afterEach(() => { + if (prevEnv === undefined) delete process.env.CLAUDE_PROJECTS_PATH; + else process.env.CLAUDE_PROJECTS_PATH = prevEnv; + rmSync(root, { recursive: true, force: true }); +}); + +describe("listClaudeTranscripts — subagent nesting", () => { + it("finds a transcript nested under subagents/workflows//", () => { + transcript("-home-u-proj", `${PARENT_A}.jsonl`); + transcript("-home-u-proj", PARENT_A, "subagents", "agent-direct.jsonl"); + transcript("-home-u-proj", PARENT_A, "subagents", "workflows", "wf_123-abc", "agent-nested.jsonl"); + + const found = allTranscripts(); + + // The regression: the nested one used to be dropped entirely. + expect(found).toHaveLength(3); + expect(found.filter((t) => t.isSubagent)).toHaveLength(2); + expect(found.map((t) => t.transcriptPath).some((p) => p.includes("wf_123-abc"))).toBe(true); + }); + + it("keeps the top-level session id untouched", () => { + transcript("-home-u-proj", `${PARENT_A}.jsonl`); + const top = allTranscripts().find((t) => !t.isSubagent); + expect(top?.sessionId).toBe(PARENT_A); + }); + + it("qualifies a subagent id with its parent session and its path", () => { + transcript("-home-u-proj", PARENT_A, "subagents", "agent-direct.jsonl"); + transcript("-home-u-proj", PARENT_A, "subagents", "workflows", "wf_123-abc", "agent-nested.jsonl"); + + const ids = allTranscripts().map((t) => t.sessionId).sort(); + expect(ids).toEqual([ + `${PARENT_A}__agent-direct`, + `${PARENT_A}__workflows__wf_123-abc__agent-nested`, + ]); + }); + + it("does not collide when one workflow run id appears under two parent sessions", () => { + // Found in the real corpus, not imagined: a resumed session reuses its run + // id, so `wf_/journal.jsonl` exists under two parents in one project. + // Deriving the id from the path below `subagents/` alone merged them, and + // sessionId is what example attribution and per-session detector state are + // keyed by — so the merge would be silent rather than an error. + transcript("-home-u-proj", PARENT_A, "subagents", "workflows", "wf_shared", "journal.jsonl"); + transcript("-home-u-proj", PARENT_B, "subagents", "workflows", "wf_shared", "journal.jsonl"); + + const found = allTranscripts(); + expect(found).toHaveLength(2); + expect(new Set(found.map((t) => t.sessionId)).size).toBe(2); + }); + + it("ignores non-transcript files and does not follow symlinks", () => { + transcript("-home-u-proj", PARENT_A, "subagents", "agent-real.jsonl"); + writeFileSync(join(root, "-home-u-proj", PARENT_A, "subagents", "notes.txt"), "x"); + + // A symlink pointing back up would make a naive walk loop forever. + const subDir = join(root, "-home-u-proj", PARENT_A, "subagents"); + symlinkSync(join(root, "-home-u-proj"), join(subDir, "loop"), "dir"); + + const found = allTranscripts(); + expect(found).toHaveLength(1); + expect(found[0].sessionId).toBe(`${PARENT_A}__agent-real`); + }); + + it("stops descending past the depth cap instead of walking forever", () => { + const deep = ["subagents", "a", "b", "c", "d", "e", "f", "g"]; + transcript("-home-u-proj", PARENT_A, ...deep, "agent-too-deep.jsonl"); + expect(allTranscripts()).toHaveLength(0); + }); +}); diff --git a/app/actions/get-leaks.ts b/app/actions/get-leaks.ts new file mode 100644 index 000000000..bbaaed2fd --- /dev/null +++ b/app/actions/get-leaks.ts @@ -0,0 +1,109 @@ +"use server"; + +import { + activeFindings, + dismissFinding, + markLeakReportViewed, + readLeakRecord, +} from "@/src/audit/leak-store"; +import { readConfig, updateConfig } from "@/src/hooks/fp-config"; + +/** + * One credential, flattened for the report table. + * + * A separate shape from `LeakFinding` on purpose: this crosses into a client + * component, so it carries only what is drawn. The finding's raw sightings, its + * rule and its salt stay on the server side of the boundary — not because any + * of them is a secret (the record holds no secret by construction), but because + * a display type that mirrors the storage type drifts into rendering whatever + * gets added to storage next. + */ +export interface LeakRow { + id: string; + /** WHAT — `ghp_••••••••4f2a`. Never the value; there is no value to send. */ + display: string; + label: string; + length: number; + /** False means no console to revoke at — the advice changes completely. */ + attributed: boolean; + /** The identifier it was assigned to, when there was one. */ + name: string | null; + /** WHO — the harness whose transcript carried it. */ + cli: string; + /** WHERE — home-shortened project path. */ + project: string; + /** WHEN — ISO, formatted client-side so it lands in the reader's timezone. */ + lastSeen: string; + firstSeen: string; + /** HOW — "read from ~/…/.env". */ + mechanism: string; + /** WHY IT MATTERS — an input the agent sent can be denied next time; a + * result it received cannot be un-received. */ + direction: "input" | "result"; + occurrences: number; + sessions: number; +} + +export interface LeaksPayload { + rows: LeakRow[]; + /** Whether this machine may raise a desktop notification. Drawn as a toggle + * beside the table, because the table is where somebody decides the banner + * was worth it or not. */ + notify: boolean; +} + +export async function getLeaksAction(): Promise { + const findings = activeFindings(readLeakRecord()); + // Opening the report is the one signal worth silencing the in-CLI notice on. + // Recorded here rather than on a button, because reading it is the action — + // there is nothing further for the user to do to say "I have seen this". + markLeakReportViewed(); + const rows: LeakRow[] = findings.map((f) => { + // The most recent sighting: where the key is now, not where it debuted. + const seen = f.sightings?.[f.sightings.length - 1]; + return { + id: f.id, + display: f.fingerprint?.display ?? "[credential]", + label: f.fingerprint?.label ?? "secret", + length: f.fingerprint?.length ?? 0, + attributed: f.fingerprint?.attributed === true, + name: f.name, + cli: seen?.cli ?? "unknown", + project: seen?.cwd ?? "unknown", + lastSeen: f.lastSeen, + firstSeen: f.firstSeen, + mechanism: seen?.mechanism?.summary ?? "seen in a transcript", + direction: seen?.mechanism?.direction ?? "result", + occurrences: f.occurrences, + sessions: new Set((f.sightings ?? []).map((s) => s.sessionId)).size, + }; + }); + rows.sort((a, b) => Date.parse(b.lastSeen) - Date.parse(a.lastSeen)); + return { rows, notify: readConfig().audit.notify }; +} + +/** + * Mark one finding as not-a-secret. + * + * The finding is RETAINED, not deleted — deleting it would let the next scan + * rediscover the same value and alert again, which is the one outcome that + * teaches a user the dismiss button does not work. + */ +export async function dismissLeakAction(id: string): Promise { + return dismissFinding(id); +} + +/** + * Turn the desktop banner on or off. + * + * Writes the same `audit.notify` key `failproofai config` writes and the daemon's + * audit child reads, so the dashboard and the CLI cannot disagree about it. + */ +export async function setLeakNotifyAction(notify: boolean): Promise { + try { + updateConfig({ audit: { notify } }); + return true; + } catch { + return false; + } +} diff --git a/app/audit/_components/audit-dashboard.tsx b/app/audit/_components/audit-dashboard.tsx index 372325e8a..1c5f65e64 100644 --- a/app/audit/_components/audit-dashboard.tsx +++ b/app/audit/_components/audit-dashboard.tsx @@ -3,8 +3,8 @@ /** * Top-level client wrapper for /audit. * - * Composes the calm personality report: classify the agent into one of - * 8 archetypes, derive a score, and render the 5-section flow: + * The old five-section personality report is SWITCHED OFF and being rebuilt + * bottom-up around leak detection. What it used to compose: * * 01 AuditPoster — single-screen shareable poster * 02 StrengthsSection — what it's great at @@ -12,23 +12,36 @@ * 04 HowToImproveSection — install / configure * 05 ComeBackBetterSection — spread the audit (invite) * + * Every one of those components, and the score / persona / strengths / findings + * modules behind them, is intact and still unit-tested — only the renders and + * the imports here are commented out, so restoring any of it is deleting a pair + * of comment markers. See the header of `src/audit/scoring.ts` for why. + * + * What this page renders now is `AuditReportPlaceholder`: the scan's own + * numbers. The scan, the 12-CLI transcript reader and the emailed digest all + * still run. + * * Empty / running states fall back to EmptyState and RunProgress. */ import { useCallback, useEffect, useMemo, useRef, useState } from "react"; import { getAuditResultAction } from "@/app/actions/get-audit-result"; import type { AuditResult, RunAuditOptions } from "@/src/audit/types"; -import { classifyAgent } from "@/src/audit/archetypes"; -import { deriveScore, gradeFor, projectedScore } from "@/src/audit/scoring"; -import { deriveStrengths } from "@/src/audit/strengths"; -import { deriveFindings } from "@/src/audit/findings"; +// import { classifyAgent } from "@/src/audit/archetypes"; +// Score switched off — see the header of `src/audit/scoring.ts`. +// import { deriveScore, gradeFor, projectedScore } from "@/src/audit/scoring"; +// import { deriveStrengths } from "@/src/audit/strengths"; +// import { deriveFindings } from "@/src/audit/findings"; import { usePostHog } from "@/contexts/PostHogContext"; -import { AuditPoster } from "./audit-poster"; -import { StrengthsSection } from "./strengths-section"; -import { QuirksSection } from "./quirks-section"; -import { HowToImproveSection } from "./how-to-improve-section"; -import { ComeBackBetterSection } from "./come-back-better-section"; +// The old report's five sections — switched off, not deleted. Each component +// file and its tests are untouched; only this render is commented out. +// import { AuditPoster } from "./audit-poster"; +// import { StrengthsSection } from "./strengths-section"; +// import { QuirksSection } from "./quirks-section"; +// import { HowToImproveSection } from "./how-to-improve-section"; +// import { ComeBackBetterSection } from "./come-back-better-section"; import { ReportFooter } from "./report-footer"; +import { LeakSection } from "./leak-section"; import { EmptyState } from "./empty-state"; import { RunProgress } from "./run-progress"; import { AuditProgressStrip, type RerunStatus } from "./audit-progress-strip"; @@ -46,7 +59,9 @@ type Initial = /** Tag passed to the shared `startRerun()` handler so PostHog can tell * whether the click came from the bottom return-section button or the * empty-state CTA. */ -export type RerunSource = "return_section" | "empty_state"; +// `scan_header` is the button in section 01 — the only re-scan control the +// page has now that the old report's sections are switched off. +export type RerunSource = "return_section" | "empty_state" | "scan_header"; interface Props { initial: Initial; @@ -59,6 +74,9 @@ interface Props { totalCatalogSize: number; } +/* Switched off with the persona layer — this named the project that seeded the + archetype classifier and the leaderboard row. Restore alongside + `classifyAgent` in this file. function inferProjectName(result: AuditResult, override?: string): string { if (override && override.trim()) return override; // Pick the cwd that appears in the most examples — proxy for "your @@ -81,6 +99,7 @@ function inferProjectName(result: AuditResult, override?: string): string { if (segs.length >= 2) return `${segs[segs.length - 2]} / ${segs[segs.length - 1]}`; return segs[segs.length - 1] ?? "your agent"; } +*/ export function AuditDashboard({ initial, projectFromUrl, totalCatalogSize }: Props) { const [cache, setCache] = useState(initial); @@ -279,16 +298,22 @@ function MainReport({ onDismissRerun, }: MainReportProps) { const { capture } = usePostHog(); - const project = useMemo(() => inferProjectName(result, projectFromUrl), [result, projectFromUrl]); + // Only fed the persona classifier, which is switched off with it. + // const project = useMemo(() => inferProjectName(result, projectFromUrl), [result, projectFromUrl]); // Seed classification with the project name so the behaviour fingerprint // (used for tie-breaks + copy variants) is stable per project. - const classification = useMemo(() => classifyAgent(result, project), [result, project]); - const score = useMemo(() => deriveScore(result), [result]); - const projected = useMemo(() => projectedScore(result, score), [result, score]); - const grade = gradeFor(score); - const projectedGrade = gradeFor(projected); - const strengths = useMemo(() => deriveStrengths(result), [result]); - const findings = useMemo(() => deriveFindings(result), [result]); + // Persona classification is switched off with the rest of the old report. Its + // last live use was two telemetry properties describing a persona the product + // no longer shows anyone. + // const classification = useMemo(() => classifyAgent(result, project), [result, project]); + // Score switched off — see the header of `src/audit/scoring.ts`. The four + // functions are intact and still unit-tested; nothing calls them. + // const score = useMemo(() => deriveScore(result), [result]); + // const projected = useMemo(() => projectedScore(result, score), [result, score]); + // const grade = gradeFor(score); + // const projectedGrade = gradeFor(projected); + // const strengths = useMemo(() => deriveStrengths(result), [result]); + // const findings = useMemo(() => deriveFindings(result), [result]); // One pass over result.results: detectors triggered + missing prescribed // policies. Both feed PostHog instrumentation; `missing` also feeds the @@ -310,10 +335,13 @@ function MainReport({ if (dashboardViewedRef.current) return; dashboardViewedRef.current = true; capture("audit_dashboard_viewed", { - score, - grade, - archetype: classification.archetype, - secondary: classification.secondary ?? null, + // Score switched off — see `src/audit/scoring.ts`. `missing` is the + // prescription's denominator and is unaffected, so the copy→install + // funnel keeps both of its ends. + // score, + // grade, + // archetype: classification.archetype, + // secondary: classification.secondary ?? null, missing, transcripts_scanned: result.transcripts.scanned, results_count: result.results.length, @@ -321,10 +349,6 @@ function MainReport({ }); }, [ capture, - score, - grade, - classification.archetype, - classification.secondary, missing, result.transcripts.scanned, result.results.length, @@ -332,19 +356,43 @@ function MainReport({ ]); /** Poster ref — captured to PNG by the poster's share buttons. */ - const posterRef = useRef(null); + // const posterRef = useRef(null); return (
+ onRerun("scan_header")} + /> + {/* The rebuilt report. It reads the leak record rather than this + scan's result, deliberately: a credential found last month and + not touched by today's transcripts is still a credential to + rotate, and a report that showed only the current scan's findings + would go quiet on exactly those. */} + + {/* The whole old report is switched off — the persona poster, the + strengths and quirks sections, the punch-list with its install-all + funnel, and the invite section. It is being rebuilt bottom-up + around leak detection; see the header of `src/audit/scoring.ts` + for the reasoning and `~/Desktop/failproofai-leak-detection- + verdict-2026-09-07.md` for the measurements behind it. + + Every component below is intact and still unit-tested — only the + render is commented — so restoring any one of them is deleting a + pair of comment markers. + @@ -352,10 +400,11 @@ function MainReport({ - + + */}
@@ -363,6 +412,63 @@ function MainReport({ ); } +/** + * What `/audit` shows while the report is being rebuilt. + * + * Deliberately states the scan's own numbers rather than nothing at all: the + * scan still runs, still walks every transcript across all 12 CLIs, and still + * feeds the emailed digest. Saying so is the difference between "this page is + * under construction" and "this product is broken". + */ +function AuditReportPlaceholder( + { + transcripts, + events, + projects, + isRunning, + onRerun, + }: { + transcripts: number; + events: number; + projects: number; + isRunning: boolean; + onRerun: () => void; + }, +) { + const n = (x: number) => x.toLocaleString(); + return ( +
+
+ + 01{"// scan"} + + {/* The re-scan control. It lived in the old report's sections and went + dormant when they were commented out, which left the page with no way + to run an audit again at all — you had to go back to the terminal. */} + +
+

scan complete

+
+ {n(events)} tool call{events === 1 ? "" : "s"} across {n(transcripts)}{" "} + transcript{transcripts === 1 ? "" : "s"} + {projects > 0 ? ` · ${n(projects)} project${projects === 1 ? "" : "s"}` : ""} +
+
+ {"// the report is being rebuilt around leak detection. the scan, the"} +
+ {"// 12-CLI transcript reader and the emailed digest are unaffected."} +
+
+ ); +} + interface ShellEmptyProps { running: boolean; mode?: "no-cache" | "zero-sessions"; diff --git a/app/audit/_components/audit-poster.tsx b/app/audit/_components/audit-poster.tsx index 9822eda34..91784e9e2 100644 --- a/app/audit/_components/audit-poster.tsx +++ b/app/audit/_components/audit-poster.tsx @@ -24,7 +24,12 @@ import React, { forwardRef, useMemo, useState } from "react"; import { pickArchetypeVariant, type ArchetypeKey } from "@/src/audit/archetypes"; import { type Grade } from "@/src/audit/scoring"; -import { getArchetypeRarityPct } from "@/src/audit/social-proof"; +// Archetype rarity switched off — the percentages are eight hardcoded integers +// ("Seeded with snapshot values; swap for live aggregates once that pipeline +// lands"), and this is the ONLY surface that rendered them: baked by +// html-to-image into the PNG people post publicly. A fabricated population +// statistic on a shared card is a claim we cannot support. +// import { getArchetypeRarityPct } from "@/src/audit/social-proof"; import { copyOrDownloadCard, downloadCard, shareCardNative, shareCardToastMessage } from "@/lib/share-card"; import { toast } from "@/app/components/toast"; import { usePostHog } from "@/contexts/PostHogContext"; @@ -45,8 +50,11 @@ interface Props { archetypeKey: ArchetypeKey; /** Stable seed for variant selection (project name is the natural fit). */ seed: string; - score: number; - grade: Grade; + /** The score is switched off (see `src/audit/scoring.ts`). Kept as optional + * props rather than removed, so restoring is un-commenting rather than + * re-threading: the dashboard simply stops passing them. */ + score?: number; + grade?: Grade; /** Count of unenabled prescribed policies — passed to the share-text * templates, not rendered on the poster itself. */ missing: number; @@ -63,7 +71,7 @@ export const AuditPoster = forwardRef(function AuditPoste () => pickArchetypeVariant(archetypeKey, seed), [archetypeKey, seed], ); - const rarityPct = getArchetypeRarityPct(archetypeKey); + // const rarityPct = getArchetypeRarityPct(archetypeKey); const indexLabel = String(archetype.index).padStart(2, "0"); const auditedDate = useMemo(() => formatAuditedDate(auditedAt), [auditedAt]); @@ -134,8 +142,11 @@ export const AuditPoster = forwardRef(function AuditPoste } }; + // Score switched off — the filename is keyed on the archetype instead, which + // is what the card now actually shows. Original: + // `failproofai-${channel}-${grade.toLowerCase()}-${score}.png` const filenameFor = (channel: "x" | "linkedin" | "download") => - `failproofai-${channel}-${grade.toLowerCase()}-${score}.png`; + `failproofai-${channel}-${archetypeKey}.png`; const handleShare = async (channel: "x" | "linkedin" | "download") => { if (busy) return; @@ -143,8 +154,11 @@ export const AuditPoster = forwardRef(function AuditPoste capture("audit_card_share_clicked", { channel, source: "poster", - score, - grade, + // Score switched off — see `src/audit/scoring.ts`. Note for whoever reads + // the funnel: these two properties stop appearing from this release, so a + // dashboard keyed on them breaks rather than reading zero. + // score, + // grade, missing_policies: missing, }); try { @@ -174,10 +188,10 @@ export const AuditPoster = forwardRef(function AuditPoste } const shareCtx: ShareCtx = { - score, arch: archetype.name.toLowerCase(), - grade, missing, + // score, + // grade, }; const shareText = channel === "x" ? pickTemplate(X_TEMPLATES, seed, shareCtx) @@ -248,6 +262,7 @@ export const AuditPoster = forwardRef(function AuditPoste ))}
+ {/* Rarity switched off — see the import above. {typeof rarityPct === "number" && (
{"// only"}{" "} @@ -255,13 +270,17 @@ export const AuditPoster = forwardRef(function AuditPoste of agents are this archetype
)} + */} - {/* Score block — heroic number, centered in the card */} + {/* Score block — heroic number, centered in the card. Switched off; + see `src/audit/scoring.ts` for why. The archetype is the card's + headline now.
{score} /100
+ */}