From 7c4384303d0c815135b280c508f2f6b7381db159 Mon Sep 17 00:00:00 2001 From: rwrife Date: Sun, 5 Jul 2026 20:59:31 +0000 Subject: [PATCH] Add secret tripwire: sp scan + ls marker + promote guard (#9) --- README.md | 55 +++- cmd/sp/main.go | 7 + internal/cli/ls.go | 37 ++- internal/cli/promote.go | 54 +++- internal/cli/promote_secret_test.go | 149 ++++++++++ internal/cli/root.go | 6 +- internal/cli/scan.go | 117 ++++++++ internal/cli/scan_test.go | 90 ++++++ internal/render/json.go | 21 +- internal/render/render.go | 42 ++- internal/render/scan.go | 132 +++++++++ internal/render/scan_test.go | 105 +++++++ internal/secret/secret.go | 329 ++++++++++++++++++++++ internal/secret/secret_test.go | 183 ++++++++++++ internal/secret/testdata/bearer.txt | 5 + internal/secret/testdata/clean.txt | 10 + internal/secret/testdata/dotenv.txt | 6 + internal/secret/testdata/placeholders.txt | 8 + internal/secret/testdata/private_key.txt | 5 + 19 files changed, 1338 insertions(+), 23 deletions(-) create mode 100644 internal/cli/promote_secret_test.go create mode 100644 internal/cli/scan.go create mode 100644 internal/cli/scan_test.go create mode 100644 internal/render/scan.go create mode 100644 internal/render/scan_test.go create mode 100644 internal/secret/secret.go create mode 100644 internal/secret/secret_test.go create mode 100644 internal/secret/testdata/bearer.txt create mode 100644 internal/secret/testdata/clean.txt create mode 100644 internal/secret/testdata/dotenv.txt create mode 100644 internal/secret/testdata/placeholders.txt create mode 100644 internal/secret/testdata/private_key.txt diff --git a/README.md b/README.md index f02fce6..cb3c95f 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ sp doctor # check store health (orphans, missing file sp doctor --json | jq -e '.healthy' # gate a script on store health sp ls --json | jq '.[].id' # machine-readable output for scripting sp completion zsh > "${fpath[1]}/_sp" # tab-completion for your shell +sp scan # tripwire: does this scratch hold a secret? sp resurrect # changed your mind? pull it back sp promote # the good ones: graduate a scratch into your repo ``` @@ -34,10 +35,12 @@ sp promote # the good ones: graduate a scratch into yo `sp new` and `sp ls` are implemented (M3); the full lifecycle — `sp cat`, `sp open`, `sp rm` (soft-delete), `sp resurrect`, and `sp ls --morgue` (M4); **automatic reaping** — `sp reap`, with human-friendly TTLs and a -`--dry-run` preview (M5); and a read-only **`sp doctor`** store health check -plus scripting polish — **`sp ls --json`** / **`sp doctor --json`** and -**`sp completion`** for bash/zsh/fish, and **`sp promote`** to graduate a scratch -into your repo (M6, in progress). +`--dry-run` preview (M5); a read-only **`sp doctor`** store health check; +scripting polish — **`sp ls --json`** / **`sp doctor --json`** and +**`sp completion`** for bash/zsh/fish; **`sp promote`** to graduate a scratch +into your repo; and a **secret tripwire** — **`sp scan`** flags scratches that +look like they hold credentials, `sp ls` marks them with a 🔑, and `sp promote` +refuses them unless you pass `--allow-secrets` (M6, in progress). ### `sp new [name]` @@ -128,6 +131,7 @@ sp promote 1a2b ./notes # into a directory: ./notes/. sp promote 1a2b keep.md # to an explicit path (renames on the way out) sp promote 1a2b keep.md --force # overwrite an existing destination sp promote 1a2b --no-open # don't open it in $EDITOR afterwards +sp promote 1a2b --allow-secrets # promote even if the secret tripwire flags it ``` - With no `dest`, the file lands in the current directory under a slug of the @@ -136,6 +140,8 @@ sp promote 1a2b --no-open # don't open it in $EDITOR afterwards full target path. - Promoting **never overwrites** an existing file without `--force`, and a refused promote leaves the scratch untouched in the store. +- Promoting a scratch that trips the **secret tripwire** is refused unless you + pass `--allow-secrets` — run `sp scan ` to see the masked findings first. - After moving, the promoted file opens in `$EDITOR` (skip with `--no-open`); a missing `$EDITOR` is not fatal — the move already happened. @@ -231,6 +237,47 @@ never null). Gate a script on the store's health without parsing prose: `sp doctor --json | jq -e '.healthy'`, or list drift with `sp doctor --json | jq '.orphans[].path'`. +### `sp scan ` — the secret tripwire + +AI coding agents and tired humans leak API keys and `.env` dumps into throwaway +files without thinking. `sp scan` runs a conservative heuristic detector over a +single scratch and reports anything that looks like a credential — **with the +values masked**. It never echoes a full secret back to your terminal. + +It catches: + +- **AWS access key ids** (`AKIA…`/`ASIA…` + the fixed-length tail), +- **PEM private-key headers** (`-----BEGIN … PRIVATE KEY-----`), +- **secret-looking assignments** — `API_KEY=`, `TOKEN=`, `SECRET=`, + `PASSWORD=` and friends with a non-placeholder value, +- **long high-entropy tokens** that look generated (bearer tokens, opaque keys). + +The heuristics are deliberately conservative to avoid alarm fatigue: template +values like `API_KEY=changeme`, `TOKEN=`, and `${VAR}` references +stay quiet, as do ordinary prose, long numbers, and URLs. + +```bash +sp scan 1a2b # report masked findings by line number +sp scan 1a2b --no-color # plain, script-friendly +sp scan 1a2b --json # stable JSON object (no color, no flavor) +sp scan 1a2b || echo blocked # non-zero exit when secrets are found +``` + +A clean scratch prints a one-line bill of health and exits `0`. A tripped one +lists each finding as `L ` and exits **non-zero**, so +`sp scan` slots straight into pre-commit hooks and CI. The `--json` form carries +a top-level `tripped` flag and a `findings` array (always an array, never null; +each finding has `kind`, `line`, `rule`, and a `masked` preview — never the raw +value): `sp scan 1a2b --json | jq -e '.tripped | not'`. + +The tripwire also shows up where it matters most: + +- **`sp ls`** puts a 🔑 next to any scratch that trips, and `sp ls --json` sets + `"secret": true` on it. +- **`sp promote`** refuses to graduate a tripped scratch into your repo unless + you pass `--allow-secrets` — the last line of defense before a leaked key + lands somewhere it might get committed. + ### `sp completion ` Prints a shell completion script to stdout so `sp`'s commands and flags diff --git a/cmd/sp/main.go b/cmd/sp/main.go index 4027dfb..471b369 100644 --- a/cmd/sp/main.go +++ b/cmd/sp/main.go @@ -3,6 +3,7 @@ package main import ( + "errors" "fmt" "os" @@ -11,6 +12,12 @@ import ( func main() { if err := cli.NewRootCommand().Execute(); err != nil { + // `sp scan` signals "secrets found" with a message-less sentinel so it + // can gate hooks/CI via exit code without a redundant stderr line — the + // scan report already said everything. Exit non-zero, but stay quiet. + if errors.Is(err, cli.ErrSecretsFound) { + os.Exit(1) + } fmt.Fprintln(os.Stderr, err) os.Exit(1) } diff --git a/internal/cli/ls.go b/internal/cli/ls.go index e0284b4..e00fa4a 100644 --- a/internal/cli/ls.go +++ b/internal/cli/ls.go @@ -7,7 +7,9 @@ import ( "github.com/spf13/cobra" + "github.com/rwrife/scratchpatch/internal/index" "github.com/rwrife/scratchpatch/internal/render" + "github.com/rwrife/scratchpatch/internal/secret" "github.com/rwrife/scratchpatch/internal/store" ) @@ -26,6 +28,9 @@ func newLsCommand() *cobra.Command { "text with no color codes.\n\n" + "Pass --morgue to list soft-deleted scratches instead, showing how long\n" + "until each is purged for good.\n\n" + + "A 🔑 next to a scratch's name means it tripped the secret tripwire — run\n" + + "`sp scan ` to see the masked findings. Such scratches can't be\n" + + "promoted into a repo without --allow-secrets.\n\n" + "Pass --json for a stable, machine-readable array (no color, no flavor)\n" + "suitable for scripting: `sp ls --json | jq '.[].id'`.", Args: cobra.NoArgs, @@ -74,10 +79,38 @@ func runLs(cmd *cobra.Command, noColor, morgue, asJSON bool) error { return err } + // Flag any live scratch that trips the secret tripwire so `sp ls` shows a + // 🔑 next to it (and --json carries "secret": true). Scanning is best-effort: + // a scratch whose content can't be read just goes unflagged rather than + // failing the whole listing. + markers := secretMarkers(st, scratches) + if asJSON { - return render.TableJSON(out, scratches, now) + return render.TableMarkedJSON(out, scratches, markers, now) + } + return render.TableMarked(out, scratches, markers, now, color) +} + +// secretMarkers scans each scratch's content and returns the set of ids that +// tripped the secret tripwire, for `sp ls` to mark. It reads content directly +// and swallows per-scratch read errors: a listing should never fail because one +// file went missing, and doctor is the command that reports such drift. Returns +// nil when nothing tripped so the render layer can skip marking entirely. +func secretMarkers(st *store.Store, scratches []index.Scratch) map[string]bool { + var markers map[string]bool + for _, sc := range scratches { + content, err := st.ReadContent(sc) + if err != nil { + continue + } + if secret.Tripped(content) { + if markers == nil { + markers = make(map[string]bool) + } + markers[sc.ID] = true + } } - return render.Table(out, scratches, now, color) + return markers } // isTerminal reports whether w is a character device (a TTY), which is our diff --git a/internal/cli/promote.go b/internal/cli/promote.go index 0ba5182..6565246 100644 --- a/internal/cli/promote.go +++ b/internal/cli/promote.go @@ -9,13 +9,15 @@ import ( "github.com/spf13/cobra" "github.com/rwrife/scratchpatch/internal/index" + "github.com/rwrife/scratchpatch/internal/secret" "github.com/rwrife/scratchpatch/internal/store" ) // promoteFlags holds the parsed `sp promote` options. type promoteFlags struct { - force bool - noOpen bool + force bool + noOpen bool + allowSecrets bool } func newPromoteCommand() *cobra.Command { @@ -32,7 +34,11 @@ func newPromoteCommand() *cobra.Command { "If [dest] is an existing directory the file is placed inside it under a\n" + "slug of its name; otherwise [dest] is the full target path. Promoting\n" + "never overwrites an existing file unless --force is given. The id may be\n" + - "an unambiguous prefix.", + "an unambiguous prefix.\n\n" + + "Before moving, promote runs the secret tripwire over the scratch and\n" + + "refuses to graduate one that looks like it holds a credential (API keys,\n" + + "private keys, `TOKEN=`/`SECRET=` assignments). Run `sp scan ` to see\n" + + "the masked findings, or pass --allow-secrets to promote it anyway.", Args: cobra.RangeArgs(1, 2), RunE: func(cmd *cobra.Command, args []string) error { dest := "" @@ -45,6 +51,7 @@ func newPromoteCommand() *cobra.Command { cmd.Flags().BoolVar(&f.force, "force", false, "overwrite the destination if a file is already there") cmd.Flags().BoolVar(&f.noOpen, "no-open", false, "don't open the promoted file in $EDITOR after moving") + cmd.Flags().BoolVar(&f.allowSecrets, "allow-secrets", false, "promote even if the scratch trips the secret tripwire") return cmd } @@ -59,6 +66,19 @@ func runPromote(cmd *cobra.Command, ref, dest string, f promoteFlags) error { return err } + // Secret tripwire: refuse to graduate a scratch that looks like it holds a + // credential into the working tree, unless the user explicitly overrides. + // This is the last line of defense before a leaked key lands in a repo where + // it might get committed. Checked before any move so a blocked promote + // changes nothing. + if !f.allowSecrets { + if blocked, serr := promoteSecretGuard(st, sc); serr != nil { + return serr + } else if blocked != nil { + return blocked + } + } + target, err := promoteTarget(sc, dest) if err != nil { return err @@ -158,3 +178,31 @@ func promoteError(err error, target string) error { } return err } + +// promoteSecretGuard runs the secret tripwire over the scratch's content and, +// if it trips, returns a blocking error explaining how to inspect (`sp scan`) +// or override (--allow-secrets). A nil error and nil block mean the scratch is +// clean (or its content couldn't be read as text, in which case we don't block +// on a read error — promote itself will surface any real content problem). The +// error deliberately names the finding count but not the secret values; use +// `sp scan` to see the masked details. +func promoteSecretGuard(st *store.Store, sc index.Scratch) (blocked error, err error) { + content, rerr := st.ReadContent(sc) + if rerr != nil { + // Don't turn a content-read problem into a secret block; let the actual + // promote path report it. Returning nil,nil means "not blocked here". + return nil, nil + } + findings := secret.Scan(content) + if len(findings) == 0 { + return nil, nil + } + n := "secret" + if len(findings) != 1 { + n = "secrets" + } + return fmt.Errorf( + "refusing to promote %s (%s): %d %s detected \u2014 run `sp scan %s` to see "+ + "the (masked) findings, or pass --allow-secrets to promote anyway", + sc.ID, displayName(sc), len(findings), n, sc.ID), nil +} diff --git a/internal/cli/promote_secret_test.go b/internal/cli/promote_secret_test.go new file mode 100644 index 0000000..d7ba847 --- /dev/null +++ b/internal/cli/promote_secret_test.go @@ -0,0 +1,149 @@ +package cli + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// newScratchIDExt creates a named scratch with an explicit extension and digs +// its id back out by globbing for that ext, so tests that need two coexisting +// scratches can tell them apart (the default newScratchID globs *.txt and is +// only reliable for one txt scratch at a time). +func (s *session) newScratchIDExt(name, ext string) string { + s.t.Helper() + if out, err := s.run("new", name, "--no-edit", "--ext", ext); err != nil { + s.t.Fatalf("new %q .%s: %v (out=%s)", name, ext, err, out) + } + matches, _ := filepath.Glob(filepath.Join(s.home, "scratches", "*."+ext)) + if len(matches) == 0 { + s.t.Fatalf("no .%s scratch created for %q", ext, name) + } + base := filepath.Base(matches[len(matches)-1]) + return strings.TrimSuffix(base, "."+ext) +} + +// writeScratchBodyExt writes body into the live content file for id with the +// given ext. +func (s *session) writeScratchBodyExt(id, ext, body string) string { + s.t.Helper() + path := filepath.Join(s.home, "scratches", id+"."+ext) + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + s.t.Fatalf("write scratch body: %v", err) + } + return path +} + +func TestPromoteBlocksScratchWithSecrets(t *testing.T) { + s := newSession(t) + id := s.newScratchID("leaky-promote") + s.writeScratchBody(id, dotenvBody) + + dest := filepath.Join(t.TempDir(), "escaped.env") + _, err := s.run("promote", id, dest) + if err == nil { + t.Fatal("promote should refuse a scratch that trips the secret tripwire") + } + if !strings.Contains(err.Error(), "--allow-secrets") { + t.Errorf("block should point at --allow-secrets; got %v", err) + } + // The block must not echo the secret value. + if strings.Contains(err.Error(), "AKIAIOSFODNN7EXAMPLE") { + t.Errorf("promote block leaked a raw secret: %v", err) + } + + // Non-destructive: the file did not escape and the scratch is still listed. + if _, statErr := os.Stat(dest); statErr == nil { + t.Errorf("blocked promote must not create the destination file") + } + lsOut, _ := s.run("ls") + if !strings.Contains(lsOut, "leaky-promote") { + t.Errorf("blocked scratch should remain listed; got %q", lsOut) + } +} + +func TestPromoteAllowSecretsOverridesTheBlock(t *testing.T) { + s := newSession(t) + id := s.newScratchID("override-me") + s.writeScratchBody(id, dotenvBody) + + dest := filepath.Join(t.TempDir(), "kept.env") + out, err := s.run("promote", id, dest, "--allow-secrets") + if err != nil { + t.Fatalf("promote --allow-secrets should succeed; got %v (out=%s)", err, out) + } + if _, statErr := os.Stat(dest); statErr != nil { + t.Errorf("promote --allow-secrets should move the file: %v", statErr) + } + // And the scratch is gone from the store, like any promote. + lsOut, _ := s.run("ls") + if strings.Contains(lsOut, "override-me") { + t.Errorf("promoted scratch should vanish from ls; got %q", lsOut) + } +} + +func TestPromoteCleanScratchStillWorks(t *testing.T) { + // Regression guard: the tripwire must not interfere with normal promotes. + s := newSession(t) + id := s.newScratchID("totally-clean") + s.writeScratchBody(id, "no secrets here, just notes\n") + + dest := filepath.Join(t.TempDir(), "notes.md") + if out, err := s.run("promote", id, dest); err != nil { + t.Fatalf("clean promote should succeed; got %v (out=%s)", err, out) + } +} + +func TestLsMarksScratchesThatTripTheTripwire(t *testing.T) { + s := newSession(t) + // Distinct exts so newScratchID resolves each id unambiguously (the harness + // globs by ext, so two same-ext scratches would collide). + leakyID := s.newScratchIDExt("leaky-ls", "env") + s.writeScratchBodyExt(leakyID, "env", dotenvBody) + cleanID := s.newScratchIDExt("clean-ls", "md") + s.writeScratchBodyExt(cleanID, "md", "harmless\n") + + out, err := s.run("ls") + if err != nil { + t.Fatalf("ls: %v", err) + } + // The leaky scratch's line carries the 🔑 marker; the clean one does not. + // Match on id so the two rows are told apart unambiguously. + var sawLeaky, sawClean bool + for _, line := range strings.Split(out, "\n") { + if strings.Contains(line, leakyID) { + sawLeaky = true + if !strings.Contains(line, "🔑") { + t.Errorf("leaky scratch line should carry the 🔑 marker; got %q", line) + } + } + if strings.Contains(line, cleanID) { + sawClean = true + if strings.Contains(line, "🔑") { + t.Errorf("clean scratch line should not be marked; got %q", line) + } + } + } + if !sawLeaky || !sawClean { + t.Fatalf("expected both scratches listed; sawLeaky=%v sawClean=%v out=%q", sawLeaky, sawClean, out) + } +} + +func TestLsJSONCarriesSecretFlag(t *testing.T) { + s := newSession(t) + leakyID := s.newScratchID("leaky-json-ls") + s.writeScratchBody(leakyID, dotenvBody) + + out, err := s.run("ls", "--json") + if err != nil { + t.Fatalf("ls --json: %v", err) + } + if !strings.Contains(out, "\"secret\": true") { + t.Errorf("ls --json should mark the leaky scratch with secret=true; got %s", out) + } + // JSON stays personality-free — no marker glyph in the machine output. + if strings.Contains(out, "🔑") { + t.Errorf("ls --json must not contain the marker glyph; got %s", out) + } +} diff --git a/internal/cli/root.go b/internal/cli/root.go index c876a8b..ed6ef8c 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -8,9 +8,10 @@ // M6 begins the polish pass with `sp doctor`: a read-only store health check // that reconciles the index against what's actually on disk. M6 also adds // `--json` output to `sp ls` and `sp doctor` for scripting, `sp completion` to -// generate bash/zsh/fish completion scripts, and `sp promote`: graduate a +// generate bash/zsh/fish completion scripts, `sp promote`: graduate a // scratch out of the store into the working tree so the good ones escape the -// reaper. +// reaper, and `sp scan`: a secret tripwire that flags scratches holding +// credentials (also surfaced as a 🔑 in `sp ls` and enforced by `sp promote`). package cli import ( @@ -53,6 +54,7 @@ func NewRootCommand() *cobra.Command { newPromoteCommand(), newReapCommand(), newDoctorCommand(), + newScanCommand(), newCompletionCommand(), ) diff --git a/internal/cli/scan.go b/internal/cli/scan.go new file mode 100644 index 0000000..71ad2a8 --- /dev/null +++ b/internal/cli/scan.go @@ -0,0 +1,117 @@ +package cli + +import ( + "github.com/spf13/cobra" + + "github.com/rwrife/scratchpatch/internal/index" + "github.com/rwrife/scratchpatch/internal/render" + "github.com/rwrife/scratchpatch/internal/secret" + "github.com/rwrife/scratchpatch/internal/store" +) + +// ErrSecretsFound is returned by `sp scan` when a scratch trips the secret +// tripwire. It carries no message: the scan report has already told the user +// everything, so main maps this sentinel to a non-zero exit code WITHOUT +// printing a redundant error line. This is what lets `sp scan ` act as a +// gate in pre-commit hooks and CI (`sp scan x || block`) while staying quiet on +// stderr. +var ErrSecretsFound = errSecretsFoundError{} + +type errSecretsFoundError struct{} + +func (errSecretsFoundError) Error() string { return "" } + +// errSecretsFound is the unexported handle the command returns; it is the same +// value as the exported ErrSecretsFound so callers can errors.Is against either. +var errSecretsFound error = ErrSecretsFound + +func newScanCommand() *cobra.Command { + var noColor bool + var asJSON bool + + cmd := &cobra.Command{ + Use: "scan ", + Short: "Check a scratch for leaked secrets (masked findings)", + Long: "Run the secret tripwire over a single scratch and report anything that\n" + + "looks like a credential: AWS access keys, PEM private-key headers,\n" + + "`*_API_KEY=`/`TOKEN=` style assignments, and long high-entropy tokens.\n\n" + + "Findings are reported by line number with the offending value MASKED —\n" + + "scan never echoes a full secret back to your terminal. It is read-only\n" + + "and changes nothing; it just tells you whether a scratch is safe to\n" + + "`sp promote` into a repo (which blocks on a tripped scratch unless you\n" + + "pass --allow-secrets).\n\n" + + "The id may be an unambiguous prefix, and works on morgued scratches too.\n" + + "Pass --json for a stable, machine-readable object (no color, no flavor)\n" + + "suitable for scripting: `sp scan --json | jq '.tripped'`.\n\n" + + "Exit status is non-zero when secrets are found, so scan slots into\n" + + "pre-commit hooks and CI: `sp scan || echo blocked`.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + return runScan(cmd, args[0], noColor, asJSON) + }, + } + + cmd.Flags().BoolVar(&noColor, "no-color", false, "force plain output even on a TTY") + cmd.Flags().BoolVar(&asJSON, "json", false, "emit a JSON object instead of a report (for scripting)") + + return cmd +} + +func runScan(cmd *cobra.Command, ref string, noColor, asJSON bool) error { + st, err := store.Open() + if err != nil { + return err + } + sc, err := resolve(st, ref) + if err != nil { + return err + } + content, err := st.ReadContent(sc) + if err != nil { + return err + } + + findings := secret.Scan(content) + data := toScanReport(sc, findings) + + out := cmd.OutOrStdout() + if asJSON { + if err := render.ScanReportJSON(out, data); err != nil { + return err + } + } else { + color := !noColor && isTerminal(out) + if err := render.ScanReport(out, data, color); err != nil { + return err + } + } + + // Non-zero exit when the scratch tripped, so scan is usable as a gate in + // hooks and CI. cobra prints RunE errors, so use a silent sentinel and let + // the caller (main) map it to an exit code without a noisy message — the + // report already said everything. + if len(findings) > 0 { + return errSecretsFound + } + return nil +} + +// toScanReport flattens the store scratch + detector findings into the render +// layer's plain view, matching the adapter pattern doctor/reap use so render +// never imports the store or secret packages. +func toScanReport(sc index.Scratch, findings []secret.Finding) render.ScanReportData { + rows := make([]render.ScanFinding, len(findings)) + for i, f := range findings { + rows[i] = render.ScanFinding{ + Kind: string(f.Kind), + Line: f.Line, + Rule: f.Rule, + Masked: f.Masked, + } + } + return render.ScanReportData{ + ID: sc.ID, + Name: sc.Name, + Findings: rows, + } +} diff --git a/internal/cli/scan_test.go b/internal/cli/scan_test.go new file mode 100644 index 0000000..8cb1ff2 --- /dev/null +++ b/internal/cli/scan_test.go @@ -0,0 +1,90 @@ +package cli + +import ( + "errors" + "strings" + "testing" +) + +// dotenvBody is a small .env-style dump with real-looking secrets used across +// the scan/promote tripwire tests. +const dotenvBody = "# creds\n" + + "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n" + + "STRIPE_API_KEY=EXAMPLExQzLkdIwqZ9mNbVcXsWePqRtY\n" + + "NOTES=this is just prose and should stay quiet\n" + +func TestScanCleanScratchExitsZeroAndSaysSoClean(t *testing.T) { + s := newSession(t) + id := s.newScratchID("safe-notes") + s.writeScratchBody(id, "just some harmless notes\nnothing to see here\n") + + out, err := s.run("scan", id) + if err != nil { + t.Fatalf("scan of a clean scratch should exit zero; got %v (out=%s)", err, out) + } + if !strings.Contains(strings.ToLower(out), "clean") { + t.Errorf("scan should report a clean bill of health; got %q", out) + } +} + +func TestScanTrippedScratchReportsMaskedFindingsAndErrors(t *testing.T) { + s := newSession(t) + id := s.newScratchID("leaky") + s.writeScratchBody(id, dotenvBody) + + out, err := s.run("scan", id) + if err == nil { + t.Fatal("scan of a scratch with secrets should return the secrets-found sentinel") + } + if !errors.Is(err, ErrSecretsFound) { + t.Errorf("expected ErrSecretsFound sentinel; got %v", err) + } + + // The report must name findings by line but never echo a raw secret. + if !strings.Contains(out, "tripwire") { + t.Errorf("scan should headline the tripwire hit; got %q", out) + } + for _, raw := range []string{"AKIAIOSFODNN7EXAMPLE", "EXAMPLExQzLkdIwqZ9mNbVcXsWePqRtY"} { + if strings.Contains(out, raw) { + t.Errorf("scan output leaked a raw secret %q: %s", raw, out) + } + } + // Masked previews should be present (first-3…last-3 form). + if !strings.Contains(out, "AKI") || !strings.Contains(out, "…") { + t.Errorf("scan should show masked previews; got %q", out) + } +} + +func TestScanJSONShapeIsStable(t *testing.T) { + s := newSession(t) + id := s.newScratchID("leaky-json") + s.writeScratchBody(id, dotenvBody) + + out, _ := s.run("scan", id, "--json") + // --json must be pure data: no personality words, a tripped flag, and a + // findings array. (We assert on substrings rather than parsing to keep the + // test close to the other CLI tests in this package.) + for _, want := range []string{"\"tripped\": true", "\"findings\"", "\"kind\"", "\"line\"", "\"masked\""} { + if !strings.Contains(out, want) { + t.Errorf("scan --json missing %q; got %s", want, out) + } + } + if strings.Contains(out, "tripwire") || strings.Contains(out, "🔑") { + t.Errorf("scan --json should carry no personality/markers; got %s", out) + } +} + +func TestScanPrefixAndUnknownID(t *testing.T) { + s := newSession(t) + id := s.newScratchID("prefixy-scan") + s.writeScratchBody(id, "clean\n") + + if _, err := s.run("scan", id[:4]); err != nil { + t.Errorf("scan by prefix should work on a clean scratch; got %v", err) + } + + _, err := s.run("scan", "ghost999") + if err == nil || !strings.Contains(err.Error(), "no scratch matches") { + t.Errorf("scan of unknown id should error with no-match; got %v", err) + } +} diff --git a/internal/render/json.go b/internal/render/json.go index 4ebe995..bfc65a0 100644 --- a/internal/render/json.go +++ b/internal/render/json.go @@ -41,6 +41,10 @@ type ScratchJSON struct { // table's color buckets so a script can branch the same way the eye does. Status string `json:"status"` OriginCwd string `json:"originCwd"` + // Secret is true when the scratch tripped the secret tripwire (the same + // signal the 🔑 marker shows in `sp ls`). Scripts can gate a bulk promote on + // `.[] | select(.secret)` without shelling out to `sp scan` per id. + Secret bool `json:"secret"` } // MorgueJSON is the scriptable record for a soft-deleted scratch under @@ -67,8 +71,10 @@ type MorgueJSON struct { // scratchJSON builds a ScratchJSON view of a live scratch as of now. It reuses // the same humanizers and lifecycle classification the tables use, so the two -// renderings can never disagree about age, expiry phrasing, or status. -func scratchJSON(s index.Scratch, now time.Time) ScratchJSON { +// renderings can never disagree about age, expiry phrasing, or status. secret +// is threaded in from the caller's tripwire scan (the JSON layer stays pure and +// does no scanning itself). +func scratchJSON(s index.Scratch, now time.Time, secret bool) ScratchJSON { return ScratchJSON{ ID: s.ID, Name: s.Name, @@ -83,6 +89,7 @@ func scratchJSON(s index.Scratch, now time.Time) ScratchJSON { ExpiresHuman: humanExpiry(s.ExpiresAt.Sub(now)), Status: statusString(classify(s, now)), OriginCwd: s.OriginCwd, + Secret: secret, } } @@ -111,10 +118,18 @@ func morgueJSON(r MorgueRow, now time.Time) MorgueJSON { // than null, so consumers can always treat the output as an array. Unlike the // table, this path is intentionally color- and personality-free. func TableJSON(w io.Writer, scratches []index.Scratch, now time.Time) error { + return TableMarkedJSON(w, scratches, nil, now) +} + +// TableMarkedJSON is TableJSON with an optional per-scratch tripwire marker set: +// any id in markers gets "secret": true in its record. markers may be nil, in +// which case every record reports secret=false. As with TableMarked, the marker +// is a side map so the store/index never learn about the tripwire. +func TableMarkedJSON(w io.Writer, scratches []index.Scratch, markers map[string]bool, now time.Time) error { ordered := sortLive(scratches) records := make([]ScratchJSON, 0, len(ordered)) for _, s := range ordered { - records = append(records, scratchJSON(s, now)) + records = append(records, scratchJSON(s, now, markers[s.ID])) } return writeJSON(w, records) } diff --git a/internal/render/render.go b/internal/render/render.go index fa8e5b2..277edaa 100644 --- a/internal/render/render.go +++ b/internal/render/render.go @@ -93,6 +93,16 @@ func (p Palette) styleFor(l lifecycle) lipgloss.Style { // passed in (rather than read from the clock) so output is deterministic and // unit-testable. func Table(w io.Writer, scratches []index.Scratch, now time.Time, color bool) error { + return TableMarked(w, scratches, nil, now, color) +} + +// TableMarked is Table with an optional per-scratch marker set: any id present +// in markers is flagged in the NAME column (currently a key glyph for scratches +// that tripped the secret tripwire). markers may be nil, in which case this is +// exactly Table. Keeping the marker as a side map — rather than a field on +// index.Scratch — preserves the "index is plain storage, render decides +// presentation" boundary and keeps the store unaware of the tripwire. +func TableMarked(w io.Writer, scratches []index.Scratch, markers map[string]bool, now time.Time, color bool) error { if len(scratches) == 0 { _, err := fmt.Fprintln(w, "no scratches yet — create one with `sp new`") return err @@ -102,9 +112,9 @@ func Table(w io.Writer, scratches []index.Scratch, now time.Time, color bool) er rows := sortLive(scratches) if color { - return colorTable(w, rows, now) + return colorTable(w, rows, markers, now) } - return plainTable(w, rows, now) + return plainTable(w, rows, markers, now) } // sortLive returns a copy of scratches in the canonical live ordering: @@ -122,11 +132,13 @@ func sortLive(scratches []index.Scratch) []index.Scratch { return rows } -// rowCells builds the six display strings for a single scratch. -func rowCells(s index.Scratch, now time.Time) []string { +// rowCells builds the six display strings for a single scratch. markers may be +// nil; when it flags this scratch's id, the NAME cell is prefixed with a key +// glyph so a tripwire hit is visible at a glance without adding a whole column. +func rowCells(s index.Scratch, markers map[string]bool, now time.Time) []string { return []string{ s.ID, - nameOrDash(s.Name), + markedName(s, markers), humanAge(now.Sub(s.CreatedAt)), humanExpiry(s.ExpiresAt.Sub(now)), tagsOrDash(s.Tags), @@ -134,13 +146,25 @@ func rowCells(s index.Scratch, now time.Time) []string { } } +// markedName renders the NAME cell, prefixing a key glyph when the scratch +// tripped the secret tripwire (its id is in markers). The marker rides on the +// name rather than in its own column so existing table widths and the +// plain/JSON contracts stay stable for scratches that didn't trip. +func markedName(s index.Scratch, markers map[string]bool) string { + name := nameOrDash(s.Name) + if markers[s.ID] { + return "🔑 " + name + } + return name +} + // plainTable writes a no-escape, tab-separated table suitable for pipes. -func plainTable(w io.Writer, rows []index.Scratch, now time.Time) error { +func plainTable(w io.Writer, rows []index.Scratch, markers map[string]bool, now time.Time) error { var b strings.Builder b.WriteString(strings.Join(columns, "\t")) b.WriteByte('\n') for _, s := range rows { - b.WriteString(strings.Join(rowCells(s, now), "\t")) + b.WriteString(strings.Join(rowCells(s, markers, now), "\t")) b.WriteByte('\n') } _, err := io.WriteString(w, b.String()) @@ -148,7 +172,7 @@ func plainTable(w io.Writer, rows []index.Scratch, now time.Time) error { } // colorTable draws the lipgloss table, tinting each row by lifecycle. -func colorTable(w io.Writer, rows []index.Scratch, now time.Time) error { +func colorTable(w io.Writer, rows []index.Scratch, markers map[string]bool, now time.Time) error { pal := defaultPalette() // Compute column widths from the (untinted) cell content so styling @@ -160,7 +184,7 @@ func colorTable(w io.Writer, rows []index.Scratch, now time.Time) error { cells := make([][]string, len(rows)) lifes := make([]lifecycle, len(rows)) for r, s := range rows { - cells[r] = rowCells(s, now) + cells[r] = rowCells(s, markers, now) lifes[r] = classify(s, now) for c, val := range cells[r] { if wdt := lipgloss.Width(val); wdt > widths[c] { diff --git a/internal/render/scan.go b/internal/render/scan.go new file mode 100644 index 0000000..0c37037 --- /dev/null +++ b/internal/render/scan.go @@ -0,0 +1,132 @@ +// scan.go renders the secret tripwire's findings for `sp scan`. +// +// As with every other command, the detector (internal/secret) returns plain +// data and render decides how to phrase and tint it. render never sees a full +// secret: the Masked field arrives already redacted from the detector, and this +// file only ever prints that masked preview. A clean scratch gets a reassuring +// (green) line in scratchpatch's tombstone voice; a tripped one gets an amber +// header and one red line per finding, naming the line number, the rule, and +// the masked value. +// +// The --json path here mirrors ls/doctor: color-free, personality-free, stable +// keys, findings always an array (never null) so a script can iterate +// unconditionally and gate on `.tripped`. +package render + +import ( + "encoding/json" + "fmt" + "io" + "strings" +) + +// ScanFinding is render's flat view of one secret-detector hit: the kind of +// secret, the 1-based line it sat on, the human rule label, and a masked +// preview safe to print. render imports neither the store nor the secret +// package; the CLI adapts those into this shape (same pattern as DoctorOrphan). +type ScanFinding struct { + Kind string + Line int + Rule string + Masked string +} + +// ScanReportData is the plain summary `sp scan` renders: which scratch was +// scanned and every place it tripped. An empty Findings slice means the scratch +// is clean. +type ScanReportData struct { + ID string + Name string + Findings []ScanFinding +} + +// tripped reports whether the scanned scratch hit anything. +func (d ScanReportData) tripped() bool { return len(d.Findings) > 0 } + +// ScanReport writes a secret-scan report to w. A clean scratch gets a single +// green line; a tripped one leads with an amber headline naming the count, then +// lists each finding on its own red line as "L ", and +// closes with a pointer at the safe next step (promote blocks unless +// --allow-secrets). On a non-TTY it's plain text. The masked values are printed +// verbatim from the detector — this function never has, and never prints, a raw +// secret. +func ScanReport(w io.Writer, d ScanReportData, color bool) error { + pal := defaultPalette() + var b strings.Builder + + if !d.tripped() { + writeLine(&b, fmt.Sprintf("clean bill of health — no secrets found in %s (%s)", + d.ID, nameOrDash(d.Name)), color, pal.fresh) + _, err := io.WriteString(w, b.String()) + return err + } + + writeLine(&b, fmt.Sprintf("tripwire! %s in %s (%s) — do NOT let this out alive", + countFindings(len(d.Findings)), d.ID, nameOrDash(d.Name)), color, pal.header) + + for _, f := range d.Findings { + line := fmt.Sprintf(" L%-4d %-32s %s", f.Line, f.Rule, f.Masked) + writeLine(&b, line, color, pal.expired) + } + + fmt.Fprintf(&b, "\nvalues are masked; `sp promote` will refuse this scratch "+ + "unless you pass --allow-secrets.\n") + + _, err := io.WriteString(w, b.String()) + return err +} + +// ScanJSON is the scriptable record for `sp scan --json`: the scanned scratch's +// id/name, a one-field `tripped` summary so a script can gate without inspecting +// the array, and the findings themselves. Like ls/doctor JSON it is color- and +// personality-free, and findings is always a (possibly empty) array so the +// shape never flips to null on a clean scratch. +type ScanJSON struct { + ID string `json:"id"` + Name string `json:"name"` + Tripped bool `json:"tripped"` + Findings []ScanFindingJSON `json:"findings"` +} + +// ScanFindingJSON is the scriptable view of a single finding. Masked carries the +// redacted preview only; there is deliberately no field for the raw value. +type ScanFindingJSON struct { + Kind string `json:"kind"` + Line int `json:"line"` + Rule string `json:"rule"` + Masked string `json:"masked"` +} + +// ScanReportJSON writes a ScanReportData to w as a single ScanJSON object. The +// findings slice is always emitted as an array (never null), and no raw secret +// is ever included — only the detector's masked preview. +func ScanReportJSON(w io.Writer, d ScanReportData) error { + findings := make([]ScanFindingJSON, 0, len(d.Findings)) + for _, f := range d.Findings { + findings = append(findings, ScanFindingJSON{ + Kind: f.Kind, + Line: f.Line, + Rule: f.Rule, + Masked: f.Masked, + }) + } + rec := ScanJSON{ + ID: d.ID, + Name: d.Name, + Tripped: d.tripped(), + Findings: findings, + } + enc := json.NewEncoder(w) + enc.SetEscapeHTML(false) + enc.SetIndent("", " ") + return enc.Encode(rec) +} + +// countFindings renders "N secret(s)" with correct pluralization, matching the +// countScratches/countOrphans family so reports read naturally for 1 or N. +func countFindings(n int) string { + if n == 1 { + return "1 secret" + } + return fmt.Sprintf("%d secrets", n) +} diff --git a/internal/render/scan_test.go b/internal/render/scan_test.go new file mode 100644 index 0000000..7f79890 --- /dev/null +++ b/internal/render/scan_test.go @@ -0,0 +1,105 @@ +package render + +import ( + "bytes" + "strings" + "testing" + "time" + + "github.com/rwrife/scratchpatch/internal/index" +) + +func TestScanReportCleanIsReassuring(t *testing.T) { + var buf bytes.Buffer + d := ScanReportData{ID: "abc123", Name: "notes"} + if err := ScanReport(&buf, d, false); err != nil { + t.Fatalf("ScanReport: %v", err) + } + got := buf.String() + if !strings.Contains(got, "clean bill of health") { + t.Errorf("clean scan should say so; got %q", got) + } + if strings.Contains(got, "🔑") { + t.Errorf("clean report needs no marker; got %q", got) + } +} + +func TestScanReportListsMaskedFindings(t *testing.T) { + var buf bytes.Buffer + d := ScanReportData{ + ID: "def456", + Name: "leaky", + Findings: []ScanFinding{ + {Kind: "aws-access-key", Line: 2, Rule: "AWS access key id", Masked: "AKI…PLE"}, + {Kind: "assignment", Line: 3, Rule: "secret-looking assignment (TOKEN)", Masked: "EXA…nJx"}, + }, + } + if err := ScanReport(&buf, d, false); err != nil { + t.Fatalf("ScanReport: %v", err) + } + got := buf.String() + for _, want := range []string{"tripwire", "L2", "L3", "AKI…PLE", "EXA…nJx", "--allow-secrets"} { + if !strings.Contains(got, want) { + t.Errorf("scan report missing %q; got %q", want, got) + } + } +} + +func TestScanReportJSONIsPureData(t *testing.T) { + var buf bytes.Buffer + d := ScanReportData{ + ID: "ghi789", + Name: "leaky", + Findings: []ScanFinding{{Kind: "high-entropy", Line: 1, Rule: "high-entropy token", Masked: "9f8…iE0"}}, + } + if err := ScanReportJSON(&buf, d); err != nil { + t.Fatalf("ScanReportJSON: %v", err) + } + got := buf.String() + for _, want := range []string{`"id": "ghi789"`, `"tripped": true`, `"findings"`, `"masked": "9f8…iE0"`} { + if !strings.Contains(got, want) { + t.Errorf("scan JSON missing %q; got %s", want, got) + } + } + if strings.Contains(got, "tripwire") { + t.Errorf("scan JSON should carry no personality; got %s", got) + } +} + +func TestScanReportJSONEmptyFindingsIsArrayNotNull(t *testing.T) { + var buf bytes.Buffer + d := ScanReportData{ID: "x", Name: ""} + if err := ScanReportJSON(&buf, d); err != nil { + t.Fatalf("ScanReportJSON: %v", err) + } + got := buf.String() + if !strings.Contains(got, `"findings": []`) { + t.Errorf("empty findings should serialize as [], not null; got %s", got) + } + if !strings.Contains(got, `"tripped": false`) { + t.Errorf("clean scan should report tripped=false; got %s", got) + } +} + +func TestTableMarkedFlagsNameWithKey(t *testing.T) { + now := time.Now() + s := index.Scratch{ + ID: "aaa111", + Name: "secrets", + CreatedAt: now.Add(-time.Hour), + ExpiresAt: now.Add(48 * time.Hour), + } + var marked, plain bytes.Buffer + if err := TableMarked(&marked, []index.Scratch{s}, map[string]bool{"aaa111": true}, now, false); err != nil { + t.Fatalf("TableMarked: %v", err) + } + if err := TableMarked(&plain, []index.Scratch{s}, nil, now, false); err != nil { + t.Fatalf("TableMarked(nil markers): %v", err) + } + if !strings.Contains(marked.String(), "🔑") { + t.Errorf("marked table should show 🔑; got %q", marked.String()) + } + if strings.Contains(plain.String(), "🔑") { + t.Errorf("unmarked table should not show 🔑; got %q", plain.String()) + } +} diff --git a/internal/secret/secret.go b/internal/secret/secret.go new file mode 100644 index 0000000..8d73332 --- /dev/null +++ b/internal/secret/secret.go @@ -0,0 +1,329 @@ +// Package secret is scratchpatch's tripwire for leaked credentials. +// +// AI coding agents and tired humans dump API keys, .env dumps, and private +// keys into throwaway files without thinking. This package scans content for +// the *shapes* those secrets take and reports where they are — never what they +// are. It is deliberately conservative: the goal is to catch the obvious leaks +// (AWS keys, `*_API_KEY=` assignments, PEM private-key headers, long +// high-entropy tokens) without crying wolf on ordinary prose or code, because +// a detector that fires constantly gets muted and then it protects nothing. +// +// Everything here is pure and clock-free: Scan takes bytes and returns +// findings, so it is trivially unit-testable over fixture files. Like the rest +// of scratchpatch, this package knows nothing about color, terminals, or the +// CLI — it returns plain data and lets the caller decide how to phrase it. And +// per the issue's hard rule, a Finding never carries a full secret value: it +// carries a Masked preview (first/last few characters with the middle redacted) +// so a report can point at a leak without re-leaking it to the terminal or to +// logs. +package secret + +import ( + "bufio" + "bytes" + "math" + "regexp" + "strings" +) + +// Kind names the category of secret a Finding matched, so callers (and scripts, +// via `sp scan --json` later) can group or filter without parsing prose. +type Kind string + +const ( + // KindAWSAccessKey is an AWS access key id (AKIA/ASIA + 16 base32 chars). + KindAWSAccessKey Kind = "aws-access-key" + // KindPrivateKey is a PEM private-key header line ("-----BEGIN ... PRIVATE KEY-----"). + KindPrivateKey Kind = "private-key" + // KindAssignment is a key/token assignment whose *name* looks secret + // (API_KEY, SECRET, TOKEN, PASSWORD…) and whose value is non-trivial. + KindAssignment Kind = "assignment" + // KindHighEntropy is a long, high-entropy token that doesn't match a more + // specific rule but looks generated rather than written. + KindHighEntropy Kind = "high-entropy" +) + +// Finding is one place a scan tripped. It names what kind of secret shape +// matched, which 1-based line it sat on, and a masked preview safe to print — +// never the raw value. Rule is a short human label for the specific heuristic +// (e.g. "AWS access key id") used in reports. +type Finding struct { + Kind Kind + // Line is the 1-based line number the match sat on. + Line int + // Rule is a short human label for the matched heuristic. + Rule string + // Masked is a redacted preview of the offending token, safe to display. + // It never contains the full secret value. + Masked string +} + +// awsAccessKeyRe matches an AWS access key id: the AKIA/ASIA/AGPA/AIDA-family +// prefix followed by exactly 16 uppercase base32 characters. Anchored on word +// boundaries so it won't fire mid-identifier. +var awsAccessKeyRe = regexp.MustCompile(`\b(A3T|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}\b`) + +// assignmentRe matches " " where sep is '=' or ':' — the +// shape of shell/.env assignments and many config lines. The name and value are +// captured so the caller can decide (via secretName / value heuristics) whether +// this is actually a secret rather than, say, `count = 3`. Quotes around the +// value are tolerated and stripped by the caller. +var assignmentRe = regexp.MustCompile(`(?i)([A-Za-z_][A-Za-z0-9_.-]*)\s*[:=]\s*["']?([^\s"']{6,})["']?`) + +// secretNameRe recognizes assignment *names* that strongly imply a credential. +// Kept conservative on purpose: we want KEY/SECRET/TOKEN/PASSWORD-family names, +// not every variable that happens to contain "key" as a substring of a word. +var secretNameRe = regexp.MustCompile(`(?i)(^|[_.-])(api[_.-]?key|secret|token|password|passwd|access[_.-]?key|private[_.-]?key|client[_.-]?secret|auth|credential|session[_.-]?key)([_.-]|$)`) + +// pemHeaderRe matches a PEM private-key banner line. We flag on the header +// alone (the body is just base64) so a pasted key trips even if truncated. +var pemHeaderRe = regexp.MustCompile(`-----BEGIN ([A-Z0-9 ]*)?PRIVATE KEY-----`) + +// tokenRe pulls candidate standalone tokens out of a line for the high-entropy +// fallback: runs of base64url / hex characters of a generated length. +// Slashes and '+'/'=' are deliberately excluded so URL paths and query strings +// (which are naturally high-entropy across their segments) don't trip the +// fallback — a real base64 secret with those characters is almost always in an +// assignment and is caught by the assignment rule instead. Bare opaque tokens +// (bearer tokens, opaque API keys) live in the [A-Za-z0-9_-] alphabet, which is +// what we scan for here. +var tokenRe = regexp.MustCompile(`[A-Za-z0-9_-]{24,}`) + +const ( + // minEntropyBits is the Shannon-entropy floor (bits per char) a bare token + // must clear to count as high-entropy. English prose sits well below this; + // random base64/hex sits above it. Tuned to avoid flagging ordinary words, + // URLs, and hashes-of-nothing while catching generated secrets. + minEntropyBits = 3.6 + // minTokenLen is the shortest bare token the high-entropy rule considers. + // Short tokens can't carry enough entropy to be confidently a secret. + minTokenLen = 24 + // maskKeep is how many leading and trailing characters a mask preserves. + maskKeep = 3 +) + +// Scan reads content and returns every place it tripped a secret heuristic, in +// ascending line order. It is pure and allocation-light: no I/O, no clock, no +// globals mutated — the same bytes always yield the same findings, which is +// what makes the fixture tests meaningful. A line may produce more than one +// finding (e.g. an assignment whose value is also high-entropy) but each rule +// reports a line at most once to avoid piling on. Content that trips nothing +// yields a nil slice, so callers can treat len==0 as "clean". +func Scan(content []byte) []Finding { + var findings []Finding + + sc := bufio.NewScanner(bytes.NewReader(content)) + // Allow long lines (minified JSON, base64 blobs) without the scanner's + // default 64 KiB token limit aborting the scan. + sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024) + + line := 0 + for sc.Scan() { + line++ + text := sc.Text() + findings = append(findings, scanLine(text, line)...) + } + return findings +} + +// scanLine applies every rule to a single line, deduping so no single line +// emits two findings of the same Kind. Rules are checked most-specific first so +// a matched AWS key or PEM header isn't also reported as a generic high-entropy +// token. +func scanLine(text string, line int) []Finding { + var out []Finding + seen := map[Kind]bool{} + + add := func(k Kind, rule, raw string) { + if seen[k] { + return + } + seen[k] = true + out = append(out, Finding{Kind: k, Line: line, Rule: rule, Masked: Mask(raw)}) + } + + // 1. PEM private-key header — unambiguous, check first. + if m := pemHeaderRe.FindString(text); m != "" { + add(KindPrivateKey, "PEM private key header", m) + } + + // 2. AWS access key id — specific prefix + fixed length. + if m := awsAccessKeyRe.FindString(text); m != "" { + add(KindAWSAccessKey, "AWS access key id", m) + } + + // 3. Secret-named assignment — the name implies a credential and the value + // is non-trivial. This is the workhorse for .env-style leaks. + if name, val, ok := secretAssignment(text); ok { + add(KindAssignment, "secret-looking assignment ("+name+")", val) + } + + // 4. High-entropy fallback — a long, generated-looking token that none of + // the above claimed. Skipped if we already flagged this line, so we + // don't double-report the value of a secret assignment. + if len(out) == 0 { + if tok := highEntropyToken(text); tok != "" { + add(KindHighEntropy, "high-entropy token", tok) + } + } + + return out +} + +// secretAssignment reports whether text is an assignment whose name looks like +// a credential and whose value is substantive (not a placeholder). It returns +// the matched name and the raw value so the caller can mask the value. +func secretAssignment(text string) (name, value string, ok bool) { + m := assignmentRe.FindStringSubmatch(text) + if m == nil { + return "", "", false + } + name, value = m[1], m[2] + if !secretNameRe.MatchString(name) { + return "", "", false + } + if isPlaceholder(value) { + return "", "", false + } + return name, value, true +} + +// isPlaceholder filters out obvious non-secret values so a template like +// `API_KEY=changeme` or `TOKEN=` doesn't cry wolf. Conservative: we +// only skip values that are clearly stand-ins, not anything that merely looks +// low-entropy. +func isPlaceholder(v string) bool { + lv := strings.ToLower(strings.Trim(v, "<>{}[]()\"'")) + switch lv { + case "changeme", "change-me", "your-key", "your_key", "yourkey", + "your-token", "your_token", "yourtoken", "xxx", "xxxx", "todo", + "example", "placeholder", "none", "null", "nil", "empty", + "redacted", "secret", "password", "test", "dummy", "fixme": + return true + } + // A value made only of the same repeated char (xxxx, ****, ....) is a mask, + // not a secret. + if len(lv) > 0 && strings.Count(lv, string(lv[0])) == len(lv) { + return true + } + // Interpolations like ${FOO} / $(cmd) / {{ var }} are references, not + // literal secrets. + if strings.HasPrefix(v, "${") || strings.HasPrefix(v, "$(") || strings.HasPrefix(v, "{{") { + return true + } + return false +} + +// highEntropyToken returns the first long, high-entropy token on the line, or +// "" if none qualifies. It exists to catch generated secrets that don't carry a +// telltale prefix or assignment name (bare bearer tokens, opaque API keys). The +// entropy floor, length minimum, and a "looks generated" mixed-character check +// keep it off ordinary words, hex color codes, git SHAs, URLs, and paths. +func highEntropyToken(text string) string { + for _, tok := range tokenRe.FindAllString(text, -1) { + if len(tok) < minTokenLen { + continue + } + // Skip tokens that are all one character class in a way that reads as + // structured rather than secret (e.g. a run of digits). + if isAllDigits(tok) { + continue + } + // Generated secrets mix character classes; a long run of only lowercase + // letters is almost always a word or a slug, not a credential. Requiring + // a mix (upper+lower, or letters+digits) is the single biggest lever + // against flagging prose and URL path segments. + if !looksGenerated(tok) { + continue + } + if shannonEntropy(tok) >= minEntropyBits { + return tok + } + } + return "" +} + +// looksGenerated reports whether tok mixes character classes the way a random +// token does: it must contain at least two of {lowercase, uppercase, digit}. +// Single-class runs (all-lowercase words, all-uppercase constants) read as +// human-written and are left alone to keep the false-positive rate low. +func looksGenerated(tok string) bool { + var lower, upper, digit bool + for _, r := range tok { + switch { + case r >= 'a' && r <= 'z': + lower = true + case r >= 'A' && r <= 'Z': + upper = true + case r >= '0' && r <= '9': + digit = true + } + } + classes := 0 + for _, on := range []bool{lower, upper, digit} { + if on { + classes++ + } + } + return classes >= 2 +} + +// isAllDigits reports whether s is only ASCII digits (a long number is not a +// secret by our heuristics — think ids, timestamps, counts). +func isAllDigits(s string) bool { + for _, r := range s { + if r < '0' || r > '9' { + return false + } + } + return len(s) > 0 +} + +// shannonEntropy returns the Shannon entropy of s in bits per character. Random +// base64 approaches ~6 bits/char; English text sits near ~2. We use it as a +// cheap "does this look generated?" signal for the high-entropy fallback. +func shannonEntropy(s string) float64 { + if s == "" { + return 0 + } + var counts [256]float64 + n := 0 + for i := 0; i < len(s); i++ { + counts[s[i]]++ + n++ + } + var h float64 + for _, c := range counts { + if c == 0 { + continue + } + p := c / float64(n) + h -= p * math.Log2(p) + } + return h +} + +// Mask redacts a secret value for display: it keeps the first and last maskKeep +// characters and replaces the middle with a fixed-width ellipsis, so a report +// can show *which* token tripped without revealing it. Short values are fully +// starred so we never expose most of a small secret by "previewing" it. Mask is +// exported because the CLI masks values it pulls straight from a line (e.g. the +// assignment value it already parsed) as well as ones inside Findings. +func Mask(raw string) string { + if raw == "" { + return "" + } + // For anything short enough that first+last would reveal most of it, star + // the whole thing. + if len(raw) <= 2*maskKeep+2 { + return strings.Repeat("*", len(raw)) + } + return raw[:maskKeep] + "…" + raw[len(raw)-maskKeep:] +} + +// Tripped reports whether content contains any secret shape at all. It's the +// cheap gate the CLI uses for `sp ls` markers and the `sp promote` block, where +// only the yes/no matters and the individual findings don't. +func Tripped(content []byte) bool { + return len(Scan(content)) > 0 +} diff --git a/internal/secret/secret_test.go b/internal/secret/secret_test.go new file mode 100644 index 0000000..29654c2 --- /dev/null +++ b/internal/secret/secret_test.go @@ -0,0 +1,183 @@ +package secret + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// readFixture loads a testdata file or fails the test. +func readFixture(t *testing.T, name string) []byte { + t.Helper() + b, err := os.ReadFile(filepath.Join("testdata", name)) + if err != nil { + t.Fatalf("read fixture %s: %v", name, err) + } + return b +} + +// kinds collects the set of Kinds present in a finding slice. +func kinds(fs []Finding) map[Kind]bool { + m := map[Kind]bool{} + for _, f := range fs { + m[f.Kind] = true + } + return m +} + +func TestScanCleanFileTripsNothing(t *testing.T) { + got := Scan(readFixture(t, "clean.txt")) + if len(got) != 0 { + t.Fatalf("clean file should trip nothing; got %d findings: %+v", len(got), got) + } + if Tripped(readFixture(t, "clean.txt")) { + t.Error("Tripped should be false for a clean file") + } +} + +func TestScanPlaceholdersStayQuiet(t *testing.T) { + // A .env.example full of template values must not cry wolf — that's the + // whole point of alarm-fatigue avoidance. + got := Scan(readFixture(t, "placeholders.txt")) + if len(got) != 0 { + t.Fatalf("placeholder template should trip nothing; got %+v", got) + } +} + +func TestScanDotenvCatchesTheRealSecrets(t *testing.T) { + content := readFixture(t, "dotenv.txt") + got := Scan(content) + if len(got) == 0 { + t.Fatal("dotenv dump should trip the detector") + } + + ks := kinds(got) + if !ks[KindAWSAccessKey] { + t.Error("expected the AWS access key id to be caught") + } + if !ks[KindAssignment] { + t.Error("expected secret-looking assignments to be caught") + } + + // The plain-prose NOTES= line must not be flagged. + for _, f := range got { + if f.Line == 7 { + t.Errorf("prose line 7 should not trip; got %+v", f) + } + } +} + +func TestScanFindsPrivateKeyHeader(t *testing.T) { + got := Scan(readFixture(t, "private_key.txt")) + if !kinds(got)[KindPrivateKey] { + t.Fatalf("expected a private-key finding; got %+v", got) + } + // The header sits on line 1. + var found bool + for _, f := range got { + if f.Kind == KindPrivateKey { + found = true + if f.Line != 1 { + t.Errorf("private key header should be line 1; got line %d", f.Line) + } + } + } + if !found { + t.Error("private-key finding missing") + } +} + +func TestScanFindsBareBearerTokenViaEntropy(t *testing.T) { + got := Scan(readFixture(t, "bearer.txt")) + if !kinds(got)[KindHighEntropy] && !kinds(got)[KindAssignment] { + t.Fatalf("expected the bearer token to trip high-entropy (or assignment); got %+v", got) + } +} + +func TestFindingsNeverContainRawSecret(t *testing.T) { + // The cardinal rule: no finding may echo a full secret value. Check that + // the known raw secrets from the fixtures never appear verbatim in any + // Masked field. + rawSecrets := []string{ + "AKIAIOSFODNN7EXAMPLE", + "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", + "EXAMPLExQzLkdIwqZ9mNbVcXsWePqRtYuIoP", + "EXAMPLEwWPw5k4aXcaT4fNP0UcnZwJUVFk6LO0p1nJx", + "9f8Kd2LmQx7RvT1nZ4pW6sYb3cJhAeUgN5oXiE0", + } + for _, fx := range []string{"dotenv.txt", "bearer.txt", "private_key.txt"} { + for _, f := range Scan(readFixture(t, fx)) { + for _, raw := range rawSecrets { + if strings.Contains(f.Masked, raw) { + t.Errorf("%s: masked value leaked a raw secret %q in %+v", fx, raw, f) + } + } + } + } +} + +func TestMaskRedactsMiddleAndStarsShortValues(t *testing.T) { + // Long value: keep 3 + 3, redact the middle. + got := Mask("AKIAIOSFODNN7EXAMPLE") + if strings.Contains(got, "IOSFODNN") { + t.Errorf("mask must hide the middle; got %q", got) + } + if !strings.HasPrefix(got, "AKI") || !strings.HasSuffix(got, "PLE") { + t.Errorf("mask should keep first/last 3; got %q", got) + } + + // Short value: fully starred, no characters revealed. + short := Mask("abcd") + if short != "****" { + t.Errorf("short value should be fully starred; got %q", short) + } + if Mask("") != "" { + t.Errorf("empty mask should stay empty; got %q", Mask("")) + } +} + +func TestScanReportsAscendingLineNumbers(t *testing.T) { + got := Scan(readFixture(t, "dotenv.txt")) + last := 0 + for _, f := range got { + if f.Line < last { + t.Errorf("findings should be in ascending line order; %d after %d", f.Line, last) + } + if f.Line < 1 { + t.Errorf("line numbers are 1-based; got %d", f.Line) + } + last = f.Line + } +} + +func TestScanIgnoresPlainNumbersAndProse(t *testing.T) { + // Long digit runs (ids, timestamps) and ordinary sentences must not trip + // the high-entropy fallback. + cases := []string{ + "the build finished in 1234567890123456 nanoseconds", + "order id 000000001111112222223333334444445555556666", + "This is a perfectly ordinary sentence with several longish words in it.", + "see the docs at https://example.com/very/long/path/that/is/just/a/url/here", + } + for _, c := range cases { + if got := Scan([]byte(c)); len(got) != 0 { + t.Errorf("expected no findings for %q; got %+v", c, got) + } + } +} + +func TestScanDedupesPerKindPerLine(t *testing.T) { + // Two AWS keys on one line should yield a single aws-access-key finding for + // that line (we don't pile on). + line := "keys: AKIAIOSFODNN7EXAMPLE and AKIAJEXAMPLE1234567X" + var awsCount int + for _, f := range Scan([]byte(line)) { + if f.Kind == KindAWSAccessKey { + awsCount++ + } + } + if awsCount != 1 { + t.Errorf("expected one aws finding per line, got %d", awsCount) + } +} diff --git a/internal/secret/testdata/bearer.txt b/internal/secret/testdata/bearer.txt new file mode 100644 index 0000000..ed5aa5f --- /dev/null +++ b/internal/secret/testdata/bearer.txt @@ -0,0 +1,5 @@ +Bearer token pasted from a curl command someone was debugging: + +curl -H "Authorization: Bearer 9f8Kd2LmQx7RvT1nZ4pW6sYb3cJhAeUgN5oXiE0" https://api.example.com/v1/me + +The response was a 200. Everything else on this line is just words. diff --git a/internal/secret/testdata/clean.txt b/internal/secret/testdata/clean.txt new file mode 100644 index 0000000..3a1258c --- /dev/null +++ b/internal/secret/testdata/clean.txt @@ -0,0 +1,10 @@ +Deploy notes for the staging cluster. + +The rollout went fine after we bumped the replica count to 3. +Remember to check the dashboard at https://status.example.com/staging +before paging anyone. The p95 latency budget is 250ms. + +TODO: write the runbook for the cache warmer. +count = 3 +retries = 5 +timeout_seconds = 30 diff --git a/internal/secret/testdata/dotenv.txt b/internal/secret/testdata/dotenv.txt new file mode 100644 index 0000000..a0190a4 --- /dev/null +++ b/internal/secret/testdata/dotenv.txt @@ -0,0 +1,6 @@ +# scratch .env dump from the deploy box +AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +STRIPE_API_KEY=EXAMPLExQzLkdIwqZ9mNbVcXsWePqRtYuIoP +GITHUB_TOKEN=EXAMPLEwWPw5k4aXcaT4fNP0UcnZwJUVFk6LO0p1nJx +NOTES=this line is just prose and should not trip anything at all diff --git a/internal/secret/testdata/placeholders.txt b/internal/secret/testdata/placeholders.txt new file mode 100644 index 0000000..e9d58c8 --- /dev/null +++ b/internal/secret/testdata/placeholders.txt @@ -0,0 +1,8 @@ +# .env.example — a template checked into the repo on purpose. +# None of these are real; the tripwire must stay quiet. +API_KEY=changeme +SECRET_TOKEN= +DATABASE_PASSWORD=${DB_PASSWORD} +CLIENT_SECRET=your-key +AUTH_TOKEN=xxxxxxxxxxxx +SESSION_KEY=example diff --git a/internal/secret/testdata/private_key.txt b/internal/secret/testdata/private_key.txt new file mode 100644 index 0000000..066ab96 --- /dev/null +++ b/internal/secret/testdata/private_key.txt @@ -0,0 +1,5 @@ +-----BEGIN OPENSSH PRIVATE KEY----- +b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW +QyNTUxOQAAACDExampleExampleExampleExampleExampleExampleExampleExAAAAJi +NOTKEYNOTKEYAAAAC3NzaC1lZDI1NTE5AAAAIMTExampleExampleExampleExampleExam +-----END OPENSSH PRIVATE KEY-----