Skip to content

[feature] sp dedup — content-hash duplicate detection ("you already scratched this") #28

Description

@rwrife

Pitch

Throwaway files breed duplicates: you paste the same log three times, sp new the same snippet across two projects, or an agent re-captures identical API output on every run. Over time the store fills with byte-identical (or near-identical) scratches that all age out separately and waste footprint. sp dedup gives the store a duplicate report based on content hashing, and optionally collapses exact dupes — a genuinely new hygiene primitive distinct from sp doctor (which reconciles index-vs-disk drift, not content equality) and sp reap (which is TTL-based).

Behavior

  • sp dedup (read-only default): hash every live scratch's content (e.g. SHA-256), group by digest, and report clusters of identical scratches — showing which is the oldest "canonical" copy and how many bytes the redundant copies cost. Colorized report + --json for scripting.
  • sp dedup --collapse (opt-in, still non-destructive): move the redundant copies to the morgue (never hard-delete), keeping the oldest/pinned member as canonical. Consistent with scratchpatch's ironclad "destructive actions are always two-step, morgue-first" rule.
  • Optional stretch: --near to also flag near-duplicates (normalized whitespace / trivial diffs), reported separately and never auto-collapsed.

Why it fits

  • Reuses internal/store for content access + the existing move-to-morgue path; no new destructive machinery.
  • Mirrors the layering of sp doctor/sp reap: pure grouping logic in store, presentation in render, thin internal/cli wiring, --json via the existing render/json pattern.
  • Pairs naturally with [feature] sp new --stdin / quick-capture from a pipe (headless scratch creation) #27 (headless capture) — agents that re-pipe identical output get automatic "already have this" hygiene.

Acceptance criteria

  • sp dedup produces a read-only report of exact-duplicate clusters among live scratches (canonical = oldest; count + wasted bytes per cluster; total reclaimable). Clean "no duplicates" headline when the store is unique.
  • --json output mode (arrays never null, color/personality-free, same contract as sp ls --json/sp doctor --json).
  • sp dedup --collapse moves redundant copies to the morgue (never hard-deletes), preserving the canonical member; respects sp pin if/when present (pinned copies are never collapsed).
  • Default is strictly read-only; --collapse requires the explicit flag and reports exactly what it moved.
  • Tombstone-flavored wording in the human report; zero personality/color leakage into --json.
  • Handles edge cases: single scratch, empty store, unreadable/missing content file (surface, don't crash), and identical content with different names/tags (still a dupe by content).
  • Tests: exact-dupe grouping, canonical selection, wasted-byte math, --collapse morgue-move (with a read-only guarantee for the default path), and --json shape. go build/go vet/go test ./.../gofmt clean.
  • README documents sp dedup (+ the morgue-first --collapse behavior).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature / backlog item

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions