Add corpus index text dedupe command - #46
Merged
Merged
Conversation
Piece ids include the source path, so re-ingesting a post from a second export path lands it twice and append_index's id check cannot see it. dedupe-index groups pieces by normalized text (plus near-identical captures that differ only in trailing link/hashtag noise), keeps one deterministic keeper per group, and refuses to drop holdout ids. Report-only by default; --apply backs up the index before rewriting it and prints the index-voice / select / build-style-profile rebuild those counts depend on. Local run: 24 groups / 44 pieces removed, corpus 1647 -> 1603. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
append_index's id check cannot see it.dedupe-indexgroups pieces by normalized text (plus near-identical captures differing only in trailing link/hashtag noise), picks one deterministic keeper per group, and refuses to drop holdout ids.--applybacks up the index before rewriting it and prints theindex-voice/select/build-style-profilerebuild those counts depend on.Local run (not in repo)
24 duplicate-text groups / 44 pieces removed; corpus index 1647 → 1603; voice index 1600; style
draft_word_target419. No personal text, index backups, or corpus data included here.Test plan
pytest -q— 280 passedruff check .— cleantests/test_corpus_dedupe.py: exact + near duplicates, holdout keeper protection, cross-source keeper priority, report-only vs--applybackupMade with Cursor