Camp A RAG writer: product path with honest holdout GO - #44
Merged
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Compute median sentence length, short-line ratio, contractions, and you/I from selected posts, plus a banned AI-filler list, via build-style-profile. Co-authored-by: Cursor <cursoragent@cursor.com>
Mask retrieved entities and reject exemplar parroting or invented brief facts before drafts are accepted. Co-authored-by: Cursor <cursoragent@cursor.com>
Build a profile-scoped index that excludes holdouts and replaces stale vectors so RAG retrieval remains private and reproducible. Co-authored-by: Cursor <cursoragent@cursor.com>
Draft posts from retrieved exemplars on base weights: retrieve 3-5 masked exemplars, build the locked prompt, generate, and regenerate once when the parrot or invention guard trips. The --json receipt records adapter=none. MLX now blocks by default and requires PP_MLX_ALLOW=1, because importing mlx without a Metal device aborts the interpreter instead of raising. Generation is injectable and pytest installs a meta_path finder that rejects mlx imports, so sandboxed runs fail as ImportError rather than a crash dialog. Co-authored-by: Cursor <cursoragent@cursor.com>
Status now reports voice_mode=rag with adapter=none, and the README quickstart points at index-voice → build-style-profile → write instead of the legacy translator/train path. Co-authored-by: Cursor <cursoragent@cursor.com>
Carve never-indexed holdouts, mine briefs, score injectable RAG and bare-base drafts, and emit an eval-write-holdout receipt without draft bodies. Co-authored-by: Cursor <cursoragent@cursor.com>
Apply the chat template on the write path (shared with filter), mine real terse briefs instead of leaking the holdout body, persist raw prompts/drafts under gitignored dogfood/raw, and stop invent-guard from flagging ordinary nouns and prompt scaffolding. Co-authored-by: Cursor <cursoragent@cursor.com>
The holdout eval was measuring the wrong thing three ways over. Write generate handed Qwen3.5-instruct a bare text block, so it continued the document and echoed EXAMPLES/BRIEF headers back instead of writing a post. Both write paths now render through a shared chat_prompt helper, the same way filter.py already did. Brief mining set points to the entire holdout body, which handed the eval its answer key: the bare-base arm could paraphrase the target and win on stylometric distance with no voice modelling at all. Briefs are now a short topic plus terse bullets fitted to a word budget proportional to the post, tracked as brief_leakage_ratio in the receipt and enforced fail-closed. The invent guard scores against a separate guard_facts field so its fact set can never silently widen beyond what the model was shown. The invention guard flagged common nouns, calendar words, generic acronyms and our own prompt scaffolding as fabricated companies, so it fired on every draft and carried no signal. Masking stays strict (unknown capitalized words are masked); invention now uses a much wider allowlist, and both directions are tested. Raw prompts and drafts are persisted under the profile's gitignored dogfood/raw for human review, and stripped from CLI JSON and receipts. Co-authored-by: Cursor <cursoragent@cursor.com>
A [ENTITY] placeholder in the prompt is an instruction the model follows: one holdout draft came back as its own redacted exemplars with dozens of literal [ENTITY] markers. Names are now cut out and the text tidied, so nothing in the prompt advertises a slot to fill. Parrot rejection was one exact 8-gram window against the unredacted exemplars, which a dump of the redacted text walks straight past. It now also fails on prompt scaffolding in the output (bracketed placeholders, section headers, separator lines) and on 5-gram coverage for drafts long enough to measure, and it compares against the redacted exemplars the model actually saw. Exemplars are clipped to 120 words (five retrieved pieces filled a 60 KB prompt), and the prompt asks for a new post from the BRIEF with the EXAMPLES as rhythm reference only. The holdout eval now disqualifies a guard-failing draft from winning: a dump has near-perfect axis distance because it is the author's own text, which is how it got crowned. Co-authored-by: Cursor <cursoragent@cursor.com>
Redaction tidy-up was stripping every "'s" in the exemplar, not just the one left behind by a cut name, so "That's it" reached the model as "That it" — mangled rhythm in the very text the prompt asks it to imitate. The holdout re-run's one RAG "win" was the mined bullets handed straight back: no exemplar copied, so the parrot check passed, and a brief echo has excellent axis distance. The brief now joins the copy-check pool alongside the exemplars. Co-authored-by: Cursor <cursoragent@cursor.com>
Folding the brief into the parrot pool disqualified every draft in both arms: an exact shared window with the brief is expected, because the brief is where the content comes from. Brief echo is now its own looser check — near-total coverage only — so it catches the bullets handed back without condemning a post that legitimately uses them. Co-authored-by: Cursor <cursoragent@cursor.com>
The rule banning bracketed placeholders spelled one out, putting the exact string we do not want back in front of the model. Co-authored-by: Cursor <cursoragent@cursor.com>
Normalize corpus text before embedding and storage so article-export CSS cannot enter prompts. Cap mined briefs at three short bullets and fail closed on excessive source overlap. Co-authored-by: Cursor <cursoragent@cursor.com>
Route retrieval text through normalize_corpus_text so LinkedIn CSS/HTML junk never enters exemplars. Cap mined briefs and reject high source overlap so the eval cannot hand the model its own answer. Co-authored-by: Cursor <cursoragent@cursor.com>
Voice reached the model only as five retrieved posts, and it answered by
continuing them: every RAG draft came back a multi-post dump while the
exemplar-free arm wrote clean posts. The style profile Lane C already
builds was never read by the write path.
Carry voice as measured cadence targets (sentence length, short-line
ratio, post length, pronoun lean, banned filler) and cut exemplars to two
short excerpts, so the signal survives with nothing to paste.
Brief mining cut sentences at a fixed word count, which manufactured
fragments ("IATA is now", "and one more time, please") that the writer
could only glue back together. Fit bullets to clause boundaries instead
and drop tails that leave a phrase mid-air.
Co-authored-by: Cursor <cursoragent@cursor.com>
The model wrote a strong opening then recycled its own lines until the
token budget ran out ("Say no. Say no. Say no."), so a good post lost to a
short one on axis distance. Lowering max_tokens only moves the cut
mid-sentence, so the tail is dropped after generation: paragraphs that
restate an earlier one go, then the draft stops at the last paragraph
boundary inside the measured word target.
Both arms get the same trim; applying a length edit to one side would
decide the comparison by itself.
Co-authored-by: Cursor <cursoragent@cursor.com>
The distance metric mixed raw pronoun counts with rates, so it measured length instead of voice: a long draft paid a penalty per extra "you" while a three-sentence stub sat at 0.08 from a 190-word post. Base "won" by not writing a post. Compare pronoun rates per 1k words and add a symmetric log-length term, so a draft that is not the length of a post is penalized for it. Enforce the same word ceiling the prompt states, rather than a looser one. Co-authored-by: Cursor <cursoragent@cursor.com>
A lone median sentence length made the writer produce eighteen uniform one-line sentences, which reads as a caricature of the voice and lost to a draft with real paragraph structure. Measure the quartile spread and the share of paragraphs carrying more than one sentence, and state both. Co-authored-by: Cursor <cursoragent@cursor.com>
The corpus separates paragraphs with a single newline, so splitting only on blank lines counted each stored post as one paragraph and put a false 100% in the voice card. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Present the write path without internal experiment framing, failure history, or "legacy because it failed" language. Advanced train/filter commands stay as a short optional section. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
index-voice→build-style-profile→writeon Qwen3.5-9B-4bit withadapter=none(no LoRA).rag_beats_base=true, no RAG disqualifications. README updated to match.Test plan
pytestgreen (259+ Contoso-safe tests)personality-protect index-voice && build-style-profile && write --topic ... --points ...personality-protect eval-write-holdoutwith three holdout ids; receipt showsrag_beats_basestatusreportsvoice_mode=rag/adapter=noneMade with Cursor