Give the article channel its own length band and a holdout eval - #49
Merged
Conversation
Writer SFT briefs were mined verbatim from their own target, so the input sat inside the output (5-gram copy ratio median 1.0) and copying the prompt forward was a winning strategy for the objective. Add a deterministic de-voicing operator so rows are (D(y), y): entities, figures and claim vocabulary are preserved while second-person address, contractions, emphasis, discourse markers, fragment rhythm and connective scaffolding are removed. Pair construction is gated rather than merely built. Every row is measured against the shipped pair gate for cadence movement, checked for entities or figures the operator invented, and dropped unless what the brief shares with the post falls under a cap. Copy ratio drops from median 1.0 to 0.15. Also here: - select-writer-holdouts: deterministic, pinned-compatible holdout carve sized as a share of the briefable pool. A three-item gate cannot reach any significance threshold; a sign test needs more comparisons than that. - eval-writer-adapter: committed ship gate for RAG+adapter vs RAG-alone, loading each arm's weights once. Keeps an adapter only on a majority win that clears a one-sided sign test and does not raise the disqualification rate. - index-voice --from-carve: read holdout ids from the carve file instead of retyping them as flags. - train --detach: portable detached launch via start_new_session, so an unattended run does not depend on a shell staying open (and does not depend on setsid, which macOS does not ship). - Writer train recipe: 16 layers, LoRA rank 16, lr 3e-5, 10 epochs, with rank and learning rate plumbed through the chunk worker and recorded in the checkpoint meta. - release_mlx_memory now honours the MLX opt-in gate; a Metal-less session aborts in C++ where except Exception cannot catch it. Co-authored-by: Cursor <cursoragent@cursor.com>
The article channel inherited the post length band, so "articles" came out at post length and no measurement said otherwise. Length now comes from the author's own linkedin_article pieces (median aim, p90 ceiling) divided across the outline into a per-section budget, and article retrieval is restricted to articles so a post cannot become the rhythm reference for a section. Article voice was also unproven, which is a different problem from being wrong. select-article-holdouts carves articles deterministically without dropping retrieval below its floor, each holdout is reduced to a lossy outline brief that is capped in absolute words rather than as a share of a source that keeps growing, and eval-write-article writes that brief twice -- once with exemplars and the style card, once with neither -- under the same outline and budget, so distance measures the voice machinery and not the scaffolding both arms share. Co-authored-by: Cursor <cursoragent@cursor.com>
An item where both arms are disqualified scores as a tie, so a run in which every draft was rejected before distance was consulted reported the same "article_did_not_win_majority" as a run the voice arm was measured and lost. Those are different results and only one is evidence about voice. The receipt now carries distance_ever_decided and names the all-disqualified case. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The article channel could already outline, draft sections and stitch them. Two things were missing: it took its length from the post band, and nothing measured whether the result sounded like the author.
Length comes from articles now
style_profilemeasureslinkedin_articlepieces separately fromlinkedin_postpieces. The article aim is the median of the author's own articles (clamped 600–3000 words), the ceiling is p90, and the aim is divided across the outline into a per-section budget (clamped 180–600 words) so a six-section article asks for six sections rather than six posts. With no articles in the corpus the aim falls back to a stated 1100-word default instead of borrowing the post band.Three mechanical defects surfaced while wiring this up and are fixed here:
linkedin_articlepieces.The eval
The carve is deterministic, keeps previously carved ids pinned, and refuses to drop the voice index below its five-article floor. Each holdout is reduced to a lossy brief — a topic plus 3–6 bullets taken one per segment of the piece so the brief describes the shape of an article rather than its densest paragraph.
The post miner could not be reused unchanged. Its overlap cap is a share of the source, and a 25% share of a 1500-word article is a 375-word "brief"; the article cap is an absolute 60 words, so a longer source does not buy a longer head start. Both arms then write that brief under the same outline, budget and trim — the product arm with exemplars and the style card, the control arm with neither — so distance measures the voice machinery instead of the scaffolding they share.
Real MLX run: article voice is not proven
n=4 holdouts,
Qwen3.5-9B-4bit, adapter none.not_supportednot_supportedEvery draft in both arms was disqualified for inventing entities or figures, which scores as a tie, so distance never decided a single item. That is the run's actual finding and it is about the product, not the harness: the model cannot currently expand a 60-word brief into a ~1300-word article without fabricating specifics, and the control fabricates just as much. The invention guard is left at zero tolerance — loosening a factuality gate to obtain a verdict would manufacture the result the gate exists to prevent.
Because "everything was disqualified" and "the voice arm lost on cadence" both produce zero wins, the receipt now distinguishes them explicitly (
distance_ever_decided).Underneath the disqualifications the exemplars do move cadence in the right direction — the article arm is closer to the holdout at k=2 than at k=0 on all four pieces — but that is a within-arm observation, not a result against the control, and it is not a claim this PR makes.
Notes
writeis unchanged for the post channel and still runs base weights.ruff checkclean.Made with Cursor