Skip to content

Give the article channel its own length band and a holdout eval - #49

Merged
telivity-otaip merged 3 commits into
mainfrom
feature/article-voice-eval
Jul 31, 2026
Merged

Give the article channel its own length band and a holdout eval#49
telivity-otaip merged 3 commits into
mainfrom
feature/article-voice-eval

Conversation

@telivity-otaip

Copy link
Copy Markdown
Contributor

What this changes

The article channel could already outline, draft sections and stitch them. Two things were missing: it took its length from the post band, and nothing measured whether the result sounded like the author.

Length comes from articles now

style_profile measures linkedin_article pieces separately from linkedin_post pieces. The article aim is the median of the author's own articles (clamped 600–3000 words), the ceiling is p90, and the aim is divided across the outline into a per-section budget (clamped 180–600 words) so a six-section article asks for six sections rather than six posts. With no articles in the corpus the aim falls back to a stated 1100-word default instead of borrowing the post band.

Three mechanical defects surfaced while wiring this up and are fixed here:

  • article retrieval was drawing from the whole corpus, so posts could become the rhythm reference for an article section; it is restricted to linkedin_article pieces.
  • the corpus floor counted pieces on disk, so a carve large enough to empty article retrieval still passed; it now checks the voice index too and errors instead of quietly drafting from nothing.
  • independently drafted sections sometimes restated each other; near-duplicate sections are dropped before stitching.

The eval

personality-protect select-article-holdouts --apply
personality-protect index-voice --from-carve
PP_MLX_ALLOW=1 personality-protect eval-write-article --out receipt.json

The carve is deterministic, keeps previously carved ids pinned, and refuses to drop the voice index below its five-article floor. Each holdout is reduced to a lossy brief — a topic plus 3–6 bullets taken one per segment of the piece so the brief describes the shape of an article rather than its densest paragraph.

The post miner could not be reused unchanged. Its overlap cap is a share of the source, and a 25% share of a 1500-word article is a 375-word "brief"; the article cap is an absolute 60 words, so a longer source does not buy a longer head start. Both arms then write that brief under the same outline, budget and trim — the product arm with exemplars and the style card, the control arm with neither — so distance measures the voice machinery instead of the scaffolding they share.

Real MLX run: article voice is not proven

n=4 holdouts, Qwen3.5-9B-4bit, adapter none.

run wins (article/base/tie) disqualified p verdict
k=2 (default) 0 / 0 / 4 4 article, 4 base 1.0 not_supported
k=0 (no exemplars) 0 / 0 / 4 4 article, 4 base 1.0 not_supported

Every draft in both arms was disqualified for inventing entities or figures, which scores as a tie, so distance never decided a single item. That is the run's actual finding and it is about the product, not the harness: the model cannot currently expand a 60-word brief into a ~1300-word article without fabricating specifics, and the control fabricates just as much. The invention guard is left at zero tolerance — loosening a factuality gate to obtain a verdict would manufacture the result the gate exists to prevent.

Because "everything was disqualified" and "the voice arm lost on cadence" both produce zero wins, the receipt now distinguishes them explicitly (distance_ever_decided).

Underneath the disqualifications the exemplars do move cadence in the right direction — the article arm is closer to the holdout at k=2 than at k=0 on all four pieces — but that is a within-arm observation, not a result against the control, and it is not a claim this PR makes.

Notes

Made with Cursor

telivity-otaip and others added 3 commits July 30, 2026 19:10
Writer SFT briefs were mined verbatim from their own target, so the input
sat inside the output (5-gram copy ratio median 1.0) and copying the prompt
forward was a winning strategy for the objective. Add a deterministic
de-voicing operator so rows are (D(y), y): entities, figures and claim
vocabulary are preserved while second-person address, contractions, emphasis,
discourse markers, fragment rhythm and connective scaffolding are removed.

Pair construction is gated rather than merely built. Every row is measured
against the shipped pair gate for cadence movement, checked for entities or
figures the operator invented, and dropped unless what the brief shares with
the post falls under a cap. Copy ratio drops from median 1.0 to 0.15.

Also here:
- select-writer-holdouts: deterministic, pinned-compatible holdout carve
  sized as a share of the briefable pool. A three-item gate cannot reach any
  significance threshold; a sign test needs more comparisons than that.
- eval-writer-adapter: committed ship gate for RAG+adapter vs RAG-alone,
  loading each arm's weights once. Keeps an adapter only on a majority win
  that clears a one-sided sign test and does not raise the disqualification
  rate.
- index-voice --from-carve: read holdout ids from the carve file instead of
  retyping them as flags.
- train --detach: portable detached launch via start_new_session, so an
  unattended run does not depend on a shell staying open (and does not
  depend on setsid, which macOS does not ship).
- Writer train recipe: 16 layers, LoRA rank 16, lr 3e-5, 10 epochs, with
  rank and learning rate plumbed through the chunk worker and recorded in
  the checkpoint meta.
- release_mlx_memory now honours the MLX opt-in gate; a Metal-less session
  aborts in C++ where except Exception cannot catch it.

Co-authored-by: Cursor <cursoragent@cursor.com>
The article channel inherited the post length band, so "articles" came out
at post length and no measurement said otherwise. Length now comes from the
author's own linkedin_article pieces (median aim, p90 ceiling) divided across
the outline into a per-section budget, and article retrieval is restricted to
articles so a post cannot become the rhythm reference for a section.

Article voice was also unproven, which is a different problem from being
wrong. select-article-holdouts carves articles deterministically without
dropping retrieval below its floor, each holdout is reduced to a lossy
outline brief that is capped in absolute words rather than as a share of a
source that keeps growing, and eval-write-article writes that brief twice --
once with exemplars and the style card, once with neither -- under the same
outline and budget, so distance measures the voice machinery and not the
scaffolding both arms share.

Co-authored-by: Cursor <cursoragent@cursor.com>
An item where both arms are disqualified scores as a tie, so a run in which
every draft was rejected before distance was consulted reported the same
"article_did_not_win_majority" as a run the voice arm was measured and lost.
Those are different results and only one is evidence about voice. The receipt
now carries distance_ever_decided and names the all-disqualified case.

Co-authored-by: Cursor <cursoragent@cursor.com>
@telivity-otaip
telivity-otaip merged commit b60028a into main Jul 31, 2026
5 checks passed
telivity-otaip added a commit that referenced this pull request Jul 31, 2026
Keep writer-adapter CLI from #48 and article holdout/eval CLI from #49.
index-voice --from-carve excludes both carves.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant