Skip to content

Repair inventing article sections instead of dropping them - #51

Merged
telivity-otaip merged 1 commit into
mainfrom
feature/article-invent-repair
Aug 1, 2026
Merged

Repair inventing article sections instead of dropping them#51
telivity-otaip merged 1 commit into
mainfrom
feature/article-invent-repair

Conversation

@telivity-otaip

Copy link
Copy Markdown
Contributor

Why

In the article channel a section that invented was regenerated once on a shorter budget and then dropped. The retry told the model to invent less without telling it what it had invented, and any drop set invent_reject for the whole article — so a stitch whose remaining text was clean was disqualified for a gap in coverage rather than for a fabrication.

On the article holdout after #50 that disqualified three items of four, including one arm that stitched zero words and was scored against a control that had written 135.

What changed

  • Invent repair pass (section_repair_directives): the regenerate names the fabricated entities and figures the section has to remove, forbids replacements, and asks for a shorter section. Naming offenders is a calculated risk with this model, which is why the scrub stands behind it.
  • Mechanical scrub (scrub_invented_sentences): if the repair still invents, the sentences carrying the offenders are cut and the section is re-checked against invent + parrot. The unit of removal is the sentence, not the span — cutting only the name out of "Fabrikam shipped the migration in nine weeks" leaves the fabricated claim standing with its subject missing. Lineation survives, so a scrubbed section keeps its paragraph rhythm.
  • Drop is last, not second. A section is omitted only when repair and scrub both leave nothing usable.
  • Final invent flag now fires when the stitch is empty or the stitched text invents against the visible brief — not because a section was dropped. Dropped-section offenders are still reported, under their own keys.
  • Control arm parity. Both arms build sections through one draft_section_with_repair helper. A fact-lock only the product arm had to survive would separate the arms by editing policy rather than by writing.
  • Receipts gained repair/scrub/drop counts and base_attempts, as counts only — the outline titles are mined from the holdout body.
  • README: the article channel and eval paragraphs describe repair → scrub → drop.

Holdout result

Same carve, same model, k=2, real MLX run:

post-#50 this branch
wins (article / base) 1 / 3 3 / 1
invent disqualifications article 3 of 4 article 0 of 4, base 0 of 4
empty article drafts 1 0

Repair and scrub both fire on both arms, and the two items that previously lost only to an invent DQ are now the two best cadence distances in the run.

The verdict stays not_supported for one remaining reason: margin_within_chance. With four holdouts a 3–1 margin reaches p=0.3125, and even a clean sweep of four cannot clear alpha=0.10 by much — that is the carve size, not a fabrication problem, and the receipt says so rather than shipping a claim.

Test plan

  • pytest -q — 377 passed
  • ruff check src tests
  • scripts/sanitize_check.sh
  • Contoso coverage: repair keeps the section and names the offenders; scrub keeps the clean sentences when repair fails; a section is emptied only when both fail; a dropped section does not disqualify a clean stitch; the control arm repairs, scrubs, and still disqualifies when it cannot be repaired.
  • Real PP_MLX_ALLOW=1 eval-write-article on the pinned carve; Contoso-safe receipt only.

Made with Cursor

A section that invented was regenerated once on a shorter budget and then
dropped, which told the model to invent less without telling it what it had
invented. Any drop also set invent_reject for the whole article, so an
otherwise clean stitch was disqualified for a gap in coverage. On the article
holdout that disqualified three items of four, one of them a zero-word draft
scored against a control arm that had written 135 words.

The regenerate now names the fabricated entities and figures it has to lose. If
they survive that, the sentences carrying them are cut out and the section is
re-checked — the sentence rather than the span, because removing only the name
leaves the fabricated claim standing without its subject. A section is dropped
only when nothing usable is left, and the whole-article invent flag fires on an
empty stitch or a stitched text that still invents, not on the drop itself.

The control arm builds its sections through the same path. A fact-lock only the
product arm had to survive would separate the arms by editing policy rather
than by writing.

Article holdout, same carve and model: wins article 3 / base 1 (was 1 / 3),
invent disqualifications 0 of 4 on both arms (was 3 of 4 on the article arm),
and no arm now stitches an empty draft. The verdict stays not_supported for one
remaining reason: with four holdouts a 3-1 margin cannot clear alpha=0.10.

Co-authored-by: Cursor <cursoragent@cursor.com>
@telivity-otaip
telivity-otaip merged commit 5e50699 into main Aug 1, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant