Repair inventing article sections instead of dropping them - #51
Merged
Conversation
A section that invented was regenerated once on a shorter budget and then dropped, which told the model to invent less without telling it what it had invented. Any drop also set invent_reject for the whole article, so an otherwise clean stitch was disqualified for a gap in coverage. On the article holdout that disqualified three items of four, one of them a zero-word draft scored against a control arm that had written 135 words. The regenerate now names the fabricated entities and figures it has to lose. If they survive that, the sentences carrying them are cut out and the section is re-checked — the sentence rather than the span, because removing only the name leaves the fabricated claim standing without its subject. A section is dropped only when nothing usable is left, and the whole-article invent flag fires on an empty stitch or a stitched text that still invents, not on the drop itself. The control arm builds its sections through the same path. A fact-lock only the product arm had to survive would separate the arms by editing policy rather than by writing. Article holdout, same carve and model: wins article 3 / base 1 (was 1 / 3), invent disqualifications 0 of 4 on both arms (was 3 of 4 on the article arm), and no arm now stitches an empty draft. The verdict stays not_supported for one remaining reason: with four holdouts a 3-1 margin cannot clear alpha=0.10. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
In the article channel a section that invented was regenerated once on a shorter budget and then dropped. The retry told the model to invent less without telling it what it had invented, and any drop set
invent_rejectfor the whole article — so a stitch whose remaining text was clean was disqualified for a gap in coverage rather than for a fabrication.On the article holdout after #50 that disqualified three items of four, including one arm that stitched zero words and was scored against a control that had written 135.
What changed
section_repair_directives): the regenerate names the fabricated entities and figures the section has to remove, forbids replacements, and asks for a shorter section. Naming offenders is a calculated risk with this model, which is why the scrub stands behind it.scrub_invented_sentences): if the repair still invents, the sentences carrying the offenders are cut and the section is re-checked against invent + parrot. The unit of removal is the sentence, not the span — cutting only the name out of "Fabrikam shipped the migration in nine weeks" leaves the fabricated claim standing with its subject missing. Lineation survives, so a scrubbed section keeps its paragraph rhythm.draft_section_with_repairhelper. A fact-lock only the product arm had to survive would separate the arms by editing policy rather than by writing.base_attempts, as counts only — the outline titles are mined from the holdout body.Holdout result
Same carve, same model,
k=2, real MLX run:Repair and scrub both fire on both arms, and the two items that previously lost only to an invent DQ are now the two best cadence distances in the run.
The verdict stays
not_supportedfor one remaining reason:margin_within_chance. With four holdouts a 3–1 margin reaches p=0.3125, and even a clean sweep of four cannot clear alpha=0.10 by much — that is the carve size, not a fabrication problem, and the receipt says so rather than shipping a claim.Test plan
pytest -q— 377 passedruff check src testsscripts/sanitize_check.shPP_MLX_ALLOW=1 eval-write-articleon the pinned carve; Contoso-safe receipt only.Made with Cursor