You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Measuring 32 texts from the calibration corpus three ways — the whole document, a ~400-word excerpt of it, and that excerpt after a model rewrote it — produced this:
Flagged at 25/100
The whole documents
0 / 32
A ~400-word excerpt of each
6 / 32 (18.8%)
The same excerpt, rewritten
9 / 32
Same authors, same subjects, same sentences. The only change in the middle row is where the scissors fell.
Median score goes 6.1 → 12.3 and median burstiness 0.51 → 0.48, but the tail is what crosses: individual documents move 4.5 → 29.6, 6.2 → 31.9, 10.0 → 40.0, 4.4 → 36.5.
Full method and numbers: Docs/PARAPHRASE.md, section Length. Reproduce with tools/SignsOfAI.Calibration -- excerpt then -- paraphrase.
Why it happens
stat.burstiness is the coefficient of variation of sentence length. A ~400-word window holds perhaps twenty sentences, and the long three-clause sentence and the short fragment that together make a paragraph read as human may not both fall inside it. The estimate does not merely get noisier — a reader could allow for that. It moves in one direction, toward the machine.
Why it is a defect and not a caveat
VerdictBands.Holds(score, language) already refuses a verdict when the calibration corpus never measured that language, on the principle that a bound measured on one population must not be spent on another. Length is such a population. The shipped boundary of 25/100 was measured on documents with a median of 3,241 words, and today the same boundary is applied to a pasted paragraph with nothing in the interface to say so.
This is the same shape as #36, where the report inherited a false-positive rate measured on other languages. The fix pattern exists.
Suggested direction
Add a length dimension to VerdictBands, so a document below the measured range gets its score and an explicit "this is below what has been measured" rather than a verdict.
Decide the floor from the corpus rather than by choosing it: the calibration tool can report the flag rate at a range of excerpt lengths and the floor comes out of that table, the same way the 25/100 boundary does.
Do not raise the global threshold to make short text safe. That would spend the false-positive budget of long documents to fix a problem that only exists in short ones, and the published 0/90 would stop being true of the case it was measured on.
What was measured
Measuring 32 texts from the calibration corpus three ways — the whole document, a ~400-word excerpt of it, and that excerpt after a model rewrote it — produced this:
Same authors, same subjects, same sentences. The only change in the middle row is where the scissors fell.
Median score goes 6.1 → 12.3 and median burstiness 0.51 → 0.48, but the tail is what crosses: individual documents move 4.5 → 29.6, 6.2 → 31.9, 10.0 → 40.0, 4.4 → 36.5.
Full method and numbers:
Docs/PARAPHRASE.md, section Length. Reproduce withtools/SignsOfAI.Calibration -- excerptthen-- paraphrase.Why it happens
stat.burstinessis the coefficient of variation of sentence length. A ~400-word window holds perhaps twenty sentences, and the long three-clause sentence and the short fragment that together make a paragraph read as human may not both fall inside it. The estimate does not merely get noisier — a reader could allow for that. It moves in one direction, toward the machine.Why it is a defect and not a caveat
VerdictBands.Holds(score, language)already refuses a verdict when the calibration corpus never measured that language, on the principle that a bound measured on one population must not be spent on another. Length is such a population. The shipped boundary of 25/100 was measured on documents with a median of 3,241 words, and today the same boundary is applied to a pasted paragraph with nothing in the interface to say so.This is the same shape as #36, where the report inherited a false-positive rate measured on other languages. The fix pattern exists.
Suggested direction
VerdictBands, so a document below the measured range gets its score and an explicit "this is below what has been measured" rather than a verdict.VerdictBandsrather than re-deciding, since that was the failure The verdict bands were chosen by hand; the calibration is measured. Reconcile them. #32 fixed.What not to do
Do not raise the global threshold to make short text safe. That would spend the false-positive budget of long documents to fix a problem that only exists in short ones, and the published 0/90 would stop being true of the case it was measured on.