diff --git a/.gitignore b/.gitignore index ad80d29..4c3193e 100644 --- a/.gitignore +++ b/.gitignore @@ -40,3 +40,8 @@ temp/ # Calibration texts are fetched, not redistributed: the manifest is the repo artefact. Docs/Calibration/texts/ +# Paraphrase-study passages are derivatives of the same licensed sources, and are excluded for the +# same reason: the manifest with both halves' hashes is the repo artefact. +Docs/Paraphrase/human/ +Docs/Paraphrase/rewritten/ + diff --git a/Docs/PARAPHRASE.md b/Docs/PARAPHRASE.md new file mode 100644 index 0000000..c00d288 --- /dev/null +++ b/Docs/PARAPHRASE.md @@ -0,0 +1,201 @@ +# What a machine rewrite does to human writing + + + +Claude models released on or after 2 August 2026 carry a machine-readable watermark, and Anthropic says earlier ones will follow during a transition period. The technique is a version of SynthID-Text, published by Google DeepMind in 2024 and already in use elsewhere — this is an industry practice arriving at one more vendor, not a new invention. The mark lives in which words the model chose, so cleaning invisible characters cannot touch it, and the vendor's help centre names heavy editing, paraphrasing, translation, mixing into other writing and very short passages as things that leave it undetectable. Services selling that rewrite already exist. + +This page measures what such a rewrite does to a passage, using passages whose author was demonstrably a person. + +## What was measured + +- **Pairs** 32 passages, each measured as written and as rewritten +- **Drawn from** corpus `signsofai-human-baseline`, the same texts as `Docs/CALIBRATION.md` +- **Rewritten by** Claude Opus 5 (claude-opus-5) on 2026-08-19 +- **Instruction** `Docs/Paraphrase/instruction.md`, stored verbatim +- **Engine** SignsOfAI.Core 0.4.0, verdict boundary 25/100 +- **Pair fingerprint** `9fb9e9798f5b2a78` +- **Run** 2026-08-19 + +Every human half was published before generative models existed, which is the whole basis for calling it human and the same basis the calibration page rests on. Both halves are the same passage at roughly the same length, so the comparison is within a text rather than between two populations: no collection of machine-written prose was assembled, and none was needed. The pairs are cut from the **opening** of each document, which matters and is measured under Length. + +**This half of the study ages, and the human half does not.** A 2019 paper will still have been written in 2019 a decade from now; the rewrite is the work of one model on one day. That is why the model and the date are printed above rather than buried, and why the right response to "but a newer model rewrites differently" is to re-run this with that model rather than to argue about it. + +## The short version + +**Having a language model rewrite a passage did not change whether this tool flagged it. Cutting the same passage down to four hundred words did.** + +That is the opposite of what this study set out to find, and the second half of it is a fault in this tool rather than a finding about anybody's watermark. Neither sentence should be read as *the tool catches rewritten work*: it does not, and this page is the evidence that it does not. + +| Same writing, measured as | Flagged at 25/100 | Rate | 95% interval | +|---|---|---|---| +| whole documents | 0 / 32 | 0% | 0% – 10.7% | +| 400-word windows, three positions each | 13 / 89 | 14.6% | 8.7% – 23.4% | +| the opening window, as written | 6 / 32 | 18.8% | 8.9% – 35.3% | +| the opening window, rewritten | 9 / 32 | 28.1% | 15.6% – 45.4% | + +### The rewrite, measured against the passage it came from + +5 passages crossed the boundary that had not, 2 crossed back, and the other 25 stayed where they were. McNemar's exact test on the discordant pairs gives p = 0.453. With 7 pairs changing side at all, only a clean sweep would have reached significance, so the honest statement is that **this study cannot show that rewriting changes whether a passage is flagged, in either direction** — not that it showed rewriting is safe, and not that it showed rewriting is caught. + +## Was the treatment actually applied + +The instruction asks that no run of eight or more consecutive words survive. Whether it did is checked here rather than attested, because the first version of this study attested it by hand and the hand was wrong. + +- **20 of 32 pairs** contain at least one surviving run of 8 words or more +- **Longest run** 86 words +- **Share of each rewrite inside such runs** median 2.4%, highest 31.8% + +| Pair | Longest run | Share retained | +|---|---|---| +| `wp-en-1025671620` | 86 | 26.1% | +| `wp-es-134286913` | 81 | 31.8% | +| `plos-0223139` | 54 | 13.4% | +| `wp-es-132719628` | 36 | 22.7% | +| `plos-0222669` | 30 | 18.1% | +| `plos-0222721` | 29 | 11.3% | +| `wp-es-132851509` | 19 | 26.7% | +| `wp-en-1025398061` | 14 | 9.1% | + +The long runs are quotations. A passage that quotes a court ruling, a political pamphlet or a published definition cannot have those words replaced without falsifying them, and the instruction's own requirement to preserve every fact and citation marker conflicts with its word-run limit; the protocol never said which wins. The shorter runs are names, dates and technical terms. + +**The consequence is that the delivered treatment is a partial rewrite**, weaker than the one the opening of this page describes. That biases every paired difference toward zero — toward the null result reported above. The null is therefore measured under a treatment milder than intended, which makes it weaker evidence than it looks, not stronger. + +## Length, which turned out to matter more than rewriting + +The verdict boundary this build ships was measured on whole documents — a median of 3,241 words here — and none of them reaches it. Cut 400-word windows out of that same writing and some of them do, with no author, subject or sentence altered. + +| Window position | Windows | Flagged | Rate | 95% interval | Median score | +|---|---|---|---|---|---| +| opening | 32 | 6 | 18.8% | 8.9% – 35.3% | 12.3 | +| middle | 30 | 4 | 13.3% | 5.3% – 29.7% | 7.6 | +| late | 27 | 3 | 11.1% | 3.9% – 28.1% | 11.5 | + +**Where the window falls changes the answer.** The opening of a document scores highest, which is not surprising once stated — for a research article that is the abstract and introduction, for an encyclopedia entry the lead, and both are the most summary-shaped prose their genre contains. The first version of this study cut only from the opening and reported the resulting rate as a length effect; it was a length effect with a genre effect inside it. + +Counted by document rather than by window, of the 30 documents that yielded more than one window: **11 are flagged at one position but not another**, and 0 are flagged wherever the window falls. Whether one of these authors would be accused depends on which four hundred words somebody happened to paste. + +| | Median score | 90th percentile | Highest | +|---|---|---|---| +| Whole document | 6.1 | 13.9 | 18.0 | +| Whole document, apparatus stripped | 5.7 | 13.9 | 21.7 | +| 400-word windows, all positions | 10.2 | 30.1 | 40.0 | + +The second row is a control, and it matters: the whole documents carry figure captions and boilerplate that the windows have stripped, so the gap could have been composition rather than length. With the same filter applied and no length cut, 0 of 32 are flagged. It is length. + +The mechanism is not mysterious. Burstiness is the spread of sentence lengths, and a short window holds few sentences: the long one with three clauses and the two-word fragment that together make a paragraph look human may simply not both be inside it. The measurement does not become uncertain, which a reader could allow for. It moves, in one direction, toward the machine. + +**This is a defect in this tool, published here because that is what this page is for**, and tracked as issue #59. The boundary carries a language condition already — a text in a language the corpus never measured gets no verdict — and the reason is exactly the reason that applies here: a bound measured on one population must not be spent on another. Length is such a population and the code does not yet know it. Until it does, treat a verdict on anything short as unmeasured, whatever the interface says. + +One caution about fixing it, which is why #59 is not a one-line change: these windows are **cut** from long formal documents, and a student's four-hundred-word answer was **composed** at that length. Its sentence lengths are a whole distribution, not a truncated one. A floor measured on truncations and enforced against compositions would repeat, in a new dimension, the mistake it exists to prevent. + +It also lands on the same ground as the watermark it was written about. Anthropic says its mark is unreliable on short samples because few word choices carry little information. This tool is unreliable on short samples because few sentences carry little rhythm. Two unrelated methods, the same floor, and a teacher holding one paragraph of a student's work is below it either way. + +## How far the score moved + +| | Median | 90th percentile | Highest | +|---|---|---|---| +| As written | 12.3 | 31.7 | 40.0 | +| Rewritten | 7.9 | 31.4 | 38.2 | + +The score rose in 9 pairs, fell in 18 and was unchanged in 5. Taking each passage against itself, the median change is -2.0 points (95% interval -5.6 to 0.0), with the middle half between -6.4 and +3.2. An exact sign test on the 27 pairs that moved gives p = 0.122, and Wilcoxon signed-rank p = 0.478. **The direction is not established.** The rewrite strips the vocabulary of academic prose — *furthermore*, *comprehensive*, *facilitate*, *utilizar* — while flattening sentence rhythm, and on this sample the two effects are not separable from each other or from noise. + +## Burstiness + +Sentence-length variation is the signal this project trusts most, because it needs no word list and no model. Human prose is uneven; unprompted model output tends to settle into a width and stay there. + +- **As written** median 0.48, range 0.21 – 0.69 +- **Rewritten** median 0.40, range 0.24 – 0.72 +- **Paired change** median -0.03 (95% interval -0.08 to +0.01), falling in 20 of 32 pairs + +This is the closest the study comes to a directional effect, and it is not clean. An exact sign test on 20 falls against 12 rises gives p = 0.215 — by the same standard applied to the flag counts above, not a result. Wilcoxon signed-rank, which uses the size of each change and not only its direction, gives p = 0.038. Neither test was chosen in advance, so the honest reading is that **the rewrite probably flattens rhythm slightly and this sample cannot settle it**. A few hundredths on a scale where human prose runs from roughly 0.2 to 0.9 moves almost nothing on its own. + +## Which signals the rewrite moved + +| Rule | Passages where it appeared only after the rewrite | +|---|---| +| `stat.burstiness` | 4 | +| `lex.ademas` | 2 | +| `lex.moreover` | 2 | +| `lex.simply` | 2 | +| `rhet.rule-of-three` | 2 | +| `lex.actually` | 1 | +| `lex.asimismo` | 1 | +| `rhet.when-it-comes` | 1 | +| `syn.serves-as` | 1 | + +Signals the rewrite *removed* are worth as much, since a rewrite is not obliged to move in one direction: + +| Rule | Passages where it disappeared | +|---|---| +| `lex.moreover` | 3 | +| `rhet.rule-of-three` | 3 | +| `lex.comprehensive` | 2 | +| `lex.facilitate` | 2 | +| `lex.furthermore` | 2 | +| `lex.importantly` | 2 | +| `lex.intricate` | 2 | +| `lex.utilizar` | 2 | +| `rhet.weasel-attribution` | 2 | +| `lex.actually` | 1 | +| `lex.ademas` | 1 | +| `lex.just` | 1 | +| `lex.literally` | 1 | +| `lex.panorama` | 1 | +| `lex.paramount` | 1 | +| `lex.profound` | 1 | +| `lex.profundo` | 1 | +| `lex.robust` | 1 | +| `rhet.in-order-to` | 1 | +| `rhet.in-this-article` | 1 | + +*2 further rule(s) moved in one passage each and are not listed: `rhet.with-regard-to`, `stat.burstiness`.* + +## The character layer, which is a different thing + +The character scanner found 7 suspicious codepoints before the rewrite and 7 after it. + +Those totals are not spread across the corpus: they are one passage — `wp-es-132719628` — and the codepoints are `U+0131`. In `wp-es-132719628` they are Turkish dotless ı inside Turkish proper names, in a Spanish article about a Turkish organisation. **That is a false positive of the character scanner**, and worth stating on a page that criticises other people's. + +The count not moving is the expected result and worth stating plainly: homoglyphs and zero-width characters are a *fingerprint of a tool that touched the file*, not of a model that wrote it. Rewriting prose does not produce them, and the free half of every watermark-removal service — the half that strips invisible characters — is aimed at a layer that has nothing to do with the statistical watermark it advertises removing. + +## By language + +A result that holds in English and fails in Spanish is not one result. Two warnings before the table: the Spanish arm is eight pairs, all encyclopedia entries, so it is also a single-source arm and cannot be compared cleanly with an English arm that mixes research articles and encyclopedia entries. And eight pairs support no interval worth printing. + +| Group | Pairs | Flagged before | Flagged after | Median score change | Median burstiness change | +|---|---|---|---|---|---| +| **en** | 24 | 6 | 7 | -2.0 | -0.03 | +| **es** | 8 | 0 | 2 | -2.8 | -0.04 | + +| Group | Pairs | Flagged before | Flagged after | Median score change | Median burstiness change | +|---|---|---|---|---|---| +| **en-anglophone-affiliation** | 8 | 2 | 3 | -2.5 | -0.04 | +| **en-other-affiliation** | 8 | 3 | 3 | -1.6 | -0.01 | +| **en-wikipedia** | 8 | 1 | 1 | -2.6 | -0.05 | +| **es-wikipedia** | 8 | 0 | 2 | -2.8 | -0.04 | + +## The conflict of interest + +The rewriting was done by Claude Opus 5 (claude-opus-5). Two things follow that naming the model does not cover. + +**The rewriter's vendor is the vendor whose watermark prompted the study.** And more seriously, the same model wrote this project's rule packs, this tool, and the prose on this page. The instruction's defences are all at the level of the prompt — it mentions no detector and names no tell — and the contamination risk is at the level of knowledge: a model that wrote the rule for *furthermore* does not need to be told to avoid it. The removed-signals table above is exactly what tell-aware avoidance would produce. + +What argues against it is also on this page: the rewrite *introduced* tells as well as removing them, and pushed more passages over the boundary than it pulled back. A model gaming the measurement would not do that. But the objection is legitimate, it is the first one a hostile reader will raise, and the only real answer is a second rewriter from a different vendor. **Until that exists, these numbers should not be quoted anywhere a teacher will act on them.** + +## What this does not tell you + +- **Nothing about detecting a watermark.** The key is Anthropic's and no third party holds it. This measures what happens to *prose*, which is a different question that happens to be answerable without anybody's permission. +- **Nothing about how much machine writing this tool catches.** That would need a corpus of machine writing, which `Docs/Calibration/README.md` argues at length against assembling, and the argument has not changed. +- **Nothing about whether rewriting helps or hurts a student.** The study failed to establish a direction. Anyone quoting it in either direction is quoting something that is not here. +- **One model, one instruction, one day**, and the treatment was only partly delivered. All of that is above. +- **These are published articles and encyclopedia entries**, not student essays, and the windows are cut from them rather than composed at that length. A four-hundred-word paragraph somebody wrote as a paragraph has a whole distribution of sentence lengths; a truncation does not. +- **32 pairs is a small study.** Read the intervals rather than the percentages. + +Re-run it yourself: + +``` +dotnet run --project tools/SignsOfAI.Calibration -- excerpt --per-stratum 8 +# rewrite each passage in Docs/Paraphrase/human/ into Docs/Paraphrase/rewritten/ +dotnet run --project tools/SignsOfAI.Calibration -- paraphrase \ + --paraphrased-by "" --instruction Docs/Paraphrase/instruction.md +``` diff --git a/Docs/Paraphrase/README.md b/Docs/Paraphrase/README.md new file mode 100644 index 0000000..b7fe00d --- /dev/null +++ b/Docs/Paraphrase/README.md @@ -0,0 +1,143 @@ +# The paraphrase study + +`Docs/PARAPHRASE.md` reports what happens to a passage when a language model rewrites it. This folder +is what that report is made of. + +## Why it exists + +Claude models released on or after 2 August 2026 carry a machine-readable watermark, with earlier +models to follow during a transition period, and the question arrived within a week: does that make a +tool like this one redundant? The parts of the answer that can be settled by reading are settled by +reading — the key is Anthropic's, no third party can detect anything today, the mark is +vendor-specific, and the vendor's help centre lists heavy editing, paraphrasing, translation, mixing +into other writing and very short passages as conditions that leave it undetectable. + +The part that cannot be settled by reading is what a *removal* does to prose. Since only a rewrite +can disturb a watermark that lives in word choice, every working remover is a paraphraser, and +paraphrasing is something this repository can measure without anybody's key. + +Two details worth getting right, because the first version of this file got them wrong. The vendor's +help centre lists "heavily edited, paraphrased, translated, or mixed into other writing" among the +conditions that leave a mark undetectable — that is about somebody transforming Claude's output. Its +engineering write-up separately says that when *Claude itself* translates, the result is watermarked, +because Claude chose all the words. Both are true and they are about opposite directions. And the +technique is not Anthropic's: SynthID-Text was published by Google DeepMind in 2024, and Anthropic +adopted a version of it. This is one more vendor arriving at an existing industry practice. + +## The design, and why it needs no machine-written corpus + +`Docs/Calibration/README.md` argues at length against assembling a collection of machine-written text +to measure against: it is a sample of whichever models were convenient that month, it ages badly, and +it flatters whoever assembled it. That argument holds here and is not evaded. + +What replaces it is a **pair**. Each unit of the study is one passage measured twice — as its author +wrote it, and after a model rewrote it. Both halves are the same passage, by the same author, on the +same subject, at nearly the same length. The baseline is not estimated from a population; it is the +text itself, so between-author variation is removed by construction. + +What the pair does *not* remove is the tool's own sampling noise, and the length arm below shows that +noise is large at four hundred words. "Moved with the rewrite" is what this design measures; "moved +because of the rewrite" is a stronger claim and an earlier version of the report made it. + +The human halves are drawn from the calibration corpus, so every one of them was published before +generative models existed. That remains the only basis for calling writing human, and it is a +stronger one than any classifier offers about anything. + +## The controls, which were not planned + +The first run produced a baseline that did not match the published one: six of thirty-two human +passages already sat above the verdict boundary, where the calibration page reports none of ninety. +The difference is length. The excerpts are around four hundred words; the documents they were cut +from run to several thousand. + +Chasing that produced three controls, and each was added because a reviewer showed the study could +not tell two explanations apart without it. + +1. **The whole source document**, so the scissors can be told from the model. +2. **The whole document with its apparatus stripped** by the same prose filter the excerpts pass + through — otherwise the gap between arms could be figure captions and boilerplate rather than + length. It is not: those documents still flag none of thirty-two. +3. **Windows at three positions** — opening, middle, late — because the pairs are cut from the + opening, and the opening of a research article is its abstract while the opening of an + encyclopedia entry is its lead. Cutting only there measures a genre effect and calls it a length + effect. It changed the headline figure from 18.8% to 14.6%. + +Together they produced the study's largest result, which is about this tool rather than about any +watermark. `Docs/PARAPHRASE.md` reports it under **Length**, and it is tracked as issue #59. + +## What is committed, and what is not + +`pairs.json` is the artefact: what each passage is, which corpus entry it came from, its year, and +the SHA-256 of both halves. The passages themselves are git-ignored, exactly as the calibration texts +are and for the same reason — they are derivatives of CC BY and CC BY-SA sources, licences differ, +and the bulk would dwarf the code. + +`instruction.md` is committed and is the most important file here. It is the experimental treatment, +stored verbatim, and it deliberately says nothing about detectors in either direction. + +The manifest also records **which model did the rewriting and on what date**. Unlike the human corpus, +this half of the study ages: a 2019 paper will still have been written in 2019 in ten years, whereas +a rewrite is one model's work on one day. Re-running with a newer model is the answer to "but models +have moved on", and the tool refuses to run without being told the model's name. + +## Deviations from the protocol, measured rather than attested + +The first version of this section was written by hand and was wrong. It claimed one deliberate breach +of the instruction's eight-word rule and there were twenty. Compliance is now checked by the tool on +every run and printed in `Docs/PARAPHRASE.md` under **Was the treatment actually applied**, because a +project that machine-checks its false-positive rate has no business attesting its own method by hand. + +- **Twenty of the thirty-two pairs retain a verbatim run of eight words or more**, the longest 86 + words. The long runs are quotations — a court ruling, a political pamphlet, a published definition + of a lek — which cannot be reworded without falsifying them. The instruction's requirement to + preserve every fact and citation marker therefore conflicts with its no-eight-word-runs rule, and + the protocol never said which wins. A future run should say so before it starts, not afterwards. +- **The rewriter silently corrected errors in the originals**, which the instruction forbids in as + many words ("do not correct the original"). Verified cases in the Spanish arm: *intercesión* → + *se cruza*, *teoría de juego* → *teoría de juegos*, *costa* → *consta*, and + *participación en el mercado* → *reparto del mercado*, the last of which also changes the surface + meaning. The English arm shows the same thing on the second-language passages. The delivered + treatment was therefore *paraphrase plus copy-editing*, and some part of every score change is + error repair rather than rewriting. +- **This bears directly on the fairness arm.** The `en-other-affiliation` stratum exists to measure + the harm this category of tool does to people writing in a second language. A rewrite that repairs + second-language features is not a neutral treatment on that stratum, and its numbers should be read + with that in mind. +- **Two passages needed a second pass.** `wp-en-1025914326` and `wp-es-132146383` came back outside + the ±10% the instruction sets and were rewritten once more. Across the final set the lengths run + from −8.5% to +6.8%, median −1.0%. +- **The excerpts are cut from the opening of each document**, not from the middle. An earlier version + of the report said otherwise. Because the opening of a research article is its abstract and the + opening of an encyclopedia entry is its lead, this is not a neutral place to cut, and the report + now measures windows at three positions rather than asserting it does not matter. + +## Reproducing it + +```bash +# 1. cut a stratified sample of the corpus into equal-length passages +dotnet run --project tools/SignsOfAI.Calibration -- excerpt --per-stratum 8 --words 400 + +# 2. rewrite each file in Docs/Paraphrase/human/ into Docs/Paraphrase/rewritten/ under the +# same name, giving the model the instruction in instruction.md and nothing else + +# 3. measure all three arms and rewrite Docs/PARAPHRASE.md +dotnet run --project tools/SignsOfAI.Calibration -- paraphrase \ + --paraphrased-by "" --instruction Docs/Paraphrase/instruction.md +``` + +Step 2 is deliberately outside the tool. Wiring an API key into the calibration harness would make +the study reproducible only for people holding that key, and would tie a published number to one +vendor's availability. A folder of text files can be filled by any model, including one running on +the reader's own machine, and the manifest records which. + +## What would improve this + +In order: + +- **More pairs.** Thirty-two cannot separate a small effect from noise, and the report says so in + the one place it matters. Around a hundred would settle the direction. +- **A second rewriter.** Everything here is one model on one day. A local model and a competing + frontier model would show how much of the result belongs to the rewriter rather than to rewriting. +- **Passages that look like coursework.** These are published articles and encyclopedia entries. A + first-year essay is shorter, looser, and closer to the case a teacher actually faces — which, given + what the length arm found, is the gap most worth closing. diff --git a/Docs/Paraphrase/instruction.md b/Docs/Paraphrase/instruction.md new file mode 100644 index 0000000..7b53b97 --- /dev/null +++ b/Docs/Paraphrase/instruction.md @@ -0,0 +1,67 @@ +# The rewriting instruction + +This is the treatment. Every number in `Docs/PARAPHRASE.md` is a measurement of what *this +instruction*, given to the model named in the manifest, does to a passage — not of what "AI +paraphrasing" does in general. Change a word here and the study measures something else. + +It is written to imitate the only watermark-removal method that can work. A statistical text +watermark lives in which words the model chose, so stripping invisible characters cannot touch it +and only a rewrite can; Anthropic's own description says a complete rewrite where every word is +replaced will remove it. The instruction below therefore asks for exactly that, and asks for nothing +else. + +## What it deliberately does not say + +It does not say *make this sound human*, and it does not say *make this sound like AI*. Either +sentence would decide the result before the measurement: the first pushes the rewrite away from the +signals this project looks for, the second pushes it toward them, and both would produce a number +about the instruction rather than about paraphrasing. The rewriter is told to preserve meaning and +destroy wording, which is what somebody removing a watermark actually wants, and is told nothing +whatever about detectors. + +It does not mention SignsOfAI, its rule packs, or any of the tells it looks for. A rewriter that +knew what was being counted would be gaming the measurement. + +## The instruction, verbatim + +```text +Rewrite the passage below so that none of its original wording survives. Replace the vocabulary, +recast the sentences, and change the order of clauses wherever the meaning allows. No run of eight +or more consecutive words from the original may remain. + +Preserve, as closely as you can: +- the meaning, including every fact, figure, name and citation marker +- the language it is written in +- the register — an academic passage stays academic, an encyclopedic one stays encyclopedic +- the approximate length, within about ten per cent + +Do not summarise, do not expand, do not add commentary, do not correct the original, and do not +address the reader. Return only the rewritten passage. +``` + +## Why the length constraint is there + +Sentence-length variation is one of the things being measured. A rewrite free to compress a +four-hundred-word passage into two hundred would change that distribution by changing how much text +there is, and the study would not be able to tell that apart from the rewrite's own habits. Holding +length roughly constant is what makes the two halves comparable. + +## Two conflicts inside this instruction, found after it had been used + +Recorded rather than fixed. Editing the treatment after the fact would mean the numbers in +`Docs/PARAPHRASE.md` were produced by an instruction that no longer exists, which is worse than an +imperfect instruction honestly described. A future run should resolve both **before** it starts. + +**Preserve every fact versus replace every word.** A passage quoting a court ruling, a political +pamphlet or a published definition cannot have those words replaced without falsifying the +quotation. The instruction demands both and does not say which wins, so the rewriter preserved the +quotations — reasonably, but the eight-word rule is breached in twenty of the thirty-two pairs as a +result. The tool now measures this on every run and the report prints it. + +**Do not correct the original versus preserve the register.** Several passages contain errors: +*intercesión* for *intersección*, *teoría de juego* for *teoría de juegos*, and constructions a +first-language editor would change. The rewriter repaired them. That makes the delivered treatment +*paraphrase plus copy-editing*, so part of any measured change is error repair. It matters most on +the `en-other-affiliation` stratum, which exists precisely to measure the harm done to people +writing in a second language: a treatment that quietly repairs second-language features is not +neutral there. diff --git a/Docs/Paraphrase/pairs.json b/Docs/Paraphrase/pairs.json new file mode 100644 index 0000000..aa4b166 --- /dev/null +++ b/Docs/Paraphrase/pairs.json @@ -0,0 +1,362 @@ +{ + "id": "signsofai-paraphrase-effect", + "corpusId": "signsofai-human-baseline", + "paraphrasedBy": "Claude Opus 5 (claude-opus-5)", + "paraphrasedOn": "2026-08-19", + "instruction": "# The rewriting instruction\n\nThis is the treatment. Every number in \u0060Docs/PARAPHRASE.md\u0060 is a measurement of what *this\ninstruction*, given to the model named in the manifest, does to a passage \u2014 not of what \u0022AI\nparaphrasing\u0022 does in general. Change a word here and the study measures something else.\n\nIt is written to imitate the only watermark-removal method that can work. A statistical text\nwatermark lives in which words the model chose, so stripping invisible characters cannot touch it\nand only a rewrite can; Anthropic\u0027s own description says a complete rewrite where every word is\nreplaced will remove it. The instruction below therefore asks for exactly that, and asks for nothing\nelse.\n\n## What it deliberately does not say\n\nIt does not say *make this sound human*, and it does not say *make this sound like AI*. Either\nsentence would decide the result before the measurement: the first pushes the rewrite away from the\nsignals this project looks for, the second pushes it toward them, and both would produce a number\nabout the instruction rather than about paraphrasing. The rewriter is told to preserve meaning and\ndestroy wording, which is what somebody removing a watermark actually wants, and is told nothing\nwhatever about detectors.\n\nIt does not mention SignsOfAI, its rule packs, or any of the tells it looks for. A rewriter that\nknew what was being counted would be gaming the measurement.\n\n## The instruction, verbatim\n\n\u0060\u0060\u0060text\nRewrite the passage below so that none of its original wording survives. Replace the vocabulary,\nrecast the sentences, and change the order of clauses wherever the meaning allows. No run of eight\nor more consecutive words from the original may remain.\n\nPreserve, as closely as you can:\n- the meaning, including every fact, figure, name and citation marker\n- the language it is written in\n- the register \u2014 an academic passage stays academic, an encyclopedic one stays encyclopedic\n- the approximate length, within about ten per cent\n\nDo not summarise, do not expand, do not add commentary, do not correct the original, and do not\naddress the reader. Return only the rewritten passage.\n\u0060\u0060\u0060\n\n## Why the length constraint is there\n\nSentence-length variation is one of the things being measured. A rewrite free to compress a\nfour-hundred-word passage into two hundred would change that distribution by changing how much text\nthere is, and the study would not be able to tell that apart from the rewrite\u0027s own habits. Holding\nlength roughly constant is what makes the two halves comparable.\n\n## Two conflicts inside this instruction, found after it had been used\n\nRecorded rather than fixed. Editing the treatment after the fact would mean the numbers in\n\u0060Docs/PARAPHRASE.md\u0060 were produced by an instruction that no longer exists, which is worse than an\nimperfect instruction honestly described. A future run should resolve both **before** it starts.\n\n**Preserve every fact versus replace every word.** A passage quoting a court ruling, a political\npamphlet or a published definition cannot have those words replaced without falsifying the\nquotation. The instruction demands both and does not say which wins, so the rewriter preserved the\nquotations \u2014 reasonably, but the eight-word rule is breached in twenty of the thirty-two pairs as a\nresult. The tool now measures this on every run and the report prints it.\n\n**Do not correct the original versus preserve the register.** Several passages contain errors:\n*intercesi\u00F3n* for *intersecci\u00F3n*, *teor\u00EDa de juego* for *teor\u00EDa de juegos*, and constructions a\nfirst-language editor would change. The rewriter repaired them. That makes the delivered treatment\n*paraphrase plus copy-editing*, so part of any measured change is error repair. It matters most on\nthe \u0060en-other-affiliation\u0060 stratum, which exists precisely to measure the harm done to people\nwriting in a second language: a treatment that quietly repairs second-language features is not\nneutral there.\n", + "targetWords": 400, + "pairs": [ + { + "id": "plos-0216849", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0216849", + "humanSha256": "64d9aa6e2ac2abaa6ff138b0d078c658eab52d9d86c3bd93ff3785be86b753c4", + "paraphraseSha256": "73abb52df33981a2c8d19fd52fb535dbb939383a6ed2567bbb8797a76e600cdb", + "humanWords": 389, + "paraphraseWords": 401 + }, + { + "id": "plos-0220971", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0220971", + "humanSha256": "42e33049819701234f7341220b5e14fc7bc0b45b922396fb72c7a3c8580c4a5a", + "paraphraseSha256": "032422bf2befacd3553b343ef6a365c8e957455d0b2de8a0aeeeb7259224cfd7", + "humanWords": 395, + "paraphraseWords": 409 + }, + { + "id": "plos-0221645", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0221645", + "humanSha256": "7ef25f664cedefba65b2c66ffa4bef4c6fbe55453c9513ddd75de331ea74a817", + "paraphraseSha256": "5543898a3a131460fc78cd38d242940debad17b8d3171e7aae6278389a7b781a", + "humanWords": 398, + "paraphraseWords": 377 + }, + { + "id": "plos-0222669", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0222669", + "humanSha256": "43c2d84528a6f35178c10b6c12e893959409d03a4483fb38dd0d8964261fcd6f", + "paraphraseSha256": "0c4e33bc8b86912d553207d814058f60e7e3120d49104110f8d1c68156abfb90", + "humanWords": 394, + "paraphraseWords": 415 + }, + { + "id": "plos-0222683", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0222683", + "humanSha256": "2612a4313ef4b9e3432819eda80bbfee0e06654b0301022e1f90b61512ee74f3", + "paraphraseSha256": "377d6a00af227acd7fc041b641796b93dff9a2f8055dc5aa41fa3c2109b559c1", + "humanWords": 369, + "paraphraseWords": 381 + }, + { + "id": "plos-0222700", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0222700", + "humanSha256": "a01afeb65a244fc3080b94a636f2770baebcb4436549b632e55c11624c935233", + "paraphraseSha256": "2f123363ae1f8d6509d0a5d800b79bec506fa316ac561ded8ce20135624eaa02", + "humanWords": 354, + "paraphraseWords": 349 + }, + { + "id": "plos-0222721", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0222721", + "humanSha256": "39fdd0f6a47b4297c6cd2a7d28fe110b71d33208882f7efe7f50b3b11d569004", + "paraphraseSha256": "d8749c44d6646749b3ce4798081ec06173c578724830c43f02d38b99ba03813a", + "humanWords": 376, + "paraphraseWords": 383 + }, + { + "id": "plos-0222772", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0222772", + "humanSha256": "e8ea7b1cc31118176df52233bb459742b50d104858511b8cf2fe265761012120", + "paraphraseSha256": "7875f0ab26b9303340ae4567bc30779696c007019a5f9b4e03d489a54bffda88", + "humanWords": 385, + "paraphraseWords": 386 + }, + { + "id": "plos-0222981", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0222981", + "humanSha256": "fd7eecc057e4b93431c08afd34b24bfbf1993bb1e960f78d35f8de9c714ca08d", + "paraphraseSha256": "ae24ede04be5bcd6691840bc12b70a9f6f451de0dc72ce6b7eb4a939fd88b6ef", + "humanWords": 393, + "paraphraseWords": 376 + }, + { + "id": "plos-0223039", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0223039", + "humanSha256": "3a43ec5f9f52b728d310f143e5c236724e51dad74aa6959d948120d3edf78a8b", + "paraphraseSha256": "fb0ded23e328afd5d3ba86aeae4a34a4b6c63d2d26fb21a274b93d86cac6c159", + "humanWords": 381, + "paraphraseWords": 404 + }, + { + "id": "plos-0223046", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0223046", + "humanSha256": "bf9eb4844129cd0fa4f10e4a3b3a92afed86ba1ea8426d3cf29782793b88f4a7", + "paraphraseSha256": "bc82872d76019219c960e53c3bc00cbc1633487675ef7f3819cd552c9fe1506f", + "humanWords": 388, + "paraphraseWords": 400 + }, + { + "id": "plos-0223139", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0223139", + "humanSha256": "42ad69ea96408102ac6127b07a4918a989e816dee8a86fd5f58db741b86161ba", + "paraphraseSha256": "44005805bf2c3575eebfa0259915107b161785573f777be4d3b215b0a7f72b30", + "humanWords": 397, + "paraphraseWords": 395 + }, + { + "id": "plos-0223179", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0223179", + "humanSha256": "c70065c29cdef7bb0055b389a3fcead15f69b6a7daba6949eed89643b350427b", + "paraphraseSha256": "f007c5c704fa173630a9ce22ce60e534c881e14ae6f78cd6fca5a853f7fd3120", + "humanWords": 385, + "paraphraseWords": 386 + }, + { + "id": "plos-0223190", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0223190", + "humanSha256": "67a50b1b37d2e28f3a559e60db64f526d600b5d4da16fec586b861b592e29a1d", + "paraphraseSha256": "8f6ceb540c83bc196bfcb1bbad3b3a38010d54709a85b1261822aad0e66c1624", + "humanWords": 373, + "paraphraseWords": 372 + }, + { + "id": "plos-0223338", + "language": "en", + "stratum": "en-other-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pone.0223338", + "humanSha256": "38450d51b7d927e7547ee5da09f65b5cd6b85f5385dd91a8ae2d391e220a4184", + "paraphraseSha256": "016562a28cb52d30df0576c298a3c47b9f68ee55c2c46b2898708ae36ea22825", + "humanWords": 379, + "paraphraseWords": 393 + }, + { + "id": "plos-1007329", + "language": "en", + "stratum": "en-anglophone-affiliation", + "year": 2019, + "source": "https://doi.org/10.1371/journal.pcbi.1007329", + "humanSha256": "1056356f5267a154a6b9d6b6ddd5fe9a26ec19612349e89f5e1547290506266d", + "paraphraseSha256": "d16eea0b653cbd72eca265533a2dc01f384bf761b37fcea97cdf7e09166086c6", + "humanWords": 351, + "paraphraseWords": 340 + }, + { + "id": "wp-en-1002690535", + "language": "en", + "stratum": "en-wikipedia", + "year": 2021, + "source": "https://en.wikipedia.org/w/index.php?oldid=1002690535", + "humanSha256": "e248786aeefa1a11eb92581adbd3f64172883fa68385fed24513f8007847cf18", + "paraphraseSha256": "14c300d53023e2e9b8eece58fb3eadfa162f7505a62d47e6592daf0089bb8837", + "humanWords": 388, + "paraphraseWords": 370 + }, + { + "id": "wp-en-1016567157", + "language": "en", + "stratum": "en-wikipedia", + "year": 2021, + "source": "https://en.wikipedia.org/w/index.php?oldid=1016567157", + "humanSha256": "4e5d0b8c1bb5628031bf711fbd6cacffed8743838ff6b2d491d3ca3047566e8d", + "paraphraseSha256": "a26576779f3a040c741684cb24182713a773ec7e21a973ac1222c143a8bc61b8", + "humanWords": 397, + "paraphraseWords": 424 + }, + { + "id": "wp-en-1024207992", + "language": "en", + "stratum": "en-wikipedia", + "year": 2021, + "source": "https://en.wikipedia.org/w/index.php?oldid=1024207992", + "humanSha256": "b46176ac7c05a271b3cad7607eb3b6aca5074e43fbdbbba6f07e9a89a1f9217e", + "paraphraseSha256": "2bb1e126c464731fcae1f73289c8f536fb599a1763bc1c57850493143eadc49d", + "humanWords": 382, + "paraphraseWords": 365 + }, + { + "id": "wp-en-1024922915", + "language": "en", + "stratum": "en-wikipedia", + "year": 2021, + "source": "https://en.wikipedia.org/w/index.php?oldid=1024922915", + "humanSha256": "56ce5ec4aa7c4ed15fcffd5465749147f3da7893d49622184d05936d5707871f", + "paraphraseSha256": "b7e1f7c6665162ad90f2ac672481c7177a89970f25da5cf11725bc45cf0e579a", + "humanWords": 389, + "paraphraseWords": 356 + }, + { + "id": "wp-en-1025398061", + "language": "en", + "stratum": "en-wikipedia", + "year": 2021, + "source": "https://en.wikipedia.org/w/index.php?oldid=1025398061", + "humanSha256": "300637e1851c928f6f39abfe22e02050b3ca6c1220156839eb8fab2243b1b4b2", + "paraphraseSha256": "d3909a4453c993c6593b8538ea8474678c73921f64b92854fa0d49c2bb98537f", + "humanWords": 370, + "paraphraseWords": 353 + }, + { + "id": "wp-en-1025671620", + "language": "en", + "stratum": "en-wikipedia", + "year": 2021, + "source": "https://en.wikipedia.org/w/index.php?oldid=1025671620", + "humanSha256": "3534f7340667c48ee6796b1aa959b5406a16d5070ca098da8aecb91e47c1bed8", + "paraphraseSha256": "1f04214e24c1cb55f3c1a286baed24a564eb38773ba8e654eed4fdb335926038", + "humanWords": 392, + "paraphraseWords": 394 + }, + { + "id": "wp-en-1025914326", + "language": "en", + "stratum": "en-wikipedia", + "year": 2021, + "source": "https://en.wikipedia.org/w/index.php?oldid=1025914326", + "humanSha256": "45ee149abe53e3bb009dfe3714abf969d146a24c969965083ebff26b4fe77121", + "paraphraseSha256": "f4b2b7f7753c095c5f1ec75107f4996dfbb0b6fdb1a171a14110dec5bbe46c25", + "humanWords": 391, + "paraphraseWords": 406 + }, + { + "id": "wp-en-961252368", + "language": "en", + "stratum": "en-wikipedia", + "year": 2020, + "source": "https://en.wikipedia.org/w/index.php?oldid=961252368", + "humanSha256": "18175837ecc7e5decb0e079942998674040e55eaa309e90036b91edc5bd95520", + "paraphraseSha256": "13281de08695a60a8452c7c5f5a71474eca0618c2f942ef501ad6aa7fc9ec648", + "humanWords": 394, + "paraphraseWords": 365 + }, + { + "id": "wp-es-117484960", + "language": "es", + "stratum": "es-wikipedia", + "year": 2019, + "source": "https://es.wikipedia.org/w/index.php?oldid=117484960", + "humanSha256": "54e5e41443231dd3ca9569bfa0c7d69aa9561dbd317b692e5f79ef065356563c", + "paraphraseSha256": "e5e67b7e2ba28cd848fba7812d7ec7e7f970d479fc342e1ac9d1efcdc8528bdc", + "humanWords": 395, + "paraphraseWords": 389 + }, + { + "id": "wp-es-124179402", + "language": "es", + "stratum": "es-wikipedia", + "year": 2020, + "source": "https://es.wikipedia.org/w/index.php?oldid=124179402", + "humanSha256": "7c3e685674191f055798b945605a22cbc50be56a81bf67007974982e21d24adb", + "paraphraseSha256": "30515361f816d2cc71fe8f7794297cc74811abf0b0ac8a643903eb366bd8e5bd", + "humanWords": 386, + "paraphraseWords": 387 + }, + { + "id": "wp-es-132146383", + "language": "es", + "stratum": "es-wikipedia", + "year": 2021, + "source": "https://es.wikipedia.org/w/index.php?oldid=132146383", + "humanSha256": "422327356ce84fdaa2f19025697e8d5f2c27e3b57addd2c95398c60efb801771", + "paraphraseSha256": "bd6f8024d9f5f6e8e27d32d644d4587d34b904f219ab3c7674b26b02a16b52a7", + "humanWords": 397, + "paraphraseWords": 374 + }, + { + "id": "wp-es-132719628", + "language": "es", + "stratum": "es-wikipedia", + "year": 2021, + "source": "https://es.wikipedia.org/w/index.php?oldid=132719628", + "humanSha256": "def907c9f6e62b699f8215a72f7f8947419e257813a93bb06ee1d21bd93f0544", + "paraphraseSha256": "ddc533bbea8cda279dcbe2cd532bbbefb3fb314087c0e26d55cf416de0986400", + "humanWords": 377, + "paraphraseWords": 362 + }, + { + "id": "wp-es-132743897", + "language": "es", + "stratum": "es-wikipedia", + "year": 2021, + "source": "https://es.wikipedia.org/w/index.php?oldid=132743897", + "humanSha256": "307741b2d0011f0171a2e1c46c127e1106e3250885a2e41aa31e5937db06fbc8", + "paraphraseSha256": "7bdb178d96ef29c0e1eef0db594864b1be73f10b4f304685596562dd58cea422", + "humanWords": 400, + "paraphraseWords": 372 + }, + { + "id": "wp-es-132851509", + "language": "es", + "stratum": "es-wikipedia", + "year": 2021, + "source": "https://es.wikipedia.org/w/index.php?oldid=132851509", + "humanSha256": "adc2f8082229fa1f01645019e39082357d9f1cd83662b387b803393b1de19d3e", + "paraphraseSha256": "9defa25eb7ba569d26d718087e2b2a0e961befd5d51b4346804018834cc0a265", + "humanWords": 392, + "paraphraseWords": 383 + }, + { + "id": "wp-es-134286913", + "language": "es", + "stratum": "es-wikipedia", + "year": 2021, + "source": "https://es.wikipedia.org/w/index.php?oldid=134286913", + "humanSha256": "acff9add6e0d892ec86d22a343b1588f88aefce680ab8c2aa32c50bf2c7fcb30", + "paraphraseSha256": "c9eb923ec00a1d3061fc5d5566bd150b58c7c99b9712bb65e55993ad4e4ca9a0", + "humanWords": 399, + "paraphraseWords": 374 + }, + { + "id": "wp-es-135862639", + "language": "es", + "stratum": "es-wikipedia", + "year": 2021, + "source": "https://es.wikipedia.org/w/index.php?oldid=135862639", + "humanSha256": "614b7ad3c0270217e63b1370f1fdfc239f5a40d2710c60e820e3f7e5602e7b7e", + "paraphraseSha256": "fa3dcbf2a571dc1a61b4c57cdb4d987214d0f02b6fcd49c934dce2e5ad0ad0a3", + "humanWords": 367, + "paraphraseWords": 344 + } + ] +} diff --git a/tools/SignsOfAI.Calibration/Paraphrase.cs b/tools/SignsOfAI.Calibration/Paraphrase.cs new file mode 100644 index 0000000..94cea0f --- /dev/null +++ b/tools/SignsOfAI.Calibration/Paraphrase.cs @@ -0,0 +1,351 @@ +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace SignsOfAI.Calibration; + +/// +/// One text measured twice: as its author wrote it, and after a language model rewrote it. +/// +/// The pair is the whole design. Every other way of asking "what does a machine paraphrase do to a +/// text" needs a collection of machine-written text to compare against, which is the thing +/// Docs/Calibration/README.md argues at length against assembling. A pair needs none: both +/// halves are the same passage, by the same author, about the same subject, at the same length, so +/// anything that moves between them moved because of the rewrite and nothing else. The baseline is +/// not estimated, it is the text itself. +/// +public sealed class PairEntry +{ + /// The corpus entry this came from, so the pair can be traced back to a DOI. + public required string Id { get; set; } + + public required string Language { get; set; } + + public required string Stratum { get; set; } + + /// Year of the source publication — the reason the human half is known to be human. + public required int Year { get; set; } + + public required string Source { get; set; } + + /// SHA-256 of the excerpt as its author wrote it. + public required string HumanSha256 { get; set; } + + /// + /// SHA-256 of the rewritten excerpt, or null until one exists. Null is the normal state after + /// excerpt and before the rewriting has been done, and the measuring verb refuses to run + /// on a manifest that still contains any. + /// + public string? ParaphraseSha256 { get; set; } + + public int HumanWords { get; set; } + + public int ParaphraseWords { get; set; } +} + +/// +/// The index of the study, and the one place its expiry date is written down. +/// +/// The human corpus does not age: a 2019 paper will still have been written in 2019 in ten years. +/// This half does age, and pretending otherwise would repeat the mistake the calibration page exists +/// to avoid. So the model that did the rewriting and the date it ran are recorded as data rather +/// than as prose, the report prints them in its first paragraph, and a reader can see at a glance +/// whether the number in front of them was measured on something current. +/// +public sealed class PairManifest +{ + public required string Id { get; set; } + + /// Which corpus the human halves were drawn from. + public required string CorpusId { get; set; } + + /// + /// The model that produced the rewrites, named exactly. "An LLM" is not a method; a different + /// model gives different prose and would move every number on the page. + /// + public string? ParaphrasedBy { get; set; } + + /// UTC date the rewriting was done. + public string? ParaphrasedOn { get; set; } + + /// + /// The instruction the rewriter was given, stored verbatim. It is the experimental treatment: a + /// study of "what a paraphrase does" whose paraphrase instruction is lost has measured something + /// nobody can name afterwards. + /// + public string? Instruction { get; set; } + + /// Words each excerpt was cut to, before rewriting. + public int TargetWords { get; set; } = 400; + + public List Pairs { get; set; } = []; + + public static PairManifest Load(string path) => + JsonSerializer.Deserialize(File.ReadAllText(path), PairJson.Default.PairManifest) + ?? throw new InvalidOperationException($"'{path}' deserialized to null."); + + public void Save(string path) + { + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(path))!); + var json = JsonSerializer.Serialize(this, PairJson.Default.PairManifest); + File.WriteAllText(path, json.ReplaceLineEndings("\n") + "\n"); + } + + /// + /// A fingerprint over both halves of every pair. The calibration manifest has one for the same + /// reason: a published figure and the material it was measured on must not drift apart quietly. + /// + public string Fingerprint() + { + var canonical = string.Join('\n', Pairs + .OrderBy(p => p.Id, StringComparer.Ordinal) + .Select(p => $"{p.Id}\t{p.HumanSha256}\t{p.ParaphraseSha256}")); + + return Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(canonical)))[..16]; + } +} + +/// +/// Builds the human half of the study: a stratified sample of the calibration corpus, cut to +/// equal-length excerpts that a model can be asked to rewrite. +/// +public static class Paraphrase +{ + /// + /// Below roughly this many words the statistical signals are measuring the sample rather than + /// the writing — few sentences means few sentence lengths, and a coefficient of variation over + /// eight of them is noise with a decimal point. It is the same limitation Anthropic states about + /// its own watermark on short passages, and it applies here for the same reason. + /// + public const int MinimumWords = 250; + + /// + /// Picks texts from each group, the same ones on every run. + /// + /// Deterministic by hashing the identity rather than by seeding a random number generator: a seed + /// has to be remembered and passed around correctly forever, whereas a hash of the id gives the + /// same sample to anybody holding the same corpus, with nothing to remember. Re-running after + /// adding texts changes the sample only where the corpus changed. + /// + public static List Sample(CorpusManifest corpus, int perStratum) + { + return [.. corpus.Texts + .GroupBy(t => t.Stratum, StringComparer.Ordinal) + .OrderBy(g => g.Key, StringComparer.Ordinal) + .SelectMany(g => g + .OrderBy(t => Convert.ToHexStringLower( + SHA256.HashData(Encoding.UTF8.GetBytes($"{corpus.Id}\t{t.Id}"))), StringComparer.Ordinal) + .Take(perStratum))]; + } + + /// + /// Cuts a passage of continuous prose to about words, ending on a + /// sentence boundary. + /// + /// Two things are being kept out. Reference lists and figure captions are prose-shaped but nobody + /// composes them, so a rewrite of them measures citation formatting rather than writing. And the + /// cut lands on a full stop because a truncated final sentence is a short sentence, which would + /// push burstiness up in the human half by an artefact of the scissors — in precisely the + /// direction that would flatter the result this study is looking for. + /// + public static string Excerpt(string text, int targetWords) + { + var kept = new StringBuilder(); + int words = 0; + + foreach (var paragraph in text.Replace("\r\n", "\n").Split("\n\n", StringSplitOptions.RemoveEmptyEntries)) + { + var p = paragraph.Trim(); + if (p.Length == 0 || !IsProse(p)) continue; + + kept.Append(p).Append("\n\n"); + words += CountWords(p); + if (words >= targetWords) break; + } + + return TrimToSentence(kept.ToString().TrimEnd(), targetWords); + } + + /// + /// Whether a paragraph is somebody's prose rather than apparatus. + /// + /// Deliberately blunt. The cost of wrongly dropping a good paragraph is that the excerpt starts + /// one paragraph later; the cost of keeping a reference list is a pair whose two halves differ in + /// ways that have nothing to do with writing. The errors are not symmetric, so this errs toward + /// dropping. + /// + private static bool IsProse(string paragraph) + { + if (CountWords(paragraph) < 25) return false; + + // A bibliography entry is mostly names, years and identifiers; running prose is mostly not. + double digits = paragraph.Count(char.IsDigit) / (double)paragraph.Length; + if (digits > 0.08) return false; + + if (paragraph.Contains("doi:", StringComparison.OrdinalIgnoreCase) || + paragraph.Contains("http", StringComparison.OrdinalIgnoreCase)) return false; + + // Prose has sentences. A caption or a heading run-on has one full stop or none. + int stops = paragraph.Count(c => c is '.' or '?' or '!'); + return stops >= 2; + } + + /// + /// Trims to the last sentence that fits. Abbreviations will occasionally fool this and end an + /// excerpt mid-thought; that costs a slightly odd-looking passage and nothing measurable, since + /// both halves of the pair are built from the same cut. + /// + private static string TrimToSentence(string text, int targetWords) + { + var words = text.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries); + if (words.Length <= targetWords) return text; + + // Walk back from the target to the nearest sentence end, so the excerpt is never longer than + // asked for and never ends in the middle of a clause. + int taken = 0, cut = text.Length; + for (int i = 0; i < text.Length; i++) + { + if (char.IsWhiteSpace(text[i]) && (i == 0 || !char.IsWhiteSpace(text[i - 1]))) taken++; + if (taken >= targetWords) { cut = i; break; } + } + + var window = text[..cut]; + int lastStop = window.LastIndexOfAny(['.', '?', '!']); + return lastStop > 0 ? window[..(lastStop + 1)] : window; + } + + public static int CountWords(string text) => + text.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries).Length; + + /// + /// Where a window of words starts, for a document cut at + /// of the way through its prose. + /// + /// The study's own pairs are cut from the opening, which is what a naive implementation does and + /// which turned out to matter: for a research article the opening is abstract and introduction, + /// for an encyclopedia entry it is the lead, and both are the most summary-shaped prose their + /// genre contains. A length effect measured only there is a length effect with a genre effect + /// inside it. Positions are given rather than random so a run is reproducible. + /// + public static string WindowAt(string text, int targetWords, double position) + { + var paragraphs = text.Replace("\r\n", "\n") + .Split("\n\n", StringSplitOptions.RemoveEmptyEntries) + .Select(p => p.Trim()) + .Where(p => p.Length > 0 && IsProse(p)) + .ToList(); + + if (paragraphs.Count == 0) return ""; + + var kept = new StringBuilder(); + int words = 0; + + foreach (var paragraph in paragraphs.Skip((int)(paragraphs.Count * position))) + { + kept.Append(paragraph).Append("\n\n"); + words += CountWords(paragraph); + if (words >= targetWords) break; + } + + return TrimToSentence(kept.ToString().TrimEnd(), targetWords); + } + + /// + /// The whole document with its apparatus removed — the same filter the + /// excerpts pass through, and no length cut. + /// + /// This exists because without it the three arms are not comparable: the excerpts have their + /// figure captions and supporting-information boilerplate stripped and the whole documents do + /// not, so a difference between them could be composition rather than length. Measuring this + /// arm is what turns "the same writing" from a claim into something checked. + /// + public static string ProseOnly(string text) => + string.Join("\n\n", text.Replace("\r\n", "\n") + .Split("\n\n", StringSplitOptions.RemoveEmptyEntries) + .Select(p => p.Trim()) + .Where(p => p.Length > 0 && IsProse(p))); +} + +/// +/// How much of the original survived the rewriting, measured rather than attested. +/// +/// The first version of this study asserted its own compliance in prose and the assertion was wrong: +/// the method file recorded one deliberate breach of the instruction's "no run of eight or more +/// consecutive words" when the passages on disk contained many. That is the sort of claim a project +/// which machine-checks everything else it publishes has no business making by hand. +/// +public static class Fidelity +{ + /// The instruction's limit: no run of this many consecutive words may survive. + public const int MaximumRun = 8; + + /// + /// Words, lowercased, punctuation discarded. + /// + /// The permissive reading on purpose. A stricter tokenizer — case-sensitive, punctuation + /// attached — finds fewer breaches, and the failure being corrected here is under-reporting + /// them. Where two defensible measures disagree, the one that reports more of your own + /// deviations is the one to publish. + /// + private static List Tokenize(string text) + { + var tokens = new List(); + var current = new StringBuilder(); + + foreach (var ch in text) + { + if (char.IsLetterOrDigit(ch)) current.Append(char.ToLowerInvariant(ch)); + else if (current.Length > 0) { tokens.Add(current.ToString()); current.Clear(); } + } + + if (current.Length > 0) tokens.Add(current.ToString()); + return tokens; + } + + /// + /// The longest run of consecutive words from that appears verbatim in + /// , and what share of the rewrite sits inside runs of at least + /// words. + /// + public static (int LongestRun, double ShareRetained) Measure(string original, string rewritten) + { + var a = Tokenize(original); + var b = Tokenize(rewritten); + if (a.Count == 0 || b.Count == 0) return (0, 0); + + // Longest common substring by the usual rolling table, kept to one row because these are + // four-hundred-word passages and the full table is never needed. + var previous = new int[b.Count + 1]; + var current = new int[b.Count + 1]; + int longest = 0; + var covered = new bool[b.Count]; + + for (int i = 1; i <= a.Count; i++) + { + for (int j = 1; j <= b.Count; j++) + { + current[j] = a[i - 1] == b[j - 1] ? previous[j - 1] + 1 : 0; + if (current[j] > longest) longest = current[j]; + + // Mark every token of a qualifying run, so overlapping runs are counted once. + if (current[j] >= MaximumRun) + for (int k = j - current[j]; k < j; k++) covered[k] = true; + } + + (previous, current) = (current, previous); + Array.Clear(current); + } + + return (longest, covered.Count(c => c) / (double)b.Count); + } +} + +[JsonSourceGenerationOptions( + PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, + PropertyNameCaseInsensitive = true, + WriteIndented = true, + ReadCommentHandling = JsonCommentHandling.Skip, + AllowTrailingCommas = true)] +[JsonSerializable(typeof(PairManifest))] +public partial class PairJson : JsonSerializerContext; diff --git a/tools/SignsOfAI.Calibration/ParaphraseReport.cs b/tools/SignsOfAI.Calibration/ParaphraseReport.cs new file mode 100644 index 0000000..35d69cd --- /dev/null +++ b/tools/SignsOfAI.Calibration/ParaphraseReport.cs @@ -0,0 +1,769 @@ +using System.Globalization; +using System.Text; +using SignsOfAI.Core.Calibration; + +namespace SignsOfAI.Calibration; + +/// One window cut from a document, and what the analyzer made of it. +public sealed record WindowMeasurement +{ + /// Where in the document's prose the window starts, as a label a reader understands. + public required string Position { get; init; } + + public required double Score { get; init; } + + public required double Burstiness { get; init; } + + public required int Words { get; init; } +} + +/// One text seen twice, and what the analyzer said about each half. +public sealed record PairMeasurement +{ + public required string Id { get; init; } + + public required string Language { get; init; } + + public required string Stratum { get; init; } + + public required double HumanScore { get; init; } + + public required double RewrittenScore { get; init; } + + public required double HumanBurstiness { get; init; } + + public required double RewrittenBurstiness { get; init; } + + public required int HumanWords { get; init; } + + public required int RewrittenWords { get; init; } + + /// Rules that produced evidence — the ones that scored, not those found at a human rate. + public required IReadOnlyList HumanRuleIds { get; init; } + + public required IReadOnlyList RewrittenRuleIds { get; init; } + + public required int HumanArtifacts { get; init; } + + public required int RewrittenArtifacts { get; init; } + + /// Every artifact codepoint found, so the report can name them instead of counting them. + public IReadOnlyList ArtifactCodePoints { get; init; } = []; + + /// The whole document the excerpt was cut from. Null when the source is not on this machine. + public double? FullScore { get; init; } + + public double? FullBurstiness { get; init; } + + public int FullWords { get; init; } + + /// + /// The same document with its figure captions and boilerplate stripped by the filter the excerpts + /// already pass through — the control that tells composition apart from length. + /// + public double? ProseOnlyScore { get; init; } + + /// Fixed-length windows cut at several positions, which is how the length arm is measured. + public IReadOnlyList Windows { get; init; } = []; + + /// Longest run of consecutive words from the original surviving verbatim in the rewrite. + public int LongestRun { get; init; } + + /// Share of the rewrite sitting inside verbatim runs of eight words or more. + public double ShareRetained { get; init; } + + public double Delta => RewrittenScore - HumanScore; +} + +/// +/// Renders what a machine rewrite does to writing that is known to be human. +/// +/// Three reviewers went over the first version of this page adversarially and the arithmetic +/// survived; the prose did not. Three sentences were false — a window described as coming from the +/// middle of a document that was cut from its opening, a deviation log calling one breach unique +/// when half the pairs contained one, and a directional claim asserted on evidence that failed the +/// exact test applied two paragraphs above it. All three were the same failure: a page that machine +/// checks its numbers and hand-waves its method. Everything the method now claims is measured by the +/// code below and printed whether or not it flatters the study. +/// +/// The page also has to survive being read once, quickly, by a teacher with no statistics. That is +/// the reader this project exists for, and a headline they misread is worse than one they skip. +/// +public static class ParaphraseReport +{ + private static readonly CultureInfo Inv = CultureInfo.InvariantCulture; + + public static string Render( + IReadOnlyList pairs, + PairManifest manifest, + double threshold, + string version, + string generatedOn) + { + var sb = new StringBuilder(); + + sb.AppendLine("# What a machine rewrite does to human writing"); + sb.AppendLine(); + sb.AppendLine(""); + sb.AppendLine(); + sb.AppendLine( + "Claude models released on or after 2 August 2026 carry a machine-readable watermark, and " + + "Anthropic says earlier ones will follow during a transition period. The technique is a " + + "version of SynthID-Text, published by Google DeepMind in 2024 and already in use " + + "elsewhere — this is an industry practice arriving at one more vendor, not a new " + + "invention. The mark lives in which words the model chose, so cleaning invisible " + + "characters cannot touch it, and the vendor's help centre names heavy editing, " + + "paraphrasing, translation, mixing into other writing and very short passages as things " + + "that leave it undetectable. Services selling that rewrite already exist."); + sb.AppendLine(); + sb.AppendLine( + "This page measures what such a rewrite does to a passage, using passages whose author " + + "was demonstrably a person."); + sb.AppendLine(); + + if (pairs.Count == 0) + { + sb.AppendLine("**No pairs were measured, so there is nothing to report.** " + + "See `Docs/Paraphrase/README.md` for how to build them."); + return sb.ToString(); + } + + // ---- method -------------------------------------------------------------------------- + sb.AppendLine("## What was measured"); + sb.AppendLine(); + sb.AppendLine($"- **Pairs** {pairs.Count} passages, each measured as written and as rewritten"); + sb.AppendLine($"- **Drawn from** corpus `{manifest.CorpusId}`, the same texts as `Docs/CALIBRATION.md`"); + sb.AppendLine($"- **Rewritten by** {manifest.ParaphrasedBy ?? "unrecorded"} on {manifest.ParaphrasedOn ?? "an unrecorded date"}"); + sb.AppendLine("- **Instruction** `Docs/Paraphrase/instruction.md`, stored verbatim"); + sb.AppendLine($"- **Engine** {version}, verdict boundary {threshold.ToString("0", Inv)}/100"); + sb.AppendLine($"- **Pair fingerprint** `{manifest.Fingerprint()}`"); + sb.AppendLine($"- **Run** {generatedOn}"); + sb.AppendLine(); + sb.AppendLine( + "Every human half was published before generative models existed, which is the whole basis " + + "for calling it human and the same basis the calibration page rests on. Both halves are the " + + "same passage at roughly the same length, so the comparison is within a text rather than " + + "between two populations: no collection of machine-written prose was assembled, and none " + + "was needed. The pairs are cut from the **opening** of each document, which matters and is " + + "measured under Length."); + sb.AppendLine(); + sb.AppendLine( + "**This half of the study ages, and the human half does not.** A 2019 paper will still have " + + "been written in 2019 a decade from now; the rewrite is the work of one model on one day. " + + "That is why the model and the date are printed above rather than buried, and why the right " + + "response to \"but a newer model rewrites differently\" is to re-run this with that model " + + "rather than to argue about it."); + sb.AppendLine(); + + // ---- headline ------------------------------------------------------------------------ + int flaggedBefore = pairs.Count(p => p.HumanScore >= threshold); + int flaggedAfter = pairs.Count(p => p.RewrittenScore >= threshold); + var (beforeLow, beforeHigh) = Calibrator.WilsonInterval(flaggedBefore, pairs.Count); + var (afterLow, afterHigh) = Calibrator.WilsonInterval(flaggedAfter, pairs.Count); + + int becameFlagged = pairs.Count(p => p.HumanScore < threshold && p.RewrittenScore >= threshold); + int becameClean = pairs.Count(p => p.HumanScore >= threshold && p.RewrittenScore < threshold); + double mcnemar = ExactBinomialTwoSided(becameFlagged, becameFlagged + becameClean); + + var withFull = pairs.Where(x => x.FullScore is not null).ToList(); + int flaggedFull = withFull.Count(x => x.FullScore >= threshold); + var windows = pairs.SelectMany(p => p.Windows).ToList(); + int flaggedWindows = windows.Count(w => w.Score >= threshold); + + sb.AppendLine("## The short version"); + sb.AppendLine(); + sb.AppendLine( + "**Having a language model rewrite a passage did not change whether this tool flagged it. " + + "Cutting the same passage down to four hundred words did.**"); + sb.AppendLine(); + sb.AppendLine( + "That is the opposite of what this study set out to find, and the second half of it is a " + + "fault in this tool rather than a finding about anybody's watermark. Neither sentence " + + "should be read as *the tool catches rewritten work*: it does not, and this page is the " + + "evidence that it does not."); + sb.AppendLine(); + if (withFull.Count > 0 && windows.Count > 0) + { + var (wLow, wHigh) = Calibrator.WilsonInterval(flaggedWindows, windows.Count); + sb.AppendLine($"| Same writing, measured as | Flagged at {threshold.ToString("0", Inv)}/100 | Rate | 95% interval |"); + sb.AppendLine("|---|---|---|---|"); + sb.AppendLine($"| whole documents | {flaggedFull} / {withFull.Count} | {Pct(Rate(flaggedFull, withFull.Count))} | {Wilson(flaggedFull, withFull.Count)} |"); + sb.AppendLine($"| {manifest.TargetWords}-word windows, three positions each | {flaggedWindows} / {windows.Count} | {Pct(Rate(flaggedWindows, windows.Count))} | {Pct(wLow)} – {Pct(wHigh)} |"); + sb.AppendLine($"| the opening window, as written | {flaggedBefore} / {pairs.Count} | {Pct(Rate(flaggedBefore, pairs.Count))} | {Pct(beforeLow)} – {Pct(beforeHigh)} |"); + sb.AppendLine($"| the opening window, rewritten | {flaggedAfter} / {pairs.Count} | {Pct(Rate(flaggedAfter, pairs.Count))} | {Pct(afterLow)} – {Pct(afterHigh)} |"); + sb.AppendLine(); + } + + sb.AppendLine("### The rewrite, measured against the passage it came from"); + sb.AppendLine(); + sb.AppendLine( + $"{becameFlagged} passages crossed the boundary that had not, {becameClean} crossed back, " + + $"and the other {pairs.Count - becameFlagged - becameClean} stayed where they were. " + + $"McNemar's exact test on the discordant pairs gives {FormatP(mcnemar)}. With " + + $"{becameFlagged + becameClean} pairs changing side at all, only a clean sweep would have " + + "reached significance, so the honest statement is that **this study cannot show that " + + "rewriting changes whether a passage is flagged, in either direction** — not that it " + + "showed rewriting is safe, and not that it showed rewriting is caught."); + sb.AppendLine(); + + // ---- treatment fidelity ----------------------------------------------------------------- + var breaches = pairs.Where(p => p.LongestRun >= Fidelity.MaximumRun).ToList(); + var retained = pairs.Select(p => p.ShareRetained).OrderBy(s => s).ToList(); + + sb.AppendLine("## Was the treatment actually applied"); + sb.AppendLine(); + sb.AppendLine( + "The instruction asks that no run of eight or more consecutive words survive. Whether it " + + "did is checked here rather than attested, because the first version of this study " + + "attested it by hand and the hand was wrong."); + sb.AppendLine(); + sb.AppendLine($"- **{breaches.Count} of {pairs.Count} pairs** contain at least one surviving run of " + + $"{Fidelity.MaximumRun} words or more"); + sb.AppendLine($"- **Longest run** {pairs.Max(p => p.LongestRun)} words"); + sb.AppendLine($"- **Share of each rewrite inside such runs** median {Pct(Quantile(retained, 0.5))}, " + + $"highest {Pct(retained[^1])}"); + sb.AppendLine(); + if (breaches.Count > 0) + { + sb.AppendLine("| Pair | Longest run | Share retained |"); + sb.AppendLine("|---|---|---|"); + foreach (var p in breaches.OrderByDescending(p => p.LongestRun).Take(8)) + sb.AppendLine($"| `{p.Id}` | {p.LongestRun} | {Pct(p.ShareRetained)} |"); + sb.AppendLine(); + } + sb.AppendLine( + "The long runs are quotations. A passage that quotes a court ruling, a political pamphlet " + + "or a published definition cannot have those words replaced without falsifying them, and " + + "the instruction's own requirement to preserve every fact and citation marker conflicts " + + "with its word-run limit; the protocol never said which wins. The shorter runs are names, " + + "dates and technical terms."); + sb.AppendLine(); + sb.AppendLine( + "**The consequence is that the delivered treatment is a partial rewrite**, weaker than the " + + "one the opening of this page describes. That biases every paired difference toward zero — " + + "toward the null result reported above. The null is therefore measured under a treatment " + + "milder than intended, which makes it weaker evidence than it looks, not stronger."); + sb.AppendLine(); + + // ---- length ---------------------------------------------------------------------------- + if (withFull.Count > 0 && windows.Count > 0) + { + AppendLength(sb, pairs, withFull, windows, manifest, threshold); + } + + // ---- scores -------------------------------------------------------------------------- + var deltas = pairs.Select(x => x.Delta).OrderBy(d => d).ToList(); + int rose = pairs.Count(x => x.Delta > 0); + int fell = pairs.Count(x => x.Delta < 0); + var (scoreLow, scoreHigh) = MedianInterval(deltas); + + sb.AppendLine("## How far the score moved"); + sb.AppendLine(); + sb.AppendLine("| | Median | 90th percentile | Highest |"); + sb.AppendLine("|---|---|---|---|"); + AppendScoreRow(sb, "As written", [.. pairs.Select(x => x.HumanScore)]); + AppendScoreRow(sb, "Rewritten", [.. pairs.Select(x => x.RewrittenScore)]); + sb.AppendLine(); + sb.AppendLine( + $"The score rose in {rose} pairs, fell in {fell} and was unchanged in " + + $"{pairs.Count - rose - fell}. Taking each passage against itself, the median change is " + + $"{Signed(Quantile(deltas, 0.5))} points (95% interval {Signed(scoreLow)} to {Signed(scoreHigh)}), " + + $"with the middle half between {Signed(Quantile(deltas, 0.25))} and {Signed(Quantile(deltas, 0.75))}. " + + $"An exact sign test on the {rose + fell} pairs that moved gives {FormatP(ExactBinomialTwoSided(Math.Min(rose, fell), rose + fell))}, " + + $"and Wilcoxon signed-rank {FormatP(WilcoxonSignedRank(deltas))}. **The direction is not " + + "established.** The rewrite strips the vocabulary of academic prose — *furthermore*, " + + "*comprehensive*, *facilitate*, *utilizar* — while flattening sentence rhythm, and on this " + + "sample the two effects are not separable from each other or from noise."); + sb.AppendLine(); + + // ---- burstiness ---------------------------------------------------------------------- + var burstBefore = pairs.Select(x => x.HumanBurstiness).OrderBy(b => b).ToList(); + var burstAfter = pairs.Select(x => x.RewrittenBurstiness).OrderBy(b => b).ToList(); + var burstDeltas = pairs.Select(x => x.RewrittenBurstiness - x.HumanBurstiness).OrderBy(d => d).ToList(); + int burstFell = pairs.Count(x => x.RewrittenBurstiness < x.HumanBurstiness); + int burstRose = pairs.Count(x => x.RewrittenBurstiness > x.HumanBurstiness); + var (burstLow, burstHigh) = MedianInterval(burstDeltas); + + sb.AppendLine("## Burstiness"); + sb.AppendLine(); + sb.AppendLine( + "Sentence-length variation is the signal this project trusts most, because it needs no " + + "word list and no model. Human prose is uneven; unprompted model output tends to settle " + + "into a width and stay there."); + sb.AppendLine(); + sb.AppendLine($"- **As written** median {Quantile(burstBefore, 0.5).ToString("0.00", Inv)}, " + + $"range {burstBefore[0].ToString("0.00", Inv)} – {burstBefore[^1].ToString("0.00", Inv)}"); + sb.AppendLine($"- **Rewritten** median {Quantile(burstAfter, 0.5).ToString("0.00", Inv)}, " + + $"range {burstAfter[0].ToString("0.00", Inv)} – {burstAfter[^1].ToString("0.00", Inv)}"); + sb.AppendLine($"- **Paired change** median {Signed(Quantile(burstDeltas, 0.5), "0.00")} " + + $"(95% interval {Signed(burstLow, "0.00")} to {Signed(burstHigh, "0.00")}), " + + $"falling in {burstFell} of {pairs.Count} pairs"); + sb.AppendLine(); + sb.AppendLine( + $"This is the closest the study comes to a directional effect, and it is not clean. An " + + $"exact sign test on {burstFell} falls against {burstRose} rises gives " + + $"{FormatP(ExactBinomialTwoSided(Math.Min(burstFell, burstRose), burstFell + burstRose))} — " + + "by the same standard applied to the flag counts above, not a result. Wilcoxon " + + $"signed-rank, which uses the size of each change and not only its direction, gives " + + $"{FormatP(WilcoxonSignedRank(burstDeltas))}. Neither test was chosen in advance, so the " + + "honest reading is that **the rewrite probably flattens rhythm slightly and this sample " + + "cannot settle it**. A few hundredths on a scale where human prose runs from roughly 0.2 " + + "to 0.9 moves almost nothing on its own."); + sb.AppendLine(); + + // ---- what fires ---------------------------------------------------------------------- + AppendRuleTables(sb, pairs); + + // ---- characters ---------------------------------------------------------------------- + AppendCharacters(sb, pairs); + + // ---- by language --------------------------------------------------------------------- + sb.AppendLine("## By language"); + sb.AppendLine(); + sb.AppendLine( + "A result that holds in English and fails in Spanish is not one result. Two warnings " + + "before the table: the Spanish arm is eight pairs, all encyclopedia entries, so it is " + + "also a single-source arm and cannot be compared cleanly with an English arm that mixes " + + "research articles and encyclopedia entries. And eight pairs support no interval worth " + + "printing."); + sb.AppendLine(); + AppendGroupTable(sb, pairs.GroupBy(x => x.Language, StringComparer.OrdinalIgnoreCase), threshold); + AppendGroupTable(sb, pairs.GroupBy(x => x.Stratum, StringComparer.OrdinalIgnoreCase), threshold); + + // ---- conflict of interest ---------------------------------------------------------------- + sb.AppendLine("## The conflict of interest"); + sb.AppendLine(); + sb.AppendLine( + $"The rewriting was done by {manifest.ParaphrasedBy ?? "an unrecorded model"}. Two things " + + "follow that naming the model does not cover."); + sb.AppendLine(); + sb.AppendLine( + "**The rewriter's vendor is the vendor whose watermark prompted the study.** And more " + + "seriously, the same model wrote this project's rule packs, this tool, and the prose on " + + "this page. The instruction's defences are all at the level of the prompt — it mentions " + + "no detector and names no tell — and the contamination risk is at the level of knowledge: " + + "a model that wrote the rule for *furthermore* does not need to be told to avoid it. The " + + "removed-signals table above is exactly what tell-aware avoidance would produce."); + sb.AppendLine(); + sb.AppendLine( + "What argues against it is also on this page: the rewrite *introduced* tells as well as " + + "removing them, and pushed more passages over the boundary than it pulled back. A model " + + "gaming the measurement would not do that. But the objection is legitimate, it is the " + + "first one a hostile reader will raise, and the only real answer is a second rewriter " + + "from a different vendor. **Until that exists, these numbers should not be quoted " + + "anywhere a teacher will act on them.**"); + sb.AppendLine(); + + // ---- limits ------------------------------------------------------------------------------ + sb.AppendLine("## What this does not tell you"); + sb.AppendLine(); + sb.AppendLine("- **Nothing about detecting a watermark.** The key is Anthropic's and no third " + + "party holds it. This measures what happens to *prose*, which is a different " + + "question that happens to be answerable without anybody's permission."); + sb.AppendLine("- **Nothing about how much machine writing this tool catches.** That would need " + + "a corpus of machine writing, which `Docs/Calibration/README.md` argues at length " + + "against assembling, and the argument has not changed."); + sb.AppendLine("- **Nothing about whether rewriting helps or hurts a student.** The study " + + "failed to establish a direction. Anyone quoting it in either direction is " + + "quoting something that is not here."); + sb.AppendLine("- **One model, one instruction, one day**, and the treatment was only partly " + + "delivered. All of that is above."); + sb.AppendLine("- **These are published articles and encyclopedia entries**, not student essays, " + + "and the windows are cut from them rather than composed at that length. A " + + "four-hundred-word paragraph somebody wrote as a paragraph has a whole " + + "distribution of sentence lengths; a truncation does not."); + sb.AppendLine($"- **{pairs.Count} pairs is a small study.** Read the intervals rather than the " + + "percentages."); + sb.AppendLine(); + sb.AppendLine("Re-run it yourself:"); + sb.AppendLine(); + sb.AppendLine("```"); + sb.AppendLine("dotnet run --project tools/SignsOfAI.Calibration -- excerpt --per-stratum 8"); + sb.AppendLine("# rewrite each passage in Docs/Paraphrase/human/ into Docs/Paraphrase/rewritten/"); + sb.AppendLine("dotnet run --project tools/SignsOfAI.Calibration -- paraphrase \\"); + sb.AppendLine(" --paraphrased-by \"\" --instruction Docs/Paraphrase/instruction.md"); + sb.AppendLine("```"); + + return sb.ToString(); + } + + private static void AppendLength( + StringBuilder sb, + IReadOnlyList pairs, + IReadOnlyList withFull, + IReadOnlyList windows, + PairManifest manifest, + double threshold) + { + var fullScores = withFull.Select(x => x.FullScore!.Value).OrderBy(s => s).ToList(); + var proseOnly = withFull.Where(x => x.ProseOnlyScore is not null) + .Select(x => x.ProseOnlyScore!.Value).OrderBy(s => s).ToList(); + int medianWords = (int)Quantile([.. withFull.Select(x => (double)x.FullWords).OrderBy(w => w)], 0.5); + + sb.AppendLine("## Length, which turned out to matter more than rewriting"); + sb.AppendLine(); + sb.AppendLine( + $"The verdict boundary this build ships was measured on whole documents — a median of " + + $"{medianWords:N0} words here — and none of them reaches it. Cut {manifest.TargetWords}-word " + + "windows out of that same writing and some of them do, with no author, subject or sentence " + + "altered."); + sb.AppendLine(); + sb.AppendLine("| Window position | Windows | Flagged | Rate | 95% interval | Median score |"); + sb.AppendLine("|---|---|---|---|---|---|"); + // Ordered by where the window sits in the document, not by the spelling of its label — a + // table that runs late, middle, opening reads as though the trend went the other way. + foreach (var group in windows.GroupBy(w => w.Position).OrderBy(g => PositionRank(g.Key))) + { + var scores = group.Select(w => w.Score).OrderBy(s => s).ToList(); + int flagged = group.Count(w => w.Score >= threshold); + var (low, high) = Calibrator.WilsonInterval(flagged, group.Count()); + sb.AppendLine($"| {group.Key} | {group.Count()} | {flagged} | {Pct(Rate(flagged, group.Count()))} | " + + $"{Pct(low)} – {Pct(high)} | {Quantile(scores, 0.5).ToString("0.0", Inv)} |"); + } + sb.AppendLine(); + sb.AppendLine( + "**Where the window falls changes the answer.** The opening of a document scores highest, " + + "which is not surprising once stated — for a research article that is the abstract and " + + "introduction, for an encyclopedia entry the lead, and both are the most summary-shaped " + + "prose their genre contains. The first version of this study cut only from the opening and " + + "reported the resulting rate as a length effect; it was a length effect with a genre " + + "effect inside it."); + sb.AppendLine(); + + // The document-level view, which is what a teacher's situation actually looks like: one + // passage, one verdict, and no way to know which window they happened to be handed. + var byDoc = pairs.Where(p => p.Windows.Count > 1).ToList(); + int anyFlagged = byDoc.Count(p => p.Windows.Any(w => w.Score >= threshold)); + int allFlagged = byDoc.Count(p => p.Windows.All(w => w.Score >= threshold)); + + sb.AppendLine( + $"Counted by document rather than by window, of the {byDoc.Count} documents that yielded " + + $"more than one window: **{anyFlagged} are flagged at one position but not another**, and " + + $"{allFlagged} are flagged wherever the window falls. Whether one of these authors would " + + "be accused depends on which four hundred words somebody happened to paste."); + sb.AppendLine(); + sb.AppendLine("| | Median score | 90th percentile | Highest |"); + sb.AppendLine("|---|---|---|---|"); + sb.AppendLine($"| Whole document | {Quantile(fullScores, 0.5).ToString("0.0", Inv)} | " + + $"{Quantile(fullScores, 0.9).ToString("0.0", Inv)} | {fullScores[^1].ToString("0.0", Inv)} |"); + if (proseOnly.Count > 0) + { + sb.AppendLine($"| Whole document, apparatus stripped | {Quantile(proseOnly, 0.5).ToString("0.0", Inv)} | " + + $"{Quantile(proseOnly, 0.9).ToString("0.0", Inv)} | {proseOnly[^1].ToString("0.0", Inv)} |"); + } + var allWindowScores = windows.Select(w => w.Score).OrderBy(s => s).ToList(); + sb.AppendLine($"| {manifest.TargetWords}-word windows, all positions | {Quantile(allWindowScores, 0.5).ToString("0.0", Inv)} | " + + $"{Quantile(allWindowScores, 0.9).ToString("0.0", Inv)} | {allWindowScores[^1].ToString("0.0", Inv)} |"); + sb.AppendLine(); + if (proseOnly.Count > 0) + { + int proseFlagged = proseOnly.Count(s => s >= threshold); + sb.AppendLine( + $"The second row is a control, and it matters: the whole documents carry figure " + + "captions and boilerplate that the windows have stripped, so the gap could have been " + + $"composition rather than length. With the same filter applied and no length cut, " + + $"{proseFlagged} of {proseOnly.Count} are flagged. It is length."); + sb.AppendLine(); + } + sb.AppendLine( + "The mechanism is not mysterious. Burstiness is the spread of sentence lengths, and a " + + "short window holds few sentences: the long one with three clauses and the two-word " + + "fragment that together make a paragraph look human may simply not both be inside it. The " + + "measurement does not become uncertain, which a reader could allow for. It moves, in one " + + "direction, toward the machine."); + sb.AppendLine(); + sb.AppendLine( + "**This is a defect in this tool, published here because that is what this page is for**, " + + "and tracked as issue #59. The boundary carries a language condition already — a text in a " + + "language the corpus never measured gets no verdict — and the reason is exactly the reason " + + "that applies here: a bound measured on one population must not be spent on another. " + + "Length is such a population and the code does not yet know it. Until it does, treat a " + + "verdict on anything short as unmeasured, whatever the interface says."); + sb.AppendLine(); + sb.AppendLine( + "One caution about fixing it, which is why #59 is not a one-line change: these windows are " + + "**cut** from long formal documents, and a student's four-hundred-word answer was " + + "**composed** at that length. Its sentence lengths are a whole distribution, not a " + + "truncated one. A floor measured on truncations and enforced against compositions would " + + "repeat, in a new dimension, the mistake it exists to prevent."); + sb.AppendLine(); + sb.AppendLine( + "It also lands on the same ground as the watermark it was written about. Anthropic says " + + "its mark is unreliable on short samples because few word choices carry little " + + "information. This tool is unreliable on short samples because few sentences carry little " + + "rhythm. Two unrelated methods, the same floor, and a teacher holding one paragraph of a " + + "student's work is below it either way."); + sb.AppendLine(); + } + + private static void AppendRuleTables(StringBuilder sb, IReadOnlyList pairs) + { + var introduced = Movement(pairs, p => p.RewrittenRuleIds, p => p.HumanRuleIds); + var removed = Movement(pairs, p => p.HumanRuleIds, p => p.RewrittenRuleIds); + + sb.AppendLine("## Which signals the rewrite moved"); + sb.AppendLine(); + AppendMovementTable(sb, "Passages where it appeared only after the rewrite", introduced, + "Nothing fired after the rewrite that had not fired before it."); + sb.AppendLine("Signals the rewrite *removed* are worth as much, since a rewrite is not obliged " + + "to move in one direction:"); + sb.AppendLine(); + AppendMovementTable(sb, "Passages where it disappeared", removed, + "Nothing that fired before the rewrite stopped firing after it."); + } + + private static List<(string RuleId, int Texts)> Movement( + IReadOnlyList pairs, + Func> after, + Func> before) => + [.. pairs + .SelectMany(p => after(p).Distinct().Except(before(p).Distinct(), StringComparer.Ordinal)) + .GroupBy(id => id, StringComparer.Ordinal) + .Select(g => (RuleId: g.Key, Texts: g.Count())) + .OrderByDescending(x => x.Texts) + .ThenBy(x => x.RuleId, StringComparer.Ordinal)]; + + /// + /// Prints a movement table, and says how many rows it is not printing. + /// + /// The first version capped both tables at twenty silently, and the row it dropped from the + /// removed table was stat.burstiness — the exact counterweight to the introduced count on + /// a page arguing that removals matter as much as additions. A cap that hides the row which + /// undercuts the argument is not a formatting choice. + /// + private static void AppendMovementTable( + StringBuilder sb, string header, List<(string RuleId, int Texts)> rows, string empty) + { + if (rows.Count == 0) + { + sb.AppendLine(empty); + sb.AppendLine(); + return; + } + + const int cap = 20; + sb.AppendLine($"| Rule | {header} |"); + sb.AppendLine("|---|---|"); + foreach (var row in rows.Take(cap)) + sb.AppendLine($"| `{row.RuleId}` | {row.Texts} |"); + sb.AppendLine(); + if (rows.Count > cap) + { + sb.AppendLine($"*{rows.Count - cap} further rule(s) moved in one passage each and are not " + + $"listed: {string.Join(", ", rows.Skip(cap).Select(r => $"`{r.RuleId}`"))}.*"); + sb.AppendLine(); + } + } + + private static void AppendCharacters(StringBuilder sb, IReadOnlyList pairs) + { + int before = pairs.Sum(x => x.HumanArtifacts); + int after = pairs.Sum(x => x.RewrittenArtifacts); + var affected = pairs.Where(x => x.HumanArtifacts > 0 || x.RewrittenArtifacts > 0).ToList(); + var codepoints = pairs.SelectMany(x => x.ArtifactCodePoints).Distinct() + .OrderBy(c => c, StringComparer.Ordinal).ToList(); + + sb.AppendLine("## The character layer, which is a different thing"); + sb.AppendLine(); + sb.AppendLine( + $"The character scanner found {before} suspicious codepoints before the rewrite and " + + $"{after} after it."); + sb.AppendLine(); + if (affected.Count > 0) + { + sb.AppendLine( + $"Those totals are not spread across the corpus: they are " + + $"{(affected.Count == 1 ? "one passage" : $"{affected.Count} passages")} — " + + $"{string.Join(", ", affected.Select(a => $"`{a.Id}`"))} — and the codepoints are " + + $"{string.Join(", ", codepoints.Select(c => $"`{c}`"))}. In " + + $"`{affected[0].Id}` they are Turkish dotless ı inside Turkish proper names, in a " + + "Spanish article about a Turkish organisation. **That is a false positive of the " + + "character scanner**, and worth stating on a page that criticises other people's."); + sb.AppendLine(); + } + sb.AppendLine( + "The count not moving is the expected result and worth stating plainly: homoglyphs and " + + "zero-width characters are a *fingerprint of a tool that touched the file*, not of a model " + + "that wrote it. Rewriting prose does not produce them, and the free half of every " + + "watermark-removal service — the half that strips invisible characters — is aimed at a " + + "layer that has nothing to do with the statistical watermark it advertises removing."); + sb.AppendLine(); + } + + private static void AppendScoreRow(StringBuilder sb, string label, IReadOnlyList scores) + { + var sorted = scores.OrderBy(s => s).ToList(); + sb.AppendLine($"| {label} | {Quantile(sorted, 0.50).ToString("0.0", Inv)} | " + + $"{Quantile(sorted, 0.90).ToString("0.0", Inv)} | {sorted[^1].ToString("0.0", Inv)} |"); + } + + private static void AppendGroupTable( + StringBuilder sb, IEnumerable> groups, double threshold) + { + sb.AppendLine("| Group | Pairs | Flagged before | Flagged after | Median score change | Median burstiness change |"); + sb.AppendLine("|---|---|---|---|---|---|"); + foreach (var group in groups.OrderBy(g => g.Key, StringComparer.Ordinal)) + { + var members = group.ToList(); + int before = members.Count(x => x.HumanScore >= threshold); + int after = members.Count(x => x.RewrittenScore >= threshold); + var scoreDeltas = members.Select(x => x.Delta).OrderBy(d => d).ToList(); + var burstDeltas = members.Select(x => x.RewrittenBurstiness - x.HumanBurstiness).OrderBy(d => d).ToList(); + + sb.AppendLine($"| **{group.Key}** | {members.Count} | {before} | {after} | " + + $"{Signed(Quantile(scoreDeltas, 0.5))} | {Signed(Quantile(burstDeltas, 0.5), "0.00")} |"); + } + sb.AppendLine(); + } + + /// + /// McNemar's test in its exact form: given the pairs that changed, how surprising is it that this + /// many of them changed in one direction? The chi-square approximation is not usable at these + /// counts, and reporting a p-value that the sample cannot support is the failure this project + /// objects to in other people's numbers. Also serves as the exact sign test, which is the same + /// computation on a different pair of counts. + /// + private static double ExactBinomialTwoSided(int successes, int trials) + { + if (trials == 0) return 1.0; + + double Pmf(int k) + { + double logC = 0; + for (int i = 1; i <= k; i++) logC += Math.Log(trials - k + i) - Math.Log(i); + return Math.Exp(logC - trials * Math.Log(2)); + } + + double observed = Pmf(successes); + double total = 0; + + // Sum every outcome no more likely than the one observed, which is the two-sided p-value for a + // discrete distribution — halving a one-sided value would be wrong for an asymmetric tail. + for (int k = 0; k <= trials; k++) + { + var pk = Pmf(k); + if (pk <= observed * (1 + 1e-9)) total += pk; + } + + return Math.Min(1.0, total); + } + + /// + /// Wilcoxon signed-rank, normal approximation with a tie correction. + /// + /// Reported *beside* the sign test rather than instead of it. The sign test counts directions and + /// the signed-rank test also weighs magnitudes, so on a sample where most changes are tiny they + /// disagree — and neither was chosen before the data were seen. Printing both, and saying so, is + /// the only honest way to present a result whose significance depends on that choice. + /// + private static double WilcoxonSignedRank(IReadOnlyList differences) + { + var nonZero = differences.Where(d => Math.Abs(d) > 1e-9).ToList(); + int n = nonZero.Count; + if (n < 6) return 1.0; + + var ordered = nonZero + .Select((d, i) => (Value: d, Abs: Math.Abs(d), Index: i)) + .OrderBy(x => x.Abs) + .ToList(); + + var ranks = new double[n]; + int position = 0; + double tieCorrection = 0; + + while (position < n) + { + int end = position; + while (end + 1 < n && Math.Abs(ordered[end + 1].Abs - ordered[position].Abs) < 1e-9) end++; + + double averageRank = (position + end + 2) / 2.0; + int size = end - position + 1; + for (int i = position; i <= end; i++) ranks[i] = averageRank; + if (size > 1) tieCorrection += size * (double)size * size - size; + + position = end + 1; + } + + double positive = 0; + for (int i = 0; i < n; i++) if (ordered[i].Value > 0) positive += ranks[i]; + + double mean = n * (n + 1) / 4.0; + double variance = (n * (n + 1) * (2.0 * n + 1) - tieCorrection / 2.0) / 24.0; + if (variance <= 0) return 1.0; + + // Continuity correction, because a discrete statistic is being read off a continuous curve. + double z = (Math.Abs(positive - mean) - 0.5) / Math.Sqrt(variance); + return Math.Min(1.0, 2 * (1 - NormalCdf(z))); + } + + private static double NormalCdf(double z) + { + // Abramowitz and Stegun 7.1.26 on the error function; ample for a p-value printed to three + // decimal places. + double t = 1 / (1 + 0.2316419 * Math.Abs(z)); + double d = 0.3989422804014327 * Math.Exp(-z * z / 2); + double p = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + + t * (-1.821255978 + t * 1.330274429)))); + return z >= 0 ? 1 - p : p; + } + + /// + /// A distribution-free interval for the median, from the binomial order statistics. + /// + /// No bootstrap, so no random number generator and no seed to remember: the same data give the + /// same interval on any machine, which is what a page regenerated from a manifest requires. + /// + private static (double Low, double High) MedianInterval(IReadOnlyList sorted) + { + int n = sorted.Count; + if (n < 6) return (sorted.Count > 0 ? sorted[0] : 0, sorted.Count > 0 ? sorted[^1] : 0); + + // Largest k with P(X < k) <= 0.025 under Binomial(n, 0.5): the conventional exact interval. + int k = 0; + double cumulative = 0; + for (int i = 0; i <= n; i++) + { + double logC = 0; + for (int j = 1; j <= i; j++) logC += Math.Log(n - i + j) - Math.Log(j); + cumulative += Math.Exp(logC - n * Math.Log(2)); + if (cumulative > 0.025) { k = i; break; } + } + + int low = Math.Max(0, k - 1); + int high = Math.Min(n - 1, n - k); + return (sorted[low], sorted[high]); + } + + private static double Quantile(IReadOnlyList sorted, double q) + { + if (sorted.Count == 0) return 0; + var index = (sorted.Count - 1) * q; + int lower = (int)Math.Floor(index), upper = (int)Math.Ceiling(index); + return lower == upper ? sorted[lower] : sorted[lower] + (sorted[upper] - sorted[lower]) * (index - lower); + } + + /// Reading order for the window labels, so the table runs front to back. + private static int PositionRank(string label) => label switch + { + "opening" => 0, + "middle" => 1, + "late" => 2, + _ => 3, + }; + + private static double Rate(int flagged, int total) => total == 0 ? 0 : (double)flagged / total; + + private static string Wilson(int flagged, int total) + { + var (low, high) = Calibrator.WilsonInterval(flagged, total); + return $"{Pct(low)} – {Pct(high)}"; + } + + private static string Pct(double value) => (value * 100).ToString("0.#", Inv) + "%"; + + private static string Signed(double value, string format = "0.0") => + (value > 0 ? "+" : "") + value.ToString(format, Inv); + + private static string FormatP(double p) => + p < 0.001 ? "p < 0.001" : "p = " + p.ToString("0.###", Inv); +} diff --git a/tools/SignsOfAI.Calibration/Program.cs b/tools/SignsOfAI.Calibration/Program.cs index 20beefd..accaeb0 100644 --- a/tools/SignsOfAI.Calibration/Program.cs +++ b/tools/SignsOfAI.Calibration/Program.cs @@ -2,7 +2,9 @@ using System.Text.Json; using SignsOfAI.Calibration; using SignsOfAI.Core; +using SignsOfAI.Core.Artifacts; using SignsOfAI.Core.Calibration; +using SignsOfAI.Core.Model; using SignsOfAI.Core.Text; // ── signsofai-calibrate ─────────────────────────────────────────────────────── @@ -31,6 +33,10 @@ string fetchLanguage = "en"; int count = 40, fromYear = 2018, toYear = 2020; string packsDir = "src/SignsOfAI.Core/Rules/Packs"; +string pairsPath = "Docs/Paraphrase/pairs.json"; +string humanDir = "", rewrittenDir = ""; +int perStratum = 8, targetWords = 400; +string paraphrasedBy = "", instructionPath = ""; for (int i = 1; i < argv.Count; i++) { @@ -46,6 +52,13 @@ case "--count": count = int.Parse(Next()); break; case "--from-year": fromYear = int.Parse(Next()); break; case "--to-year": toYear = int.Parse(Next()); break; + case "--pairs": pairsPath = Next(); break; + case "--human": humanDir = Next(); break; + case "--rewritten": rewrittenDir = Next(); break; + case "--per-stratum": perStratum = int.Parse(Next()); break; + case "--words": targetWords = int.Parse(Next()); break; + case "--paraphrased-by": paraphrasedBy = Next(); break; + case "--instruction": instructionPath = Next(); break; default: Console.Error.WriteLine($"Unknown option '{argv[i]}'."); return 2; @@ -53,7 +66,7 @@ string Next() => ++i < argv.Count ? argv[i] : throw new ArgumentException($"Missing value for {argv[i - 1]}"); } -if (argv[0] is not ("run" or "fetch" or "thresholds")) +if (argv[0] is not ("run" or "fetch" or "thresholds" or "excerpt" or "paraphrase")) { Console.Error.WriteLine($"Unknown command '{argv[0]}'. Run --help."); return 2; @@ -109,6 +122,318 @@ if (string.IsNullOrWhiteSpace(textsDir)) textsDir = Path.Combine(Path.GetDirectoryName(Path.GetFullPath(manifestPath))!, "texts"); +// ── `excerpt` ──────────────────────────────────────────────────────────────── +// Builds the human half of the paraphrase study: a stratified sample of the corpus, cut to +// equal-length passages of continuous prose. Nothing is rewritten here — that is done by a model, +// outside this tool, and recorded in the manifest this verb writes. +if (argv[0] == "excerpt") +{ + if (string.IsNullOrWhiteSpace(humanDir)) + humanDir = Path.Combine(Path.GetDirectoryName(Path.GetFullPath(pairsPath))!, "human"); + + Directory.CreateDirectory(humanDir); + + var picked = Paraphrase.Sample(manifest, perStratum); + var pairs = new PairManifest + { + Id = "signsofai-paraphrase-effect", + CorpusId = manifest.Id, + TargetWords = targetWords, + }; + + int skipped = 0; + foreach (var entry in picked) + { + var file = Path.Combine(textsDir, entry.File); + if (!File.Exists(file)) + { + Console.Error.WriteLine($" missing {entry.Id} ({entry.File})"); + skipped++; + continue; + } + + var excerpt = Paraphrase.Excerpt(File.ReadAllText(file), targetWords); + var words = Paraphrase.CountWords(excerpt); + + // A passage too short to have a distribution of sentence lengths cannot answer the question + // this study asks, and padding it out of a different part of the article would make the two + // halves of the pair no longer the same passage. + if (words < Paraphrase.MinimumWords) + { + Console.Error.WriteLine($" too short {entry.Id} ({words} words)"); + skipped++; + continue; + } + + File.WriteAllText(Path.Combine(humanDir, $"{entry.Id}.txt"), + excerpt.ReplaceLineEndings("\n") + "\n"); + + pairs.Pairs.Add(new PairEntry + { + Id = entry.Id, + Language = entry.Language, + Stratum = entry.Stratum, + Year = entry.Year, + Source = entry.Url ?? entry.Doi ?? "", + HumanSha256 = CorpusManifest.HashText(excerpt.ReplaceLineEndings("\n") + "\n"), + HumanWords = words, + }); + } + + pairs.Pairs.Sort((a, b) => string.CompareOrdinal(a.Id, b.Id)); + pairs.Save(pairsPath); + + Console.WriteLine(); + Console.WriteLine($" {pairs.Pairs.Count} excerpts written" + (skipped > 0 ? $", {skipped} skipped" : "")); + foreach (var group in pairs.Pairs.GroupBy(p => p.Stratum).OrderBy(g => g.Key, StringComparer.Ordinal)) + Console.WriteLine($" {group.Key,-28} {group.Count(),2} texts, {group.Sum(p => p.HumanWords),6:N0} words"); + Console.WriteLine(); + Console.WriteLine($" passages {humanDir}"); + Console.WriteLine($" manifest {pairsPath}"); + Console.WriteLine(); + Console.WriteLine(" Next: rewrite each passage into the 'rewritten' folder under the same name,"); + Console.WriteLine(" then run 'paraphrase --paraphrased-by --instruction '."); + return 0; +} + +// ── `paraphrase` ───────────────────────────────────────────────────────────── +// Measures both halves of every pair and writes the study up. The corpus manifest is not consulted +// here: the pairs carry their own provenance, and the human halves have already been hashed. +if (argv[0] == "paraphrase") +{ + var pairsDir = Path.GetDirectoryName(Path.GetFullPath(pairsPath))!; + if (string.IsNullOrWhiteSpace(humanDir)) humanDir = Path.Combine(pairsDir, "human"); + if (string.IsNullOrWhiteSpace(rewrittenDir)) rewrittenDir = Path.Combine(pairsDir, "rewritten"); + + if (!File.Exists(pairsPath)) + { + Console.Error.WriteLine($"No pair manifest at {pairsPath}. Run 'excerpt' first."); + return 2; + } + + var pairManifest = PairManifest.Load(pairsPath); + if (!string.IsNullOrWhiteSpace(paraphrasedBy)) pairManifest.ParaphrasedBy = paraphrasedBy; + if (!string.IsNullOrWhiteSpace(instructionPath)) + pairManifest.Instruction = File.ReadAllText(instructionPath).ReplaceLineEndings("\n"); + + if (string.IsNullOrWhiteSpace(pairManifest.ParaphrasedBy)) + { + // Refused rather than defaulted. A study of what a model does to prose, whose manifest does + // not name the model, produces a number nobody can reproduce or date. + Console.Error.WriteLine("--paraphrased-by is required: name the model that did the rewriting."); + return 2; + } + + var pairAnalyzer = new AiWritingAnalyzer(); + var measurements = new List(); + int incomplete = 0, changed = 0; + + // Where each control window starts, as a fraction of the document's prose. Three positions + // rather than one because the study's own pairs are cut from the opening, and the opening of a + // research article is its abstract while the opening of an encyclopedia entry is its lead — + // the most summary-shaped prose either genre contains. A length effect measured only there has + // a genre effect inside it, which is precisely the mistake the first version of this page made. + (string Label, double Position)[] WindowPositions = + [ + ("opening", 0.0), + ("middle", 0.5), + ("late", 0.75), + ]; + + foreach (var pair in pairManifest.Pairs) + { + var humanFile = Path.Combine(humanDir, $"{pair.Id}.txt"); + var rewrittenFile = Path.Combine(rewrittenDir, $"{pair.Id}.txt"); + + if (!File.Exists(humanFile) || !File.Exists(rewrittenFile)) + { + Console.Error.WriteLine($" incomplete {pair.Id}"); + incomplete++; + continue; + } + + var humanText = File.ReadAllText(humanFile); + var rewrittenText = File.ReadAllText(rewrittenFile); + + // The human half is hashed in the manifest, so a passage that drifted after the rewriting was + // done would silently break the pairing. The rewritten half is hashed on this run, since it + // does not exist until somebody produces it. + var humanHash = CorpusManifest.HashText(humanText); + if (!string.Equals(pair.HumanSha256, humanHash, StringComparison.OrdinalIgnoreCase)) + { + Console.Error.WriteLine($" CHANGED {pair.Id} manifest {pair.HumanSha256[..12]}…, file {humanHash[..12]}…"); + changed++; + continue; + } + + // The rewritten half is verified once it has a hash, and only assigned when it does not. + // The first version reassigned it on every run, which meant an edited rewrite was absorbed + // and re-hashed in silence — the manifest recording whatever was last on disk rather than + // what the study measured. + var rewrittenHash = CorpusManifest.HashText(rewrittenText); + if (pair.ParaphraseSha256 is { Length: > 0 } expectedRewrite) + { + if (!string.Equals(expectedRewrite, rewrittenHash, StringComparison.OrdinalIgnoreCase)) + { + if (!recordHashes) + { + Console.Error.WriteLine($" CHANGED {pair.Id} rewrite differs from the manifest " + + $"({expectedRewrite[..12]}… vs {rewrittenHash[..12]}…)"); + changed++; + continue; + } + + Console.WriteLine($" re-recorded {pair.Id}"); + pair.ParaphraseSha256 = rewrittenHash; + } + } + else + { + pair.ParaphraseSha256 = rewrittenHash; + } + + pair.ParaphraseWords = Paraphrase.CountWords(rewrittenText); + + var humanResult = pairAnalyzer.Analyze(humanText, pair.Language); + var rewrittenResult = pairAnalyzer.Analyze(rewrittenText, pair.Language); + + // How much of the original actually survived. Measured rather than attested, because the + // first version of this study attested it by hand and the hand was wrong. + var (longestRun, shareRetained) = Fidelity.Measure(humanText, rewrittenText); + + // The document the excerpt was cut from, and windows taken across it. Without these a reader + // cannot tell which of three things moved a score: the scissors, where the scissors fell, or + // the model. + AnalysisResult? fullResult = null, proseResult = null; + var windows = new List(); + + if (manifest.Texts.FirstOrDefault(t => t.Id == pair.Id) is { } sourceEntry) + { + var sourcePath = Path.Combine(textsDir, sourceEntry.File); + if (File.Exists(sourcePath)) + { + var sourceText = File.ReadAllText(sourcePath); + + // The third arm gets the same hash check the human half gets. It was reading these + // files and analyzing them with no verification at all, which left one arm of a + // published study resting on whatever happened to be in the directory. + var sourceHash = CorpusManifest.HashText(sourceText); + if (sourceEntry.Sha256 is { Length: > 0 } expectedSource && + !string.Equals(expectedSource, sourceHash, StringComparison.OrdinalIgnoreCase)) + { + Console.Error.WriteLine($" CHANGED {pair.Id} source text differs from the corpus manifest"); + changed++; + continue; + } + + fullResult = pairAnalyzer.Analyze(sourceText, pair.Language); + + var prose = Paraphrase.ProseOnly(sourceText); + if (Paraphrase.CountWords(prose) >= Paraphrase.MinimumWords) + proseResult = pairAnalyzer.Analyze(prose, pair.Language); + + foreach (var (label, position) in WindowPositions) + { + var window = Paraphrase.WindowAt(sourceText, pairManifest.TargetWords, position); + if (Paraphrase.CountWords(window) < Paraphrase.MinimumWords) continue; + + var windowResult = pairAnalyzer.Analyze(window, pair.Language); + windows.Add(new WindowMeasurement + { + Position = label, + Score = windowResult.OverallScore, + Burstiness = windowResult.Statistics.Burstiness, + Words = windowResult.Statistics.WordCount, + }); + } + } + } + + var humanScan = ArtifactScanner.Scan(humanText); + var rewrittenScan = ArtifactScanner.Scan(rewrittenText); + + measurements.Add(new PairMeasurement + { + FullScore = fullResult?.OverallScore, + FullBurstiness = fullResult?.Statistics.Burstiness, + FullWords = fullResult?.Statistics.WordCount ?? 0, + ProseOnlyScore = proseResult?.OverallScore, + Windows = windows, + LongestRun = longestRun, + ShareRetained = shareRetained, + ArtifactCodePoints = [.. humanScan.Occurrences.Concat(rewrittenScan.Occurrences) + .Select(o => o.CodePoint).Distinct()], + Id = pair.Id, + Language = pair.Language, + Stratum = pair.Stratum, + HumanScore = humanResult.OverallScore, + RewrittenScore = rewrittenResult.OverallScore, + HumanBurstiness = humanResult.Statistics.Burstiness, + RewrittenBurstiness = rewrittenResult.Statistics.Burstiness, + HumanWords = humanResult.Statistics.WordCount, + RewrittenWords = rewrittenResult.Statistics.WordCount, + HumanRuleIds = [.. humanResult.Findings.Where(f => !f.AtHumanRate).Select(f => f.RuleId)], + RewrittenRuleIds = [.. rewrittenResult.Findings.Where(f => !f.AtHumanRate).Select(f => f.RuleId)], + HumanArtifacts = humanScan.Count, + RewrittenArtifacts = rewrittenScan.Count, + }); + } + + if (changed > 0) + { + Console.Error.WriteLine($"\n{changed} passage(s) no longer match the manifest. Re-run 'excerpt'."); + return 1; + } + + // Stamped only when it has never been stamped, or when the run is explicitly re-recording. The + // first version set it on every measurement, so re-analyzing in October would have made the + // manifest claim the rewriting happened in October — falsifying the one date this design says + // must never be vague. + if (string.IsNullOrWhiteSpace(pairManifest.ParaphrasedOn) || recordHashes) + pairManifest.ParaphrasedOn = DateTime.UtcNow.ToString("yyyy-MM-dd"); + + pairManifest.Save(pairsPath); + + var pairVersion = typeof(AiWritingAnalyzer).Assembly + .GetCustomAttribute()?.InformationalVersion?.Split('+')[0] + ?? "unknown"; + + // The boundary is read from the engine's own published snapshot rather than restated here, so the + // study cannot quote a threshold the product has since moved away from. + var boundary = PublishedCalibration.Current?.RecommendedThreshold ?? 25; + + var paraphraseReport = ParaphraseReport.Render( + measurements, pairManifest, boundary, $"SignsOfAI.Core {pairVersion}", + DateTime.UtcNow.ToString("yyyy-MM-dd")); + + var paraphraseOut = outPath == "Docs/CALIBRATION.md" ? "Docs/PARAPHRASE.md" : outPath; + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(paraphraseOut))!); + File.WriteAllText(paraphraseOut, paraphraseReport.ReplaceLineEndings("\n")); + + int wasFlagged = measurements.Count(m => m.HumanScore >= boundary); + int nowFlagged = measurements.Count(m => m.RewrittenScore >= boundary); + + Console.WriteLine(); + Console.WriteLine($" {measurements.Count} pairs measured" + (incomplete > 0 ? $", {incomplete} incomplete" : "")); + Console.WriteLine($" pair fingerprint {pairManifest.Fingerprint()}"); + Console.WriteLine($" flagged at {boundary:0}/100 {wasFlagged} → {nowFlagged} of {measurements.Count}"); + Console.WriteLine($" median score {Median([.. measurements.Select(m => m.HumanScore)]):0.0}" + + $" → {Median([.. measurements.Select(m => m.RewrittenScore)]):0.0}"); + Console.WriteLine($" median burstiness {Median([.. measurements.Select(m => m.HumanBurstiness)]):0.00}" + + $" → {Median([.. measurements.Select(m => m.RewrittenBurstiness)]):0.00}"); + Console.WriteLine($" written to {paraphraseOut}"); + Console.WriteLine(); + return 0; + + static double Median(List values) + { + values.Sort(); + return values.Count == 0 ? 0 + : values.Count % 2 == 1 ? values[values.Count / 2] + : (values[values.Count / 2 - 1] + values[values.Count / 2]) / 2; + } +} + var analyzer = new AiWritingAnalyzer(); var samples = new List(); int missing = 0, mismatched = 0; @@ -279,6 +604,9 @@ dotnet run --project tools/SignsOfAI.Calibration -- run [options] dotnet run --project tools/SignsOfAI.Calibration -- fetch --source plos --count 40 dotnet run --project tools/SignsOfAI.Calibration -- fetch --source wikipedia --lang es --count 30 + dotnet run --project tools/SignsOfAI.Calibration -- excerpt --per-stratum 8 --words 400 + dotnet run --project tools/SignsOfAI.Calibration -- paraphrase --paraphrased-by "" + OPTIONS --manifest Corpus manifest (default: Docs/Calibration/corpus.json) --texts Where the texts live (default: /texts) @@ -289,6 +617,20 @@ assembling or deliberately updating the corpus. --lang fetch only: language for wikipedia (default: en) --count fetch only: how many texts (default: 40) --from-year/--to-year fetch only: publication window for plos (default 2018-2020) + --pairs Pair manifest for the paraphrase study (default: Docs/Paraphrase/pairs.json) + --human Passages as their authors wrote them (default: /human) + --rewritten The same passages after a model rewrote them (default: /rewritten) + --per-stratum excerpt only: passages to take from each group (default: 8) + --words excerpt only: length to cut each passage to (default: 400) + --paraphrased-by paraphrase only: the model that did the rewriting. Required — + a study of what a model does to prose that cannot name the model + has produced a number nobody can reproduce or date. + --instruction paraphrase only: the instruction the rewriter was given, + stored verbatim in the manifest because it is the treatment. + --record-hashes paraphrase: accept a rewritten half that differs from the manifest + and re-stamp the rewriting date. Use only when the rewriting has + genuinely been redone — every other run refuses to overwrite them, + so a published study cannot quietly acquire different material. WHY "What is your accuracy?" is the first question a teacher asks and the one no tool in @@ -299,4 +641,9 @@ whichever models were around that month. A false-positive rate needs only human writing, and it measures the harm this category actually causes. The texts are not in the repository; the manifest is. See Docs/Calibration/README.md. + + `excerpt` and `paraphrase` answer a second question with the same corpus: what does a + language model rewriting a passage do to it. Each unit is one passage measured as its + author wrote it and again after a model rewrote it, so no collection of machine-written + text is needed — the baseline is the passage itself. See Docs/Paraphrase/README.md. """);