Ask whether removing a watermark leaves a trace, and answer no

[peopleworks]

What this is

A measured answer to the question that arrived the week Anthropic started watermarking Claude's output: does a detector like this one still matter, and does stripping a watermark leave a trace we can see.

Each unit is one passage measured twice — as its author wrote it, and after a model rewrote it under a de-watermarking instruction. Both halves are the same passage at the same length, so no corpus of machine-written text was assembled; the argument in Docs/Calibration/README.md against ever assembling one still stands, and the baseline is the passage itself rather than a population.

What it found

Not what it went looking for. The expectation was that a rewrite performed to strip a watermark would leave the passage looking more like machine writing to this tool. Five passages crossed the verdict boundary, two crossed back, McNemar exact p = 0.453. The expectation is withdrawn in bold on the page.

A fault of ours, which is the real result. The verdict boundary was measured on documents of a few thousand words and is applied to text of any length:

Same writing, measured as	Flagged at 25/100	95% interval
whole documents	0 / 32	0% – 10.7%
400-word windows, three positions each	13 / 89	8.7% – 23.4%

Eleven of thirty documents are flagged at one position in the text but not another, and none is flagged wherever the window falls. Whether an author is accused depends on which four hundred words somebody pasted. Filed as #59.

Review

Three reviewers went over it adversarially, each told not to read the others' verdicts. The arithmetic survived independent recomputation — McNemar, Wilson, quantiles, counts, and byte-identical regeneration. Three published sentences did not:

• the pairs are cut from each document's opening, while the page claimed the middle — a length effect with a genre effect inside it, which moved the headline from 18.8% to 14.6% once measured at three positions;
• the deviation log called one breach of the eight-word rule unique when twenty of thirty-two pairs contain one;
• a directional claim about burstiness was asserted on evidence that fails the exact test applied two paragraphs above it (sign test p = 0.215; Wilcoxon p = 0.038; both now printed, with the note that neither was chosen in advance).

All three were the same failure: machine-checked numbers beside a hand-waved method. Treatment compliance is now measured by the tool on every run and printed whether or not it flatters the study.

Also in here

• ParaphrasedOn no longer re-stamped by every measurement run, which would have falsified the study's expiry date
• Both halves of each pair and the source text hash-verified, where only one of three was before
• Truncated rule tables now name what they dropped — the silently dropped row was stat.burstiness, the counterweight to its own introduced count
• The seven "suspicious codepoints" named: all U+0131, Turkish dotless ı in Turkish proper names inside a Spanish article — a false positive of our own character scanner, said out loud
• A conflict-of-interest section: the model that wrote these rule packs also did the rewriting, and until a second vendor's model repeats this, these numbers should not be quoted anywhere a teacher will act on them

Not in here

The passages themselves are git-ignored, like the calibration texts and for the same licence reasons; pairs.json carries both halves' hashes. The #59 fix is deliberately not attempted — a floor measured on truncations of long documents must not be enforced against text somebody composed at that length, and the page says so.

283 tests green.

🤖 Generated with Claude Code