What happened
Docs/PARAPHRASE.md reports seven suspicious codepoints across sixty-four passages. All seven are the same character in the same passage: U+0131 LATIN SMALL LETTER DOTLESS I, in the Turkish proper names Fazıl, Kıbrıs, Komandoları and Rıza, inside a Spanish Wikipedia article about a Turkish organisation.
Reproduce:
signsofai check Docs/Paraphrase/human/wp-es-132719628.txt --lang es --json
Seven occurrences, all U+0131, lines 3, 5 and 13.
Why it is wrong
The scanner's promise, stated in its own summary text, is that an artifact is a fingerprint of a tool that touched the file — a homoglyph substituted to disguise text, a zero-width character inserted by a rewriting service. U+0131 here is none of that. It is how Turkish is spelled, in a name, in a document whose subject is Turkish. Nobody substituted anything.
Characters.cs is explicit that its Latin-letter test covers "accents and all — legitimate in English and Spanish", and equally explicit in NeedsJoinControls about not flagging scripts that legitimately need join controls — Arabic, Hebrew, the Indic scripts, Myanmar. Turkish uses the Latin script and so slips past that reasoning: its dotless ı is a Latin letter that happens to look like a lookalike of i when the surrounding text is Spanish.
Why it matters more than seven characters
This project's artifact check is the one signal it presents as a fact rather than a judgement — a character is or is not there, and a teacher can verify it in any editor. A fact that is wrong is worse than a score that is uncertain. And the failure lands on proper names from a language other than the document's, which is the same shape of harm the calibration page exists to measure: the tool penalising text for being multilingual.
What to do
Not a simple allowlist of U+0131 — that trades one blunt rule for another. The question is what evidence distinguishes a disguise from an orthography, and there is some:
- Does the character appear inside a word whose other letters are also that script's? A single Cyrillic а inside otherwise-Latin análisis is a substitution. A dotless ı inside Kıbrıs alongside a dotless-ı-using name pattern is not.
- Does it recur across the document in the same lexical items? A disguise sprays; an orthography clusters in names that repeat.
- Is the document already flagged as containing a second language? Proper nouns from another language are the ordinary case in encyclopedia and academic prose.
Whatever the rule turns out to be, it belongs in Characters.cs with the reasoning written down, not in a rule pack — the existing comment explains why: this file encodes facts from the Unicode standard rather than editorial judgement, and a pack that quietly dropped a mapping would weaken a check without anyone noticing.
Test
tests/SignsOfAI.Core.Tests should gain a case asserting that Turkish proper names inside Spanish and English prose produce no artifact, alongside the existing cases asserting that a Cyrillic homoglyph mid-word does.
What happened
Docs/PARAPHRASE.mdreports seven suspicious codepoints across sixty-four passages. All seven are the same character in the same passage: U+0131 LATIN SMALL LETTER DOTLESS I, in the Turkish proper names Fazıl, Kıbrıs, Komandoları and Rıza, inside a Spanish Wikipedia article about a Turkish organisation.Reproduce:
Seven occurrences, all
U+0131, lines 3, 5 and 13.Why it is wrong
The scanner's promise, stated in its own summary text, is that an artifact is a fingerprint of a tool that touched the file — a homoglyph substituted to disguise text, a zero-width character inserted by a rewriting service. U+0131 here is none of that. It is how Turkish is spelled, in a name, in a document whose subject is Turkish. Nobody substituted anything.
Characters.csis explicit that its Latin-letter test covers "accents and all — legitimate in English and Spanish", and equally explicit inNeedsJoinControlsabout not flagging scripts that legitimately need join controls — Arabic, Hebrew, the Indic scripts, Myanmar. Turkish uses the Latin script and so slips past that reasoning: its dotless ı is a Latin letter that happens to look like a lookalike ofiwhen the surrounding text is Spanish.Why it matters more than seven characters
This project's artifact check is the one signal it presents as a fact rather than a judgement — a character is or is not there, and a teacher can verify it in any editor. A fact that is wrong is worse than a score that is uncertain. And the failure lands on proper names from a language other than the document's, which is the same shape of harm the calibration page exists to measure: the tool penalising text for being multilingual.
What to do
Not a simple allowlist of U+0131 — that trades one blunt rule for another. The question is what evidence distinguishes a disguise from an orthography, and there is some:
Whatever the rule turns out to be, it belongs in
Characters.cswith the reasoning written down, not in a rule pack — the existing comment explains why: this file encodes facts from the Unicode standard rather than editorial judgement, and a pack that quietly dropped a mapping would weaken a check without anyone noticing.Test
tests/SignsOfAI.Core.Testsshould gain a case asserting that Turkish proper names inside Spanish and English prose produce no artifact, alongside the existing cases asserting that a Cyrillic homoglyph mid-word does.