diff --git a/AGENTS.md b/AGENTS.md index da54fbdc..9b7f0115 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -87,7 +87,7 @@ uv run sphinx-build -b html docs dist/docs # tools/differential/expected_since_.toml. Until both are done a # bare compare.py measures against two minors back while reporting the # previous one, and _allowlist_for hard-errors on the missing file. -# tests/v2/test_regex_sync.py sweeps every expected_since_*.toml (#333), so +# tests/v2/test_ledger_guards.py sweeps every expected_since_*.toml (#333), so # the new ledger's hand copies of _SCRIPT_RANGES and of the honorific and # Latin vocabularies are checked from the day the file lands. All three # rosters find copies by their SYNTAX -- a span class or an alternation -- @@ -188,7 +188,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content. - **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, and values are deliberately NOT re-validated (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` keeps its own copy of the guard (layering) plus the `mappingproxy` slot rebuild; a new unpicklable slot type needs the same treatment plus a round-trip test. - **One sanctioned global**: the (future) cached default `Parser`. Lazily cached FROZEN singletons (`Lexicon.default()`'s `functools.cache`, the future default parser) are constants, not state; any second piece of module-level MUTABLE state requires amending the conventions doc, on purpose, in review. Sanctioned exceptions, facade layer only: `_config_shim.CONSTANTS` (the v1 shared singleton, mutable by design) and `_facade._WARNED_SUBCLASSES` (the once-per-subclass hook-warning dedup set) — both deleted with the layer in 3.0. -- **Tests**: all v2 tests live in the `tests/v2/` package (its `conftest.py` overrides the v1 dual-run fixture — v2 code never reads shared `CONSTANTS`), one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, and `test_regex_sync.py`, which pins every hand-copied pattern or codepoint table against its source wherever the copy lives — including copies outside the package), names stating behavior. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception. +- **Tests**: all v2 tests live in the `tests/v2/` package (its `conftest.py` overrides the v1 dual-run fixture — v2 code never reads shared `CONSTANTS`), one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, `test_regex_sync.py`, which pins the pipeline's hand-copied patterns against their sources, and `test_ledger_guards.py`, which pins the differential harness's hand copies and bounds what each ledger rule may claim — sharing `_differential_fixtures.py` with `test_differential.py` for the handles on `tools/differential/`), names stating behavior. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception. ## Extension Patterns diff --git a/tests/v2/_differential_fixtures.py b/tests/v2/_differential_fixtures.py new file mode 100644 index 00000000..2e32975f --- /dev/null +++ b/tests/v2/_differential_fixtures.py @@ -0,0 +1,99 @@ +"""Shared handles on the differential harness's own data. + +Imported by tests/v2/test_ledger_guards.py and tests/v2/test_differential.py, +which both need to reach `tools/differential/` -- the ledgers, the +corpora, and compare.py itself. Neither is importable the ordinary way: +`tools/` is outside `testpaths` and is not a package, so compare.py is +loaded by path, and the ledgers and corpora are data files a Python +module could not import if it wanted to. + +Not a conftest: these are constants and plain helpers, not fixtures in +pytest's sense, and two modules want them by name. Not a test module +either -- the guards that keep this data honest live next to the tests +that rely on it, in test_ledger_guards.py. +""" +import importlib.util +import json +import re +import tomllib +from pathlib import Path +from types import ModuleType + +from nameparser import _policy + +_TOOLS = Path(__file__).parents[2] / "tools" / "differential" + +#: Every baseline's ledger, swept rather than named. #332 added a second +#: file whose four hand copies went unchecked because the pins in +#: test_ledger_guards.py named the 1.4 one by filename, and the count grows by one per +#: release -- see AGENTS.md's release step 8. +_LEDGERS = sorted(_TOOLS.glob("expected_since_*.toml")) + +#: Every name the harness classifies, deduplicated. The ledgers exist +#: to explain diffs on THESE strings and no others, so "what does this +#: rule claim?" is answerable here without parsing anything -- a plain +#: regex search, no baseline wheel, no network. +#: +#: This is what test_ledger_guards.py's corpus checks read, and why +#: they hold where four rounds of syntactic ones did not. Depth-0 pipes, nesting +#: levels and probe strings are all proxies for the question that +#: actually matters; a rule cannot widen its corpus reach and still +#: answer this one the same way, however it is spelled. +_CORPUS_NAMES = sorted({ + json.loads(line) + for path in sorted(_TOOLS.glob("corpus*.jsonl")) + for line in path.read_text(encoding="utf-8").splitlines() if line.strip()}) + + +def _claimed(name_regex: str) -> list[str]: + """Corpus names a rule's regex matches.""" + return [name for name in _CORPUS_NAMES if re.search(name_regex, name)] + + +def _unclassified_names() -> list[str]: + """Corpus names carrying no codepoint _SCRIPT_RANGES classifies.""" + has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) + return [name for name in _CORPUS_NAMES if not has_classified(name)] + + +#: Built once. The expression this replaced sat inside a +#: comprehension's condition, so it rebuilt the script matcher AND +#: rescanned all 751 names per candidate name rather than per rule -- +#: measured around 400x a frozenset lookup, machine-dependent. The +#: rescan was the cost; the rebuild alone is minor. +_UNCLASSIFIED_NAMES = frozenset(_unclassified_names()) + + +def _rules(ledger: Path) -> list[dict]: + """The [[change]] table of one ledger.""" + # .get, matching compare.py. The open cycle's ledger is created at + # release with no `change` key at all -- an empty [[change]] array + # cannot be appended to in TOML -- so an absent key IS the empty + # ledger here, not a malformed file. What stops that leniency from + # hiding a typo'd table header lives in tests/v2/test_differential.py: + # every other ledger must be non-empty, and the open one may define + # nothing but `change`. + return tomllib.loads( + ledger.read_text(encoding="utf-8")).get("change", []) + + +def load_tool(stem: str) -> ModuleType: + """A module from tools/differential/, loaded by path. + + `tools/` is outside testpaths, and adding it would run + --doctest-modules over the corpus builders, so these are imported + this way rather than made importable. Two callers need it and + wrote the same six lines each: test_differential.py for compare.py, + test_ledger_guards.py for build_cjk_corpus.py. + + Neither has import-time side effects -- compare.py's main() is + behind a __name__ guard, and build_cjk_corpus.py only defines + functions -- so importing them to read a constant or call one + function is safe. + """ + spec = importlib.util.spec_from_file_location( + f"differential_{stem}", _TOOLS / f"{stem}.py") + assert spec is not None and spec.loader is not None, stem + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module diff --git a/tests/v2/test_differential.py b/tests/v2/test_differential.py index 443809ad..a26d20d4 100644 --- a/tests/v2/test_differential.py +++ b/tests/v2/test_differential.py @@ -2,33 +2,21 @@ `tools/` is outside `testpaths`, and adding it would run `--doctest-modules` over the corpus builders, so `compare.py` is -imported by path here -- the same way `test_regex_sync.py` already -imports `build_cjk_corpus`. +imported by path -- through `_differential_fixtures.load_compare`, +shared with `test_ledger_guards.py` so the loader exists once. Only pure logic is covered: nothing here spawns `uv` or the network. What is tested is what produces FALSE CONFIDENCE when it silently misbehaves -- which surfaces get compared, which ledger gets consulted, and above all whether a version tell is believed. """ -import importlib.util from pathlib import Path -from types import ModuleType import pytest -_TOOLS = Path(__file__).parents[2] / "tools" / "differential" +from ._differential_fixtures import _TOOLS, load_tool - -def _load_compare() -> ModuleType: - spec = importlib.util.spec_from_file_location( - "differential_compare", _TOOLS / "compare.py") - assert spec is not None and spec.loader is not None - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -compare = _load_compare() +compare = load_tool("compare") def test_parse_version_pads_a_short_release_to_three_parts() -> None: diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py new file mode 100644 index 00000000..56e5aad0 --- /dev/null +++ b/tests/v2/test_ledger_guards.py @@ -0,0 +1,1117 @@ +"""Pins the differential harness's hand copies to their sources, and +bounds what each ledger rule may claim. + +tools/differential/expected_since_*.toml classifies a v1-vs-v2 diff as +an INTENDED change. A rule matching too widely absorbs a real +regression and the run exits 0, so these files carry hand copies of +nameparser constants -- script ranges, honorific and maiden vocabulary +-- that a TOML file could not import if it wanted to. Nothing else +checks them. + +Most of what is here reads a rule's SYNTAX -- which spans a character +class declares, which members an alternation offers. Those are exact +where they apply and blind where they do not: five review rounds each +found a widening spelled just outside whichever one had been added +last (#333, #350). The rest read the CORPORA, asking what a rule +actually claims of the names the harness will ever be asked about; +those cannot be dodged by notation, because they do not read notation +-- though a recorded claim can be re-recorded, which is why the syntax +guards stay. + +Three tests are neither kind. They keep this file's own inputs honest: +the ledger glob, the roster-versus-filesystem staleness check, and the +pin holding a GENERATED corpus equal to what its generator would +write. + +Split from test_regex_sync.py, which shares none of this (#352). +""" +import hashlib +import json +import re +from pathlib import Path +from typing import NamedTuple + +import pytest + +from nameparser import _policy +from nameparser._policy import Script +# The parser's own fold, imported rather than reimplemented: a +# hand-written one here stripped commas, parens, brackets and quotes, +# five character classes neither the lexicon's fold nor config's +# assert_normalized touches -- looser in the dangerous direction, and a +# hand copy of a constant with a source of truth, inside the module +# written to forbid exactly that. +from nameparser._lexicon import _normalize +from nameparser.config.maiden_markers import MAIDEN_MARKERS +from nameparser.config.suffixes import ( + GLUED_HONORIFICS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_NOT_ACRONYMS) + +from ._differential_fixtures import ( + _CORPUS_NAMES, _LEDGERS, _TOOLS, _UNCLASSIFIED_NAMES, _claimed, _rules, + _unclassified_names, load_tool) + + +# The one sanctioned divergence between the differential rules' +# character classes and _SCRIPT_RANGES: the halfwidth middle dot +# separates tokens without being classified (halfwidth kana stays out +# of the table on purpose). U+00B7 is deliberately NOT here -- its +# flank guard means every name it can change matches through a +# classified flanking character already. Single-sourced: read by the +# span sweep below, and by the membership guard that keeps "sanctioned" +# meaning something -- an extra that becomes classified belongs in the +# table, not in this list. +_SANCTIONED_EXTRAS = frozenset({(0xFF65, 0xFF65)}) + + +def test_the_corpus_population_is_not_degenerate() -> None: + """The floors themselves live in compare.py and are asserted by + tests/v2/test_differential.py, which already checks every shipped + corpus clears one and that every floor names a file that exists. + Restating that here was a second, independently-drifting copy of a + guarantee the harness owns. + + What is local to THIS module is the population the guards actually + measure, which is not the same thing: _CORPUS_NAMES is + deduplicated, so a corpus rewritten as 486 copies of one line + clears its floor while the set collapses. And guard A is inert if + nothing in that set is unclassified. + """ + assert len(_CORPUS_NAMES) > 700, ( + f"_CORPUS_NAMES holds {len(_CORPUS_NAMES)} distinct names; the " + f"corpora clear their floors in compare.py but deduplicate to far " + f"fewer than usual, so every guard here is measuring a smaller " + f"population than it appears to") + assert _unclassified_names(), ( + "no corpus name lacks a classified codepoint, so the span rules' " + "unclassified-reach check has nothing to test against") + + +def test_ledger_glob_is_not_empty() -> None: + """A parametrize over an empty list generates a single silent SKIP + rather than a failure -- the exact silence this module exists to + break. The swept pins cannot assert this for themselves, since a + test that is never generated cannot complain, so it lives here.""" + assert _LEDGERS, f"no expected_since_*.toml under {_TOOLS}" + + +def test_span_bearing_roster_names_exactly_the_ledgers_on_disk() -> None: + """The roster's per-ledger sweep asserts every ledger has an entry, + but it runs over _LEDGERS, so it can never visit an entry naming a + file that is gone. A deleted or renamed ledger would leave its key + behind indefinitely -- the same staleness the roster's tag-level + check exists to prevent, one level up.""" + assert set(_SPAN_BEARING_RULES) == {ledger.name for ledger in _LEDGERS}, ( + f"_SPAN_BEARING_RULES names ledgers that do not exist: " + f"{sorted(set(_SPAN_BEARING_RULES) - {L.name for L in _LEDGERS})}; " + f"and is missing: " + f"{sorted({L.name for L in _LEDGERS} - set(_SPAN_BEARING_RULES))}") + + +_SPAN = re.compile(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})") + + +def _declared_spans(name_regex: str) -> set[tuple[int, int]]: + """The \\uXXXX-\\uXXXX span pairs a rule's character class declares.""" + return {(int(lo, 16), int(hi, 16)) + for lo, hi in _SPAN.findall(name_regex)} + + +def _unrecognized_class_content(name_regex: str) -> list[str]: + """Whatever a span-declaring character class holds BESIDES spans. + + _declared_spans reads one notation and is blind to every other, so + set equality against the table only pins what is written as an + escaped span. Anything appended in another notation -- a literal + range, a bare character, a leading "^" negating the whole class -- + is invisible to it and rides along unchecked. That is not + hypothetical: the ledgers' own convention mixes both spellings (see + the interpunct note in expected_since_2.0.0.toml), and their + non-span classes are written literally. + + The consequence is worst in the widening direction the equality is + supposed to cover. Appending "a-z" to a pinned CJK class passes both + that equality and compare.validate_rules' sentinel probe, and lets + the rule claim every Latin diff in the corpus as intended -- exactly + the regression-absorbing failure the harness exists to prevent. + + So: a class that declares any span must declare NOTHING else. + Classes carrying no spans (the delimiter sets) are a different + decision surface and are out of scope here -- as is widening spelled + OUTSIDE the brackets, which _no_top_level_alternation covers. + + Two known false positives, both deliberate: a single non-range + escape and a trailing literal "-" are legal regex and would be + rejected. Write them as a one-codepoint span instead. A class + metacharacter like \\s has no span spelling at all, so a rule + genuinely needing one has to move it out of the span-bearing class. + + The bracket scan is a simple findall, not a regex parser: an escaped + "\\]" inside a class truncates the body early and a nested "[" reads + as content. Both surface as unrecognized content rather than as + silence, which is the safe direction, and no ledger rule uses either. + """ + return [rest + for body in re.findall(r"\[([^\]]*)\]", name_regex) + if _SPAN.search(body) + for rest in [_SPAN.sub("", body)] + if rest] + + +def _top_level_alternation(name_regex: str) -> bool: + """Whether the rule has a "|" at nesting depth 0. + + The sibling of the hole above, three characters to the right of the + "]". Appending "|[A-Za-z]" to a span-bearing rule widens it exactly + as appending "a-z" inside the class would, and passes BOTH layers + that are supposed to stop that: the span equality sees an unchanged + class, and compare.validate_rules' sentinel probe clears it because + "Хосе Сантос" fails to match, breaking the matches-everything + conjunction. A CJK-scoped rule would then claim a Latin name's diff + as intended. + + No ledger rule has one today, and a rule that genuinely needs an + alternation can wrap it in "(?:...)", so requiring depth-0 purity + costs nothing and closes the hatch. + """ + depth = in_class = 0 + i = 0 + while i < len(name_regex): + char = name_regex[i] + if char == "\\": + i += 2 + continue + if in_class: + in_class = char != "]" + elif char == "[": + in_class = 1 + elif char == "(": + depth += 1 + elif char == ")": + depth -= 1 + elif char == "|" and depth == 0: + return True + i += 1 + return False + + +#: Codepoints the script table classifies that a ledger rule may still +#: spell literally inside a character class. The nakaguro separators +#: sit in the katakana block by Unicode block assignment while +#: functioning as punctuation, which is exactly why the delimiter rules +#: write them as themselves alongside the corner brackets. +_LITERAL_IN_CLASS_OK = frozenset("・･") + +_SPAN_TOKEN = re.compile(r"\\u[0-9A-Fa-f]{4}") + + +def _literally_spelled_script_chars(name_regex: str) -> list[str]: + """Classified codepoints a class spells as themselves, not as spans. + + This is the escape hatch every masking attack on the roster below + actually uses. Respell a rule's class in literal characters and the + regex means the same thing while the rule vanishes from discovery -- + its hand copy is then unpinned, and _SPAN_BEARING_RULES cannot tell + a rule that LEFT from a rule that was never there, because a set of + names cannot see that two different rules answer to one name (the + 1.4 ledger already has two rules tagged feat(#269)). + + Naming rules could never fix that on its own. Closing the hatch can: + a classified codepoint written literally in a class is refused, so a + class covering script content has to stay in the notation discovery + reads. + """ + return sorted({char + for body in re.findall(r"\[([^\]]*)\]", name_regex) + for char in _SPAN_TOKEN.sub("", body) + if char not in _LITERAL_IN_CLASS_OK + and _policy._script_matcher(*_policy._SCRIPT_RANGES)(char)}) + + +def _expected_bmp_spans() -> set[tuple[int, int]]: + """What a full CJK character class in the toml must declare: the + table's BMP spans plus the sanctioned extras. + + Han's astral block is the single table entry out of scope, on both + sides: the ledger rules omit it deliberately because no corpus name + reaches it -- see the comment on the canonical rule in + expected_since_1.4.0.toml, which is the only place that reasoning is + written down -- so the comparisons run over the BMP spans only + rather than failing forever on a difference everyone agreed to. + """ + return {span + for spans in _policy._SCRIPT_RANGES.values() + for span in spans + if span[1] <= 0xFFFF} | set(_SANCTIONED_EXTRAS) + + +def test_script_ranges_membership_is_decided() -> None: + """The two guards that belong to the script TABLE rather than to any + one ledger's copy of it. + + Every table entry is in scope for the differential rules. The + canonical rule covered HAN and HANGUL alone while the kana members + existed only for classification, but #272 gave HIRAGANA a default + order entry and made the kana blocks part of the same + first/middle/last diff shape, so scoping by issue no longer draws a + real line. Comparing against the whole table is the stronger + promise: a script added to _SCRIPT_RANGES for ANY reason fails here + until someone decides, in writing, whether the rules should cover + it. + + The second assert is what makes _SANCTIONED_EXTRAS mean something. + That set is the ledgers' licence to be WIDER than the table -- see + its definition above for why U+FF65 is in it and U+00B7 is not -- + and a licence nobody audits is just a hole. An extra that becomes + classified belongs in the table, not in the exception list, and + fails here until it moves. + + There is deliberately no canonical-rule selector here any more. It + picked rules by the literal '#271'/'#272' substrings and asserted + uniqueness, which #332 broke: expected_since_2.0.0.toml has two such + rules. Its equality check was in any case fully subsumed by + test_every_span_bearing_rule_matches_the_script_ranges, since the + canonical rule is itself span-bearing. Splitting the two guarantees + is the point -- the sweep owns "every hand copy equals the table", + this test owns "the table did not change shape without a decision" + -- so a selector break can no longer take the decision gate out as + collateral. Rule authors are correspondingly free to put #271 or + #272 in a compound slug. + """ + assert set(_policy._SCRIPT_RANGES) == { + Script.HAN, Script.HANGUL, Script.HIRAGANA, Script.KATAKANA}, ( + "a Script joined _SCRIPT_RANGES: decide whether the differential " + "rules in tools/differential/expected_since_*.toml should cover " + "it, then update this assertion") + for xlo, xhi in _SANCTIONED_EXTRAS: + assert not any(lo <= xhi and xlo <= hi + for spans in _policy._SCRIPT_RANGES.values() + for lo, hi in spans), ( + f"U+{xlo:04X}-U+{xhi:04X} is classified now; drop it " + "from _SANCTIONED_EXTRAS") + + +#: Which rules each ledger is known to carry a script-span copy in, +#: named by the leading fix(...)/feat(...) tag of their `issue`. +#: +#: Declared rather than counted. A count is identity-free, so one copy +#: could leave discovery -- rewritten as literal characters, say -- while +#: an unrelated span-bearing rule was added, and the total would hold +#: steady while a hand copy went unpinned (measured). Naming them also +#: buys the staleness direction the count never had, and that this +#: module's other two rosters already have: a tag here that matches no +#: rule fails, so a renamed or deleted rule cannot leave its entry +#: behind. +#: +#: Membership is the forcing function: a new baseline's ledger fails as +#: unrecorded until someone writes its rules down. An empty set is fine +#: and correct for a release that changed nothing CJK. +_SPAN_BEARING_RULES: dict[str, frozenset[str]] = { + "expected_since_1.4.0.toml": frozenset({ + "fix(#271/#272/#298)", # the canonical class + "fix(cjk-delimited-nickname)", # the three compounds, whose + "fix(cjk-fullwidth-paren-nickname)", # lookaheads each carry + "fix(cjk-comma-compound)", # their own copy + }), + "expected_since_2.0.0.toml": frozenset({ + "fix(#271/#272/#298)", # the canonical class + "fix(#298)", # the 间隔号 lookahead + }), + "expected_since_2.1.0.toml": frozenset(), # open cycle, no rules yet +} + +#: The leading `fix(...)`/`feat(...)` tag of a rule's `issue`, which is +#: what _SPAN_BEARING_RULES names rules by. Unique within each ledger +#: among span-bearing rules (asserted below), and stable across the +#: prose that follows it. +_ISSUE_TAG = re.compile(r"^[a-z]+\([^)]*\)") + + +def _tag(issue: str) -> str: + """The leading tag, required only of rules the roster has to name. + + Nothing obliges a ledger rule to carry a tag in general, and one in + the 1.4 file does not ("ambiguous-surname-acronym data change: + ..."). That is fine while it declares no script span. If such a rule + ever gains one it lands here, and the fix is to give it a tag rather + than to loosen the roster. + """ + match = _ISSUE_TAG.match(issue) + assert match, ( + f"a span-bearing rule's issue must open with a fix(...) or " + f"feat(...) tag so _SPAN_BEARING_RULES can name it; this one " + f"does not: {issue!r}") + return match.group(0) + + +@pytest.mark.parametrize("ledger", _LEDGERS, ids=lambda p: p.name) +def test_every_span_bearing_rule_matches_the_script_ranges( + ledger: Path) -> None: + """Auto-discovered pin for every hand copy of the script spans in + every ledger: any rule whose character class declares spans + intersecting _SCRIPT_RANGES must declare the whole expected class + (table BMP spans + sanctioned extras). + + A TOML file cannot import _policy._SCRIPT_RANGES, so these are the + copies with no possible alternative -- and the ones whose divergence + is quietest, because the harness is run by hand rather than in CI. + + Both failure directions matter, which is why this compares sets + rather than checking coverage. A span MISSING from a class turns an + intended change into an UNEXPLAINED diff (a release blocker for the + wrong reason); a span that should not be there silently classifies a + real regression as intended, which is the failure the whole harness + exists to prevent. + + The compound rules' require-a-classified-codepoint lookaheads exist + so their trigger sets alone (delimiters; a comma) cannot claim a + Latin name's regression -- and each such lookahead is a copy nothing + else checks. Discovery, not enrollment, is what subjects a rule to + the equality above: a new compound rule's copy is checked because it + exists, not because an author remembered it. _SPAN_BEARING_RULES + then holds discovery itself to account, since a copy rewritten in a + notation discovery cannot see would otherwise just vanish from the + sweep. Rules whose spans touch OTHER scripts (Cyrillic, say) are out + of scope and skipped by the intersection test. + """ + assert ledger.name in _SPAN_BEARING_RULES, ( + f"{ledger.name} is a new ledger whose span-bearing rules are not " + f"recorded; add it to _SPAN_BEARING_RULES (an empty set is a " + f"legal answer for a release that changed nothing CJK)") + table_spans = _expected_bmp_spans() + tags = [] + for rule in _rules(ledger): + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + declared = _declared_spans(regex) + if not declared & table_spans: + continue + tags.append(_tag(rule["issue"])) + assert declared == table_spans, ( + f"{ledger.name}: {rule['issue']!r} declares " + f"{sorted(declared)}; expected {sorted(table_spans)}") + extra = _unrecognized_class_content(regex) + assert not extra, ( + f"{ledger.name}: {rule['issue']!r} has a span-declaring " + f"character class holding {extra!r} besides its spans. The " + f"span equality above cannot see that, so it would widen the " + f"rule unchecked -- write it as an escaped span or not at all") + assert not _top_level_alternation(regex), ( + f"{ledger.name}: {rule['issue']!r} has a '|' at depth 0, so " + f"the whole rule is an alternation and the pinned class " + f"governs only one branch. Wrap it in '(?:...)'") + # The property the syntactic check above is only a proxy for. + # A rule scoped to classified scripts must not reach a name + # written in none of them -- and unlike a depth test, this does + # not care how the widening is spelled. "(?:CJK|[A-Za-z])" + # hides the pipe at depth 1 where the check above stops + # looking, and claims 644 of the 654 unclassified corpus names; + # this sees + # it. Both are kept: the depth test gives the clearer message + # for the naive spelling, and catches a widening toward a + # script the corpora happen not to contain. + unclassified = _UNCLASSIFIED_NAMES.intersection(_claimed(regex)) + assert not unclassified, ( + f"{ledger.name}: {rule['issue']!r} declares the script table's " + f"spans but claims {len(unclassified)} corpus names carrying " + f"no classified codepoint at all, e.g. {sorted(unclassified)[:3]}. A rule scoped " + f"to these scripts cannot explain a diff on those names, so it " + f"would absorb one instead") + # Every rule, not just the discovered ones: this is what stops a + # class being respelled out of discovery in the first place, and it + # has to reach the rules that are NOT currently span-bearing to do + # that job. + for rule in _rules(ledger): + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + literal = _literally_spelled_script_chars(regex) + assert not literal, ( + f"{ledger.name}: {rule['issue']!r} spells the classified " + f"codepoints {literal} literally inside a character class. " + f"Write them as \\uXXXX-\\uXXXX spans, or the rule drops out " + f"of the sweep above while meaning the same thing") + # two rules sharing a tag would collapse into one set member and + # read as a disappearance below, which is a confusing way to learn + # that the naming scheme broke + assert len(tags) == len(set(tags)), ( + f"{ledger.name}: two span-bearing rules share an issue tag " + f"({sorted(tags)}); _SPAN_BEARING_RULES cannot name them apart") + found = set(tags) + assert found == _SPAN_BEARING_RULES[ledger.name], ( + f"{ledger.name}'s span-bearing rules are not the recorded set. " + f"Left discovery (a hand copy is now unpinned): " + f"{sorted(_SPAN_BEARING_RULES[ledger.name] - found)}. " + f"Newly discovered (pinned now, but record it): " + f"{sorted(found - _SPAN_BEARING_RULES[ledger.name])}") + + +#: The delimiter compound's trigger set is its own decision surface, +#: separate from the script spans: these are the characters whose mere +#: presence lets the rule claim a diff. Written once here rather than +#: inline: the check this replaces tested the regex against this set +#: spelled as unicode escapes OR against it spelled as the characters +#: themselves, which in a non-raw literal are the same str -- an `or` +#: whose two operands could never disagree. One spelling, named once. +_NICKNAME_DELIMITERS = "[「」『』・･]" + + +def test_nickname_delimiter_sets_are_deliberate() -> None: + """Swept rather than pinned to one file and one rule. The 1.4 ledger + has exactly one cjk-delimited-nickname rule and the 2.0 ledger has + none, so a per-file `== 1` is already false and a global one holds + only by accident of there being a single rule today. What is + actually invariant is that EVERY such rule carries the sanctioned + trigger set, and that at least one exists somewhere or this is + checking nothing. + + Note the scope this does NOT have: it is a decision surface, not a + sync pin. _NICKNAME_DELIMITERS is a literal because the rule's class + is not Policy.nickname_delimiters and is not meant to be -- it is + the two CJK corner-bracket pairs, plus the nakaguro separators, + which delimit nothing, and minus the nine other pairs the config + ships. Deriving it would mean deciding all of that in code rather + than writing it down. + + So a delimiter pair removed from the config does not fail here; it + fails tests/v2/test_cases.py, on the cjk_white_corner_bracket_ + nickname row (measured -- tests/v2/pipeline/ stays green, which is + why the pointer is worth being exact about).""" + found = [] + for ledger in _LEDGERS: + for rule in _rules(ledger): + if "cjk-delimited-nickname" not in rule["issue"]: + continue + found.append(f"{ledger.name}: {rule['issue']}") + # .get rather than [] on purpose: a rule that dropped its + # name_regex outright should land on the assertion below, + # not raise KeyError out of the sweep + assert _NICKNAME_DELIMITERS in rule.get("name_regex", ""), ( + f"{ledger.name}: the compound rule's delimiter set " + f"changed, or the rule lost its name_regex; decide " + f"deliberately, then update _NICKNAME_DELIMITERS") + assert found, ( + "no cjk-delimited-nickname rule in any ledger; this check is " + "passing vacuously") + + +def test_cjk_corpus_matches_the_case_table() -> None: + """corpus_cjk.jsonl is GENERATED, not curated (#295): every + distinct case-table text bearing a codepoint the script table + classifies, sorted -- see build_cjk_corpus.py for why the other + two corpora cannot carry these names. The checked-in file must + equal what the generator would write, so a CJK case row added + without regenerating fails HERE instead of silently narrowing + the differential gate back toward the blind spot #295 closed. + Same promise as the toml pin above, aimed at a generated artifact + instead of a hand copy. + """ + module = load_tool("build_cjk_corpus") + checked_in = [json.loads(line) for line in + (_TOOLS / "corpus_cjk.jsonl") + .read_text(encoding="utf-8").splitlines()] + assert checked_in == module.selected_names(), ( + "corpus_cjk.jsonl is stale: regenerate with " + "`uv run python tools/differential/build_cjk_corpus.py`") + + +#: Which vocabulary constant each ledger rule's alternation is a hand +#: copy of. A roster rather than an inference: GLUED_HONORIFICS is a +#: SUBSET of SUFFIX_NOT_ACRONYMS (asserted at the bottom of +#: nameparser/config/suffixes.py), so "equals one of the two known sets" +#: would let a spaced rule that silently narrowed to exactly the glued +#: set pass by matching the other member -- a subset check wearing a +#: disguise, and the subset direction is precisely the one that removal +#: drift travels in. +#: +#: Keys are matched as substrings of a rule's `issue` and must select +#: exactly one entry. The full issue lists are the keys, not a bare +#: '#308': both 2.0 rules cite #308 while copying different constants. +_HONORIFIC_SOURCES: dict[str, set[str]] = { + "cjk-honorific-suffix": SUFFIX_NOT_ACRONYMS, # 1.4 + "#307/#308/#320": SUFFIX_NOT_ACRONYMS, # 2.0, spaced + "#308/#312/#319/#320": GLUED_HONORIFICS, # 2.0, glued +} + +#: An alternation group with two or more members and no nested +#: parentheses, capturing or not. Together with the classified-member +#: filter in _cjk_alternations it selects exactly the three honorific +#: alternations across both ledgers; on its own it also matches +#: "(?:^| )" from the 1.4 rule and the two Latin maiden/acronym +#: alternations, all of which the filter drops. +#: +#: The "(?:" alternative is spelled out and the plain "(" is guarded by +#: (?!\?) so that lookarounds -- "(?=$|[ ,])", "(?<=[^\s,])" -- do not +#: parse as alternations of their own syntax. Matching plain capturing +#: groups matters: a rule written "(씨|님|先生)" instead of +#: "(?:씨|님|先生)" would otherwise carry a hand copy this pin cannot +#: see, which is the silent-unpinning this module exists to prevent. +#: +#: Shapes it still cannot read correctly. All of them fail, but by two +#: different mechanisms, and the message you get depends on which +#: (every row measured): +#: +#: STALE half of the roster check -- the group stops matching, so the +#: rule's key goes unused: +#: a nested group whose inner members are UNclassified; +#: a paren inside a character class; +#: a single-member "group", which is not an alternation at all; +#: the named and inline-flag forms "(?Pa|b)" and "(?i:a|b)". +#: +#: Declared-vs-expected equality -- the group still matches, but its +#: members are MISread, since the split is a plain str.split("|"): +#: "|" inside a character class: "[a|b]" yields "[a" and "b]"; +#: a nested group whose inner members ARE classified: the inner +#: group is extracted and shows up as a member of the outer one. +_ALTERNATION = re.compile(r"\((?:\?:|(?!\?))((?:[^()|]+\|)+[^()|]+)\)") + + +def _alternations(name_regex: str) -> list[set[str]]: + """Every alternation group's members.""" + return [set(body.split("|")) for body in _ALTERNATION.findall(name_regex)] + + +def _cjk_alternations(name_regex: str) -> list[set[str]]: + """Every alternation in a rule with a script-classified member.""" + has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) + return [members for members in _alternations(name_regex) + if any(has_classified(m) for m in members)] + + +def test_differential_honorific_rules_match_their_vocabulary() -> None: + """The honorific rules' alternations are hand copies of the CJK + entries of SUFFIX_NOT_ACRONYMS (#307) and of GLUED_HONORIFICS + (#308) -- a toml cannot import them. Each expected set is DERIVED + from the config by script membership (a classified codepoint + anywhere in the entry), so adding a CJK honorific without widening + the rule, or widening a rule with something the vocabulary does not + ship, fails here. + + Swept over every ledger and every alternation, because the three + copies are anchored three different ways -- a leading '(?:^| )' in + 1.4, a character class and a lookbehind in 2.0 -- so the old + startswith/endswith parser could not read two of them. Discovery + plus a declared source is what replaces it. + + The span-bearing pins skip these rules on purpose: their trigger is + the alternation, not a character class. Note GLUED_HONORIFICS had no + pinned copy anywhere in the tree before this. + + Two completeness directions, both required. An alternation matching + no roster key fails as UNDECLARED, so a new rule's copy is pinned by + existing rather than by an author remembering to enroll it. A roster + key matching no rule fails as STALE, catching an entry left behind + after a rule was renamed or deleted. + """ + has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) + used: set[str] = set() + found = 0 + for ledger in _LEDGERS: + for rule in _rules(ledger): + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + for declared in _cjk_alternations(regex): + keys = [k for k in _HONORIFIC_SOURCES if k in rule["issue"]] + assert len(keys) == 1, ( + f"{ledger.name}: rule {rule['issue']!r} carries a CJK " + f"alternation matching {len(keys)} roster keys " + f"({keys}); every such hand copy must name exactly " + f"one source in _HONORIFIC_SOURCES") + used.add(keys[0]) + found += 1 + expected = {entry for entry in _HONORIFIC_SOURCES[keys[0]] + if has_classified(entry)} + assert declared == expected, ( + f"{ledger.name}: {rule['issue']!r} declares " + f"{sorted(declared)}; the config's CJK entries for " + f"{keys[0]!r} are {sorted(expected)}") + assert found, ( + "no CJK honorific alternation found in any ledger; this pin is " + "passing vacuously") + assert used == set(_HONORIFIC_SOURCES), ( + f"_HONORIFIC_SOURCES keys matching no rule: " + f"{sorted(set(_HONORIFIC_SOURCES) - used)}. Either a renamed or " + f"deleted rule left its entry behind -- drop it -- or that " + f"rule's alternation stopped matching _ALTERNATION (a nested " + f"group, a paren inside a character class, or a lone member), " + f"which is how a still-present hand copy silently leaves this " + f"pin.") + + +class _LatinCopy(NamedTuple): + """A ledger alternation that hand-copies a Latin-script vocabulary. + + Two set-shaped fields that are emphatically not the same kind of + thing, which is why they are named rather than positional: + `vocabulary` is the source of truth, `covers` an audited snapshot of + which of its entries the rule's members reach. + """ + vocabulary: set[str] + covers: frozenset[str] + + +#: Ledger rules whose alternation hand-copies a LATIN vocabulary, keyed +#: by a substring of the rule's `issue`. Kept apart from +#: _HONORIFIC_SOURCES because the relationship is not set equality: +#: these members are regex FRAGMENTS, not entries -- "n[ée]e" covers two +#: markers at once, "geb\.?" and "roz\.?" one each -- so there is no set +#: to compare against. +#: +#: `covers` is recorded rather than equated to the whole vocabulary. +#: Equality would force a rule to grow alternatives for markers it has +#: no reason to claim -- see the note on fix(#274) in the 1.4 ledger for +#: why 旧姓 in particular is not one of them. Recording still catches +#: removal: drop an entry a member covers and the snapshot shrinks. +#: +#: Three nearby counts differ and are easy to conflate, all for +#: fix(#274) specifically: MAIDEN_MARKERS ships 17 entries; that rule's +#: members reach 4 of them; the corpora contain 3 markers in total +#: (geb, née, 旧姓), only 2 of which it covers. +_LATIN_ALTERNATION_SOURCES: dict[str, _LatinCopy] = { + "fix(#274)": _LatinCopy( + vocabulary=MAIDEN_MARKERS, + covers=frozenset({"geb", "nee", "née", "roz"})), + "ambiguous-surname-acronym": _LatinCopy( + vocabulary=SUFFIX_ACRONYMS_AMBIGUOUS, + covers=frozenset({"do", "ma"})), +} + +#: Alternations that copy no vocabulary, so discovery must not demand a +#: source for them. Declared rather than inferred, on the principle +#: test_regex_sync.py's _SOURCES sets with its None entries: an undeclared alternation is a +#: question someone answers in writing, not something to skip past. +_NOT_A_VOCABULARY_COPY = frozenset({ + frozenset({"^", " "}), # the honorific rule's leading anchor +}) + +def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: + """Corpus names the whole rule claims that none of its own members + reach. + + The member checks bound what each ALTERNATIVE matches. This bounds + the rule built around them, which is a different question and the + one four rounds of syntactic guards kept failing to ask. Nesting a + rule's own alternation one level down and adding a branch -- + "(?:[(\"'](m\\.?a\\.?|d\\.?o\\.?)[)\"']|[A-Za-z]{4,})" -- leaves every + member innocent, hides the pipe below the depth test, and hides the + outer group from _ALTERNATION, which refuses nested parens. It + claimed 622 of the corpus while every check passed. + + Keyed on the roster rather than on `fields`, which is what makes it + reach that rule: the acronym copy claims `suffix`/`nickname`, so + the field-keyed guard below never looks at it. + + A rule claiming nothing scores zero, which is the strongest answer + rather than a vacuous one -- so the non-vacuity assertion is over + the union of all rostered rules, not per rule. + """ + reachable = [re.compile(member, re.IGNORECASE) for member in members] + return [name for name in _claimed(name_regex) + if not any(pattern.search(name) for pattern in reachable)] + + +def _reaches_non_vocabulary(member: str, vocabulary: set[str]) -> list[str]: + """Corpus text this member matches that is NOT a vocabulary entry. + + fullmatch against the vocabulary bounds what a member matches + WITHIN those entries and says nothing about what it matches in a + NAME. That gap was first filled with a tuple of eight hand-picked + probe strings, and the tuple was defeated by a wider rule than the + one it was added to stop: every entry this rule needs is three + characters, so a member must accept some 3-character string and is + unconstrained everywhere else -- "[acdf-uw-z]{3,}" covers `roz`, + dodges all eight probes, and reaches 634 of the 751 corpus names + as a fourth alternative (the rule carrying it claims 542). + Measured with the IGNORECASE this function applies; the + case-sensitive figure, 592, is not what runs. Counts here and below are + against _CORPUS_NAMES, which deduplicates the 783 corpus lines. + + Eight strings could never be more than a spot check. The corpus is + the whole population the rule will ever be asked about, so ask it + instead: every fragment a member matches must BE an entry. + + Searched unanchored, which is stricter than the rule's own \\b + anchoring and so cannot produce a false negative from it. + Normalized before comparison because the corpus writes "geb." where + the config stores "geb" -- without that, the shipped member fails. + Exempting only THIS vocabulary, not the union of all of them, keeps + a maiden member from being excused by an acronym entry. + """ + matches = set() + for name in _CORPUS_NAMES: + for found in re.finditer(member, name, re.IGNORECASE): + if found.group() and _normalize(found.group()) not in vocabulary: + matches.add(found.group()) + return sorted(matches) + + +def test_latin_alternations_mean_something_the_vocabulary_ships() -> None: + """The Latin twin of the honorific pin, shaped by what a regex + alternation over a vocabulary can honestly promise. + + Discovery-first, like its twin: every alternation in every ledger + must be a declared vocabulary copy or a declared non-copy. Keying + off the roster and skipping everything else would mean a future rule + that hand-copies a vocabulary under a new tag is pinned only if its + author remembers to enroll it -- the failure this module exists to + prevent, not a shape it should adopt. + + Members are matched against entries as regexes rather than compared + as strings, because the members ARE regex syntax: "geb\\.?" stands + for the entry "geb", which the config stores normalized (lowercase, + no trailing period). + + Two ways a MEMBER can be wrong, checked per member: it can match + nothing in the vocabulary -- then it cannot describe a real change + and can only claim other names' diffs, which is how "born" survived + from the harness's first commit to #350 -- or it can reach corpus + text that is not vocabulary. A third assertion is about the RULE + around them, which can widen at depth 0 and leave the pinned + alternation governing one branch of an unchecked whole. An invalid + member raises rather than asserts, since nothing else can proceed. + """ + has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) + used: set[str] = set() + found = 0 + reach_checked = 0 + for ledger in _LEDGERS: + for rule in _rules(ledger): + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + for members in _alternations(regex): + if any(has_classified(m) for m in members): + continue # the honorific pin owns these + if frozenset(members) in _NOT_A_VOCABULARY_COPY: + continue + found += 1 + keys = [k for k in _LATIN_ALTERNATION_SOURCES + if k in rule["issue"]] + assert len(keys) == 1, ( + f"{ledger.name}: {rule['issue']!r} carries the Latin " + f"alternation {sorted(members)}, which matches " + f"{len(keys)} roster keys ({keys}). Declare the " + f"vocabulary it copies in _LATIN_ALTERNATION_SOURCES, " + f"or add it to _NOT_A_VOCABULARY_COPY if it copies " + f"nothing") + used.add(keys[0]) + source = _LATIN_ALTERNATION_SOURCES[keys[0]] + unjustified = _unjustified_reach(regex, members) + assert not unjustified, ( + f"{ledger.name}: {rule['issue']!r} claims " + f"{len(unjustified)} corpus names that none of its own " + f"vocabulary members reach, e.g. {unjustified[:3]}. The " + f"members are innocent and the rule around them is not " + f"-- whatever it matches beyond them, it claims") + reach_checked += len(_claimed(regex)) + assert not _top_level_alternation(regex), ( + f"{ledger.name}: {rule['issue']!r} has a '|' at depth " + f"0, so the pinned alternation governs only one branch " + f"and the rest of the rule is unchecked. Wrap it in " + f"'(?:...)'") + covered = set() + for member in members: + try: + pattern = re.compile(member, re.IGNORECASE) + except re.error as exc: + raise AssertionError( + f"{ledger.name}: {rule['issue']!r} member " + f"{member!r} is not a valid regex ({exc}). A " + f"mis-split alternation can produce this -- see " + f"_ALTERNATION's notes") from None + matched = {entry for entry in source.vocabulary + if pattern.fullmatch(entry)} + loose = _reaches_non_vocabulary(member, source.vocabulary) + assert not loose, ( + f"{ledger.name}: {rule['issue']!r} member " + f"{member!r} matches corpus text that is not " + f"vocabulary: {loose[:6]}. Spell it literally -- a " + f"member broad enough to reach a real name lets the " + f"rule claim that name's diff as intended") + assert matched, ( + f"{ledger.name}: {rule['issue']!r} offers the " + f"alternative {member!r}, which matches no entry in " + f"the vocabulary it copies. It cannot correspond to " + f"a real change, so it can only claim other names' " + f"diffs as intended -- drop it, or ship it as " + f"vocabulary first") + covered |= matched + assert covered == source.covers, ( + f"{ledger.name}: {rule['issue']!r} covers " + f"{sorted(covered)}; recorded {sorted(source.covers)}. " + f"Lost: {sorted(source.covers - covered)} (an entry the " + f"rule relied on left the vocabulary). Gained: " + f"{sorted(covered - source.covers)} (record it)") + assert found, ( + "no Latin vocabulary alternation found in any ledger; this pin is " + "passing vacuously") + assert reach_checked, ( + "no rostered rule claims any corpus name, so the reach check above " + "measured nothing -- verify the corpora loaded") + assert used == set(_LATIN_ALTERNATION_SOURCES), ( + f"_LATIN_ALTERNATION_SOURCES keys matching no rule: " + f"{sorted(set(_LATIN_ALTERNATION_SOURCES) - used)}") + + +#: A role a rule can claim, mapped to the vocabulary a name must carry +#: for that claim to be possible. Keyed on `fields` rather than on the +#: regex, which is the point: the two pins above discover hand copies +#: by their SYNTAX, so a copy that is not written as an alternation or +#: a character class is not undeclared, it is unseen. A brand-new rule +#: whose name_regex is "(?i)\b[a-z]{3}\b" with maiden fields passed +#: every check in this module while claiming 308 corpus names. +#: +#: `fields` cannot be dodged the same way. A rule that does not claim +#: `maiden` cannot be labelled a maiden change at all, so keying here +#: reaches every spelling -- including fix(cjk-maiden-marker), whose +#: regex is the bare literal "旧姓" and which no roster in this module +#: could see. +#: +#: Only `maiden` today, because only its vocabulary is small and +#: mandatory enough for the implication to hold: a maiden diff needs a +#: marker in the name. `suffix` and `title` are not like that -- most +#: of their diffs come from routing, not from a vocabulary word being +#: present -- so adding them would be false rather than strict. +_FIELD_VOCABULARIES: dict[str, set[str]] = { + "maiden": MAIDEN_MARKERS, +} + + +def _carries(name: str, vocabulary: set[str]) -> bool: + """Whether a name contains a vocabulary entry. + + Whole-token for ASCII entries, substring for the rest, because a + marker like 旧姓 is written against the name it marks rather than + spaced off it. + + Note what the isascii() split actually covers: 12 of the 17 + entries, not only the CJK one. `né` is two characters, so the + substring branch reads `René` as carrying a marker. Every + over-match here SHRINKS the set of unexplained names and so + weakens the guard -- the direction this module exists to close -- + but exactly one corpus name reaches that branch today, and it is + the 旧姓 one. Tighten this before admitting a vocabulary whose + short non-ASCII entries occur inside ordinary names. + """ + tokens = {_normalize(token) for token in name.split()} + return bool(tokens & vocabulary) or any( + entry in name for entry in vocabulary if not entry.isascii()) + + +def test_rules_claiming_a_vocabulary_role_need_the_vocabulary_present() -> None: + """The guard that does not care how a rule is spelled. + + Every pin above starts from regex syntax -- an alternation, a + character class, a span. Each closed the hole it was built for and + left the next spelling open, four rounds running. This one starts + from what the rule CLAIMS: if a rule says a diff is a maiden-marker + change, then every corpus name it claims must actually carry a + maiden marker. A rule cannot escape that by changing notation, + because it is not reading the notation. + + Deliberately narrow. It does not say the rule is correct, only that + it cannot be explaining a marker on a name that has none. That + covers the maiden widenings review found; the ones claiming + `nickname` and `suffix` fall outside it, and _CORPUS_CLAIMS is + what catches those. + """ + checked = 0 + for ledger in _LEDGERS: + for rule in _rules(ledger): + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + for field, vocabulary in _FIELD_VOCABULARIES.items(): + if field not in (rule.get("fields") or []): + continue + checked += 1 + bare = [name for name in _claimed(regex) + if not _carries(name, vocabulary)] + assert not bare, ( + f"{ledger.name}: {rule['issue']!r} claims {field!r} " + f"diffs on {len(bare)} corpus names carrying no {field} " + f"vocabulary, e.g. {bare[:3]}. It cannot be explaining a " + f"marker that is not there, so on those names it would " + f"absorb a regression instead") + assert checked, ( + "no rule claims a role in _FIELD_VOCABULARIES; this pin is passing " + "vacuously") + + +class _Claim(NamedTuple): + """What a rule claims, in the three dimensions classify() uses. + + A count alone is identity-free -- the weakness this module rejects + at _SPAN_BEARING_RULES -- and review proved it here twice. Swapping + feat(#273)'s delimiter class for a single accented letter holds the + count at 6 while claiming six entirely different names. And + classify() narrows on `fields` as well as `name_regex`, so widening + a rule's roles moves nothing a regex-only count can see: the + comma rule kept its `,` regex and its 236 names while going from + explaining 6 of the corpus to 242. + """ + #: corpus names the name_regex reaches; the whole corpus when a + #: rule has none, which is the most unbounded shape validate_rules + #: permits and the one most worth writing down + names: int + #: the roles it narrows by, sorted; () when it narrows by regex alone + roles: tuple[str, ...] + #: sha256[:12] of the claimed names, so a swap that holds the count + #: still fails. Unreadable by design -- the count above is what a + #: reviewer reads, and the failure message prints what moved. + digest: str + + +def _claim(rule: dict) -> _Claim: + regex = rule.get("name_regex") + names = (_claimed(regex) if isinstance(regex, str) else list(_CORPUS_NAMES)) + fields = rule.get("fields") + return _Claim( + names=len(names), + roles=tuple(sorted(fields)) if isinstance(fields, list) else (), + digest=hashlib.sha256( + "\n".join(names).encode("utf-8")).hexdigest()[:12]) + + +#: How many corpus names each rule's name_regex matches, per ledger. +#: +#: The backstop the other guards each failed to be. Every one of them +#: is scoped to a CATEGORY -- a span class, an alternation's members, a +#: `maiden` field, a member's reach -- and five review rounds each found +#: a rule outside whichever category the last fix had covered. The +#: categories are the test's, not the ledger's. What every rule shares +#: is what it claims: how much corpus its regex reaches, which roles +#: it narrows by, and WHICH names those are. Scoped to the corpus -- +#: and only there -- a widening cannot change what a rule explains +#: without moving one of the three. Restoring 'born' moves none of +#: them, because no corpus name contains it; the member guards catch +#: that, which is why this does not replace them. +#: +#: So this is deliberately dumb: it knows nothing about vocabularies, +#: scripts or roles, and it cannot say whether a number is RIGHT. It +#: says only that it moved, which is the question a widening cannot +#: dodge. The specific guards above stay because they explain WHY a +#: rule may claim what it claims, and their messages are the ones worth +#: reading; this one just refuses to let the number change quietly. +#: +#: Keyed by the full `issue` rather than by tag: the 1.4 ledger has two +#: rules tagged feat(#269), and a tag-keyed roster cannot tell them +#: apart -- the same identity-free weakness recorded at +#: _SPAN_BEARING_RULES. +#: +#: These numbers move when the CORPORA move, not only when a rule does. +#: That is the intended cost: a corpus name added under an existing +#: rule is a real change in what that rule explains, and it should be +#: read once rather than absorbed silently. +_CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { + "expected_since_1.4.0.toml": { + "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": + _Claim(97, ('family', 'given', 'middle'), "66e71d60a075"), + "fix(#274) maiden markers consumed": + _Claim(4, ('family', 'maiden', 'middle'), "b31dc2e2bbc4"), + "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": + _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + "fix(comma-family) lone post-comma piece routes to suffix/title, not first": + _Claim(236, ('given', 'suffix', 'title'), "3416f69d0ce4"), + "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": + _Claim(751, ('family', 'given', 'suffix'), "231640fc7535"), + "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": + _Claim(0, ('suffix',), "e3b0c44298fc"), + "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": + _Claim(0, ('nickname', 'suffix'), "e3b0c44298fc"), + "feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)": + _Claim(2, ('family', 'middle'), "3e2b5c6d1f4d"), + "feat(#273) typographic nickname delimiters recognized by default": + _Claim(6, ('middle', 'nickname'), "a03c9763c8c4"), + "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJK order flip": + _Claim(6, ('family', 'given', 'nickname'), "ae1dffa01608"), + "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": + _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7"), + "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": + _Claim(20, ('family', 'given', 'middle', 'suffix', 'title'), "b2ea8fa59eea"), + "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": + _Claim(14, ('family', 'given', 'middle', 'suffix'), "d49ce901bdce"), + "feat(#269) non-Latin titles/conjunctions recognized": + _Claim(2, ('given', 'middle', 'title'), "c14187bb08f8"), + "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": + _Claim(1, ('given', 'middle', 'suffix', 'title'), "390e7f814d13"), + }, + "expected_since_2.0.0.toml": { + "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": + _Claim(97, ('_ambiguities', 'family', 'given', 'middle'), "66e71d60a075"), + "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": + _Claim(34, ('family', 'given', 'suffix'), "877ab3246d33"), + "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": + _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2"), + "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": + _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + "fix(#272) nakaguro inside delimited content renders as a space, compounding with the CJK order flip": + _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), + "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": + _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), + }, + "expected_since_2.1.0.toml": {}, +} + + +def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: + """The guard that knows nothing, and therefore cannot be dodged. + + Five rounds of review defeated five guards, each by moving to a + rule the last fix did not cover: a span rule, then an alternation's + members, then a non-maiden role, then a rule whose narrowing lived + outside its alternation entirely. Every one of those attacks + changed what the rule claimed -- 6 to 668 names, 0 to 193, and the + comma rule from 3 roles to 6 while its regex and its 236 names + stood still -- because that is what widening a rule means WITHIN + the corpus. A widening reaching only names the corpora lack moves + nothing here; the guards above are what see those. + + Note which layer is which, because it is the opposite of what it + looks like. This is a change DETECTOR, not an enforcer: it is + inert for a brand-new rule, whose author simply records whatever + number it produces, and its own failure message invites the + remedy that defeats it -- re-record and the attack lands. The + member and vocabulary guards above are the walls, because they + judge a rule wrong at any time INCLUDING at recording time. This + catches the widenings none of them is scoped to see, and holds + them still long enough for someone to look. + """ + for ledger in _LEDGERS: + assert ledger.name in _CORPUS_CLAIMS, ( + f"{ledger.name} is a new ledger with no recorded corpus claims; " + f"add it to _CORPUS_CLAIMS (an empty mapping if it has no rules)") + recorded = _CORPUS_CLAIMS[ledger.name] + # Keyed on `issue`, so a duplicate silently collapses to the + # last rule written and the other goes unmeasured -- coverage + # by file order, which is no coverage. Nothing else asserts + # this: validate_rules only requires a non-empty string, and + # the tag-uniqueness check above covers span-bearing rules + # alone. _LATIN_ALTERNATION_SOURCES and _HONORIFIC_SOURCES key + # on issue SUBSTRINGS, so they lean on it too. + issues = [rule["issue"] for rule in _rules(ledger)] + assert len(set(issues)) == len(issues), ( + f"{ledger.name} has rules sharing an `issue`: " + f"{sorted({i for i in issues if issues.count(i) > 1})}. Every " + f"roster here keys on it, so one of them would go unmeasured") + # A rule with no name_regex narrows by `fields` alone and so + # reaches EVERY name -- the most unbounded shape validate_rules + # permits, and the one most worth recording. Counting it as the + # whole corpus is not a placeholder; it is what it claims. + actual = {rule["issue"]: _claim(rule) + for rule in _rules(ledger)} + moved = {issue: (recorded.get(issue), count) + for issue, count in actual.items() + if recorded.get(issue) != count} + assert actual == recorded, ( + f"{ledger.name}: the corpus each rule claims is not what is " + f"recorded. Moved (recorded, now): {moved}. Gone: " + f"{sorted(set(recorded) - set(actual))}. New: " + f"{sorted(set(actual) - set(recorded))}. A number that GREW " + f"means the rule claims more of the corpus than it did -- " + f"check it is not absorbing a regression, then record it") + assert set(_CORPUS_CLAIMS) == {led.name for led in _LEDGERS}, ( + f"_CORPUS_CLAIMS names ledgers that do not exist: " + f"{sorted(set(_CORPUS_CLAIMS) - {L.name for L in _LEDGERS})}") + diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 8c09fef1..c1fa9955 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -9,42 +9,22 @@ CI signal. Tests may legally import both sides (test_layering.py's own convention), so this module is where the promise gets checked. -Layering is the usual reason for a copy but not the only one, so this -module's scope is the PROMISE rather than that one pair of packages: -the comma-set pin below reads _pipeline._state instead of config, and -several tests reach outside the package altogether: eight read the -differential ledgers, which could not import a Python constant if they -wanted to, and four of those eight also read the corpora as a DATA -POPULATION rather than as an artifact -- asking what a ledger rule -actually claims, which is the question no amount of inspecting its -syntax could answer. One more pins a generated corpus against its -generator, which can and must stay run. +Layering is the usual reason for a copy but not the only one, so the +scope here is the PROMISE rather than that one pair of packages: the +comma-set pin below reads _pipeline._state instead of config. + +The differential harness makes the same kind of copy by hand, into +files that could not import a Python constant if they wanted to. Those +are pinned in test_ledger_guards.py, which shares nothing with this +module but the idea (#352). """ -import hashlib -import importlib.util -import json import re -import tomllib -from pathlib import Path -from typing import NamedTuple import pytest from nameparser.config import regexes as _config from nameparser._pipeline import _assign, _post_rules, _tokenize, _vocab -from nameparser import _policy -from nameparser._policy import Script from nameparser import _render -# The parser's own fold, imported rather than reimplemented: a -# hand-written one here stripped commas, parens, brackets and -# quotes, five classes neither the lexicon's fold nor config's -# assert_normalized touches -- looser in the dangerous direction, -# and a hand copy of a constant with a source of truth, inside the -# module written to forbid exactly that. -from nameparser._lexicon import _normalize -from nameparser.config.maiden_markers import MAIDEN_MARKERS -from nameparser.config.suffixes import ( - GLUED_HONORIFICS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_NOT_ACRONYMS) def test_emoji_ranges_match_config() -> None: @@ -178,1128 +158,3 @@ def test_comma_char_matches_the_pipeline_comma_set() -> None: from nameparser._pipeline._state import COMMA_CHARS assert set(_render._COMMA_CHAR.pattern.strip("[]")) == set(COMMA_CHARS) - - -# The one sanctioned divergence between the differential rules' -# character classes and _SCRIPT_RANGES: the halfwidth middle dot -# separates tokens without being classified (halfwidth kana stays out -# of the table on purpose). U+00B7 is deliberately NOT here -- its -# flank guard means every name it can change matches through a -# classified flanking character already. Single-sourced: read by the -# span sweep below, and by the membership guard that keeps "sanctioned" -# meaning something -- an extra that becomes classified belongs in the -# table, not in this list. -_SANCTIONED_EXTRAS = frozenset({(0xFF65, 0xFF65)}) - -_TOOLS = Path(__file__).parents[2] / "tools" / "differential" - -#: Every baseline's ledger, swept rather than named. #332 added a second -#: file whose four hand copies went unchecked because the pins below -#: named the 1.4 one by filename, and the count grows by one per -#: release -- see AGENTS.md's release step 8. -_LEDGERS = sorted(_TOOLS.glob("expected_since_*.toml")) - -#: Every name the harness classifies, deduplicated. The ledgers exist -#: to explain diffs on THESE strings and no others, so "what does this -#: rule claim?" is answerable here without parsing anything -- a plain -#: regex search, no baseline wheel, no network. -#: -#: This is what the guards below check against, and it is why they hold -#: where four rounds of syntactic ones did not. Depth-0 pipes, nesting -#: levels and probe strings are all proxies for the question that -#: actually matters; a rule cannot widen its corpus reach and still -#: answer this one the same way, however it is spelled. -_CORPUS_NAMES = sorted({ - json.loads(line) - for path in sorted(_TOOLS.glob("corpus*.jsonl")) - for line in path.read_text(encoding="utf-8").splitlines() if line.strip()}) - - -def _claimed(name_regex: str) -> list[str]: - """Corpus names a rule's regex matches.""" - return [name for name in _CORPUS_NAMES if re.search(name_regex, name)] - - -def _unclassified_names() -> list[str]: - """Corpus names carrying no codepoint _SCRIPT_RANGES classifies.""" - has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) - return [name for name in _CORPUS_NAMES if not has_classified(name)] - - -#: Built once. The expression this replaced sat inside a -#: comprehension's condition, so it rebuilt the script matcher AND -#: rescanned all 751 names per candidate name rather than per rule -- -#: measured around 400x a frozenset lookup, machine-dependent. The -#: rescan was the cost; the rebuild alone is minor. -_UNCLASSIFIED_NAMES = frozenset(_unclassified_names()) - - -def test_the_corpus_population_is_not_degenerate() -> None: - """The floors themselves live in compare.py and are asserted by - tests/v2/test_differential.py, which already checks every shipped - corpus clears one and that every floor names a file that exists. - Restating that here was a second, independently-drifting copy of a - guarantee the harness owns. - - What is local to THIS module is the population the guards actually - measure, which is not the same thing: _CORPUS_NAMES is - deduplicated, so a corpus rewritten as 486 copies of one line - clears its floor while the set collapses. And guard A is inert if - nothing in that set is unclassified. - """ - assert len(_CORPUS_NAMES) > 700, ( - f"_CORPUS_NAMES holds {len(_CORPUS_NAMES)} distinct names; the " - f"corpora clear their floors in compare.py but deduplicate to far " - f"fewer than usual, so every guard here is measuring a smaller " - f"population than it appears to") - assert _unclassified_names(), ( - "no corpus name lacks a classified codepoint, so the span rules' " - "unclassified-reach check has nothing to test against") - - -def test_ledger_glob_is_not_empty() -> None: - """A parametrize over an empty list generates a single silent SKIP - rather than a failure -- the exact silence this module exists to - break. The swept pins cannot assert this for themselves, since a - test that is never generated cannot complain, so it lives here.""" - assert _LEDGERS, f"no expected_since_*.toml under {_TOOLS}" - - -def _rules(ledger: Path) -> list[dict]: - """The [[change]] table of one ledger.""" - # .get, matching compare.py. The open cycle's ledger is created at - # release with no `change` key at all -- an empty [[change]] array - # cannot be appended to in TOML -- so an absent key IS the empty - # ledger here, not a malformed file. What stops that leniency from - # hiding a typo'd table header lives in tests/v2/test_differential.py: - # every other ledger must be non-empty, and the open one may define - # nothing but `change`. - return tomllib.loads( - ledger.read_text(encoding="utf-8")).get("change", []) - - -def test_span_bearing_roster_names_exactly_the_ledgers_on_disk() -> None: - """The roster's per-ledger sweep asserts every ledger has an entry, - but it runs over _LEDGERS, so it can never visit an entry naming a - file that is gone. A deleted or renamed ledger would leave its key - behind indefinitely -- the same staleness the roster's tag-level - check exists to prevent, one level up.""" - assert set(_SPAN_BEARING_RULES) == {ledger.name for ledger in _LEDGERS}, ( - f"_SPAN_BEARING_RULES names ledgers that do not exist: " - f"{sorted(set(_SPAN_BEARING_RULES) - {L.name for L in _LEDGERS})}; " - f"and is missing: " - f"{sorted({L.name for L in _LEDGERS} - set(_SPAN_BEARING_RULES))}") - - -_SPAN = re.compile(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})") - - -def _declared_spans(name_regex: str) -> set[tuple[int, int]]: - """The \\uXXXX-\\uXXXX span pairs a rule's character class declares.""" - return {(int(lo, 16), int(hi, 16)) - for lo, hi in _SPAN.findall(name_regex)} - - -def _unrecognized_class_content(name_regex: str) -> list[str]: - """Whatever a span-declaring character class holds BESIDES spans. - - _declared_spans reads one notation and is blind to every other, so - set equality against the table only pins what is written as an - escaped span. Anything appended in another notation -- a literal - range, a bare character, a leading "^" negating the whole class -- - is invisible to it and rides along unchecked. That is not - hypothetical: the ledgers' own convention mixes both spellings (see - the interpunct note in expected_since_2.0.0.toml), and their - non-span classes are written literally. - - The consequence is worst in the widening direction the equality is - supposed to cover. Appending "a-z" to a pinned CJK class passes both - that equality and compare.validate_rules' sentinel probe, and lets - the rule claim every Latin diff in the corpus as intended -- exactly - the regression-absorbing failure the harness exists to prevent. - - So: a class that declares any span must declare NOTHING else. - Classes carrying no spans (the delimiter sets) are a different - decision surface and are out of scope here -- as is widening spelled - OUTSIDE the brackets, which _no_top_level_alternation covers. - - Two known false positives, both deliberate: a single non-range - escape and a trailing literal "-" are legal regex and would be - rejected. Write them as a one-codepoint span instead. A class - metacharacter like \\s has no span spelling at all, so a rule - genuinely needing one has to move it out of the span-bearing class. - - The bracket scan is a simple findall, not a regex parser: an escaped - "\\]" inside a class truncates the body early and a nested "[" reads - as content. Both surface as unrecognized content rather than as - silence, which is the safe direction, and no ledger rule uses either. - """ - return [rest - for body in re.findall(r"\[([^\]]*)\]", name_regex) - if _SPAN.search(body) - for rest in [_SPAN.sub("", body)] - if rest] - - -def _top_level_alternation(name_regex: str) -> bool: - """Whether the rule has a "|" at nesting depth 0. - - The sibling of the hole above, three characters to the right of the - "]". Appending "|[A-Za-z]" to a span-bearing rule widens it exactly - as appending "a-z" inside the class would, and passes BOTH layers - that are supposed to stop that: the span equality sees an unchanged - class, and compare.validate_rules' sentinel probe clears it because - "Хосе Сантос" fails to match, breaking the matches-everything - conjunction. A CJK-scoped rule would then claim a Latin name's diff - as intended. - - No ledger rule has one today, and a rule that genuinely needs an - alternation can wrap it in "(?:...)", so requiring depth-0 purity - costs nothing and closes the hatch. - """ - depth = in_class = 0 - i = 0 - while i < len(name_regex): - char = name_regex[i] - if char == "\\": - i += 2 - continue - if in_class: - in_class = char != "]" - elif char == "[": - in_class = 1 - elif char == "(": - depth += 1 - elif char == ")": - depth -= 1 - elif char == "|" and depth == 0: - return True - i += 1 - return False - - -#: Codepoints the script table classifies that a ledger rule may still -#: spell literally inside a character class. The nakaguro separators -#: sit in the katakana block by Unicode block assignment while -#: functioning as punctuation, which is exactly why the delimiter rules -#: write them as themselves alongside the corner brackets. -_LITERAL_IN_CLASS_OK = frozenset("・･") - -_SPAN_TOKEN = re.compile(r"\\u[0-9A-Fa-f]{4}") - - -def _literally_spelled_script_chars(name_regex: str) -> list[str]: - """Classified codepoints a class spells as themselves, not as spans. - - This is the escape hatch every masking attack on the roster below - actually uses. Respell a rule's class in literal characters and the - regex means the same thing while the rule vanishes from discovery -- - its hand copy is then unpinned, and _SPAN_BEARING_RULES cannot tell - a rule that LEFT from a rule that was never there, because a set of - names cannot see that two different rules answer to one name (the - 1.4 ledger already has two rules tagged feat(#269)). - - Naming rules could never fix that on its own. Closing the hatch can: - a classified codepoint written literally in a class is refused, so a - class covering script content has to stay in the notation discovery - reads. - """ - return sorted({char - for body in re.findall(r"\[([^\]]*)\]", name_regex) - for char in _SPAN_TOKEN.sub("", body) - if char not in _LITERAL_IN_CLASS_OK - and _policy._script_matcher(*_policy._SCRIPT_RANGES)(char)}) - - -def _expected_bmp_spans() -> set[tuple[int, int]]: - """What a full CJK character class in the toml must declare: the - table's BMP spans plus the sanctioned extras. - - Han's astral block is the single table entry out of scope, on both - sides: the ledger rules omit it deliberately because no corpus name - reaches it -- see the comment on the canonical rule in - expected_since_1.4.0.toml, which is the only place that reasoning is - written down -- so the comparisons run over the BMP spans only - rather than failing forever on a difference everyone agreed to. - """ - return {span - for spans in _policy._SCRIPT_RANGES.values() - for span in spans - if span[1] <= 0xFFFF} | set(_SANCTIONED_EXTRAS) - - -def test_script_ranges_membership_is_decided() -> None: - """The two guards that belong to the script TABLE rather than to any - one ledger's copy of it. - - Every table entry is in scope for the differential rules. The - canonical rule covered HAN and HANGUL alone while the kana members - existed only for classification, but #272 gave HIRAGANA a default - order entry and made the kana blocks part of the same - first/middle/last diff shape, so scoping by issue no longer draws a - real line. Comparing against the whole table is the stronger - promise: a script added to _SCRIPT_RANGES for ANY reason fails here - until someone decides, in writing, whether the rules should cover - it. - - The second assert is what makes _SANCTIONED_EXTRAS mean something. - That set is the ledgers' licence to be WIDER than the table -- see - its definition above for why U+FF65 is in it and U+00B7 is not -- - and a licence nobody audits is just a hole. An extra that becomes - classified belongs in the table, not in the exception list, and - fails here until it moves. - - There is deliberately no canonical-rule selector here any more. It - picked rules by the literal '#271'/'#272' substrings and asserted - uniqueness, which #332 broke: expected_since_2.0.0.toml has two such - rules. Its equality check was in any case fully subsumed by - test_every_span_bearing_rule_matches_the_script_ranges, since the - canonical rule is itself span-bearing. Splitting the two guarantees - is the point -- the sweep owns "every hand copy equals the table", - this test owns "the table did not change shape without a decision" - -- so a selector break can no longer take the decision gate out as - collateral. Rule authors are correspondingly free to put #271 or - #272 in a compound slug. - """ - assert set(_policy._SCRIPT_RANGES) == { - Script.HAN, Script.HANGUL, Script.HIRAGANA, Script.KATAKANA}, ( - "a Script joined _SCRIPT_RANGES: decide whether the differential " - "rules in tools/differential/expected_since_*.toml should cover " - "it, then update this assertion") - for xlo, xhi in _SANCTIONED_EXTRAS: - assert not any(lo <= xhi and xlo <= hi - for spans in _policy._SCRIPT_RANGES.values() - for lo, hi in spans), ( - f"U+{xlo:04X}-U+{xhi:04X} is classified now; drop it " - "from _SANCTIONED_EXTRAS") - - -#: Which rules each ledger is known to carry a script-span copy in, -#: named by the leading fix(...)/feat(...) tag of their `issue`. -#: -#: Declared rather than counted. A count is identity-free, so one copy -#: could leave discovery -- rewritten as literal characters, say -- while -#: an unrelated span-bearing rule was added, and the total would hold -#: steady while a hand copy went unpinned (measured). Naming them also -#: buys the staleness direction the count never had, and that this -#: module's other two rosters already have: a tag here that matches no -#: rule fails, so a renamed or deleted rule cannot leave its entry -#: behind. -#: -#: Membership is the forcing function: a new baseline's ledger fails as -#: unrecorded until someone writes its rules down. An empty set is fine -#: and correct for a release that changed nothing CJK. -_SPAN_BEARING_RULES: dict[str, frozenset[str]] = { - "expected_since_1.4.0.toml": frozenset({ - "fix(#271/#272/#298)", # the canonical class - "fix(cjk-delimited-nickname)", # the three compounds, whose - "fix(cjk-fullwidth-paren-nickname)", # lookaheads each carry - "fix(cjk-comma-compound)", # their own copy - }), - "expected_since_2.0.0.toml": frozenset({ - "fix(#271/#272/#298)", # the canonical class - "fix(#298)", # the 间隔号 lookahead - }), - "expected_since_2.1.0.toml": frozenset(), # open cycle, no rules yet -} - -#: The leading `fix(...)`/`feat(...)` tag of a rule's `issue`, which is -#: what _SPAN_BEARING_RULES names rules by. Unique within each ledger -#: among span-bearing rules (asserted below), and stable across the -#: prose that follows it. -_ISSUE_TAG = re.compile(r"^[a-z]+\([^)]*\)") - - -def _tag(issue: str) -> str: - """The leading tag, required only of rules the roster has to name. - - Nothing obliges a ledger rule to carry a tag in general, and one in - the 1.4 file does not ("ambiguous-surname-acronym data change: - ..."). That is fine while it declares no script span. If such a rule - ever gains one it lands here, and the fix is to give it a tag rather - than to loosen the roster. - """ - match = _ISSUE_TAG.match(issue) - assert match, ( - f"a span-bearing rule's issue must open with a fix(...) or " - f"feat(...) tag so _SPAN_BEARING_RULES can name it; this one " - f"does not: {issue!r}") - return match.group(0) - - -@pytest.mark.parametrize("ledger", _LEDGERS, ids=lambda p: p.name) -def test_every_span_bearing_rule_matches_the_script_ranges( - ledger: Path) -> None: - """Auto-discovered pin for every hand copy of the script spans in - every ledger: any rule whose character class declares spans - intersecting _SCRIPT_RANGES must declare the whole expected class - (table BMP spans + sanctioned extras). - - A TOML file cannot import _policy._SCRIPT_RANGES, so these are the - copies with no possible alternative -- and the ones whose divergence - is quietest, because the harness is run by hand rather than in CI. - - Both failure directions matter, which is why this compares sets - rather than checking coverage. A span MISSING from a class turns an - intended change into an UNEXPLAINED diff (a release blocker for the - wrong reason); a span that should not be there silently classifies a - real regression as intended, which is the failure the whole harness - exists to prevent. - - The compound rules' require-a-classified-codepoint lookaheads exist - so their trigger sets alone (delimiters; a comma) cannot claim a - Latin name's regression -- and each such lookahead is a copy nothing - else checks. Discovery, not enrollment, is what subjects a rule to - the equality above: a new compound rule's copy is checked because it - exists, not because an author remembered it. _SPAN_BEARING_RULES - then holds discovery itself to account, since a copy rewritten in a - notation discovery cannot see would otherwise just vanish from the - sweep. Rules whose spans touch OTHER scripts (Cyrillic, say) are out - of scope and skipped by the intersection test. - """ - assert ledger.name in _SPAN_BEARING_RULES, ( - f"{ledger.name} is a new ledger whose span-bearing rules are not " - f"recorded; add it to _SPAN_BEARING_RULES (an empty set is a " - f"legal answer for a release that changed nothing CJK)") - table_spans = _expected_bmp_spans() - tags = [] - for rule in _rules(ledger): - regex = rule.get("name_regex") - if not isinstance(regex, str): - continue - declared = _declared_spans(regex) - if not declared & table_spans: - continue - tags.append(_tag(rule["issue"])) - assert declared == table_spans, ( - f"{ledger.name}: {rule['issue']!r} declares " - f"{sorted(declared)}; expected {sorted(table_spans)}") - extra = _unrecognized_class_content(regex) - assert not extra, ( - f"{ledger.name}: {rule['issue']!r} has a span-declaring " - f"character class holding {extra!r} besides its spans. The " - f"span equality above cannot see that, so it would widen the " - f"rule unchecked -- write it as an escaped span or not at all") - assert not _top_level_alternation(regex), ( - f"{ledger.name}: {rule['issue']!r} has a '|' at depth 0, so " - f"the whole rule is an alternation and the pinned class " - f"governs only one branch. Wrap it in '(?:...)'") - # The property the syntactic check above is only a proxy for. - # A rule scoped to classified scripts must not reach a name - # written in none of them -- and unlike a depth test, this does - # not care how the widening is spelled. "(?:CJK|[A-Za-z])" - # hides the pipe at depth 1 where the check above stops - # looking, and claims 644 of the 654 unclassified corpus names; - # this sees - # it. Both are kept: the depth test gives the clearer message - # for the naive spelling, and catches a widening toward a - # script the corpora happen not to contain. - unclassified = _UNCLASSIFIED_NAMES.intersection(_claimed(regex)) - assert not unclassified, ( - f"{ledger.name}: {rule['issue']!r} declares the script table's " - f"spans but claims {len(unclassified)} corpus names carrying " - f"no classified codepoint at all, e.g. {sorted(unclassified)[:3]}. A rule scoped " - f"to these scripts cannot explain a diff on those names, so it " - f"would absorb one instead") - # Every rule, not just the discovered ones: this is what stops a - # class being respelled out of discovery in the first place, and it - # has to reach the rules that are NOT currently span-bearing to do - # that job. - for rule in _rules(ledger): - regex = rule.get("name_regex") - if not isinstance(regex, str): - continue - literal = _literally_spelled_script_chars(regex) - assert not literal, ( - f"{ledger.name}: {rule['issue']!r} spells the classified " - f"codepoints {literal} literally inside a character class. " - f"Write them as \\uXXXX-\\uXXXX spans, or the rule drops out " - f"of the sweep above while meaning the same thing") - # two rules sharing a tag would collapse into one set member and - # read as a disappearance below, which is a confusing way to learn - # that the naming scheme broke - assert len(tags) == len(set(tags)), ( - f"{ledger.name}: two span-bearing rules share an issue tag " - f"({sorted(tags)}); _SPAN_BEARING_RULES cannot name them apart") - found = set(tags) - assert found == _SPAN_BEARING_RULES[ledger.name], ( - f"{ledger.name}'s span-bearing rules are not the recorded set. " - f"Left discovery (a hand copy is now unpinned): " - f"{sorted(_SPAN_BEARING_RULES[ledger.name] - found)}. " - f"Newly discovered (pinned now, but record it): " - f"{sorted(found - _SPAN_BEARING_RULES[ledger.name])}") - - -#: The delimiter compound's trigger set is its own decision surface, -#: separate from the script spans: these are the characters whose mere -#: presence lets the rule claim a diff. Written once here rather than -#: inline: the check this replaces tested the regex against this set -#: spelled as unicode escapes OR against it spelled as the characters -#: themselves, which in a non-raw literal are the same str -- an `or` -#: whose two operands could never disagree. One spelling, named once. -_NICKNAME_DELIMITERS = "[「」『』・･]" - - -def test_nickname_delimiter_sets_are_deliberate() -> None: - """Swept rather than pinned to one file and one rule. The 1.4 ledger - has exactly one cjk-delimited-nickname rule and the 2.0 ledger has - none, so a per-file `== 1` is already false and a global one holds - only by accident of there being a single rule today. What is - actually invariant is that EVERY such rule carries the sanctioned - trigger set, and that at least one exists somewhere or this is - checking nothing. - - Note the scope this does NOT have: it is a decision surface, not a - sync pin. _NICKNAME_DELIMITERS is a literal because the rule's class - is not Policy.nickname_delimiters and is not meant to be -- it is - the two CJK corner-bracket pairs, plus the nakaguro separators, - which delimit nothing, and minus the nine other pairs the config - ships. Deriving it would mean deciding all of that in code rather - than writing it down. - - So a delimiter pair removed from the config does not fail here; it - fails tests/v2/test_cases.py, on the cjk_white_corner_bracket_ - nickname row (measured -- tests/v2/pipeline/ stays green, which is - why the pointer is worth being exact about).""" - found = [] - for ledger in _LEDGERS: - for rule in _rules(ledger): - if "cjk-delimited-nickname" not in rule["issue"]: - continue - found.append(f"{ledger.name}: {rule['issue']}") - # .get rather than [] on purpose: a rule that dropped its - # name_regex outright should land on the assertion below, - # not raise KeyError out of the sweep - assert _NICKNAME_DELIMITERS in rule.get("name_regex", ""), ( - f"{ledger.name}: the compound rule's delimiter set " - f"changed, or the rule lost its name_regex; decide " - f"deliberately, then update _NICKNAME_DELIMITERS") - assert found, ( - "no cjk-delimited-nickname rule in any ledger; this check is " - "passing vacuously") - - -def test_cjk_corpus_matches_the_case_table() -> None: - """corpus_cjk.jsonl is GENERATED, not curated (#295): every - distinct case-table text bearing a codepoint the script table - classifies, sorted -- see build_cjk_corpus.py for why the other - two corpora cannot carry these names. The checked-in file must - equal what the generator would write, so a CJK case row added - without regenerating fails HERE instead of silently narrowing - the differential gate back toward the blind spot #295 closed. - Same promise as the toml pin above, aimed at a generated artifact - instead of a hand copy. - """ - tools = Path(__file__).parents[2] / "tools" / "differential" - spec = importlib.util.spec_from_file_location( - "build_cjk_corpus", tools / "build_cjk_corpus.py") - assert spec is not None and spec.loader is not None - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - checked_in = [json.loads(line) for line in - (tools / "corpus_cjk.jsonl") - .read_text(encoding="utf-8").splitlines()] - assert checked_in == module.selected_names(), ( - "corpus_cjk.jsonl is stale: regenerate with " - "`uv run python tools/differential/build_cjk_corpus.py`") - - -#: Which vocabulary constant each ledger rule's alternation is a hand -#: copy of. A roster rather than an inference: GLUED_HONORIFICS is a -#: SUBSET of SUFFIX_NOT_ACRONYMS (asserted at the bottom of -#: nameparser/config/suffixes.py), so "equals one of the two known sets" -#: would let a spaced rule that silently narrowed to exactly the glued -#: set pass by matching the other member -- a subset check wearing a -#: disguise, and the subset direction is precisely the one that removal -#: drift travels in. -#: -#: Keys are matched as substrings of a rule's `issue` and must select -#: exactly one entry. The full issue lists are the keys, not a bare -#: '#308': both 2.0 rules cite #308 while copying different constants. -_HONORIFIC_SOURCES: dict[str, set[str]] = { - "cjk-honorific-suffix": SUFFIX_NOT_ACRONYMS, # 1.4 - "#307/#308/#320": SUFFIX_NOT_ACRONYMS, # 2.0, spaced - "#308/#312/#319/#320": GLUED_HONORIFICS, # 2.0, glued -} - -#: An alternation group with two or more members and no nested -#: parentheses, capturing or not. Together with the classified-member -#: filter in _cjk_alternations it selects exactly the three honorific -#: alternations across both ledgers; on its own it also matches -#: "(?:^| )" from the 1.4 rule and the two Latin maiden/acronym -#: alternations, all of which the filter drops. -#: -#: The "(?:" alternative is spelled out and the plain "(" is guarded by -#: (?!\?) so that lookarounds -- "(?=$|[ ,])", "(?<=[^\s,])" -- do not -#: parse as alternations of their own syntax. Matching plain capturing -#: groups matters: a rule written "(씨|님|先生)" instead of -#: "(?:씨|님|先生)" would otherwise carry a hand copy this pin cannot -#: see, which is the silent-unpinning this module exists to prevent. -#: -#: Shapes it still cannot read correctly. All of them fail, but by two -#: different mechanisms, and the message you get depends on which -#: (every row measured): -#: -#: STALE half of the roster check -- the group stops matching, so the -#: rule's key goes unused: -#: a nested group whose inner members are UNclassified; -#: a paren inside a character class; -#: a single-member "group", which is not an alternation at all; -#: the named and inline-flag forms "(?Pa|b)" and "(?i:a|b)". -#: -#: Declared-vs-expected equality -- the group still matches, but its -#: members are MISread, since the split is a plain str.split("|"): -#: "|" inside a character class: "[a|b]" yields "[a" and "b]"; -#: a nested group whose inner members ARE classified: the inner -#: group is extracted and shows up as a member of the outer one. -_ALTERNATION = re.compile(r"\((?:\?:|(?!\?))((?:[^()|]+\|)+[^()|]+)\)") - - -def _alternations(name_regex: str) -> list[set[str]]: - """Every alternation group's members.""" - return [set(body.split("|")) for body in _ALTERNATION.findall(name_regex)] - - -def _cjk_alternations(name_regex: str) -> list[set[str]]: - """Every alternation in a rule with a script-classified member.""" - has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) - return [members for members in _alternations(name_regex) - if any(has_classified(m) for m in members)] - - -def test_differential_honorific_rules_match_their_vocabulary() -> None: - """The honorific rules' alternations are hand copies of the CJK - entries of SUFFIX_NOT_ACRONYMS (#307) and of GLUED_HONORIFICS - (#308) -- a toml cannot import them. Each expected set is DERIVED - from the config by script membership (a classified codepoint - anywhere in the entry), so adding a CJK honorific without widening - the rule, or widening a rule with something the vocabulary does not - ship, fails here. - - Swept over every ledger and every alternation, because the three - copies are anchored three different ways -- a leading '(?:^| )' in - 1.4, a character class and a lookbehind in 2.0 -- so the old - startswith/endswith parser could not read two of them. Discovery - plus a declared source is what replaces it. - - The span-bearing pins skip these rules on purpose: their trigger is - the alternation, not a character class. Note GLUED_HONORIFICS had no - pinned copy anywhere in the tree before this. - - Two completeness directions, both required. An alternation matching - no roster key fails as UNDECLARED, so a new rule's copy is pinned by - existing rather than by an author remembering to enroll it. A roster - key matching no rule fails as STALE, catching an entry left behind - after a rule was renamed or deleted. - """ - has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) - used: set[str] = set() - found = 0 - for ledger in _LEDGERS: - for rule in _rules(ledger): - regex = rule.get("name_regex") - if not isinstance(regex, str): - continue - for declared in _cjk_alternations(regex): - keys = [k for k in _HONORIFIC_SOURCES if k in rule["issue"]] - assert len(keys) == 1, ( - f"{ledger.name}: rule {rule['issue']!r} carries a CJK " - f"alternation matching {len(keys)} roster keys " - f"({keys}); every such hand copy must name exactly " - f"one source in _HONORIFIC_SOURCES") - used.add(keys[0]) - found += 1 - expected = {entry for entry in _HONORIFIC_SOURCES[keys[0]] - if has_classified(entry)} - assert declared == expected, ( - f"{ledger.name}: {rule['issue']!r} declares " - f"{sorted(declared)}; the config's CJK entries for " - f"{keys[0]!r} are {sorted(expected)}") - assert found, ( - "no CJK honorific alternation found in any ledger; this pin is " - "passing vacuously") - assert used == set(_HONORIFIC_SOURCES), ( - f"_HONORIFIC_SOURCES keys matching no rule: " - f"{sorted(set(_HONORIFIC_SOURCES) - used)}. Either a renamed or " - f"deleted rule left its entry behind -- drop it -- or that " - f"rule's alternation stopped matching _ALTERNATION (a nested " - f"group, a paren inside a character class, or a lone member), " - f"which is how a still-present hand copy silently leaves this " - f"pin.") - - -class _LatinCopy(NamedTuple): - """A ledger alternation that hand-copies a Latin-script vocabulary. - - Two set-shaped fields that are emphatically not the same kind of - thing, which is why they are named rather than positional: - `vocabulary` is the source of truth, `covers` an audited snapshot of - which of its entries the rule's members reach. - """ - vocabulary: set[str] - covers: frozenset[str] - - -#: Ledger rules whose alternation hand-copies a LATIN vocabulary, keyed -#: by a substring of the rule's `issue`. Kept apart from -#: _HONORIFIC_SOURCES because the relationship is not set equality: -#: these members are regex FRAGMENTS, not entries -- "n[ée]e" covers two -#: markers at once, "geb\.?" and "roz\.?" one each -- so there is no set -#: to compare against. -#: -#: `covers` is recorded rather than equated to the whole vocabulary. -#: Equality would force a rule to grow alternatives for markers it has -#: no reason to claim -- see the note on fix(#274) in the 1.4 ledger for -#: why 旧姓 in particular is not one of them. Recording still catches -#: removal: drop an entry a member covers and the snapshot shrinks. -#: -#: Three nearby counts differ and are easy to conflate, all for -#: fix(#274) specifically: MAIDEN_MARKERS ships 17 entries; that rule's -#: members reach 4 of them; the corpora contain 3 markers in total -#: (geb, née, 旧姓), only 2 of which it covers. -_LATIN_ALTERNATION_SOURCES: dict[str, _LatinCopy] = { - "fix(#274)": _LatinCopy( - vocabulary=MAIDEN_MARKERS, - covers=frozenset({"geb", "nee", "née", "roz"})), - "ambiguous-surname-acronym": _LatinCopy( - vocabulary=SUFFIX_ACRONYMS_AMBIGUOUS, - covers=frozenset({"do", "ma"})), -} - -#: Alternations that copy no vocabulary, so discovery must not demand a -#: source for them. Declared rather than inferred, on the principle -#: _SOURCES' None entries already set: an undeclared alternation is a -#: question someone answers in writing, not something to skip past. -_NOT_A_VOCABULARY_COPY = frozenset({ - frozenset({"^", " "}), # the honorific rule's leading anchor -}) - -def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: - """Corpus names the whole rule claims that none of its own members - reach. - - The member checks bound what each ALTERNATIVE matches. This bounds - the rule built around them, which is a different question and the - one four rounds of syntactic guards kept failing to ask. Nesting a - rule's own alternation one level down and adding a branch -- - "(?:[(\"'](m\\.?a\\.?|d\\.?o\\.?)[)\"']|[A-Za-z]{4,})" -- leaves every - member innocent, hides the pipe below the depth test, and hides the - outer group from _ALTERNATION, which refuses nested parens. It - claimed 622 of the corpus while every check passed. - - Keyed on the roster rather than on `fields`, which is what makes it - reach that rule: the acronym copy claims `suffix`/`nickname`, so - the field-keyed guard below never looks at it. - - A rule claiming nothing scores zero, which is the strongest answer - rather than a vacuous one -- so the non-vacuity assertion is over - the union of all rostered rules, not per rule. - """ - reachable = [re.compile(member, re.IGNORECASE) for member in members] - return [name for name in _claimed(name_regex) - if not any(pattern.search(name) for pattern in reachable)] - - -def _reaches_non_vocabulary(member: str, vocabulary: set[str]) -> list[str]: - """Corpus text this member matches that is NOT a vocabulary entry. - - fullmatch against the vocabulary bounds what a member matches - WITHIN those entries and says nothing about what it matches in a - NAME. That gap was first filled with a tuple of eight hand-picked - probe strings, and the tuple was defeated by a wider rule than the - one it was added to stop: every entry this rule needs is three - characters, so a member must accept some 3-character string and is - unconstrained everywhere else -- "[acdf-uw-z]{3,}" covers `roz`, - dodges all eight probes, and reaches 634 of the 751 corpus names - as a fourth alternative (the rule carrying it claims 542). - Measured with the IGNORECASE this function applies; the - case-sensitive figure, 592, is not what runs. Counts here and below are - against _CORPUS_NAMES, which deduplicates the 783 corpus lines. - - Eight strings could never be more than a spot check. The corpus is - the whole population the rule will ever be asked about, so ask it - instead: every fragment a member matches must BE an entry. - - Searched unanchored, which is stricter than the rule's own \\b - anchoring and so cannot produce a false negative from it. - Normalized before comparison because the corpus writes "geb." where - the config stores "geb" -- without that, the shipped member fails. - Exempting only THIS vocabulary, not the union of all of them, keeps - a maiden member from being excused by an acronym entry. - """ - matches = set() - for name in _CORPUS_NAMES: - for found in re.finditer(member, name, re.IGNORECASE): - if found.group() and _normalize(found.group()) not in vocabulary: - matches.add(found.group()) - return sorted(matches) - - -def test_latin_alternations_mean_something_the_vocabulary_ships() -> None: - """The Latin twin of the honorific pin, shaped by what a regex - alternation over a vocabulary can honestly promise. - - Discovery-first, like its twin: every alternation in every ledger - must be a declared vocabulary copy or a declared non-copy. Keying - off the roster and skipping everything else would mean a future rule - that hand-copies a vocabulary under a new tag is pinned only if its - author remembers to enroll it -- the failure this module exists to - prevent, not a shape it should adopt. - - Members are matched against entries as regexes rather than compared - as strings, because the members ARE regex syntax: "geb\\.?" stands - for the entry "geb", which the config stores normalized (lowercase, - no trailing period). - - Two ways a MEMBER can be wrong, checked per member: it can match - nothing in the vocabulary -- then it cannot describe a real change - and can only claim other names' diffs, which is how "born" survived - from the harness's first commit to #350 -- or it can reach corpus - text that is not vocabulary. A third assertion is about the RULE - around them, which can widen at depth 0 and leave the pinned - alternation governing one branch of an unchecked whole. An invalid - member raises rather than asserts, since nothing else can proceed. - """ - has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) - used: set[str] = set() - found = 0 - reach_checked = 0 - for ledger in _LEDGERS: - for rule in _rules(ledger): - regex = rule.get("name_regex") - if not isinstance(regex, str): - continue - for members in _alternations(regex): - if any(has_classified(m) for m in members): - continue # the honorific pin owns these - if frozenset(members) in _NOT_A_VOCABULARY_COPY: - continue - found += 1 - keys = [k for k in _LATIN_ALTERNATION_SOURCES - if k in rule["issue"]] - assert len(keys) == 1, ( - f"{ledger.name}: {rule['issue']!r} carries the Latin " - f"alternation {sorted(members)}, which matches " - f"{len(keys)} roster keys ({keys}). Declare the " - f"vocabulary it copies in _LATIN_ALTERNATION_SOURCES, " - f"or add it to _NOT_A_VOCABULARY_COPY if it copies " - f"nothing") - used.add(keys[0]) - source = _LATIN_ALTERNATION_SOURCES[keys[0]] - unjustified = _unjustified_reach(regex, members) - assert not unjustified, ( - f"{ledger.name}: {rule['issue']!r} claims " - f"{len(unjustified)} corpus names that none of its own " - f"vocabulary members reach, e.g. {unjustified[:3]}. The " - f"members are innocent and the rule around them is not " - f"-- whatever it matches beyond them, it claims") - reach_checked += len(_claimed(regex)) - assert not _top_level_alternation(regex), ( - f"{ledger.name}: {rule['issue']!r} has a '|' at depth " - f"0, so the pinned alternation governs only one branch " - f"and the rest of the rule is unchecked. Wrap it in " - f"'(?:...)'") - covered = set() - for member in members: - try: - pattern = re.compile(member, re.IGNORECASE) - except re.error as exc: - raise AssertionError( - f"{ledger.name}: {rule['issue']!r} member " - f"{member!r} is not a valid regex ({exc}). A " - f"mis-split alternation can produce this -- see " - f"_ALTERNATION's notes") from None - matched = {entry for entry in source.vocabulary - if pattern.fullmatch(entry)} - loose = _reaches_non_vocabulary(member, source.vocabulary) - assert not loose, ( - f"{ledger.name}: {rule['issue']!r} member " - f"{member!r} matches corpus text that is not " - f"vocabulary: {loose[:6]}. Spell it literally -- a " - f"member broad enough to reach a real name lets the " - f"rule claim that name's diff as intended") - assert matched, ( - f"{ledger.name}: {rule['issue']!r} offers the " - f"alternative {member!r}, which matches no entry in " - f"the vocabulary it copies. It cannot correspond to " - f"a real change, so it can only claim other names' " - f"diffs as intended -- drop it, or ship it as " - f"vocabulary first") - covered |= matched - assert covered == source.covers, ( - f"{ledger.name}: {rule['issue']!r} covers " - f"{sorted(covered)}; recorded {sorted(source.covers)}. " - f"Lost: {sorted(source.covers - covered)} (an entry the " - f"rule relied on left the vocabulary). Gained: " - f"{sorted(covered - source.covers)} (record it)") - assert found, ( - "no Latin vocabulary alternation found in any ledger; this pin is " - "passing vacuously") - assert reach_checked, ( - "no rostered rule claims any corpus name, so the reach check above " - "measured nothing -- verify the corpora loaded") - assert used == set(_LATIN_ALTERNATION_SOURCES), ( - f"_LATIN_ALTERNATION_SOURCES keys matching no rule: " - f"{sorted(set(_LATIN_ALTERNATION_SOURCES) - used)}") - - -#: A role a rule can claim, mapped to the vocabulary a name must carry -#: for that claim to be possible. Keyed on `fields` rather than on the -#: regex, which is the point: the two pins above discover hand copies -#: by their SYNTAX, so a copy that is not written as an alternation or -#: a character class is not undeclared, it is unseen. A brand-new rule -#: whose name_regex is "(?i)\b[a-z]{3}\b" with maiden fields passed -#: every check in this module while claiming 308 corpus names. -#: -#: `fields` cannot be dodged the same way. A rule that does not claim -#: `maiden` cannot be labelled a maiden change at all, so keying here -#: reaches every spelling -- including fix(cjk-maiden-marker), whose -#: regex is the bare literal "旧姓" and which no roster in this module -#: could see. -#: -#: Only `maiden` today, because only its vocabulary is small and -#: mandatory enough for the implication to hold: a maiden diff needs a -#: marker in the name. `suffix` and `title` are not like that -- most -#: of their diffs come from routing, not from a vocabulary word being -#: present -- so adding them would be false rather than strict. -_FIELD_VOCABULARIES: dict[str, set[str]] = { - "maiden": MAIDEN_MARKERS, -} - - -def _carries(name: str, vocabulary: set[str]) -> bool: - """Whether a name contains a vocabulary entry. - - Whole-token for ASCII entries, substring for the rest, because a - marker like 旧姓 is written against the name it marks rather than - spaced off it. - - Note what the isascii() split actually covers: 12 of the 17 - entries, not only the CJK one. `né` is two characters, so the - substring branch reads `René` as carrying a marker. Every - over-match here SHRINKS the set of unexplained names and so - weakens the guard -- the direction this module exists to close -- - but exactly one corpus name reaches that branch today, and it is - the 旧姓 one. Tighten this before admitting a vocabulary whose - short non-ASCII entries occur inside ordinary names. - """ - tokens = {_normalize(token) for token in name.split()} - return bool(tokens & vocabulary) or any( - entry in name for entry in vocabulary if not entry.isascii()) - - -def test_rules_claiming_a_vocabulary_role_need_the_vocabulary_present() -> None: - """The guard that does not care how a rule is spelled. - - Every pin above starts from regex syntax -- an alternation, a - character class, a span. Each closed the hole it was built for and - left the next spelling open, four rounds running. This one starts - from what the rule CLAIMS: if a rule says a diff is a maiden-marker - change, then every corpus name it claims must actually carry a - maiden marker. A rule cannot escape that by changing notation, - because it is not reading the notation. - - Deliberately narrow. It does not say the rule is correct, only that - it cannot be explaining a marker on a name that has none. That - covers the maiden widenings review found; the ones claiming - `nickname` and `suffix` fall outside it, and _CORPUS_CLAIMS is - what catches those. - """ - checked = 0 - for ledger in _LEDGERS: - for rule in _rules(ledger): - regex = rule.get("name_regex") - if not isinstance(regex, str): - continue - for field, vocabulary in _FIELD_VOCABULARIES.items(): - if field not in (rule.get("fields") or []): - continue - checked += 1 - bare = [name for name in _claimed(regex) - if not _carries(name, vocabulary)] - assert not bare, ( - f"{ledger.name}: {rule['issue']!r} claims {field!r} " - f"diffs on {len(bare)} corpus names carrying no {field} " - f"vocabulary, e.g. {bare[:3]}. It cannot be explaining a " - f"marker that is not there, so on those names it would " - f"absorb a regression instead") - assert checked, ( - "no rule claims a role in _FIELD_VOCABULARIES; this pin is passing " - "vacuously") - - -class _Claim(NamedTuple): - """What a rule claims, in the three dimensions classify() uses. - - A count alone is identity-free -- the weakness this module rejects - at _SPAN_BEARING_RULES -- and review proved it here twice. Swapping - feat(#273)'s delimiter class for a single accented letter holds the - count at 6 while claiming six entirely different names. And - classify() narrows on `fields` as well as `name_regex`, so widening - a rule's roles moves nothing a regex-only count can see: the - comma rule kept its `,` regex and its 236 names while going from - explaining 6 of the corpus to 242. - """ - #: corpus names the name_regex reaches; the whole corpus when a - #: rule has none, which is the most unbounded shape validate_rules - #: permits and the one most worth writing down - names: int - #: the roles it narrows by, sorted; () when it narrows by regex alone - roles: tuple[str, ...] - #: sha256[:12] of the claimed names, so a swap that holds the count - #: still fails. Unreadable by design -- the count above is what a - #: reviewer reads, and the failure message prints what moved. - digest: str - - -def _claim(rule: dict) -> _Claim: - regex = rule.get("name_regex") - names = (_claimed(regex) if isinstance(regex, str) else list(_CORPUS_NAMES)) - fields = rule.get("fields") - return _Claim( - names=len(names), - roles=tuple(sorted(fields)) if isinstance(fields, list) else (), - digest=hashlib.sha256( - "\n".join(names).encode("utf-8")).hexdigest()[:12]) - - -#: How many corpus names each rule's name_regex matches, per ledger. -#: -#: The backstop the other guards each failed to be. Every one of them -#: is scoped to a CATEGORY -- a span class, an alternation's members, a -#: `maiden` field, a member's reach -- and five review rounds each found -#: a rule outside whichever category the last fix had covered. The -#: categories are the test's, not the ledger's. What every rule shares -#: is what it claims: how much corpus its regex reaches, which roles -#: it narrows by, and WHICH names those are. Scoped to the corpus -- -#: and only there -- a widening cannot change what a rule explains -#: without moving one of the three. Restoring 'born' moves none of -#: them, because no corpus name contains it; the member guards catch -#: that, which is why this does not replace them. -#: -#: So this is deliberately dumb: it knows nothing about vocabularies, -#: scripts or roles, and it cannot say whether a number is RIGHT. It -#: says only that it moved, which is the question a widening cannot -#: dodge. The specific guards above stay because they explain WHY a -#: rule may claim what it claims, and their messages are the ones worth -#: reading; this one just refuses to let the number change quietly. -#: -#: Keyed by the full `issue` rather than by tag: the 1.4 ledger has two -#: rules tagged feat(#269), and a tag-keyed roster cannot tell them -#: apart -- the same identity-free weakness recorded at -#: _SPAN_BEARING_RULES. -#: -#: These numbers move when the CORPORA move, not only when a rule does. -#: That is the intended cost: a corpus name added under an existing -#: rule is a real change in what that rule explains, and it should be -#: read once rather than absorbed silently. -_CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { - "expected_since_1.4.0.toml": { - "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('family', 'given', 'middle'), "66e71d60a075"), - "fix(#274) maiden markers consumed": - _Claim(4, ('family', 'maiden', 'middle'), "b31dc2e2bbc4"), - "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), - "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(236, ('given', 'suffix', 'title'), "3416f69d0ce4"), - "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(751, ('family', 'given', 'suffix'), "231640fc7535"), - "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": - _Claim(0, ('suffix',), "e3b0c44298fc"), - "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": - _Claim(0, ('nickname', 'suffix'), "e3b0c44298fc"), - "feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)": - _Claim(2, ('family', 'middle'), "3e2b5c6d1f4d"), - "feat(#273) typographic nickname delimiters recognized by default": - _Claim(6, ('middle', 'nickname'), "a03c9763c8c4"), - "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJK order flip": - _Claim(6, ('family', 'given', 'nickname'), "ae1dffa01608"), - "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": - _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7"), - "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": - _Claim(20, ('family', 'given', 'middle', 'suffix', 'title'), "b2ea8fa59eea"), - "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": - _Claim(14, ('family', 'given', 'middle', 'suffix'), "d49ce901bdce"), - "feat(#269) non-Latin titles/conjunctions recognized": - _Claim(2, ('given', 'middle', 'title'), "c14187bb08f8"), - "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": - _Claim(1, ('given', 'middle', 'suffix', 'title'), "390e7f814d13"), - }, - "expected_since_2.0.0.toml": { - "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('_ambiguities', 'family', 'given', 'middle'), "66e71d60a075"), - "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": - _Claim(34, ('family', 'given', 'suffix'), "877ab3246d33"), - "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": - _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2"), - "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), - "fix(#272) nakaguro inside delimited content renders as a space, compounding with the CJK order flip": - _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), - "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": - _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), - }, - "expected_since_2.1.0.toml": {}, -} - - -def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: - """The guard that knows nothing, and therefore cannot be dodged. - - Five rounds of review defeated five guards, each by moving to a - rule the last fix did not cover: a span rule, then an alternation's - members, then a non-maiden role, then a rule whose narrowing lived - outside its alternation entirely. Every one of those attacks - changed what the rule claimed -- 6 to 668 names, 0 to 193, and the - comma rule from 3 roles to 6 while its regex and its 236 names - stood still -- because that is what widening a rule means WITHIN - the corpus. A widening reaching only names the corpora lack moves - nothing here; the guards above are what see those. - - Note which layer is which, because it is the opposite of what it - looks like. This is a change DETECTOR, not an enforcer: it is - inert for a brand-new rule, whose author simply records whatever - number it produces, and its own failure message invites the - remedy that defeats it -- re-record and the attack lands. The - member and vocabulary guards above are the walls, because they - judge a rule wrong at any time INCLUDING at recording time. This - catches the widenings none of them is scoped to see, and holds - them still long enough for someone to look. - """ - for ledger in _LEDGERS: - assert ledger.name in _CORPUS_CLAIMS, ( - f"{ledger.name} is a new ledger with no recorded corpus claims; " - f"add it to _CORPUS_CLAIMS (an empty mapping if it has no rules)") - recorded = _CORPUS_CLAIMS[ledger.name] - # Keyed on `issue`, so a duplicate silently collapses to the - # last rule written and the other goes unmeasured -- coverage - # by file order, which is no coverage. Nothing else asserts - # this: validate_rules only requires a non-empty string, and - # the tag-uniqueness check above covers span-bearing rules - # alone. _LATIN_ALTERNATION_SOURCES and _HONORIFIC_SOURCES key - # on issue SUBSTRINGS, so they lean on it too. - issues = [rule["issue"] for rule in _rules(ledger)] - assert len(set(issues)) == len(issues), ( - f"{ledger.name} has rules sharing an `issue`: " - f"{sorted({i for i in issues if issues.count(i) > 1})}. Every " - f"roster here keys on it, so one of them would go unmeasured") - # A rule with no name_regex narrows by `fields` alone and so - # reaches EVERY name -- the most unbounded shape validate_rules - # permits, and the one most worth recording. Counting it as the - # whole corpus is not a placeholder; it is what it claims. - actual = {rule["issue"]: _claim(rule) - for rule in _rules(ledger)} - moved = {issue: (recorded.get(issue), count) - for issue, count in actual.items() - if recorded.get(issue) != count} - assert actual == recorded, ( - f"{ledger.name}: the corpus each rule claims is not what is " - f"recorded. Moved (recorded, now): {moved}. Gone: " - f"{sorted(set(recorded) - set(actual))}. New: " - f"{sorted(set(actual) - set(recorded))}. A number that GREW " - f"means the rule claims more of the corpus than it did -- " - f"check it is not absorbing a regression, then record it") - assert set(_CORPUS_CLAIMS) == {led.name for led in _LEDGERS}, ( - f"_CORPUS_CLAIMS names ledgers that do not exist: " - f"{sorted(set(_CORPUS_CLAIMS) - {L.name for L in _LEDGERS})}") diff --git a/tools/differential/README.md b/tools/differential/README.md index 808cf89c..b502711d 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -192,7 +192,7 @@ explicit, reviewable act that can only add names — and the CJK corpus exists because BOTH of those are structurally blind to unspaced CJK (v1's banks never tested it; `build_issues_corpus.py` requires an internal space, which unspaced names never have). It regenerates from the case -table, and `tests/v2/test_regex_sync.py` pins the checked-in file +table, and `tests/v2/test_ledger_guards.py` pins the checked-in file against the generator's selection, so a CJK case row added without regenerating fails the suite instead of silently narrowing this gate. diff --git a/tools/differential/build_cjk_corpus.py b/tools/differential/build_cjk_corpus.py index 4e0c8f6a..38b2c860 100644 --- a/tools/differential/build_cjk_corpus.py +++ b/tools/differential/build_cjk_corpus.py @@ -23,7 +23,7 @@ uv run python tools/differential/build_cjk_corpus.py -tests/v2/test_regex_sync.py pins the checked-in file against this +tests/v2/test_ledger_guards.py pins the checked-in file against this module's selection, so a stale corpus fails the suite rather than silently narrowing the differential gate. """ diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 36636572..24137ed0 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -27,7 +27,7 @@ issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segme # astral block (U+20000-U+323AF) is deliberately left out of the class: # no name in either corpus reaches it, and a rule should be no wider # than the diffs it has to explain. Extend it -- and the sync pin in -# tests/v2/test_regex_sync.py, which fails if the two disagree -- if +# tests/v2/test_ledger_guards.py, which fails if the two disagree -- if # one ever does. The kana spans are BMP-only for a different reason: # supplementary-plane kana is hentaigana and other archaic forms no # modern name uses, so _SCRIPT_RANGES does not list it either. @@ -64,7 +64,7 @@ fields = ["given", "middle", "family"] [[change]] issue = "fix(#274) maiden markers consumed" # The alternation is a hand copy of MAIDEN_MARKERS, pinned by -# tests/v2/test_regex_sync.py: every alternative must match at least +# tests/v2/test_ledger_guards.py: every alternative must match at least # one entry the config ships, and none may match ordinary name text. # # Its members reach 4 of the 17 entries -- geb, nee, née, roz -- which @@ -270,7 +270,7 @@ issue = "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition com # middle-name word -- a spaced corner-bracket name never does that. # Adding `middle` there would pre-excuse a bare middle regression on # '山田「タロ」太郎'. Its delimiter set is separately pinned in -# tests/v2/test_regex_sync.py as "its own decision surface", so +# tests/v2/test_ledger_guards.py as "its own decision surface", so # widening it is a deliberate act with a test to update, and there is # nothing to gain by making it here. # @@ -325,7 +325,7 @@ issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compoundin # ('Wang Xiaoming 先生') is inside its shadow, accepted because the # recognized honorific is the diff's cause there too, and the # alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries -- -# pinned by tests/v2/test_regex_sync.py, which derives the expected +# pinned by tests/v2/test_ledger_guards.py, which derives the expected # set from the config by script membership. Anchored to a WHOLE # trailing token ((?:^| )...$), judged on the NAME STRING: without # it, any name string merely ENDING in 양/군 -- the glued given name diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 3372eedd..7b167ce4 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -54,7 +54,7 @@ issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segme # expected_since_1.4.0.toml rather than rewritten in literal # characters, so the two ledgers spell the same class the same way. # -# tests/v2/test_regex_sync.py pins this copy, and the honorific +# tests/v2/test_ledger_guards.py pins this copy, and the honorific # alternations below, by sweeping every expected_since_*.toml rather # than naming one (#333). A span removed from the script table, or an # entry removed from SUFFIX_NOT_ACRONYMS or GLUED_HONORIFICS, now diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index cb5f5bf4..817e4184 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -27,7 +27,7 @@ # must carry rules like any other, so a 2.2 that changed no behavior at # all would need that decision made deliberately rather than inherited. # -# tests/v2/test_regex_sync.py's _SPAN_BEARING_RULES already records this +# tests/v2/test_ledger_guards.py's _SPAN_BEARING_RULES already records this # file with an empty set: no rule here hand-copies _SCRIPT_RANGES yet, # and the first one that does has to be recorded there or the sweep # fails.