diff --git a/AGENTS.md b/AGENTS.md index 39a9f545..9c5ec46c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -153,7 +153,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reports given `de Mesnil` and no family at all in the default order, which is the same guard declining on a chained piece; that output is believed WRONG and is tracked as #367, so do not cite it as a line this rule means to draw.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reported given `de Mesnil` and no family at all in the default order, which was the same guard declining on a chained piece; #367 removed the chain rather than touching this rule — a title is now transparent to the leading-particle exception, so the piece is lone again, 1b fires, and the name reads family `de Mesnil` like the untitled form.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` @@ -196,7 +196,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker** — nameparser ships `py.typed`. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Dr. Van Jr.", where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when a title shifts it off index 0 and the prefix chain claims it, so both report; for two years only the first did. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, so both report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates — standing ahead of the chained particle with nothing but titles before it. That word need not LEAD the input: `Dr. Do van Johnson` reaches the emitter with a plain title in front of it, and `Do St Johnson` reaches it with the chained particle itself in both vocabularies. `Freiherr von Richthofen` is the canonical shape rather than the only one; when checking whether that emitter is dead, a both-vocabulary word is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission (see the comma paths in concepts.rst) over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/docs/release_log.rst b/docs/release_log.rst index cdb1da00..45aa83a7 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -10,11 +10,14 @@ Release Log editing one in place as a way to change a default and replaces it with configuring a ``Lexicon`` or a private ``Constants``. - Nothing moved between vocabularies and no parse changes in the - default name order: over the 751 names of the differential - corpora, every one of the seven fields is identical to 2.1 through - both the 2.0 and the 1.x API. Both family-first orders do change, - below -- the same seven names under each. + Nothing moved between vocabularies. The rename itself changes no + parse at all; the parsing changes below are separate fixes, and + only one of them reaches the default name order -- a title no + longer changing how the name behind it is read, which moves one of + the 751 names of the differential corpora. Both family-first + orders change too, below -- the same eight names under each, seven + of them from the family-first fix and the eighth the one name the + title fix moves in every order. What breaks is code that *writes* to a default word list, and code that imports one by its 1.x name has until 3.0. @@ -24,6 +27,8 @@ Release Log **Behavior Changes** + - Fix a title changing how the name behind it is read. A leading particle deliberately does not chain onto the words after it -- that is what makes ``"Van Johnson"`` read as given ``Van``, family ``Johnson`` rather than as one surname -- but the exception was keyed on the first *piece of the input*, so a title displaced the particle out of that position and the chain fired. Identical name text parsed two ways depending on whether a title preceded it: ``"Dr. Van Johnson"`` gave family ``Van Johnson`` with no given name, and ``"Sir Van Johnson"`` gave given ``Van Johnson`` with no *family* name at all, a given-name title then handing the whole chain to ``given``. A title is not part of the name, so it can no longer decide whether the name begins with a particle: the exception is keyed on the first piece of the **name**, and a titled name now reads exactly as the same text reads untitled. What moves is every titled name whose first name-piece is a particle. Every reading in this bullet is the default ``name_order``, which is where the fields differ by order: ``"Dr. Van Johnson"`` and ``"Mr. Van Nguyen"`` go to given ``Van`` plus family ``Johnson``/``Nguyen``, ``"Dr. Van Johnson Smith"`` to given ``Van``, middle ``Johnson``, family ``Smith``, and ``"Sir de Mesnil"`` to family ``de Mesnil``. The same grouping change reaches ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``, where the leading particle takes the family rather than the given: ``"Dr. Van Johnson"`` reads family ``Van``, given ``Johnson`` under both. Three fields carry nearly all of the redistribution -- ``given``, ``middle`` and ``family`` -- but two more move on names inside this class, and only ``suffix`` and ``nickname`` move on none of them. ``title`` GROWS where the word that stops the transparency scan is one the chain used to swallow: ``"Dr. St John Smith"`` was title ``Dr.``, family ``St John Smith`` and is now title ``Dr. St``, given ``John``, family ``Smith``, ``st`` being in both vocabularies and so staying a title piece instead of being chained onto the name. ``maiden`` appears where un-chaining lets a marker standing behind the particle be seen at all: ``"Mr. Van Johnson nee Brown"`` was family ``Van Johnson nee Brown`` with no maiden name and is now given ``Van``, family ``Johnson``, maiden ``Brown``. Untitled names are untouched, and so is every name with no particle behind its title: ``"Mr. Smith"``, ``"Mr. John Smith"``, ``"Sir Ian McKellen"`` and ``"King Henry"`` are byte-identical. So is a name whose leading word is *both* a title and a particle -- ``st``, ``do`` and ``freiherr`` are each in both vocabularies, so ``"St John Smith"``, ``"Do John Smith"`` and ``"Freiherr von Richthofen"`` keep their readings: a word that could be the name's own first piece stops the transparency scan rather than being stepped over. The fork is still reported, and still as ``PARTICLE_OR_GIVEN``; it now comes from the same place the untitled ``"Van Johnson"`` reports it, so the ``detail`` text changes from "was chained onto the following name piece" to "leading 'Van' may be a family-name particle; read as a given name". This fixes a defect v1 shipped as a known-failing test for the life of the 1.x line, with the note "Not sure how to fix this without breaking Mr and Mrs"; it does not break Mr and Mrs, which have no particle to displace. One of the 751 differential corpus names moves, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike -- and it is the same one name under each family-first order too, ``"Mr. Van Nguyen"``. One case regresses and is tracked as `#369 `_: ``"Sheik Abu Bakar"`` was given ``Abu Bakar`` and is now given ``Abu``, family ``Bakar``. It read correctly before only because ``abu`` happens to be a particle as well as a bound given name -- ``"Sheik abdul salam"`` shows that being a bound given name alone does not chain -- so the reading was a side effect of the bug rather than a rule (closes #367) + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index a2c5d1b8..89de00ab 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -334,10 +334,13 @@ class Lexicon: particles: frozenset[str] = frozenset() #: Subset of particles that can also BE a given name ("Van #: Johnson", but also "Van Buren"). Membership decides nothing - #: about chaining: the prefix chain skips index 0 unconditionally + #: about chaining: the prefix chain skips the name's first piece #: and never consults this set, so it leaves a leading particle a #: piece of its own whether listed or not -- "de Mesnil" groups - #: into two pieces exactly as "van Gogh" does. What membership + #: into two pieces exactly as "van Gogh" does, and since #367 the + #: NAME in "Dr. de Mesnil" and "Dr. Van Johnson" groups into those + #: same two pieces behind the title piece, a title not being + #: part of the name it precedes. What membership #: decides is what becomes of that piece afterwards. Under ANY #: ``name_order`` a member records a particle-or-given ambiguity #: and a non-member records none, and a non-member is additionally diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 3975af76..4178fc6a 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -22,8 +22,9 @@ non-suffixy ones COMMA_STRUCTURE). SUFFIX_COMMA: segment 0 as NO_COMMA; segments 1+ wholly SUFFIX. Emits PARTICLE_OR_GIVEN when the leading name piece is a lone -particles_ambiguous token with more pieces following ("Van Johnson") -- -whatever role name_order assigns that position. +particles_ambiguous token with more pieces following ("Van Johnson", +and since #367 "Dr. Van Johnson" too, a title no longer displacing the +particle out of that position) -- whatever role name_order assigns. """ from __future__ import annotations diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index daad03ef..5fc6af63 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -199,9 +199,53 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # prefix chains: a non-leading prefix run absorbs everything to # the next prefix or suffix (v1's leading_first_name rule keeps # the first piece a name: "Van Johnson") + # + # "Leading" means the first piece of the NAME, not of the input + # (#367): a title is not part of the name, so it must not decide + # whether the name begins with a particle. Keyed on index 0, a + # title displaced the particle and the chain fired, so identical + # name text parsed two ways ("Van Johnson" -> given Van, family + # Johnson; "Dr. Van Johnson" -> family "Van Johnson"). + # + # "Title AND NOT prefix" rather than the plain "not a title" the + # rule is stated as, and the difference is not academic: `st`, + # `do` and `freiherr` are each BOTH a title and an ambiguous + # particle, so the plain test skipped over the very piece the + # exception exists to protect and "St John Smith" -- no title in + # front of it at all -- collapsed from title St, given John, + # family Smith into one given "St John Smith". A piece that + # could be the name's own first piece stops the scan; only a + # piece that can ONLY be a title is stepped over. + # + # Computed once, before the loop: every merge below starts at + # some k at or past this index, so no merge can move it. + # + # Suffix pieces are deliberately NOT skipped, and the reason is + # what skipping them WOULD do rather than what it would cost. + # A credential written with spaces already parses without a + # family name -- "Ph. D. Van Johnson" is given 'Van Johnson', + # suffix 'Ph. D.', family '' -- and skipping the suffix piece + # would actually give it one (given 'Van', family 'Johnson'). + # The shapes that decide it are the ones whose leading piece + # lands in `given` instead: "Ph.D. Van Johnson", "II Van + # Johnson" and "Msc.Ed. Van Johnson" each read given + # 'Ph.D.'/'II'/'Msc.Ed.' with family 'Van Johnson', and + # skipping the piece moves `Van` out of the family and into the + # middle name (given 'Ph.D.', middle 'Van', family 'Johnson') + # -- a worse reading, on three shapes, to fix none. "Jr. Van + # Johnson", the shape that looks like it needs the skip, + # classifies its leading piece as a TITLE and is already + # covered here. + # + # The `, 0` fallback is inert by construction rather than a + # default worth testing: it is reached only when every piece is + # a title and none is a prefix, and the loop below merges + # nothing unless some piece is a prefix. + leading = next((k for k in range(len(pieces)) + if not title(k) or prefix(k)), 0) k = 0 while k < len(pieces): - if k == 0 or not prefix(k): + if k == leading or not prefix(k): k += 1 continue j = k + 1 @@ -212,11 +256,41 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # The other half of PARTICLE_OR_GIVEN. _assign reports the # fork when an ambiguous particle stays a lone leading piece # ("Van Johnson" -> given under the default order, family - # under FAMILY_FIRST); the chain here takes the - # opposite branch whenever a title shifts it off index 0 - # ("Dr. Van Johnson" -> family "Van Johnson"). A fork whose - # two sides are decided in different stages needs an emitter - # in each. + # under FAMILY_FIRST); the chain here takes the opposite + # branch when the particle is not the name's leading piece. + # A fork whose two sides are decided in different stages + # needs an emitter in each. + # + # Narrow, and #367 is why. `all(title(x) for x in range(k))` + # says every piece ahead of this one is a title, and the + # loop skipped k == leading, so `leading` is STRICTLY + # before k -- and being before k it is one of those titles, + # while being `leading` it satisfies `not title or prefix`. + # For both, it must be a prefix as well: a word in both + # vocabularies (`st`, `do`, `freiherr` by default, or any + # overlap a caller configures). A plain title alone can no + # longer put a particle off the name's leading piece; it is + # stepped over and _assign reports the fork instead. + # + # What that leaves is wider than one shape: any number of + # plain title pieces, then a piece in BOTH vocabularies, + # then any number of further titles, then the ambiguous + # particle whose chain claims something. "Freiherr von + # Richthofen" is the canonical spelling and the one + # tests/v2/cases.py and tests/v2/test_parser.py lead with, + # but "St Van Johnson", "Do St Johnson" (the chained + # particle itself in both vocabularies) and "Dr. Do van + # Johnson" (a plain title AHEAD of the both-vocabulary + # word) all reach here too. What none of them can do is + # dispense with the both-vocabulary WORD. The conjunction + # merge is the only other way a piece acquires `title` or + # `prefix`, and it cannot manufacture the pair: it derives + # from ONE neighbor, which is the left one whenever there + # is a left one, and its right operands are always fresh + # pieces (the loop runs left to right, so nothing to the + # right has been merged yet). Both tags therefore have to + # come from the piece it extends, which bottoms out at a + # lone token in both vocabularies. # # j > k + 1 is what makes this a DECISION rather than a # shape: when the next piece is a suffix the inner scan diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index f14e540c..78ae161b 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -145,10 +145,13 @@ def post_rules(state: ParseState) -> ParseState: # particle, so 1b declines and given='de la Vega' stands; #359 # records that case as working as intended. And the degenerate bare # 'de' keeps given='de', having nothing to fold into. - # "Sir de Mesnil" is the same guard declining on a chained piece, - # but do NOT cite it as a limit this rule means to draw: it reports - # given='de Mesnil' with no family at all, which is believed wrong - # and is tracked as #367. + # "Sir de Mesnil" used to be this guard declining on a chained + # piece, reporting given='de Mesnil' with no family at all. That + # was never a limit this rule meant to draw, and #367 removed the + # chain rather than touching the rule: a title is transparent to + # the leading-particle exception, so 'de' is a lone piece again, + # this guard fires, and the name reads family='de Mesnil' like the + # untitled form. # Those two sites are the whole scope, and the MIDDLE position is # deliberately not one of them -- which shows: "Mesnil Garcia de" # strands middle='de' under FAMILY_FIRST, while under @@ -169,14 +172,18 @@ def post_rules(state: ParseState) -> ParseState: # FAMILY_FIRST the opening piece is the family and the given sits # behind it, and reading the role alone let "de Mesnil" split. The # single-token test says the same thing in each shape: a particle - # group already chained forward is not a lone particle. "Mr. de - # Mesnil" is three tokens in two pieces -- the title alone, then - # the particle GROUP -- so both sites are two tokens long and 1b - # declines on each; the family reading there is rule 1's in the - # default order and assign's under a family-first one. Both shapes - # then need another name token to fold with, which leaves a - # degenerate bare 'de' as it stands rather than inventing a - # surname. + # group already chained forward is not a lone particle -- the + # FAMILY_FIRST "Juan de la Vega" above is what that looks like. + # "Mr. de Mesnil" is NOT one, and since #367 not even close to + # one: it is three tokens in THREE pieces -- the title, the + # particle, the surname -- because a title no longer displaces the + # particle out of the leading name position, so nothing chains. + # Both sites are one token long, so this guard FIRES and the + # family reading is its own. Rule 1 above cannot be what produces + # it: rule 1 is gated on `not families`, and 'Mesnil' is already + # the family. Both shapes need another name token to fold with, + # which leaves a degenerate bare 'de' as it stands rather than + # inventing a surname. sites = (_leading_name_piece(state, tokens), tuple(givens)) if len(givens) + len(middles) + len(families) > 1 and any( len(site) == 1 diff --git a/nameparser/_types.py b/nameparser/_types.py index 77e941f4..8da6a6eb 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -366,10 +366,15 @@ class AmbiguityKind(StrEnum): #: names ("read as a given name") -- that role is whatever #: assignment gave it, so it follows ``name_order`` and any #: ``script_orders`` entry, which is why the kind cannot name it. - #: A particle a title shifted off the front ("Dr. Van Johnson") - #: was instead claimed by the prefix chain, and ``detail`` says - #: that and names no field at all: grouping runs before roles - #: exist, so that text is the same under every order. + #: A particle that something ahead of it shifted off the front of + #: the name was instead claimed by the prefix chain, and ``detail`` + #: says that and names no field at all: grouping runs before roles + #: exist, so that text is the same under every order. Since #367 a + #: plain title is not such a thing -- "Dr. Van Johnson" reads as + #: the untitled "Van Johnson" does and takes the first shape -- + #: and what remains is a leading word that is both a title and a + #: particle, so it stays a name piece and the particle behind it is + #: genuinely not leading ("Freiherr von Richthofen"). PARTICLE_OR_GIVEN = "particle-or-given" #: A nickname/maiden delimiter opened without closing (or closed #: without opening); the text was kept as literal name content, so diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index ea5a46aa..806df8f4 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -18,9 +18,13 @@ #: the given position too, folding it into the family beside it, so that #: neither shape leaves a given name behind -- as long as there is another #: name token to fold into. A bare "de" stays as it is. Where a chain is -#: reported as the given name anyway -- "Sir de Mesnil" gives given -#: "de Mesnil" and no surname at all -- that is an open bug (#367), not a -#: limit this set means to draw. +#: reported as the given name anyway it is because the member is no +#: longer standing alone: under ``Policy(name_order=FAMILY_FIRST)`` the +#: given position of "Juan de la Vega" holds the whole three-token +#: chain, so the rule declines and given "de la Vega" stands. A title +#: in front is NOT such a case -- since #367 a title is transparent to +#: the leading-particle exception, so "Sir de Mesnil" leaves "de" a lone +#: piece and reads family "de Mesnil", exactly as the untitled form does. #: Membership also decides the ambiguity report -- see #: :py:data:`PARTICLES` below. #: Curated to exclude anything that can be a given name in some culture @@ -97,8 +101,17 @@ #: name "von bergen wessels", while the same chaining in "Smith, Juan #: de la Cruz" gives the middle name "de la Cruz". A leading #: particle is the exception and chains nothing: the chain skips the -#: first piece unconditionally, membership in this set or any other -#: never entering into it. Where the pieces then land is again a later +#: first piece of the NAME, its membership in this set or in +#: :py:data:`NON_GIVEN_NAME_PARTICLES` never entering into it. Leading +#: is read off the name rather than off the input (#367): a title is +#: not part of the name, so it is stepped over and "Dr. Van Johnson" +#: reads as the untitled "Van Johnson" does. One kind of word is not +#: stepped over, and this is the one place a title's vocabulary and +#: this one interact -- ``st``, ``do`` and ``freiherr`` are each BOTH +#: a title and a particle, so any of them could be the name's own +#: first piece and stops the scan, leaving a particle behind it +#: non-leading and free to chain ("Freiherr von Richthofen"). +#: Where the pieces then land is again a later #: question, and this is where membership decides something: a leading #: :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole name a #: family name ("de la Vega") under every ``name_order``, because a word diff --git a/tests/test_first_name.py b/tests/test_first_name.py index 38ffa4bd..c2dc88b2 100644 --- a/tests/test_first_name.py +++ b/tests/test_first_name.py @@ -1,5 +1,3 @@ -import pytest - from nameparser import HumanName from tests.base import HumanNameTestBase @@ -61,9 +59,31 @@ def test_first_name_is_not_prefix_if_only_two_parts_comma(self) -> None: self.m(hn.first, "Van", hn) self.m(hn.last, "Nguyen", hn) - @pytest.mark.xfail def test_first_name_is_prefix_if_three_parts(self) -> None: - """Not sure how to fix this without breaking Mr and Mrs""" + """Fixed in 2.2 (#367) by making titles transparent to the + leading-particle exception. This carried v1's ``xfail`` and the + docstring "Not sure how to fix this without breaking Mr and + Mrs" for the whole life of the 1.x line -- the exception was + keyed on piece index 0, so any title displaced the particle and + the prefix chain fired, giving last='Van Nguyen'. It is keyed on + the first piece that is not a title OR is a particle -- not the + naive "first piece that is not a title", which collapses + 'St John Smith' into one given name and breaks + tests/test_constants.py::ConstantsCustomizationTests + ::test_add_title, 15 failures in all. The Mr/Mrs conclusion + holds under either rule, which is why the Mr/Mrs shapes are + asserted here rather than only asserted about: a bare 'Mr./Mrs. Surname' + has no particle to displace, so nothing about it reaches the + rule that moved. v1's concern was about the fix it had in mind, + not about this one.""" hn = HumanName("Mr. Van Nguyen") self.m(hn.first, "Van", hn) self.m(hn.last, "Nguyen", hn) + for text, last in [("Mr. Smith", "Smith"), ("Mrs. Smith", "Smith"), + ("Mr. Nguyen", "Nguyen")]: + titled = HumanName(text) + self.m(titled.first, "", titled) + self.m(titled.last, last, titled) + titled = HumanName("Mr. John Smith") + self.m(titled.first, "John", titled) + self.m(titled.last, "Smith", titled) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index b420958c..6f8ccd64 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -114,20 +114,100 @@ def __post_init__(self) -> None: {"given": "John", "middle": "Q", "family": "Smith", "suffix": "MA"}, ambiguities=("suffix-or-name",)), - Case("titled_ambiguous_particle_chains", "Dr. Van Johnson", - {"title": "Dr.", "family": "Van Johnson"}, + Case("titled_ambiguous_particle_does_not_chain", "Dr. Van Johnson", + {"title": "Dr.", "given": "Van", "family": "Johnson"}, + classification="fix(#367)", ambiguities=("particle-or-given",), - notes="the other branch of 'Van Johnson': a leading title " - "shifts Van off the given position, the prefix chain " - "fires, and the fork is reported from group rather than " - "assign (v1 parity on the fields)"), - Case("titled_ambiguous_particle_no_op_chain", "Dr. Van Jr.", - {"title": "Dr.", "given": "Van", "suffix": "Jr."}, + notes="reads exactly as the untitled 'Van Johnson' does, " + "because a title is not part of the name and so cannot " + "decide whether the NAME begins with a particle (#367). " + "This row pinned the opposite until 2.2 -- title 'Dr.', " + "family 'Van Johnson', the chain having fired because " + "the title shifted Van off piece index 0 -- and cited " + "v1 parity for it. Parity was real (1.4.0 gives last " + "'Van Johnson') and still not the tiebreaker it looked " + "like: this is the SAME shape as 'Mr. Van Nguyen', " + "which v1 shipped as an xfail calling the reading " + "wrong, so v1 pinned one shape both as correct and as " + "broken. Resolved toward the xfail, which now passes " + "(tests/test_first_name.py::" + "test_first_name_is_prefix_if_three_parts). The fork is " + "still reported, from assign rather than group -- the " + "same place the untitled 'Van Johnson' reports it"), + Case("titled_ambiguous_particle_keeps_its_middles", "Dr. Van Johnson Smith", + {"title": "Dr.", "given": "Van", "middle": "Johnson", + "family": "Smith"}, + classification="fix(#367)", + ambiguities=("particle-or-given",), + notes="the release log names this shape and nothing asserted " + "it. 1.4.0 gives title 'Dr.', last 'Van Johnson Smith'; " + "un-chaining leaves three name pieces, so the middle " + "appears where the whole thing used to be one surname"), + Case("given_name_title_ambiguous_particle", "Sir Van Johnson", + {"title": "Sir", "given": "Van", "family": "Johnson"}, + classification="fix(#367)", + ambiguities=("particle-or-given",), + notes="a GIVEN-NAME title, which is the worse half of the bug " + "#367 fixed: 1.4.0 and 2.1 alike gave first " + "'Van Johnson' and no family name at all, the chain " + "having fired and then been handed whole to `given`. " + "Now identical to the untitled 'Van Johnson'"), + Case("given_name_title_never_given_particle", "Sir de Mesnil", + {"title": "Sir", "family": "de Mesnil"}, + classification="fix(#367)", + notes="the never-given half: `de` is not an ambiguous " + "particle, so there is no fork to report and post_rules " + "1b folds the name into the family. 1.4.0 and 2.1 gave " + "first 'de Mesnil' with no family, because the chain " + "left 1b nothing standing alone to fire on"), + Case("suffix_word_title_ambiguous_particle", "Jr. Van Johnson", + {"title": "Jr.", "given": "Van", "family": "Johnson"}, + classification="fix(#367)", + ambiguities=("particle-or-given",), + notes="a leading 'Jr.' classifies as a TITLE, not a suffix, " + "which is why the transparency scan does not step over " + "suffix pieces -- see tests/v2/pipeline/test_group.py::" + "test_a_suffix_shaped_leading_piece_is_not_stepped_over. " + "1.4.0 gives title 'Jr.', last 'Van Johnson'"), + Case("titled_particle_chain_survives_a_title_that_is_also_a_particle", + "Freiherr von Richthofen", + {"title": "Freiherr", "family": "von Richthofen"}, + ambiguities=("particle-or-given",), + notes="#367's title transparency skips a piece that can ONLY " + "be a title, never one that could be the name's own " + "first piece. 'freiherr' is both a title and an " + "ambiguous particle, so it stops the scan and stays the " + "leading NAME piece; 'von' behind it is therefore " + "non-leading and chains, exactly as before 2.2. This is " + "also the CANONICAL shape reaching group's " + "PARTICLE_OR_GIVEN emitter, not the only one -- 'St Van " + "Johnson', 'Do St Johnson' and 'Dr. Do van Johnson' " + "reach it too, the last with a plain title ahead of the " + "both-vocabulary word; see tests/v2/test_parser.py -- " + "and the class that a plain " + "'first piece that is not a title' test broke: it " + "skipped 'St'/'Do'/'Freiherr' and collapsed the " + "untitled 'St John Smith' into one given name"), + Case("titled_ambiguous_particle_no_op_chain", "Do Van Jr.", + {"title": "Do", "given": "Van", "suffix": "Jr."}, notes="the piece after the particle is a suffix, so the chain " "scan never advances and the merge is a no-op -- nothing " "was chained, so there is no fork to report (the emitter " "fired here for all 39 ambiguous particles, and _assign " - "double-reported the same token)"), + "double-reported the same token). Spelled with 'Do' " + "rather than the 'Dr.' this row carried until 2.2: " + "under #367 a plain title is transparent, so 'Dr. Van " + "Jr.' leaves Van the leading name piece and the chain " + "loop skips it without ever reaching the no-op. 'Do' is " + "a title AND a particle, which stops the transparency " + "scan, so the chain does fire on Van and the j > k + 1 " + "guard is what declines it -- the same reading of the " + "name, reached through the branch the row exists to pin. " + "Not parity, and not #367's doing either: 1.4.0 reads " + "'Do Van Jr.' as first 'Do Van', last 'Jr.', so the " + "divergence is 2.0's suffix routing plus 'do' being a " + "title -- both older than this row's respelling", + classification="fix"), Case("initial_shaped_not_conjunction", "john e. smith", {"given": "john", "middle": "e.", "family": "smith"}, notes="v1 is_conjunction excludes initials at classify too"), diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 7f3208f8..6140ea46 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -1,3 +1,5 @@ +import dataclasses + from nameparser._lexicon import Lexicon from nameparser._pipeline._classify import classify from nameparser._pipeline._extract import extract_delimited @@ -76,6 +78,46 @@ def test_leading_prefix_is_never_chained() -> None: assert _piece_texts(out) == [["Van", "Johnson"]] +def test_a_title_does_not_make_the_particle_behind_it_non_leading() -> None: + # #367: "leading" is the first piece of the NAME. A title is not + # part of the name, so it is stepped over and the grouping is the + # untitled one with the title in front of it. + assert _piece_texts(_grouped("Mr. Van Johnson")) == \ + [["Mr.", "Van", "Johnson"]] + assert _piece_texts(_grouped("Mr. Van Johnson Smith")) == \ + [["Mr.", "Van", "Johnson", "Smith"]] + + +def test_a_title_that_is_also_a_particle_stops_the_scan() -> None: + # the other half of the same rule, and the reason it is not spelled + # "the first piece that is not a title": the default vocabulary + # puts `freiherr`, `st` and `do` in BOTH sets, so any of them could + # be the name's own first piece. Stepping over one would make the + # particle behind it chain, and would break a name with no title in + # front of it at all ("St John Smith" -> title St, given John, + # family Smith). Spelled here as the overlap a CONFIG can create, + # which is the same shape: tests/test_constants.py::test_add_title + # adds `te` to the titles while `te` ships as a particle. + overlap = dataclasses.replace(_LEX, titles=_LEX.titles | {"van"}) + assert _piece_texts(_grouped("Van von Richthofen", lexicon=overlap)) == \ + [["Van", "von Richthofen"]] + assert _piece_texts(_grouped("Van Johnson Smith", lexicon=overlap)) == \ + [["Van", "Johnson", "Smith"]] + + +def test_a_suffix_shaped_leading_piece_is_not_stepped_over() -> None: + # #367 steps over TITLE pieces only. Adding suffix pieces to that + # scan survives the whole suite while changing what these names + # parse to, so pin them here: with the skip, `leading` moves to + # "Van", the chain loop passes over it, and the two pieces below + # become three -- which puts Van in the middle name rather than the + # family once roles exist. See _group.py for why that reading is + # worse rather than merely different. + assert _piece_texts(_grouped("Ph. D. Van Johnson")) == \ + [["Ph. D.", "Van Johnson"]] + assert _piece_texts(_grouped("II Van Johnson")) == [["II", "Van Johnson"]] + + def test_von_und_zu_bridges() -> None: # conjunction "und" joins two prefixes; the joined piece is a derived # prefix and still chains onto the following name (v1 PR #191) diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 7ad21966..850c6ff7 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1036,6 +1036,8 @@ def _claim(rule: dict) -> _Claim: _Claim(2, ('given', 'middle', 'title'), "c14187bb08f8"), "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": _Claim(1, ('given', 'middle', 'suffix', 'title'), "390e7f814d13"), + "fix(#367) a title no longer displaces a leading particle out of the leading position": + _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, "expected_since_2.0.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": @@ -1050,8 +1052,13 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), + "fix(#367) a title no longer displaces a leading particle out of the leading position": + _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), + }, + "expected_since_2.1.0.toml": { + "fix(#367) a title no longer displaces a leading particle out of the leading position": + _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, - "expected_since_2.1.0.toml": {}, } diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index a1ab63af..a2ae0e28 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -288,22 +288,73 @@ def test_trailing_roman_numeral_reports_the_fork() -> None: assert parse("John Q. V").ambiguities == () +#: The word the _group-emitter tests below lead with. It has to be BOTH a +#: title and an ambiguous particle -- see test_the_chained_emitter_is_still +#: _reachable for why, and for what to do when this stops being true. +_TITLE_PARTICLE = "Freiherr" + + +def test_the_chained_emitter_is_still_reachable() -> None: + """_group's PARTICLE_OR_GIVEN emitter needs a piece that is both a title + and an ambiguous particle somewhere ahead of the chained particle, and + since #367 nothing else will do -- an ordinary title is transparent to + the leading-particle exception, so on its own it takes _assign's branch + instead. That word need not lead the input: "Dr. Do van Johnson" reaches + the emitter with a plain title in front of it. What it cannot be is + absent. + + Two different failures, wanting two different fixes. If the intersection + is merely missing the word the tests below use, pick another from the + set. If it is EMPTY, the emitter is unreachable: every test of it is + then measuring nothing, and the emitter itself should go rather than be + re-pointed. #360 may move members of this set, which is why the coupling + is executable here instead of being a comment. + """ + lex = Lexicon.default() + both = lex.titles & lex.particles_ambiguous + assert both, ( + "no title is an ambiguous particle, so _group's PARTICLE_OR_GIVEN " + "emitter is unreachable and its tests measure nothing -- remove the " + "emitter rather than repointing them") + assert _TITLE_PARTICLE.lower() in both, ( + f"{_TITLE_PARTICLE!r} is no longer both a title and an ambiguous " + f"particle; the _group-emitter tests need a lead from {sorted(both)}") + + def test_ambiguous_particle_reports_both_branches_of_its_fork() -> None: - # "Van Johnson" reads Van as a given name and says so. A leading - # title shifts Van off index 0, the prefix-chain merge fires, and - # Van becomes a particle instead -- the SAME fork, called the other - # way. The two branches are taken in different stages (_assign vs - # _group), so only the one with an emitter used to report. - given_reading = parse("Van Johnson") - assert given_reading.given == "Van" + # "von Richthofen" reads von as a given name and says so. Put a + # piece in front of it that is BOTH a title and a particle and von + # is no longer the name's leading piece, the prefix-chain merge + # fires, and von becomes a particle instead -- the SAME fork, + # called the other way. The two branches are taken in different + # stages (_assign vs _group), so only the one with an emitter used + # to report. + # + # Spelled with 'Freiherr' rather than the 'Dr.' this test used + # until 2.2: an ordinary title is now transparent to the + # leading-particle exception (#367), so "Dr. Van Johnson" takes the + # _assign branch like everything else. `freiherr`/`st`/`do` -- a + # title that could also be the name's own first piece -- is what + # still reaches _group's emitter. This is the canonical spelling of + # that, not the only one: "St Van Johnson", "Do St Johnson" and + # "Dr. Do van Johnson" reach it as well. + given_reading = parse("von Richthofen") + assert given_reading.given == "von" assert [a.kind for a in given_reading.ambiguities] == \ [AmbiguityKind.PARTICLE_OR_GIVEN] - particle_reading = parse("Dr. Van Johnson") - assert particle_reading.family == "Van Johnson" + particle_reading = parse("Freiherr von Richthofen") + assert particle_reading.family == "von Richthofen" assert [a.kind for a in particle_reading.ambiguities] == \ [AmbiguityKind.PARTICLE_OR_GIVEN] - assert [t.text for t in particle_reading.ambiguities[0].tokens] == ["Van"] + assert [t.text for t in particle_reading.ambiguities[0].tokens] == ["von"] + # and the branch the title no longer takes: "Dr. Van Johnson" is + # now byte-identical to the bare "Van Johnson", fork included + titled, bare = parse("Dr. Van Johnson"), parse("Van Johnson") + assert (titled.given, titled.family) == (bare.given, bare.family) \ + == ("Van", "Johnson") + assert [a.detail for a in titled.ambiguities] == \ + [a.detail for a in bare.ambiguities] def test_unambiguous_particle_chain_reports_nothing() -> None: @@ -311,10 +362,33 @@ def test_unambiguous_particle_chain_reports_nothing() -> None: assert parse("Dr. de la Vega").ambiguities == () +@pytest.mark.xfail(strict=True, reason="#369: 'abu' is a bound given name " + "as well as an ambiguous particle") +def test_bound_given_name_that_is_also_a_particle() -> None: + # The one case #367 regressed, asserted as the DESIRED post-#369 + # output so that fixing #369 announces itself here rather than + # silently. Today: given 'Abu', family 'Bakar'. Through 2.1 it read + # correctly only because the title displaced 'abu' out of the + # leading position and the prefix chain claimed 'Bakar' -- a side + # effect of the bug, not a rule: "Sheik abdul salam", whose lead is + # a bound given name and NOT a particle, chains no such thing. + # The name is in none of the differential corpora, so nothing else + # would notice it moving again. + assert parse("Sheik Abu Bakar").given == "Abu Bakar" + + +# The first three reach the chain loop and decline inside it: the piece +# after the particle is a suffix, so the scan never advances and merge() +# is a no-op -- nothing was chained, no fork taken. They are spelled with +# a title that is ALSO a particle ('Do', 'St'), because that is what +# still puts an ambiguous particle off the name's leading position since +# #367. The last three are the same strings with a plain title, which +# now decline one step earlier -- the particle IS the leading name piece +# and the loop skips it -- and are kept so the pair stays visible: two +# different reasons, one output, and neither may start reporting a fork. @pytest.mark.parametrize("text", [ - "Dr. Van Jr.", # the piece after the particle is a suffix, so - "Dr. Van MD", # the chain scan never advances and merge() is - "Dr. Do Jr.", # a no-op -- nothing was chained, no fork taken + "Do Van Jr.", "Do Van MD", "St Van Jr.", + "Dr. Van Jr.", "Dr. Van MD", "Dr. Do Jr.", ]) def test_no_op_prefix_chain_is_not_a_fork(text: str) -> None: assert parse(text).ambiguities == () @@ -322,18 +396,23 @@ def test_no_op_prefix_chain_is_not_a_fork(text: str) -> None: def test_a_fork_is_reported_by_exactly_one_stage() -> None: # the no-op merge left the particle a lone leading piece, which is - # _assign's trigger, so both stages reported the same token - n = parse("Dr. Van Jr Smith") + # _assign's trigger, so both stages reported the same token. + # 'Do' rather than the 'Dr.' this used until 2.2, for the reason + # above: with a plain title the chain loop never fires at all now, + # so the double-report it guards against is out of reach there. + n = parse("Do Van Jr Smith") assert n.given == "Van" assert len(n.ambiguities) == 1 def test_chained_particle_detail_does_not_claim_a_role() -> None: # _group runs before assignment, so it cannot know which field the - # chained piece lands in -- "Dr. Van Johnson de la Cruz" puts it in - # GIVEN. The detail must describe the decision, not guess a role. - n = parse("Dr. Van Johnson de la Cruz") - assert n.given == "Van Johnson" + # chained piece lands in -- "Freiherr von Richthofen de la Cruz" + # puts it in GIVEN, while the bare "Freiherr von Richthofen" above + # puts it in FAMILY. The detail must describe the decision, not + # guess a role. + n = parse("Freiherr von Richthofen de la Cruz") + assert n.given == "von Richthofen" (amb,) = n.ambiguities assert "family name" not in amb.detail @@ -347,17 +426,36 @@ def test_chained_particle_detail_is_order_invariant(policy: Policy) -> None: # _group's emitter is the reason the docs can scope leading-particle # DESTINATIONS to the default order without qualifying this text: # it names no field, and the chain it reports is a grouping-stage - # decision taken before any role exists. "Dr. Van Johnson" takes the - # chained branch under every order, so the string is the same one - # three times -- pin it, or the invariant is only an intention. - n = Parser(policy=policy).parse("Dr. Van Johnson") - assert (n.given, n.family) == ("", "Van Johnson") + # decision taken before any role exists. "Freiherr von Richthofen" + # takes the chained branch under every order, so the string is the + # same one three times -- pin it, or the invariant is only an + # intention. ("Dr. Van Johnson" carried this until 2.2; #367 made a + # plain title transparent, so it no longer chains at all.) + n = Parser(policy=policy).parse("Freiherr von Richthofen") + assert (n.given, n.family) == ("", "von Richthofen") (amb,) = n.ambiguities assert amb.kind is AmbiguityKind.PARTICLE_OR_GIVEN assert amb.detail == ( - "'Van' was chained onto the following name piece; " + "'von' was chained onto the following name piece; " "it is also a given name in other names") + # The shape above is the one #367 did NOT move, so pin the one it + # did in the same three orders: a plain title is transparent, so + # "Dr. Van Johnson" is byte-identical to the bare "Van Johnson" + # under every name_order, and the fork comes from _assign -- whose + # detail DOES name the role, unlike the grouping-stage text above. + titled = Parser(policy=policy).parse("Dr. Van Johnson") + bare = Parser(policy=policy).parse("Van Johnson") + assert titled.title == "Dr." + assert (titled.given, titled.middle, titled.family) == \ + (bare.given, bare.middle, bare.family) + (titled_amb,) = titled.ambiguities + assert titled_amb.detail == bare.ambiguities[0].detail + role = "family" if policy.name_order[0] is Role.FAMILY else "given" + assert titled_amb.detail == ( + f"leading 'Van' may be a family-name particle; " + f"read as a {role} name") + def test_each_suffix_or_name_branch_describes_itself() -> None: # one kind, two causes: the acronym branch turns on periods, the diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 8b6f4cb5..586d0c4b 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -103,9 +103,13 @@ def _fork_count(state: ParseState) -> int: def test_a_leading_ambiguous_particle_is_reported_once_and_only_once( ) -> None: """PARTICLE_OR_GIVEN is the one kind two stages emit: _group takes - the particle branch when a title shifts the particle off index 0 and - the chain claims something ("Dr. Van Johnson"), _assign takes the - given branch when it stays a lone leading piece ("Van Johnson"). + the particle branch when something shifts the particle off the + name's leading piece and the chain claims something ("Freiherr von + Richthofen"), _assign takes the given branch when it stays a lone + leading piece ("Van Johnson", and since #367 "Dr. Van Johnson" as + well -- a plain title is transparent, so only a leading word that + is BOTH a title and a particle still reaches _group's emitter, + which is what the 'Freiherr ' lead below exercises). Each reports the side it decides -- see the ParseState docstring -- but they coordinate only through _group's `j > k + 1` guard, which mirrors _assign's reachability by hand. Nothing checked the mirror. @@ -131,7 +135,7 @@ def test_a_leading_ambiguous_particle_is_reported_once_and_only_once( assert particles, "no ambiguous particles to exercise" failures = [] for particle in particles: - for lead in ("", "Dr. ", "Dr. Ann "): + for lead in ("", "Dr. ", "Dr. Ann ", "Freiherr "): for body in ("", "Johnson ", "Johnson Smith "): for tail in ("", "Jr.", "MD", "III"): text = f"{lead}{particle} {body}{tail}".strip() diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 37d22b23..2b956301 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -414,3 +414,59 @@ fields = ["title", "given", "middle", "suffix"] # again, which is the thing the sort exists to prevent. Giving # classify() a real specificity order is the fix, and it is a change # to the harness rather than to a rule. + +[[change]] +issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" +# 'Mr. Van Nguyen': a leading particle deliberately does not chain -- +# that is what makes 'Van Nguyen' read as given Van, family Nguyen. +# Through 2.1 the exception was keyed on piece index 0, so a TITLE +# displaced the particle and the chain fired, giving family +# 'Van Nguyen' (v1's reading, which is why this name is parity with +# 1.4.0 until now). 2.2 keys it on the first piece of the NAME, so a +# titled name reads as the same text reads untitled. +# +# The wider class is any title followed by a particle -- 'Dr. Van +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and this +# regex names one title and one particle because 'Mr. Van Nguyen' is +# the one name in the corpora that moves. Deliberately narrower than +# the class: a corpus name that joins it later should arrive +# UNEXPLAINED and be read once, not be absorbed silently. Widen the +# literals then, and record the new count. +# +# The title slot is a LITERAL for that reason and not by accident. +# Nothing in a regex over the raw string can ask whether a word is a +# title, so the `(?i)^\S+\.?\s+van\b` this rule shipped with matched +# ANY first word: it reached 11 corpus names and explained 1. The +# other ten include 'Vincent van Gogh', 'VINCENT VAN GOGH' and +# 'Alex van Johnson' -- the "Van Johnson" family AGENTS.md names as a +# standard regression canary. name_regex rules sort ahead of +# fields-only ones and `fields` matches by subset, so a future +# regression on any of those would have been labelled fix(#367) and +# exited 0 -- a strict loss at this baseline, whose ledger was empty +# before. (`\.?` was inert into the bargain: `\S+` is greedy, so it +# had already taken the period.) +# +# `fields` is what carries the rest of the tightness -- and it is narrower +# than the change rather than a description of it. The change moves +# `title` as well: a word in BOTH vocabularies stops the transparency +# scan and stays a title piece instead of being chained onto the name, +# so 'Dr. St John Smith' goes from title 'Dr.', family 'St John Smith' +# to title 'Dr. St', given 'John', family 'Smith'. It moves `maiden` +# as well: un-chaining lets a marker standing behind the particle be +# seen at all, so 'Mr. Van Johnson nee Brown' goes from family +# 'Van Johnson nee Brown' with no maiden name to given 'Van', family +# 'Johnson', maiden 'Brown'. `suffix` and `nickname` are the two that +# genuinely never move. Naming three roles anyway is deliberate, for +# the reason the regex names only `van`: a corpus name that moves +# `title` or `maiden` should arrive UNEXPLAINED and be read once, +# rather than be absorbed here. +# +# This rule has to exist even though the run already exited 0 without +# it. The diff was being classified by fix(suffix-routing) below, a +# fields-only rule whose ["given", "family", "suffix"] is a superset of +# this diff's ["given", "family"] -- a rule about a trailing credential +# absorbing a leading-particle change, which is the mis-classification +# the README warns about. A name_regex rule outranks every fields-only +# one, so this claims it back. +name_regex = "(?i)^mr\\.\\s+van\\b" +fields = ["given", "middle", "family"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 75374fda..d8cb0fe7 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -228,3 +228,58 @@ issue = "fix(#298) 间隔号 division changes the comma reading, sending the cre # screen that a literal here would be unreviewable. name_regex = "(?s)(?=.*,)(?=.*\\u00B7)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["given", "family", "title", "suffix"] + +[[change]] +issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" +# 'Mr. Van Nguyen': a leading particle deliberately does not chain -- +# that is what makes 'Van Nguyen' read as given Van, family Nguyen. +# Through 2.1 the exception was keyed on piece index 0, so a TITLE +# displaced the particle and the chain fired, giving family +# 'Van Nguyen' (v1's reading, which is why this name is parity with +# 1.4.0 until now). 2.2 keys it on the first piece of the NAME, so a +# titled name reads as the same text reads untitled. +# +# The wider class is any title followed by a particle -- 'Dr. Van +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and this +# regex names one title and one particle because 'Mr. Van Nguyen' is +# the one name in the corpora that moves. Deliberately narrower than +# the class: a corpus name that joins it later should arrive +# UNEXPLAINED and be read once, not be absorbed silently. Widen the +# literals then, and record the new count. +# +# The title slot is a LITERAL for that reason and not by accident. +# Nothing in a regex over the raw string can ask whether a word is a +# title, so the `(?i)^\S+\.?\s+van\b` this rule shipped with matched +# ANY first word: it reached 11 corpus names and explained 1. The +# other ten include 'Vincent van Gogh', 'VINCENT VAN GOGH' and +# 'Alex van Johnson' -- the "Van Johnson" family AGENTS.md names as a +# standard regression canary. name_regex rules sort ahead of +# fields-only ones and `fields` matches by subset, so a future +# regression on any of those would have been labelled fix(#367) and +# exited 0 -- a strict loss at this baseline, whose ledger was empty +# before. (`\.?` was inert into the bargain: `\S+` is greedy, so it +# had already taken the period.) +# +# `fields` is what carries the rest of the tightness -- and it is narrower +# than the change rather than a description of it. The change moves +# `title` as well: a word in BOTH vocabularies stops the transparency +# scan and stays a title piece instead of being chained onto the name, +# so 'Dr. St John Smith' goes from title 'Dr.', family 'St John Smith' +# to title 'Dr. St', given 'John', family 'Smith'. It moves `maiden` +# as well: un-chaining lets a marker standing behind the particle be +# seen at all, so 'Mr. Van Johnson nee Brown' goes from family +# 'Van Johnson nee Brown' with no maiden name to given 'Van', family +# 'Johnson', maiden 'Brown'. `suffix` and `nickname` are the two that +# genuinely never move. Naming three roles anyway is deliberate, for +# the reason the regex names only `van`: a corpus name that moves +# `title` or `maiden` should arrive UNEXPLAINED and be read once, +# rather than be absorbed here. +# +# The 1.4 ledger carries the same rule, where it also has a second job: +# there the diff was already being absorbed by that file's fields-only +# fix(suffix-routing), a rule about a trailing credential whose +# ["given", "family", "suffix"] is a superset of this diff's fields. No +# fields-only rule exists here, so this file's copy is the plain +# classification. +name_regex = "(?i)^mr\\.\\s+van\\b" +fields = ["given", "middle", "family"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 817e4184..796b1062 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -3,40 +3,94 @@ # needs `issue`; optional `name_regex` and `fields` narrow it, and # compare.py sorts name_regex rules ahead of fields-only ones. # -# EMPTY ON PURPOSE, and correct while it stays that way. A ledger is -# opened the day its baseline ships (AGENTS.md release step 8), before -# the cycle has produced a single behavior change -- so there is nothing -# to classify yet. `nameparser/` is byte-identical to the v2.1.0 tag as -# this file is added, so a run against this baseline reports zero diffs -# and zero UNEXPLAINED. -# -# It exists rather than waiting because DEFAULT_BASELINE and the ledger -# are coupled: _allowlist_for treats a missing file as a hard error, on -# the reasoning that an absent ledger classifies nothing and would make -# every diff report as unexplained. So the baseline cannot advance to -# 2.1.0 without this file. -# -# Add the first rule when the first 2.2 behavior change lands. Do not -# copy rules across from expected_since_2.0.0.toml: that ledger +# This file was opened empty the day 2.1.0 shipped (AGENTS.md release +# step 8) and stayed that way through #354, #358 and #361, none of +# which moved the default-order corpus. #367 is the first 2.2 change +# that does, and its rule below is the first entry here. +# +# The ledger is opened before it has anything to say because +# DEFAULT_BASELINE and the ledger are coupled: _allowlist_for treats a +# missing file as a hard error, on the reasoning that an absent ledger +# classifies nothing and would make every diff report as unexplained. +# So the baseline could not advance to 2.1.0 without this file. +# +# Do not copy rules across from expected_since_2.0.0.toml: that ledger # classifies what 2.1 changed on top of 2.0, and a rule copied without # checking either over-matches -- hiding a real 2.2 regression behind a -# 2.1-era label -- or never fires at all. +# 2.1-era label -- or never fires at all. The #367 rule below IS shared +# with the other two ledgers, and that is not an exception to this: the +# change is in the 2.2 cycle, so every baseline before it sees the same +# diff, and each copy was checked against its own run rather than +# assumed. # # tests/v2/test_differential.py permits exactly one empty ledger, the -# one DEFAULT_BASELINE names. Once this stops being the open cycle it -# must carry rules like any other, so a 2.2 that changed no behavior at -# all would need that decision made deliberately rather than inherited. +# one DEFAULT_BASELINE names -- a permission this file no longer needs. # -# tests/v2/test_ledger_guards.py's _SPAN_BEARING_RULES already records this -# file with an empty set: no rule here hand-copies _SCRIPT_RANGES yet, -# and the first one that does has to be recorded there or the sweep -# fails. +# tests/v2/test_ledger_guards.py's _SPAN_BEARING_RULES records this +# file with an empty set: no rule here hand-copies _SCRIPT_RANGES, and +# the first one that does has to be recorded there or the sweep fails. +# #367's rule is Latin-only and declares no script span. # # There is deliberately no `change = []` line. TOML forbids appending a # [[change]] table to a statically defined array, so that line would -# block the exact next step this comment asks for -- and -# every reader uses .get("change", []) -- four call sites across -# three files -- so the key's absence is already the empty -# ledger. tests/v2/test_differential.py checks that nothing else is -# defined at the top level here, since a mistyped table name would -# otherwise read as an empty ledger rather than as a broken one. +# have blocked the entry below -- and every reader uses +# .get("change", []) -- four call sites across three files -- so the +# key's absence was already the empty ledger. +# tests/v2/test_differential.py checks that nothing else is defined at +# the top level here, since a mistyped table name would otherwise read +# as an empty ledger rather than as a broken one. + +[[change]] +issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" +# 'Mr. Van Nguyen': a leading particle deliberately does not chain -- +# that is what makes 'Van Nguyen' read as given Van, family Nguyen. +# Through 2.1 the exception was keyed on piece index 0, so a TITLE +# displaced the particle and the chain fired, giving family +# 'Van Nguyen' (v1's reading, which is why this name is parity with +# 1.4.0 until now). 2.2 keys it on the first piece of the NAME, so a +# titled name reads as the same text reads untitled. +# +# The wider class is any title followed by a particle -- 'Dr. Van +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and this +# regex names one title and one particle because 'Mr. Van Nguyen' is +# the one name in the corpora that moves. Deliberately narrower than +# the class: a corpus name that joins it later should arrive +# UNEXPLAINED and be read once, not be absorbed silently. Widen the +# literals then, and record the new count. +# +# The title slot is a LITERAL for that reason and not by accident. +# Nothing in a regex over the raw string can ask whether a word is a +# title, so the `(?i)^\S+\.?\s+van\b` this rule shipped with matched +# ANY first word: it reached 11 corpus names and explained 1. The +# other ten include 'Vincent van Gogh', 'VINCENT VAN GOGH' and +# 'Alex van Johnson' -- the "Van Johnson" family AGENTS.md names as a +# standard regression canary. name_regex rules sort ahead of +# fields-only ones and `fields` matches by subset, so a future +# regression on any of those would have been labelled fix(#367) and +# exited 0 -- a strict loss at this baseline, whose ledger was empty +# before. (`\.?` was inert into the bargain: `\S+` is greedy, so it +# had already taken the period.) +# +# `fields` is what carries the rest of the tightness -- and it is narrower +# than the change rather than a description of it. The change moves +# `title` as well: a word in BOTH vocabularies stops the transparency +# scan and stays a title piece instead of being chained onto the name, +# so 'Dr. St John Smith' goes from title 'Dr.', family 'St John Smith' +# to title 'Dr. St', given 'John', family 'Smith'. It moves `maiden` +# as well: un-chaining lets a marker standing behind the particle be +# seen at all, so 'Mr. Van Johnson nee Brown' goes from family +# 'Van Johnson nee Brown' with no maiden name to given 'Van', family +# 'Johnson', maiden 'Brown'. `suffix` and `nickname` are the two that +# genuinely never move. Naming three roles anyway is deliberate, for +# the reason the regex names only `van`: a corpus name that moves +# `title` or `maiden` should arrive UNEXPLAINED and be read once, +# rather than be absorbed here. +# +# The 1.4 ledger carries the same rule, where it also has a second job: +# there the diff was already being absorbed by that file's fields-only +# fix(suffix-routing), a rule about a trailing credential whose +# ["given", "family", "suffix"] is a superset of this diff's fields. No +# fields-only rule exists here, so this file's copy is the plain +# classification. +name_regex = "(?i)^mr\\.\\s+van\\b" +fields = ["given", "middle", "family"]