Make a title transparent to the leading-particle exception (#367)

docs/release_log.rst

@@ -10,11 +10,14 @@ Release Log
     editing one in place as a way to change a default and replaces it
     with configuring a ``Lexicon`` or a private ``Constants``.
 
-    Nothing moved between vocabularies and no parse changes in the
-    default name order: over the 751 names of the differential
-    corpora, every one of the seven fields is identical to 2.1 through
-    both the 2.0 and the 1.x API. Both family-first orders do change,
-    below -- the same seven names under each.
+    Nothing moved between vocabularies. The rename itself changes no
+    parse at all; the parsing changes below are separate fixes, and
+    only one of them reaches the default name order -- a title no
+    longer changing how the name behind it is read, which moves one of
+    the 751 names of the differential corpora. Both family-first
+    orders change too, below -- the same eight names under each, seven
+    of them from the family-first fix and the eighth the one name the
+    title fix moves in every order.
     What breaks is code that *writes* to a default word list, and code
     that imports one by its 1.x name has until 3.0.
 
@@ -24,6 +27,8 @@ Release Log
 
     **Behavior Changes**
 
+    - Fix a title changing how the name behind it is read. A leading particle deliberately does not chain onto the words after it -- that is what makes ``"Van Johnson"`` read as given ``Van``, family ``Johnson`` rather than as one surname -- but the exception was keyed on the first *piece of the input*, so a title displaced the particle out of that position and the chain fired. Identical name text parsed two ways depending on whether a title preceded it: ``"Dr. Van Johnson"`` gave family ``Van Johnson`` with no given name, and ``"Sir Van Johnson"`` gave given ``Van Johnson`` with no *family* name at all, a given-name title then handing the whole chain to ``given``. A title is not part of the name, so it can no longer decide whether the name begins with a particle: the exception is keyed on the first piece of the **name**, and a titled name now reads exactly as the same text reads untitled. What moves is every titled name whose first name-piece is a particle. Every reading in this bullet is the default ``name_order``, which is where the fields differ by order: ``"Dr. Van Johnson"`` and ``"Mr. Van Nguyen"`` go to given ``Van`` plus family ``Johnson``/``Nguyen``, ``"Dr. Van Johnson Smith"`` to given ``Van``, middle ``Johnson``, family ``Smith``, and ``"Sir de Mesnil"`` to family ``de Mesnil``. The same grouping change reaches ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``, where the leading particle takes the family rather than the given: ``"Dr. Van Johnson"`` reads family ``Van``, given ``Johnson`` under both. Three fields carry nearly all of the redistribution -- ``given``, ``middle`` and ``family`` -- but two more move on names inside this class, and only ``suffix`` and ``nickname`` move on none of them. ``title`` GROWS where the word that stops the transparency scan is one the chain used to swallow: ``"Dr. St John Smith"`` was title ``Dr.``, family ``St John Smith`` and is now title ``Dr. St``, given ``John``, family ``Smith``, ``st`` being in both vocabularies and so staying a title piece instead of being chained onto the name. ``maiden`` appears where un-chaining lets a marker standing behind the particle be seen at all: ``"Mr. Van Johnson nee Brown"`` was family ``Van Johnson nee Brown`` with no maiden name and is now given ``Van``, family ``Johnson``, maiden ``Brown``. Untitled names are untouched, and so is every name with no particle behind its title: ``"Mr. Smith"``, ``"Mr. John Smith"``, ``"Sir Ian McKellen"`` and ``"King Henry"`` are byte-identical. So is a name whose leading word is *both* a title and a particle -- ``st``, ``do`` and ``freiherr`` are each in both vocabularies, so ``"St John Smith"``, ``"Do John Smith"`` and ``"Freiherr von Richthofen"`` keep their readings: a word that could be the name's own first piece stops the transparency scan rather than being stepped over. The fork is still reported, and still as ``PARTICLE_OR_GIVEN``; it now comes from the same place the untitled ``"Van Johnson"`` reports it, so the ``detail`` text changes from "was chained onto the following name piece" to "leading 'Van' may be a family-name particle; read as a given name". This fixes a defect v1 shipped as a known-failing test for the life of the 1.x line, with the note "Not sure how to fix this without breaking Mr and Mrs"; it does not break Mr and Mrs, which have no particle to displace. One of the 751 differential corpus names moves, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike -- and it is the same one name under each family-first order too, ``"Mr. Van Nguyen"``. One case regresses and is tracked as `#369 <https://github.com/derek73/python-nameparser/issues/369>`_: ``"Sheik Abu Bakar"`` was given ``Abu Bakar`` and is now given ``Abu``, family ``Bakar``. It read correctly before only because ``abu`` happens to be a particle as well as a bound given name -- ``"Sheik abdul salam"`` shows that being a bound given name alone does not chain -- so the reading was a side effect of the bug rather than a rule (closes #367)
+
     - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359)
 
     - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355)

nameparser/_lexicon.py

@@ -334,10 +334,13 @@ class Lexicon:
     particles: frozenset[str] = frozenset()
     #: Subset of particles that can also BE a given name ("Van
     #: Johnson", but also "Van Buren"). Membership decides nothing
-    #: about chaining: the prefix chain skips index 0 unconditionally
+    #: about chaining: the prefix chain skips the name's first piece
     #: and never consults this set, so it leaves a leading particle a
     #: piece of its own whether listed or not -- "de Mesnil" groups
-    #: into two pieces exactly as "van Gogh" does. What membership
+    #: into two pieces exactly as "van Gogh" does, and since #367 the
+    #: NAME in "Dr. de Mesnil" and "Dr. Van Johnson" groups into those
+    #: same two pieces behind the title piece, a title not being
+    #: part of the name it precedes. What membership
     #: decides is what becomes of that piece afterwards. Under ANY
     #: ``name_order`` a member records a particle-or-given ambiguity
     #: and a non-member records none, and a non-member is additionally

nameparser/_pipeline/_assign.py

@@ -22,8 +22,9 @@
 non-suffixy ones COMMA_STRUCTURE).
 SUFFIX_COMMA: segment 0 as NO_COMMA; segments 1+ wholly SUFFIX.
 Emits PARTICLE_OR_GIVEN when the leading name piece is a lone
-particles_ambiguous token with more pieces following ("Van Johnson") --
-whatever role name_order assigns that position.
+particles_ambiguous token with more pieces following ("Van Johnson",
+and since #367 "Dr. Van Johnson" too, a title no longer displacing the
+particle out of that position) -- whatever role name_order assigns.
 """
 from __future__ import annotations
 

nameparser/_pipeline/_group.py

@@ -199,9 +199,53 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(),
         # prefix chains: a non-leading prefix run absorbs everything to
         # the next prefix or suffix (v1's leading_first_name rule keeps
         # the first piece a name: "Van Johnson")
+        #
+        # "Leading" means the first piece of the NAME, not of the input
+        # (#367): a title is not part of the name, so it must not decide
+        # whether the name begins with a particle. Keyed on index 0, a
+        # title displaced the particle and the chain fired, so identical
+        # name text parsed two ways ("Van Johnson" -> given Van, family
+        # Johnson; "Dr. Van Johnson" -> family "Van Johnson").
+        #
+        # "Title AND NOT prefix" rather than the plain "not a title" the
+        # rule is stated as, and the difference is not academic: `st`,
+        # `do` and `freiherr` are each BOTH a title and an ambiguous
+        # particle, so the plain test skipped over the very piece the
+        # exception exists to protect and "St John Smith" -- no title in
+        # front of it at all -- collapsed from title St, given John,
+        # family Smith into one given "St John Smith". A piece that
+        # could be the name's own first piece stops the scan; only a
+        # piece that can ONLY be a title is stepped over.
+        #
+        # Computed once, before the loop: every merge below starts at
+        # some k at or past this index, so no merge can move it.
+        #
+        # Suffix pieces are deliberately NOT skipped, and the reason is
+        # what skipping them WOULD do rather than what it would cost.
+        # A credential written with spaces already parses without a
+        # family name -- "Ph. D. Van Johnson" is given 'Van Johnson',
+        # suffix 'Ph. D.', family '' -- and skipping the suffix piece
+        # would actually give it one (given 'Van', family 'Johnson').
+        # The shapes that decide it are the ones whose leading piece
+        # lands in `given` instead: "Ph.D. Van Johnson", "II Van
+        # Johnson" and "Msc.Ed. Van Johnson" each read given
+        # 'Ph.D.'/'II'/'Msc.Ed.' with family 'Van Johnson', and
+        # skipping the piece moves `Van` out of the family and into the
+        # middle name (given 'Ph.D.', middle 'Van', family 'Johnson')
+        # -- a worse reading, on three shapes, to fix none. "Jr. Van
+        # Johnson", the shape that looks like it needs the skip,
+        # classifies its leading piece as a TITLE and is already
+        # covered here.
+        #
+        # The `, 0` fallback is inert by construction rather than a
+        # default worth testing: it is reached only when every piece is
+        # a title and none is a prefix, and the loop below merges
+        # nothing unless some piece is a prefix.
+        leading = next((k for k in range(len(pieces))
+                        if not title(k) or prefix(k)), 0)
         k = 0
         while k < len(pieces):
-            if k == 0 or not prefix(k):
+            if k == leading or not prefix(k):
                 k += 1
                 continue
             j = k + 1
@@ -212,11 +256,41 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(),
             # The other half of PARTICLE_OR_GIVEN. _assign reports the
             # fork when an ambiguous particle stays a lone leading piece
             # ("Van Johnson" -> given under the default order, family
-            # under FAMILY_FIRST); the chain here takes the
-            # opposite branch whenever a title shifts it off index 0
-            # ("Dr. Van Johnson" -> family "Van Johnson"). A fork whose
-            # two sides are decided in different stages needs an emitter
-            # in each.
+            # under FAMILY_FIRST); the chain here takes the opposite
+            # branch when the particle is not the name's leading piece.
+            # A fork whose two sides are decided in different stages
+            # needs an emitter in each.
+            #
+            # Narrow, and #367 is why. `all(title(x) for x in range(k))`
+            # says every piece ahead of this one is a title, and the
+            # loop skipped k == leading, so `leading` is STRICTLY
+            # before k -- and being before k it is one of those titles,
+            # while being `leading` it satisfies `not title or prefix`.
+            # For both, it must be a prefix as well: a word in both
+            # vocabularies (`st`, `do`, `freiherr` by default, or any
+            # overlap a caller configures). A plain title alone can no
+            # longer put a particle off the name's leading piece; it is
+            # stepped over and _assign reports the fork instead.
+            #
+            # What that leaves is wider than one shape: any number of
+            # plain title pieces, then a piece in BOTH vocabularies,
+            # then any number of further titles, then the ambiguous
+            # particle whose chain claims something. "Freiherr von
+            # Richthofen" is the canonical spelling and the one
+            # tests/v2/cases.py and tests/v2/test_parser.py lead with,
+            # but "St Van Johnson", "Do St Johnson" (the chained
+            # particle itself in both vocabularies) and "Dr. Do van
+            # Johnson" (a plain title AHEAD of the both-vocabulary
+            # word) all reach here too. What none of them can do is
+            # dispense with the both-vocabulary WORD. The conjunction
+            # merge is the only other way a piece acquires `title` or
+            # `prefix`, and it cannot manufacture the pair: it derives
+            # from ONE neighbor, which is the left one whenever there
+            # is a left one, and its right operands are always fresh
+            # pieces (the loop runs left to right, so nothing to the
+            # right has been merged yet). Both tags therefore have to
+            # come from the piece it extends, which bottoms out at a
+            # lone token in both vocabularies.
             #
             # j > k + 1 is what makes this a DECISION rather than a
             # shape: when the next piece is a suffix the inner scan

nameparser/_pipeline/_post_rules.py

@@ -145,10 +145,13 @@ def post_rules(state: ParseState) -> ParseState:
     # particle, so 1b declines and given='de la Vega' stands; #359
     # records that case as working as intended. And the degenerate bare
     # 'de' keeps given='de', having nothing to fold into.
-    # "Sir de Mesnil" is the same guard declining on a chained piece,
-    # but do NOT cite it as a limit this rule means to draw: it reports
-    # given='de Mesnil' with no family at all, which is believed wrong
-    # and is tracked as #367.
+    # "Sir de Mesnil" used to be this guard declining on a chained
+    # piece, reporting given='de Mesnil' with no family at all. That
+    # was never a limit this rule meant to draw, and #367 removed the
+    # chain rather than touching the rule: a title is transparent to
+    # the leading-particle exception, so 'de' is a lone piece again,
+    # this guard fires, and the name reads family='de Mesnil' like the
+    # untitled form.
     # Those two sites are the whole scope, and the MIDDLE position is
     # deliberately not one of them -- which shows: "Mesnil Garcia de"
     # strands middle='de' under FAMILY_FIRST, while under
@@ -169,14 +172,18 @@ def post_rules(state: ParseState) -> ParseState:
     # FAMILY_FIRST the opening piece is the family and the given sits
     # behind it, and reading the role alone let "de Mesnil" split. The
     # single-token test says the same thing in each shape: a particle
-    # group already chained forward is not a lone particle. "Mr. de
-    # Mesnil" is three tokens in two pieces -- the title alone, then
-    # the particle GROUP -- so both sites are two tokens long and 1b
-    # declines on each; the family reading there is rule 1's in the
-    # default order and assign's under a family-first one. Both shapes
-    # then need another name token to fold with, which leaves a
-    # degenerate bare 'de' as it stands rather than inventing a
-    # surname.
+    # group already chained forward is not a lone particle -- the
+    # FAMILY_FIRST "Juan de la Vega" above is what that looks like.
+    # "Mr. de Mesnil" is NOT one, and since #367 not even close to
+    # one: it is three tokens in THREE pieces -- the title, the
+    # particle, the surname -- because a title no longer displaces the
+    # particle out of the leading name position, so nothing chains.
+    # Both sites are one token long, so this guard FIRES and the
+    # family reading is its own. Rule 1 above cannot be what produces
+    # it: rule 1 is gated on `not families`, and 'Mesnil' is already
+    # the family. Both shapes need another name token to fold with,
+    # which leaves a degenerate bare 'de' as it stands rather than
+    # inventing a surname.
     sites = (_leading_name_piece(state, tokens), tuple(givens))
     if len(givens) + len(middles) + len(families) > 1 and any(
             len(site) == 1

nameparser/_types.py

@@ -366,10 +366,15 @@ class AmbiguityKind(StrEnum):
     #: names ("read as a given name") -- that role is whatever
     #: assignment gave it, so it follows ``name_order`` and any
     #: ``script_orders`` entry, which is why the kind cannot name it.
-    #: A particle a title shifted off the front ("Dr. Van Johnson")
-    #: was instead claimed by the prefix chain, and ``detail`` says
-    #: that and names no field at all: grouping runs before roles
-    #: exist, so that text is the same under every order.
+    #: A particle that something ahead of it shifted off the front of
+    #: the name was instead claimed by the prefix chain, and ``detail``
+    #: says that and names no field at all: grouping runs before roles
+    #: exist, so that text is the same under every order. Since #367 a
+    #: plain title is not such a thing -- "Dr. Van Johnson" reads as
+    #: the untitled "Van Johnson" does and takes the first shape --
+    #: and what remains is a leading word that is both a title and a
+    #: particle, so it stays a name piece and the particle behind it is
+    #: genuinely not leading ("Freiherr von Richthofen").
     PARTICLE_OR_GIVEN = "particle-or-given"
     #: A nickname/maiden delimiter opened without closing (or closed
     #: without opening); the text was kept as literal name content, so