Keep a never-given particle attached under any name_order (#359)

AGENTS.md

@@ -133,7 +133,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and
 
 - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family)
 - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on
-- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (under the DEFAULT given-first order a name starting with one is all surname: "de Mesnil" — but that is `name_order`'s half of the sentence, not this set's, and `Policy(name_order=FAMILY_FIRST)` reads the same input as family "de", given "Mesnil"; what the set decides under either order is that a leading particle outside it records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either
+- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reports given `de Mesnil` and no family at all in the default order, which is the same guard declining on a chained piece; that output is believed WRONG and is tracked as #367, so do not cite it as a line this rule means to draw.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either
 - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code)
 - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles
 - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden`

docs/customize.rst

@@ -168,12 +168,14 @@ a suffix only when written with periods:
 ``particles_ambiguous`` is the same idea for surname particles. A
 particle listed there may also be a given name, which is what makes a
 leading one a decision to take; a particle *not* listed there never
-is, so there is nothing to decide. Under the default name order that
-shows up as whether the name has a given name at all: one that starts
-with a listed particle keeps it, while one starting with an unlisted
-particle has no given name — the whole thing is the surname. (Which
-field each piece lands in is ``name_order``'s question, covered
-below.)
+is, so there is nothing to decide. That shows up in what a particle
+standing *alone* at the front of a name does: a listed one is a name
+part in its own right, while an unlisted one pulls the rest of the
+name into the surname and leaves no given name at all. Which field a
+*listed* particle lands in is ``name_order``'s question, covered
+below; an unlisted one opening the name is the surname under every
+order, because a word that can never be a given name leaves the order
+nothing to decide.
 
 .. doctest::
 
@@ -184,10 +186,20 @@ below.)
     >>> parse("de Mesnil").family
     'de Mesnil'
 
+A comma forestalls the question rather than answering it. Writing the
+surname before the comma has already said which words are the surname,
+so a particle at the front of them decides nothing, and whatever
+follows the comma is the given name as usual:
+
+.. doctest::
+
+    >>> parse("de Mesnil, Juan").given   # the comma named the surname
+    'Juan'
+
 If your data never uses ``Van`` as a given name, take it out of the
 ambiguous set: a leading ``van`` is then no decision at all, so no
-ambiguity is recorded, and under the default order it becomes part of
-the surname:
+ambiguity is recorded and it becomes part of the surname — under any
+``name_order``, since that is what taking the word out asserted:
 
 .. doctest::
 
@@ -312,7 +324,8 @@ Family-first name order
 
 ``name_order`` is the one most likely to matter for data that is not
 in Western order. Positional input is assigned in the order you
-declare, so a name written family-first — Hungarian, here — parses as
+declare — with the two vocabulary exceptions noted at the end of this
+section — so a name written family-first — Hungarian, here — parses as
 written instead of needing to be rearranged afterwards:
 
 .. doctest::
@@ -356,14 +369,29 @@ no order of its own — so it applies only where you set it, and there
 is no ``vn`` locale pack yet (issue `#146
 <https://github.com/derek73/python-nameparser/issues/146>`_).
 
-One caution, which is why the example above is not the more obvious
+Two cautions, both places where the vocabulary layer answers before
+``name_order`` is consulted at all.
+
+The first is why the example above is not the more obvious
 ``"Nguyen Van Minh"``: a middle word that is also a shipped particle
-is claimed by the vocabulary layer before ``name_order`` is consulted
-at all. ``Van`` is the Dutch particle ``van``, so that name reads
-family ``Nguyen`` with ``Van Minh`` given under *both* family-first
-orders, and the choice between them makes no difference. `Words that
-are also ordinary names`_ covers dropping such a word from the
-vocabulary.
+is claimed by the vocabulary layer. ``Van`` is the Dutch particle
+``van``, so that name reads family ``Nguyen`` with ``Van Minh`` given
+under *both* family-first orders, and the choice between them makes no
+difference.
+
+The second is at the *front* of a name, and there the vocabulary
+overrides the declared order outright: where a particle that can never
+be a given name stands alone as the opening piece, the whole name is
+the surname, in every ``name_order``. ``"de Mesnil"`` is family ``de
+Mesnil`` under both family-first orders exactly as it is by default,
+not family ``de`` with ``Mesnil`` given — a word that can never be a
+given name leaves the order nothing to decide. Only the never-given
+set does this: ``"van Gogh"`` reads family ``van``, given ``Gogh``
+under a family-first order, because ``van`` *can* be a given name and
+so leaves a real question to answer.
+
+`Words that are also ordinary names`_ covers dropping a word from a
+vocabulary, or moving one between those two sets.
 
 East Asian defaults, and turning them off
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/release_log.rst

@@ -10,9 +10,11 @@ Release Log
     editing one in place as a way to change a default and replaces it
     with configuring a ``Lexicon`` or a private ``Constants``.
 
-    Nothing moved between vocabularies and no parse changes: over the
-    751 names of the differential corpora, every one of the seven
-    fields is identical to 2.1 through both the 2.0 and the 1.x API.
+    Nothing moved between vocabularies and no parse changes in the
+    default name order: over the 751 names of the differential
+    corpora, every one of the seven fields is identical to 2.1 through
+    both the 2.0 and the 1.x API. Both family-first orders do change,
+    below -- the same seven names under each.
     What breaks is code that *writes* to a default word list, and code
     that imports one by its 1.x name has until 3.0.
 
@@ -22,6 +24,8 @@ Release Log
 
     **Behavior Changes**
 
+    - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359)
+
     - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355)
 
     **Deprecations**

docs/usage.rst

@@ -115,13 +115,14 @@ names together as easily as two surnames:
     >>> parse("Juan de la Vega y Rodriguez").family
     'de la Vega y Rodriguez'
 
-Position matters in exactly one place: the start of a name. A particle
-there has no surname to attach to yet, so what decides the reading is
-whether it is one that can double as a given name. Where the pieces
-then land is ``name_order``'s question — see :doc:`customize` — and
-the destinations below are the default given-first order's: the
-particle either becomes the given name or turns the whole name into a
-surname:
+Position matters in exactly one place: a particle standing on its own
+at the start of a name. It has no surname to attach to yet, so what
+decides the reading is whether it is one that can double as a given
+name: the particle either becomes the given name or turns the whole
+name into a surname. Only the first of those is ``name_order``'s
+question — see :doc:`customize`, and read the given name below as the
+default given-first order's — since a particle that can never be a
+given name is the surname whatever order you declare:
 
 .. doctest::
 
@@ -132,6 +133,15 @@ surname:
     >>> parse("de Mesnil").family
     'de Mesnil'
 
+A comma gets there first. It names the surname outright, so a particle
+opening that surname has nothing left to decide and the part after the
+comma is the given name:
+
+.. doctest::
+
+    >>> parse("de Mesnil, Juan").given
+    'Juan'
+
 :doc:`customize` covers how to change which words are in each of these
 sets, including which particles may double as given names. One shipped
 vocabulary works the other way round and so is not in the table above:

nameparser/_lexicon.py

@@ -338,14 +338,15 @@ class Lexicon:
     #: and never consults this set, so it leaves a leading particle a
     #: piece of its own whether listed or not -- "de Mesnil" groups
     #: into two pieces exactly as "van Gogh" does. What membership
-    #: decides is what becomes of that piece afterwards. Under EITHER
+    #: decides is what becomes of that piece afterwards. Under ANY
     #: ``name_order`` a member records a particle-or-given ambiguity
-    #: and a non-member records none; under the default given-first
-    #: order a non-member is additionally folded back into the family
-    #: name once roles exist, so the whole name is the surname ("de
-    #: Mesnil" -- a bare "de", with nothing to fold into, is left
-    #: alone). Which field each piece lands in is ``name_order``'s
-    #: question, not this set's.
+    #: and a non-member records none, and a non-member is additionally
+    #: folded back into the family name once roles exist, so the whole
+    #: name is the surname ("de Mesnil" -- a bare "de", with nothing to
+    #: fold into, is left alone). That fold is order-independent too
+    #: (#359): a word that can never be a given name leaves
+    #: ``name_order`` nothing to decide. Which field a MEMBER's piece
+    #: lands in is ``name_order``'s question, not this set's.
     #: No constant of its own -- the default derives
     #: as particles minus
     #: :data:`~nameparser.config.particles.NON_GIVEN_NAME_PARTICLES`