From dacc700bf648e582a2ac514b6549e66e765de192 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 14:44:40 -0700 Subject: [PATCH 1/7] Key the leading-particle fold on position, not on the GIVEN role (#359) post_rules rule 1b folds a name that opens with a particle that is never a given name into the family: "de Mesnil" is all surname. It asked for that particle by ROLE -- the single GIVEN token -- and under Policy(name_order=FAMILY_FIRST) the opening particle is already FAMILY and the GIVEN role belongs to the token behind it. So the tag test inspected the wrong word, found no particle, and declined: family 'de', given 'Mesnil'. The rule's intent is order-agnostic and its implementation was not. The decision recorded in #359 is that a never-given particle keeps its particle whatever order the caller declared: a word that can never be a given name leaves name_order nothing to place, so declaring family-first is not a reason to make 'de' a surname on its own. Only the trigger moves; the fold itself (givens and middles into family) was already written in roles rather than positions and reads correctly under any order. _leading_name_piece reads the opening name piece off `pieces` instead: the first piece carrying a name role, skipping the titles and group-flagged suffixes assign already peeled, in segment 0 -- or in segment 1 under a family comma, where segment 0 is fixed as the surname and the name continues after the comma. That last clause is what keeps the family-comma reading identical rather than gating it off: master folds 'Smith, de Mesnil' into family 'Smith de Mesnil' TODAY, in the default order, so a NO_COMMA-only gate would have been a default-order change, not a guard against one. The two conditions the old trigger carried survive in position form. A single-token piece is what `len(givens) == 1` was saying: a particle group already chained forward ('Mr. de Mesnil' is one piece) is not a lone leading particle, and a title makes the particle non-leading, so the chain has it. And there must be another name token to fold, which leaves a degenerate bare 'de' as it stands rather than inventing a surname. Every default-order parse is unchanged, which is the property the change turns on: the differential harness reports byte-identical output over all 751 corpus names against 1.4.0, 2.0.0 and 2.1.0 alike. One shape moves the other way, and its test says so: 'Mesnil de' under FAMILY_FIRST put a bare 'de' in the given POSITION, which the old role test folded. The rule is about a leading particle and 'Mesnil' is not one, so the position-keyed form declines it. Co-Authored-By: Claude Opus 5 --- nameparser/_pipeline/_post_rules.py | 48 ++++++++++++-- tests/v2/pipeline/test_post_rules.py | 95 +++++++++++++++++++++++++++- 2 files changed, 135 insertions(+), 8 deletions(-) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 40c0ac54..66318888 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -9,6 +9,10 @@ 1. v1 handle_firstnames: when the parse is exactly a title plus ONE given token (no other roles), and the title is not a given-name title ('Sir'), that token is a family name -- "Mr. Johnson". +1b. a lone never-given particle OPENING the name folds the rest of it + into the family -- "de la Vega". Alone among these rules it reads + position from `pieces` rather than from the roles assign left, so + it fires the same way under every name_order (#359). 2. EAST_SLAVIC (opt-in): positional GIVEN/MIDDLE/FAMILY each exactly one token, the FAMILY-position token carries an East Slavic patronymic ending, and the MIDDLE-position token does NOT (given + @@ -23,10 +27,12 @@ Both rotations fire only on Structure.NO_COMMA (v1 gates them on `not self._had_comma`): a comma already established the family. -These rules reconstruct token POSITION from roles, which is faithful +The rotations reconstruct token POSITION from roles, which is faithful to v1 only under the default GIVEN_FIRST order; their interaction with other name_order values is an open design question for the locale-pack -work (#270). +work (#270). Rule 1b was the third and was re-keyed on position in +#359, the decision there being that a never-given particle keeps its +particle whatever order the caller declared. """ from __future__ import annotations @@ -53,10 +59,30 @@ r"^(оглу|оглы|оғлу|ўғли|угли|кызы|гызы|қызы|қизи|улы|ұлы|уулу)$", re.I) +_NAME_ROLES = (Role.GIVEN, Role.MIDDLE, Role.FAMILY) + + def _idx(tokens: list[WorkToken], role: Role) -> list[int]: return [i for i, t in enumerate(tokens) if t.role is role] +def _leading_name_piece(state: ParseState, + tokens: list[WorkToken]) -> tuple[int, ...]: + """The piece that OPENS the name, whatever role name_order gave it: + the first name-position piece (titles and group-flagged suffixes + already skipped) of the segment the positional read governs. That is + segment 0, except under a family comma, where segment 0 is already + fixed as the surname and the name continues in segment 1. Empty when + the segment has no name piece at all.""" + seg = 1 if state.structure is Structure.FAMILY_COMMA else 0 + if seg >= len(state.pieces): + return () + for piece in state.pieces[seg]: + if any(tokens[i].role in _NAME_ROLES for i in piece): + return piece + return () + + def _retag(tokens: list[WorkToken], i: int, role: Role) -> None: tokens[i] = dataclasses.replace(tokens[i], role=role) @@ -80,11 +106,19 @@ def post_rules(state: ParseState) -> ParseState: # rule 1b: a leading particle that is NEVER a given name means the # whole name is a surname -- fold given (and middles) into family # (v1 handle_non_first_name_prefix; 'de la Vega' -> family, while - # ambiguous 'van Gogh' keeps the given reading). The middle/family - # guard leaves a degenerate bare 'de' as given rather than - # inventing a surname. - if len(givens) == 1 and (middles or families): - gtags = tokens[givens[0]].tags + # ambiguous 'van Gogh' keeps the given reading). Keyed on POSITION, + # not on the GIVEN role (#359): under the default order the opening + # piece IS the given, but under name_order=FAMILY_FIRST it is the + # family and the given sits behind it -- the same input, and a + # never-given particle keeps its particle either way. The + # one-token test is what the role-keyed `len(givens) == 1` was + # saying: a particle that group already chained forward ('Mr. de + # Mesnil') is not a lone leading particle. The second guard wants + # another name token to fold, leaving a degenerate bare 'de' as it + # stands rather than inventing a surname. + head = _leading_name_piece(state, tokens) + if len(head) == 1 and len(givens) + len(middles) + len(families) > 1: + gtags = tokens[head[0]].tags if "particle" in gtags and "vocab:particle-ambiguous" not in gtags: for i in givens + middles: _retag(tokens, i, Role.FAMILY) diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index ca7da24d..c6465825 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -1,7 +1,9 @@ +import pytest + from nameparser._lexicon import Lexicon from nameparser._pipeline import run from nameparser._pipeline._state import ParseState -from nameparser._policy import PatronymicRule, Policy +from nameparser._policy import FAMILY_FIRST, PatronymicRule, Policy from nameparser._types import Role _LEX = Lexicon( @@ -9,6 +11,7 @@ given_name_titles=frozenset({"sir"}), particles=frozenset({"de", "la", "van"}), particles_ambiguous=frozenset({"van"}), + suffix_words=frozenset({"md"}), ) @@ -108,6 +111,96 @@ def test_degenerate_bare_particle_stays_given() -> None: assert not _by_role(out, Role.FAMILY) +_FF = Policy(name_order=FAMILY_FIRST) + + +# --- rule 1b under name_order=FAMILY_FIRST (#359) --------------------- +# The fold keys on POSITION, not on the GIVEN role: a never-given +# particle keeps its particle whatever name_order says. + +@pytest.mark.parametrize("text,family,given,suffix", [ + # the leading particle chains the rest of the name into the family + ("de Mesnil", "de Mesnil", "", ""), + ("de la Vega", "de la Vega", "", ""), + # ... and the trailing suffix run is peeled before the rule looks, + # comma or no comma (NO_COMMA and SUFFIX_COMMA both fold) + ("de Mesnil MD", "de Mesnil", "", "MD"), + ("De Mesnil, MD", "De Mesnil", "", "MD"), +]) +def test_family_first_folds_leading_never_given_particle( + text: str, family: str, given: str, suffix: str) -> None: + out = _parsed(text, _FF) + assert _by_role(out, Role.FAMILY) == family + assert _by_role(out, Role.GIVEN) == given + assert _by_role(out, Role.SUFFIX) == suffix + + +@pytest.mark.parametrize("text,family,given", [ + # a title makes the particle non-leading, so group already chained + # it into one piece -- one name piece, wholly family under FF + ("Mr. de Mesnil", "de Mesnil", ""), + # a family comma has already fixed the family; the post-comma part + # is the given name and must not be folded into it + ("de Mesnil, Juan", "de Mesnil", "Juan"), + # degenerate: nothing to fold, so no surname is invented + ("de", "de", ""), + # the leading piece is not a particle + ("Juan de Mesnil", "Juan", "de Mesnil"), + # 'van' is particles_ambiguous -- out of the rule's scope in EVERY + # order, so FAMILY_FIRST still splits at the particle (#360) + ("van Gogh", "van", "Gogh"), +]) +def test_family_first_leading_particle_cases_that_do_not_fold( + text: str, family: str, given: str) -> None: + out = _parsed(text, _FF) + assert _by_role(out, Role.FAMILY) == family + assert _by_role(out, Role.GIVEN) == given + + +@pytest.mark.parametrize("text,title,given,middle,family,suffix", [ + ("de Mesnil", "", "", "", "de Mesnil", ""), + ("de la Vega", "", "", "", "de la Vega", ""), + ("de Mesnil MD", "", "", "", "de Mesnil", "MD"), + ("De Mesnil, MD", "", "", "", "De Mesnil", "MD"), + ("Mr. de Mesnil", "Mr.", "", "", "de Mesnil", ""), + ("de Mesnil, Juan", "", "Juan", "", "de Mesnil", ""), + ("de", "", "de", "", "", ""), + ("Juan de Mesnil", "", "Juan", "", "de Mesnil", ""), + ("van Gogh", "", "van", "", "Gogh", ""), + # a family comma folds the post-comma part when IT opens with a + # never-given particle -- long-standing behaviour, order-independent + # (assign ignores name_order after a family comma), pinned here so + # the re-key cannot quietly drop it + ("Smith, de Mesnil", "", "", "", "Smith de Mesnil", ""), + ("Smith, van Gogh", "", "van", "Gogh", "Smith", ""), +]) +def test_default_order_is_unchanged_by_the_family_first_fold( + text: str, title: str, given: str, middle: str, family: str, + suffix: str) -> None: + out = _parsed(text) + assert _by_role(out, Role.TITLE) == title + assert _by_role(out, Role.GIVEN) == given + assert _by_role(out, Role.MIDDLE) == middle + assert _by_role(out, Role.FAMILY) == family + assert _by_role(out, Role.SUFFIX) == suffix + + +def test_family_comma_fold_is_order_independent() -> None: + out = _parsed("Smith, de Mesnil", _FF) + assert _by_role(out, Role.FAMILY) == "Smith de Mesnil" + assert not _by_role(out, Role.GIVEN) + + +def test_trailing_particle_is_not_a_leading_particle() -> None: + # consequence of keying on position: under FAMILY_FIRST the + # given-POSITION piece here is the bare 'de', which the role-keyed + # rule used to fold ("Mesnil de" -> family). The rule is about a + # LEADING particle, and 'Mesnil' is not one, so it declines. + out = _parsed("Mesnil de", _FF) + assert _by_role(out, Role.FAMILY) == "Mesnil" + assert _by_role(out, Role.GIVEN) == "de" + + def test_middle_as_family_folds_middles() -> None: # v1 handle_middle_name_as_last, opt-in: middles prepend to family out = _parsed("John Quincy Adams Smith", From 3d02563d618becf7eb8412223d7fff115dc97edd Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 14:44:58 -0700 Subject: [PATCH 2/7] Move the docs that said FAMILY_FIRST splits at the particle (#359) Four sites asserted, correctly until the previous commit, that Policy(name_order=FAMILY_FIRST) reads "de Mesnil" as family 'de', given 'Mesnil'. #354 wrote two of them and #358 scoped the other two to the default order precisely so they would not bless that output. Now that the fold is order-independent, the first two are false and the scoping on the others is over-cautious in the one place a reader is most likely to test it. - config/particles.py: both docstrings. NON_GIVEN_NAME_PARTICLES no longer scopes its "the whole thing is a surname" reading to the default order, and PARTICLES now splits the sentence where the behavior splits -- a leading member is the surname under every order, while a leading particle OUTSIDE the set is genuinely order-dependent and still reads as family 'van', given 'Gogh' under FAMILY_FIRST. - _lexicon.py: particles_ambiguous said a non-member is folded "under the default given-first order". It is folded under all of them; what stays name_order's question is where a MEMBER's piece lands. - AGENTS.md's config-layer entry, same correction, plus the mechanism (the fold keys on position) since that is what a future reader needs to know before touching the rule. - docs/customize.rst and docs/usage.rst: no false claim to fix, but both fenced the reading off to the default order in a sentence that is now about two different things. Each says which half name_order still governs. customize.rst's particles_ambiguous section gets the point of #359 as well: taking a word out of the set now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Every replacement claim was measured by parsing, not reasoned: FAMILY_FIRST gives family 'de Mesnil' and 'de la Vega' with no given name, family 'van' + given 'Gogh' for "van Gogh", family 'de' for the bare "de", and family 'van Gogh' once 'van' is removed from particles_ambiguous. FAMILY_FIRST_GIVEN_LAST agrees with FAMILY_FIRST on all of them. The release log bullet records the FAMILY_FIRST change and the "Mesnil de" shape that moves the other way; the 2.2.0 preamble's "no parse changes" is narrowed to the default order, which is what its own 751-name measurement covers. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/customize.rst | 17 ++++++++------- docs/release_log.rst | 10 ++++++--- docs/usage.rst | 11 +++++----- nameparser/_lexicon.py | 13 ++++++------ nameparser/config/particles.py | 38 ++++++++++++++++++---------------- 6 files changed, 50 insertions(+), 41 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9df9b0b5..3a2e40e5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (under the DEFAULT given-first order a name starting with one is all surname: "de Mesnil" — but that is `name_order`'s half of the sentence, not this set's, and `Policy(name_order=FAMILY_FIRST)` reads the same input as family "de", given "Mesnil"; what the set decides under either order is that a leading particle outside it records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name starting with one is all surname — "de Mesnil" — under EVERY `name_order` since #359: the post_rules fold keys on the token's opening POSITION, not on the GIVEN role it happened to get, so `Policy(name_order=FAMILY_FIRST)` reads "de Mesnil" as the family name too, and the degenerate bare "de" with nothing to fold into still stays as it is. A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under `FAMILY_FIRST` — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under either order is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` diff --git a/docs/customize.rst b/docs/customize.rst index e2a10816..c74cebe7 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -168,12 +168,13 @@ a suffix only when written with periods: ``particles_ambiguous`` is the same idea for surname particles. A particle listed there may also be a given name, which is what makes a leading one a decision to take; a particle *not* listed there never -is, so there is nothing to decide. Under the default name order that -shows up as whether the name has a given name at all: one that starts -with a listed particle keeps it, while one starting with an unlisted -particle has no given name — the whole thing is the surname. (Which -field each piece lands in is ``name_order``'s question, covered -below.) +is, so there is nothing to decide. That shows up as whether the name +has a given name at all: one that starts with a listed particle keeps +it, while one starting with an unlisted particle has no given name — +the whole thing is the surname. Which field a *listed* particle lands +in is ``name_order``'s question, covered below; an unlisted one is the +surname under every order, because a word that can never be a given +name leaves the order nothing to decide. .. doctest:: @@ -186,8 +187,8 @@ below.) If your data never uses ``Van`` as a given name, take it out of the ambiguous set: a leading ``van`` is then no decision at all, so no -ambiguity is recorded, and under the default order it becomes part of -the surname: +ambiguity is recorded and it becomes part of the surname — under any +``name_order``, since that is what taking the word out asserted: .. doctest:: diff --git a/docs/release_log.rst b/docs/release_log.rst index 6ba2086b..0be92053 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -10,9 +10,11 @@ Release Log editing one in place as a way to change a default and replaces it with configuring a ``Lexicon`` or a private ``Constants``. - Nothing moved between vocabularies and no parse changes: over the - 751 names of the differential corpora, every one of the seven - fields is identical to 2.1 through both the 2.0 and the 1.x API. + Nothing moved between vocabularies and no parse changes in the + default name order: over the 751 names of the differential + corpora, every one of the seven fields is identical to 2.1 through + both the 2.0 and the 1.x API. One non-default order does change, + below. What breaks is code that *writes* to a default word list, and code that imports one by its 1.x name has until 3.0. @@ -22,6 +24,8 @@ Release Log **Behavior Changes** + - Fix a name opening with a particle that is *never* a given name being split at the particle under ``Policy(name_order=FAMILY_FIRST)``: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule that folds a leading never-given particle into the family keyed on the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined; it now keys on the position that opens the name and fires under every ``name_order``. The decision behind that: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under ``FAMILY_FIRST``. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. One shape moves the other way, on the same re-key: a bare never-given particle in the *given* position rather than the leading one -- ``"Mesnil de"`` under ``FAMILY_FIRST`` -- was folded by the old role test and now reads as family ``Mesnil``, given ``de``. Default-order output is byte-identical over all 751 corpus names, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) + - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) **Deprecations** diff --git a/docs/usage.rst b/docs/usage.rst index 59007b76..37bc6e2a 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -117,11 +117,12 @@ names together as easily as two surnames: Position matters in exactly one place: the start of a name. A particle there has no surname to attach to yet, so what decides the reading is -whether it is one that can double as a given name. Where the pieces -then land is ``name_order``'s question — see :doc:`customize` — and -the destinations below are the default given-first order's: the -particle either becomes the given name or turns the whole name into a -surname: +whether it is one that can double as a given name: the particle either +becomes the given name or turns the whole name into a surname. Only +the first of those is ``name_order``'s question — see +:doc:`customize`, and read the given name below as the default +given-first order's — since a particle that can never be a given name +is the surname whatever order you declare: .. doctest:: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index c8611df1..ef273924 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -340,12 +340,13 @@ class Lexicon: #: into two pieces exactly as "van Gogh" does. What membership #: decides is what becomes of that piece afterwards. Under EITHER #: ``name_order`` a member records a particle-or-given ambiguity - #: and a non-member records none; under the default given-first - #: order a non-member is additionally folded back into the family - #: name once roles exist, so the whole name is the surname ("de - #: Mesnil" -- a bare "de", with nothing to fold into, is left - #: alone). Which field each piece lands in is ``name_order``'s - #: question, not this set's. + #: and a non-member records none, and a non-member is additionally + #: folded back into the family name once roles exist, so the whole + #: name is the surname ("de Mesnil" -- a bare "de", with nothing to + #: fold into, is left alone). That fold is order-independent too + #: (#359): a word that can never be a given name leaves + #: ``name_order`` nothing to decide. Which field a MEMBER's piece + #: lands in is ``name_order``'s question, not this set's. #: No constant of its own -- the default derives #: as particles minus #: :data:`~nameparser.config.particles.NON_GIVEN_NAME_PARTICLES` diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index 7d751185..193ceb10 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -2,14 +2,15 @@ from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES #: The sub-set of :py:data:`PARTICLES` that are *never* a standalone given -#: name. Under the default given-first order that means a name *starting* -#: with one of these has no given name -- the whole thing is a surname -#: (e.g. "de Mesnil" -> family name "de Mesnil"). The reading is scoped to -#: the order on purpose: ``Policy(name_order=FAMILY_FIRST)`` parses the -#: same input as family "de", given "Mesnil", because which side of a -#: leading particle the family name sits on is ``name_order``'s question, -#: not this set's. What membership decides under either order is the -#: ambiguity report -- see :py:data:`PARTICLES` below. +#: name. A name *starting* with one of these has no given name -- the +#: whole thing is a surname (e.g. "de Mesnil" -> family name "de Mesnil") +#: -- and that reading holds under EVERY ``name_order`` (#359). It is not +#: scoped to the default order the way the rest of the positional read is: +#: ``name_order`` says which side of the name the family sits on, and a +#: word that can never be a given name leaves it nothing to decide, so +#: ``Policy(name_order=FAMILY_FIRST)`` reads "de Mesnil" as the family +#: name too. Membership also decides the ambiguity report -- see +#: :py:data:`PARTICLES` below. #: Curated to exclude anything that can be a given name in some culture #: (`al`, `van`, `von`, `della`, `di`, `del`, `da`, `vander`, ...) and #: anything that is also a bound given-name particle (`abu`). When unsure, @@ -86,16 +87,17 @@ #: particle is the exception and chains nothing: the chain skips the #: first piece unconditionally, membership in this set or any other #: never entering into it. Where the pieces then land is again a later -#: question, -#: and this one is ``name_order``'s: under the default given-first order -#: a leading :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole -#: name a family name ("de la Vega"), while a leading particle outside -#: that set is read as the given name ("Van Johnson") -- whereas -#: ``Policy(name_order=FAMILY_FIRST)`` splits both at the leading -#: particle alike ("de la Vega" -> family "de", given "la Vega"; "Van -#: Johnson" -> family "Van", given "Johnson"), which is the same -#: chains-nothing grouping read the other way round. What membership -#: decides under EITHER order is the report: a leading particle outside +#: question, and this is where membership decides something: a leading +#: :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole name a +#: family name ("de la Vega") under every ``name_order``, because a word +#: that is never a given name leaves the order nothing to place. A +#: leading particle OUTSIDE that set could be either, so there +#: ``name_order`` decides after all: the default given-first order reads +#: it as the given name ("Van Johnson"), while +#: ``Policy(name_order=FAMILY_FIRST)`` splits the same chains-nothing +#: grouping the other way round ("Van Johnson" -> family "Van", given +#: "Johnson"). What membership decides under EITHER order is also the +#: report: a leading particle outside #: :py:data:`NON_GIVEN_NAME_PARTICLES` records a particle-or-given #: ambiguity for the reading not taken, and one inside it records none. #: From 3e0c3c351b9c72cde668c7e92b377e256ac7ff53 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 14:50:32 -0700 Subject: [PATCH 3/7] State rule 1b as the invariant it enforces, not as a leading-particle rule (#359) The re-key in dacc700 traded one shape of a single invariant for another. Keying on the opening position fixed "de Mesnil" under FAMILY_FIRST and lost "Mesnil de", where a bare 'de' sits in the given POSITION -- the trailing piece under that order -- and used to fold by the role test. The branch reported given='de' for it, which contradicts the vocabulary the rule consults: NON_GIVEN_NAME_PARTICLES exists to say that word is never a standalone given name. The two shapes are not two rules. The invariant is: a never-given particle is never REPORTED as the given name and the repair is identical in both -- given and middles join the family. Opening the name, the particle pulls the rest of it in ("de la Vega"); left alone in the given position, it folds into the family beside it ("Mesnil de"). That is why the old role test caught the trailing case seemingly by accident: under the default order the opening piece IS the given, so one test covered both. Only under a family-first order do the two come apart, and both need asking. So the rule now asks at both sites -- the opening piece from `pieces`, and the given-position tokens -- with the single-token and another-name-token-to-fold-with guards shared. The comment and the stage header state the invariant first and the two shapes under it, rather than presenting the second site as a legacy arm. 'Mesnil de' is pinned under BOTH orders, with a comment saying what it protects: it is the shape a refactor reading the rule as leading-particle-only drops, silently and under a non-default order. Verified rather than asserted, since "an extra site can only add firings" is exactly the kind of claim that is true until it isn't: the differential harness reports byte-identical output to master over all 751 corpus names at 1.4.0, 2.0.0 and 2.1.0. Against the branch's own pre-restore measurements, one line of the 34-name probe matrix moves -- 'Mesnil de' under FAMILY_FIRST, from family='Mesnil' given='de' back to family='Mesnil de' -- and nothing else, in either order. Prose that had inherited the leading-only framing moves with it: config/particles.py's NON_GIVEN_NAME_PARTICLES docstring, AGENTS.md's config-layer entry (which now records the misreading itself, since it is how the bug got in), and the release-log bullet, which no longer announces the 'Mesnil de' regression it no longer has. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/release_log.rst | 2 +- nameparser/_pipeline/_post_rules.py | 72 +++++++++++++++++----------- nameparser/config/particles.py | 6 ++- tests/v2/pipeline/test_post_rules.py | 23 ++++++--- 5 files changed, 66 insertions(+), 39 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3a2e40e5..c140264d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name starting with one is all surname — "de Mesnil" — under EVERY `name_order` since #359: the post_rules fold keys on the token's opening POSITION, not on the GIVEN role it happened to get, so `Policy(name_order=FAMILY_FIRST)` reads "de Mesnil" as the family name too, and the degenerate bare "de" with nothing to fold into still stays as it is. A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under `FAMILY_FIRST` — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under either order is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name starting with one is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. The invariant post_rules rule 1b actually enforces is one line wider than that, and reading it as leading-only is how the FAMILY_FIRST bug got in: a member is never *reported* as the given name. Two shapes, one repair — opening the name it pulls the rest in, and left ALONE in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under `FAMILY_FIRST` — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under either order is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` diff --git a/docs/release_log.rst b/docs/release_log.rst index 0be92053..134e8d11 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -24,7 +24,7 @@ Release Log **Behavior Changes** - - Fix a name opening with a particle that is *never* a given name being split at the particle under ``Policy(name_order=FAMILY_FIRST)``: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule that folds a leading never-given particle into the family keyed on the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined; it now keys on the position that opens the name and fires under every ``name_order``. The decision behind that: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under ``FAMILY_FIRST``. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. One shape moves the other way, on the same re-key: a bare never-given particle in the *given* position rather than the leading one -- ``"Mesnil de"`` under ``FAMILY_FIRST`` -- was folded by the old role test and now reads as family ``Mesnil``, given ``de``. Default-order output is byte-identical over all 751 corpus names, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) + - Fix a name opening with a particle that is *never* a given name being split at the particle under ``Policy(name_order=FAMILY_FIRST)``: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing that such a particle is never *reported* as the given name asked for it by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same invariant are caught: a particle opening the name pulls the rest of it into the family, and a particle left alone in the given position folds into the family beside it. The decision behind it: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under ``FAMILY_FIRST``. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Default-order output is byte-identical over all 751 corpus names, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 66318888..b1371984 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -9,10 +9,12 @@ 1. v1 handle_firstnames: when the parse is exactly a title plus ONE given token (no other roles), and the title is not a given-name title ('Sir'), that token is a family name -- "Mr. Johnson". -1b. a lone never-given particle OPENING the name folds the rest of it - into the family -- "de la Vega". Alone among these rules it reads +1b. a particle that is never a given name is never reported as one: + opening the name it pulls the rest of it into the family ("de la + Vega"), and left alone in the given position it folds into the + family beside it. Alone among these rules it reads the opening position from `pieces` rather than from the roles assign left, so - it fires the same way under every name_order (#359). + the first shape holds under every name_order (#359). 2. EAST_SLAVIC (opt-in): positional GIVEN/MIDDLE/FAMILY each exactly one token, the FAMILY-position token carries an East Slavic patronymic ending, and the MIDDLE-position token does NOT (given + @@ -30,9 +32,10 @@ The rotations reconstruct token POSITION from roles, which is faithful to v1 only under the default GIVEN_FIRST order; their interaction with other name_order values is an open design question for the locale-pack -work (#270). Rule 1b was the third and was re-keyed on position in -#359, the decision there being that a never-given particle keeps its -particle whatever order the caller declared. +work (#270). Rule 1b read its particle the same way until #359 gave +it the position test as well, the decision there being that a +never-given particle keeps its particle whatever order the caller +declared. """ from __future__ import annotations @@ -103,29 +106,40 @@ def post_rules(state: ParseState) -> ParseState: for i in givens: _retag(tokens, i, Role.FAMILY) - # rule 1b: a leading particle that is NEVER a given name means the - # whole name is a surname -- fold given (and middles) into family - # (v1 handle_non_first_name_prefix; 'de la Vega' -> family, while - # ambiguous 'van Gogh' keeps the given reading). Keyed on POSITION, - # not on the GIVEN role (#359): under the default order the opening - # piece IS the given, but under name_order=FAMILY_FIRST it is the - # family and the given sits behind it -- the same input, and a - # never-given particle keeps its particle either way. The - # one-token test is what the role-keyed `len(givens) == 1` was - # saying: a particle that group already chained forward ('Mr. de - # Mesnil') is not a lone leading particle. The second guard wants - # another name token to fold, leaving a degenerate bare 'de' as it - # stands rather than inventing a surname. - head = _leading_name_piece(state, tokens) - if len(head) == 1 and len(givens) + len(middles) + len(families) > 1: - gtags = tokens[head[0]].tags - if "particle" in gtags and "vocab:particle-ambiguous" not in gtags: - for i in givens + middles: - _retag(tokens, i, Role.FAMILY) - # downstream rules key on the role counts: recompute - givens = _idx(tokens, Role.GIVEN) - middles = _idx(tokens, Role.MIDDLE) - families = _idx(tokens, Role.FAMILY) + # rule 1b enforces one invariant (v1 handle_non_first_name_prefix): + # a particle that is NEVER a given name is never REPORTED as the + # given name. It reaches the parse in two shapes, and the repair is + # the same in both -- the given and the middles join the family: + # * the particle OPENS the name, so the whole name is a surname + # and it pulls the rest in -- "de la Vega"; + # * the particle is left ALONE in the given position, so it folds + # into the family beside it -- "Mesnil de" under + # name_order=FAMILY_FIRST, where the given position is the + # trailing piece. + # Only a never-given particle is in scope: ambiguous 'van Gogh' + # keeps its given reading, and #360 tracks the vocabulary line. + # The opening shape is read from `pieces` rather than from the role + # assign left (#359). Under the default order the opening piece IS + # the given, so the one role test used to catch both shapes; under + # FAMILY_FIRST the opening piece is the family and the given sits + # behind it, and reading the role alone let "de Mesnil" split. The + # single-token test says the same thing in each shape: a particle + # group already chained forward ('Mr. de Mesnil' is one piece) is + # not a lone particle. Both shapes then need another name token to + # fold with, which leaves a degenerate bare 'de' as it stands + # rather than inventing a surname. + sites = (_leading_name_piece(state, tokens), tuple(givens)) + if len(givens) + len(middles) + len(families) > 1 and any( + len(site) == 1 + and "particle" in tokens[site[0]].tags + and "vocab:particle-ambiguous" not in tokens[site[0]].tags + for site in sites): + for i in givens + middles: + _retag(tokens, i, Role.FAMILY) + # downstream rules key on the role counts: recompute + givens = _idx(tokens, Role.GIVEN) + middles = _idx(tokens, Role.MIDDLE) + families = _idx(tokens, Role.FAMILY) # v1 gates both rotations on `not self._had_comma`; the # middle_as_family fold below runs comma or not (v1 order: diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index 193ceb10..d269d7b7 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -9,7 +9,11 @@ #: ``name_order`` says which side of the name the family sits on, and a #: word that can never be a given name leaves it nothing to decide, so #: ``Policy(name_order=FAMILY_FIRST)`` reads "de Mesnil" as the family -#: name too. Membership also decides the ambiguity report -- see +#: name too. Leading is only the commonest way that comes up: what the +#: parse guarantees is that a member is never *reported* as the given +#: name, so one that lands alone in the given position under a +#: non-default order folds into the family beside it instead. +#: Membership also decides the ambiguity report -- see #: :py:data:`PARTICLES` below. #: Curated to exclude anything that can be a given name in some culture #: (`al`, `van`, `von`, `della`, `di`, `del`, `da`, `vander`, ...) and diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index c6465825..e0dff7f7 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -191,14 +191,23 @@ def test_family_comma_fold_is_order_independent() -> None: assert not _by_role(out, Role.GIVEN) -def test_trailing_particle_is_not_a_leading_particle() -> None: - # consequence of keying on position: under FAMILY_FIRST the - # given-POSITION piece here is the bare 'de', which the role-keyed - # rule used to fold ("Mesnil de" -> family). The rule is about a - # LEADING particle, and 'Mesnil' is not one, so it declines. +def test_lone_never_given_particle_in_given_position_folds() -> None: + # The invariant is "never REPORTED as the given name", which the + # opening-position test alone does not carry: under FAMILY_FIRST + # the given position is the TRAILING piece, and a bare 'de' landing + # there has to fold into the family beside it or the parse + # contradicts the very set that says 'de' is never a given name. + # Guarded here because a refactor that reads the rule as + # leading-particle-only drops exactly this shape, silently and + # under a non-default order (#359 review). out = _parsed("Mesnil de", _FF) - assert _by_role(out, Role.FAMILY) == "Mesnil" - assert _by_role(out, Role.GIVEN) == "de" + assert _by_role(out, Role.FAMILY) == "Mesnil de" + assert not _by_role(out, Role.GIVEN) + # the default order reaches the invariant from the other side: the + # particle is already the family, so there is nothing to repair + default = _parsed("Mesnil de") + assert _by_role(default, Role.GIVEN) == "Mesnil" + assert _by_role(default, Role.FAMILY) == "de" def test_middle_as_family_folds_middles() -> None: From 4db3ca6908448fb2d9bfd1c30d06855ba2b691a6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 15:28:46 -0700 Subject: [PATCH 4/7] Cover the shapes the fold's guards actually depend on (#359) Four documents claim rule 1b for every `name_order` and one order was never parsed. `FAMILY_FIRST_GIVEN_LAST` folds a leading never-given particle exactly as `FAMILY_FIRST` does -- measured over the whole differential corpus, the same seven names move under each -- so the family-first cases are parametrized over both orders rather than duplicated. `de Mesnil Garcia` joins them: it is one of those seven, and the only no-comma family-first shape whose fold has a MIDDLE to move as well as a given. `_leading_name_piece`'s skip loop had no test. Two mutations survived the entire suite -- dropping its name-role filter, and reading only `pieces[seg][:1]` -- because the one case with a title in front, `Mr. de Mesnil`, chains the particle into a two-token piece where the rule declines with or without the skip. A mid-name suffix word breaks that chain: `Dr. de MD Mesnil` leaves the particle a lone piece behind the title piece, and both mutations then split it into given `MD`, middle `Mesnil`, family `de`. Both now fail. Every other case rides on the fixture lexicon's `de`. The rule is claimed of the whole never-given class, so sweep it off the live `Lexicon.default()` in all three orders, the way `test_properties.py` already sweeps `particles_ambiguous` -- asserting the behaviour, not the membership, so an addition to the set is covered the day it lands. Rule 1 retags GIVEN->FAMILY without recomputing the role index lists, so `givens`/`middles`/`families` are stale by the time rule 1b reads them: over the 751 corpus names in four policies, that arm fires 48 times and the lists are stale at 1b on all 48. Harmless today -- 1b fires on none of them -- but it is the shape of the bug this branch fixes, a guard reading a token list that no longer means what its name says. Recompute, as 1b already does after its own fold. Parse output is byte-identical afterwards in all three orders over the corpus. Co-Authored-By: Claude Opus 5 --- nameparser/_pipeline/_post_rules.py | 11 +++ tests/v2/pipeline/test_post_rules.py | 121 ++++++++++++++++++++++----- 2 files changed, 111 insertions(+), 21 deletions(-) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index b1371984..73b5a070 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -105,6 +105,17 @@ def post_rules(state: ParseState) -> ParseState: if joined not in state.lexicon.given_name_titles: for i in givens: _retag(tokens, i, Role.FAMILY) + # every rule below reads these lists; recompute them the way + # 1b does after its own fold, so no guard can inspect a name + # that has already moved. Measured harmless today -- over the + # 751 differential names in four policies this arm fires 48 + # times, and 1b fires on none of them -- but reading a stale + # token list is the shape of the bug #359 fixed. `middles` + # is empty by the guard above and recomputed anyway, so + # relaxing that guard cannot leave it stale. + givens = _idx(tokens, Role.GIVEN) + middles = _idx(tokens, Role.MIDDLE) + families = _idx(tokens, Role.FAMILY) # rule 1b enforces one invariant (v1 handle_non_first_name_prefix): # a particle that is NEVER a given name is never REPORTED as the diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index e0dff7f7..44a44a7d 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -3,7 +3,8 @@ from nameparser._lexicon import Lexicon from nameparser._pipeline import run from nameparser._pipeline._state import ParseState -from nameparser._policy import FAMILY_FIRST, PatronymicRule, Policy +from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, + PatronymicRule, Policy) from nameparser._types import Role _LEX = Lexicon( @@ -112,32 +113,69 @@ def test_degenerate_bare_particle_stays_given() -> None: _FF = Policy(name_order=FAMILY_FIRST) +_FFGL = Policy(name_order=FAMILY_FIRST_GIVEN_LAST) +#: Both family-first orders, since the rule is claimed of every one of +#: them and only one was ever parsed. They differ in where the given +#: name lands behind the family, which is exactly what the leading +#: shape must not depend on; the cases below fold identically under +#: both. +_FAMILY_FIRST = [pytest.param(_FF, id="FAMILY_FIRST"), + pytest.param(_FFGL, id="FAMILY_FIRST_GIVEN_LAST")] -# --- rule 1b under name_order=FAMILY_FIRST (#359) --------------------- + +# --- rule 1b under the family-first orders (#359) --------------------- # The fold keys on POSITION, not on the GIVEN role: a never-given # particle keeps its particle whatever name_order says. +@pytest.mark.parametrize("policy", _FAMILY_FIRST) @pytest.mark.parametrize("text,family,given,suffix", [ # the leading particle chains the rest of the name into the family ("de Mesnil", "de Mesnil", "", ""), ("de la Vega", "de la Vega", "", ""), + # three pieces, so the fold has a MIDDLE to move as well as the + # given -- the `givens + middles` half of the repair, and the only + # no-comma corpus name that reaches it + ("de Mesnil Garcia", "de Mesnil Garcia", "", ""), # ... and the trailing suffix run is peeled before the rule looks, # comma or no comma (NO_COMMA and SUFFIX_COMMA both fold) ("de Mesnil MD", "de Mesnil", "", "MD"), ("De Mesnil, MD", "De Mesnil", "", "MD"), ]) def test_family_first_folds_leading_never_given_particle( - text: str, family: str, given: str, suffix: str) -> None: - out = _parsed(text, _FF) + policy: Policy, text: str, family: str, given: str, + suffix: str) -> None: + out = _parsed(text, policy) assert _by_role(out, Role.FAMILY) == family assert _by_role(out, Role.GIVEN) == given assert _by_role(out, Role.SUFFIX) == suffix +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_leading_piece_scan_skips_pieces_that_hold_no_name( + policy: Policy) -> None: + # `_leading_name_piece` walks PAST pieces carrying no name role + # rather than reading piece 0 -- and past the first such piece, not + # only over a single title. 'Mr. de Mesnil' cannot show that: its + # particle is chained into one piece with 'Mesnil', so the scan + # lands on a two-token piece and the rule declines either way. + # Here a mid-name suffix word breaks that chain, leaving the + # particle a piece of its own BEHIND a title piece. Without the + # skip, or reading only pieces[0], the scan finds the title (or + # nothing) and the name splits: given='MD', middle='Mesnil', + # family='de'. + out = _parsed("Dr. de MD Mesnil", policy) + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.FAMILY) == "de MD Mesnil" + assert not _by_role(out, Role.GIVEN) + assert not _by_role(out, Role.MIDDLE) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) @pytest.mark.parametrize("text,family,given", [ # a title makes the particle non-leading, so group already chained - # it into one piece -- one name piece, wholly family under FF + # it into one piece -- one name piece, wholly family under both + # family-first orders ("Mr. de Mesnil", "de Mesnil", ""), # a family comma has already fixed the family; the post-comma part # is the given name and must not be folded into it @@ -147,12 +185,12 @@ def test_family_first_folds_leading_never_given_particle( # the leading piece is not a particle ("Juan de Mesnil", "Juan", "de Mesnil"), # 'van' is particles_ambiguous -- out of the rule's scope in EVERY - # order, so FAMILY_FIRST still splits at the particle (#360) + # order, so a family-first order still splits at the particle (#360) ("van Gogh", "van", "Gogh"), ]) def test_family_first_leading_particle_cases_that_do_not_fold( - text: str, family: str, given: str) -> None: - out = _parsed(text, _FF) + policy: Policy, text: str, family: str, given: str) -> None: + out = _parsed(text, policy) assert _by_role(out, Role.FAMILY) == family assert _by_role(out, Role.GIVEN) == given @@ -160,6 +198,8 @@ def test_family_first_leading_particle_cases_that_do_not_fold( @pytest.mark.parametrize("text,title,given,middle,family,suffix", [ ("de Mesnil", "", "", "", "de Mesnil", ""), ("de la Vega", "", "", "", "de la Vega", ""), + ("de Mesnil Garcia", "", "", "", "de Mesnil Garcia", ""), + ("Dr. de MD Mesnil", "Dr.", "", "", "de MD Mesnil", ""), ("de Mesnil MD", "", "", "", "de Mesnil", "MD"), ("De Mesnil, MD", "", "", "", "De Mesnil", "MD"), ("Mr. de Mesnil", "Mr.", "", "", "de Mesnil", ""), @@ -185,31 +225,70 @@ def test_default_order_is_unchanged_by_the_family_first_fold( assert _by_role(out, Role.SUFFIX) == suffix -def test_family_comma_fold_is_order_independent() -> None: - out = _parsed("Smith, de Mesnil", _FF) +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_family_comma_fold_is_order_independent(policy: Policy) -> None: + out = _parsed("Smith, de Mesnil", policy) assert _by_role(out, Role.FAMILY) == "Smith de Mesnil" assert not _by_role(out, Role.GIVEN) -def test_lone_never_given_particle_in_given_position_folds() -> None: - # The invariant is "never REPORTED as the given name", which the - # opening-position test alone does not carry: under FAMILY_FIRST - # the given position is the TRAILING piece, and a bare 'de' landing - # there has to fold into the family beside it or the parse - # contradicts the very set that says 'de' is never a given name. - # Guarded here because a refactor that reads the rule as - # leading-particle-only drops exactly this shape, silently and - # under a non-default order (#359 review). - out = _parsed("Mesnil de", _FF) +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_lone_never_given_particle_in_given_position_folds( + policy: Policy) -> None: + # The opening-position test alone does not carry the rule: under a + # family-first order the given position is the TRAILING piece, and + # a lone 'de' landing there has to fold into the family beside it + # or the parse leaves the whole given name as a word the vocabulary + # says is never a given name. Guarded here because a refactor that + # reads the rule as leading-particle-only drops exactly this shape, + # silently and under a non-default order (#359 review). + out = _parsed("Mesnil de", policy) assert _by_role(out, Role.FAMILY) == "Mesnil de" assert not _by_role(out, Role.GIVEN) - # the default order reaches the invariant from the other side: the + + +def test_lone_never_given_particle_needs_no_repair_by_default() -> None: + # the default order reaches the same rule from the other side: the # particle is already the family, so there is nothing to repair default = _parsed("Mesnil de") assert _by_role(default, Role.GIVEN) == "Mesnil" assert _by_role(default, Role.FAMILY) == "de" +# --- the whole never-given class, not just the fixture's 'de' --------- + +_ALL_ORDERS = [pytest.param(Policy(), id="GIVEN_FIRST"), *_FAMILY_FIRST] + + +@pytest.mark.parametrize("policy", _ALL_ORDERS) +def test_no_never_given_particle_is_left_as_the_given_name( + policy: Policy) -> None: + """Every case above rides on the fixture lexicon's 'de'. The rule + is claimed of the whole never-given class in every name_order, so + sweep the live class rather than pinning another word or two of it + -- a hardcoded handful would document those entries and catch + nothing else in the set (AGENTS.md, "Prefer behavior tests over + constant-content tests"). Derived from the lexicon, so an addition + to NON_GIVEN_NAME_PARTICLES is swept the day it lands, and asserts + nothing about which words are in the set. + """ + lex = Lexicon.default() + never_given = sorted(lex.particles - lex.particles_ambiguous) + assert never_given, "no never-given particles to exercise" + failures = [] + for particle in never_given: + text = f"{particle} Mesnil" + out = run(ParseState(original=text, lexicon=lex, policy=policy)) + given = _by_role(out, Role.GIVEN) + family = _by_role(out, Role.FAMILY) + if given or family != text: + failures.append( + f"{text!r}: given={given!r} family={family!r}") + assert not failures, ( + f"{len(failures)} of {len(never_given)} left as a given name " + f"or unfolded:\n" + "\n".join(failures[:15])) + + def test_middle_as_family_folds_middles() -> None: # v1 handle_middle_name_as_last, opt-in: middles prepend to family out = _parsed("John Quincy Adams Smith", From 230978388602f24dbb05e10689402490827396d6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 15:33:51 -0700 Subject: [PATCH 5/7] Say what rule 1b enforces, and name both family-first orders (#359) Five documents stated the rule as "a particle that is never a given name is never *reported* as the given name", and this branch's own test file ships four counterexamples: `parse("de").given` is `'de'`, `parse("Sir de Mesnil").given` is `'de Mesnil'` in the default order, and under FAMILY_FIRST `"Juan de la Vega"` reports given `de la Vega` -- which #359 blesses in as many words. What the code enforces, and what the `len(site) == 1` guard says, is one clause narrower: where a never-given particle stands ALONE as a piece, opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family -- as long as there is another name token to fold into. Checked over 3,591 parses (751 corpus names plus 446 synthetic shapes, three orders): the guard holds in 303 of them and the parse has no given and no middle in all 303, against 82 failures of the same check on the pre-fix tree. The release note named one non-default order. Both change, and by the same seven of the 751 corpus names -- a reader on FAMILY_FIRST_GIVEN_LAST would have concluded they were unaffected. Smaller corrections, each re-measured by parsing: - "ambiguous 'van Gogh' keeps its given reading" is default-order only, and sits in the paragraph explaining that the rule stopped being order-scoped. Under either family-first order `van` is the family. - "a name *starting* with one of these has no given name, under EVERY name_order" over-reaches for the same reason `Sir de Mesnil` does. It is the opening PIECE that is asked about. - "'Mr. de Mesnil' is one piece" is false -- it is two, `((0,), (1, 2))`. The intended claim is about the particle group; 1b declines on both sites there, and the family reading is rule 1's in the default order. - the stage header still said `Consumes: tokens (roles assigned)` after this branch made post_rules a consumer of `pieces`, and it read `structure` unannounced before that. - `_leading_name_piece` skips by ROLE, so NICKNAME, MAIDEN and unroled pieces are skipped too, not just "titles and group-flagged suffixes"; and it has two empty exits, not one. - the family-comma test credited `assign` for a fold rule 1b does. - "EITHER order" at three sites: there are three orders. `docs/customize.rst` claimed positional input is assigned in the order you declare, which now has a second exception; it joins the `Nguyen Van Minh` caution already in that section. Not done, deliberately: the "under every name_order" prose in customize.rst and usage.rst sits beside bare `parse(...)` doctests, and a family-first line in each would make the claim executable under `sphinx -b doctest`. The repo's convention is prose plus a unit test rather than a doctest block, and the previous commit's sweep already executes exactly that claim over the whole never-given class in all three orders -- so the doctests stay as the vocabulary examples they are. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/customize.rst | 32 +++++++++---- docs/release_log.rst | 6 +-- nameparser/_lexicon.py | 2 +- nameparser/_pipeline/_post_rules.py | 67 +++++++++++++++++++--------- nameparser/config/particles.py | 35 ++++++++------- tests/v2/pipeline/test_post_rules.py | 8 ++-- 7 files changed, 100 insertions(+), 52 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c140264d..a3ae18b5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name starting with one is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. The invariant post_rules rule 1b actually enforces is one line wider than that, and reading it as leading-only is how the FAMILY_FIRST bug got in: a member is never *reported* as the given name. Two shapes, one repair — opening the name it pulls the rest in, and left ALONE in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under `FAMILY_FIRST` — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under either order is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: `Sir de Mesnil` reports given `de Mesnil` in the default order and `Juan de la Vega` reports given `de la Vega` under `FAMILY_FIRST` — #359 blesses that second one explicitly — because in each the particle chained onto the next word and is not standing alone. A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` diff --git a/docs/customize.rst b/docs/customize.rst index c74cebe7..fe9518b1 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -313,7 +313,8 @@ Family-first name order ``name_order`` is the one most likely to matter for data that is not in Western order. Positional input is assigned in the order you -declare, so a name written family-first — Hungarian, here — parses as +declare — with the two vocabulary exceptions noted at the end of this +section — so a name written family-first — Hungarian, here — parses as written instead of needing to be rearranged afterwards: .. doctest:: @@ -357,14 +358,29 @@ no order of its own — so it applies only where you set it, and there is no ``vn`` locale pack yet (issue `#146 `_). -One caution, which is why the example above is not the more obvious +Two cautions, both places where the vocabulary layer answers before +``name_order`` is consulted at all. + +The first is why the example above is not the more obvious ``"Nguyen Van Minh"``: a middle word that is also a shipped particle -is claimed by the vocabulary layer before ``name_order`` is consulted -at all. ``Van`` is the Dutch particle ``van``, so that name reads -family ``Nguyen`` with ``Van Minh`` given under *both* family-first -orders, and the choice between them makes no difference. `Words that -are also ordinary names`_ covers dropping such a word from the -vocabulary. +is claimed by the vocabulary layer. ``Van`` is the Dutch particle +``van``, so that name reads family ``Nguyen`` with ``Van Minh`` given +under *both* family-first orders, and the choice between them makes no +difference. + +The second is at the *front* of a name, and there the vocabulary +overrides the declared order outright: where a particle that can never +be a given name stands alone as the opening piece, the whole name is +the surname, in every ``name_order``. ``"de Mesnil"`` is family ``de +Mesnil`` under both family-first orders exactly as it is by default, +not family ``de`` with ``Mesnil`` given — a word that can never be a +given name leaves the order nothing to decide. Only the never-given +set does this: ``"van Gogh"`` reads family ``van``, given ``Gogh`` +under a family-first order, because ``van`` *can* be a given name and +so leaves a real question to answer. + +`Words that are also ordinary names`_ covers dropping a word from a +vocabulary, or moving one between those two sets. East Asian defaults, and turning them off ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/release_log.rst b/docs/release_log.rst index 134e8d11..cdb1da00 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -13,8 +13,8 @@ Release Log Nothing moved between vocabularies and no parse changes in the default name order: over the 751 names of the differential corpora, every one of the seven fields is identical to 2.1 through - both the 2.0 and the 1.x API. One non-default order does change, - below. + both the 2.0 and the 1.x API. Both family-first orders do change, + below -- the same seven names under each. What breaks is code that *writes* to a default word list, and code that imports one by its 1.x name has until 3.0. @@ -24,7 +24,7 @@ Release Log **Behavior Changes** - - Fix a name opening with a particle that is *never* a given name being split at the particle under ``Policy(name_order=FAMILY_FIRST)``: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing that such a particle is never *reported* as the given name asked for it by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same invariant are caught: a particle opening the name pulls the rest of it into the family, and a particle left alone in the given position folds into the family beside it. The decision behind it: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under ``FAMILY_FIRST``. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Default-order output is byte-identical over all 751 corpus names, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index ef273924..a2c5d1b8 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -338,7 +338,7 @@ class Lexicon: #: and never consults this set, so it leaves a leading particle a #: piece of its own whether listed or not -- "de Mesnil" groups #: into two pieces exactly as "van Gogh" does. What membership - #: decides is what becomes of that piece afterwards. Under EITHER + #: decides is what becomes of that piece afterwards. Under ANY #: ``name_order`` a member records a particle-or-given ambiguity #: and a non-member records none, and a non-member is additionally #: folded back into the family name once roles exist, so the whole diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 73b5a070..9194cc2d 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -1,6 +1,8 @@ """Stage: post_rules. -Consumes: tokens (roles assigned). +Consumes: tokens (roles assigned), plus pieces and structure -- rule 1b +reads the opening piece of segment 0, or of segment 1 under a family +comma (#359). structure was always read here, for the rotation gate. Produces: tokens with roles adjusted by the post rules. Reads: Policy.patronymic_rules, Policy.middle_as_family; Lexicon.given_name_titles. @@ -9,12 +11,16 @@ 1. v1 handle_firstnames: when the parse is exactly a title plus ONE given token (no other roles), and the title is not a given-name title ('Sir'), that token is a family name -- "Mr. Johnson". -1b. a particle that is never a given name is never reported as one: - opening the name it pulls the rest of it into the family ("de la - Vega"), and left alone in the given position it folds into the - family beside it. Alone among these rules it reads the opening - position from `pieces` rather than from the roles assign left, so - the first shape holds under every name_order (#359). +1b. where a particle that is never a given name stands ALONE as a + piece -- either opening the name or in the given position -- the + name is left with no given name at all: the given and the middles + fold into the family. Opening the name it pulls the rest of it in + ("de la Vega"); in the given position it folds into the family + beside it ("Mesnil de" under a family-first order). Needs another + name token to fold into, so a bare "de" stays as it is. Alone among + these rules it reads the opening position from `pieces` rather than + from the roles assign left, so that shape holds for a lone leading + particle piece under every name_order (#359). 2. EAST_SLAVIC (opt-in): positional GIVEN/MIDDLE/FAMILY each exactly one token, the FAMILY-position token carries an East Slavic patronymic ending, and the MIDDLE-position token does NOT (given + @@ -72,11 +78,15 @@ def _idx(tokens: list[WorkToken], role: Role) -> list[int]: def _leading_name_piece(state: ParseState, tokens: list[WorkToken]) -> tuple[int, ...]: """The piece that OPENS the name, whatever role name_order gave it: - the first name-position piece (titles and group-flagged suffixes - already skipped) of the segment the positional read governs. That is - segment 0, except under a family comma, where segment 0 is already - fixed as the surname and the name continues in segment 1. Empty when - the segment has no name piece at all.""" + the first piece holding a GIVEN, MIDDLE or FAMILY token, in the + segment the positional read governs. Every piece holding none of + those is walked past -- title and suffix pieces, but NICKNAME and + MAIDEN as well, and anything assign left unroled -- and any number + of them, not only a single leading title. The segment is 0, except + under a family comma, where segment 0 is already fixed as the + surname and the name continues in segment 1. Empty on either of + two exits: that segment does not exist, or none of its pieces + holds a name token.""" seg = 1 if state.structure is Structure.FAMILY_COMMA else 0 if seg >= len(state.pieces): return () @@ -118,27 +128,42 @@ def post_rules(state: ParseState) -> ParseState: families = _idx(tokens, Role.FAMILY) # rule 1b enforces one invariant (v1 handle_non_first_name_prefix): - # a particle that is NEVER a given name is never REPORTED as the - # given name. It reaches the parse in two shapes, and the repair is - # the same in both -- the given and the middles join the family: + # where a particle that is NEVER a given name stands ALONE as a + # piece -- either opening the name, or in the given position -- the + # name is left with no given name at all, the given and the middles + # joining the family. Two shapes, one repair: # * the particle OPENS the name, so the whole name is a surname # and it pulls the rest in -- "de la Vega"; # * the particle is left ALONE in the given position, so it folds # into the family beside it -- "Mesnil de" under # name_order=FAMILY_FIRST, where the given position is the # trailing piece. - # Only a never-given particle is in scope: ambiguous 'van Gogh' - # keeps its given reading, and #360 tracks the vocabulary line. + # A lone PIECE is the whole of it, which is a clause narrower than + # "a member is never reported as the given name" -- that reading + # would be false, and #359 blesses the first of its counterexamples + # outright: "Juan de la Vega" under FAMILY_FIRST reports given='de + # la Vega', "Sir de Mesnil" reports given='de Mesnil' in the + # default order, and the degenerate bare 'de' keeps given='de'. In + # each the particle is in a piece with something else, or has + # nothing to fold into. + # Only a never-given particle is in scope: an ambiguous one keeps + # whatever reading name_order gives it -- 'van Gogh' is given + # 'van' in the default order and family 'van' under a family-first + # one -- and #360 tracks the vocabulary line. # The opening shape is read from `pieces` rather than from the role # assign left (#359). Under the default order the opening piece IS # the given, so the one role test used to catch both shapes; under # FAMILY_FIRST the opening piece is the family and the given sits # behind it, and reading the role alone let "de Mesnil" split. The # single-token test says the same thing in each shape: a particle - # group already chained forward ('Mr. de Mesnil' is one piece) is - # not a lone particle. Both shapes then need another name token to - # fold with, which leaves a degenerate bare 'de' as it stands - # rather than inventing a surname. + # group already chained forward is not a lone particle. "Mr. de + # Mesnil" is three tokens in two pieces -- the title alone, then + # the particle GROUP -- so both sites are two tokens long and 1b + # declines on each; the family reading there is rule 1's in the + # default order and assign's under a family-first one. Both shapes + # then need another name token to fold with, which leaves a + # degenerate bare 'de' as it stands rather than inventing a + # surname. sites = (_leading_name_piece(state, tokens), tuple(givens)) if len(givens) + len(middles) + len(families) > 1 and any( len(site) == 1 diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index d269d7b7..54937246 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -2,17 +2,22 @@ from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES #: The sub-set of :py:data:`PARTICLES` that are *never* a standalone given -#: name. A name *starting* with one of these has no given name -- the -#: whole thing is a surname (e.g. "de Mesnil" -> family name "de Mesnil") -#: -- and that reading holds under EVERY ``name_order`` (#359). It is not -#: scoped to the default order the way the rest of the positional read is: -#: ``name_order`` says which side of the name the family sits on, and a -#: word that can never be a given name leaves it nothing to decide, so -#: ``Policy(name_order=FAMILY_FIRST)`` reads "de Mesnil" as the family -#: name too. Leading is only the commonest way that comes up: what the -#: parse guarantees is that a member is never *reported* as the given -#: name, so one that lands alone in the given position under a -#: non-default order folds into the family beside it instead. +#: name. Where one of these stands ALONE as the piece opening a name, that +#: name has no given name -- the whole thing is a surname (e.g. "de Mesnil" +#: -> family name "de Mesnil") -- and that reading holds under EVERY +#: ``name_order`` (#359). It is not scoped to the default order the way the +#: rest of the positional read is: ``name_order`` says which side of the +#: name the family sits on, and a word that can never be a given name +#: leaves it nothing to decide, so ``Policy(name_order=FAMILY_FIRST)`` +#: reads "de Mesnil" as the family name too. What is asked about is the +#: opening *piece*, not the first word of the string: in "Sir de Mesnil" +#: the particle has already chained onto "Mesnil", so it is not standing +#: alone and the default order reads that piece as the given name. +#: Opening the name is only the commonest shape. The rule enforcing it +#: (``post_rules`` rule 1b) reaches a member standing alone as a piece in +#: the given position too, folding it into the family beside it, so that +#: neither shape leaves a given name behind -- as long as there is another +#: name token to fold into. A bare "de" stays as it is. #: Membership also decides the ambiguity report -- see #: :py:data:`PARTICLES` below. #: Curated to exclude anything that can be a given name in some culture @@ -97,10 +102,10 @@ #: that is never a given name leaves the order nothing to place. A #: leading particle OUTSIDE that set could be either, so there #: ``name_order`` decides after all: the default given-first order reads -#: it as the given name ("Van Johnson"), while -#: ``Policy(name_order=FAMILY_FIRST)`` splits the same chains-nothing -#: grouping the other way round ("Van Johnson" -> family "Van", given -#: "Johnson"). What membership decides under EITHER order is also the +#: it as the given name ("Van Johnson"), while either family-first order +#: splits the same chains-nothing grouping the other way round ("Van +#: Johnson" -> family "Van", given "Johnson"). +#: What membership decides under ANY of the three orders is also the #: report: a leading particle outside #: :py:data:`NON_GIVEN_NAME_PARTICLES` records a particle-or-given #: ambiguity for the reading not taken, and one inside it records none. diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 44a44a7d..ad456f60 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -208,9 +208,11 @@ def test_family_first_leading_particle_cases_that_do_not_fold( ("Juan de Mesnil", "", "Juan", "", "de Mesnil", ""), ("van Gogh", "", "van", "", "Gogh", ""), # a family comma folds the post-comma part when IT opens with a - # never-given particle -- long-standing behaviour, order-independent - # (assign ignores name_order after a family comma), pinned here so - # the re-key cannot quietly drop it + # never-given particle -- long-standing behaviour, and rule 1b's + # own doing: assign hands it the same roles in every order (the + # comma already fixed the family), and both of 1b's sites then + # agree, the opening piece of segment 1 and the lone given being + # the same token. Pinned here so the re-key cannot quietly drop it ("Smith, de Mesnil", "", "", "", "Smith de Mesnil", ""), ("Smith, van Gogh", "", "van", "Gogh", "Smith", ""), ]) From 47aa70d9c4810f851fbfb3f1a1c54374b8ab6a7b Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 15:38:09 -0700 Subject: [PATCH 6/7] Scope the last two leading-particle claims, and cite the open questions (#359) `docs/customize.rst` and `docs/usage.rst` carried the same first-word-versus-lone-piece over-reach the previous commit corrected in `config/particles.py` and `AGENTS.md` -- "one starting with an unlisted particle has no given name", "position matters in exactly one place: the start of a name". Both are falsified in the DEFAULT order, so neither needed a family-first parser to catch: `Sir de Mesnil` reports given `de Mesnil`, because the title pushed `de` off the front and it chained onto `Mesnil` there, and `de Mesnil, Juan` reports given `Juan`, because the comma named the surname first. Each page now says standing ALONE at the front, in its own register, and both counter-examples go into the adjacent doctest block rather than into prose -- they are bare `parse(...)` calls in the default order, which is exactly what those blocks already are, so `sphinx -b doctest` now executes the qualification instead of the reader taking it on trust (223 -> 227 doctests). The rule-1b comment now records what the rule does NOT cover, since that is where a reader will ask. MIDDLE is deliberately outside its two sites and shows it -- `Mesnil Garcia de` strands `middle='de'` under FAMILY_FIRST while FAMILY_FIRST_GIVEN_LAST, whose given position is that same trailing piece, folds the whole name to `family='Mesnil Garcia de'` (#365). And how much the fold takes once it fires is the other open question: `de Mesnil Juan` goes wholly to the family in every order, matching the default rather than stopping at the particle group (#364). No logic changed; parse output over the 751 corpus names is identical in all three orders. Co-Authored-By: Claude Opus 5 --- docs/customize.rst | 28 +++++++++++++++++++++------- docs/usage.rst | 27 +++++++++++++++++++-------- nameparser/_pipeline/_post_rules.py | 10 ++++++++++ 3 files changed, 50 insertions(+), 15 deletions(-) diff --git a/docs/customize.rst b/docs/customize.rst index fe9518b1..9bf069a6 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -168,13 +168,14 @@ a suffix only when written with periods: ``particles_ambiguous`` is the same idea for surname particles. A particle listed there may also be a given name, which is what makes a leading one a decision to take; a particle *not* listed there never -is, so there is nothing to decide. That shows up as whether the name -has a given name at all: one that starts with a listed particle keeps -it, while one starting with an unlisted particle has no given name — -the whole thing is the surname. Which field a *listed* particle lands -in is ``name_order``'s question, covered below; an unlisted one is the -surname under every order, because a word that can never be a given -name leaves the order nothing to decide. +is, so there is nothing to decide. That shows up in what a particle +standing *alone* at the front of a name does: a listed one is a name +part in its own right, while an unlisted one pulls the rest of the +name into the surname and leaves no given name at all. Which field a +*listed* particle lands in is ``name_order``'s question, covered +below; an unlisted one opening the name is the surname under every +order, because a word that can never be a given name leaves the order +nothing to decide. .. doctest:: @@ -185,6 +186,19 @@ name leaves the order nothing to decide. >>> parse("de Mesnil").family 'de Mesnil' +Standing alone is the condition, not merely being the first word. +Anything ahead of the particle lets it join the word behind it +instead, and a comma has named the surname before the question is +reached at all — so neither of these is the case above, and both keep +a given name: + +.. doctest:: + + >>> parse("Sir de Mesnil").given # 'de' joined 'Mesnil' + 'de Mesnil' + >>> parse("de Mesnil, Juan").given # the comma settled the surname + 'Juan' + If your data never uses ``Van`` as a given name, take it out of the ambiguous set: a leading ``van`` is then no decision at all, so no ambiguity is recorded and it becomes part of the surname — under any diff --git a/docs/usage.rst b/docs/usage.rst index 37bc6e2a..f69ce5b5 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -115,14 +115,14 @@ names together as easily as two surnames: >>> parse("Juan de la Vega y Rodriguez").family 'de la Vega y Rodriguez' -Position matters in exactly one place: the start of a name. A particle -there has no surname to attach to yet, so what decides the reading is -whether it is one that can double as a given name: the particle either -becomes the given name or turns the whole name into a surname. Only -the first of those is ``name_order``'s question — see -:doc:`customize`, and read the given name below as the default -given-first order's — since a particle that can never be a given name -is the surname whatever order you declare: +Position matters in exactly one place: a particle standing on its own +at the start of a name. It has no surname to attach to yet, so what +decides the reading is whether it is one that can double as a given +name: the particle either becomes the given name or turns the whole +name into a surname. Only the first of those is ``name_order``'s +question — see :doc:`customize`, and read the given name below as the +default given-first order's — since a particle that can never be a +given name is the surname whatever order you declare: .. doctest:: @@ -133,6 +133,17 @@ is the surname whatever order you declare: >>> parse("de Mesnil").family 'de Mesnil' +On its own is the whole of it. Put anything ahead of the particle and +it joins the word behind it as usual, and a comma names the surname +before the question comes up, so both of these keep a given name: + +.. doctest:: + + >>> parse("Sir de Mesnil").given + 'de Mesnil' + >>> parse("de Mesnil, Juan").given + 'Juan' + :doc:`customize` covers how to change which words are in each of these sets, including which particles may double as given names. One shipped vocabulary works the other way round and so is not in the table above: diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 9194cc2d..02711797 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -146,6 +146,16 @@ def post_rules(state: ParseState) -> ParseState: # default order, and the degenerate bare 'de' keeps given='de'. In # each the particle is in a piece with something else, or has # nothing to fold into. + # Those two sites are the whole scope, and the MIDDLE position is + # deliberately not one of them -- which shows: "Mesnil Garcia de" + # strands middle='de' under FAMILY_FIRST, while under + # FAMILY_FIRST_GIVEN_LAST the same trailing piece IS the given + # position, so it folds to family='Mesnil Garcia de'. Whether that + # difference should stand is #365, not this rule's to settle. How + # much the fold takes once it fires is the other open question: + # "de Mesnil Juan" goes wholly to the family in every order, + # matching the default rather than stopping at the particle group + # (#364). # Only a never-given particle is in scope: an ambiguous one keeps # whatever reading name_order gives it -- 'van Gogh' is given # 'van' in the default order and family 'van' under a family-first From 990e8f34ec321592b759b4f929b7e9fece8ffd44 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 18:14:43 -0700 Subject: [PATCH 7/7] Stop documenting the Sir de Mesnil reading as intended (#359, #367) The previous commit scoped two doc sites with a pair of counter-examples, and pinned both as doctests. One of them is now believed to be a bug: `parse("Sir de Mesnil")` reports given `de Mesnil` with family `''`, and a particle chain is a surname by construction -- `Lexicon.particles` chains onto the following piece to BUILD a family name, and `NON_GIVEN_NAME_PARTICLES` exists to say its members can never be a given name. A chain sitting in `given`, with no surname at all, contradicts that vocabulary rather than qualifying it. It is filed as #367. The mechanism is a collision, not a decision about particles: `sir` is in `GIVEN_NAME_TITLES`, which suppresses post_rules rule 1, so the chain is left in the given position -- `Dr.`, `Lady` and `Prof.` all give family `de Mesnil`. Longstanding rather than a 2.x regression: `nameparser==1.4.0` from PyPI gives title `Sir`, first `de Mesnil`, last `''`. So the previous commit presented a bug as a deliberate design qualification, and made doctests of it -- which would both read wrong and FAIL the moment #367 lands. Removed here rather than left to be found then. What replaced it in each page is the comma alone, which is correct under any reading and which #367 does not touch: a comma names the surname before a leading particle decides anything, so what follows it is the given name. Each page keeps its own register, and the surrounding "standing alone" framing stays -- it is a sufficient condition, still true after #367, not the exclusivity claim that was the over-reach. `config/particles.py`, `AGENTS.md` and the rule-1b comment cited the case from 2309783 as JUSTIFICATION for the `len(site) == 1` guard's shape. It is still worth naming there, because mechanically it IS what that guard does -- so it is named as suspected-wrong with a pointer to #367, and each site now says outright not to cite it as a line the rule means to draw. The reasoning itself now rests on the two sound counter-examples: `Juan de la Vega`, whose given position under FAMILY_FIRST holds a three-token chain rather than a lone particle, and the degenerate bare `de` with nothing to fold into. The first is stated as what the guard tests rather than as a blessed output, since #367 disputes that row too even though #359 records it as intended. Comments and doctests only; no executable line changed. Parse output is identical -- verified over `Sir de Mesnil`, `Dr. de Mesnil`, `Lady de Mesnil`, `Prof. de Mesnil`, `de Mesnil, Juan`, `de Mesnil`, `Juan de la Vega` and bare `de` in all three name orders. Doctests 227 -> 225, exactly the two removed `Sir de Mesnil` assertions. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/customize.rst | 13 +++++-------- docs/usage.rst | 8 +++----- nameparser/_pipeline/_post_rules.py | 15 +++++++++------ nameparser/config/particles.py | 11 +++++++---- 5 files changed, 25 insertions(+), 24 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a3ae18b5..bae62f71 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: `Sir de Mesnil` reports given `de Mesnil` in the default order and `Juan de la Vega` reports given `de la Vega` under `FAMILY_FIRST` — #359 blesses that second one explicitly — because in each the particle chained onto the next word and is not standing alone. A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reports given `de Mesnil` and no family at all in the default order, which is the same guard declining on a chained piece; that output is believed WRONG and is tracked as #367, so do not cite it as a line this rule means to draw.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` diff --git a/docs/customize.rst b/docs/customize.rst index 9bf069a6..d4815d0d 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -186,17 +186,14 @@ nothing to decide. >>> parse("de Mesnil").family 'de Mesnil' -Standing alone is the condition, not merely being the first word. -Anything ahead of the particle lets it join the word behind it -instead, and a comma has named the surname before the question is -reached at all — so neither of these is the case above, and both keep -a given name: +A comma forestalls the question rather than answering it. Writing the +surname before the comma has already said which words are the surname, +so a particle at the front of them decides nothing, and whatever +follows the comma is the given name as usual: .. doctest:: - >>> parse("Sir de Mesnil").given # 'de' joined 'Mesnil' - 'de Mesnil' - >>> parse("de Mesnil, Juan").given # the comma settled the surname + >>> parse("de Mesnil, Juan").given # the comma named the surname 'Juan' If your data never uses ``Van`` as a given name, take it out of the diff --git a/docs/usage.rst b/docs/usage.rst index f69ce5b5..857b2621 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -133,14 +133,12 @@ given name is the surname whatever order you declare: >>> parse("de Mesnil").family 'de Mesnil' -On its own is the whole of it. Put anything ahead of the particle and -it joins the word behind it as usual, and a comma names the surname -before the question comes up, so both of these keep a given name: +A comma gets there first. It names the surname outright, so a particle +opening that surname has nothing left to decide and the part after the +comma is the given name: .. doctest:: - >>> parse("Sir de Mesnil").given - 'de Mesnil' >>> parse("de Mesnil, Juan").given 'Juan' diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 02711797..f14e540c 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -140,12 +140,15 @@ def post_rules(state: ParseState) -> ParseState: # trailing piece. # A lone PIECE is the whole of it, which is a clause narrower than # "a member is never reported as the given name" -- that reading - # would be false, and #359 blesses the first of its counterexamples - # outright: "Juan de la Vega" under FAMILY_FIRST reports given='de - # la Vega', "Sir de Mesnil" reports given='de Mesnil' in the - # default order, and the degenerate bare 'de' keeps given='de'. In - # each the particle is in a piece with something else, or has - # nothing to fold into. + # would be false. Under FAMILY_FIRST the given position of "Juan de + # la Vega" holds the whole chain, three tokens rather than a lone + # particle, so 1b declines and given='de la Vega' stands; #359 + # records that case as working as intended. And the degenerate bare + # 'de' keeps given='de', having nothing to fold into. + # "Sir de Mesnil" is the same guard declining on a chained piece, + # but do NOT cite it as a limit this rule means to draw: it reports + # given='de Mesnil' with no family at all, which is believed wrong + # and is tracked as #367. # Those two sites are the whole scope, and the MIDDLE position is # deliberately not one of them -- which shows: "Mesnil Garcia de" # strands middle='de' under FAMILY_FIRST, while under diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index 54937246..ea5a46aa 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -10,14 +10,17 @@ #: name the family sits on, and a word that can never be a given name #: leaves it nothing to decide, so ``Policy(name_order=FAMILY_FIRST)`` #: reads "de Mesnil" as the family name too. What is asked about is the -#: opening *piece*, not the first word of the string: in "Sir de Mesnil" -#: the particle has already chained onto "Mesnil", so it is not standing -#: alone and the default order reads that piece as the given name. +#: opening *piece*, not the first word of the string: a particle that has +#: already chained onto the word behind it is part of that piece rather +#: than standing alone. #: Opening the name is only the commonest shape. The rule enforcing it #: (``post_rules`` rule 1b) reaches a member standing alone as a piece in #: the given position too, folding it into the family beside it, so that #: neither shape leaves a given name behind -- as long as there is another -#: name token to fold into. A bare "de" stays as it is. +#: name token to fold into. A bare "de" stays as it is. Where a chain is +#: reported as the given name anyway -- "Sir de Mesnil" gives given +#: "de Mesnil" and no surname at all -- that is an open bug (#367), not a +#: limit this set means to draw. #: Membership also decides the ambiguity report -- see #: :py:data:`PARTICLES` below. #: Curated to exclude anything that can be a given name in some culture