diff --git a/AGENTS.md b/AGENTS.md index 9df9b0b5..bae62f71 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (under the DEFAULT given-first order a name starting with one is all surname: "de Mesnil" — but that is `name_order`'s half of the sentence, not this set's, and `Policy(name_order=FAMILY_FIRST)` reads the same input as family "de", given "Mesnil"; what the set decides under either order is that a leading particle outside it records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reports given `de Mesnil` and no family at all in the default order, which is the same guard declining on a chained piece; that output is believed WRONG and is tracked as #367, so do not cite it as a line this rule means to draw.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` diff --git a/docs/customize.rst b/docs/customize.rst index e2a10816..d4815d0d 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -168,12 +168,14 @@ a suffix only when written with periods: ``particles_ambiguous`` is the same idea for surname particles. A particle listed there may also be a given name, which is what makes a leading one a decision to take; a particle *not* listed there never -is, so there is nothing to decide. Under the default name order that -shows up as whether the name has a given name at all: one that starts -with a listed particle keeps it, while one starting with an unlisted -particle has no given name — the whole thing is the surname. (Which -field each piece lands in is ``name_order``'s question, covered -below.) +is, so there is nothing to decide. That shows up in what a particle +standing *alone* at the front of a name does: a listed one is a name +part in its own right, while an unlisted one pulls the rest of the +name into the surname and leaves no given name at all. Which field a +*listed* particle lands in is ``name_order``'s question, covered +below; an unlisted one opening the name is the surname under every +order, because a word that can never be a given name leaves the order +nothing to decide. .. doctest:: @@ -184,10 +186,20 @@ below.) >>> parse("de Mesnil").family 'de Mesnil' +A comma forestalls the question rather than answering it. Writing the +surname before the comma has already said which words are the surname, +so a particle at the front of them decides nothing, and whatever +follows the comma is the given name as usual: + +.. doctest:: + + >>> parse("de Mesnil, Juan").given # the comma named the surname + 'Juan' + If your data never uses ``Van`` as a given name, take it out of the ambiguous set: a leading ``van`` is then no decision at all, so no -ambiguity is recorded, and under the default order it becomes part of -the surname: +ambiguity is recorded and it becomes part of the surname — under any +``name_order``, since that is what taking the word out asserted: .. doctest:: @@ -312,7 +324,8 @@ Family-first name order ``name_order`` is the one most likely to matter for data that is not in Western order. Positional input is assigned in the order you -declare, so a name written family-first — Hungarian, here — parses as +declare — with the two vocabulary exceptions noted at the end of this +section — so a name written family-first — Hungarian, here — parses as written instead of needing to be rearranged afterwards: .. doctest:: @@ -356,14 +369,29 @@ no order of its own — so it applies only where you set it, and there is no ``vn`` locale pack yet (issue `#146 `_). -One caution, which is why the example above is not the more obvious +Two cautions, both places where the vocabulary layer answers before +``name_order`` is consulted at all. + +The first is why the example above is not the more obvious ``"Nguyen Van Minh"``: a middle word that is also a shipped particle -is claimed by the vocabulary layer before ``name_order`` is consulted -at all. ``Van`` is the Dutch particle ``van``, so that name reads -family ``Nguyen`` with ``Van Minh`` given under *both* family-first -orders, and the choice between them makes no difference. `Words that -are also ordinary names`_ covers dropping such a word from the -vocabulary. +is claimed by the vocabulary layer. ``Van`` is the Dutch particle +``van``, so that name reads family ``Nguyen`` with ``Van Minh`` given +under *both* family-first orders, and the choice between them makes no +difference. + +The second is at the *front* of a name, and there the vocabulary +overrides the declared order outright: where a particle that can never +be a given name stands alone as the opening piece, the whole name is +the surname, in every ``name_order``. ``"de Mesnil"`` is family ``de +Mesnil`` under both family-first orders exactly as it is by default, +not family ``de`` with ``Mesnil`` given — a word that can never be a +given name leaves the order nothing to decide. Only the never-given +set does this: ``"van Gogh"`` reads family ``van``, given ``Gogh`` +under a family-first order, because ``van`` *can* be a given name and +so leaves a real question to answer. + +`Words that are also ordinary names`_ covers dropping a word from a +vocabulary, or moving one between those two sets. East Asian defaults, and turning them off ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/release_log.rst b/docs/release_log.rst index 6ba2086b..cdb1da00 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -10,9 +10,11 @@ Release Log editing one in place as a way to change a default and replaces it with configuring a ``Lexicon`` or a private ``Constants``. - Nothing moved between vocabularies and no parse changes: over the - 751 names of the differential corpora, every one of the seven - fields is identical to 2.1 through both the 2.0 and the 1.x API. + Nothing moved between vocabularies and no parse changes in the + default name order: over the 751 names of the differential + corpora, every one of the seven fields is identical to 2.1 through + both the 2.0 and the 1.x API. Both family-first orders do change, + below -- the same seven names under each. What breaks is code that *writes* to a default word list, and code that imports one by its 1.x name has until 3.0. @@ -22,6 +24,8 @@ Release Log **Behavior Changes** + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) + - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) **Deprecations** diff --git a/docs/usage.rst b/docs/usage.rst index 59007b76..857b2621 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -115,13 +115,14 @@ names together as easily as two surnames: >>> parse("Juan de la Vega y Rodriguez").family 'de la Vega y Rodriguez' -Position matters in exactly one place: the start of a name. A particle -there has no surname to attach to yet, so what decides the reading is -whether it is one that can double as a given name. Where the pieces -then land is ``name_order``'s question — see :doc:`customize` — and -the destinations below are the default given-first order's: the -particle either becomes the given name or turns the whole name into a -surname: +Position matters in exactly one place: a particle standing on its own +at the start of a name. It has no surname to attach to yet, so what +decides the reading is whether it is one that can double as a given +name: the particle either becomes the given name or turns the whole +name into a surname. Only the first of those is ``name_order``'s +question — see :doc:`customize`, and read the given name below as the +default given-first order's — since a particle that can never be a +given name is the surname whatever order you declare: .. doctest:: @@ -132,6 +133,15 @@ surname: >>> parse("de Mesnil").family 'de Mesnil' +A comma gets there first. It names the surname outright, so a particle +opening that surname has nothing left to decide and the part after the +comma is the given name: + +.. doctest:: + + >>> parse("de Mesnil, Juan").given + 'Juan' + :doc:`customize` covers how to change which words are in each of these sets, including which particles may double as given names. One shipped vocabulary works the other way round and so is not in the table above: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index c8611df1..a2c5d1b8 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -338,14 +338,15 @@ class Lexicon: #: and never consults this set, so it leaves a leading particle a #: piece of its own whether listed or not -- "de Mesnil" groups #: into two pieces exactly as "van Gogh" does. What membership - #: decides is what becomes of that piece afterwards. Under EITHER + #: decides is what becomes of that piece afterwards. Under ANY #: ``name_order`` a member records a particle-or-given ambiguity - #: and a non-member records none; under the default given-first - #: order a non-member is additionally folded back into the family - #: name once roles exist, so the whole name is the surname ("de - #: Mesnil" -- a bare "de", with nothing to fold into, is left - #: alone). Which field each piece lands in is ``name_order``'s - #: question, not this set's. + #: and a non-member records none, and a non-member is additionally + #: folded back into the family name once roles exist, so the whole + #: name is the surname ("de Mesnil" -- a bare "de", with nothing to + #: fold into, is left alone). That fold is order-independent too + #: (#359): a word that can never be a given name leaves + #: ``name_order`` nothing to decide. Which field a MEMBER's piece + #: lands in is ``name_order``'s question, not this set's. #: No constant of its own -- the default derives #: as particles minus #: :data:`~nameparser.config.particles.NON_GIVEN_NAME_PARTICLES` diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 40c0ac54..f14e540c 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -1,6 +1,8 @@ """Stage: post_rules. -Consumes: tokens (roles assigned). +Consumes: tokens (roles assigned), plus pieces and structure -- rule 1b +reads the opening piece of segment 0, or of segment 1 under a family +comma (#359). structure was always read here, for the rotation gate. Produces: tokens with roles adjusted by the post rules. Reads: Policy.patronymic_rules, Policy.middle_as_family; Lexicon.given_name_titles. @@ -9,6 +11,16 @@ 1. v1 handle_firstnames: when the parse is exactly a title plus ONE given token (no other roles), and the title is not a given-name title ('Sir'), that token is a family name -- "Mr. Johnson". +1b. where a particle that is never a given name stands ALONE as a + piece -- either opening the name or in the given position -- the + name is left with no given name at all: the given and the middles + fold into the family. Opening the name it pulls the rest of it in + ("de la Vega"); in the given position it folds into the family + beside it ("Mesnil de" under a family-first order). Needs another + name token to fold into, so a bare "de" stays as it is. Alone among + these rules it reads the opening position from `pieces` rather than + from the roles assign left, so that shape holds for a lone leading + particle piece under every name_order (#359). 2. EAST_SLAVIC (opt-in): positional GIVEN/MIDDLE/FAMILY each exactly one token, the FAMILY-position token carries an East Slavic patronymic ending, and the MIDDLE-position token does NOT (given + @@ -23,10 +35,13 @@ Both rotations fire only on Structure.NO_COMMA (v1 gates them on `not self._had_comma`): a comma already established the family. -These rules reconstruct token POSITION from roles, which is faithful +The rotations reconstruct token POSITION from roles, which is faithful to v1 only under the default GIVEN_FIRST order; their interaction with other name_order values is an open design question for the locale-pack -work (#270). +work (#270). Rule 1b read its particle the same way until #359 gave +it the position test as well, the decision there being that a +never-given particle keeps its particle whatever order the caller +declared. """ from __future__ import annotations @@ -53,10 +68,34 @@ r"^(оглу|оглы|оғлу|ўғли|угли|кызы|гызы|қызы|қизи|улы|ұлы|уулу)$", re.I) +_NAME_ROLES = (Role.GIVEN, Role.MIDDLE, Role.FAMILY) + + def _idx(tokens: list[WorkToken], role: Role) -> list[int]: return [i for i, t in enumerate(tokens) if t.role is role] +def _leading_name_piece(state: ParseState, + tokens: list[WorkToken]) -> tuple[int, ...]: + """The piece that OPENS the name, whatever role name_order gave it: + the first piece holding a GIVEN, MIDDLE or FAMILY token, in the + segment the positional read governs. Every piece holding none of + those is walked past -- title and suffix pieces, but NICKNAME and + MAIDEN as well, and anything assign left unroled -- and any number + of them, not only a single leading title. The segment is 0, except + under a family comma, where segment 0 is already fixed as the + surname and the name continues in segment 1. Empty on either of + two exits: that segment does not exist, or none of its pieces + holds a name token.""" + seg = 1 if state.structure is Structure.FAMILY_COMMA else 0 + if seg >= len(state.pieces): + return () + for piece in state.pieces[seg]: + if any(tokens[i].role in _NAME_ROLES for i in piece): + return piece + return () + + def _retag(tokens: list[WorkToken], i: int, role: Role) -> None: tokens[i] = dataclasses.replace(tokens[i], role=role) @@ -76,23 +115,81 @@ def post_rules(state: ParseState) -> ParseState: if joined not in state.lexicon.given_name_titles: for i in givens: _retag(tokens, i, Role.FAMILY) - - # rule 1b: a leading particle that is NEVER a given name means the - # whole name is a surname -- fold given (and middles) into family - # (v1 handle_non_first_name_prefix; 'de la Vega' -> family, while - # ambiguous 'van Gogh' keeps the given reading). The middle/family - # guard leaves a degenerate bare 'de' as given rather than - # inventing a surname. - if len(givens) == 1 and (middles or families): - gtags = tokens[givens[0]].tags - if "particle" in gtags and "vocab:particle-ambiguous" not in gtags: - for i in givens + middles: - _retag(tokens, i, Role.FAMILY) - # downstream rules key on the role counts: recompute + # every rule below reads these lists; recompute them the way + # 1b does after its own fold, so no guard can inspect a name + # that has already moved. Measured harmless today -- over the + # 751 differential names in four policies this arm fires 48 + # times, and 1b fires on none of them -- but reading a stale + # token list is the shape of the bug #359 fixed. `middles` + # is empty by the guard above and recomputed anyway, so + # relaxing that guard cannot leave it stale. givens = _idx(tokens, Role.GIVEN) middles = _idx(tokens, Role.MIDDLE) families = _idx(tokens, Role.FAMILY) + # rule 1b enforces one invariant (v1 handle_non_first_name_prefix): + # where a particle that is NEVER a given name stands ALONE as a + # piece -- either opening the name, or in the given position -- the + # name is left with no given name at all, the given and the middles + # joining the family. Two shapes, one repair: + # * the particle OPENS the name, so the whole name is a surname + # and it pulls the rest in -- "de la Vega"; + # * the particle is left ALONE in the given position, so it folds + # into the family beside it -- "Mesnil de" under + # name_order=FAMILY_FIRST, where the given position is the + # trailing piece. + # A lone PIECE is the whole of it, which is a clause narrower than + # "a member is never reported as the given name" -- that reading + # would be false. Under FAMILY_FIRST the given position of "Juan de + # la Vega" holds the whole chain, three tokens rather than a lone + # particle, so 1b declines and given='de la Vega' stands; #359 + # records that case as working as intended. And the degenerate bare + # 'de' keeps given='de', having nothing to fold into. + # "Sir de Mesnil" is the same guard declining on a chained piece, + # but do NOT cite it as a limit this rule means to draw: it reports + # given='de Mesnil' with no family at all, which is believed wrong + # and is tracked as #367. + # Those two sites are the whole scope, and the MIDDLE position is + # deliberately not one of them -- which shows: "Mesnil Garcia de" + # strands middle='de' under FAMILY_FIRST, while under + # FAMILY_FIRST_GIVEN_LAST the same trailing piece IS the given + # position, so it folds to family='Mesnil Garcia de'. Whether that + # difference should stand is #365, not this rule's to settle. How + # much the fold takes once it fires is the other open question: + # "de Mesnil Juan" goes wholly to the family in every order, + # matching the default rather than stopping at the particle group + # (#364). + # Only a never-given particle is in scope: an ambiguous one keeps + # whatever reading name_order gives it -- 'van Gogh' is given + # 'van' in the default order and family 'van' under a family-first + # one -- and #360 tracks the vocabulary line. + # The opening shape is read from `pieces` rather than from the role + # assign left (#359). Under the default order the opening piece IS + # the given, so the one role test used to catch both shapes; under + # FAMILY_FIRST the opening piece is the family and the given sits + # behind it, and reading the role alone let "de Mesnil" split. The + # single-token test says the same thing in each shape: a particle + # group already chained forward is not a lone particle. "Mr. de + # Mesnil" is three tokens in two pieces -- the title alone, then + # the particle GROUP -- so both sites are two tokens long and 1b + # declines on each; the family reading there is rule 1's in the + # default order and assign's under a family-first one. Both shapes + # then need another name token to fold with, which leaves a + # degenerate bare 'de' as it stands rather than inventing a + # surname. + sites = (_leading_name_piece(state, tokens), tuple(givens)) + if len(givens) + len(middles) + len(families) > 1 and any( + len(site) == 1 + and "particle" in tokens[site[0]].tags + and "vocab:particle-ambiguous" not in tokens[site[0]].tags + for site in sites): + for i in givens + middles: + _retag(tokens, i, Role.FAMILY) + # downstream rules key on the role counts: recompute + givens = _idx(tokens, Role.GIVEN) + middles = _idx(tokens, Role.MIDDLE) + families = _idx(tokens, Role.FAMILY) + # v1 gates both rotations on `not self._had_comma`; the # middle_as_family fold below runs comma or not (v1 order: # patronymics first, then handle_middle_name_as_last) diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index 7d751185..ea5a46aa 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -2,14 +2,27 @@ from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES #: The sub-set of :py:data:`PARTICLES` that are *never* a standalone given -#: name. Under the default given-first order that means a name *starting* -#: with one of these has no given name -- the whole thing is a surname -#: (e.g. "de Mesnil" -> family name "de Mesnil"). The reading is scoped to -#: the order on purpose: ``Policy(name_order=FAMILY_FIRST)`` parses the -#: same input as family "de", given "Mesnil", because which side of a -#: leading particle the family name sits on is ``name_order``'s question, -#: not this set's. What membership decides under either order is the -#: ambiguity report -- see :py:data:`PARTICLES` below. +#: name. Where one of these stands ALONE as the piece opening a name, that +#: name has no given name -- the whole thing is a surname (e.g. "de Mesnil" +#: -> family name "de Mesnil") -- and that reading holds under EVERY +#: ``name_order`` (#359). It is not scoped to the default order the way the +#: rest of the positional read is: ``name_order`` says which side of the +#: name the family sits on, and a word that can never be a given name +#: leaves it nothing to decide, so ``Policy(name_order=FAMILY_FIRST)`` +#: reads "de Mesnil" as the family name too. What is asked about is the +#: opening *piece*, not the first word of the string: a particle that has +#: already chained onto the word behind it is part of that piece rather +#: than standing alone. +#: Opening the name is only the commonest shape. The rule enforcing it +#: (``post_rules`` rule 1b) reaches a member standing alone as a piece in +#: the given position too, folding it into the family beside it, so that +#: neither shape leaves a given name behind -- as long as there is another +#: name token to fold into. A bare "de" stays as it is. Where a chain is +#: reported as the given name anyway -- "Sir de Mesnil" gives given +#: "de Mesnil" and no surname at all -- that is an open bug (#367), not a +#: limit this set means to draw. +#: Membership also decides the ambiguity report -- see +#: :py:data:`PARTICLES` below. #: Curated to exclude anything that can be a given name in some culture #: (`al`, `van`, `von`, `della`, `di`, `del`, `da`, `vander`, ...) and #: anything that is also a bound given-name particle (`abu`). When unsure, @@ -86,16 +99,17 @@ #: particle is the exception and chains nothing: the chain skips the #: first piece unconditionally, membership in this set or any other #: never entering into it. Where the pieces then land is again a later -#: question, -#: and this one is ``name_order``'s: under the default given-first order -#: a leading :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole -#: name a family name ("de la Vega"), while a leading particle outside -#: that set is read as the given name ("Van Johnson") -- whereas -#: ``Policy(name_order=FAMILY_FIRST)`` splits both at the leading -#: particle alike ("de la Vega" -> family "de", given "la Vega"; "Van -#: Johnson" -> family "Van", given "Johnson"), which is the same -#: chains-nothing grouping read the other way round. What membership -#: decides under EITHER order is the report: a leading particle outside +#: question, and this is where membership decides something: a leading +#: :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole name a +#: family name ("de la Vega") under every ``name_order``, because a word +#: that is never a given name leaves the order nothing to place. A +#: leading particle OUTSIDE that set could be either, so there +#: ``name_order`` decides after all: the default given-first order reads +#: it as the given name ("Van Johnson"), while either family-first order +#: splits the same chains-nothing grouping the other way round ("Van +#: Johnson" -> family "Van", given "Johnson"). +#: What membership decides under ANY of the three orders is also the +#: report: a leading particle outside #: :py:data:`NON_GIVEN_NAME_PARTICLES` records a particle-or-given #: ambiguity for the reading not taken, and one inside it records none. #: diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index ca7da24d..ad456f60 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -1,7 +1,10 @@ +import pytest + from nameparser._lexicon import Lexicon from nameparser._pipeline import run from nameparser._pipeline._state import ParseState -from nameparser._policy import PatronymicRule, Policy +from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, + PatronymicRule, Policy) from nameparser._types import Role _LEX = Lexicon( @@ -9,6 +12,7 @@ given_name_titles=frozenset({"sir"}), particles=frozenset({"de", "la", "van"}), particles_ambiguous=frozenset({"van"}), + suffix_words=frozenset({"md"}), ) @@ -108,6 +112,185 @@ def test_degenerate_bare_particle_stays_given() -> None: assert not _by_role(out, Role.FAMILY) +_FF = Policy(name_order=FAMILY_FIRST) +_FFGL = Policy(name_order=FAMILY_FIRST_GIVEN_LAST) + +#: Both family-first orders, since the rule is claimed of every one of +#: them and only one was ever parsed. They differ in where the given +#: name lands behind the family, which is exactly what the leading +#: shape must not depend on; the cases below fold identically under +#: both. +_FAMILY_FIRST = [pytest.param(_FF, id="FAMILY_FIRST"), + pytest.param(_FFGL, id="FAMILY_FIRST_GIVEN_LAST")] + + +# --- rule 1b under the family-first orders (#359) --------------------- +# The fold keys on POSITION, not on the GIVEN role: a never-given +# particle keeps its particle whatever name_order says. + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +@pytest.mark.parametrize("text,family,given,suffix", [ + # the leading particle chains the rest of the name into the family + ("de Mesnil", "de Mesnil", "", ""), + ("de la Vega", "de la Vega", "", ""), + # three pieces, so the fold has a MIDDLE to move as well as the + # given -- the `givens + middles` half of the repair, and the only + # no-comma corpus name that reaches it + ("de Mesnil Garcia", "de Mesnil Garcia", "", ""), + # ... and the trailing suffix run is peeled before the rule looks, + # comma or no comma (NO_COMMA and SUFFIX_COMMA both fold) + ("de Mesnil MD", "de Mesnil", "", "MD"), + ("De Mesnil, MD", "De Mesnil", "", "MD"), +]) +def test_family_first_folds_leading_never_given_particle( + policy: Policy, text: str, family: str, given: str, + suffix: str) -> None: + out = _parsed(text, policy) + assert _by_role(out, Role.FAMILY) == family + assert _by_role(out, Role.GIVEN) == given + assert _by_role(out, Role.SUFFIX) == suffix + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_leading_piece_scan_skips_pieces_that_hold_no_name( + policy: Policy) -> None: + # `_leading_name_piece` walks PAST pieces carrying no name role + # rather than reading piece 0 -- and past the first such piece, not + # only over a single title. 'Mr. de Mesnil' cannot show that: its + # particle is chained into one piece with 'Mesnil', so the scan + # lands on a two-token piece and the rule declines either way. + # Here a mid-name suffix word breaks that chain, leaving the + # particle a piece of its own BEHIND a title piece. Without the + # skip, or reading only pieces[0], the scan finds the title (or + # nothing) and the name splits: given='MD', middle='Mesnil', + # family='de'. + out = _parsed("Dr. de MD Mesnil", policy) + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.FAMILY) == "de MD Mesnil" + assert not _by_role(out, Role.GIVEN) + assert not _by_role(out, Role.MIDDLE) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +@pytest.mark.parametrize("text,family,given", [ + # a title makes the particle non-leading, so group already chained + # it into one piece -- one name piece, wholly family under both + # family-first orders + ("Mr. de Mesnil", "de Mesnil", ""), + # a family comma has already fixed the family; the post-comma part + # is the given name and must not be folded into it + ("de Mesnil, Juan", "de Mesnil", "Juan"), + # degenerate: nothing to fold, so no surname is invented + ("de", "de", ""), + # the leading piece is not a particle + ("Juan de Mesnil", "Juan", "de Mesnil"), + # 'van' is particles_ambiguous -- out of the rule's scope in EVERY + # order, so a family-first order still splits at the particle (#360) + ("van Gogh", "van", "Gogh"), +]) +def test_family_first_leading_particle_cases_that_do_not_fold( + policy: Policy, text: str, family: str, given: str) -> None: + out = _parsed(text, policy) + assert _by_role(out, Role.FAMILY) == family + assert _by_role(out, Role.GIVEN) == given + + +@pytest.mark.parametrize("text,title,given,middle,family,suffix", [ + ("de Mesnil", "", "", "", "de Mesnil", ""), + ("de la Vega", "", "", "", "de la Vega", ""), + ("de Mesnil Garcia", "", "", "", "de Mesnil Garcia", ""), + ("Dr. de MD Mesnil", "Dr.", "", "", "de MD Mesnil", ""), + ("de Mesnil MD", "", "", "", "de Mesnil", "MD"), + ("De Mesnil, MD", "", "", "", "De Mesnil", "MD"), + ("Mr. de Mesnil", "Mr.", "", "", "de Mesnil", ""), + ("de Mesnil, Juan", "", "Juan", "", "de Mesnil", ""), + ("de", "", "de", "", "", ""), + ("Juan de Mesnil", "", "Juan", "", "de Mesnil", ""), + ("van Gogh", "", "van", "", "Gogh", ""), + # a family comma folds the post-comma part when IT opens with a + # never-given particle -- long-standing behaviour, and rule 1b's + # own doing: assign hands it the same roles in every order (the + # comma already fixed the family), and both of 1b's sites then + # agree, the opening piece of segment 1 and the lone given being + # the same token. Pinned here so the re-key cannot quietly drop it + ("Smith, de Mesnil", "", "", "", "Smith de Mesnil", ""), + ("Smith, van Gogh", "", "van", "Gogh", "Smith", ""), +]) +def test_default_order_is_unchanged_by_the_family_first_fold( + text: str, title: str, given: str, middle: str, family: str, + suffix: str) -> None: + out = _parsed(text) + assert _by_role(out, Role.TITLE) == title + assert _by_role(out, Role.GIVEN) == given + assert _by_role(out, Role.MIDDLE) == middle + assert _by_role(out, Role.FAMILY) == family + assert _by_role(out, Role.SUFFIX) == suffix + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_family_comma_fold_is_order_independent(policy: Policy) -> None: + out = _parsed("Smith, de Mesnil", policy) + assert _by_role(out, Role.FAMILY) == "Smith de Mesnil" + assert not _by_role(out, Role.GIVEN) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_lone_never_given_particle_in_given_position_folds( + policy: Policy) -> None: + # The opening-position test alone does not carry the rule: under a + # family-first order the given position is the TRAILING piece, and + # a lone 'de' landing there has to fold into the family beside it + # or the parse leaves the whole given name as a word the vocabulary + # says is never a given name. Guarded here because a refactor that + # reads the rule as leading-particle-only drops exactly this shape, + # silently and under a non-default order (#359 review). + out = _parsed("Mesnil de", policy) + assert _by_role(out, Role.FAMILY) == "Mesnil de" + assert not _by_role(out, Role.GIVEN) + + +def test_lone_never_given_particle_needs_no_repair_by_default() -> None: + # the default order reaches the same rule from the other side: the + # particle is already the family, so there is nothing to repair + default = _parsed("Mesnil de") + assert _by_role(default, Role.GIVEN) == "Mesnil" + assert _by_role(default, Role.FAMILY) == "de" + + +# --- the whole never-given class, not just the fixture's 'de' --------- + +_ALL_ORDERS = [pytest.param(Policy(), id="GIVEN_FIRST"), *_FAMILY_FIRST] + + +@pytest.mark.parametrize("policy", _ALL_ORDERS) +def test_no_never_given_particle_is_left_as_the_given_name( + policy: Policy) -> None: + """Every case above rides on the fixture lexicon's 'de'. The rule + is claimed of the whole never-given class in every name_order, so + sweep the live class rather than pinning another word or two of it + -- a hardcoded handful would document those entries and catch + nothing else in the set (AGENTS.md, "Prefer behavior tests over + constant-content tests"). Derived from the lexicon, so an addition + to NON_GIVEN_NAME_PARTICLES is swept the day it lands, and asserts + nothing about which words are in the set. + """ + lex = Lexicon.default() + never_given = sorted(lex.particles - lex.particles_ambiguous) + assert never_given, "no never-given particles to exercise" + failures = [] + for particle in never_given: + text = f"{particle} Mesnil" + out = run(ParseState(original=text, lexicon=lex, policy=policy)) + given = _by_role(out, Role.GIVEN) + family = _by_role(out, Role.FAMILY) + if given or family != text: + failures.append( + f"{text!r}: given={given!r} family={family!r}") + assert not failures, ( + f"{len(failures)} of {len(never_given)} left as a given name " + f"or unfolded:\n" + "\n".join(failures[:15])) + + def test_middle_as_family_folds_middles() -> None: # v1 handle_middle_name_as_last, opt-in: middles prepend to family out = _parsed("John Quincy Adams Smith",