From a89aecea3672da7cc9f0a7c1c7cdf84b6c36bebd Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 20:20:56 -0700 Subject: [PATCH 1/9] Make a title transparent to the leading-particle exception (#367) A leading particle deliberately does not chain -- that is what makes "Van Johnson" read as given Van, family Johnson rather than as one surname. But the exception was keyed on piece index 0, so a TITLE displaced the particle out of leading position and the chain fired. Identical name text then parsed differently depending on whether a title preceded it: Van Johnson given='Van' family='Johnson' Dr. Van Johnson given='' family='Van Johnson' Sir Van Johnson given='Van Johnson' family='' The last row compounded it: with a given-name title, post_rules rule 1's carve-out handed the whole chain to `given`, so the name came out with no surname at all. A title is not part of the name, so it cannot decide whether the NAME begins with a particle. The chain loop's `k == 0` becomes "the first piece of the name", computed once before the loop. Not the plain "first piece that is not a title" the rule is stated as, and the difference is not academic: `st`, `do` and `freiherr` are each BOTH a title and an ambiguous particle, so a plain title test skipped over the very piece the exception exists to protect. Measured, the untitled "St John Smith" collapsed from title St, given John, family Smith into one given name "St John Smith", and "Do John Smith" and "Freiherr Blah Blah" with it -- none of which has a title in front of it at all. It also broke test_constants.py::test_add_title, whose "Te Awanui-a-Rangi Black" adds `te` to the titles while `te` ships as an ambiguous particle, manufacturing the same overlap from config. So the scan steps over a piece that can ONLY be a title and stops at one that could be the name's own first piece; with that, every untitled shape is byte-identical and test_add_title needs no change. Suffix-flagged pieces are deliberately not skipped too. The shapes that look like they need it -- "Jr. Van Johnson", "MD Van Johnson", "PhD Van Johnson" -- classify their leading piece as a TITLE and are already covered. What remains is a leading piece assign puts in GIVEN ("Ph.D. Van Johnson", "II Van Johnson", "Msc.Ed. Van Johnson"), which is part of the name in the output and holds the leading name position exactly as John does in "John Van Johnson"; skipping it also measured "Ph. D. Van Johnson" losing its family name outright. What moves, all of it a titled name whose first name-piece is a particle: Dr. Van Johnson family='Van Johnson' -> given='Van' family='Johnson' Dr. Van Johnson Smith family='Van Johnson Smith' -> given='Van' middle='Johnson' family='Smith' Mr. Van Nguyen family='Van Nguyen' -> given='Van' family='Nguyen' Sir Van Johnson given='Van Johnson' -> given='Van' family='Johnson' Sir Van Johnson Smith given='Van Johnson Smith' -> given='Van' middle='Johnson' family='Smith' Sir de Mesnil given='de Mesnil' -> family='de Mesnil' Sheik Abu Bakar given='Abu Bakar' -> given='Abu' family='Bakar' The last one is a regression, tracked as #369: it worked before only because `abu` happens to be a particle as well as a bound given name, and "Sheik abdul salam" shows bound-given alone does not chain. test_first_name_is_prefix_if_three_parts drops a v1-era xfail whose docstring said "Not sure how to fix this without breaking Mr and Mrs". It does not break Mr and Mrs -- a bare "Mr./Mrs. Surname" has no particle to displace, so nothing about it reaches the rule that moved -- and those shapes are now asserted in the test rather than only asserted about. cases.py's titled_ambiguous_particle_chains pinned the opposite and cited v1 parity. Parity was real (1.4.0 gives last 'Van Johnson') and still not the tiebreaker it looked like: "Dr. Van Johnson" and "Mr. Van Nguyen" are the same shape, so v1 pinned one shape both as correct and as broken. Resolved toward the xfail. The fork is still reported either way, from assign rather than group -- the same place the untitled "Van Johnson" reports it. _group's PARTICLE_OR_GIVEN emitter is NOT dead, though every test that used to reach it went through a plain title. It fires only when a piece before the particle is both a title and a prefix, which is the `st`/`do`/`freiherr` class again: "Freiherr von Richthofen" is the one natural spelling of it, and the four tests that pinned the emitter move onto it. The no-op-chain tests move to "Do Van Jr." for the same reason -- with a plain title the loop now skips the particle before the j > k + 1 guard is ever consulted, so they were passing without exercising the branch they exist to pin. Co-Authored-By: Claude Opus 5 --- nameparser/_pipeline/_group.py | 28 +++++++++++- tests/test_first_name.py | 24 ++++++++-- tests/v2/cases.py | 52 +++++++++++++++++---- tests/v2/test_parser.py | 82 +++++++++++++++++++++++----------- 4 files changed, 148 insertions(+), 38 deletions(-) diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index daad03ef..41e43766 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -199,9 +199,35 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # prefix chains: a non-leading prefix run absorbs everything to # the next prefix or suffix (v1's leading_first_name rule keeps # the first piece a name: "Van Johnson") + # + # "Leading" means the first piece of the NAME, not of the input + # (#367): a title is not part of the name, so it must not decide + # whether the name begins with a particle. Keyed on index 0, a + # title displaced the particle and the chain fired, so identical + # name text parsed two ways ("Van Johnson" -> given Van, family + # Johnson; "Dr. Van Johnson" -> family "Van Johnson"). + # + # "Title AND NOT prefix" rather than the plain "not a title" the + # rule is stated as, and the difference is not academic: `st`, + # `do` and `freiherr` are each BOTH a title and an ambiguous + # particle, so the plain test skipped over the very piece the + # exception exists to protect and "St John Smith" -- no title in + # front of it at all -- collapsed from title St, given John, + # family Smith into one given "St John Smith". A piece that + # could be the name's own first piece stops the scan; only a + # piece that can ONLY be a title is stepped over. + # + # Computed once, before the loop: every merge below starts at + # some k at or past this index, so no merge can move it. Suffix + # pieces are deliberately NOT skipped -- see the release log for + # 2.2.0; the shapes that look like they need it ("Jr. Van + # Johnson") classify their leading piece as a TITLE and are + # already covered here. + leading = next((k for k in range(len(pieces)) + if not title(k) or prefix(k)), 0) k = 0 while k < len(pieces): - if k == 0 or not prefix(k): + if k == leading or not prefix(k): k += 1 continue j = k + 1 diff --git a/tests/test_first_name.py b/tests/test_first_name.py index 38ffa4bd..3c01a263 100644 --- a/tests/test_first_name.py +++ b/tests/test_first_name.py @@ -1,5 +1,3 @@ -import pytest - from nameparser import HumanName from tests.base import HumanNameTestBase @@ -61,9 +59,27 @@ def test_first_name_is_not_prefix_if_only_two_parts_comma(self) -> None: self.m(hn.first, "Van", hn) self.m(hn.last, "Nguyen", hn) - @pytest.mark.xfail def test_first_name_is_prefix_if_three_parts(self) -> None: - """Not sure how to fix this without breaking Mr and Mrs""" + """Fixed in 2.2 (#367) by making titles transparent to the + leading-particle exception. This carried v1's ``xfail`` and the + docstring "Not sure how to fix this without breaking Mr and + Mrs" for the whole life of the 1.x line -- the exception was + keyed on piece index 0, so any title displaced the particle and + the prefix chain fired, giving last='Van Nguyen'. Keying it on + the first piece that is not a title instead leaves Mr and Mrs + provably untouched, which is why the Mr/Mrs shapes are asserted + here rather than only asserted about: a bare 'Mr./Mrs. Surname' + has no particle to displace, so nothing about it reaches the + rule that moved. v1's concern was about the fix it had in mind, + not about this one.""" hn = HumanName("Mr. Van Nguyen") self.m(hn.first, "Van", hn) self.m(hn.last, "Nguyen", hn) + for text, last in [("Mr. Smith", "Smith"), ("Mrs. Smith", "Smith"), + ("Mr. Nguyen", "Nguyen")]: + titled = HumanName(text) + self.m(titled.first, "", titled) + self.m(titled.last, last, titled) + titled = HumanName("Mr. John Smith") + self.m(titled.first, "John", titled) + self.m(titled.last, "Smith", titled) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index b420958c..5c4bb29e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -115,19 +115,55 @@ def __post_init__(self) -> None: "suffix": "MA"}, ambiguities=("suffix-or-name",)), Case("titled_ambiguous_particle_chains", "Dr. Van Johnson", - {"title": "Dr.", "family": "Van Johnson"}, + {"title": "Dr.", "given": "Van", "family": "Johnson"}, ambiguities=("particle-or-given",), - notes="the other branch of 'Van Johnson': a leading title " - "shifts Van off the given position, the prefix chain " - "fires, and the fork is reported from group rather than " - "assign (v1 parity on the fields)"), - Case("titled_ambiguous_particle_no_op_chain", "Dr. Van Jr.", - {"title": "Dr.", "given": "Van", "suffix": "Jr."}, + notes="reads exactly as the untitled 'Van Johnson' does, " + "because a title is not part of the name and so cannot " + "decide whether the NAME begins with a particle (#367). " + "This row pinned the opposite until 2.2 -- title 'Dr.', " + "family 'Van Johnson', the chain having fired because " + "the title shifted Van off piece index 0 -- and cited " + "v1 parity for it. Parity was real (1.4.0 gives last " + "'Van Johnson') and still not the tiebreaker it looked " + "like: this is the SAME shape as 'Mr. Van Nguyen', " + "which v1 shipped as an xfail calling the reading " + "wrong, so v1 pinned one shape both as correct and as " + "broken. Resolved toward the xfail, which now passes " + "(tests/test_first_name.py::" + "test_first_name_is_prefix_if_three_parts). The fork is " + "still reported, from assign rather than group -- the " + "same place the untitled 'Van Johnson' reports it"), + Case("titled_particle_chain_survives_a_title_that_is_also_a_particle", + "Freiherr von Richthofen", + {"title": "Freiherr", "family": "von Richthofen"}, + ambiguities=("particle-or-given",), + notes="#367's title transparency skips a piece that can ONLY " + "be a title, never one that could be the name's own " + "first piece. 'freiherr' is both a title and an " + "ambiguous particle, so it stops the scan and stays the " + "leading NAME piece; 'von' behind it is therefore " + "non-leading and chains, exactly as before 2.2. This is " + "also the only shape that still reaches group's " + "PARTICLE_OR_GIVEN emitter -- see " + "tests/v2/test_parser.py -- and the class that a plain " + "'first piece that is not a title' test broke: it " + "skipped 'St'/'Do'/'Freiherr' and collapsed the " + "untitled 'St John Smith' into one given name"), + Case("titled_ambiguous_particle_no_op_chain", "Do Van Jr.", + {"title": "Do", "given": "Van", "suffix": "Jr."}, notes="the piece after the particle is a suffix, so the chain " "scan never advances and the merge is a no-op -- nothing " "was chained, so there is no fork to report (the emitter " "fired here for all 39 ambiguous particles, and _assign " - "double-reported the same token)"), + "double-reported the same token). Spelled with 'Do' " + "rather than the 'Dr.' this row carried until 2.2: " + "under #367 a plain title is transparent, so 'Dr. Van " + "Jr.' leaves Van the leading name piece and the chain " + "loop skips it without ever reaching the no-op. 'Do' is " + "a title AND a particle, which stops the transparency " + "scan, so the chain does fire on Van and the j > k + 1 " + "guard is what declines it -- the same output, reached " + "through the branch the row exists to pin"), Case("initial_shaped_not_conjunction", "john e. smith", {"given": "john", "middle": "e.", "family": "smith"}, notes="v1 is_conjunction excludes initials at classify too"), diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index a1ab63af..43158d17 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -289,21 +289,38 @@ def test_trailing_roman_numeral_reports_the_fork() -> None: def test_ambiguous_particle_reports_both_branches_of_its_fork() -> None: - # "Van Johnson" reads Van as a given name and says so. A leading - # title shifts Van off index 0, the prefix-chain merge fires, and - # Van becomes a particle instead -- the SAME fork, called the other - # way. The two branches are taken in different stages (_assign vs - # _group), so only the one with an emitter used to report. - given_reading = parse("Van Johnson") - assert given_reading.given == "Van" + # "von Richthofen" reads von as a given name and says so. Put a + # piece in front of it that is BOTH a title and a particle and von + # is no longer the name's leading piece, the prefix-chain merge + # fires, and von becomes a particle instead -- the SAME fork, + # called the other way. The two branches are taken in different + # stages (_assign vs _group), so only the one with an emitter used + # to report. + # + # Spelled with 'Freiherr' rather than the 'Dr.' this test used + # until 2.2: an ordinary title is now transparent to the + # leading-particle exception (#367), so "Dr. Van Johnson" takes the + # _assign branch like everything else. `freiherr`/`st`/`do` -- a + # title that could also be the name's own first piece -- is what + # still reaches _group's emitter, and this is the only shape that + # does. + given_reading = parse("von Richthofen") + assert given_reading.given == "von" assert [a.kind for a in given_reading.ambiguities] == \ [AmbiguityKind.PARTICLE_OR_GIVEN] - particle_reading = parse("Dr. Van Johnson") - assert particle_reading.family == "Van Johnson" + particle_reading = parse("Freiherr von Richthofen") + assert particle_reading.family == "von Richthofen" assert [a.kind for a in particle_reading.ambiguities] == \ [AmbiguityKind.PARTICLE_OR_GIVEN] - assert [t.text for t in particle_reading.ambiguities[0].tokens] == ["Van"] + assert [t.text for t in particle_reading.ambiguities[0].tokens] == ["von"] + # and the branch the title no longer takes: "Dr. Van Johnson" is + # now byte-identical to the bare "Van Johnson", fork included + titled, bare = parse("Dr. Van Johnson"), parse("Van Johnson") + assert (titled.given, titled.family) == (bare.given, bare.family) \ + == ("Van", "Johnson") + assert [a.detail for a in titled.ambiguities] == \ + [a.detail for a in bare.ambiguities] def test_unambiguous_particle_chain_reports_nothing() -> None: @@ -311,10 +328,18 @@ def test_unambiguous_particle_chain_reports_nothing() -> None: assert parse("Dr. de la Vega").ambiguities == () +# The first three reach the chain loop and decline inside it: the piece +# after the particle is a suffix, so the scan never advances and merge() +# is a no-op -- nothing was chained, no fork taken. They are spelled with +# a title that is ALSO a particle ('Do', 'St'), because that is what +# still puts an ambiguous particle off the name's leading position since +# #367. The last three are the same strings with a plain title, which +# now decline one step earlier -- the particle IS the leading name piece +# and the loop skips it -- and are kept so the pair stays visible: two +# different reasons, one output, and neither may start reporting a fork. @pytest.mark.parametrize("text", [ - "Dr. Van Jr.", # the piece after the particle is a suffix, so - "Dr. Van MD", # the chain scan never advances and merge() is - "Dr. Do Jr.", # a no-op -- nothing was chained, no fork taken + "Do Van Jr.", "Do Van MD", "St Van Jr.", + "Dr. Van Jr.", "Dr. Van MD", "Dr. Do Jr.", ]) def test_no_op_prefix_chain_is_not_a_fork(text: str) -> None: assert parse(text).ambiguities == () @@ -322,18 +347,23 @@ def test_no_op_prefix_chain_is_not_a_fork(text: str) -> None: def test_a_fork_is_reported_by_exactly_one_stage() -> None: # the no-op merge left the particle a lone leading piece, which is - # _assign's trigger, so both stages reported the same token - n = parse("Dr. Van Jr Smith") + # _assign's trigger, so both stages reported the same token. + # 'Do' rather than the 'Dr.' this used until 2.2, for the reason + # above: with a plain title the chain loop never fires at all now, + # so the double-report it guards against is out of reach there. + n = parse("Do Van Jr Smith") assert n.given == "Van" assert len(n.ambiguities) == 1 def test_chained_particle_detail_does_not_claim_a_role() -> None: # _group runs before assignment, so it cannot know which field the - # chained piece lands in -- "Dr. Van Johnson de la Cruz" puts it in - # GIVEN. The detail must describe the decision, not guess a role. - n = parse("Dr. Van Johnson de la Cruz") - assert n.given == "Van Johnson" + # chained piece lands in -- "Freiherr von Richthofen de la Cruz" + # puts it in GIVEN, while the bare "Freiherr von Richthofen" above + # puts it in FAMILY. The detail must describe the decision, not + # guess a role. + n = parse("Freiherr von Richthofen de la Cruz") + assert n.given == "von Richthofen" (amb,) = n.ambiguities assert "family name" not in amb.detail @@ -347,15 +377,17 @@ def test_chained_particle_detail_is_order_invariant(policy: Policy) -> None: # _group's emitter is the reason the docs can scope leading-particle # DESTINATIONS to the default order without qualifying this text: # it names no field, and the chain it reports is a grouping-stage - # decision taken before any role exists. "Dr. Van Johnson" takes the - # chained branch under every order, so the string is the same one - # three times -- pin it, or the invariant is only an intention. - n = Parser(policy=policy).parse("Dr. Van Johnson") - assert (n.given, n.family) == ("", "Van Johnson") + # decision taken before any role exists. "Freiherr von Richthofen" + # takes the chained branch under every order, so the string is the + # same one three times -- pin it, or the invariant is only an + # intention. ("Dr. Van Johnson" carried this until 2.2; #367 made a + # plain title transparent, so it no longer chains at all.) + n = Parser(policy=policy).parse("Freiherr von Richthofen") + assert (n.given, n.family) == ("", "von Richthofen") (amb,) = n.ambiguities assert amb.kind is AmbiguityKind.PARTICLE_OR_GIVEN assert amb.detail == ( - "'Van' was chained onto the following name piece; " + "'von' was chained onto the following name piece; " "it is also a given name in other names") From 8376a0c267874c2208944e9d5c869dfa8cbbeee8 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 20:26:27 -0700 Subject: [PATCH 2/9] Classify #367's corpus diff at all three baselines The first 2.2 change to move the default-order corpus, so expected_since_2.1.0.toml gains its first rule and its header stops saying it is empty on purpose. Exactly one of 751 corpus names moves -- 'Mr. Van Nguyen', given '' -> 'Van' and family 'Van Nguyen' -> 'Nguyen' -- measured by running every corpus name through fd8dd8d and through the tree rather than by reading the harness's summary, so the count covers names that already diff from a baseline for other reasons. No field outside given/middle/family moves on any name. baseline intentional unexplained 2.1.0 1 0 2.0.0 90 0 1.4.0 108 0 The 1.4.0 copy has a second job. That run already exited 0 WITHOUT a rule, because the diff was landing on fix(suffix-routing), a fields-only rule about a trailing credential whose ["given", "family", "suffix"] is a superset of this diff's fields -- a rule silently absorbing a change it does not describe, which is the mis-classification the README warns about and is invisible in an exit code. A name_regex rule outranks every fields-only one, so the new rule claims it back: that class goes 26 -> 25 and #367 takes 1, with no other class moving at any baseline. The regex is deliberately narrower than the class it documents. The class is any title followed by a particle -- 'Dr. Van Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and only `van` is written, because it is the only member the corpora exercise; a name that joins the class later should arrive UNEXPLAINED and be read once. The leading `\S+\.?` title slot is the loose half, since no regex over the raw string can ask whether a word is a title, and `fields` carries the tightness instead. _CORPUS_CLAIMS records the rule at 11 names in each ledger -- the corpus names the regex reaches, not the one it explains, which is what that roster measures. Co-Authored-By: Claude Opus 5 --- tests/v2/test_ledger_guards.py | 9 +- tools/differential/expected_since_1.4.0.toml | 33 ++++++++ tools/differential/expected_since_2.0.0.toml | 32 +++++++ tools/differential/expected_since_2.1.0.toml | 89 +++++++++++++------- 4 files changed, 133 insertions(+), 30 deletions(-) diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 7ad21966..aaf148b8 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1036,6 +1036,8 @@ def _claim(rule: dict) -> _Claim: _Claim(2, ('given', 'middle', 'title'), "c14187bb08f8"), "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": _Claim(1, ('given', 'middle', 'suffix', 'title'), "390e7f814d13"), + "fix(#367) a title no longer displaces a leading particle out of the leading position": + _Claim(11, ('family', 'given', 'middle'), "b9d738c0e73b"), }, "expected_since_2.0.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": @@ -1050,8 +1052,13 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), + "fix(#367) a title no longer displaces a leading particle out of the leading position": + _Claim(11, ('family', 'given', 'middle'), "b9d738c0e73b"), + }, + "expected_since_2.1.0.toml": { + "fix(#367) a title no longer displaces a leading particle out of the leading position": + _Claim(11, ('family', 'given', 'middle'), "b9d738c0e73b"), }, - "expected_since_2.1.0.toml": {}, } diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 37d22b23..79bce9b9 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -414,3 +414,36 @@ fields = ["title", "given", "middle", "suffix"] # again, which is the thing the sort exists to prevent. Giving # classify() a real specificity order is the fix, and it is a change # to the harness rather than to a rule. + +[[change]] +issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" +# 'Mr. Van Nguyen': a leading particle deliberately does not chain -- +# that is what makes 'Van Nguyen' read as given Van, family Nguyen. +# Through 2.1 the exception was keyed on piece index 0, so a TITLE +# displaced the particle and the chain fired, giving family +# 'Van Nguyen' (v1's reading, which is why this name is parity with +# 1.4.0 until now). 2.2 keys it on the first piece of the NAME, so a +# titled name reads as the same text reads untitled. +# +# The wider class is any title followed by a particle -- 'Dr. Van +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and the +# regex names only `van` because that is the only member the corpora +# exercise. Deliberately narrower than the class: a corpus name that +# joins it later should arrive UNEXPLAINED and be read once, not be +# absorbed silently. Widen the literal then, and record the new count. +# +# The leading `\S+\.?` is the title slot, and it is the loose half: +# nothing in a regex over the raw string can ask whether a word is a +# title. `fields` is what carries the tightness -- the change only ever +# redistributes name pieces between given/middle/family, and never +# touches title, suffix, nickname or maiden. +# +# This rule has to exist even though the run already exited 0 without +# it. The diff was being classified by fix(suffix-routing) below, a +# fields-only rule whose ["given", "family", "suffix"] is a superset of +# this diff's ["given", "family"] -- a rule about a trailing credential +# absorbing a leading-particle change, which is the mis-classification +# the README warns about. A name_regex rule outranks every fields-only +# one, so this claims it back. +name_regex = "(?i)^\\S+\\.?\\s+van\\b" +fields = ["given", "middle", "family"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 75374fda..ba0a0e04 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -228,3 +228,35 @@ issue = "fix(#298) 间隔号 division changes the comma reading, sending the cre # screen that a literal here would be unreviewable. name_regex = "(?s)(?=.*,)(?=.*\\u00B7)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["given", "family", "title", "suffix"] + +[[change]] +issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" +# 'Mr. Van Nguyen': a leading particle deliberately does not chain -- +# that is what makes 'Van Nguyen' read as given Van, family Nguyen. +# Through 2.1 the exception was keyed on piece index 0, so a TITLE +# displaced the particle and the chain fired, giving family +# 'Van Nguyen' (v1's reading, which is why this name is parity with +# 1.4.0 until now). 2.2 keys it on the first piece of the NAME, so a +# titled name reads as the same text reads untitled. +# +# The wider class is any title followed by a particle -- 'Dr. Van +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and the +# regex names only `van` because that is the only member the corpora +# exercise. Deliberately narrower than the class: a corpus name that +# joins it later should arrive UNEXPLAINED and be read once, not be +# absorbed silently. Widen the literal then, and record the new count. +# +# The leading `\S+\.?` is the title slot, and it is the loose half: +# nothing in a regex over the raw string can ask whether a word is a +# title. `fields` is what carries the tightness -- the change only ever +# redistributes name pieces between given/middle/family, and never +# touches title, suffix, nickname or maiden. +# +# The 1.4 ledger carries the same rule, where it also has a second job: +# there the diff was already being absorbed by that file's fields-only +# fix(suffix-routing), a rule about a trailing credential whose +# ["given", "family", "suffix"] is a superset of this diff's fields. No +# fields-only rule exists here, so this file's copy is the plain +# classification. +name_regex = "(?i)^\\S+\\.?\\s+van\\b" +fields = ["given", "middle", "family"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 817e4184..fb69e9ae 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -3,40 +3,71 @@ # needs `issue`; optional `name_regex` and `fields` narrow it, and # compare.py sorts name_regex rules ahead of fields-only ones. # -# EMPTY ON PURPOSE, and correct while it stays that way. A ledger is -# opened the day its baseline ships (AGENTS.md release step 8), before -# the cycle has produced a single behavior change -- so there is nothing -# to classify yet. `nameparser/` is byte-identical to the v2.1.0 tag as -# this file is added, so a run against this baseline reports zero diffs -# and zero UNEXPLAINED. -# -# It exists rather than waiting because DEFAULT_BASELINE and the ledger -# are coupled: _allowlist_for treats a missing file as a hard error, on -# the reasoning that an absent ledger classifies nothing and would make -# every diff report as unexplained. So the baseline cannot advance to -# 2.1.0 without this file. -# -# Add the first rule when the first 2.2 behavior change lands. Do not -# copy rules across from expected_since_2.0.0.toml: that ledger +# This file was opened empty the day 2.1.0 shipped (AGENTS.md release +# step 8) and stayed that way through #354, #358 and #361, none of +# which moved the default-order corpus. #367 is the first 2.2 change +# that does, and its rule below is the first entry here. +# +# The ledger is opened before it has anything to say because +# DEFAULT_BASELINE and the ledger are coupled: _allowlist_for treats a +# missing file as a hard error, on the reasoning that an absent ledger +# classifies nothing and would make every diff report as unexplained. +# So the baseline could not advance to 2.1.0 without this file. +# +# Do not copy rules across from expected_since_2.0.0.toml: that ledger # classifies what 2.1 changed on top of 2.0, and a rule copied without # checking either over-matches -- hiding a real 2.2 regression behind a -# 2.1-era label -- or never fires at all. +# 2.1-era label -- or never fires at all. The #367 rule below IS shared +# with the other two ledgers, and that is not an exception to this: the +# change is in the 2.2 cycle, so every baseline before it sees the same +# diff, and each copy was checked against its own run rather than +# assumed. # # tests/v2/test_differential.py permits exactly one empty ledger, the -# one DEFAULT_BASELINE names. Once this stops being the open cycle it -# must carry rules like any other, so a 2.2 that changed no behavior at -# all would need that decision made deliberately rather than inherited. +# one DEFAULT_BASELINE names -- a permission this file no longer needs. # -# tests/v2/test_ledger_guards.py's _SPAN_BEARING_RULES already records this -# file with an empty set: no rule here hand-copies _SCRIPT_RANGES yet, -# and the first one that does has to be recorded there or the sweep -# fails. +# tests/v2/test_ledger_guards.py's _SPAN_BEARING_RULES records this +# file with an empty set: no rule here hand-copies _SCRIPT_RANGES, and +# the first one that does has to be recorded there or the sweep fails. +# #367's rule is Latin-only and declares no script span. # # There is deliberately no `change = []` line. TOML forbids appending a # [[change]] table to a statically defined array, so that line would -# block the exact next step this comment asks for -- and -# every reader uses .get("change", []) -- four call sites across -# three files -- so the key's absence is already the empty -# ledger. tests/v2/test_differential.py checks that nothing else is -# defined at the top level here, since a mistyped table name would -# otherwise read as an empty ledger rather than as a broken one. +# have blocked the entry below -- and every reader uses +# .get("change", []) -- four call sites across three files -- so the +# key's absence was already the empty ledger. +# tests/v2/test_differential.py checks that nothing else is defined at +# the top level here, since a mistyped table name would otherwise read +# as an empty ledger rather than as a broken one. + +[[change]] +issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" +# 'Mr. Van Nguyen': a leading particle deliberately does not chain -- +# that is what makes 'Van Nguyen' read as given Van, family Nguyen. +# Through 2.1 the exception was keyed on piece index 0, so a TITLE +# displaced the particle and the chain fired, giving family +# 'Van Nguyen' (v1's reading, which is why this name is parity with +# 1.4.0 until now). 2.2 keys it on the first piece of the NAME, so a +# titled name reads as the same text reads untitled. +# +# The wider class is any title followed by a particle -- 'Dr. Van +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and the +# regex names only `van` because that is the only member the corpora +# exercise. Deliberately narrower than the class: a corpus name that +# joins it later should arrive UNEXPLAINED and be read once, not be +# absorbed silently. Widen the literal then, and record the new count. +# +# The leading `\S+\.?` is the title slot, and it is the loose half: +# nothing in a regex over the raw string can ask whether a word is a +# title. `fields` is what carries the tightness -- the change only ever +# redistributes name pieces between given/middle/family, and never +# touches title, suffix, nickname or maiden. +# +# The 1.4 ledger carries the same rule, where it also has a second job: +# there the diff was already being absorbed by that file's fields-only +# fix(suffix-routing), a rule about a trailing credential whose +# ["given", "family", "suffix"] is a superset of this diff's fields. No +# fields-only rule exists here, so this file's copy is the plain +# classification. +name_regex = "(?i)^\\S+\\.?\\s+van\\b" +fields = ["given", "middle", "family"] From d1b0431a591c771b74c84a50a35a6fd9350e2e36 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 20:31:45 -0700 Subject: [PATCH 3/9] Pin the emitter path #367 left, and the scan's stopping rule The four tests that pinned _group's PARTICLE_OR_GIVEN emitter moved to "Freiherr von Richthofen" in the first commit, which leaves the mechanism pinned only from the public API. Two more pins, one per half of the rule, at the stage that owns it: test_a_title_does_not_make_the_particle_behind_it_non_leading is the change itself in `pieces` -- "Mr. Van Johnson" groups as ["Mr.", "Van", "Johnson"], the untitled grouping with a title in front of it. test_a_title_that_is_also_a_particle_stops_the_scan is why the rule is not spelled "the first piece that is not a title", and it is written against a CONFIGURED overlap rather than against `st`/`do`/`freiherr`. That is the same shape from the other direction: this module's fixture lexicon is a small hand-built one, and adding `van` to its titles is exactly what tests/test_constants.py::test_add_title does with `te` -- which is how the plain-title spelling was caught, since it broke that test as well as the untitled "St John Smith". test_a_leading_ambiguous_particle_is_reported_once_and_only_once gains "Freiherr " to its lead tuple. It swept "", "Dr. " and "Dr. Ann " over all 39 ambiguous particles and four tails, and after #367 not one of those reaches _group's emitter -- the sweep of the coordination between two emitters would have been measuring one. The new lead is the shape that still reaches it. Co-Authored-By: Claude Opus 5 --- tests/v2/pipeline/test_group.py | 29 +++++++++++++++++++++++++++++ tests/v2/test_properties.py | 12 ++++++++---- 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 7f3208f8..81a4bb8e 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -1,3 +1,5 @@ +import dataclasses + from nameparser._lexicon import Lexicon from nameparser._pipeline._classify import classify from nameparser._pipeline._extract import extract_delimited @@ -76,6 +78,33 @@ def test_leading_prefix_is_never_chained() -> None: assert _piece_texts(out) == [["Van", "Johnson"]] +def test_a_title_does_not_make_the_particle_behind_it_non_leading() -> None: + # #367: "leading" is the first piece of the NAME. A title is not + # part of the name, so it is stepped over and the grouping is the + # untitled one with the title in front of it. + assert _piece_texts(_grouped("Mr. Van Johnson")) == \ + [["Mr.", "Van", "Johnson"]] + assert _piece_texts(_grouped("Mr. Van Johnson Smith")) == \ + [["Mr.", "Van", "Johnson", "Smith"]] + + +def test_a_title_that_is_also_a_particle_stops_the_scan() -> None: + # the other half of the same rule, and the reason it is not spelled + # "the first piece that is not a title": the default vocabulary + # puts `freiherr`, `st` and `do` in BOTH sets, so any of them could + # be the name's own first piece. Stepping over one would make the + # particle behind it chain, and would break a name with no title in + # front of it at all ("St John Smith" -> title St, given John, + # family Smith). Spelled here as the overlap a CONFIG can create, + # which is the same shape: tests/test_constants.py::test_add_title + # adds `te` to the titles while `te` ships as a particle. + overlap = dataclasses.replace(_LEX, titles=_LEX.titles | {"van"}) + assert _piece_texts(_grouped("Van von Richthofen", lexicon=overlap)) == \ + [["Van", "von Richthofen"]] + assert _piece_texts(_grouped("Van Johnson Smith", lexicon=overlap)) == \ + [["Van", "Johnson", "Smith"]] + + def test_von_und_zu_bridges() -> None: # conjunction "und" joins two prefixes; the joined piece is a derived # prefix and still chains onto the following name (v1 PR #191) diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 8b6f4cb5..586d0c4b 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -103,9 +103,13 @@ def _fork_count(state: ParseState) -> int: def test_a_leading_ambiguous_particle_is_reported_once_and_only_once( ) -> None: """PARTICLE_OR_GIVEN is the one kind two stages emit: _group takes - the particle branch when a title shifts the particle off index 0 and - the chain claims something ("Dr. Van Johnson"), _assign takes the - given branch when it stays a lone leading piece ("Van Johnson"). + the particle branch when something shifts the particle off the + name's leading piece and the chain claims something ("Freiherr von + Richthofen"), _assign takes the given branch when it stays a lone + leading piece ("Van Johnson", and since #367 "Dr. Van Johnson" as + well -- a plain title is transparent, so only a leading word that + is BOTH a title and a particle still reaches _group's emitter, + which is what the 'Freiherr ' lead below exercises). Each reports the side it decides -- see the ParseState docstring -- but they coordinate only through _group's `j > k + 1` guard, which mirrors _assign's reachability by hand. Nothing checked the mirror. @@ -131,7 +135,7 @@ def test_a_leading_ambiguous_particle_is_reported_once_and_only_once( assert particles, "no ambiguous particles to exercise" failures = [] for particle in particles: - for lead in ("", "Dr. ", "Dr. Ann "): + for lead in ("", "Dr. ", "Dr. Ann ", "Freiherr "): for body in ("", "Johnson ", "Johnson Smith "): for tail in ("", "Jr.", "MD", "III"): text = f"{lead}{particle} {body}{tail}".strip() From 0eb64f88d7fcd81da10cf6628729a0fd3681b74c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 20:32:03 -0700 Subject: [PATCH 4/9] Sweep the prose that says a title changes the name behind it Every claim below was re-measured by parsing rather than reasoned about, per AGENTS.md's rule for mechanism claims. docs/release_log.rst gains the Behavior Changes bullet, which has to say plainly what moves because it reverses documented behaviour. It names the shapes that move, the shapes that provably do not (bare Mr./Mrs., and the title-that-is-also-a-particle class), the detail-text change that follows from the fork moving stages, and the "Sheik Abu Bakar" regression tracked as #369. The 2.2 summary paragraph above it claimed no default-order parse changed at all; that was true through #354, #358 and #361 and is not true now. AGENTS.md, in the two places it asserted the old reading. The particles.py entry recorded "Sir de Mesnil" -> given "de Mesnil", no family, as an output believed wrong and tracked as #367; it is now the untitled reading, and #367 got there by removing the chain rather than by touching post_rules rule 1b, which is the part worth keeping. The ambiguity-emitter entry said _group reports "when a title shifts it off index 0", which was the whole of that emitter's reach and is no longer any of it -- and since "is this emitter now dead?" is the question a reader arrives with, the answer and the input that settles it are written down. nameparser/_types.py's PARTICLE_OR_GIVEN docstring, published API reference: the second shape it describes is no longer "a particle a title shifted off the front". nameparser/config/particles.py said the chain "skips the first piece unconditionally, membership in this set or any other never entering into it". The first half now reads "the first piece of the NAME", and the second half is false as written -- the titles vocabulary does enter into it, in exactly one way, which is the st/do/freiherr overlap and is worth naming where a reader adding a particle will see it. nameparser/_lexicon.py's particles_ambiguous docstring and _pipeline/_assign.py's module docstring each named the leading piece in a way the change makes ambiguous; both now say which "leading" they mean. _group.py's emitter comment is rewritten around the reachability argument rather than the example: `all(title(x) for x in range(k))` and `leading` can now both hold only if one of the leading titles is also a particle, which is what makes the branch narrow rather than dead. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 4 ++-- docs/release_log.rst | 13 ++++++++----- nameparser/_lexicon.py | 6 ++++-- nameparser/_pipeline/_assign.py | 5 +++-- nameparser/_pipeline/_group.py | 23 ++++++++++++++++++----- nameparser/_types.py | 13 +++++++++---- nameparser/config/particles.py | 13 +++++++++++-- 7 files changed, 55 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 39a9f545..f3a9c0e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -153,7 +153,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and - `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) - `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reports given `de Mesnil` and no family at all in the default order, which is the same guard declining on a chained piece; that output is believed WRONG and is tracked as #367, so do not cite it as a line this rule means to draw.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reported given `de Mesnil` and no family at all in the default order, which was the same guard declining on a chained piece; #367 removed the chain rather than touching this rule — a title is now transparent to the leading-particle exception, so the piece is lone again, 1b fires, and the name reads family `de Mesnil` like the untitled form.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` @@ -196,7 +196,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker** — nameparser ships `py.typed`. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Dr. Van Jr.", where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when a title shifts it off index 0 and the prefix chain claims it, so both report; for two years only the first did. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, so both report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter's whole remaining reach is a leading word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates. `Freiherr von Richthofen` is the shape; when checking whether that emitter is dead, that is the input to look for, and the answer is that it is not. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission (see the comma paths in concepts.rst) over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/docs/release_log.rst b/docs/release_log.rst index cdb1da00..60195777 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -10,11 +10,12 @@ Release Log editing one in place as a way to change a default and replaces it with configuring a ``Lexicon`` or a private ``Constants``. - Nothing moved between vocabularies and no parse changes in the - default name order: over the 751 names of the differential - corpora, every one of the seven fields is identical to 2.1 through - both the 2.0 and the 1.x API. Both family-first orders do change, - below -- the same seven names under each. + Nothing moved between vocabularies. The rename itself changes no + parse at all; the parsing changes below are separate fixes, and + only one of them reaches the default name order -- a title no + longer changing how the name behind it is read, which moves one of + the 751 names of the differential corpora. Both family-first + orders change too, below -- the same seven names under each. What breaks is code that *writes* to a default word list, and code that imports one by its 1.x name has until 3.0. @@ -24,6 +25,8 @@ Release Log **Behavior Changes** + - Fix a title changing how the name behind it is read. A leading particle deliberately does not chain onto the words after it -- that is what makes ``"Van Johnson"`` read as given ``Van``, family ``Johnson`` rather than as one surname -- but the exception was keyed on the first *piece of the input*, so a title displaced the particle out of that position and the chain fired. Identical name text parsed two ways depending on whether a title preceded it: ``"Dr. Van Johnson"`` gave family ``Van Johnson`` with no given name, and ``"Sir Van Johnson"`` gave given ``Van Johnson`` with no *family* name at all, a given-name title then handing the whole chain to ``given``. A title is not part of the name, so it can no longer decide whether the name begins with a particle: the exception is keyed on the first piece of the **name**, and a titled name now reads exactly as the same text reads untitled. What moves is every titled name whose first name-piece is a particle -- ``"Dr. Van Johnson"`` and ``"Mr. Van Nguyen"`` to given ``Van`` plus family ``Johnson``/``Nguyen``, ``"Dr. Van Johnson Smith"`` to given ``Van``, middle ``Johnson``, family ``Smith``, ``"Sir de Mesnil"`` to family ``de Mesnil``. Only ``given``, ``middle`` and ``family`` are ever redistributed; no other field moves on any name. Untitled names are untouched, and so is every name with no particle behind its title: ``"Mr. Smith"``, ``"Mr. John Smith"``, ``"Sir Ian McKellen"`` and ``"King Henry"`` are byte-identical. So is a name whose leading word is *both* a title and a particle -- ``st``, ``do`` and ``freiherr`` are each in both vocabularies, so ``"St John Smith"``, ``"Do John Smith"`` and ``"Freiherr von Richthofen"`` keep their readings: a word that could be the name's own first piece stops the transparency scan rather than being stepped over. The fork is still reported, and still as ``PARTICLE_OR_GIVEN``; it now comes from the same place the untitled ``"Van Johnson"`` reports it, so the ``detail`` text changes from "was chained onto the following name piece" to "leading 'Van' may be a family-name particle; read as a given name". This fixes a defect v1 shipped as a known-failing test for the life of the 1.x line, with the note "Not sure how to fix this without breaking Mr and Mrs"; it does not break Mr and Mrs, which have no particle to displace. One of the 751 differential corpus names moves, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike. One case regresses and is tracked as `#369 `_: ``"Sheik Abu Bakar"`` was given ``Abu Bakar`` and is now given ``Abu``, family ``Bakar``. It read correctly before only because ``abu`` happens to be a particle as well as a bound given name -- ``"Sheik abdul salam"`` shows that being a bound given name alone does not chain -- so the reading was a side effect of the bug rather than a rule (closes #367) + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index a2c5d1b8..aa46c46e 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -334,10 +334,12 @@ class Lexicon: particles: frozenset[str] = frozenset() #: Subset of particles that can also BE a given name ("Van #: Johnson", but also "Van Buren"). Membership decides nothing - #: about chaining: the prefix chain skips index 0 unconditionally + #: about chaining: the prefix chain skips the name's first piece #: and never consults this set, so it leaves a leading particle a #: piece of its own whether listed or not -- "de Mesnil" groups - #: into two pieces exactly as "van Gogh" does. What membership + #: into two pieces exactly as "van Gogh" does, and since #367 so + #: do "Dr. de Mesnil" and "Dr. Van Johnson", a title not being + #: part of the name it precedes. What membership #: decides is what becomes of that piece afterwards. Under ANY #: ``name_order`` a member records a particle-or-given ambiguity #: and a non-member records none, and a non-member is additionally diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 3975af76..4178fc6a 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -22,8 +22,9 @@ non-suffixy ones COMMA_STRUCTURE). SUFFIX_COMMA: segment 0 as NO_COMMA; segments 1+ wholly SUFFIX. Emits PARTICLE_OR_GIVEN when the leading name piece is a lone -particles_ambiguous token with more pieces following ("Van Johnson") -- -whatever role name_order assigns that position. +particles_ambiguous token with more pieces following ("Van Johnson", +and since #367 "Dr. Van Johnson" too, a title no longer displacing the +particle out of that position) -- whatever role name_order assigns. """ from __future__ import annotations diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 41e43766..a540fd27 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -238,11 +238,24 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # The other half of PARTICLE_OR_GIVEN. _assign reports the # fork when an ambiguous particle stays a lone leading piece # ("Van Johnson" -> given under the default order, family - # under FAMILY_FIRST); the chain here takes the - # opposite branch whenever a title shifts it off index 0 - # ("Dr. Van Johnson" -> family "Van Johnson"). A fork whose - # two sides are decided in different stages needs an emitter - # in each. + # under FAMILY_FIRST); the chain here takes the opposite + # branch when the particle is not the name's leading piece. + # A fork whose two sides are decided in different stages + # needs an emitter in each. + # + # Narrow, and #367 is why. `all(title(x) for x in range(k))` + # says every piece ahead of this one is a title, and + # `leading` above says the FIRST piece that could be a name + # is at or before k -- so for both to hold, one of those + # leading titles must be a particle too, i.e. a word in both + # vocabularies (`st`, `do`, `freiherr` by default, or any + # overlap a caller configures). A plain title no longer + # reaches here at all: it is stepped over, the particle is + # the leading name piece, and _assign reports it. So this + # branch's whole remaining reach is "Freiherr von + # Richthofen" -- not dead, but pinned by exactly one shape, + # which tests/v2/cases.py and tests/v2/test_parser.py both + # now spell that way rather than with "Dr.". # # j > k + 1 is what makes this a DECISION rather than a # shape: when the next piece is a suffix the inner scan diff --git a/nameparser/_types.py b/nameparser/_types.py index 77e941f4..8da6a6eb 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -366,10 +366,15 @@ class AmbiguityKind(StrEnum): #: names ("read as a given name") -- that role is whatever #: assignment gave it, so it follows ``name_order`` and any #: ``script_orders`` entry, which is why the kind cannot name it. - #: A particle a title shifted off the front ("Dr. Van Johnson") - #: was instead claimed by the prefix chain, and ``detail`` says - #: that and names no field at all: grouping runs before roles - #: exist, so that text is the same under every order. + #: A particle that something ahead of it shifted off the front of + #: the name was instead claimed by the prefix chain, and ``detail`` + #: says that and names no field at all: grouping runs before roles + #: exist, so that text is the same under every order. Since #367 a + #: plain title is not such a thing -- "Dr. Van Johnson" reads as + #: the untitled "Van Johnson" does and takes the first shape -- + #: and what remains is a leading word that is both a title and a + #: particle, so it stays a name piece and the particle behind it is + #: genuinely not leading ("Freiherr von Richthofen"). PARTICLE_OR_GIVEN = "particle-or-given" #: A nickname/maiden delimiter opened without closing (or closed #: without opening); the text was kept as literal name content, so diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index ea5a46aa..ebf9daae 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -97,8 +97,17 @@ #: name "von bergen wessels", while the same chaining in "Smith, Juan #: de la Cruz" gives the middle name "de la Cruz". A leading #: particle is the exception and chains nothing: the chain skips the -#: first piece unconditionally, membership in this set or any other -#: never entering into it. Where the pieces then land is again a later +#: first piece of the NAME, its membership in this set or in +#: :py:data:`NON_GIVEN_NAME_PARTICLES` never entering into it. Leading +#: is read off the name rather than off the input (#367): a title is +#: not part of the name, so it is stepped over and "Dr. Van Johnson" +#: reads as the untitled "Van Johnson" does. One kind of word is not +#: stepped over, and this is the one place a title's vocabulary and +#: this one interact -- ``st``, ``do`` and ``freiherr`` are each BOTH +#: a title and a particle, so any of them could be the name's own +#: first piece and stops the scan, leaving a particle behind it +#: non-leading and free to chain ("Freiherr von Richthofen"). +#: Where the pieces then land is again a later #: question, and this is where membership decides something: a leading #: :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole name a #: family name ("de la Vega") under every ``name_order``, because a word From edb64b9c15c040d43cf6198143b178b979bd0be2 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 21:29:23 -0700 Subject: [PATCH 5/9] Make the chained emitter's vocabulary precondition executable (#367) _group's PARTICLE_OR_GIVEN emitter needs a leading piece that is both a title and an ambiguous particle, and since this branch that is the ONLY shape reaching it -- an ordinary title is now transparent to the leading-particle exception, so it takes _assign's branch. The five tests covering the emitter therefore lead with "Freiherr", and a comment names the class (freiherr/st/do) they depend on. A comment is the wrong medium for that. #360 is considering moving words between the particle sets, and freiherr and st are both on its candidate list. If either moves, those tests fail as five unrelated-looking parse mismatches and the reader has to find the comment to learn why. The precondition is now a test, and it distinguishes the two cases, which want opposite fixes. A missing WORD means pick another from the set -- the message prints what is left. An EMPTY intersection means the emitter is unreachable, every test of it is measuring nothing, and the right move is to delete the emitter rather than repoint its tests. Verified both branches by evaluating the guard against mutated sets: dropping 'freiherr' reports "need a lead from ['do', 'st']", dropping all three reports "remove the emitter rather than repointing". Co-Authored-By: Claude Opus 5 --- tests/v2/test_parser.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 43158d17..5cb2e72d 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -288,6 +288,36 @@ def test_trailing_roman_numeral_reports_the_fork() -> None: assert parse("John Q. V").ambiguities == () +#: The word the _group-emitter tests below lead with. It has to be BOTH a +#: title and an ambiguous particle -- see test_the_chained_emitter_is_still +#: _reachable for why, and for what to do when this stops being true. +_TITLE_PARTICLE = "Freiherr" + + +def test_the_chained_emitter_is_still_reachable() -> None: + """_group's PARTICLE_OR_GIVEN emitter needs a leading piece that is both + a title and an ambiguous particle, and since #367 that is the only shape + reaching it -- an ordinary title is transparent to the leading-particle + exception, so it takes _assign's branch instead. + + Two different failures, wanting two different fixes. If the intersection + is merely missing the word the tests below use, pick another from the + set. If it is EMPTY, the emitter is unreachable: every test of it is + then measuring nothing, and the emitter itself should go rather than be + re-pointed. #360 may move members of this set, which is why the coupling + is executable here instead of being a comment. + """ + lex = Lexicon.default() + both = lex.titles & lex.particles_ambiguous + assert both, ( + "no title is an ambiguous particle, so _group's PARTICLE_OR_GIVEN " + "emitter is unreachable and its tests measure nothing -- remove the " + "emitter rather than repointing them") + assert _TITLE_PARTICLE.lower() in both, ( + f"{_TITLE_PARTICLE!r} is no longer both a title and an ambiguous " + f"particle; the _group-emitter tests need a lead from {sorted(both)}") + + def test_ambiguous_particle_reports_both_branches_of_its_fork() -> None: # "von Richthofen" reads von as a given name and says so. Put a # piece in front of it that is BOTH a title and a particle and von From d0834ca56893456208c780368c2f641c66351c4c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 22:10:03 -0700 Subject: [PATCH 6/9] Say which fields the title fix actually moves (#367) The release log and all three ledgers claimed the change only ever redistributes given/middle/family and "never touches title, suffix, nickname or maiden". Two of those four are wrong. `title` moves: a word in BOTH vocabularies stops the transparency scan and stays a title piece instead of being chained onto the name, so "Dr. St John Smith" goes from title 'Dr.', family 'St John Smith' to title 'Dr. St', given 'John', family 'Smith'. `maiden` moves: un-chaining lets a marker standing behind the particle be seen at all, so "Mr. Van Johnson nee Brown" goes from family 'Van Johnson nee Brown' with no maiden name to given 'Van', family 'Johnson', maiden 'Brown'. Over a 20,979-name stress set of this class, `title` moved on 2,583 names (always growing, never shrinking) and `maiden` on 1,638 (always appearing, never lost); `suffix` and `nickname` moved on none. The ledgers keep fields = ["given", "middle", "family"], which is still what they should claim -- it is now stated as deliberately narrower than the change, so a title- or maiden-moving corpus name arrives UNEXPLAINED rather than being absorbed, the same posture the `van` literal already takes. Two more counts in the release log: * the release summary said the family-first orders move "the same seven names under each". Measured against 2.1.0 over the 751 corpus names it is eight under each -- the seven from #359 plus "Mr. Van Nguyen", which the title fix moves in every order. * the #367 bullet stated every worked example in default-order fields without saying so. Under both family-first orders "Dr. Van Johnson" reads family 'Van', given 'Johnson'; the #359 bullet below already names its orders explicitly, so this one now does too. Co-Authored-By: Claude Opus 5 --- docs/release_log.rst | 6 ++++-- tools/differential/expected_since_1.4.0.toml | 17 ++++++++++++++--- tools/differential/expected_since_2.0.0.toml | 17 ++++++++++++++--- tools/differential/expected_since_2.1.0.toml | 17 ++++++++++++++--- 4 files changed, 46 insertions(+), 11 deletions(-) diff --git a/docs/release_log.rst b/docs/release_log.rst index 60195777..45aa83a7 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -15,7 +15,9 @@ Release Log only one of them reaches the default name order -- a title no longer changing how the name behind it is read, which moves one of the 751 names of the differential corpora. Both family-first - orders change too, below -- the same seven names under each. + orders change too, below -- the same eight names under each, seven + of them from the family-first fix and the eighth the one name the + title fix moves in every order. What breaks is code that *writes* to a default word list, and code that imports one by its 1.x name has until 3.0. @@ -25,7 +27,7 @@ Release Log **Behavior Changes** - - Fix a title changing how the name behind it is read. A leading particle deliberately does not chain onto the words after it -- that is what makes ``"Van Johnson"`` read as given ``Van``, family ``Johnson`` rather than as one surname -- but the exception was keyed on the first *piece of the input*, so a title displaced the particle out of that position and the chain fired. Identical name text parsed two ways depending on whether a title preceded it: ``"Dr. Van Johnson"`` gave family ``Van Johnson`` with no given name, and ``"Sir Van Johnson"`` gave given ``Van Johnson`` with no *family* name at all, a given-name title then handing the whole chain to ``given``. A title is not part of the name, so it can no longer decide whether the name begins with a particle: the exception is keyed on the first piece of the **name**, and a titled name now reads exactly as the same text reads untitled. What moves is every titled name whose first name-piece is a particle -- ``"Dr. Van Johnson"`` and ``"Mr. Van Nguyen"`` to given ``Van`` plus family ``Johnson``/``Nguyen``, ``"Dr. Van Johnson Smith"`` to given ``Van``, middle ``Johnson``, family ``Smith``, ``"Sir de Mesnil"`` to family ``de Mesnil``. Only ``given``, ``middle`` and ``family`` are ever redistributed; no other field moves on any name. Untitled names are untouched, and so is every name with no particle behind its title: ``"Mr. Smith"``, ``"Mr. John Smith"``, ``"Sir Ian McKellen"`` and ``"King Henry"`` are byte-identical. So is a name whose leading word is *both* a title and a particle -- ``st``, ``do`` and ``freiherr`` are each in both vocabularies, so ``"St John Smith"``, ``"Do John Smith"`` and ``"Freiherr von Richthofen"`` keep their readings: a word that could be the name's own first piece stops the transparency scan rather than being stepped over. The fork is still reported, and still as ``PARTICLE_OR_GIVEN``; it now comes from the same place the untitled ``"Van Johnson"`` reports it, so the ``detail`` text changes from "was chained onto the following name piece" to "leading 'Van' may be a family-name particle; read as a given name". This fixes a defect v1 shipped as a known-failing test for the life of the 1.x line, with the note "Not sure how to fix this without breaking Mr and Mrs"; it does not break Mr and Mrs, which have no particle to displace. One of the 751 differential corpus names moves, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike. One case regresses and is tracked as `#369 `_: ``"Sheik Abu Bakar"`` was given ``Abu Bakar`` and is now given ``Abu``, family ``Bakar``. It read correctly before only because ``abu`` happens to be a particle as well as a bound given name -- ``"Sheik abdul salam"`` shows that being a bound given name alone does not chain -- so the reading was a side effect of the bug rather than a rule (closes #367) + - Fix a title changing how the name behind it is read. A leading particle deliberately does not chain onto the words after it -- that is what makes ``"Van Johnson"`` read as given ``Van``, family ``Johnson`` rather than as one surname -- but the exception was keyed on the first *piece of the input*, so a title displaced the particle out of that position and the chain fired. Identical name text parsed two ways depending on whether a title preceded it: ``"Dr. Van Johnson"`` gave family ``Van Johnson`` with no given name, and ``"Sir Van Johnson"`` gave given ``Van Johnson`` with no *family* name at all, a given-name title then handing the whole chain to ``given``. A title is not part of the name, so it can no longer decide whether the name begins with a particle: the exception is keyed on the first piece of the **name**, and a titled name now reads exactly as the same text reads untitled. What moves is every titled name whose first name-piece is a particle. Every reading in this bullet is the default ``name_order``, which is where the fields differ by order: ``"Dr. Van Johnson"`` and ``"Mr. Van Nguyen"`` go to given ``Van`` plus family ``Johnson``/``Nguyen``, ``"Dr. Van Johnson Smith"`` to given ``Van``, middle ``Johnson``, family ``Smith``, and ``"Sir de Mesnil"`` to family ``de Mesnil``. The same grouping change reaches ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``, where the leading particle takes the family rather than the given: ``"Dr. Van Johnson"`` reads family ``Van``, given ``Johnson`` under both. Three fields carry nearly all of the redistribution -- ``given``, ``middle`` and ``family`` -- but two more move on names inside this class, and only ``suffix`` and ``nickname`` move on none of them. ``title`` GROWS where the word that stops the transparency scan is one the chain used to swallow: ``"Dr. St John Smith"`` was title ``Dr.``, family ``St John Smith`` and is now title ``Dr. St``, given ``John``, family ``Smith``, ``st`` being in both vocabularies and so staying a title piece instead of being chained onto the name. ``maiden`` appears where un-chaining lets a marker standing behind the particle be seen at all: ``"Mr. Van Johnson nee Brown"`` was family ``Van Johnson nee Brown`` with no maiden name and is now given ``Van``, family ``Johnson``, maiden ``Brown``. Untitled names are untouched, and so is every name with no particle behind its title: ``"Mr. Smith"``, ``"Mr. John Smith"``, ``"Sir Ian McKellen"`` and ``"King Henry"`` are byte-identical. So is a name whose leading word is *both* a title and a particle -- ``st``, ``do`` and ``freiherr`` are each in both vocabularies, so ``"St John Smith"``, ``"Do John Smith"`` and ``"Freiherr von Richthofen"`` keep their readings: a word that could be the name's own first piece stops the transparency scan rather than being stepped over. The fork is still reported, and still as ``PARTICLE_OR_GIVEN``; it now comes from the same place the untitled ``"Van Johnson"`` reports it, so the ``detail`` text changes from "was chained onto the following name piece" to "leading 'Van' may be a family-name particle; read as a given name". This fixes a defect v1 shipped as a known-failing test for the life of the 1.x line, with the note "Not sure how to fix this without breaking Mr and Mrs"; it does not break Mr and Mrs, which have no particle to displace. One of the 751 differential corpus names moves, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike -- and it is the same one name under each family-first order too, ``"Mr. Van Nguyen"``. One case regresses and is tracked as `#369 `_: ``"Sheik Abu Bakar"`` was given ``Abu Bakar`` and is now given ``Abu``, family ``Bakar``. It read correctly before only because ``abu`` happens to be a particle as well as a bound given name -- ``"Sheik abdul salam"`` shows that being a bound given name alone does not chain -- so the reading was a side effect of the bug rather than a rule (closes #367) - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 79bce9b9..e18eb791 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -434,9 +434,20 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # # The leading `\S+\.?` is the title slot, and it is the loose half: # nothing in a regex over the raw string can ask whether a word is a -# title. `fields` is what carries the tightness -- the change only ever -# redistributes name pieces between given/middle/family, and never -# touches title, suffix, nickname or maiden. +# title. `fields` is what carries the tightness -- and it is narrower +# than the change rather than a description of it. The change moves +# `title` as well: a word in BOTH vocabularies stops the transparency +# scan and stays a title piece instead of being chained onto the name, +# so 'Dr. St John Smith' goes from title 'Dr.', family 'St John Smith' +# to title 'Dr. St', given 'John', family 'Smith'. It moves `maiden` +# as well: un-chaining lets a marker standing behind the particle be +# seen at all, so 'Mr. Van Johnson nee Brown' goes from family +# 'Van Johnson nee Brown' with no maiden name to given 'Van', family +# 'Johnson', maiden 'Brown'. `suffix` and `nickname` are the two that +# genuinely never move. Naming three roles anyway is deliberate, for +# the reason the regex names only `van`: a corpus name that moves +# `title` or `maiden` should arrive UNEXPLAINED and be read once, +# rather than be absorbed here. # # This rule has to exist even though the run already exited 0 without # it. The diff was being classified by fix(suffix-routing) below, a diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index ba0a0e04..d9f9ade6 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -248,9 +248,20 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # # The leading `\S+\.?` is the title slot, and it is the loose half: # nothing in a regex over the raw string can ask whether a word is a -# title. `fields` is what carries the tightness -- the change only ever -# redistributes name pieces between given/middle/family, and never -# touches title, suffix, nickname or maiden. +# title. `fields` is what carries the tightness -- and it is narrower +# than the change rather than a description of it. The change moves +# `title` as well: a word in BOTH vocabularies stops the transparency +# scan and stays a title piece instead of being chained onto the name, +# so 'Dr. St John Smith' goes from title 'Dr.', family 'St John Smith' +# to title 'Dr. St', given 'John', family 'Smith'. It moves `maiden` +# as well: un-chaining lets a marker standing behind the particle be +# seen at all, so 'Mr. Van Johnson nee Brown' goes from family +# 'Van Johnson nee Brown' with no maiden name to given 'Van', family +# 'Johnson', maiden 'Brown'. `suffix` and `nickname` are the two that +# genuinely never move. Naming three roles anyway is deliberate, for +# the reason the regex names only `van`: a corpus name that moves +# `title` or `maiden` should arrive UNEXPLAINED and be read once, +# rather than be absorbed here. # # The 1.4 ledger carries the same rule, where it also has a second job: # there the diff was already being absorbed by that file's fields-only diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index fb69e9ae..06fdfcbb 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -59,9 +59,20 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # # The leading `\S+\.?` is the title slot, and it is the loose half: # nothing in a regex over the raw string can ask whether a word is a -# title. `fields` is what carries the tightness -- the change only ever -# redistributes name pieces between given/middle/family, and never -# touches title, suffix, nickname or maiden. +# title. `fields` is what carries the tightness -- and it is narrower +# than the change rather than a description of it. The change moves +# `title` as well: a word in BOTH vocabularies stops the transparency +# scan and stays a title piece instead of being chained onto the name, +# so 'Dr. St John Smith' goes from title 'Dr.', family 'St John Smith' +# to title 'Dr. St', given 'John', family 'Smith'. It moves `maiden` +# as well: un-chaining lets a marker standing behind the particle be +# seen at all, so 'Mr. Van Johnson nee Brown' goes from family +# 'Van Johnson nee Brown' with no maiden name to given 'Van', family +# 'Johnson', maiden 'Brown'. `suffix` and `nickname` are the two that +# genuinely never move. Naming three roles anyway is deliberate, for +# the reason the regex names only `van`: a corpus name that moves +# `title` or `maiden` should arrive UNEXPLAINED and be read once, +# rather than be absorbed here. # # The 1.4 ledger carries the same rule, where it also has a second job: # there the diff was already being absorbed by that file's fields-only From a7add5507d35ac1b5edfc9dd1bd499135c7f802c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 22:26:34 -0700 Subject: [PATCH 7/9] Correct the claims #367's own change falsified (#367) Three sites still described #367 as an open bug. The prose sweep caught AGENTS.md and missed these: * nameparser/config/particles.py -- "'Sir de Mesnil' gives given 'de Mesnil' and no surname at all -- that is an open bug (#367)". Measured on HEAD: title 'Sir', family 'de Mesnil'. The remaining case where a chain IS reported as the given name is the one this set means to draw -- FAMILY_FIRST "Juan de la Vega", given 'de la Vega' -- so that is what it now cites. This is a `#:` comment and renders into the published API docs. * _post_rules.py's 1b preamble -- same claim, and the guard now fires. * _post_rules.py's worked example for 1b's single-token invariant was wrong end to end. "Mr. de Mesnil" is not "three tokens in two pieces"; it is three tokens in THREE pieces, [['Mr.'], ['de'], ['Mesnil']]. Both sites are one token long, so 1b FIRES rather than declines, and rule 1 cannot be producing the family reading -- rule 1 is gated on `not families` and 'Mesnil' is already the family. Output is unchanged either way, which is why no test caught it. "Exactly one shape" / "the only shape" reaches _group's PARTICLE_OR_GIVEN emitter was asserted at five sites (_group.py, tests/v2/cases.py, tests/v2/test_parser.py twice, AGENTS.md) and is false. Measured over a 20,979-name stress set: 2,520 hits, and "Freiherr von Richthofen", "St Van Johnson", "Do St Johnson" and "Dr. Do van Johnson" are four structurally different ways in. The last one matters most, because _group.py said "a plain title no longer reaches here at all" and it has one. The real reach is: any number of plain title pieces, then a piece in BOTH vocabularies, then any number of further titles, then an ambiguous particle whose chain claims something. The reachability argument itself is sound and is now stated the way it actually works. `leading` is STRICTLY before k (the loop skips k == leading), so it is one of the titles ahead of k and must also be a prefix -- a word in both vocabularies. The conjunction merge is the only other way a piece acquires those tags and cannot manufacture the pair: it derives from one neighbor, the left one whenever there is one, and its right operands are always fresh pieces. 243,280 conjunction-bearing probes produced no title+prefix piece without a both-vocabulary word. tests/test_first_name.py's docstring stated the rule as "the first piece that is not a title" -- precisely the naive rule this PR shows is wrong. Measured: that predicate collapses "St John Smith" into one given name and fails 15 tests including test_add_title. The shipped predicate is `not title(k) or prefix(k)`; the Mr/Mrs conclusion holds under either, which is why no test caught the docstring. _group.py's suffix rationale had the sign backwards. Skipping suffix pieces does not cost "Ph. D. Van Johnson" its family name -- shipped gives given 'Van Johnson', family '', and the skipping variant gives it given 'Van', family 'Johnson'. The real reason is the other half: "Ph.D. Van Johnson", "II Van Johnson" and "Msc.Ed. Van Johnson" put their leading piece in `given` and read family 'Van Johnson', and skipping moves Van out of the family into the middle name. The pointer to the 2.2.0 release log, which says nothing about suffixes, is replaced by the reasoning inline. Minors, all measured: _lexicon.py said "Dr. de Mesnil" groups into two pieces (it is three; the NAME is two); the `, 0` fallback now says it is inert by construction, since no piece being a prefix means the loop merges nothing; and cases.py's `titled_ambiguous_particle_chains` is renamed to `..._does_not_chain`, its id having come to contradict its own expectation. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- nameparser/_lexicon.py | 5 ++- nameparser/_pipeline/_group.py | 67 ++++++++++++++++++++++------- nameparser/_pipeline/_post_rules.py | 31 +++++++------ nameparser/config/particles.py | 10 +++-- tests/test_first_name.py | 12 ++++-- tests/v2/cases.py | 11 +++-- tests/v2/test_parser.py | 16 ++++--- 8 files changed, 106 insertions(+), 48 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f3a9c0e7..9c5ec46c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -196,7 +196,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker** — nameparser ships `py.typed`. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, so both report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter's whole remaining reach is a leading word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates. `Freiherr von Richthofen` is the shape; when checking whether that emitter is dead, that is the input to look for, and the answer is that it is not. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, so both report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates — standing ahead of the chained particle with nothing but titles before it. That word need not LEAD the input: `Dr. Do van Johnson` reaches the emitter with a plain title in front of it, and `Do St Johnson` reaches it with the chained particle itself in both vocabularies. `Freiherr von Richthofen` is the canonical shape rather than the only one; when checking whether that emitter is dead, a both-vocabulary word is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission (see the comma paths in concepts.rst) over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index aa46c46e..89de00ab 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -337,8 +337,9 @@ class Lexicon: #: about chaining: the prefix chain skips the name's first piece #: and never consults this set, so it leaves a leading particle a #: piece of its own whether listed or not -- "de Mesnil" groups - #: into two pieces exactly as "van Gogh" does, and since #367 so - #: do "Dr. de Mesnil" and "Dr. Van Johnson", a title not being + #: into two pieces exactly as "van Gogh" does, and since #367 the + #: NAME in "Dr. de Mesnil" and "Dr. Van Johnson" groups into those + #: same two pieces behind the title piece, a title not being #: part of the name it precedes. What membership #: decides is what becomes of that piece afterwards. Under ANY #: ``name_order`` a member records a particle-or-given ambiguity diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index a540fd27..5fc6af63 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -218,11 +218,29 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # piece that can ONLY be a title is stepped over. # # Computed once, before the loop: every merge below starts at - # some k at or past this index, so no merge can move it. Suffix - # pieces are deliberately NOT skipped -- see the release log for - # 2.2.0; the shapes that look like they need it ("Jr. Van - # Johnson") classify their leading piece as a TITLE and are - # already covered here. + # some k at or past this index, so no merge can move it. + # + # Suffix pieces are deliberately NOT skipped, and the reason is + # what skipping them WOULD do rather than what it would cost. + # A credential written with spaces already parses without a + # family name -- "Ph. D. Van Johnson" is given 'Van Johnson', + # suffix 'Ph. D.', family '' -- and skipping the suffix piece + # would actually give it one (given 'Van', family 'Johnson'). + # The shapes that decide it are the ones whose leading piece + # lands in `given` instead: "Ph.D. Van Johnson", "II Van + # Johnson" and "Msc.Ed. Van Johnson" each read given + # 'Ph.D.'/'II'/'Msc.Ed.' with family 'Van Johnson', and + # skipping the piece moves `Van` out of the family and into the + # middle name (given 'Ph.D.', middle 'Van', family 'Johnson') + # -- a worse reading, on three shapes, to fix none. "Jr. Van + # Johnson", the shape that looks like it needs the skip, + # classifies its leading piece as a TITLE and is already + # covered here. + # + # The `, 0` fallback is inert by construction rather than a + # default worth testing: it is reached only when every piece is + # a title and none is a prefix, and the loop below merges + # nothing unless some piece is a prefix. leading = next((k for k in range(len(pieces)) if not title(k) or prefix(k)), 0) k = 0 @@ -244,18 +262,35 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # needs an emitter in each. # # Narrow, and #367 is why. `all(title(x) for x in range(k))` - # says every piece ahead of this one is a title, and - # `leading` above says the FIRST piece that could be a name - # is at or before k -- so for both to hold, one of those - # leading titles must be a particle too, i.e. a word in both + # says every piece ahead of this one is a title, and the + # loop skipped k == leading, so `leading` is STRICTLY + # before k -- and being before k it is one of those titles, + # while being `leading` it satisfies `not title or prefix`. + # For both, it must be a prefix as well: a word in both # vocabularies (`st`, `do`, `freiherr` by default, or any - # overlap a caller configures). A plain title no longer - # reaches here at all: it is stepped over, the particle is - # the leading name piece, and _assign reports it. So this - # branch's whole remaining reach is "Freiherr von - # Richthofen" -- not dead, but pinned by exactly one shape, - # which tests/v2/cases.py and tests/v2/test_parser.py both - # now spell that way rather than with "Dr.". + # overlap a caller configures). A plain title alone can no + # longer put a particle off the name's leading piece; it is + # stepped over and _assign reports the fork instead. + # + # What that leaves is wider than one shape: any number of + # plain title pieces, then a piece in BOTH vocabularies, + # then any number of further titles, then the ambiguous + # particle whose chain claims something. "Freiherr von + # Richthofen" is the canonical spelling and the one + # tests/v2/cases.py and tests/v2/test_parser.py lead with, + # but "St Van Johnson", "Do St Johnson" (the chained + # particle itself in both vocabularies) and "Dr. Do van + # Johnson" (a plain title AHEAD of the both-vocabulary + # word) all reach here too. What none of them can do is + # dispense with the both-vocabulary WORD. The conjunction + # merge is the only other way a piece acquires `title` or + # `prefix`, and it cannot manufacture the pair: it derives + # from ONE neighbor, which is the left one whenever there + # is a left one, and its right operands are always fresh + # pieces (the loop runs left to right, so nothing to the + # right has been merged yet). Both tags therefore have to + # come from the piece it extends, which bottoms out at a + # lone token in both vocabularies. # # j > k + 1 is what makes this a DECISION rather than a # shape: when the next piece is a suffix the inner scan diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index f14e540c..78ae161b 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -145,10 +145,13 @@ def post_rules(state: ParseState) -> ParseState: # particle, so 1b declines and given='de la Vega' stands; #359 # records that case as working as intended. And the degenerate bare # 'de' keeps given='de', having nothing to fold into. - # "Sir de Mesnil" is the same guard declining on a chained piece, - # but do NOT cite it as a limit this rule means to draw: it reports - # given='de Mesnil' with no family at all, which is believed wrong - # and is tracked as #367. + # "Sir de Mesnil" used to be this guard declining on a chained + # piece, reporting given='de Mesnil' with no family at all. That + # was never a limit this rule meant to draw, and #367 removed the + # chain rather than touching the rule: a title is transparent to + # the leading-particle exception, so 'de' is a lone piece again, + # this guard fires, and the name reads family='de Mesnil' like the + # untitled form. # Those two sites are the whole scope, and the MIDDLE position is # deliberately not one of them -- which shows: "Mesnil Garcia de" # strands middle='de' under FAMILY_FIRST, while under @@ -169,14 +172,18 @@ def post_rules(state: ParseState) -> ParseState: # FAMILY_FIRST the opening piece is the family and the given sits # behind it, and reading the role alone let "de Mesnil" split. The # single-token test says the same thing in each shape: a particle - # group already chained forward is not a lone particle. "Mr. de - # Mesnil" is three tokens in two pieces -- the title alone, then - # the particle GROUP -- so both sites are two tokens long and 1b - # declines on each; the family reading there is rule 1's in the - # default order and assign's under a family-first one. Both shapes - # then need another name token to fold with, which leaves a - # degenerate bare 'de' as it stands rather than inventing a - # surname. + # group already chained forward is not a lone particle -- the + # FAMILY_FIRST "Juan de la Vega" above is what that looks like. + # "Mr. de Mesnil" is NOT one, and since #367 not even close to + # one: it is three tokens in THREE pieces -- the title, the + # particle, the surname -- because a title no longer displaces the + # particle out of the leading name position, so nothing chains. + # Both sites are one token long, so this guard FIRES and the + # family reading is its own. Rule 1 above cannot be what produces + # it: rule 1 is gated on `not families`, and 'Mesnil' is already + # the family. Both shapes need another name token to fold with, + # which leaves a degenerate bare 'de' as it stands rather than + # inventing a surname. sites = (_leading_name_piece(state, tokens), tuple(givens)) if len(givens) + len(middles) + len(families) > 1 and any( len(site) == 1 diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index ebf9daae..806df8f4 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -18,9 +18,13 @@ #: the given position too, folding it into the family beside it, so that #: neither shape leaves a given name behind -- as long as there is another #: name token to fold into. A bare "de" stays as it is. Where a chain is -#: reported as the given name anyway -- "Sir de Mesnil" gives given -#: "de Mesnil" and no surname at all -- that is an open bug (#367), not a -#: limit this set means to draw. +#: reported as the given name anyway it is because the member is no +#: longer standing alone: under ``Policy(name_order=FAMILY_FIRST)`` the +#: given position of "Juan de la Vega" holds the whole three-token +#: chain, so the rule declines and given "de la Vega" stands. A title +#: in front is NOT such a case -- since #367 a title is transparent to +#: the leading-particle exception, so "Sir de Mesnil" leaves "de" a lone +#: piece and reads family "de Mesnil", exactly as the untitled form does. #: Membership also decides the ambiguity report -- see #: :py:data:`PARTICLES` below. #: Curated to exclude anything that can be a given name in some culture diff --git a/tests/test_first_name.py b/tests/test_first_name.py index 3c01a263..c2dc88b2 100644 --- a/tests/test_first_name.py +++ b/tests/test_first_name.py @@ -65,10 +65,14 @@ def test_first_name_is_prefix_if_three_parts(self) -> None: docstring "Not sure how to fix this without breaking Mr and Mrs" for the whole life of the 1.x line -- the exception was keyed on piece index 0, so any title displaced the particle and - the prefix chain fired, giving last='Van Nguyen'. Keying it on - the first piece that is not a title instead leaves Mr and Mrs - provably untouched, which is why the Mr/Mrs shapes are asserted - here rather than only asserted about: a bare 'Mr./Mrs. Surname' + the prefix chain fired, giving last='Van Nguyen'. It is keyed on + the first piece that is not a title OR is a particle -- not the + naive "first piece that is not a title", which collapses + 'St John Smith' into one given name and breaks + tests/test_constants.py::ConstantsCustomizationTests + ::test_add_title, 15 failures in all. The Mr/Mrs conclusion + holds under either rule, which is why the Mr/Mrs shapes are + asserted here rather than only asserted about: a bare 'Mr./Mrs. Surname' has no particle to displace, so nothing about it reaches the rule that moved. v1's concern was about the fix it had in mind, not about this one.""" diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 5c4bb29e..a4ceeb1e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -114,7 +114,7 @@ def __post_init__(self) -> None: {"given": "John", "middle": "Q", "family": "Smith", "suffix": "MA"}, ambiguities=("suffix-or-name",)), - Case("titled_ambiguous_particle_chains", "Dr. Van Johnson", + Case("titled_ambiguous_particle_does_not_chain", "Dr. Van Johnson", {"title": "Dr.", "given": "Van", "family": "Johnson"}, ambiguities=("particle-or-given",), notes="reads exactly as the untitled 'Van Johnson' does, " @@ -143,9 +143,12 @@ def __post_init__(self) -> None: "ambiguous particle, so it stops the scan and stays the " "leading NAME piece; 'von' behind it is therefore " "non-leading and chains, exactly as before 2.2. This is " - "also the only shape that still reaches group's " - "PARTICLE_OR_GIVEN emitter -- see " - "tests/v2/test_parser.py -- and the class that a plain " + "also the CANONICAL shape reaching group's " + "PARTICLE_OR_GIVEN emitter, not the only one -- 'St Van " + "Johnson', 'Do St Johnson' and 'Dr. Do van Johnson' " + "reach it too, the last with a plain title ahead of the " + "both-vocabulary word; see tests/v2/test_parser.py -- " + "and the class that a plain " "'first piece that is not a title' test broke: it " "skipped 'St'/'Do'/'Freiherr' and collapsed the " "untitled 'St John Smith' into one given name"), diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 5cb2e72d..5aa1360b 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -295,10 +295,13 @@ def test_trailing_roman_numeral_reports_the_fork() -> None: def test_the_chained_emitter_is_still_reachable() -> None: - """_group's PARTICLE_OR_GIVEN emitter needs a leading piece that is both - a title and an ambiguous particle, and since #367 that is the only shape - reaching it -- an ordinary title is transparent to the leading-particle - exception, so it takes _assign's branch instead. + """_group's PARTICLE_OR_GIVEN emitter needs a piece that is both a title + and an ambiguous particle somewhere ahead of the chained particle, and + since #367 nothing else will do -- an ordinary title is transparent to + the leading-particle exception, so on its own it takes _assign's branch + instead. That word need not lead the input: "Dr. Do van Johnson" reaches + the emitter with a plain title in front of it. What it cannot be is + absent. Two different failures, wanting two different fixes. If the intersection is merely missing the word the tests below use, pick another from the @@ -332,8 +335,9 @@ def test_ambiguous_particle_reports_both_branches_of_its_fork() -> None: # leading-particle exception (#367), so "Dr. Van Johnson" takes the # _assign branch like everything else. `freiherr`/`st`/`do` -- a # title that could also be the name's own first piece -- is what - # still reaches _group's emitter, and this is the only shape that - # does. + # still reaches _group's emitter. This is the canonical spelling of + # that, not the only one: "St Van Johnson", "Do St Johnson" and + # "Dr. Do van Johnson" reach it as well. given_reading = parse("von Richthofen") assert given_reading.given == "von" assert [a.kind for a in given_reading.ambiguities] == \ From d56d0840bb56cedb7351332c5ff2bffeaeb09992 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 22:32:41 -0700 Subject: [PATCH 8/9] Pin the shapes #367 moves, and stop its ledger rule claiming the canaries (#367) The fix(#367) rule shipped with name_regex = "(?i)^\S+\.?\s+van\b". The title slot matched any first word at all, so the rule reached 11 corpus names and explained 1. The other ten include 'Vincent van Gogh', 'VINCENT VAN GOGH' and 'Alex van Johnson' -- the "Van Johnson" family AGENTS.md names as a standard regression canary. Since name_regex rules sort ahead of fields-only ones and `fields` matches by subset, a future regression on those names was labelled fix(#367) and exited 0. At the 2.1.0 baseline that is a strict loss: the ledger was empty before this PR. It is the same defect this PR found in fix(suffix-routing). Narrowed to "(?i)^mr\.\s+van\b", matching the comment's own deliberately-narrower-than-the-class posture. `\.?` went with it, being inert -- `\S+` is greedy, so it had already taken the period; both spellings reach the same 11 names. Demonstrated rather than argued. With a simulated chain regression in a scratch tree, the wide regex absorbs 7 corpus names under fix(#367) -- 'Alex van Johnson', 'Vincent van Gogh', 'VINCENT VAN GOGH', 'Vincent van Gogh van Beethoven', 'Charles van der van der Berg', 'Mike van der Velt' and the one it explains -- while the narrow one absorbs 1 and every canary arrives UNEXPLAINED. The three _CORPUS_CLAIMS entries go from _Claim(11, ..., "b9d738c0e73b") to _Claim(1, ..., "dce0ae6df4be"), read off the guard rather than predicted. All three baselines still exit 0 and still classify 'Mr. Van Nguyen'. Four things the prose asserted and nothing tested: * The suffix decision. "Also skip suffix pieces" survives all 3169 tests while changing output, so test_a_suffix_shaped_leading_piece_is_not_stepped_over pins the two shapes that decide it. Kill proven: with the mutation applied to a scratch copy of this tree, that test is the only failure (1 failed, 3169 passed). * Four shapes the release log names with zero assertions anywhere get cases.py rows: "Dr. Van Johnson Smith", "Sir Van Johnson", "Sir de Mesnil" and "Jr. Van Johnson", each classified fix(#367) against measured 1.4.0 output. The existing "Dr. Van Johnson" row is reclassified from the default "parity" for the same reason: 1.4.0 gives last 'Van Johnson', so parity stopped holding when this PR changed the row's expectation. * #369's regression. "Sheik Abu Bakar" lived only in ledger comments and is in no corpus, so nothing would notice it moving again. A strict xfail now asserts the DESIRED post-#369 output; verified strict by flipping the assertion to today's output in a scratch tree and watching it fail XPASS(strict). * test_chained_particle_detail_is_order_invariant covered all three orders for "Freiherr von Richthofen" -- the shape #367 did NOT move. It now also pins "Dr. Van Johnson" against the bare "Van Johnson" in each order, including family 'Van', given 'Johnson' under both family-first ones. Co-Authored-By: Claude Opus 5 --- tests/v2/cases.py | 36 ++++++++++++++++++++ tests/v2/pipeline/test_group.py | 13 +++++++ tests/v2/test_ledger_guards.py | 6 ++-- tests/v2/test_parser.py | 32 +++++++++++++++++ tools/differential/expected_since_1.4.0.toml | 30 +++++++++++----- tools/differential/expected_since_2.0.0.toml | 30 +++++++++++----- tools/differential/expected_since_2.1.0.toml | 30 +++++++++++----- 7 files changed, 147 insertions(+), 30 deletions(-) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index a4ceeb1e..c6b58d05 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -116,6 +116,7 @@ def __post_init__(self) -> None: ambiguities=("suffix-or-name",)), Case("titled_ambiguous_particle_does_not_chain", "Dr. Van Johnson", {"title": "Dr.", "given": "Van", "family": "Johnson"}, + classification="fix(#367)", ambiguities=("particle-or-given",), notes="reads exactly as the untitled 'Van Johnson' does, " "because a title is not part of the name and so cannot " @@ -133,6 +134,41 @@ def __post_init__(self) -> None: "test_first_name_is_prefix_if_three_parts). The fork is " "still reported, from assign rather than group -- the " "same place the untitled 'Van Johnson' reports it"), + Case("titled_ambiguous_particle_keeps_its_middles", "Dr. Van Johnson Smith", + {"title": "Dr.", "given": "Van", "middle": "Johnson", + "family": "Smith"}, + classification="fix(#367)", + ambiguities=("particle-or-given",), + notes="the release log names this shape and nothing asserted " + "it. 1.4.0 gives title 'Dr.', last 'Van Johnson Smith'; " + "un-chaining leaves three name pieces, so the middle " + "appears where the whole thing used to be one surname"), + Case("given_name_title_ambiguous_particle", "Sir Van Johnson", + {"title": "Sir", "given": "Van", "family": "Johnson"}, + classification="fix(#367)", + ambiguities=("particle-or-given",), + notes="a GIVEN-NAME title, which is the worse half of the bug " + "#367 fixed: 1.4.0 and 2.1 alike gave first " + "'Van Johnson' and no family name at all, the chain " + "having fired and then been handed whole to `given`. " + "Now identical to the untitled 'Van Johnson'"), + Case("given_name_title_never_given_particle", "Sir de Mesnil", + {"title": "Sir", "family": "de Mesnil"}, + classification="fix(#367)", + notes="the never-given half: `de` is not an ambiguous " + "particle, so there is no fork to report and post_rules " + "1b folds the name into the family. 1.4.0 and 2.1 gave " + "first 'de Mesnil' with no family, because the chain " + "left 1b nothing standing alone to fire on"), + Case("suffix_word_title_ambiguous_particle", "Jr. Van Johnson", + {"title": "Jr.", "given": "Van", "family": "Johnson"}, + classification="fix(#367)", + ambiguities=("particle-or-given",), + notes="a leading 'Jr.' classifies as a TITLE, not a suffix, " + "which is why the transparency scan does not step over " + "suffix pieces -- see tests/v2/pipeline/test_group.py::" + "test_a_suffix_shaped_leading_piece_is_not_stepped_over. " + "1.4.0 gives title 'Jr.', last 'Van Johnson'"), Case("titled_particle_chain_survives_a_title_that_is_also_a_particle", "Freiherr von Richthofen", {"title": "Freiherr", "family": "von Richthofen"}, diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 81a4bb8e..6140ea46 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -105,6 +105,19 @@ def test_a_title_that_is_also_a_particle_stops_the_scan() -> None: [["Van", "Johnson", "Smith"]] +def test_a_suffix_shaped_leading_piece_is_not_stepped_over() -> None: + # #367 steps over TITLE pieces only. Adding suffix pieces to that + # scan survives the whole suite while changing what these names + # parse to, so pin them here: with the skip, `leading` moves to + # "Van", the chain loop passes over it, and the two pieces below + # become three -- which puts Van in the middle name rather than the + # family once roles exist. See _group.py for why that reading is + # worse rather than merely different. + assert _piece_texts(_grouped("Ph. D. Van Johnson")) == \ + [["Ph. D.", "Van Johnson"]] + assert _piece_texts(_grouped("II Van Johnson")) == [["II", "Van Johnson"]] + + def test_von_und_zu_bridges() -> None: # conjunction "und" joins two prefixes; the joined piece is a derived # prefix and still chains onto the following name (v1 PR #191) diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index aaf148b8..850c6ff7 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1037,7 +1037,7 @@ def _claim(rule: dict) -> _Claim: "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": _Claim(1, ('given', 'middle', 'suffix', 'title'), "390e7f814d13"), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(11, ('family', 'given', 'middle'), "b9d738c0e73b"), + _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, "expected_since_2.0.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": @@ -1053,11 +1053,11 @@ def _claim(rule: dict) -> _Claim: "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(11, ('family', 'given', 'middle'), "b9d738c0e73b"), + _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, "expected_since_2.1.0.toml": { "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(11, ('family', 'given', 'middle'), "b9d738c0e73b"), + _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, } diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 5aa1360b..a2ae0e28 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -362,6 +362,21 @@ def test_unambiguous_particle_chain_reports_nothing() -> None: assert parse("Dr. de la Vega").ambiguities == () +@pytest.mark.xfail(strict=True, reason="#369: 'abu' is a bound given name " + "as well as an ambiguous particle") +def test_bound_given_name_that_is_also_a_particle() -> None: + # The one case #367 regressed, asserted as the DESIRED post-#369 + # output so that fixing #369 announces itself here rather than + # silently. Today: given 'Abu', family 'Bakar'. Through 2.1 it read + # correctly only because the title displaced 'abu' out of the + # leading position and the prefix chain claimed 'Bakar' -- a side + # effect of the bug, not a rule: "Sheik abdul salam", whose lead is + # a bound given name and NOT a particle, chains no such thing. + # The name is in none of the differential corpora, so nothing else + # would notice it moving again. + assert parse("Sheik Abu Bakar").given == "Abu Bakar" + + # The first three reach the chain loop and decline inside it: the piece # after the particle is a suffix, so the scan never advances and merge() # is a no-op -- nothing was chained, no fork taken. They are spelled with @@ -424,6 +439,23 @@ def test_chained_particle_detail_is_order_invariant(policy: Policy) -> None: "'von' was chained onto the following name piece; " "it is also a given name in other names") + # The shape above is the one #367 did NOT move, so pin the one it + # did in the same three orders: a plain title is transparent, so + # "Dr. Van Johnson" is byte-identical to the bare "Van Johnson" + # under every name_order, and the fork comes from _assign -- whose + # detail DOES name the role, unlike the grouping-stage text above. + titled = Parser(policy=policy).parse("Dr. Van Johnson") + bare = Parser(policy=policy).parse("Van Johnson") + assert titled.title == "Dr." + assert (titled.given, titled.middle, titled.family) == \ + (bare.given, bare.middle, bare.family) + (titled_amb,) = titled.ambiguities + assert titled_amb.detail == bare.ambiguities[0].detail + role = "family" if policy.name_order[0] is Role.FAMILY else "given" + assert titled_amb.detail == ( + f"leading 'Van' may be a family-name particle; " + f"read as a {role} name") + def test_each_suffix_or_name_branch_describes_itself() -> None: # one kind, two causes: the acronym branch turns on periods, the diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index e18eb791..2b956301 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -426,15 +426,27 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # titled name reads as the same text reads untitled. # # The wider class is any title followed by a particle -- 'Dr. Van -# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and the -# regex names only `van` because that is the only member the corpora -# exercise. Deliberately narrower than the class: a corpus name that -# joins it later should arrive UNEXPLAINED and be read once, not be -# absorbed silently. Widen the literal then, and record the new count. +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and this +# regex names one title and one particle because 'Mr. Van Nguyen' is +# the one name in the corpora that moves. Deliberately narrower than +# the class: a corpus name that joins it later should arrive +# UNEXPLAINED and be read once, not be absorbed silently. Widen the +# literals then, and record the new count. # -# The leading `\S+\.?` is the title slot, and it is the loose half: -# nothing in a regex over the raw string can ask whether a word is a -# title. `fields` is what carries the tightness -- and it is narrower +# The title slot is a LITERAL for that reason and not by accident. +# Nothing in a regex over the raw string can ask whether a word is a +# title, so the `(?i)^\S+\.?\s+van\b` this rule shipped with matched +# ANY first word: it reached 11 corpus names and explained 1. The +# other ten include 'Vincent van Gogh', 'VINCENT VAN GOGH' and +# 'Alex van Johnson' -- the "Van Johnson" family AGENTS.md names as a +# standard regression canary. name_regex rules sort ahead of +# fields-only ones and `fields` matches by subset, so a future +# regression on any of those would have been labelled fix(#367) and +# exited 0 -- a strict loss at this baseline, whose ledger was empty +# before. (`\.?` was inert into the bargain: `\S+` is greedy, so it +# had already taken the period.) +# +# `fields` is what carries the rest of the tightness -- and it is narrower # than the change rather than a description of it. The change moves # `title` as well: a word in BOTH vocabularies stops the transparency # scan and stays a title piece instead of being chained onto the name, @@ -456,5 +468,5 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # absorbing a leading-particle change, which is the mis-classification # the README warns about. A name_regex rule outranks every fields-only # one, so this claims it back. -name_regex = "(?i)^\\S+\\.?\\s+van\\b" +name_regex = "(?i)^mr\\.\\s+van\\b" fields = ["given", "middle", "family"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index d9f9ade6..d8cb0fe7 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -240,15 +240,27 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # titled name reads as the same text reads untitled. # # The wider class is any title followed by a particle -- 'Dr. Van -# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and the -# regex names only `van` because that is the only member the corpora -# exercise. Deliberately narrower than the class: a corpus name that -# joins it later should arrive UNEXPLAINED and be read once, not be -# absorbed silently. Widen the literal then, and record the new count. +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and this +# regex names one title and one particle because 'Mr. Van Nguyen' is +# the one name in the corpora that moves. Deliberately narrower than +# the class: a corpus name that joins it later should arrive +# UNEXPLAINED and be read once, not be absorbed silently. Widen the +# literals then, and record the new count. # -# The leading `\S+\.?` is the title slot, and it is the loose half: -# nothing in a regex over the raw string can ask whether a word is a -# title. `fields` is what carries the tightness -- and it is narrower +# The title slot is a LITERAL for that reason and not by accident. +# Nothing in a regex over the raw string can ask whether a word is a +# title, so the `(?i)^\S+\.?\s+van\b` this rule shipped with matched +# ANY first word: it reached 11 corpus names and explained 1. The +# other ten include 'Vincent van Gogh', 'VINCENT VAN GOGH' and +# 'Alex van Johnson' -- the "Van Johnson" family AGENTS.md names as a +# standard regression canary. name_regex rules sort ahead of +# fields-only ones and `fields` matches by subset, so a future +# regression on any of those would have been labelled fix(#367) and +# exited 0 -- a strict loss at this baseline, whose ledger was empty +# before. (`\.?` was inert into the bargain: `\S+` is greedy, so it +# had already taken the period.) +# +# `fields` is what carries the rest of the tightness -- and it is narrower # than the change rather than a description of it. The change moves # `title` as well: a word in BOTH vocabularies stops the transparency # scan and stays a title piece instead of being chained onto the name, @@ -269,5 +281,5 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # ["given", "family", "suffix"] is a superset of this diff's fields. No # fields-only rule exists here, so this file's copy is the plain # classification. -name_regex = "(?i)^\\S+\\.?\\s+van\\b" +name_regex = "(?i)^mr\\.\\s+van\\b" fields = ["given", "middle", "family"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 06fdfcbb..796b1062 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -51,15 +51,27 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # titled name reads as the same text reads untitled. # # The wider class is any title followed by a particle -- 'Dr. Van -# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and the -# regex names only `van` because that is the only member the corpora -# exercise. Deliberately narrower than the class: a corpus name that -# joins it later should arrive UNEXPLAINED and be read once, not be -# absorbed silently. Widen the literal then, and record the new count. +# Johnson', 'Sir de Mesnil', 'Sheik Abu Bakar' all move -- and this +# regex names one title and one particle because 'Mr. Van Nguyen' is +# the one name in the corpora that moves. Deliberately narrower than +# the class: a corpus name that joins it later should arrive +# UNEXPLAINED and be read once, not be absorbed silently. Widen the +# literals then, and record the new count. # -# The leading `\S+\.?` is the title slot, and it is the loose half: -# nothing in a regex over the raw string can ask whether a word is a -# title. `fields` is what carries the tightness -- and it is narrower +# The title slot is a LITERAL for that reason and not by accident. +# Nothing in a regex over the raw string can ask whether a word is a +# title, so the `(?i)^\S+\.?\s+van\b` this rule shipped with matched +# ANY first word: it reached 11 corpus names and explained 1. The +# other ten include 'Vincent van Gogh', 'VINCENT VAN GOGH' and +# 'Alex van Johnson' -- the "Van Johnson" family AGENTS.md names as a +# standard regression canary. name_regex rules sort ahead of +# fields-only ones and `fields` matches by subset, so a future +# regression on any of those would have been labelled fix(#367) and +# exited 0 -- a strict loss at this baseline, whose ledger was empty +# before. (`\.?` was inert into the bargain: `\S+` is greedy, so it +# had already taken the period.) +# +# `fields` is what carries the rest of the tightness -- and it is narrower # than the change rather than a description of it. The change moves # `title` as well: a word in BOTH vocabularies stops the transparency # scan and stays a title piece instead of being chained onto the name, @@ -80,5 +92,5 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # ["given", "family", "suffix"] is a superset of this diff's fields. No # fields-only rule exists here, so this file's copy is the plain # classification. -name_regex = "(?i)^\\S+\\.?\\s+van\\b" +name_regex = "(?i)^mr\\.\\s+van\\b" fields = ["given", "middle", "family"] From cc57ca2e6090bbe170dbbacec30c7bf89904ca00 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 22:36:09 -0700 Subject: [PATCH 9/9] Stop the respelled no-op row claiming 1.4 parity (#367) This PR respelled titled_ambiguous_particle_no_op_chain from "Dr. Van Jr." to "Do Van Jr." so it would still reach the chain loop's j > k + 1 guard -- a plain title is transparent now, so the old spelling never gets there. The row kept its default `parity` classification through the respelling, and the new text does not hold parity: 1.4.0 first='Do Van' last='Jr.' this branch title='Do' given='Van' suffix='Jr.' cases.py's own header says changing a row means changing its classification, and the respelling is this PR's doing even though the divergence is not: 'do' being a title and 'Jr.' routing to suffix are both 2.0-era, so neither belongs to #367. Classified `fix` rather than a specific slug because there are two independent causes, and the note now records what they are so the next reader does not have to re-measure 1.4.0 to find out. Co-Authored-By: Claude Opus 5 --- tests/v2/cases.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index c6b58d05..6f8ccd64 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -201,8 +201,13 @@ def __post_init__(self) -> None: "loop skips it without ever reaching the no-op. 'Do' is " "a title AND a particle, which stops the transparency " "scan, so the chain does fire on Van and the j > k + 1 " - "guard is what declines it -- the same output, reached " - "through the branch the row exists to pin"), + "guard is what declines it -- the same reading of the " + "name, reached through the branch the row exists to pin. " + "Not parity, and not #367's doing either: 1.4.0 reads " + "'Do Van Jr.' as first 'Do Van', last 'Jr.', so the " + "divergence is 2.0's suffix routing plus 'do' being a " + "title -- both older than this row's respelling", + classification="fix"), Case("initial_shaped_not_conjunction", "john e. smith", {"given": "john", "middle": "e.", "family": "smith"}, notes="v1 is_conjunction excludes initials at classify too"),