From 0d5fea407c90467a3144e228defafa10d14546b5 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 10 Oct 2026 12:03:24 -0700 Subject: [PATCH 1/2] Name every field a tail part's words land in (#629) Segment's comma-structure report on a part past the second comma said the part was "consumed as suffix best-effort". Since #603 a title word there reads as a title, so `John Smith, Jr., Dr. Bart` was told 'Dr.' was a suffix (this cycle, never shipped). #626's field_tail names only the first referent's field, and one part's words land in two, so PendingAmbiguity gains fields_tail: assemble names every field the referents hold, by role name in canonical order ("consumed as title and suffix best-effort"). A part landing wholly in suffix keeps the wording every release gave it. The field sweep reads the phrase and checks it for equality; its control is 6 failures on master, and the recorded claim count moves from 1155 to 1180. Measured against master over the corpora, the case table and a 97,650-text tail grid under three orders: only the detail of comma-structure reports on a part holding a title moves. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/decisions.md | 6 +++- docs/design/mechanisms.md | 2 +- docs/design/rules.md | 3 +- docs/release_log.rst | 2 +- nameparser/_pipeline/_assemble.py | 10 ++++++- nameparser/_pipeline/_segment.py | 11 +++++-- nameparser/_pipeline/_state.py | 9 ++++++ tests/v2/pipeline/test_assemble.py | 21 +++++++++++++ tests/v2/pipeline/test_segment.py | 28 +++++++++++++++++ tests/v2/test_cases.py | 48 ++++++++++++++++++++++-------- 11 files changed, 121 insertions(+), 21 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e15f80b4..43b22d98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -269,7 +269,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John "` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23), as were `rai`, `cha`, `ba` and `mc` before `decisions.md#suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions.md#vocabulary-collisions`, with #360's positional qualifier. -**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's "consumed as suffix" is not in it (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. +**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, or beside it as segment's "consumed as title and suffix" went, checked for equality since a report naming several fields can be wrong by naming one too many (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. **Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 25ba09eb..94aff26c 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -23,7 +23,11 @@ Entry conventions: MISSED, then caught in review: the first draft converted five emitters and its grid had no empty-head comma with a trailing title, the shape AGENTS.md's comma sweep already asks for; four of the review's five readers found the sixth emitter on `, Ma Dr.`. The draft also named only H1 and P6 as movers; P1's fold was found by stepping the stages, and the review of the fix found the opt-in movers O1 to O3, which no grid over the three orders reaches -- reverting the leading-particle emitter passed the whole suite until rows under those policies pinned it. MEASURED 2026-10-08, each report's field phrase compared against its tokens' final roles. RECIPE: texts are the 1,536 names of `tools/differential/corpus*.jsonl` and 347 `tests/v2/cases.py` texts (the table at 0c54662b, the parent of #614's first commit, less the corpus names; today's table adds rows this entry's review wrote for the shapes the grid missed, so it reads more wrong claims on master, 6 more as measured on the review), plus every head in {'John', 'John Smith', 'John Q. Smith', 'Mary Ann Smith', 'John van Smith', 'Juan de la Vega', 'John van', 'anh van', 'Jan Freiherr von Berg', 'Freiherr von Berg', 'Freiherr St', 'St', 'Dr. St', 'Freiherr', 'Dr.', 'Mr. and Mrs.', 'Attorney General of', 'abdul salam', 'Juan y Garcia', 'Jane Doe nee van der Berg', 'Dr. John Smith', 'JOHN SMITH', 'john smith', 'Smith', 'de Mesnil', 'Dr. de Mesnil', 'Kim Min', 'Kim Min Jun', 'Ortega y Gasset', 'van', 'Do', ''} followed by one or two words (repetition allowed) from the list [PhD, MA, Ma, DO, Do, do, vd, mc, Jr, III, V, V., VI, X.Y.Z., Ph. D., Esq., Prof., Sir, Ed, van, and, Jones, ba, John, Smith, Minnesota, de, St, Freiherr], space-joined and stripped; 20,000 draws after Python's `random.seed(626)`, each `random.choice(heads)`, then `random.randint(3, 4)` words each `random.choice(words)`, space-joined and stripped, both lists in the order written here; then, unstripped, `{head}, {word}`, `Smith, {head} {word}` with any double space replaced by one, and `{word}, {head}` for every head and word, and `, {w1} {w2}` and `(Bob), {w1} {w2}` for every ordered pair of words; the empty text dropped and duplicates removed -- 53,517 texts; each parsed under the three orders, so the opt-in movers are outside these figures; a claim is a match of `(reads? as|joins) ()` for the phrases `_FIELD_CLAIMS` in tests/v2/test_cases.py lists plus "the given name", and it is wrong when its tokens' roles are not among the fields the phrase names. On master (26cdb891) 2,725 of 89,092 claims were wrong, in three emitters: title-or-name's join clause (1,575), the trailing peel's picks (1,114) and the comma report (36); on the tree, 0. The comparator is master, which words the field at assign; the tree words it at assemble. Fingerprint against master over the differential corpora, master's case-table names and a seeded comma grid (13,090 names) under six policies (the three orders, `lenient_comma_suffixes=False`, a `/` suffix delimiter and `CapsSuffixes.OFF`): no role, kind, initials or rendering moves; the details that change are 6,395 comma reports on 1,086 names reworded from "the given name" to "a given name", 85 on 15 names now naming the family they land in, and 40 title-or-name join reports on 10 names now naming the family. Pinned by `test_a_report_names_the_field_its_word_lands_in` in tests/v2/test_cases.py, over every case row under all three orders except the locale rows (a row with its own policy only as declared), whose recorded control is 14 failures on master, and by `test_the_field_sweep_sees_the_claims_it_checks`, which records how many claims that sweep checks; the opt-in movers by the rows `Van Ivan Petrovich` (East Slavic), `Van Ali Veli oglu` (Turkic) and `Do Bishop Do` (FAMILY_FIRST with `middle_as_family`), which the sweep reads as declared. -Open: [#629](https://github.com/derek73/python-nameparser/issues/629) a comma-structure report on a part past the second comma says the part was "consumed as suffix" where a title in it now reads as a title (`John Smith, Jr., Freiherr von Richthofen`): the one report naming a field that the #626 entry's mechanism does not reach. +- 2026-10-10 #629 — A REPORT ON WORDS IN TWO FIELDS NAMES BOTH. Segment's `comma-structure` report on a part past the second comma said the part was "consumed as suffix best-effort", true through 2.3.0, when the whole part read as suffix. Since #603 (rules.md#C2) a title word there reads as a title, so `John Smith, Jr., Dr. Bart` was told 'Dr.' was a suffix with 'Dr.' in the title (this cycle, never shipped). The #626 entry's `field_tail` names the FIRST referent's field, which is wrong here: one part's words land in two fields. And the report is made in segment, before classify has tagged a word, so which words read as titles is not known at the emitter. + DECIDED: assemble words the fields from the final roles, as for #626, through a second tail, `PendingAmbiguity.fields_tail`: every field the referents hold, by bare role name, in canonical order, joined by "and" ("consumed as title and suffix best-effort"). Role names rather than FIELD_READING's phrases keep a part that lands wholly in suffix at the wording every release gave it. Not withdrawn where the part reads wholly as titles (`Secretary of State`): C2 keeps that flag on purpose, so it now says "consumed as title". + Declined: a wording that names no field ("beyond the recognized comma structures; read best-effort"), the issue's other option. It is true by construction, but it tells the caller less than the fields do and leaves nothing for the field guard to check. + MEASURED 2026-10-10, tree against master bc243869, every report's kind, detail, referents and roles plus the seven fields: no field, kind or referent moves; only the detail of `comma-structure` reports on a part holding a title moves. RECIPE: the names of `tools/differential/corpus*.jsonl` plus every `tests/v2/cases.py` text (1,925 texts), and a tail grid of 97,650 texts, `{head} {w1 … wk}` for k from 1 to 3 over the heads `John Smith, Jr.,`, `Smith, John,`, `Smith, Jr.,`, `,,`, `Smith,,`, `John Smith, PhD, Jr.,` and the words `Dr.`, `Dr`, `Attorney`, `Secretary of State`, `Lt.Gov.`, `Freiherr`, `St`, `von`, `van der`, `Bart`, `Jones`, `Jr.`, `PhD`, `MA`, `Ma`, `X.Y.Z.`, `nee`, `VI`, `V`, `'Bob'`, `(Bob)`, `Prof.`, `Mr. and Mrs.`, `I`, `y`; each under the default order, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, through `Parser(policy=...)` with the default lexicon. Named texts: 3 move (`John Smith, Jr., Freiherr von Richthofen` to "title and suffix", `Eric H. Holder, Jr., Secretary of State` and `John Smith, Jr., and Secretary of State` to "title"), each under three orders; 81 suffix-only reports keep their wording. Grid: 150,576 reports on 50,192 texts move to "title and suffix" and 12,396 on 4,132 to "title"; 84,381 suffix-only reports keep theirs. No referent landed outside title and suffix. + Held by `test_a_report_names_the_field_its_word_lands_in`, which reads "consumed as …" beside `_FIELD_CLAIMS` and requires EQUALITY with the fields landed in, since a phrase naming several fields is wrong by naming one too many as much as by leaving one out; its recorded control for this phrasing is 6 failures on master, the two rows above under three orders. ### A2 — the empty name diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 068715cc..50d8a171 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -114,7 +114,7 @@ Problem shape. A retired default rides in on old pickles, but a user may have de ## AMBIGUITY-AT-THE-DECISION-SITE — emit where the branch is taken -Problem shape. An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary. Contract statement. Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626; a comma-structure report on a part past the second comma still names "suffix" for a part holding a title, #629). How it works. PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on +Problem shape. An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary. Contract statement. Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626), or for a report on words that land in more than one field, every field they hold (`fields_tail`, #629, segment's report on a part past the second comma). How it works. PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on "the code got here" instead of "the outcome differed" once reported a fork for all ambiguous particles on "Dr. Van Jr.". Lives in. The AmbiguityKind emitters across _pipeline/ (rule A1 is the observable contract). Reach for it when. Adding any ambiguous vocabulary or any new fork — count the deciding sites, then count the emitters. diff --git a/docs/design/rules.md b/docs/design/rules.md index 3be3c5fc..9ff2c7f4 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -2431,7 +2431,8 @@ A1. Rationale: a caller can only act on doubt that is reported. connective-or-initial fork offered, and only that fork's report goes — the generation's own stands. And a report that names the field its word was read into names the field the word holds in - the result, whatever rule moved it after the report was made. + the result, whatever rule moved it after the report was made; a + report on words that land in more than one field names each. "Van Johnson" → ambiguities=("particle-or-given",) "JOHN QUINCY SMITH I" → ambiguities=("suffix-or-name",) "Jane „JD Smith" → ambiguities=("unbalanced-delimiter",) diff --git a/docs/release_log.rst b/docs/release_log.rst index 1097d6db..c3e5b477 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -44,7 +44,7 @@ Release Log - **Fix a comma part read as credentials being taken apart again by a later join.** ``HumanName("John Smith, Ph. D. and Mary Jones")`` gives first ``John``, last ``Smith``, suffix ``Ph. D. and Mary Jones``, where 2.3.0 gave first ``Ph. D. and Mary``, middle ``Jones``, last ``John Smith``, and ``parse()`` reports each name word the part took: the parser now decides once, after every word is recognized, whether the part after a comma holds the credentials, and nothing joins across that decision afterwards. Titles joined by ``and`` stay one title (``John Smith, Mr. and Mrs.`` keeps title ``Mr. and Mrs.``). A surname joined by ``y`` counts as its words, as #575 already counted it in front of a credential: ``Ortega y Gasset, Dr.`` gives title ``Dr.``, first ``Ortega``, middle ``y``, last ``Gasset``, where 2.3.0 gave last ``Ortega y Gasset``. And a surname with a suffix in it before the comma is the surname, as ``Smith Jr., John`` always read it: ``Smith Jr., Esq.`` gives last ``Smith``, suffix ``Jr., Esq.``, where 2.3.0 gave first ``Smith``, and ``Smith Jr., V`` gives first ``V``, last ``Smith``, suffix ``Jr.``, where 2.3.0 gave first ``Smith``, suffix ``Jr., V``. See the #613 entry under ``C1`` in ``docs/design/decisions.md`` (closes #613) - - **Change a title word in a part after a second comma to read as a title.** ``HumanName("Eric H. Holder, Jr., Attorney General")`` gives title ``Attorney General``, suffix ``Jr.``, where 1.4.0 through 2.3.0 gave suffix ``Jr., Attorney General``, and the part no longer reports ``comma-structure``. A word that is also a suffix stays a suffix (``John Smith, MD, Ms``). A title joined by a connective is read as a title, but its part keeps the report (``Secretary of State``). (#603) + - **Change a title word in a part after a second comma to read as a title.** ``HumanName("Eric H. Holder, Jr., Attorney General")`` gives title ``Attorney General``, suffix ``Jr.``, where 1.4.0 through 2.3.0 gave suffix ``Jr., Attorney General``, and the part no longer reports ``comma-structure``. A word that is also a suffix stays a suffix (``John Smith, MD, Ms``). A title joined by a connective is read as a title, but its part keeps the report (``Secretary of State``), which names the fields the part's words land in: ``John Smith, Jr., Dr. Bart`` reports its last part "consumed as title and suffix". (#603, #629) - **Add the Catalan and Polish surname link.** ``parse("Josep Carod i Rovira")`` gives family ``Carod i Rovira``, where every release from 1.4.0 through 2.3.0 gave middle ``Carod i`` with family ``Rovira``; ``Josep Lluis Carod i Rovira`` gives middle ``Lluis`` with that same family; and ``Carod i Rovira, Josep`` gives it too, where they read family ``Carod Rovira`` and took the link into ``suffix`` as a generation marker. ``i`` is connective vocabulary now, the way ``y`` already was, and a connective counts as a name word wherever the three-word carve-out counts them -- whatever else the vocabulary says the word is, which matters here because ``i`` is also the roman numeral. A connective that is also generational vocabulary joins only where a name word stands on each side of it, so ``John Quincy Smith i`` keeps suffix ``i``, ``Josep Lluis Carod i III`` keeps suffix ``i III``, and the two-word ``Carod i`` keeps its generation reading. Written wholly in one case the letter reads as an initial and says so: ``JOSEP CAROD I ROVIRA`` and ``josep carod i rovira`` keep the fields they had and gain a ``conjunction-or-initial`` report, which a one-case name gains wherever a bare ``i`` or ``I`` stands among the name's own words -- a letter inside a maiden clause is read by the clause's rules and stays silent, as ``e`` already was -- and in an all-lower name that reading can move a field, each such name now reading as its all-caps twin already did (``parse("john smith i jr")`` gives middle ``smith``, family ``i`` and suffix ``jr`` where it gave family ``smith`` and suffix ``i jr``). Case repair follows the reading: a lower-case ``i`` the parse read as the generation is still title-cased by ``capitalize(force=True)`` (``Carod i`` gives ``Carod I``, as every release did), while one standing among the name words keeps its lower case as ``y`` always has (``Carod i Rovira`` gives ``Carod i Rovira``, where ``Carod I Rovira`` was the pre-2.4 answer). A link inside a maiden clause stays in the birth name, which no release read that way: ``HumanName("Jane Doe nee Puig i Soler")`` gives maiden ``Puig i Soler`` with last ``Doe``, where 2.0 through 2.3 ended the birth name at the link and gave maiden ``Puig`` with middle ``Doe i``, last ``Soler`` -- and the same words would have joined into last ``Doe i Soler`` under the change above, carrying a word of the birth name into the current surname. ``Jane Doe née Kowalska i Nowak`` moves with it, as does the all-lower ``jane doe nee puig i soler``; the ``y`` spelling always read this way and is untouched. The link still has to be joining: ``Jane Doe nee Puig i`` keeps maiden ``Puig`` with suffix ``i``, and ``Jane Doe nee Puig i III`` suffix ``i III``. A caller with Catalan or Polish data removes the entry from ``conjunctions_ambiguous`` and gets the join in the one-case names too; a caller who wants none of this removes ``i`` from ``conjunctions``, which restores every prior FIELD and every prior report, with two readings it does not restore and cannot: a letter the two vocabularies disagree about being an initial reads as one here and as the generation there, and case repair leaves a connective the parse placed among the NAME words in lower case where the off switch title-cases it -- ``parse("Dr. John i Smith").capitalized(force=True)`` keeps ``i`` where the off switch gives ``Dr. John I Smith``, and ``Carod i Rovira`` and ``Josep i Rovira`` are the same shape. Those two are the whole of what the switch does not undo, and ``tests/v2/test_properties.py`` states them as its invariants' only exemptions. A delimiter the caller declares through ``Policy(extra_suffix_delimiters=...)`` parts a trailing suffix part as a comma does (see the suffix-delimiter entry below), so no link joins across one: under ``(" - ",)``, ``Smith, John, PhD - i Soler`` keeps suffix ``PhD, i Soler``, as 2.3.0 read it. The default policy declares no such delimiter. See the ``P3`` and ``M2`` entries of ``docs/design/decisions.md`` (closes #397, closes #538) diff --git a/nameparser/_pipeline/_assemble.py b/nameparser/_pipeline/_assemble.py index a3921d4f..8adeb74e 100644 --- a/nameparser/_pipeline/_assemble.py +++ b/nameparser/_pipeline/_assemble.py @@ -2,7 +2,8 @@ Consumes: tokens (all roles set), dropped, ambiguities (by index; one carrying ``field_tail`` is worded from its word's final role, -#626). +#626, and one carrying ``fields_tail`` from its words' final roles, +#629). Produces: a validated ParsedName -- the constructor re-checks every invariant (span order/bounds, ambiguity subset), so a pipeline bug that would produce an invalid result dies HERE, not in a renderer @@ -124,6 +125,13 @@ def assemble(state: ParseState) -> ParsedName: continue detail = (f"{detail}{FIELD_READING[materialized[0].role]}" f"{pending.field_tail}") + elif pending.fields_tail is not None: + if not materialized: + continue + held = {t.role for t in materialized} + detail = (f"{detail}" + f"{' and '.join([r.value for r in Role if r in held])}" + f"{pending.fields_tail}") ambiguities.append(Ambiguity(pending.kind, detail, materialized)) return ParsedName(original=state.original, tokens=tuple(final.values()), diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index e6171e27..cbb10b4e 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -195,11 +195,18 @@ def titles_and_suffixes(seg: tuple[int, ...]) -> bool: if (seg and not suffixy(seg) and not suffixy(seg, case_class()) and not class_run(seg) and not titles_and_suffixes(seg)): texts_joined = " ".join([state.tokens[i].text for i in seg]) + # The part's words land in two fields since #603 -- a title + # word reads as a title, the rest as suffixes -- and which + # words read as titles is group's and assign's call, so + # the fields are worded at assemble from the final roles + # (`PendingAmbiguity.fields_tail`, #629): 'consumed as + # suffix' had told 'John Smith, Jr., Dr. Bart' its 'Dr.' + # was a suffix. ambiguities.append(PendingAmbiguity( AmbiguityKind.COMMA_STRUCTURE, f"segment {texts_joined!r} beyond the recognized comma " - f"structures; consumed as suffix best-effort", - tuple(seg))) + f"structures; consumed as ", + tuple(seg), fields_tail=" best-effort")) return copy_with(state, segments=tuple(groups), structure=Structure.FAMILY_COMMA, ambiguities=tuple(ambiguities), diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index dd37336d..08e73f36 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -141,6 +141,14 @@ class PendingAmbiguity: assemble tests ``is not None`` -- and assemble joins them around the FIELD_READING of the first referent's final role, the one place that knows it (#626). Until then ``detail`` is a fragment. + + ``fields_tail`` is the same for a report on a run of words that + can land in more than one field: assemble names every field its + referents hold, as bare role names in canonical order joined by + "and" ("title and suffix"). Its one emitter is segment's report on + a part past the second comma, whose title words read as titles + and the rest as suffixes (rules.md#C2, #629). An emitter sets one + tail or neither. """ kind: AmbiguityKind @@ -148,6 +156,7 @@ class PendingAmbiguity: indices: tuple[int, ...] = () origin: int | None = None field_tail: str | None = None + fields_tail: str | None = None #: How a report names the field a word was read into, by final role. diff --git a/tests/v2/pipeline/test_assemble.py b/tests/v2/pipeline/test_assemble.py index 79efbad2..40497108 100644 --- a/tests/v2/pipeline/test_assemble.py +++ b/tests/v2/pipeline/test_assemble.py @@ -250,3 +250,24 @@ def test_a_field_tail_report_names_its_first_referent_s_field() -> None: PendingAmbiguity(AmbiguityKind.TITLE_OR_NAME, "read as ", (0, 1), field_tail="")) assert [a.detail for a in pn.ambiguities] == ["read as a family name"] + + +def test_a_fields_tail_report_names_every_field_in_canonical_order() -> None: + # #629: a report on a run whose words land in more than one field + # names each field once, by role name, in canonical order however + # the referents are ordered, and in an emptied name it goes like a + # field_tail report. + tokens = (WorkToken("Jr.", Span(0, 3), role=Role.SUFFIX), + WorkToken("Dr.", Span(4, 7), role=Role.TITLE), + WorkToken("Bart", Span(8, 12), role=Role.SUFFIX)) + pn = _with_reports( + "Jr. Dr. Bart", tokens, + PendingAmbiguity(AmbiguityKind.COMMA_STRUCTURE, "consumed as ", + (0, 1, 2), fields_tail=" best-effort")) + assert [a.detail for a in pn.ambiguities] == [ + "consumed as title and suffix best-effort"] + pn = _with_reports( + "-", (WorkToken("-", Span(0, 1), role=Role.SUFFIX),), + PendingAmbiguity(AmbiguityKind.COMMA_STRUCTURE, "consumed as ", + (0,), fields_tail=" best-effort")) + assert not pn.ambiguities diff --git a/tests/v2/pipeline/test_segment.py b/tests/v2/pipeline/test_segment.py index 21291ece..873eebf6 100644 --- a/tests/v2/pipeline/test_segment.py +++ b/tests/v2/pipeline/test_segment.py @@ -1,4 +1,6 @@ from nameparser._lexicon import Lexicon +from nameparser._pipeline import run +from nameparser._pipeline._assemble import assemble from nameparser._pipeline._extract import extract_delimited from nameparser._pipeline._segment import segment from nameparser._pipeline._state import ParseState, Structure @@ -49,6 +51,32 @@ def test_excess_non_suffix_segment_flags_comma_structure() -> None: assert AmbiguityKind.COMMA_STRUCTURE in kinds +def test_the_comma_structure_report_names_the_fields_its_part_lands_in() -> None: + # #629: a title word past the second comma reads as a title + # (rules.md#C2, #603), so the part lands in two fields and the + # report names what each word landed in, worded at assemble from + # the final roles. A part that lands wholly in suffix keeps the + # wording every release through 2.3.0 gave it. + lex = _LEX.add(titles={"dr", "secretary", "state"}, conjunctions={"of"}) + + def detail(text: str) -> str: + [report] = [a for a in assemble(run(ParseState( + original=text, lexicon=lex, policy=Policy()))).ambiguities + if a.kind is AmbiguityKind.COMMA_STRUCTURE] + return report.detail + + assert detail("John Smith, Jr., Dr. Bart") == ( + "segment 'Dr. Bart' beyond the recognized comma structures; " + "consumed as title and suffix best-effort") + assert detail("John Smith, Jr., Bart") == ( + "segment 'Bart' beyond the recognized comma structures; " + "consumed as suffix best-effort") + # C2: a part a connective joins into one title keeps the flag + assert detail("John Smith, Jr., Secretary of State") == ( + "segment 'Secretary of State' beyond the recognized comma " + "structures; consumed as title best-effort") + + def test_empty_input_yields_no_segments() -> None: assert _segmented("").segments == () diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 300e0729..3891e909 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -2,6 +2,7 @@ runner (migration plan) consumes the same CASES.""" import functools import re +from collections.abc import Iterator from typing import Any import pytest @@ -120,6 +121,21 @@ def test_the_family_partitions_into_particles_and_base( r"\b(?:reads? as|joins) (" + "|".join(map(re.escape, sorted(_FIELD_CLAIMS, key=len, reverse=True))) + r")\b") +#: segment's report on a part past the second comma names EVERY field +#: the part's words land in, as role names ("consumed as title and +#: suffix best-effort", #629), so it is checked for equality rather +#: than inclusion: a named field nothing landed in is as wrong as a +#: field left out. +_FIELDS_CLAIM = re.compile(r"\bconsumed as ([a-z]+(?: and [a-z]+)*) best-effort") + + +def _claims(detail: str) -> Iterator[tuple[str, frozenset[Role], bool]]: + """The field claims in a report's detail: the phrase, the fields + that make it true, and whether they must ALL be held.""" + if m := _FIELD_CLAIM.search(detail): + yield m[0], _FIELD_CLAIMS[m[1]], False + if m := _FIELDS_CLAIM.search(detail): + yield m[0], frozenset(map(Role, m[1].split(" and "))), True @pytest.mark.parametrize("case,order", _SWEPT_ROWS, ids=_SWEPT_IDS) @@ -153,17 +169,23 @@ def test_a_report_names_the_field_its_word_lands_in( and so not one `_FIELD_CLAIMS` lists, and `, Ma Dr.` (H1 moving it to the family) is pinned instead by test_assign.py's verbatim detail. No row reached that shape until #626's review. + + Segment's report on a part past the second comma is read by + `_FIELDS_CLAIM` and checked for equality (#629). Its control, + measured 2026-10-10 over master's parser (bc243869) with this + test: 6 failures, the two rows holding a title past the second + comma (`John Smith, Jr., Freiherr von Richthofen` and `John + Smith, Jr., and Secretary of State`) under all three orders, + each told 'consumed as suffix'. """ pn = _swept(case.id, order) for a in pn.ambiguities: - m = _FIELD_CLAIM.search(a.detail) - if m is None: - continue landed = {t.role for t in a.tokens} - assert landed <= _FIELD_CLAIMS[m[1]], ( - f"{case.text!r}: {a.kind.value} says {m[0]!r} but its " - f"tokens landed in {sorted(r.value for r in landed)}: " - f"{a.detail!r}") + for phrase, fields, every in _claims(a.detail): + assert landed == fields if every else landed <= fields, ( + f"{case.text!r}: {a.kind.value} says {phrase!r} but its " + f"tokens landed in {sorted(r.value for r in landed)}: " + f"{a.detail!r}") def test_the_field_sweep_sees_the_claims_it_checks() -> None: @@ -172,12 +194,12 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: dropped -- so the count of claims it checks is recorded here. Move it deliberately when a row or a phrasing moves, never to 0.""" claims = sum( - _FIELD_CLAIM.search(a.detail) is not None - for case, order in _SWEPT_ROWS - for a in _swept(case.id, order).ambiguities) - assert claims == 1155, ( - f"the field sweep checks {claims} claims, recorded as 1155 on " - f"2026-10-08") + 1 for case, order in _SWEPT_ROWS + for a in _swept(case.id, order).ambiguities + for _ in _claims(a.detail)) + assert claims == 1180, ( + f"the field sweep checks {claims} claims, recorded as 1180 on " + f"2026-10-10") #: Case.__post_init__'s shape checks, each probed for the message that From 14c750decb7a3e5961cb761af78ae28fa0c82344 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 10 Oct 2026 12:18:41 -0700 Subject: [PATCH 2/2] Word #629's report without naming a field Replaces 0d5fea40's mechanism with the simpler fix the issue offered: the comma-structure report on a part past the second comma now says "beyond the recognized comma structures; read best-effort" and names no field, since the part's words can land in two and the fields say where each went. The input is malformed, so the report is not worth machinery. Reverts PendingAmbiguity.fields_tail, assemble's branch for it, the field guard's equality check and the A1 clause. Review of 0d5fea40 found that branch withdrew the report from a name emptied of content (", , -"), against rules.md#A2; with nothing worded at assemble that case keeps its report as on master. Measured against master over the corpora, the case table and the tail grid under three orders: only the detail of comma-structure reports moves, and no report appears or goes. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/decisions.md | 9 +++--- docs/design/mechanisms.md | 2 +- docs/design/rules.md | 3 +- docs/release_log.rst | 2 +- nameparser/_pipeline/_assemble.py | 10 +------ nameparser/_pipeline/_segment.py | 15 ++++------ nameparser/_pipeline/_state.py | 9 ------ tests/v2/pipeline/test_assemble.py | 21 ------------- tests/v2/pipeline/test_segment.py | 32 +++++--------------- tests/v2/test_cases.py | 48 ++++++++---------------------- 11 files changed, 36 insertions(+), 117 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 43b22d98..2f44f85a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -269,7 +269,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John "` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23), as were `rai`, `cha`, `ba` and `mc` before `decisions.md#suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions.md#vocabulary-collisions`, with #360's positional qualifier. -**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, or beside it as segment's "consumed as title and suffix" went, checked for equality since a report naming several fields can be wrong by naming one too many (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. +**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. **Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 94aff26c..6029fb2e 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -23,11 +23,10 @@ Entry conventions: MISSED, then caught in review: the first draft converted five emitters and its grid had no empty-head comma with a trailing title, the shape AGENTS.md's comma sweep already asks for; four of the review's five readers found the sixth emitter on `, Ma Dr.`. The draft also named only H1 and P6 as movers; P1's fold was found by stepping the stages, and the review of the fix found the opt-in movers O1 to O3, which no grid over the three orders reaches -- reverting the leading-particle emitter passed the whole suite until rows under those policies pinned it. MEASURED 2026-10-08, each report's field phrase compared against its tokens' final roles. RECIPE: texts are the 1,536 names of `tools/differential/corpus*.jsonl` and 347 `tests/v2/cases.py` texts (the table at 0c54662b, the parent of #614's first commit, less the corpus names; today's table adds rows this entry's review wrote for the shapes the grid missed, so it reads more wrong claims on master, 6 more as measured on the review), plus every head in {'John', 'John Smith', 'John Q. Smith', 'Mary Ann Smith', 'John van Smith', 'Juan de la Vega', 'John van', 'anh van', 'Jan Freiherr von Berg', 'Freiherr von Berg', 'Freiherr St', 'St', 'Dr. St', 'Freiherr', 'Dr.', 'Mr. and Mrs.', 'Attorney General of', 'abdul salam', 'Juan y Garcia', 'Jane Doe nee van der Berg', 'Dr. John Smith', 'JOHN SMITH', 'john smith', 'Smith', 'de Mesnil', 'Dr. de Mesnil', 'Kim Min', 'Kim Min Jun', 'Ortega y Gasset', 'van', 'Do', ''} followed by one or two words (repetition allowed) from the list [PhD, MA, Ma, DO, Do, do, vd, mc, Jr, III, V, V., VI, X.Y.Z., Ph. D., Esq., Prof., Sir, Ed, van, and, Jones, ba, John, Smith, Minnesota, de, St, Freiherr], space-joined and stripped; 20,000 draws after Python's `random.seed(626)`, each `random.choice(heads)`, then `random.randint(3, 4)` words each `random.choice(words)`, space-joined and stripped, both lists in the order written here; then, unstripped, `{head}, {word}`, `Smith, {head} {word}` with any double space replaced by one, and `{word}, {head}` for every head and word, and `, {w1} {w2}` and `(Bob), {w1} {w2}` for every ordered pair of words; the empty text dropped and duplicates removed -- 53,517 texts; each parsed under the three orders, so the opt-in movers are outside these figures; a claim is a match of `(reads? as|joins) ()` for the phrases `_FIELD_CLAIMS` in tests/v2/test_cases.py lists plus "the given name", and it is wrong when its tokens' roles are not among the fields the phrase names. On master (26cdb891) 2,725 of 89,092 claims were wrong, in three emitters: title-or-name's join clause (1,575), the trailing peel's picks (1,114) and the comma report (36); on the tree, 0. The comparator is master, which words the field at assign; the tree words it at assemble. Fingerprint against master over the differential corpora, master's case-table names and a seeded comma grid (13,090 names) under six policies (the three orders, `lenient_comma_suffixes=False`, a `/` suffix delimiter and `CapsSuffixes.OFF`): no role, kind, initials or rendering moves; the details that change are 6,395 comma reports on 1,086 names reworded from "the given name" to "a given name", 85 on 15 names now naming the family they land in, and 40 title-or-name join reports on 10 names now naming the family. Pinned by `test_a_report_names_the_field_its_word_lands_in` in tests/v2/test_cases.py, over every case row under all three orders except the locale rows (a row with its own policy only as declared), whose recorded control is 14 failures on master, and by `test_the_field_sweep_sees_the_claims_it_checks`, which records how many claims that sweep checks; the opt-in movers by the rows `Van Ivan Petrovich` (East Slavic), `Van Ali Veli oglu` (Turkic) and `Do Bishop Do` (FAMILY_FIRST with `middle_as_family`), which the sweep reads as declared. -- 2026-10-10 #629 — A REPORT ON WORDS IN TWO FIELDS NAMES BOTH. Segment's `comma-structure` report on a part past the second comma said the part was "consumed as suffix best-effort", true through 2.3.0, when the whole part read as suffix. Since #603 (rules.md#C2) a title word there reads as a title, so `John Smith, Jr., Dr. Bart` was told 'Dr.' was a suffix with 'Dr.' in the title (this cycle, never shipped). The #626 entry's `field_tail` names the FIRST referent's field, which is wrong here: one part's words land in two fields. And the report is made in segment, before classify has tagged a word, so which words read as titles is not known at the emitter. - DECIDED: assemble words the fields from the final roles, as for #626, through a second tail, `PendingAmbiguity.fields_tail`: every field the referents hold, by bare role name, in canonical order, joined by "and" ("consumed as title and suffix best-effort"). Role names rather than FIELD_READING's phrases keep a part that lands wholly in suffix at the wording every release gave it. Not withdrawn where the part reads wholly as titles (`Secretary of State`): C2 keeps that flag on purpose, so it now says "consumed as title". - Declined: a wording that names no field ("beyond the recognized comma structures; read best-effort"), the issue's other option. It is true by construction, but it tells the caller less than the fields do and leaves nothing for the field guard to check. - MEASURED 2026-10-10, tree against master bc243869, every report's kind, detail, referents and roles plus the seven fields: no field, kind or referent moves; only the detail of `comma-structure` reports on a part holding a title moves. RECIPE: the names of `tools/differential/corpus*.jsonl` plus every `tests/v2/cases.py` text (1,925 texts), and a tail grid of 97,650 texts, `{head} {w1 … wk}` for k from 1 to 3 over the heads `John Smith, Jr.,`, `Smith, John,`, `Smith, Jr.,`, `,,`, `Smith,,`, `John Smith, PhD, Jr.,` and the words `Dr.`, `Dr`, `Attorney`, `Secretary of State`, `Lt.Gov.`, `Freiherr`, `St`, `von`, `van der`, `Bart`, `Jones`, `Jr.`, `PhD`, `MA`, `Ma`, `X.Y.Z.`, `nee`, `VI`, `V`, `'Bob'`, `(Bob)`, `Prof.`, `Mr. and Mrs.`, `I`, `y`; each under the default order, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, through `Parser(policy=...)` with the default lexicon. Named texts: 3 move (`John Smith, Jr., Freiherr von Richthofen` to "title and suffix", `Eric H. Holder, Jr., Secretary of State` and `John Smith, Jr., and Secretary of State` to "title"), each under three orders; 81 suffix-only reports keep their wording. Grid: 150,576 reports on 50,192 texts move to "title and suffix" and 12,396 on 4,132 to "title"; 84,381 suffix-only reports keep theirs. No referent landed outside title and suffix. - Held by `test_a_report_names_the_field_its_word_lands_in`, which reads "consumed as …" beside `_FIELD_CLAIMS` and requires EQUALITY with the fields landed in, since a phrase naming several fields is wrong by naming one too many as much as by leaving one out; its recorded control for this phrasing is 6 failures on master, the two rows above under three orders. +- 2026-10-10 #629 — A REPORT ON WORDS IN TWO FIELDS NAMES NEITHER. Segment's `comma-structure` report on a part past the second comma said the part was "consumed as suffix best-effort", true through 2.3.0, when the whole part read as suffix. Since #603 (rules.md#C2) a title word there reads as a title, so `John Smith, Jr., Dr. Bart` was told 'Dr.' was a suffix with 'Dr.' in the title (this cycle, never shipped). + DECIDED: the report names no field: "segment 'Dr. Bart' beyond the recognized comma structures; read best-effort". It says what is certain, that the part is outside the recognized comma structures, and the fields say where each word went. The input is malformed, so the report is not worth machinery: the fix is one string, and a part that reads wholly as suffix loses the word "suffix" from its detail, which is not contract (kinds are). + Declined: naming every field the part's words land in, worded at assemble from the final roles like #626's `field_tail` ("consumed as title and suffix best-effort"). It was built (0d5fea40, PR #630) and reviewed: it needed a second tail on `PendingAmbiguity`, an assemble branch and an equality check in the field guard, and the branch withdrew the report from a name emptied of content (`", , -"`), against rules.md#A2's "ambiguities born from its punctuation survive on the empty result", which no grid caught because every grid word had letters. More code to say more about garbage input; the user-facing gain did not pay for it. + MEASURED 2026-10-10, tree against master bc243869 over the names of `tools/differential/corpus*.jsonl` plus every `tests/v2/cases.py` text, each under the three orders, comparing every report's kind, detail and referents and the seven fields: only the detail of `comma-structure` reports moves, and no report appears or goes. ### A2 — the empty name diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 50d8a171..8bb1a215 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -114,7 +114,7 @@ Problem shape. A retired default rides in on old pickles, but a user may have de ## AMBIGUITY-AT-THE-DECISION-SITE — emit where the branch is taken -Problem shape. An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary. Contract statement. Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626), or for a report on words that land in more than one field, every field they hold (`fields_tail`, #629, segment's report on a part past the second comma). How it works. PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on +Problem shape. An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary. Contract statement. Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626). Segment's report on a part past the second comma names no field at all, since that part's words can land in two (#629). How it works. PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on "the code got here" instead of "the outcome differed" once reported a fork for all ambiguous particles on "Dr. Van Jr.". Lives in. The AmbiguityKind emitters across _pipeline/ (rule A1 is the observable contract). Reach for it when. Adding any ambiguous vocabulary or any new fork — count the deciding sites, then count the emitters. diff --git a/docs/design/rules.md b/docs/design/rules.md index 9ff2c7f4..3be3c5fc 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -2431,8 +2431,7 @@ A1. Rationale: a caller can only act on doubt that is reported. connective-or-initial fork offered, and only that fork's report goes — the generation's own stands. And a report that names the field its word was read into names the field the word holds in - the result, whatever rule moved it after the report was made; a - report on words that land in more than one field names each. + the result, whatever rule moved it after the report was made. "Van Johnson" → ambiguities=("particle-or-given",) "JOHN QUINCY SMITH I" → ambiguities=("suffix-or-name",) "Jane „JD Smith" → ambiguities=("unbalanced-delimiter",) diff --git a/docs/release_log.rst b/docs/release_log.rst index c3e5b477..ff83478e 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -44,7 +44,7 @@ Release Log - **Fix a comma part read as credentials being taken apart again by a later join.** ``HumanName("John Smith, Ph. D. and Mary Jones")`` gives first ``John``, last ``Smith``, suffix ``Ph. D. and Mary Jones``, where 2.3.0 gave first ``Ph. D. and Mary``, middle ``Jones``, last ``John Smith``, and ``parse()`` reports each name word the part took: the parser now decides once, after every word is recognized, whether the part after a comma holds the credentials, and nothing joins across that decision afterwards. Titles joined by ``and`` stay one title (``John Smith, Mr. and Mrs.`` keeps title ``Mr. and Mrs.``). A surname joined by ``y`` counts as its words, as #575 already counted it in front of a credential: ``Ortega y Gasset, Dr.`` gives title ``Dr.``, first ``Ortega``, middle ``y``, last ``Gasset``, where 2.3.0 gave last ``Ortega y Gasset``. And a surname with a suffix in it before the comma is the surname, as ``Smith Jr., John`` always read it: ``Smith Jr., Esq.`` gives last ``Smith``, suffix ``Jr., Esq.``, where 2.3.0 gave first ``Smith``, and ``Smith Jr., V`` gives first ``V``, last ``Smith``, suffix ``Jr.``, where 2.3.0 gave first ``Smith``, suffix ``Jr., V``. See the #613 entry under ``C1`` in ``docs/design/decisions.md`` (closes #613) - - **Change a title word in a part after a second comma to read as a title.** ``HumanName("Eric H. Holder, Jr., Attorney General")`` gives title ``Attorney General``, suffix ``Jr.``, where 1.4.0 through 2.3.0 gave suffix ``Jr., Attorney General``, and the part no longer reports ``comma-structure``. A word that is also a suffix stays a suffix (``John Smith, MD, Ms``). A title joined by a connective is read as a title, but its part keeps the report (``Secretary of State``), which names the fields the part's words land in: ``John Smith, Jr., Dr. Bart`` reports its last part "consumed as title and suffix". (#603, #629) + - **Change a title word in a part after a second comma to read as a title.** ``HumanName("Eric H. Holder, Jr., Attorney General")`` gives title ``Attorney General``, suffix ``Jr.``, where 1.4.0 through 2.3.0 gave suffix ``Jr., Attorney General``, and the part no longer reports ``comma-structure``. A word that is also a suffix stays a suffix (``John Smith, MD, Ms``). A title joined by a connective is read as a title, but its part keeps the report (``Secretary of State``). Since the part's words can land in two fields, the report no longer names one: ``parse("John Smith, Jr., Dr. Bart")`` reports its last part as "beyond the recognized comma structures; read best-effort", where 2.3.0 said "consumed as suffix best-effort". (#603, closes #629) - **Add the Catalan and Polish surname link.** ``parse("Josep Carod i Rovira")`` gives family ``Carod i Rovira``, where every release from 1.4.0 through 2.3.0 gave middle ``Carod i`` with family ``Rovira``; ``Josep Lluis Carod i Rovira`` gives middle ``Lluis`` with that same family; and ``Carod i Rovira, Josep`` gives it too, where they read family ``Carod Rovira`` and took the link into ``suffix`` as a generation marker. ``i`` is connective vocabulary now, the way ``y`` already was, and a connective counts as a name word wherever the three-word carve-out counts them -- whatever else the vocabulary says the word is, which matters here because ``i`` is also the roman numeral. A connective that is also generational vocabulary joins only where a name word stands on each side of it, so ``John Quincy Smith i`` keeps suffix ``i``, ``Josep Lluis Carod i III`` keeps suffix ``i III``, and the two-word ``Carod i`` keeps its generation reading. Written wholly in one case the letter reads as an initial and says so: ``JOSEP CAROD I ROVIRA`` and ``josep carod i rovira`` keep the fields they had and gain a ``conjunction-or-initial`` report, which a one-case name gains wherever a bare ``i`` or ``I`` stands among the name's own words -- a letter inside a maiden clause is read by the clause's rules and stays silent, as ``e`` already was -- and in an all-lower name that reading can move a field, each such name now reading as its all-caps twin already did (``parse("john smith i jr")`` gives middle ``smith``, family ``i`` and suffix ``jr`` where it gave family ``smith`` and suffix ``i jr``). Case repair follows the reading: a lower-case ``i`` the parse read as the generation is still title-cased by ``capitalize(force=True)`` (``Carod i`` gives ``Carod I``, as every release did), while one standing among the name words keeps its lower case as ``y`` always has (``Carod i Rovira`` gives ``Carod i Rovira``, where ``Carod I Rovira`` was the pre-2.4 answer). A link inside a maiden clause stays in the birth name, which no release read that way: ``HumanName("Jane Doe nee Puig i Soler")`` gives maiden ``Puig i Soler`` with last ``Doe``, where 2.0 through 2.3 ended the birth name at the link and gave maiden ``Puig`` with middle ``Doe i``, last ``Soler`` -- and the same words would have joined into last ``Doe i Soler`` under the change above, carrying a word of the birth name into the current surname. ``Jane Doe née Kowalska i Nowak`` moves with it, as does the all-lower ``jane doe nee puig i soler``; the ``y`` spelling always read this way and is untouched. The link still has to be joining: ``Jane Doe nee Puig i`` keeps maiden ``Puig`` with suffix ``i``, and ``Jane Doe nee Puig i III`` suffix ``i III``. A caller with Catalan or Polish data removes the entry from ``conjunctions_ambiguous`` and gets the join in the one-case names too; a caller who wants none of this removes ``i`` from ``conjunctions``, which restores every prior FIELD and every prior report, with two readings it does not restore and cannot: a letter the two vocabularies disagree about being an initial reads as one here and as the generation there, and case repair leaves a connective the parse placed among the NAME words in lower case where the off switch title-cases it -- ``parse("Dr. John i Smith").capitalized(force=True)`` keeps ``i`` where the off switch gives ``Dr. John I Smith``, and ``Carod i Rovira`` and ``Josep i Rovira`` are the same shape. Those two are the whole of what the switch does not undo, and ``tests/v2/test_properties.py`` states them as its invariants' only exemptions. A delimiter the caller declares through ``Policy(extra_suffix_delimiters=...)`` parts a trailing suffix part as a comma does (see the suffix-delimiter entry below), so no link joins across one: under ``(" - ",)``, ``Smith, John, PhD - i Soler`` keeps suffix ``PhD, i Soler``, as 2.3.0 read it. The default policy declares no such delimiter. See the ``P3`` and ``M2`` entries of ``docs/design/decisions.md`` (closes #397, closes #538) diff --git a/nameparser/_pipeline/_assemble.py b/nameparser/_pipeline/_assemble.py index 8adeb74e..a3921d4f 100644 --- a/nameparser/_pipeline/_assemble.py +++ b/nameparser/_pipeline/_assemble.py @@ -2,8 +2,7 @@ Consumes: tokens (all roles set), dropped, ambiguities (by index; one carrying ``field_tail`` is worded from its word's final role, -#626, and one carrying ``fields_tail`` from its words' final roles, -#629). +#626). Produces: a validated ParsedName -- the constructor re-checks every invariant (span order/bounds, ambiguity subset), so a pipeline bug that would produce an invalid result dies HERE, not in a renderer @@ -125,13 +124,6 @@ def assemble(state: ParseState) -> ParsedName: continue detail = (f"{detail}{FIELD_READING[materialized[0].role]}" f"{pending.field_tail}") - elif pending.fields_tail is not None: - if not materialized: - continue - held = {t.role for t in materialized} - detail = (f"{detail}" - f"{' and '.join([r.value for r in Role if r in held])}" - f"{pending.fields_tail}") ambiguities.append(Ambiguity(pending.kind, detail, materialized)) return ParsedName(original=state.original, tokens=tuple(final.values()), diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index cbb10b4e..ac95b615 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -195,18 +195,15 @@ def titles_and_suffixes(seg: tuple[int, ...]) -> bool: if (seg and not suffixy(seg) and not suffixy(seg, case_class()) and not class_run(seg) and not titles_and_suffixes(seg)): texts_joined = " ".join([state.tokens[i].text for i in seg]) - # The part's words land in two fields since #603 -- a title - # word reads as a title, the rest as suffixes -- and which - # words read as titles is group's and assign's call, so - # the fields are worded at assemble from the final roles - # (`PendingAmbiguity.fields_tail`, #629): 'consumed as - # suffix' had told 'John Smith, Jr., Dr. Bart' its 'Dr.' - # was a suffix. + # Names no field: since #603 a title word in the part reads + # as a title, so 'consumed as suffix' had told + # 'John Smith, Jr., Dr. Bart' its 'Dr.' was a suffix, and + # the fields themselves say where each word went (#629). ambiguities.append(PendingAmbiguity( AmbiguityKind.COMMA_STRUCTURE, f"segment {texts_joined!r} beyond the recognized comma " - f"structures; consumed as ", - tuple(seg), fields_tail=" best-effort")) + f"structures; read best-effort", + tuple(seg))) return copy_with(state, segments=tuple(groups), structure=Structure.FAMILY_COMMA, ambiguities=tuple(ambiguities), diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 08e73f36..dd37336d 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -141,14 +141,6 @@ class PendingAmbiguity: assemble tests ``is not None`` -- and assemble joins them around the FIELD_READING of the first referent's final role, the one place that knows it (#626). Until then ``detail`` is a fragment. - - ``fields_tail`` is the same for a report on a run of words that - can land in more than one field: assemble names every field its - referents hold, as bare role names in canonical order joined by - "and" ("title and suffix"). Its one emitter is segment's report on - a part past the second comma, whose title words read as titles - and the rest as suffixes (rules.md#C2, #629). An emitter sets one - tail or neither. """ kind: AmbiguityKind @@ -156,7 +148,6 @@ class PendingAmbiguity: indices: tuple[int, ...] = () origin: int | None = None field_tail: str | None = None - fields_tail: str | None = None #: How a report names the field a word was read into, by final role. diff --git a/tests/v2/pipeline/test_assemble.py b/tests/v2/pipeline/test_assemble.py index 40497108..79efbad2 100644 --- a/tests/v2/pipeline/test_assemble.py +++ b/tests/v2/pipeline/test_assemble.py @@ -250,24 +250,3 @@ def test_a_field_tail_report_names_its_first_referent_s_field() -> None: PendingAmbiguity(AmbiguityKind.TITLE_OR_NAME, "read as ", (0, 1), field_tail="")) assert [a.detail for a in pn.ambiguities] == ["read as a family name"] - - -def test_a_fields_tail_report_names_every_field_in_canonical_order() -> None: - # #629: a report on a run whose words land in more than one field - # names each field once, by role name, in canonical order however - # the referents are ordered, and in an emptied name it goes like a - # field_tail report. - tokens = (WorkToken("Jr.", Span(0, 3), role=Role.SUFFIX), - WorkToken("Dr.", Span(4, 7), role=Role.TITLE), - WorkToken("Bart", Span(8, 12), role=Role.SUFFIX)) - pn = _with_reports( - "Jr. Dr. Bart", tokens, - PendingAmbiguity(AmbiguityKind.COMMA_STRUCTURE, "consumed as ", - (0, 1, 2), fields_tail=" best-effort")) - assert [a.detail for a in pn.ambiguities] == [ - "consumed as title and suffix best-effort"] - pn = _with_reports( - "-", (WorkToken("-", Span(0, 1), role=Role.SUFFIX),), - PendingAmbiguity(AmbiguityKind.COMMA_STRUCTURE, "consumed as ", - (0,), fields_tail=" best-effort")) - assert not pn.ambiguities diff --git a/tests/v2/pipeline/test_segment.py b/tests/v2/pipeline/test_segment.py index 873eebf6..364e2088 100644 --- a/tests/v2/pipeline/test_segment.py +++ b/tests/v2/pipeline/test_segment.py @@ -1,6 +1,4 @@ from nameparser._lexicon import Lexicon -from nameparser._pipeline import run -from nameparser._pipeline._assemble import assemble from nameparser._pipeline._extract import extract_delimited from nameparser._pipeline._segment import segment from nameparser._pipeline._state import ParseState, Structure @@ -51,30 +49,16 @@ def test_excess_non_suffix_segment_flags_comma_structure() -> None: assert AmbiguityKind.COMMA_STRUCTURE in kinds -def test_the_comma_structure_report_names_the_fields_its_part_lands_in() -> None: +def test_the_comma_structure_report_names_no_field() -> None: # #629: a title word past the second comma reads as a title - # (rules.md#C2, #603), so the part lands in two fields and the - # report names what each word landed in, worded at assemble from - # the final roles. A part that lands wholly in suffix keeps the - # wording every release through 2.3.0 gave it. - lex = _LEX.add(titles={"dr", "secretary", "state"}, conjunctions={"of"}) - - def detail(text: str) -> str: - [report] = [a for a in assemble(run(ParseState( - original=text, lexicon=lex, policy=Policy()))).ambiguities - if a.kind is AmbiguityKind.COMMA_STRUCTURE] - return report.detail - - assert detail("John Smith, Jr., Dr. Bart") == ( + # (rules.md#C2, #603), so the part can land in two fields; the + # report says the part was not recognized and leaves where each + # word went to the fields. 2.3.0 said 'consumed as suffix'. + out = _segmented("John Smith, Jr., Dr. Bart") + assert [a.detail for a in out.ambiguities + if a.kind is AmbiguityKind.COMMA_STRUCTURE] == [ "segment 'Dr. Bart' beyond the recognized comma structures; " - "consumed as title and suffix best-effort") - assert detail("John Smith, Jr., Bart") == ( - "segment 'Bart' beyond the recognized comma structures; " - "consumed as suffix best-effort") - # C2: a part a connective joins into one title keeps the flag - assert detail("John Smith, Jr., Secretary of State") == ( - "segment 'Secretary of State' beyond the recognized comma " - "structures; consumed as title best-effort") + "read best-effort"] def test_empty_input_yields_no_segments() -> None: diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 3891e909..300e0729 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -2,7 +2,6 @@ runner (migration plan) consumes the same CASES.""" import functools import re -from collections.abc import Iterator from typing import Any import pytest @@ -121,21 +120,6 @@ def test_the_family_partitions_into_particles_and_base( r"\b(?:reads? as|joins) (" + "|".join(map(re.escape, sorted(_FIELD_CLAIMS, key=len, reverse=True))) + r")\b") -#: segment's report on a part past the second comma names EVERY field -#: the part's words land in, as role names ("consumed as title and -#: suffix best-effort", #629), so it is checked for equality rather -#: than inclusion: a named field nothing landed in is as wrong as a -#: field left out. -_FIELDS_CLAIM = re.compile(r"\bconsumed as ([a-z]+(?: and [a-z]+)*) best-effort") - - -def _claims(detail: str) -> Iterator[tuple[str, frozenset[Role], bool]]: - """The field claims in a report's detail: the phrase, the fields - that make it true, and whether they must ALL be held.""" - if m := _FIELD_CLAIM.search(detail): - yield m[0], _FIELD_CLAIMS[m[1]], False - if m := _FIELDS_CLAIM.search(detail): - yield m[0], frozenset(map(Role, m[1].split(" and "))), True @pytest.mark.parametrize("case,order", _SWEPT_ROWS, ids=_SWEPT_IDS) @@ -169,23 +153,17 @@ def test_a_report_names_the_field_its_word_lands_in( and so not one `_FIELD_CLAIMS` lists, and `, Ma Dr.` (H1 moving it to the family) is pinned instead by test_assign.py's verbatim detail. No row reached that shape until #626's review. - - Segment's report on a part past the second comma is read by - `_FIELDS_CLAIM` and checked for equality (#629). Its control, - measured 2026-10-10 over master's parser (bc243869) with this - test: 6 failures, the two rows holding a title past the second - comma (`John Smith, Jr., Freiherr von Richthofen` and `John - Smith, Jr., and Secretary of State`) under all three orders, - each told 'consumed as suffix'. """ pn = _swept(case.id, order) for a in pn.ambiguities: + m = _FIELD_CLAIM.search(a.detail) + if m is None: + continue landed = {t.role for t in a.tokens} - for phrase, fields, every in _claims(a.detail): - assert landed == fields if every else landed <= fields, ( - f"{case.text!r}: {a.kind.value} says {phrase!r} but its " - f"tokens landed in {sorted(r.value for r in landed)}: " - f"{a.detail!r}") + assert landed <= _FIELD_CLAIMS[m[1]], ( + f"{case.text!r}: {a.kind.value} says {m[0]!r} but its " + f"tokens landed in {sorted(r.value for r in landed)}: " + f"{a.detail!r}") def test_the_field_sweep_sees_the_claims_it_checks() -> None: @@ -194,12 +172,12 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: dropped -- so the count of claims it checks is recorded here. Move it deliberately when a row or a phrasing moves, never to 0.""" claims = sum( - 1 for case, order in _SWEPT_ROWS - for a in _swept(case.id, order).ambiguities - for _ in _claims(a.detail)) - assert claims == 1180, ( - f"the field sweep checks {claims} claims, recorded as 1180 on " - f"2026-10-10") + _FIELD_CLAIM.search(a.detail) is not None + for case, order in _SWEPT_ROWS + for a in _swept(case.id, order).ambiguities) + assert claims == 1155, ( + f"the field sweep checks {claims} claims, recorded as 1155 on " + f"2026-10-08") #: Case.__post_init__'s shape checks, each probed for the message that