From b8e5d635129fa12b3c5c3b6285b1214e6f1e0962 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 20:47:19 -0700 Subject: [PATCH] Revert #731's maiden-clause numeral fix; accept the shape in T4 Derek: rules/T4 and maiden names do not really interact, because nobody writes a maiden marker followed by an initial. #731 (PR #732, 6008c8ce) bought 'john smith nee x x' and 'john smith nee j x' their capitals' reading with a case argument on trailing_candidates, a whole-name case test in the take, an exception to the candidates' superset contract, two grid shapes and a second invariant -- and the gap it left would need the peel to carry two case facts. Reverted. The readings go back to 2.2.0 through 2.4.2's ('john smith nee x x' suffix 'x', maiden 'x'), stated as an Accepted clause of T4 with both spellings as examples. Kept: M2 naming S10 among its trailing rules, true before #731. decisions/T4 and decisions/M2 keep their #731 entries and gain superseding ones. Code and tests match ad486f66. Co-Authored-By: Claude Opus 5.5 --- .../design/decisions/M2-maiden-marker-rule.md | 1 + .../decisions/T4-bare-letter-as-an-initial.md | 1 + .../rules/T4-bare-letter-as-an-initial.md | 13 ++- docs/release_log.rst | 2 +- nameparser/_pipeline/_group.py | 13 +-- nameparser/_pipeline/_pieces.py | 19 +--- nameparser/_pipeline/_vocab.py | 9 +- tests/v2/cases.py | 16 ---- tests/v2/pipeline/test_pieces.py | 4 +- tests/v2/test_properties.py | 94 ++----------------- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 3 +- 12 files changed, 33 insertions(+), 143 deletions(-) diff --git a/docs/design/decisions/M2-maiden-marker-rule.md b/docs/design/decisions/M2-maiden-marker-rule.md index 717c6657..54d56db7 100644 --- a/docs/design/decisions/M2-maiden-marker-rule.md +++ b/docs/design/decisions/M2-maiden-marker-rule.md @@ -76,3 +76,4 @@ the fullwidth-colon marker (旧姓：佐藤 arrives as one word; the head-peel q - 2026-10-04 (Derek), #601 — A MARKER COUNTS ONLY BEHIND A SURNAME, AND IT TAKES THE WORDS AFTER IT UP TO THE TRAILING RUN THE NAME WOULD END WITH IF THE CLAUSE WERE NOT WRITTEN. This SUPERSEDES the release machinery of the #533 and #535 entries above (the release question, the per-stop checks, the anchored-head boundary of the #544 entry) and the given-part and suffix-comma reach the 2026-07-03 rule had; the #399, #420, #434 and delimiter entries stand. The trigger was #548, `Dr. nee Smith PhD Prof.`, where 2.3.0 read family 'PhD', maiden 'Smith': a clause whose head was nothing but a title, and a release check asking whether each word it gave up could still read as a post-nominal. #548 was closed into this issue and #602 (S2's run) because both halves of its answer were rules rather than repairs. THE HEAD RULE (Derek): a marker is one only in the name before any comma or the surname part before a family comma, behind at least one name word past the leading title run, and not straight behind an unambiguous suffix word or a connective. Anywhere else it is an ordinary word — in the given part after a family comma (`Doe, Jane nee Smith` reads middle 'nee Smith', 1.4.0's reading) and in a part after a suffix comma (`Smith, John, PhD née Jones` reads suffix 'PhD née Jones', also 1.4.0's). Derek's framing was that a maiden marker always follows a family name, and that a marker following something the parser has decided is a title, suffix, particle or connective is not a marker. THE PARTICLE HALF WAS DROPPED BY MEASUREMENT: prototype variant 1 refused a marker behind particle vocabulary and so declined `Anh Do née Tran` and `Mai Le née Nguyen`, Do, Le and Van being particles that are also the commonest Vietnamese surnames; a particle counts as the surname (`Jane van nee Smith` reads family 'van', maiden 'Smith'). THE BOUND (Derek): the clause takes its words up to the run of post-nominals and titles that the trailing rules read at the end of the name with the clause removed, and the words it gives up take the roles that reading gives them — the TRAILING rule's reading of the clause-free name, not the full parse of it, so a head the full parse would read differently (H1's lone title, a P5 join) cannot change where the clause ends. The first word after the marker is always taken, the marker having announced a name — a lone roman numeral included, so `Jane Smith née V` reads maiden 'V' where the clause-free `Jane Smith V` reads suffix 'V', accepted rather than given a numeral exception (rules/M2's Accepted block) — unless it is an unambiguous suffix word, where the marker declines (`Jane Smith nee PhD` reads family 'nee', suffix 'PhD', as before). Before a family comma the clause keeps the 2026-07-03 stop at the first suffix word, since no trailing rule reads that part's end. VARIANTS, prototyped 2026-10-03 behind an environment switch on one tree (the numbers are in #601's comments): v2/v3 grew the clause until the existing release check accepted it and reached 0 invariant violations but inherited the check's caution about particles (`Anh Do geb. de la Vega PhD Prof.` kept 'PhD Prof.' in the maiden name); v4 read the run once over the name as written without binding it, and is the recorded negative control in tests/v2/test_properties.py (350 of the clause grid's 4,482 parses fail under it, measured 2026-10-04); v5/v6 bound that reading as roles but still counted the clause's words among the words to spare, so `John nee Smith ba` read 'ba' as a credential where `John ba` reads a name; v7, the clause-free reading bound as roles, was adopted. Against #535's grids v7 left 0 violations of the M2 invariant where the tree then had 15,586 (grid A, 327,936 texts) and 8,713 (grid B, 173,057), measured 2026-10-03 with an invariant check written to #535's recipe. WHAT THE PROTOTYPE MISSED, found implementing it: the numeral fork. A trailing roman numeral is a suffix only by its shape against the word before it, and v7 read the clause's last word without that test, so `John née Jones Smith VI` kept 'VI' in the maiden name; the take now asks `is_trailing_numeral_suffix` of the last piece, and cases.py's `a_numeral_the_clause_free_name_reads_ends_the_clause` pins it. And the head's title run is measured over the words before the marker, a lone title excepted: measured over the whole segment, `Lord Chancellor née Jones` counted 'Chancellor' as a title and refused the marker. TRIAGE, 2026-10-04: the prototype's 68 failing case rows (40 facade failures among them) were re-pinned as `fix(#601)` with their group named in the note, bar the head-rule bug above; tests/v2/test_parser.py's corpus-append invariant became two-sided (appending ` née X` moves no other field, OR the take records why it declined), with a named set for the suffixes that themselves hold a marker. DEGENERATE INPUT, NO READING DESIGNED FOR (Derek, 2026-10-04: garbage in, garbage out). Two names that read worse by eye moved, and they are recorded here so the move is not mistaken for a regression, not because either reading is wanted: `Berg, Jane van der nee Smith DO` reads middle 'van der nee Smith', family 'Berg' where 2.3.0 read family 'van der Berg', maiden 'Smith DO' — the marker in a given part being a word, the particles no longer trail the given name (P6); and `Jane Doe, PhD née Smith` reads given 'PhD', middle 'née Smith', family 'Jane Doe' where 2.3.0 read given 'Jane', family 'Doe', suffix 'PhD', maiden 'Smith', which 2.3.0 reached only through the clause taking the marker (the comma part opening with a credential is #603's shape). Measured 2026-10-04 against 2.3.0. COST: the take is one function and no release helper survives; the #533 review's comma-head grids could no longer reach a take at all and were replaced by counting heads (tests/v2/test_properties.py's `_maiden_clause_grid`, 4,482 parses). - 2026-10-04 #610 — THE MAIDEN TAKE'S CANDIDATE WORDS ARE ONE FUNCTION BESIDE THE PEEL, AND SHARING IT FIXED A GAP THE HAND COPY HAD. `_maiden_take` chose the clause words to read its clause-free view over with a filter of its own (`_clause_tail_word` plus a numeral clause), a hand copy of the peel's and the chain's admissions whose contract — a superset of what `tail_reading` takes — nothing stated or tested. It is now `_pieces.trailing_candidates`, written beside `peel_trailing`, and tests/v2/pipeline/test_pieces.py pins the superset at the take itself, on the unjoined pieces the take is handed, over a generated grid (37,044 clauses through `group`). The grid found the copy wrong twice. The numeral fork reads a shape-only roman numeral (`VI`, in no wordlist) as the WALK's last piece, and two things leave it last without standing at the end of the name: a period-marked title behind it, which the chain takes first, and the merged `Ph. D.`, which group flags and the walk never holds. The copy admitted the numeral only as the clause's last word, so `Jane Doe nee Smith VI Prof.` kept maiden 'Smith VI' where `John Smith VI Prof.` reads suffix 'VI', and `Jane Doe nee Smith VI Ph. D.` the same way. Both now end the clause at the numeral, as rules/M2's statement already said they should. Recorded negative control: the loop as #601 shipped it misses 432 of that grid's clauses, every one this shape. The test's first version read plain names through all of `group`, on pieces the joins had already merged, which review caught: there the old loop missed 288 names, and a first draft of the fix, covering the title half only, still missed 111 — the `Ph. D.` half, which that grid is what found. Against master 8a8459a8 over the 95,813-text equivalence grid the change moved 10 readings, all of this class. The same issue lifted P6's particle-tail walk into `_pieces.particle_tail`, read off roles, so post_rules' attachment and assign's given-part credential run (#602) ask one walk; that half moved none of the 95,813 readings. - 2026-10-18, #731 — THE CANDIDATE TEST IS A SUPERSET WITH ONE STATED EXCEPTION. The #610 entry's contract — `trailing_candidates` a superset of what `tail_reading` takes — now has one exception: a numeral whose word in front reads as an initial is refused, since the clause keeps that word and the view would otherwise read the numeral behind a different one. And the statement names S10 among the trailing rules the take asks, the fork that reads a shape-only numeral being S10's and not S2's. History: decisions/T4's #731 entry. +- 2026-10-18 (Derek), #731 reverted — the #731 entry above is superseded: `trailing_candidates` takes no case fact again and is a superset of what `tail_reading` takes, without the exception. Its statement keeps naming S10. History: decisions/T4's revert entry. diff --git a/docs/design/decisions/T4-bare-letter-as-an-initial.md b/docs/design/decisions/T4-bare-letter-as-an-initial.md index f7432bb8..c3123151 100644 --- a/docs/design/decisions/T4-bare-letter-as-an-initial.md +++ b/docs/design/decisions/T4-bare-letter-as-an-initial.md @@ -14,3 +14,4 @@ A GAP LEFT OPEN, on ad486f66 as on this tree: the view the take reads asks the own words' case, so a lower-case clause numeral behind own words in capitals is counted as a capital would be, where behind Title case it is S10's "lower case in a name of mixed case", the generation and unreported. With two own name words the fields agree and only the report differs (108 pairs of the committed grid: `JOHN SMITH nee x x` reports `suffix-or-name`, `John Smith nee x x` does not); with ONE own name word there is no word to spare and the fields differ too (`SMITH NEE j x` maiden 'j x', `Smith Nee j x` suffix 'x', maiden 'j'; the code review's finding). The committed grid therefore holds two own words in front only, and says so. Declined, measured: passing the whole name's fact to the view as well. It closes both (0 field and 0 report differences on a grid with one-word, two-word and titled heads) but the view also asks S6's credential lean, which P3 says is asked of the own words, and moving it there fails `test_a_maiden_clause_does_not_change_how_a_trailing_word_reads` (tests/v2/test_properties.py) on 270 of its 20,412 pairs (with it, `J. nee Smith BA` would read the credential where `J. BA` reads a name). Closing the gap needs the peel to take the numeral's case fact apart from the lean's — a design change beyond this fix. Measured: the one-case grid gains `{a} {b} nee {n} {n}` and `{a} {b} nee j {n}`. ad486f66 disagreed on 36 of its 4,752 pairs, all in those shapes; this tree on none, and the fourth recorded control is that 36. 2.2.0 through 2.4.2 gave the lowercase reading above, and 2.0.0 and 2.1.0 the reading this restores (maiden 'x x'), so the release bullet for #723 carries this too. No corpus name moved. +- 2026-10-18 (Derek), #731 reverted — A MAIDEN CLAUSE'S NUMERAL IS NOT HELD TO ITS CAPITALS. The #731 entry above is superseded and its change reverted. Derek: this rule and maiden names do not really interact, because nobody writes a maiden marker followed by an initial, or two. Asked whether settling the gap that entry left open would simplify the parser, the answer was the opposite: it needs the peel to carry two case facts, one for S6's lean and one for the numeral. And #731 itself had bought `john smith nee x x` and `john smith nee j x` — both that unwritten shape — a case argument on `trailing_candidates`, a whole-name case test in the take, a stated exception to the candidates' superset contract (decisions/M2's #731 entry), two grid shapes and a second invariant with their controls, all reverted. The readings go back to 2.2.0 through 2.4.2's (`john smith nee x x` suffix 'x', maiden 'x', reported), stated as an Accepted clause of this rule with both spellings as examples. Kept from that PR: M2 naming S10 among the trailing rules its take asks, which was true before #731 and stays true. What this rule still says of a clause's letters — classify reads one by the whole name's case (the #723 code-review entry) — stands: the shape that guards is a connective inside a clause (`MARIA GARCIA NEE LOPEZ y FERNANDEZ`), which names do take. diff --git a/docs/design/rules/T4-bare-letter-as-an-initial.md b/docs/design/rules/T4-bare-letter-as-an-initial.md index babe41c0..79fe6a30 100644 --- a/docs/design/rules/T4-bare-letter-as-an-initial.md +++ b/docs/design/rules/T4-bare-letter-as-an-initial.md @@ -17,8 +17,7 @@ T4. Rationale: one letter standing for a name word is how an count passes over one, S10 reads a numeral written as one as the capital numeral it is, and a suffix test that vetoes an initial vetoes it — C1's strict mode, and the scan-back of W2's - peel — and M2's take keeps a numeral its clause ends with where - the letter in front is one. A letter whose capital is not one + peel. A letter whose capital is not one of those — é, ж — is no initial in either spelling, and in a name of mixed case a bare lower-case letter is no initial. As P3 asks it, the case question is asked of the name's own words, @@ -39,11 +38,17 @@ T4. Rationale: one letter standing for a name word is how an "smith, j y lopez" → given="j" "dr. née juan y lopez" → middle="juan y" "john j v" → family="v" - "john smith nee x x" → maiden="x x" "Jose j y Lopez" → family="j y Lopez" · boundary "é nee smith" → family="é" · boundary "j nee Smith" → given="j" "DR. NÉE JUAN y LOPEZ" → family="JUAN y LOPEZ" · boundary "jose e lopez" → ambiguities=("conjunction-or-initial",) "jose y lopez" → middle="y" - history: decisions/T4 · interacts: M2, M4, O5, P2, P3, P7, S10, R3, R4, C1, W2 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_vocab.py + Accepted: the numeral ending a maiden clause is M2's question, + and the take asks the letter in front of it by its own writing + alone, so a clause written in lower case can end otherwise than + its capitals. Nobody writes an initial straight after a maiden + marker, so the shape this reaches is not one names take. + "john smith nee x x" → suffix="x" + "JOHN SMITH NEE X X" → maiden="X X" + history: decisions/T4 · interacts: M2, M4, O5, P2, P3, P7, S10, R3, R4, C1, W2 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_vocab.py diff --git a/docs/release_log.rst b/docs/release_log.rst index 80f5f6c1..69975190 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -16,7 +16,7 @@ Release Log - **Fix case repair capitalizing the Dutch particle 't.** ``jacobus henricus van 't hoff`` repairs to ``Jacobus Henricus van 't Hoff``, where 1.4.0 through 2.4.2 gave ``van 'T Hoff``. :issue:`700` - **Add Gov as a title.** ``Gov Smith`` gives title ``Gov``, last ``Smith``, where 1.4.0 through 2.4.2 gave first ``Gov``, and ``Lt. Gov. Smith, John`` gives title ``Lt. Gov.`` with the change above. :issue:`701` - **Fix a capital V, I or X after a lone first name being read as a generational suffix, leaving no last name.** ``John V`` gives first ``John``, last ``V``, and ``Malcolm X`` last ``X``, where 2.0.0 through 2.4.2 gave suffix ``V`` and ``X`` with no last name, restoring 1.4.0's reading. A title is no word to spare, so ``Dr. Smith V`` gives first ``Smith``, last ``V`` (1.4.0's reading too) where 2.0.0 through 2.4.2 gave last ``Smith``, suffix ``V``; a name with a word to spare keeps the suffix, and ``John Smith V`` still gives suffix ``V``. A name written all in lower case reads the same way (``john v`` gives last ``v``). :issue:`718`, :issue:`721` - - **Fix a single letter in a name written all in lower case reading differently from the same name in capitals.** ``jose j y lopez`` gives first ``jose``, middle ``j y``, last ``lopez``, as ``JOSE J Y LOPEZ`` does, where 1.4.0 through 2.4.2 gave last ``j y lopez``. A lone letter before a maiden marker stays the first name (``j nee smith`` as ``J NEE SMITH``), ``田中さん v`` reads as ``田中さん V`` does, and ``Policy(lenient_comma_suffixes=False)`` reads ``john smith, v`` as it reads ``JOHN SMITH, V``. :issue:`723`, :issue:`727`, :issue:`731` + - **Fix a single letter in a name written all in lower case reading differently from the same name in capitals.** ``jose j y lopez`` gives first ``jose``, middle ``j y``, last ``lopez``, as ``JOSE J Y LOPEZ`` does, where 1.4.0 through 2.4.2 gave last ``j y lopez``. A lone letter before a maiden marker stays the first name (``j nee smith`` as ``J NEE SMITH``), ``田中さん v`` reads as ``田中さん V`` does, and ``Policy(lenient_comma_suffixes=False)`` reads ``john smith, v`` as it reads ``JOHN SMITH, V``. :issue:`723`, :issue:`727` * 2.4.2 - October 13, 2026 diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 28301f7a..3383bf12 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -81,8 +81,7 @@ import nameparser._pipeline._comma as _comma from nameparser._pipeline._vocab import D, PH from nameparser._pipeline._vocab import ( - comma_follows_marker, delimiter_cores, is_one_case, - reads_as_a_reported_numeral, + comma_follows_marker, delimiter_cores, reads_as_a_reported_numeral, ) from nameparser._types import AmbiguityKind, Role @@ -304,15 +303,7 @@ def _maiden_take(pieces: Sequence[Sequence[int]], for j in range(lo, end)] if site is not ClauseSite.TRAILING: assert_never(site) - # rules/T4: a letter outside the name's own words "asks the case - # of the whole name, the clause included" -- and the word in front - # of a candidate numeral is always a clause word, so the candidate - # test takes the whole name's fact, not the own words' (#731's - # review: 'MARY SMITH NEE j x' is mixed case, and its 'j' no - # initial). Asked only where the own words are one case. - c = trailing_candidates( - lo, pieces, ptags, tokens, - one_case and is_one_case([t.text for t in tokens])) + c = trailing_candidates(lo, pieces, ptags, tokens) end = len(pieces) released: list[tuple[int, Role]] = [] # the view's picks, for the kept member below (#718) diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 2e71cccc..c659f057 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -1506,15 +1506,11 @@ def is_trailing_title_word(piece: Sequence[int], ptags: Set[str], # beside the peel and the chain whose admissions it has to cover (#610) def trailing_candidates(lo: int, pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], - tokens: Sequence[WorkToken], - one_case: bool | None) -> int: + tokens: Sequence[WorkToken]) -> int: """The first of the pieces after `lo` that `tail_reading` might read into the trailing run: `pieces[c:]`, a SUPERSET of what it takes, for the maiden take to build its clause-free view over and let `tail_reading` decide. `pieces[lo]` itself is never a candidate. - One exception, on purpose: a numeral whose word in front reads as - an initial is refused even where a view without that word would - read it, since the clause keeps the word (#731, below). Back from the end over every piece the peel or the chain admits on its own terms -- a suffix piece, a member of the ambiguous class, @@ -1524,16 +1520,7 @@ def trailing_candidates(lo: int, pieces: Sequence[Sequence[int]], but such title words and group-flagged suffix pieces: the chain takes the titles first and the walk never holds the flagged pieces ('Ph. D.'), so either way the numeral is the walk's last piece and - the fork reads it. The shape test is asked with a one-case fact - -- the WHOLE name's, which the caller passes, since the word in - front is a clause word (rules/T4), where the view's reading keeps - the own words' fact: a candidate is not only - admitted but cut out of the view, so the view puts a different - word in front of it, and a numeral the fork would refuse for the - letter in front is a numeral the view reads. A bare lowercase - letter in front of one in a one-case name is the initial its - capital is (rules/T4), so 'john smith nee x x' keeps its clause as - 'JOHN SMITH NEE X X' does (#731). Then back to the first + the fork reads it. Then back to the first credential that starts #602's run, whose words the run takes whatever they are. A BARE title word is no candidate: H5's chain does not take one, and in the view it would only inflate the count @@ -1554,7 +1541,7 @@ def trailing_candidates(lo: int, pieces: Sequence[Sequence[int]], not tokens[piece[0]].tags.isdisjoint(_TRAILING_WORD_TAGS) or (titles_behind and is_trailing_numeral_suffix( tokens[piece[0]].text, - tokens[pieces[k - 1][0]].text, one_case))))): + tokens[pieces[k - 1][0]].text, None))))): break # a group-flagged suffix piece is outside the peel's walk # altogether (`peel_walk` drops it), so it is behind the diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 39e80ecc..abd31427 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -277,12 +277,9 @@ def is_trailing_numeral_suffix(text: str, preceding: str, the word in front is read as its capital would be, so a bare 'j' is initial-shaped as 'J' is (rules/S10, #721's review: 'john j v' reads as 'JOHN J V'), while 'é', 'ж' or an uncased word, whose - capitals are not, stay as they were. None is the shape test - alone. The maiden take's candidate test once passed it as a safe - superset, but admitting a candidate also cuts it out of the view, - changing the word in front of the numeral, so it passes a fact - now: the whole name's, the word in front being a clause word - (#731), where the view's own reading keeps the own words'.""" + capitals are not, stay as they were. None (the maiden take's + superset view passes it) gets the shape test alone, which admits + more and so stays a superset.""" return (_ROMAN.match(text) is not None and not is_initial_shaped(preceding) and not (one_case and is_initial_shaped(preceding.upper()))) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 67704b88..966d7aa0 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -7686,22 +7686,6 @@ def _check_cjk_shape_purity(self) -> None: "capitals, but the name is not written in one case, so " "the lower-case 'y' past the marker is no initial (T4) " "and joins (#723's review)"), - Case("a_one_case_doubled_numeral_ending_a_clause_stays_in_it", - "john smith nee x x", - {"given": "john", "family": "smith", "maiden": "x x"}, - classification="fix(#274)", - notes="read as 'JOHN SMITH NEE X X' is (T4): the first 'x' is " - "the initial its capital is, so the take's view does " - "not cut the second out of the clause and read it as " - "the generation; 2.2 through 2.4.2 read suffix 'x', " - "maiden 'x', and 2.0 and 2.1 read as here (#731). 1.4.0 " - "had no maiden markers"), - Case("a_one_case_numeral_behind_a_letter_ending_a_clause", - "john smith nee j x", - {"given": "john", "family": "smith", "maiden": "j x"}, - classification="fix(#274)", - notes="the row above with a plain letter in front, as 'JOHN " - "SMITH NEE J X' reads (#731)"), Case("a_lowercase_numeral_after_a_marker_among_capitals_is_no_initial", "MARY SMITH NEE v JONES", {"given": "MARY", "middle": "SMITH NEE v", "family": "JONES"}, diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index 1165d980..79bb3198 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -921,8 +921,8 @@ def _candidate_misses(monkeypatch: pytest.MonkeyPatch, def cut(lo: int, pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], - tokens: Sequence[WorkToken], one_case: bool | None) -> int: - c = candidates(lo, pieces, ptags, tokens, one_case) + tokens: Sequence[WorkToken]) -> int: + c = candidates(lo, pieces, ptags, tokens) seen.append((lo, c)) return c diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 4de6a6c0..07a77f53 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3347,7 +3347,7 @@ def test_every_one_letter_particle_veto_is_load_bearing( # again under Policy(lenient_comma_suffixes=False), the one switch # whose suffix test vetoes an initial by the word's own shape (#727); # trimmed by shape, since the switch reaches only a comma's parts: -# 4,752 pairs, about 9,500 parses, 0.75s on py3.11 (2026-10-18). +# 4,554 pairs, about 9,100 parses, 0.73s on py3.11 (2026-10-18). _ONE_CASE_SHAPES = ( "{a} {n}", "{a} {b} {n}", "{a} {n}, phd", "{a} {b} {n}, phd", "{a} {n}, jr.", "{b}, {a} {n}", ", {a} {n}", "{a} nee {b} {n}", @@ -3367,11 +3367,7 @@ def test_every_one_letter_particle_veto_is_load_bearing( # #727: the letter alone after a comma, behind a whole name or in a # third part, and opening a credential run; and after a glued CJK # honorific, whose peel asks the suffix test by text - "{a} {b}, {n}", "{b}, {a}, {n}", "{a} {b}, {n} phd", "田中さん {n}", - # #731: the letter doubled, and behind a bare letter, ending a - # maiden clause, where the take reads the numeral fork's shape test - # to choose the words its clause-free view holds - "{a} {b} nee {n} {n}", "{a} {b} nee j {n}") + "{a} {b}, {n}", "{b}, {a}, {n}", "{a} {b}, {n} phd", "田中さん {n}") #: the one-letter numerals and a listed and a shape-only spelling of #: each longer length, then one letter from each other fork a bare #: letter meets (#723): no vocabulary, a connective, a marked one @@ -3380,18 +3376,14 @@ def test_every_one_letter_particle_veto_is_load_bearing( _ONE_CASE_HEADS = (("john", "smith", "jones"), ("carod", "puig", "rovira"), ("mary", "ma", "lee")) #: the negative controls, measured 2026-10-18 on this grid, which -#: ad486f66 (before #731) disagreed on 36 of: with classify's one-case +#: 2444f847 (before #727) disagreed on 72 of: with classify's one-case #: letter initial taken out, with the numeral fork's one-case letter -#: in front taken out (the fork and the maiden take's candidate test -#: alike, the two sharing the name in `_pieces`), with the strict -#: suffix test's one-case veto taken out (#727: 54 strict comma pairs -#: and 18 honorific ones), and with the maiden take's candidate test -#: alone given no one-case fact (#731: ad486f66's 36) -- each guard -#: seen on its own -_ONE_CASE_UNGUARDED = 423 -_ONE_CASE_LETTER_IN_FRONT_UNGUARDED = 153 +#: in front taken out, and with the strict suffix test's one-case veto +#: taken out (#727: the 54 strict comma pairs and the 18 honorific +#: ones, which are 2444f847's 72) -- each guard seen on its own +_ONE_CASE_UNGUARDED = 387 +_ONE_CASE_LETTER_IN_FRONT_UNGUARDED = 117 _ONE_CASE_STRICT_VETO_UNGUARDED = 72 -_ONE_CASE_MAIDEN_CANDIDATES_UNGUARDED = 36 def _one_case_disagreements() -> list[str]: @@ -3439,76 +3431,6 @@ def test_the_one_case_grid_sees_the_strict_veto( == _ONE_CASE_STRICT_VETO_UNGUARDED) -# --- #731's review: a clause's letter asks the whole name's case -- -# T4 reads a letter outside the name's own words by the case of the -# WHOLE name, so a lower-case letter in a maiden clause behind own -# words in capitals is mixed-case writing and reads as it does behind -# the same words in Title case. The one-case grid above cannot see -# this (both spellings of each of its pairs are wholly one case); this -# pairs capitals against Title case, the clause letters lower-case in -# both: 396 pairs, about 800 parses, 0.07s on py3.11 (2026-10-18). -# FIELDS only, and TWO own name words in front only, which is this -# grid's whole claim: the take's view reads the numeral by the own -# words' case, so behind capitals a lower-case clause numeral is -# counted like a capital -- the report differs on 108 of these pairs -# (on ad486f66 as here), and with ONE own name word, no word to -# spare, the fields differ too ('SMITH NEE j x' maiden 'j x', 'Smith -# Nee j x' suffix 'x'). Recorded in decisions/T4's #731 entry with -# why the view keeps the own words' fact. -_CLAUSE_CASE_SHAPES = ("{a} {b} nee {n} {n}", "{a} {b} nee j {n}", - "{a} {b} nee {c} j {n}", "{a} {b} nee {c} {n} {n}") -#: the negative control, measured 2026-10-18: the maiden take's -#: candidate test given the own-words fact instead (#731's first -#: version), which disagreed on that many -_CLAUSE_CASE_OWN_WORDS = 72 - - -def _clause_case_disagreements() -> list[str]: - def sig(name: ParsedName) -> dict[str, str]: - return {k: v.lower() for k, v in name.as_dict().items() if v} - out = [] - for order in (GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST): - parser = Parser(policy=Policy(name_order=order)) - for shape, letter, (a, b, c) in itertools.product( - _CLAUSE_CASE_SHAPES, _ONE_CASE_LETTERS, _ONE_CASE_HEADS): - caps = shape.format(a=a.upper(), b=b.upper(), c=c.upper(), - n=letter) - title = shape.format(a=a.title(), b=b.title(), c=c.title(), - n=letter) - if sig(parser.parse(caps)) != sig(parser.parse(title)): - out.append(f"[{order}] {caps!r}") - return out - - -def test_a_clause_letter_beside_capitals_reads_as_beside_title_case( - ) -> None: - failures = _clause_case_disagreements() - assert not failures, "\n".join(failures[:10]) - - -def test_the_clause_case_grid_sees_the_whole_name_fact( - monkeypatch: pytest.MonkeyPatch) -> None: - # the take asks `one_case and is_one_case()`; with the - # second half always true it is the own-words fact alone, which is - # what #731's first version passed - monkeypatch.setattr("nameparser._pipeline._group.is_one_case", - lambda texts: True) - assert len(_clause_case_disagreements()) == _CLAUSE_CASE_OWN_WORDS - - -def test_the_one_case_grid_sees_the_maiden_candidates( - monkeypatch: pytest.MonkeyPatch) -> None: - # the maiden take's candidate test without the one-case fact - # (#731): the shape test alone, as before - from nameparser._pipeline import _group, _pieces - monkeypatch.setattr( - _group, "trailing_candidates", - lambda lo, pieces, ptags, tokens, one_case: - _pieces.trailing_candidates(lo, pieces, ptags, tokens, None)) - assert (len(_one_case_disagreements()) - == _ONE_CASE_MAIDEN_CANDIDATES_UNGUARDED) - - def test_the_one_case_grid_sees_the_letter_in_front( monkeypatch: pytest.MonkeyPatch) -> None: from nameparser._pipeline import _vocab diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 59a2dbcb..12992ae0 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -137,6 +137,7 @@ "JOHN PROF. MA" "JOHN QUINCY SMITH I" "JOHN SMITH MA" +"JOHN SMITH NEE X X" "JOHN SMITH PH.D." "JOHN SMITH, VD MA" "JOHN V" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 2dd6964e..4dee8dd6 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1270,6 +1270,7 @@ {"name": "JOHN QUINCY SMITH I", "given": "JOHN", "middle": "QUINCY", "family": "SMITH", "suffix": "I", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. Q. S."} {"name": "JOHN SMITH", "given": "JOHN", "family": "SMITH", "initials": "J. S."} {"name": "JOHN SMITH MA", "given": "JOHN", "family": "SMITH", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "JOHN SMITH NEE X X", "given": "JOHN", "family": "SMITH", "maiden": "X X", "initials": "J. S."} {"name": "JOHN SMITH PH.D.", "given": "JOHN", "family": "SMITH", "suffix": "PH.D.", "initials": "J. S."} {"name": "JOHN SMITH, MA", "given": "JOHN", "family": "SMITH", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "JOHN SMITH, VD MA", "given": "JOHN", "family": "SMITH", "suffix": "VD MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} @@ -3412,7 +3413,7 @@ {"name": "john smith mcse", "given": "john", "family": "smith", "suffix": "mcse", "initials": "j. s."} {"name": "john smith md", "given": "john", "family": "smith", "suffix": "md", "initials": "j. s."} {"name": "john smith meng", "given": "john", "family": "smith", "suffix": "meng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} -{"name": "john smith nee x x", "given": "john", "family": "smith", "maiden": "x x", "initials": "j. s."} +{"name": "john smith nee x x", "given": "john", "family": "smith", "suffix": "x", "maiden": "x", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} {"name": "john smith p.h.d.", "given": "john", "family": "smith", "suffix": "p.h.d.", "initials": "j. s."} {"name": "john smith ph. d.", "given": "john", "family": "smith", "suffix": "ph. d.", "initials": "j. s."} {"name": "john smith ph.d.", "given": "john", "family": "smith", "suffix": "ph.d.", "initials": "j. s."}