diff --git a/AGENTS.md b/AGENTS.md index 3298f5eb..78bb0925 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -173,7 +173,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 disagreeing pairs at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules/H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index 50f01a02..aa00b9a0 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -80,6 +80,7 @@ Entry conventions: - [T1 — separators, not joiners](T1-separators-not-joiners.md) - [T3 — the interpunct's flank guard](T3-interpuncts-flank-guard.md) +- [T4 — the bare letter as an initial](T4-bare-letter-as-an-initial.md) ## Ambiguity & tie-breaking (A) diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index 28691d24..32477623 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -20,3 +20,4 @@ THE PREMISE BEHIND #718, REVISED. #718's entry quotes Derek that a generational V is "mostly archaic". On this PR he found otherwise: searching shows many people named John Ingram with every variation of generational marker and middle initial, written without a comma unless a credential follows ("MBA", "MD"). Nothing decided here moves — the two-word count reads each of those as the generation — but #718's lone-numeral reading (`John V`, family 'V') now rests on the count alone, a name of two words having no word to spare for a generation, not on the generation being rare. - 2026-10-18, #722's /simplify pass — P3'S "CONNECTS NOTHING" IS ASKED IN ONE PLACE, OVER FINAL ROLES. The last two rounds answered rules/P3's "unless no name word of its part stands to its right" twice: classify skipped its report on a letter that was its part's last own word by position (`ends`), and assign withdrew it at the end of the stage for a letter followed only by suffix and title roles. Both go; assemble, which already withdrew this report where the letter was roled SUFFIX or TITLE (rules/A1), withdraws it too where no given, middle or family word follows the letter in its segment — the roles being final there and nothing downstream moving them. Measured, the one site reads everything the two did, and restores one reading the position cut had wrongly silenced: where no clause forms and the marker is read as a name word, a name word does follow the letter, so `jose e nee` (family 'nee') and `smith, e nee smith` report `conjunction-or-initial` again, as master did. The two reports on a numeral a post-nominal follows (assign's, and the maiden take's twin) share one predicate, `_vocab.reads_as_a_reported_numeral`. Open: "a name written wholly in one case says nothing about any letter in it" is applied to the roman-numeral letters only. In a one-case name a bare capital is tagged an initial and a bare lowercase letter is not, unless it is `v`, `i` or `x` (this PR) or a marked connective (P3), so the lowercase and all-caps spellings of a one-case name read differently wherever a reader honors the tag — before #721 too: M4 makes a lone name word before a marker the family but not an initial (`j nee smith` family 'j', `J NEE SMITH` given 'J'), O5 is silenced by one (a lone `j` reports given-or-family, `J` does not), and the connective join defers to one (`jose j y lopez` family 'j y lopez', `JOSE J Y LOPEZ` middle 'J Y', family 'LOPEZ'). Tagging every one-case single letter as its capital is tagged is a behavior change beyond this PR: #723. +- 2026-10-18, #723 — THE OPEN QUESTION ABOVE IS ANSWERED, FOR EVERY LETTER. A bare letter in a one-case name is the initial its capital is (rules/T4), and the one-letter numerals are an instance of that rule rather than a case of their own: S10's own one-case sentences are unchanged, and classify's numeral test is gone into T4's. History: decisions/T4. diff --git a/docs/design/decisions/T4-bare-letter-as-an-initial.md b/docs/design/decisions/T4-bare-letter-as-an-initial.md new file mode 100644 index 00000000..33c3e555 --- /dev/null +++ b/docs/design/decisions/T4-bare-letter-as-an-initial.md @@ -0,0 +1,9 @@ +# T4 — the bare letter as an initial + +- 2026-10-18, #723 — A ONE-CASE BARE LETTER IS THE INITIAL ITS CAPITAL IS. #721 applied S10's "a name written wholly in one case says nothing about any letter in it" to the one-letter roman numerals only (decisions/S10, the Open line of its /simplify entry), so a bare lowercase `j` was a name word where `J` was an initial, and every rule that reads an initial differently read the two spellings of one name differently: M4 (`j nee smith` family 'j', `J NEE SMITH` given 'J'), O5 (a lone `j` reported `given-or-family`, `J` did not), and P3's count, which passes over an initial (`jose j y lopez` family 'j y lopez', `JOSE J Y LOPEZ` middle 'J Y', family 'LOPEZ', which 1.4.0 read too). The rule is now stated once, for every letter, and the numeral case is an instance of it: in a name written in one case, a bare letter is tagged an initial wherever its capital would be. Its capital is asked of the bare-capital shape (one ASCII capital), so `é` and `ж`, whose capitals are no initial, read as before in both spellings. `ı` and `ſ`, whose capitals are `I` and `S`, are tagged initials too; `ı`'s twin is not a true one, since `I` is also the marked Catalan connective and `ı` is in no vocabulary (`JOSE I LOPEZ` reports `conjunction-or-initial`, `jose ı lopez` nothing). Measured on the first commit's tree against 16ee2851, a dated snapshot from a scratch grid not committed: letters a–z, `é` and `ж`; heads (a, b, c) = (john, smith, jones), (jose, lopez, garcia), (mary, ma, lee); all three orders; the 35 shapes `{n}`, `{n} nee {b}`, `{a} {n}`, `{n} {a}`, `{a} {n} {b}`, `{a} {n} y {b}`, `{a} {n} e {b}`, `{b}, {n}`, `{b}, {n} y {c}`, `{b}, {a} {n}`, `, {n}`, `{n}, phd`, `{a} {n}, phd`, `{a} {n}, jr.`, `{a} nee {b} {n}`, `{a} nee {n}`, `{a} {b} nee {n} {c}`, `dr {n}`, `dr {n} {b}`, `sir {n} {b}`, `{a} van {n}`, `{n} van {b}`, `{a} van {n} {b}`, `abdul {n}`, `{n} {a} {b}`, `{a} {n} {n} {b}`, `{a} {n}. {b}`, `({c}) {n} {b}`, `{n} y {b}`, `{a} {n} phd`, `{a} {b} {n} prof.`, `{n} {b} {n}`, `{n} {n}`, `{a} {n} de la {b}`, `{b} {n} jr`; each lowercase text against its `.upper()`, compared on case-folded fields, sorted report kinds and `initials()`, keyed by order and text, so the 8,820 products collapse to 7,794 distinct pairs where a shape has no head. 16ee2851 disagreed on 504 of them, this tree on none, and no capital spelling moved. The committed invariant is `test_a_one_case_name_reads_as_its_capitals` (tests/v2/test_properties.py), whose grid gains the issue's shapes and the letters `j`, `y` and `e`; its stated controls are the recompute. + WHAT MOVES: lower-case letters in a name whose own words are written in one case, each onto its capital twin's reading — wholly lower-case names, and (since the case question is asked of the own words, P3) a one-case name beside a clause of another case, so `j nee Smith` reads given 'j' as `J nee Smith` does. In three classes — a lone letter loses O5's report, a letter before a maiden marker keeps the given name M4 would have taken, and a letter beside a single-letter link is passed over by P3's count, so the link in what is then a three-word name stays a name word. No corpus name moved (the snapshot and the CLDR, GND and native-script agreement records are unchanged). And P3's Accepted paragraph on a declined marker loses its "one shape" in which the name's own words read differently in the two spellings: `dr. née juan y lopez` read family 'juan y lopez' where `DR. NÉE JUAN Y LOPEZ` reads middle 'JUAN Y', because the `y` past the declined marker is outside the own words P3's fork reads; it is this rule's letter now, and reads as its capital does. + And one move the snapshot cannot see (R4): case repair of a connective letter in a one-case maiden clause, now tagged an initial as its capital is, so `jose lopez nee y garcia` repairs to `Jose Lopez (Y Garcia)` where it gave `(y Garcia)` — the same tag in the two spellings that P3's rewritten paragraph intends. No corpus name's `capitalized()` moved (the design-docs review compared every one across the two trees). + A GAP THE RULE DOES NOT CLOSE, found by the design-docs review and present on 16ee2851 too: under `Policy(lenient_comma_suffixes=False)` the strict comma-suffix test asks a one-letter numeral's shape as written, not its tag, so `john smith, v` reads suffix 'v' where `JOHN SMITH, V` reads given 'V', and `smith, john, i` reports nothing where `SMITH, JOHN, I` reports `comma-structure`. The committed grid runs the default policy only. Left for its own change. + Open: #727 — whether the strict comma test should read the tag. + The connective letters are left where P3 put them. Among the name's own words, P3's fork answers by vocabulary before this rule is asked — the marked `e` and `i` read as initials and report, the unmarked `y` as a connective — and does so in both spellings, so the invariant already held for them; a connective letter outside the own words was always read by the bare-capital shape in its capital spelling, and its lower-case spelling now follows. +- 2026-10-18, #723's code review — A LETTER OUTSIDE THE OWN WORDS ASKS THE WHOLE NAME'S CASE. The first version gated the new tag on the one-case fact classify records, which is asked of the name's own words only (P3), so a lower-case letter in a clause of a name whose own words are capitals passed it: `MARIA GARCIA NEE LOPEZ y FERNANDEZ` and `MARIA GARCIA (LOPEZ y FERNANDEZ)` tagged the `y` an initial (case repair wrote `Lopez Y Fernandez`), and `DR. NÉE JUAN y LOPEZ` moved from family 'JUAN y LOPEZ' to middle 'JUAN y' — mixed-case writing, which this rule reads as no initial, and moves the twin-only grids could not see, since every pair in them is wholly one case. The fix asks a word outside the own words about the whole name instead; an own word keeps the own-words fact, so a clause beside the name still changes nothing about the name's own letters (P3). The defect predated #723 for the numeral letters, which #721 gated the same way; they are read by this rule now and move with it, and the move is not only a tag (found by the design-docs review). After a marker among capitals, a lower-case numeral no longer holds the clause up as an initial would: `MARY SMITH NEE v JONES` reads middle 'SMITH NEE v', family 'JONES' (on 16ee2851, maiden 'v JONES'), `MARY SMITH NEE i JONES` family 'NEE i JONES', and `JANE DOE NEE SMITH v` drops `suffix-or-name`. Each is the reading of its mixed-case spelling (`Mary Smith nee v Jones`) and of 2.4.2 (and 1.4.0 for the first), so only the unreleased #721 had moved them and no release bullet is owed; the capital twin `MARY SMITH NEE V JONES` keeps maiden 'V JONES'. Pinned by the row `a_lowercase_numeral_after_a_marker_among_capitals_is_no_initial`. Pinned by `test_a_lowercase_clause_letter_among_capitals_is_no_initial` and the case row `a_lowercase_letter_past_a_declined_marker_among_capitals_joins`. The first entry's figures were measured on the first version, and the twin grids are unchanged by the fix (both spellings of every pair in them are wholly one case). diff --git a/docs/design/rules/M4-maiden-marker-makes-the-lone-name-word-the-family.md b/docs/design/rules/M4-maiden-marker-makes-the-lone-name-word-the-family.md index 22ea8695..5393b536 100644 --- a/docs/design/rules/M4-maiden-marker-makes-the-lone-name-word-the-family.md +++ b/docs/design/rules/M4-maiden-marker-makes-the-lone-name-word-the-family.md @@ -36,5 +36,6 @@ M4. Rationale: a maiden name is a FORMER family name, and a former "Smith née Jones PhD" → family="Smith" "abd née Jones" → given="abd" · boundary "J. née Jones Smith V" → given="J." · boundary - history: decisions/M4 · interacts: M1, M2, M3, H1, N3, O5 · implemented: nameparser/_pipeline/_post_rules.py + "j nee smith" → given="j" · boundary + history: decisions/M4 · interacts: M1, M2, M3, H1, N3, O5, T4 · implemented: nameparser/_pipeline/_post_rules.py diff --git a/docs/design/rules/O5-lone-name-words-convention-and-reporting-it.md b/docs/design/rules/O5-lone-name-words-convention-and-reporting-it.md index 35296db3..57ff0714 100644 --- a/docs/design/rules/O5-lone-name-words-convention-and-reporting-it.md +++ b/docs/design/rules/O5-lone-name-words-convention-and-reporting-it.md @@ -27,7 +27,7 @@ O5. Rationale: O4 reads a name by comparing where its words stand, where it fires, a family comma — a comma with a name segment after it — silences it, a script whose own convention settles the order (W4) silences it, and so does the word's own claim — a - particle, a bound given name, an initial's shape. A comma with + particle, a bound given name, an initial (T4). A comma with nothing after it names no family and silences nothing, so `سلمان،` and `Smith,` report where `Smith, Andrew` does not. A comma with only a TITLE after it names no family either, but the @@ -56,5 +56,6 @@ O5. Rationale: O4 reads a name by comparing where its words stand, "Sir John" → ambiguities=() "Smith née Jones" → ambiguities=() "abd née Jones" → ambiguities=() - history: decisions/O5 · interacts: O4, H1, N3, M4, H4, W4, P3, A2 · implemented: nameparser/_pipeline/_assign.py + "j" → ambiguities=() + history: decisions/O5 · interacts: O4, H1, N3, M4, H4, W4, P3, A2, T4 · implemented: nameparser/_pipeline/_assign.py diff --git a/docs/design/rules/P2-particles-join-forward.md b/docs/design/rules/P2-particles-join-forward.md index 9ec268b2..9f270754 100644 --- a/docs/design/rules/P2-particles-join-forward.md +++ b/docs/design/rules/P2-particles-join-forward.md @@ -31,8 +31,9 @@ P2. Rationale: a particle is written as part of the surname it word or nothing behind it, it reads as the suffix it is there. A lone letter behind it reads as it does in the same part without the particle: a capital is a name word ('SMITH VD V, JOHN' as - 'SMITH V, JOHN'), a lowercase letter the numeral ('smith vd v, - john' as 'smith v, john'). + 'SMITH V, JOHN'), a lowercase letter in a name of mixed case the + numeral ('Smith vd v, John' as 'Smith v, John'), and a name + written in one case reads as its capitals (T4). "John van der Berg" → family="van der Berg" "John van der Berg Smith" → family="van der Berg Smith" "Vincent van Gogh van Beethoven" → middle="van Gogh" diff --git a/docs/design/rules/P3-connectives.md b/docs/design/rules/P3-connectives.md index 01dd0e06..8d20f112 100644 --- a/docs/design/rules/P3-connectives.md +++ b/docs/design/rules/P3-connectives.md @@ -52,8 +52,8 @@ P3. Rationale: connective words ("y", "of the") bind name words into (M2), are not among them, and neither is a delimited clause (N1, M1). So a clause beside the name changes neither whether the connective joins nor how a letter in the name reads, and a letter - inside such a clause is the clause's word, read as it always was - and not by this rule. The two questions part at one point: a + inside such a clause is the clause's word, read by T4 and not by + this rule. The two questions part at one point: a marker the pass declines and leaves as a word (M2) is a word, and counts toward the three — but its case is still not asked. That span is not this rule's alone: wherever another rule asks @@ -137,17 +137,13 @@ P3. Rationale: connective words ("y", "of the") bind name words into class, which shows when it is the only differently-cased token, so "JUAN Y GARCIA née" joins where "JUAN Y GARCIA née Jones" keeps "Y" a name word. And it leaves any connective standing - after it to the mixed-case rule rather than to the one-case fork, - so a bare Latin capital there reads as an initial while its - lowercase spelling joins — the one shape in which the name's OWN - words read differently in its two one-case spellings. So "DR. - NÉE JUAN Y LOPEZ" reads middle "JUAN Y" with its "Y" an initial, - while "dr. née juan y lopez" reads family "juan y lopez" with its - own "y" joined. A clause's words are read as they always were, - and that is not an exception to this: such a letter still takes - a different tag in the two spellings, so its case repair can - differ between them, which is a repair difference and not a - reading of the name's own words. + after it to T4 rather than to the one-case fork, so a letter + there reads as the initial its capital is, in both spellings, + where the fork would have joined it: both "DR. NÉE JUAN Y LOPEZ" + and "dr. née juan y lopez" read middle "JUAN Y" with the letter + an initial. A clause's words are + read by T4 the same way, so such a letter takes the same tag in + the two spellings. Accepted rather than repaired: classify cannot know what group will decline, and a clause's words are read by the clause's own rules. @@ -157,5 +153,5 @@ P3. Rationale: connective words ("y", "of the") bind name words into same two words unjoined are two name words and H1 does not fire. P1's leading run is the second (#395, landed): its run takes the "Vega y Santos" join whole or stops before it. - history: decisions/P3 · interacts: H1, P1, M2, R1, R3, R4, C1, S6, S10 · implemented: nameparser/_pipeline/_assemble.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py + history: decisions/P3 · interacts: H1, P1, M2, R1, R3, R4, C1, S6, S10, T4 · implemented: nameparser/_pipeline/_assemble.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index 5dd3969f..7723f48d 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -99,6 +99,7 @@ The marker's unit is the whole rule, so a clause that reaches past the tolerated - [T1 — separators, not joiners](T1-separators-not-joiners.md) - [T2 — the katakana middle dot divides like a space](T2-katakana-middle-dot-divides-like-a-space.md) - [T3 — the interpunct's flank guard](T3-interpuncts-flank-guard.md) +- [T4 — the bare letter as an initial](T4-bare-letter-as-an-initial.md) ## [Ambiguity & tie-breaking (A)](A-ambiguity-and-tie-breaking.md) diff --git a/docs/design/rules/S10-trailing-roman-numeral.md b/docs/design/rules/S10-trailing-roman-numeral.md index fcbe7459..3c6ec800 100644 --- a/docs/design/rules/S10-trailing-roman-numeral.md +++ b/docs/design/rules/S10-trailing-roman-numeral.md @@ -64,4 +64,4 @@ S10. Rationale: a roman numeral ending a name is its generation, as one with the numeral in front of it, the run taking both, as 1.4.0 read it. "John Smith V Ph. D." → suffix="V Ph. D." - history: decisions/S10 · interacts: S2, S4, S7, C1, P2, P3, M2, H1, H3 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py + history: decisions/S10 · interacts: S2, S4, S7, C1, P2, P3, M2, H1, H3, T4 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/docs/design/rules/T4-bare-letter-as-an-initial.md b/docs/design/rules/T4-bare-letter-as-an-initial.md new file mode 100644 index 00000000..4ff914ae --- /dev/null +++ b/docs/design/rules/T4-bare-letter-as-an-initial.md @@ -0,0 +1,45 @@ +# T4 — the bare letter as an initial + +T4. Rationale: one letter standing for a name word is how an + initial is written, and the writing marks it: by a period after + the letter, or by a capital among mixed case, since nobody + writes an initial in lower case beside capitals. A name written + wholly in one case marks nothing, so its two spellings — all + lower case and all capitals — are one name, and read alike. + A one-letter word written with its period is an initial, in a + script that writes initials, and so is a bare unaccented Latin + capital, A to Z; in a name written wholly in one case, + which says nothing about any letter in it, a bare letter is an + initial wherever its capital would be. So every rule that reads + an initial differently from a name word reads the lower-case + spelling as it reads the capitals: M4 leaves an initial its + given-name reading, O5 reports no convention for one, P3's + count passes over one, and S10 reads a numeral written as one + as the capital numeral it is. A letter whose capital is not one + of those — é, ж — is no initial in either spelling, and in a + name of mixed case a bare lower-case letter is no initial. As + P3 asks it, the case question is asked of the name's own words, + so a clause beside the name changes nothing about the name's own + letters: a lone letter before a maiden name in capitals is still + a name written in one case. A single-letter connective among the name's own + words is P3's question and not this one's: there the vocabulary + decides, and the letter it does not mark reads as a connective + in both spellings; a letter outside the name's own words — in + a maiden clause, or past a marker the pass declines (M2) — is + this rule's, and asks the case of the whole name, the clause + included, since the own words' case says nothing of how the + clause beside them was written. + "J NEE SMITH" → given="J" + "j nee smith" → given="j" + "j" → ambiguities=() + "jose j y lopez" → middle="j y" + "smith, j y lopez" → given="j" + "dr. née juan y lopez" → middle="juan y" + "john j v" → family="v" + "Jose j y Lopez" → family="j y Lopez" · boundary + "é nee smith" → family="é" · boundary + "j nee Smith" → given="j" + "DR. NÉE JUAN y LOPEZ" → family="JUAN y LOPEZ" · boundary + "jose e lopez" → ambiguities=("conjunction-or-initial",) + "jose y lopez" → middle="y" + history: decisions/T4 · interacts: M4, O5, P2, P3, P7, S10, R3, R4 · implemented: nameparser/_pipeline/_classify.py diff --git a/docs/modules.rst b/docs/modules.rst index 5bb12bbc..b0897026 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -51,8 +51,10 @@ Results actual family particles), ``conjunction`` (a joining word, "and"/"y"), ``initial`` (a word READ as an initial — initial-shaped in a script that HAS initials, - "J." or "А.", never "씨.", or a marked single-letter connective in - a name written in one case, rules/P3), and ``joined`` (a + "J." or "А.", never "씨.", a marked single-letter connective in a + name written in one case, rules/P3, or any other bare letter in a + name written in one case where its capital would be one, "j" as + "J", rules/T4), and ``joined`` (a continuation of the token before it — within one merged piece the tag is role-blind and every view joins the pair with a space, so it renders "Ph. D." as one diff --git a/docs/release_log.rst b/docs/release_log.rst index a02be1e6..2b1e3bf1 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -15,6 +15,7 @@ Release Log - **Fix case repair capitalizing the Dutch particle 't.** ``jacobus henricus van 't hoff`` repairs to ``Jacobus Henricus van 't Hoff``, where 1.4.0 through 2.4.2 gave ``van 'T Hoff``. :issue:`700` - **Add Gov as a title.** ``Gov Smith`` gives title ``Gov``, last ``Smith``, where 1.4.0 through 2.4.2 gave first ``Gov``, and ``Lt. Gov. Smith, John`` gives title ``Lt. Gov.`` with the change above. :issue:`701` - **Fix a capital V, I or X after a lone first name being read as a generational suffix, leaving no last name.** ``John V`` gives first ``John``, last ``V``, and ``Malcolm X`` last ``X``, where 2.0.0 through 2.4.2 gave suffix ``V`` and ``X`` with no last name, restoring 1.4.0's reading. A title is no word to spare, so ``Dr. Smith V`` gives first ``Smith``, last ``V`` (1.4.0's reading too) where 2.0.0 through 2.4.2 gave last ``Smith``, suffix ``V``; a name with a word to spare keeps the suffix, and ``John Smith V`` still gives suffix ``V``. A name written all in lower case reads the same way (``john v`` gives last ``v``). :issue:`718`, :issue:`721` + - **Fix a single letter in a name written all in lower case reading differently from the same name in capitals.** ``jose j y lopez`` gives first ``jose``, middle ``j y``, last ``lopez``, as ``JOSE J Y LOPEZ`` does, where 1.4.0 through 2.4.2 gave last ``j y lopez``. A lone letter before a maiden marker stays the first name (``j nee smith`` as ``J NEE SMITH``). :issue:`723` * 2.4.2 - October 13, 2026 diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 5e9b8f78..639b65fe 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -62,7 +62,7 @@ ) from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( - _ROMAN, caps_shape_candidate, composed_len, is_initial, is_one_case, + _INITIAL, caps_shape_candidate, composed_len, is_initial, is_one_case, period_joined_vocab, suffix_as_written, tag_marker_runs, ) from nameparser._pipeline._pieces import own_words @@ -80,7 +80,7 @@ # spare" def _tags_for(token: WorkToken, n: str, state: ParseState, marker_tag: str | None, one_case_own: bool, - one_case: bool) -> frozenset[str]: + one_case: bool, letter_one_case: bool) -> frozenset[str]: """`n` is _normalize(token.text), folded once by the caller and shared with the marker pass; `marker_tag` is what that pass decided for this token, or None. The marker DECISION is entirely @@ -106,7 +106,15 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, clause cut, never from the name's own writing (#516 review round, a second reviewer's finding). `single_letter_connective` below keeps `one_case_own`: that fork is genuinely about the OWN-WORDS - span, and reads a clause's word as no evidence on purpose.""" + span, and reads a clause's word as no evidence on purpose. + + `letter_one_case` is rules/T4's question for THIS token: the + own-words fact for one of the name's own words, and for a word + outside them -- a clause's, or past a declined marker -- the fact + about the WHOLE name, since the own words' case says nothing + about how the clause beside them was written: 'y' in 'MARIA + GARCIA NEE LOPEZ y FERNANDEZ' is a lower-case letter among + capitals, which T4 reads as no initial (#723's review).""" lex = state.lexicon tags = set(token.tags) if marker_tag is not None: @@ -159,21 +167,24 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # the mixed-case rule, unchanged. v1's is_conjunction excludes # initials: 'e.' in 'john e. smith' is a middle initial, not # the Spanish conjunction 'e' - # rules/S10: "A name written wholly in one case says nothing - # about any letter in it" -- so a one-letter roman numeral - # there is the initial its capital is, and every reader of the - # tag after classify vetoes 'v' as it vetoes 'V' (#721) -- not - # the honorific peel upstream, which asks the shape itself - # ('田中さん v' still reads otherwise than '田中さん V'). - # `one_case` is the fact about the name's own words, applied to - # a clause's words too, as #516's caps branch applies it, since - # a capital 'V' in a clause is an initial as well. One - # character already, so `_ROMAN` alone asks what - # `is_single_letter_numeral` would, at C level: no frame per - # one-letter word + # rules/T4: "in a name written wholly in one case, which says + # nothing about any letter in it, a bare letter is an initial + # wherever its capital would be" -- so every reader of the tag + # after classify reads 'j' as it reads 'J' (#723; the roman + # numerals first, #721) -- not the honorific peel upstream, + # which asks the shape itself ('田中さん v' still reads + # otherwise than '田中さん V'). `one_case` is the fact about + # the name's own words, applied to a clause's words too, as + # #516's caps branch applies it, since a capital 'J' in a + # clause is an initial as well. The capital is asked of the + # bare pattern at C level, no frame per one-letter word, and + # needs no script test: no capital `_INITIAL` matches is in a + # script without initials. 'é' and 'ж' stay as they were, + # their capitals being no initial either. A clause's letter + # asks the whole name's case (`letter_one_case`, docstring) initial = is_initial(token.text) or ( - one_case and len(token.text) == 1 - and _ROMAN.match(token.text) is not None) + letter_one_case and len(token.text) == 1 + and _INITIAL.fullmatch(token.text.upper()) is not None) if n in lex.conjunctions and not initial: tags.add("conjunction") if initial: @@ -306,11 +317,21 @@ def classify(state: ParseState) -> ParseState: # per token, and the fork and its emitter then agree with the case # class they consult. No extra frame -- it is one more boolean in a # comprehension that already walks every token. + # rules/T4 asks a word outside the own words about the WHOLE + # name's case (`_tags_for`'s `letter_one_case`). Where every token + # is an own word the two facts are one, so only a name with a + # clause pays the second test's frame. + whole_one_case = one_case and ( + len(own) == len(state.tokens) + or is_one_case([t.text for t in state.tokens])) tokens = tuple( copy_with( t, tags=_tags_for(t, folded[i], state, marker_tags.get(i), one_case_own=one_case and i < clause_at - and t.role is None, one_case=one_case)) + and t.role is None, one_case=one_case, + letter_one_case=whole_one_case or ( + one_case and i < clause_at + and t.role is None))) for i, t in enumerate(state.tokens)) # Delimited content whose vocabulary cannot settle it: extract's # escape sends an UNambiguous suffix straight through ("(MBA)" -> @@ -340,11 +361,11 @@ def classify(state: ParseState) -> ParseState: # `is_initial` matches an ASCII capital only, so a bare # LOWERCASE letter is tagged "initial" only by the fork's # branch ("jose e maria santos" tags a lowercase e), for a - # conjunctions_ambiguous member, or by the one-case numeral - # (#721), which never tags an own word that is a single-letter - # connective (the fork's branch takes those first). So for a - # letter of EITHER case in `conjunctions`, "initial" plus len 1 - # implies membership. It is kept only because `_tags_for` + # conjunctions_ambiguous member, or by the one-case letter + # (#721, #723), which never tags an own word that is a + # single-letter connective (the fork's branch takes those + # first). So for a letter of EITHER case in `conjunctions`, + # "initial" plus len 1 implies membership. It is kept only because `_tags_for` # starts from `set(token.tags)`, so # this clause is what keeps the emitter honest if a runner ever # hands classify tokens it did not build; no such path exists diff --git a/nameparser/_types.py b/nameparser/_types.py index b6b1f6aa..2be39eed 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -88,9 +88,9 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: word READ as an initial -- initial-shaped in a script that HAS #: initials ("J." or "А.", never "씨.", #320), or a marked #: single-letter connective in a name written in one case -#: (rules/P3, see CONJUNCTION_OR_INITIAL), or a one-letter roman -#: numeral in a name whose own words are written in one case, as its -#: capital is (rules/S10); +#: (rules/P3, see CONJUNCTION_OR_INITIAL), or any other bare letter +#: in a name whose own words are written in one case, where its +#: capital would be one ("j" as "J", never "é", rules/T4); #: "joined" a continuation of the token before it -- within one #: merged piece the tag is role-blind and every view joins the pair #: with a space ("Ph." + "D."; 'Ph. D., Jr.' gives last_list diff --git a/tests/v2/cases.py b/tests/v2/cases.py index eaed1afb..d7774342 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -7624,6 +7624,58 @@ def _check_cjk_shape_purity(self) -> None: notes="read as 'JOHN J V' is: in a one-case name the bare 'j' " "in front is the initial its capital is (S10). 1.4.0 " "read last 'j', suffix 'v'"), + Case("a_one_case_lone_letter_is_an_initial_and_unreported", + "j", {"given": "j"}, + notes="read as 'J' is (T4): an initial claims the word, so " + "O5's convention is not reported, where before #723 the " + "lowercase letter reported given-or-family"), + Case("a_one_case_letter_before_a_maiden_marker_stays_given", + "j nee smith", {"given": "j", "maiden": "smith"}, + classification="fix(#274)", + notes="read as 'J NEE SMITH' is (T4): M4 does not make an " + "initial the family name, where before #723 the " + "lowercase letter read family 'j'; 1.4.0 had no maiden " + "markers"), + Case("a_one_case_letter_before_a_link_does_not_join", + "jose j y lopez", + {"given": "jose", "middle": "j y", "family": "lopez"}, + classification="fix(#723)", + notes="read as 'JOSE J Y LOPEZ' is, in 1.4.0 too (T4): the " + "initial is no name word to P3's count, so the " + "single-letter link is in a three-word name and stays a " + "name word; 1.4.0 and 2.0-2.4 read the lowercase " + "spelling family 'j y lopez'"), + Case("a_one_case_letter_before_a_link_after_a_family_comma", + "smith, j y lopez", + {"given": "j", "middle": "y lopez", "family": "smith"}, + classification="fix(#723)", + notes="read as 'SMITH, J Y LOPEZ' is, in 1.4.0 too (T4), " + "for the row above's reason; 1.4.0 and 2.0-2.4 read the " + "lowercase spelling given 'j y lopez'"), + Case("a_one_case_letter_past_a_declined_marker_reads_as_its_capital", + "dr. née juan y lopez", + {"title": "dr.", "given": "née", "middle": "juan y", + "family": "lopez"}, + classification="fix(#723)", + notes="the declined marker leaves 'y' outside the name's own " + "words, so P3's one-case fork does not read it and T4 " + "does: the initial its capital is, as in 'DR. NÉE JUAN " + "Y LOPEZ'. 1.4.0 and 2.0-2.4 read family 'juan y lopez'"), + Case("a_lowercase_letter_past_a_declined_marker_among_capitals_joins", + "DR. NÉE JUAN y LOPEZ", + {"title": "DR.", "given": "NÉE", "family": "JUAN y LOPEZ"}, + notes="the row above's mixed-case twin: the own words are " + "capitals, but the name is not written in one case, so " + "the lower-case 'y' past the marker is no initial (T4) " + "and joins (#723's review)"), + Case("a_lowercase_numeral_after_a_marker_among_capitals_is_no_initial", + "MARY SMITH NEE v JONES", + {"given": "MARY", "middle": "SMITH NEE v", "family": "JONES"}, + notes="the name is not written in one case, so the lower-case " + "'v' is no initial (T4) and the marker reads as " + "'Mary Smith nee v Jones' does, as 1.4.0 and 2.4.2 read " + "it; its capital twin 'NEE V JONES' keeps maiden 'V " + "JONES'"), Case("an_uncased_word_in_front_is_no_initial", "김 v", {"given": "김", "family": "v"}, ambiguities=("suffix-or-name",), diff --git a/tests/v2/pipeline/test_classify.py b/tests/v2/pipeline/test_classify.py index fc10c6e9..38ee33fc 100644 --- a/tests/v2/pipeline/test_classify.py +++ b/tests/v2/pipeline/test_classify.py @@ -354,15 +354,37 @@ def test_the_fork_itself_never_reads_a_delimited_clauses_tokens() -> None: assert e.ambiguities == () -def test_a_word_after_the_maiden_marker_reads_as_plain_vocabulary() -> None: +@pytest.mark.parametrize("letter", ["e", "y"]) +def test_a_letter_after_the_maiden_marker_reads_as_its_capital( + letter: str) -> None: # symmetric with the delimited-clause case: a word inside the # maiden clause itself is not one of the name's own words either, - # so it never enters the fork -- 'e' after 'née' is bare vocabulary - # membership, the same reading it had before this PR. - out = _classified("juan garcia lopez née e") - assert "conjunction" in _tags(out, "e") - assert "initial" not in _tags(out, "e") - assert out.ambiguities == () + # so it never enters P3's fork, which would read the marked 'e' as + # an initial WITH a report and the unmarked 'y' as a connective. + # It is read as its capital in the clause is (rules/T4, #723): an + # initial, reported by nobody. + lower = _classified(f"juan garcia lopez née {letter}") + upper = _classified(f"JUAN GARCIA LOPEZ NÉE {letter.upper()}") + assert "initial" in _tags(lower, letter) + assert "conjunction" not in _tags(lower, letter) + assert _tags(lower, letter) == _tags(upper, letter.upper()) + assert lower.ambiguities == upper.ambiguities == () + + +@pytest.mark.parametrize("text", [ + "MARIA GARCIA NEE LOPEZ y FERNANDEZ", + "MARIA GARCIA (LOPEZ y FERNANDEZ)", + "DR. NÉE JUAN y LOPEZ", +]) +def test_a_lowercase_clause_letter_among_capitals_is_no_initial( + text: str) -> None: + # #723's review: the own words are capitals, but the clause beside + # them is not, so the name is written in more than one case and a + # bare lower-case letter there is no initial (rules/T4) -- the + # own-words fact alone would have tagged it one + out = _classified(text) + assert "initial" not in _tags(out, "y") + assert "conjunction" in _tags(out, "y") def test_a_clauses_own_marker_word_does_not_truncate_the_own_words() -> None: @@ -681,9 +703,10 @@ def test_the_caps_branch_reads_the_name_level_case_not_the_own_span( def reverted(token: WorkToken, n: str, state: ParseState, marker_tag: str | None, one_case_own: bool, - one_case: bool) -> frozenset[str]: + one_case: bool, letter_one_case: bool) -> frozenset[str]: return real(token, n, state, marker_tag, - one_case_own=one_case_own, one_case=one_case_own) + one_case_own=one_case_own, one_case=one_case_own, + letter_one_case=letter_one_case) monkeypatch.setattr(_classify_module, "_tags_for", reverted) broken = Parser(policy=on).parse("née JONES XYZ") diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 45ac067b..d8d9b470 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3334,17 +3334,17 @@ def test_every_one_letter_particle_veto_is_load_bearing( assert len(_p7_disagreements(lexicon)) == _P7_SITE_EFFECT[module] -# --- #721: a name written in one case reads as its capitals --------- -# S10 (the trailing roman numeral) says a name written wholly in one +# --- #721, #723: a name written in one case reads as its capitals --- +# T4 (the bare letter as an initial) says a name written wholly in one # case says nothing about any letter in it -- so a wholly lower-case # name parses exactly as the same name in capitals, fields (compared # case-folded) and report kinds alike. The grid crosses the shapes a one-letter numeral meets # -- every comma shape, a maiden clause, a title, a particle chain, a # bound given name, an initial, a nickname and a credential before or -# after it -- with the numerals a listed and a shape-only spelling of -# each length, three heads (one Catalan, one holding an ambiguous +# after it -- and the shapes any bare letter meets (#723), with the +# letters below, three heads (one Catalan, one holding an ambiguous # acronym) and all three orders: -# 1,872 pairs, about 3,750 parses, 0.29s on py3.11 (2026-10-18). +# 3,168 pairs, about 6,350 parses, 0.48s on py3.11 (2026-10-18). _ONE_CASE_SHAPES = ( "{a} {n}", "{a} {b} {n}", "{a} {n}, phd", "{a} {b} {n}, phd", "{a} {n}, jr.", "{b}, {a} {n}", ", {a} {n}", "{a} nee {b} {n}", @@ -3356,15 +3356,24 @@ def test_every_one_letter_particle_veto_is_load_bearing( "{a} b {n}", "{a} e {n}", "{a} {n} prof.", "{a} {b} {n} prof.", "{a} {b} {n} {n}", "{a} {b} {n}.", # a non-ASCII letter in front, whose capital is no initial - "{a} é {n}") -_ONE_CASE_NUMERALS = ("v", "i", "x", "vi", "iii", "ii", "iv", "vii") + "{a} é {n}", + # #723: the letter alone, before a maiden marker, before or after a + # connective, opening the name and after an empty head + "{n}", "{n} nee {b}", "{a} {n} y {b}", "{b}, {n} y {c}", "{n} {a}", + ", {n}") +#: the one-letter numerals and a listed and a shape-only spelling of +#: each longer length, then one letter from each other fork a bare +#: letter meets (#723): no vocabulary, a connective, a marked one +_ONE_CASE_LETTERS = ("v", "i", "x", "vi", "iii", "ii", "iv", "vii", + "j", "y", "e") _ONE_CASE_HEADS = (("john", "smith", "jones"), ("carod", "puig", "rovira"), ("mary", "ma", "lee")) #: the negative controls, measured 2026-10-18 on this grid, which -#: master (2bfa9533) disagreed on 315 of: with classify's one-case -#: numeral initial taken out, and with the numeral fork's one-case -#: letter in front taken out -- each guard seen on its own -_ONE_CASE_UNGUARDED = 225 +#: 16ee2851 (before #723, its numerals already guarded) disagreed on +#: 30 of: with classify's one-case letter initial taken out, and with +#: the numeral fork's one-case letter in front taken out -- each guard +#: seen on its own +_ONE_CASE_UNGUARDED = 306 _ONE_CASE_LETTER_IN_FRONT_UNGUARDED = 117 @@ -3375,9 +3384,9 @@ def sig(name: ParsedName) -> tuple[object, ...]: out = [] for order in (GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST): parser = Parser(policy=Policy(name_order=order)) - for shape, numeral, (a, b, c) in itertools.product( - _ONE_CASE_SHAPES, _ONE_CASE_NUMERALS, _ONE_CASE_HEADS): - lower = shape.format(a=a, b=b, c=c, n=numeral) + for shape, letter, (a, b, c) in itertools.product( + _ONE_CASE_SHAPES, _ONE_CASE_LETTERS, _ONE_CASE_HEADS): + lower = shape.format(a=a, b=b, c=c, n=letter) if sig(parser.parse(lower)) != sig(parser.parse(lower.upper())): out.append(f"[{order}] {lower!r}") return out @@ -3388,10 +3397,10 @@ def test_a_one_case_name_reads_as_its_capitals() -> None: assert not failures, "\n".join(failures[:10]) -def test_the_one_case_grid_sees_the_numeral_initial( +def test_the_one_case_grid_sees_the_letter_initial( monkeypatch: pytest.MonkeyPatch) -> None: - # classify's one use of `_ROMAN` is the one-case numeral initial - monkeypatch.setattr("nameparser._pipeline._classify._ROMAN", + # classify's one use of `_INITIAL` is the one-case letter initial + monkeypatch.setattr("nameparser._pipeline._classify._INITIAL", re.compile(r"(?!)")) assert len(_one_case_disagreements()) == _ONE_CASE_UNGUARDED diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 03338891..a871f168 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -142,8 +142,6 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # SAME object, imported for the trailing read's weighed-word test # (#620), as _PERIOD_ABBREV is above ("_pieces", "_ROMAN"): "roman_numeral", - # SAME object, imported for the one-case numeral initial (#721) - ("_classify", "_ROMAN"): "roman_numeral", # SAME object, imported for the report on a numeral before a # credential (#721) ("_assign", "_ROMAN"): "roman_numeral", @@ -163,6 +161,8 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # above, which assert the documented RELATIONSHIP instead: ("_render", "_INITIAL"): None, # config's pattern minus one "?" ("_vocab", "_INITIAL"): None, # same + # SAME object, imported for the one-case letter initial (#723) + ("_classify", "_INITIAL"): None, ("_render", "_DOTTED_INITIAL"): None, # _INITIAL's period alternative # #542: the name run after a Mac/Mc prefix, read on the word as # written once _MAC has matched its composed spelling; no config diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index e19b471d..9c78ba7c 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -42,6 +42,7 @@ "DOE, JANE MA VD PHD" "DOE, JANE PHD VD" "DOE, JANE PHD VD MA" +"DR. NÉE JUAN y LOPEZ" "Davis Royce, Ed" "De La Cruz, Ed" "De La Cruz, M.J. K.L." @@ -126,6 +127,7 @@ "II Van Johnson" "IVOI, PAUL D'" "Ivoi, Paul d'" +"J NEE SMITH" "J-E-P DUPONT" "J. Smith" "J. née Jones Smith V" @@ -310,6 +312,7 @@ "Jose E Maria Santos" "Jose Ortega-Y-Gasset" "Jose e Maria Santos" +"Jose j y Lopez" "Josep Carod i Rovira" "Josep Lluis Carod i III" "Josep i Rovira" @@ -508,9 +511,13 @@ "doe, jane v phd do" "donovan mcnabb-smith" "dr. juan garcia III" +"dr. née juan y lopez" "émile zola" "ibn Awf abdul Rahman" "ina binti navalamar" +"j" +"j nee Smith" +"j nee smith" "j.-e.-p. dupont" "jacobus henricus van 't hoff" "john doe q.c." @@ -547,9 +554,12 @@ "john v" "john van der berg ma" "john van e phd" +"jose e lopez" "jose e maria santos" "jose e phd" +"jose j y lopez" "jose ortega-y-gasset" +"jose y lopez" "josé garcía" "juan de la vega" "juan e-f smith" @@ -566,6 +576,7 @@ "née Jones" "qc mp" "smith, de ma" +"smith, j y lopez" "smith, john e phd" "tran lac" "van Berg Jan de" @@ -575,6 +586,7 @@ "wang meng" "zaʿīm, ḥusnī az-" "Ó. Pérez" +"é nee smith" "Агриппина Младшая" "Ана Хорват старији Џонс" "Гай Плиний Старший Иванов" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 8aded66a..70e509f1 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -526,6 +526,7 @@ {"name": "DOE, JOHN MA", "given": "JOHN", "family": "DOE", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "DOE, MARY JO MA", "given": "MARY", "middle": "JO", "family": "DOE", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. J. D."} {"name": "DR DOE", "title": "DR", "family": "DOE", "initials": "D."} +{"name": "DR. NÉE JUAN y LOPEZ", "title": "DR.", "given": "NÉE", "family": "JUAN y LOPEZ", "initials": "N. J. L."} {"name": "Dale, Antonius van", "given": "Antonius", "family": "van Dale", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "A. D."} {"name": "Dalen, Cornelis van", "given": "Cornelis", "family": "van Dalen", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "C. D."} {"name": "Dame Mary", "title": "Dame", "given": "Mary", "initials": "M."} @@ -1236,6 +1237,7 @@ {"name": "Iñaki", "given": "Iñaki", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "I."} {"name": "Iñaki", "order": "FAMILY_FIRST", "family": "Iñaki", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "I."} {"name": "J A D", "given": "J", "middle": "A", "family": "D", "initials": "J. A. D."} +{"name": "J NEE SMITH", "given": "J", "maiden": "SMITH", "initials": "J."} {"name": "J-E-P DUPONT", "given": "J-E-P", "family": "DUPONT", "initials": "J. D."} {"name": "J. A. D.", "given": "J.", "middle": "A.", "family": "D.", "initials": "J. A. D."} {"name": "J. None D.", "given": "J.", "middle": "None", "family": "D.", "initials": "J. N. D."} @@ -1678,6 +1680,7 @@ {"name": "Jose Ortega-Y-Gasset", "given": "Jose", "family": "Ortega-Y-Gasset", "initials": "J. O."} {"name": "Jose e Maria", "given": "Jose", "middle": "e", "family": "Maria", "initials": "J. e. M."} {"name": "Jose e Maria Santos", "given": "Jose e Maria", "family": "Santos", "initials": "J. M. S."} +{"name": "Jose j y Lopez", "given": "Jose", "family": "j y Lopez", "initials": "J. j. L."} {"name": "Josef Václav Svoboda", "given": "Josef", "middle": "Václav", "family": "Svoboda", "initials": "J. V. S."} {"name": "Josep Balari i Jovany", "given": "Josep", "family": "Balari i Jovany", "initials": "J. B. J."} {"name": "Josep Campabadal i Calvet", "given": "Josep", "family": "Campabadal i Calvet", "initials": "J. C. C."} @@ -3357,6 +3360,7 @@ {"name": "dr. john p. doe-Ray, CLU, CFP, LUTC", "title": "dr.", "given": "john", "middle": "p.", "family": "doe-Ray", "suffix": "CLU, CFP, LUTC", "ambiguities": ["COMMA_STRUCTURE"], "initials": "j. p. d."} {"name": "dr. juan de la vega jr.", "title": "dr.", "given": "juan", "family": "de la vega", "suffix": "jr.", "initials": "j. v."} {"name": "dr. juan garcia III", "title": "dr.", "given": "juan", "family": "garcia", "suffix": "III", "initials": "j. g."} +{"name": "dr. née juan y lopez", "title": "dr.", "given": "née", "middle": "juan y", "family": "lopez", "initials": "n. j. y. l."} {"name": "dr. p. Gabrielė Aldona Petraitytė", "title": "dr.", "given": "p.", "middle": "Gabrielė Aldona", "family": "Petraitytė", "initials": "p. G. A. P."} {"name": "dr. sc. Branko Hrvoje Marić mlađi, prof.", "title": "dr. sc. prof.", "given": "Branko", "middle": "Hrvoje", "family": "Marić", "suffix": "mlađi", "initials": "B. H. M."} {"name": "e and e", "given": "e and e", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "e. e."} @@ -3372,6 +3376,9 @@ {"name": "hr. Hans Henrik Sven Åge Møller, MF", "title": "hr.", "given": "MF", "family": "Hans Henrik Sven Åge Møller", "initials": "M. H. H. S. Å. M."} {"name": "ibn Awf abdul Rahman", "family": "ibn Awf abdul Rahman", "initials": "A. a. R."} {"name": "ina binti navalamar", "given": "ina", "family": "binti navalamar", "initials": "i. n."} +{"name": "j", "given": "j", "initials": "j."} +{"name": "j nee Smith", "given": "j", "maiden": "Smith", "initials": "j."} +{"name": "j nee smith", "given": "j", "maiden": "smith", "initials": "j."} {"name": "j.-e.-p. dupont", "given": "j.-e.-p.", "family": "dupont", "initials": "j. d."} {"name": "jack ma", "given": "jack", "family": "ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. m."} {"name": "jacobus henricus van 't hoff", "given": "jacobus", "middle": "henricus", "family": "van 't hoff", "ambiguities": ["UNBALANCED_DELIMITER"], "initials": "j. h. h."} @@ -3423,9 +3430,12 @@ {"name": "john van e phd", "given": "john", "family": "van e", "suffix": "phd", "initials": "j. e."} {"name": "johnny y", "given": "johnny", "family": "y", "initials": "j. y."} {"name": "jones, john e", "given": "john", "middle": "e", "family": "jones", "initials": "j. e. j."} +{"name": "jose e lopez", "given": "jose", "middle": "e", "family": "lopez", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. l."} {"name": "jose e maria santos", "given": "jose", "middle": "e maria", "family": "santos", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. m. s."} {"name": "jose e phd", "given": "jose", "family": "e", "suffix": "phd", "initials": "j. e."} +{"name": "jose j y lopez", "given": "jose", "middle": "j y", "family": "lopez", "initials": "j. j. l."} {"name": "jose ortega-y-gasset", "given": "jose", "family": "ortega-y-gasset", "initials": "j. o."} +{"name": "jose y lopez", "given": "jose", "middle": "y", "family": "lopez", "initials": "j. y. l."} {"name": "josep carod i rovira", "given": "josep", "middle": "carod i", "family": "rovira", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. c. i. r."} {"name": "josé garcía", "given": "josé", "family": "garcía", "initials": "j. g."} {"name": "juan de la vega", "given": "juan", "family": "de la vega", "initials": "j. v."} @@ -3492,6 +3502,7 @@ {"name": "scott l. werner", "given": "scott", "middle": "l.", "family": "werner", "initials": "s. l. w."} {"name": "señora María García", "title": "señora", "given": "María", "family": "García", "initials": "M. G."} {"name": "smith, de ma", "family": "smith de ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "s. m."} +{"name": "smith, j y lopez", "given": "j", "middle": "y lopez", "family": "smith", "initials": "j. l. s."} {"name": "smith, john e phd", "given": "john", "middle": "e", "family": "smith", "suffix": "phd", "initials": "j. e. s."} {"name": "st.repr. Gregers Werle Marve Almar von Fleksnes jr.", "title": "st.repr.", "given": "Gregers", "middle": "Werle Marve Almar", "family": "von Fleksnes", "suffix": "jr.", "initials": "G. W. M. A. F."} {"name": "st.repr. von Fleksnes Gregers Werle Marve Almar", "order": "FAMILY_FIRST", "title": "st.repr.", "given": "Fleksnes", "middle": "Gregers Werle Marve Almar", "family": "von", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "F. G. W. M. A. v."} @@ -3604,6 +3615,7 @@ {"name": "Þorsteinn Sveinsson", "given": "Þorsteinn", "family": "Sveinsson", "initials": "Þ. S."} {"name": "Þuríður Pálsdóttir", "given": "Þuríður", "family": "Pálsdóttir", "initials": "Þ. P."} {"name": "Þórunn Egilsdóttir", "given": "Þórunn", "family": "Egilsdóttir", "initials": "Þ. E."} +{"name": "é nee smith", "family": "é", "maiden": "smith", "initials": "é."} {"name": "Āqā Buzurg aṭ-Ṭihrānī", "given": "Āqā", "middle": "Buzurg", "family": "aṭ-Ṭihrānī", "initials": "Ā. B. a."} {"name": "ĐBQH Nguyễn Văn Anh", "order": "FAMILY_FIRST", "given": "Nguyễn", "middle": "Văn Anh", "family": "ĐBQH", "initials": "N. V. A. Đ."} {"name": "Ğawīdān _ Hānum", "given": "Ğawīdān", "middle": "_", "family": "Hānum", "initials": "Ğ. _. H."}