From 83a2df33580cae745f9510639cc5f42820caacfb Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 18:11:47 -0700 Subject: [PATCH 1/3] Record why H6 reads no title by shape before a family comma decisions/H6 point (1) gave only the doctrine and the absence of need for skipping H2's shape before a family comma. #710's close-out asked again whether `Xyz. Smith, John` could read title 'Xyz.'. There is a cost: a surname can open with an abbreviated word, and Filipino records write Santa and Santo as `Sta.` and `Sto.`, which the shape would take as titles (`Sta. Maria, Juan` -> title 'Sta.'). The decisions entry records the Census counts and what reopening would need. rules/H6's rationale names the class, and gains `Sta. Maria, Juan` as a boundary example. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H6-title-before-a-family-comma.md | 2 ++ docs/design/rules/H6-title-before-a-family-comma.md | 9 ++++++--- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 1 + 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/H6-title-before-a-family-comma.md b/docs/design/decisions/H6-title-before-a-family-comma.md index 1a9cf959..011d0088 100644 --- a/docs/design/decisions/H6-title-before-a-family-comma.md +++ b/docs/design/decisions/H6-title-before-a-family-comma.md @@ -7,3 +7,5 @@ MEASURED 2026-10-17 against master `eb2c1fa5`, py3.11. Snapshot (`uv run python tools/differential/snapshot.py --since master`, before this change's own rows entered the corpora): 20 entries move, all one shape — a title leaves the family for the title field — and 18 of their names are held only by corpus_cldr.jsonl. Several of those are names C1 reads as a family comma though the part after it is a credential CLDR's vocabulary lacks (`hr. Hans Henrik Sven Åge Møller, MF`); the title still comes out right there, and the family stays as wrong as before. CLDR agreement: 29 fields now agree and 28 more get closer to CLDR's value, with none regressing. Merging master after #705 listed `bapak` adds one more of the same shape, `Bapak Adinata, Dwi Putro bin MP` (title 'Bapak' out of family 'bin Bapak Adinata'), two more CLDR fields agreeing and one closer. Cost (`sys.setprofile` call events, one warm-up): `Dr. Juan Q. Xavier de la Vega III` 366 → 366 (the benchmark's `_REFERENCE` row, 307, is unmoved too), `Smith, John` 161 → 161, `Smith Jones, John` 203 → 203 (an inline tag test ahead of `is_title_piece` keeps a part no title opens free of frames), `Dr. Smith, John` 200 → 201 (one `is_title_piece`; the floor's `is_suffix_piece` on 'Smith' replaces the family loop's, which no longer asks the first family piece), `Prof. Dr. Brühl, Ada` 279 → 280. The family loop does not re-ask a piece at or past the floor, so a long post-nominal run costs one frame more than on master, not one per member: `Dr. Smith Jr. PhD MD, John` 306 → 307 (310 before the /simplify round), `Dr. Smith` plus twenty `PhD`, `, John` 903 → 904 (924 before). Those two changes moved no reading on a grid of 88,740 family-comma heads (one to four words from `Dr.`, `Prof`, `St`, `Freiherr`, `Sr.`, `Ms`, `Smith`, `van`, `Vd`, `PhD`, `Jr.`, `MA`, `V`, `de`, `Rev.`, `Mc`, `née`, before `, John`) under the default and `FAMILY_FIRST` orders, comparing fields, ambiguity kinds and `initials()`. THE PR REVIEW (2026-10-17), three decisions. (a) `gov` JOINS TITLES (Derek). `Lt. Gov. Smith, John` read title 'Lt.', family 'Gov. Smith' — a compound split across two fields, where master kept it whole in the family and the comma-less `Lt. Gov. John Smith` reads title 'Lt. Gov.' by H2's shape. The run is vocabulary-only, so the fix is the vocabulary: `gov` is the abbreviation of the listed `governor`. Counts (`uv run python tools/name_usage.py Gov`, 2026-10-17): 236 US Census 2010 surname bearers, rank 80,419, 91.53% Asian/Pacific Islander; SSA given names under 5 a year; Wikidata 3 family-name bearers, all Israeli. The claim acts only in front, so `John Gov` keeps family 'Gov', and a surname written first loses it (`Gov Sokha` → title 'Gov'), the graf-class trade (#salutation-titles) at a smaller incidence. No corpus name moved. The other unlisted second words the review found (`insp`, `sen`, `supt`, `cmdr`, `atty`, `msgr`) still split; each is a vocabulary question on its own. (b) `Dr. PhD Smith, John` KEEPS the branch reading, family 'PhD Smith' (Derek): the first piece the run leaves is family whatever it is, as piece 0 is with no title (`PhD Smith, John`, read that way by 1.4.0 too); master and 1.4.0 gave `Dr. PhD Smith, John` family 'Dr. Smith', suffix 'PhD'. Garbage input, so no row. (c) `Graf Lambsdorff, Otto` reads title 'Graf' while `Freiherr von Berg, Ed` keeps the rank, and the clause first written for it gave the 1919 legal-surname argument as the reason — which holds for every German rank alike. The reason is the particle stop: `freiherr` is the one rank in `particles`. rules/H6's Accepted clause now says so, and the vocabulary question (should `graf`, `fürst`, `prinz` and the rest read as `freiherr` does, with and without a comma) is [#709](https://github.com/derek73/python-nameparser/issues/709). +- 2026-10-18 (Derek), #710 — POINT (1) KEEPS ITS VOCABULARY-ONLY READING, AND NOW HAS A NAMED COST BEHIND IT. Closing #710 raised it again: `Xyz. Smith, John` keeps family 'Xyz. Smith', and since no surname ends in a period, title 'Xyz.' looked like a fair reading. The 2026-10-17 entry gave only the doctrine (the slot is a surname's slot, as at #H5) and the absence of need (every CLDR case is a listed title). It named no surname the shape would break. There is one: a surname can OPEN with an abbreviated word. Filipino records write Santa and Santo as `Sta.` and `Sto.`, and H2's shape (two or more letters, one trailing period) claims both. So `Sta. Maria, Juan`, which reads family 'Sta. Maria' today, would read title 'Sta.', family 'Maria'. US Census 2010 surname bearers, measured 2026-10-18 with `uv run python tools/name_usage.py "Sta. Maria"` (and likewise for each): Sta. Ana 408, Sta. Maria 376, Sto. Domingo 163, Sta. Cruz 110, each over 90% Asian/Pacific Islander. The Census strips punctuation (it looks these up as `STAMARIA`), so the counts show how many people bear the surname, not how many write it abbreviated. `ste` is in `particles`, so point (3)'s stop already keeps `Ste. Marie, Jean` whole; `sta` and `sto` are in no vocabulary set, so nothing would stop the shape. `Mt.` and `Ft.` (`Mt. Pleasant`, `Ft. Smith`) have the same shape; their incidence was not measured. rules/H6's rationale now names the class, and `Sta. Maria, Juan` is a boundary example there. + Declined: peeling by H2's shape before a family comma, again. Reopening it would need a stop for `sta`/`sto`, most likely listing them as particles beside `ste`, which is a vocabulary question of its own: as particles they would also join forward inside a name (`Juan Sta. Maria`), and that has not been measured. diff --git a/docs/design/rules/H6-title-before-a-family-comma.md b/docs/design/rules/H6-title-before-a-family-comma.md index 3fe34e9a..98673778 100644 --- a/docs/design/rules/H6-title-before-a-family-comma.md +++ b/docs/design/rules/H6-title-before-a-family-comma.md @@ -5,9 +5,11 @@ H6. Rationale: the listing form moves the surname to the front and ahead of the name ends up ahead of the surname — `Dr. Smith, John` is `Dr. John Smith` indexed. The comma names the family, and a title is no part of it. But the slot in front of a family - comma is a surname's slot as well, so only the vocabulary claims - a word there, never H2's shape, and a word that is a particle as - well as a title opens a surname rather than ranking a person. + comma is a surname's slot as well, and a surname can open with + an abbreviated word (Sta. Maria, Sto. Domingo), so only the + vocabulary claims a word there, never H2's shape, and a word + that is a particle as well as a title opens a surname rather + than ranking a person. A run of title words opening the part before a family comma is the title, joining the title field in input order with any title after the comma. The run admits a title word only as vocabulary @@ -28,6 +30,7 @@ H6. Rationale: the listing form moves the surname to the front and "Prof. Smith, John, PhD" → family="Smith" "Dr., John" → family="Dr." · boundary "Dr. King, John" → family="King" · boundary + "Sta. Maria, Juan" → family="Sta. Maria" · boundary "MD PhD Jr., John" → family="MD" · boundary "St Pierre, Ed" → family="St Pierre" · boundary "Dr. St. Pierre, Ed" → family="St. Pierre" · boundary diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index e19b471d..2a428f56 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -463,6 +463,7 @@ "Smith, de Mesnil Jean" "Smith. John" "St Pierre, Ed" +"Sta. Maria, Juan" "Stariji Bruegel" "Steven Hardman, MD, DO, DDS" "Steven Hardman, RN - CRNA" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 8aded66a..51a35c03 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -2918,6 +2918,7 @@ {"name": "Srta. Andréia da Silva", "title": "Srta.", "given": "Andréia", "family": "da Silva", "initials": "A. S."} {"name": "St Clair", "title": "St", "family": "Clair", "initials": "C."} {"name": "St Pierre, Ed", "given": "Ed", "family": "St Pierre", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "E. P."} +{"name": "Sta. Maria, Juan", "given": "Juan", "family": "Sta. Maria", "initials": "J. S. M."} {"name": "Stander Zelda Herman", "order": "FAMILY_FIRST", "given": "Zelda", "middle": "Herman", "family": "Stander", "initials": "Z. H. S."} {"name": "Stander, Zelda Herman", "given": "Zelda", "middle": "Herman", "family": "Stander", "initials": "Z. H. S."} {"name": "Stariji Bruegel", "given": "Stariji", "family": "Bruegel", "initials": "S. B."} From 7d8be67566814f82a0ce7a83fd527fd9aac70bf4 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 18:17:58 -0700 Subject: [PATCH 2/3] Correct the H6 Sta./Sto. entry after review The Juan Sta. Maria example presented a fix as a risk: today it already splits the surname (middle 'Sta.'), and particle status fixes it. The real open cost is P1's fold on a bare leading Sta, which the ambiguous listing avoids (both measured). Ste. Marie, Jean stays whole because ste is no title, not because of point (3)'s stop. The Census counts are bearers recorded with the abbreviation (Santa Maria is a separate 12,375-bearer entry). Mt./Ft. counts recorded. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H6-title-before-a-family-comma.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/design/decisions/H6-title-before-a-family-comma.md b/docs/design/decisions/H6-title-before-a-family-comma.md index 011d0088..d5dba56f 100644 --- a/docs/design/decisions/H6-title-before-a-family-comma.md +++ b/docs/design/decisions/H6-title-before-a-family-comma.md @@ -7,5 +7,6 @@ MEASURED 2026-10-17 against master `eb2c1fa5`, py3.11. Snapshot (`uv run python tools/differential/snapshot.py --since master`, before this change's own rows entered the corpora): 20 entries move, all one shape — a title leaves the family for the title field — and 18 of their names are held only by corpus_cldr.jsonl. Several of those are names C1 reads as a family comma though the part after it is a credential CLDR's vocabulary lacks (`hr. Hans Henrik Sven Åge Møller, MF`); the title still comes out right there, and the family stays as wrong as before. CLDR agreement: 29 fields now agree and 28 more get closer to CLDR's value, with none regressing. Merging master after #705 listed `bapak` adds one more of the same shape, `Bapak Adinata, Dwi Putro bin MP` (title 'Bapak' out of family 'bin Bapak Adinata'), two more CLDR fields agreeing and one closer. Cost (`sys.setprofile` call events, one warm-up): `Dr. Juan Q. Xavier de la Vega III` 366 → 366 (the benchmark's `_REFERENCE` row, 307, is unmoved too), `Smith, John` 161 → 161, `Smith Jones, John` 203 → 203 (an inline tag test ahead of `is_title_piece` keeps a part no title opens free of frames), `Dr. Smith, John` 200 → 201 (one `is_title_piece`; the floor's `is_suffix_piece` on 'Smith' replaces the family loop's, which no longer asks the first family piece), `Prof. Dr. Brühl, Ada` 279 → 280. The family loop does not re-ask a piece at or past the floor, so a long post-nominal run costs one frame more than on master, not one per member: `Dr. Smith Jr. PhD MD, John` 306 → 307 (310 before the /simplify round), `Dr. Smith` plus twenty `PhD`, `, John` 903 → 904 (924 before). Those two changes moved no reading on a grid of 88,740 family-comma heads (one to four words from `Dr.`, `Prof`, `St`, `Freiherr`, `Sr.`, `Ms`, `Smith`, `van`, `Vd`, `PhD`, `Jr.`, `MA`, `V`, `de`, `Rev.`, `Mc`, `née`, before `, John`) under the default and `FAMILY_FIRST` orders, comparing fields, ambiguity kinds and `initials()`. THE PR REVIEW (2026-10-17), three decisions. (a) `gov` JOINS TITLES (Derek). `Lt. Gov. Smith, John` read title 'Lt.', family 'Gov. Smith' — a compound split across two fields, where master kept it whole in the family and the comma-less `Lt. Gov. John Smith` reads title 'Lt. Gov.' by H2's shape. The run is vocabulary-only, so the fix is the vocabulary: `gov` is the abbreviation of the listed `governor`. Counts (`uv run python tools/name_usage.py Gov`, 2026-10-17): 236 US Census 2010 surname bearers, rank 80,419, 91.53% Asian/Pacific Islander; SSA given names under 5 a year; Wikidata 3 family-name bearers, all Israeli. The claim acts only in front, so `John Gov` keeps family 'Gov', and a surname written first loses it (`Gov Sokha` → title 'Gov'), the graf-class trade (#salutation-titles) at a smaller incidence. No corpus name moved. The other unlisted second words the review found (`insp`, `sen`, `supt`, `cmdr`, `atty`, `msgr`) still split; each is a vocabulary question on its own. (b) `Dr. PhD Smith, John` KEEPS the branch reading, family 'PhD Smith' (Derek): the first piece the run leaves is family whatever it is, as piece 0 is with no title (`PhD Smith, John`, read that way by 1.4.0 too); master and 1.4.0 gave `Dr. PhD Smith, John` family 'Dr. Smith', suffix 'PhD'. Garbage input, so no row. (c) `Graf Lambsdorff, Otto` reads title 'Graf' while `Freiherr von Berg, Ed` keeps the rank, and the clause first written for it gave the 1919 legal-surname argument as the reason — which holds for every German rank alike. The reason is the particle stop: `freiherr` is the one rank in `particles`. rules/H6's Accepted clause now says so, and the vocabulary question (should `graf`, `fürst`, `prinz` and the rest read as `freiherr` does, with and without a comma) is [#709](https://github.com/derek73/python-nameparser/issues/709). -- 2026-10-18 (Derek), #710 — POINT (1) KEEPS ITS VOCABULARY-ONLY READING, AND NOW HAS A NAMED COST BEHIND IT. Closing #710 raised it again: `Xyz. Smith, John` keeps family 'Xyz. Smith', and since no surname ends in a period, title 'Xyz.' looked like a fair reading. The 2026-10-17 entry gave only the doctrine (the slot is a surname's slot, as at #H5) and the absence of need (every CLDR case is a listed title). It named no surname the shape would break. There is one: a surname can OPEN with an abbreviated word. Filipino records write Santa and Santo as `Sta.` and `Sto.`, and H2's shape (two or more letters, one trailing period) claims both. So `Sta. Maria, Juan`, which reads family 'Sta. Maria' today, would read title 'Sta.', family 'Maria'. US Census 2010 surname bearers, measured 2026-10-18 with `uv run python tools/name_usage.py "Sta. Maria"` (and likewise for each): Sta. Ana 408, Sta. Maria 376, Sto. Domingo 163, Sta. Cruz 110, each over 90% Asian/Pacific Islander. The Census strips punctuation (it looks these up as `STAMARIA`), so the counts show how many people bear the surname, not how many write it abbreviated. `ste` is in `particles`, so point (3)'s stop already keeps `Ste. Marie, Jean` whole; `sta` and `sto` are in no vocabulary set, so nothing would stop the shape. `Mt.` and `Ft.` (`Mt. Pleasant`, `Ft. Smith`) have the same shape; their incidence was not measured. rules/H6's rationale now names the class, and `Sta. Maria, Juan` is a boundary example there. - Declined: peeling by H2's shape before a family comma, again. Reopening it would need a stop for `sta`/`sto`, most likely listing them as particles beside `ste`, which is a vocabulary question of its own: as particles they would also join forward inside a name (`Juan Sta. Maria`), and that has not been measured. +- 2026-10-18 (Derek), #710 — POINT (1) KEEPS ITS VOCABULARY-ONLY READING, AND NOW HAS A NAMED COST BEHIND IT. Closing #710 raised it again: `Xyz. Smith, John` keeps family 'Xyz. Smith', and since no surname ends in a period, title 'Xyz.' looked like a fair reading. The 2026-10-17 entry gave only the doctrine (the slot is a surname's slot, as at #H5) and the absence of need (every CLDR case is a listed title). It named no surname the shape would break. There is one: a surname can OPEN with an abbreviated word. Filipino records write Santa and Santo as `Sta.` and `Sto.`, and H2's shape (two or more letters, one trailing period) claims both. So `Sta. Maria, Juan`, which reads family 'Sta. Maria' today, would read title 'Sta.', family 'Maria'. US Census 2010 surname bearers, measured 2026-10-18 with `uv run python tools/name_usage.py "Sta. Maria"` (and likewise for each): Sta. Ana 408, Sta. Maria 376, Sto. Domingo 163, Sta. Cruz 110, each over 90% Asian/Pacific Islander. The Census strips punctuation (it looks these up as `STAMARIA`), but the unabbreviated spelling is a separate entry (`Santa Maria`, 12,375, 77% Hispanic), so these counts are the bearers recorded with the abbreviation; what the fold hides is whether they wrote its period, which is what H2's shape needs. `ste` is a particle and no title, so `Ste. Marie, Jean` never enters H6's run today; under a shape peel, point (3)'s particle stop is what would keep it whole. `sta` and `sto` are in no vocabulary set, so nothing would stop the shape on them. `Mt.` and `Ft.` have the same shape and are rare as surnames (`Mt. Pleasant` and `Ft. Smith` each under 100 Census bearers, same date). rules/H6's rationale now names the class, and `Sta. Maria, Juan` is a boundary example there. + NOT FIXED BY THIS RULE: the comma-less writing splits the same surname. `Juan Sta. Maria` reads given 'Juan', middle 'Sta.', family 'Maria' (and `Juan Sto. Domingo` likewise), so it disagrees with `Sta. Maria, Juan` on the family; `Jean Ste. Marie` reads family 'Ste. Marie' because `ste` is a particle. Listing `sta`/`sto` as particles (measured with `Lexicon.default().add(particles={"sta", "sto"})`) fixes both comma-less readings and leaves `Sta. Maria, Juan` as it is. Added that way they are never-given, so P1's fold takes a bare leading one: `Sta Juan` moves from given 'Sta', family 'Juan' to family 'Sta Juan' (bare `Sta` has 120 US Census surname bearers). Listed as ambiguous instead (adding them to `particles_ambiguous` as well), the comma-less readings are fixed the same way and `Sta Juan` keeps given 'Sta', with a `PARTICLE_OR_GIVEN` report. Neither listing has been decided. + Declined: peeling by H2's shape before a family comma, again. Reopening it would need a stop for `sta`/`sto`, which the particle listing above would supply. From ba7a3baaec5fc737126a35b2e16461b9b280919e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 18:19:06 -0700 Subject: [PATCH 3/3] State the condition on H6's Sta./Sto. stop A shape peel gets a stop from a particle listing only if it keeps point (3)'s particle stop, which H2's comma-less head does not have (Ste. Marie Jean reads title 'Ste.'). Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/H6-title-before-a-family-comma.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/decisions/H6-title-before-a-family-comma.md b/docs/design/decisions/H6-title-before-a-family-comma.md index d5dba56f..59b39d19 100644 --- a/docs/design/decisions/H6-title-before-a-family-comma.md +++ b/docs/design/decisions/H6-title-before-a-family-comma.md @@ -9,4 +9,4 @@ - 2026-10-18 (Derek), #710 — POINT (1) KEEPS ITS VOCABULARY-ONLY READING, AND NOW HAS A NAMED COST BEHIND IT. Closing #710 raised it again: `Xyz. Smith, John` keeps family 'Xyz. Smith', and since no surname ends in a period, title 'Xyz.' looked like a fair reading. The 2026-10-17 entry gave only the doctrine (the slot is a surname's slot, as at #H5) and the absence of need (every CLDR case is a listed title). It named no surname the shape would break. There is one: a surname can OPEN with an abbreviated word. Filipino records write Santa and Santo as `Sta.` and `Sto.`, and H2's shape (two or more letters, one trailing period) claims both. So `Sta. Maria, Juan`, which reads family 'Sta. Maria' today, would read title 'Sta.', family 'Maria'. US Census 2010 surname bearers, measured 2026-10-18 with `uv run python tools/name_usage.py "Sta. Maria"` (and likewise for each): Sta. Ana 408, Sta. Maria 376, Sto. Domingo 163, Sta. Cruz 110, each over 90% Asian/Pacific Islander. The Census strips punctuation (it looks these up as `STAMARIA`), but the unabbreviated spelling is a separate entry (`Santa Maria`, 12,375, 77% Hispanic), so these counts are the bearers recorded with the abbreviation; what the fold hides is whether they wrote its period, which is what H2's shape needs. `ste` is a particle and no title, so `Ste. Marie, Jean` never enters H6's run today; under a shape peel, point (3)'s particle stop is what would keep it whole. `sta` and `sto` are in no vocabulary set, so nothing would stop the shape on them. `Mt.` and `Ft.` have the same shape and are rare as surnames (`Mt. Pleasant` and `Ft. Smith` each under 100 Census bearers, same date). rules/H6's rationale now names the class, and `Sta. Maria, Juan` is a boundary example there. NOT FIXED BY THIS RULE: the comma-less writing splits the same surname. `Juan Sta. Maria` reads given 'Juan', middle 'Sta.', family 'Maria' (and `Juan Sto. Domingo` likewise), so it disagrees with `Sta. Maria, Juan` on the family; `Jean Ste. Marie` reads family 'Ste. Marie' because `ste` is a particle. Listing `sta`/`sto` as particles (measured with `Lexicon.default().add(particles={"sta", "sto"})`) fixes both comma-less readings and leaves `Sta. Maria, Juan` as it is. Added that way they are never-given, so P1's fold takes a bare leading one: `Sta Juan` moves from given 'Sta', family 'Juan' to family 'Sta Juan' (bare `Sta` has 120 US Census surname bearers). Listed as ambiguous instead (adding them to `particles_ambiguous` as well), the comma-less readings are fixed the same way and `Sta Juan` keeps given 'Sta', with a `PARTICLE_OR_GIVEN` report. Neither listing has been decided. - Declined: peeling by H2's shape before a family comma, again. Reopening it would need a stop for `sta`/`sto`, which the particle listing above would supply. + Declined: peeling by H2's shape before a family comma, again. Reopening it would need a stop for `sta`/`sto`. The particle listing above would supply one only if the peel kept point (3)'s particle stop, and that has to be done on purpose: H2's shape at the head of a name with no comma has no such stop, so `Ste. Marie Jean` reads title 'Ste.' today.