Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_

**Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.<key>` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo.

**Adding a word to a config set** — first check the *other* sets for the same word (grep `nameparser/config/` or intersect the sets in a `python3 -c`). Real overlaps exist: `do`/`st`/`mc` ∈ `PARTICLES` ∩ `TITLES`/`SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; dropping `abd` from `BOUND_GIVEN_NAMES`). Then check the word against real names, not memory: `uv run python tools/name_usage.py <word>` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. It reports and does not decide — read the counts against the position the set's claim acts on.
**Adding a word to a config set** — first check the *other* sets for the same word (grep `nameparser/config/` or intersect the sets in a `python3 -c`). Real overlaps exist: `do`/`st`/`mc` ∈ `PARTICLES` ∩ `TITLES`/`SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; dropping `abd` from `BOUND_GIVEN_NAMES`). Then check the word against real names, not memory: `uv run python tools/name_usage.py <word>` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. It reports and does not decide — read the counts against the position the set's claim acts on. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented rules.md#S2's "Latin" as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements.

**Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191).

Expand Down
10 changes: 10 additions & 0 deletions docs/design/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -819,6 +819,16 @@ Closes #606. No parser code moves: 41 words join TITLES and one, knt (Knight, be
- **gravos was dropped, not excluded.** The issue listed it as the Latin-script Greek for Count; the spot-check could not confirm it is the Greek word at all, which is κόμης (kómis). Nothing was shown, so there is nothing to keep out — it is recorded here so that a later sweep starts from that rather than from the issue's table. κόμης itself is NOT added and NOT excluded: it is a Greek surname (~420 bearers in Greece, official lists writing the surname first), which by its counts sits inside the class above beside hrabia, and it was left out of this change for having no use in current Greek address rather than for its collision. Adding it is a decision still to make.
- **Measurement (2026-10-04).** One corpus name moves, `Greve Anna`, given Greve to title Greve: the rules.md#H1 example this entry added, and the only one of its three examples that moves. Recompute: take the 41 entries of the #606 block in titles.py — the lines that are a quoted string, not the words quoted inside its comments, which name existing entries and excluded ones — parse every distinct name in the tools/differential/corpus*.jsonl glob (1,518 on the day) with the shipped parser and with `Lexicon.default().remove(titles=<those words>, suffix_acronyms={"knt"})`, and diff `as_dict()`. Before the rules.md examples were added, the same diff moved nothing over the 1,809 corpus lines of the tree before #604 merged, gravos still in the word list. The old readings the release-log bullet quotes were checked against the 1.4.0 and 2.3.0 wheels.

### generational-words — junior and senior in other languages (2026-10-14, #661)

Closes #661. 31 spelled-out generational words join SUFFIX_WORDS, and one predicate moves with them (the run-start bullet below), so `Antonio Pavlović mlađi` reads family Pavlović, suffix mlađi, where 1.4.0 and 2.4.2 read middle Pavlović, family mlađi (checked against both wheels). The class is decided once — a word meaning junior/senior (or the Younger/the Elder) written spaced AFTER the name — and each word was then checked two ways. First, collisions: a suffix claim acts on the trailing word, so the risk is a borne name there, and `tools/name_usage.py` found none (US Census, SSA, Wikidata; nuorempi's one Wikidata bearer of each kind is noise, as the issue said). Second, usage, because the issue's list came partly from ilib's machine-translated test data: each word was found in the trailing spaced position in real titles on its own language's Wikipedia (2026-10-13, e.g. "Hans Holbein mlađi", "Плиний Младший", "Pieter Brueghel vanhempi", "बलबीर सिंह सीनियर"). The per-word list, grouped by language, is the #661 block in nameparser/config/suffixes.py. Feminine forms ship beside the masculine wherever the language has them (mlađa, младшая, молодша, jaunesnioji); lt vyresnioji is the grammatical feminine with no titled bearer found, and ships on grammar. ru/uk and cs/sk were not in the issue and were added on the same evidence (Derek, 2026-10-14).

- **CLDR is corroboration, not a source.** Its person-name samples write mlađi/млађи (hr, sr — on a woman's name too), младши/младша (bg), ifj. (hu) and जूनियर (hi), but its `generation` field also holds translations of the field LABEL rather than of the word: isizukulwane (zu), जेनेरेसन (ne), نسل (ps) and jẹnereṣọn (pcm) all mean "generation". A word is not added because CLDR has it. The wider roster (fils, yngri, الابن and the rest) is open at [#679](https://github.com/derek73/python-nameparser/issues/679), to be vetted word by word the same way.
- **Excluded, each with its reason.** kıdemli (tr): a rank word (Kıdemli onbaşı); Turkish writes Jr. or a parenthesis, (oğul). sóisearach/sinsearach (ga): rank words too; Irish writes Óg and Mór, which are also inside given names (Seán Óg Ó hAilpín), so they are a separate decision and not taken here. ifj./id. (hu): written BEFORE the name (ifj. Kovács János, which H2's shape rule already reads as a title), and a bare `id` would make any trailing ID a suffix. млада (bg feminine of млади): Mlada is a borne given name, and under a family-first order the given name is the trailing word. ml./мл./st.: the abbreviations stay under the #269 deferral in suffixes.py, st. colliding with Saint besides.
- **Spaced only, and what that leaves.** bg, ru and uk write a modern Jr. hyphen-glued (Теодор Рузвелт-младши, Дональд Трамп-молодший), which no whole-token set reaches; for those three the gain is mainly the historical the Elder/the Younger names, written title case (Младший), which the lowercase fold matches.
- **The #602 run start reads the letters' script, not ASCII (Derek chose the Latin-script test, 2026-10-14).** rules.md#S2 lets "an unambiguous Latin suffix word of two or more letters" start the run, and `_pieces.starts_a_credential_run` had implemented Latin as `letters.isascii()`, exact while no accented Latin word was suffix vocabulary. This entry adds the first ones (mlađi, mlađa, mladší, starší), so without a change `Pieter Bruegel stariji Jones` would read suffix 'stariji Jones' while `Antonio Pavlović mlađi Jones` kept family Jones -- one language, one construction, split by the `đ`. The test now asks a non-ASCII word whether every letter's Unicode name is a LATIN one; the ASCII test stays first, so an ASCII word pays no frame for it. The same script test is what refuses the non-Latin honorifics (`Taro Tanaka さん Jones` keeps family Jones), and it refuses this entry's Cyrillic and Devanagari generational words with them, though the reason S2 gives for the refusal -- an honorific carries no credential mark -- is not true of them. So Serbian splits by script: `Ana Horvat stariji Jones` reads suffix 'stariji Jones' and `Ана Хорват старији Џонс` family Џонс, and after a comma C1 asks the same predicate, so `Ана Хорват, старији Џонс` reads given 'старији'. ACCEPTED for this entry (Derek, 2026-10-14) and stated at rules.md#S2's Accepted clause: the vocabulary does not mark which suffix words are honorifics, a script test is the only line available, and only a name word written after the generational word reaches it. Settling it is open at [#679](https://github.com/derek73/python-nameparser/issues/679), where the non-Latin generational words multiply. The first draft of this entry credited the ASCII test to S2's honorific exclusion instead and proposed deleting it, on the evidence that deleting it moved no test and no corpus name. The design-docs review found the attribution wrong and the evidence empty: nothing pinned the test, and deleting it reads `Taro Tanaka さん Jones` as suffix 'さん Jones' and `Гай Плиний Старший Иванов` as suffix 'Старший Иванов'. Both are rules.md#S2 boundary examples now. #678 was filed on the first draft and corrected.
- **Measurement (2026-10-14).** Six corpus names move, all in corpus_cldr.jsonl and all the CLDR hr/sr renderings carrying mlađi/млађи, each gaining suffix mlađi (or млађи) from a name field; cldr_agreement records 14 changes, every one toward CLDR's fields. The bg and hi renderings do not move because those locales' formats drop the generation field. The run-start change moves no name the corpus held before this entry; its one mover is S2's own `Antonio Pavlović mlađi Jones`. Of the 161 rows of tools/differential/cldr_fields.jsonl whose `generation` value `cldr_agreement.occurs()` in the text, none writing a non-ASCII generation word has a name word after it in the same part (two write given names after `II` across a comma, which ends the run's part). Recompute: `uv run python tools/differential/snapshot.py --since <the commit before this entry>`.

### cjk-comma-demotion — the script shapes are pure, the wrappers are tolerated (2026-09-01, #469)

Closes #469, and continues the corpus-tier arc below rather than standing apart from it: the tier split gave the differential somewhere to WATCH a name without promising it, and this is the first doctrine narrowing to spend that. No parser behavior moves anywhere in it. Counts are this session's and every one is recomputable from the checked-in tree — `wc -l` over tools/differential/, `uv run python tools/differential/build_shapes_corpus.py --coverage`, and one `uv run python tools/differential/compare.py --baseline X` run per baseline, read off its `corpora:` and `corpus:` lines.
Expand Down
17 changes: 17 additions & 0 deletions docs/design/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -1238,6 +1238,11 @@ S2. Rationale: generational suffixes and credentials are recognized
part in silence, unless paired initials speak only for each
other.
"John Smith Jr." → suffix="Jr."
"Antonio Pavlović mlađi" → family="Pavlović"
"Antonio Pavlović mlađi" → suffix="mlađi"
"Агриппина Младшая" → suffix="Младшая"
"Pavlović, Antonio mlađi" → suffix="mlađi"
"Stariji Bruegel" → given="Stariji" · boundary
"John Smith M.A." → suffix="M.A."
"John Smith PhD" → suffix="PhD"
"John Ma" → family="Ma" · boundary
Expand Down Expand Up @@ -1302,6 +1307,10 @@ S2. Rationale: generational suffixes and credentials are recognized
"Mary Jane King Smith" → family="Smith"
"Josep Carod i Rovira" → family="Carod i Rovira"
"Mohamed Ali Abd Allah" → family="Allah"
"Antonio Pavlović mlađi Jones" → suffix="mlađi Jones"
"Pieter Bruegel stariji Jones" → suffix="stariji Jones"
"Гай Плиний Старший Иванов" → family="Иванов" · boundary
"Taro Tanaka さん Jones" → family="Jones" · boundary
"John PhD Smith" → family="Smith" · boundary
Accepted: a title before the credential keeps it a credential,
and the name loses its surname exactly as it did before this
Expand Down Expand Up @@ -1353,6 +1362,14 @@ S2. Rationale: generational suffixes and credentials are recognized
and unchanged (decisions.md#v1-xfail-triage: `king` stays a
title, for the addressing forms).
"Dr Jr" → suffix="Jr"
Accepted: the non-Latin refusal is drawn by script, because the
vocabulary does not say which suffix words are honorifics, so a
generational word in another script does not start the run
either, and Serbian reads by the script it is written in. Only a
name word written after the generational word reaches the
difference (decisions.md#generational-words, open at #679).
"Ana Horvat stariji Jones" → suffix="stariji Jones"
"Ана Хорват старији Џонс" → family="Џонс" · boundary
history: decisions.md#S2 · interacts: H1, H2, H3, H5, C1, C2, S3, P1, P2, P3, P5, P6, M2 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py

S3. Rationale: credentials are often written run together with
Expand Down
4 changes: 4 additions & 0 deletions docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ Release Log

nameparser 2.5 is under development.

**Behavior Changes**

- **Add junior and senior in other languages as suffixes.** ``Antonio Pavlović mlađi`` gives last ``Pavlović``, suffix ``mlađi``, where 1.4.0 through 2.4.2 gave middle ``Pavlović``, last ``mlađi``; the words are Croatian, Serbian, Bosnian, Bulgarian, Russian, Ukrainian, Czech, Slovak, Finnish, Lithuanian and Hindi, written as a separate word after the name. :issue:`661`

* 2.4.2 - October 13, 2026

nameparser 2.4.2 fixes two readings: a name typed with decomposed accents or hangul, and the Hausa given name Shehu.
Expand Down
17 changes: 15 additions & 2 deletions nameparser/_pipeline/_pieces.py
Original file line number Diff line number Diff line change
Expand Up @@ -475,7 +475,11 @@ def starts_a_credential_run(piece: Sequence[int], ptags: Set[str],
`i` or `v` mid-name is an initial or a connective far more often
than a numeral, and carries no `initial` tag), a connective, a
word that is also bound given-name or particle vocabulary (`abd`
heads `Abd Allah`; `vd`, `mc`), and the non-Latin honorific words.
heads `Abd Allah`; `vd`, `mc`), and a word with a letter of another
script, which is what refuses the non-Latin honorifics (`Taro
Tanaka さん Jones` keeps family 'Jones'). Latin means the letters'
Unicode names, not ASCII: #661's `mlađi` starts the run as `stariji`
does (decisions.md#generational-words, #678).

The split credential group merges ('Ph. D.', the one piece carrying
the "suffix" piece tag) starts it as its one-token spelling 'PhD'
Expand All @@ -500,7 +504,16 @@ def starts_a_credential_run(piece: Sequence[int], ptags: Set[str],
return False
# C-level, no generator frame per character
letters = "".join(filter(str.isalpha, tok.text.lower()))
return len(letters) >= 2 and letters.isascii()
return len(letters) >= 2 and (letters.isascii()
or _all_latin(letters))


def _all_latin(letters: str) -> bool:
"""Whether every letter is a Latin one by its Unicode name. Asked
only of a non-ASCII word, so the ASCII test above keeps the common
case free of a frame per letter."""
return all(unicodedata.name(c, "").startswith("LATIN ")
for c in letters)


#: tags of suffix words another vocabulary claims as a name word or a
Expand Down
Loading
Loading