Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p

**A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John <word>"` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23), as were `rai`, `cha`, `ba` and `mc` before `decisions.md#suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions.md#vocabulary-collisions`, with #360's positional qualifier.

**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's "consumed as suffix" is not in it (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves.
**Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves.

**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path.

Expand Down
5 changes: 4 additions & 1 deletion docs/design/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,10 @@ Entry conventions:
MISSED, then caught in review: the first draft converted five emitters and its grid had no empty-head comma with a trailing title, the shape AGENTS.md's comma sweep already asks for; four of the review's five readers found the sixth emitter on `, Ma Dr.`. The draft also named only H1 and P6 as movers; P1's fold was found by stepping the stages, and the review of the fix found the opt-in movers O1 to O3, which no grid over the three orders reaches -- reverting the leading-particle emitter passed the whole suite until rows under those policies pinned it.
MEASURED 2026-10-08, each report's field phrase compared against its tokens' final roles. RECIPE: texts are the 1,536 names of `tools/differential/corpus*.jsonl` and 347 `tests/v2/cases.py` texts (the table at 0c54662b, the parent of #614's first commit, less the corpus names; today's table adds rows this entry's review wrote for the shapes the grid missed, so it reads more wrong claims on master, 6 more as measured on the review), plus every head in {'John', 'John Smith', 'John Q. Smith', 'Mary Ann Smith', 'John van Smith', 'Juan de la Vega', 'John van', 'anh van', 'Jan Freiherr von Berg', 'Freiherr von Berg', 'Freiherr St', 'St', 'Dr. St', 'Freiherr', 'Dr.', 'Mr. and Mrs.', 'Attorney General of', 'abdul salam', 'Juan y Garcia', 'Jane Doe nee van der Berg', 'Dr. John Smith', 'JOHN SMITH', 'john smith', 'Smith', 'de Mesnil', 'Dr. de Mesnil', 'Kim Min', 'Kim Min Jun', 'Ortega y Gasset', 'van', 'Do', ''} followed by one or two words (repetition allowed) from the list [PhD, MA, Ma, DO, Do, do, vd, mc, Jr, III, V, V., VI, X.Y.Z., Ph. D., Esq., Prof., Sir, Ed, van, and, Jones, ba, John, Smith, Minnesota, de, St, Freiherr], space-joined and stripped; 20,000 draws after Python's `random.seed(626)`, each `random.choice(heads)`, then `random.randint(3, 4)` words each `random.choice(words)`, space-joined and stripped, both lists in the order written here; then, unstripped, `{head}, {word}`, `Smith, {head} {word}` with any double space replaced by one, and `{word}, {head}` for every head and word, and `, {w1} {w2}` and `(Bob), {w1} {w2}` for every ordered pair of words; the empty text dropped and duplicates removed -- 53,517 texts; each parsed under the three orders, so the opt-in movers are outside these figures; a claim is a match of `(reads? as|joins) (<phrase>)` for the phrases `_FIELD_CLAIMS` in tests/v2/test_cases.py lists plus "the given name", and it is wrong when its tokens' roles are not among the fields the phrase names. On master (26cdb891) 2,725 of 89,092 claims were wrong, in three emitters: title-or-name's join clause (1,575), the trailing peel's picks (1,114) and the comma report (36); on the tree, 0. The comparator is master, which words the field at assign; the tree words it at assemble. Fingerprint against master over the differential corpora, master's case-table names and a seeded comma grid (13,090 names) under six policies (the three orders, `lenient_comma_suffixes=False`, a `/` suffix delimiter and `CapsSuffixes.OFF`): no role, kind, initials or rendering moves; the details that change are 6,395 comma reports on 1,086 names reworded from "the given name" to "a given name", 85 on 15 names now naming the family they land in, and 40 title-or-name join reports on 10 names now naming the family. Pinned by `test_a_report_names_the_field_its_word_lands_in` in tests/v2/test_cases.py, over every case row under all three orders except the locale rows (a row with its own policy only as declared), whose recorded control is 14 failures on master, and by `test_the_field_sweep_sees_the_claims_it_checks`, which records how many claims that sweep checks; the opt-in movers by the rows `Van Ivan Petrovich` (East Slavic), `Van Ali Veli oglu` (Turkic) and `Do Bishop Do` (FAMILY_FIRST with `middle_as_family`), which the sweep reads as declared.

Open: [#629](https://github.com/derek73/python-nameparser/issues/629) a comma-structure report on a part past the second comma says the part was "consumed as suffix" where a title in it now reads as a title (`John Smith, Jr., Freiherr von Richthofen`): the one report naming a field that the #626 entry's mechanism does not reach.
- 2026-10-10 #629 — A REPORT ON WORDS IN TWO FIELDS NAMES NEITHER. Segment's `comma-structure` report on a part past the second comma said the part was "consumed as suffix best-effort", true through 2.3.0, when the whole part read as suffix. Since #603 (rules.md#C2) a title word there reads as a title, so `John Smith, Jr., Dr. Bart` was told 'Dr.' was a suffix with 'Dr.' in the title (this cycle, never shipped).
DECIDED: the report names no field: "segment 'Dr. Bart' beyond the recognized comma structures; read best-effort". It says what is certain, that the part is outside the recognized comma structures, and the fields say where each word went. The input is malformed, so the report is not worth machinery: the fix is one string, and a part that reads wholly as suffix loses the word "suffix" from its detail, which is not contract (kinds are).
Declined: naming every field the part's words land in, worded at assemble from the final roles like #626's `field_tail` ("consumed as title and suffix best-effort"). It was built (0d5fea40, PR #630) and reviewed: it needed a second tail on `PendingAmbiguity`, an assemble branch and an equality check in the field guard, and the branch withdrew the report from a name emptied of content (`", , -"`), against rules.md#A2's "ambiguities born from its punctuation survive on the empty result", which no grid caught because every grid word had letters. More code to say more about garbage input; the user-facing gain did not pay for it.
MEASURED 2026-10-10, tree against master bc243869 over the names of `tools/differential/corpus*.jsonl` plus every `tests/v2/cases.py` text, each under the three orders, comparing every report's kind, detail and referents and the seven fields: only the detail of `comma-structure` reports moves, and no report appears or goes.

### A2 — the empty name

Expand Down
2 changes: 1 addition & 1 deletion docs/design/mechanisms.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ Problem shape. A retired default rides in on old pickles, but a user may have de

## AMBIGUITY-AT-THE-DECISION-SITE — emit where the branch is taken

Problem shape. An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary. Contract statement. Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626; a comma-structure report on a part past the second comma still names "suffix" for a part holding a title, #629). How it works. PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on
Problem shape. An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary. Contract statement. Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. The decision is the site's, but a FIELD the report names is not: a later rule can still move the word, so the emitter leaves the field to assemble (`PendingAmbiguity.field_tail`), which words it from the final role (#626). Segment's report on a part past the second comma names no field at all, since that part's words can land in two (#629). How it works. PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on
"the code got here" instead of "the outcome differed" once reported
a fork for all ambiguous particles on "Dr. Van Jr.". Lives in. The AmbiguityKind emitters across _pipeline/ (rule A1 is the observable contract). Reach for it when. Adding any ambiguous vocabulary or any new fork — count the deciding sites, then count the emitters.

Expand Down
Loading
Loading