diff --git a/AGENTS.md b/AGENTS.md index 9ceda453..256b66d9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -173,7 +173,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. -**And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured, `Smith, José` is 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. Not covered, deliberately: `_INITIAL` and the `_render` patterns (`\w\.`, v1-verbatim; a lone initial parses alike, but `É. y É. Smith` does not) and the decomposed conjunction `й` (decisions.md#nfd-shapes lists both as open, #665) and the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules.md#H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of the new test's rows pass with every shape regex reverted (measured) and are parity pins; `_DOTTED` has no observable input and is pinned by a direct call. +**And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules.md#H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. ### Configuration layer (`nameparser/config/`) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 64ffdd9e..55bfcd51 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -383,6 +383,7 @@ The reconciled v1-style banks (`tests/test_*.py`) carried eight `@pytest.mark.xf - 2026-10-12 (#652) — a shape regex spelled a letter `[^\W\d_]`, and `\w` does not match a combining mark, so a name typed in NFD (`inž.` as i, n, z, U+030C) ended its word at the z: `Dr. inž. Janez Novak` made `inž.` the given name, and `Janez Novak, ŽŠČ` reversed because `str.isalpha()` refuses the caps shape. Vocabulary was never affected (`_normalize` NFC-folds); only the shape path was. The bug is as old as 1.4.0, whose `period_abbreviation` had the same class, so this is a fix and not a restoration. DECIDED: a letter in the four letter-shape regexes is `_vocab._LETTER` (letter plus the general combining blocks), `config`'s `period_abbreviation` carries the same class, and the caps predicates NFC-compose a word that is not already NFC before asking `isalpha()`, and the two-letter pair test counts letters rather than code points (`composed_len`; the first version of the fix split `John Smith, MÁ` the other way, found in review). The tests as written (`str.isalpha()`, `unicodedata.is_normalized`) are C-level, so a word already composed pays no frame: `Smith, José` 193 and `Janez Novak, Žiga` 237 frames before and after, default and EVERYWHERE (measured 2026-10-12, py3.11). Output keeps the encoding written; only the reading is made encoding-independent. MEASURED MOVES the release log cannot see, against the released 2.4.0: NFD `Janez Novak, ŽŠČ` went from given `ŽŠČ` (reversed) to suffix `ŽŠČ`, and NFD `Janez Novak, Ž.Š.` went from suffix `Ž.Š.` to given `Ž.Š.`, family `Janez Novak` — the composed twin's reading, which is C1's (paired initials are a name), so the second moves AWAY from the suffix reading and toward parity. Neither is in a corpus. - DECLINED to widen the class to every script's marks: Indic vowel signs and Arabic and Hebrew points are marks too, but rules.md#H2's Accepted clause already records that an abugida abbreviation never reads as a title by shape, and a general `\p{M}` would change that without a measurement. KNOWN LIMITS, left open (#665): `_INITIAL` / `_render._INITIAL` / `config` `initial` (`\w\.`) are v1-verbatim and public, and an NFD single initial (`É.`) is not initial-shaped there. No role or `initials()` result differs for a lone initial (`Janez Ž. Novak`, 17 shapes × 3 orders), but one does beside a connective: NFC `É. y É. Smith` reads given `É.`, middle `y É.`, NFD reads given `É. y É.`. The same width problem makes the decomposed Cyrillic conjunction `й` (two code points) miss the single-letter tests at `_classify.py` (`len(token.text) == 1`) and `_pieces.py` (`len(text) == 1`): NFD `John й Smith` reads family `John й Smith`. Both predate #652 and read identically on 2.4.0 and master; neither is in a corpus. +- 2026-10-13 #665 — DONE, closing both KNOWN LIMITS above. The initial shape is asked of the composed spelling: `_vocab.is_initial_shaped` NFC-composes a non-ASCII word before `_INITIAL`, as `_render` already did for its two copies (#542). Composing was chosen over widening the class because the pattern is the public v1 `initial` regex, pinned verbatim by `test_regex_sync`, and because it gives parity BY CONSTRUCTION — a decomposed word reads as the word it composes to, and a letter-plus-mark with no precomposed form has no composed twin to disagree with, so it is left as the pattern reads it — and likewise the width tests count composed code points, so a caller's one-letter connective with no precomposed form (`q́`, added to `conjunctions`) is two letters to them and joins as a longer connective does (`John q́ Smith` reads given `John q́ Smith`), as rules.md#P3's "composes to one character" states. The three single-letter width tests (`_classify`'s connective fork and its one-case report, `_pieces`' three-word carve-out) count letters: the classify pair asks `composed_len` only for non-ASCII text that the vocabulary membership has already admitted and the C-level `len(...) == 1` has declined, and `_pieces` composes in C before `isalpha()`. ASCII text pays no frame and the width tests cost composed text nothing (`Dr. Juan Q. Xavier de la Vega III` 366, `Smith, José` 194, `Juan y Eva Garcia` 254, `John й Smith` 255 composed, all equal before and after); the veto below costs one frame per non-ASCII word that reaches it, its matcher being called through a wrapper (`É. y É. Smith` 305 → 307, `Janez Ž. Novak` 237 → 238, `田中. 太郎` 207 → 208; measured 2026-10-13, py3.11, all composed). Kept rather than composing inline at its four call sites: a rare shape no guard counts, against four copies of the one step whose omission this entry is about (AGENTS.md's frame rule, question 4). Decomposed `John й Smith` reads given John, middle й, family Smith, where 2.3.0 through 2.4.1 read the whole name as the given name and reported given-or-family (the KNOWN LIMITS bullet above says family; measured on 2.4.0, 2.4.1 and f9b2b81c it was given, and this bullet corrects it) — a regression of 2.3.0, whose NFC lookup fold (#cjk-full-stops) let the decomposed й match the conjunction vocabulary while the width tests still counted two code points; 1.4.0 through 2.2.0 never matched it and read it as middle й, so this restores their reading. Decomposed `É. y É. Smith` reads given É., middle `y É.`, where every release from 1.4.0 read given `É. y É.`, so that half is a fix. Both checked on the 1.4.0, 2.0.0, 2.2.0, 2.3.0, 2.4.0 and 2.4.1 wheels (2026-10-13). The #320 veto beside the shape is asked of the composed spelling too (`_vocab.in_initialless_script`, ASCII answering before the matcher): decomposed hangul is conjoining jamo, which no `_NO_INITIALS` range holds, so composing the shape alone made a decomposed `씨.` an initial — `김민준, 씨.` read title `씨.` — which the first version of this fix did, caught by `test_suffix_lookup_reads_every_full_stop`. The same veto read on jamo had long let a decomposed hangul word with a period take the leading-abbreviation shape (#H2): `김. 민준` read title `김.` decomposed in every release from 1.4.0 (checked on the 1.4.0, 2.0.0, 2.2.0, 2.3.0 and 2.4.0 wheels), and reads family `김.`, given `민준` as composed now; rules.md#H2's veto clause states it and carries the decomposed example. An UNSPACED decomposed hangul name is still not split (docs/usage.rst's documented exception), so `김민준. 이서연` keeps differing by encoding. The guard is `tests/v2/test_parser.py`'s twin test, which now compares ambiguity kinds and `initials()` as well as roles; its docstring records which rows each of the four sites holds. ### render-default — the default view's format diff --git a/docs/design/rules.md b/docs/design/rules.md index 0afa80ca..d886558b 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -138,8 +138,12 @@ H2. Rationale: before a name, an abbreviation is almost always a title by shape (#323; decisions.md#cjk-full-stops) — the same veto that keeps 씨. from reading as an initial. The Latin reading is unchanged, and W3's example block carries the CJK - reading this clause vetoes into. + reading this clause vetoes into. The veto reads the composed + spelling, so hangul typed decomposed, as conjoining jamo the + script table does not hold, is hangul to it too (#665); the + second line below is typed that way. "Smith. John" → title="Smith." + "김. 민준" → title="" Accepted: before a family comma the pre-comma text is wholly the family name (C1), so no shape or vocabulary reading makes a title there. @@ -611,6 +615,8 @@ P3. Rationale: connective words ("y", "of the") bind name words into "Carod i" → suffix="i" "Josep Lluis Carod i III" → suffix="i III" "Carod y de Rovira i" → middle="y" + "John й Smith" → middle="й" + "É. y É. Smith" → given="É." Both exceptions are about the written FORM, not the word: the three-word carve-out counts letters, so a symbol connective joins at any length, and it reaches every single-letter connective the @@ -622,7 +628,11 @@ P3. Rationale: connective words ("y", "of the") bind name words into Cyrillic abbreviates with a dotted letter — so in a name of more than one case it joins, the reading #267 blessed, and in a name of one case it takes the vocabulary's answer like any other cased - letter. + letter. A letter written as a base letter plus a combining mark + (NFD) that composes to one character is one letter to every + count here, and a period-marked initial written that way is one + initial (#665), so the last two examples above, both typed + decomposed, read as their composed twins. Which single letters a tradition actually wants joined differs by language, and the marked set is where that answer lives: "y" is the commonest Hispanic compound and stays out of it, while "e" and diff --git a/docs/release_log.rst b/docs/release_log.rst index 0165857b..ad77633e 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -6,6 +6,7 @@ Release Log **Behavior Changes** + - **Fix a name typed with decomposed (NFD) accents or hangul reading differently from its composed spelling.** ``John й Smith`` typed with ``й`` as ``и`` plus a combining breve gives middle ``й``, where 2.3.0 through 2.4.1 read the whole name as the first name, restoring 1.4.0's reading. :issue:`665` - **Fix the Hausa given name Shehu being read as a title.** ``Shehu Shagari`` gives first ``Shehu``, last ``Shagari``, where 1.4.0 through 2.4.1 gave title ``Shehu`` and no first name. :issue:`660` * 2.4.1 - October 13, 2026 diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 6f1115b2..b104497e 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -62,8 +62,8 @@ ) from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( - caps_shape_candidate, is_initial, is_one_case, period_joined_vocab, - suffix_as_written, tag_marker_runs, + caps_shape_candidate, composed_len, is_initial, is_one_case, + period_joined_vocab, suffix_as_written, tag_marker_runs, ) from nameparser._pipeline._pieces import own_words @@ -134,9 +134,14 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # written wholly in one case, where nothing says so — where the # letter is one the vocabulary marks as reading both ways" # (#383/#479; history: decisions.md#P3) - single_letter_connective = (len(token.text) == 1 + # one letter as the writer sees it: a decomposed 'й' is two code + # points (#665), so non-ASCII text is counted composed, after the + # C-level tests have declined everything else + single_letter_connective = (n in lex.conjunctions and token.text.upper() != token.text.lower() - and n in lex.conjunctions) + and (len(token.text) == 1 + or not token.text.isascii() + and composed_len(token.text) == 1)) if single_letter_connective and one_case_own: # No case evidence, so the vocabulary decides. No namespaced # tag beside it: the emitted ambiguity IS the record of the @@ -334,10 +339,12 @@ def classify(state: ParseState) -> ParseState: # site (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE), per # token -- 'e and e' reports twice. if (one_case and "initial" in token.tags - and len(token.text) == 1 and folded[i] in state.lexicon.conjunctions_ambiguous and folded[i] in state.lexicon.conjunctions - and i < clause_at and token.role is None): + and i < clause_at and token.role is None + and (len(token.text) == 1 + or not token.text.isascii() + and composed_len(token.text) == 1)): ambiguities.append(PendingAmbiguity( AmbiguityKind.CONJUNCTION_OR_INITIAL, f"{token.text!r} is both a connective and an initial; " diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index cd56c346..fabca60e 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -52,6 +52,7 @@ """ from __future__ import annotations +import unicodedata from collections.abc import ( Callable, Container, Iterable, Mapping, Sequence, Set) from typing import NamedTuple @@ -314,6 +315,10 @@ def join_connectives(pieces: list[list[int]], ptags: list[set[str]], continue if letter_stays and len(pieces[k]) == 1: text = tokens[pieces[k][0]].text + if not text.isascii(): + # a decomposed 'й' is one letter in two code points + # (#665); compose it, as the lookup does + text = unicodedata.normalize("NFC", text) if len(text) == 1 and text.isalpha(): k += 1 continue diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index e8521f9d..b20c1551 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -143,7 +143,18 @@ def _drop_kana_voicing_marks(text: str) -> str: # vetoes the whole word -- 'Kim김.' is refused as a title along with # '田中.' -- and that is deliberate: a word carrying a script with no # abbreviations is not wearing an abbreviation's period. -in_initialless_script = _script_matcher(*_NO_INITIALS, whole=False) +_has_initialless_char = _script_matcher(*_NO_INITIALS, whole=False) + + +def in_initialless_script(text: str) -> bool: + """Whether TEXT holds a character of a script that writes no + initials (#320), asked of the composed spelling: a decomposed + hangul syllable is conjoining jamo, which no _NO_INITIALS range + holds, so '씨.' typed decomposed escaped the veto every shape test + beside it reads composed (#665). ASCII answers without the + matcher's frame, as the callers' own `isascii()` gates do.""" + return (not text.isascii() + and _has_initialless_char(unicodedata.normalize("NFC", text))) # H2's own shape, text-level: an unlisted period-marked abbreviation, @@ -224,7 +235,15 @@ def is_initial_shaped(text: str) -> bool: name want is_initial (#320). The two answers differ only inside _NO_INITIALS scripts, where '씨.' is initial-SHAPED but is not an initial -- see assign's roman-numeral fork, the shape caller, for - what picking the wrong one costs.""" + what picking the wrong one costs. + + Asked of the composed spelling, as `_render` asks it (#542): `\\w` + does not match a combining mark, so a decomposed 'É.' is three + characters to the pattern and no initial (#665). Composing rather + than widening keeps the pattern v1-verbatim, and both calls are + C-level, so the predicate costs no frame it did not before.""" + if not text.isascii(): + text = unicodedata.normalize("NFC", text) return bool(_INITIAL.fullmatch(text)) diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 8195a4b3..a84f0851 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1962,6 +1962,34 @@ def test_the_catalan_recipe_unmarks_the_link_for_one_case_names() -> None: "Janez Novak, Ž.Š.Č.", "Janez Novak, Žž.Šš.", "Smith, Ž.Š. Jones", + # #665: a period-marked initial (`\w\.`) beside a connective, in + # each slot and comma shape -- a lone one read alike before the fix + "É. y É. Smith", + "Й. John Smith", + "Ana É. y Luis", + "Smith, É. y É.", + ", É. y É.", + "(Bob), É. y É.", + # ... and the one-letter connective written in two code points: + # three words (P3's carve-out keeps it a name word), four words + # (it joins), one case, and after a comma + "John й Smith", + "JOHN Й SMITH", + "john й smith", + "Ivan й Petrov Sidorov", + "Smith, John й Jones", + # ... and the veto on scripts that write no initials (#320), asked + # of the composed spelling too: decomposed hangul is conjoining + # jamo, outside every _NO_INITIALS range, so composing the initial + # shape alone made a decomposed '씨.' an initial (a title or a + # given name). Spaced or after a comma, since an UNSPACED + # decomposed hangul name is not split (docs/usage.rst) and so has + # no composed twin to match + "김민준, 씨.", + "김 민준 씨.", + "김 민준, 씨.", + ", 씨.", + "김. 민준", ] _NFD_POLICIES = [ Policy(), @@ -1976,6 +2004,15 @@ def _composed_roles(parser: Parser, text: str) -> dict[str, str]: for k, v in parser.parse(text).as_dict().items()} +def _composed_reading(parser: Parser, text: str) -> tuple[object, ...]: + """The roles, the reports' kinds and initials(), composed: #665's + `John й Smith` differed in its report as well as its fields.""" + name = parser.parse(text) + return (_composed_roles(parser, text), + [a.kind for a in name.ambiguities], + unicodedata.normalize("NFC", name.initials())) + + @pytest.mark.parametrize("policy", _NFD_POLICIES, ids=["default", "family-first", "family-first-given-last", "caps-everywhere"]) @@ -1996,13 +2033,43 @@ def test_a_decomposed_abbreviation_reads_as_its_composed_twin( two-letter rows pass there; the two-letter rows exist for the caps fix's own length count, which counted code points the first time (`John Smith, MÁ` read as a pair composed and a credential - decomposed, found in the review of 08c0dd60).""" + decomposed, found in the review of 08c0dd60). + + #665 added sixteen rows and compares kinds and initials() too. + Recorded control, measured 2026-10-13 against f9b2b81c's + nameparser/: 38 rows x 4 policies = 152 twin cases plus the four + direct tests, of which 41 fail -- the six initial rows under all + four policies (24; the same 24 with only `is_initial_shaped`'s + compose reverted), the three-word `й` rows (12; `_pieces`' + carve-out alone), `김. 민준` (4), and the one-case report test + below (1; either `_classify` width site alone). The other four + hangul rows pass there: they guard the fix itself, failing (with + `김. 민준`, 20 in all) when `in_initialless_script`'s compose + alone is reverted, since the composed initial shape without the + composed veto made a decomposed '씨.' an initial. `Ivan й Petrov + Sidorov` and `Smith, John й Jones` pass everywhere and pin parity + only.""" composed = unicodedata.normalize("NFC", text) decomposed = unicodedata.normalize("NFD", text) assert decomposed != composed parser = Parser(policy=policy) - assert (_composed_roles(parser, decomposed) - == _composed_roles(parser, composed)) + assert (_composed_reading(parser, decomposed) + == _composed_reading(parser, composed)) + + +def test_a_decomposed_conjunction_reports_as_its_composed_twin() -> None: + """#665: the one-case report on a letter that is both a connective + and an initial (rules.md#P3) counted code points, so it is reached + by `й` only once a caller marks it ambiguous -- no default entry is + decomposable.""" + parser = Parser(lexicon=Lexicon.default().add( + conjunctions_ambiguous={"й"})) + text = "ivan й petrov" + composed = parser.parse(text) + assert [a.kind for a in composed.ambiguities] == [ + AmbiguityKind.CONJUNCTION_OR_INITIAL] + assert (_composed_reading(parser, unicodedata.normalize("NFD", text)) + == _composed_reading(parser, text)) def test_a_decomposed_caps_credential_is_still_a_suffix() -> None: diff --git a/tests/v2/test_rules_doc.py b/tests/v2/test_rules_doc.py index 25828d07..bec234b9 100644 --- a/tests/v2/test_rules_doc.py +++ b/tests/v2/test_rules_doc.py @@ -145,12 +145,22 @@ def test_example(rule: Rule, example: Example) -> None: assert actual == example.value -def test_the_h2_nfd_example_is_still_decomposed() -> None: - """#652: rules.md#H2's NFD example is the only executable witness - of the encoding clause, and an editor or formatter that - NFC-normalizes the file turns it into the composed case, which - always worked and still passes. Its bytes are z + U+030C.""" +@pytest.mark.parametrize("decomposed", [ + "inž.", # H2's abbreviation shape (#652): z + U+030C + "John й Smith", # P3's one-letter connective (#665) + "É. y É. Smith", # P3's initial beside a connective (#665) + "김. 민준", # H2's script veto on hangul typed as jamo (#665) +]) +def test_the_nfd_examples_are_still_decomposed(decomposed: str) -> None: + """The decomposed example lines in rules.md are the only executable + witnesses of its encoding clauses (H2, P3), and an editor or + formatter that NFC-normalizes the file turns each into the + composed case, which always worked and still passes. Each entry + here is typed decomposed, which the first assert checks so that + normalizing this file cannot hollow the test out either.""" + import unicodedata from pathlib import Path + assert not unicodedata.is_normalized("NFC", decomposed) text = (Path(__file__).parents[2] / "docs" / "design" / "rules.md" ).read_text(encoding="utf-8") - assert "inž." in text, "H2's NFD example was recomposed" + assert decomposed in text, f"the NFD example {decomposed!r} was recomposed" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index c21fe8ac..f7fea4ef 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -87,6 +87,7 @@ "Eric H. Holder, Jr., Attorney General" "Eric H. Holder, Jr., Secretary of State" "Esq. Smith" +"É. y É. Smith" "Freiherr St John Smith" "Freiherr von Berg MA" "Freiherr von Berg, Ed" @@ -265,6 +266,7 @@ "John van der Berg née Jones" "John van der J. V" "John von PhD Jones" +"John й Smith" "John ｢Jack｣ Smith" "John😀Smith" "Jon Dough and" @@ -517,6 +519,7 @@ "Хосе И Мария Сантос" "سلمان، محمد" "প্রকৌশলী. Sen" +"김. 민준" "ア゙イ タロウ" "マイケル ジャクソン" "マイケル ジャクソン" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 1840ff9a..0e9f3c66 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -514,6 +514,7 @@ {"name": "Excm. Sr. Lloret Palol II, Josep Antoni Carles Joan", "order": "FAMILY_FIRST", "given": "Josep", "middle": "Antoni Carles Joan", "family": "Excm. Lloret Palol", "suffix": "Sr., II", "initials": "J. A. C. J. E. L. P."} {"name": "Excma. Dra. Maria da Graça César Martín van Dalen Hoogeveen", "title": "Excma. Dra.", "given": "Maria", "middle": "da Graça César Martín", "family": "van Dalen Hoogeveen", "initials": "M. G. C. M. D. H."} {"name": "Excma. Dra. Van Dalen Hoogeveen II, Maria da Graça César Martín", "order": "FAMILY_FIRST", "given": "Maria", "middle": "da Graça César Martín", "family": "Excma. Dra. Van Dalen Hoogeveen", "suffix": "II", "initials": "M. G. C. M. E. D. D. H."} +{"name": "É. y É. Smith", "given": "É.", "middle": "y É.", "family": "Smith", "initials": "É. É. S."} {"name": "Fabbri Irene", "order": "FAMILY_FIRST", "given": "Irene", "family": "Fabbri", "initials": "I. F."} {"name": "Fabbri, Irene", "given": "Irene", "family": "Fabbri", "initials": "I. F."} {"name": "Falke, Iris", "given": "Iris", "family": "Falke", "initials": "I. F."} @@ -1031,6 +1032,7 @@ {"name": "John van der J. V", "given": "John", "family": "van der J. V", "initials": "J. J. V."} {"name": "John von PhD Jones", "given": "John", "family": "von", "suffix": "PhD Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. v."} {"name": "John y Jane", "given": "John", "middle": "y", "family": "Jane", "initials": "J. y. J."} +{"name": "John й Smith", "given": "John", "middle": "й", "family": "Smith", "initials": "J. й. S."} {"name": "John “Jack” Kennedy", "given": "John", "family": "Kennedy", "nickname": "Jack", "initials": "J. K."} {"name": "John 王", "given": "John", "family": "王", "initials": "J. 王."} {"name": "John 田中さん, XYZ", "given": "John", "family": "田中", "suffix": "さん, XYZ", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. 田."} @@ -3099,6 +3101,7 @@ {"name": "ფონ ბრიული გონსალეს დომინგო, პროფ. დოქტ. ადა კორნელია სეზარ მარტინ მეცნ. დოქტ.", "title": "პროფ. დოქტ.", "given": "ადა", "middle": "კორნელია სეზარ მარტინ მეცნ. დოქტ.", "family": "ფონ ბრიული გონსალეს დომინგო", "initials": "ა. კ. ს. მ. მ. დ. ფ. ბ. გ. დ."} {"name": "შტიობერი ცეცილია ჰეიმიშ", "order": "FAMILY_FIRST", "given": "ცეცილია", "middle": "ჰეიმიშ", "family": "შტიობერი", "initials": "ც. ჰ. შ."} {"name": "ცეცილია ჰეიმიშ შტიობერი", "given": "ცეცილია", "middle": "ჰეიმიშ", "family": "შტიობერი", "initials": "ც. ჰ. შ."} +{"name": "김. 민준", "given": "민준", "family": "김.", "initials": "민. 김."} {"name": "ሙለር ካቲ", "order": "FAMILY_FIRST", "given": "ካቲ", "family": "ሙለር", "initials": "ካ. ሙ."} {"name": "ሙለር፣ ካቲ", "order": "FAMILY_FIRST", "given": "ካቲ", "family": "ሙለር፣", "initials": "ካ. ሙ."} {"name": "ሜሪ ሱ ሃሚሽ ዋትሰን", "given": "ሜሪ", "middle": "ሱ ሃሚሽ", "family": "ዋትሰን", "initials": "ሜ. ሱ. ሃ. ዋ."}