Read a decomposed initial, connective and CJK word as its composed twin

AGENTS.md

@@ -173,7 +173,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p
 
 **Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator.
 
-**And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured, `Smith, José` is 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. Not covered, deliberately: `_INITIAL` and the `_render` patterns (`\w\.`, v1-verbatim; a lone initial parses alike, but `É. y É. Smith` does not) and the decomposed conjunction `й` (decisions.md#nfd-shapes lists both as open, #665) and the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules.md#H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of the new test's rows pass with every shape regex reverted (measured) and are parity pins; `_DOTTED` has no observable input and is pinned by a direct call.
+**And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules.md#H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call.
 
 ### Configuration layer (`nameparser/config/`)
 

docs/design/decisions.md

@@ -383,6 +383,7 @@ The reconciled v1-style banks (`tests/test_*.py`) carried eight `@pytest.mark.xf
 
 - 2026-10-12 (#652) — a shape regex spelled a letter `[^\W\d_]`, and `\w` does not match a combining mark, so a name typed in NFD (`inž.` as i, n, z, U+030C) ended its word at the z: `Dr. inž. Janez Novak` made `inž.` the given name, and `Janez Novak, ŽŠČ` reversed because `str.isalpha()` refuses the caps shape. Vocabulary was never affected (`_normalize` NFC-folds); only the shape path was. The bug is as old as 1.4.0, whose `period_abbreviation` had the same class, so this is a fix and not a restoration. DECIDED: a letter in the four letter-shape regexes is `_vocab._LETTER` (letter plus the general combining blocks), `config`'s `period_abbreviation` carries the same class, and the caps predicates NFC-compose a word that is not already NFC before asking `isalpha()`, and the two-letter pair test counts letters rather than code points (`composed_len`; the first version of the fix split `John Smith, MÁ` the other way, found in review). The tests as written (`str.isalpha()`, `unicodedata.is_normalized`) are C-level, so a word already composed pays no frame: `Smith, José` 193 and `Janez Novak, Žiga` 237 frames before and after, default and EVERYWHERE (measured 2026-10-12, py3.11). Output keeps the encoding written; only the reading is made encoding-independent. MEASURED MOVES the release log cannot see, against the released 2.4.0: NFD `Janez Novak, ŽŠČ` went from given `ŽŠČ` (reversed) to suffix `ŽŠČ`, and NFD `Janez Novak, Ž.Š.` went from suffix `Ž.Š.` to given `Ž.Š.`, family `Janez Novak` — the composed twin's reading, which is C1's (paired initials are a name), so the second moves AWAY from the suffix reading and toward parity. Neither is in a corpus.
 - DECLINED to widen the class to every script's marks: Indic vowel signs and Arabic and Hebrew points are marks too, but rules.md#H2's Accepted clause already records that an abugida abbreviation never reads as a title by shape, and a general `\p{M}` would change that without a measurement. KNOWN LIMITS, left open (#665): `_INITIAL` / `_render._INITIAL` / `config` `initial` (`\w\.`) are v1-verbatim and public, and an NFD single initial (`É.`) is not initial-shaped there. No role or `initials()` result differs for a lone initial (`Janez Ž. Novak`, 17 shapes × 3 orders), but one does beside a connective: NFC `É. y É. Smith` reads given `É.`, middle `y É.`, NFD reads given `É. y É.`. The same width problem makes the decomposed Cyrillic conjunction `й` (two code points) miss the single-letter tests at `_classify.py` (`len(token.text) == 1`) and `_pieces.py` (`len(text) == 1`): NFD `John й Smith` reads family `John й Smith`. Both predate #652 and read identically on 2.4.0 and master; neither is in a corpus.
+- 2026-10-13 #665 — DONE, closing both KNOWN LIMITS above. The initial shape is asked of the composed spelling: `_vocab.is_initial_shaped` NFC-composes a non-ASCII word before `_INITIAL`, as `_render` already did for its two copies (#542). Composing was chosen over widening the class because the pattern is the public v1 `initial` regex, pinned verbatim by `test_regex_sync`, and because it gives parity BY CONSTRUCTION — a decomposed word reads as the word it composes to, and a letter-plus-mark with no precomposed form has no composed twin to disagree with, so it is left as the pattern reads it — and likewise the width tests count composed code points, so a caller's one-letter connective with no precomposed form (`q́`, added to `conjunctions`) is two letters to them and joins as a longer connective does (`John q́ Smith` reads given `John q́ Smith`), as rules.md#P3's "composes to one character" states. The three single-letter width tests (`_classify`'s connective fork and its one-case report, `_pieces`' three-word carve-out) count letters: the classify pair asks `composed_len` only for non-ASCII text that the vocabulary membership has already admitted and the C-level `len(...) == 1` has declined, and `_pieces` composes in C before `isalpha()`. ASCII text pays no frame and the width tests cost composed text nothing (`Dr. Juan Q. Xavier de la Vega III` 366, `Smith, José` 194, `Juan y Eva Garcia` 254, `John й Smith` 255 composed, all equal before and after); the veto below costs one frame per non-ASCII word that reaches it, its matcher being called through a wrapper (`É. y É. Smith` 305 → 307, `Janez Ž. Novak` 237 → 238, `田中. 太郎` 207 → 208; measured 2026-10-13, py3.11, all composed). Kept rather than composing inline at its four call sites: a rare shape no guard counts, against four copies of the one step whose omission this entry is about (AGENTS.md's frame rule, question 4). Decomposed `John й Smith` reads given John, middle й, family Smith, where 2.3.0 through 2.4.1 read the whole name as the given name and reported given-or-family (the KNOWN LIMITS bullet above says family; measured on 2.4.0, 2.4.1 and f9b2b81c it was given, and this bullet corrects it) — a regression of 2.3.0, whose NFC lookup fold (#cjk-full-stops) let the decomposed й match the conjunction vocabulary while the width tests still counted two code points; 1.4.0 through 2.2.0 never matched it and read it as middle й, so this restores their reading. Decomposed `É. y É. Smith` reads given É., middle `y É.`, where every release from 1.4.0 read given `É. y É.`, so that half is a fix. Both checked on the 1.4.0, 2.0.0, 2.2.0, 2.3.0, 2.4.0 and 2.4.1 wheels (2026-10-13). The #320 veto beside the shape is asked of the composed spelling too (`_vocab.in_initialless_script`, ASCII answering before the matcher): decomposed hangul is conjoining jamo, which no `_NO_INITIALS` range holds, so composing the shape alone made a decomposed `씨.` an initial — `김민준, 씨.` read title `씨.` — which the first version of this fix did, caught by `test_suffix_lookup_reads_every_full_stop`. The same veto read on jamo had long let a decomposed hangul word with a period take the leading-abbreviation shape (#H2): `김. 민준` read title `김.` decomposed in every release from 1.4.0 (checked on the 1.4.0, 2.0.0, 2.2.0, 2.3.0 and 2.4.0 wheels), and reads family `김.`, given `민준` as composed now; rules.md#H2's veto clause states it and carries the decomposed example. An UNSPACED decomposed hangul name is still not split (docs/usage.rst's documented exception), so `김민준. 이서연` keeps differing by encoding. The guard is `tests/v2/test_parser.py`'s twin test, which now compares ambiguity kinds and `initials()` as well as roles; its docstring records which rows each of the four sites holds.
 
 ### render-default — the default view's format
 

docs/design/rules.md

@@ -138,8 +138,12 @@ H2. Rationale: before a name, an abbreviation is almost always a
     title by shape (#323; decisions.md#cjk-full-stops) — the same
     veto that keeps 씨. from reading as an initial. The Latin
     reading is unchanged, and W3's example block carries the CJK
-    reading this clause vetoes into.
+    reading this clause vetoes into. The veto reads the composed
+    spelling, so hangul typed decomposed, as conjoining jamo the
+    script table does not hold, is hangul to it too (#665); the
+    second line below is typed that way.
       "Smith. John"               →  title="Smith."
+      "김. 민준"                    →  title=""
     Accepted: before a family comma the pre-comma text is wholly the
     family name (C1), so no shape or vocabulary reading makes a
     title there.
@@ -611,6 +615,8 @@ P3. Rationale: connective words ("y", "of the") bind name words into
       "Carod i"                   →  suffix="i"
       "Josep Lluis Carod i III"   →  suffix="i III"
       "Carod y de Rovira i"       →  middle="y"
+      "John й Smith"              →  middle="й"
+      "É. y É. Smith"             →  given="É."
     Both exceptions are about the written FORM, not the word: the
     three-word carve-out counts letters, so a symbol connective joins
     at any length, and it reaches every single-letter connective the
@@ -622,7 +628,11 @@ P3. Rationale: connective words ("y", "of the") bind name words into
     Cyrillic abbreviates with a dotted letter — so in a name of more
     than one case it joins, the reading #267 blessed, and in a name
     of one case it takes the vocabulary's answer like any other cased
-    letter.
+    letter. A letter written as a base letter plus a combining mark
+    (NFD) that composes to one character is one letter to every
+    count here, and a period-marked initial written that way is one
+    initial (#665), so the last two examples above, both typed
+    decomposed, read as their composed twins.
     Which single letters a tradition actually wants joined differs by
     language, and the marked set is where that answer lives: "y" is
     the commonest Hispanic compound and stays out of it, while "e" and

docs/release_log.rst

@@ -6,6 +6,7 @@ Release Log
 
     **Behavior Changes**
 
+    - **Fix a name typed with decomposed (NFD) accents or hangul reading differently from its composed spelling.** ``John й Smith`` typed with ``й`` as ``и`` plus a combining breve gives middle ``й``, where 2.3.0 through 2.4.1 read the whole name as the first name, restoring 1.4.0's reading. :issue:`665`
     - **Fix the Hausa given name Shehu being read as a title.** ``Shehu Shagari`` gives first ``Shehu``, last ``Shagari``, where 1.4.0 through 2.4.1 gave title ``Shehu`` and no first name. :issue:`660`
 
 * 2.4.1 - October 13, 2026

nameparser/_pipeline/_classify.py

@@ -62,8 +62,8 @@
 )
 from nameparser._types import AmbiguityKind, Role
 from nameparser._pipeline._vocab import (
-    caps_shape_candidate, is_initial, is_one_case, period_joined_vocab,
-    suffix_as_written, tag_marker_runs,
+    caps_shape_candidate, composed_len, is_initial, is_one_case,
+    period_joined_vocab, suffix_as_written, tag_marker_runs,
 )
 from nameparser._pipeline._pieces import own_words
 
@@ -134,9 +134,14 @@ def _tags_for(token: WorkToken, n: str, state: ParseState,
     # written wholly in one case, where nothing says so — where the
     # letter is one the vocabulary marks as reading both ways"
     # (#383/#479; history: decisions.md#P3)
-    single_letter_connective = (len(token.text) == 1
+    # one letter as the writer sees it: a decomposed 'й' is two code
+    # points (#665), so non-ASCII text is counted composed, after the
+    # C-level tests have declined everything else
+    single_letter_connective = (n in lex.conjunctions
                                 and token.text.upper() != token.text.lower()
-                                and n in lex.conjunctions)
+                                and (len(token.text) == 1
+                                     or not token.text.isascii()
+                                     and composed_len(token.text) == 1))
     if single_letter_connective and one_case_own:
         # No case evidence, so the vocabulary decides. No namespaced
         # tag beside it: the emitted ambiguity IS the record of the
@@ -334,10 +339,12 @@ def classify(state: ParseState) -> ParseState:
         # site (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE), per
         # token -- 'e and e' reports twice.
         if (one_case and "initial" in token.tags
-                and len(token.text) == 1
                 and folded[i] in state.lexicon.conjunctions_ambiguous
                 and folded[i] in state.lexicon.conjunctions
-                and i < clause_at and token.role is None):
+                and i < clause_at and token.role is None
+                and (len(token.text) == 1
+                     or not token.text.isascii()
+                     and composed_len(token.text) == 1)):
             ambiguities.append(PendingAmbiguity(
                 AmbiguityKind.CONJUNCTION_OR_INITIAL,
                 f"{token.text!r} is both a connective and an initial; "

nameparser/_pipeline/_pieces.py

@@ -52,6 +52,7 @@
 """
 from __future__ import annotations
 
+import unicodedata
 from collections.abc import (
     Callable, Container, Iterable, Mapping, Sequence, Set)
 from typing import NamedTuple
@@ -314,6 +315,10 @@ def join_connectives(pieces: list[list[int]], ptags: list[set[str]],
             continue
         if letter_stays and len(pieces[k]) == 1:
             text = tokens[pieces[k][0]].text
+            if not text.isascii():
+                # a decomposed 'й' is one letter in two code points
+                # (#665); compose it, as the lookup does
+                text = unicodedata.normalize("NFC", text)
             if len(text) == 1 and text.isalpha():
                 k += 1
                 continue