Extend the ZH surname list past the census top 100 (#695, #657)

docs/design/decisions/W1-unspaced-cjk-division.md

@@ -5,9 +5,20 @@
 
 - 2026-07-29 (ja amendment §1a) — script CLASSIFICATION NFC-normalizes its input (NFD katakana carries combining marks outside the block; NFD hangul decomposes to jamo, which would miss the order rule for macOS-origin names) while SEGMENTATION stays raw — chosen over offset-mapping complexity because NFD then degrades to no-split, never to a wrong split. The read-only folds are the deliberate exceptions to "nothing rewrites the text" (rules.md T Background), and neither ever enters a token. (Corrected 2026-09-10, #cjk-full-stops: this read "segmentation MATCHING stays raw" and "the one deliberate exception". Vocabulary matching composes NFC too since #322 — `_lexicon._normalize` folds every lookup and every stored entry the same way — so the folds are two, and what stays raw is segmentation proper: the surname site's membership test and the peel's tail slice, which index the token's own text. The no-split degradation is what still holds there.)
 
-Declined:
+- 2026-08 — zh and ja packs are corpus ALTERNATIVES, one per corpus; stacking them (parser_for(ZH, JA, segmenter=...)) is for genuinely mixed data that accepts the trade: a listed Chinese surname wins before the segmenter is consulted, so 高橋一郎 still divides 高 + 橋一郎 under ZH+JA exactly as under ZH alone. Kana gating resolves through the same script function order uses, so kana-licensed composites (高橋みなみ) gate in under JA while pure-katakana tokens never do (amendment 2026-07-27: activation is per script because the ambiguity is per script).
 
-- JMnedict as bundled segmentation data (#272's body, 2026-07-07) — the pip packaging (jamdict-data) was last compiled 2021-04; JMnedict carries no frequency data, so it cannot resolve the ambiguous 2+2/3+1 splits that motivate a segmenter at all; and bundling raised licensing questions the core avoids by delegating to namedivider — whose code and GBDT model are MIT, whose surname data carries Myoji-Yurai terms permitting exactly this name-dividing use, and whose CC-BY-SA BERT model is unused (the 2026-07-29 amendment corrected an earlier flat-"MIT" description; so does this entry). The kind of decline someone re-proposes in two years.
+- 2026-10-18 #695 (weighing; folds in #657) — the ZH surname list reaches past the 2020 census top 100. The question was what a caller who applies `locales.ZH` wants, and the answer taken is Chinese names divided: applying the pack is already the "my data is Chinese" declaration (decisions/W1's 2026-08 entry, on zh and ja as corpus alternatives), and a Japanese corpus has `locales.JA`. So a common Chinese name left whole (柯洁 → family 柯洁) counts as a failure, not as the safe side of W1's preference for an undivided word over a wrong division. The single-character list became the census rows plus the union of two rankings' top 300: Taiwan's Ministry of the Interior surname ranking (2023, Open Government Data License v1) and Wikidata's Chinese family names (Q1093580) ranked by bearers (CC0). Both, because each is biased by region in its own direction: Taiwan ranks Hokkien surnames high (詹 37, 柯 46), and Wikidata counts notable people, Korean and Vietnamese bearers of the same character included, and types some common surnames outside Q1093580 — 46 of the 344 selected surnames reach the list only through Taiwan, among them 关 and 区 (both named in #657), 颜, 东 and 兰. On the native-script corpus alone Wikidata's list did as well for less (the cutoff measurements in this entry); the union was taken for the surnames that corpus does not hold. `tools/zh_surnames.py` derives the block and writes it into zh.py; it adds a traditional character's simplified form from Unihan when Unihan gives exactly one in the BMP, and adds a traditional form only where a source attests it, because the reverse mapping is one-to-many (余's traditional variants include 餘, which is no one's surname).
+  Measured on 16ee2851 against this change, with `parser_for(locales.ZH)`: the 80 Chinese rows of the native-script corpus went from 74 to 80 agreeing (`wikidata_native_agreement_locales.json` holds it); the CLDR locale view stayed at 42/96, and every rendering that moved there was wrong before and after (a given name alone, 文傑 → 文 + 傑 where it had been family 文傑; the transcription 辛巴达; the given-first 安怡张). Recompute: `uv run python tools/differential/cldr_agreement.py`.
+  Accepted: Japanese names read under ZH split wrongly more often. On the 60 Japanese rows of the native-script corpus, wrong splits went from 2 to 11 (安達禎 → 安, 平賀淳 → 平, 吉川亨 → 吉, …), taken from "left whole"; the rows read right stayed at 12. On CLDR's native Japanese renderings, 安藤一郎 now gives 安 in the two family-first orders. Transcribed foreign names written without the 间隔号 split too (安娜 → 安 + 娜), since 安, 艾 and 伊 are surnames inside the cut. Recompute: parse each `ja-jp` row of `wikidata_native_fields.jsonl` under `parser_for(locales.ZH)` at the parent commit and at this one, and count a family that is a proper prefix of the text but not the stated surname.
+  Accepted: a listed compound wins over its first character in every name that opens with it, whatever the name's length — longest match first, the same mechanism the married-double exclusion in decisions/vocabulary-exclusions guards against. 张简明 reads 张简 + 明 where it had read 张 + 简明, 贺兰芳 reads 贺兰 + 芳, 范姜伟 reads 范姜 + 伟, and a two-character name that spells a compound stays whole (张简 is family 张简, not 张 + 简, a token that is wholly a surname not being divided). This was already true of the 34 shipped compounds (司马), and the compounds admitted are borne as one surname — 張簡 by 9,162 people in the Taiwan file — where a married double is two. The cost is on record: Wikidata has 25 people whose family name is the 张 item alone and whose Chinese label opens with 张简 (measured 2026-10-18). 14 are labelled with the two characters alone, 9 of them with given name 简 stated, so they now read whole where they had read 张 + 简; the longer ones are Taiwanese 張簡 bearers tagged with the single character or undetermined (the Tang and Ming 张简修). Recompute: people with P734 = Q804909 and a zh, zh-hans, zh-hant, zh-cn or zh-tw label starting 张简 or 張簡, with their P735 labels.
+  The cutoff (wrong Japanese splits of 60, Chinese rows right of 80, each list run through the tool's `select` with the census rows and the shipped compounds): Wikidata alone at 200, 4 and 79; at 300, 9 and 80; at 400, 12 and 80. Taiwan alone at 200, 4 and 78; at 300, 10 and 79 (茅 is its 326th, in `read_moi`'s order); at 400, 12 and 80. Both at 200, 5 and 79; at 300, 11 and 80; at 400, 14 and 80. The corpus does not force 300: both lists at 212 read all 80 rows with 7 wrong Japanese splits, Wikidata alone at 212 with 6, and 211 misses a row. 300 leaves a margin over a corpus of 80 public figures, who skew toward common surnames (the reason #695 gave for expecting ordinary data to miss more), at the cost of 4 more Japanese wrong splits there than 212. (The first measurement, taken before the Unihan step existed, had Taiwan at 300 reading 77: without simplified forms it missed 卫 and 华.)
+  Compounds stay hand-curated (the note above `_COMPOUND` in zh.py states the admission test). A compound ships where Wikidata or the Taiwan file attests a bearer, and where no Wikidata bearer of its first character alone has a name opening with it; the check was read by hand, since most of its hits are compound bearers tagged with the single character (左丘明 under 左, 鍾離權 under 鍾). Amended the same day, after the second review: the check also fails where such a bearer's stated given name (P735) opens with the compound's second character, and a compound that fails it still ships when its living bearers in the Taiwan file outnumber those readings by orders of magnitude. 張簡 is the case (9,162 bearers against 9 people named 張 + 簡, the cost this entry's compound Accepted clause records); 东野 fails without it (no Taiwan bearers against 2 people named 东 + 野) and is left out. 25 simplified compounds were added, 钟离 and 段干 among them. Recompute the check: for each compound, people whose P734 is the item labelled with its first character, whose Chinese label opens with the compound, and whose P735 label opens with its second character.
 
-- 2026-08 — zh and ja packs are corpus ALTERNATIVES, one per corpus; stacking them (parser_for(ZH, JA, segmenter=...)) is for genuinely mixed data that accepts the trade: a listed Chinese surname wins before the segmenter is consulted, so 高橋一郎 still divides 高 + 橋一郎 under ZH+JA exactly as under ZH alone. Kana gating resolves through the same script function order uses, so kana-licensed composites (高橋みなみ) gate in under JA while pure-katakana tokens never do (amendment 2026-07-27: activation is per script because the ambiguity is per script).
+Declined:
 
+- JMnedict as bundled segmentation data (#272's body, 2026-07-07) — the pip packaging (jamdict-data) was last compiled 2021-04; JMnedict carries no frequency data, so it cannot resolve the ambiguous 2+2/3+1 splits that motivate a segmenter at all; and bundling raised licensing questions the core avoids by delegating to namedivider — whose code and GBDT model are MIT, whose surname data carries Myoji-Yurai terms permitting exactly this name-dividing use, and whose CC-BY-SA BERT model is unused (the 2026-07-29 amendment corrected an earlier flat-"MIT" description; so does this entry). The kind of decline someone re-proposes in two years.
+- A pack with every surname either source ranks (#695 review, 2026-10-18). Taiwan's single-character surnames past the 300th hold 77,355 of its 23,373,283 people (0.33%). Recompute over the order `tools/zh_surnames.py`'s `read_moi` returns — single unified ideographs ranked among themselves, which is not the CSV's `ranking` column, since that column counts two-character surnames too — summing the population column for every surname past index 300. The tail is where the characters that open Japanese surnames sit (in that order: 山 369, 川 656, 大 787, 小 788, 中 920, 本 1008). For data that is Chinese, a complete list acts as "divide after the first character unless a compound opens the name"; the list's work past the head is screening out text that is not a Chinese name, and the tail stops paying for it. A stricter "divide after the first character" mode was raised in the same review and declined: one ZH pack.
+- Sources not used as data: the PRC Ministry of Public Security name reports (top 100 only, no licence); Yuan Yida's 300- and 400-surname books (copyrighted; usable as a cross-check only); the CRAN package ChineseNames (GPL-3, not redistributable here); zh.wikipedia's 中国姓氏排名 tables (CC BY-SA, compiled from the sources above).
+- Hundred Family Surnames compounds no source attests a bearer of (宗政, 谷梁, 东门, 南门, 归海 and others). They divide nothing real, and where the first character is a listed surname they would cut a name that opens with it (宗 + 政X).
+- 东野 — two Wikidata people named 东野 have 野 as their stated given name, and the Taiwan file has no 東野 bearer to outweigh them (the amended compound test, 2026-10-18).
+- 成公 — 成公亮, the guqin player, is 成 + 公亮, and a Wikidata bearer of 成 whose name opens with 成公 is the case the compound test exists to refuse.

docs/design/decisions/vocabulary-exclusions.md

@@ -109,3 +109,8 @@ Excluded (CAPITALIZATION_EXCEPTIONS — meng, edd, lac, ded, left out of the 202
 Excluded (CAPITALIZATION_EXCEPTIONS — md, ii, iii, iv, removed 2026-09-23, #459):
 
 - `md`, `ii`, `iii` and `iv` are OUT of the exceptions map and must not be put back by a sweep that finds one of them title-cased somewhere. As a SUFFIX each is written in capitals with no entry — `md` by the acronym clause, the numerals by the numeral clause — and in any other role each repairs as the parse read it by decision (decisions/R4's item-4 bullet), so an entry would only bring back the role-free capitals that bullet retired: `iv smith` → `IV Smith`, `Md Abdul Karim` → `MD Abdul Karim`. A caller who wants them back adds them to a private lexicon, as masks.
+
+Excluded (locales.ZH surnames, nameparser/locales/zh.py — the two-character entries):
+
+- Married double surnames (王李, 張李, 陳李, 李林, 張陳 and the rest of the Taiwan file's two-character rows that join two single surnames). The longest match runs first, so 李林 would cut 李林伟 into 李林 + 伟, a real Chinese name read wrong where today it reads right (2026-10-18, #695). 張廖, 張簡 and 范姜 are not this class: they are clan compounds borne as one surname, and ship.
+- Japanese surnames and transcriptions in the same file (佐藤, 鈴木, 馬丁). The pack divides Chinese names; a Japanese surname belongs to `locales.JA`'s segmenter (2026-10-18, #695).

docs/design/rules/W1-unspaced-cjk-division.md

@@ -20,6 +20,9 @@ W1. Rationale: hangul is monoglot Korean and its surnames are a
       "毛泽东"                    →  family="毛泽东"  · boundary
       "毛泽东"  [zh]              →  family="毛"
       "高橋一郎"  [zh]            →  family="高"
+      "柯洁"  [zh]                →  family="柯"
+      "钟离小凤"  [zh]            →  family="钟离"
+      "张简"  [zh]                →  family="张简"
     Accepted: a name the interpunct divides is already divided in
     the sense that matters — division stands down entirely there,
     vocabulary and segmenter alike (#298; decisions/T3).

docs/locales.rst

@@ -180,7 +180,11 @@ Shipped packs
        surname and given name (``毛泽东`` → family ``毛``, given
        ``泽东``) against the surname list the pack ships. It sets no
        name order: native-script Han already reads family-first
-       without a pack.
+       without a pack. The list holds the commonest surnames of the
+       mainland and Taiwan, in both scripts, and the compound surnames
+       still borne; a name whose surname is not on it stays whole. A
+       Japanese name read with it can split after its first character
+       (``安藤一郎`` → family ``安``), so use ``ja`` for Japanese data.
 
 ``ja``, ``ru`` and ``tr_az`` are policy-only — they carry no vocabulary
 of their own. ``zh`` is both halves at once: a surname list, plus the

docs/release_log.rst

@@ -9,6 +9,7 @@ Release Log
     - **Add junior and senior in other languages as suffixes.** ``Antonio Pavlović mlađi`` gives last ``Pavlović``, suffix ``mlađi``, where 1.4.0 through 2.4.2 gave middle ``Pavlović``, last ``mlađi``. The words come from 45 languages and are read when written as a separate word after the name. :issue:`661`, :issue:`679`
     - **Remove mai from the default post-nominal acronyms, so a trailing Mai keeps the name.** ``Tran Thi Mai`` read in Vietnamese order (``FAMILY_FIRST_GIVEN_LAST``) gives given ``Mai``, middle ``Thi``, family ``Tran``, where 2.0.0 through 2.4.2 gave suffix ``Mai`` and no given name; ``Nguyen Mai`` gives family ``Mai``, as 1.4.0 did. The cost is that ``John Smith MAI`` gives last ``MAI`` instead of suffix; ``Smith, John, MAI`` and ``John Smith M.A.I.`` still read as credentials. A caller who needs it back adds it: ``Lexicon.default().add(suffix_acronyms={"mai"})``. See the ``suffix-acronym-collisions`` entry of ``docs/design/decisions/`` (:issue:`687`)
     - **Add Chinese and Japanese credential, office and generational words as suffixes.** ``彭德威 议员`` under ``locales.ZH`` gives family ``彭``, given ``德威``, suffix ``议员``, where 2.1 through 2.4.2 gave middle ``议员``; ``田中社長`` gives family ``田中``, suffix ``社長``, where 2.1 through 2.4.2 gave family ``田中社長``. Read when written as a separate word, after a comma, or glued to the end of the name, except that 部長 and 部长 are not split off a glued name (``田中部長`` is unchanged) unless ``locales.ZH`` divides it at a listed surname (``李部長``). :issue:`671`, :issue:`707`
+    - **Extend the Chinese surname list in locales.ZH past the census top 100, and add the compound surnames it was missing.** ``柯洁`` under ``locales.ZH`` gives family ``柯``, given ``洁``, where 2.1.0 through 2.4.2 gave family ``柯洁``; ``钟离小凤`` gives family ``钟离`` where they gave ``钟``. A Japanese name read under ``locales.ZH`` splits after its first character more often (``安藤一郎`` gives family ``安``), so use ``locales.JA`` for Japanese names. :issue:`695`, :issue:`657`
     - **Add the Akan and Indonesian salutations owura and bapak as titles.** ``Bapak Dwi Putro`` gives title ``Bapak``, first ``Dwi``, last ``Putro``, where 1.4.0 through 2.4.2 gave first ``Bapak``, middle ``Dwi``. :issue:`662`
     - **Fix a title written before a family comma reading as part of the family name.** ``Dr. Smith, John`` gives title ``Dr.``, last ``Smith``, where 1.4.0 through 2.4.2 gave last ``Dr. Smith``. Only listed titles are read there, so ``Xyz. Smith, John`` keeps its family, and a title that is also a surname particle, such as ``St`` or ``Freiherr``, stays in the surname. :issue:`701`
     - **Fix a surname particle such as d' or al- reading as a middle name when a listing puts it after the given name.** ``Ivoi, Paul d'`` gives last ``d' Ivoi``, where 1.4.0 through 2.4.2 gave middle ``d'``. The Arabic article reads the same in its plain and assimilated spellings (``al-``, ``az-``, ``aš-``). A particle written joined to the surname, as in ``Paul d'Ivoi``, is unchanged. :issue:`700`

nameparser/_pipeline/_script_segment.py

@@ -688,7 +688,7 @@ def _split_surname_site(state: ParseState) -> ParseState:
     # boundary its writer drew just as deliberately as a Han one: under
     # the JA pack "山田太郎 マイケル" declines, though katakana is in no
     # activation set. A Latin title or suffix is NOT such a boundary:
-    # it says nothing about where the CJK name splits, so "Dr 阿明日,
+    # it says nothing about where the CJK name splits, so "Dr 小明日,
     # Jr." still reaches the segmenter. Vocabulary keeps its own rule
     # -- a listed surname is a certainty about that exact string,
     # whoever else stands beside it.