diff --git a/docs/design/decisions/W1-unspaced-cjk-division.md b/docs/design/decisions/W1-unspaced-cjk-division.md index 62f16f2e..027690a6 100644 --- a/docs/design/decisions/W1-unspaced-cjk-division.md +++ b/docs/design/decisions/W1-unspaced-cjk-division.md @@ -5,9 +5,20 @@ - 2026-07-29 (ja amendment §1a) — script CLASSIFICATION NFC-normalizes its input (NFD katakana carries combining marks outside the block; NFD hangul decomposes to jamo, which would miss the order rule for macOS-origin names) while SEGMENTATION stays raw — chosen over offset-mapping complexity because NFD then degrades to no-split, never to a wrong split. The read-only folds are the deliberate exceptions to "nothing rewrites the text" (rules.md T Background), and neither ever enters a token. (Corrected 2026-09-10, #cjk-full-stops: this read "segmentation MATCHING stays raw" and "the one deliberate exception". Vocabulary matching composes NFC too since #322 — `_lexicon._normalize` folds every lookup and every stored entry the same way — so the folds are two, and what stays raw is segmentation proper: the surname site's membership test and the peel's tail slice, which index the token's own text. The no-split degradation is what still holds there.) -Declined: +- 2026-08 — zh and ja packs are corpus ALTERNATIVES, one per corpus; stacking them (parser_for(ZH, JA, segmenter=...)) is for genuinely mixed data that accepts the trade: a listed Chinese surname wins before the segmenter is consulted, so 高橋一郎 still divides 高 + 橋一郎 under ZH+JA exactly as under ZH alone. Kana gating resolves through the same script function order uses, so kana-licensed composites (高橋みなみ) gate in under JA while pure-katakana tokens never do (amendment 2026-07-27: activation is per script because the ambiguity is per script). -- JMnedict as bundled segmentation data (#272's body, 2026-07-07) — the pip packaging (jamdict-data) was last compiled 2021-04; JMnedict carries no frequency data, so it cannot resolve the ambiguous 2+2/3+1 splits that motivate a segmenter at all; and bundling raised licensing questions the core avoids by delegating to namedivider — whose code and GBDT model are MIT, whose surname data carries Myoji-Yurai terms permitting exactly this name-dividing use, and whose CC-BY-SA BERT model is unused (the 2026-07-29 amendment corrected an earlier flat-"MIT" description; so does this entry). The kind of decline someone re-proposes in two years. +- 2026-10-18 #695 (weighing; folds in #657) — the ZH surname list reaches past the 2020 census top 100. The question was what a caller who applies `locales.ZH` wants, and the answer taken is Chinese names divided: applying the pack is already the "my data is Chinese" declaration (decisions/W1's 2026-08 entry, on zh and ja as corpus alternatives), and a Japanese corpus has `locales.JA`. So a common Chinese name left whole (柯洁 → family 柯洁) counts as a failure, not as the safe side of W1's preference for an undivided word over a wrong division. The single-character list became the census rows plus the union of two rankings' top 300: Taiwan's Ministry of the Interior surname ranking (2023, Open Government Data License v1) and Wikidata's Chinese family names (Q1093580) ranked by bearers (CC0). Both, because each is biased by region in its own direction: Taiwan ranks Hokkien surnames high (詹 37, 柯 46), and Wikidata counts notable people, Korean and Vietnamese bearers of the same character included, and types some common surnames outside Q1093580 — 46 of the 344 selected surnames reach the list only through Taiwan, among them 关 and 区 (both named in #657), 颜, 东 and 兰. On the native-script corpus alone Wikidata's list did as well for less (the cutoff measurements in this entry); the union was taken for the surnames that corpus does not hold. `tools/zh_surnames.py` derives the block and writes it into zh.py; it adds a traditional character's simplified form from Unihan when Unihan gives exactly one in the BMP, and adds a traditional form only where a source attests it, because the reverse mapping is one-to-many (余's traditional variants include 餘, which is no one's surname). + Measured on 16ee2851 against this change, with `parser_for(locales.ZH)`: the 80 Chinese rows of the native-script corpus went from 74 to 80 agreeing (`wikidata_native_agreement_locales.json` holds it); the CLDR locale view stayed at 42/96, and every rendering that moved there was wrong before and after (a given name alone, 文傑 → 文 + 傑 where it had been family 文傑; the transcription 辛巴达; the given-first 安怡张). Recompute: `uv run python tools/differential/cldr_agreement.py`. + Accepted: Japanese names read under ZH split wrongly more often. On the 60 Japanese rows of the native-script corpus, wrong splits went from 2 to 11 (安達禎 → 安, 平賀淳 → 平, 吉川亨 → 吉, …), taken from "left whole"; the rows read right stayed at 12. On CLDR's native Japanese renderings, 安藤一郎 now gives 安 in the two family-first orders. Transcribed foreign names written without the 间隔号 split too (安娜 → 安 + 娜), since 安, 艾 and 伊 are surnames inside the cut. Recompute: parse each `ja-jp` row of `wikidata_native_fields.jsonl` under `parser_for(locales.ZH)` at the parent commit and at this one, and count a family that is a proper prefix of the text but not the stated surname. + Accepted: a listed compound wins over its first character in every name that opens with it, whatever the name's length — longest match first, the same mechanism the married-double exclusion in decisions/vocabulary-exclusions guards against. 张简明 reads 张简 + 明 where it had read 张 + 简明, 贺兰芳 reads 贺兰 + 芳, 范姜伟 reads 范姜 + 伟, and a two-character name that spells a compound stays whole (张简 is family 张简, not 张 + 简, a token that is wholly a surname not being divided). This was already true of the 34 shipped compounds (司马), and the compounds admitted are borne as one surname — 張簡 by 9,162 people in the Taiwan file — where a married double is two. The cost is on record: Wikidata has 25 people whose family name is the 张 item alone and whose Chinese label opens with 张简 (measured 2026-10-18). 14 are labelled with the two characters alone, 9 of them with given name 简 stated, so they now read whole where they had read 张 + 简; the longer ones are Taiwanese 張簡 bearers tagged with the single character or undetermined (the Tang and Ming 张简修). Recompute: people with P734 = Q804909 and a zh, zh-hans, zh-hant, zh-cn or zh-tw label starting 张简 or 張簡, with their P735 labels. + The cutoff (wrong Japanese splits of 60, Chinese rows right of 80, each list run through the tool's `select` with the census rows and the shipped compounds): Wikidata alone at 200, 4 and 79; at 300, 9 and 80; at 400, 12 and 80. Taiwan alone at 200, 4 and 78; at 300, 10 and 79 (茅 is its 326th, in `read_moi`'s order); at 400, 12 and 80. Both at 200, 5 and 79; at 300, 11 and 80; at 400, 14 and 80. The corpus does not force 300: both lists at 212 read all 80 rows with 7 wrong Japanese splits, Wikidata alone at 212 with 6, and 211 misses a row. 300 leaves a margin over a corpus of 80 public figures, who skew toward common surnames (the reason #695 gave for expecting ordinary data to miss more), at the cost of 4 more Japanese wrong splits there than 212. (The first measurement, taken before the Unihan step existed, had Taiwan at 300 reading 77: without simplified forms it missed 卫 and 华.) + Compounds stay hand-curated (the note above `_COMPOUND` in zh.py states the admission test). A compound ships where Wikidata or the Taiwan file attests a bearer, and where no Wikidata bearer of its first character alone has a name opening with it; the check was read by hand, since most of its hits are compound bearers tagged with the single character (左丘明 under 左, 鍾離權 under 鍾). Amended the same day, after the second review: the check also fails where such a bearer's stated given name (P735) opens with the compound's second character, and a compound that fails it still ships when its living bearers in the Taiwan file outnumber those readings by orders of magnitude. 張簡 is the case (9,162 bearers against 9 people named 張 + 簡, the cost this entry's compound Accepted clause records); 东野 fails without it (no Taiwan bearers against 2 people named 东 + 野) and is left out. 25 simplified compounds were added, 钟离 and 段干 among them. Recompute the check: for each compound, people whose P734 is the item labelled with its first character, whose Chinese label opens with the compound, and whose P735 label opens with its second character. -- 2026-08 — zh and ja packs are corpus ALTERNATIVES, one per corpus; stacking them (parser_for(ZH, JA, segmenter=...)) is for genuinely mixed data that accepts the trade: a listed Chinese surname wins before the segmenter is consulted, so 高橋一郎 still divides 高 + 橋一郎 under ZH+JA exactly as under ZH alone. Kana gating resolves through the same script function order uses, so kana-licensed composites (高橋みなみ) gate in under JA while pure-katakana tokens never do (amendment 2026-07-27: activation is per script because the ambiguity is per script). +Declined: +- JMnedict as bundled segmentation data (#272's body, 2026-07-07) — the pip packaging (jamdict-data) was last compiled 2021-04; JMnedict carries no frequency data, so it cannot resolve the ambiguous 2+2/3+1 splits that motivate a segmenter at all; and bundling raised licensing questions the core avoids by delegating to namedivider — whose code and GBDT model are MIT, whose surname data carries Myoji-Yurai terms permitting exactly this name-dividing use, and whose CC-BY-SA BERT model is unused (the 2026-07-29 amendment corrected an earlier flat-"MIT" description; so does this entry). The kind of decline someone re-proposes in two years. +- A pack with every surname either source ranks (#695 review, 2026-10-18). Taiwan's single-character surnames past the 300th hold 77,355 of its 23,373,283 people (0.33%). Recompute over the order `tools/zh_surnames.py`'s `read_moi` returns — single unified ideographs ranked among themselves, which is not the CSV's `ranking` column, since that column counts two-character surnames too — summing the population column for every surname past index 300. The tail is where the characters that open Japanese surnames sit (in that order: 山 369, 川 656, 大 787, 小 788, 中 920, 本 1008). For data that is Chinese, a complete list acts as "divide after the first character unless a compound opens the name"; the list's work past the head is screening out text that is not a Chinese name, and the tail stops paying for it. A stricter "divide after the first character" mode was raised in the same review and declined: one ZH pack. +- Sources not used as data: the PRC Ministry of Public Security name reports (top 100 only, no licence); Yuan Yida's 300- and 400-surname books (copyrighted; usable as a cross-check only); the CRAN package ChineseNames (GPL-3, not redistributable here); zh.wikipedia's 中国姓氏排名 tables (CC BY-SA, compiled from the sources above). +- Hundred Family Surnames compounds no source attests a bearer of (宗政, 谷梁, 东门, 南门, 归海 and others). They divide nothing real, and where the first character is a listed surname they would cut a name that opens with it (宗 + 政X). +- 东野 — two Wikidata people named 东野 have 野 as their stated given name, and the Taiwan file has no 東野 bearer to outweigh them (the amended compound test, 2026-10-18). +- 成公 — 成公亮, the guqin player, is 成 + 公亮, and a Wikidata bearer of 成 whose name opens with 成公 is the case the compound test exists to refuse. diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index ec9626a7..2a57d966 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -109,3 +109,8 @@ Excluded (CAPITALIZATION_EXCEPTIONS — meng, edd, lac, ded, left out of the 202 Excluded (CAPITALIZATION_EXCEPTIONS — md, ii, iii, iv, removed 2026-09-23, #459): - `md`, `ii`, `iii` and `iv` are OUT of the exceptions map and must not be put back by a sweep that finds one of them title-cased somewhere. As a SUFFIX each is written in capitals with no entry — `md` by the acronym clause, the numerals by the numeral clause — and in any other role each repairs as the parse read it by decision (decisions/R4's item-4 bullet), so an entry would only bring back the role-free capitals that bullet retired: `iv smith` → `IV Smith`, `Md Abdul Karim` → `MD Abdul Karim`. A caller who wants them back adds them to a private lexicon, as masks. + +Excluded (locales.ZH surnames, nameparser/locales/zh.py — the two-character entries): + +- Married double surnames (王李, 張李, 陳李, 李林, 張陳 and the rest of the Taiwan file's two-character rows that join two single surnames). The longest match runs first, so 李林 would cut 李林伟 into 李林 + 伟, a real Chinese name read wrong where today it reads right (2026-10-18, #695). 張廖, 張簡 and 范姜 are not this class: they are clan compounds borne as one surname, and ship. +- Japanese surnames and transcriptions in the same file (佐藤, 鈴木, 馬丁). The pack divides Chinese names; a Japanese surname belongs to `locales.JA`'s segmenter (2026-10-18, #695). diff --git a/docs/design/rules/W1-unspaced-cjk-division.md b/docs/design/rules/W1-unspaced-cjk-division.md index 45a03cfc..2256d8e1 100644 --- a/docs/design/rules/W1-unspaced-cjk-division.md +++ b/docs/design/rules/W1-unspaced-cjk-division.md @@ -20,6 +20,9 @@ W1. Rationale: hangul is monoglot Korean and its surnames are a "毛泽东" → family="毛泽东" · boundary "毛泽东" [zh] → family="毛" "高橋一郎" [zh] → family="高" + "柯洁" [zh] → family="柯" + "钟离小凤" [zh] → family="钟离" + "张简" [zh] → family="张简" Accepted: a name the interpunct divides is already divided in the sense that matters — division stands down entirely there, vocabulary and segmenter alike (#298; decisions/T3). diff --git a/docs/locales.rst b/docs/locales.rst index 603379d1..74913ccd 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -180,7 +180,11 @@ Shipped packs surname and given name (``毛泽东`` → family ``毛``, given ``泽东``) against the surname list the pack ships. It sets no name order: native-script Han already reads family-first - without a pack. + without a pack. The list holds the commonest surnames of the + mainland and Taiwan, in both scripts, and the compound surnames + still borne; a name whose surname is not on it stays whole. A + Japanese name read with it can split after its first character + (``安藤一郎`` → family ``安``), so use ``ja`` for Japanese data. ``ja``, ``ru`` and ``tr_az`` are policy-only — they carry no vocabulary of their own. ``zh`` is both halves at once: a surname list, plus the diff --git a/docs/release_log.rst b/docs/release_log.rst index a02be1e6..bd32da51 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -9,6 +9,7 @@ Release Log - **Add junior and senior in other languages as suffixes.** ``Antonio Pavlović mlađi`` gives last ``Pavlović``, suffix ``mlađi``, where 1.4.0 through 2.4.2 gave middle ``Pavlović``, last ``mlađi``. The words come from 45 languages and are read when written as a separate word after the name. :issue:`661`, :issue:`679` - **Remove mai from the default post-nominal acronyms, so a trailing Mai keeps the name.** ``Tran Thi Mai`` read in Vietnamese order (``FAMILY_FIRST_GIVEN_LAST``) gives given ``Mai``, middle ``Thi``, family ``Tran``, where 2.0.0 through 2.4.2 gave suffix ``Mai`` and no given name; ``Nguyen Mai`` gives family ``Mai``, as 1.4.0 did. The cost is that ``John Smith MAI`` gives last ``MAI`` instead of suffix; ``Smith, John, MAI`` and ``John Smith M.A.I.`` still read as credentials. A caller who needs it back adds it: ``Lexicon.default().add(suffix_acronyms={"mai"})``. See the ``suffix-acronym-collisions`` entry of ``docs/design/decisions/`` (:issue:`687`) - **Add Chinese and Japanese credential, office and generational words as suffixes.** ``彭德威 议员`` under ``locales.ZH`` gives family ``彭``, given ``德威``, suffix ``议员``, where 2.1 through 2.4.2 gave middle ``议员``; ``田中社長`` gives family ``田中``, suffix ``社長``, where 2.1 through 2.4.2 gave family ``田中社長``. Read when written as a separate word, after a comma, or glued to the end of the name, except that 部長 and 部长 are not split off a glued name (``田中部長`` is unchanged) unless ``locales.ZH`` divides it at a listed surname (``李部長``). :issue:`671`, :issue:`707` + - **Extend the Chinese surname list in locales.ZH past the census top 100, and add the compound surnames it was missing.** ``柯洁`` under ``locales.ZH`` gives family ``柯``, given ``洁``, where 2.1.0 through 2.4.2 gave family ``柯洁``; ``钟离小凤`` gives family ``钟离`` where they gave ``钟``. A Japanese name read under ``locales.ZH`` splits after its first character more often (``安藤一郎`` gives family ``安``), so use ``locales.JA`` for Japanese names. :issue:`695`, :issue:`657` - **Add the Akan and Indonesian salutations owura and bapak as titles.** ``Bapak Dwi Putro`` gives title ``Bapak``, first ``Dwi``, last ``Putro``, where 1.4.0 through 2.4.2 gave first ``Bapak``, middle ``Dwi``. :issue:`662` - **Fix a title written before a family comma reading as part of the family name.** ``Dr. Smith, John`` gives title ``Dr.``, last ``Smith``, where 1.4.0 through 2.4.2 gave last ``Dr. Smith``. Only listed titles are read there, so ``Xyz. Smith, John`` keeps its family, and a title that is also a surname particle, such as ``St`` or ``Freiherr``, stays in the surname. :issue:`701` - **Fix a surname particle such as d' or al- reading as a middle name when a listing puts it after the given name.** ``Ivoi, Paul d'`` gives last ``d' Ivoi``, where 1.4.0 through 2.4.2 gave middle ``d'``. The Arabic article reads the same in its plain and assimilated spellings (``al-``, ``az-``, ``aš-``). A particle written joined to the surname, as in ``Paul d'Ivoi``, is unchanged. :issue:`700` diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index e5f756d1..e20d2485 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -688,7 +688,7 @@ def _split_surname_site(state: ParseState) -> ParseState: # boundary its writer drew just as deliberately as a Han one: under # the JA pack "山田太郎 マイケル" declines, though katakana is in no # activation set. A Latin title or suffix is NOT such a boundary: - # it says nothing about where the CJK name splits, so "Dr 阿明日, + # it says nothing about where the CJK name splits, so "Dr 小明日, # Jr." still reaches the segmenter. Vocabulary keeps its own rule # -- a listed surname is a certainty about that exact string, # whoever else stands beside it. diff --git a/nameparser/locales/zh.py b/nameparser/locales/zh.py index d8689aed..c4bbd722 100644 --- a/nameparser/locales/zh.py +++ b/nameparser/locales/zh.py @@ -12,10 +12,16 @@ divides a kanji name. Data sources: single-character surnames are the top-100 of the 2020 -PRC census; traditional-script variant forms are included for names -written in Taiwan/Hong Kong/overseas orthography; two-character -compounds are the Hundred Family Surnames compounds still in modern -use. A coverage floor, not a census roster: extend by giving +PRC census, plus the union of two rankings' top 300 (#695): Taiwan's +Ministry of the Interior surname ranking (內政部戶政司, 2023, used under +the Open Government Data License, version 1) and Wikidata's Chinese +family names ranked by bearers (CC0). tools/zh_surnames.py derives +that block and writes it below; do not edit it by hand. +Traditional-script variant forms are included for names written in +Taiwan/Hong Kong/overseas orthography; two-character compounds are +hand-curated (see the note above _COMPOUND), never generated, since a +married double surname like 李林 would cut 李林伟 into 李林 + 伟. A +coverage floor, not a census roster: extend by giving parser_for a base whose lexicon already carries the additions -- ``parser_for(locales.ZH, base=Parser(lexicon=Lexicon.default().add( surnames={...})))``. (Adding to the finished parser's lexicon instead @@ -74,25 +80,83 @@ "盧", "鍾", "譚", "陸", "韋", "賈", "鄒", "閆", "閻", "龍", "賀", "顧", "龔", "萬", "錢", "嚴", "蔣", ) -# Hundred Family Surnames compounds still in modern use, simplified. +# Both rankings' top 300 past the census rows, in combined rank order +# (each surname's better rank of the two), both scripts. +# -- BEGIN generated by tools/zh_surnames.py; edit the tool, not this block -- +_SINGLE_RANKED = ( + "洪", "賴", "赖", "庄", "莊", "简", "簡", "鐘", "锺", "游", + "詹", "施", "顏", "颜", "柯", "翁", "阮", "卓", "裴", "释", + "釋", "俞", "温", "溫", "蓝", "藍", "古", "章", "紀", "纪", + "连", "連", "欧", "歐", "汤", "湯", "康", "尤", "巫", "塗", + "涂", "柳", "元", "窦", "竇", "童", "凃", "伍", "倪", "丘", + "駱", "骆", "葛", "樊", "甘", "官", "邢", "鮑", "鲍", "牛", + "文", "祝", "粘", "饒", "饶", "左", "闕", "阙", "乔", "喬", + "凌", "淩", "齊", "齐", "舒", "姬", "翟", "辛", "华", "華", + "辜", "魯", "鲁", "盛", "成", "易", "殷", "池", "耿", "褚", + "卫", "衛", "衞", "麥", "麦", "路", "毕", "畢", "庞", "龐", + "焦", "管", "关", "關", "包", "虞", "項", "项", "梅", "聂", + "聶", "利", "房", "全", "佘", "花", "費", "费", "安", "喻", + "霍", "申", "穆", "应", "應", "瞿", "蒲", "柴", "谷", "臧", + "季", "練", "练", "苗", "边", "邊", "邬", "鄔", "查", "阳", + "陽", "楼", "樓", "岳", "符", "尚", "解", "屠", "繆", "缪", + "宗", "屈", "乐", "樂", "单", "單", "曲", "篮", "籃", "閔", + "闵", "滕", "戚", "靳", "留", "桂", "鄞", "諸", "诸", "卜", + "厐", "龎", "車", "车", "艾", "郎", "封", "鹿", "𡍼", "岑", + "商", "时", "時", "买", "買", "卞", "祁", "力", "景", "云", + "雲", "郁", "吉", "席", "茅", "仇", "巴", "牟", "荆", "荊", + "司", "晏", "甄", "幸", "來", "来", "寇", "晁", "諶", "谌", + "邝", "鄺", "南", "储", "儲", "沙", "談", "谈", "栾", "欒", + "狄", "奚", "宫", "宮", "榮", "荣", "敬", "刁", "桑", "权", + "權", "和", "师", "師", "支", "松", "佟", "柏", "宁", "寧", + "甯", "祖", "冉", "麻", "隋", "冷", "風", "风", "子", "聞", + "闻", "鄂", "髙", "浦", "仲", "裘", "燕", "庾", "湛", "迟", + "遲", "东", "東", "米", "劳", "勞", "匡", "昌", "党", "兰", + "蘭", "阴", "陰", "娄", "婁", "敖", "惠", "戈", "羊", "伊", + "盖", "蓋", "茹", "強", "强", "平", "丛", "叢", "蒙", "羋", + "芈", "桓", "区", "區", "源", "偕", "栗", "初", "恽", "阿", + "兵", "冀", "計", "计", "索", "明", "蔺", "藺", "危", "鞠", + "廉", "茆", "雍", "蹇", "苻", "宣", "盘", "盤", "𨶒", "厉", + "厲", "机", "归", "歸", +) +# -- END generated -- +# Two-character surnames, simplified. The first 34 rows are the +# Hundred Family Surnames compounds still in modern use; the rows after +# them (#695) admit a compound surname -- never a married double +# surname (王李) or a Japanese one -- only where Wikidata or the Taiwan +# ranking attests a bearer, and only where no Wikidata bearer of its +# first character alone has a name opening with it (成公 is left out: +# 成公亮 is 成 + 公亮), unless the compound's living bearers in the +# Taiwan file outnumber those readings by orders of magnitude (張簡: +# 9,162 bearers against 9 people named 張 + 簡; 东野, with none +# against 2, is left out). A listed compound wins over its first +# character in every name that opens with it (张简明 is 张简 + 明, and +# 张简 alone stays whole); that cost and the measurements are in +# decisions/W1. _COMPOUND = ( "欧阳", "司马", "诸葛", "上官", "夏侯", "皇甫", "尉迟", "公孙", "长孙", "慕容", "司徒", "司空", "令狐", "宇文", "东方", "独孤", "南宫", "西门", "澹台", "淳于", "单于", "申屠", "公羊", "仲孙", "轩辕", "呼延", "端木", "百里", "东郭", "闻人", "拓跋", "万俟", "夹谷", "太史", + "钟离", "鲜于", "赫连", "贺兰", "相里", "叔孙", "孟孙", "季孙", + "士孙", "羊舌", "濮阳", "陆费", "梁丘", "公西", "公冶", "乐正", + "东里", "张简", "范姜", "张廖", "第五", "颛孙", "段干", "闾丘", + "左丘", ) # Traditional forms of the compounds where any glyph differs. _COMPOUND_TRADITIONAL = ( "歐陽", "司馬", "諸葛", "尉遲", "公孫", "長孫", "東方", "獨孤", "南宮", "西門", "澹臺", "單于", "仲孫", "軒轅", "東郭", "聞人", "萬俟", "夾谷", + "鍾離", "鮮于", "赫連", "賀蘭", "叔孫", "孟孫", "季孫", "士孫", + "濮陽", "陸費", "樂正", "東里", "張簡", "張廖", "顓孫", "閭丘", ) # Private: ZH.lexicon.surnames is the supported way to read this, and # publishing a name later is compatible where un-publishing is not. _SURNAMES = frozenset( - _SINGLE + _SINGLE_TRADITIONAL + _COMPOUND + _COMPOUND_TRADITIONAL) + _SINGLE + _SINGLE_TRADITIONAL + _SINGLE_RANKED + _COMPOUND + + _COMPOUND_TRADITIONAL) ZH = Locale( code="zh", diff --git a/tests/v2/cases.py b/tests/v2/cases.py index eaed1afb..8204a817 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -11040,16 +11040,20 @@ def _check_cjk_shape_purity(self) -> None: locale="zh", classification="fix(#271)", ambiguities=("segmentation",), notes="夏 (rank 65) is itself listed, so longest-first " - "decides a real fork here and records it; most " - "compounds' first chars are NOT listed"), + "decides a real fork here and records it"), Case("zh_compound_two_char_given", "司马相如", {"family": "司马", "given": "相如"}, - locale="zh", classification="fix(#271)"), + locale="zh", classification="fix(#271)", + ambiguities=("segmentation",), + notes="司 is listed since #695's ranked surnames, so the " + "compound wins a fork, as 夏侯 does"), Case("zh_traditional_compound", "諸葛亮", {"family": "諸葛", "given": "亮"}, - locale="zh", classification="fix(#271)"), - Case("zh_no_surname_match", "阿明", - {"family": "阿明"}, + locale="zh", classification="fix(#271)", + ambiguities=("segmentation",), + notes="諸 is listed since #695's ranked surnames"), + Case("zh_no_surname_match", "小明", + {"family": "小明"}, locale="zh", classification="fix(#271)", notes="no surname prefix in the vocabulary: the token stays " "whole and takes the script order's first role"), diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index da3610f6..e478c07d 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1601,7 +1601,7 @@ def boom(text: str) -> Segmentation | None: p = parser_for(locales.get("zh"), base=Parser(segmenter=boom)) with pytest.raises(RuntimeError, match="segmenter bug"): - p.parse("阿明") # zh pack active, 阿 unmatched by vocabulary -> + p.parse("小明") # zh pack active, 小 unmatched by vocabulary -> # the stage consults the segmenter @@ -1615,10 +1615,10 @@ def always(text: str) -> Segmentation | None: return Segmentation((1,), confidence=1.0) p = parser_for(locales.get("zh"), base=Parser(segmenter=always)) - assert p.parse("阿明").family == "阿" # one token: consulted - n = p.parse("阿明 日月") # already divided - assert (n.family, n.given) == ("阿明", "日月") - assert p.parse("Dr 阿明").family == "阿" # a title is not a split + assert p.parse("小明").family == "小" # one token: consulted + n = p.parse("小明 日月") # already divided + assert (n.family, n.given) == ("小明", "日月") + assert p.parse("Dr 小明").family == "小" # a title is not a split def test_a_segmenter_split_reaches_the_fields() -> None: @@ -1629,8 +1629,8 @@ def two(text: str) -> Segmentation | None: return Segmentation((1,), confidence=0.5) p = parser_for(locales.get("zh"), base=Parser(segmenter=two)) - n = p.parse("阿明") - assert (n.family, n.given) == ("阿", "明") + n = p.parse("小明") + assert (n.family, n.given) == ("小", "明") assert [a.kind for a in n.ambiguities] == [AmbiguityKind.SEGMENTATION] # two cuts, three pieces -- and the token indices the comma @@ -1639,9 +1639,9 @@ def three(text: str) -> Segmentation | None: return Segmentation((1, 2)) q = parser_for(locales.get("zh"), base=Parser(segmenter=three)) - n3 = q.parse("Dr 阿明日, Jr.") + n3 = q.parse("Dr 小明日, Jr.") assert (n3.title, n3.family, n3.given, n3.middle, n3.suffix) == ( - "Dr", "阿", "明", "日", "Jr.") + "Dr", "小", "明", "日", "Jr.") def _noop_segmenter(text: str) -> None: diff --git a/tests/v2/test_zh_surnames_tool.py b/tests/v2/test_zh_surnames_tool.py new file mode 100644 index 00000000..e9796f53 --- /dev/null +++ b/tests/v2/test_zh_surnames_tool.py @@ -0,0 +1,145 @@ +"""Offline checks on tools/zh_surnames.py, which writes the ZH pack's +ranked surname block (#695). + +The tool's sources are two downloads and a live SPARQL query, none of +which a test may touch, so these pin the reading and the selection on +hand-built inputs: the failure that matters is a wrong surname shipped +silently -- a placeholder row, a variant no one bears as a surname -- +since nothing downstream can tell a shipped entry from a meant one. +""" +from __future__ import annotations + +import importlib.util +import sys +import urllib.request +from pathlib import Path +from types import ModuleType + +import pytest + +from nameparser.locales import zh + +_PATH = Path(__file__).parents[2] / "tools" / "zh_surnames.py" + + +def _load() -> ModuleType: + spec = importlib.util.spec_from_file_location("zh_surnames", _PATH) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module # @dataclass looks itself up here + spec.loader.exec_module(module) + return module + + +tool = _load() + + +@pytest.fixture(autouse=True) +def _no_network(monkeypatch: pytest.MonkeyPatch) -> None: + def refuse(*args: object, **kwargs: object) -> None: + raise AssertionError("a test reached the network") + monkeypatch.setattr(urllib.request, "urlopen", refuse) + + +# One row per surname per age band, as the Taiwan file has it, with its +# placeholder row and a compatibility ideograph (U+F9A8, which NFC-folds +# to 令) among them. +_MOI = """年度,ranking,lastname,age,人口數 +112,1,陳,0-14,1 +112,1,陳,15-64,1 +112,1,陳,65+,1 +112,2,其他,0-14,1 +112,3,令,0-14,1 +112,4,衛,0-14,1 +112,5,張簡,0-14,1 +""" + + +def test_the_taiwan_file_reads_as_single_unified_surnames_in_rank_order() -> None: + # 其他 ("other") is a placeholder, 張簡 a compound (compounds are + # curated by hand), and the compatibility ideograph no keyboard + # produces for a name. + assert tool.read_moi(_MOI) == ["陳", "衛"] + + +def test_an_error_page_is_refused_rather_than_read_as_empty() -> None: + with pytest.raises(ValueError, match="columns"): + tool.read_moi("Service unavailable") + + +def _item(qid: str, bearers: int, *labels: str) -> list[dict[str, str]]: + return [{"item": f"http://www.wikidata.org/entity/{qid}", + "n": str(bearers), "label": label} for label in labels] + + +def test_a_label_written_as_surname_x_names_the_character() -> None: + # 易, 鄺 and 閆 are labelled this way on Wikidata; read literally, + # they had dropped out of the ranking. + (item,) = tool.read_wikidata(_item("Q1", 9, "鄺姓", "邝氏")) + assert item.singles() == {"鄺", "邝"} + + +def test_items_rank_by_bearers_and_compounds_are_not_singles() -> None: + rows = (_item("Q1", 5, "卫", "衛") + _item("Q2", 9, "欧阳") + + _item("Q3", 7, "柯")) + assert [i.qid for i in tool.read_wikidata(rows)] == ["Q3", "Q1"] + + +def _forms(surnames: list[object]) -> list[set[str]]: + return [s.forms for s in surnames] # type: ignore[attr-defined] + + +def test_one_surname_in_two_sources_is_one_entry_with_both_ranks() -> None: + items = tool.read_wikidata(_item("Q1", 5, "卫", "衛")) + (surname,) = tool.select(["衛"], items, {}, top=10) + assert surname.forms == {"卫", "衛"} + assert (surname.moi_rank, surname.wikidata_rank) == (1, 1) + + +def test_an_item_bridging_two_entries_merges_them_keeping_the_best_ranks() -> None: + # Two ranked rows that are one surname in two scripts become one + # entry once an item labelled with both joins them, keeping the + # better rank of each source. + items = tool.read_wikidata(_item("Q1", 5, "卫", "衛")) + surnames = tool.select(["陳", "衛", "卫"], items, {}, top=10) + (merged,) = [s for s in surnames if "衛" in s.forms] + assert len(surnames) == 2 and merged.forms == {"卫", "衛"} + assert (merged.moi_rank, merged.wikidata_rank) == (2, 1) + + +def test_a_traditional_form_gains_its_one_simplified_variant() -> None: + assert _forms(tool.select(["關"], [], {"關": {"关"}})) == [{"關", "关"}] + + +def test_a_supplementary_plane_variant_does_not_count_against_the_bmp_one() -> None: + # Unihan gives 蘭 both 兰 and a GB 18030 extension form; a name is + # typed with 兰, so 兰 is the one variant. + variants = tool.read_unihan("U+862D\tkSimplifiedVariant\tU+5170 U+2C795\n") + assert _forms(tool.select(["蘭"], [], variants)) == [{"蘭", "兰"}] + + +def test_a_simplified_form_never_gains_an_unattested_traditional_one() -> None: + # 余's traditional variants include 餘, which is no one's surname; + # only the simplified direction is read from Unihan. + variants = tool.read_unihan("U+4F59\tkTraditionalVariant\tU+4F59 U+9918\n" + "U+9918\tkSimplifiedVariant\tU+4F59\n") + assert _forms(tool.select(["余"], [], variants)) == [{"余"}] + + +def test_each_ranking_contributes_only_its_top() -> None: + surnames = tool.select(["陳", "林", "詹"], [], {}, top=2) + assert _forms(surnames) == [{"陳"}, {"林"}] + + +def test_the_shipped_block_is_what_the_tool_writes_for_its_own_contents() -> None: + # A hand edit inside the generated block, or a block the tool could + # not have written, is caught here without the sources. + text = tool.ZH_MODULE.read_text(encoding="utf-8") + assert tool.rewrite(text, tool.render(list(zh._SINGLE_RANKED))) == text + + +def test_the_block_repeats_no_census_row_and_holds_only_single_ideographs() -> None: + by_hand = set(zh._SINGLE) | set(zh._SINGLE_TRADITIONAL) + assert tool.listed_by_hand(tool.ZH_MODULE.read_text(encoding="utf-8")) == by_hand + assert not by_hand & set(zh._SINGLE_RANKED) + assert all(tool.is_han(c) for c in zh._SINGLE_RANKED) diff --git a/tools/differential/cldr_agreement_locales.json b/tools/differential/cldr_agreement_locales.json index bbe862d8..4882ae5a 100644 --- a/tools/differential/cldr_agreement_locales.json +++ b/tools/differential/cldr_agreement_locales.json @@ -655,33 +655,33 @@ "yue/nativeG/givenFirst": { "family": { "expected": "", - "got": "文傑" + "got": "文" }, "given_names": { "expected": "文傑", - "got": "" + "got": "傑" }, "text": "文傑" }, "yue/nativeG/sorting": { "family": { "expected": "", - "got": "文傑" + "got": "文" }, "given_names": { "expected": "文傑", - "got": "" + "got": "傑" }, "text": "文傑" }, "yue/nativeG/surnameFirst": { "family": { "expected": "", - "got": "文傑" + "got": "文" }, "given_names": { "expected": "文傑", - "got": "" + "got": "傑" }, "text": "文傑" }, @@ -854,33 +854,33 @@ "yue_Hans/nativeG/givenFirst": { "family": { "expected": "", - "got": "文杰" + "got": "文" }, "given_names": { "expected": "文杰", - "got": "" + "got": "杰" }, "text": "文杰" }, "yue_Hans/nativeG/sorting": { "family": { "expected": "", - "got": "文杰" + "got": "文" }, "given_names": { "expected": "文杰", - "got": "" + "got": "杰" }, "text": "文杰" }, "yue_Hans/nativeG/surnameFirst": { "family": { "expected": "", - "got": "文杰" + "got": "文" }, "given_names": { "expected": "文杰", - "got": "" + "got": "杰" }, "text": "文杰" }, @@ -978,33 +978,33 @@ "zh/foreignG/givenFirst": { "family": { "expected": "", - "got": "辛巴达" + "got": "辛" }, "given_names": { "expected": "辛巴达", - "got": "" + "got": "巴达" }, "text": "辛巴达" }, "zh/foreignG/sorting": { "family": { "expected": "", - "got": "辛巴达" + "got": "辛" }, "given_names": { "expected": "辛巴达", - "got": "" + "got": "巴达" }, "text": "辛巴达" }, "zh/foreignG/surnameFirst": { "family": { "expected": "", - "got": "辛巴达" + "got": "辛" }, "given_names": { "expected": "辛巴达", - "got": "" + "got": "巴达" }, "text": "辛巴达" }, @@ -1101,11 +1101,11 @@ "zh/nativeGS/givenFirst": { "family": { "expected": "张", - "got": "安怡张" + "got": "安" }, "given_names": { "expected": "安怡", - "got": "" + "got": "怡张" }, "text": "安怡张" }, @@ -1270,33 +1270,33 @@ "zh_Hant/nativeG/givenFirst": { "family": { "expected": "", - "got": "文傑" + "got": "文" }, "given_names": { "expected": "文傑", - "got": "" + "got": "傑" }, "text": "文傑" }, "zh_Hant/nativeG/sorting": { "family": { "expected": "", - "got": "文傑" + "got": "文" }, "given_names": { "expected": "文傑", - "got": "" + "got": "傑" }, "text": "文傑" }, "zh_Hant/nativeG/surnameFirst": { "family": { "expected": "", - "got": "文傑" + "got": "文" }, "given_names": { "expected": "文傑", - "got": "" + "got": "傑" }, "text": "文傑" }, diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index d18a8191..db29cf13 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -40,6 +40,7 @@ "威廉·莎士比亚, PhD" "威廉·莎士比亚さん" "威廉・莎士比亚" +"小明" "山田 ア゙イ" "山田 エミ" "山田 太郎 (マイケル・ジャクソン)" @@ -99,7 +100,6 @@ "田中社長" "田中部長" "諸葛亮" -"阿明" "陳怡君 二世" "陳怡君二世" "陳怡君，醫學博士" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 2a428f56..1aa3d4ff 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -601,6 +601,8 @@ "山田 花子 旧姓：佐藤" "山田 ﾀﾛｳ" "岸田文雄衆議院議員" +"张简" +"柯洁" "毛·泽东" "毛泽东" "王君" @@ -614,6 +616,7 @@ "田中さん, PhD" "田中社長" "田中部長" +"钟离小凤" "陳子媚 二世" "陳怡君二世" "马丁·路德·金씨" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 51a35c03..042c79b4 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -5487,6 +5487,7 @@ {"name": "小冯·布鲁赫·艾达·科妮莉亚·塞萨尔·马丁·博士教授", "order": "FAMILY_FIRST", "given": "布鲁赫", "middle": "艾达 科妮莉亚 塞萨尔 马丁", "family": "小冯", "suffix": "博士 教授", "initials": "布. 艾. 科. 塞. 马. 小."} {"name": "小彭德威东升议员先生", "order": "FAMILY_FIRST", "family": "小彭德威东升议员", "suffix": "先生", "initials": "小."} {"name": "小德威东升彭议员先生", "family": "小德威东升彭议员", "suffix": "先生", "initials": "小."} +{"name": "小明", "family": "小明", "initials": "小."} {"name": "小杉 啓太", "given": "啓太", "family": "小杉", "initials": "啓. 小."} {"name": "小林宏", "family": "小林宏", "initials": "小."} {"name": "小林由佳", "family": "小林由佳", "initials": "小."} @@ -5518,6 +5519,7 @@ {"name": "引地良一", "family": "引地良一", "initials": "引."} {"name": "张伟", "family": "张伟", "initials": "张."} {"name": "张安怡", "order": "FAMILY_FIRST", "family": "张安怡", "initials": "张."} +{"name": "张简", "family": "张简", "initials": "张."} {"name": "张翰", "family": "张翰", "initials": "张."} {"name": "张雅婷", "order": "FAMILY_FIRST", "family": "张雅婷", "initials": "张."} {"name": "張瑞", "family": "張瑞", "initials": "張."} @@ -5667,7 +5669,7 @@ {"name": "郭靜", "family": "郭靜", "initials": "郭."} {"name": "鈴木亀蔵", "family": "鈴木亀蔵", "initials": "鈴."} {"name": "鎌田 弥彦", "given": "弥彦", "family": "鎌田", "initials": "弥. 鎌."} -{"name": "阿明", "family": "阿明", "initials": "阿."} +{"name": "钟离小凤", "family": "钟离小凤", "initials": "钟."} {"name": "陈·怡君·达印", "order": "FAMILY_FIRST", "given": "怡君", "middle": "达印", "family": "陈", "initials": "怡. 达. 陈."} {"name": "陈·陈·怡君·教授，小·达印·博士", "order": "FAMILY_FIRST", "given": "小", "middle": "达印", "family": "陈 陈 怡君", "suffix": "教授, 博士", "initials": "小. 达. 陈. 陈. 怡."} {"name": "陈俊年杰思", "order": "FAMILY_FIRST", "family": "陈俊年杰思", "initials": "陈."} diff --git a/tools/differential/wikidata_native_agreement_locales.json b/tools/differential/wikidata_native_agreement_locales.json index dbb05b3e..0a35e32b 100644 --- a/tools/differential/wikidata_native_agreement_locales.json +++ b/tools/differential/wikidata_native_agreement_locales.json @@ -1,15 +1,4 @@ { - "Q103785817/zh-tw/surnameFirst": { - "family": { - "expected": "詹", - "got": "詹桓" - }, - "given_names": { - "expected": "桓", - "got": "" - }, - "text": "詹桓" - }, "Q111081424/ja-jp/surnameFirst": { "family": { "expected": "安達", @@ -20,60 +9,5 @@ "got": "達禎" }, "text": "安達禎" - }, - "Q127605824/zh-tw/surnameFirst": { - "family": { - "expected": "卫", - "got": "卫惠林" - }, - "given_names": { - "expected": "惠林", - "got": "" - }, - "text": "卫惠林" - }, - "Q16262307/zh-cn/surnameFirst": { - "family": { - "expected": "柯", - "got": "柯洁" - }, - "given_names": { - "expected": "洁", - "got": "" - }, - "text": "柯洁" - }, - "Q19825302/zh-cn/surnameFirst": { - "family": { - "expected": "茅", - "got": "茅祖權" - }, - "given_names": { - "expected": "祖權", - "got": "" - }, - "text": "茅祖權" - }, - "Q39040747/zh-cn/surnameFirst": { - "family": { - "expected": "华", - "got": "华楠" - }, - "given_names": { - "expected": "楠", - "got": "" - }, - "text": "华楠" - }, - "Q7927961/zh-cn/surnameFirst": { - "family": { - "expected": "文", - "got": "文强" - }, - "given_names": { - "expected": "强", - "got": "" - }, - "text": "文强" } } diff --git a/tools/zh_surnames.py b/tools/zh_surnames.py new file mode 100644 index 00000000..2bd34c59 --- /dev/null +++ b/tools/zh_surnames.py @@ -0,0 +1,320 @@ +"""Build the single-character surname list locales.ZH ships past the census. + +WHY THIS IS IN THE TREE. The ZH pack divides an unspaced Chinese name +after a listed surname and leaves it whole otherwise (rules/W1). Up to +2.4 it listed the 2020 PRC census top 100, and 6 of the 80 real Chinese +names in the native-script corpus fell outside it (柯洁, 詹桓, #695). The +census publishes nothing past rank 100, so the list is extended from two +openly licensed rankings instead, and this script is how the extension +is re-derived rather than transcribed by hand: + +- Taiwan, Ministry of the Interior, Department of Household + Registration (內政部戶政司), surname ranking by population, 2023 + (data.gov.tw dataset 126774; Open Government Data License, version 1, + which asks for attribution). Traditional script, Taiwan's own + frequencies. +- Wikidata (CC0): items that are an instance of "Chinese family name" + (Q1093580), ranked by how many people carry them as family name + (P734), with their Chinese labels. Both scripts; it skews toward + notable people and counts Korean and Vietnamese bearers of the same + character. + +Each ranking contributes its top N single-character surnames (N=300 by +decision, decisions/W1 of 2026-10-18), and the union is what ships, so +neither region's bias decides alone. Where a source gives one script +only, Unicode's Unihan variant table supplies the other, in ONE +direction: a traditional character's simplified form is added when +Unihan gives exactly one in the Basic Multilingual Plane. The reverse is one-to-many (余 has 餘 among its +traditional variants, and 餘 is no one's surname), so a traditional form +ships only where a source attests it -- a Wikidata item labelled with +both, or the Taiwan ranking itself. + +Two-character surnames are NOT taken from here. Taiwan's multi-character +rows are mostly married double surnames (王李, 李林), and an entry like +李林 would cut 李林伟 into 李林 + 伟, so compounds stay a hand-curated +list in zh.py. + + uv run python tools/zh_surnames.py # the selection, with ranks + uv run python tools/zh_surnames.py --write # rewrite zh.py's block + uv run python tools/zh_surnames.py --refresh # fetch the sources again + +Sources are cached outside the repository, in $NAMEPARSER_TOOLS_CACHE +or ~/.cache/nameparser-tools (shared with name_usage.py); --refresh +fetches them again, and the Wikidata ranking moves when it does. +""" +from __future__ import annotations + +import argparse +import csv +import io +import json +import re +import sys +import unicodedata +import urllib.request +import zipfile +from dataclasses import dataclass, field +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) +import name_usage # noqa: E402 (a sibling script, not a package) + +ZH_MODULE = Path(__file__).parents[1] / "nameparser" / "locales" / "zh.py" +TOP = 300 + +MOI_URL = ( + "https://opdadm.moi.gov.tw/api/v1/no-auth/resource/api/dataset/" + "5D318346-EDFF-42C7-B914-946665D483F5/resource/" + "E4A6518E-FF4C-4925-8D59-48ECE4401820/download") +MOI_FILE = "moi_surnames_2023.csv" +UNIHAN_URL = "https://www.unicode.org/Public/16.0.0/ucd/Unihan.zip" +UNIHAN_FILE = "unihan_16.0.0.zip" +UNIHAN_MEMBER = "Unihan_Variants.txt" +WIKIDATA_FILE = "wikidata_chinese_family_names.json" + +#: Chinese label languages; Wikidata keeps each script under several. +LABEL_LANGUAGES = ("zh", "zh-hans", "zh-hant", "zh-cn", "zh-tw", "zh-hk", + "zh-sg", "zh-mo") +WIKIDATA_QUERY = f""" +SELECT ?item ?n ?label WHERE {{ + {{ SELECT ?item (COUNT(DISTINCT ?person) AS ?n) WHERE {{ + ?item wdt:P31 wd:Q1093580 . ?person wdt:P734 ?item . + }} GROUP BY ?item }} + OPTIONAL {{ ?item rdfs:label ?label + FILTER(LANG(?label) IN ({", ".join(f'"{x}"' for x in LABEL_LANGUAGES)})) }} +}}""" + +BEGIN = "# -- BEGIN generated by tools/zh_surnames.py; edit the tool, not this block --" +END = "# -- END generated --" + +#: Some items are labelled "X姓" or "X氏" ("surname X"), which is how +#: 易, 鄺 and 閆 appear; the label names the single character. +_SURNAME_WORD = re.compile(r"^(.)[姓氏]$") + + +def is_han(char: str) -> bool: + """A unified ideograph. Compatibility ideographs and private-use + codepoints (about 60 rows of the Taiwan file) are left out: no + keyboard produces them for a name, and the compatibility block + NFC-folds to a unified twin anyway.""" + return len(char) == 1 and unicodedata.name(char, "").startswith( + "CJK UNIFIED IDEOGRAPH") + + +def read_moi(csv_text: str) -> list[str]: + """Taiwan's single-character surnames in rank order. The file has + one row per surname per age band, all carrying the surname's rank, + and a placeholder row 其他 ("other") that is not a surname.""" + reader = csv.DictReader(io.StringIO(csv_text.lstrip(""))) + if not reader.fieldnames or not {"ranking", "lastname"} <= set( + reader.fieldnames): + raise ValueError(f"unexpected Taiwan surname columns: " + f"{reader.fieldnames}") + rank: dict[str, int] = {} + for row in reader: + name = row["lastname"].strip() + rank.setdefault(name, int(row["ranking"])) + if not rank: + raise ValueError("the Taiwan surname file holds no rows") + return [name for name in sorted(rank, key=lambda n: (rank[n], n)) + if is_han(name)] + + +@dataclass +class Item: + qid: str + bearers: int + labels: set[str] = field(default_factory=set) + + def singles(self) -> set[str]: + out = set() + for label in self.labels: + match = _SURNAME_WORD.match(label) + word = match.group(1) if match else label + if is_han(word): + out.add(word) + return out + + +def read_wikidata(rows: list[dict[str, str]]) -> list[Item]: + """Chinese family-name items with at least one single-character + label, most-borne first.""" + items: dict[str, Item] = {} + for row in rows: + qid = row["item"].rsplit("/", 1)[-1] + item = items.setdefault(qid, Item(qid, int(row["n"]))) + if row.get("label"): + item.labels.add(row["label"]) + ranked = [item for item in items.values() if item.singles()] + return sorted(ranked, key=lambda item: (-item.bearers, item.qid)) + + +def read_unihan(text: str) -> dict[str, set[str]]: + """Each traditional character's simplified variants.""" + simplified: dict[str, set[str]] = {} + for line in text.splitlines(): + if line.startswith("#") or "\tkSimplifiedVariant\t" not in line: + continue + source, _, targets = line.split("\t") + char = chr(int(source[2:], 16)) + simplified[char] = {chr(int(t[2:], 16)) for t in targets.split()} + return simplified + + +@dataclass +class Surname: + forms: set[str] + moi_rank: int | None = None + wikidata_rank: int | None = None + + @property + def rank(self) -> int: + return min(r for r in (self.moi_rank, self.wikidata_rank) + if r is not None) + + +def select(moi: list[str], items: list[Item], + simplified: dict[str, set[str]], top: int = TOP) -> list[Surname]: + """The union of each ranking's top `top` single-character surnames, + as surnames (one per set of script forms), best rank first. A + character two sources share joins their entries, so 衛 from Taiwan + and the item labelled 卫 and 衛 are one surname.""" + surnames: list[Surname] = [] + by_form: dict[str, Surname] = {} + + def add(forms: set[str], **ranks: int) -> None: + found = {id(by_form[f]): by_form[f] for f in forms if f in by_form} + if found: + target, *others = found.values() + for other in others: # two entries this one bridges + target.forms |= other.forms + for key in ("moi_rank", "wikidata_rank"): + mine, theirs = getattr(target, key), getattr(other, key) + if theirs is not None and (mine is None or theirs < mine): + setattr(target, key, theirs) + surnames.remove(other) + else: + target = Surname(set()) + surnames.append(target) + target.forms |= forms + for key, value in ranks.items(): + if getattr(target, key) is None: + setattr(target, key, value) + for form in target.forms: + by_form[form] = target + + for i, char in enumerate(moi[:top], 1): + add({char}, moi_rank=i) + for i, item in enumerate(items[:top], 1): + add(item.singles(), wikidata_rank=i) + for surname in list(surnames): + for form in list(surname.forms): + # Unihan also names supplementary-plane forms only GB 18030 + # extensions spell (蘭 → 兰 or 𬞕), an alternative to the BMP + # spelling a name is typed with. A surname whose ONLY + # spelling is supplementary (Taiwan's 𡍼) ships as ranked. + variants = {v for v in simplified.get(form, ()) if ord(v) < 0x10000} + if len(variants) == 1 and not variants <= surname.forms: + add(variants | {form}) + return sorted(surnames, key=lambda s: (s.rank, min(s.forms))) + + +def additions(surnames: list[Surname], listed: set[str]) -> list[str]: + """The forms to generate: every form of every selected surname that + the hand-kept census rows do not already hold, in rank order.""" + out: list[str] = [] + for surname in surnames: + out.extend(sorted(surname.forms - listed - set(out))) + return out + + +def render(forms: list[str]) -> str: + rows = [", ".join(f'"{c}"' for c in forms[i:i + 10]) + for i in range(0, len(forms), 10)] + body = "\n".join(f" {row}," for row in rows) + return f"{BEGIN}\n_SINGLE_RANKED = (\n{body}\n)\n{END}" + + +def rewrite(module_text: str, block: str) -> str: + start, end = module_text.find(BEGIN), module_text.find(END) + if start < 0 or end < start: + raise ValueError(f"{ZH_MODULE} has no generated block to replace") + return module_text[:start] + block + module_text[end + len(END):] + + +def listed_by_hand(module_text: str) -> set[str]: + """The census rows zh.py keeps by hand, read from its source so the + script never imports the package it writes.""" + start = module_text.find(BEGIN) + head = module_text if start < 0 else module_text[:start] + head = head.split("_COMPOUND = (")[0] + return {c for c in re.findall(r'"(.)"', head) if is_han(c)} + + +# -- sources -------------------------------------------------------------- + +def _fetch(url: str) -> bytes: + request = urllib.request.Request( + url, headers={"User-Agent": name_usage.USER_AGENT}) + with urllib.request.urlopen(request, timeout=120) as response: + data: bytes = response.read() + return data + + +def load_moi(refresh: bool) -> list[str]: + path = name_usage.cache_dir() / MOI_FILE + if refresh or not path.exists(): + text = _fetch(MOI_URL).decode("utf-8-sig") + read_moi(text) # refuse to cache an error page + path.write_text(text, encoding="utf-8") + return read_moi(path.read_text(encoding="utf-8")) + + +def load_wikidata(refresh: bool) -> list[Item]: + path = name_usage.cache_dir() / WIKIDATA_FILE + if refresh or not path.exists(): + rows = name_usage._bindings(name_usage._sparql(WIKIDATA_QUERY)) + if not read_wikidata(rows): + raise ValueError("the Wikidata query returned no surnames") + path.write_text(json.dumps(rows, ensure_ascii=False), + encoding="utf-8") + return read_wikidata(json.loads(path.read_text(encoding="utf-8"))) + + +def load_unihan(refresh: bool) -> dict[str, set[str]]: + path = name_usage.cache_dir() / UNIHAN_FILE + if refresh or not path.exists(): + name_usage._download(UNIHAN_URL, path, UNIHAN_MEMBER) + with zipfile.ZipFile(path) as zf: + return read_unihan(zf.read(UNIHAN_MEMBER).decode("utf-8")) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + parser.add_argument("--write", action="store_true", + help="rewrite the generated block in zh.py") + parser.add_argument("--refresh", action="store_true", + help="fetch the sources again") + parser.add_argument("--top", type=int, default=TOP) + args = parser.parse_args(argv) + surnames = select(load_moi(args.refresh), load_wikidata(args.refresh), + load_unihan(args.refresh), args.top) + module_text = ZH_MODULE.read_text(encoding="utf-8") + listed = listed_by_hand(module_text) + forms = additions(surnames, listed) + for s in surnames: + new = "+" if s.forms - listed else " " + print(f"{new} {s.rank:4} Taiwan {s.moi_rank or '-':>4} " + f"Wikidata {s.wikidata_rank or '-':>4} " + f"{' '.join(sorted(s.forms))}") + print(f"{len(surnames)} surnames; {len(forms)} forms not in the census " + "rows", file=sys.stderr) + if args.write: + ZH_MODULE.write_text(rewrite(module_text, render(forms)), + encoding="utf-8") + print(f"wrote {ZH_MODULE}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main())