Skip to content

Extend the ZH surname list past the census top 100 (#695, #657) - #730

Merged
derek73 merged 5 commits into
masterfrom
feat/issue-695-zh-surnames
Oct 19, 2026
Merged

derek73 merged 5 commits into
masterfrom
feat/issue-695-zh-surnames

Conversation

@derek73

@derek73 derek73 commented Oct 19, 2026

Copy link
Copy Markdown
Owner

Closes #695. Also covers #657 (closed, folded into #695): the missing common surnames and the missing compounds.

What changes

locales.ZH divided an unspaced Chinese name only after one of the 2020 PRC census top-100 surnames, so 柯洁 stayed whole as family 柯洁. The single-character list is now the census rows plus the union of two openly licensed rankings' top 300:

  • Taiwan's Ministry of the Interior surname ranking (2023, data.gov.tw 126774, Open Government Data License v1; attribution is in zh.py's docstring).
  • Wikidata's Chinese family names ranked by bearers (CC0).

tools/zh_surnames.py fetches both and writes a generated block into nameparser/locales/zh.py, so the list is re-derived rather than transcribed. It also uses Unihan's variant table to add simplified forms, in one direction only: 余's traditional variants include 餘, which nobody bears as a surname.

Compounds stay hand-curated. 25 were added (钟离 and 段干 among them) under an admission test stated in zh.py, and married double surnames like 李林 are excluded.

柯洁      →  family '柯', given '洁'      (was family '柯洁')
钟离小凤  →  family '钟离', given '小凤'  (was family '钟')

Measured

before after
Chinese rows of the native-script corpus right 74/80 80/80
Japanese rows split wrongly under ZH 2/60 11/60
CLDR ZH locale view 42/96 42/96 (moved renderings wrong before and after)

The default parse moves no corpus name: the pack is opt-in, and the snapshot only gains the new example names.

Decisions taken in the session (recorded in decisions/W1 (unspaced CJK division))

  • The pack is for Chinese names. Applying locales.ZH declares the data Chinese, and Japanese data has locales.JA. So a common Chinese name left whole is a failure, and more Japanese wrong splits under ZH are an accepted cost.
  • One pack, cut at 300. A full-roster pack and a "split after the first character" mode were both declined. Taiwan's tail past 300 covers 0.33% of its population and holds the characters that open Japanese surnames. The corpus doesn't force 300: a cut of 212 reads all 80 rows with 7 Japanese wrong splits. 300 leaves a margin over a corpus of public figures.
  • Both sources, not one. On the corpus alone, Wikidata's list matched the union with 2 fewer Japanese wrong splits. But 46 surnames reach the list only through Taiwan, including 关 and 区 (both named in The Chinese surname list misses common surnames, and splits compound ones wrongly: 钟离小凤 gives family 钟 #657), 颜, 东 and 兰.
  • The compound test weighs living bearers. 張簡 stays: 9,162 bearers in the Taiwan file against 9 Wikidata people named 張 + 簡, who now read whole. 东野 is left out (no bearers in the Taiwan file, against 2 such people), and so is 成公 (成公亮 is 成 + 公亮).

Accepted costs

  • A Japanese name read under ZH splits after its first character more often (安藤一郎 → family 安).
  • A transcribed foreign name written without the 间隔号 splits too (安娜 → 安 + 娜).
  • A listed compound wins over its first character in every name that opens with it (张简明 → 张简 + 明).
  • The nickname style 阿明 now reads 阿 + 明; 阿 is a Taiwan surname at rank 282.

Also in the diff

  • Five tests and a case row used 阿 as an unlisted surname; they now use 小.
  • 司马相如 and 諸葛亮 now record a segmentation ambiguity, because 司 and 諸 are listed.
  • Re-recorded agreement files, CJK and rules corpora, and the snapshot.
  • New W1 examples, a vocabulary-exclusions block for married double surnames, the user docs' zh row, and a release-log bullet.

Review

Two design-docs review rounds and one code review ran; their findings are fixed in commits 5945de4, 6fad6ae and the last commit. The suite passes without the ja extra installed, the path CI's build jobs take. mypy, ruff and the Sphinx doctest build are clean.

🤖 Generated with Claude Code

derek73 and others added 5 commits October 18, 2026 19:39
Union of two openly licensed rankings' top 300 single-character
surnames -- Taiwan's Ministry of the Interior ranking (2023, OGDL v1)
and Wikidata's Chinese family names by bearers (CC0) -- written into
zh.py by tools/zh_surnames.py, which derives simplified forms from
Unihan in one direction only. Compounds stay hand-curated: 26 more,
钟离 and 段干 among them, admitted on an attested bearer and a
first-character check (#657).

The native-script corpus's Chinese rows go from 74/80 to 80/80. Case
rows and segmenter tests that used 阿 as an unlisted surname move to 小,
which no source ranks; 司马 and 諸葛 now win a recorded fork, their
first characters being listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decisions/W1 weighs the cutoff and the sources and records the
accepted costs (Japanese names read under ZH split wrongly more often:
2 -> 11 of 60 corpus rows; a listed compound reads a two-character name
whole); vocabulary-exclusions keeps married double surnames out of the
compounds. User docs and the release log say what the list holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cutoff table is re-measured through the shipped select(), Unihan
step included (Taiwan alone at 300 reads 79, not 77; both at 400 give
14 wrong, not 13), and says why the union was kept although Wikidata
alone matched it on the corpus: 46 surnames, 关 and 区 among them,
reach the list only through Taiwan. The compound cost covers every
name opening with a compound (张简明), not only two-character ones. The
tail figure's recipe now names the order it ranks by, and the new
entry moves above Declined:, with the 2026-08 one. The bridge branch
of select() gains a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
300 is a judgment, not the smallest cut the corpus allows (212 reads
all 80 rows with 7 Japanese wrong splits); the 张简 cost is on record
(25 Wikidata people tagged 张 whose label opens with 张简, 9 of them
two-character 张简 with given name 简); 茅's position is counted in the
recipe's order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The compound test also asks a first-character bearer's stated given
name, and a compound that fails it still ships when its living bearers
outnumber those readings by orders of magnitude: 張簡 (9,162 in the
Taiwan file against 9 people named 張 + 簡) stays, 东野 (none against
2) leaves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 2.5 milestone Oct 19, 2026
@derek73 derek73 self-assigned this Oct 19, 2026
@codecov

codecov Bot commented Oct 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.04%. Comparing base (c726650) to head (ba53232).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #730   +/-   ##
=======================================
  Coverage   99.04%   99.04%           
=======================================
  Files          46       46           
  Lines        4585     4587    +2     
=======================================
+ Hits         4541     4543    +2     
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@derek73
derek73 merged commit 8915679 into master Oct 19, 2026
12 checks passed
@derek73
derek73 deleted the feat/issue-695-zh-surnames branch October 19, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

柯洁 reads as one family name under locales.ZH: should the Chinese surname list go past the census top 100?

1 participant