Repository navigation
Extend the ZH surname list past the census top 100 (#695, #657) - #730
Merged
Merged
Conversation
Union of two openly licensed rankings' top 300 single-character surnames -- Taiwan's Ministry of the Interior ranking (2023, OGDL v1) and Wikidata's Chinese family names by bearers (CC0) -- written into zh.py by tools/zh_surnames.py, which derives simplified forms from Unihan in one direction only. Compounds stay hand-curated: 26 more, 钟离 and 段干 among them, admitted on an attested bearer and a first-character check (#657). The native-script corpus's Chinese rows go from 74/80 to 80/80. Case rows and segmenter tests that used 阿 as an unlisted surname move to 小, which no source ranks; 司马 and 諸葛 now win a recorded fork, their first characters being listed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decisions/W1 weighs the cutoff and the sources and records the accepted costs (Japanese names read under ZH split wrongly more often: 2 -> 11 of 60 corpus rows; a listed compound reads a two-character name whole); vocabulary-exclusions keeps married double surnames out of the compounds. User docs and the release log say what the list holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cutoff table is re-measured through the shipped select(), Unihan step included (Taiwan alone at 300 reads 79, not 77; both at 400 give 14 wrong, not 13), and says why the union was kept although Wikidata alone matched it on the corpus: 46 surnames, 关 and 区 among them, reach the list only through Taiwan. The compound cost covers every name opening with a compound (张简明), not only two-character ones. The tail figure's recipe now names the order it ranks by, and the new entry moves above Declined:, with the 2026-08 one. The bridge branch of select() gains a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
300 is a judgment, not the smallest cut the corpus allows (212 reads all 80 rows with 7 Japanese wrong splits); the 张简 cost is on record (25 Wikidata people tagged 张 whose label opens with 张简, 9 of them two-character 张简 with given name 简); 茅's position is counted in the recipe's order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The compound test also asks a first-character bearer's stated given name, and a compound that fails it still ships when its living bearers outnumber those readings by orders of magnitude: 張簡 (9,162 in the Taiwan file against 9 people named 張 + 簡) stays, 东野 (none against 2) leaves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #730 +/- ##
=======================================
Coverage 99.04% 99.04%
=======================================
Files 46 46
Lines 4585 4587 +2
=======================================
+ Hits 4541 4543 +2
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #695. Also covers #657 (closed, folded into #695): the missing common surnames and the missing compounds.
What changes
locales.ZHdivided an unspaced Chinese name only after one of the 2020 PRC census top-100 surnames, so 柯洁 stayed whole as family柯洁. The single-character list is now the census rows plus the union of two openly licensed rankings' top 300:zh.py's docstring).tools/zh_surnames.pyfetches both and writes a generated block intonameparser/locales/zh.py, so the list is re-derived rather than transcribed. It also uses Unihan's variant table to add simplified forms, in one direction only: 余's traditional variants include 餘, which nobody bears as a surname.Compounds stay hand-curated. 25 were added (钟离 and 段干 among them) under an admission test stated in
zh.py, and married double surnames like 李林 are excluded.Measured
The default parse moves no corpus name: the pack is opt-in, and the snapshot only gains the new example names.
Decisions taken in the session (recorded in
decisions/W1(unspaced CJK division))locales.ZHdeclares the data Chinese, and Japanese data haslocales.JA. So a common Chinese name left whole is a failure, and more Japanese wrong splits under ZH are an accepted cost.钟离小凤gives family钟#657), 颜, 东 and 兰.Accepted costs
安藤一郎→ family安).安娜→安+娜).张简明→张简+明).阿明now reads阿+明; 阿 is a Taiwan surname at rank 282.Also in the diff
zhrow, and a release-log bullet.Review
Two design-docs review rounds and one code review ran; their findings are fixed in commits 5945de4, 6fad6ae and the last commit. The suite passes without the
jaextra installed, the path CI's build jobs take. mypy, ruff and the Sphinx doctest build are clean.🤖 Generated with Claude Code