You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Should halfwidth katakana (タロウ) read family-first like full-width katakana? #594
parse("山田 エミ") reads family 山田, given エミ: kanji with kana is licensed as Japanese and assigned family-first. The same name written with halfwidth katakana, parse("山田 タロウ"), reads given 山田, family タロウ, which is the positional reading. Script.KATAKANA covers U+30A0–30FF only, so halfwidth kana (U+FF65–FF9F) is unclassified and the name never reaches the kana license.
The same gap reaches the segmenter. A halfwidth-katakana word beside a Han token doesn't count as a second East Asian word, so "高橋一郎 タロウ" still asks the segmenter to divide 高橋一郎, where "高橋一郎 タロウ" doesn't.
Halfwidth katakana still turns up in legacy Japanese data (old JIS systems, bank and payroll exports). The question is whether to treat it as katakana for script classification (NFKC maps タロウ → タロウ), and if so, whether that belongs in classification (widen the range) or as normalization at tokenize. Note that U+FF65, the halfwidth nakaguro, is already a token separator.
Found during the docs review in #593. The docs describe the license as "kanji and kana other than katakana alone" and leave halfwidth unmentioned until this is decided.
Activity
test: pin the span's ゚/ー edges, the unspaced twin and the ja adapter (#…
docs: fix what the PR review found in #594's comments and prose
test: say which of the span's codepoints the ペーター row pins (#594)
docs: AGENTS.md no longer says halfwidth data writes every name in ka…
Read halfwidth katakana as katakana (derek73#594) (derek73#595)
Drop a voicing mark only after a kana; fix what review found (derek73…
Read halfwidth corner brackets as a nickname delimiter (derek73#597)
Fix what the review of derek73#597 found