Repository navigation
Read a decomposed initial, connective and CJK word as its composed twin - #675
Merged
Merged
Conversation
…in (#665) The initial shape (`\w\.`, the public v1 `initial` pattern) and the single-letter connective tests counted code points, so a letter typed as base plus combining mark was not one letter: decomposed `É. y É. Smith` read given `É. y É.`, and decomposed `John й Smith` read the whole name as the given name (a 2.3.0 regression, when the NFC lookup fold let `й` match the conjunction vocabulary). `is_initial_shaped` now asks the composed spelling, as `_render` already did (#542), keeping the pattern verbatim; the three width tests in classify and pieces count composed letters. Composing the shape alone made a decomposed hangul `씨.` an initial, because the #320 script veto read conjoining jamo, so `in_initialless_script` composes too -- which also fixes decomposed `김. 민준` reading `김.` as a title in every release. The twin test gains 16 rows and compares ambiguity kinds and initials(); its docstring records the per-site negative controls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules.md#H2 states that the script veto reads the composed spelling and carries a decomposed `김. 민준` example (title ""). - rules.md#P3 narrows "one letter" to a letter-plus-mark that composes to one character, which is what the width tests count. - test_rules_doc's byte guard covers every decomposed example, not just H2's `inž.`. - decisions.md: correct the KNOWN LIMITS field for `John й Smith` (given, not family), the composed_len gating, and the q́ limit. - The release-log bullet keeps one example; AGENTS.md's 193-frame count is dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #675 +/- ##
=======================================
Coverage 99.02% 99.03%
=======================================
Files 46 46
Lines 4534 4541 +7
=======================================
+ Hits 4490 4497 +7
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #665.
A name typed decomposed (NFD: a base letter plus a combining mark) read differently from its composed spelling in three places. Each one asked a shape question of code points instead of letters.
йregressed in 2.3.0. That release's NFC lookup fold let the decomposedйmatch the conjunction vocabulary, but the width tests still counted two code points. 1.4.0 through 2.2.0 read it as middleй, so this restores their reading.É.and김.were wrong in every release from 1.4.0.All three were checked on the 1.4.0, 2.0.0, 2.2.0, 2.3.0, 2.4.0 and 2.4.1 wheels.
Changes
_vocab.is_initial_shapedcomposes a non-ASCII word before matching_INITIAL, as_renderalready did (capitalize()breaks a decomposed (NFD) word at its combining mark:josé garcíatyped NFD givesJosé GarcíA#542). I composed the word rather than widening the class, so the public v1initialregex stays verbatim, and the result matches the composed spelling by construction._vocab.in_initialless_script, the A trailing period stops a CJK honorific being recognized ("김민준, 씨." → given "씨.") #320 script veto, composes too. Decomposed hangul is conjoining jamo, which no_NO_INITIALSrange holds. Composing only the shape made a decomposed씨.an initial (김민준, 씨.read title씨.), andtest_suffix_lookup_reads_every_full_stopcaught it._classify's connective fork and its one-case report, and_pieces' three-word carve-out.김. 민준 → title=""example.Cost
ASCII text costs no new frames. Composed non-ASCII text pays nothing for the width tests, and one frame per word that reaches the veto. The veto's matcher is now called through a wrapper:
É. y É. SmithJanez Ž. Novak田中. 太郎The reference name and
Smith, Joséare unchanged. I kept the wrapper rather than composing at the veto's four call sites, because a missing compose step is exactly this bug.Tests
test_a_decomposed_abbreviation_reads_as_its_composed_twingains 16 rows. It now compares ambiguity kinds andinitials()as well as roles, sinceJohn й Smithalso differed in its report. The negative control, re-taken after the last row was added against f9b2b81'snameparser/, fails 41 of 156. The per-site breakdown is in the docstring: initial 24,_pieces12, each_classifysite 1, and the veto 20.test_rules_doc's byte guard now covers every decomposed rules.md example, not just H2'sinž..Measured moves (
snapshot.py --since v2.4.1): two new corpus names from the rules examples. No existing corpus name moved.Not covered
Unspaced decomposed hangul is still not split into surname and given name, as documented in docs/usage.rst. So
김민준. 이서연keeps differing by encoding.A caller-added one-letter connective with no precomposed form (
q́) counts as two letters. P3 states this ("composes to one character"), and decisions.md records it.Review
Full suite passes with namedivider absent (matching CI's build jobs). mypy and ruff are clean.
🤖 Generated with Claude Code