Skip to content

Read a decomposed initial, connective and CJK word as its composed twin - #675

Merged
derek73 merged 2 commits into
masterfrom
fix/issue-665-nfd-initial-conjunction
Oct 14, 2026
Merged

derek73 merged 2 commits into
masterfrom
fix/issue-665-nfd-initial-conjunction

Conversation

@derek73

@derek73 derek73 commented Oct 14, 2026

Copy link
Copy Markdown
Owner

Closes #665.

A name typed decomposed (NFD: a base letter plus a combining mark) read differently from its composed spelling in three places. Each one asked a shape question of code points instead of letters.

                  decomposed, before                     now (= composed)
John й Smith      given 'John й Smith' + given-or-family given 'John', middle 'й', family 'Smith'
É. y É. Smith     given 'É. y É.'                        given 'É.', middle 'y É.'
김. 민준           title '김.'                             family '김.', given '민준'
  • й regressed in 2.3.0. That release's NFC lookup fold let the decomposed й match the conjunction vocabulary, but the width tests still counted two code points. 1.4.0 through 2.2.0 read it as middle й, so this restores their reading.
  • É. and 김. were wrong in every release from 1.4.0.

All three were checked on the 1.4.0, 2.0.0, 2.2.0, 2.3.0, 2.4.0 and 2.4.1 wheels.

Changes

  • _vocab.is_initial_shaped composes a non-ASCII word before matching _INITIAL, as _render already did (capitalize() breaks a decomposed (NFD) word at its combining mark: josé garcía typed NFD gives José GarcíA #542). I composed the word rather than widening the class, so the public v1 initial regex stays verbatim, and the result matches the composed spelling by construction.
  • _vocab.in_initialless_script, the A trailing period stops a CJK honorific being recognized ("김민준, 씨." → given "씨.") #320 script veto, composes too. Decomposed hangul is conjoining jamo, which no _NO_INITIALS range holds. Composing only the shape made a decomposed 씨. an initial (김민준, 씨. read title 씨.), and test_suffix_lookup_reads_every_full_stop caught it.
  • Three single-letter width tests now count composed letters: _classify's connective fork and its one-case report, and _pieces' three-word carve-out.
  • Docs:
    • rules.md#P3 gets the encoding statement and two decomposed examples.
    • rules.md#H2's veto clause gets a sentence and a decomposed 김. 민준 → title="" example.
    • decisions.md#nfd-shapes closes its two known limits and corrects one field.
    • AGENTS.md's encoding paragraph says to compose the whole predicate.
    • Release-log bullet.

Cost

ASCII text costs no new frames. Composed non-ASCII text pays nothing for the width tests, and one frame per word that reaches the veto. The veto's matcher is now called through a wrapper:

Name (composed) Frames before → after
É. y É. Smith 305 → 307
Janez Ž. Novak 237 → 238
田中. 太郎 207 → 208

The reference name and Smith, José are unchanged. I kept the wrapper rather than composing at the veto's four call sites, because a missing compose step is exactly this bug.

Tests

test_a_decomposed_abbreviation_reads_as_its_composed_twin gains 16 rows. It now compares ambiguity kinds and initials() as well as roles, since John й Smith also differed in its report. The negative control, re-taken after the last row was added against f9b2b81's nameparser/, fails 41 of 156. The per-site breakdown is in the docstring: initial 24, _pieces 12, each _classify site 1, and the veto 20.

test_rules_doc's byte guard now covers every decomposed rules.md example, not just H2's inž..

Measured moves (snapshot.py --since v2.4.1): two new corpus names from the rules examples. No existing corpus name moved.

Not covered

Unspaced decomposed hangul is still not split into surname and given name, as documented in docs/usage.rst. So 김민준. 이서연 keeps differing by encoding.

A caller-added one-letter connective with no precomposed form (q́) counts as two letters. P3 states this ("composes to one character"), and decisions.md records it.

Review

  • Code review found no issues. 283,710 parses confirmed composed and ASCII input is unchanged, and NFD-vs-NFC sweeps showed 0 differences, against 11,027 and 5,892 on the parent.
  • The first docs review found six issues, all fixed in d390685.
  • A second review of that fix commit found none.

Full suite passes with namedivider absent (matching CI's build jobs). mypy and ruff are clean.

🤖 Generated with Claude Code

derek73 and others added 2 commits October 13, 2026 03:06
…in (#665)

The initial shape (`\w\.`, the public v1 `initial` pattern) and the
single-letter connective tests counted code points, so a letter typed
as base plus combining mark was not one letter: decomposed `É. y É.
Smith` read given `É. y É.`, and decomposed `John й Smith` read the
whole name as the given name (a 2.3.0 regression, when the NFC lookup
fold let `й` match the conjunction vocabulary).

`is_initial_shaped` now asks the composed spelling, as `_render` already
did (#542), keeping the pattern verbatim; the three width tests in
classify and pieces count composed letters. Composing the shape alone
made a decomposed hangul `씨.` an initial, because the #320 script veto
read conjoining jamo, so `in_initialless_script` composes too -- which
also fixes decomposed `김. 민준` reading `김.` as a title in every
release.

The twin test gains 16 rows and compares ambiguity kinds and initials();
its docstring records the per-site negative controls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rules.md#H2 states that the script veto reads the composed spelling
  and carries a decomposed `김. 민준` example (title "").
- rules.md#P3 narrows "one letter" to a letter-plus-mark that composes
  to one character, which is what the width tests count.
- test_rules_doc's byte guard covers every decomposed example, not just
  H2's `inž.`.
- decisions.md: correct the KNOWN LIMITS field for `John й Smith`
  (given, not family), the composed_len gating, and the q́ limit.
- The release-log bullet keeps one example; AGENTS.md's 193-frame count
  is dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added the bug label Oct 14, 2026
@derek73 derek73 self-assigned this Oct 14, 2026
@codecov

codecov Bot commented Oct 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (f9b2b81) to head (d390685).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #675   +/-   ##
=======================================
  Coverage   99.02%   99.03%           
=======================================
  Files          46       46           
  Lines        4534     4541    +7     
=======================================
+ Hits         4490     4497    +7     
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@derek73
derek73 merged commit 26d5bfb into master Oct 14, 2026
12 checks passed
@derek73
derek73 deleted the fix/issue-665-nfd-initial-conjunction branch October 14, 2026 04:06
@derek73 derek73 added this to the 2.4.2 milestone Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A decomposed (NFD) initial or conjunction reads differently from its composed twin: É. y É. Smith and John й Smith

1 participant