Repository navigation
Ship hi and bn locale packs for Latin-transliterated Indic honorifics #345
Description
#343 and #344 add native-script Indic honorifics to the default vocabulary, on #269's reasoning that native-script entries cannot collide with Latin-script names. The transliterated forms — which is how most Western-sourced datasets store these names — have no route into the default set, because they collide exactly as #269 said:
Shri Narendra Modi -> given='Shri' family='Modi' ❌
Pandit Ravi Shankar -> given='Pandit' family='Shankar' ❌
Sardar Vallabhbhai Patel -> given='Sardar' family='Patel' ❌
Also unrecognized: smt, shrimati, kumari, sushri, shriyut, babu, acharya, janab, begum. (md already ships as a title, so Md Abdul Karim parses correctly today — no change needed.)
A locale pack is the right home: it scopes the collision to the datasets whose owners know it is the right reading.
Prototype — verified working at 2.1.0
HI = Locale(code="hi", lexicon=Lexicon(titles=frozenset(
{"shri","sri","smt","shrimati","pandit","pt","sardar",
"kumari","sushri","shriyut","babu","acharya"})))
BN = Locale(code="bn", lexicon=Lexicon(titles=frozenset(
{"janab","begum","shri","sri","shrimati","adhyapak"})))
p = parser_for(HI, BN)Shri Narendra Modi -> title='Shri' given='Narendra' family='Modi' ✅
Pandit Ravi Shankar -> title='Pandit' given='Ravi' family='Shankar' ✅
Sardar Vallabhbhai Patel -> title='Sardar' given='Vallabhbhai' family='Patel' ✅
Janab Abul Kalam -> title='Janab' given='Abul' family='Kalam' ✅
Begum Khaleda Zia -> title='Begum' given='Khaleda' family='Zia' ✅
Acharya Vinoba Bhave -> title='Acharya' given='Vinoba' family='Bhave' ✅
And the cost, which stays opt-in:
Sri Mulyani Indrawati default -> given='Sri' ✅
pack -> title='Sri' given='Mulyani' ❌
That is the intended trade — the ambiguity is irreducible, so it is resolved by whoever knows whether the dataset is Indic honorifics or Indonesian given names.
Two packs, not one
parser_for composes left-to-right, so parser_for(locales.HI, locales.BN) works and the vocabularies union — verified above. The lists overlap (shri, sri, shrimati) but diverge meaningfully (pandit/sardar vs janab/begum), and separate codes let a Bangladesh-only dataset avoid the Hindi collisions it does not need. tr_az bundles two languages, but it carries no vocabulary at all — the shared thing there is a mechanism, not a word list.
Notes for implementation
This would be the first title-carrying pack: ru and tr_az are policy-only (Lexicon.empty()), and zh carries surnames. Nothing in the design forbids it — Locale.lexicon unions onto the base — but it is a new shape for the pack layer.
The registry contract (tests/v2/test_locales.py:28) makes adding a pack one _REGISTRY entry plus one rotator list, with a meta-test that fails until both exist.
DEVIATES is straightforward for a vocabulary-only pack — any token in the pack's added vocabulary. Over-declaring is the safe direction, per the ru pack's note:
def DEVIATES(name: str) -> bool:
return any(t.strip(".").lower() in _WORDS for t in name.split())Explicit non-goal
These packs will not fix initials-first South Indian names:
P. V. Sindhu -> given='P.' middle='V.' family='Sindhu'
"Sindhu" is the personal name; "P. V." is father/village. That convention varies by person rather than by language, so there is no key — script, locale, or otherwise — to hang it on. It belongs in docs/locales.rst as a stated limit, not in a pack.
Open questions
- Vocabulary scope per pack. The prototype lists are a starting point, not a vetted set; each entry needs the Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269-style check against real Latin given names and surnames.
begum,kumari. Both are also name components (Kumariis a common given name). Inside an opt-in pack the bar is lower than for the default set — but "lower" still is not "none", since the pack's users will have both readings in their data.- Marathi/Nepali. Both write Devanagari and share much of the
hivocabulary. Fold in, or separate codes later?
Activity
fix(vocab,assign): give each title/suffix overlap word its true class…
docs(design): record derek73#296/derek73#325 -- the listing form's po…
docs: review locales.rst and migrate.rst against the tree; state the …
feat(titles): Devanagari honorifics
feat(titles): Bengali honorifics