Skip to content

Ship hi and bn locale packs for Latin-transliterated Indic honorifics #345

Open

Description

@derek73

#343 and #344 add native-script Indic honorifics to the default vocabulary, on #269's reasoning that native-script entries cannot collide with Latin-script names. The transliterated forms — which is how most Western-sourced datasets store these names — have no route into the default set, because they collide exactly as #269 said:

Shri Narendra Modi       -> given='Shri'   family='Modi'      ❌
Pandit Ravi Shankar      -> given='Pandit' family='Shankar'   ❌
Sardar Vallabhbhai Patel -> given='Sardar' family='Patel'     ❌

Also unrecognized: smt, shrimati, kumari, sushri, shriyut, babu, acharya, janab, begum. (md already ships as a title, so Md Abdul Karim parses correctly today — no change needed.)

A locale pack is the right home: it scopes the collision to the datasets whose owners know it is the right reading.

Prototype — verified working at 2.1.0

HI = Locale(code="hi", lexicon=Lexicon(titles=frozenset(
    {"shri","sri","smt","shrimati","pandit","pt","sardar",
     "kumari","sushri","shriyut","babu","acharya"})))
BN = Locale(code="bn", lexicon=Lexicon(titles=frozenset(
    {"janab","begum","shri","sri","shrimati","adhyapak"})))

p = parser_for(HI, BN)
Shri Narendra Modi        -> title='Shri'    given='Narendra'  family='Modi'    ✅
Pandit Ravi Shankar       -> title='Pandit'  given='Ravi'      family='Shankar' ✅
Sardar Vallabhbhai Patel  -> title='Sardar'  given='Vallabhbhai' family='Patel' ✅
Janab Abul Kalam          -> title='Janab'   given='Abul'      family='Kalam'   ✅
Begum Khaleda Zia         -> title='Begum'   given='Khaleda'   family='Zia'     ✅
Acharya Vinoba Bhave      -> title='Acharya' given='Vinoba'    family='Bhave'   ✅

And the cost, which stays opt-in:

Sri Mulyani Indrawati   default -> given='Sri'                 ✅
                        pack    -> title='Sri' given='Mulyani' ❌

That is the intended trade — the ambiguity is irreducible, so it is resolved by whoever knows whether the dataset is Indic honorifics or Indonesian given names.

Two packs, not one

parser_for composes left-to-right, so parser_for(locales.HI, locales.BN) works and the vocabularies union — verified above. The lists overlap (shri, sri, shrimati) but diverge meaningfully (pandit/sardar vs janab/begum), and separate codes let a Bangladesh-only dataset avoid the Hindi collisions it does not need. tr_az bundles two languages, but it carries no vocabulary at all — the shared thing there is a mechanism, not a word list.

Notes for implementation

This would be the first title-carrying pack: ru and tr_az are policy-only (Lexicon.empty()), and zh carries surnames. Nothing in the design forbids it — Locale.lexicon unions onto the base — but it is a new shape for the pack layer.

The registry contract (tests/v2/test_locales.py:28) makes adding a pack one _REGISTRY entry plus one rotator list, with a meta-test that fails until both exist.

DEVIATES is straightforward for a vocabulary-only pack — any token in the pack's added vocabulary. Over-declaring is the safe direction, per the ru pack's note:

def DEVIATES(name: str) -> bool:
    return any(t.strip(".").lower() in _WORDS for t in name.split())

Explicit non-goal

These packs will not fix initials-first South Indian names:

P. V. Sindhu -> given='P.' middle='V.' family='Sindhu'

"Sindhu" is the personal name; "P. V." is father/village. That convention varies by person rather than by language, so there is no key — script, locale, or otherwise — to hang it on. It belongs in docs/locales.rst as a stated limit, not in a pack.

Open questions

  • Vocabulary scope per pack. The prototype lists are a starting point, not a vetted set; each entry needs the Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269-style check against real Latin given names and surnames.
  • begum, kumari. Both are also name components (Kumari is a common given name). Inside an opt-in pack the bar is lower than for the default set — but "lower" still is not "none", since the pack's users will have both readings in their data.
  • Marathi/Nepali. Both write Devanagari and share much of the hi vocabulary. Fold in, or separate codes later?

Metadata

Metadata

Assignees

Labels

enhancementvocabularyPer-word vocabulary model and language packs

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions