Type-Driven Tokenization for Brahmic Scripts

Sai Hemanth Kapila, Rakshika Bagavathy

Standard tokenizers used in large language models produce malformed text when applied to Brahmic scripts. They are a family of abugidas, writing systems whose consonants carry an i…