Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

Milan Miletić, Julie Kallini, Ekaterina Shutova

Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches f…