To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

Franck Signe, Hippolyte Pilchen, François Yvon +1

Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, t…