LangMAP: A Language-Adaptive Approach to Tokenization

Clara Meister, Suchir Salhan, Andrzej Szablewski +3

Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. However, using such a tokenizer comes at a cost: either a new…