Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models

Iaroslav Chelombitko, Egor Safronov, Aleksey Komissarov

In the development of Large Language Models (LLMs), considerable attention has been given to the quality of training datasets. However, the role of tokenizers in the LLM training p…