AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3

Mark Kashirskiy, Artiom Lipinski, Ilya Makarov

Tokenization is a critical preprocessing step for large language models (LLMs), directly impacting training efficiency and downstream performance. General-purpose tokenizers traine…