Length-MAX Tokenizer for Language Models

Dong Dong, Weijie Su

We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and…