Counting and Min-Cost Encoding for Tokenization in Large Language Models

Shuming Shi, Xiang Zhang, Hao Yu +7

Mainstream large language models rely on a tokenizer to encode text into a token sequence. Different tokenizers may yield token sequences of substantially different lengths for the…