Byte BPE Tokenization as an Inverse string Homomorphism

Saibo Geng, Sankalp Gambhir, Chris Wendler +1

Tokenization is an important preprocessing step in the training and inference of large language models (LLMs). While there has been extensive research on the expressive power of th…