SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling

Dong Liu, Yanxuan Yu

Tokenization plays a critical role in language modeling, yet existing approaches such as Byte-Pair Encoding (BPE) or WordPiece operate purely on frequency statistics, ignoring the…