Entropy-Driven Pre-Tokenization for Byte-Pair Encoding

Yifan Hu, Frank Liang, Dachuan Zhao +4

Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream…