Distilling Token-Trained Models into Byte-Level Models

Zishuo Bao, Jiaqi Leng, Junxiong Wang +2

Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization. However, existing BLMs typically require training from scratch on…