Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

Jie Wang, Shiwei Luo, Qi Zhang +1

Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computati…