The Efficiency Gap in Byte Modeling

Celine Lee, Jing Nathan Yan, Chen Liang +9

Modern language models have historically relied on two dominant design choices: subword tokenization and autoregressive (AR) ordering. These design decisions bake in priors that di…