5 papers · 1 filter
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4
Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in ter…
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Dynamic Multi-Byte Prediction With Hierarchical Language Models
Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant +2
Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byt…
Compute Optimal Tokenization
Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6
Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this wo…
Fast Byte Latent Transformer
Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5
Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte auto…