4 papers
Compute Optimal Tokenization
Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6
Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this wo…
Fast Byte Latent Transformer
Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5
Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte auto…
Byte Latent Transformer: Patches Scale Better Than Tokens
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez +11
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant imp…
Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling
Margaret Li, Weijia Shi, Artidoro Pagnoni +2
RLHF-aligned LMs have shown unprecedented ability on both benchmarks and long-form text generation, yet they struggle with one foundational task: next-token prediction. As RLHF mod…