5 papers
Thinking into the Future: Latent Lookahead Training for Transformers
Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1
Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time. Although very scalable, this objective forces the model to…
The Potential of CoT for Reasoning: A Closer Look at Trace Dynamics
Gregor Bachmann, Yichen Jiang, Seyed Mohsen Moosavi Dezfooli +1
Chain-of-thought (CoT) prompting is a de-facto standard technique to elicit reasoning-like responses from large language models (LLMs), allowing them to spell out individual steps…
The pitfalls of next-token prediction
Gregor Bachmann, Vaishnavh Nagarajan
Can a mere next-token predictor faithfully model human intelligence? We crystallize this emerging concern and correct popular misconceptions surrounding it, and advocate a simple m…
Tokenisation is NP-Complete
Philip Whittington, Gregor Bachmann, Tiago Pimentel
In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most symbols by either finding a vocabulary d…
Interpolated-MLPs: Controllable Inductive Bias
Sean Wu, Jordan Hong, Keyu Bai +1
Due to their weak inductive bias, Multi-Layer Perceptrons (MLPs) have subpar performance at low-compute levels compared to standard architectures such as convolution-based networks…