111 citations · 403 across the 22 of their papers we have counts for
4 papers · 1 filter
Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
Alexandre Galashov, Matt Jones, Rosemary Ke +3
Within the landscape of inference-time scaling methods for foundation models, a width-based approach to scaling -- which involves the insertion of <pause> tokens in the input strea…
The pitfalls of next-token prediction
Gregor Bachmann, Vaishnavh Nagarajan
Can a mere next-token predictor faithfully model human intelligence? We crystallize this emerging concern and correct popular misconceptions surrounding it, and advocate a simple m…
The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning
Tian Jin, Nolan Clement, Xin Dong +4
How does scaling the number of parameters in large language models (LLMs) affect their core capabilities? We study two natural scaling techniques -- weight pruning and simply train…
Think before you speak: Training Language Models With Pause Tokens
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat +3
Language models generate responses by producing a series of tokens in immediate succession: the token is an outcome of manipulating hidden vectors per layer, one v…