6 papers
Softmax Linear: Transformers may learn to classify in-context by kernel gradient descent
Sara DragutinoviÄ, Andrew M. Saxe, Aaditya K. Singh
The remarkable ability of transformers to learn new concepts solely by reading examples within the input prompt, termed in-context learning (ICL), is a crucial aspect of intelligen…
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
Jin Hwa Lee, Andrew K. Lampinen, Aaditya K. Singh +1
In-context learning (ICL) research often considers learning a function in-context through a uniform sample of input-output pairs. Here, we investigate how presenting a compositiona…
The broader spectrum of in-context learning
Andrew Kyle Lampinen, Stephanie C. Y. Chan, Aaditya K. Singh +1
The ability of language models to learn a task from a few examples in context has generated substantial interest. Here, we provide a perspective that situates this type of supervis…
Training Dynamics of In-Context Learning in Linear Attention
Yedi Zhang, Aaditya K. Singh, Peter E. Latham +1
While attention-based models have demonstrated the remarkable ability of in-context learning (ICL), the theoretical understanding of how these models acquired this ability through…
Strategy Coopetition Explains the Emergence and Transience of In-Context Learning
Aaditya K. Singh, Ted Moskovitz, Sara Dragutinovic +3
In-context learning (ICL) is a powerful ability that emerges in transformer models, enabling them to learn from context without weight updates. Recent work has established emergent…
HARP: A challenging human-annotated math reasoning benchmark
Albert S. Yue, Lovish Madaan, Ted Moskovitz +2
Math reasoning is becoming an ever increasing area of focus as we scale large language models. However, even the previously-toughest evals like MATH are now close to saturated by f…