9 citations · 13 across the 11 of their papers we have counts for
12 papers
In-context classification: Calibration and noise in linear attention with softmax readout
Arman Rysmakhanov, Mary Letey, Yue M. Lu +2
We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its…
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Mary Letey, Yue M. Lu, Cengiz Pehlevan +1
Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability…
Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learning
Nicholas Barnfield, Juno Kim, Eshaan Nichani +2
How many key-value associations can a linear memory store? The answer depends not only on the degrees of freedom in the memory matrix, but also on the retrieval c…
Asymptotic Theory of Iterated Empirical Risk Minimization, with Applications to Active Learning
Hugo Cui, Yue M. Lu
We study a class of iterated empirical risk minimization (ERM) procedures in which two successive ERMs are performed on the same dataset, and the predictions of the first estimator…
Pretrain-Test Task Alignment Governs Generalization in In-Context Learning
Mary I. Letey, Jacob A. Zavatone-Veth, Yue M. Lu +1
In-context learning (ICL) is a central capability of Transformer models, but the structures in data that enable its emergence and govern its robustness remain poorly understood. In…
High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification
Nicholas Barnfield, Hugo Cui, Yue M. Lu
When and how can an attention mechanism learn to selectively attend to informative tokens, thereby enabling detection of weak, rare, and sparsely located features? We address these…
A solvable model of learning generative diffusion: theory and insights
Hugo Cui, Cengiz Pehlevan, Yue M. Lu
In this manuscript, we consider the problem of learning a flow or diffusion-based generative model parametrized by a two-layer auto-encoder, trained with online stochastic gradient…
A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities
Yatin Dandi, Luca Pesce, Hugo Cui +3
A key property of neural networks is their capacity of adapting to data during training. Yet, our current mathematical understanding of feature learning and its relationship to gen…
Asymptotic theory of in-context learning by linear attention
Yue M. Lu, Mary I. Letey, Jacob A. Zavatone-Veth +2
Transformers have a remarkable ability to learn and execute tasks based on examples provided within the input itself, without explicit prior training. It has been argued that this…
Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime
Hong Hu, Yue M. Lu, Theodor Misiakiewicz
Recent advances in machine learning have been achieved by using overparametrized models trained until near interpolation of the training data. It was shown, e.g., through the doubl…
Asymptotics of feature learning in two-layer networks after one gradient-step
Hugo Cui, Luca Pesce, Yatin Dandi +4
In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single grad…
Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
Lechao Xiao, Hong Hu, Theodor Misiakiewicz +2
As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improve…