3 citations · 7 across the 12 of their papers we have counts for
5 papers · 1 filter
In-context classification: Calibration and noise in linear attention with softmax readout
Arman Rysmakhanov, Mary Letey, Yue M. Lu +2
We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its…
What can linear attention learn from nonlinear teachers in-context?
Mary Letey, Arman Rysmakhanov, Yue M. Lu +2
Linear attention is a tractable model for understanding the mechanisms governing in-context learning in transformers. For linear regression tasks, recent asymptotic analyses have c…
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Mary Letey, Yue M. Lu, Cengiz Pehlevan +1
Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability…
Pretrain-Test Task Alignment Governs Generalization in In-Context Learning
Mary I. Letey, Jacob A. Zavatone-Veth, Yue M. Lu +1
In-context learning (ICL) is a central capability of Transformer models, but the structures in data that enable its emergence and govern its robustness remain poorly understood. In…
Risk and cross validation in ridge regression with correlated samples
Alexander Atanasov, Jacob A. Zavatone-Veth, Cengiz Pehlevan
Recent years have seen substantial advances in our understanding of high-dimensional ridge regression, but existing theories assume that training examples are independent. By lever…