3 citations · 7 across the 12 of their papers we have counts for
15 papers
In-context classification: Calibration and noise in linear attention with softmax readout
Arman Rysmakhanov, Mary Letey, Yue M. Lu +2
We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its…
What can linear attention learn from nonlinear teachers in-context?
Mary Letey, Arman Rysmakhanov, Yue M. Lu +2
Linear attention is a tractable model for understanding the mechanisms governing in-context learning in transformers. For linear regression tasks, recent asymptotic analyses have c…
Broken scale symmetries in undercomplete linear autoencoders
Farhad Pashakhanloo, Jacob A. Zavatone-Veth
Neural network loss landscapes have many symmetries, which are preserved by gradient flow but broken by finite-stepsize stochastic gradient descent (SGD). A canonical example of su…
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Mary Letey, Yue M. Lu, Cengiz Pehlevan +1
Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability…
Stationary covariance spectra of discrete-time non-normal random recurrent dynamics
Jacob A. Zavatone-Veth
Principal component analysis is widely used to characterize structure in the dynamics of recurrent neural networks. For stationary noise-driven dynamics, the distribution of varian…
Stimulus symmetries can confound representational similarity analyses
Farhad Pashakhanloo, Jacob A. Zavatone-Veth
What can representational similarity matrices (RSMs) tell us about a neural code? As the popularity of these summary statistics grows, so too does the need for a more complete char…
A Random Matrix Theory Perspective on the Consistency of Diffusion Models
Binxu Wang, Jacob Zavatone-Veth, Cengiz Pehlevan
Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a…
Pretrain-Test Task Alignment Governs Generalization in In-Context Learning
Mary I. Letey, Jacob A. Zavatone-Veth, Yue M. Lu +1
In-context learning (ICL) is a central capability of Transformer models, but the structures in data that enable its emergence and govern its robustness remain poorly understood. In…
A note on the dynamics of extended-context disordered kinetic spin models
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Inspired by striking advances in language modeling, there has recently been much interest in developing autogressive sequence models that are amenable to analytical study. In this…
Summary statistics of learning link changing neural representations to behavior
Jacob A. Zavatone-Veth, Blake Bordelon, Cengiz Pehlevan
How can we make sense of large-scale recordings of neural activity across learning? Theories of neural network learning with their origins in statistical physics offer a potential…
Dynamics of learning to integrate in linear recurrent neural networks
Blake Bordelon, Jordan Cotler, Cengiz Pehlevan +1
Learning recurrent connectivity that supports memory over long intrinsic timescales is a basic problem in the theory of dynamical computation. While continuous attractor and integr…
Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models
Alexander Atanasov, Blake Bordelon, Jacob A. Zavatone-Veth +2
We derive a novel deterministic equivalence for the two-point function of a random matrix resolvent. Using this result, we give a unified derivation of the performance of a wide va…
Risk and cross validation in ridge regression with correlated samples
Alexander Atanasov, Jacob A. Zavatone-Veth, Cengiz Pehlevan
Recent years have seen substantial advances in our understanding of high-dimensional ridge regression, but existing theories assume that training examples are independent. By lever…
Nadaraya-Watson kernel smoothing as a random energy model
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Precise asymptotics have revealed many surprises in high-dimensional regression. These advances, however, have not extended to perhaps the simplest estimator: direct Nadaraya-Watso…
Spectral regularization for adversarially-robust representation learning
Sheng Yang, Jacob A. Zavatone-Veth, Cengiz Pehlevan
The vulnerability of neural network classifiers to adversarial attacks is a major obstacle to their deployment in safety-critical applications. Regularization of network parameters…