24 citations · 115 across the 11 of their papers we have counts for
11 papers
Asymptotic theory of in-context learning by linear attention
Yue M. Lu, Mary I. Letey, Jacob A. Zavatone-Veth +2
Transformers have a remarkable ability to learn and execute tasks based on examples provided within the input itself, without explicit prior training. It has been argued that this…
Scaling and renormalization in high-dimensional regression
Alexander Atanasov, Jacob A. Zavatone-Veth, Cengiz Pehlevan
From benign overfitting in overparameterized models to rich power-law scalings in performance, simple ridge regression displays surprising behaviors sometimes thought to be limited…
Long Sequence Hopfield Memory
Hamza Tahir Chaudhry, Jacob A. Zavatone-Veth, Dmitry Krotov +1
Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Comput…
Learning curves for deep structured Gaussian feature models
Jacob A. Zavatone-Veth, Cengiz Pehlevan
In recent years, significant attention in deep learning theory has been devoted to analyzing when models that interpolate their training data can still generalize well to unseen ex…
How does training shape the Riemannian geometry of neural network representations?
Jacob A. Zavatone-Veth, Sheng Yang, Julian A. Rubinfien +1
In machine learning, there is a long history of trying to build neural networks that can learn from fewer example data by baking in strong geometric priors. However, it is not alwa…
Replica method for eigenvalues of real Wishart product matrices
Jacob A. Zavatone-Veth, Cengiz Pehlevan
We show how the replica method can be used to compute the asymptotic eigenvalue spectrum of a real Wishart product matrix. For unstructured factors, this provides a compact, elemen…
Contrasting random and learned features in deep Bayesian linear regression
Jacob A. Zavatone-Veth, William L. Tong, Cengiz Pehlevan
Understanding how feature learning affects generalization is among the foremost goals of modern deep learning theory. Here, we study how the ability to learn representations affect…
On neural network kernels and the storage capacity problem
Jacob A. Zavatone-Veth, Cengiz Pehlevan
In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on ke…
Parallel locomotor control strategies in mice and flies
Ana I. Gonçalves, Jacob A. Zavatone-Veth, Megan R. Carey +1
Our understanding of the neural basis of locomotor behavior can be informed by careful quantification of animal movement. Classical descriptions of legged locomotion have defined d…
Depth induces scale-averaging in overparameterized linear Bayesian neural networks
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Inference in deep Bayesian neural networks is only fully understood in the infinite-width limit, where the posterior flexibility afforded by increased depth washes out and the post…
Activation function dependence of the storage capacity of treelike neural networks
Jacob A. Zavatone-Veth, Cengiz Pehlevan
The expressive power of artificial neural networks crucially depends on the nonlinearity of their activation functions. Though a wide variety of nonlinear activation functions have…