106 citations · 197 across the 33 of their papers we have counts for
17 papers · 1 filter
There Will Be a Scientific Theory of Deep Learning
Jamie Simon, Daniel Kunin, Alexander Atanasov +11
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the train…
Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
Blake Bordelon, Mary I. Letey, Cengiz Pehlevan
We study in-context learning (ICL) of linear regression in a deep linear self-attention model, characterizing how performance depends on various computational and statistical resou…
How Feature Learning Can Improve Neural Scaling Laws
Blake Bordelon, Alexander Atanasov, Cengiz Pehlevan
We develop a solvable model of neural scaling laws beyond the kernel limit. Theoretical analysis of this model shows how performance scales with model size, training time, and the…
Infinite Limits of Multi-head Transformer Dynamics
Blake Bordelon, Hamza Tahir Chaudhry, Cengiz Pehlevan
In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime. We identify the set of parameterizations that admit w…
A Dynamical Model of Neural Scaling Laws
Blake Bordelon, Alexander Atanasov, Cengiz Pehlevan
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is k…
Grokking as the Transition from Lazy to Rich Training Dynamics
Tanishq Kumar, Blake Bordelon, Samuel J. Gershman +1
We propose that the grokking phenomenon, where the train loss of a neural network decreases much earlier than its test loss, can arise due to a neural network transitioning from la…
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
Blake Bordelon, Lorenzo Noci, Mufan Bill Li +2
The cost of hyperparameter tuning in deep learning has been rising with model sizes, prompting practitioners to find new tuning methods using a proxy of smaller networks. One such…
Loss Dynamics of Temporal Difference Reinforcement Learning
Blake Bordelon, Paul Masset, Henry Kuo +1
Reinforcement learning has been successful across several applications in which agents have to learn to act in environments with sparse feedback. However, despite this empirical su…
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
Blake Bordelon, Cengiz Pehlevan
We analyze the dynamics of finite width effects in wide but finite feature learning neural networks. Starting from a dynamical mean field theory description of infinite width deep…
The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan +1
For small training set sizes , the generalization error of wide neural networks is well-approximated by the error of an infinite width neural network (NN), either in the kernel…
The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks
Blake Bordelon, Cengiz Pehlevan
It is unclear how changing the learning rule of a deep neural network alters its learning dynamics and representations. To gain insight into the relationship between learned featur…
Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks
Blake Bordelon, Cengiz Pehlevan
We analyze feature learning in infinite-width neural networks trained with gradient flow through a self-consistent dynamical field theory. We construct a collection of deterministi…
Neural Networks as Kernel Learners: The Silent Alignment Effect
Alexander Atanasov, Blake Bordelon, Cengiz Pehlevan
Neural networks in the lazy training regime converge to kernel machines. Can neural networks in the rich feature learning regime learn a kernel machine with a data-dependent kernel…
Learning Curves for SGD on Structured Features
Blake Bordelon, Cengiz Pehlevan
The generalization performance of a machine learning algorithm such as a neural network depends in a non-trivial way on the structure of the data distribution. To analyze the influ…
Out-of-Distribution Generalization in Kernel Regression
Abdulkadir Canatar, Blake Bordelon, Cengiz Pehlevan
In real word applications, data generating process for training a machine learning model often differs from what the model encounters in the test stage. Understanding how and wheth…
A Theory of Neural Tangent Kernel Alignment and Its Influence on Training
Haozhe Shan, Blake Bordelon
The training dynamics and generalization properties of neural networks (NN) can be precisely characterized in function space via the neural tangent kernel (NTK). Structural changes…
Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
Abdulkadir Canatar, Blake Bordelon, Cengiz Pehlevan
Generalization beyond a training dataset is a main goal of machine learning, but theoretical understanding of generalization remains an open problem for many models. The need for a…