4 citations · 17 across the 14 of their papers we have counts for
14 papers
The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations
Enric Boix-Adsera, Neil Mallinar, James B. Simon +1
It is a central challenge in deep learning to understand how neural networks learn representations. A leading approach is the Neural Feature Ansatz (NFA) (Radhakrishnan et al. 2024…
Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape
Ioannis Bantzis, James B. Simon, Arthur Jacot
When a deep ReLU network is initialized with small weights, gradient descent (GD) is at first dominated by the saddle at the origin in parameter space. We study the so-called escap…
The Optimization Landscape of SGD Across the Feature Learning Strength
Alexander Atanasov, Alexandru Meterez, James B. Simon +1
We consider neural networks (NNs) where the final layer is down-scaled by a fixed hyperparameter . Recent work has identified as controlling the strength of feature learning…
A Spectral Condition for Feature Learning
Greg Yang, James B. Simon, Jeremy Bernstein
The push to train ever larger neural networks has motivated the study of initialization and training at large network width. A key challenge is to scale training so that a network'…
Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training
Abraham J. Fetterman, Ellie Kitanidis, Joshua Albrecht +6
Hyperparameter tuning of deep learning models can lead to order-of-magnitude performance gains for the same amount of compute. Despite this, systematic tuning is uncommon, particul…
An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression
Lijia Zhou, James B. Simon, Gal Vardi +1
We study the cost of overfitting in noisy kernel ridge regression (KRR), which we define as the ratio between the test error of the interpolating ridgeless model and the test error…
On the Stepwise Nature of Self-Supervised Learning
James B. Simon, Maksis Knutins, Liu Ziyin +3
We present a simple picture of the training process of joint embedding self-supervised learning methods. We find that these methods learn their high-dimensional embeddings one dime…
Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds
Joshua Albrecht, Abraham J. Fetterman, Bryden Fogelman +8
Despite impressive successes, deep reinforcement learning (RL) systems still fall short of human performance on generalization to new tasks and environments that differ from their…
On Kernel Regression with Data-Dependent Kernels
James B. Simon
The primary hyperparameter in kernel regression (KR) is the choice of kernel. In most theoretical studies of KR, one assumes the kernel is fixed before seeing the training data. Un…
Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
Neil Mallinar, James B. Simon, Amirhesam Abedsoltan +3
The practical success of overparameterized neural networks has motivated the recent scientific study of interpolating methods, which perfectly fit their training data. Certain inte…
The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks
James B. Simon, Madeline Dickens, Dhruva Karkada +1
We derive simple closed-form estimates for the test risk and other generalization metrics of kernel ridge regression (KRR). Relative to prior work, our derivations are greatly simp…
SGD with a Constant Large Learning Rate Can Converge to Local Maxima
Liu Ziyin, Botao Li, James B. Simon +1
Previous works on stochastic gradient descent (SGD) often focus on its success. In this work, we construct worst-case optimization problems illustrating that, when not in the regim…
Reverse Engineering the Neural Tangent Kernel
James B. Simon, Sajant Anand, Michael R. DeWeese
The development of methods to guide the design of neural networks is an important open challenge for deep learning theory. As a paradigm for principled neural architecture design,…
Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses
Charles G. Frye, James Simon, Neha S. Wadia +3
Despite the fact that the loss functions of deep neural networks are highly non-convex, gradient-based optimization algorithms converge to approximately the same performance from m…