198 citations · 339 across the 18 of their papers we have counts for
13 papers · 1 filter
Training Transformers with Enforced Lipschitz Constants
Laker Newhouse, R. Preston Hess, Franz Cesista +3
Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, diverge…
Modular Duality in Deep Learning
Jeremy Bernstein, Laker Newhouse
An old idea in optimization theory says that since the gradient is a dual vector it may not be subtracted from the weights without first being mapped to the primal space where the…
Old Optimizer, New Norm: An Anthology
Jeremy Bernstein, Laker Newhouse
Deep learning optimizers are often motivated through a mix of convex and approximate second-order theory. We select three such methods -- Adam, Shampoo and Prodigy -- and argue tha…
Scalable Optimization in the Modular Norm
Tim Large, Yang Liu, Minyoung Huh +3
To improve performance in contemporary deep learning, one is interested in scaling up the neural network in terms of both the number and the size of the layers. When ramping up the…
Training Neural Networks from Scratch with Parallel Low-Rank Adapters
Minyoung Huh, Brian Cheung, Jeremy Bernstein +2
The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced th…
A Spectral Condition for Feature Learning
Greg Yang, James B. Simon, Jeremy Bernstein
The push to train ever larger neural networks has motivated the study of initialization and training at large network width. A key challenge is to scale training so that a network'…
Automatic Gradient Descent: Deep Learning without Hyperparameters
Jeremy Bernstein, Chris Mingard, Kevin Huang +2
The architecture of a deep neural network is defined explicitly in terms of the number of layers, the width of each layer and the general network topology. Existing optimisation fr…
Investigating Generalization by Controlling Normalized Margin
Alexander R. Farhang, Jeremy Bernstein, Kushal Tirumala +2
Weight norm and margin participate in learning theory via the normalized margin . Since standard neural net optimizers do not control normalized margin, it is…
Kernel Interpolation as a Bayes Point Machine
Jeremy Bernstein, Alex Farhang, Yisong Yue
A Bayes point machine is a single classifier that approximates the majority decision of an ensemble of classifiers. This paper observes that kernel interpolation is a Bayes point m…
Computing the Information Content of Trained Neural Networks
Jeremy Bernstein, Yisong Yue
How much information does a learning algorithm extract from the training data and store in a neural network's weights? Too much, and the network would overfit to the training data.…
On the distance between two neural networks and the stability of learning
Jeremy Bernstein, Arash Vahdat, Yisong Yue +1
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relati…
Stochastic Activation Pruning for Robust Adversarial Defense
Guneet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton +4
Neural networks are known to be vulnerable to adversarial examples. Carefully chosen perturbations to real images, while imperceptible to humans, induce misclassification and threa…
signSGD: Compressed Optimisation for Non-Convex Problems
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli +1
Training large neural networks requires distributing learning across multiple workers, where the cost of communicating gradients can be a significant bottleneck. signSGD alleviates…