198 citations · 339 across the 18 of their papers we have counts for
18 papers
Training Transformers with Enforced Lipschitz Constants
Laker Newhouse, R. Preston Hess, Franz Cesista +3
Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, diverge…
Modular Duality in Deep Learning
Jeremy Bernstein, Laker Newhouse
An old idea in optimization theory says that since the gradient is a dual vector it may not be subtracted from the weights without first being mapped to the primal space where the…
Old Optimizer, New Norm: An Anthology
Jeremy Bernstein, Laker Newhouse
Deep learning optimizers are often motivated through a mix of convex and approximate second-order theory. We select three such methods -- Adam, Shampoo and Prodigy -- and argue tha…
Scalable Optimization in the Modular Norm
Tim Large, Yang Liu, Minyoung Huh +3
To improve performance in contemporary deep learning, one is interested in scaling up the neural network in terms of both the number and the size of the layers. When ramping up the…
Training Neural Networks from Scratch with Parallel Low-Rank Adapters
Minyoung Huh, Brian Cheung, Jeremy Bernstein +2
The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced th…
A Spectral Condition for Feature Learning
Greg Yang, James B. Simon, Jeremy Bernstein
The push to train ever larger neural networks has motivated the study of initialization and training at large network width. A key challenge is to scale training so that a network'…
Devising and Detecting Phishing: Large Language Models vs. Smaller Human Models
Fredrik Heiding, Bruce Schneier, Arun Vishwanath +2
AI programs, built using large language models, make it possible to automatically create phishing emails based on a few data points about a user. They stand in contrast to traditio…
Automatic Gradient Descent: Deep Learning without Hyperparameters
Jeremy Bernstein, Chris Mingard, Kevin Huang +2
The architecture of a deep neural network is defined explicitly in terms of the number of layers, the width of each layer and the general network topology. Existing optimisation fr…
Optimisation & Generalisation in Networks of Neurons
Jeremy Bernstein
The goal of this thesis is to develop the optimisation and generalisation theoretic foundations of learning in artificial neural networks. On optimisation, a new theoretical framew…
Investigating Generalization by Controlling Normalized Margin
Alexander R. Farhang, Jeremy Bernstein, Kushal Tirumala +2
Weight norm and margin participate in learning theory via the normalized margin . Since standard neural net optimizers do not control normalized margin, it is…
Kernel Interpolation as a Bayes Point Machine
Jeremy Bernstein, Alex Farhang, Yisong Yue
A Bayes point machine is a single classifier that approximates the majority decision of an ensemble of classifiers. This paper observes that kernel interpolation is a Bayes point m…
Fine-Grained System Identification of Nonlinear Neural Circuits
Dawna Bagherian, James Gornet, Jeremy Bernstein +3
We study the problem of sparse nonlinear model recovery of high dimensional compositional functions. Our study is motivated by emerging opportunities in neuroscience to recover fin…
Computing the Information Content of Trained Neural Networks
Jeremy Bernstein, Yisong Yue
How much information does a learning algorithm extract from the training data and store in a neural network's weights? Too much, and the network would overfit to the training data.…
Learning by Turning: Neural Architecture Aware Optimisation
Yang Liu, Jeremy Bernstein, Markus Meister +1
Descent methods for deep networks are notoriously capricious: they require careful tuning of step size, momentum and weight decay, and which method will work best on a new benchmar…
Learning compositional functions via multiplicative weight updates
Jeremy Bernstein, Jiawei Zhao, Markus Meister +3
Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems l…
On the distance between two neural networks and the stability of learning
Jeremy Bernstein, Arash Vahdat, Yisong Yue +1
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relati…
Stochastic Activation Pruning for Robust Adversarial Defense
Guneet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton +4
Neural networks are known to be vulnerable to adversarial examples. Carefully chosen perturbations to real images, while imperceptible to humans, induce misclassification and threa…
signSGD: Compressed Optimisation for Non-Convex Problems
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli +1
Training large neural networks requires distributing learning across multiple workers, where the cost of communicating gradients can be a significant bottleneck. signSGD alleviates…