198 citations · 339 across the 18 of their papers we have counts for
4 papers · 1 filter
Modular Duality in Deep Learning
Jeremy Bernstein, Laker Newhouse
An old idea in optimization theory says that since the gradient is a dual vector it may not be subtracted from the weights without first being mapped to the primal space where the…
Old Optimizer, New Norm: An Anthology
Jeremy Bernstein, Laker Newhouse
Deep learning optimizers are often motivated through a mix of convex and approximate second-order theory. We select three such methods -- Adam, Shampoo and Prodigy -- and argue tha…
Scalable Optimization in the Modular Norm
Tim Large, Yang Liu, Minyoung Huh +3
To improve performance in contemporary deep learning, one is interested in scaling up the neural network in terms of both the number and the size of the layers. When ramping up the…
Training Neural Networks from Scratch with Parallel Low-Rank Adapters
Minyoung Huh, Brian Cheung, Jeremy Bernstein +2
The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced th…