538 citations · 2.3k across the 54 of their papers we have counts for
4 papers · 2 filters
BiSLS/SPS: Auto-tune Step Sizes for Stable Bi-level Optimization
Chen Fan, Gaspard Choné-Ducasse, Mark Schmidt +1
The popularity of bi-level optimization (BO) in deep learning has spurred a growing interest in studying gradient-based BO algorithms. However, existing algorithms involve two coup…
Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington +1
The success of the Adam optimizer on a wide array of architectures has made it the default in settings where stochastic gradient descent (SGD) performs poorly. However, our theoret…
Fast Convergence of Random Reshuffling under Over-Parameterization and the Polyak-Łojasiewicz Condition
Chen Fan, Christos Thrampoulidis, Mark Schmidt
Modern machine learning models are often over-parameterized and as a result they can interpolate the training data. Under such a scenario, we study the convergence properties of a…
Target-based Surrogates for Stochastic Optimization
Jonathan Wilder Lavington, Sharan Vaswani, Reza Babanezhad +2
We consider minimizing functions for which it is expensive to compute the (possibly stochastic) gradient. Such functions are prevalent in reinforcement learning, imitation learning…