Dual Averaging is Surprisingly Effective for Deep Learning Optimization
arXiv:2010.10502
Abstract
First-order stochastic optimization methods are currently the most widely used class of methods for training deep neural networks. However, the choice of the optimizer has become an ad-hoc rule that can significantly affect the performance. For instance, SGD with momentum (SGD+M) is typically used in computer vision (CV) and Adam is used for training transformer models for Natural Language Processing (NLP). Using the wrong method can lead to significant performance degradation. Inspired by the dual averaging algorithm, we propose Modernized Dual Averaging (MDA), an optimizer that is able to perform as well as SGD+M in CV and as Adam in NLP. Our method is not adaptive and is significantly simpler than Adam. We show that MDA induces a decaying uncentered -regularization compared to vanilla SGD+M and hypothesize that this may explain why it works on NLP problems where SGD+M fails.
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
- Pointer Sentinel Mixture Models
- On Empirical Comparisons of Optimizers for Deep Learning
- AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
- Learning Deep Transformer Models for Machine Translation
- Optimal Black-Box Reductions Between Optimization Objectives
- How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGD
- On the Convergence Rate of Stochastic Mirror Descent for Nonsmooth Nonconvex Optimization
- Gossip Dual Averaging for Decentralized Optimization of Pairwise Functions
- Deep Frank-Wolfe For Neural Network Optimization
- Asymptotic Optimality in Stochastic Optimization
- Offset Sampling Improves Deep Learning based Accelerated MRI Reconstructions by Exploiting Symmetry