AdaSGD: Bridging the gap between SGD and Adam
arXiv:2006.16541
Abstract
In the context of stochastic gradient descent(SGD) and adaptive moment estimation (Adam),researchers have recently proposed optimization techniques that transition from Adam to SGD with the goal of improving both convergence and generalization performance. However, precisely how each approach trades off early progress and generalization is not well understood; thus, it is unclear when or even if, one should transition from one approach to the other. In this work, by first studying the convex setting, we identify potential contributors to observed differences in performance between SGD and Adam. In particular,we provide theoretical insights for when and why Adam outperforms SGD and vice versa. We ad-dress the performance gap by adapting a single global learning rate for SGD, which we refer to as AdaSGD. We justify this proposed approach with empirical analyses in non-convex settings. On several datasets that span three different domains,we demonstrate how AdaSGD combines the benefits of both SGD and Adam, eliminating the need for approaches that transition from Adam to SGD.
References in corpus (14)
- Decoupled Weight Decay Regularization
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- An overview of gradient descent optimization algorithms
- On the Convergence of Adam and Beyond
- AutoAugment: Learning Augmentation Policies from Data
- Pointer Sentinel Mixture Models
- Improving Generalization Performance by Switching from Adam to SGD
- Shake-Shake regularization
- On Empirical Comparisons of Optimizers for Deep Learning
- Gradient Descent Happens in a Tiny Subspace
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
- On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
- AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods