An Optimistic Acceleration of AMSGrad for Nonconvex Optimization
arXiv:1903.01435
Abstract
We propose a new variant of AMSGrad, a popular adaptive gradient based optimization algorithm widely used for training deep neural networks. Our algorithm adds prior knowledge about the sequence of consecutive mini-batch gradients and leverages its underlying structure making the gradients sequentially predictable. By exploiting the predictability and ideas from optimistic online learning, the proposed algorithm can accelerate the convergence and increase sample efficiency. After establishing a tighter upper bound under some convexity conditions on the regret, we offer a complimentary view of our algorithm which generalizes the offline and stochastic version of nonconvex optimization. In the nonconvex case, we establish a non-asymptotic convergence bound independently of the initialization. We illustrate the practical speedup on several deep learning models via numerical experiments.
References in corpus (12)
- ADADELTA: An Adaptive Learning Rate Method
- Playing Atari with Deep Reinforcement Learning
- On the Convergence of Adam and Beyond
- AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
- Adaptive Bound Optimization for Online Convex Optimization
- Optimization, Learning, and Games with Predictable Sequences
- Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile
- Fast Convergence of Regularized Learning in Games
- On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
- Faster Rates for Convex-Concave Games
- Efficient Full-Matrix Adaptive Regularization
- Several Tunable GMM Kernels