SAdam: A Variant of Adam for Strongly Convex Functions

Abstract

The Adam algorithm has become extremely popular for large-scale machine learning. Under convexity condition, it has been proved to enjoy a data-dependant $O(\sqrt{T})$ regret bound where $T$ is the time horizon. However, whether strong convexity can be utilized to further improve the performance remains an open problem. In this paper, we give an affirmative answer by developing a variant of Adam (referred to as SAdam) which achieves a data-dependant $O(\log T)$ regret bound for strongly convex functions. The essential idea is to maintain a faster decaying yet under controlled step size for exploiting strong convexity. In addition, under a special configuration of hyperparameters, our SAdam reduces to SC-RMSprop, a recently proposed variant of RMSprop for strongly convex functions, for which we provide the first data-dependent logarithmic regret bound. Empirical results on optimizing strongly convex functions and training deep networks demonstrate the effectiveness of our method.

arXiv comment19 pages, 9 figures

References in corpus (8)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• DRAW: A Recurrent Neural Network For Image Generation
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration

Cited by in corpus (1)

• Adaptive Gradient Methods Can Be Provably Faster than SGD after Finite Epochs