An Adaptive Gradient Method with Energy and Momentum

Abstract

We introduce a novel algorithm for gradient-based optimization of stochastic objective functions. The method may be seen as a variant of SGD with momentum equipped with an adaptive learning rate automatically adjusted by an 'energy' variable. The method is simple to implement, computationally efficient, and well suited for large-scale machine learning problems. The method exhibits unconditional energy stability for any size of the base learning rate. We provide a regret bound on the convergence rate under the online convex optimization framework. We also establish the energy-dependent convergence rate of the algorithm to a stationary point in the stochastic non-convex setting. In addition, a sufficient condition is provided to guarantee a positive lower threshold for the energy variable. Our experiments demonstrate that the algorithm converges fast while generalizing better than or as well as SGD with momentum in training deep neural networks, and compares also favorably to Adam.

arXiv comment29 pages, 6 figures

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Decoupled Weight Decay Regularization
• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Improving Generalization Performance by Switching from Adam to SGD
• Non-convex Finite-Sum Optimization Via SCSG Methods
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent