Time-Delay Momentum: A Regularization Perspective on the Convergence and Generalization of Stochastic Momentum for Deep Learning
arXiv:1903.00760
Abstract
In this paper we study the problem of convergence and generalization error bound of stochastic momentum for deep learning from the perspective of regularization. To do so, we first interpret momentum as solving an -regularized minimization problem to learn the offsets between arbitrary two successive model parameters. We call this {\em time-delay momentum} because the model parameter is updated after a few iterations towards finding the minimizer. We then propose our learning algorithm, \ie stochastic gradient descent (SGD) with time-delay momentum. We show that our algorithm can be interpreted as solving a sequence of strongly convex optimization problems using SGD. We prove that under mild conditions our algorithm can converge to a stationary point with rate of and generalization error bound of with probability at least , where are the numbers of model updates and training samples, respectively. We demonstrate the empirical superiority of our algorithm in deep learning in comparison with the state-of-the-art deep learning solvers.
has errors
References in corpus (9)
- Decoupled Weight Decay Regularization
- ADADELTA: An Adaptive Learning Rate Method
- On the Convergence of Adam and Beyond
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
- Data-Dependent Stability of Stochastic Gradient Descent
- Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
- Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
- Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates