Adaptive Learning Rate and Momentum for Training Deep Neural Networks
arXiv:2106.11548
Abstract
Recent progress on deep learning relies heavily on the quality and efficiency of training algorithms. In this paper, we develop a fast training method motivated by the nonlinear Conjugate Gradient (CG) framework. We propose the Conjugate Gradient with Quadratic line-search (CGQ) method. On the one hand, a quadratic line-search determines the step size according to current loss landscape. On the other hand, the momentum factor is dynamically updated in computing the conjugate gradient parameter (like Polak-Ribiere). Theoretical results to ensure the convergence of our method in strong convex settings is developed. And experiments in image classification datasets show that our method yields faster convergence than other local solvers and has better generalization capability (test set accuracy). One major advantage of the paper method is that tedious hand tuning of hyperparameters like the learning rate and momentum is avoided.
accepted to ECML PKDD 2021
References in corpus (4)
- On the Convergence of Adam and Beyond
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- Conjugate-gradient-based Adam for stochastic optimization and its application to deep learning
- On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent