Local Convergence of Adaptive Gradient Descent Optimizers
arXiv:2102.09804
Abstract
Adaptive Moment Estimation (ADAM) is a very popular training algorithm for deep neural networks and belongs to the family of adaptive gradient descent optimizers. However to the best of the authors knowledge no complete convergence analysis exists for ADAM. The contribution of this paper is a method for the local convergence analysis in batch mode for a deterministic fixed training set, which gives necessary conditions for the hyperparameters of the ADAM algorithm. Due to the local nature of the arguments the objective function can be non-convex but must be at least twice continuously differentiable. Then we apply this procedure to other adaptive gradient descent algorithms and show for most of them local convergence with hyperparameter bounds.
References in corpus (10)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- ADADELTA: An Adaptive Learning Rate Method
- On the Convergence of Adam and Beyond
- Improving Generalization Performance by Switching from Adam to SGD
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- An improvement of the convergence proof of the ADAM-Optimizer
- Lyapunov Theory for Discrete Time Systems
- A general system of differential equations to model first order adaptive algorithms
- Non-Convergence and Limit Cycles in the Adam optimizer