Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration

Abstract

Adam is one of the most influential adaptive stochastic algorithms for training deep neural networks, which has been pointed out to be divergent even in the simple convex setting via a few simple counterexamples. Many attempts, such as decreasing an adaptive learning rate, adopting a big batch size, incorporating a temporal decorrelation technique, seeking an analogous surrogate, \textit{etc.}, have been tried to promote Adam-type algorithms to converge. In contrast with existing approaches, we introduce an alternative easy-to-check sufficient condition, which merely depends on the parameters of the base learning rate and combinations of historical second-order moments, to guarantee the global convergence of generic Adam for solving large-scale non-convex stochastic optimization. This observation, coupled with this sufficient condition, gives much deeper interpretations on the divergence of Adam. On the other hand, in practice, mini-Adam and distributed-Adam are widely used without any theoretical guarantee. We further give an analysis on how the batch size or the number of nodes in the distributed system affects the convergence of Adam, which theoretically shows that mini-batch and distributed Adam can be linearly accelerated by using a larger mini-batch size or a larger number of nodes.At last, we apply the generic Adam and mini-batch Adam with the sufficient condition for solving the counterexample and training several neural networks on various real-world datasets. Experimental results are exactly in accord with our theoretical analysis.

arXiv commentAccepted to JMLR(JMLR). arXiv admin note: substantial text overlap with arXiv:1811.09358

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• On the Convergence of Adam and Beyond
• One weird trick for parallelizing convolutional neural networks
• Large Batch Training of Convolutional Networks
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Adaptive Federated Optimization
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates

Cited by in corpus (2)

• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization