Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration

Abstract

RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the running time. Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov's Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter ${\beta }_1$. We show that at very high values of the momentum parameter (${\beta }_1=0.99$) ADAM outperforms a carefully tuned NAG on most of our experiments, in terms of getting lower training and test losses. On the other hand, NAG can sometimes do better when ADAM's ${\beta }_1$ is set to the most commonly used value: ${\beta }_1=0.9$, indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance. We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms, and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates.

Presented on 14th July 2018 at the ICML Workshop on Modern Trends in Nonconvex Optimization for Machine Learning. In this version, we have made changes to the setup of our Theorem 3.1, and added additional experimental results

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• On the Convergence of Adam and Beyond
• DRAW: A Recurrent Neural Network For Image Generation
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Improving Generalization Performance by Switching from Adam to SGD
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Stop Wasting My Gradients: Practical SVRG
• Training Deep AutoEncoders for Collaborative Filtering
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Why Regularized Auto-Encoders learn Sparse Representation?

Cited by in corpus (19)

• A Comparison of Optimization Algorithms for Deep Learning
• On Empirical Comparisons of Optimizers for Deep Learning
• Optimization for deep learning: theory and algorithms
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Wave Function Engineering for Spectrally-Uncorrelated Biphotons in the Telecommunication Band based on a Machine-Learning Framework
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• Gradient descent with momentum --- to accelerate or to super-accelerate?
• CNN aided Weighted Interpolation for Channel Estimation in Vehicular Communications
• Time-Delay Momentum: A Regularization Perspective on the Convergence and Generalization of Stochastic Momentum for Deep Learning
• SAdam: A Variant of Adam for Strongly Convex Functions
• Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
• Quantized Adam with Error Feedback
• A Genetic Algorithm based Kernel-size Selection Approach for a Multi-column Convolutional Neural Network
• Rapidly Adapting Moment Estimation
• A Skip-connected Multi-column Network for Isolated Handwritten Bangla Character and Digit recognition
• Nonlinear Matrix Approximation with Radial Basis Function Components