On the Convergence Proof of AMSGrad and a New Version

Abstract

The adaptive moment estimation algorithm Adam (Kingma and Ba) is a popular optimizer in the training of deep neural networks. However, Reddi et al. have recently shown that the convergence proof of Adam is problematic and proposed a variant of Adam called AMSGrad as a fix. In this paper, we show that the convergence proof of AMSGrad is also problematic. Concretely, the problem in the convergence proof of AMSGrad is in handling the hyper-parameters, treating them as equal while they are not. This is also the neglected issue in the convergence proof of Adam. We provide an explicit counter-example of a simple convex optimization setting to show this neglected issue. Depending on manipulating the hyper-parameters, we present various fixes for this issue. We provide a new convergence proof for AMSGrad as the first fix. We also propose a new version of AMSGrad called AdamX as another fix. Our experiments on the benchmark dataset also support our theoretical results.

arXiv commentUpdate publication information

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• On the Convergence of Adam and Beyond
• Adaptive Bound Optimization for Online Convex Optimization
• An improvement of the convergence proof of the ADAM-Optimizer
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Cited by in corpus (13)

• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Robust quantum gates using smooth pulses and physics-informed neural networks
• MintNet: Building Invertible Neural Networks with Masked Convolutions
• Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
• A Decentralized Adaptive Momentum Method for Solving a Class of Min-Max Optimization Problems
• Order in the Court: Explainable AI Methods Prone to Disagreement
• Precipitation Forecasting via Multi-Scale Deconstructed ConvLSTM
• Recursive Estimation for Sparse Gaussian Process Regression
• Parallel and distributed asynchronous adaptive stochastic gradient methods
• Adversarially Robust Kernel Smoothing
• Expressive equivalence of classical and quantum restricted Boltzmann machines
• On Generalization of Adaptive Methods for Over-parameterized Linear Regression
• Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series