An improvement of the convergence proof of the ADAM-Optimizer

Abstract

A common way to train neural networks is the Backpropagation. This algorithm includes a gradient descent method, which needs an adaptive step size. In the area of neural networks, the ADAM-Optimizer is one of the most popular adaptive step size methods. It was invented in \cite{Kingma.2015} by Kingma and Ba. The $5865$ citations in only three years shows additionally the importance of the given paper. We discovered that the given convergence proof of the optimizer contains some mistakes, so that the proof will be wrong. In this paper we give an improvement to the convergence proof of the ADAM-Optimizer.

References in corpus (1)

• An overview of gradient descent optimization algorithms

Cited by in corpus (9)

• On the Convergence Proof of AMSGrad and a New Version
• A multi-task learning for cavitation detection and cavitation intensity recognition of valve acoustic signals
• AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence
• RUL forecasting for wind turbine predictive maintenance based on deep learning
• p2pGNN: A Decentralized Graph Neural Network for Node Classification in Peer-to-Peer Networks
• PDE-Inspired Algorithms for Semi-Supervised Learning on Point Clouds
• On Deep Learning Solutions for Joint Transmitter and Noncoherent Receiver Design in MU-MIMO Systems
• Local Convergence of Adaptive Gradient Descent Optimizers
• Deep Learning-based Stress Determinator for Mouse Psychiatric Analysis using Hippocampus Activity