Convergence of the Iterates for Momentum and RMSProp for Local Smooth Functions: Adaptation is the Key

Bilel Bensaid, Gaël Poëtte, Rodolphe Turpault

Both accelerated and adaptive gradient methods are among state of the art algorithms to train neural networks. The tuning of hyperparameters is needed to make them work efficiently…