Quasi-hyperbolic momentum and Adam for deep learning

Jerry Ma, Denis Yarats

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple al…