Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization

Tao Sun, Huaming Ling, Zuoqiang Shi +2

Heavy ball momentum is crucial in accelerating (stochastic) gradient-based optimization algorithms for machine learning. Existing heavy ball momentum is usually weighted by a unifo…