(Accelerated) Noise-adaptive Stochastic Heavy-Ball Momentum

Anh Dang, Reza Babanezhad, Sharan Vaswani

Stochastic heavy ball momentum (SHB) is commonly used to train machine learning models, and often provides empirical improvements over stochastic gradient descent. By primarily foc…