On the insufficiency of existing momentum schemes for Stochastic Optimization

Rahul Kidambi, Praneeth Netrapalli, Prateek Jain +1

Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks a…