Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods

Depen Morwani, Alexandru Meterez, Pranav Nair +1

Stochastic momentum methods such as heavy ball (HB), Nesterov momentum, and variants of Accelerated SGD (ASGD) [Kidambi et al., 2018] are widely used in modern training, but their…