Flatter, faster: scaling momentum for optimal speedup of SGD

Aditya Cowsik, Tankut Can, Paolo Glorioso

Commonly used optimization algorithms often show a trade-off between good generalization and fast training times. For instance, stochastic gradient descent (SGD) tends to have good…