Scaling Distributed Training with Adaptive Summation

Saeed Maleki, Madan Musuvathi, Todd Mytkowicz +5

Stochastic gradient descent (SGD) is an inherently sequential training algorithm--computing the gradient at batch $i$ depends on the model parameters learned from batch $i-1$. Prio…