Communication-efficient SGD: From Local SGD to One-Shot Averaging

Artin Spiridonoff, Alex Olshevsky, Ioannis Ch. Paschalidis

We consider speeding up stochastic gradient descent (SGD) by parallelizing it across multiple workers. We assume the same data set is shared among $N$ workers, who can take SGD ste…