Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalization

Yunwen Lei, Tao Sun, Mingrui Liu

The increasing scale of data propels the popularity of leveraging parallelism to speed up the optimization. Minibatch stochastic gradient descent (minibatch SGD) and local SGD are…