Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training

Kwangmin Yu, Thomas Flynn, Shinjae Yoo +1

Stochastic Gradient Descent (SGD) is the most popular algorithm for training deep neural networks (DNNs). As larger networks and datasets cause longer training times, training on d…