Sparse Communication for Training Deep Networks

Negar Foroutan Eghlidi, Martin Jaggi

Synchronous stochastic gradient descent (SGD) is the most common method used for distributed training of deep learning models. In this algorithm, each worker shares its local gradi…