Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD

Abstract

Large scale machine learning is increasingly relying on distributed optimization, whereby several machines contribute to the training process of a statistical model. In this work we study the performance of asynchronous, distributed settings, when applying sparsification, a technique used to reduce communication overheads. In particular, for the first time in an asynchronous, non-convex setting, we theoretically prove that, in presence of staleness, sparsification does not harm SGD performance: the ergodic convergence rate matches the known result of standard SGD, that is $\mathcal{O}\left(1/\sqrt{T}\right)$. We also carry out an empirical study to complement our theory, and confirm that the effects of sparsification on the convergence rate are negligible, when compared to 'vanilla' SGD, even in the challenging scenario of an asynchronous, distributed system.

References in corpus (18)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Revisiting Distributed Synchronous SGD
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Sparsified SGD with Memory
• Don't Use Large Mini-Batches, Use Local SGD
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• Essentially No Barriers in Neural Network Energy Landscape
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Parallel SGD: When does averaging help?
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Asynchronous Byzantine Machine Learning (the case of SGD)
• Toward Understanding the Impact of Staleness in Distributed Machine Learning

Cited by in corpus (1)

• Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates